NSX IPSec Throughput in IBM Softlayer

To understand the real throughput capacity of NSX IPSec in Softlayer, I built a quick IPSec performance testing environment.

Below are the network topology of my testing environment:

NSX_IPSec_Performance_Topology

NSX version: 6.2.4
NSX Edge: X-Large (6 vCPUs and 8G Memory), which is the largest size NSX offers. All of Edges in this testing enviroment reside in the same vSphere cluster which include 3 ESXi hosts. Each ESXi host has 64GB DDR4 Memory and 2 processors (2.4GHz Intel Xeon-Haswell (E5-2620-V3-HexCore))
IPerf Client: Redhat 7.1 (2 vCPUs and 4GB Memory)
IPerf Server: Redhat 7.1 (2 vCPUs and 4GB Memory)
IPerf version: IPerf3

2 IPsec tunnels are built as the above diagram. IPSec setting is:

  • Encryption: AES-GCM
  • Diff-Hellman Group: DH5
  • PFS(Perfect forward secrecy): Enabled
  • AESNI: Enabled
I include 3 test cases in my testing:
Test1_Bandwidth_Utilisation
  • Test Case 2: 2 IPerf Clients (172.16.31.0/24) to 2 IPerf Servers (172.16.38.0/24) via 1 IPsec Tunnel. Result: around 1.6-2.3Gbit/s in total
Test2_Bandwidth_Utilisation
Test3_Bandwidth_Utilisation
Please note:
  1. Firewall function on NSX Edge is disabled in all test cases.
  2. TCP traffic is used in all 3 test cases. 10 parallel streams are used to push the performance test to the max on each IPerf Client.
  3. I didn’t see any CPU or Memory contention in all test cases: the CPU utilisation of NSX Edge was  less than 40% and memory utilisation is nearly zero.

CPU_Mem

How to lock down your Softlayer Vyatta

In Softlayer, Vyatta Network Gateway is offered to provide routing, firewall and VPN gateway function. As a network security device, we have to protect Vyatta gateway itself properly.

Softlayer suggest:

“This Vyatta gateway is administered directly by the customer.  The customer has the ability to login directly to the device and make extensive configurations for servicing their network traffic. The customer is responsible for maintaining proper backups of the device’s configuration files.”

So as a customer of Softlayer, it is YOUR responsibility to secure the Vyatta gateway.

Here I will try to give you a few tips to lock down your Vyatta gateway:

  1. Disable un-secure and unused services running on Vyatta gateway. We are lucky: only SSH and HTTPs are enabled by default with Softlayer Vyatta build.
  2. Softlayer Vyatta build allows you to SSH to Vyatta gateway through Internet by default. You have two ways to make it more secure:
  3. Set SSH service only listens on Vyatta private network

set service ssh listen-address private-ip

  1. Apply firewall rules on the Vyatta gateway public interface to only allowed trusted network to access your Vyatta gateway. Note the firewall rules should apply as “local”.
  2. Apply the principle of least privilege by use of Role-based access control (RBAC). Vyatta defines 3 roles (operator, administrator and superuser) by default.
  3. Integrate with your central AAA server if you have one for access control. TACACS+ and Radius are supported by Vyatta gateway.
  4. Configure SNMP and Syslog to monitor the operation of Vyatta gateway.
  5. BY default, Softlayer Vyatta gateway NTP is configured to sync the Vyatta clock with Softlayer NTP server. You can change to sync the clock with your own NTP server if you like. Don’t forget to change the time-zone to reflect your local time!
  6. Control the device access in customer portal to only allow your network administrator has access to Vyatta gateway. Make sure the user name and password of Vyatta is only visible to them.
  7. Follow your password management policy and change your password regularly.

Citrix Netscaler L2 and L3 mode

Citrix NetScaler as an L2 Device

A NetScaler functioning as an L2 device is said to operate in L2 mode. In L2 mode, the NetScaler

forwards packets between network interfaces when all of the following conditions are met:

• The packets are destined to another device’s media access control (MAC) address.

• The destination MAC address is on a different network interface.

• The network interface is a member of the same virtual LAN (VLAN).

By default, all network interfaces are members of a pre-defined VLAN, VLAN 1. Address Resolution

Protocol (ARP) requests and responses are forwarded to all network interfaces that are members of

the same VLAN. To avoid bridging loops, L2 mode must be disabled if another L2 device is working

in parallel with the NetScaler.

Citrix NetScaler as a Packet Forwarding Device

A NetScaler can function as a packet forwarding device, and this mode of operation is called

L3 mode. With L3 mode enabled, the NetScaler forwards any received unicast packets that are

destined for an IP address that it does not have internally configured, if there is a route to the

destination. A NetScaler can also route packets between VLANs.

In both modes of operation, L2 and L3, a NetScaler generally drops packets that are in:

• Multicast frames

• Unknown protocol frames destined for a NetScaler’s

• Spanning Tree protocol (unless BridgeBPDUs is ON)

Citrix Netscaler CloudBridge L3 mode lab

The following lab is going to run through the steps to build a working L3 NetScaler Cloud Bridge tunnel. The lab is built on VMware workstation 9.2.

This solution shows in this lab can be applied to Softlayer Cloud Offering for the secure connectivity from your own environment to Softlayer DC.

 

Lab environment component

NetScaler VPX Platinum Evaluation version 10.1-119.7. (You can download this edition from Citrix.com)

Vyatta Router (Note: No routing at Vyatta for 192.168.108.0/24 or 192.168.175.0/24)

Lab Topology

CloudBridge L3

IP addressing

Please see the above diagram for the IP addressing

Lab Steps

Step 0. Perform initial configuration of Netscaler including NSIP, SNIP and gateway as the above topology

Step 1. Log in the Netscaler GUI management, verify Netscaler works in L3 mode in System-Settings-Configuring mode on both Netscalers

Netscaler l3mode

Step 2. Enable CloudBridge feature under System-Settings-Configuring advances features on both Netscalers

CloudBridge feature

Step 3. Under System-CloudBridge Connector, click Getting Started to open the CloudBridge configuration wizard at Netscaler@DC-A

CloudBridge Wizard

Step 4. In the wizard, select Netscaler icon

CloudBridge Wizard1

Step 5. Type in the remote Netscaler@DC-B NSIP and user/password

CloudBridge Wizard2

Step 6. Configure the Cloud BridgeConnector

CloudBridge Wizard3

After click Continue button, the wizard will complete the configuration for you on both Netscalers.

Step 7. Configure bridge SNIP. Netscaler @DC-A: 172.16.31.1/30 Netscaler @DC-B: 172.16.31.2/30

Netscaler BridgeSNIP

Netscaler BridgeSNIP1

Step 8. Add routing from local DC to remote DC for network in the peering DC

On Netscaler@DC-A

Netsacler Routing1

On Netscaler@DC-B

Netsacler Routing2

Step 9. Verify the CloudBridge Tunnel works well

In GUI, you can see the tunnel status is up as the below:

Netscaler IP Tunnels

Personally, I prefer to perform the status check by CLI.

Netscaler@DC-A

> ping 172.16.31.2

PING 172.16.31.2 (172.16.31.2): 56 data bytes

64 bytes from 172.16.31.2: icmp_seq=0 ttl=255 time=18.184 ms

64 bytes from 172.16.31.2: icmp_seq=1 ttl=255 time=2.586 ms

64 bytes from 172.16.31.2: icmp_seq=2 ttl=255 time=3.075 ms

64 bytes from 172.16.31.2: icmp_seq=3 ttl=255 time=2.590 ms

^C

— 172.16.31.2 ping statistics —

4 packets transmitted, 4 packets received, 0% packet loss

 

round-trip min/avg/max/stddev = 2.586/6.609/18.184/6.686 ms

> show arp

IP               MAC                Iface VLAN  TD     Origin     TTL

—               —                —– —-  —     ——     —

1)      127.0.0.1        00:0c:29:93:a6:c7  LO/1  1     0      PERMANENT  N/A

2)      172.16.31.2      00:0c:29:17:ea:7f  TUN1  1     0      DYNAMIC    1196

3)      192.168.107.20   00:0c:29:93:a6:c7  LO/1  1     0      PERMANENT  N/A

4)      192.168.107.21   00:0c:29:93:a6:c7  LO/1  1     0      PERMANENT  N/A

5)      192.168.107.10   00:0c:29:86:7a:18  0/1   1     0      DYNAMIC    1189

6)      192.168.107.100  00:0c:29:1a:15:a2  0/1   1     0      DYNAMIC    1184

Done

> show ip

Ipaddress        TD    Type             Mode     Arp      Icmp     Vserver  State

———        —    —-             —-     —      —-     ——-  ——

1)      192.168.107.20   0     NetScaler IP     Active   Enabled  Enabled  NA       Enabled

2)      192.168.107.21   0     SNIP             Active   Enabled  Enabled  NA       Enabled

3)      172.16.31.1      0     SNIP             Active   Enabled  Enabled  NA       Enabled

> show iptunnel

1) Domain…….:               0

Name………:  cbbridge1 (TUN1)

Remote…….:  192.168.174.21   Mask……: 255.255.255.255

Local……..:  192.168.107.21   Encap…..:  192.168.107.21

Protocol…..:             GRE   Type……:               C

IPSec Profile Name…….:       cbbridge1

IPSec Tunnel Status……:              UP

 

Done

> show route

Network          Netmask          Gateway/OwnedIP  State   TD     Type

——-          ——-          —————  —–   —     —-

1)      0.0.0.0          0.0.0.0          192.168.107.10   UP      0     STATIC

2)      127.0.0.0        255.0.0.0        127.0.0.1        UP      0     PERMANENT

3)      172.16.31.0      255.255.255.252  172.16.31.1      UP      0     DIRECT

4)      192.168.107.0    255.255.255.0    192.168.107.20   UP      0     DIRECT

Done

> stat ipsec counters

 

Secure tunnel(s) summary

Rate (/s)                Total

Bytes Received                                     0                  176

Bytes Sent                                         0                  352

Packets Received                                   0                    2

Packets Sent                                       0                    4

Done

Netscaler@DC-B

> ping 172.16.31.1

PING 172.16.31.1 (172.16.31.1): 56 data bytes

64 bytes from 172.16.31.1: icmp_seq=0 ttl=255 time=0.485 ms

64 bytes from 172.16.31.1: icmp_seq=1 ttl=255 time=0.559 ms

^C

— 172.16.31.1 ping statistics —

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max/stddev = 0.485/0.522/0.559/0.037 ms

Done

 

> show arp

IP               MAC                Iface VLAN  TD     Origin     TTL

—               —                —– —-  —     ——     —

1)      127.0.0.1        00:0c:29:17:ea:7f  LO/1  1     0      PERMANENT  N/A

2)      172.16.31.1      00:0c:29:93:a6:c7  TUN1  1     0      DYNAMIC    1065

3)      192.168.174.10   00:0c:29:86:7a:22  0/1   1     0      DYNAMIC    1190

4)      192.168.174.20   00:0c:29:17:ea:7f  LO/1  1     0      PERMANENT  N/A

5)      192.168.174.21   00:0c:29:17:ea:7f  LO/1  1     0      PERMANENT  N/A

Done

> show ip

Ipaddress        TD    Type             Mode     Arp      Icmp     Vserver  State

———        —    —-             —-     —      —-     ——-  ——

1)      192.168.174.20   0     NetScaler IP     Active   Enabled  Enabled  NA       Enabled

2)      192.168.174.21   0     SNIP             Active   Enabled  Enabled  NA       Enabled

3)      172.16.31.2      0     SNIP             Active   Enabled  Enabled  NA       Enabled

Done

> show iptunnel

1) Domain…….:               0

Name………:  cbbridge1 (TUN1)

Remote…….:  192.168.107.21   Mask……: 255.255.255.255

Local……..:  192.168.174.21   Encap…..:  192.168.174.21

Protocol…..:             GRE   Type……:               C

IPSec Profile Name…….:       cbbridge1

IPSec Tunnel Status……:              UP

 

Done

> show route

Network          Netmask          Gateway/OwnedIP  State   TD     Type

——-          ——-          —————  —–   —     —-

1)      0.0.0.0          0.0.0.0          192.168.174.10   UP      0     STATIC

2)      127.0.0.0        255.0.0.0        127.0.0.1        UP      0     PERMANENT

3)      172.16.31.0      255.255.255.252  172.16.31.2      UP      0     DIRECT

4)      192.168.174.0    255.255.255.0    192.168.174.20   UP      0     DIRECT

> stat ipsec counters

 

Secure tunnel(s) summary

Rate (/s)                Total

Bytes Received                                     0                  304

Bytes Sent                                         0                  204

Packets Received                                   0                    4

Packets Sent                                       0                    2

Done

Ping Test from DC-A to DC-B

> ping -S 192.168.108.20 192.168.175.20

PING 192.168.175.20 (192.168.175.20) from 192.168.108.20: 56 data bytes

64 bytes from 192.168.175.20: icmp_seq=0 ttl=255 time=9.419 ms

64 bytes from 192.168.175.20: icmp_seq=1 ttl=255 time=2.559 ms

64 bytes from 192.168.175.20: icmp_seq=2 ttl=255 time=3.598 ms

64 bytes from 192.168.175.20: icmp_seq=3 ttl=255 time=2.561 ms

64 bytes from 192.168.175.20: icmp_seq=4 ttl=255 time=2.592 ms

64 bytes from 192.168.175.20: icmp_seq=5 ttl=255 time=3.107 ms

How to Use Citrix NetScaler CloudBridge Connector with the SoftLayer Cloud

Very good doc and video about how to use Citrix Netscaler CloudBridge Connector with Softlayer Cloud

How to Use Citrix NetScaler CloudBridge Connector with the SoftLayer Cloud

Points to Consider when Configuring a CloudBridge Tunnel with a NAT Device

For configuring a CloudBridge tunnel, consider the following points:

-You must not deploy any dynamic NAT device before a CloudBridge tunnel end point. At least one of the CloudBridge appliances must be directly connected or behind a static NAT device.

-Make sure that the IP tunnel entity configured on each CloudBridge appliance specifies the correct IP address of the appliance at the other end point. If one of the CloudBridge appliances is behind a static NAT device, the IP tunnel entity on the peer tunnel end point (the peer CloudBridge appliance) must specify the remote tunnel end-point IP as the IP address of the NAT device, not the IP address of the CloudBridge appliance that is behind the NAT device.

Prerequisites for Configuring a Cloudbridge Tunnel

Configure firewalls to allow UDP and ESP traffic – Configure firewalls, deployed on the network edge of each of the CloudBridge tunnel end points, as follows:

-If no NAT device is deployed before each of the CloudBridge tunnel end points, that is, the public IP addresses of both the tunnel end points are directly accessible to each other, you must configure the firewall to allow the following:

Any UDP packets for port 500

Any ESP (IP protocol number 50) packets

-If a NAT device is deployed before any or each of the CloudBridge tunnel end points, that is, the public IP addresses of at least one tunnel end point is not directly accessible to the other, you must configure the firewall to allow the following:

Any UDP packets for port 500

Any UDP packets for port 4500

Any ESP (IP protocol number 50) packets

Step by Step: how to route a VLAN to Vyatta Gateway using Softlayer Customer Portal

The Vyatta Network Gateway device places a customer configurable routing device in front of specific customer configured VLANs.

The primary benefit of using a Network Gateway is control. A Network Gateway provides the customer expanded control over features and functionality, including:

 

  • Terminate public IPSec VPN tunnels on the Network Gateway device.
  • NAT based configurations where “private-network only” servers are reached via NAT on the Network Gateway.
  • Custom routing designs adjustable on the fly by your administration team.
  • Run extra diagnostic commands such as traffic monitoring on the Network Gateway.
  • Firewall capabilities in addition to other benefits

In Softlayer customer portal, you can specify which vlan you want to route to Vaytta gateway so that CCI and bare metal server in that vlan will use Vyatta as default gateway/next-hop.

Below will show you the steps:

Step 1: Network > Gateway Appliances

vyatta1

Step 2: Click the Gateway Name for the desired Network Gateway to access the Gateway Details screen.

vyatta2

Step 3: Select the desired VLAN from the Associate a VLAN drop down list and Click the Associate button to associate the VLAN

vyatta3

Step 4: Locate the desired VLAN in the Associated VLANs section and Select Route VLAN from the Actions drop down menu

vyatta4

Step 5: Log in the Vyatta to configure the gateway interface for vlan 1942

set interfaces bonding bond0 vif 1942 address ’10.x.x.193/26′

If you have HA Vyatta gateway, configure the VRRP as well

NOTE: the VRRP virtual IP is always the 1st usable IP in your subnet.

SoftLayer Citrix Netscaler VPX

The NetScaler VPX product line has three family editions available supporting the same feature set as the NetScaler appliances:

  • Standard Edition
  • Enterprise Edition
  • Platinum Edition

Softlayer has provided the following options for Netscaler VPX. Standard and Platinum edition are available from license point of viewSoftlayer Citrix VPX

The features are supported by the Platinum edition are:

Citrix VPX license

The features are supported by the Standard edition are:

Citrix Standard version

The following reference about VPX performance can help you to pick up the rigth edition of VPX.

VPX Performance

VPX Performance1

Upgrade Licenses For NetScaler VPX Virtual Appliance

Customers can upgrade their NetScaler virtual appliance from one family edition to another and from one performance range to another by purchasing an upgrade license.

  • There are two types of upgrades for this product:
    Edition Upgrades—Standard to Enterprise, Standard to Platinum and Enterprise to Platinum. Edition upgrades must be within the same bandwidth.
  • Performance (Bandwidth) Upgrades—10Mbps to 200 Mbps, 10Mbps to 1000Mbps and 200Mbps to 1000Mbps. Performance upgrades can only be performed on the same Edition (Standard, Enterprise,or Platinum).