Setup NSX L2VPN on Standalone Edge

With NSX L2VPN, you can extend your VLAN/VXLAN across multiple data centers.  Even in a non-NSX environment, you can achieve this as well by use of standalone edge. In this blog, I will show you how to set up NSX L2VPN between Standalone Edge and NSX edge.

Topology:2018-06-27_162648

As the above, we have 1 NSX edge as L2VPN server and 1 standalone edge which resides in the remote DC which is non-NSX environment. Our target is to stretch two VXLAN backed networks (172.16.136.0/24 and 172.16.137.0/24) to 2 VLAN (VLAN100 and VLAN200) backed networks in remote DC via L2VPN. In addition, we will leverage 4 virtual machines for our L2VPN communication testing.

2 virtual machines in NSX environment:

test1000: 10.172.136.100 gw 172.16.136.1 which is connected to VXLAN10032;

test1002: 10.172.137.100 gw 172.16.137.1 which is connected to VXLAN10033;

2 virtual machines in non-NSX environment:

test1001: 10.172.136.101 gw 172.16.136.1 which is connected to a dVS port-group with access vlan 100;

test1003: 10.172.137.101 gw 172.16.137.1 which is connected to a dVS port-group with access vlan 200;

Step 1: Configure NSX Edge as L2VPN Server

  • Create 2 sub interfaces(sub100: 172.16.136.1/24 and sub200: 172.16.137.1) by two VXLANs under trunk port

L2VPN Server03

Two VXLAN sub-interfaces, please note that 1st sub-interface is mapped to vNic10 and 2nd sub-interface is mapped to vNic11.

L2VPN Server04

Sub-interface sub100: tunnel Id 100/172.16.136.1 (VXLAN 10032)

L2VPN Server05

Sub-interface sub200 tunnel Id 200/172.16.137.1 (VXLAN 10033)

L2VPN Server06

  • L2VPN Server setting as below:
    • Listener IP: 172.16.133.1
    • Listener Port: 443
    • Encryption Algorithm: AES128-GCM-SHA256
    • Site Configuration:
      • name: remote
      • User Id/Password: admin/credential
      • Stretched Interfaces: sub100 and sub200

L2VPN Server01

L2VPN Server02

Step 2: Deploy and Setup L2VPN virtual appliance

Use standard process of deploying a virtual appliance.

  • Start the deploy OVF template wizard

1.2

  • Select the standalone Edge ovf file which is downloaded from vmware.com

1.3

1.4

  • Accept extra configuration options

1.5

  • Select name and folder1.6

1.7

  • Select storage

1.8

  • Setup Networks: here we use one dVS port-group for the standalone trunk interface. We will provide more details around the setting of this port-group later1.9
  • Customize template. We will configure L2VPN client here as well.

The configuration includes multiple parts:

Part1: standalone edge admin credentials:

1.10

Part2: standalone edge network setting:

1.11

Part 3: L2VPN setting, which required to exactly match the L2VPN server configuration which you did in Step1 including cipher suite, L2VPN Server address/service port and L2VPN username/password for authentication

1.12

Part4: L2VPN Sub Interfaces

1.13.1

Part5: other setting, e.g. proxy if your standalone edge need proxy to establish connectivity to L2VPN server.

1.14

  • Accept all setting and submit for the standalone edge deployment.

1.14.1

Once the standalone edge deployment is completed and powered on, you should be able to see the L2VPN tunnel is up either on NSX edge L2VPN server or standalone edge via CLI (show service l2vpn).

On NSX edge L2VPN server:

L2VPN up

On standalone edge:

l2vpn status_client

Step 3: Verification of communication

I simply use PING to verify the communication. My initial test is failed. Yes, you still need to configure port group DPortGroup_ClientTrunk to support L2VPN although L2VPN tunnel is up. You don’t need to do the same for NSX edge as it is completed automatically for you when you configure L2VPN on it.

  • VLAN trunking with VLAN100 and VLAN200

PG_ClientTrunk03

PG_ClientTrunk02

After completing of the above configuration, you will be able to ping all testing virtual machines between each other:

  • test1001 to test1000 (communication within 172.16.136.0/24 via L2VPN)

test01

  • test1003 to test1002 (communication for 172.16.137.0/24 via L2VPN)

test02

  • test1001 to test1003 (communication between 172.16.136.0/24 and 172.16.137.0/24 via L2VPN)

test03

You can check the mac-address and L2VPN mapping relationship via CLI “show service l2vpn bridge”

show_service_l2vpn_bridge

Possibly you noted there is an interface called na1 in the above, which is tunnel interface is created at NSX edge for L2VPN, you can find more details via show interface na1″

interface_na1

On standalone edge L2VPN client end, you will find 2 new vNiCs (vNic_110 and vNic_210) for VLAN 100 and 200 are created as well like vNic10 and vNic11 on the NSX Edge L2VPN server end.

L2VPN client new vNic

In addition, you can find a L2VPN tunnel interface tap0 on standalone edge.

l2vpn client trunk

Wireshark Filter for SSL Traffic

Useful Wireshark filter for analysis of SSL Traffic.

Client Hello:

ssl.handshake.type == 1

Server Hello:

ssl.handshake.type == 2

NewSessionTicket:

ssl.handshake.type == 4

Certificate:

ssl.handshake.type == 11

CertificateRequest

ssl.handshake.type == 13

ServerHelloDone:

ssl.handshake.type == 14

Note: “ServerHellpDone” means full-handshake TLS session.

Cipher Suites:

ssl.handshake.ciphersuite

Please note:

More and more deployment require more secure mechnism e.g.Perfect Forward Secrecy. To provide PFS, cipher suite need to leverage  Elliptic-curve Diffie–Hellman (ECDH) or Ephemeral Diffie-Hellman during the key exchange.  In those cases, we can’t use private key to de-encrypt the traffic.

 

Create XML file in vRealize Orchestrator for NSX Automation

NSX API uses XML format for API communication. To automate NSX in VMware vRealize Orchestror, it is always required to create a XML file with javascript  as vRO workflow supports javascript only.Here i only shows you an example to how to do it.

The target here is to create a security group and add a simple firewall rule in this newly created security group.

Note: this vRO workflow has 2 inputs:
securityGroupName, description
And 2 properties:
nsxManagerRestHost, realtime(equal to sgID in Step1)

Step1: create a security group

var xmlbody = new XML('<securitygroup />');
xmlbody.objectId = " ";
xmlbody.type.typeName = " ";
xmlbody.description = description;
xmlbody.name = securityGroupName;
xmlbody.revision = 0;
xmlbody.objectTypeName = " ";
System.log(xmlbody);
var request = nsxManagerRestHost.createRequest("POST", "/2.0/services/securitygroup/bulk/globalroot-0", xmlbody.toString());
request.contentType = "application/xml";
System.log("Creating a SecurityGroup " + securityGroupName);
System.log("POST Request URL: " + request.fullUrl);
var response = request.execute();
if (response.statusCode == 201) {
	System.debug("Successfully created Security Group " + securityGroupName);
	}
else {
	throw("Failed to SecurityGroup " + securityGroupName);
	}
sgID = response.getAllHeaders().get("Location").split('/').pop();
realtime=sgID

Step2: add a section in DFW and add a firewall rules

//create XML object for DFW source;
var rulesources = new XML('<sources excluded="false" />');
rulesources.source.name = " ";
rulesources.source.value = "10.47.161.23";
rulesources.source.type = "Ipv4Address";
rulesources.source.isValid = 'true';
System.log("Source: "+rulesources);

//create XML object for DFW destination;
var ruledestionations = new XML('<destinations excluded="false" />');
ruledestionations.destination.name = " ";
ruledestionations.destination.value = "10.47.161.24";
ruledestionations.destination.type = "Ipv4Address";
ruledestionations.destination.isValid = 'true';
System.log("Destination: "+ruledestionations);

//create XML object for DFW service
var ruleservices = new XML('<services />');
ruleservices.service.destinationPort = "80";
ruleservices.service.protocol = "6";
ruleservices.service.subProtocol = "6";
ruleservices.service.isValid = 'true';
System.log("Service: "+ruleservices);

//create XML object for the whole rule
var xmlbodyrule = new XML('<rule disabled="false" logged="true" />');
xmlbodyrule.name = "vro created rule";
xmlbodyrule.action = "allow";
xmlbodyrule.notes = " ";
xmlbodyrule.appliedToList.appliedTo.name = securityGroupName;
xmlbodyrule.appliedToList.appliedTo.value = realtime;
xmlbodyrule.appliedToList.appliedTo.type = 'SecurityGroup';
xmlbodyrule.appliedToList.appliedTo.isValid = 'true';
xmlbodyrule.sectionId = " ";
xmlbodyrule.sources = rulesources;
xmlbodyrule.destinations = ruledestionations;
xmlbodyrule.services = ruleservices;

//create XML object for DFW section
var xmlbody = new XML(
<section name ={securityGroupName} />);
//xmlbody.rule = 'disabled="false" logged="true" />';
xmlbody.rule=xmlbodyrule;
System.log("XML file for new rules: "+xmlbody);

var request = nsxManagerRestHost.createRequest("POST", "/4.0/firewall/globalroot-0/config/layer3sections", xmlbody.toString());
request.contentType = "application/xml";
var response = request.execute();
if (response.statusCode == 201) {
	System.debug("Successfully created Security Group Section" + securityGroupName);
	}
else {
	throw("Failed to SecurityGroup Section" + securityGroupName);
	}

Below is the output of XML file for creating a security group:

<securitygroup>
  <objectId></objectId>
  <type>
    <typeName></typeName>
  </type>
  <description>nsx1001test</description>
  <name>nsx1001test</name>
  <revision>0</revision>
  <objectTypeName></objectTypeName>
</securitygroup>

XML file for creating a NSX DFW section and adding a new simple firewall rules:

<section name="nsx1001test">
  <rule disabled="false" logged="true">
    <name>vro created rule</name>
    <action>allow</action>
    <notes></notes>
    <appliedToList>
      <appliedTo>
        <name>nsx1001test</name>
        <value>securitygroup-947</value>
        <type>SecurityGroup</type>
        <isValid>true</isValid>
      </appliedTo>
    </appliedToList>
    <sectionId></sectionId>
    <sources excluded="false">
      <source>
        <name></name>
        <value>10.47.161.23</value>
        <type>Ipv4Address</type>
        <isValid>true</isValid>
      </source>
    </sources>
    <destinations excluded="false">
      <destination>
        <name></name>
        <value>10.47.161.24</value>
        <type>Ipv4Address</type>
        <isValid>true</isValid>
      </destination>
    </destinations>
    <services>
      <service>
        <destinationPort>80</destinationPort>
        <protocol>6</protocol>
        <subProtocol>6</subProtocol>
        <isValid>true</isValid>
      </service>
    </services>
  </rule>
</section>

NSX Load Balancer Qucik Summary

Recently, I was asked a lot of questions around the capability of NSX load balancer by team and customers. So I put a quick summary of NSX load balancer to ease my life.

NSX can perform L4 or L7 load balancing:

  • L4 Load Balancing (packet-based load balancing) : support TCP and UDP load balancing, which is based on Linux Virtual Server.
  • L7 Load Balancing (socket-based load balancing): Support TCP and TCP-based application (e.g. HTTPs_ load balancing, which is based on HAProxy.

Regarding SSL load balancing, it requests L7 load balancing.

3 options for SSL load balancing:

  • SSL Passthrough:
    • NSX load balancer won’t terminate the client session and only pass through the SSL traffic;
    • Session persistence: SSL session id or source IP
  • SSL Offload:
    • client SSL session will be terminated on NSX load balancer and a clear-text (e.g. HTTP) session will be initiated from NSX load balancer to backend server;
    • Session persistence: cookie, SSL session id or source IP
  • SSL end to end:
    • client SSL session will be terminated on NSX load balancer and a new SSL session will be initiated from NSX load balancer to backend server;
    • Session persistence: cookie, SSL session id or source IP

Tips:

  1. L4 and L7 virtual server can co-exist on the same NSX load balancer;
  2. NSX load balancer can use 1 or multiple security groups as pool member, which means Virtual machines will be added into the load balancing pool automatically if they are added into right security group; This feature is especially useful when your Cloud VM is re-provisioned and its IP is changed;
  3. Transparent mode load balancing is not recommended due to the complexity and potential performance issue;
  4. In proxy mode, you can try to use HTTP x-forwarded-for to maintain the source IP information in the request;

Limitation and Constraints:

  1. Don’t supprt the integration with HSM;
  2. As NSX load balancer use the secondary IPs of vNIC, the size of virtual IP can’t scale up well;
  3. Lack of fine security control for traffic to virtual server;
  4. NSX can’t provide good service monitoring like F5 BIGIP or Citrix Netscaler;

 

New Ansible F5 HTTPs Health Monitor Module

Just got time this weekend to test the newly released dev version of Ansible F5 HTTPs health monitor. The result of testing looks good: most of common use cases have been covered properly.

Below is my first playbook for my testing:

# This version is to create a new https health monitor
---
- name: f5 config
  hosts:  lb.davidwzhang.com
  connection: local
  vars:
    ports:
      - 443
  tasks:
    - name: creat http healthmonitor
      bigip_monitor_https:
        state:  "present"
        #state: "absent"
        name: "ansible-httpshealthmonitor"
        password: "password"
        server: "10.1.1.122"
        user: "admin"
        validate_certs: "no"
        send: "Get /cgi-bin/env.sh HTTP/1.1\r\nHost:192.168.72.28\r\nConnection: Close\r\n"
        receive: "web"
        interval: "3"
        timeout: "10"
      delegate_to:  localhost

After run the playbook, I log in my F5 BIGIP VE and see the https health monitor has been created successfully.
f5 https healthmonitor

I tried to create another HTTPs health monitor, which includes basic authentication(admin/password) and customized alias address and alias service port(8443).
Playbook:

# This version is to create a new HTTP health monitor
---
- name: f5 config
  hosts:  lb.davidwzhang.com
  connection: local
  vars:
    ports:
      - 443
  tasks:
    - name: creat http healthmonitor
      bigip_monitor_https:
        state:  "present"
        #state: "absent"
        name: "ansible-httpshealthmonitor02"
        password: "password"
        server: "10.1.1.122"
        user: "admin"
        validate_certs: "no"
        ip: "192.168.72.128"
        port: "8443"
        send: "Get /cgi-bin/env.sh\r\n"
        receive: "200"
        interval: "3"
        timeout: "10"
        target_username: "admin"
        target_password: "password"
      delegate_to:  localhost

In F5, you can see the below:
f5 https healthmonitor02

In addition, you possibly noticed that I comment a line in the above 2 playbooks:

#state: "absent"

You can use it to remove the health monitor.

vRA7.3 and NSX Integration: Network Security Data Collection Failure

We are building vRA 7.3 . We added vCenter and NSX manager as endpoint in vRA. And associate NSX manager with vCenter. All of computing resource data collection works well but not NSX (network and security):

So in vRA reservation, we only can see vSphere cluster, vDS port-group/logical switch but not Transport zone, security group/tags

When check the log, we see the following:

Workflow ‘vSphereVCNSInventory’ failed with the following exception:

One or more errors occurred.

Inner Exception: An error occurred while sending the request.

at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)

at DynamicOps.VCNSModel.Interface.NSXClient.GetDatacenters()

at DynamicOps.VCNSModel.Activities.CollectDatacenters.Execute(CodeActivityContext context)

at System.Activities.CodeActivity.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager)

at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

Inner Exception:

VCNS Workflow failure

I tried to delete NSX end point and recreate from vRA but no luck. I raised the issue in vmware community but can’t get any real valuable feedback.

After a few hours investigation, finally I find a fix:

run the “create a NSX endpoint” workflow in vRO as the below

2017-07-26_184701

Then I re-start network & security data collection in vRA. Everything works and I can see all defined NSX Transport Zone, security groups and DLR in vRA network reservations.

Hope this fix can help others who have the same issue.

Bad OSPF summary routing due to poor IP addressing

The below is a real life example. Due to the poor IP addressing, you will see wrong OSPF summary routing has been added in the routing table under specific condition.ospf

As RFC 2328 suggests,the metric for summary routes in which the component of the summary route with the maximum / largest metric would determine the metric of the summary.

On Router 1, you will see the two routing entries for 10.16.192.0/23 have been added into routing table.

routing table

FYI:Route summary metric in RFC 1583

Area border routers then summarize the area contents (for distribution to the backbone) by advertising a single route for each address range.  The cost of the route is the minimum cost to any of the networks falling in the specified range.