Layer 2 DCI with eVPN

Today’s virtualized data centers are typically deployed at geographically diverse sites in order to optimize the performance of application delivery to end users, and to maintain high availability of applications in the event of site disruption.

These benefits are achieved via virtual machine live migration.

The virtual machine live migration raises 2 main requirements to the Data Center Interconnect:

1. When the virtual machine live migration happens, IP address/Subnet Mask/Default Gateway of the VM need to remain the same so that the application was not broken.

2. In addition, the DC network infrastructure is also relied upon to ensure that traffic flows to and from the VMs are forwarded along the most direct path, before, as well as after migration; that bandwidth on all available links is efficiently utilized; and, that the network recovers quickly to minimize downtime in the event of a link or node failure.

EVPN (Ethernet VPN) was recently published as a standard by IETF as RFC 7432. As a new technology, EVPN has attributes specifically designed to address the networking requirements of interconnected data centers, which demonstrate better flexibility and scalability than other existing DCI technologies like EoMPLS, VPLS and Cisco OTV or other proprietary solutions.

Today, I am here to show you how to configure eVPN on Juniper vMX. The topology of our configuration example is as below:

eVPN

MPLS configuration: OSPF+LDP+MPBGP (AS# 65001)eVPN routing instance: BD1 (type evpn)

IRB routing instance: L3 (type vrf). Local gateway IP at each DC is as the diagram above.

CE1:

root@CE1> show configuration | display set
set version 14.1R1.10
set system host-name CE1
set system root-authentication encrypted-password “$1$iiqdh9ru$CBQygri2MkpfgtgdGo8GO1”
set system login user lab uid 2000
set system login user lab class super-user
set system login user lab authentication encrypted-password “$1$oW1NeMUu$N7IVMfy8GfJ02jzQzJ39h/”
set system services telnet connection-limit 5
set system syslog user * any emergency
set system syslog file messages any notice
set system syslog file messages authorization info
set system syslog file interactive-commands interactive-commands any
set interfaces ge-0/0/1 vlan-tagging
set interfaces ge-0/0/1 unit 0 vlan-id 101
set interfaces ge-0/0/1 unit 0 family inet address 192.168.101.11/22
set interfaces em0 unit 0 family inet address 192.168.56.1/32
set routing-options static route 0.0.0.0/0 next-hop 192.168.101.1

PE1:

root@vMX-1> show configuration | display set
set version 14.1R1.10
set system host-name vMX-1
set system root-authentication encrypted-password “$1$iiqdh9ru$CBQygri2MkpfgtgdGo8GO1”
set system login user lab uid 2000
set system login user lab class super-user
set system login user lab authentication encrypted-password “$1$oW1NeMUu$N7IVMfy8GfJ02jzQzJ39h/”
set system services telnet connection-limit 5
set system syslog user * any emergency
set system syslog file messages any notice
set system syslog file messages authorization info
set system syslog file interactive-commands interactive-commands any
set interfaces ge-0/0/0 unit 0 family inet address 100.64.1.11/24
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 flexible-vlan-tagging
set interfaces ge-0/0/1 encapsulation flexible-ethernet-services
set interfaces ge-0/0/1 unit 101 encapsulation vlan-bridge
set interfaces ge-0/0/1 unit 101 vlan-id 101
set interfaces em0 unit 0 family inet address 192.168.56.11/32
set interfaces irb unit 0 family inet address 192.168.101.1/22
set interfaces lo0 unit 0 family inet address 192.168.1.1/32
set interfaces lo0 unit 0 family mpls
set routing-options router-id 192.168.1.1
set routing-options autonomous-system 65001
set routing-options forwarding-table chained-composite-next-hop ingress evpn
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface lo0.0
set protocols bgp group INTERNAL type internal
set protocols bgp group INTERNAL local-address 192.168.1.1
set protocols bgp group INTERNAL family inet-vpn unicast
set protocols bgp group INTERNAL family evpn signaling
set protocols bgp group INTERNAL neighbor 192.168.1.2
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface lo0.0
set routing-instances BD1 instance-type evpn
set routing-instances BD1 vlan-id 101
set routing-instances BD1 interface ge-0/0/1.101
set routing-instances BD1 routing-interface irb.0
set routing-instances BD1 route-distinguisher 1001:1001
set routing-instances BD1 vrf-target target:1001:1001
set routing-instances BD1 protocols evpn interface ge-0/0/1.101
set routing-instances L3 instance-type vrf
set routing-instances L3 interface irb.0
set routing-instances L3 route-distinguisher 1001:100
set routing-instances L3 vrf-target target:1001:100
set routing-instances L3 vrf-table-label

PE2:

root@vMX-2> show configuration | display set
set version 14.1R1.10
set system host-name vMX-2
set system root-authentication encrypted-password “$1$iiqdh9ru$CBQygri2MkpfgtgdGo8GO1”
set system login user lab uid 2000
set system login user lab class super-user
set system login user lab authentication encrypted-password “$1$oW1NeMUu$N7IVMfy8GfJ02jzQzJ39h/”
set system services telnet connection-limit 5
set system syslog user * any emergency
set system syslog file messages any notice
set system syslog file messages authorization info
set system syslog file interactive-commands interactive-commands any
set interfaces ge-0/0/0 unit 0 family inet address 100.64.1.22/24
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 flexible-vlan-tagging
set interfaces ge-0/0/1 encapsulation flexible-ethernet-services
set interfaces ge-0/0/1 unit 101 encapsulation vlan-bridge
set interfaces ge-0/0/1 unit 101 vlan-id 101
set interfaces em0 unit 0 family inet address 192.168.56.22/32
set interfaces irb unit 0 family inet address 192.168.103.1/22
set interfaces lo0 unit 0 family inet address 192.168.1.2/32
set interfaces lo0 unit 0 family mpls
set routing-options router-id 192.168.1.2
set routing-options autonomous-system 65001
set routing-options forwarding-table chained-composite-next-hop ingress evpn
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface lo0.0
set protocols bgp group INTERNAL type internal
set protocols bgp group INTERNAL local-address 192.168.1.2
set protocols bgp group INTERNAL family inet-vpn unicast
set protocols bgp group INTERNAL family evpn signaling
set protocols bgp group INTERNAL neighbor 192.168.1.1
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0
set protocols ldp interface ge-0/0/0.0
set protocols ldp interface lo0.0
set routing-instances BD1 instance-type evpn
set routing-instances BD1 vlan-id 101
set routing-instances BD1 interface ge-0/0/1.101
set routing-instances BD1 routing-interface irb.0
set routing-instances BD1 route-distinguisher 1001:1001
set routing-instances BD1 vrf-target target:1001:1001
set routing-instances BD1 protocols evpn interface ge-0/0/1.101
set routing-instances L3 instance-type vrf
set routing-instances L3 interface irb.0
set routing-instances L3 route-distinguisher 1001:100
set routing-instances L3 vrf-target target:1001:100
set routing-instances L3 vrf-table-label

CE2:

root@CE2> show configuration | display set
set version 14.1R1.10
set system host-name CE2
set system root-authentication encrypted-password “$1$iiqdh9ru$CBQygri2MkpfgtgdGo8GO1”
set system login user lab uid 2000
set system login user lab class super-user
set system login user lab authentication encrypted-password “$1$oW1NeMUu$N7IVMfy8GfJ02jzQzJ39h/”
set system services telnet connection-limit 5
set system syslog user * any emergency
set system syslog file messages any notice
set system syslog file messages authorization info
set system syslog file interactive-commands interactive-commands any
set interfaces ge-0/0/1 vlan-tagging
set interfaces ge-0/0/1 unit 0 vlan-id 101
set interfaces ge-0/0/1 unit 0 family inet address 192.168.101.12/22
set interfaces em0 unit 0 family inet address 192.168.56.2/32
set routing-options static route 0.0.0.0/0 next-hop 192.168.103.1

Verification

Section 1: Basic eVPN operation

CE1 MAC:

root@CE1> show interfaces ge-0/0/1
Physical interface: ge-0/0/1, Enabled, Physical link is Up
Interface index: 138, SNMP ifIndex: 514
Link-level type: Ethernet, MTU: 1518, MRU: 1526, Speed: 1000mbps,
BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
Source filtering: Disabled, Flow control: Enabled
Pad to minimum frame size: Disabled
Device flags   : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
CoS queues     : 8 supported, 8 maximum usable queues
Current address: 00:05:86:71:79:01, Hardware address: 00:05:86:71:79:01

CE2 MAC:

root@CE2> show interfaces ge-0/0/1
Physical interface: ge-0/0/1, Enabled, Physical link is Up
Interface index: 138, SNMP ifIndex: 514
Link-level type: Ethernet, MTU: 1518, MRU: 1526, Speed: 1000mbps,
BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
Source filtering: Disabled, Flow control: Enabled
Pad to minimum frame size: Disabled
Device flags   : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
CoS queues     : 8 supported, 8 maximum usable queues
Current address: 00:05:86:71:8e:01, Hardware address: 00:05:86:71:8e:01

On PE1, we can see the remote MAC and local MAC in the eVPN mac table.

root@vMX-1> show evpn mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Routing instance : BD1
Bridging domain : __BD1__, VLAN : 101
MAC                 MAC      Logical          NH     RTR
address             flags    interface        Index  ID
 00:05:86:71:79:01   D        ge-0/0/1.101
00:05:86:71:8e:01   DC                        1048574 1048574

We perform a ping test. The ping test shows the Ping CE2 from CE1 is successful.devil

root@CE1> ping 192.168.101.12
PING 192.168.101.12 (192.168.101.12): 56 data bytes
64 bytes from 192.168.101.12: icmp_seq=0 ttl=64 time=10.048 ms
64 bytes from 192.168.101.12: icmp_seq=1 ttl=64 time=8.530 ms
64 bytes from 192.168.101.12: icmp_seq=2 ttl=64 time=7.547 ms
^C
— 192.168.101.12 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 7.547/8.708/10.048/1.029 ms

We can verify that the remote MAC is learned via MP-BGP:

root@vMX-1> show route table BD1.evpn.0 | find 00:05:86:71:8e:01
2:1001:1001::101::00:05:86:71:8e:01/304
*[BGP/170] 00:33:27, localpref 100, from 192.168.1.2
AS path: I, validation-state: unverified
> to 100.64.1.22 via ge-0/0/0.0

The local MAC is injected to the eVPN table as well

root@vMX-1> show route table BD1.evpn.0 | find 00:05:86:71:79:01
2:1001:1001::101::00:05:86:71:79:01/304
*[EVPN/170] 00:36:00
Indirect

In addition, we can see that another routing entry for each MAC, which not only contains MAC but also corresponding IP:

root@vMX-1> …pn.0 | find 00:05:86:71:79:01::192.168.101.11
2:1001:1001::101::00:05:86:71:79:01::192.168.101.11/304
*[EVPN/170] 00:41:00
Indirect

root@vMX-1> … table BD1.evpn.0 | find 00:05:86:71:8e:01::192.168.101.12
2:1001:1001::101::00:05:86:71:8e:01::192.168.101.12/304
*[BGP/170] 00:41:07, localpref 100, from 192.168.1.2
AS path: I, validation-state: unverified
> to 100.64.1.22 via ge-0/0/0.0

Section 2: Layer 3 All-Active Gateways

Gateway MAC (IRB Interface) on PE1

root@vMX-1> show interfaces irb
Physical interface: irb, Enabled, Physical link is Up
Interface index: 132, SNMP ifIndex: 503
Type: Ethernet, Link-level type: Ethernet, MTU: 1514
Device flags   : Present Running
Interface flags: SNMP-Traps
Link type      : Full-Duplex
Link flags     : None
Current address: 00:05:86:71:bd:f0, Hardware address: 00:05:86:71:bd:f0
Gateway MAC (IRB Interface) on PE2

root@vMX-2> show interfaces irb
Physical interface: irb, Enabled, Physical link is Up
Interface index: 132, SNMP ifIndex: 503
Type: Ethernet, Link-level type: Ethernet, MTU: 1514
Device flags   : Present Running
Interface flags: SNMP-Traps
Link type      : Full-Duplex
Link flags     : None
Current address: 00:05:86:71:c7:f0, Hardware address: 00:05:86:71:c7:f0

Local gateway MAC address will be advertised to remote PE in our case PE2 with “evpn-default-gateway”  extended community.

root@vMX-1> …route table BD1.evpn.0 extensive | find 00:05:86:71:bd:f0
2:1001:1001::101::00:05:86:71:bd:f0/304 (1 entry, 1 announced)
TSI:
Page 0 idx 0, (group INTERNAL type Internal) Type 1 val 0x957e958 (adv_entry)
Advertised metrics:
Flags: Nexthop Change
Nexthop: Self
Localpref: 100
AS path: [65001] I
Communities: target:1001:1001 evpn-default-gateway
Path 2:1001:1001::101::00:05:86:71:bd:f0 Vector len 4.  Val: 0
*EVPN   Preference: 170
Next hop type: Indirect
Address: 0x940de94
Next-hop reference count: 7
Protocol next hop: 192.168.1.1
Indirect next hop: 0x0 – INH Session ID: 0x0
State:
Age: 57:46
Validation State: unverified
Task: BD1-evpn
Announcement bits (1): 1-BGP_RT_Background
AS path: I
Communities: evpn-default-gateway
Route Label: 299776
ESI: 00:00:00:00:00:00:00:00:00:00

root@vMX-1> ….0 extensive | find 00:05:86:71:bd:f0::192.168.101.1
2:1001:1001::101::00:05:86:71:bd:f0::192.168.101.1/304 (1 entry, 1 announced)
TSI:
Page 0 idx 0, (group INTERNAL type Internal) Type 1 val 0x957e9c8 (adv_entry)
Advertised metrics:
Flags: Nexthop Change
Nexthop: Self
Localpref: 100
AS path: [65001] I
Communities: target:1001:1001 evpn-default-gateway
Path 2:1001:1001::101::00:05:86:71:bd:f0::192.168.101.1 Vector len 4.  Val: 0
*EVPN   Preference: 170
Next hop type: Indirect
Address: 0x940de94
Next-hop reference count: 7
Protocol next hop: 192.168.1.1
Indirect next hop: 0x0 – INH Session ID: 0x0
State:
Age: 1:15:05
Validation State: unverified
Task: BD1-evpn
Announcement bits (1): 1-BGP_RT_Background
AS path: I
Communities: evpn-default-gateway
Route Label: 299776
ESI: 00:00:00:00:00:00:00:00:00:00

Similarly, PE1 receives a routing entry from PE2 for PE2 IRB (remote gateway at DC2). The receiving PE (PE1 here) will create a forwarding state to route packets destined for the gateway MAC, and a proxy ARP is done for the gateway IP with the MAC advertised in the route.

root@vMX-1> …route table BD1.evpn.0 extensive | find 00:05:86:71:c7:f0
2:1001:1001::101::00:05:86:71:c7:f0/304 (1 entry, 1 announced)
*BGP    Preference: 170/-101
Route Distinguisher: 1001:1001
Next hop type: Indirect
Address: 0x97a8454
Next-hop reference count: 10
Source: 192.168.1.2
Protocol next hop: 192.168.1.2
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
State:
Local AS: 65001 Peer AS: 65001
Age: 59:35      Metric2: 1
Validation State: unverified
Task: BGP_65001.192.168.1.2+63423
Announcement bits (1): 0-BD1-evpn
AS path: I
Communities: target:1001:1001 evpn-default-gateway
Import Accepted
Route Label: 299776
ESI: 00:00:00:00:00:00:00:00:00:00
Localpref: 100
Router ID: 192.168.1.2
Primary Routing Table bgp.evpn.0
Indirect next hops: 1
Protocol next hop: 192.168.1.2 Metric: 1
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: 100.64.1.22 via ge-0/0/0.0
Session Id: 0x0
192.168.1.2/32 Originating RIB: inet.3
Metric: 1                       Node path count: 1
Forwarding nexthops: 1
Nexthop: 100.64.1.22 via ge-0/0/0.0

root@vMX-1> ….evpn.0 extensive | find 00:05:86:71:c7:f0::192.168.103.1
2:1001:1001::101::00:05:86:71:c7:f0::192.168.103.1/304 (1 entry, 1 announced)
*BGP    Preference: 170/-101
Route Distinguisher: 1001:1001
Next hop type: Indirect
Address: 0x97a8454
Next-hop reference count: 10
Source: 192.168.1.2
Protocol next hop: 192.168.1.2
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
State:
Local AS: 65001 Peer AS: 65001
Age: 1:14:00    Metric2: 1
Validation State: unverified
Task: BGP_65001.192.168.1.2+63423
Announcement bits (1): 0-BD1-evpn
AS path: I
Communities: target:1001:1001 evpn-default-gateway
Import Accepted
Route Label: 299776
ESI: 00:00:00:00:00:00:00:00:00:00
Localpref: 100
Router ID: 192.168.1.2
Primary Routing Table bgp.evpn.0
Indirect next hops: 1
Protocol next hop: 192.168.1.2 Metric: 1
Indirect next hop: 0x2 no-forward INH Session ID: 0x0
Indirect path forwarding next hops: 1
Next hop type: Router
Next hop: 100.64.1.22 via ge-0/0/0.0
Session Id: 0x0
192.168.1.2/32 Originating RIB: inet.3
Metric: 1                       Node path count: 1
Forwarding nexthops: 1
Nexthop: 100.64.1.22 via ge-0/0/0.0

root@vMX-1> show evpn peer-gateway-macs

Routing instance : BD1
Bridging domain : __BD1__, VLAN : 101
Installed GW MAC addresses:
00:05:86:71:c7:f0

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s