Intermittent IPSEC tunnel connectivity issues on Vyatta

I just experienced an Intermittent IPSEC tunnel connectivity issues on Vyatta. Customer suggested they intermittently lose the connectivity to their Softlayer VMs via IPSec between their corporate VPN gateway and Softlayer Vyatta gateway.

Finally, I found out the issue is due to a Vyatta underlying Linux OS Debian system kernel network setting:xfrm4_gc_thresh.

The xfrm_gc_thresh controls the size of the IPSEC connection routing table which contains a list of the source and target systems. When the thresh value is reached the table is cleaned up very aggressively which inadvertently removes information related to active connections containing the Syn. Since the Syn entry for the active location can no longer be found the second part of the tcp handshake which contains the Syn-Ack ( Syn,Ack) response is silently drop.

On Vyatta OS version 6.7, the vaule of xfrm4_gc_thresh is set too low:1024, which bring intermittent IPSEC tunnel connectivity issues to customer if customer has a “big number” of sessions (like a few of hundred sessions. I know it is not big number at all). You can verify your system setting by using CLI:

vyatta@vyatta:~$ cat /proc/sys/net/ipv4/xfrm4_gc_thresh
1024

In addition, you can use CLI below to verify if you are experiencing this issue or not.
vyatta@vyatta:~$ cat /proc/net/xfrm_stat | grep BundleGen
XfrmOutBundleGenError 164671

Increasing the size of this threshold means that active connections will no longer be removed from the IPSEC connection routing table when the xfrm4_gc_thresh value is reached.

Recommended value for xfrm4_gc_thresh is 32768.

To fix the issue on the fly, you can do the following on Vyatta gateway:

sudo sysctl -w net.ipv4.xfrm4_gc_thresh=32768
sudo sysctl -p

The above change to the kernel network setting will lost after next reboot. To avoid that, you need to add the setting “net.ipv4.xfrm4_gc_thresh=32768” into /etc/sysctl.conf.

Note: the above issue is confirmed as Vyatta firmware bug by Brocade and will be fixed in the upcoming release 6.7R9.

In addition, another symptom for the issue is as below:

vyatta@vyatta:~$ ping 10.2.128.81
connect: No buffer space available

Update:

Version 6.7R9 has been released by Brocade. The bug is fixed in this version.

Version:      VSE6.7R9
Description:  Brocade Vyatta 5415 vRouter 6.7 R9
Copyright:    2006-2015 Vyatta, Inc.
Last login: Fri Oct  2 14:17:02 2015 from 60-242-58-167.static.tpgi.com.au
vyatta@vyatta:~$  cat /proc/sys/net/ipv4/xfrm4_gc_thresh
32768

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s