How to achieve maximum TCP throughput on LFN

Firstly, what’s LFN? LFN means long fat network, often pronounced “elephan”.

In RFC 1072, a network is considered an LFN if its bandwidth-delay product is significantly larger than 105 bits  (12500 bytes).

Then you will possibly have another question: what bandwidth-delay product is?

As Wiki suggested, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.

When using TCP to transfer data, the maximum possible throughput of a data transfer between sender and receiver will be “How many data the receiver says it can receive” or “How many data the sender thinks it can send”, whichever is lower.

The factors which impact “How many data the receiver says it can receive” mainly includes: TCP recieve windows and SOCKET receive buffer size.

TCP receive window

The receive window is the window that controls how much unacknowledged data can be in flight from the sender to the receiver. In another words, TCP receive window suggests the possible maximum size of Bytes in Flight (the amount of unacknowledged data already sent by sender).

SOCKET receive buffer size: Receive Buffer size defines the receive buffer in Socket layer. Programs based on Socket gets data from that buffer. When a program receives more data than this buffer is configured to hold, all data received up to this count must be transferred to the program before receiving continues. When this happens, an acknowledgement will be sent to the sender.

Similarly, the factors which impact “How many data the sender thinks it can send” mainly includes: SOCKET send buffer size and current size of Bytes in Flight.

SOCKET send buffer size: Send buffer size is the size of the socket send buffer. This is the buffer that the application writes data to for TCP to send.

Almost of all modern OS (Windows and Linux) are not really designed for LFN. Unfortunately, most of wide area network links are LFN at the same time. So you have to manually tune the above 3 parameters to achieve better performance when you are talking about throughput over WAN.

Recently, I just helped one client on their network throughput for their Cloud migration. They have quite big bandwidth but only achieve 1Mbit/s throughput from their own network to a Cloud DC. After I tuned the above 3 settings on their windows servers, they immediately achieves 10 times throughput improvement!!!

Please note you should have the SOCKET send buffer size and SOCKET receive buffer size the same value as TCP windows size.

There are other parameters which will decrease the TCP throughput a lot when these features are disabled:

SACK (selective acknowledgement):

SACK allows a TCP receiver inform the sender exactly which data is missing and needs to be retransmitted.

TCP Window Scaling:

For larger window sizes to accommodate high-speed transmission paths, RFC 1323 ( defines window scaling that allows a receiver to advertise a window size larger than 65,535 bytes. A TCP Window Scale option includes a window scaling factor that, when combined with the 16-bit Window field in the TCP header, can increase the receive window size to a maximum of approximately 1GB.

F5 wireshark plugin

When you perform packet capture on F5 LTM, you possibly notice there are some unknow fileds in the packet capture.

These unknown data fileds are the additional diagnostic data which is encoded on tcpdump captures by F5 LTM. F5 has provided the wireshark plugin to decode the unknown fileds. You can download the plugin from the link below:

What you need to is to create a customize wireshark build to include the plugin.

Please follow the steps below to build your own wireshark :

1. Acquire the Wireshark source tarball at:

2. Extract out the files:
tar xjf wireshark-{version}.tar.bz2

3. Enter into the directory, and extract the files in the F5 package:
cd wireshark-{version}/
tar xzf wireshark.plugin.f5ethtrailer.1.3.tar.gz

4. Apply the patch:
patch -p1 < f5ethtrailer.makefiles.{version}.patch

5a. If you are on Windows, proceed to compilation following the instructions at:

5b. If you are on a GNU GCC based platform, proceed to compilation by following the instructions at:

6. Install Wireshark to your target system

When you get your own wireshark build, open the F5 LTM tcpdump file, you will see something like below: