How to achieve maximum TCP throughput on LFN

Firstly, what’s LFN? LFN means long fat network, often pronounced “elephan”.

In RFC 1072, a network is considered an LFN if its bandwidth-delay product is significantly larger than 105 bits  (12500 bytes).

Then you will possibly have another question: what bandwidth-delay product is?

As Wiki suggested, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.

When using TCP to transfer data, the maximum possible throughput of a data transfer between sender and receiver will be “How many data the receiver says it can receive” or “How many data the sender thinks it can send”, whichever is lower.

The factors which impact “How many data the receiver says it can receive” mainly includes: TCP recieve windows and SOCKET receive buffer size.

TCP receive window

The receive window is the window that controls how much unacknowledged data can be in flight from the sender to the receiver. In another words, TCP receive window suggests the possible maximum size of Bytes in Flight (the amount of unacknowledged data already sent by sender).

SOCKET receive buffer size: Receive Buffer size defines the receive buffer in Socket layer. Programs based on Socket gets data from that buffer. When a program receives more data than this buffer is configured to hold, all data received up to this count must be transferred to the program before receiving continues. When this happens, an acknowledgement will be sent to the sender.

Similarly, the factors which impact “How many data the sender thinks it can send” mainly includes: SOCKET send buffer size and current size of Bytes in Flight.

SOCKET send buffer size: Send buffer size is the size of the socket send buffer. This is the buffer that the application writes data to for TCP to send.

Almost of all modern OS (Windows and Linux) are not really designed for LFN. Unfortunately, most of wide area network links are LFN at the same time. So you have to manually tune the above 3 parameters to achieve better performance when you are talking about throughput over WAN.

Recently, I just helped one client on their network throughput for their Cloud migration. They have quite big bandwidth but only achieve 1Mbit/s throughput from their own network to a Cloud DC. After I tuned the above 3 settings on their windows servers, they immediately achieves 10 times throughput improvement!!!

Please note you should have the SOCKET send buffer size and SOCKET receive buffer size the same value as TCP windows size.

There are other parameters which will decrease the TCP throughput a lot when these features are disabled:

SACK (selective acknowledgement):

SACK allows a TCP receiver inform the sender exactly which data is missing and needs to be retransmitted.

TCP Window Scaling:

For larger window sizes to accommodate high-speed transmission paths, RFC 1323 (ietf.org/rfc/rfc1323.txt) defines window scaling that allows a receiver to advertise a window size larger than 65,535 bytes. A TCP Window Scale option includes a window scaling factor that, when combined with the 16-bit Window field in the TCP header, can increase the receive window size to a maximum of approximately 1GB.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s