Duplicate Packet or TCP retransimission?

When you analyse the packet capture in wireshark, you sometimes see the similar/nearly same packets more than 1 time. It is maybe due to duplicate packet or TCP retransimission. If it is TCP retransmission, you have to pay more attention on it.

 

The difference between duplicate packet and TCP restransmission is IP ID. For duplicate packets, IP ID of two packets is same besides the TCP sequence number is same. (Why you can see the duplicate packets in the capture is not in the scope of this discussion. If you have interest, please google it. 🙂

 

Duplicate Packet example (same IP ID and same TCP sequence number)wKioL1OP4ZmjE6pcAAH8SGETuT8295

TCP Original packet

wKioL1OP5QOCghZ7AARaSvfdasI156

TCP retransmission packet

wKiom1OP5Vvz6xwOAAOy38Fp7JA403

Same TCP sequence number (2171548184)but different IP ID (frame 553 ID is 48058 and frame 578 ID is 48083) for orgianl packet and retransmission packet.

F5 packet capture CLI

F5 offers the capacity for packet capture by use of tcpdump command. In version 10.x, F5 doesn’t support you to perform tcpdump in the non-default route domain.

F5 recommends that you run the tcpdump command from the default route domain (route domain 0), and specify interface 0.0 as below:

tcpdump -s0  -w /var/tmp/WOI1.pcap -fnni 0.0:nnn  host x.x.x.x (x.x.x.x works as a filter which match the source IP or destination IP of a packet)

In addition, F5 has a CLI for SSL traffic capture which is good for the analysis of SSL traffic

ssldump -aAden -N -r <dump file> -k <key file> >> /var/tmp/<output file>

 

UPDATE:

In BigIP version 13.1, you can do the following like other linux

[root@bigip1:Active:In Sync] config # tcpdump -i 1.3 -w hsm.pcap
tcpdump: listening on 1.3, link-type EN10MB (Ethernet), capture size 65535 bytes
^C297 packets captured
297 packets received by filter
0 packets dropped by kernel

How to achieve maximum TCP throughput on LFN

Firstly, what’s LFN? LFN means long fat network, often pronounced “elephan”.

In RFC 1072, a network is considered an LFN if its bandwidth-delay product is significantly larger than 105 bits  (12500 bytes).

Then you will possibly have another question: what bandwidth-delay product is?

As Wiki suggested, bandwidth-delay product refers to the product of a data link’s capacity (in bits per second) and its round-trip delay time (in seconds). The result, an amount of data measured in bits (or bytes), is equivalent to the maximum amount of data on the network circuit at any given time, i.e., data that has been transmitted but not yet acknowledged.

When using TCP to transfer data, the maximum possible throughput of a data transfer between sender and receiver will be “How many data the receiver says it can receive” or “How many data the sender thinks it can send”, whichever is lower.

The factors which impact “How many data the receiver says it can receive” mainly includes: TCP recieve windows and SOCKET receive buffer size.

TCP receive window

The receive window is the window that controls how much unacknowledged data can be in flight from the sender to the receiver. In another words, TCP receive window suggests the possible maximum size of Bytes in Flight (the amount of unacknowledged data already sent by sender).

SOCKET receive buffer size: Receive Buffer size defines the receive buffer in Socket layer. Programs based on Socket gets data from that buffer. When a program receives more data than this buffer is configured to hold, all data received up to this count must be transferred to the program before receiving continues. When this happens, an acknowledgement will be sent to the sender.

Similarly, the factors which impact “How many data the sender thinks it can send” mainly includes: SOCKET send buffer size and current size of Bytes in Flight.

SOCKET send buffer size: Send buffer size is the size of the socket send buffer. This is the buffer that the application writes data to for TCP to send.

Almost of all modern OS (Windows and Linux) are not really designed for LFN. Unfortunately, most of wide area network links are LFN at the same time. So you have to manually tune the above 3 parameters to achieve better performance when you are talking about throughput over WAN.

Recently, I just helped one client on their network throughput for their Cloud migration. They have quite big bandwidth but only achieve 1Mbit/s throughput from their own network to a Cloud DC. After I tuned the above 3 settings on their windows servers, they immediately achieves 10 times throughput improvement!!!

Please note you should have the SOCKET send buffer size and SOCKET receive buffer size the same value as TCP windows size.

There are other parameters which will decrease the TCP throughput a lot when these features are disabled:

SACK (selective acknowledgement):

SACK allows a TCP receiver inform the sender exactly which data is missing and needs to be retransmitted.

TCP Window Scaling:

For larger window sizes to accommodate high-speed transmission paths, RFC 1323 (ietf.org/rfc/rfc1323.txt) defines window scaling that allows a receiver to advertise a window size larger than 65,535 bytes. A TCP Window Scale option includes a window scaling factor that, when combined with the 16-bit Window field in the TCP header, can increase the receive window size to a maximum of approximately 1GB.

F5 wireshark plugin

When you perform packet capture on F5 LTM, you possibly notice there are some unknow fileds in the packet capture.

These unknown data fileds are the additional diagnostic data which is encoded on tcpdump captures by F5 LTM. F5 has provided the wireshark plugin to decode the unknown fileds. You can download the plugin from the link below:

https://devcentral.f5.com/d/wireshark-plugin

What you need to is to create a customize wireshark build to include the plugin.

Please follow the steps below to build your own wireshark :

Installation
1. Acquire the Wireshark source tarball at:
http://www.wireshark.org/download/src/wireshark-version.tar.bz2

2. Extract out the files:
tar xjf wireshark-{version}.tar.bz2

3. Enter into the directory, and extract the files in the F5 package:
cd wireshark-{version}/
tar xzf wireshark.plugin.f5ethtrailer.1.3.tar.gz

4. Apply the patch:
patch -p1 < f5ethtrailer.makefiles.{version}.patch

5a. If you are on Windows, proceed to compilation following the instructions at:
http://www.wireshark.org/docs/wsdg_html_chunked/ChSetupWin32.html

5b. If you are on a GNU GCC based platform, proceed to compilation by following the instructions at:
http://www.wireshark.org/docs/wsdg_html_chunked/ChapterSetup.html

6. Install Wireshark to your target system

When you get your own wireshark build, open the F5 LTM tcpdump file, you will see something like below:

f5wireshark