Category Archives: tcp-ip

Slow file transfer and TCP Zero Window Probe

Slow file transfers must be really bothering everyone. I have a ZyXEL NSA325 NAS device which has a gigabit interface but I am getting extremely low throughput. Unfortunately this has been a problem I think since I bought this device. Now I could finally get hold of time to troubleshoot the issue. Here is my topology I used in testing this scenario.

nas-tcp-window-zero

As per the topology above, my laptop and this NAS device are connected to two ports of this Juniper EX2200 switch. I have enabled jumbo frame on the ports, laptop and NAS device.

Continue reading

Packetization Layer PMTU Discovery

Path MTU discovery that is in place today is relying on ICMP based MTU discovery i.e you send an oversize packet which can’t be forwarded by an intermediate host in the path because the next hop link has a lower MTU size, then the source host is notified by this hop which can’t forward this packet. It is this notification that is sent to the source in an ICMP Destination Unreachable “Fragmentation needed and DF set” message but what happens if this ICMP notifications are blocked? Then we have a big problem and sometimes it may be difficult to identify.
So in this post I would like to show the mitigation technique in case ICMPs are blocked in the network. Let’s first see this ICMP block situation and how we can mitigate this problem by using packetization layer MTU discovery method which is explained in RFC4821 “Packetization Layer Path MTU Discovery”

Following is our topology that we carry out the tests.

mtu-probing-topology-2

Let’s first lower the MTU on segment 2. We do this on Host B(LAB1021-R1)

Yes we have a lower MTU now.

Continue reading

Wireshark [TCP Window Full] & [Zero Window]

TCP sliding window is very crucial concept in understanding how TCP behaves. In order to see how this mechanism works, I have rate limited an HTTP download and observed what happens during this scenario in which we will see reports from Wireshark that [TCP Window Full] and [TCP ZeroWindow]. The aim of this post is to try to show how wireshark understands that Window is full.

tcp-window-full-topology

We have a web server and a client machine on this setup. We intentionally rate limit the traffic by using wget to allow us investigate this scenario.

Continue reading

Traceroute and meaning of outputs

Van Jacobson is a prominent person in networking, especially for TCP/IP. What I didn’t know was (according to wikipedia) original traceroute was also written by him. As this tool is the swiss knife of a Tech Support Engineer, I would like to share the meaning of some of the outputs. If you have any other error, please do share here to improve the list.

!N

This sample output indicates that network that 10.1.2.4 IP belongs doesn’t exist on the 10.11.1.1 host if you take a packet capture when you see this error, you will see that you receive, ICMP destination unreachable “Network Unreachable” message from this host.

!H

This error however indicates that IP network is available but the individual host 10.11.2.4 can’t be reached. It isn’t available. The last host (10.11.1.1) which is supposed to provide connectivity to the destination device returns ICMP destination unreachable “Host unreachable” to the source host.

!F

This error is received if you are trying some PMTU discovery. Intermediate host which can’t deliver this oversized packet returns ICMP Destination unreachable “Fragmentation needed and DF bit set”

Any other error letter you have seen? Drop your comment here!

Traceroute behaviour in MPLS

Traceroute is a great tool to discover the path a packet traverses in outgoing direction but if you have an MPLS cloud, you may have some unexpected behavior if you don’t do some tweaks. First of all let’s see how traceroute discovers a path when there isn’t any MPLS cloud.

traceroute-ttl

The network above is using IP to route packets and we are running traceroute on GW2 device towards Debian1 device.

We can clearly see the two hops in our traceroute. IP addresses displayed on the output are from ingress interface of our probe packets. For this traceroute I also took a packet capture on ingress interface of GW1 i.e 212.6.1.1 side.

Junos and Linux traceroute by default use UDP to send probe packets and each hop receives 3 UDP segments.
Continue reading

Effect of TCP SACK on throughput

On this Saturday evening, I have finally completed my work with TCP SACK analysis. This post was in my mind for sometime but now I have done it after building my big local Internet at home. You will also find some stuff about receive segmentation offload, wireshark tips etc. Here is the topology used for TCP SACK tests.

tcp_sack_bgp_setupFirst of all, this big setup isn’t really necessary but I am using this setup for my BGP tests and have found it suitable for a real world scenario. What are we testing?

Continue reading

Fragmented IP packet forwarding

I couldn’t really find a suitable topic for this post actually but I will try to find answers for the following questions:

  • How can we fragment an IP packet manually in scapy
  • How does a fragmented packet look like and how the transport layer (TCP/UDP) header is located
  • How do we forward fragmented packets, do we reassemle them?
  • If we don’t reassemble, can we force reassembly?

First of all a bit of a theory: if an incoming IP packet is to be forwarded to another next hop and the MTU of this new path is smaller than the packet to be transmitted, we must find a way to forward the packet. If the packet has DF (Don’t Fragment) bit on i.e we are instructed not to fragment the packet most probably by the source, then normally we are expected to send an ICMP packet with type “Fragmentation needed” and pray that on the way back to the source no devices block all ICMP type of traffic. Second scenario is that what if the source lets us fragment the packet. Then we need to fragment it and story from now on is about this part of the scenario and the topology we will use is something like below.

fragmented_packets Continue reading

IP Identification why zero?

I must say that network troubleshooting is not an easy task. Especially if you need to analyze thousands of packets in packet captures or lines of flow traces. IP ID is a field I use to compare captures taken at different segments most of the time. Also it is a crucial field for me to find the right packet in the flow trace. I didn’t know that this field can be zero till I notice it in a flow trace. Following captured packet is a SYN-ACK segment from a Linux box. ip_identification_zero

and Identification field is 0. If you have multiple SYN-ACKs from the same source your Seq,Ack numbers will also be the same which means you have literally no way to distinguish two packets apart from timing. I have searched for a way to disable this zero ID feature to make it unique in Linux but there doesn’t seem to exist any way. When I was searching some documentation to find the reason for this zero ID, I have found a very recent RFC Updated Specification of the IPv4 ID Field. Here is the text that I really didn’t like on this RFC;

RFC doesn’t enforce anything on the value and also states that originating source MAY set the field of atomic datagrams to any value. RFC also touches on the performance impact of uniqueness of ID field for a given source/destination.

In my humble opinion, this field should never have zero value at least not for me:) and Linux should have a sysctl switch to disable this behaviour.

Especially if you have an IPSEC VPN connection and you need to take a packet capture, you have almost no way to make a comparison between packet captures. Wouldn’t it be nice temporarily copy ip identification field value to the outside header of an ESP packet or a way to let the troubleshooter match the clear text packet and encrypted one. It would be terrific!

IPsec TCP-MSS, DF-BIT and Fragmentation

In my previous ipsec troubleshooting post, I haven’t talked about how we approach performance issues. Which is probably not a JNCIE-SEC topic but this is a very important topic for the real networks.

tcp-mss-df-bit-topology2

In this topology I will examine how throughput changes between two end points of an IPSEC tunnel depending on the configuration of IPSEC tunnel.

Change 1) Setting DF-BIT to copy

An IPsec tunnel between J23 and J41 is established and no extra configuration is done. I initiate a huge 1.6GB file download via HTTP
Continue reading

Port Scanner in Python

Python is a great tool to do some socket operations. I have written a piece of code by which I can scan a port range.
It is very basic and missing bunch of checks as aim is the simplicity here.

You can run the script in the following way by which you scan ports between 1 and 1024:

Continue reading