Category Archives: linux

tmux multiple terminals

I think most of us watched the movie Inception in which there was dream within a dream. It was a bit of confusing at some point. There is a Linux version of it 🙂 terminal within terminal provided by the fantastic tool tmux (terminal multiplexer). I use this tool during some troubleshooting scenarios when I would like to see two continuous pings on two terminals. Sometimes, I also divide the screen to 4 terminals. It may not be needed for everyone but in my opinion it is really handy. Here is an example of what you can achieve by tmux on a single terminal.


It is pretty cool right.

Continue reading

htop an alternative to top command

Today I have stumbled across a system resource utilization tool htop which is way more visual than top and it is easier to use too. For example, following is from my single core amazon cloud centos server.
You can easily see that sorting is by CPU and if you want to change it, just press F6 and you get a new menu to choose as to which column you want to sort by. It is really cool!


PS: I have disabled my mail notifications for a while for minor posts as I don’t want to bother subscribers with my tiny posts.

Packetization Layer PMTU Discovery

Path MTU discovery that is in place today is relying on ICMP based MTU discovery i.e you send an oversize packet which can’t be forwarded by an intermediate host in the path because the next hop link has a lower MTU size, then the source host is notified by this hop which can’t forward this packet. It is this notification that is sent to the source in an ICMP Destination Unreachable “Fragmentation needed and DF set” message but what happens if this ICMP notifications are blocked? Then we have a big problem and sometimes it may be difficult to identify.
So in this post I would like to show the mitigation technique in case ICMPs are blocked in the network. Let’s first see this ICMP block situation and how we can mitigate this problem by using packetization layer MTU discovery method which is explained in RFC4821 “Packetization Layer Path MTU Discovery”

Following is our topology that we carry out the tests.


Let’s first lower the MTU on segment 2. We do this on Host B(LAB1021-R1)

Yes we have a lower MTU now.

Continue reading

Linux Namespace – Routing Instance

In Linux, in the past I was using iproute2 and multiple routing tables to do some more advance stuff but when I became aware of Namespace, things really changed for me. Namespaces in Linux seem to be similar to logical systems in Junos. It seems to be a bit more than a routing instance in my opinion. I believe this much of introduction is sufficient. Now I would like to show several commands by which we can create a new routing instance inside a Linux box. To this new routing instance we will also assign a VLAN interface.

I have a linux box named vHost2. It has 2 physical ethernet interfaces as you can see. (Actually more but I am hiding some to make the output more brief) eth1 is connected to a trunk port on the switch side so that I can create vlan interfaces on this physical interface.

First create the vlan interface.

Continue reading

Source address selection in traceroute

Have you ever thought how the IP addresses are chosen/selected in icmp time exceeded error messages when you run a traceroute command? Recently I was analyzing an issue and this really made a difference in troubleshooting. I have done the analysis on an SRX firewall and a Linux device and I have got different results. I haven’t really prepared a setup for this but I will try to show this in a sample traceroute output.

This is just a snippet of my traceroute. is actually an SRX device and this IP address is tied to vlan.103 interface to which my packet (UDP traceroute) has entered. What if packet returned to me for example via vlan.104 interface. Would I still see the IP address in the output? Answer is yes according to my tests. What I have noticed is the following on an SRX box if there is asymmetric routing;

    • In flow mode SRX: device is sending icmp time exceeded message with source address of vlan.103 through the interface that the packet entered. Regardless of the routing error is sent via the incoming path
    • In packet mode SRX: device is sending icmp time exceeded message with the source address of vlan.103 through the vlan.104


As you can see even if the device can follow a different path to you, ICMP time exceeded error message is sourced from the incoming interface. This makes troubleshooting a bit difficult actually as you don’t really understand how the remote device is returning the traffic back to you unless you take packet capture.

What about Linux? Linux’s behaviour is also different. If we put Linux instead of SRX on this topology and packet enters via vlan.103 and return via vlan.104, Linux will send ICMP time exceeded packet through the vlan.104 interface with source address of vlan.104. This means if your UDP packet (traceroute) is forwarded through a Linux device and there is an asymmetric route, you can notice immediately if you know the interface IP assignments as traceroute will display a different IP address then you expect.

To be honest, I have searched several RFCs in order to find if there is any RFC requirement in ICMP time exceeded source address selection but couldn’t find anything. If you know anything, please leave a comment:)

Update: Later I have found the nanog traceroute presentation in which I have found the answer for this mystery.

Kamailio SIP server

Today I wanted to test kamailio SIP server but I didn’t have prior experience on this software
and I experienced several problems.

After following the installation manual I created the username rtoo.

so far so good. Then I tried to connect to this server but I got
403 Not relaying” then I put the server in debug mode by adding

on top of /usr/local/etc/kamailio/kamailio.cfg file and restarted the server

then I have found that I must add my domain name somewhere in the config.
After this, I added domain as follows in the same config file
and restarted the server again.

Then it worked. However at the very beginning I even got an authentication error like

I don’t know how this error occurred after adding username via CLI. Then I
found the description of the authentication table of kamailio which is subscriber.

At the URL,
you can see that ha1 and ha1b is an MD5 of some values. According to the
ha1 description, I got the MD5, and updated the subscriber table than it worked.

You may not have this error as I didn’t receive it again.

I have found kamailio server quite powerful so far. I hope I will do some test with SRX as well.

VIM syntax highlighting on SecureCRT

My SecureCRT terminal window wasn’t displaying ANSI colors which was really annoying me. To be honest I dislike procrastination and finally I have enabled ANSI colors:)

In Ubuntu installing “vim” package is sufficient I suppose. In other distributions you may have to install supplementary packages. Here are the possible steps;

1) Set your terminal to xterm or linux if xterm does not work

2) On SecureCRT


3) Open a file and see if highlighting is on

4) If not, type the command inside the vim editor.

if you can see the highlighting, then add “syntax on” to your .vimrc file under your home directory to make it permanent. According to SecureCRT article, you can also add the followings to your .vimrc file

Meanings of Access,Modify and Change in stat command

I keep forgetting the “subtle” differences between access,modify and change attributes of an inode. I will try to take some note here for you and me. My file test.txt has got the text “abc” in it which is 3 bytes + 1 new line = 4 bytes and I pasted inode information below via stat command.

Continue reading

Increasing the number of inodes

If you are keeping huge number of small files in your disks, this may one day cause running out of available inodes because of which even though you might have sufficient disk space, file system cannot create new files. Lets see how it works in my 1GB disk partition.

The parameter “-i” allows us to play with the number of inodes we can have in a partition. It is called “bytes-per-inode”. The bigger it is less number of inodes you will have. I have set a relatively big size “163840” bytes per inode and the result is I have 6768 inodes. What this means indeed is I can’t create more than 6768 files. (even a bit less than this)

If you don’t believe me, lets do a test. I will mount this partition and create this number of files;

Continue reading

Linux file attributes

Have you ever received such an error when you are logged in as root? This error surprised me as I asked myself: Hang on a second! I am the super user, I should be able to change the permission of this regular file.

Look what prevents me from doing this;

Culprit is the file attribute “a” which means append. With “644” mode, I was trying to remove the write permission from the file but kernel didn’t let me do it because of this bit on. I would appreciate a cleaner error message indeed:) Once I clear the append flag, I successfully changed the permission.

There are a couple of more flags I like;

  • A : atime record of the file isn’t modified if set
  • S : changes are written synchronously on the disk
  • i : My favorite, the immutable bit. If set none can modify the file even can create a link