Category Archives: troubleshooting

Wonderful tool traceroute monitor

There is one traceroute option which you might not have noticed so far: It is the monitor. I use this option during packet drop issues from time to time to see if there is any hop on the path which might be causing some drop or latency. It is extremely handy and you can also modify your monitoring parameters. I believe this is a very useful troubleshooting option. The column names are also intuitive but you can take a look at the documentation here for this command if you like.

This command helped you in any troubleshooting, then drop here a comment!

According to Matt’s comment, both Linux and FreeBSD has this tool. For more info about the tool and its history check the wiki page. Tool was originally called Matt’s traceroute due to its first writer 🙂 On an ubuntu host, “apt-get install mtr” will bring you this handy tool in a few seconds!

flow trace without commit

On SRX, there is now a handy feature introduced in 12.1X46-D10. You can enable flow trace without going into configuration on the operational mode. I believe this will make troubleshooting easier as it saves time if you need to try different flow filters. Here is how you can enable a sample ICMP flow trace for a specific IP address e.g 192.168.1.10

Create your filters named incoming-filter,outgoing-filter to catch the traffic

Give a file name to save the flow trace

File will be saved under /var/log folder, you can also set size option if you like

Check the filters

Yes we have created the filters but they are not active as you can see on the Status field.
Continue reading

JNCIE-SEC: Traceoptions & IPSEC troubleshooting

In IPSEC topic, I am continuing with traceoptions and troubleshooting section. In this post, I will try to explain how I troubleshoot IPSEC VPNs mostly initial setup.

IPsec VPNs

NAT

  • Implementation of NAT
  • Source NAT
  • Destination NAT
  • Static NAT
  • Implementation of NAT with IPSec
  • Overlapping IPs between sites

ipsec_multipoint_route_policy_based_vpn

One of the challenging parts of JNCIE-SEC must be the troubleshooting part for which I need to understand under what sort of problems what type of error logs are generated. Because of this, I enabled IKE traceoptions and simulated several type of possible problems and observed the error logs.

But first let’s see how a successful IKE Phase 1 and IKE Phase 2 log looks like;

PS: All errors below are between ike peers 192.168.179.2 and 212.45.64.2

IKE & IPSEC SUCCESSFUL LOG

Phase 1

You can see the “IKE negotiation done” log in here.
Continue reading

Packet mode and host-inbound traffic

Did you know that if you enable packet-mode in traffic interface of an SRX box,
host inbound traffic isn’t allowed anymore? Device can still process transit traffic
but inbound traffic won’t work. For example, apply a filter like below to an interface
and try to SSH to IP 98.1.1.1, you shouldn’t be allowed.

If you are wondering why: Documentation says that host-inbound traffic is only allowed
if it is sent to flow daemon for inspection.

PS: You can selectively exclude local IP from packet mode but here I just want to show the behavior change.

How to take packet capture in SRX

I wasn’t planning to put my notes about packet capture here today but I have got an issue with my ESX server file upload component. I kept receiving “I/O Error” during a file upload to datastore directly or big (e.g 1-2 GB) ova file deployments. I took several captures with no positive result in the end. I was thinking that it is the problem of ESX server but when I connected my vsphere client directly to the SRX box, I saw that neither SRX nor ESX server is the culprit. Some device in between my client and server behaving abnormally which I couldn’t figure out yet. I will keep troubleshooting:) Now I would like to put my notes here to easily find them again how we can take packet capture in a branch SRX device.

1) First set the capture file and maximum packet size. (It is better to set 1500 not to miss any part of a packet)

2) Apply the capture on the interface you would like to collect traffic in both direction

Once you commit you will see that capture files will be collected under /var/tmp/ folder:

Packet debug in SRX

If you want to debug a packet flow you can use the following config by which testdebug.log file will contain icmp traffic debugs.