JNCIE-SEC: Traceoptions & IPSEC troubleshooting

In IPSEC topic, I am continuing with traceoptions and troubleshooting section. In this post, I will try to explain how I troubleshoot IPSEC VPNs mostly initial setup.

IPsec VPNs


  • Implementation of NAT
  • Source NAT
  • Destination NAT
  • Static NAT
  • Implementation of NAT with IPSec
  • Overlapping IPs between sites


One of the challenging parts of JNCIE-SEC must be the troubleshooting part for which I need to understand under what sort of problems what type of error logs are generated. Because of this, I enabled IKE traceoptions and simulated several type of possible problems and observed the error logs.

But first let’s see how a successful IKE Phase 1 and IKE Phase 2 log looks like;

PS: All errors below are between ike peers and


Phase 1

You can see the “IKE negotiation done” log in here.

Phase 2

You can see “QM; MESSAGE: Phase 2 connection succeeded” message.

For this specific connection here is the CLI outputs;

ERROR 1: “IKEv1 Error : Invalid payload type”
If your pre-shared keys aren’t matching you will get a similar error log like below.

Error message “IKEv1 Error: Invalid payload type” is a likely indication of a pre-shared key mismatch. You can also see “Error text = Incorrect pe-shared-key”

Error 2: “IKEv1 Error : No proposal chosen”
You will get the following error if one of the followings mismatches in your IKE config;

  • dh-group
  • authentication algorithm
  • encryption algorithm

WARNING!!!: In addition to these mismatches, you will get the same error under the following conditions

  • if you forget to set “bind-interface st0.0” under your vpn configuration,
  • if st0.0 interface isn’t created with family inet and/or assigned to a security zone
  • if you are using routing instances, also make sure st0.0 interface is assigned to the right routing instance

ERROR 3: “IKEv1 Error : Timeout”
If IKE port 500 isn’t reachable at the peer, you will get an error like this. Another problem you might encounter is that for example, you forget to enable IKE service in a zone only in one peer (e.g Peer B) but Peer A is still allowing IKE. This means peer A can’t be the initiator but only responder. Because A can’t connect to IKE port but B can.

ERROR 4: “IKEv1 Error : No proposal chosen”
This is the same error like ERROR 2 but it is actually caused by IPSEC proposals not IKE. Thus if one of the following two mismatches, you will get this error.

  • Authentication algorithm
  • Encryption algorithm

Note: I had thought that ipsec lifetime is also something that has to match but my tests showed a different result. As far as I can see peers agree on the lowest lifetime configured i.e if peer A has 3600secs and peer B has 7200secs, they agree on 3600secs.

FLOW Troubleshooting
So far I have done IKE troubleshooting. Now I will do some flow troubleshooting.

I enable traceoptions for the traffic that I am going to generate.

First two packet filters will show us clear text packets but outgoing-esp is for the encrypted packets

Let’s send 1 ICMP packet with 1000 bytes (1008 bytes with ICMP header)

Now examine the file ipsec-traf.log

outgoing-filter match

What happens here is ;

  • packet with size 1028 (extra 20 byte from IP header) with identification number 3812 matches: packet [1028] ipid = 3812
  • A new session is created “flow_first_create_session”
  • self-traffic-policy allows the traffic as it is locally generated
  • Tunnel id is identified for the traffic “flow_first_get_tun_info: tunnel out 0x577cf0ec, tun id 131073”
  • Physical outgoing interface is chosen: “choose interface ge-0/0/0.0 as outgoing phy if”
  • and finally packet is encrypted “flow_encrypt: tun 0x577cf0ec, type 1”

outgoing-esp filter match

In this filter we can see that:
Packet is in the tunnel and grew in size to 1080 bytes (i.e total length of the new IP packet with ESP header and encryption) and with outside ip id 405

incoming encrypted traffic
I have seen that though I haven’t configured returned esp traffic filter, outgoing-esp filter caught this traffic.

  • Return traffic enters from ge-0/0/0.0 interface : ge-0/0/0.0:>, 50
  • So called source port: 58759 and destination port: 3018
  • It hits the flow session with id 1 which is a one direction ESP session
  • I hope to have covered various scenarios in this post related to traceoptions and troubleshooting of IPSEC VPN sessions. In the future posts, I will do some troubleshooting if required.

    If you have any other error you have received which isn’t covered here, please do share.

17 thoughts on “JNCIE-SEC: Traceoptions & IPSEC troubleshooting

  1. rtoodtoo Post author

    Hi Philip,
    I haven’t worked with Appsecure much actually. Maybe in the future who knows:)


  2. Didzis Ozolins


    I’m having trouble with SRX IKE debugging output.. I can’t get the same detailed output as you are showing. I’ve turned on all the traceoptions (level 15, flags all, etc), no luck so far.
    Your output is from SRX device?


    1. rtoodtoo Post author

      To be honest, these outputs are from X44 release which might produce more verbose output but I didn’t really compare the outputs between different release and yes output is from SRX device.

    1. rtoodtoo Post author

      I don’t write specifically for ENT exam but I do several tests for routing. I am mostly interested in security indeed. Security itself is huge to digest anyway!

    1. rtoodtoo Post author

      I think it isn’t easy to say what went wrong here. In trace file, in a normal rekey, you see ike_get_sa to find the available SA between this peers and normally it should be followed by “ike_sa_find: Found SA” which isn’t your case. It seems there wasn’t any SA for this ISAKMP message. Maybe an abrupt shutdown on one side.

  3. Vlad

    Hi rtoodtoo,

    Thank you for your tremendous posts. I read them all the time and find information that you provide very helpfull in my day to day work. I was wondering if you were planning to make a post for Ikev2. Im running into the following message when trying to force Ikev2 only. It works fine when in ikev1.

    [Mar 24 12:49:12]IPSec negotiation failed for SA-CFG xxxxxx for local:x.x.x.x., remote:y.y.y.y IKEv2. status: Invalid argument

    and it generates the following syslog:

    %DAEMON-3: IKE negotiation failed with error: Invalid argument. IKE Version: 2, VPN: xxxxxxx Gateway: xxxxxx, Local: x.x.x.x/500, Remote: y.y.y.y/500, Local IKE-ID: Not-Available, Remote IKE-ID: Not-Available, VR-ID: 6

  4. David Mackintosh

    iked_pm_id_validate id NOT matched.

    I was doing a VPN with a Cisco running ASA 8.0, and it was expecting IKE-IDs by default, and so the options for the same were not present in the Cisco’s config. The result was a phase-1 that appeared to come up and then drop 2 to 5 seconds later before phase-2 could be negotiated. It wasn’t clear which side was doing the dropping.

    The solution is to set the general IKE ID option:

    set security ike gateway $GATEWAY general-ikeid

  5. lutfe habib khan

    Do you have configuration sample for IPSec between MX and SRX???


You have a feedback?

This site uses Akismet to reduce spam. Learn how your comment data is processed.