Panorama address object mismatch with firewall

Panorama is a nice management tool. It is nice compared to NSM and Security Director:) On the other hand, I had to deal with an issue which is address group content on panorama was different than the firewall. Here is an example;

Panorama had AddGroup1 = Addr1 , Addr2, Addr3
Firewall had AddGroup1 = Addr1, Addr2, Addr3, Addr4

Security rule (Block_IPs) referencing AddGroup1 address group object had the action block but we needed to delete this Addr4. I don’t even want to think how this sync issue happened. The problem is that panorama pushed objects are read-only, you can’t delete them. What did I do?

I first unchecked this useless (at least for me) setting as I wanted to delete the address group on the firewall and re-push from panorama. However in order to remove the referenced object, I first had to delete the Block_IPs rule. That is what I did. I removed the Block_IPs rule on this firewall from panorama. I had a wrong expectation that AddrGroup1 will also be deleted but it wasn’t. Apparently my PAN knowledge is getting worse. Anyway, then I pushed the same rule Block_IPs once again and then hey!!!

 > show config pushed-shared-policy | match Addr4

On the firewall, I didn’t see the pushed object any more. Then I checked the traffic logs but traffic was still being blocked hitting the same rule! Unbelievable…

 > show running security-policy

Then I checked with this command to see if the data plane has the updated config without Addr4 IP address. I didn’t show the address either. Why the traffic was still blocked? I have no idea. After this I took a bold step and disabled policy objects from panorama on the firewall under panorama settings.

My intention was to copy everything from Panorama locally, delete these rules and objects and then do a fresh push once again but due to this Unused object sharing checkbox, all the unused objects were residing on the firewall, cleaning them up took me really time. Finally I got it working though. I am writing this for a few lessons:

• Shared unused address setting doesn’t seem to be a useful one. People have local firewall rules as well and sometimes use panorama pushed objects on their local firewall rules. Apparently this isn’t really good in the long term. I just don’t feel comfortable with this approach.
• Disabling panorama policy and objects isn’t easily reversible as far as I know and must be used very very cautiously. If you want to enable the objects once again, you will have bunch of duplicate objects causing commit failures.