well now . . . 90 seconds after watching this video my own machine gave a BSOD. Perhaps your gremlin crawled out the interwebs from your system into mine. :)
When you upgrade an Active-Passive pair, you normally disable “preempt” as to control who the primary is. You never bounce both units after an upgrade as you should upgrade one at a time and rollback upon failure. As all traffic is flowing thru the primary device, you can upgrade the secondary unit and bounce it (power cycle). If the secondary upgrade was successful, then you would trigger failover from primary to the secondary unit and verify all traffic is successfully working on the secondary. If not, you would fail back traffic to the primary, verify primary is functional and stop all work. If the secondary upgrade was unsuccessful, you would roll-back to the primary, verify all traffic is still successfully on the primary and stop all work. Anytime you perform any maintenance work you should create a “Cut-Sheet w-verification steps) as well as a “Roll-Back” plan should failure occur.
I did disable pre-emptive failover. But remember, there was no PANOS upgrade involved. This was simply a reboot. However, I didn't gracefully failover to the secondary, I disabled preemptive on both primary and secondary, committed, and then simply rebooted the primary. I've never done that before. Never will again. God bless!
Will do. The really weird thing is that no other tunnels were affected. Just traffic on a specific port to this specific server. I'll have an update for next week. God bless!
Your storytelling is so relaxing to listen to that i sometimes just put it on in the background to relax while I'm doing other things. Please stay this gentle soul for all eternity and keep up the good work :)
Hey Network Admin Life. In regards to the firewall issue about the fin app VPN issue. Sometimes firewalls have issues if they have mechanical spinning hard drives. If they are mechanical HD I would replace the mechanical HD in the Palo Alto firewalls. A long time ago I did a VM lab scenario with older Palo Alto vm's versions OS and ran into issues because the Firewall's where running on virtualization platform on a external HD (Mechanical spinning hard drives). After rebooting and clearing alot of cache and reboot the main OS the configuration I setup worked. Sometimes as techs,admins, & engineers we forget physical hardware plays a huge role in troubleshooting.
Our edge (Internet) firewalls do not have spinning disks. Our segmentation firewall does have spinning disks. All the performance metrics *seem* to be okay. But, you never know.
VRF = virtual routing and forwarding. Crazy how others implement similar configurations. I do miss paloalto gear. I had to settle with fortinet and an abomination known as the Sophos firewall, a company who can’t even spell VRF! The dang thing even routes out its physical management interface! 🤦♀️ I always look back to the last change before a problem starts, and cert changes in firewall firmware is a legit source of grief. Good luck to you!
Well, actually one of our building engineers who knows very little about networking found the issue! He just happened to find a cable plugged into to ports on a mini-hub. When he removed the loop, the problem went away. God bless!
Say you didn't watch the whole video without saying you didn't watch the whole video. :-) The reboot was recommended by the manufacturer. What would you and your friends have done differently? God bless!
Seems like a bad week for firewalls.... We applied a hotfix to fix an SSL VPN vulnerability on our Fortigates this week - that fix apparently had changes to hairpin NAT in it. It took out our ERP for the whole day on Monday. That was not a fun day!
Dang. I wish I had an answer for you brother. But sounds like PA may be to blame here. Was working before the update. This is a good lesson for me, a new CCNA. Keep it coming
I've check everything and nothing seems to be wrong. Still, I was the one to make the last change (the reboots). So it's on me to definitively rule out my firewalls before pointing my finger elsewhere. God bless!
I agree. Wish I had time to do all that. One person to manage the network for an entire hospital is just not a good thing. Hoping for relief in the near future. God bless!
I was up the coast Wednesday to Friday this last week … Paso Robles to Lompoc … to refresh the networks at 9 small offices. I was thinking of you. I removed a lot of old equipment, unplugged many unneeded Ethernet cords, etc. clearly your VPN use some of them. 😅 I’ll pray for quick resolution to your issue and hope to see a video about it soon.
Wish I had been in Paso Robles, or better yet, Avila Beach instead of where I was last week. Also, my cousin used to live in Lompoc when he worked at Vandenberg. God bless!
I've conducted maintenance work that is not 100% necessary that goes bad, it's an extremely bad feeling. I always regret being proactive... It's sad that your vendor has not been able to fix this after multiple days, usually I'd hesitate running around trying things behinnd a vendors back while they are fult finding an issue, but in this instance I think I'd be at that stage...
If there was another network person here at work, I would be trying more on my own. As it is just be, paying the price of being proactive as you say, I'm very hesitant to try anything without the vendor on board. Going to talk to the head of county networking tomorrow afternoon and see if he has any thoughts on this. God bless!
Wow what an update! I wish I had some type of good advice to offer, but I’m still learning networking. Your chronicles showcase great insight into your campus. I greatly appreciate your content. God Bless and I pray that they find what caused the havoc.
Glad you enjoyed the video. I do have some breathing room now and I generally think better when I'm NOT under pressure. I'm also going to be talking to the head of networking for the county and see if he has any ideas. God bless!
Grace & Peace 2 you brother! Hope all is well. This is a very interesting issue you got here! It seems that all of the necessary steps were taken as far as failover is concerned. I'm not a firewall expert but as far as the routing is concerned, it sounds like it's "By the book". Maybe because an OS and firmware upgrade was done on the firewall, those couple of systems that were affected needs software/driver updates locally as well to run compatibly with the hardware updates, just a theory I'm sure you will figure it all out and remedy the situation, you always do with God guiding you! Well thank you 4 another great video. Be well, be safe and blessed brother! 🙏🏽
The only thing I did differently from the last time these firewalls rebooted was that I did NOT gracefully fail them over. I simply rebooted the primary and let the secondary take over. Wondering if I should have taken a more graceful approach. Well, there is a workaround in place so the heat is off. Back at it tomorrow. God bless!
Because off these things I dont like too complicated networks with many updates and similar things. But that problem with vpn is similary like in microtic routerboards , when site to site vpn disconect it needs to be manualy connected to our network thrue firewall to have comunication in 2 way. and Im not familiar with palo alto but site to site vpn on microtic works in a way that every remote site has own intenet with fixed ip and hes ovn internal network wich is different from each other network connecting with sstp vpn and user and password to our main microtic. He connect to our sstp vpn server with fixed ip , in main microtic we have 2 different network one is created to sstp vpn and the second is our main network , tho connection its goes like this their network on remote site-firewall - sstpvpn- internet- on our side internet- sstp vpn server- second network pool ip adress- firewalls- our network. the whole block is in the second network pool ips- there we go and connact 2 networks to go communication in 2 way. and the second thing what I can thing is that the procesor in main router is overloaded and he is breaking the connections.
It does. PAN support did check the CPU as part of their troubleshooting. Nothing seemed out of the ordinary. Extreme GTAC did a show khi per cpu on my cores. However, the management network bouncing may be explained by an even I learned about subsequently. I'll talk about that in the next video. God bless!
Really love these videos! I’d be looking at if there’s any snmp traps for the firewall. cpu load memory usage etc. might help to monitor those to throw it back to PA. Any way to roll back the updates and monitor for a few hours then roll forward and monitor again. 🤷♂️
That is the thing, there were no updates. There were dynamic updates. I could try rolling back to one from before Monday morning. We have had problems like that in the past. God bless!
I’m not a PA guy so don’t really understand the difference between an update and a dynamic update, wasn’t aware it’s not possible to roll back changes after a dynamic update. 🙏
