@@366KRGaming Also Telstra used their old Frame Relay network for OOB into the exchanges and IP network but that was turned off recently. At least Telstra has “Network Outage Phones” and “Outage Ethernet ports” connected to a satellite in their major exchanges so if their network goes down techs can talk to the GOC And download configs… but out of all the networks the TPG Telecom network seems to be the most reliable, they run the NSW Gov network and have clients like Qantas, KFC, Hungry Jacks and never go down
After I learned the hard way in the past, set a timed reboot on the router/switch when you making your changes, if you make a mistake, wait for 30 minutes and it will re boot back into the saved config and not the running configuration. If all goes well cancel the reboot, and save the running configuration
Yes! You quickly learn to use "reload in 10" command (at least on Cisco). Once, I was making a change to a MPLS core switch and fumbled the ACS. I managed to take down the internet for the whole company. I had to rush to the data center to pull the power on the switch and reboot it. Ever since then, I will NEVER make a major config change on a remote switch without first telling it to reboot in 10 minutes. Just don't forget to cancel the reboot!
You can use mikrotik for your management oob networks. Can leave access and ports shut, use usb lte modem on managenebt router. Use script to read incoming txt messages, can send unique code/txt to signal oob to open management ports. Can theoretically also push serial configuration via sms to routers on oob network if you can't get ip link back in.
@3:40 "Hello Palo Alto/Cisco support, can you describe the situation?" "Yeah, all the routers just shat themselves." Yes, you typed sh it instead of sh int...."
It's nice to see the physical connections for out of band management demonstrated in this video. Also the reminder to have alternate an authentication method for OOB login.
Great as always Paul. This video remembers me; In a previous netadmin job we used raspberry pi as a kind of console server with own 3G modem, bunch console cables and ser2net which provides telnet interface for serial connections. These RPis was spread across different locations and all was connected to the home VPN firewall.
Lol Optus/singtel : "Our 90 PE Routers went into safety mode when a north American node propagated too many IPv6 routes and they overloaded because that's how they come default (cough) and that's Ciscos fault" Pikachu face
Just last week I locked myself out of a switch by updating the ACL a bit too quick before thinking about 'order' and where I'm connected from. Had to do the factory reset walk of shame.
not only you need local passwords or certificates set up, but knowledge of all your important IP addresses and management interfaces if the local DNS and documentation server is down on the main network. I had luckily access to local backups of that info - and it helped me really well. Only when doing regular tests it will tell you if everything will work. The problem is to test production systems with strict SLA's
The problem with doing it this way: you need to effectively double your switching/routing infrastructure or you add another "access vector" that needs to be secured against (if using a cellular device to maintain OOB access). It might work for a small office that you're an MSP for, but good luck justifying the additional expense/MRC to the company. 🤜🤛
I was working at a WISP, where I quickly learned that the order of operations does matter a lot. I got cut off maybe two or three times, but Ubiquiti's test mode has saved my back much more times. My colleague was better though. He pushed a firmware update to a site on a Friday at 4 PM, just half an hour before the end of our shift. He did all the client facing APs first and then the uplink device, but that one didn't get back online. At 4:15 he called one of our field techs, told him the story and asked him to go back to reflash the unit locally. He was already a few hundred meters from the office, and he agreed only to return to the tower with the condition that my colleague is going to climb it (because at that site the switch was about 12-15m off the ground). That's when I learned not to touch a working device on a Friday afternoon. All the updates and preventive stuff can wait until the next working day :)
It’s a real pain in the arse when somebody’s offshore company offshores the outsource into the offshore outsourcer, and they are known next to nothing about BGP causing and outage - Only a bunch of greedy charlatans would do that
Ah i remember taking down 1/2 call centre during the day when if forgot to add “add” to the vlan command then adding a new voice vlan to the trunks. I thought the switch took a bit to come back after the first link, then did the second link and that’s all she wrote. Then was scrambling to get in contact with someone on site that has access to the comms room to power cycle switch.
@@TallPaulTech I know of a few people who have done my fuckup on production networks. I watched a vendor do a source Nat rule incorrectly (not specify the source network range, and nated everything to a private IP address). that was fun taking a whole government department offline. But we were laughing on the other side of the pond when the whole OPTUS outage. (we use them for a few of our 8 redundant links in OZ). I think a lot companies/government departments have learned a valuable lesson. Diverse paths aren't equal to having redundant providers.
@@Darkk6969 Or you can use we used to create scripts to reload the device after X mins (or at a specific time) for high risk changes. For the Juniper devices we just run "commit confirm" and if you lose access the device will roll back in 10mins. I know People who forgot to do a commit within that 10mins and lost all their changes.
Lost network connection to remote devices twice in my career. First was downloaded the wrong ip to a field device. Luckily was a test device, but still needed to walk someone through how to reset it. Second was recently. Laggy connection to a production server, meant a mistimed click happened to hit exactly on 'eject this device' for the network adapter as it poppod up. I've never seen that prompt before for anything other than usb devices, but there it was. Needless to say I discovered that prompt _does not_ require a confirmation.
Thank you youtube, this (very good and well explained btw) tutorial will most certainly come in handy when I need to manage my single computer connected directly to my single router/modem by ethernet
This is standard practice to have fail over connections where if one connection gets lost, another still exists which allows for remote access. If you're a business or whatever and you do networking stuff or whatever that requires network management, you need 2 Internet connections into two routers, and then bridging them into a single VLAN utilizing BOTH connections and doubling speed since it's shared across both networks once they've been bridged to a single line, ISP's will offer businesses with redundant networking connections by installing a second line if there isn't one there already. While this ensures Internet will maintain a 99.9% uptime, you still also need multiple network switches for routing traffic between more devices and giving them fail over connections by plugging in multiple RJ45 cables into them to also ensure that if anything happens to one of those 2 connections or more that suddenly becomes inaccessible or goes down you still have a live connection.
Out of band management is a security nightmare though. And regulators are starting to enforce them. Because it can make changes to the network. So you need to have that port only available through a session host so activity is monitored, recorded, and enforced. Ports where that is a webpage, like ipmi/idrac/I'll is even more or a pain
@@Darkk6969 yup. The https seasons have been the most difficult ones to deal with. Forcing 3389/22 thru a session host to record and manage data is easy. But when you need to deal with https, that gets a bit messy and a ton of gotchas in an enterprise environment
Ive done this mistake too many times, but youd think devices like these would be smart enough to give you a warning that youre about to kick yourself off
Or the machines think you're smart enough that you'll have a second way in... Also the chicken before the egg issue happens, you might need to lose access for a major upgrade and you can't have something reverting configuration when you have to make huge changes throughout the network.
@@jamess1787 they wouldnt need to think that because there is always a second way in via physical access, and i didnt mention anything about a revert, i just mean a warning that applying the new config will boot your current connection through this port. Maybe even list all the available ports to reconnect via and if there are none it gives you a second warning that you will lose access all together
Nah. Maybe for consumer stuff, but I wouldn't want that in enterprise grade equipment. Sometimes you just need the equipment to listen and run a command.
Interesting concept… I run a number of retail sites across the country which all are on SDWAN, cloud managed connections. If their primary & secondary internet connection is down, the site is down and no work is being done. I can’t think how out of band management would help in this case? Unless someone just no shut the interface on the router. Every other case for shutting down portions of the network would be easily resolvable once the router has a cloud connection again. Maybe for different / bigger deployments this would be more important? Would love the input.
Would it be safe you if use something like Tailscale or zero tier to say link all your sites out of band and have the authentication remote on the ZT or TS control pc ? Are there any security risks of doing it this way ?
I'm all for it. That PA-200 is on ancient code because the licence it once had has long expired. That's one of the issues with this industry. It's pretty wasteful of the hardware. It still does what it could do the day I got it though, unlike the shitty Tesla.
@@TallPaulTech the waste has gotten to a level of insanity lol. i chuckle every time i think about this last cisco live in vegas. major theme of the event was being eco-friendly and renewables....while hosting the event in the global capital of waste and excess lolol. the irony was just immaculate. what a world we live in lolol
At $job-4 for a national "junior telco" (now owned by a "top 4" aussie telco), we used exclusively other telco's services for our out of band networks. And for more remote PoPs where primary connectivity was leased circuits from "A" we tried our very best to make sure OOB was from "not A". Having said that, i also lived through the 256000 routes apocalypse which was a non-issue for us because we understood how to make sure it didnt impact our network.
Excellent walkthrough but can I ask - how is connecting the RasPi to the management port any more beneficial than being into a standard port on the Switch but having out of band connection to the the RasPi such as a 4G modem? Also I love the desk - whats it called / where did you get it from?
I don't quite get your question. If someone's connected to the management port, then it's out of band... as long as the means to connect to that device connected to the port is truly out of band (ie, via a different network).
I see you did Management and serial connections to each. The console serial connections are all that's needed correct? The connected management ports just in cause you didn't have the console connection?
Well it depends how you want to manage it. In my example I could have don the 'enable port' on the GUI, or a 'no shut' on the CLI. Either would be the same. A serial is the real deal if you have to do some nitty gritty stuff, or see the boot up process.
when i try to adjust the video quality a minute into the video the background color and the text color changes to white on mt mobile phone. is this something to do with how the bitstream of the video/audio stram is screwing with my youtube client. using offical cluent on android 12
strange a reset of the phone cleared it up. something strange with the packets must have been happening for just a minute that affected the operationand colours of the settings and editing dialog boxes. staight through my vpn and 5g connection. hmmmm. multiple times too.... very suspicious.
The Palo is just kind of sitting there, as its code is old and can't get new without subscription, so as usual it all ends up as ewaste when it's perfectly good gear.
@@TallPaulTech if i set up port 48 as mgmt vlan and connect it to the management network (separated from main network), then this should work as a mgmt port. i can only assume the mgmt port has priority access to cpu and disabled all the extra protocols/routing/gateway (but this can be done on any port i guess).
It's a different data plane. What if you switch ports have say a loop or something so nothing works with them? ... or some other thing we haven't thought of.@@olaff667
@@Darkk6969 In a production environment this is a huge no-no. This is a huge nightmare for any STIG and configuration management office. The things you find on youtube lol.
@@TallPaulTech What does this even mean? I like the youtube video, it's a great idea...not really original. Lots of small companies use out-of-band management over 5G. That said, in any major organization this is a huge vulnerability and wouldn't pass any type of organizational accreditation. This is more of a hobbiest thing but still a bad idea.
