Why Did Facebook Go Down? - Computerphile 

What's a phone book you ask? Well, it's like DNS, but for people and their telephone numbers.

The world briefly became a better place

Ironically, Facebook's mission statement is best fulfilled when Facebook is disconnected from the Internet.

it's like Stackoverflow going down and their devs not having where to search for the fix

Love the Apple magic mouse in its natural position, upside down, asking you to recharge it.

It was beautiful, but as everything beautiful it didn't last long.

imagine a world where it took days rather than hours to fix personally, i think this timeline lost

The phone book names are the same that the teacher calls in "Ferris Bueller's day off". Brilliant

One thing I learned the hard way on this, whenever working on a remote router always set a timed reboot on the router before you make the configuration changes. So if the worst happens you just wait for the reboot into the saved configuration and not the running one. instead of jumping in your car to the remote location.

It may have been a throwaway question, but i was talking to my 11 year old nephew a couple of weeks ago and i mentioned a phonebook and he didnt have a clue. His best guess was a diary that you write your names and numbers into but he couldn't get his head around the idea that we used to get a catalogue of everyones phone number posted out to us every year.

What's a phone book, grandpa? "Well, an old philosopher put it like this: it was the great book of doxing. It told you the names, numbers, and sometimes locations of everyone in town."

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair." Douglas Adams

Whoever slipped in that Bueller Bueller Bueller joke in deserves a raise.

Today I learned that with the BGP protocol, you have to constantly scream that you still exist, or everyone will assume you died.

We need to start explaining things the other way around: a phone book is like a DNS for phone numbers.

I love how you're talking about DNS, IPs, ISP and several technical terms but need to explain what a phonebook is

A phone book is like a DNS server that's updated every 12 months.

I don’t understand how Facebook didn’t have an out-of-band back door in case their ASN completely vanished from routing tables everywhere. I mean, these guys helped write the book on data center infrastructure Best Practices

I like that FB designed things to lock themselves out of FB.

This reminds when I worked on tech support at an antivirus company. The IT admin of a mining company called us that he had accidentally pushed the wrong configuration out all of their computers, including the mission critical ones down in the mine. I cannot recall what his config was meant to look like, but the wrong one was set up to run a total scan of all hard drives every 10 minutes. Since a single scan took significantly longer than 10 minutes to complete, many parallel scans were running, slowing the system down even more and every 10 minutes a new scan job would start. We had to use a special, last resort tool, which would kill the antivirus kernel, but getting hold of it was hard (it was given out only with a special permission). My advice for that IT admin was to try new antivirus configurations on his own PC first (or get a sacrificial one for tests), then push out the config to 10 machines (preferably of tech savvy people). If there are still no problems, push out the config to 20-50 more computers, and so on, but never ever do it to the entire network.

it's kinda wild to me how people LOST THEIR MINDs over this. Back only a few years ago, stuff would go down and you'd just say, "idk, shit's down right now, I'll try again later"

People thought that their internet went kaput during Facebook's outage, even my ISP had to clarify that it was Facebook's fault and not their network's. Just shows how large Facebook is.

So could one make an analogy that it's like a company deleting their address from their website, revoking it from Google maps and any other database? The roads could still take you there, but knowledge of where it is quickly dies out

The national network I have worked on since 1975 used to have a thing called "the Shout-down". It consisted of analogue audio circuits over copper only, between major control rooms or what-have-you. Everyone had to call in every morning to check its integrity. It was intended to circumvent network failures and keep communication alive robustly between these places. (As a young man then it reminded me of those old marine speaking tubes from the early 20th century; Phweet! "Half astern!" "Aye-aye, Captain! Half astern it is!") I knew the rot had set in when about a decade ago some genius decided that it was old-fashioned copper tech and they moved it to a VOIP connection.

I also recall phonebooks having addresses and advertisements for local companies.

It's remarkeable that one update affected all their servers. I imagined their EU and NA networks for example would be kind of seperated.

When I was studying to be a software engineer we had a semester on networks. We were assigned network equipment and tasked with setting up a network with security, services and so on. It was common for students to update the router configuration resulting in cutting off the connection that they were using to configure the router and having to get help to reset things. Almost everyone did this (including our group) and it was like a rite of passage.

'A JCB through a fibre-optic' - such an understatement and incredible how often it happens.

What I found really interesting is how long it took to get back. You're absolutely right that there's no way to fully prevent something like this from happening (and let's assume they did everything they could), but if you can believe the rumours about keycards, locksmiths, and angle grinders, they probably could have designed their physical security in a way that makes recovery in the event of a catastrophic failure a lot easier than it was.

Somewhere out there there are now two people one with a C64 the other an Atari 800XL with dial up modems. Their whole job is to call into the secret Facebook BBS interface to jump start the network the next time this happens.

A minor correction is that iBGP is not a separate protocol for use internally but just part of BGP that is called Internal BGP (iBGP) when the peering between devices is within the same AS (autonomous system). An AS is defined by its AS number and if they are the same then that’s iBGP.

Whoever did that, thank you for making this world a better place, even just for several hours.

It honestly couldn't have happened to a better company.

I'm 20 and clearly remember going through phone books as a kid

BGP updates very sloooowly... and is full of wonderful little quirks and traps for the unwary, which made it so much fun learning nearly 20 years ago....

Everybody has a robust resilient network until the day it goes down. Same with security, everyone has a secure network until it gets hacked. One thing I have see in IT big time is that they develops all these processes, procedures and policies to deal with emergencies and disasters but rarely do they ever test it.

Their badge access failed because they had no route to their service provider. A local authenticator or “break glass” process would work. Physical structures have “Fireman” mode, allowing access to First Responders.

The phone book entry "beuler, f" is awesome 😂

Another amazing video. I love that computing is such a huge topic that most experts only know about their wee part of it, except Steve seems to know everything about everything lol

"Let your fingers do the walking." A phrase more apt nowadays than when it used to advertise Yellow Pages.

this is amazing how clear you can explain it that clearly ! I don't know much about tech and still managed to understand the majority of the video :D

It's like the South Park episode when the Internet went down, but for Facebook. :D Edit: I remember a story from the husband of my sister. He is a Cisco architect working for a very big and well know company at the time. One day he came back home laughing, because one of his colleagues made a mistake when entering IP addresses. Apparently he copy/pasted the wrong address and caused a power outage in a city in China.

Fascinating. Thank you On a different note, your laptop on the edge of the desk was stressing me out!

I have learned the hard way: "Reload in 5". If in 5 minutes I'm not able to cancel the reload, that's my lifeline.

Loving all the retro computer goodness in the background. I'm seeing an early IBM PC, Atari ST monitor, Apple II and BBC Micro. Sure there's other stuff too.

We're destroying evidence just as fast as we can, and expect our services to be restored as soon as possible. We apologize for the inconvenience ...

This reminded me of an event that happened here in the US back in something like 2014. An engineer at Verizon made some bad commit to a routing configuration and brought down a substantial portion of the internet in the States. That was a fun day. On October 4 2021 the sun shined a bit brighter.

That is so true. Federated approaches are much more resilient to these sorts of events.

What are the odds of a once in al lifetime catastrophic failure happening exactly when the company was embroiled in a scandal?

The scariest thing for me, during the outage, it made me appreciate open-source, decentralized software even more, because, I get the advantages of having a centralized infrastructure allowing better monitoring of the services of the various products of a company, allowing better communications with each other (integration)... But this is the edge case scenario that reminds us to not build everything under the same roots

Anyone know what the BGP path visualization software was? That looked pretty amazing!

The clarity of the explaination

11:00 And this, my friends, is why we always keep two cups and a stretched wire as a communication backup.

Love the retro hardware in the back! For my part, I always kept a physical key to our company building because I did not trust the access system. We are now in a new building completely run by keytag accesses. I guess i'm going to add a physical key to my request list to the new building operator and thank Facebook for giving arguments in favor of this.

Almost 2 million. Congrats guys.

The thumbnail for this is pure genius

At least *NSA* "Prism" network was still statically *connected* .

"The thing that's probably happened here is that someone has had a bad day at work, they've pushed something up to their servers and to their switches and things, and it just didn't work." Now, that was detailed, unexpected and fact-based! Someone then thought of disconnecting the Molex cable but it didn't improve things much.

Brilliant n00b-friendly explanation and hilarious thumbnail 😅. This made me think of some kind of thriller movie, where a technician gets trapped in a massive datacenter in an outage like this. So they gotta go all Gordon Freeman to get through the complex's security systems - to get the servers back online in time, before a bomb goes off or something idk 😂

Thank you Mr Philip Seymour Hoffman for your outstanding explanation!

Really well explained - thanks. Having recently retired after years of data networking, my first thought was for the poor guy that pushed out the change ... and whether he (and his boss) will still be employed next week. I hope they (fb) learn from this and maybe segregate their networks a bit more ... especially (if the media stories are to be believed) the physical security and building access networks so staff can at least have a chance to start diagnosis

Reminds me of the AT&T long distance failure. Cascaded due to a code issue.

Thanks for the great explanation! I loved that wire diagram.

Hey, almost 2 million subscribers, great job, computerphile!😀👍

For a short while, the world was simply normal. Wish FB stayed offline for good.

It was quite interesting to see how much of facebook and instagram is buffered locally on the smartphone, I didn't spot it being offline until starting to watch a video and it failed mid-way._

Cool, an Acorn RiscPC, just sat in the background!! One of my favourite machines of all time. I own a StrongARM SA110 upgraded RiscPC 600, which I use, occasionally.

Loved this analysis. Watched it at the time and just rewatched it in light of the optus outage here. Dr Steve is there any chance you could take a look at that using the same tool to see if it's the same problem. Thx

Pushing your new code to production that you felt like "it was just some minor errors that don't have be re-tested" be like:

They had to reboot to deploy the skynet version

Great video! What is the software you are using to display the network connections?

Thanks for this intresting video. I knew DNS but needed a bit more context to this!

It's funny how at the end they seem to be saying it's impossible for Facebook to create something that doesn't have a single point of failure.

Email is exact example I was thinking about during whole video! I think it would be nice topic for Computerphile to discuss: open protocol based services and systems, with various implementation such as email is, vs. proprietary and centralized ones like popular instant messengers (from ICQ, MSN and Skype to modern such as WhatsApp, Viber, FB Messenger etc.) or whole platforms like Facebook, Instagram are.

One video that best explained what really happen with WB. Kudos @computerphile

There's a hint of Facebook's arrogance here. "They wrote the book on best practice", people say. They're world leaders and have extensive backup systems, apparently. And so on. The thing is, their perception was that they could put all their eggs -- external, internal, backup, everything systems all in one notional basket. I run the tiniest of networks and even I think to put some systems "out of band", so if the unforeseen occurs and kills my primary systems, I have a little alternative that I can reach/access/use. Note my use of the word UNFORESEEN. There's your "best practice" -- write the book, follow the book, and know there is more BEYOND said book.

You make good videos. Looking forward to the next one!

2:43 "You see...a phonebook was like a physical DNS server, where you had this book that you would look up the person's telephone number by their name"

Very interesting and clarified many things for me, thank you!

Awesome explaination!

IT person: "Mark, our servers are down!" The Zucker: "Have you tried turning them off and on again?"

We had something like this happen to the Australian internet a few years back. The internet provider Dodo (who I worked for years prior) sent out an update that forced all traffic in the country through Dodo's routers, slowing down the entire country's internet.

I love the fact that he has an old IBM PC running on the background, an original Mac keyboard sitting around, and a big old CRT on a shelf, too.

Thanks for the video! I have one question. I thought BGP just published changes. I mean, BGP once a route is published, there is no need to announce it again. Are you sure it is correct that thing that Steve is saying "Facebook stopped announcing...." (min 8:30)? This would mean that border routers are continually announcing their routes. Thanks again. Your videos are so nice! Love them!

The guy who pushed the wrong config got probably very hard time after he realized what happened. I wouldn’t like to be in his shoes. Poor guy.

What program did he use to seed the network paths?

The "Ferris Buller's Day Off" thumbnail is awesome.

@computerphile at 7:33 when You refer say cost are you referring to the number of lost packets?

I could actually feel the positivity returning to people around me and people engaging more with eachother. Can't we just end Facebook?

This incident is a good argument against the IoT. If a single-point failure can knock out your phone system and lock you out of your own building, it's a real vulnerability.

I would love to be in those retro meetings.

Really informative! Do you think you could make a similarly in-depth video on how Apple’s Private Relay will work/works?

Would have been nice to add ASN numbers in the explanation. You could still keep it fairly simple and explain how BGP4 pathing works.

Every person I know felt better without facebook, they realized that there is so much free time :)

This was a great video thanks!

Was that DNS routing map a website or some desktop app? Is that or something similar available to the public? Thanks.

Isn't the DNS component a bit of a red herring? Surely if the Facebook hosts are unreachable due to BGP announcements (or lack thereof) then it doesn't matter whether DNS is functioning.

The "coincidence" of the whistleblower revelations and Facebook going down is remarkable.

Excellent review thanks.

Where can we see the connection visual on the PC for ourself? That's pretty cool.