How Roblox Went Down For 73 Hours

Подписаться 155 тыс.

Просмотров 220 тыс.

50% 1

A look into what happened behind the scenes during the longest outage in Roblox history.
Sources:
blog.roblox.com/2022/01/roblo...
www.hashicorp.com/resources/h...
roblox.fandom.com/wiki/2021_R...
roblox.fandom.com/wiki/Timeli...
news.ycombinator.com/item?id=...
raft.github.io/
www.lmdb.tech/media/20130329-d...
www.lmdb.tech/doc/
db.cs.cmu.edu/mmap-cidr2022/
• Free To Use Gameplay |...
Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution
Corrections:
- At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a *buffered channel of size 1* however the overall point still stands.
Music Credits:
- Firecracker by LEMMiNO ( • LEMMiNO - Firecracker ... )
- Impact Prelude by Kevin MacLeod
- We're Finally Landing by Home

Наука

Опубликовано:

28 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 525

@sirgamsay3596 20 дней назад

Imagine just doing a Hobby project to understand a piece of Software and suddenly the complete Roblox Infrastrucure is build on it.

@AjaxGb 20 дней назад

Open source developers: "Hey guys check out this thing I built in my spare time! It's not perfect but I'm making it freely available so other people can learn from--" Large corporations: "FREE?? 👀👀🥵🥵👀👀"

@JusticeNDOU 19 дней назад

i thought this was a joke,

@xelspeth 19 дней назад

xkcd 2347

@shantilkhadatkar1195 19 дней назад

@@xelspeth what is xkcd

@michael_betts 19 дней назад

@@shantilkhadatkar1195 webcomic. type that text into goold and youll get the comic.

@agnor9978 20 дней назад

every time I hear that someones hobby project caused a major outage somewhere I get the feeling that maybe big corporations should maybe check what software they are built on and support it's development/maintenance

@fisch37 19 дней назад

XKCD 2347

@humza890 19 дней назад

Except that software relies on another software, which then relies on another software, which then relies on another software.... It can turn into an endless loop

@agnor9978 19 дней назад

@@humza890 it can't circular dependencies are usually rare and you can stop looking for dependencies once you've seen it. I also didn't mean, that every company has to look through all of their dependencies and maintain them all, but maybe picking a few or doing an audit of some of them every now and then would be beneficial to not only them, but the world as a whole

@GoogleDoesEvil 19 дней назад

The Unix philosophy of "do one thing" and link against a ton of dependencies was a mistake.

@bookle5829 19 дней назад

It's what FUTO stands for.

@c1ph9r 20 дней назад

the negative 900 million dollars hits hard 😭

@_GhostMiner 19 дней назад

Why? Trash game has trash income

@Luna5829 19 дней назад

@@_GhostMiner its not a game tho its a game engine and hoster

@_GhostMiner 19 дней назад

@@Luna5829 **H O S T*

@_GhostMiner 19 дней назад

@@Luna5829 ackhually

@ducksongfans 19 дней назад

@@_GhostMiner plenty of trash games hosted on roblox, plenty of great ones

@k4l1hm4n 20 дней назад

Turns out, this video could be a great introduction to modern backend architecture and development.

@SrKinko 20 дней назад

I think all of his videos are a good resource for understanding different architectures and subsequently how fragile they can be lol

@wangjiefan8939 19 дней назад

I worked at a global e-commerce company a year ago and their platform infrastructure is pretty similar, down to their use of etcd and go channel spaghetti 💀

@百合仙子 18 дней назад

and a great counter-example for troubleshooting....

@bounceysteve 9 дней назад

the leaks are too

@ratm0 20 дней назад

"A massive company with ... -$924 million net income" 💀

@klafbang 19 дней назад

"Each minute of downtime costs us negative $1750, this must be fixed ASAP!"

@kosmonautofficial296 19 дней назад

@@klafbanglolol

@Mempler 19 дней назад

That is absurd lmao

@zaper2904 19 дней назад

@@klafbang So does that mean they were earning money when they were down? 🤔

@LibertyMonk 19 дней назад

@@zaper2904no, because they still had expenses (developers trying to fix the servers) but reduced income (no micro transactions available).

@erie7452 10 дней назад

Crowdstrike video when?

@ccccy-o7x 19 дней назад

Hi Kevin, amazing content as always! One minor correction @9:54 tho: Go unbuffered channel's length is 0, instead of 1, and it means the sender will get blocked until a receiver receives the value. What the video @9:54 showcase is actually a buffered channel with length 1 (e.g. result of make(chan string, 1)).

@theprantadutta 20 дней назад

This is one of the biggest challenges of modern programming, depending on various 3rd party packages, not knowing what that package is, what it does, or whether it's even reliable, and moreover knowing what are the dependency of that 3rd party package and whether they are safe or not.

@Paulo27 19 дней назад

Also never update anything

@juniorwmg 19 дней назад

*If its not a security fix @@Paulo27

@tbuk8350 19 дней назад

Or HashiCorp, being a multi-billion dollar company, could just maintain the fucking project themselves instead of blindly using a 4-year-old abandoned pet project from some random person's GitHub page and trusting it to work in a large production environment.

@ironcanon4920 18 дней назад

And that's before the issues of relying on additional 3rd party companies to supply the correct 3rd party packages. Supply chain issues the whole way down.

@yeetyeet7070 19 дней назад

github repo: "it was a toy project never meant for production" multibillion dollar company: "YAYEET"

@Mihacappy 13 дней назад

"multibillion dollar company" _-$924 million net income_

@bruhmoment5974 9 дней назад

L pfp

@mrdabup 20 дней назад

I still remember the day that it went down, people were blaming Chipotle (american fast casual chain) because they had an event that same day where you could claim a free burrito. People suspected that it was due to a mass influx of people, I knew (and a bunch of people too) that this wasn't an issue with influx of people. At the end of the day, it was a fun journey (more or less with the conspiracies, guessing correctly that it went down for 3 days months before this outage, and youtubers just milking on the outage). Thank you for making a video about this.

@frezzingaces 19 дней назад

Wait. How in tf could Chipolte's traffic affect Roblox's servers. Whats the theoretical connection?

@baribari1000 19 дней назад

@@frezzingaces it was a sort of partnership between chipotle and roblox, so if you installed roblox and did a bunch of stuff you'd get a free burrito. I think that's what it was, roblox has done tons of these

@fitmotheyap 19 дней назад

Oh this happened during that time? Man the memes about the roblox crashes during its downtime were so enjoyable

@bruhlake 14 дней назад

@@baribari1000 Yeah, it was super easy too, you could do it in like 2 minutes on a new account, it gave you a free entrée instead of a free burrito, so you could actually choose most meals you wanted. The few times they did the event with chipotle, I probably earned like 35 or so free entrees, which is pretty decent!

@argynews2825 14 дней назад

wasnt there also a massive adopt me update at the time which also probably caused a large increase of active accounts

@useruser-ti1og 19 дней назад

This is like the XKCD of all of the world depending on a toy project someone abandoned 10 years ago

@imgladnotu9527 19 дней назад

probably 2347... as someone mentioned in some comment above.....

@michaellin7936 9 дней назад

Crowdstrike video incoming in 2 years

@mahnibba2674 10 дней назад

Came here to look for crowdstrike, seems like im way too late🤣

@pdlbackup 19 дней назад

Roblox players figuring out about the DNS steering and sharing ips for early access is kinda crazy 💀

@mortred4144 10 дней назад

yo when is the CrowdStrike video coming

@TheeSirRandom 17 дней назад

Imagine how it must feel, starting a free project just as a hobby, and planning to abandon it eventually, then pretty much half the internet starts using it as an important building block to support the web. Now you're just sitting there, and have a choice to make. Stop maintaining the software, and pretty much break half the internet or keep going, getting zero thanks, and zero dollars for your work.

@Sam_Hue 10 дней назад

The Crowdstrike video is going to hit pretty hard

@levimatheri7682 9 дней назад

Waiting for the Crowdstrike outage video!

@shalodey 10 дней назад

IT global outage vid gonna go crazy

@andreyabrz 10 дней назад

Well.. now we know the next video

@sergelorenzvillasica2361 10 дней назад

Can't wait for the CrowdStrike episode 😀

@arnavn2554 9 дней назад

You gotta make a video about the CrowdStrike outage

@darthmaul5413 9 дней назад

can you do a video about the current CrowdStrike Outage?

@MaximumADHD 19 дней назад

Oh shit I was gonna suggest this as an idea, awesome to see that you did it!

@0x7f2c 19 дней назад

Lol nice you're here

@glefyr 19 дней назад

is that

@use2l 17 дней назад

@@glefyrhello call of duty black ops guy

@KieranMahoney 9 дней назад

WHENS THE CLOUDSTRIKE EPISODE COMING OUT???

@bummbumm6 19 дней назад

This happened in the middle of my friends sleepover, when we were COMPLETELY into Roblox. He pretty much just came to play it. We checked like every 5 minutes if it got better. We eventually just slept. THROUGH THE WHOLE THING Edit: Are some of you really watching videos on Roblox and just hate people in the comment section who used to like the game? Find something better to do jeez

@ProblematicParag0n 19 дней назад

Seems like you guys need to find better games

@N30ZUK1 19 дней назад

@@ProblematicParag0n Isn't your avatar from a ripoff of Minetest?

@Hellscaped 19 дней назад

@@N30ZUK1 minetest is a clone of minecraft...

@tbuk8350 19 дней назад

@@N30ZUK1 Calling Minecraft a ripoff of Minetest is the most sweaty nerd Redditor thing you could do

@dexahtheman 19 дней назад

@@tbuk8350 Tbh nothing is correct here. Minetest is not trying to be minecraft it's trying to be a general purpose voxel game engine (check out it's other gamemodes there's some pretty unique cool stuff in there)

@BananasAintCheap 19 дней назад

It’s crazy how much of the internet as a whole is in the hands of solo developers who made a thing in their spare time for fun

@nebufabu 20 дней назад

Whatever it took to make a video about a Roblox server crash and not use the "oof" SFX even once... I salute it.

@MartijnvanBerkel 20 дней назад

It's on 6:45

@yeetyeet7070 19 дней назад

@@MartijnvanBerkel gottem

@aze4308 19 дней назад

6:45

@nebufabu 19 дней назад

@@MartijnvanBerkel I stand corrected. Frankly, using it only once is even more impressive.

@vincentschumann937 17 дней назад

i read this 2 seconds before the oof sound played, well done sir

@ski3r3n 19 дней назад

the kids enter angry the kids leave confused

@TheRealStevenPolley 9 дней назад

Kevin Fang, big fan here. Please cover the clownstrike incident

@yaakovwaxman4807 19 дней назад

This is by far my favorite documentary channel on yt

@ChineseKiwi 10 дней назад

Kevin, get busy and make the Crowdstrike video 😂😢

@superbobsaget9000 20 дней назад

Thank you for all the work you put into making this!!

@iqmal 10 дней назад

Great. Hopefully you'll make a video about Windows bsod due to CrowdStrike

@HarishDoredla 10 дней назад

Next video on Crowd Strike update causing global outage!!

@ChineseKiwi 10 дней назад

It was Crowdstrike, not Microsoft

@_tylerkinney 20 дней назад

Thank you for this, been waiting for this one for awhile now!

@pompomaddons 20 дней назад

KEVIN FANG JUST DROPPED A VIDEO ABOUT THE HALLOWS OUTAGE OH MY GOD

@jakfjfrgnei 18 дней назад

The burrito incident

@pompomaddons 18 дней назад

@@jakfjfrgnei the slippery cord incident

@i-am-linja 19 дней назад

I'd imagine programmer Hell is just a bug like this which takes all of Eternity to fix, also it takes down the company's internal issue tracker and communication system.

@davidslevs 19 дней назад

Roblox is actually a bigger company than most think. Thanks for doing a video on it.

@tekratek4077 20 дней назад

Nice technical aspect of the outage!

@pitust 19 дней назад

9:45 "A default channel can only hold one piece of data at a time" It's actually even worse than this: an unbuffered channel also requires that this piece of data is received before a send can complete (!)

@patahgaming 20 дней назад

Saddest day ever for 7 Y.O i hope they can recovery from this 😢

@dagdnoob 19 дней назад

😂😂😂😂😂😂😂😂😂😂😂

@gn2b445 12 дней назад

developers probably missed out on millions of dollars too!

@rusprice 17 дней назад

Thanks! I submitted this in as a suggestion a while ago, never thought it’d be published.

@Hopgop1 19 дней назад

Man I love your videos, this was a particularly technical one, but still really well presented and interesting.

@Aunarky 19 дней назад

I'm glad you made a video on that. I had no idea how it went down behind the scenes! :D

@Komas19Gaming 13 дней назад

there was an blog post made after the outage

@zenobikraweznick 19 дней назад

Amazing CGI as always, thanks !!!

@hdgrove5567 19 дней назад

Love these videos please keep them coming!

@xFrednet 19 дней назад

Awesome summary, as always. Thank you! :D

@MohamedAruham 20 дней назад

Damn I was waiting for this one

@Cmanorange 19 дней назад

daily appreciation of kevin's visual style, i love how you're able to break down the language i might take for granted and make it easily followable

@asmith7966 17 дней назад

Haven't finished the video yet, but this already makes me feel better about the half-day internet outage I fixed at work

@poketopa1234 16 дней назад

Great great video, I seriously love the format and I learn so much

@flokibyarian6832 17 дней назад

Thank you for the great information and entertainment video like always😊

@Viniter 19 дней назад

I love this series. It's like true crime or airplane disaster videos, but it can be fun, because nobody really gets hurt. Except for big corporations and Roblox players, and well... screw them.

@hagangray8006 19 дней назад

That’s a bit harsh on Roblox players… I mean most of them are like 9 years old

@absoultethings4213 18 дней назад

@@hagangray8006if they aren’t 9 there’s a 50% chance they’re a predator or another kind of scum

@apersoniguess_ 17 дней назад

@@absoultethings4213 or… just normal people. Big shocker I know

@adityaramadhan1708 16 дней назад

@@apersoniguess_impossible😱😱😱😱😱

@enthuscimandiri1640 16 дней назад

unti money some rando get involved, yeahhh its really fun

@wormonastring6888 19 дней назад

Another super interesting well researched + explained video! As a back end game dev, thanks for the nightmares!

@passenger175 19 дней назад

Good work, these are both interesting from the tech perspective and just plain fun hah

@warw 20 дней назад

Great video!

@Dudex11a 19 дней назад

This video is very well executed!

@TheGrimravager 20 дней назад

> And probably some machine learning and block chain for good measure lmao nice

@mwalton9526 9 дней назад

How fast can you pop out a video? I think there might be something video worthy.

@ElioAllen-sb6by 20 дней назад

I like your stuff keep it up make more security related stuff!

@Evercreeper 19 дней назад

YAY GLAD YOU DROPPED THIS

@_xord 20 дней назад

new kevin fang video today is a good day

@hasanpatel9029 19 дней назад

The oof sound was a chef kiss to this master piece of video. Great work as always.

@mat-hu5ys 20 дней назад

love your vids! please make more

@JustDeeevin 16 дней назад

Tons of love for captioning your videos❤❤

@ishan6771 17 дней назад

Well done as always

@3rdalbum 13 дней назад

Another great video, I really enjoyed it. There's probably heaps of outages you can do next, but perhaps you could do a video on the "OpenOffice can't print on Tuesdays" bug?

@gareth2021 19 дней назад

great video, thanks dude

@Mihacappy 13 дней назад

Ah yes, that day in 2021 that i was working in Studio and the toolbox stopped working, and my ass almost had a heart attack because i though i got banned.

@heyjakeay 20 дней назад

yo honey wake up, new Kevin Fang video to watch while at work

@frosty4769 19 дней назад

the goat's back with another banger

@thegammingbros6231 9 дней назад

love this video makes everything understandable!

@matze489 20 дней назад

its a good day when there ia a new kevin fang video

@ImTotallyTechy 20 дней назад

In life... you have roblox (another BANGER kevin fang video, cant wait for the next)

@greatcanadianmoose3965 19 дней назад

Always love kevin fang videos... but would you mind using I Home's we're finally landing closer to the end of the video please? Thx

@vash47 18 дней назад

your videos are quality over quantity

@NoobieNoodle89 19 дней назад

I love the way that you explain these complex incidents. You deserve a 冰淇淋🍦

@fang-penlin4482 19 дней назад

Oh man, I love your channel so much. I can't wait to see the XZ backdoor video made by you, it's gonna be fun 😂

@randomazzy11 19 дней назад

5:04 I also heard avatars broke before the whole game went out, and some players were able to play roblox but most of the scripts were missing so it was pretty unplayable. Is it because the game couldnt fetch those? Wow

@ibis8566 19 дней назад

these videos make me feel like im watching a some type of CSI crime documentary

@Spiffycaius 20 дней назад

Man I remember when this happened this was crazy.

@H-E-S-C 15 дней назад

finally, a good video on the infamous outage

@brawldude2656 17 дней назад

These server incident always feels like a surgery where you have to save a person in its current form ASAP

@No-day-off 7 дней назад

Let’s go bro. CrowdStrike is giving free material to your next video.

@GardenOfUna 20 дней назад

I don't understand a single thing but I'm so incredibly curious that I want to know more I genuinely really love this for some reason

@5TC 19 дней назад

Wasn't expecting him to talk about this but man I remember when this happened

@zhitaburnurli9631 19 дней назад

Wow2 new upload🎉🎉

@CiY3 19 дней назад

Finally, a Kevin Fang video about an outage I was witness to.

@siz1700 19 дней назад

Nice! I wish Roblox never recover from that!

@mementomori8856 19 дней назад

GO mentioned! So you're telling me that I should continue to be paranoid about how every single line of code of my personal projects is not efficient or secure enough? Deal! Love this thank you!

@BeaStScoPesHD 19 дней назад

3 days to figure out that they turned on a bad feature a day ago is actually insane

@Jmcgee1125 19 дней назад

I paused the video when that perf screenshot came up. 5 seconds later I'm like "why the hell did nobody check this before?" We love lock contention.

@06NinjaKid06 20 дней назад

best roblox video

@leosh9026 13 дней назад

Thanks for the explanation dude"

@Core533 2 дня назад

Waiting for the crowdstrike video

@xSNJVideos 19 дней назад

Great video but why does it sound like an AI voice when you say "Availability Zone" at the end??

@caduhidalgo4996 20 дней назад

Baby, wake up! New Kevin Fang lore just dropped 🎉

@Sealqua 12 дней назад

I remember this happening, it ruined arsenal’s Halloween update.. a 2 part update with a boss fight and over 20 items to collect for free became a 1 part update with no boss fight and only 12 items to get. (also for free but the ones we got took awhile to get)