The CrowdStrike Problem Isn’t A Simple Fix… 

Sorry about the frame rate issues, CrowdStrike took down my main recording rig and I had to do this on my Mac :(

This is the best named company in history. This is the exact same outcome as if the entire crowd went on strike.

To everyone cleaning up this mess: my condolences, may your weekend rest in peace.

When I got multiple calls at 2AM I knew this was going to go down as one of the worst days in recent IT history.

Imagine having "I broke the planet" as a hold my beer anecdote whenever you and your colleagues start trying to one-up each other on times you screwed up at work.

The largest disruption in human history caused by a missing try/catch block

Definitely a "zero" day problem. The only things saving CrowdStrike from a class action is most law firms are Windows users too :)

I literally just turned down an offer from Crowdstrike two weeks ago in favor of another job offer…it was a tough decision to make at the time but now it’s definitely looking like I made the right decision! 😬

Crowdstrike is ransomware, they just have a different payment plan. You pay up front for the privilege of being ransomwared at some unknown point in the future. Turns out the unknown point in the future was today! Surprise!

Sending out a sys file filled with nulls looks to me like sabotage

I mean, all the malware also targets windows because that's the big user facing desktop OS.

They failed to do a smoketest of their agent after build but before deploying it worldwide. Sounds like their software and update development process is just really not up to professional software engineering standards. At Meta, we had to have other engineers, sometimes multiple, review diffs before they would be accepted. And then there were multiple layers of CI/CD testing before exponential deployment with canary testing. You don't just push new code to all the machines all at once, because it's way too dangerous.

I literally spent my entire friday manually fixing computers and explaining to people at remote locations how to fix their computers. Our entire IT department became helpdesk because of this update. You don't know the pain of explaining to a non-tech person over the phone - how to make a bootable USB, boot to it, and then enter their bitlocker recovery key so they can delete a file via command prompt until you've done it personally. I got to do that dozens of times on Friday and theres going to be lots more of this for the foreseeable future... I cannot express how much this sucks to fix even though its a relatively simple fix. It just can't be automated and it's horrible for that reason.

I mean, Windows might be the least secure how most people use it, but there's another huge facet to why it's the target of randsomware: it's absolutely dominates the end-user/workstation market, especially when you are wagering the victim can't just restore from a backup and ignore you.

The problem with the bootable USB thing is that a lot of corporate devices block booting from USB by default, which means the IT Team would have to tell the end user the BIOS password to get into the BIOS to change the boot order to enable booting from USB. Its a total nightmare!

the fact that one company can take everything down like this is scary, one bad actor and this could've been a mass malware attack instead of a simple driver error

Where in the hell is the testing cluster they should have deployed to first? CrowdStrike should deploy their Falcon updates to all their own machines and if they don’t BSOD after a week THEN release to the entire galactic empire.

I was dealing with this today as an IT tech. Oh boy, What a joke. Took around 20 minutes on average per person affected. Since hundreds of devices were affected it made for a long day... :(

hey grandma, all you have to do is start up in safe mode, grandma? Grandma?

uh, no. Crowdstrike on mac is just as deep, and slows down my work mac just as much.

"Windows is the only OS that is insecure enough to have problems like this" Let me tell you why that's bullshit - Crowdstrike did this to our production Linux fleet back on April 19th.

loved the title "The day the world went blue"

Our company installed CS on thousands of Windows clients recently. A few weeks ago they uninstalled it because it was causing massive system performance issues. Giant bullet dodged. On the Windows side, it's just nuts that any driver causing continual stop errors is not auto disabled /quarantined by the OS.

Let's not forget all the people who probably put their very important bit locker passwords... inside of their bit lockers.

Root cause analysis - 1) Using Windows for mission critical work in 2024 2) Terrible code that somehow made it past code review 3) Build system that corrupts files 4) No validation checks prior to rollouts 5) Rollouts to the entire install base rather than a staged rollout

They don't want to apologize cuz they don't want to admit fault and open them up to lawsuits.

Linux versions of Falcon also hook into the kernel. Talking about how this fuck is somehow because it is Windows is disingenuous. Why can't we just blame Crowdstrike and just Crowdstrike instead of bringing up and blaming Microsoft?

Crowdstrike painted the town BLUE !

My favourite part of the disappearing air traffic example is that while they will occasionally get crippling downtime from their infrastructure, Southwest still running primarily Windows 3.1 with a sprinkling of Windows 95 here and there rather isolated them from the CrowdStrike issue.

This is going to be a LONG weekend for some folks in tech

Crowdstrike is the company that shuts down your computer when you're hacked, and they wont allow it to turn on until they check manually how bad is the hack...

"Pretty much every PC in the world just BSOD" - Incorrect, only PC's that ran CrowdStrike.

There was an issue with CrowdStrike on Debian several months back, that caused the OS to not boot… This isn’t the first time CrowdStrike has massively broken an operating system

its shockingly inept. the fact they did a rollout with no percentage based rollout and metrics on how the rollout was performing and no rollback plan is literally insane and then on top of that the fact their deploy pipeline distributed this with tests either not run or failed.... makes it seem like theres more to this story than just oh we just pushed some bad code. There are safeguards in code and policy that should prevent this.

One point to make, (I am a Mac user so don’t come at me 😂) Microsoft has the biggest market share for desktop by a large margin so make sense for hackers to focus on them, not sayings it shouldn’t be more secure but also theywill get bigger focus from hackers just due to market share

If you search hard enough, you'll find that there were two Linux Distros that got a bad update from CrowdStrike that resulted in the same issue. I think I read that it was cases of kernel panics in this case.

its insane that crowdstrike didnt integration test the update, and even more insane that mission-critical infrastructure is OK with automatic OTA patches.

Lesson learned: use Linux (I use arch btw(I use arch btw))

3:40 Not true. Falcon sensor is available for Mac and for Linux too. The real reason that it has happened on Windows only, because 1.: CrowdStrike seemingly made a mistake only in the Windows driver, 2.: Windows is waaaay more popular in businneses (both server and desktop side) than anything else.

CrowdStrike engineers: “Update is ready, let’s deploy it to the world Friday morning, and let’s test on production”

WOW. I'm a developer currently on vacation, and I'll be back to my job on Monday. My computer has been off for the past two weeks so I guess I'm lucky? Assuming right now they are no longer shipping the bad update and I can safely turn on my PC, but will definitely make sure next week!

Every single time something really shitty like this happens.. almost without fail. EVERY SINGLE TIME: Look at the education of the CEO. George Kurtz: Degree in Accounting. (no formal science, tech, engineering, or computer education of any kind... I can see he claims he can program but as far as I can tell, he's never actually worked a job involving programming or science of any kind.) The fact that this company pushed a driver rollout to hundreds of millions of people SIMULTANEOUSLY without checking it worked first tells you EVERYTHING you need to know about how this dude runs this company. If he had any actual knowledge of computer systems he wouldn't have allowed that to happen. And yet he did. When will the world wake up and start to realize that shit like this always happens when you put business people in charge of technology they don't comprehend. Now I'm not saying George Kurtz knows nothing about programming, its completely fair to be self-taught. But as far as I can tell he never did anything science-related at his job in any capacity, and his claim to fame is that he co-wrote a book about computer hacking with some actual computer scientists back in the 90s. Ever since then everyone has treated him as though he himself is a computer scientist, even though he's not actually. And after a #$#@ up this extraordinarily bad, it seems it was wrong to believe he knew what he's doing. There's NO WAY a mistake this bad could have happened without his express knowledge and instruction that this is how they operate. Stop trusting bean counters with important technology.

One thing though, using a kernel mode driver is not exclusive to crowdstrike. Many other AV/EDR systems use drivers as well. In fact a very similar thing happened with Symantec a while ago.

“Pretty much every PC in the world”… What? No. Nobody’s home computer would have CrowdStrike software. No Mac or Unix or Linux systems would be affected. The number of business computers running Windows with this software loaded is absolutely mind-boggling, though. I mean, we’re in seriously WTF territory. I hope people can get things straightened out quickly, especially for the more critical areas.

Imagine a robotaxi malfunction due to crowdstrike

OMFG, Theo thank you I had no idea how terrible this was!! The encrypted bitlocker problem is absolutely horrendous! Oh my god, I could maybe get this sorted but most people definitely can't, this is BAD!

I don't understand why everyone is so caught up over this "kernel level driver" thing - this is not built for consumer PC's. EDR solutions REQUIRE kernel level access to even be effective at catching as much malicious software as possible - it gives you such an upper hand and allows you to check and scan EVERYTHING on the machine. For a consumer non-business user device, this is super undesirable and would not be a good solution - but for a business that requires intense security to protect their data? At least for the moment, there is no other way. Those saying that Windows is the only OS unsecure enough to need this and jabbing at windows... Yeah. Just Windows things. I mean it's not like Linux and MacOS are perfectly secure, but the general consensus is those OS's are more resilient to viruses than windows. (Though it's not a bad thing to point out, that most malware is written for windows, because of how ubiquitous it is in industry.)

@t3dotgg Your description of the Windows / Active Directory / BitLocker login process is inaccurate. The bitlocker key is not retrieved from AD or other remote DB when you auth, but rather from the device’s local TPM.

Companies that run something like CrowdStrike often use BitLocker AND have take measures to block USB devices. You have to lock a Windows down way more to be ‘compliant’ for auditing

this seems like one of those applications where you'd expect every pull requests to go through a "committee" such that you don't have some one-guy write a bug into the code..

Problem with bootable USB device is that, a lot of corporate systems also disable USB for security reasons, this gets more interesting 😂

more like clown strike haha gottem

Do note Crowdstrike has a kmode driver for Linux as well. And that also broke RHEL recently... OOPS

3:34 - People always think hackers target Windows because it is insecure but that's actually not true. Microsoft Windows is the most used at 72.22%, followed by Apple's macOS at 14.73%, desktop Linux at 3.88%. Just by looking at this, one can easily deduce what operating system a sensible hacker would target if they wanted to create malware. So, its not that the other Operating Systems are secure but a matter of ROI. If you spend a month creating malware for windows you get 72% possible targets while on the other hand spending a month creating macOS malware will give you only 14.73% possible targets.

Ooooh yeah I love the voice that tech influencers have when they weren't expecting to go recording and their humanness comes out more??? Like, idk what it is about it, it sounds like you just woke up (compliment)

I’m shocked how much windows there is in infrastructure

Friday morning at the office I was jokingly asked if I caused it. The day before in the company Town Hall I announced that I cancelled the CrowdStike contract and we have mostly removed Falcon from all devices. Only two left over machines had issues.

Running bit-locker feels more like a "Hurt Locker"...

3rd party access to kernel mode plus cloud service = recipe for disaster. Crowdstrike - aptly named - soon to be a Null company pointer.

At first glance this is actually hilarious to see, but I feel so bad for the patients at hospitals affected by this. That's the worst part. Sometime like this will literally cost lives. Crazy

I find a lot of the take in this video to be misguided. Modern anti-malware software works by monitoring application behavior in addition to traditional known signature matching. The only way to get the level of access to protect at the level is through kernel modules (aka drivers). The falcon scanner for linux also has a kernel module for this reason. While there are a number of vulnerabilities in Windows, it is also the most used desktop OS. Securing desktops is a lot harder than securing a server because you have a user who may or may not do stupid stuff that you worry less about than a server. There are multiple reports of crowdstrike falcon causing linux kernel panics...they just are not as wide spread as windows.

And Hammond gets a shoutout here as well 😎. Also, great video of course, really enjoyed the take on how to and how not to communicate in such a situation.

RIP to the intern😢

Hah! That USB boot idea would be fine, if it weren't booting you into safe mode after unlocking your bitlocker. That's prime attack territory for planting a rootkit while "helping" you clean up that crowdstrike BSOD. You'd have to audit everything about how that USB key came to be and all the software on there, from extremely trustworthy sources.

The real questions that everyone should be asking is ...... What happened during, DURING the bootloop, because that's when anyone can access ANYTHING....and yesterday was an entire day of chances!

Not sure what is scarier, that CrowdStroke released a bad version or that so many companies just blindly went with it without testing and staging it first.

I don't quite get the bit about most antivirus not using drivers. Drivers have been an important part of AV and other security software going back to the Windows 95 days, I worked for a company doing security back in those days and we had a file system driver that was the core component for the file system security portion of the product, and a keyboard driver that was part of the system that ensured commands given to our system were coming from the interactive user and not from some script or application. Drivers are so important as they are outside of Windows. A normal application, even if running as a privileged user, cannot kill or modify the driver, nor can it bypass the driver when talking to the devices controlled by the driver.

Southwest Airlines unaffected. Use older version of Windows. Brilliant. Latest/Greatest not always good.

Somebody changed one life of code on a Friday afternoon, pushed the pipeline, and now we get this

Every time I hear about this it starts saying (pretty much every PC in the world was affected). I never heard about CrowdStrike before. Every person I know is unaffected and I didn't hear about any company or service affected where I live. Anyway, good luck guys.

The amount of nonsense I haven't dealt with over two decades, since switching away from Windows to Linux in 2006. I was on most versions of Windows since 3.1 in 1991 (3.1, 3.11, 95, 2k, XP, NT, 7, 8, 10, the last three for work), and Linux has been like a breath of fresh air for 18 years. It's really nice to be able to actually control everything on my system.

The take on windows being the least secure OS is a bit biased. The real reason why most cyber attacks happen through windows is because 1. It is the most used OS by a huge margin. 2. Since it is so widely used, hackers focus on finding vulnerabilities in windows instead of other OS like Linux.

The problem is the monopoly crowdstike has on critical infrastructure

I mean there's no way a company with that many skilled people rolls out a zero bytes file does anyone think this was deliberate? Theres more to this

The fact their stock price barely took a dent is insane. This is the kind of mistake I'd expect from a solo startup with 30 users, not a multi billion dollar corporation!

This had to happen during my vacation week

Basically, they over-complicated everything in the name of 'security' to the point of keeping themselves out. Bravo! 👏

It's not Microsoft bug/error. it's a Crowdstrike bug/error

Oopsy daisy 😂 good thing I don’t have auto update. All the companies are responsible for this feature

Why does a badly written driver stop the machine from booting? Shouldn’t that driver just be skipped and whatever device it targets not work? Seems like a terrible design within Windows.

(IT) "ok sir we need to put your computer in safe mode..." (User) Hold on I don't think it will fit in there.. this thing is heavy!"

alot of hospital computers are down too which is very dangerous

a null file? No it's a low level admin, it's flat out a cyberattack by a disgruntled employee or malicious actor with access. Or someone playing around and MASSIVELY fucking up. For you to zero a file is deliberate, any checksum bundling the update would've instantly failed. Anyone with hexedit can zero a file.

CEOs can't apologise to that extent for legal reasons. That proposed comment would bankrupt the company.

To clarify; it is not confirmed whether it was a _driver_ or just data used by the driver that caused the incident. The .sys extension is not only used for kernel drivers, but also data (i.e. see Windows' own KEYBOARD.sys which are is not a driver in and of itself, but contain a database of keyboard layout and P-Code sequences that a interpreter in the KEYB driver can execute) Removing the file in question has solved the issue for people, meaning it is not a critical part of the driver itself in that case.

The file full of zeros look suspicious. Could it be supply-chain attack?

At my workplace i responded to this i incident at 12:45 am, the local dispatch center had all the conputers down. I was not given approval by my higherups to run the fix until around 8 am that same day. Man did I sleep good Friday evening.

You really lost me on this one theo. There isn't anything here that couldn't happen on Linux, in fact, it did -- about a month ago. It's not guarding your bootup and preventing you from booting in an unsafe environment, its just a broken kernel mode driver. I got bluescreened. You know what happened? Windows dumped it's ram, rebooted into last know good config, i shrugged, and moved on. I can't believe this overshadowed Trump being shot to the point i hadnt heard about the latter until an hour ago

"Imagine explaining this to Karl in Sales". I spit coffee on my keyboard, and now my Windows machine is locked up.

While I hate Microsoft about as much as Apple nowadays, your take on Windows in this specific case is totally wrong and uncalled for. Only Windows was affected because only that update was affected. Windows doesn't have anything worse than Mac or Linux here. Also Mac and Linux could've been just as much affected. That whole section is very cringe. Edit: I mean the section starting at 3:35 The rest of the video is a-ok

At this moment there are probably hundreds of foreign agents planted at all levels in important companies. Imagine the damage that can/will be done when the time comes.

I am a support partner for Microsoft. To resolve this issue, boot into safe mode or recovery environment and then go to the C:\Windows\System32\drivers\CrowdStrike directory and delete the “C-00000291*.sys.” file. Then restart the system in normal mode. That should fix the issue.

Real men test in production.

Yea, being locked out by a faulty update of a security thingy dodat! JUST one question: who will pay for all the hours of work for this faulty update? Are lawyers already drafting suits? 😦🤑

My rage at everyone downplaying this for CrowdStrike is immeasurable. This is a billion dollar company, with a B, trusted by critical government, public, and private services and they shafted each and everyone. The lack of outrage from our authorities is absolutely disgusting. Speaks a lot to the state of cybersecurity and tech in general

wonderful day to run mac and linux

This being an accident is more terrifying that if it was done on purpose.

i personally think that while the emergency is occurring i don't want the CEO wasting time crafting apologies. Their goal should be dispensing accurate actionable information to help people recover. apologies can be crafted after.

Apart from obviously missing QA on the shipped update, they also pushed it to everyone at once, instead of first having a limited roll out, that could signal potential problems ahead of time, on a limited number of systems.

She wrote the tweet with ChatGPT. 😂