CrowdStrike Destroyed The Internet 

Man it was such a treat to finally join you on a stream -- thanks for having me, and looking forward to more! Will prep some ciphers for DEFCON 😎

"I don't always test code, but when I do, I do it in Production." - CrowdStrike

They found the ultimate protection against malware. No working machine = no malware

"Hey, boss, I removed this useless regression test to save time. It was called 'boot just one single machine"

Let's rollout a kernel level patch globally on a Friday Yolo 😂

Kudos to the one that came up with the name Crowdstrike; spot on!

They should win a Guinness World Record for blue screening the whole world 😂

CrowdStrike needs kernel space to override syscalls like reading files, mmap, etc. Rootkits and other malware will rewrite syscalls as well. There is no way to intercept calls/access memory for other processes in userspace, and AV is perpetually trying to be "on top", hence the kernel-mode drivers. All AV works like this - once it's hooked in, processes that e.g. read files will be accessing it through a rewritten fopen() syscall that goes through CrowdStrike's driver. "Channel update" means CrowdStrike's updates - they pushed a new DLL to their release channel, machines downloaded and applied it. There was some kind of error where the file that was pushed (to CDN?) was corrupted, and CrowdStrike's "channel updates" don't employ checksums, so machines just downloaded, applied it, and BSOD'ed cause the driver was invalid. Very hard to imagine how their process possibly could have done an immediate rollout of a corrupt file to everybody... Clearly not a great test engineering culture... Why is kernelmode AV needed? If I get RCE on Windows or Linux, I can install background software. It doesn't make a difference if it's Windows or Linux, but there's much more money in mass-targeting Windows machines with e.g. ransomware whereas Linux is usually more specifically targeted with 0day exploits. With AV you have a shot at preventing this without patching the software (CrowdStrike is essentially patching it without relying on the vendor); on Linux you're definitely vulnerable until you patch, but Linux also has a much better patching culture ¯\_(ツ)_/¯ Basically it's not exactly clear whether it's good to have something like this or not, but shitty software is the problem in both cases (rewrite it in Rust lmao)

What's funny is that 2 days ago the company I work in (a bank) released a post on it's internal network celebrating the acquisition of the "Falcon" tool to make the work computers more secure. I guess it was a really bad timing

Just so you know, Prime says "server" but this affects clients too. That is, hundreds of thousands of SCCM deployed laptops and workstations...if not millions. Everywhere. If you are doing remote work and your work issued laptop is running this trash, then it's hosed. But so is your whole organization.

My wife works for an insurance company as a software engineer. She and her team has been asked to report to work today (Saturday) to help the IT guys fix the PCs affected. The number of machines affected is too many for just one team to fix.

3:00 Not knowing Ryanair and being confused about it might be the most American thing I have heard in a while. It is not the Irish Spirit, it is more like Spirit is the American Ryanair. Spirit carries around 20 million passengers per year. Ryanair carries 180 million.

I'm loving this. All the times I had to explain to management why we should wait a few days before implementing an update, only to be met with blank stares. lol

I flew American home from a commissioning trip today. Luckily my flight was only delayed an hour, but there was a like 250+ft line from almost the end of the terminal up the customer service desk, and I shit you not, most of the monitors in the terminal were blue screened lol

I feel so bad for the engineer who made this mistake. He's probably going to lose his job even though there were a 100 different failure points from management, procedures, redundancy, and QA testing point of view. I would never want to work for a company like this where one mistake could literally lead to someone dead in a hospital.

0:32 "Security Expert John Hammond" Something ain't Jurassicing in my park

CrowdStrike destroyed the best Rootkit ever made*

You said the right thing: "Why are you using windows for a serious thing, in first place? "

As someone in the financial services industries, I'm too well aware of this type of software. It's essentially required to run this stuff to pass audit.

Unfortunately critical infrastructure like hospitals and government running Windows doesn't surprise me one bit. What *did* surprise me with this whole thing is how many billboards, signs, etc. runs Windows... You could EASILY power those with probably even a Raspberry Pi Zero, yet they licensed Windows for that...

First time I heard of Crowdstrike, I was on call on a Saturday night. I happened to be on the computer at the time just checking our systems. Suddenly all of our ETL jobs were failing, databases down. Turns out they installed CrowdStrike and it blocked network communications and shut down a bunch of our containers. Yeah, that was a fun (not) overnight work session. In that case, it was doing what it was suppose to. Just nobody told us they installed it.

When I hear "What is Ryanair?" I know internet has brought nothing together, Americans still live in their own little bubble and literally and figuratively there's still an ocean between us.

Back in late 90's/early 2000's a lot of European airlines were using Linux but complained after a few years that it stopped working correctly and abandoned it. Root cause of their problems lie in that they never ran updates on their systems. they somehow thought that they never had to run updates and that their systems would just continue working fine forever.

It shut down the airlines... except for Southwest who were spared because they are apparently still on Windows 3.1

My mom is the head nurse of a department at a big hospital in my city. She went in at 5 when she usually does at 8. She said today was an absolute nightmare. Like 4-5 usable computers in the whole hospital that were being shared by every department. Nurses writing everything down by hand. She said shes never seen anything like it before.

CrowdStrike and Kernel Panic on Linux happened like a month or a few months ago. So...this isn't a Windows VS Linux thing. I work at a Cloud Provider and I've seen these security solutions tear up Linux environments too.

At this point it's clear they didn't perform integrity checks on the update when sending it on the client end and there is no rollback mechanism for an update failure. The bug causing a null payload is severe, but nothing compared to a total lack of sanity checking, rollout testing and staging.

I’m an old unix /linux guy currently working at a windows managed services company. You have no idea how little knowledge, especially basic engineering knowledge, 98% of windows administrators have. Including basic engineers street-wise knowledge. And they are working with an OS which is an order of magnitude bigger and more complex than Linux. They have zero mental image of how stuff works. THIS is why this happens.

Apple had a similar problem with a content update for their XProtect a few months ago. It falsely identified iOS simulators as containing a virus and would remove them. It only affected developers working in Xcode for about a day. It does show how automatic security updates can create big problems. I unchecked the “auto install security updates” box after that.

(FORCED) Pusheed to Prod at Fridaaayyy -- Burned by its sins. In all seriousness, forced remote updates are horrible. And it was pushed to millions of users without proper testing...

2:58 "who is Ryanair and why does he have his own company; why should I trust Ryan?" I can't stop laughing 😂😂😂

Server class hardware has out of band management (HP has ILO, Dell has iDrac) which can be simply described as KVM over the network. The machine does not even need to be switched on. Many client machines in enterprise environments have similar functionality, such as Intel AMT. So no, if configured correctly, no one needs to physically visit each machine.

31 seconds ago is wild, it's neat to be in here at the same time as the scam bots for the first time in a while.

In the US, folks woke up to this, but in Australia, this all happened at 3 pm, peak hours

Jokes aside, imagine your life or the life of a loved one depending on systems like these (for travel, insurance, or healthcare) and getting stuck without any immediate resolution. Hope no one died because of this.

I drank a shot every time you said CLOUD STRIKE as the words CROWD STRIKE were on the screen right in front of you. Now I'm being rushed to ER

Crowdstrike: Security so good, it attacks itself.

You'd be surprised what an amateur hour the airport, medical and banking world is sometimes, so many "server" applications which are just a GUI running on some desktop machine. If they're lucky they get a dedicated machine, but often it's just running under someone's desk, being also used as a normal client computer.

Longtime security professional here and I must say that I am shocked by the lack of awareness around how all of this stuff works. It should be noted that enterprises run EDR/XDR agents such as Crowdstrike on Linux, Mac, and Windows machines. To be able to detect modern, sophisticated malware, you need low-level/kernel access to the machines. Enterprises manage a ton of machines and to protect our environment from endpoints (servers/laptops/etc.), we need to monitor them as users are traditionally the riskiest thing in an environment.

"Isn't EU all about privacy and security?" Privacy? Yes. Security? Not really. Expecially not enforced to a foreign country entity. That's more of a USA thing. 😅

So, to me, it appears that CrowdStrike seemingly did not test this on any actual machines before deploying it globally, think about the negligence of that move.

I’m surprised how much infra uses Windows.

The hospital staff didn't know which medication my dad was scheduled to receive today.. This is absolutely embarrassing for the hospital in my opinion. They should've never setup their infrastructure like that.

My team was in the middle of a production go live when our systems started getting struck down one by one. thankfully, my own machine would only bsod intermittently and not on boot-up. When googling the issue, I found that this wasnt even the first time crowdstrike has caused these issues (my company adopted crowdstrike late last year). There were forum posts from july 2023, and march 2023 of the exact same issue.

Effected my department. I had to go around recovering my coworkers' conputers.

5,000 isn’t that bad. My company has 7,000 workstations that will need to be manually recovered in addition to a few thousand servers. Gotta feel bad for the IT guys

✅️Confidentially ✅️Integrity ❌️Availability

It did not turn off any Internet, it turned off machines that use Internet. There were no internet outages.

This happened to some Debian servers in April, just the blast radius wasn’t big enough to make news

A lot of companies run crowdstrike or generally cybersecurity suits on linux/unix too, this is not a windows problem. And generally enterprise runs on windows because of active directory and office. Also .net and c# is quite common for monolith applications

This turned my normally pretty dead Friday morning into a hellscape. My organization has Falcon on all endpoints, and many of our customers are on Windows, and we had a LOT of tickets come in. As for servers, unfortunately there are a good bit of windows only application servers, it does suck

Prime doesn’t have an IT ops background. To him servers are ephemeral but that’s not how traditional IT systems work.

>thank the day off >I'm an IT Tech MORE LIKE ENJOY THE HELL ON.

I actually love that "anti cheat" is like a point on the scale of how intrusive something is

In grocery: my beverage company had an issue in sales, some system went down. Another beverage company, their warehouse picker system for beer went down. A grocery store(singular to my knowledge) clicklist system went down, no online shopping allowed. Starbucks mobile ordering went down (nation wide I heard).

Why does a BILLBOARD need to be linked up to a computer with windows installed? What a waste.

Y2K finally came but it was 24 years late

my phone started ringing at 12:49 am - "we are down, have BSD on many machines, can't reach the server screens", fun way to wake up. Long night,

Two questions: 1) Why didn’t they see this bug in testings???? 2) Why didn’t they push this update incrementally to a smaller amount of customers?

The short pause to slander United Airlines was cathartic. I’ve been saying the same thing for the last few years and I finally feel heard

EDR is for Linux and MacOS too. Not only Windows. EDR for linux server is the first cell to detect a security breach - as long as it works 🤣.

How can you not make a million Jurassic Park jokes?

On updating old systems to new ones: 6 years ago, when I was working at Walmart, we had someone updating our Self Check-Out machines with newer software. They updated the computers from XP to Vista. Yikes! And people wonder why our security is such a big issue.

* Ryan and John push a global kernal update * "Wait Ryan, are you seeing what I'm seeing?" "Shit."

This would've never happened if they did a internal test before they push out a update.

I'm in IT. Our servers came back up pretty quickly. The bigger issue was the endpoint client. We couldn't just write a PowerShell script and push it for a fix because none of then endpoint had internet access. We had to access the Recovery option, get into CMD, remove the bad update file and reboot manually; on. every. single. machine. (sometimes guiding our user over the phone). The reason for the shut down, from what I can tell, was not the severs being down, but the endpoints.

Mr. Hammond, I think we're back in business

Also best impression of Seth Rogen in Cybersecurity

My cousin works there and said he was on call but it wasn’t his team. Wild shit

This is possibly the best named company in history. This is exactly the same result if the entire crowd goes on strike.

Ryanair is the largest airline on earth. Known for wanting to sell stand up "seats" in the aircraft, basically they tie you down to a vertical pipe.

For my company, it was a 100% manual/in-person fix. Walking around the office for an entire day, booting into safe mode, and deleting the file.

CrowdStrike really made the dream of wannabe hackers come true

Ryan air is the airline "owned" by the US military. From what i jave heard, they are used to transport US troops over seas.

I feel left out, my IT infrastructure didn't get taken out today, all our stuff (including some windows boxes) are on-prem and don't have it installed. I still had to work.

01:00 The answer is that probably 95% of all business and B2B related software (in offices) runs on Windows and Windows only even on the Server side. Try to teach your average office worker who struggles to tell if the PC is turned on or not (when the screen is black because its turned off) to install some random Linux Software with 7 dependencies that you need to install via shell (Good luck with that one) or in other words: "Why no one cares about Linux in offices and no one ever will". And no: Ubuntu is not a good example of "easy to use" by MacOS or Windows standards that are already considered to be "hard to use" by average people. Average people dont't even know the difference between "user" and "password" when they get prompted to login. Any more questions?

IT management: we need to restrict employee's permissions for security. Employee: please approve I'll need some permissions to do my work. Security team: wtf you need that permissions. Crowdstrike: I need your super admin to install patch on your keneral. IT management and security team: go for it. Thanks so much. Given: that CS CEO was McAfee CTO who created a big disaster crashed tens of thousands of computers. That guy is much more reliable than your loyal employee.:)

Ryanair is the other cheapest flight travel provider but in EU , They might fly from airports nobody else makes money from and shure dont expect much in term of service but its cheap for students

"i'll never use linux, it doesn't have antivirus!" meanwhile, antivirus:

My employer was unaffected because we don't use CrowdStrike. I also wouldn't have been affected because I use Linux. GG EZ

Crowdstrike is the perfect name for the company that did this

MacAfee come back! We have cocaine here too!

this happened: Bullying and 'politics' in the Psycho companies... and this is the result... when 'soft skills' are more appreciated than 'technical skills.' NOTE: 'soft skills' = Bullying and 'politics' ;)

Somebody's getting fired for releasing this to Production on a Friday.

Most companies and governments in the EU are lobbied into oblivion to use Microsoft, antivirus and such. There is somehow a strong urge to be dependent on US big tech. Open source efforts are usually belittled and soon de-funded. It is quite frustrating.

Dunno why everyone's complaining about Microsoft and Windows here. Crowdstrike isn't their product, and it has Linux and Mac versions, it's just they happened to not get hit by this one. It's not like there isn't plenty of *good* reasons to point and laugh at Microsoft security and reliability: they recently took like half a year to squash all the print spooler vulnerabilities, for example.

Software engineer here. Answering your question about "why does anybody want Windows". I have experience working in both public and private sector. The rule of the thumb is as follows. Public sector will always use Windows and Windows based software. Private sector will always use Linux and open source software. Windows is part of the stablishment, and for the life of me, I can't see this in any other way than "on purpose"

How did it pass through QA checks, do they really have bad deployment setup, like they are the best in the business, how can that slip

I work on a team that runs the email system for a state and its sub agencies. This state is usually up our butts about having a test environment and running all system patches through that environment before applying it into production. So seeing crowd strike make this mistake is wild to me. Like how could it not be caught then go on to affect so many systems. THEN you have the IT personnel who turn on auto updates (I’m guessing) and just let stuff run. I know I’m speaking from hindsight here but this makes me appreciate the CAB process that I’ve always thought was a hindrance.

CrowndStrike - the company that helped to jail Julian Assange. DNC-Mails anyone?

Really, the World-Wide Fail wasn't caused by the bad update. The FAIL was in rolling out the update to EVERYONE at once -- a blitz. Had they done a Canary deployment and slow-roll, the problems would have tiny

Thinking about this differntly; what did they just install on millions of machines?

Imagine if you had 5,000 servers that you needed to manually fix, and you decided to outsource this work, and then you got malware again. 😂

The hassle was global. The company must be held responsible.

I called in sick on Friday, at like 6:30 in the morning. Went back to sleep. Woke up 5 hours later: "Oh wow, IT is globally on fire, the Internet is imploding, Windows is dying..." Went back to sleep. If there is a God, homie's got my back lol.

Our GP practice in the UK wasn't able to access medical records or see any patients today because of it

If you're creating and/or maintaining code that is mission-critical for so many people, ALWAYS test EVERY SINGLE update BEFORE it gets to any customer on a few machines that you yourself control. So that if you make a grave mistake (like e.g. causing a boot loop), you can catch it before any costomer gets it.

Sky-Net went Online.

1:02 IMHO it is companies, not people / employees, who choose to use Windows because of the features provided by Active Directory. Also, a lot of creative software does not natively deploy to Linux. Not to mention the driver headache that non technical people don't want to hear about. BSD doesn't seem to be exactly made for desktops. And justifying the pricey mac to the financial department doesn't seem to work. Hey, great to see John Hammond here!

This is 50% on Crowdstrike and 50% on the customer. Falcon admins can set update policies and allow small groups to get the latest N version while keeping the majority on N-1.