CrowdStrike Blew Up The Internet 

Oooouuuch

They were the cyberattack all along

You can’t perform a cyberattack on a device that’s offline now ;)

Nothing to see here, chop chop, move!!

if your product phones home and installs updates all behind the user's back, it's just part of a botnet really

Can’t get a BSOD from Malware if CrowdStrike beats you to it!

How one company, did more for Linux than Penguins on ice.

Genius!

Oops. “The best” 😮

AI level thinking there

Is that a joke

​@@lhutton1yes ?

Almost choked on my tea

This brings back memories. When I worked for a retail shop in Dallas, we would say this line when we were making fun of the vendors for not performing proper testing on the code they delivered.

@@lhutton1 That is THE joke. Because pushing test code into production is an asinine thing to do, but it's probably what actually happened.

At least you folks had a disaster readiness plan in place! Good job! Weirdest and ugliest disaster I ever dealt with was a flood - in the middle of a parched Middle Eastern desert. A large water pipe ruptured, the water running on a calcium carbonate layer under the sand, to fill manholes with telecom cabling inside, saturating the phone network into oblivion. Unbeknown to us for a week, one primary emergency generator's underground fuel supply was also flooded, the tank now full of water and eagerly awaiting the next week's power transformer failure to drop that same telephone network, server rooms and all war theater communications. Who has flood emergency plans in the desert?! I did after that! Boy, but my face was red. And the most fetching shade of purple usually reserved for the pointy haired boss... When asked to come up with an emergency response for the future, I quietly handed over the new plan.

Super safe mode 😂

@@hugohabicht9957bruhhh hahahah😅

😂

Stole this and memed it

Comment of the day

Seeing how much infrastructure relies on this makes how irresponsible a practice that is that much worse. They seriously couldn't test it?

As software test automation engineer, "Fuck it, lets deploy" - I did once when I was new at my job lol. Learned my lesson.

Never auto accept patches. Let the others take the hit first.

That's not really best practice for cyber security stuff​@@TheBooban

Well... needs to be signed by M$ and C$... hmmm

Add changes on at COB Friday’ fired on Monday

nah, you can change all you want, just don't push to prod on Friday.

Think maybe was from X Flare that happened at same time, not sure why they don't want to mention that

If you have a mature deployment model you can deploy at anytime with confidence. They obviously don't have a mature deployment model. That's the tenant behind CI/CD. You should be able to integrate and deploy at any time, e.g. continuously. The C stands for continuous in CI/CD. Small changes with continuous testing, usually via automated testing methods, can help mature your model. I'll bet their branches have thousands of commits and they keep branches alive forever. They probably don't use trunk based or anything else that would indicate a mature pipeline.

​@@BitwiseMobile Not pushing something to production on a Friday would be a part of a mature production model. There is always, always, always a chance of something going wrong. You can never completely prevent that from happening. Because of that, it is unwise to have something go live on a Friday.

Literally on the profit side...

2) apply for a job at MediSecure... oh wait.

I work in helpdesk, today was a rough day 😂

4.) Sell Company Stock

change name by deed poll, someone wants your ass

😇😎😇

Tbh this is worse than a windows update because it is done automatically, and since it was a driver it wouldn't even let windows fully load before it crashed

It wasn't a Windows Update! Did anyone watch the video?????

@@x2desmit the people it is affecting are saying it was a Microsoft update but the news is all blaming crowdstrike. I think its possible microsoft is bullying another smaller company again.

Scotty, from Star Trek III: "The More they Overthink the Plumbing, the easier it is to stop up the drain.". Accidentally in this case.

Seems like patch distribution 101. The "I'm sure it will be fine" mentality is behind a lot of major screw ups.

Any company which is IT dependent and running more than a couple of systems in their network should be vetting updates on an isolated test system. I never understood the trust people have in these software developers.

@@Bob-cx4ze I used to have this mentality when I started in IT, now I test 100 times before sending something to QA :P

​@@rmcq1999 Microsoft didn't give them a choice. They back end patched it themselves

We need system diversify ... They are focusing on DEi for people yet there are no diversity to services they are using ...

CASH IS KING.

@@oojimmyflip crypto could have been our lord and savior. but the hype&scam bros fucked it up so badly…

Whoever goes full on cashless as payment method are stupid and idiots.

Cashless? How about money free? Even if that point of sale terminal can accept cash, it won't be able to process the transaction anyway.

Cashless is not a problem, no backup / failproof rerouting is a problem. If there's no electricity your new electronic cash register won't let you use cash inside - unless it has manual override. Same with using digital resources.

True , companies are still looking into the issue

lolololol

You should see his tweets lol this is nothing 😂

Many people already use thix fix for over an hour. also it doesn't work on machines that have bitlocker - which majority of corp/large companies would be on - won't.

A official workaround was posted earlier on reddit already though,

Amen Brother ‼️Amen‼️‼️

They released the update on Thrusday btw.

zero updates for zero-days... i'm not against holding feature updates back for testing but for security critical systems like this you'd want the program detecting and blocking new threats asap

No, no! Friday morning gets the low level driver updates. Friday afternoon, at COB, DNS gets a major update. All one needs then is a bit of gasoline... The bear of it is, Crowdstrike defaults to automagic updates through their own channel. All of this angst over a damned logger.

I've been in IT for 30+ years. You'd be surprised. Special shout out to the Online Game Devs. We used to joke about how Blizzard moved to Friday game launches, after Korea took the week off to play the new Starcraft expansion.

After 30 years in development, I've removed social apps, and Google (as much as I can) to minimize problems. I'd like to go back to pen and paper rotary phones, and talking face to face.

Linux is here

Use Linux and you will say that software development is finally a science

I've also done 30 years, what seriously worries is the fact that MS pander to the American government... the guys that operate a business model based on g3n0c1de and Microsoft products are vulnerable by design and always have been. Hmmm 😑

The main reason I strip as much of google out of my systems​@m12652

@RyanClone As a fellow veteran, I approve this analogy

Dumbest shit ever had to contact a whole different team for a response they never gave , to be fair I work for a shit company

The RyanAir of CyberSecurity

Terrifying lol Hope you people never have to go to war

The fact the hospitals dont have paper fallbacks is disgusting. Our healthcare shouldn't be bound to a computer so tightly.

@@ImperiumLibertasthere is paper charting fallbacks but it’s tedious and a lot of processes take longer and coordination of care is less efficient.

​@@ImperiumLibertasproblem is, paper takes significantly more time and effort. That's like saying everyone should be driving around with paper maps. A lot of people do, but after years of depending on smart phones and GPS navigation, even those people with paper maps couldn't easily and quickly find their way.

The problem is that thanks to Obamacare hospitals were forced to use electronic medical records or they would get penalized on their medicare reimbursement by 3% and since that's the the entire profit margin they all switched a decade ago. So how do you retrain a 1k different roles from aids to physicians on process in less than thirty minutes with no access to charts or orders? Blame congress and Obama

Disrupt is gonna love this

I mean doesnt take much to see that it definitely seems like BY FAR the most expensive already. It's been affecting everything

@@raymondqiu8202 I think the company - CrowdStrike - by now is just done, they just don't know yet... Imagine the whole world filing a lawsuit against your company for damages...

This probably is. Cloudstrike stock is going down like the planes that are going down and getting grounded

What about a talking bug that lays golden eggs?

Sounds about right!!! Every boss does this 😂😂😅

Man that sucks, hope you found an alternative?

Sorry to hear. Best wishes to your father.

Sorry to hear, Hope he get well soon! ❤

So sorry. May God protect him

Really, critical services like that should not rely completely on their computer systems.

Haha same here

show me what is behind your lol

​@PriCap what do you mean?

Refer me

@@Motinha-l7c It affects mostly companys and infrastructure world wide, partuially in china and russia too...

@@marcelogonzalezdanke I work for a law firm in Brisbane. Our system was unaffected but Monday morning might be interesting!

A tech company in Melbourne, most of windows laptop experience BSOD. We have to start our weekend early, while MacBook users still need to work, lol.

I'm a coles employee in Brisbane, went to buy some groceries, but could only pay in cash. According to an old coworker, all the back end systems (there are a lot!) have gone haywire. Thank god I have today off, but praying its back on by tomorrow.

Not good, justice will get delayed. Courts should not depend on the system.

Apples anyone?

How furtuitous 🤣🤣accident by chance or just coincidence.

I might try this 😂

Not the Fix for the company I work for where we’ve been fighting with 20k down Systems. After 4 days we’re down to 1k

The bonus stupidity of that company releasing the update to everyone at the same time, brilliant. Great to know we have such security experts being trusted by corporate idiots everywhere :D

Tell that to Cloudflare being at least half of the world provider

@@Vysair The same rules apply. Centralization is bad.

@@Ungood-jl5ep And of course, the same applies to the fact these corporations exist in the first place, and affect a disturbing portion of our world. But even if "only" 1% of companies used this... the impact would still be absolutely ridiculous. _Especially_ considering some of those completely eliminate the competition around them. And all this is built on the solid foundations of "trust me, bro".

Not really. The issue is ppl. Making soft good and secure requires way more knowledge and time. Would that be my "perfect" setup, it would be back up in few minutes. Not to mention in perfect scenario it would simple not happen. Keep doing "good" job companies. 😅😅

Yep no cash no buy

People forget about that part

Handwriting and cash, exactly what ALWAYS worked. Go figure 😅

And I remember the bad old days, when mainframe computers ruled the roost. If a crap Windows update or driver for Windows update pooped the bed, worst case, they'd dust off the old terminal to the mainframe.

​@@spvillano "Finally! My COBOL studies will pay off!"

Heh, had a ruptured pipe flood manholes with all of our base telephone lines in them once. Boy, it was a peaceful, quiet day, then someone had to come in and wreck it by reporting the outage and flooding underground... Boy, but I was tempted to shut down the VOSIP server, as network services remained operational...

Not being able to use Teams is always a blessing in disguise. Teams is the worst team chat platform.

Indeed! I only use Linux so I had no idea of anything that had happened until I saw the news.

Lucky that ssh vulnerability was found otherwise it would’ve been worse with all the servers running Linux lol

There's linux version too (though it wasn't affected this time).

@@overlordmae9090 who uses antivirus with linux😂😂

@@khelben1979 my colleague told me and both were like "thats why linux"

*fist bump* I'm a phone jockey as well WFH, been enjoying the quiet, my customers call when their car breaks down or they wreck and need a tow or jump start or whatever, but I don't get new callers, I get the escalated ones that have been waiting 4-10hours and longer and I'm the guy that takes the cussing saying that Im a sup and blah blah blah, easy money tonight. I so feel for day shift, they just may all walk out and be done lol.

My work is also affected. And we also use crowdrstrike. But lucky enough my bank is working fine :).

Running Linux & didn't have a single clue till my mate mentioned it. Lmao

the things i need still work

​@@DutchPyro2011 lol same here went from Windows to Linux Mint after Windows 7 expired. Never regreted it.

@@UnhingedSystems they'll tell you that's why they pay for IT, right before they realize it's outsourced and 100% remote. This is the day freelancers have an opportunity to make BANK

Despite warnings that The Cloud might be Pie in the Sky, people were still CONvinced that putting all their data to someone else's computer was a great idea. After this event, perhaps some will start waking up to the clear and present danger of totally trusting technology?

@@antispindr8613 they laughed at us when we 1) bought our own CDs and records instead of relying only on streaming 2) bought our own Blu-rays and books instead of "digital downloads" 3) run an in-home NAS instead of paying "only a few dollars a month" for cloud storage Today, I'm gonna throw on a movie, read a book, and thank myself for having a shred of foresight. Might check Kasperky's tweets later to see how smug they get

@@antispindr8613 We're already starting to see more conversations about relying more on on-prem solutions and to use cloud technology sparingly. It isn't the magic bullet it sells itself to be. In my opinion the cloud has been a bait and switch game. We were promised it would be cheap and affordable and would continue to become more affordable as time went on. Instead we see companies with cloud products increasing prices aggressively and turning cloud services into a predatory system where they make companies reliant on them and then raise prices arbitrarily to appease shareholders. The cloud should be aptly named "Digital Landlords" because they now control the rent prices and when maintenance is needed they just pass the problems and the costs onto their customers. Shareholders want another big house or boat? Just raise prices 8-10% per year until your customers can't afford it anymore. I utilize cloud services, but sparingly. Never put your eggs in one basket.

What are you smoking? This situation is a great ad for cloud services. The cloud services all fixed this issue on their end already and all their customers are back up and running without having to do anything. Everyone who ISN'T running on "the cloud" is having a very bad day right now.

Freudian slip

Accurate

This was the company involved with the "hacked DNC server"... for some strange reason this private firm did the investigation and not cybersecurity experts within the FBI/NSA.

classic australian media being ignorant as fuck

It's lemmingware. Everyone uses it because everyone else uses it.

Assuming that Crowdstrike has the capability to do that. They control a lot of things to a certain extent.

@@x2desmit I saw in a reddit thread someone claim they pushed for their company to turn off auto-update for CrowdStrike when they adopted it and they had no BSODs. Anecdotal, but I don't see a reason to disbelieve it.

Crowndstrike have remote access for drive updates on corporate windows machine all around the world, so a employee usually can’t control what gets update or not on their windows machine. The problems is even deeper, its related to how windows OS is designed, how corporations manage their OS and more

What time to be alive

I feel you.

I agree with the guy on hackernews it's the companies fault to enable automatic update for everything(using windows for everything is pretty stupid too but that's other problem)

Imagine making your entire company dependent on a single point of failure caused by a random software running in kernel level of Windows which can push an update any time. What could go wrong lmao

@@zocker1600 exactly lol

Agreed.

Hidden in plain sight

Stormbreaker

@@AParkedCar1 Tinfoil hat is an hilarious game.

They were used by the DNC to find out who "hacked" their computers and their CEO, Shawn Henry, admitted under oath that there was no evidence that Russia (or anyone) hacked the DNC computers.

Rofl

I was about to be like "wait isn't that the thing on my OPNsense" but nah that's crowdsec. Nevermind.

Crowdstrike runs on linux as well...

Laughs in Linux and MacOS

@@ragayclark Linuix for the WIN!!!!

Big sack coming from both microsoft and crowdstrike

Digital money no work? You no eat. Brave new world.

All of the cash registers in big business run on windows whether they accept card or not.

I heard that CBA today going fully cashless by Xmas 2024

@@aidancoutts2341 I'm old enough to remember that if the power went out or a register went down that you'd break out a ledger paper and write it all down.

Banks use unix systems and other legacy systems that are okay

Reminded me of Evil Corp from Mr.Robot.

@@nmisoo exactly, maybe we are experiencing something alike :D

Yupp. Just another reason why I'm glad I'm running Linux instead. And having a live booting Linux USB helps in case needs be

It's even worse that it affects all versions. You didn't have a choice on whether or not you got the update.

@@jholden0 woah, didn't know that, well brilliant move unfortunately if it was intentional..

Same I work at dixie we're down.

No disaster recovery

Same here, working at another shipping company that everyone had heard. Half of our work laptop went bsod, but our web services are miraculously still up. We felt like we’re less affected, but I’m sure our aviation sector probably went dead.

Good

Ouch.

1) maybe. 2) doubt it. Crowdstrike is one of the best antivirus and security companies on the planet. For sure, this was a big f up, but I doubt this will be more than a blip in our memory, in retrospect. Solarwinds recovered. Equifax is still around. ConnectWise got through there mess up. Heck, McAfee and Norton are still around.

Remove the software, you'd terminated compliance for logging. Your company would be done. But, you do you, I'll pay pennies on the thousand for the old office equipment. Crowdstrike is a logging software and monitoring solution. If it's done on a system, it's logged and stored, fully searchable in their console. It's actually very good software that I've personally used in the SOC. Even chased down a Chinese government incursion that had been ongoing until Crowdstrike was deployed and we could gain visibility in realtime ongoing attacks. To get the company to spring for it, well a couple of SOX audits tend to be convincting. Long and short, with that software the attack was traced to a literally forgotten DMZ test machine and the incursion into the global network was halted. That all said, they're quite likely liable for this debacle - to each and every disserved client organization. Rightfully so, this isn't some Act of God, it's an Act of Moron. One never, ever, *ever* pushes a patch out on a Friday! For this very reason.

@@WarrenGarabrandt this. so depressing. so unfortunately, true...

​@@WarrenGarabrandtFirms taking financial loss will likely litigate Crowdstrike to recover damages

@@hojo70 Per crowdstrike's TOS (I posted a link but my comment disappeared), they are not liable for any lost revenue, business opportunities, damages, etc. beyond the remedy of cancelling the license agreement and refunding any prepaid subscription fees. They also say they do not provide warranty for the software that it is defect free, etc. There's no case that can be brought against them unless a customer operates in a country that forbids those kinds of disclaimers and limitations of liability.

There’s gonna be some congressional hearings lol. I heard that using a company like crowd strike is required by many corporate insurance companies

🎯🎯👍

I am a huge fan of open source software, yet I would still remove the "closed source" from that sentence. Diversity is necessary even in an open source ecosystem. Systems should be interoperable with open protocols, but the implementation should never reach dominant market share for any critical component.

Many ppl dont appreciate IT at all. Budget for visible stuff (painting, digging) is always higher than budget for crucial operating systems. Doesn't matter company size. The ones that got best tech for what they need is the ones that smallest that would die in a day without it. All the rest that can still get away with it, do just that. And then it happens. I have 0 empathy. 😂

Do not try to normalize pushing untested software during production.

Ironically, it's a logger that blew up. Forwards all system events to a log server for examination and analysis for if there's an intrusion or other mass event going on. Unfortunately, to gain such granular access, one needs a low level driver and apparently, this one got a modification and was thoroughly tested by Helen Keller. I've had patches sail through testing in small test groups, only to blow up in production. This one though, it's winning a booby prize.

@@pawelhyzopski6456 not even a budget issue. A borked patch for a logger's low level driver is it, all of the IT budget in the universe wouldn't prevent or mitigate that. I am glad of one thing. That I'm nowhere near Crowdstrike's corporate office or boardroom. I'm fairly certain that they're gonna need a shit ton of new paint for all of the now peeling paint...

@@spvillano That probably means the deployment model isn't mature. If you have a mature pipeline with healthy CI/CD things like that don't happen. You should be deploying to a staging environment that is identical to production. That would have found the issue on the first smoke test. A unit test should have been written already for the logger, and that would have been caught in development. When you code like a cowboy you have to expect some issues. It sounds like Crowdstrike is a bunch of cowboys.

You are a wizard, Harry

Let me guess it had a gay black woman as the hero...

bruh

The same thing happened to my great grandfather out at sea.

Me too... I was just thinking about that.

Truly befitting for a calendar bug.

Why are you making me feel old though...?

Right! This was what everyone thought might happen on Y2K. I even wrote an essay on it in school 😅

@@Tarodev Because we are old. Lol

@@nozeonfroze1 Just seasoned, 55 is old

😂

Literally!! 😂

Same here brother😂

Crowdstrike - we brought you the 4 day work week!

What if you can't work because of this? Wont be funny then

and you know it didnt "blow up the internet" when you can already see that site

​@koneofsilence5896 phones aren't affected, mate.

@@nopers369 Phones visiting websites that rely on servers w/ crowdstrike software running would be affected

Stop installing software from BG. Install the virus instead.

Not how soypedia works but ok

All thanks to AI

we had a Power cut at the local supermarket about 15 years ago, and the only person who could sell anything was near retirement. no body else could do the mental arithmatic.

Let’s go to the Old School way Folks! 🤣

@@grokitall😂❤

No it's not. Trust me, it's not. Having experienced work in retail, hospital, Amazon warehouse, a hospital, web development, no one ever said just write it down. And the only places that had safety in place for this were those that were in IT AND did their IT themselves. With tech as a tool, for most businesses it's safer to wait out until it's fixed than rely on workers writing down important info. In retail, due to regulations and laws. Unless we go back to cash, which we won't, for consumer safety reasons you are not allowed to write down information needed to charge someone's credit card. The period of those mechanical credit card machines you use to slide the inked paper over is long gone. Does anyone even make those anymore? In major hospitals not giving medicine to patient is legally safer than giving wrong medicine or a wrong dose. At the hospital I've been only white boards and clipboards in patient rooms, and front desks with their posted notes had stuff written down by hand. Again due to regulations patient information can not be written down. Because they would get sued if it came back that due to their negligence their patience had their identity stolen in such a stupid way. Amazone warehouse, would be no different than if conveyor belt died. Eventually it will be fixed. There is only so much you can do without equipment working. Especially computers. Web development, most applications and websites are not as important as their owners would like to think they are. That has to do with western business mindset. The whole idea of - if it's not making money, it's losing money - is ridiculous. Commercial websites are not losing money if they experience a brief outage. So, you spent few hours or a day with your website down, do you really think IT company will fix your problem first. Most online businesses don't spend that much on their websites as they think. So, that paper is staying in a printing machine.

Obviously 🙄

Spent 98-99 testing payroll software to make sure it handled 4 digit yrs. Lol

Oh noes! The nukes flew, the world blew up, we all died and the lions and lambs started interbreeding! Why, according to our intrepid host, even Amazon is offline - ignore that their site is up and merrily running.

Yep, everyone thinks Y2K was overblown. It wasn't overblown at all. It was a monumental success.

How many of those will have to be resampled?

i also work as a med lab and all our instruments are down. Thousands of patients on hold rn.

I work on lab equipment…should be an interesting day. Best of luck, all!

Do you have an IT guy? Reboot in Safe Mode, then type: del "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys" Then reboot the PC normally and it should be good

@@aisle_of_viewabhorrent troll attempt

Lollllllll... perfect 5/7.

LMFAO

They aren't blaming Microsoft, rather that it's only affecting Microsoft computers.

I had news on my radio say a sorts of "it's a problem only on devices with windows but it's CrowdStrike's fault"

They both are shite and not worth a good word.

It’s news outlets. What do you expect? No one knows CrowdStrike.. everyone knows Microsoft.

Doesnt Microsoft have responsibility since they work with them?

This is how stupid internet rumors get started.

@@dawnfire82 Keep being sheeple and believe the media.

@@Zagirus Keep spouting baseless bullshit.

​@@dawnfire82lol

Where I work I'm suppose to tell the customers how system issues are just updates so I can't help but feel this is a bunch of BS here and they have leaked a bunch of data. Given they are a security company it ain't gonna help them to be honest, if they can hide it will be best for that money they love....

Well it isn’t a security icident. If your device can’t boot, it isn’t vulnerable.

Reboot .... all good 👍

@@niceone2731 Availability is a major part of security - So the machine can't be hacked, great, but only if you are protecting data on it, or that it has access to. However, what if that machine is responsible for something critical? For example, patients can't get medication because the electronic prescription and patient record system is down.

@@TigCMWell I am aware about that and dealing with scalability with proper availability is a part of my job day in day out. I was just trying to make a joke, guess it didn’t land just like the flights. Oh wait, they did land finally.

Same here.

Teams is working? Damn.

I also work in semiconductor. Luckily we separate networks, so only one of my 3 computers had a minor malfunction with database to email not working.

Nvidia? AMD? Intel? As if there are more than these in the entire world lmao. Nice job trying to obfuscate it.

@@SahilP2648 Nvidia and AMD don't make any chips themselves. They're purely design firms and outsource production to TSMC. Intel has their own fabs though.

lawyers are going to be busy for a looooooong time

@@Bristolcentaurus I think they are missing out right now, they should probably be running to the airports... Lots of money waiting there to be made :D Imagine all those angry people heading for holidays... Class action material...

@@Bristolcentauruswhy are they gonna be busy?

Please explain. I've been on hold with my bank for almost an hour.

I was supposed to get paid this morning. I’m guessing this means all direct deposits are on hold or something??

@@Ausnapify My bank is working fine, its not affected with the issue.

@@Ausnapify Well, IF they are affected, they're on hold by themselves too. This issue just locks them in place, like the KRAGLE from the Lego movie. Most people, that work at a bank have no idea how to fix this issue and even if they did, they do not have the access needed to fix it. This is one hell of a ride for the IT-Department, especially if the device is encrypted, because only they have the key to open said device up. Now imagine a whole IT-Landscape being locked up and you have 3 people, that are able to unlock it, because the IT is notoriously understaffed and underpayed.

Ah, my nightmare.

Nasty. That's a bit worrying knowing my brother is in for heart surgery atm!

Holy crap!

That is crap , why are your computers connected to internet anyway ?

@@jovictor3007so we can chart. Epic runs on internet.

Well, you couldn'thave made this much of impact.

Imagine how overqualified the engineering team at CS is 🤦🏿

I can see why that’s good practice, but what if the software is a critical piece of your security infrastructure? I don’t have the CrowdStrike release notes but what if it was an urgent update that needs to be sent out ASAP?

@@lukeh990 priority 1 it and get on it asap. It still needs to follow change procedure though

​@@lukeh990update notifications are vital, mandatory updates are a sign to move to a different provider. it is basically saying that it is fine for someone else to decide your machine is so unimportant that it is fine for them to immediately stop what you are doing, and change something so you have to have a complete reinstall, and hope you have everything backed up and that everything else can wait. if you cannot see why that might be a problem, go back to pencil and paper.

A company that needs to be up to date on threat detection

auto updates

Major updates are normally done on a Friday evening to avoid disruptions to business operations.

Barclays Bank does so does Tescos and Asda.

@@andrewortiz8044 Who tf doesn't at least deploy on a test machine to make sure it boots, THEN send the update out, though? Like, I do tests for my frekking Minecraft datapack, HOW is this slipping past an actual company that pays people???

And this is a great thing as I hope it will be a wake up call to stop using Windows servers for backend and critical stuff. What I have read here traumatizes me: emergency servers are on Windows? WTF is that??

​@@TraumatreeIt's not a windows issue, it's a crowdstrike problem.

That's a "Crowdstrike" experience. Know the difference.

​@@something86783yes it is a windows issue. the whole reason the malware market exists is because microsoft ran security by obscurity for decades, resulting in the windows malware ecosystem. add to that the need for pretty much every program to need root access to every file on the machine to do the install, and you need antivirus and other windows security software to get around the insecurity model of windows. add to that that it sounds like the channel file in question sounds like a config file, and anyone doing tdd or ci writes a config file tester which is run before any config file is committed to version control, precisely so that this sort of thing is not likely to happen. a lot of this type of software does not even have a mechanism where you can defer the update, with it happening immediately it is available. if you could designate one machine as a canary, and require a successful update and reboot to publish a file that all your other machines can access to gatekeep the update, it wouldn't be so bad.

Tine to switch to Linux or Mac folks

95 percent of european companies😂

No need to, I worked for the US DoD, the overwhelming majority of systems run on Windows. Odd though that you'd ignorantly blame an operating system for the failure of an independent logging software low level driver. Tell me, have you found the any key yet? Can you tell us what a syslog server is and what it does? Don't worry, I'm worse on idiots that push patches out through their proprietary channels on a Friday, violating the first law of system patches - never, ever, *fucking ever* push a patch out on a Friday. That's basically like going to the breakroom and defrosting a fine bottle of nitroglycerin in a pot of boiling water. Nitro losing its temper easily anywhere above freezing. "Hey, on the way out tonight, prepare for the weekend and pour a few gallons of gas out onto the floors."

​@@spvillanoi still remember them having the engines on the uss yorktown (i think) fail due to a botched windows automatic update and having to be towed back into port until it could be fixed.

All major companies and other large entities like banks and governments use Windows. You may not like it but that's just reality.

Imagine being a major company and having your mission critical systems running on Somebody Else's PC (aka The Cloud). Imagine being a major company and having your mission critical systems accepting pushed updates from parties you don't control. I could go on...

Word

Meetings Mondays, patch Tuesdays, pray Wednesday, rollback Thursday, chill Fridays 😅

in their defense, they're an anti-virus company, so if a new vulnerability is discovered they probably have to deploy a patch as fast as possible. Even on a Friday

​@@ahdog8there are ways to test this stuff before it hits all of the public. netflix do rolling updates, where they pick the least active region, do gradual migration, check if it does not scale, and then roll back if it has problems. this allows them to do multiple updates per day, and if they can do it, so can others.

dude I just saw that lmfaoooooooo

​@@vilelive Yes, it references "Maybe the real treasure is the friends we made along the way"

Most sane Reddit user.

And us, IT still fixing up the mess

hmmm yeah... i mean deployment should be localized first at least 😂

Yeah, we got BSOD one after another around 3:40 PM, then my friends said their companies experienced the same issue about the same time. At the time, I know something crazy is going on……

It actually didn't, which is why it was seen as a rolling outage starting in Australia (first place with significant modern infrastructure where 0:00 occurs earliest)

​@earlnoli or make someone else try it first 😂

Well for that reason you normally have Clients run n-1 Agent Versions behind the latest Update. Sorry to say, but all companys affected did not follow proper Patch Management Guidelines when it comes to Deployment of XDR Agents

It's a loaded name

Share price is also going to get struck down tonight

They really lived up to their name, didn't they? lol

- cutting costs to the extreme - pushing AI hard, and not relying on experienced humans two main factors

Crowdstrike evolved to Worldstrike

I doubt my insurance would cover 1/4 million.

It shouldn't cost 250,000

😢

More nerve wracking that a drug costs $250,000

​@@solido888facts

don't worry bro just like calm down though everything is gonna be fucking terrible!

@@skittle-m you too broke to fly stfu

The thing is, apparently no one that uses crowdstrike can vet anything and the whole point of it is to do zero touch patch exploits for emerging threats. So it’s a lose/lose situation, you remain vulnerable until you manually vet and patch and hope you don’t get exploited in the mean time every time there’s a new vulnerability, or you get a patch like on occasion this that ‘fails safe’ and locks even your users out.

@@peter65zzfdfh If you run a multi-billion dollar company, they should employ a couple in-house IT people. It is insane to rely 100% of your security on a 3rd party. Especially when it is software that has direct access to the registry. This is computer usage 101. Just the fact that so many companies are using this single point of failure without any oversight should be a massive wake-up call to the world to realize how fragile these things are - for both the companies and the public.

You are totally correct in your company's approach to system updates. They are used internationally by vital infrastructure companies, so it was totally irresponsible for them to have not tested and retested the update BEFORE sending it to the world!

​@@peter65zzfdfhif you can't vet it, and are stuck with mandatory automatic updates, start looking for alternatives. you cannot outsource your resilience planning and still expect things to work, ever.

This has been reality for a while. Everyone runs on cloud systems where updates are forced, not announced, and continuous. Well other than the airlines that still use floppy disks for updates. Doesn't seem so crazy anymore!

"Finally". Love your optimism! Unfortunately, it was always like this, only difference is now they can deploy the updates even faster...

Exactly! All these companies had ticked a checkbox saying "security". I guess systems that are not running are the most secure.

It was pushed by Crowdstrike globally bypassing any staging configuration their clients had. The companies affected are not to blame, Crowdstrike massively, and I emphasize, MASSIVELY fucked up with this one.

for most critical infrastructure, air gapping them is not an option. how can you run your airline if customers cannot connect to your booking system, and neither can check-in. what about not knowing where the planes are, when they are going to arrive, when the connecting flight is going to leave, etc. not having things connected does not work for most systems. unfortunately most systems are designed without even considering if they are fragile. the answer is resiliency planning, where you start from the assumption that the thing will fail and ask what happens next. so all the flight booking information for a particular flight would be stored on the company server at the airport where it was going to depart from. if the internet link goes down, you use the backup copy uploaded to regional hq when the Internet was last up, but don't confirm the specific seat until the Internet comes back up. similarly, the aircraft should have something on board which contacts the head office every minute and sends a gps location. this should not be something the pilots or crew can turn off for obvious reasons. both of these techniques date back to before windows 95, so not new tech.

@@grokitall I'm not saying you can air-gap everything, but you can air-gap a lot, and you can most certainly prevent auto-updates. You can use stuff like AWS private links, etc., to create your own closed-off intra-inter-net. It's not really about the Internet connection nearly as much as it's about using Windows for anything mission-critical, and not just for end-user desktops. Even ignoring this incident, would you really want to be on an aeroplane that's controlled by a system with auto-updates enabled? No way.

@@NatiiixLP i agree, i would not want to fly automatic updates airways either, but then that is why every version of the civil aviation authority insists that updates happen on the ground. also, you were the one who said that most important systems run air gapped. i was just pointing out that while there are ways to mitigate the problems with connected computers, air gapping does not work for lots of software and you need resilience planning instead. i also would not say it is about windows per say (although there are good reasons to avoid it for anything important), but more about not using anything with mandatory automatic update policies overriding your security, stability and resilience policies.

@@grokitall The wording is ambiguous, I admit, and that was not really intended, but I said "most important systems" as in "most OF important systems", not as in "THE most important systems". And the value of these systems is relative, but I was primarily referring to banks and government organisations, which don't necessarily require real-time access to the general Internet. I'm quite sure many airlines also air-gapped mainframes/clusters, just not the ones used for ATC or other airport procedures.

@@NatiiixLP i don't think air gapped means what you think it means from your usage. if you remember the scene in mission impossible where they steal the identities of all the spies, that machine was air gapped. stuck in a faraday cage, with no tech in the room, and then only way to get data onto the system was to either type it in manually, or to create the equivalent of a custom usb stick which is then scanned on another machine before you can even take it into the same room as the machine. when in flight, a planes computer acts like it is air gapped for software updates, but still receives weather data from external sources. the moment it has a data connection to another machine, it is not air gapped. thus very few systems in practice use this method, as they need to talk to other machines to get software and data updates. with resilience planning, you can mitigate then risks, and build systems which are less fragile, and you can use chaos engineering to test that those steps work as intended.

Even though it was 20 sec ago that’s underrated

I got my money. 🤷🏿‍♀️

Our company sent us home early and boi am glad for that.

I was shocked I got my check this morning, I pray for everyone else does too... I can't help but if you mean ADP...... I work nights and from home and still on the clock, while this has been fun tonight won't be and well day shift just may quit after they have to deal with the cussin about how the night went. My callers are plp calling their insurance needing a tow or roadside, I deal with escalated customers in the manager compacity so I hope they don't fix it before my shift ends I don't want to have to get on the phone this morning it will be a nightmare.

​@@TethoSama one benefit for doing updates/changes on a Friday I guess!

@@PatrickBaptist hahaha oh boy. Not going to reply to your comment but while “we” got everything back online this afternoon the next few weeks are going to be great fun replying to the millions of clients asking for an update

i’ve said exactly this so many times in my career 😂

The problem is a corrupted channel file. I don't know what that is, but I suspect it contains malware lists or something similar. If that is the case, it makes sense that it is pushed immediately. Still should have tested it though. This should not have passed tests, but it does make sense that it was released on a Friday, and quickly at that.

@@entcraft44 nope, no malware lists at all. Crowdstrike is a fancy logger, with an analysis panel at the security operations center that utilizes a Splunk styled software interface. Overall, I quite like their software and implementation - until now. They push the updates to their software. It's inexcusable to ever consider pushing a low level driver out on a Friday. If anything does sideways, it's equivalent to pouring gasoline on a foundry floor while it's in use.

@@spvillano If so then I retract my statement. But it could still affect other products in the future.

​@@entcraft44it can effect any product with automatic updates, especially if it needs write permissions everywhere to install. this is why every tech lead worth the name won't install mandatory automatic update software on any machine they are responsible for.

And where exactly are you allowed to 115mph. I can only think of a couple of roads in Germany and that is it.

If your car was damaged by windows, you have bigger issues than this...

@@alexpervanoglu7420 To be fair, he never said anything about driving legally. And thus driving 115 mph is possible on a lot of roads.🤓🤓

​@@alexpervanoglu7420 driving that illegally is still possible if your car hasn't been electronically limited, with potential repercussions depending on where you're at.

@@morpheon_xyz 3 decades as a continental truck driver, so yes I know this, and sadly have seen the aftermath too many times. But yeah, it is possible.

Skynet is useful. This is stupidity 😂

Speaking of which I just recently repaired a robot. 😂😂😂

@@someoneout-there2165😶

@@someoneout-there2165 Did it call you a meat bag or ask for a plasma rifle in the 40-watt range?

​@@blshouseI understood that reference 😂

They are not smart enough for this 😂

Or have backups for essential systems

Same problem can happen with Linux. I've installed Linux updates that resulted in a reboot loop too. The issue is processes. Don't install updates without testing them. Don't install updates without a backout plan.

@@username7763 Updates are the ones supposed to save you from the security hole hell... kek

the contractor they hired was afraid of the penguin or under a microsoft related contract, so there you have it

Betcha they're going to call you in. lol

Testing Testing Testing 😊

Oops 😬 🤭 😊

That's your Friday evening F'd 😂

​@@aisle_of_viewjust don't pick up

Worst part is since this is happening on every pc, if they had tested just one it would have been fine

Yeah, went through that. Had 10k users, was only allowed a test group of 20. Obviously my fault. So, I left them to swim in their own swill and amazingly, the entire organization failed. Guess that was my fault too. Remotely. Without access. Without even being in the country. For I am all powerful with that mighty can of beans.

FedEx? 🤣🤣

It also takes a company that invests in resources and people to be able to test. Most run so lean we're all just hoping nothing breaks.

You need an ambulance ride to the hospital for an anxiety attack???? Seems a bit overkill.

@@LoganChristiansonsome people seriously can’t handle stress or plans changing

Sounds like a pretty normal night on reception back when I did it.

@@LoganChristianson When you're running a business you don't take any cha- oh wait, I just remembered what this video is about lol

Does that mean I don’t needa go to work today at my hotel:)?

"Do it. What could possibly go wrong?"

"We can do this Agile, release it... analyse potential issues, mtp update" "Yeah great idea, let's go.. TGIF!:

Push to prod, pray to god

"Dew it"

"Don't worry, it's a pre-alpha release. Nothing will go wrong!"

fragile no, careless on allowing automatic updates from a mediocre company, i blame microspft, who promoted automatic updates since 2015 with win 10, everybody followed them

​@@betag24cnactually they started it a lot earlier, but you had your system installed by someone who knew what they were doing and turned them to notifications before the machine even went out the door. it was only when people started buying from white goods suppliers instead of computer shops that it went down hill.

Thanks for the spoilers

@@ZakuroBananabrother that came out decades ago

🤔 you may be onto something