Global IT chaos - CrowdStrike boss warns return to normal will take time | BBC News 

They get yelled at everyday in this modern “I need it now” world.

Yeah. Sucks they are being paid so low and have to deal with karens because this company screwed up

I am one of those receptionists & people yesterday called me "rude" & wanted to make a complaint bcuz "I refused to give them a doctors appointment!" 🙄

The receptionists at my surgery had one HELL of a day! They got yelled at a lot!

❤

Do you know Jesus Christ can set you free from sins and save you from hell today Jesus Christ is the only hope in this world no other gods will lead you to heaven There is no security or hope with out Jesus Christ in this world come and repent of all sins today Today is the day of salvation come to the loving savior Today repent and do not go to hell Come to Jesus Christ today Jesus Christ is only way to heaven Repent and follow him today seek his heart Jesus Christ can fill the emptiness he can fill the void Heaven and hell is real cone to the loving savior today Today is the day of salvation tomorrow might be to late come to the loving savior today Romans 6.23 For the wages of sin is death; but the gift of God is eternal life through Jesus Christ our Lord. John 3:16-21 16 For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life. 17 For God sent not his Son into the world to condemn the world; but that the world through him might be saved. 18 He that believeth on him is not condemned: but he that believeth not is condemned already, because he hath not believed in the name of the only begotten Son of God. 19 And this is the condemnation, that light is come into the world, and men loved darkness rather than light, because their deeds were evil. 20 For every one that doeth evil hateth the light, neither cometh to the light, lest his deeds should be reproved. 21 But he that doeth truth cometh to the light, that his deeds may be made manifest, that they are wrought in God. Mark 1.15 15 And saying, The time is fulfilled, and the kingdom of God is at hand: repent ye, and believe the gospel. 2 Peter 3:9 The Lord is not slack concerning his promise, as some men count slackness; but is longsuffering to us-ward, not willing that any should perish, but that all should come to repentance. Hebrews 11:6 6 But without faith it is impossible to please him: for he that cometh to God must believe that he is, and that he is a rewarder of them that diligently seek him. Jesus

Spare a thought for the IT department? Nope. What about all the individual users without access to a tech department?

​@@outlawJosieFoxIndividual users wouldn't be a crowdstrike customer.

​@@outlawJosieFox individual users aren't use a top tier cyber security product like crowdstrike falcon on their machines.

​@@outlawJosieFoxIndividual users? Have you checked your laptop today?

This is why I still prefer paper money transactions, always handy to have cash around

What IT outage? There was no chaos, life went on as normal. Of course I'm just a normal person going about her daily life.

Except, we’re not so fragile. This was a few hours worth of disruption. Then back to normal. It actually shows just how resilient we are.

@@fionastevenson4366well said

They used and Excel sheet to track the CoVid in the UK at the start of the pandemic. This is many things but surprising is not one.

Nostradamus been sleeping on the job

No it sounds like south park

@@skootergirl22I highly doubt you even existed during y2k

@@Zxlok I was born in 88 and I remember the 2000s panic of the millennium bug

​@@skootergirl22No trailer park

😂😂😂

Welcome to the Hotel California. You, sir, won the comment section, my respect!!

@@Arsenic71 - ach ja, Thanks!

@@Ann_T_Social - if you say so 🙂

Huge! Been trying to trade on my own for a while now, but it isn’t going well. few weeks ago I lost about $70,000 in the trade. Can you please at least advise me on what to do?

Well, I picked the challenge to put my finances in order. Then I invested in cryptocurrency,stocks,through the assistance of my discretionary fund manager,

James Werden

Such a genuine personality!! He is really a good investment advisor. I was privileged to attend some of his seminars. That’s how I start my crypto investment.

Wow, that’s very nice. Please how can i be able to reach out to your broker. My income is in a mess please

true skynets aware...

@@fpvDRE yes.

Source?

No it didn’t… Skynet was an artificial neural network-based conscious group mind and artificial general superintelligence system for a start. This is a system for receptionists that possesses no intelligence whatsoever. It also doesn’t control the military. It also can’t become self aware so we won’t have to try and turn it off which won’t cause it to attack us with nukes to preserve itself. The irony is that you’re probably a bot yourself. Almost all of a RU-vid is robots at this point.

And evil villens exist

Agree.

The main issue is, Windows 10 and 11 are sort of designed to prevent this kind of power for antivirus software. The problem is, they all started creating aggressive workarounds to regain all the power they used to have, pretty much like a rootkit malware. When that breaks, it breaks everything as it is now the core of Windows. I suspect these companies will get even less access come Windows 12. Other nightmare scenarios is, if the third party antimalware stops responding, Windows is often left completely without any protection. I suspect CrowdStrike's relatively fast growth from nothing to being "everywhere" in little over a decade is partly because everybody got the impression their software wasn't as reckless with all of this, as the old Antiviru-zillas Symantec and McAffee.

​@NOT.MI5.MI6The problem is not electric, it is the over reliance in computers and software.

Tell that to Microsoft

Exactly. We need to go back to the olden days where stuff like when media companies weren't owned by a couple of big companies like Disney, Warner brothers, comcast etc and when thousands of companies owned the media

Just shows you how this is deliberate.And every summer they have issues with travelling at Airports.something they are doing. Anyways cash is king.Paper too.

​@newwineskin5494 Yes, let's just use paper and pen like the olden days

@@skootergirl22 😂lol both.Like what's all this new system at the gp surgery for online only?People are sheep and they are just following all this changes blindly.

Proper networks have WSUS server. Just shows how many poor designed networks out there

The CrowdStrike software loads on pre-startup which is usually before any backup system that a company may have runs. If there’s a startup exception, the system automatically halts the rest of the startup and reboots. In this case, the system then loops itself with the glitch and causes the blue screen. Long story short, any backup system then requires the user to start their device in safe mode which is basically “isolation mode” which can’t be done automatically. It is sad, but these companies basically couldn’t do anything to prevent it. They also can’t opt out of CrowdStrike updates since the software updates regularly to patch exploits/vulnerabilities.

Exactly their audience must be all females (just like the all BBC presenters are in this video) who would believe it pure ignorance.

It definitely was an update. A cyber attack gains nothing from turning machines off. Why it failed testing is the question we want the answer to. Crowdstrike shouldn’t be having QA issues like that.

​@@stuartmorgan9327 A cyber attack gains nothing from turning machines off? There are plenty of ways to benefit from that. Short crowd strike stock. Be a competitor. Etc.

Try keeping your dumb conspiracies to yourself Martin.

"without testing"......like the covid vaccine.

I worked for a large security vendor that was breached and they do indeed pay through insurance in most cases.

Cash is always a good plan I will say though AI is more a tool atm than replacing jobs.

Wait what digital currency coming and computer systems go down

You can't hold a whole company responsible for stuff like this. This stuff happens all the time in software development. Sometimes it's as simple as you not having a particular hardware configuration. You enjoy RU-vid for free, and it's entirely possible a small code change could bring the entire platform down. If that was a huge financial fine every time it happens, no one would be able to afford the software we've come to rely on day to day.

@@kh_trendy Complete lack of engineering oversight and procedural governance. Who let's an update go out automatically without thorough debugging-CrowdStrike. This should be the end of this amateur outfit, watch their share price collapse. Microsoft also needs to bring in greater checks of the work of third parties, a system they have continually automated to get rid of paid human staff.

It was the best available software for the job apparently, until it wasn't.

Also why tf are they connected let alone auto updating

@@postminchoppaprobably becausen they’re all running on azure vm’s or vm’s. Which by default….auto update.

The cloud... I knew there was rain in that cloud..😂

It was the best software to use and no one expected this to happen for sure

See, the reason for not having all these tech for one day, will cause so much damage, if you do not rely on tech, then basically it means you are going to experience it every single day.

Reliance on tech is the future, whether we like it or not. The real problem is poor QA. I would think a simple test on a test system would have shown the problem. Evidently, they didn't even do that. I can't imagine how that could happen.

Your over reliance on tech, made you point out the obvious in a RU-vid comment section… Well done Captain Hindsight.

Says someone using a device that does pretty much everything

Ok boomer

But that’s not the lesson. Crowdstrike is probably about the most expensive XDR solution you can get. So companies being affected by this have not been underfunding IT. Crowdstrike might invest more money in their QA team but that’s about it. The issue is the only way to avoid these sorts of issues is to duplicate all of your systems but using completely different hardware and software so that you’re 100% completely different. That includes internet and networking as well. Which is completely impossible.

​@@stuartmorgan9327 What I was referring to was that due to compliance and profit target requirements, corporations resort to the cheapest security solution - which is a one-size-fits-all cloud-delivered product with 24x7 updates directly into the kernel, which replaces a security team of salaried qualified personnel dedicated to security. One qualified security person's salary is much higher than a Crowdstrike subscription. There are so many millions of businesses of all sizes that do not run Crowdstrike and do not get hacked everyday in the absence of a similar product. I'm not against automation either. Macs have better kernel security. Crowdstrike has been messing up their Linux updates similarly, but there is at least an option in Linux to run Crowdstrike in a lower privilege mode. Qualified DevSecOps personnel have setups which allow PXE boot as an alternative, which is how one guy brought back 1100 servers in 30 min yesterday. The point is again, if your management believes that a cloud solution can replace your security team or IT team, they will do it, and later when the cloud product fails, you don't have people to fix the mess. Now imagine if CS's CDN were compromised and the payload was not a boot-loop inducing null pointer deference due to poor data validation, but an actual payload from some state sponsored actor.

It is a "fix" in that it does undo the damage, but it is't an automated patch that can just be sent directly to the machines. Many years ago, I worked in an emergency services job (think 911 responder) and it always amazed me that community members would randomly bring us food, particularly on holidays. For those folks working in IT, particularly at really, really large organizations, their communities (e.g. their co-workers in other departments) need to bring them some good food and beverage to help them to keep going through an undoubtedly long, tedious weekend, as they manually fix each machine!

Yes, saying they’ve deployed a fix isn’t half the story. Being in IT that is going to be a nightmare to have to go around to every computer and do that, especially on a Friday aswell when we’re advised you should never roll out a update on a Friday(or the early hours) cause of this scenario

@@im.empimp you a bit hungry? have a byte!

@@timturner7772 🤣 Since you asked, I'd 🧡quite a few bytes of some 🍕. 😋

Especially now after pandemic, my company has so many outsourced so many manpower in few neighboring countries. Imagine having to fix that. Good luck.

Including our ability to spell ;)

​@@stuartburns8657😂😂😂😂

We have not, we are been forced.

Dude you're using a device that does everything

@@skootergirl22 it doesn't do everything. It mostly spies on us.

Wakeup call to include secondary redundancy. An alternative operating system rather only depending on Windows.

Hear Hear!

I am going to an island.

And what's the alternative genius?

The world coped for years without computers, genius?​@@fallenhero4550

Try keeping up with millions of data points with pen and paper. LOL

So many sheep believe whatever they’re told to believe by their corrupt governments and media.

What a coincidence..!

Pelosi shorted cyber security stock the other week. Similar to that company shorting trump stock last week. Just coincidences lol

It seems planned. Crowd strike, making massive crowds having a strike.. Interesting.

Doesn’t sound benign does it? Trojan horse?

Sounds like a my little pony name

Antivirus updates are important. Out of date software leads to much greater risks. This is an extremely rare incident.

@@DC-wq6hx antiviruses are scams. They only slow down your computer. The standard Windows Defender is all you need

This kind of kernel level Cyber security software usually updates automatically. If it were a manual update and you forgot to apply it or failed to apply it on time, your computer could be hacked

CrowdStroke

Don’t forget Microsoft. They also didn’t do any testing.

I like how Even this report blames Microsoft. As if they forced users to use CS.

It is definitely not clear which company is responsible. However, as some cloud developers have posted, the airlines and banks et al ARE responsible too, for their lack of QA and/or contingency plans.

Crowdsrroke has root access to the systems they are installed on.

lol

Yeah, 15 times out of

😂😂😂

There is a way out, its happening right now. Digital global economy incoming.

Lol 😂. So let me get this right. If you think your computer has been compromised how are you updating your passwords. If bad actors have access to your computer wouldn't they just have your updated password.

It's funny that my boomer mum didn't understand what's going on

It's not so much that as it is an overreliance on largely one software system from America, that being Microsoft.

​@@debbiegilmour6171Microsoft isnt the issue here and there arent any alternatives anyways.

@@poro167 There being no alternatives is very much an issue.

If you hate digital then why are you here. Go back to your cave

Yes executives force this spyware on you under the pretense of "Compliance". For them it is a chekbox in excel sheet that is finished and their yearly bonus secured for "excellent performance"

Of course not

Correct. They also have root access. It is spy software that views and logs what users are doing and reports back to crowdstrike. They have cost lives with this update.

The company’s name is Crowdstrike. FYI

@@hereandnow3534 Yes, and third-party security software relies heavily on Windows Firewall and other Microsoft software...

That's why personal windows users weren't affected, I suppose.

@@mosesrocco6614 Just like is its name, only crowd pc or public pc that got the strike from CrowdStrike

It's really not a matter of electric cars, but rather modern cars. All new cars being produced are controlled by computers. I've always loved electric vehicles since long before the Tesla motor company was a thing, there isn't anything inherently wrong with them. You could have an electric car without computer controls or internet connections. The issue is our modern reliance on control systems.

Lol petroleum cars still have electronics - a lot of them.

@@PikaPluff The relatively older ones don't generally receive OTA software updates that control safety critical parts of the car. They are just the same control feedback models programmed on the same microcontrollers across all the cars.

Nope. Stop lying boomer 😂

@@mohammadrizwaan1890 that's true. Any car has software that can be midified wirelessly is prone to hackers

What if some patients travelled to another part of the country and fell ill? How would the hospitals there access the patients medical records without Internet access? How would you monitor or track epidemics? Also how would the NHS know how much of a particular drug has been used and what to replace if every single hospital or local surgery was running its own closed system? Also if the computers are never connected to the Internet, would their Operating systems, antivirus and other software not be hopelessly out of date? What happens if someone then plugs in a memory stick that has a nasty virus in it? All your computers could be wrecked and all the information lost because you never backed up the data to a central database because "no Internet"

@@enadegheeghaghe6369 All valid points. However, we managed, fifty years ago. The patient’s file was sent by the old GP to the new GP. Information was obtained by letter, or in an emergency, by telephone or telegram.

💯 - This is exactly what I've been thinking all day long.

Oh bless you. That costs money.

Rolling out an update on a Friday is a big no no

@@im.empimp It's literally impossible to test a billion-wide PC update launch, it cannot be done

@@MrWillyMrBrightside LOL you can send your software to MS and they will test it for you... for free... they just dont like MS.

Only if it’s accepted by person or enterprise you’re transacting with!

@@karenshanley487 You're right. And that's why we should all try to use cash right now - whilst we still can.

@@andrewelliott4436Nah you’re alright. I’m glad I don’t have to walk round with paper in my wallet and coins in my pocket.

@@imconfused1237 Have you been mugged?

@@andrewelliott4436 I have yeah, back in 2010. Took my wallet, which had all my cards and money in it. Instant financial loss and a faff cancelling everything. Mug me of my phone these days? No problem: block it and claim on insurance. Minor inconvenience.

Does not dose.

@@garylovell6017I don’t think anyone cares, it’s clearly a typo

I think that's just the NHS not a global IT chaos.

@@bawilson999 "insecurely attached" lil bro what are you even talking about?

@@bawilson999 "Special private connectivity to 3rd party systems like Crowdstrike." yes... that's how this entire fiasco happened...

Crowdstrike had a similar incident in April with their Linux version, causing a kernel panic. It just went largely unnoticed in the media.

Chrome OS ?

Soon Microsoft will will find it's way to replace CrowdStrike with it's competitor.

@@skp5725 Actually, Microsoft has its own product (Sentinel) for end point security.

This the only sensible question I’ve seen asked anywhere in the last 24 hours, including in the news. It is the right question to ask, when everybody else is running around with their hair on fire, pulling out all the corny cliches and shouting about chaos. I’m a cloud developer who spent all day working with colleagues around the globe and nobody mentioned it. Granted I wasn’t flying, but how come the planes haven’t invested in hybrid cloud redundancy? It would pay for itself. We need to ask better questions, as this post hints at, about software quality and better code release practices and improved safeguards.

​@@pic101cloud developer here too...one of the main important rules we love to follow is to not deploy anything on a Friday....also this feels like totally untested too

Read a post on Y combinator that suggests they might have bypassed QA.

@@vancedkirukanthis is the golden run in software development.

Probably cutting corners for financial reasons like always

Due to this only affecting Windows OS machines.

Microsoft have to digitally sign software for their systems. Without Microsoft approval the software wouldn't be able to execute. They approved the update.

@@notjustforhackers4252 Microsoft does not authenticate software updates for non-Microsoft products. Digital signatures serve only to verify the source of the software, not the quality of the code. They indicate the author or issuing party of the software.

@@notjustforhackers4252no, applications can download non executable data for their own updating and patching without being certified by windows. Also, this could have happened on any OS, it just happened to be windows.

Ignorance. Everyone is an expert in everything, or at least they beleive they are.

Customers.

I spoke to a guy that does cloud solutions for a big company, he said it's likely Russians but he isn't sure. He advised me to take all of my money out of the ATM

@@Annathroy LOL its not the Russians... these companies dont do QA anymore... thats why they cheaper than trend/mcafee. Just people paying for what they got.

​@russian panic botAnnathroy

@@take2762 bruh

I don't even update unless it prevents my computer from running properly. Still using Windows 7!

@@isag.7468So you’re using an operating system which is no longer supported and is actively targeted by hackers. I mean what could possibly go wrong.

Crowdstrike Falcon software update is always automatic. That's the whole point of it. It updates quickly and quietly in the background so it can respond very quickly to new and emerging threats.

Liar

Sounds like the ramblings of a schizo

Starbucks 😂, never had one am not paying a fiver for a cup of coffee

Those damn furries

Dimwit muct aren't you, gammon?

Oh right. So “System B” must be incredibly vulnerable then, given it’s never updated. What’s that, you updated it on 19/07? Whoops. Poor planning on your part.

It’s not about no backup, going to backups has issues itself. Firstly what is your RPO depending on that depends on how much data you lose, is it better to lose that data or fix the issue and lose no data? Secondly, you rollback the system, you then also have to prevent the update from taking place again. This is security software by its very nature it’s designed not to allow people to turn it off easily. Also if you turn it off you’ll be exposing the organisation to other risks. Do you accept that risk or wait for crowdstrike to issue a fix knowing that their engineers will all be fully committed to solving this issue. A lot of armchair experts who think they know how things should be whilst not understanding the realities of IT.

​@@stuartmorgan9327finally, someone with sense commented.

@@stuartmorgan9327 A backup plan has nothing to do with an actual software backup. If the hardware of 10 computers stop working, do you have replacement hardware in the closet? If the building is on fire, do you have another location? that's a backup plan in case of a disaster.

@@imconfused1237 Irrelevant, system B contains the previous version, I said revert to the previous version, not go back in 1980. I guess you have no clue how it works.

Oh, the lawyers are already prepping the suit filing.

​@@user-ro7ps1rr3pexcept that the customers of crowdstrike is not the "you" as a person, but the companies they service. Who actually have the power to sue them to oblivion. That's very obvious if you actually have a a corporate job. The IT department installs software on your company provided PC and you do your job. If a software they had installed broke then it's not your problem. It's the company's problem.

Pretty sure their Terms of Use already cover this scenario. And whoever used it, took their chances since, apparently, it only (publicly?) happened once so far since 2011 (unverified). They lost 10% on their stock prices and that's already a lot. It had to happen one day. Whoever used the service with 0 supervision should be sued.

It likely was tested, but the particular vulnerability was not covered in the test environment. Most people don't know how complicated software can be.

They should check it in every version even with virtual os

@@GH-oi2jf This affected all PCs running it. Not just a few. Usually I'd agree but I am a software engineer and I know how many assumptions are made right before new releases.

Indeed, it is. One damn company and their incomptence.

This was thought to happen

Eh? A lot of people who work from home were affected too.

On paper and pen right? Pigeon mail?

@@skootergirl22by work from home, maybe they meant twiddling their thumbs and waiting for their UC to come in

Right haha. I woke up was told by my partner that there was a global IT outage. Had mini panic attack that it was a cyber attack. Realized it didn't affect me and continued on with my day. Seeing all the incompetent people with their conspiracies has been entertaining though.

​@@Aaaa-1zntonly those using Crowdstrike falcon software

Best comment 😂😂😂

This happened to my private windows pc 3 days ago.....took hours of work to get it to reboot to windows and a lot of patience...

Third party companies. Microsoft doesn't check the code for every update done by everyone. If a popular gaming software company creates an update for their "Super Duper Game" and it crashes everyone's PC because of their incompetent programmers, is Microsoft responsible?

@@roachtoasties That's what I said.

crowd tech guy:ok the update is ready buuuut it knocks out the os system for a while should we still go ahead with it crowdstrike:yes i wana see the world burn hell good job on the linux and mac debacle aswell crowd tech guy: why thank you😅

“Lock the doors.”

Mostly to reassure shareholders who might lose quite a lot of money

@@isag.7468 Yes, I noticed on Crowdstrike website when looking for updates which there was none there was the word 'Why Crowdstrike", that's going to be an interesting line for upcoming RFP's :)

Windows is used on nearly all commercial computers, with extremely rare exceptions, so this information is not secret by any mean. You won’t see major system ran on Linux, except some servers, and definitely not on Mac.

I suspect that some of the less important computers (e.g., food-cart ordering, even reservations) can/will use commercial operating systems like Windows. However critical systems like safety, navigation, communications, etc. will almost certainly use either a proprietary OS or one that is a heavily customized version of an industrial OS (e.g., one from VXWorks). For example, the Stuxnet virus was made to target the Siemens software running Iran's (assumed) nuclear plants. I know Linux is use for the entertainment system for several airlines.

@@TarasShyn "Some servers" yeah just a cool 80% of them.

Which out of all 2 options... and virtually all computers used in enterprise contexts like this are running Windows.

bruh it could be windows or linux you dont need a security company to disclose this