This global internet outage is insane! All airlines grounded and i was stock the airport and even banks, media, and offices from the U.S. to Australia. How can CrowdStrike have such a monopoly that could help restore such a massive amount of tech?
Right? It makes you think about the stability of our systems. But hey, I barely spend time online. When I checked my portfolio with Desiree Ruth Hoffman, we were still in the greens. That’s been the case for 16 months straight!
Probably from her forecast on Nvidia before the pump. But how are you in the greens with all the fluctuations due to the election and everything else? Can you share her strategy?
Honestly, just schedule a call with her. She has vast knowledge in finance and really knows how to navigate these times. I handed over my portfolio to her so I can focus on my family. These days, things just get scarier and scarier.
"Some experts have speculated that perhaps it was a lack of testing"...LOL! It was not a "lack of" testing, it was the "complete and total absence" of testing! They pushed out an update...to every, single one of their customers at the same time, that broke every machine. It clearly was not tested AT ALL. Also, this is the reason that you DO NOT push out an update to everyone at once! You roll out to a small number of customers and make sure there are no problems before expanding. This is BASIC operating procedure. The fact that a "SECURITY" company failed this so spectacularly says everything you need to know about them as a company. Any organization that continues to use this product after this fiasco is run by complete and utter morons.
In principal a field test for a system software is mandatory. But with AV software it is a race between viruses that spread with crazy speed and the AV updates. And many companies have a paranoia when it comes to virus attacks. So they blindly trust into the services of AV companies. In the end it is the good ol' blame game. If the AV company fails - not my fault. If a virus causes harm and the Admin did not rollout at max speed - his fault. Now you know what responsible managers choose.
I studied cybersecurity. I applied to Crowdstike and was turned down because other applicants were "more qualified". Little did I know that Crowdstrike pushes untested software to production which is something I would have never done.
I never studied cybersecurity. In my day as a software tester, I was blackbox because anyone who could actually code, would actually code and get paid a lot more money. I'm gonna guess that the code itself was not 'untested'. It was the implementation of the patch that was not tested. Crowdstrike, having tested the fixes, trusted a process of rolling the updates out. It would say the very fact it has customers, has proven that process works, so why question it? Crowdstrike probably didn't see the value in further delay by rolling the patches out to itself first and do yet another test to see that what it thought it rolled out, actually rolled out.
Updates can be rushed to customers due to zero day attacks, but I agree. I work in IT and they are supposed to test updates in a sandbox environment before being deployed to customers. Its IT security fundamentals.
What is more ironic is the CEO is a former CTO of mcafee antivirus, and this happened on his watch "Defective McAfee update causes worldwide meltdown of XP PCs" so dude has not learned from past mistakes.
CrowdStrike was the cause of the major outage, not Microsoft. Despite mentioning Microsoft Windows in the first 30 seconds, it takes until 2 minutes 20 seconds to mention the actual company responsible.
Okay thanks, Microsoft spokesperson. Can you also explain to us, why Microsoft doesn't have guardrails against these type of events? Where's the automatic rollback, for example?
@@valdragu107 You do know that pretty much the same thing happened on Linux just a month or so ago? Debian 12 + crowdstrike caused kernel panics in April RedHat + crowdstrike caused kernel panics in June
@valdragu107 This isn't a driver, this is a kernel level software. I hate MS as much as the next Linux-user, but you can't blame Windows for failing when the core of its OS is messed with
So, let's give them the benefit of the doubt and say they were totally incompetent. BUT, if you wanted to cause the max trouble, could you choose a better inside job in a better company, a better OS, a better file...to attack? I'm not a conspiracy theorist, I'm just saying a competent risk manager would have exactly this scenario at the top of their list.
<a href="#" class="seekto" data-time="412">6:52</a> - what an unnecessarily rude interruption of a reporter giving a pretty good analysis of the situation.
I read people for a living as a counsellor. Her face spoke ALOT more than her words did there. I noticed the way she was smiling alot as she said it, and then she took her answer and directed down a totally unrelated path. She was quick to jump in, and then quickly ask "Does this mean technology is advancing too much and we should go backwards, or are you saying the march of technology and AI must continue no matter what". She strongly gives the impression of someone who hates technology, (she's an older woman, you can tell beneath the makeup with how some parts of her face move), so I'm going to guess she feels left behind by tech or finds it frustrating having to use it, and wanted to make a point about "LOOK HOW BAD THIS IS, WE SHOULD HAVE LESS OF THIS" and was smiling because she saw a chance to say it.
Yes because you running on a similar algorithm as financial institutions are , medical billing and medical coding will be the next big thing once AI is becoming integral,good luck and lots of patience.
Monopolies aren't going to work anymore in a complex world like today's. Competition is healthy - and so are backup systems from across multiple global sectors.
CrowdStrike's Wikipedia page has an entire section dedicated to mergers and acquisitions. The root cause is not kernel architecture or even testing procedures but anti-trust law.
The problem is that digital markets have completely failed competition and pushed us toward a handful of companies for each general need. There are only 3 computer operating systems and only Linux is open source. Google dominants with over 90% of the search engine market and Bing trails at like less than 5% and all the many other options are far far less than that. These systems don't work well with each other or communicate well between each other making it difficult to switch or use alternatives. Developers also just don't want to develop for too many and it's expensive to support everything. On top of all this governments have largely ignored the issue (and granted these few tech companies are purposefully making it difficult to understand and see the problem). The best we see is the EU and they still aren't really getting to the heart of it. We've just become so complacent. In theory overreliance on technology shouldn't even be an issue, but we've allowed ourselves to all depend so greatly on a handful of systems, platforms, and services. No one even thinks about it. It's just, "hey crowdstrike is like the number one cybersecurity option got windows enterprise systems and cloud computing. Everyone's using them. So why don't you?" then over time they become so stubborn and think they're above simple mistakes or needing to follow basic procedures. And eventually you get a situation like this.
Bro, shut the funck up. I can clearly see your not from the area of IT. You DOnt know anything. A simple dual boot would prevent this from hapening. If windows cant start, start another OS on another partition. Thats extremely simple. You common people dont know anything and then start lecturing devs around the world wtf xd
@@sentiment7644 OSs like openSUSE, Fedora Silverblue, macOS, and Chrome OS use automatic rollback mechanisms to revert to a stable state if an update or configuration change causes a system failure, preventing widespread issues. If you do not use Windows you do not need crowdstrike in the 1st place.
Yes and the fact it runs in KERNEL MODE which is like really fragile and 1 slight error can knock down everything. KERNEL MODE FILES SHOULD ALWAYS BE FULLY TESTED.
This was NOT Microsoft's fault, this was a complete failure on CrowdStrike and their QA process for testing patches, updates etc, before pushing to production. I suspect the tools that they use for input validation against the code was not up to speed on modern programming error detection. Microsoft is a victim of this just as much as all others affected. Windows Kernel did what it's designed to do. CrowdStrike needs to test, test, test and more test, before pushing to production. The fact that they pushed this globally too is mind-blowing. Where is there QA director? Why didn't they deploy this in small regions to test and ensure no errors? I see a lot of law suits against CrowdStrike from major companies. Airlines are not going to eat those delay bills.
Everyone in business should fire Crowdstrike and replace them with a company that actually tests software before pushing out updates. Also surprised that so many airlines are running windows versus linux.
Each organization should vet software updates prior to them being distributed to their organization en masse. Allowing Crowdstrike to circumvent such processes is the responsibility of each organization. Companies should not abdicate this responsibility to a third party such as CrowdStrike.
No good IT people appreciate having to run business critical systems on Windows. But all the MBA Chads who run the companies don't understand that and run what they're used to. And They're not the sorts of people to listen to their own experts over salesmen. Source: I'm an IT expert who almost got fired by my ignorant business grad managers for speaking against the idiotic products they were being sold by slimy salespeople.
You used to be able to boot into safe mode by holding down a key while the machine restarted. Windows 10 and up require multiple reboots to access safe mode to keep regular users with no idea how computers work from getting into safe mode and messing up the computer (yes it's possible).
it's won't help, that sht need to boot in safe mode disable all the drivers and remove the faulty file, it' long process and that shtty company should inform their client the moment they discovered it,
It's simple to avoid -- trivial in fact. Do not EVER allow forced day-zero updates. You use rolling patches, and let the users decide when/if they want to install them. Any company using the crowdstrike and knowing it had KERNAL LEVEL patches being installed arbitrarily were just stupid -- no other way to call it. Those companies relying on CS were idiots.
I totally agree with your assessment. But what fool released an update without testing? There is no way any type of meaningful testing was done prior to release or this would have been discovered.
Also the fact that Windows will happily execute a kernel-mode driver it just downloaded off the internet and not jetison it if it causes boot failures is just insane. There needs to be graceful fallback when something like this happens The awful design of Windows is just as much to blame as the awful rollout of this rootkit.
I love the grounded reality of this channel!!,Despite the recession, I no longer depend on Government Grants since I acquire $16,400 bi weekly profits.
Huge! Been trying to trade on my own for a while now, but it isn’t going well. few weeks ago I lost about $70,000 in the trade. Can you please at least advise me on what to do?
Well, I picked the challenge to put my finances in order. Then I invested in cryptocurrency,stocks,through the assistance of my discretionary fund manager,
Such a genuine personality!! He is really a good investment advisor. I was privileged to attend some of his seminars. That’s how I start my crypto investment.
IT 101 ! Test the update before deploying, then slowly roll out update one machine at a time, always provide easy recall of the update to return to the previous version.
No it's really not IMHO. The people that needed to hear of them (massive enterprise) already knew who they are and now they will be looking at their competitors. You may not have heard of sentinelone or other similar companies in this space but security execs at the 100s of fortune 500 companies that were using crowdstrike have. This is a monumental screw up that is not a mistake, this is indicative of underlying issues in crowdstrike business procedures.
It’s not a antivirus. It’s a cloud soc tool which monitors network traffic for hackers. They have an agent which can respond to threats. It’s the agent that updated and blew up all machines
Most reputable software companies have a thing called quality assurance. It basically means getting someone other than the person that wrote the code to test the code BEFORE it gets sent out. It would appear that this whole thing is a failure to do basic quality assurance.
Microsoft used to have it. Then they fired them all and let their customers be the beta testers. Almost a decade ago, when Windows 10 was still being rolled out, a Microsoft Sales Exec got caught telling a concerned Enterprise customer that the updates would be rolled out to their home and small biz customers first to catch any bugs.
This wasn't a Microsoft testing gap but the update is released by Crowd strike. Just like updating chrome on windows is released by google and not Microsoft
QA is an antiquated process that needs to be phased out. It slows down the rollout of updates, and stifles progress. The sooner QA is eliminated, and the faster we employ tight-integration of AI with kernel level access to network-wide resources, the sooner we can reach singularity and get off the path of entropy. If Warbands was rolled out as soon as it was done, without archaic "QA", then we could already complete quests for any character you want without impacting Reputation or progress; whichever character completes them first will earn the Reputation for the Warband, which is all we want!
@@rocketraccoon1976 rather than play that old record over and over again, you should do some better research and read up on what Microsoft actually did, why they did it and what the overall impact was in the end. Because when you see the statistics of what they have done, you can only say that it was the right decision.
2 minutes and 22 seconds it took BBC to even mention the culprit, CrowdStrike. Though they mentioned Windows immediately. Why bury the lede? The CEO of CrowdStrike was the CTO at McAfee in 2010, when a similar thing happened. This time, he chose to lie to people, claiming that the "fix" would be automatic. When in reality, each individual computer would need to be booted in Safe Mode, and the bad file deleted. Can't be done remotely. And IT people can't travel because of the outage. I'd bet there are some secretaries being guided through this by absent IT people.
*100% THIS WAS AN INSERTION ATTACK* its was a bit of code with 00000000 00000000 00000000 00000000 00000000 at a critical location Some p'd off employee inserted it after testing but before it shipped is my guess, especially as they just ****** 2,000 employees out of severance pay to boost the stock price
@@pillettadoinswartsh4974 Jesus that's insane didn't even think of that, this one s soo negligible I can only think it was planned the update had zero bytes
A hacker could only dream of an "attack" of this magnitude, wow The fact the the only way to recover its a manual reboot into safe mode and removing a *sys file is just beyond believing. Its like all computers were infected with a evil root kit
This happened because executives didn't want to pay as much money testing this update before releasing it. I believe this is what business majors call "fiduciary responsibility." One of the executives at Cloud Strike was previously at another antivirus company that did more or less the same thing. This is what happens when you reward failure at the executive level.
The update was caused by a corrupt file, not a code bug. Testing won't necessarily catch that. It's still an unforced error, but it's more that the client software didn't validate the file and that the update rollout wasn't staggered.
@@xxgna corrupt file, by your terms, that crashes systems, can 100% be tested. It literally crashed systems. You think testing it wouldn't make it crash the test system? Does 2+2 not equal 4 to you? How many COVID vaccines have you drank?
As an alternate theory, what if this faulty update was deliberately pushed? It's quite unlikely that a well known cyber security firm will make such a silly mistake.
People have very short memories for failing technology. Me included. A few years ago someone dug a hole in the main water pipe down the street leaving us without tap water. After that experience I bought 6 bottles of water so I wouldn't be without drinking water again. Of course that water got used (and not replaced) so when the water pump in our apartment building broke down 18 months later I was once again without water. Did I learn from that? Nope, I currently have no bottled water in the house.
Your self-honesty is admirable. I'm going to subscribe to your channel and watch for a notice from a relative when your inability to adapt gets the best of you. I hope it won't be soon. Good luck.
Fortunately in my house there is big water boxes. So much so that i usually find out when there is disruption in the water supply only when the neighbors come asking for a few buckets.
@jbrc1322 I added water to today's shopping list, so I have once again an emergency supply. And I made the resolve (again) to maintain the supply this time.
When I was managing a large company, I wouldn’t let IT to do any software updates on Thursdays or Fridays. And we’d never do any updates before they had been vetted.
The Power circle of Engineers I sat with would meet bi-weekly to provide Change Control. Change requests (such as software updates/patches), no matter how small of the change, had to be fully tested for two weeks, FULLY documented, and the change request required a Backout procedure. All that and we only had about 40 engineers on staff! A few monopoly dollars to a big company like Cloudstrike. Sounds like they "sprinted" past any sort of testing.
It wasn't due to a bug in the code, it was tested as a code (I assume) It was bug in the design updating process of CS A) no checksum verification, a file got corrupted in the compile process nobody did a check for this B) they pushed the update to everyone Then there are MS design flaws... how a software driver can bsod like it's 1995
This! 💯, the amount of people saying "they didn't do a shred of testing", without: 1. Knowing their dev process/pipeline. 2. Looking just a little but deeper into the issue. Edit: Woodzta is right, if Falcon can't start up, it's designed to assume that the device is now vulnerable and shouldn't start up (I believe) and that makes perfect sense.
While true, the BSOD itself here is to prevent exposure to a manipulation exploit and I believe Linux would crash the same way under the exact same circumstances. Obviously, the protection works vastly different on Linux so this didn't occur (this time, but has occurred similarly in the past). Also, it's pretty easy to force a bug check. There's even a built in way to do so for testing purposes. Microsoft really do have very little liability here.
@@andrewtran9870 Of couse, they did not test it. A failure like this you just need to deploy to one PC to check that is broken. The issue is that some security companies just steps because of some speed paranoia. In conclusion, if we do not have better test that run fast plus a new way of making sure that this pipeline runs we can have this issue again and again. Basic necessities companies need to change OS, Microsoft should not be the default to avoid this situation. After this global disaster if companies do not do nothing it will happen again.
@@diogotrindade444 So far, we know that somewhere along the update pipeline, one of the update files became corrupted (all null). During the boot process, another piece of code attempts to dereference something within the file, resulting in an error in the boot process and the blue screen of death. It is likely that the "update", i.e., the software that was SUPPOSED to be in the update was thoroughly tested. But it was something during the process of pushing to production that went wrong. Until we receive further reports, we're making a lot of assumptions here. What if the error doesn't always happen when pushing to production? What if they did test on a number of devices, but the error with corruption never occurred? Yes, they should've tested the process or software that pushes to production more extensively. Yes, a rolling update would have minimised the impact. But to accuse them of ZERO testing? Do you really think the largest security vendor in the world would do that? Bottom line we still don't know what happened specifically and until we do we're stipulating an awful lot.
@@Woodzta there is no excuse, Microsoft should have implemented an ilo 10 years ago so that we could access the machines even if they are turned off, And then use that internet access to verify the integrity of their code, This is literally pathetic I am a veteran in c++ I write code for over 20 years, and also work on IT.. what I've described is what we do for so many other products
@Lashley-jp9bo shrieking that is fact isn't helping you champ... are you saying that have voting control over every f500 company and control over the day to day running... or are you getting swept up in internet hysteria.
As a former programmer, imagine how bad the programmers themselves must be feeling. I wouldn't go so far as to call them the devil and for being all at fault for it is a difficult job and it is easy to miss stuff. Although if something goes into blue screen immideatly after update is installed, then it probably wasn't tested at all. Sometimes deadlines push too hard.
I dont blame the programmers because programming mistakes can happen. I blame the QUALITY CONTROL/TESTING team for not doing their job (if they even exist at all). Av software runs in KERNEL MODE and thus 1 bad line of code can crash the whole system and thus KERNEL DRIVERS SHOULD BE TESTED FULLY before being released to prevent the kernel from crashing
Clearly you don't know what you're talking about, but just so you know CrowdStrike is not a cheap solution. Who knows, maybe after this our renewal might be in for a substantial discount.
@LandgrabbingIndia Using the Heroic Launcher and Steam, you can get just about everything running but games with deep level anti-cheat, anything the Steam Deck runs will work on Linux.
Let me just go over this as someone who is in IT and knows the best practices. So this has 24,000 or so consumers for btb. Each business would have to vet the updates but it is encouraged to have automated updates for stuff like this. Basically a lot of these major servers are critical so they trust this vendor to vet the updates properly. It’s kind of hard to say who is at fault when its critical, needs updates for security, but also isn’t given the time to proper time to test. Also, despite the blame pushing from a lot of these news outlets. It really is on the business side of things that should be blamed. The bigger question is why are these multi-billion dollar companies not investing in critical backups and disaster recovery processes properly. Personally sounds like ignorance to me.
I worked in a small kitchen hosp in nz, and i can only say i have nothing but respect for the staff of any hosp reliant on computers for menus, patient admissions, etc. What a nightmare
It has NOTHING to do with Microsoft! It was a third party security program that caused the problem. The same thing could happen to any operating system.
Windows is high risk since its start. It's a design problem in the Kernel and its update service as well as other problems. The blue screen of death has been around for too long. Critical systems should be migrated to Linux or Unix.
Ah yes, beause Linux doesn't have any issues. I love how everyone has forgotten just how close we were to the recent SSH supply chain attack landing into distros.
Which will leave the US with two cybersecurity companies, so that leaves us with a 50/50 chance of this happening again. There would also have to be a instant replacement. @user-zc5lf9xb2g
@@user-zc5lf9xb2g Funny thing about those SLA's... most of them promise 99.9% over a year, which is easy to do, don't be down for more than half a day at given time. Which they weren't they had an answer for it within hours. The application of that answer takes longer, but you can't sue for that part.
Spent about two hours stressing over this before finding out about this 😂 I was scared, this computer is two months old. I just kept saying "no way man"
After IBM's acquisition of Red Hat, the core values and mission is impacted thereby jeopardizing the quality of offerings and especially internal working environment of the company. I'm saying this as an ex Red Hatter
You have no clue what's going on. CrowdStrike on Linux auto-updates too. You are probably confused with system updates. Mint users are the lowest form of Linux users so I don't expect you to know much.
@Yxalitis windows 7 stopped getting security updates in Jan 2023, if they didn't stop windows 7 updates we would all be fine..in my ill informed opinion
True, but this isn't anything to do with Microsoft. It was a 3rd party security update to their security program. The same thing could be caused by any other brand of internet security software.
This amounts to criminal damage on a global scale. I was not that badly affected except for NHS GP systems have still not recovered. I wonder if there should be criminal prosecutions?
and an advice from an expert I saw on television, who worked for the government as white hat (good hacker if you so will), don't come here telling what system you run, the black hats, the bad people down underneath the real internet, are already waiting to hear that from you and waiting to use the opportunity. just stay calm and quiet. These are dangerous times concerning the dark internet.
Resiliency means having multiple (not just one) Plan B and ways of doing things. Don't rely on only technology. Carry cash as well if you want to be able to get coffee on a day when all the banks are down or cashless payment systems are down. And merchants: this means you can't be entirely cashless; you need to be able to transact with good old cash. See how both sides must be resilient?
All this technology and look how it’s brought down by a single update from a small no name company. Now imagine a meteor strike or a global catastrophe. We are doomed
Most software updates create more problems than they solve (eg, unwanted features; lost settings; etc.). This current [ _CrowdStrike_ ]fiasco is a well-pronounced example of why software updates are cold pieces of h3ll.
I dont trust the credit card system. I had more then one case where i couldnt pay for anything at the register because the system was down. I love my physical cash.
@@sundhaug92 I see, probably a bad idea. Where no user is able to touch or has a tightly controlled UI with a watchdog if that dies,, other measures seems to be a better approach than rolling out standard Windows images with anti virus.
@@gentuxable Vpn and subnet are two totally different things your an IT noob with no clue what your talking about. How tf is being on a different subnet if it's still on the network going to prevent anything ?
@madhurgupta854 if you have any Windows machine that needs to get information over the internet you may want to protect it from any other machine that can attack it. The best way I know to protect while still having it connected is by using a VPN so that it tunnels all communication over one controllable path. So an attacker needs to breach the VPN first in order to attack the machine that could possibly be used in a botnet.
All nowadays think : cloud is now more safe, secure and on premise technology is outdated. This outage will definitely make IT auditors across the globe to rethink. Before crowdstrike rolled out this patch, they could have atleast tested it in a windows based network locally. Lack of testing and overconfidence on their product has caused this chaos
Well the thing is that with Y2K you had a calendar-date you could plan for, and millions of dollars were spent on preparing systems. This just happened without warning.
This is why I have no problem with Apple's standards for the App Store and them being exclusively what can go on my device, it's not monopolistic, it's quality control.
This is a wake-up call for those who allow kernel based software into their system. The people who warned about the risks of it rang true. I just didn't realize Crowdstrike is this huge until now.
All this things aren't just happening over the last 5 years by coincidence. I've got 30kgs of rice to drop off at the food bank as its ready to go out of day, could you feed you're family without a debit card?
@luka1790 I certainly have an opinion which is probably far from correct, my point don't take for granted the as ability to go to a supermarket for life sustainment or even use you're debit card if they do have supplies.
@charlesbenca5357 You are completely ignoring my point, if you understand what's happened so well then surely the brightest minds in tech would've been had fail safes in place ...
Yeah, somehow a channel sys file got zeroed out before being pushed to clients. Since it’s considered required, Windows will refuse to boot when it can’t run it. It makes me wonder why they don’t canary their updates. It would greatly mitigate this type of failure. Why are they pushing it to all their clients simultaneously.
@GH-oi2jf I agree in that I don't see why a rolling update would be a bad thing. But I think automatic updates makes sense. Ultimately it is up to the customer to decide whether they want to use a product with automatic vs manual updating. However, MANY MANY companies have poor updating procedures and policies. And when it comes to security, we don't want a zero day to be left unpatched because companies failed to stay up-to-date with the news (as there would be a lot). Thing is, for a lot of companies it just makes more sense to have an external 3rd party manage the updates for this sort of thing. I.e. who better than the leading cyber security firm, CrowdStike, themselves... of course, this is now being called into question. But I still think it's the better approach for the vast majority of customers. Airlines, banks, defence, though... you would hope they have sufficient resources to manage manual updates and version control themselves. But the risk is still there: "do I stay on this version with a zero day? or do I immediately update to get the patch?" Of course the latter.
@jordank249 could be... but crowdstrike has had hacking issues before, at this point even if it was hacked they can't admit it or they would have big issues company wise... it could be simple update, wild to think one thing could effect so much is rather dangerous. Even are load system for trucking company I work for were unable to process loads yesterday, scary tbh.
