Welcome to episode 269 of the Cloud Pod Podcast - where the forecast is always cloudy! Justin, Matthew and Ryan are your hosts this week as we talk about - you guessed it - the Crowdstrike update that broke, well, everything! We’re also looking at Databricks, Google potentially buying Wiz, NY Summit news, and more!
Titles we almost went with this week:
• You can’t take Justin down; but a 23-hour flight to India (or Crowdstrike updates) can
• Google wants Wiz, and Crowdstrike Strikes all
• Crowdstrike, does anyone know the Graviton of this situation?
• We are called to this summit to talk AWS AI Supremacy
• Crowdstrike, Wiz and Chat GPT 4o Mini… oh my
• An Impatient Wiz builds his own data centers not impacted by Crowdstrike
A big thanks to this week’s sponsor:
We’re sponsorless! Want to reach a dedicated audience of cloud engineers? Send us an email or hit us up on our Slack Channel and let’s chat!
General News
00:58 You Guessed It - Crowdstrike
Microsoft, CrowdStrike outage disrupts travel and business worldwide (www.washington...)
Our Statement on Today’s Outage (www.crowdstrik...) (listener note: paywall article)
• It’s not every day you get to experience one of the largest IT Outages in history, and it even impacted our recording of the show last week.
• Crowdstrike, a popular EDR solution caused major disruption to the worlds IT systems with an errant update to their software that caused servers to BSOD, disrupting travel (airplanes, trains, etc), governments, news organizations and more.
• Crowdstrike removed the errant file quickly, but still the damage was done with tons of systems requiring manual intervention to be recovered.
• The fix required booting into safe mode, and removing a file from the crowdstrike directory.
• This was all complicated by bitlocker and lack of local admin rights for many end user devices.
• Sometimes doing up to 15 reboots would bring the server back to life.
• Swinging the hard drives from one broken server to a working server manually removes the files and puts them back.
• The issue also caused a large-scale outage in the Azure Central region.
• In addition to services on AWS being impacted that run Windows (Amazon is a well-known large Crowdstrike customer)
• Crowdstrike CEO Goerge Kurtz (who happened to be the CTO at Mcafee during the 2010 Update Fiasco that impacted Mcafee clients globally) stated that he was deeply sorry and vowed to make sure every customer is fully recovered.
• By the time of this recording, most clients should be mostly fixed and recovered, and we are all anxiously waiting to hear how this could have happened.
04:50 Justin - “It’s really an Achilles heel of the cloud. I mean, to fix this, you need to be able to boot a server into safe mode or into recovery mode and then remove this file manually, which requires that you have console access, which, you know, Amazon just added a couple of years ago.”
07:45 Matthew - “It’s always fun when you’re like, okay, everyone sit down, no stupid ideas. Like these crazy ideas that you have, like end up being the ones that work, but you would never realistically have the opportunity to try them because you know, one, how often and God, I hope in your day job, you’re not actively logging into the serial port for fun or how to automate your deployments. Just sounds like you’re doing something horribly wrong at that point.”
15:20 Justin - “I saw that article this morning about the EU might be the reason why Microsoft doesn’t protect the kernel more. I think that’s a cop out. Basically the EU saying we want fair and equal competition. And basically what Mac did or Apple did was they basically created a custom API that basically does what CrowdStrike needs to do in the kernel and provides that to serve to CrowdStrike and other vendors. They’re all on equal footing. They all get access to the same API. They can all implement the same features, but Mac controls it at the API.”
22:09 Google Has Been in Talks to Acquire Wiz for $23 Billion (www.theinforma...)
(listener note: paywall article)
• Over the weekend it was rumored that Google has been in talks to acquire Wiz, a four-year old cybersecurity startup, for around 23B. The deal could stil...
12 сен 2024
