Actually it wasn’t a faction of a percent it was drastically slower than it should be. If I recall it was like .5s slower than it should be which is a lot in the computer world
It is amusing that, for benchmarking purposes, the engineer who found it was sending SSH requests that shouldn't even pass a sanity check ("wrong username, etc") which explains why he got suspicious of some excess cpu cycles so quickly.
I think Seytonic covered this a month ago. But it doesn't hurt to remind ourselves: 1. Social engineering is a thing, 2. Pay developers what they are worth.
I'm going to have to say: it's not an adage, it was a (very real) threat to Margaret Thatcher. It's still applicable in this case, just a better phrasing might have been: I'm reminded of the IRA's threat to Thatcher: "Today we were unlucky, but remember we only have to be lucky once - you will have to be lucky always."
The fact this was just discovered by chance really brings into question how many other packages have similar backdoors. This is the kind of stuff that should spur a major investigation.
@An_EqualNot the FBI or CIA obviously, one of the founders of telegram said the FBI was trying to trick him into using open source libraries for telegram.
Like Andreas said himself, this was just incredibly lucky, just a massive coincidence that he happened by chance to be in the perfect position to find it (and _just barely_ in time). A confluence of events like this rarely happens, so it's possible that there is indeed a lot of stuff going undetected. 😕
There is a "major investigation". This backdoor has sparked discussion on how to prevent something similar from happening again, made some free software contributors try to audit other software and once again demonstrated the absurdity of a "software supply chain" where the companies don't pay a dime to their "suppliers" yet expect them to do the most rigorous work to avoid hurting their (the corporations) bottom line
This is the biggest weakness with OSS, but also the greatest strength of it. Anyone can worm their way into a seemingly innocuous part of the Linux ecosystem and taint it. But also anyone and everyone can topple years of nefarious actions through simple curiosity.
the fact this got caught, while Windows CVEs get put out and many admins don't update, leaving vulnerabilities in place for years! Notpetya took advantage of an years old vulnerability in Windows, and caused over $11 billion globally
Yeah nothing's going to be 100% secure at all times, the payout from successful attacks is just too big. What OSS has is a living immune system, the ability to heal.
Technically no matter how strong your password is this back door completely bypasses all passwords because it injects the hacker’s ssh keys onto the infected device.
I don't know how you got my passwords but you don't scare me I already changed it into something more secure. With six digits it almost impossible to guess mynew one.
The whole linux sphere has been talking about this a lot, but yeah, I think you're the first tech channel with a more general focus I've seen bring it up.
@@UmVtCg I wouldn't say I'm in the Linux sphere and I'm definitely not in the cyber community and I still knew about it. It was pretty much impossible not to hear about it if you're in the "IT scene" in any capacity.
As a software developer, I have no doubts that this kind of vulnerability (probably multiple) is already deployed everywhere, undetected. Never underestimate the power of social engineering, and these attacks being very easy to miss. Also I remember when ThioJoe had very few subscribers, I'm delighted to see the channel grow like this. I wonder if he remembers me 🤔
Not so crazy when you consider that at some point some one probably got themselves hired in order to put in the Juniper back door. This was found about 10 years ago.
Jia Tan is a Chinese name, Jigar Kumar is an Indian name. People who want to stay anonymous won't use their names, but also doesn't want to introduce a rival nation to investigate (so he didn't use a german name for example), so quite likely a hacker of russian origin. Isn't geopolitics wonderful.
So the backdoor would likely quietly delete itself if it detected a Russian keyboard. Making it legal for your citizens to attack any system as long as it's not one your nation owns is a stroke of genius, ngl. I wonder how much money they've saved on buying day zero exploits from the usual sources?
'Cheng" is a cantonese name while "Jia" isn't. This indicates however made the backdoor just tried to think of a name that sounds Chinese enough. Such sloppiness is typical of the US But trying to deduce the perpetrator from the name is stupid anyway, we could go in circles all day talking about potential 5d chess by the perpetrator
As someone on the defensive line working at scale (170,000 users), you do what you can with the control that you've *got* to avoid these issues, but you are mostly at the mercy of others. Where you *really* need to focus your efforts as a defender is being able to detect *when* you've been breached. Our goals are pretty clear - detect within 10 minutes, contain within 60 minutes. That's how fast you need to be, and some would argue that's not fast enough.
My Dad's mate was managing server infrastructure at a hosting company around 2010, and decided to deploy a crypto miner as a cheeky experiment for his team. It was a bit after a fortnight when the team found out, and they chewed him out for misusing company resources, but he immediately returned the blame to them. 'You're saying, if there was actually a piece of malicious software running on our systems, it'd take you two weeks before anyone realizes something's wrong?'
Nothing had made it to the news where I live regarding this. Some tech channels on YT I follow covered the bare bones when this was first discovered, yet the background you've provided has created such a broader and more chilling account of what was really happening.
We've gotten so spoiled with our technology, we need more code and more programs and more features to cover every base. Thing is, the more we have, the more hands and minds work on the code that run on our machines. That definitely comes with its risks. The truly scary thing to think about is that... logic dictates that the worst is yet to come.
Like the xkcd comic Joe showed said, there are a LOT of bits of archaic code that underlie the world's software. We've seen cases where half the Internet broke because software relies on a single function that someone wrote for themselves 25 years and everybody copied. Software is more fragile than people would like to think.
At first I thought this video is quite a bit late. I've already seen multiple videos about this backdoor right around the time it was discovered. However I'm glad I watched till the end because this video provided some additional information and context I didn't know of yet.
Remember: As an open source maintainer, you should keep an eye on the stuff coming in and just not accept incoming stuff if you don't know WTF it even DOES. (That's the technical term.) But I also realise that if you have relinquished the nominal control to someone else, you're not culpable.
I fully agree with this, but the problem was these so called contributors were intentionally bringing up so called "problems" causing the developer to burn out. Nobody remembers that these people do all this without guaranteed pay, they volunteer their time to better the open source atmosphere. It's really sad that people have to take advantage of good hearted people like this. This is why I always chip a few dollars their way whenever I can. We should try and keep these people happiness high. The actual owner was on Hiatus and gave the reigns to a person he thought he could trust, well that person was taking advantage of his burnout.
My first contribution was the Harvard's cs50 class CLI tool, translating it to my language Indonesian so my high school students can use it more easily. The maintainer raised this exact issue, "how do we know he pushed something legit, not troll translations?" That's how I realized that while open source contribution is a cool way to collaborate, some people might have malicious intentions and maintainers should try their best to prevent it.
@@dputra They could probably chuck it into deepl translate and most of it would make sense. Having a native speaker translating seems like an improvement over any automated translations though.
I'd heard of this, but not the full story. Essentially just heard "some guy was drag racing his computer for fun and noticed a tiny inefficiency which was a brand new back door, catastrophe prevented" and not all the cool details you gave! Thanks for this video.
@@milentoshev8409 Their antivirus software became the virus. Granted it was opt in but there were multiple popups urging you to opt in telling you how great crypto is. They failed to mention about the wear and tear of hardware and the performance impact on other tasks. To top it all of they would not only skip paying the electricity bill they also took a 15% cut from your earnings
@@milentoshev8409 I dont remember all the details, but norton or one of their products had or has a crypto miner within them. They stealthily made it opt-in by default, and when found out tried some justification.
This backdoor only affected amd64 systems (so ARM computers wouldn’t have been affected) and it would likely take some time before it got into Debian and Ubuntu LTS (used by a ton of servers), as they only receive non-security updates every ~2 years, so if it was discovered 1 month later, we would probably be fine.
1. If I remember correctly, there's a code that check specifically for amd64 (and x86?) architecture for it to run. (sus imo) 2. We're very lucky that the backdoor was found before it was released into stable Ubuntu LTS 24.04 release on April 2024. That might be the attacker's main target. 3. The fact that it was found by coincidence by microbenchmarking, ~500ms delay, is very concerning. 4. The attacker will learn from this mistake and might pull something like this again / another party is inspired by this move will do it in the future.
I wouldn't really say this "just" happened, it was a decent bit ago, but yeah, it could have been bad. One thing that's kind of annoying though is, well actually sort of two things, but both are alarmism 1 : "HOW MANY MORE COULD EXIST THAT WE DON'T KNOW ABOUT?!?!?!" No, this was discovered like a month after the impacted version first released precisely *_because_* everyone had eyes on it. This is the point of open source, *_everyone_* is watching, so millions and millions of tax dollars mean nothing. One random dude doing some basic benchmarking spotted that what *_should_* have been a basically instant no-op was taking over half a second and a decent chunk of compute, then identified this backdoor years in the making. If you are wearing a bulletproof vest, then get shot, and it gets stopped, you don't then say "woah, imagine how many other times I've been shot and I don't even realize it! I mean sure this vest I put on specifically to stop it stopped it, but that was just pure chance!" 2 : "See?! Open source isn't more secure!", except, 2.1 : see above, 2.2 : this exploit specifically abused how XZ was being distributed to have what is basically a closed source component be delivered with the final product. We do not have the source code for the exploit so, definitionally, the exploit was not open source. Since it was part of the project, that means part of the project was not open source. This attack had to reinvent the wheel several times over to hide itself ( and, again, still got caught instantly ) precisely *_because_* it was in an open source repo. 2.3 : your jordans are fake; this argument relies on the tacit assertion that this is an attack that hasn't been carried out on proprietary software several dozen times over. Not finding a vulnerability or not having it disclosed isn't real security, it's just feigned security. The reality is this is a prime example of exactly why open source is so secure. Allllll of this time and money, full fledged psychological warfare, completely innovative attack vectors, etc. were all rendered completely and utterly meaningless because, against the millions and millions of nerds running automated checks and test scripts, they *_are_* utterly meaningless.
That's exactly how I feel too. Tons of doomer talk over this attack even though literally nobody was impacted lmao and the attackers wasted years of effort to be shutdown over night. Malicious code can't just magically silently interact with software and hardware. It always has some impact on the system via the fact it has to run code, and that can be detected. Files not in the source code can be found with programs as simple as winmerge. This was always a doomed attack vector hinging on luck that no one would notice before they hit their target.
@@uponeric36 I wouldn't quite go that far, this is a very real attack vector that does need to be addressed. The fact that the wider open source community was *_ever_* okay with non-source-controlled files being included in the release builds is a massive red flag, as is the fact that build pipelines aren't themselves included in the repository. This wasn't a purely 'luck' attack, it did exploit some legitimate vulnerabilities with the culture around open source, particularly around people's willingness to accept "ohh it's just a hobby project their not getting paid for" as an argument that works in tandem with " *_the entire linux ecosystem relies on this package_* ". The reality is the build and release pipeline for open source *_has_* been overlooked as meaningless tertiary shit, when that's far from reality. There are some legitimate security holes in that area of how open source is typically managed, but that doesn't mean it's doom and gloom. Those are vulnerabilities that we should address, but my point is that a lot of people treat this like it's some cautionary tale when, by basically any account, thjis was handled borderline flawlessly. Sure, there was some evidence going back years, but if we're honest, go back to January of this year and all of that evidence is circumstantial and non-actionable. It was a few dodgy accounts that might be sock puppets, a few odd commits, etc. Realistically speaking this was caught like a month after it was even possible to catch it. There are definitely some lessons that need to be learned here, but it's not the sign of the end times so many people are acting like it is.
@@robonator2945 You're right, I'm just layman lol. It is wild to me in retrospect that having files like that in the final build is "part of the culture" I still feel like this was a perfect storm moment - the attack formed partially by negligence, but also against the odds of all the other security measures in place. Ultimately, the odds worked against it. There's what I was considering luck, but "pure" *is* a strong word.
Hey, I just want to take a second to say thank you and congrats. I found you ages ago through all the pranks. Was funny at the time, but I can see why you moved away from it. Over time, you've given us some really amazing videos that are very informative and make it easy to digest for those who are less educated on tech. Thank you for the years of entertainment and information and congrats on how far you've come. Much love, bro.
Heard about this the other day on the 2.5 Admins Podcast and the Late Night Linux Podcast, good to hear from some other people. It's a pretty big deal.
stuff like this is why I'm so serious about secure coding. Put as many self-checks in your software as possible. For example, something that might've protected against this attack: don't load libraries that don't match the checksum you expect. if that library that was used by SSH had been checked for integrity (which likely could have been done with libraries SSH was already using, since it's already doing some cryptography) this attack would have failed
I knew about the backdoor since the time it was discovered in March 2024. The backdoor was discussed, it seemed, everywhere, I also watched a few videos which explained what it was and the consequences if it weren't discovered in time.
This would make me paranoid about security if I wasn't already paranoid from the time I (temporarily, to test something) opened up SSH access over the internet to a Linux machine on my network and saw it immediately get hit with constant brute-force login attempts.
Imagine the backdoors and obfuscated malware code we _don't_ know about. Too much code to review in a time where people are barely paid to do the minimum requirements. A ticking timebomb.
It's kind of reassuring that software improvements make backdoors in other projects obsolete. If the good guys keep fighting, they can plug holes they don't even know about.
The 2 scariest things for me are the realisation of how connected everything has become with a backdoor happening in a software in the corner of the digital web somewhere capable to potentially affecting us all dearly, and how unfortunately fragile yet essential open source can be. It is really interesting yet terrifying to think about. 🤔 Thank you Thio for this video!
As an IT Professional for over 30+ years, the world is a SCARY place. I just wanna go live in the mountains. Tired of playing whack-a-mole all the time. No matter what we do, there's always someone or something else out there that is better.
Just casually acts like every server on the internet uses Java. That said, this kind of supply chain attack showcases one of the biggest challenges of open source software, in my opinion.
Microbenchmarking, huh? What do you think the odds are that a different state actor has been monitoring the codebase and looking for inserted backdoors. They might even have been behind the security enhancement to the other library that would've disabled the backdoor and only revealed it publicly when the backdoored library's release looked like it might be getting pushed forward.
I just came up with a theory. The backdoor was actually discovered by some counter-intelligence agency obtaining information about this backdoor but not wanting to burn their asset, they came up with a cover-up story about some Microsoft engineer noticing a 100 millisecond delay.
this would have only affected systemd systems running openssh (with custom patches for openssh). What remote access tool should people be using to control their remote systems?
@@dekeonus VPN, and yeah... One can run anything through a vpn tunnel even an SSH connection. As I said no firewall or edge device should EVER have port 22 exposed directly to the internet.
@@UmVtCg I think you've overstated the risk. You've just switched one attack vector for another: With SSH one should be using ip[v4|v6] whitelisting, restricting to certain users (or group) and only permitting public-key auth (and possibly gssapi) your attack surface is largely similar to a VPN.
Not going to lie, but I was lost about a minute into the video because I know squat about computers. I just dig your videos because they always show up in my feed. Who knows, maybe I am slowly learning things. What would be the point of doing something that malicious? To put that much time and effort into something that destructive, there must have been something to gain, was it money? A form of terrorism? It’s sad that people with that kind of talent can’t do something good with it
Everybody about Open Source: „It is so secure because everybody can REVIEW THE CODE to make sure that there are no backdoors and stuff.“ University of Minnesota in 2021: „Hey, let‘s commit some code that contains vulnerabilities to the Linux kernel! For science!“ The maintainers didn‘t notice and the students had to tell them what they had done. Has nobody REVIEWED THE CODE? XZ attacker: „Welp, if it wouldn‘t have performed so bad in a benchmark, they wouldn‘t have figured it out. Because nobody found the backdoor by REVIEWING THE CODE beforehand.“ edit: typo
So one thing to add, this was included in rolling release ditros like Arch as well, but my understanding is that, the way Arch used xz and the way Red Hat and Ubuntu used it were different enough that it wouldn't actually effect Arch systems
I once thought about open source risks and then forgot, since a whole community watches what changes... but if the hacker is patient (2 years!), it finally can happen... No I'm not afraid at all. _casually updates USB hub firmware, nothing bad can happen_
I saw this with another OSS project where Bitcoin mining was added to a library which was being used by a commercial project which was used by the company I used to work at. Our Anti-virus caught it being installed by the Dev and the company that put out the update had to release a new update.
It's even bizarre that Jia Tan contributed to Google's repos to disable or remove some packages so that his backdoor doesn't get added to Google's repos, he obviously knows that they'll be able to detect his malicious code. Very meticulous indeed.
Bitdefender is good stuff, I have Win 10 on two desktops, but I'm usually on either machine using the main hard drive which one is Linux Mint's LMDE 6, and the other machine is Linux Mint's 21.3 Cinnamon. I have Bitdefender on my Mini PC that has Win 11 Pro on it, I have Bitdefender on my phone also, it's one I recommend to clients of mine also. It's a shame they don't make a version for Linux that would have caught this XZ problem. I rarely use Windows, so this one hit close to home hitting Linux with a well used tool.
Maybe packages that are being maintained by one burned out guy who only does it sporadically as a hobby and isn't consistently communicative or responsive should get phased out of use instead of kept as pieces of vital infrastructure. Just saying, there's a lesson here, and we're not learning it fast enough: when you create software people really depend on for important things, don't set yourself up for one random guy's old piece of software to be put into a multiplier-effect position where it can cause absolutely horrible consequences in a widespread manner.
So what I'm getting here is that they found out that XZ had one dev, and that dev was a target because of their health. And then they put pressure there to get themselves as maintainer. xkcd memes are just reality, I feel I see this with all software closed or open. There's like one thing that everyone relies on but has zero resources put towards it. It would be nice to see these distros run by big teams to go over what they use. And send resources to software like XZ.
Yes, I have heard of it before. I watch a few IT channels. Your unscheduled video yesterday reminded me about this backdoor in that it looks like certain actors are attempting long cons to create vulnerabilities.
I knew about it during the Easter weekend thanks to a general channel Discord and some Linux/programming youtubers. Arch really quickly updated the package and posted to their news page when discovered. Also just happened to be 1 week after I updated Arch WSL for a Samba setup involving Windows 98. OpenSSH does not normally use liblzma but got patched by Debian/Fedora/systemd systems to work with libsystemd which did use liblzma but then 7:58 this pull request was going to make systemd not automatically load liblzma all the time which pretty much doomed the backdoor.
Off topic Thio but as an Australian I felt obligated to flag the importance of regular melanomia checks given your complexion. Thanks for your work spreading knowledge to keep us safe online and make sure you're looking out for yourself in the real world too :) PS to answer your question, yes I had heard of it but not the detail - wild stuff. It sounds like there's an architectual change required to harden OS's against these kinds of attacks. This should be a wakeup call for the tech giants that they should start making sure that the maintainers of core utilities in FOSS are adequately resourced - without any strings attached. Perhaps some kind of FOSS security group funded by for profit users of open source software that does white hat work across the core projects that underpin the internet.
I did hear about this through mainstream media but I don’t remember where. I do recall the detail of it being discovered at Microsoft but there was no more detail than that.
The US Government has programmers. The Army has programmers. I almost went the programming rout when I joined up, but I went into a different MOS. They created America's Army, the game. The US Government (and other governments around the world) should be writing their own operating systems.
