Why Hundreds of Mazdas Tuned to 94.9 Broke Simultaneously 

i live in Seattle, and would like to point out that while all of this was happening they kept joking about holding the Mazda drivers hostage. "thanks for tuning into KUOW, unless you're in a mazda, because you have no choice". good times

Cellist here: Sometimes we still get hired to play in dashboards, but the number of folks willing to pay such a high rate gets smaller every year. Support your local arts. Also, that's a double bass being played with a German bow, not a cello with the standard French bow. I expect to see this in your next corrections video.

This issue was 100% preventable, just by following LONG established programming practices and having someone review the design and code.

This reminds me of a case in Finland years back when some TV towers, due to configuration mistake, broadcasted completely corrupted data, essentially random data, for some time (I think it was a few hours). I've forgot details, but basically there was one specific model of set-top-box that apparently did a poor job with checksums or something, and at some point the random data happened to make up a sequence that the set-top-box took as a command to get a firmware update that would follow. That didn't end well.

That % formatting bug is a huge issue. Whatever programmer caused that absolutely should have known better.

I've made a few coding mistakes in my time as a dev but "hardware destroyed due to unexpected input" isn't one of them.

Having worked in IT for almost 20 years, I can very confidently state that I *DO NOT* want my cars to have any type of connection to the outside world other than the OBD port and only the most limited computing power necessary. While there are actual advantages to having certain systems being electronic vs purely mechanical, there are many systems that have absolutely no reason to be computerized and I feel like it's just a way to force owners to bring them in to the dealership because they can no longer diagnose or fix it themselves. And a justification for charging thousands more than anymore needs to pay for a car. It is also a way for manufacturers to institute control over what you can do with 'their' products. Your car is not yours even though you own it, much like most PC software is not yours - Adobe (for example) just lets you pay to use it for a while. Sadly this practice of turning everything into a service is only going to get worse, see for example the subscription to the heated seat or remote start 'service' you will need for some new cars. Stop paying extra every month? Remote start stops working, in spite of still having everything you need for it. Don't even get me started on Tesla and the battery life thing...

Having worked in vehicle infotainment engineering, I find it amazing that worse doesn't happen more often. ROM images on these head units can take hours to compile, have software from tens of suppliers, and end-to-end testing is practically non-existent, let alone exhaustive suites of edge-cases like this.

Car companies: "We take cyber security very seriously. It's impossible for a hacker to hack your car... trust us" Also car companies: "If you use a file without file extension it will literally brick the onboard computer"

Preventing this types of errors is pretty important, but being able to recover from them is far more important. This is why we have these hard to reach factory reset buttons.

Tech info: HD Radio wasn't the first radio standard to be compatible with a digital download service. FM radio had RDS back in it's later days. I remember radio stations using it mainly for two purposes. Firstly, to give the listener the station's name on their radios (applicable when the listener has a radio with a digital screen and RDS support). Secondly, to allow the listener to jump between the station's broadcast towers if the listener is on the move and moving between ranges of the station's broadcast towers (Applicable, if the listener's radio has RDS and AF support).

1:56 RDS, or Radio Data System which was developed in the 1980’s could send things like what song you’re listening to, or traffic information, or even sync your radio clock with the data received. So FM radio did have a way to send at least some type of data to your radio.

I love this channel so much. The humor is *terrible*, which is actually great!

A random manager at Mazda: nah, this is too weird, it'll never happen. The Engineers who found the bug and reported it: I told you so!

“When criminal irresponsibility becomes an industry norm.” Really, we need to stop telling people that software engineering is easy and anyone can and should do it. It's as hard as any other engineering profession and should only be attempted by those who (a) actually know what they're doing and (b) understand and take seriously the many risks they're subjecting their users to. We honestly wouldn't suffer from having less, higher quality software in the world-software re-use is a virtue for this very reason.

Okay, correction to this article -- the Mazdas affected by the issue only needed to be factory reset using a USB drive with an update. They did NOT have to be totally replaced to fix what amounted to a software bug. Mazda had a solution ready within 3 weeks and dealers were deploying the fix fleet-wide that month. Most cars didn't see this bug ( obviously ) and had their infotainment systems fixed using an "OTA" ( over the air ) update for cars that had mobile connectivity -- or a USB based upgrade that could be downloaded.

At 0:20 in the video, the stock clip is of a radio I spent 3 years helping make for GM as a consultant to a supplier. Fun fact: That radio was awesome for its time (2006~2013)... three models capable of RDS Text, talking to XM radio and OnStar modules and it handled all the various sounds your car needed to make (chimes, turn signals, etc...) The two upper tier models played MP3 tracks from CD-Rs, the top tier had a 6-disc changer built into the unit. Could a percent sign in a message mess up that radio? Possibly... I only had a direct hand in the radio's code for a small bit, most of my work was to support the radio from a development, QA, and service perspective. Anyway, that was my "cool story"

The percent issue is because a lot of programming-languages use % for formatted text for example "%d" means "take the next parameter, and add it to this part of the text as a decimal number. If however there are no more parameters to use, things will crash. Because format-string vulnerabilities are so common, some tools have started to ban user-controlled formatted strings entirely.

Back in "the old days" you got in your car, turned a knob to turn the radio on (click) and then turned it further clockwise to adjust the volume to the desired level. You then twiddled the other knob until you found the radio station you wanted. And you could do all this usually without taking your eyes off the road and you got simple AM music. Then clever technicians then started to outdo each other and made mechanical push button radios to store and change the station by pushing one of about five different buttons that you had to program in. Then someone else decided to make the tone adjustable, there by making it even more complicated and tedious. Then came loudness adjustment and then multiple speakers and balance controls, left and right, front and back. Then came FM radio, so more complications. Next was a cassette tape player included which regularly chewed up the new tape you had only just bought. Then it was music CDs ; more complications! Just when we thought it couldn't get any worse we had a second FM choice, FM1 or FM2. And then it turned into a bloody computer that had to be programed for about 20 minutes and it all erased if the car battery was disconnected. Old people had to get their grandchildren to program these SOB devices. So, I don't listen to the radio anymore, it's all too hard! Oh for the "good old days" with two knobs when you could just go click it on and twiddle the other knob to tune it in!

I'm from Germany and during my uni time here I learned about how cars need to have certain fail-safes and risk assessments and need to fullfil certain coding guidelines and have fallbacks in place. How on earth is this possible? Well I guess in Mazdas case they determined: "What's the worse that could happen if someone had no infotainment? Not much, they'd be bored."

I especially appreciated the five-second aside to shit on NFTs. It's like a national pastime at this point, I love it.

I had a colleague who worked at Mazda on the infotainment system, and I forwarded that podcast episode to him. He was thrilled since this sort of thing was his team's job and they had no idea.

I'm old school, I repair things that break. Instead of just throwing it away and spending another $1500 for a hard to find an original, try having someone trained in electronics ( like me, 40 plus years) actually finding the reset pin on the microprocessor and physically resetting the radio. DOH! We are such a throw away society.

4:53 At least *she* is holding the soldering iron *the correct way* ! Good on you, HAI, for sourcing a real-life-compatible stock video clip! 😁😁

"These sort of software bugs are going to have higher and higher stakes as cars become more and more computerised" As a software developer, I'm keenly aware of this. The fact that some cars today are entirely incapable of retaining their basic functions of "go, turn and stop" if the one or more of the onboard computers aren't functioning correctly is an immediate no-buy from me. Limp modes are okay, they protect your car if something goes wrong but still let you keep control of it. But having no mechanical backup system that makes the electronics redundant is a huge issue of newer cars IMHO. One of the reasons my daily driver was made in '91 and is still my daily driver is that everything is mechanical. If something fails in the electronics, the worst that'll happen is the check engine light coming on and maybe you'll lose some of the electronic functionality. The car will still drive to your destination, albeit likely with reduced performance and efficiency. Cars today simply don't seem to be able to claim that same feat.

“And much like the family dog, the only way to fix these terminally ill car computers was by shelling out for a new one”. That really got me more than any of his other jokes have

This video makes me appreciate my '99 Buick Lesabre so much more than I already do...

Sanitize your inputs is security 101. That dev needs to be writing PHP in Siberia.

"automatically downloaded and attempted to open those files" jesus christ that is a terrifying sentence

As a matter of principal there should be nothing you could transmit that would interfere with anything on your car. The fact that it was possible means that Mazda systems are probably faul of various certifications/compliances. This radio station might have done it by accident, but anyone can transmit a signal; some might transmit something malicious on purpose. No fault with the radio station, it all lays with Mazda.

Thanks so much for covering this! I heard about the news whilst conversing with some of radio station managers and I was wondering what the public's take would be. Truly a car that bricked because of a file extension is like maximum fail to me. And I like the HD IBOC.

Had a similar experience with a model of BMW about 10 years ago with satellite radio. Sirius sent a signal patch out that created an error and stopped the tuner. It only effected people who were using the radio at the time it went out. And the only way to remedy it was to wait for the next patch update. I work at a dealer and there was nothing we could do to fix it.

I'm surprised this didn't happen sooner. With 10,000 FM radio stations in the US (not sure how many are digital) a benign mistake like the station's was bound to happen.

When I first saw "tuned" in the title, I thought of car modification/tuning. The amount of modified Mazdas is insane

How has these software devs that work on these car systems not heard of the #1 rule: never trust external data? Verify it, sanitize it, scrutinize it, assume it to be malicious at all times. If they applied this philosophy then we'd probably never have these issues until someone found a way to get data into the system in a way thats not checked

HR Radio in Southeast Connecticut: a) works UP TO 5% of the time, b) available on only four or five out of literally dozens of stations, c) drops out when clouds pass by overhead, or when you're descending a hill, or turning a corner, or driving under an overpass, or driving under trees, or when it starts raining/snowing, or you open a window, or you raise the trunk lid... I'm sure it's WONDERFUL in flat, unobstructed areas, like if you were parked in a parking lot in the midwest, and your local station spends the extra money to invest in themselves. WHEN it works, it's a night-and-day difference in audio quality, and the subchannels offer additional listening options. The caveat is it requires a strong unobstructed signal that often can't be maintained while sitting parked. In motion, you're constantly in-and-out of HD which is annoying - you can't turn the feature off, so you either live with it, listen to channels without HD, or use another source. HD Radio works almost as well as the RDS system. Wrong song title/artist displayed, garbled text, odd combinations of title and artist appearing as one word, and I had to turn off the "Auto Clock Set" function since my radio's clock would change with every RDS-capable station. Imagine my shock when the clock read 0:39 for the time - it didn't allow a zero to be set for the hour manually - and sometimes it's off by multiple hours AND minutes, as if the radio station never set their time correctly.

Wow, I never thought too deep about HD Radio because, when it first came out, most of the time the radio was still not connected to the car in any means other than to draw power and output to the speakers. But now that infotainment centers are so intertwined with the rest of the vehicle, I really see how HD Radio becomes a vulnerability. Someone could broadcast malicious code on those frequencies, and it would be automatically downloaded, without question (assuming it meets requirements the computer enforces to accept the signal for processing). I'm not sure all of what's possible with that, but this may just be the tip of the iceberg. Let's hope I'm wrong.

Wait, does that mean that instead of sending a .jpeg, .png or .gif, (or a file with a missing extension), someone could instead send an executable file and get the car to run arbitrary code?

This reminds me an awful lot of what happened to me and a fellow apprentice. Our programming teacher wrote a little bat to delete all those unnecessary files compilers like to generate. Files that the completed projects no longer need and that the compiler can easily generate again if necessary. We used Borland C++ builder, which also generates files without an ending. In the batchfile you'd simply write "del *." to get rid of them, and in school, on my USB-drive, and even the local harddrive of my work-PC this worked fine. However, on the server it got interpeted slightly differently: first of all it got treated like a "del *. *", deleting ALL files, and it also worked on the whole drive instead of my folder and its subfolders. Basically I ended up deleting all the projects I had access to which due to me hopping between departments were quite a lot. Thankfully a friend in IT and the recovery function dealt with that rather quickly, but my friend showed the batchfile to his boss (telling him that I'd asked whether this is safe to use) and I got a nice letter to give to my teacher explaining how dangerous this can be and that if we really had to bother with saving a few kilobytes we should do it manually

i live in Seattle and listen to 94.9 very often. I did a double take when i read the title and thumbnail lol. ya learn something new every day

I remember back in 2004 a local radio station broadcast a signal that would kill the batteries in Volkswagen Touregs. I worked at a vw dealer and we had dozens show up with dead batteries. Turns out the radio signals were making the radios stay on and kill the batteries.

As a CS student i think that the requirement to buy a new inbord computer is a scam. You should be able to reset the memory of the computer by cutting its power/deleting its memory.

Great Channel, interesting stories told in 5 minutes, not dragged out for a half hour.

Here's another "glitch" involving car radio: If you have XM Radio in your car, sometimes your cities' traffic lights will emit enough electromagnetic interference to "swamp" the satellite signal incoming to your XM receiver. No damage is done, but you may have to sit through that long stop light with no sound from your car stereo (it gets muted with a bad signal).

Love my analog car! And by that, I mean it has a few dozen various ecu’s that control everything from the engine timing to knowing when to set off the airbags. What I like is my car has a manual transmission, standard dials for heating and cooling, and no touchscreens. It’s about as analogue as you can get these days, in the used market that you can daily drive.

"... because hiring cellists to sit on your dashboard was getting too expensive." Ahh, yes. I can remember my grandfather telling me about the dashboard cellists they used to hire back in the day.

Been a Seattle resident for my entire life (3+ decades) and used to listen to kuow a lot, I hadn't heard of this. Wild stuff.

One of my first jobs in engineering back in the 1980s was in a system test facility where we got to hammer away on new software and/or firmware releases for a satellite communications system and its network control software. In addition to operating the system per the normal instructions, myself and some of the other folks would enter random things into text fields and press weird keyboard sequences and so on. We wrote more bug reports as a result than all the other groups combined :)

Imagine the captive audience that NPR station has for their fundraising drives!

Sam's mom: "Why don't you call more often?" Sam: "You seeeeee...."

Chrysler had a problem similar to this back in the 90's when OBDII came out but it had nothing to do with radios. On those early Chrysler cars if you scanned the computer and unplugged the scanner without turning the key off first it would glitch the computer and set the charging voltage to 0 volts and the alt would never turn on again unless you flashed the computer with new specs. Shops were replacing alternators left & right before they found out it was the computer.

"This was partially a result of how these cars were built, but it's also a result of how this radio station broadcast it's radio signals" Dude, this was entirely on Mazda. "Never trust user input" is one of the fundamentals of coding. If this obvious but accidental example was missed then how many deeper rooted issues do you think are just waiting to be exploited? Honestly, HD radio sounds like a bad idea in the first place. What stops anybody transmitting data that your car automatically downloads and tries to open?

I’m not even surprised at the fact that glitches like these occurred, but why did these relatively small errors have to result in the entire computer bricking? Imagine that happening to your PC. I don’t much about software, but that’s why I’m wondering why failing to open one file had to lead to an infinite loop.

That %i isn't even about using it somewhere, it's about trying to do something stupid. It's almost like someone decided it'd be a good idea to eval station names instead of just, I dunno, doing sanity checking and outputting it. That way lie exploits.

The new car dashboards are on a size of a tablett with many functions. You actully have to look on them when you want to switch the currently playing song, turn on AC, turn of the lane assist systems that wants to push your car into the incomming traffic because you are on a narrow street with no middle line. But you are not allowed and gets fined alot (rightfully) when using a phone while driving. there is something wrong with new cars and it is getting worse.

I love me some carburetors and point ignition.. and the little computers I build in the basement from component parts... and program myself using skills I started using in 1984. I'm livin' the life. Ain't got no flies on me.

You can blame the radio station. And you'd be wrong. In computer security, principle number 1 is that everyone is responsible for their own safety. You can't just ask everyone in the world nicely not to brick your car. You can blame Mazda. And that's a bit better. Not expecting a file parsing error? Major mistake. But I myself blame someone else. The FCC. Couldn't you have come up with a standard where the same type of data is always transferred in the same type of way? The standard doesn't specify what to do with arbitrary files. So why the hell does my radio require arbitrary file transfer capacities? It's like the FCC was asking for a faulty implementation.

Soooo... basically what would happen if SQL injection and Windows OS had a baby.

To quote Mr. Scott of Star Trek. "The more you over kink the plumbing, the easier it is to stop up the drain." I'm a K.I.S.S. guy, don't need or want all the whistles and bells available in audio video electronics these days.

Anything but NPR purgatory! Honestly though I love listening to the opening montage to NPR Viewpoints. It always amazes me the professions that people get paid for.

I would hope self driving cars would decouple their driving OS from the radio (and all the other stuff like HVAC).

Every computer engineer knows that computers are a mistake. Having seen the horror upclose I'm very concerned about more and more of them controlling larger parts of our lives.

Man, that’s wild! If only they had cinderblocks in the passenger seat to protect them, like you alluded to. Please do a video on that and about cinderblocks in general

as someone who works with computers, having a computerized car terrifies me. the more complex your system, the more often it will break. i do not want my car to randomly break.

Ha! Jokes on you, the Mazda I drive near Seattle is too old to be fried

This is ludicrous. I mean, apart from the fact that even very simple computers _decades_ ago could work out file types without extensions, if Mazda _really_ found it too hard to implement such a feature that… again… has been available for decades… any unrecognised file in an embedded car computer system should just be ignored, and probably also deleted. As it stands right now, you could literally drive around in a car with a pretty low power (hard to detect) pirate radio transmitter outputting on the same frequency as a popular local radio station, outputting tiny unrecognised files, and brick a good number of the Mazdas you drove past. To be completely clear I’m not endorsing that anyone try this, but most people with my (pretty good but not genius) level of computer skills _could_ do this, and if I lacked the ethical beliefs I have, I was bored, and I wanted to troll some drivers, I would _totally_ be doing this.

Two of our local So. Cal cops found the time to average 7 Disabled Parking Tickets a shift. The tickets cost $305 each, except for those with unlawful display or fraudulent placards. Those cars also were impounded and the Person committing the crime cited or arrested. Their record was 31 tickets on Christmas Eve.

The real lesson here is never listen to NPR.

If that cache is in RAM, not FLASH, it should be fixable by pulling a fuse. Removing power from the radio will clear all of the RAM. If it's in FLASH this won't do anything but if there is a way to do a hard reset that should fix it.

2021 VW Jetta w/ manual transmission. I downshifted on the highway, engine blew and I started warranty process. VW said the computer reads I was over speed limit, what gear I shifter into, what song I was listening to, and used that to determine they wouldn't cover the warranty. My insurance luckily stepped in and called BS but companies are now going to use your driving analytics to determine whether they will cover the warranties part.

I learned from a dishwasher in the mid 70s that computer controls would make a lot of equipment equipped with them worthless very quickly . In the case of the dishwasher, the steam from the drying cycle would seep into the computerized control panel permanently, rendering it useless . I salvaged the machine by replacing the entire control panel with a mechanical one from an earlier model . The problems have never been resolved, and it just keeps getting spread to more and more and more appliances and equipment . The controls of any and all devices should be the last thing to fail, not the first . To this day, if a model exists that doesn't have a computer controller, that is the one i purchase. By the way, I made my living repaing computer systems in cars . I chose that line of work because I knew I would always have lots of work and lots of customers.

I've heard of a similar incident affecting TVs. Some station sent bad closed caption data and it overflowed some memory buffer in the receiver. Depending on the TV it would suddenly change settings, mute itself, or stop working entirely. (Or, in most cases, not do anything.)

Software engineer here. The funny thing to me is that the JPEG standard requires a “magic byte”, which is essentially a bit of metadata at the beginning of the file where it says “hello! I’m a JPEG”. Reading this magic byte is necessary for correctly decoding the JPEG, so crashing is extra odd.

This is actually a very good simplified explanation of several relatively complex topics. Good job HAI

It's actually happened all over the country to Mazdas that weren't up to date. happened it to my wife's car about a week after we canceled our XM radio and started listening to FM. If we were still under warranty it would have been replaced. unfortunately it had been so long that we had to buy a new unit. but I was also able to install carplay for her at the same time so it was sort of a win.

Make cars more complex, add computers to them, wire all the systems together so that if a superfluous part fails it can damage the main function, then fight to end "right to repair". Modern cars are trash, get an old one. I can't wait for those remote kill switches to go live. Those will be abused.

I was on the team who promoted and supported GIF on CompuServe, and it pleases me to no end that it's still being used - and debated - 34+ years later. :) -team soft g

You'd think that you could just reflash the infotainment system's hard drive or SSD to clear the software issue rather than replace the whole unit.

This type of thing has happened to the best of the lot. IBM once had a program that only had one single instruction. It was discovered to have an error and was corrected. They then discovered that they had introduced a second error. It also was corrected. Luckily no further errors were detected with this program.

I don’t see how this is a permanently damaged unit. There’s no damage to the hardware. Mazda could reformat the units, and reinstall the software. My guess is they’re choosing not to.

I work on Mazda's for a living and have heard about this issue - apparently, a previously released software update fixed this issue - both the % and the extension-less image problem. The problem being that to *install* the update, you needed to have at some point brought it into a Mazda dealer and ask them to update the CMU. If you hadn't updated in time, you'd need a new CMU.

Wait so both Mazda and NPR took zero responsibility and didn't pay for anything? I don't expect it from NPR, but Mazda definitely should--seems like a safety hazard that might be prevalent in other systems and definitely worthy of a voluntary recall.

*bird lands on radio transmitter * Mazda computer: *CRASHES DRAMATICALLY *

F.M. stations were around commercially as early as 1941, and even earlier experimentally (wide-band frequency modulation was developed as early as 1933). The first F.M. band was actually from 42-50MHz, which the upper 6 MHz were claimed for T.V. Channel…ONE!

This is why most cars have separate networks for low priority signals (radio, infortainment) and safety critical systems.

Worth noting that files will often have a "magic", from which you can detect what file-type it is

The $15 motion alarm I bought from Harbor Freight, as it turns out, can brick the info console of my 2019 Ford Raptor. Lucky for me, it clears itself by power cycling the truck. The wireless detector has a ~435 mHz signal to the separate base unit/alarm.

What a funny coincidence. Your two AM examples of KDKA Pittsburgh and KBOI Boise are the two cities which I have, and still, live.

I can't believe that there is no "Hold these three buttons down simultaneously" or "Poke unbent paperclip here" or "disconnect battery for 60 minutes" fix for this.

This is another reason why I'm keeping my '99 Buick Century until the wheels fall off.

This “Seattle” issue happened to me in Delaware. One day my cx5 info system just started endlessly rebooting. Two different Mazda dealers lied to my face, saying they couldn’t replicate the issue, which happened every time the car was on.

Is this fixable by Mazda Toolkit? First time hearing about it and I've got a 2017 Mazda 3

Arguments that Electric Vehicles are good/ bad as instructed by Sam

As someone who works in Automotive software, Problems like this are a daily thing we tackle during development. However, there is usually a containment action for issues like this, like creating an emergency software release and allowing dealers to flash the fix on production vehicles.

Well, I do have a computer science background - and the fix for Mazda is relatively simple. Likely, the operating system used in their infotainmen.....car computers, is some form of Linux - which has an army of open source utilities available to it - including GNU File Magic, which offers the ability to inspect file contents to determine the type of file. A relatively simple modification to look at the magic result instead of trusting the file extension would/could have fixed this issue. Sadly, auto manufacturers are like phone manufacturers - once you buy the device, they aren't interested in doing software upgrades.

factgory resetting the radios would clear the cache and stop the system from bootlooping. it wasn't just an issue from the HD radio, if you had songs on a device with clipart that had a weird file extension, it would cause the system to loop also.

Fun fact, basic info like station identification, artist, and song titles COULD be embedded in analog FM signals using something called Radio Data System (RDS). Basically it was a small digital signal sent on a sub-channel that radios with the right circuitry could pickup and decode.

Was the person at KUOW that released the download called Bobby Tables?

1:55 this is incorrect, RDS (radio data service) hides a signal in the transmission that allows compatible radios (quite common) to decode text with info about the station and its media

Death threat to Sam for mispronouncing gif as instructed by the editor