The Computer Hack That Saved Apollo 14 

Nasa safety check: we have a possible issue involving the "Burn Baby Burn!" command. Everyone: *turns slowly to that one guy who would call the engine ignition command that*

As a hardware guy I think Apollo 13 was the greatest helldesk call in history. It had everything from bad reception, thermal hardware failures, cranky customers, computer reboots, hardware/process hacks and yes even a firey decent.

I remember this when it was happening. I was amazed, and confused at how they could “reprogram” at a Lunar “arm’s length”. At about the same time, I was working on a computerized typesetting machine (Photon Pacesetter), which had for-bit words. The machine developed a hardware problem with the character flash power supply. I was able to access the computer via a control panel, keying in values which slowed the machine so the flash power supply was able to recover for the next character. The machine ran at about 1/3 speed, but we got the paper out, and the next day, the repair technician replaced the power supply. I was so impressed with having been able to “fix hardware” with software, that I eventually made software my career.

I saw a T-shirt once that had an octal joke on it: "Why do programmers get Christmas and Halloween confused?" "Because DEC25 = OCT31" Very clever, I thought!

The only one appreciated tech support guy in the history of tech support guys.

I knew I could count on Scott to use "computer hack" in its original context

"You have six seconds to key this in after the engines fire. And whatever you do, don't make a typo."

Well researched video! In the case anybody wondered, on Apollo 15 and later the astronauts had a very easy way to tell the AGC to ignore the abort button and the abort stage button. That way the same issue couldn't have happened again. The AGC specialists developed a bunch of procedures for all the different switches that interact with the computer. So if any of these switches became faulty, they were prepared for it.

"Houston we have a problem" *Indian voice* "Hello sir my name is Rajheed how may I help you today?"

Those woven memory units looks like some bonkers sci fi. Pretty cool.

A little bit OT but I recently notices KSP saying "flying safe" in the loading screen - i love it!

The issue was actually caused by a small metal fragment left in the switch at assembly. In low gravity it floated around and could shorten the switch contacts on every acceleration. Source: Gene Kranz, 'Failure is not an option'.

I truly enjoy stories from the history of computer science. This kind of ultra low level command of wire jungles flying space ships to the Moon has such an appeal to me. And Scott, you are an incredible RU-vidr and an all around awesome human being (from what we see in the videos!). You just can't fake the passion for spaceflight that you have. Fly safe!

Great video! Don is still around and is still awesome. The listing shown in the video is from Don's own personal collection, which he has allowed us (at the VirtualAGC project) to have scanned over the course of the past year. We actually just finished transcribing the last of them, Zerlina 56, this week. Other great programs Don has given us include: * Aurora 12, a "fork" of the LM system test program, which has enabled us to greatly improve the accuracy of our emulator * Sunburst 37, an early development build of the Apollo 5 software * Sunburst 120, the flown Apollo 5 unmanned LM software * Luminary 69, almost the flown Apollo 10 LM software (69 Rev 2 is what flew; this one is missing a last-minute lunar gravity model change) * AP11ROPE, a 1970 remake of the Apollo 11 software (which has increased our confidence that Luminary 99 Rev 1, which we've had for a while, is what actually flew) * Luminary 116, the flown Apollo 12 LM software * Luminary 131 (shown in the video), an early release for the Apollo 13 LM (this underwent many changes before flying, however) * Luminary 210, the flown Apollo 15-17 LM software * Zerlina 56, a super advanced LM program featuring a variable servicer and P66 LPD, which you can read about on Don's site: www.doneyles.com/LM/Tales.html This was considered too risky, so NASA didn't improve the major changes for Luminary. He actually has more listings than every other source we've had put together! In an ironic twist of fate, Apollo 14 is one of the only two LM missions he doesn't have software for (the other being Sundance from Apollo 9). We're still searching for it though! You can find his collection as well as all of the other listings we've managed to have scanned in our Internet Archive collection here: archive.org/details/virtualagcproject Or if you like GitHub, they have all been transcribed and are in our repository here: github.com/virtualagc/virtualagc

And THIS is one of the reasons I subbed. So many well-researched space shenanigans and, of course, checking the staging

Was his first response "just switch it of and on again"?

I got to literally touch one of these computers as an intern at NASA this summer....pretty awesome

“I’m sorry Don, I’m afraid I can’t do that”

Well the guy at 1:40 certainly looks he was woken up midnight during his sleep

Good thing they found the problem before landing.... I can just imagine them all bouncing around on the moon, only to see the lander "abort" and fly off into orbit.

I think Apollo 14 demonstrates how 'by the seat of our pants' the Apollo missions were. In truth, a flawless mission would have been an anomaly.

Nice to see that even in the 60s programmers commented their code silly.

Scott, you take me back to my youth. I was a member of MITs Apollo project (the older I get, the better I was). Your are right, Don Eyles was one of the essential geniuses, second only to Hal Laning, who architected runing 6 or 7 layers of interrupt on that little computer. My claim to fame was that (early on) I proposed and did the early work on the core rope (hard to break code you cannot change), as well as the DSKY concept with Verb & Noun and the scheme for getting the angle and velocity increments without having to execute a subroutine. IBM labeled that scheme "cycle stealing," which I thought was brilliant. So many people were "essential!" What a priviledge (and luck) for me to have been part of that project! Ramón Alonso

computer science major here..... that programming is so....simple.....yet elegant and powerful

I still find it pretty amazing that we made it to the moon using hardware less powerful than some wrist watches.

Thank you for using "hack" in a story that doesn't involve food or home DIY repairs. Subbed.

"This is what we call 'lies to children'" +1 for Discworld reference:)

Imagine jumping around on the moon and then the module takes off on it's own

Apollo 14: "Houston, we typed in the command codes. The computer now shows 'Err 404'." Houston: "NOOOOOO!!!!!!!!"

Even the computer geeks on the ground had brass balls back then. Fix something on the far side of the moon computer code in just a couple hours, no problem.

I don't understand 98% of this video but I'm still thumbing up for good reasons.

As a proud Bethpage resident, thank you for calling it the Lunar Excursion Module or LEM. My best friends father was in charge of wiring for the original LEM. Over 85,000 people worked round the clock @ Grumman during that time as they had other projects (E2C, F14, etc) they were working on. They also made two identical LEMs for the first moon landing. One was transported by NASA in the Super Guppy to Fl, the other one stayed in the plant. Everyone that HaD anything to do with the LEM was to be at the plant when it was descending, on the moon surface and when ot lifted off. After the successful return of the astronauts, the 2nd LEM was pushed into the parking lot and sprayed with camo green. It stayed there for over a year. I used to pass it every day when I went to work. I always thought they should have given it to the Smithsonian before they painted it green. Today the plant in Bethpage is closed down, and although a few buildings remain (one is called Grumman studios) the property now has apartment building on it. So sad to see this piece of history treated like this. There are plaques placed around the property as a memorial to this great achievement. My house was 1/4 mile from the end of the runway. I saw the guppy come in to pick up the LEM. also would regularly see E2C, F14 and many other aircraft land there. I have photos of a pilot of an E2C waving at me. Yes I could see them very clearly in my backyard

"I'm the technician that will be helping you today... have you tried turning off and back on again?"

Great piece. As someone who has been awakened at 2 am to fix a production system, I am super impressed with Don Eyles.

"Burn, baby, burn?" Sounds like a variable name I'd use lmao

3:28 yeah of all possible ways to lose astronauts in space. Having the LEM abort to orbit while you are out walking around on the moon might be the worst.

wow, rewrite the flight computer in flight... What could possibly go wrong? Thx for sharing this stuff

I worked in I.T. for 35 years. Whenever we had serious computer issues, I could never help but think "well at least we're not in Space."

..this bring me memories of my days in college..Fortran , Cobol, punched cards and floopy discs!

No pressure. Damn, I would crap my drawers if asked to punch those keys. That's why I'm not an astronaut. Yep, really, that's the ONLY reason.

And on the 2nd place on the list goes to the team behind the Remote Agent for the NASA Deep Space 1 mission. The probe stopped working due to a race condition bug. Fortunately, the people behind the Remote Agent had written most of their code in LISP (which was unheard of at the time for code meant for critical space missions). So the probe even had a LISP REPL, Debugger and everything. Using the NASA Deep Space Network, they managed to talk to the Remote Agent's LISP REPL to get a dump of the program state, find the culprit and fix the bug using the REPL debugger. Had the team listened to the skeptics and used C and/or assembly instead (as was tradition), the bug would have been catastrophic and unfixable.

Should have put more struts on in the computer

Wow!! Thanks for such a thorough video! I really enjoyed the programming vocabulary (verb/noun/etc)

Don might be the world's most ethical hacker

Imagine being out on a moonwalk and seeing your lander launch on its own.

"Have you tried turning it off and on again?"

That was awesome. *_"Computer Nerd Saves Apollo 14 Mission"_* read the headlines. 🤓 Computer nerds make today's world go round. 👨🏼‍💻

I am a USN submarine veteran and watching this reminds me of the old ballistic missile submarines navigation computers I worked on during the late 80s and early 90s. Have to love core memory and making sure the currents were adjusted just right to prevent errors.

"Applied Cryptography" is one of the books on my shelf, too!

And I thought learning to code in BASIC in the 80's was rough. But it was still easier than being on the phone with an 83 year old woman trying to explain to her how to program her new cable remote. Worked on it for over a half an hour before I realized she had the wrong remote. Once we sorted that all out, she says, " Let me look at my TV a little closer. I'm sorry it says Phillips not Panasonic on it." I literally banged my head on the desk. "Are you okay Hun?" Yes mam. Took about another hour to get it all set up and squared away. And was just the first call of the 10 hour shift I had chosen. There is a reason retention is a problem in the Cable Call Centers, and they call everyday. But I still wouldn't want to run that long distance hack. Those folks had serious skills.

"I know we're not really running the abort sequence Dave...."

Great video!!! Oh, and look at how long they left the engine going once they touched down... BEAUTIFUL streaks of dust plume splintering out from where the engine exhaust hit the surface. Awesome.

4:32 I never expected for Scott to give me a flip :(

ground control to major Tom, commencing countdwn engines on, check ignition and my God's love be with you

The original machine had a base plate of prefabulated amulite, surmounted by a malleable logarithmic casing in such a way that the two main spurving bearings were in a direct line with the panametric fan. The latter consisted simply of six hydrocoptic marzlevanes, so fitted to the ambifacient lunar waneshaft that side fumbling was effectively prevented. The main winding was of the normal lotus-o-deltoid type placed in panendermic semi-boloid slots in the stator, every seventh conductor being connected by a nonreversible tremmie pipe to the differential girdlespring on the "up" end of the grammeters.

and one of the only tech support calls where problems were actually solved

And today the best hacks seem to be just setting a few flags or changing a few numbers that are checked by the software. Don't need to reprogram the software when you can simply change the data that it's using.

Thank you for posting a detailed explanation. I've heard about hacks in the lunar program software many times, but this is by far the most detailed account of a particular issue. I'm a programmer specializing in life-critical software, plus I'm old enough to remember Apollo 14, so this sort of story is of great interest to me.

The story of a programmer who saved billions of dollars. And that was back when billions of dollars used to be a lot of money!

As an ex-IT guy (now retired) this was THE greatest tech support call!

Hey Scott. Could you do a video on earth's artificial ring? Maybe even replicate it in KSP. I've been wanting to ask it for a while. Always forgot. It's Project West Ford. Before the age of satellites and the communication nodes we now call "internet", it was feared that an atomic war would disrupt radio communications by messing up the ionosphere. So they used a rocket to disperse some 20Kg of copper wire "needles", that acted like dipole antennas, through medium orbit, in order to create an artificial ring that could bounce back radio comms. It worked, and another two launches were intended to create full coverage through the planet. However, the project was scrapped mostly due to satellite communications becoming feasible, but also due to concerns over space debris. I think it's a really interesting idea, albeit obsolete. You can use the ionosphere to send radio signals to someone literally on the other side of the planet, but it's finicky. This ring of crap solved it.

This is probably one of my favorite videos from your channel, thanks for making this!

Imagine what a misstype could do in that situation.

This is one of the best RU-vid videos I've ever seen.

Did they try just rebooting the computer first? That's the first thing the "help" desk always tells you to do. :D Seriously though, for some reason this video made me miss the days when one person could grok the entire contents of the computer at the bit level. I can still remember the hexadecimal address for the entry points to various subroutines in Apple ][ and VIC-20 ROMs. "Call -155" ring a bell for anyone? :) (Or -151 to do it quietly.. and of course 3D0G to get back.)

Videos like this are why I like the Apollo program so much. Going to the moon wasn't about the technology so much as it was about the people. Dedication to detail -- "I know what that is!" -- and the preparation necessary to make it all work even with faulty hardware (which is the only kind of hardware humans can produce).

And this show us just how rushed the Apollo mission were. They had the time to design the mission to work, but not the time to consider and prepare for all needed exception scenarios. When designing a complex, life-critical piece of equipment, one needs to consider all possible failure modes, and have them documented. Something as simple *and likely* as a stuck/intermittent/nonfunctional control switch should have had pre-documented info on its effect on mission, possible workarounds, etc. Note that in subsequent missions this functionality was built in, no problem, at least for a subset of the controls. The problem is, that doing this sort of analysis and preparation takes time. a LOT of time. And with the schedule concerns of the moon landing deadline looming, they only had time to get the systems working, not smooth out all the burrs. It is scary when your first actual test of a complex system, involves strapping 3 humans to a large explosive device and hurling them at the moon.

First video of yours that I've watched. You have an amazing story telling voice and the subtle accent doesn't hurt :) Keep up the great work!

Scott flipped me off at 4:28 ;-;

I've just finished the Don Eyles book "Sunburst and Luminary: An Apollo Memoir", great read!

Haven't watched yet, but the programmer "fooled" the computer into not searching for the Abort Mode by telling it that it was already in Abort Mode... Learned that from HBO's excellent mini-series FROM THE EARTH TO THE MOON!!!

The most unbelievable in all this, is that the guy who wrote the program decided to sleep during the descent phase.

it cant be tech support, they didnt suggest restarting it

When you're about to descend on the Mun but MechJeb starts to fail...

"Houston, we have a computer problem here" "I'll be glad to assist you today, can I first get your 6-digit customer code?" "Our lunar lander might abort!" "I understand, can you please verify your phone number?" "The abort switch keeps flickering!" "Absolutely, what is your user name?" "(sigh) Antares" "Okay, I have A as in apple, N as in neck, T as in table, A as in apple, R as in rabbit, E as in extra, and S as in salad...the user name is Antares, correct?" "Yes, now we've got to fix our computer immediately!" "We can help you, can you give me the 23 digit key that came with your computer to verify license?" "The entire mission up here is in jeopardy!!!" "Before we get started, can you please confirm your email?"

I knew who don eyles was when I was working at draper lab, previously called the instrumentation lab, and knew who his boss was, Margaret Hamilton. Draper lab was (is?) a heady place to work. Really enjoyed your video. Incidentally, Apollo 12 was hit by lightning on the way to orbit. I was in the SCAMA room when Houston called and asked if they had a go for TLI, trans lunar injection. Houston was told to turn the guidance off because it was trying to torque itself out of gimb lock, then do a P51and P52 to align the platform using the sextant. One orbit later the platform alignment was checked, the alignment drift was negligible and we gave Houston a go for TLI. I was on the fence on that decision but calmer minds prevailed.

Why didn´t they just F5 before the descent?

At 4:19, the following transcript shows that the last digit entered was a 1 not a zero.

The infamous hacker known as 4Chan probably did this

Probably one of the main reasons why pre-programmed computers didn't catch on as much as programmable ones have. As soon as a situation arises that the programmers never had any reason to conceive of during development you'd better hope there's a workaround based on what they did end up putting in.

So, basically a build in GameShark in Apollo 14?

Pretty neat. I'd never heard this before. Listening to the commands to octal brought me back in time. I started programming in 1964 and used all sort of low level languages and bases like octal. And to one of the commenters, yes commenting was especially important on languages like assembler or machine language as no one would be able to really follow logic without them. Of course if no comments you could write a logic map to help you but it would be difficult.

This is that nerdiest video ever. I'm slightly embarrassed to admit that I've learned just enough about computer science to understand all of it. It's actually pretty interesting! Just not the kind of thing I can talk about at cocktail parties.

Outstanding. Thank you. Ed has been my hero for 40 years. I have never heard this story. Carry on.

Lies told, Apollo 13. How to hack the Carbon dioxide filters. Man enters room throws 20 pieces on to table, team of 20-30 start work.....NO! Nope! did not happen!. One man on his own, did the hole how to square peg- round hole. When asked did you feel cheated? he replied Hell no! I'm that good, even Hollywood replaced me with a team of 25 people.

Not many probably find this interesting, but I learned Unix Coding while an intelligence analyst for the Army, attended MCSE course until deployment orders cancelled that. This is fascinating and appreciable

Is that the old Xtronaut board game behind you? lmao

Pedantic note. The 'excursion' part was dropped from the lunar module's designation in 1965, 6 years before Apollo 14. Though it is pronounced 'lem', it is a LM. But a cracking video. Thanks. From another Scot.

The best tech support call: - Houston, we have a problem. - This call can be recorded for training and quality assurance. If have a fire, press 1, if you blown an Oxygen tank, press 2, if you need to adapt a square CO2 into a round filter socket, press 3, if Newton laws of motion are not working, press 5, if your abort guidance computer wants to abort at any time, press 7, to repeat these options, press 9, to talk with a NASA engineer, press the ñ key followed by the Avogadro constant to the 47th decimal.

Scott's commentary seems indicate the fix was devised at Mission control in Houston. It was in fact at the MIT Instrumentation Laboratory in Cambridge Mass.There were 5 of us there working on the problem, and Don Eyles, who had written the descent code, and was probably the smartest of the 5, came up with the fix. We tested it on our simulator, and then give the information to Houston. Larry Berman

Ah the days when computer programs were small enough that a person could have it all in their head. Good episode.

7:37 - buffering...

Just noticed your cool Ninkasi NSP boxes on the shelf up there. I live up by Eugene and they’re one of my favorite breweries. Had no idea about this “Ninkasi Space Program” project until I saw that logo and had to look it up. Imperial Stout made with yeast that survived a trip to space and back. Very cool stuff! Get in my belly!!

If the computer inputs were all in octal, were the "8" and "9" keys just never used?

What insane Assembler directives. My favourite still Z80.

Benifits of writing a guidance computer software in such a low level language. You know exactly what memory addresses to change if something goes wrong.

Thanks, I used that command on my refrigerator and it works fine....

Nobody can say "Hello, it's Scott Manley here" like Scott Manley can

I want your applied cryptography book

As someone who daily works with very, very old Fortran, such cryptic algorithms are both musing and traumatic.