Reverse Engineering - Computerphile 

"gnireenignE" - Easy.

Recently Super Mario 64, Ocarina Of Time and even Jak & Daxter with its Lisp dialect GOAL got reverse engineered and even ported to pc. It's a work of art. For those interested: tools with disassemblers and decompilers like Ghidra or IDA are pretty helpful.

Reverse engineering is a very useful skill to learn early on, but also one with a very steep learning curve. It happened to be how I learned programming back in the 80s on the C64 as access to any useful technical literature in my rural town was impossible to come by. I spent much more time in the machine code monitor examining games and demos to slowly learn how things were done than I ever did actually playing the games themselves. Perhaps it was a happy coincidence as I may not have gone that direction if I could simply have learned everything from reading books. Practising reverse engineering can greatly improve your analytical skills and pattern recognition as well as a more in depth knowledge of the underlying platform; skills that are, in fact, very useful in regular engineering. So learning reverse engineering enables you to become a much better engineer.

"Nothing is ever a waterfall in Computer Science except you are doing it wrong" 100% on point

May not be completely true, but i watched a documentary about Compaq where they actually had a software engineer de-compile the IBM Bios, just to see how it worked. Once he did so, he reported to Compaq that "this isn't hard, there's nothing special here." But since he had seen the code, he was not allowed to work on the project anymore, since he might be influenced by the code he had seen. Compaq had to hire an entire team of programmers to look at how the bios calls worked, without seeing the coded behind the calls, so that they could replicate the outcome, without necessarily replicating the IBM code that generated the original outcome. So when IBM sued Compaq, they could legitimately claim that their compatible BIOS was not based on the code IBM had copyrighted from Microsoft. Nor was the Compaq DOS based on the code of Microsoft DOS.

A very famous, difficult and consequential case of reserve engineering: Alan Turing and his team's cracking of the Enigma machine. It's hard to think of more important technical problems to solve than that..

One thing not mentioned in the video is that reverse engineering is more or less regulated depending on the country. For instance, my understanding is that, in the US, the person who reverse engineers some software and the person who implements a "clone" of said software need to be two different people. Furthermore, the latter must have no prior experience with the original software and must use only the specification written by the former person to write the implementation. The DMCA added further restrictions on reverse engineering. Other countries may have their own restrictions.

I loved xoreaxeaxeax's talks about his movfuscator: First, he stumbles upon the fact that the MOV instruction turns out to be Turing complete. He then proceeds to write a compiler that turns any piece of code, even the compiler itself, into a program that ONLY uses MOV instructions. (At a hefty performance cost obviously.) And he then spends his time making the reverse engineered flow graphs in IDA to render a picture of himself, and some profanities. It's at a next level.

I maintain several legacy systems that were written in C. They were written in an idiosyncratic style, often by people who wanted to demonstrate to the world how smart they were. It can take a while to figure out just what the code is doing. Sometimes you get a head start: this code/data/whatever was written by the same people at about the same time, so it will resemble other things they've done.

Reverse engineering also applies to archives/files, i.e. watching for patterns in data and figuring out how it is used in order to reconstruct or convert/export. common practice for the modification or extension of a program’s assets. Most modern formats follow the structure of [header + chunk(s)], so finding the definitions leaves only making use of the data in each of the chunks

Nothing's ever a waterfall unless u are doing it wrong. Beautiful

I'm always amazed and overwhelmed by the sheer number of lines that even the smallest executable has when decompiled to assembly.

Been Reverse-engineering for over 30 years, no University degree and learned everything from software design patterns to how the SoC bring up is done. You learn the real way it works by reverse-engineering. I started out with C-64 and the super snapshot cartridge and moved on from there to the latest ARM hardware .

Some of my favorite (hardware) reverse-engineering going on in recent times is the work by CuriousMarc and his cohorts at the Computer History Museum, reverse-engineering the computers and radio equipment used in the Apollo missions. Their many-part series can be found here on RU-vid.

I am kind of surprised you didn't bring up the example of ReactOS being a reverse engineering of the Windows NT kernel. A truely massive undertaking since the NT kernel is big, complex, and probably bloated.

The best example of clean room reverse engineering is GTA4. One of the best trainer programs was built with the software in the next room. Engineers with gta looked at how certain actions worked in memory and passed their findings to totally separate engineers who worked out how to interrupt and alter what it did. Rockstar tried to sue them for hacking and theft, but it failed because they hadn't built any software using any of the GTA code.

Already back in the 90s some GPU drivers recognized benchmarking and gave "incredible results". ;-)

I have actually had to reverse engineer more than one industrial automation system with nothing more than a poorly drawn schematic (which wasn't up to date) and raw code. First step was identifying all I/O points, so i knew what a particular input or output did. Then based on that you could figure out a lot of the basic logic of the machine. This is a start/stop circuit. This is a closed loop control system. This is a permissive or interlock logic block. Still takes a lot of time, but it can be very rewarding when you finally get to the point where you can actually maintain and/or improve the machine automation.

With the BIOS Compaq had 2 separate teams, one team analyzing the IBM BIOS and another team implementing their BIOS clone. The only communication between the teams was the documentation the analyzing team created by reverse engineering. No one from the analyzing team was allowed to speak with the implementation team or join it. IBM was never able to sue Compaq for their BIOS because it was a clean room implementation, running on standard hardware everyone can get. IBM was too slow for the consumer market, they are building business machines, even up to today, and that's part of the story why the IBM clones took over the market. They were compatible, cheaper, had agreements with Microsoft for MSDOS and could adopt much faster to newer hardware while IBM worked on the PS/2 machines. I think we can clearly say: We are lucky it went this way!

You haven't mentioned reverse engineering to bypass copy protection/licensing. I've reverse engineered a few small things that i liked but never had the opportunity to license. In one case, it was as simple as finding the reference to a string that bothered you about purchasing a license and in the jump instruction simply make it so it never performs the jump and proceeds as normal. In another case i reverse engineered the licensing key algorithm. Then i wrote a keygen for it. Reverse engineering is a lot of fun, sometimes hair pulling frustrating, but a lot of fun and the moment you make a breakthrough you feel like you are on top of the world.

Good ol' hexadecimal hacking, I love it! 😍 Great explaination of the methodology needed for reverse engineering! 👍

A video on executable formats would be amazing

Amazing video! Loved this topic! Thanks for sharing that

Three comments saying "first"? What a dilemma

definitely do the video on executable formats! Is there a video on how modern computers boot their operation systems? If not, booting/BIOS video would be very much appreciated.

Season 1 of the TV Series "Halt and Catch Fire" (available on Netflix) shows the reverse-engineering technique explained early in this video. Although not specifically calling out Compaq, it's their story without the name :). Take it with a grain of salt, though. It shows them using a volt meter and oscilloscope to come up with the BIOS code ...but the show is very entertaining overall.

Great video as always!

A whole video on what a space is would educate some people, that's for sure. It's practically impossible to get people to agree on it.

Awesome video, many thanks! Can you please make the videos to which you are referring throuhhout the video?

Reverse engineering is when an engineer looks at a mirror

LOVE you guys!

The funny thing is, this is 1 to 1 applicable in getting from idea to a product. Having this conversation with managers is important

Brilliant video!

I'm a web developer and came across reverse engineer recently. I wanted to learn but lack of information online, courses, and such are a pain in the ass. I will try to learn it again in tbe weekend because it seems like a cool thing to do.

I think this video is more like an introduction to the technical aspect of reverse engineering. Maybe for a "fun little youtube video" would have been cool to discuss the ethic/legal aspects and what is done in practice to accomplish a successful reverse-engineered solution without being exposed to legal trouble... I worked for a company that provided alternatives to IBM solutions. We made mainframe emulators, we had clean rooms and fun like that! :) We also had employees who were former IBM employees and that made "reverse" engineering (of something they knew exactly how it is coded) even more fun! :)

that was a brilliant video !

"leave that as an exercise to the viewer" is the most university professor thing to say ever! I almost got PTSD from such lecture notes or textbooks 😂

Have an idea for a similar episode in the future. Steve should pull out a BBC Micro, C64, ZX Spectrum or any other 8-bit machine having BASIC. Then off camera develop a program which first asks a user to input any number of words. Then the program would calculate their number and display it. Then show only the running program to the camera, and ask the cameraman to also come up with a BASIC (or any other language both can code) program that does the same. Suppose the BASIC will be different. But there you have it - Reverse Engineering.

Great content. And the endless paper.. great memories :) By the way, are you also doing RE of 32/64 intel on your m1? If so, how? Thank you.

What a coincidence! I've been into cracking software lately.

3:26 Unix executables were originally "just machine code". Later on, when they wanted a format with headers and whatever, they made that format, but didn't actually change the implementation of exec. Instead, the binary started with the machine code to jump past the headers. And that's the story of the a.out magic number.

Dr. Steve has been on a roll lately. I've rather enjoyed the past few episodes that he's hosted.

I'm curious as to why that executable contains several Java class files, as shown by the magic number showing up several times.

Can you do a video on the Learning With Errors problem and why it's supposedly quantum-proof? Thanks.

My favourite reverse engineering task is to port a program to another programming language. If you have the source code this is a fun task, if you don't have source code then the first step is a pain, trying to disassemble binary into machine instructions and translating from there.

Is this the first computerphile video in 50 fps? Because I noticed and liked it.

Wish I knew how to code. The concepts just seem like logic. Thanks

Well done.

I use the principles of reverse engineering to deal with unexpected conditions. An example is an industrial flash programmer offset entire parts of the firmware in ROM. The question: how did it boot in the first place?

Do a video about reverse engineering of Diablo I

please make a video about what a space is :D

What about reversing into a pseudo-C like Ghidra? I've heard of it a bit, but not a lot of info about it

I do a little reverse engineering. Usually with mobile apps, or web code, to look for vulnerabilities. Found some interesting communications protocol in the 'Wind' scooter app (the older yellow hire scooters), no free rides, but a denial of service if you were careful. Those scooters were in Nottingham, too, the same place these guys come from.

Isn’t thinking of something and trying to figure out how it works (inventing it) basically reverse engineering something that hasn’t been designed yet?

Oh yes, please do avideo on spaces. I never understood why there are line feeds and carriage returns and why those are two different things...

I didn't come away with a good impression of how good a job reverse engineering can do in an average case. Can it understand the whole program? Does it penetrate one level deeper than assembly and then give up?

thank you

Surely by now, a machine learning system could be trained on billions of lines of code for various kinds of programs alongside the binaries, and after training be able to take a binary as an input and generate a complete, high-level code that, when compiled, is functionally completely identical to the original binary?

Might have been good to mention self-modifying code, and the challenge of reverse-engineering that.

I'm trying to reverse engineer a Z8002 system ATM , managed to get it disassembled , and can work out some of the routines . I not sure if it was written in assembler or C (knowning sometimes helps) pity Ghidra doesn't support the Z8k

Step 1: Define all the things Step 2: Figure out the data structures, algorithms, logic, and control that goes into the programs Step 3: Do C things because C :) Step 4: Go to 11:06 of the video for an IDA practical example Step 5: ??? Step 6: PROFIT!1!1!

i hoped he talked more about how "disassable" bytecode differs from mashine code or that you can also reverse assambler code back to c/cpp

9:00 yes, waterfall is doing it wrong :) And i agree with the "show me your data structures and your program will become obvious" to a large degree. Though a lot of the interesting part of programming is how to model edge-cases in the domain you are working on, and how you structure your code to handle adapting to new functionality or handling of bugs and edge cases. Programs are not (anymore) made as a single release, and then a few years later you get the next major version that is made from the bottom as a different program that may be a newer and better implementation of the same high level ideas and concepts. As systems get larger, maintenance/support and feature expansion time and time between bottom-up full rewrites get longer because the costs become much larger. Also, with agile development mindsets, the focus becomes on continuous development and deployment and delivering incremental value at low incremental risk of things going wrong.

moral of the story, document your ABI as well (I never do either), 'but implementation details are suppoesd to be hidden' lies, just one of the many lies we tell are selves to make today's work 'more efficient' where in a number of years some poor sod is going through your modules disassembly line by line, although it's surprisingly fun to do, piece such things together

surprised the word *debugger* wasn't mentioned

If the code you are analyzing is written in some high level OOP language, chances are a modern decompiler will give you at least some C equivalent, not just assembly code; in some cases it might be able to give you some snippets in the original programming language, since OOP compilers are pretty deterministic in theory. And then some languages don't even complie their code, it might not even be obfuscated, so why even bother...

So if someone makes their own offline launcher for adobe and removes the subscription DRM, is that not Reverse engineering? Could they not legally sell that then? if not why not? and how then?

If you’re attempting to disassemble non x86/64 code using IDA, take a look at the pricing….I bet you’ll be surprised. To save a lookup, it’s more than $10,000 for the full package, $5,000 for just MIPS and MIPS 64….

Can it helps to find who hacked your phone or find who cloned your phone or PC?

Video about executable pls

Reverse engineering is basically learning how things work. That is why all the restrictions in licenses that prohibit reverse engineering / de-compilation etc. should be ignored and made irrelevant. To bar humans from seeking knowledge and understanding is to be anti-competitive and anti-science.

I wanna reverse engineer a script in lua and I don't know how, can u please explain, it's for a exploit in a game , I wanna see how it works so I can make my own

What if I am hiding a Exe in a picture. How would I go around finding such information? And how would I Reverse engineer such file?

When do their stock of old printer paper become empty?

Kind of left out “vulnerability research” as a reason to reverse engineer.

Where can I buy a shirt like yours? Love it!

Where can I get Ida? I don't find it in the Ubuntu repo.

Please create a video about ReactOS, an open source Windows clone.

Can an IA reverse engineer a binary? Is there any example?

A space is anything that matches a \s in a Perl regex :)

Pretty sure step 1 should be to write tests that define the behaviour for the original system. Once you have those then you can start thinking about writing your own program to pass the tests.

Reverse engineering: some video cards had the string "Compatibility require the string (c) IBM here"

Could deeplearning improve reverse engeniering to get a highlevel output?

15:33 Library or external function calls and system calls are being conflated. Typical user code doesn't directly make system calls. These are implemented as "magic" instruction (this is all generalized) which allow a userland program to execute a small chunk of kernel code. By magic instruction I mean "an instruction which causes the kernel to shut up and pay attention", i.e. throws an interrupt of some sort. The mnemonic "syscall" is an assembly instruction for several different architecture's instruction, even on the same hardware, various operating systems (and even versions of them) can use different instructions. (Nothing is stopping you from saying "A syscall instruction is any instruction which causes a protection fault when writing to a valid syscall number." Provided your kernel can tell that happened and react to it appropriately, the mechanism doesn't matter. ) While the mechanism you use to call them varies, all of them need you to tell the kernel _what_ you want them to do as well as _that_ you want them to do something. The parameters to the syscalls go in registers (or in a specific region of memory) and include the system call number you want to run, telling the kernel how the arguments should be interpreted. Once you have set up the parameters, you execute your magic instruction, which interrupts the kernel, looks at your request and fills in the reply. Then the process returns from kernel mode, and continues. These operations are often by system library code, because different systems might use different system calls, but as long as the library call takes the same arguments, your code will still work. That's why there's a distinction between library code and system calls. The only other times you need to know what a system call is are if you are reversing a statically linked binary (which puts all the library code it is going to use directly into the program. And it can make them massive) or if you are working with shellcode (like an exploit for a vulnerability might use.)

We really need a law requiring software vendors to distribute full, annotated Source Code with everything they sell. If everybody had to do it, nobody could get away with plagiarism.

wait why can't we take the assembly code and pass it to a "decompiler" and generate C code from that? that'd be much easier. Like yk compilers can generate assembly from C, why can't you write a software that does the exact opposite process that a compiler does and generate C from the asm?

"Nothing is ever a waterfall in computer science, unless you're doing it wrong." Casually roasting one of the standard software development models 😂 But I get his point, the waterfall model is just an oversimplified, idealized concept of how software development could go, which in theory looks nice, but practically never matches reality.

If you're a junior dev working in an establish / legacy code base a big part of your job is reverse engineering. "Yeah some guy who used to work here wrote that in perl back in 1998. Nobody has any clue how it works but its a vital component of our payroll system and its broken."

Time to develop a reverse engineer coding AI - that learns the function, inputs and outputs of the target software and then writes the code for the operating environment of choice.

13:35 x86 machine code does have loops.

If you are lucky things can be much easier... if you find a decompiler that works on that specific code.

Reverse Engineering is also called dismantling~

I almost want to ask for someone to cover ECMA-48

11:34 "CAFEBABE" 😂

I reverse-Engineer games all the time, to develop hacks for them. :) Same for bypassing anti-cheats.

5:06 xnopyt

The video looks choppy, it's not filmed for 60hz screens

But there is a way to get pseudo source code from a program its called a decompiler. IDA can also do it but it requires an addon that costs several thousands of dollars, Ghidra can do it for free.

Interesting, also You kinda look like Steve Bannon

Are you guys on a ship?