Could The Internet Send You The WRONG Thing? 

yes my son got sent adult videos after he tried to download homework questions, the internet is scary

Programmer here: Linus mentioned that the data goes through a CRYPTOGRAPHIC hash. While they certainly can be used to verify file integrity, more commonly used for such purpose are NON-cryptographic hash algorithms since they are generally faster. While the cryptographic ones are generally reserved for well, cryptography. (Difference: Protection against active malicious manipulation vs just plain transmission damage) What makes a hash function cryptographic(or not)? Basically it's how hard it is for a bad actor to crack (or produce a collision). MD5 is once considered a cryptographic hash and are popular among website as a mean as storing password, but has since been deprecated and regarded as a non-cryptographic hash just for file integrity verification after people start finding fast ways to crack MD5 hashes

Actually Linus, as a CS Major it’s actually a miracle that information gets from point A to point B, fucking magic man, that’s why those low level devs always have a long beard, they’re magicians.

Fun fact, every credit/debit card has a checksum built into the number so computers can quickly determine if users accidentally typoed their number in wrong when paying for things online. Many other numbers which humans are expected to enter manually usually are designed with these sorts of checks in mind like insurance numbers, IMEI numbers and even those lil survey codes you find on receipts

7-zip can generate the checksums of files too ! It's the CRC option in the 7-zip menu in the explorer. It avoids downloading something else to check it or typing a command.

Checksums are redundant pieces of information added to data that allow the receiver to verify if the data was received correctly. A simple checksum is to use only the first seven of the 8 bits in a byte for data. The eighth bit is a sum of the first 7 bits (modulo 2) that acts as a check for the first 7 -therein the name 'checksum'. In a nutshell :)

Data transmitted with UDP does not get resent. The UDP protocol is used for things like video streaming where an occasional dropped packet won't be missed.

Not all CRCs are cryptographic, actually I'm pretty sure most fast checksums are not crypto hardened. Still, great video!

I've heard that many of the free cloud storage services use checksums to save space. They run the hash and compare to what is already on their servers. If two matching files are uploaded, only one copy gets stored. It does not matter if the files were uploaded to separate accounts, only one copy is actually stored.

I'm kind of surprised there was no mention of block-level CRC for storage media, the checksum that makes it possible for RAID-scrubbing to find faults, and for disks in general to be reasonably certain they're reading back the same values that were written in the first place, something almost everyone takes for granted.

if you have 7-zip installed (which you really should if you don't, it's amazing), it actually adds all sorts of checksum-generation options to the right-click menu in windows, its really handy

personally i prefer the method of looking at the files and going "Yeah that seems about right"

Noteworthy that a checksum on the download page being the same as the downloaded file doesn't mean that it hasn't been tampered with. If you're man in the middle-d or the site is compromised enough, hackers could also just replace the hash shown on the site to match the modified file.

And we're slowly moving to quantum-resistant hash functions, to avoid the issue of quantum computing in the future.

There is a big difference between *Cryptographic* checksum and the one used for verification that a file arrived correctly such as TCP/IP protocol

1:33 If those bad actors could replace the download with a malicious one on some website, it would be of no hassle for them to replace the checksum as well. MITM attacks are guarded against with protocols like SSL. Checksums are not used to validate the security of a file but rather to confirm it was downloaded correctly from the origin (even though TCP handles it too) so that, in the worst case scenario, your PC doesn't break down from an incorrect OS download.

Hashtab is one of the best checksum tools for Windows. It adds a tab in the properties dialog of a file to let you compare checksums.

Looking at the thumbnail, I thought this was about the new .zip TLD...

And yet hash collisions exist. We use this to crack files on some games to mod them

4:23 not every service uses TCP, for example, most online games and realtime apps use UDP to reduce latency because packets don't have to arrive correctly.

Whenever he says "bad actors", I can't help but think actors who just suck at their job doing shady things

Couple of points: - the cryptographic hash function outputs aren't guaranteed to be unique, but are generally designed to avoid collisions. - passwords aren't just hashed (or at least they shouldn't be 😅), if they were, then if the Database was leaked, the attacker would be able to tell the simple passwords. Two people with the same password would then have the same hash output. This could also mean the attacker can generate hashes from a list of common passwords and compare against the database to find people with common passwords and hack their accounts. To get around this, a "salt" is added. The salt is randomly generated and when combined with the password and then hashed, it will create a new output, even if two users have the same password. This is why you should use unique/random passwords, because if the server doesn't salt the passwords, common passwords can be found easily and then anywhere you use that same password is then potentially compromised - even if the other places do salt them.

Note that 7zip has a checksum viewer as well, so if you have that, you can view the checksum of a file easily

interesting question : could you "reverse" the SHA to get the file back from it ?

I’m surprised that this video didn’t mention google registering the .zip domain.

A good hacker that changes a file for something malicious one on some server would also change the checksum file at the same time

On the "Windows Explorer doesn't compute hashes" note; It wouldn't take a lot of resources to do that at all. They would only have to add a checksum middleman to the file transfer stream.

Checksums are only useful if you're expecting errors not malicious intervention. Anybody could just change the source, and then re-hash the source and send that as the checksum. Encryption will be necessary regardless.

Did discover something interesting with MS Teams. Apparently, it is possible to get corrupted files sent out over Teams between users. Colleague of mine had a known good file direct from the manufacturer. They then sent that file via teams to several other users that needed access to the file but couldn't access the direct download. 2 of those it was sent to could not use the firmware file because the device they were updating kept throwing an error saying the file could not be validated. I had them send me their copy through a program that I know does checksums and when I compared the file size just on its face it was smaller than the verified original. So while it seems teams attempts to deliver files, I can say first hand that it's not guaranteed to arrive in one piece.

literally right after he said "make sure they don't get corrupted" at 3:40, my blender simulation used up the last of my RAM and made the video start stuttering and I swear to god I just assumed that it was just a gag for the video

I use TeraCopy in windows to handle all file copy and moves because I can turn on its verify option as a default and never have to worry about it again.

One thing you didn't mention is that corrupt file downloads can be deliberately induced by your isp because of "traffic shaping", The worst part is that the download would have been faster and use less data/bandwidth if they had simply allowed the download to go unimpeded instead of forcing you try over and over.

Awww I really wanted Linus to mention salting in the password segment. I know it's too much of a tangent for such a short video but it's a neat solution to an unfortunately real security problem.

It's worth pointing out that the reason TCP's checksums aren't for security is because anyone who could replace the file being downloaded with malware could also just change the checksum to match the malware they inserted. That's why TLS/HTTPS uses an enhanced version of checksums called digital signatures that uses special encryption tricks to prove that the checksum was calculated by the server you're downloading the file from and not an attacker.

If you want to verify a file that's copied locally (i.e., both the source and destination file are on locally-accessible filesystems), doing a file compare (e.g., the Unix/Linux "cmp" command) should be much faster than doing a checksum, and will tell you exactly where the first different byte appears. I'm not a Windows guy, so I don't know how easy this is to do in Windows, but if you're going to get a third-party product to perform your checksums for you, you could probably get a third-party "cmp" program.

Pretty abysmal that Windows doesn't use checksums. I've had a few known corrupted files before and who knows how many unknown corrupted files.

This is still used if you have a slow internet and constant disconnection when doing downloads. Checksum is a way to check if your downloaded file is not corrupted.

Linus has been cooking in that sun

Hashing is one way encryption and used to digitally sign files. Like for root servers handing out certificates for intermediates.

CRC32 will do for a quick check, but for security it is best to use SHA256 to ensure nothing was tampered with.

Checksums vs Hashes vs Keyed Hash (MAC) and signatures could have been more clearly separated / explained. Fitting it into a technique format/speed is a challenge but would add a lot of value / clarity.

TCP/IP doesn't actually do a checksum in that way. now, it has been a while since I read up on it, but if memory serves, then TCP checks on a per packet basis instead. it also uses a kind of "session" number in order to keep track of a session of communication. sending info from A to B would look something like this: A: Sending packets 1-14 B: received packet 14 A: sending packets 15-34 B: received packet 31 A: sending packets 32-42 B: received packet 36 A: sending packets 36-40 So in addition to having a session token in all this information, A tags all sent packets with a number per packet as well. B will read every packet it gets until it has either read all packets or the packet it receives is not the one numerically after the last. so if it gets 1, 2, 3, 5, then it stops and sends back that it got packet 3. Notice how little data B actually uses by just sending a response of the latest packet it received in a series. This makes sure that TCP is not gonna use tons of data to communicate back and forth. But it still does communicate back and forth in order to keep signal integrity. UDP/IP on the other hand is not like that. UDP is like pouring a bucket of water down the drain. Most of it should arrive sequentially, but some might not arrive in order. Or at all. But it doesn't matter, since the receiver isn't checking it. Video streaming is done like this in order to keep up with the massive amounts of data being sent, where TCP might lag behind. But it comes at the cost of sometimes being out of order and have a little lag spike here and there.

The output of a hash functions are not necessarily unique since the input may be infinite but the output is finite. Its just higly unlikely to happen.

TCP/IP -along with a lot of other things - uses CRC-32, which categorically isn't a cryptographic hash (even if it's used as one sometimes).

Thought this was going to be a video on the osi model. This is just as good.

I remember tom scott going on about websites that put their checksums on there And he said if they are able to change the file sent it wouldnt be too dificult to change the hash on the website to the hackers file

Sometimes at low level shit goes wrong, but TCP also contains an ACK signal: if something does not arrive it will resend it.

You should mention that there is a difference between UDP and TCP in this instance. Because if we're doing something over UDP it's not gonna bother with resending it, lost is lost at that point.

Going by the thumbnail, here I thought this video was an incredibly speedy response to .zip top-level domains now being a thing, making phishing and tricking people into downloading malicious data stupidly easy.

Gosh darn it, Colton..

TeraCopy as a windows file transfer replacement has been my go to for years for this.

Yeah, I've been getting these weird "ads" or "sponsored segments" for every video I watch.

TY for bringning that up.

1:00 that's wrong, passwords don't become stored as hash, they become encrypted. Hash and encryption are not the same. Hash is 1 way directional so it csnt be reversed (thus hashing passwords in DB won't let users login anymore as it can't verify rhe oassword's correctness) while with encryption like AES or MD5 user authorization will work.

To address confusion. Checksums//CRC’s/Hashes etc. are often used interchangeably but basically all have the same basic goal. Can you with reasonable confidence know the file/data you have is the one that you actually wanted. Simple checksums use very lightweight insecure algorithms but their only purpose is detecting simple corruption. There’s no security component., meaning it’s relatively trivial to modify the file and tweak it such that it still has a valid, if not identical checksum if you were a malicious actor. When you bring in cryptography the intent is to prevent that attack vector. In that whilst you can modify a file, doing it such a way that leaves the files hash unchanged is non trivial. Any/all methods of hashing will suffer collisions by the very nature of containing less data than the thing it’s describing. I.e. you can’t uniquely describe a 1gb file using only 256 bytes of data, if that were true we’d all just download the file hash and magically reconstruct the original file from that. The essence of the more secure methods is to make it that the collisions will be a function of chance, not intent.

Video Suggestion: How to clean and maintain a Linux OS. Example: in windows you can delete temp files and stuff. How do we do that stuff on Linux. when i use the command prompt to install apps and frameworks; how do i know how to remove the bloat and leftover files after install? what are the common practices for keeping it clean?

PGP / cryptographic signatures are even better still. Every computer and device should come with PGP/GPG... so useful! Even works for signing email. Anyone can generate a hash, and using HMAC requires sharing the password/key (which makes it easy to fake authenticity). Public key crypto is the only real solution.

Since checksums are a much smaller set of data than the data itself, it is possible for certain permutations of data to produce the same checksum, but improbable. That fact and others means that computers are not necessarily totally reliable but may be 1 in 1 x 10 e 10 reliable per bit or so.

Hashtab inserts a extra tab "hash" into the files-options where you also can compare

I had to download a program that would verify a copy/paste. As Windows doesn't do it for some reason. As sometimes a copy/paste gets corrupted. And you don't know until you try and open the file later. And then, that could be real bad if you don't have a backup.

It is not only at TCP/IP layer but eve on the link layer the Etherner frames have CRC of that frame...

My computer literally crashed at 4:47 and rebooted on its own, a very creepy coincidence

And yet we still get corrupted downloads sometimes and have to manually download the file again.

I never got the concept of checksums on a website.. I mean, if a hacker could replace the file, he can also replace the checksum to match the new file.

This is the MOST informative Video I´ve watched so far. Thanks!!

The Internet always sends me the wrong thing. It's called Amazon 🤣

More of this!!!! plz and thank you

Love it when u make 5 minute videos with 1 minute ad

There was a forum I used to use that got confused and applied a scantily clad woman with the text SEND NUDES as my profile picture. I named my PFP the same as that one so the server was like, yeah these are the same.

A quite tingling theme regarding your recent story =)

Is it just me, or does it look like Linus got some sun over the weekend?

Why do I have to worry if U2 gets hacked or not?

You can get the wrong file even despite hashes; the files can simply be named incorrectly. For example, P2P sites that have a search function like the donkey network. You do a search, find a file, download it, the client confirms the file was transferred correctly by checking its hash, then you try to open it, and nope, it's the wrong file. Be careful with those, the file could be various different kinds of bad (not just malware…)

Meanwhile at UDP HQ: Hehe here is your corrupted file, eat it😀

One point you missed. Hashes aren't unique as you stated. That's why we can have collisions. After all, no finite value uniquely mapped to an infinite value. Intentional collisions are difficult to figure out currently due to provably hard math, but accidental collisions aren't what these are trying to protect against. Adding in details like original file size can help reduce collisions even more, but don't completely eliminate the possibility (speaking mathematically).

You better hope your password is not stored as a hash, as rainbow tables solve that problem. Salted hashes... well thats different.

OOof, at first I thought your gonna talk about googles new TLD idea that got real, .zip domains. I love downloading random zip files from the internet that are not what I thought they where. (yeah I got myself a few .zip domains)

Worth noting that TCP and UDP use a small non-cryptographic checksum. It's only 16-bits, not nearly as long as the one the animation showed. That means random collisions are far more likely (but still pretty rare), where random bitflips could pass the check, and since the checksum is part of the packet itself, it doesn't provide meaningful security from intentional changes by a "man in the middle". HTTPS provides end-to-end security that prevents that, but basic TCP and UDP don't.

But TCP already ensures a packet drop/corruption will raise an error, right? That means manually checking the full file isn't neccesary? And if hackers want to tamper with the file, they can also easily change the checksum to the one calculated from the malicious file.

*Teracopy* - It checks checksum after files transfer.

So how does that explain uploaded or downloaded files getting corrupted? This was especially an issue a with slow or unstable internet connection.

A TQ vid on cryptography, specifically password storage and Rainbow tables would be pretty cool as a sequel to this. Would love to see more security related content

Checksum != Hash A checksum is - wait for it - just a sum of bytes/words forming a number. quicker than most hashes but also very insecure, usually fit into one register. A single bit flip will often not have a big impact on the result unlike for hashes. You also forgot to mention that there are cryptographically secure hashes like SHA256, BLAKE etc. which you can generally trust because they are hard to spoof and there are cryptographically insecure hashes like MD5, FNV, etc which are easy to spoof and shouldnt be trusted. SHA1 was broken some years ago by google already btw.

Lots and lots of great notes in the comments this time. I think some of it could've been easily adressed in the video itself but somehow got overlooked.

Getting a hash for a large file while you are copying it does only take a trivial amount of time. You already have the file in memory, it's just a few instructions extra per byte. What takes time is getting a hash for a file you aren't reading anyway, as file operations is where time is spent.

The thumbnail made me think this was going to be about the .zip TLD issue currently going on...

i expected you to talk about error correction codes and how they are used in transmit, would love to see a video of it from you!

Hey guys , I heard that LTT Store Shipping Method is to expensive. Insane.!! You're guys are awesome 👌 👏 👍🏻

1:15 that doesn't mean that a strong hash value can cover for a weak password!! ALWAYS choose strong passwords guys

I am surprised to see you havent mentioned "torrent" while making a video about checksums. In torrent protocol every chunk has a checksum. In that way, you dont need to download an entire file if you recieved corrupted data.

TCP/IP _needs_ the checksum because of data collision is a real problem.

The cryptographic hash of a small file takes more than a trivial amount of time, and that time is constant for files smaller than the hash block size. Large files can take advantage of pipelining and amortize any required context switches. Hashing a gigabyte of data in one file will take much less time compared to hashing the same data divided into 2²⁸ four-byte long files.

I requested a personal data download from a website. instead they served me a Zip file that contained a full length movie. How in the fek did a MOVIE end up on that website... and how the fek did I get the movie and not my data download?

I'm not a pro developer, but I did some web programing and work with databases, and the thing you said about passwords being crypted with hash functions to avoid bad actors is wrong i'm almost 100% sure of that. Cause as long as you know what kind of hashes there are, you can always revert the hash and get the plain text. Or at least it was like that back when I used to do web program. Unless if there are stronger hash methods now, that Im not aware of, all the methods that existed back then, were easy reversed. My cousin used to get access to database dumps of private servers, and we were able to see all the stored passwords in plain text and enter in those accounts if we wanted.

if you have 7-zip installed, you can right click a file and get "all" hash-sums

Checksums are also how automatically de-duplicating filesystems for incremental backups work. Each file is stored not by its name, but by its content hash. The file metadata then just records the hash of the content, and any other file with the same content will point to the same physical extent. Tahoe-LAFS leverages this for distributed files between friends, and Freenet uses a similar process to shard and distribute files pseudo-anonymously across the entire Freenet network.

Don't wanna be that guy but here are a few exceptions to the answer to the question in the title: hash collisions (weak algorithms,) web cache poisoning, and request smuggling.

Hashtab is another great checksum utility for windows, adds a hash tab to the properties of any file, showing it's hashes in many common hashing functions, you can paste in your hash and it will verify if its correct.

Thanks for the video!

Checksums are important. People (or really just Apple sycophants) like to say ALAC is just as good as FLAC and it’s totally not a problem at all that Apple doesn’t let you use FLAC, because you can just convert between the two and lossless is lossless! Except that’s not the case at all because FLAC natively stores a checksum for every track and ALAC is entirely a-lacking in this regard.