I must have watched half a dozen videos about encoding text over the years, but this one is easily the best one! Very easy to understand with all the examples. Now to hope I don't ever have to deal with anything else than utf-8...
Probably the clearest explanation I've seen on the topic. There's lots that cover the basics pretty well, but completely skip over the byte order aspect.
This was super interesting and well done, thank you so much! I had an issue with BOM yesterday that broke my script because the shebang wasn't being recognized. I love discovering Hex Editor Neo, gave me a lot of peace of mind, actually seeing the "ef bb ef" in front of my code haha.
Quite excellent. So many little places it is possible, even easy to trip up and give sub-par or confusing information on this topic, and you just tap dance right across that minefield. Beautiful.
As a Japanese speaker, the fact that the first four glyphs in your thumbnail are "I❤日本" made me wonder if you're a fan of Japan! (日本 means Japan in Japanese)
One question at the end. There are 17 * 256 * 256 possible characters across the 17 planes, not including the various blackout or restricted sections like those used for surrogates...isn't that more like: 1,114,111 than well over 2 million? They do require 21 bits, rather than 20, because they left all but 2048 of the original BMP to lie where they were, and added another 1024 * 1024 possible new ones on top of that, so it is (256 * 256) * 16 + (256 * 256) - 2048...either way, way more than we should ever need if we don't go nuts...as they point out, Unicode is for characters, not glyphs or fonts, so if there are 500 ways you want to write A, that is still just one code point...
To be honest, I think Unicode is one of those topics which, if you keep digging, you may well go nuts. I made a conscious decision not to overthink it (i.e. "tap dance"). Perhaps Unicode will prove inadequate if SETI ever come up with the goods. :)KD
@@ComputerScienceLessons Yeah, one thing that is unfortunate is that we went from a situation where almost everything "normally" used was in BMP, to one where lots of 🤦♂️🤷♂️🤞super-popular stuff ain't. Too bad!
Great lecture! I had a question: what is that weird characters we see on the Hex editor? Because Notepad recognizes the encoding used, while Hex editor doesn't.
Hex editor displays the raw data present on the file. It could have shown that to us in binary format too, but it displays that raw binary data to us in a simpler Hexadecimal format
what web site do i get these things from? you talk about it 16:00 but were are they? how do i find them? what do i search for? this would make a good video.
It was brought to my attention by one of my viewers that the bit sequence for the Greek letter Phi, in the table of UTF-8 values, did not match the bit sequence that was (correctly) derived below the table (thank you!). All of the other information was correct. I fixed the problem and took the opportunity to de-ess the sound at the same time :)KD
Dear Sir, I have tried to convert '€' , in pure binary It is 10000000 and in denary 128, to UTF-8 format. As It is 1 byte character but also starts with 1 . Hence I am confused how to convert it to UTF-8 format. Please help. Thanks in Advance.
It doesn't fit in 7 bits, so it's not 1 byte in UTF-8. It does however fit in 11 bytes, so it's a 2 byte codepoint. Pad to 11 bits and shove it in the two bytes with the proper headers 00010_000000 => 110_00010 10_000000
4:56 in UTF-8 we have control bits which are necessary to identify the number of bytes used to represent a code point while in UTF-16 we don't have such thing because low and high surrogates are known by the the range of values so we can identify if a code point is encoded using 1 or 2 UTF-16 code units by just the range of values can some one correct me please ?
@@justcurious1940 That doesn't look quite right... Did you mean to say 2🐝∨¬2🐝? Edit: oh, I guess that was the binary string showed in the video? Then it's possible the video did a typo or something.
And there are even more stupid encodings like Oracle's CESU-8, which mostly is UTF-8, but encodes chars outside the BMP as UTF-16 and then those surrogates *again* as UTF-8, so it becomes 6 bytes... WTF?
@@ComputerScienceLessons i could not find an online website that convert UTF- 8 data to txt directly i had to convert them to unicode code points first then convert each code point to the corresponding character 😅
