Code Pages, Character Encoding, Unicode, UTF-8 and the BOM - Computer Stuff They Didn't Teach You #2 

I hit subscribe automatically after watching this single video. It's the first time I found this channel.

Never gets old this stuff, I’m trying to debug Japanese PDF generation using an ancient open source library. Hours of fun.

Sir, you're right, not everyone shares the same class, really appreciate what you're doing. I hope you're channel grows.

Great video! I’ve been designing and building software full-time (mostly large UNIX backend systems) for the last 30 years, and I still learn so much from all of your videos. I love your from-the-ground-up approach. It reminds me of things I’d forgotten, and fills in the details of things I never consciously thought about, but have just used without fully understanding them for years and years. Many thanks for helping humanity!

Thank you very much. You showed us so clearly how this is used in practice. This is a tech topic that has never been demo'ed adequately. Humbly requesting you to share a video on how Unicode text and its encoding is handled in PDF files and displayed/printed.

I love your video, you're course is more understanding than anyone. Thanks

I've had your tools page bookmarked for a decade, subbed immediately after seeing this video. Top quality 👌

I am learning so much, and I love the conversational tone. I did not finish my formal education, but I am pursuing an IT career and this really helps make the task less daunting. Thank you.

Brought back memories. My high school crush's name had an accented e, so Alt-0223 was committed to memory really early :).

Thanks Scott! I never learned this stuff. It is very useful and you present the material in a very simple way, so it is easily understood. I'll watch every single video you make in this series!

Great content! It's amazing the things you realize that you either forget or never knew. I was aware of the various encodings. The BOM , however, I either never knew or completely forgot about, Thanks Scott!

Excellent Explaination. Feels like learned something unique today. Thanks

Really cool series! The basics are often overlooked although they are essential building blocks.

Thank you for taking the time to share, and enlighten us as well, with your acquired knowledge.

Well i for one have never been taught this before, thank you for sharing your knowledge.

00:42 Character Encoding 01:26 ASCII 02:19 ASCII: writing an app 05:57 "who decides that this character means A" - understanding what encoding is 07:11 encoding: "another way to think about it" 07:24 notepad: "it looks like crap" 09:11 notepad2: new file & Chinese chars & CMD 11:32 encoding: signature/utf BOM 12:24 unicode website 12:31 windows charmap utility 13:40 encoding: you can potentially loose information 13:58 why BOM is useful 14:50 the app: hard-coding the BOM 16:36 summary This table of content was created using "Smart Bookmarks for RU-vid" chrome extension. You can import and edit them using this extension. You can install it from the official Chrome Store Page (shortened link): smb.page.link/store

Good stuff Scott, keep them coming. Thank you for all your work. 🙂

You are the teacher I never got. Thanks for uploading!

I really enjoyed this video. I was aware of encodings, but this has always been an area of confusion to me so I appreciate it. Keep it up Scott!

Love the title and the series. Thanks for helping the community!

Awesome and very useful series. Looking forward to the next videos. Thanks Scott! :)

Just found these list of videos. Really great. Thanks for this Scott

Entire series "Computer Stuff They Didn't Teach You" is excellent. Please continue and I wish you good luck ! Idea for future video: "Asynchronous/Synchronous Requests"

Back in the 1990s, Microsoft tried to get ahead of the curve by making strings made up of 16 bit characters as the preferred format for its APIs. The hope apparently was that every character in every language in the world would be assigned a unique 16 bit number. This was basically UCS2. It soon became apparent that there were too many characters in the world to accomplish this. So, people came up with UTF-16, which has a single 16 bit representation for most common characters, but other characters are represented by two 16-bit words. UTF-16 is what Windows uses internally. Although, typically strings in this format are 16 bits per character, UTF-16 is technically a variable character length encoding, so you don't truly have the advantage of one character per word, so you can't reliably index directly into strings, and string length doesn't necessarily correspond to array length. The UTF-32 encoding does have one character per word, but it wastes quite a bit of space, and isn't very popular. For both UTF-16 and UTF-32 you also have to be concerned with endianness Meanwhile, the internet moved in the opposite direction, with around 95% of all web pages now using UTF-8, in which every character is encoded in either 1, 2, 3, or 4 bytes. As of late 2019, Microsoft seems to have reversed course and is now recommending UTF-8 encoding be used internally for new apps. docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page * All the discussion above ignores the fact that you can combine two or more Unicode characters to form characters. For example, you might combine a base Unicode character with an accent Unicode character to form the accented form of that base character. So, you can't really always treat a character in a language as a single UTF-32, UTF-16, or UTF-8 character.

FYI... The Unicode standard says "Use of a BOM is neither required nor recommended for UTF-8". Using a BOM with UTF-8 is almost exclusively a "Windows thing" -- you'll rarely see it on other platforms. Most of the non-Windows world treats text files as UTF-8 by default.

Great stuff, Scott, thanks so much! I'd welcome your insights on Int32/Int64 and single- and double-precision IEEE-754. My students would appreciate it as well!

Idk who's been in your comments section telling you, you are anything other than awesome but I love your videos and the name because 1)the name feels extremely accurate (I always go to your videos for things I wasn't taught) and 2) As the Bob Ross of coding tutorials you could never deliver a subject poorly.

this channel is a gift for humanity :)

Keep making videos like this. Amazing stuff. I like how productive you are with VS Code.

Thanks for tutoring Computer Zen Mastery skills, Scott.

Providing Fantastic Quality Content as always, Thank You Sir.

I was aware of some of this but never learned it. How have I just now found your videos? I am hooked! Thank you!

WoW clear the basic concept with live example, loved it. Thank you

I love this series. Thank you so much.

Great idea! Looking forward to more

Thanks for making these, this is a great series.

Scott, thank you for share your time and large knowledge with us. For me this tips are amazing.

Thankyou Mr Hanselman, as you might have guessed, I was getting a little antsy trying to understand character encoding before finding this video.

Great explanation, I was familiar with them still learnt more about tools and background. Please keep up this series

I would thumbs up you twice if I could. I don't know where I missed this course but I've been programming for more than 20 years. Just shows you how little that means.

I agree, this is a useful series. There's just a bunch of extra information that's so helpful that you which you'd picked up earlier.

Thanks Scott! Great stuff.

A really nice explanation! I enjoyed listening to it.

Dug this. I encountered the critical importance of character encoding in the wild and it made my life hell lol. So important to get this in ones head from the jump

Thanks Scott! Great series

Such a nice video. This made me subscribe to the channel. Everybody should know these kind of fundamental concepts.

Awesome.. enlightening. Best explanation ever.

One correction, on Notepad2, when you change Enconding, you are actually getting the displayed characters to a new Code Page. So an "á" or a "ç" would still be an "á" or a "ç" (if supported) but in a different Code Page and with a different byte value (probably). What you meant to use was Enconding -> Recode. That rereads the file from disk interpreting it in the selected Code Page. Also, codepage 437 was used I think mostly in US, elsewhere codepage 850 was more common since it had more accented characters.

That was da BOM! Seriously very cool even for seasoned people.

Great stuff once more!

This video was simply awesome! I love it! Thanks Scott. Can you make one for Networking IP address and VNET from Azure Networking perspective

Thanks Scott! Can you make one explaining the Http protocol? And explaining from the bottom top What happens when the user types a URL in the browser? Also another suggestion: how TCP/IP works on the bit level?

This was great, thanks!

Finally a simple explanation of code page.

Great stuff, thank you!

I love me a good scott hanselman vid, thank you for putting these together!!!

Great Stuff -- as someone dealing with enterprise legacy systems -- aka IBM Cobol reports ... I often deal with reports formatted for "green bar" paper. So this mean reports of a given width and given page length -- typically 132 by 60 .... or is it 133 by 66? The reasons for this and the history may prove interesting and help those that see these legacy report and wonder why? Also -- while you passed over it briefly -- big-enden and little-enden encode of data may also prove interesting -- and for those of us who remember -- don't forget to mention octal notation ... Thanks again for passing along "lore" of computing.

Thanks for your lessons. Can have a video about what makes the difference b/w each objective oriented program?

Idea for a future video Scott... “Little / Big Endian, What is that all about?!” This video took me back to my school days where we wrote code with a pencil on paper. My school of about 1,500 pupils had 2 computers. I’m still not a coder, but this was fascinating. Thank you

thanks for the videos, as someone who works with many languages this series is very helpful very informative also, whats the path highlight extension, inside the console

Great video. Thanks!

Very informative, even for advanced developers. I couldn't get over the 3 spaces though :P

Great video Scott! You da BOM!

This is Gold!

Scott... you're doing good work here. Question... how do you get your VS Code terminal setup fonts/etc to match you Windows Terminal and PowerShell terminals all to match? It seems when I set one up, it doesn't apply to the others.

awesome channel!

Good job. Thanks for this.

I was unaware of code pages! thaks.

Code pages used to be part of every day computer use, and didn't need to be taught. There were a few Windows code pages for European languages, Wester, Central, Baltic, and way more DOS pages because of fewer open spots after DOS box drawing symbols. Why does the hex dump ignore the top bit and output ABCD again? I've never seen a hex dump behave like that. It should display the complete set of DOS symbols, or replace some with dots, to aid visual extraction of text. I recall a case where there was an incompatibility even within Unicode, where one non-Windows program did a simplistic conversion of Latin-1 into Unicode by putting all characters into the range 0 to 255, intead of their correct spots for things like double quotation marks down into general punctuation, and the copyright sign where that goes. This resulted in files being unopenable if those characters were in the names; and they put all crap into filenames today.

Really cool title, rich content, thanks 👍

I love this video! I found it searching for howtos on dealing with file encoding in golang programming. I have written a program that parses a text file with a very old internal syntax (character 127 is the token delimiter, character 212 is the line delimiter etc.) I found that my program runs great if and only if the file input is a utf-8 file. When I fed it a file encoded ISO-8859-1, it broke. Now, I'm trying to figure out what I should do to fix the problem. I think the first step is to "detect the encoding of the input file" and second might be to "convert from the given encoding to utf-8" before processing can begin. I know that I can encode the file on my mac in the shell using iconv, and it works file (the program processes the file with no issues after I do this manual conversion). I'm wondering if you have any comments?

man i really enjoyed this video. taught me a lot

I went through 4 years of my Computer Science degree not entirely getting to grips with all of this, and it still gives me a headache to this day.

Great video.

Thank you sir .... No they never thaught me this, even though we were using it again and again and I asked them again and again. Again ... thanks a lot sir.

thanks scott, very cool

Ran into this gem a while back when doing some data science stuff. Opened a text file and was seeing Chinese characters. Open notepad, even in Win 10. Type the following without the quotes "colourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolourcolour". Save it, then open it again with notepad.

It was so helpful, thank you could you explain the difference between encoding and encrypting

Awesome Stuff, Is that Hex dump view an Extension? Thanks Scott

16:50 Great crash course introducing the issue! 'Course, could be a post-grad, possibly even undergrad, course devoted to this subject, I would imagine.

Scott, First, Great videos! In your first two videos you've used hex. A lot of non-CS people that I've worked with don't understand hex, octal, or binary. Actually, even some CS people don't get it. If you're going with basics, maybe a short video on how all of those number systems work would be good.

I'm 41 years old, and have been wondering what happens to cause my .Txt file to turn to gibberish since the 8th grade. Thank You Scott!

sometimes we need basic to reach master.. thank for creating this video.

Suggestion: explaining the different bases, hex, octal, binary, bit shifting, xoring... could be an interesting one

Thank you so much.

@Scot Hanselman love the video and the theme on that vscode can you please tell me the name of it?

I was just wondering whether it is possible to type using a keyboard alt-number combination or other combination any characters beyond the 256 set?

Thank you. Your videos are great!

Thanks a lot for this video

This was on my TO DO list.

How do applications (notepad/notepad2 whatever) understand the first 3 bytes of a file are BOM or file data?

What is the VS Code theme used in this video, I like this theme

question, before fonts existed, how was the ascii letter drawn on screen?

In UTF-8, bytes above 0x7F require special encoding to be valid. As such, those lone 0x80, 0x81, 0x82, etc. are not a valid UTF-8 values, and a proper UTF-8 parser will give you an error there.

Scott- Thank You! One request mate, Can you please tell us a short cut while coding in Vscode.

Approach taken was very practical. I have a question, shoudnt the range of the loop be 0 - 127 instead of 0 - 128??? Why for the second loop (0 - 256) was converted to be integer ??? Byte is 7 bits right ?? So its range is from 0 - 255 ??

Great video! now I understand how I made those characters in chat lol

Thanks Scott... :)

Brilliant!

Loved the video, and the knowledge you shared. I know you're probably keeping to a specific time frame, but I have a suggestion. Can you make a video on the advanced character encodings and conversions. It's important to know that ISO-8859-* itself has issues and it has various versions that either made improvements and made breaking changes. It'd be important to know that going from ISO-8859-2 (pulling the -2 from memory) and going to UTF-8 has a block of bytes that doesn't convert over so it's lost during conversion.