One of my colleagues found a scary version of this. He was using a MCU and was using the last page in the flash memory to keep some permanent variables (a common enough hack). He was changing some printfs and suddenly the board started to reset at random points. Turns out that by making small changes on those printfs, he was shifting code *into the last page of the flash memory*, and since flash can only be erased by page, writing to the permanent variables erased part of his code and creating havoc. The moral of the story, if you decide to use the last page of flash as permanent storage *remove it from your linker map*.
Yeah that probably has less to do with the printfs and more to do with you last point haha. I mean the same thing would have happened if the programmer had just added a bunch of code or pulled in additional library functions that needed to get linked and located.
Good Video Dr. Sorber. I have encountered this issue before, when corrupting the heap, some standard C functions will cause a seg fault. Using valgrind is always helpful when having these types of issues.
Thank you for this video! I think (and that's unfortunately not the case ; at least where I was taught C) it's very important to teach beginners that memory (especially in C) is one contiguous block, so if you overwrite any "variable" then you're ultimately changing some other variable in the process (except when you're accessing unmapped memory of course). I had this happen some time, and knowing that memory is just one big block of data really helps debugging code that is behaving in "mysterious" ways :)
The real "bug" was the malloc of "values" when you thought that it's allocated enough memory for the struct (1028 bytes) when it actually allocated 8 bytes. That confusion in what size is calculated is what makes mallocs tricky. Using sizeof (struct myvalues) should do the trick.
Using sizeof on the type directly is dangerous, because the type of the variable may change without changing the malloc() call. It's recommended to do sizeof on the variable. If you want to be safer, you do: #define ALLOC(type) (type*)malloc(sizeof (type)) int *p = ALLOC(int); This way you will get errors on type mismatch when assigning.
@@monochromeart7311 casting malloc return value in C is useless, as it returns a void pointer. In C++ it is required, not that you should use malloc in a C++ program.
@@filips7158 you missed the point. If we do use #define Alloc(type) (type*)malloc(sizeof (type)) And then do: int *p = Alloc(double); The program won't compile, the type is safer. It's basically C++'s `new` operator (without the constructor call) which returns a type-correct pointer. It's an alternative to doing the following (which can be a victim to code changes): int *p = malloc(sizeof (*p)); It we mistype the variable used with sizeof, and it's still a valid variable, it can still give "incorrect" results.
I had a fun version of this bug. My program crashed whenever I freed a certain pointer. If I didn't call free, it worked perfectly fine. Turned out, I had written off of the end of an array by just a byte or so, enough to corrupt the meta data of the pointer so that free didn't know how big it was, but the pointer could still be used.
That's a classic. In my experience, bugs like this can take *days* of debugging because the crash happens in a place that is totally unrelated with the real bug (the buffer overflow).
Thank you Mr Sorber for uploading these videos despite your busy schedule. I recently switched majors to Computer Engineering and have taken an interest in embedded systems. Looking for guidance and I found this channel. Currently working on the Arduino and just started with a TI. I look forward to your uploads!
We used to call this kind of bug, "Heisenbugs". For the longest time, they were hard to debug, but once you recognize its character, you could identify it and immediately look for the cause - either an uninitialized variable or buffer overrun.
Yes, although with experience you learn to recognize the bug type, sometimes the problem is to find where you did the buffer overflow. Especially with fairly complex software, you could overrun the buffer in a completely different place. Been there...
@@riccardob9026 The worst one was when I neglected to set the direction bit in an interrupt routine. Many exciting hours trying to track that bugger down.
This line: *struct myvalues *values = malloc(sizeof values);* if you test to see the size: *int size = sizeof(values);* it will give you *8* (on 64-bit), that's is the size of the pointer, since you are declaring **values* as a pointer, *malloc* is allocating only 8 bytes for the *values* variable, you either change it to: *malloc(sizeof struct values);* or *malloc(sizeof *values);* this last one it just dereferencing the variable to get the type of the variable and that's what sizeof needs
printf (actually: Serial.printf) is the cause of 80% my segfaults on ESP32 ;-) - and I still use it instead of i.e. simpler Serial.println() from Arduino. I simply love the way printf formats the arguments ;-) And it also helps me program better (I think)
It's weird to me to judge that based on such unpredictable behaviour. You could also look at it as Linux hepling you detect a serious flaw faster instead of helping you ignore the problem for longer.
I don't understand the explanation with the heap diagram. To me, it sounds like you're describing how printf may move the struct outside of the allocated memory space, causing the program to crash. The opposite of what happens in the example.
The drawing is in the wrong order, I suppose. Imagine the purple rectangle on the right, shifting the blue one a little to the left. The blue rectangle still spills to its right, overwriting the purple but without going out the page.
Side effects! This is definitely one of those things you stare at for way too long when you're new to a language. The importance of fundamental understanding is highlighted by bugs like this, and it's even worse when you're in a high level language and the bug is underneath you. It's also one of the things that makes C really fun and interesting to learn for me, when you realise yes you can just read past what you actually allocated into garbage memory until the OS eventually complains you read an address you don't have its permission to read. A good habit for beginners is probably to print the number representing the memory you think you're requesting and see if it sounds as big as it should be.
What this really means is C is broken. For the current generation, thinking about that kind of detail is crazy. I learned by myself working in a machine with less than 1MB and had a portable machine with less than 64 when I was a kid, so clearly, I learned to think of memory, registers, reordering, and that kind of stuff. Today devoting yourself to learning that kind of stuff makes no sense. Today the compiler is better than most ASM code, all modern computers are superscalar and someone is better speding the time to truly understand functional or dynamic programming.
Well, I've got something similar, but not exactly the same. Still, thanks a lot, for it helped me to get a better grasp on what might be going on under the hood.
how do you jump back and forth your vm and mac? initially i assumed it was an ssh but then how do the files sync only plausible answer comes to mind is the vm has a shared folder. please show your dev environment setup some day
Coming from MCU land I laughed when he said printf() allocates 1K-8K. I used printf() on an MCU with 4K of RAM. The issues in that space with printf and all it's buddies is the stack allocation. If you are tight up against it with your heap and stack closing on each other printf and it's friends will happily tramp on through your heap... or trip the ARM fault handler trying to write to program memory.
In general, you're right. It happens to work in this case, because the memory is fresh from the OS, and I think all OS's fill the newly allocated pages with zeroes.
Hi guys, I need advice. I'm going to make an application. What is the best language to make an activity recommendation app and how can I implement the collaborative filtering algorithm? im newbie. advance thankyou for the answer
U don't need to zero initialize the struct when ure reading directly into it Normally U zero initialize variables when their initial state matters. The fgets will read into each buffer and will work fine
Brill. Being ancient, I have fallen into such traps. There was one really good one in HPUX, where their malloc - the contract is no guarantee of content - had the first few bytes loaded by malloc itself. Perfectly allowable. I had to debug a bit of C that worked, mostly, but sometimes seg faulted. Took a bit of time to find that.
It doesn't matter at all. You may use Notepad or even Word. C != Java, there is not IDE or anything, that, for instance, can automatically write setters and getters. Just use Codeblock or Visual Code, but overall it does not matter
But when I wrote short helloworlds like this one, printf("literal") call was optimized by gcc to a puts() call, and the literal is either on the stack or in .data, which doesn't explain heap usage. Maybe something to do with stdout or stdio buffering? Additionally, why didn't you show -fsanitize=address (or =undefined) and valgrind analysis on this example? This would be very helpful and educational.
I bet if you had used gcc you would have had a different result because it is also related to the fact that LLVM is using its own memory allocator (not glibc's).
Thank you so much for this interesting video! I certainly learn something new with each one! can you please make one for dealing with fundamental inaccuracy of floating point numbers? as far as I know, they're never 100% accurate, so how to overcome that if it's possible? the worst thing about the program is an unreliable output.
The only way to "overcome" that is not to use them 😂 Jokes aside, they're 100% accurate at representing sums of powers of two (including e.g. 3/4 = 2^-1 + 2^-2) as long as the difference in range is less than 53 bits (for float64/double) - so representing 2^53+2^-1 will lose the 2^-1. If you need to be 100% accurate, I think most banks/financial institutions work with fixed point numbers - they use an int/long, and say that each 1 represents a cent, or a thousandth of a cent, or whatever.
@@AbelShields if every 1 represents 1 cent, there must be 13 bytes room for the cents only? or using like 7 bits I guess? The problem I'm facing is that output depends on input but of course it's not just powers of 2 so some inputs might not even be representable at all like Pi, but some can be calculated exactly but may need just a big amount of length for the precision. so I wonder how i.e. scientific calculations are done in this situation. (financial ones was a good example but it's not as vast as scientific ones)
@@katiagalkina4607 ahh, there is another approach used by python and other scientific applications. You can use "bigint" types, which dynamically grow in size as the numbers you store grow. You can use these to create rational datatypes (fractions), which should cover almost anything you need. If you have a function that produces an irrational output (such as hyperbolic functions), you can store an "accuracy", keeping track of how accurate the result is as you perform the computation, and you can trade more memory and higher compute time for a more accurate rational approximation.
@@katiagalkina4607 and no, you'd still use a uint64 for the whole balance, but a value of 1 means 1 cent. 100 means a dollar, 256 means 2 dollars 56 cents etc.
Mostly we deal with it by math'ing out how much the error of our calculations have, so we can be sure to how precise our results are. Thats called error propagation. That is something a lot of fields that use math in the real world have to deal with. statistics, engineering, economy, etc. And is just a side effect of the reality and math having infinite precision while our measurements dont. In programming thou it only becomes a issue if doing scientific research, where you either accept some inaccuracy in the numbers in order to be fast. Or do it in a slow way that will assure the amount the precision you need. Or when you doing something very wrong, like accumulating the result of way too many operations in the same floating point number over a long period of time.
Not sure what you mean here. Malloc should never allocate you bad memory. The errors come in you either not allocating enough, or screwing up your pointer arithmetic to make a pointer point to the wrong place. It is possible to overwrite the stack, but it's not your allocation that's doing that.
@@taragnor i meant that the stack is in the higher memory address and heap is in lower memory address in virtual memory. Theoretically if we have enough ram can we overwrite stack with heap?
Interesting topic for a future video. Thanks. Short answer: On most machines, you can't. The address space is too large. On small microcontrollers with only a few KB of RAM, yes, it's possible. Stack/heap collisions can produce some of the most terrible bugs you've ever tried to track down.
Stupid sprintf breaks my microcontroller program and my cube programmer says there's an access violation every time this happens. Edit: yes I literally thought my code was possessed by demons and this problem has been plaguing my project for the past 4 month, every time I change something else in my program this will resolve itself by random chance and come back to haunt me later...
That's the way I always do it. Makes it auto update the malloc if you change the type of the struct. I do sizeof(*values); even though sizeof *values is completely legal. I get confused without the parentheses if there is a * count or something extra after. Works great.
struct myvalues* values = malloc(sizeof *values); is valid and legal code. It's the best way of allocating memory IMO, I also prefer to use parentheses like the previous commenter: struct myvalues *values = malloc(sizeof(*values)); Note that you don't cast the return value of malloc in C.
There's another issue in this code: line 35 in the 1st and 2nd iterations, all three members haven't been initialized yet. Not to mention, you don't free() the allocated memory.
@@user-mr3mf8lo7y This works only for helloworlds of a sole main() under an OS. What if I did this in a function in a loop? What if I happen to use a freestanding/unhosted implementation on an MCU without an RTOS (which happens frequently in embedded)? I am dissatisfied with this presentation and lack of valgrind/-fsanitize=address/UBSan, as that would help catch such naive bugs.
@@JacobSorber C is like juggling shiney knives. Java is like trying to juggle bricks. You never get cut by java but the result is ugly and slow. Of course, C++ is like juggling silver balls. Beautiful but less impressive than C.
It would be interesting for you to make a series of videos about how to create GUI applications in C such as Win32 apps. Videos about the event loop, handles, window procedures, etc. in win32 apps. I did see a library for creating cross platform GUI apps in C that is named IUP and the documentation for IUP appears to be really good. There's GTK as well which is clumsy on Windows. You can go from embedded stuff all the way up to GUIs in C. Videos about testing (unit testing and such?) in C would be nice too. Perhaps create a custom library and test it?
@Jacob Hey, I have the codeblocks IDE and downloaded the MinGW v10. The thing is that there is no /bin or gcc.exe in it. How does this get remedied? Have you heard of others having this issue? I truly don't understand why that/bin/gcc.exe isn't in the MinGW download
It would be great if you didn't use dangerous functions like atoi() (replace with strtol()) and strtok() (replace with strchr() and length calculations). Beginners would end up using them, when better alternatives exist. It's bad enough that many teachers use scanf() to teach, when it's so problematic. Also, another thing, fgets() takes the size of the buffer including a nul-terminator. So if you allocate BUFSIZE+1 characters for a buffer, you should pass BUFSIZE+1 to the 2nd parameter of fgets(). From the man pages: " _char* fgets(char* s, int size, FILE* stream)_ reads in at most _one less_ than *size* characters from *stream* and stores them into the buffer pointed to by *s* ."
What a load - using scanf you have students taking input and doing something that they feel somewhat productive on day 1. Get them up and running and motivated on day 1 - teach them proper ways later after they have some competency. If profs did what you want students wouldn’t be doing anything productive for about a month or more.
@@lef1970 or, even better, provide them with a small header file containing a simple get_int() function, so they wouldn't use the dreaded scanf(). Then the curious can look inside and see the fgets()+strtol() combo, but later on the teacher will explain it as they study C strings. Currently, many students end up doing the following: scanf("%d", buff); scanf("%d"); //failed to get 2nd number because of newline scanf("%s", buff); //get a single word but not entire line? What happened? scanf("^[ ", buff) //solution! Get entire line! But buffer may still overflow But if they avoid learning scanf(), they will use fgets() to take input. fgets() will help teach them about buffers and overflows.
@@monochromeart7311 I think that’s what that cs50 does actually, provides a library with wrapper functions to get the students up and running quickly. Not 100% sure on that as I’ve only seen code snippets from people using that library on stackoverflow. Personally I like a pure approach of standard c, ie: when I was school I was in my second year before I learned that conio.h wasn’t standard c. I don’t see the pitfalls of scanf as terrible, you have to understand streams to be able to visualize the newline being left in the buffer - and it students to read - it’s scan format - when c says format it means format ;) it doesn’t mean it’s going to read to the new line, it says it’s going scan the format you provide. Good for students to learn right away that c does what it says it’s going to do ;)
@@lef1970 I disagree, scanf() is a mess, and is also almost always the wrong tool for the job, especially for reading strings. You use "%s" to get a word, not an arbitrary string, but to get a line you need the ugly "[^ ]" format which still doesn't handle buffer overflow and other issues. It's best to learn not to use scanf() and its family.
@@monochromeart7311 then how else would you get a student to write a chunk of code that asks for their name and says hello name on day 1? With fgets youd confuse more than motivate on day 1.
Wow, I've never thought about this kind of bug. Good thing I use C++ which shouldn't allow this to happen. (I mean using C++ paradigm, malloc can be called in C++ of course... But who does that ?)
"sizeof expression" and "sizeof (expression)" are equivalent in trivial cases, it is legal C syntax. sizeof is an operator, not a function. Also, VScode uses whatever compiler you set it to, which is probably Clang if he's on Mac.
_sizeof_ is an operator (not a function or macro), it doesn't necessarily need to have its operand between brackets. It's a matter of taste whether to use brackets, some people prefer to not to so _sizeof_ doesn't get mistaken as a function.
@@pajeetsingh it's because of the use of a typename, not a requirement of "sizeof". In the video he uses "sizeof" on a variable, which can be done without parentheses.
Another brain teaser for you (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)). Sometimes it does, sometimes it doesn't. I can't reproduce it consistently (e.g. a certain optimization level): $ cat bang.c #include #include int main (void) { long x = LONG_MIN; long y = -1L; x /= y; printf ("%ld ", x); } $ gcc -Wall -o bang bang.c $ ./bang Floating point exception (core dumped) In any case: the answer is always wrong (obviously).
@@dolomighty74 I know - that's why I stated: "obviously". BTW, it returns LONG_MIN. -LONG_MIN can't be done for the same reason - although that could be calculated under 2 compliment as (~LONG_MIN)+1.
played around a bit (gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04) here) the exception gets triggered by the integer division overflow, in fact commenting the printf doesn't change that, but setting y anything else than -1 fixes it... good to know. but now i seem to recall of something similar happening ages ago with watcom c and fixed-point math code sometime... have to check
@@dolomighty74 I know why it's triggered. And yeah - like "DIV by ZERO" if you change the denominator it's fixed. It's baffling though. In some cases the optimizer catches it, but without optimization (possible) it just plainly bombs out.
