More broadly, the choice in Python not to have a prominent division between things that are pointers (still can be changed when you send them off somewhere else) and things that are not pointers (are copies when they're elsewhere) is a very easy place to get tripped up, especially for those who are used to stricter languages.
i believe everything is a pointer in python, and nothing acts like a copy, making it more easy to understand the behavior of a program… In the last example given, L was indeed a pointer, but the += reassigned the value of the pointer to a new place, and therefore, the change couldn’t be seen from the main function… The problem comes from the fact that, if not well documented, it is difficult to know what will be the behavior of basic functions (addition, +=, …) on objects.
@@JohnChaussard While yes, everything in python is an object (and thus, a pointer), the immutability of some things make them behave more like values. But as you are saying, when specific operations are ambiguous over whether they operate on the pointer or the object, that's confusing at best (and bad language design at worst). This is all over python (see descriptors for example), and while it does enable some dark magic, I'd personally argue it does not make the language more usable.
+= is fine, just always treat everything like a primitive and you'll be safe. In his example, if he would have returned the l var ([4,5,6]), he would have gotten the expected results.
Developer1: "+= has a bug, what should we do?" Developer2: "Just fix it" Developer1: "We don't have time, we must do something else" Developer2: "Tell everyone that this is a feature, not a bug" Developer1: "... Great Idea" Great video! Learnt a lot form this!
I would say this is broken semantics, more than a bug. An operator should not have two very distinct semantics (either modify the object binded to the variable, or rebind the variable to a new object). There should be two distinct operators / functions, for each semantic.
@@vbregier I wouldn't say it's broken, it's just different contexts: in all cases it means "add this to that", if I said I wanted to "add a character to a string" it's obvious what my intention is: take the string, append the character to the end and wríte that back to the variable. If I said I wanted to "add a number to a number" it's also obvious what I want to do. Same goes for "adding a string to a list (of strings)".
@@adrulb The semantics of an operator should not depend on the context. Different semantics need different operators / methods. You are not seeing it because the word "add" itself has this problem : do you mean generate a new value, or modify the value you add to ? Mutability of objects is crucial in programming, it must be clear which objects are modified and which are not for a given operation.
@@vbregier Yes, the problem is that concatenation is not the same as addition. Not having a separate concatenation operator is one of the few flaws in Python.
I’ve been writing Python code for a long time.. and a lot for my own videos, but I learn something new with each of these lessons! I think I need to hunker down and make sure I watch all of them.
Same here. I'm writing python for around 7 or 8 years now. And this is the first time i saw this problem. This is because i never used python in a way that will show this behaviour (because tuple is strictly immutable, and because i use list.append and list.extend instead of adding it). But i immediately got what will happen in this code. Just didn't happen to me ever.
There's a principal in language design known as "least astonishment". The idea is to make a feature work in the least astonishing (aka least unexpected) way. So when you create a feature, make it work in the way most developers would expect it to work. I don't think Python got the memo on this
Personally the "BadList" implementation makes more sense to me. For appending things to a list, we have append and extend. a += b should be effectively the same as a = a + b
in python 2, this was the implementation, and if you iteratively created a large list, it would use SEVERAL GBs of memory (think 125GB for a 500 000 element (500 KB+) list). My friend came across this when doing a file handling program and crashed python - It can be done in less than 10 lines of code (and it is difficult to fix).
7:30 Rarely comes up indeed. Can't recall ever encountering this specific issue after 10+ years of programming in Python. In any case, mutability is the root cause of this and should be handled with care.
The mutable integer id really blow my mind! I learn so much with your videos. One suggestion: you could put the python docs that you are refering to on the description :) Thanks!
Great topic. I had a hard time figuring this out a year ago when I was working with PyTorch - it turned out it was imperative to use x = x + foo instead of x += foo in many places since they are, counterintuitively, not the same expression... This video shed some light to the matter. Thank you!
This video is fantastic. I've been using Python for years and noticed some weirdness with these operators but never dug into it. Thanks for producing such high quality educational content!
Ahh Python, reminds me of good old True = False, and messing around with the cached small integer objects, changing their actual ctypes value. It's definitely good for quick prototyping but there are so many quirks... Looking forward to watching more!
The more I learn about other languages, the more I understand why people still write in C. It's the only language to date where things never not made sense to me. You pass references, pointers, you can modify any piece of memory anywhere in the stack, or allocate heap, it's perfectly transparent. When working with Python, that I do often use and is great for some stuff, whenever I'm working with a new function or circumstance I never know what to expect without reading the full documentation on how it works under the hood.
A talk on persistent data structures would be awesome... You don't seem that much a functional programming enthusiast, but I think it might be great to discuss limitations of standard data types in relation to copying and mutability and how it can be circumvented.
Dude. Do you sleep? Your content is of impeccable quality and is just shooting into my notif box. How do you make so many freaking good videos I love your channel ahhh
Pretty sure I just watched an 8-min Jedi mind trick designed to increase James' subscription count. Fair enough. Subscribed because it helps James and it feels good.
"that can get you into a lot of trouble if you're not familiar how python works" ... and that's why it's not a feature rather a bug(to me). That was one of the first serious case that had my and Python's relationship in trouble. "Fluent Python" also demonstrates this type of example and it had my mind blown away. You cannot just report an error on something and then do that exact thing anyways. Other than that, great video! I don't think i've seen this "bug" covered elsewhere. It'd be cool if you covered data-model more in depth. Thanks!
Yeah, the data model for __iadd__ is broken in my opinion. Instead of usually returning `self`, it shouldn't return anything. Then, instead of the += turning into `x = x.__iadd__(y)`, it should just be `x.__iadd__(y)`. The whole point of the in-place overloads is to modify yourself instead of returning a new object, as the base versions of the operators do. Not only would this help people understand the point better, it would remove your tuple bug.
yeah this really confused me at first. I think the part that got me was the reassigning the variable after doing the iadd, which other languages typically don't
interesting to see what COULD go wrong. I think the lesson is: don't write a class that inherits from list, and keep += for basic loops, and use .extend() for lists instead
2:38 Argh. This is the sort of bug you can get when you use C macros which have parameters. And yet you get bugs like that in python even if you write something which looks safe.
3:27 "I can't change a 1 into a 2" this is mostly true, but with ctypes: import ctypes ctypes.c_longlong.from_address(id(1)+24).value = 2 print(1) this code prints 2 until you change the value again, or restart python, 1 is 2
When I first started coding in python coming from c++, I was baffled by the fact that there was no '++' to augment an integer and that I had to use the '+='. How naive I was back then! But even though I understand now that the pythonic way of doing things, rarely if ever need an index for a loop, I did not understand the specific reason why It is that way. Thank you for this amazing explanation!
@@mCoding you should always use += where it makes sense (note that for lists etc. they have an extend method which is usually better) as it allows the implementation to optimise the operator, which is also the reason for the behaviour explained here. += is always the right choice, except when there's an appropriate method
I already kinda knew this inconsistency because coming from other languages you'd expect x += y to be short hand for x = x + y Which as you explained isn't true for lists. If it were just a short hand expression it would completely avoid this Problem and python could just add another operator for in-place addition to make up for the missing functionality. But yeah, nice video and i learned something new
luckily, Python has a few very neat features that allow for pure functional programs, like lambda, yield and defs inside defs... :) can't image programming without those nowadays
@@mCoding no doubt. Python and I don’t get along well though I have to use it for some parts of my work. I can see why some people really like it and I do acknowledge its strengths. But it’s just really not my preference to work with. I jokingly tell my colleagues I’d rather write pure assembly but it’s all tongue and cheek and I respect Guido and his work with python as well as everyone else involved and of course everyone who’s good at programming in python. It’s just preference and my preference is elsewhere.
I honestly didn't know the += operator can be used on anything other than arithmetic data types in python. However, is it really more convenient and clear compared to the append and extend methods for lists for the sake of argument? Nevertheless it's good of you to point this out. Thanks.
You can use any operator on any object as long as the method called by the operator is defined for the object. This is just the fundamental concept embraced by the object-oriented programming paradigm. Create a new class with the __getitem__() dunder method defined and suddenly you can use the indexing operator on any object instantiated by your new class, and it will execute whatever you defined it to execute, regardless of whether it has anything to do with indexing at all. Most of these "gotcha"-level videos revealing new information have less to do with actually introducing new information and more to do with just exposing our laziness in applying already understood knowledge. We all know operations in Python are in-place or not, so understanding whether the augmented operators are in-place operators is just a consequence of whether we bothered to read the documentation regarding them.
I would be interested in hearing your views about Pythons behaviour when returning from a finally block (how it implicitly discards any unhandled exceptions prior to the finally block)
I generally like your videos, but I really think this one is clickbait at best. You assert the issue as if += is the bad guy, but everything you talk about relates to mutability which is a concept that exists beyond the confines of +=. Mutability is a very important thing to learn, and you covered it fairly well without ever mentioning the word.
Even the exception case is not relevant, unless you relentlessly ignored it and kept running the code... Otherwise it's as bad as a syntax error and should be fixed before the code can be used.
Immutable integer types seems ridiculously inefficient, I've never understood such design decision. Ideally mutable and immutable types are available and the programmer chooses the appropriate one. In cryptography code with lots of math on big integers for example, mutable variants are far better performing unless you need a copy returned and then you want immutable. Why cant we have the best of both at our disposal.
Me: *doesn't use python at all since I use C++ on more occasions* also Me: "ooh new video from mCoding about python? time to drop my projects and watch"
To summarize, for mutable variable m1, "+=" will create a new mutable variable m2, and then copy the content of m1 into m2. Just remember "+=" will always create a new object and that's it
.. so much for the "intuitiveness" of Python. I'd rather use C. At least I can approximately predict what assembly it punches out instead of tearing my hair out trying to figure out why this thing does something I did not ask for.
Operator overloading is quite tricky for maintaining clarity and correctness whether it's C++, Python, Haskell ... I think the nature of overloading "hiding" function calls is what partly inspired the Zig language from the perspective of C++
Another sharp corner only slightly touched here is that .append() would do flawlessly on a list item in a tuple, because it preserves the constant pointer.
Python is built on C. Arrays always have the first element as the pointer, so as you increase the list size the memory address stays the same. Integers don't tend to do this in C. The value is either written into a new memory location or it uses an enumerated one.
I found all of these example completely obvious, natural and well... correct. It's not even weird, how _else_ would they work? ...I may have used Python extensively during my formative years.
So how should a custom class implement __iadd__ to not change the object itself? In other words, how should BadList be changed to behave like list, where __iadd__ works regardless of scope.
It feels as if they would have been better off not adding these operators to the language. They mean something clear in other languages but that doesn't fit well with the data model of Python so it would have been clearer to force the one extra keystroke of x=x+1 instead of x+=1.
With the benefit of bindsight, there probably should have been two different dunder methods. * One for when the return should be reassigned to the left side of the += * That returns nothing (except for possibly NotImplemented). The change would be expected to happen to the first parameter (usually 'self') That way, when trying to do += on members or elements, Python would know whether to do the final reassignment based on which is present (or which one doesn't return NotImplemented). Of course this raises the issue of what to do if both are present, or which order they should be attempted in. So it would be a new sharp corner about _implementing_ += But that moves the sharp corner to the implementer, not the user of the API. But that still would have been a worthwhile improvement.
I've almost never used it, only for increasing numerical variables, but the documentation that I check out when learning always had alternative methods that it's suggested to use
I have always been suspect of shorthand expressions. x = x + 1 is how I learned when BASIC had line numbers, C/C++ came along and made me hang a semicolon at the end of it, and Python circumcised it off for me again. x = x + 1 is more readable.
Wow I was running into a "bug" because of this feature and didn't know this was the cause. Thank you and thank RU-vid recommendation for popping this out on my feed.
3:40 Interesting that the id is different every time. I presume it is derived from the memory location, similar to the default hashCode implementation in Java?
I don't see the issue here (except for the mutate-then-error behaviour, that _is_ weird). Tuples are unmodifiable, lists are modifiable, integer primitives are immutable. You can implement iadd in an immutable or mutable manner, just like any other method. If you're publishing the code anywhere, it should be documented
Whaaat, why would integers in Python work like this? Honestly it seems so weird to me, I don't get it. The more I watch your videos the more random Python seems to me lol
Python int type is not an integer type. If it was, it would be fixed size and be able to be made arrays of and pointed to. Python is an illegitimate coding language.
Python had to be built in that way to gain traction, so that some guru could start talking about being pythonic, evangelists come trying to teach the light... at the end it's the kid coding club language. I like to see those videos, I'm reassured of my choice of skip python and I use the content of the videos to make my pythonic-friends get creazy because they still ignore the bad reality behind this language.
These sharp corners become far more blunt when you realize what these features are doing under the hood. This behavior makes perfect sense in its actual context.
If you use an integer which is sufficiently maybe the reference will not change. From my own experiements I've seen that python shares reference for small numbers, presumably because they are used very often and will be a benefit to performance.
This thing almost made me loose my sanity when I tried to implement it in one of my projects. Basically I wanted to make a dummy class that would do nothing on __iadd__ except memorising that __iadd__ was called. And due to other implementation details this __iadd__ ended up setting weird attribute values to other seemingly unrelated objects. It took me a while to figure out the reason
The id is a unique number guaranteed to be unique for the life of the program. This can be used for a hash if your objects are immutable and no two different objects should be considered equal.
A programming language that has out of the box THREE different variations of IsNumber() function, and neither is able to parse a float is in no way an "intuitive", excuse me. EDIT: ....and your video is a great proof of the above :)
Reminds me of the mutable default argument "gotcha." def frobnicate(foo=[]): foo.append(1); return foo; has surprising behavior when you run with the default argument more than once.
ngl i thought the phrase "It's not a bug, it's a feature" was just a funny meme that didn't reflect reality....but this is the most bullshit python thing I've ever seen. Even the explanation as to why this is like that is barely tangible and seems like a retcon to hide their failure.
The hardest thing I find in python is to know what changes base on other stuff changes, for example: does a change too or a and b are completely unrelated now? def func(x): return x + 1 a = 1 b = a b = func(b)
b is being rebound to an unrelated value, therefore a must not change. You can, to some extent, think of names in python as being pointers to values (and they are in the cpython implementation). Copying a pointer doesn't change the link, but generating a new pointer (x +1) and setting that unlinks the names
@@Starwort that's is where it gets confusing. Because it acts very differently for different objects. If a and b were a list and I change b like b= b +[1], a changes too
@Daniel Agostinho "b= b +[1], a changes too" it shouldn't. To me it is entierly obvious how it work, let me try to explain it with an analogy. You meet a person, and she says "Hi, I'm Anne." This is the "a = 1" , with person being the "1". There is a difference between the name and the actual person. Then she says, "Actually my name is Betty-Anne, so some call me Betty." => "b = a" Then she goes into a hairsalon, and says "The next person to come out of that hairsalon is called Betty". The analogy gets weird, but the hairsalon is the function and going into it is passing an argument. Now, you don't know who will come out of that hairsalon, but you are determined to call whoever comes out of the hairsalon "Betty", regardless. In the first instance, where you add 1 to the argument, another person comes out of the hairsalon. A big guy with braids; still you now call him "Betty", and you can still call Betty-Anne "Anne". Different people. Had your function been one that modified the argument instead, like appending to a list, the analogy would have been that Betty-Anne came out of the hairsalon, only with a different haircut. Dutifully you still record that Betty-Anne is now called "Betty". It should be _entierly_ unsurprising that after this "Anne" has a different haircut. She is the same person as "Betty" in this case. In a third case, "Betty" could go into the hairsalon, get a new haircut _and_ someone else comes out before her. "Anne" would get a new haircut, and "Betty" would be reassigned to another person. Don't actually do that in code though; it is bad form. Either create a new value that you return, or modify the passed value and _don't_ return anything.
@@dtkedtyjrtyj Okay you are right, just checked that exact code and I clearly don't understand how it works. But I remember writing some code where that actually happened. Where I assigned A to a list, made B = A, and then performed something on A and B was still equal to A. ... Okay, got back from that code I wrote and here is an example: >>> a = [1,2] >>> b = a >>> a [1, 2] >>> b [1, 2] >>> a.pop(1) 2 >>> a [1] >>> b [1] >>> Which again it first got me off guarded, because I always thought B shouldn't be dependant on A after it's assignment and any change on A wouldn't affect B. Which still confuses me a lot like does this function changes both, just the one. But thanks for that analogy, it helps.
This can help in some cases, but this will lead you astray for how the builtin list behaves. list += is x.extend(y), not x = x+y, and these two will have different behavior if x was passed a function parameter.
I've never heard anybody pronounce Tuple the way you do, it makes me wonder if you've ever heard anyone else say it. Everyone I've heard say it, says "TUH-ple", including American and British dictionaries.
As someone who comes from C++ the hidden types of the variables was already horrible for me on its own but omg there is sooo soo much more to it than I thought... You can't tell where stuff is declared, you can't tell whether stuff is copied or moved, you don't know when stuff is read only or not, you basically don't know if an operation will work until you've tried it... Python basically baits you into liking it because of its simple syntax when writing 20 line scripts but once you try anything beyond that it turns into an ugly untracable mess.
Yeah, under the hood it also compiles to mediocrely compiled C which is even more confusing to work with if you even try to optimise it, I’d say avoid python and learn swift or rust.
Python hides these things away because 99.8% of the time you don't need to know, and in that 0.2% you can just read the docs to find the answer, its really not a mess.
Yeah, I don't think anyone really uses += for anything but integer and float addition. I certainly don't. Now that I've watched this video, I'm glad for that.
yeah, this integer's "feature" is extremally confusing anyone comming from other languages, e.g. C, C++ or Java. Unfortunately, this leads to lots of bugs. Need to add "prohibittion rule" for static code analyzer for the team. Don't want to waste time trying to find such tricky stupid bugs. Thank you for this helpful info.
With your custom 'iadd' and 'add', does 'a += [1, 2, 3]' become equivalent to 'a + [1, 2, 3]'. I don't get where the assignment back to 'a' is taking place. Also what happens when you call 'append_some_to_list' with s normal list?
a += [1,2,3] is closer to a = a + [1,2,3] with the custom iadd and add. Python always does an additional assignment when you use +=, even if it is just reassigning a variable back to itself (which is the most common case). When you do the same example with the builtin list, you get [1,2,3,4,5,6], in contrast to bad list which gives [1,2,3]. Hence the confusion!