@@RainerDreyer Tospaa Be careful using things like "if not aggregate_list" and "aggregate_list or..." because these will capture all falsey types. You need to think carefully about whether it will be correct to treat all falsey values the same.
It's the only explanation that makes sense so far. I have a feeling most people that tried to explain this one to me don't really know why, they just pass the knowledge on. This is the WHY that very few people actually know.
This would have been a good one but I do want to say how much I hate how Python claimed names that are so dang useful. I'm *always* having to check my use of "type" as a variable name.
I agree that this sometimes can be annoying. A typical getaround, other than picking a different name is adding an underscore at the end. e.g. type_ . Otherwise if you happen to use reserved names too much it might just be that you're not specific enough in the naming
By the way "KeyboardInterrupt Exception is an exception" is not the only problem. Also when bare except is used there are many problems with any other errors which might arise but you cannot recognize them
@@JackofSome - actually that's simply SIGQUIT, you CAN catch it. Er, well, you can in C. sigaction(2) or such can change the behavior. There HAS to be python interfaces to that...
yeah i had no idea myself - that's a nice little hidden speedup. I thought I'd set it up vs list comprehensions as well and you're still talking significant speedup. Method A: in_names_loop | Elapsed time is 0.000108 seconds. Method B: in_names_set_loop | Elapsed time is 0.000027 seconds. Method C: in_names_list_comp | Elapsed time is 0.000089 seconds. Method D: in_names_set_list_comp | Elapsed time is 0.000019 seconds.
@@gokharol Yea true. Its good when you want to iterate through unique set of values . Otherwise I would just use numpy arrays and apply some transformations to that.
Some necessary clarifications: 1) the use of .format in the first example was intentional since I assumed f strings to be less known (and for some reason thought they were introduced in 3.7), looks like I was mistaken in that assumption. 2) Thanks to all that pointed out that the isinstance was unnecessary in the last example. I think I got burned by not using it with numpy types at some point and since have almost exclusively defaulted to it but I can't find the exact reason now... **shrug** 3) Desktop environment is xfce4, window manager is i3-gaps :D 4) *sigh*, yes I accidentally shadowed str in one example. Don't be like me.
I've looked into it but haven't had the chance to try it in detail. Definitely interested but will only switch if there's significant benefit and if I can take my i3 workflow with my without any changes.
TLDR: 1. Use if name == main 2. Don't except: without exception 3. Don't print(exception), cause it only shows message and not traceback 4. Don't loop O(n^2) on lists, use sets. 5. Don't use mutable default arguments in function definitions.
Exactly. It's not about the construction cost of about the cost of checking membership a million times. If that's not the use case then you're not gonna get a noticeable improvement.
Mutable defaults can be useful in some cases, particularly when you want to use a cache like when recursively scraping links from a website without repeating yourself. I think it's for dynamic programming.
Much more interesting issue with python: create a random float list with millions of data. Add single NaN value into the list. Call sorted(). The list returns without any exception, the list appears sorted, but under thorough investigation there are chunks in the list that are not sorted! Don’t expect python documentation to tell you about this behavior. Lesson to be learned: you must always check your float lists for NaNs before calling sorted()!
@@radeksvoboda7629 it should almost always be your priority to replace or remove NaN values from your data, before doing anything else with it. Otherwise, countless problems can arise from working with your dataset, if you don’t keep checking for NaN values every time you use the data.
One more idea about the last one: instead of having 'aggregate_list = []' we can write 'temp = ()', and 'aggregate_list = list(temp)' inside the function. This works because tuples aren't mutable
I learned to avoid using mutable types as default values the hard way. It would be great if mutable types were disallowed by the runtime, but unfortunately, I don't think there's a way to check whether an arbitrary value is mutable.
i love reading through these comments! they're the types of code reviews u just dont get if the team u work on doesn't really care about code quality or you're not working professionally in code yet. i honestly miss seeing semantics debates like these
For those unfamiliar with the broader names of python language constructs, when this guy talks about a "hashtable", he means a python dict. (which is an implementation of the hashtable concept. Python sets are basically dicts without values.)
Very helpful! I'm new to coding and I'm finding that, with these types of videos, if I have to rewatch to understand it, I'm stretching myself more mentally. Keep it up!
I actually found out the first one the hard way and it was a great example to me of how no matter how many tutorials u read, u have to do something wrong to understand why it is wrong. It wasn't game breaking but by doing a small project and making a small error I taught myself an incredibly useful concept and that has been my approach to coding since.
6:00 I usually use the logging module, which has built-in support for including tracebacks in logged messages. You don't even need to define a variable in your except statement. Just add "exc_info=True" to your logging call. docs.python.org/3/library/logging.html#logging.Logger.debug
I've watched this 2 or 3 times now. Each time I learn more because I'm more experienced and this makes more and more sense. Very well done and helpful. Thanks!
I never thought a video like this would ever be useful to me. I've been programming in python for more than a year now and everything said in this video cleared up a lot of questions I never thought I had
Nice video, makes it very easy to understand some common Python pitfalls with this eli5 way of explaining things. One note about data structures - hash sets in Python are called Dictionaries ie. `kv = {"key": "value"}` and also set can be initialized with `mySet = {"seven","8","nine"}` instead of `mySet = set("seven","8","nine")`
I prefer the explicit syntax for sets myself because I'm my experience dicts are more common and if you're quickly perusing code it's easy to mistake a set for a dict. Thanks for watching.
The mutable defaults really gave me a headache a couple of months ago. I couldn't tell my code was failing. I can't remember how I fixed it, but at least now, I can't repeat the same mistake. Thanks Jack👍👍
regarding the trace back 4:40 , if you are using the std logging module, set exc_info=True. This will automatically grab the traceback and you don't need to import a second library
number 5, regarding the aggregate_list argument, sounds like a really dumb mistake on behalf of the python language developers. It's not logical for this behaviour and it just sounds like a quick memory/performance hack that should never have happened.
@@erumabo alright, so that’s a good rationale. However why the SAME object every time? Couldn’t they create mutable objects each invocation and get mutability and avoid this problem at the same time? For a language as dynamic as python, this is kind of a weird thing to do, everything else is dynamic, why not this? See what I mean?
@@chrisalexthomas I agree, its a really weird and unexpected behaviour, I suppose that at the moment of declaring the function a reference to an object gets stored in it so when you call it it's the same, which would be quite dumb and even a safety risk so probably I'm wrong. Still, the fact that it even happens at all is really weird, better to simply tests for None as the video recommends.
The default value is an expression that gets evaluated when the function definition statement is evaluated. The result is remembered by the function as the default argument. Storing the expression itself instead and evaluating it on every function call would be more expensive: An immutable default like (1, 2, 3, 4) or even load_default_tuple() would need to be evaluated every time creating a new object which would be expensive. To make this fast, you would have to define a global name caching the value which is ugly. And even then a global name lookup would be required at every call. The way it is is the most performant in the common case.
@@geraldthaler8880 yes, we already established that, but it’s still broken and it’s still bad behavior. It’s not logical to do this. Like I said in the first comment, it’s just for performance. But I would prioritize correctness over performance any day of the week and this looks like a mistake that should never have happened but now we are stuck with it ¯\_(ツ)_/¯
OMG - The last example was one that I did not know about!! Why that is considered a good language feature just boggles my mind. I know that Python 2 was very wacky in lots of places, but why this persisted into Python 3 is just baffling.
I always forgot which symbol it is kill $ 1 # Nope kill & 1 # still wrong kill ! 1 # wow look at this idiot kill @ 1 # what even is that kill % 1 # okay yep
Actually pretty useful. I don't know if ill ever run into passing mutable variables into functions but now I'm ready for what otherwise could've been a very tough bug to sort out. Do you know why we call them 'bugs'? Fun fact: back before everything was micro and transistors were baseball sized, computers were huge and created tons of heat. This led to computers being housed in very open buildings for air-flow, which in turn gave way to problems with bugs... moths mostly would land and nest in the warmth. They were large enough and conductive enough that they could short transistors together, meaning you literally had to go "find the bug" that was hosing your system :) in modern systems of course due to the enormous change in power consumption and overall component size and also large-scale refrigeration (rather than ceiling fans and screen doors lol) we don't have insect problems anymore, only 'bugs'...and bit flipping solar flares lol
for the last one I usually do : def add_names_to_aggregate_list(names, aggregate_list=None): aggregate_list = [] if aggregate_list is None else aggregate_list Nice api, easy to ready and one lined
I applaud you for pointing out the sets/lists tip. Something I encountered a few years ago in Coldfusion which has greatly sped up many scripts for me. Just to point out for people using other languages as well, in Coldfusion we don't have sets (at least not natively), just arrays & structs (which are essentially just hash tables bts anyway). Structs can be use for much the same purpose. Just do Struct[string] = true or just some default, small value. One thing that I'll do especially if the struct will be used very heavily and if the list of strings is managed by myself or a person, is to store all the strings initially in an array, where it's a bit more straightforward what you're doing, but then afterwards create a struct using all the values in that array, and then use the resulting struct essentially as a set as described above.
Thanks so much Jack of Some. I can finally read the entire error with traceback instead of just the type of error. Extremely useful for my bug that only happens at night so I can't figure out what is happening by running it.
Hi, thank you for this video. I particularly appreciated the last part with the mutable default. I am far from being a Python expert but when trying sorting a list according to three different ways, using three different functions plus measuring the running time for each, I have noticed that when calling functions 2 and 3, I was passing the already sorted list. I didn't understand it the way you explained but rather that lists were being passed by reference while variables were being passed by value. So I just copied the list before calling each sort function.
This was a very useful and informative video. Sadly I am guilty of all 5 mistakes that were mentioned but I will use the tips in this video to improve my code. Thank you so very much for this video ….this is very very good 🙏🏻🙏🏻🙏🏻
I actually prefer doing like _result = parameter_passed _*_or_*_ some_default_value # This way the value of the right side on the _*_or_*_ will be used if the left side has been evaluated to an empty string, a zero, None or False._
I just saw it also happens on diferent objects of same classs. class Foo: def foo(self, x, y = []): y.append(x) return y a = Foo() b = Foo() a.foo(2) b.foo(3) prints [2,3] Thats interesting. But only works on lists, not on ints for example
For the last mutable list problem, it can also be solved by creating a deep copy of the passed list into another variable (as the first step of the function). This will ensure that the argument that was passed does not get changed in place
One has to be careful with converting lists to sets as lists ALLOW and may WANT duplicates. However, a set will not allow duplicates so you can find if something in the list via a set but can't check whether it appears more than once in the list via a set.
#5 happened to me once... and it was that bug that I spent a lot of time trying to solve (weeks actually), and my solution at the moment was to make sure to pass an empty array as an argument, and not use the default... eventually, I got to a similar solution, but it kind of bugged for a long time that this happened... and I have to admit until now I didn't know this happened to every mutable type! Thanks!
Thank you for this tutorial. In fact, I have the following comment: Problem #4) You mentioned that set is a HashTable. But, the set is the HashSet, and the dictionary is the HashTable or HashMap. Thanks and Regards.
I recently learned about if name = main at work and it could've changed my life if I knew this when I was taking Python courses, thanks for spreading knowledge!!!
8:30 I think you should also talk about the CPU/Memory tradeoff. Sets eat more RAM but less CPU. But since RAM is a lot cheaper than CPU, it is usually the correct choice.
The more pythonic way of dealing with default list parameter is setting that default to None, but in the function write aggregate_list = aggregate_list or []
I would definitely blame the language design for the last one. It feels so unintuitive for arguments to be initialized at the definition time... I wonder why did they implement it this way? Is there a pattern where it is somewhat useful?
That's honestly a pretty fair stance to take. Within the current design why this happens makes perfect sense to me, but I can't for the life of me think that the current design is good.
Python doesn't really have static variables so it is kind of useful in any situation where your typical static variable would be useful. (function objects can have member variables as an alternative implementation, but it's ugly).
@@shanedixon3539 I disagree, because it doesn't generalize enough to be a useful pattern. I'm surprised they didn't fix this when they had the opportunity in Python 3.
@@DanEllis Techically I see no ways to avoid it, because it will break default arguments. For example user could put some function as default argument. But functions are also mutable in a way. But nobody probably will say that this is bad idea because people usually do not mutate functions. And I see no way to distinguish what user wanted exactly. It is possible to check specifically for list/dict/etc. but it is road to hell.
The last one, about the mutable defaults, had to learn it the hard way during a course project. It was in OOP where I assigned each client a list. The list repeated itself for each client, it was as if every client shared the same list. It took me quite a few hours until I found something about it in stackoverflow.
That last one sounds like something that should happen to other languages that automatically pass mutable objects by reference, yet this is the first time I've ever saw of it
It's not quite just about mutability. The default list becomes part of the locals of the function which is different and could be avoided while keeping passing by reference.
I would also note that the final program you sampled will in-place modify any list you pass into it, so you don't even need a return statement if the list was pre-defined. This can be really annoying but at the same time really useful, and i wish there was an easier way to know when or if anything is being changed in-place
It's a life saver some times, but I absolutely hate having to use it. Out of curiosity what brought you to this video? I've had a sudden rush of viewers in the last two hours and my analytics is not being helpful about the source.
The set is keeping a vanilla array under the hood (say, with 1000 slots). The string (or really any input) is put through a hashing function that's designed to produce as unique an answer as possible for different inputs. The hash is then mapped to a number between 0 and 1000, and that becomes the index of the element (there's more details here in how collisions are handled but this is the gist). So it's more memory intensive, and there's more overall computation involved compared to just looking up an index, but the time complexity does not depend very strongly on dataset size
@@JackofSome Ahhhhhhh. I was sort of thinking "unless it knows which specific memory location to check based on the content of the string?", but I failed to make the jump to hashing and using the hash to generate an index. I'm glad big brain people come up with clever tools like that for the rest of us to use, haha. And thanks for replying!
I didn't know about the last point, and it really makes me think that Python creators must have been drunk when making this choice. I can't think of any situation where it is useful to have a default argument implicitly behave like a static variable. I can't think of any other language where this is the case.
Thanks so much for watching. Out of curiosity what brought you to this video? I've had a sudden rush of viewers in the last two hours and my analytics is not being helpful about the source.
I thought this video was rather uninteresting until I saw the last one. This is the type of behavior that creates bugs that you can spend hours fixing them on, because everything looks so logical and normal
There is also one common beginner mistake in python: using inheritance outside of frameworks/metaprogramming tasks. Python makes interfaces and compositions very easy and intuitive, so one should use them instead.
hey I suggest you two way to fix that ugly solution to agg list 1st one: a = a or [] 2nd one (readability explosion): a = a if a is not None else list()
Mutable default parameters plagued me for a long time. It's one of only two places where python's underlying interpreter rules explicitly differed from what I would consider "reasonable" expectations. The other is the unfortunate way that python scoping rules interact with "closers" (python does not have true closures, hence the problem): # Take this (very contrived) sample code def foo(): ls = [] for verb in ["start", "end", "play", "stutter"]: ls.append(lambda x: f"{x} {verb}") for fn in ls: print(fn("I")) Because of the "there can be only two" scoping of python (global and local), the variable which should be stored in the closer (verb), is instead stored in the local scope, and is overwritten on each iteration. Most people, especially coming from any sort of functional programing background would look at this and say "Oh it prints 'I start', 'I end', 'I play', and 'I stutter'". Nope, just prints 'I stutter' four times. It's not something that comes up very often; I doubt more than one percent of python programs ever encounter it, but if you're expecting lexical scoped variables (which is what python *seems* like most of the time), it can really throw you for a loop... It certainly threw me for one when I stumbled upon it. I'd almost go as far as calling both bugs in the language, but there are rational justifications for persistent default parameters and the python scoping rules, and changing either would require fundamental changes to the language. Still it's frustrating for the developer when the language seems to - at first glance - break it's own rules in unexpected ways. Edit (a bunch of times): That's what I get for changing my example half way through writing.
That's a very interesting example. Thanks for sharing. I must say I actually did expect the outcome we got since verb is more like a pointer which is being changed but I do agree that not having an obvious solution is frustrating.
@@JackofSome Yeah, in that example, the problem isn't as pronounced I think, but a real-world example is *a lot* more wordy. Simply put though, let's say you have a constructor that builds a bunch of callback functions and returns them. It builds the callbacks in a loop for code brevity. The caller receives a list of functions, each of which enclosing a different seed (based on the iterator), but because of the closure weirdness, they actually all end up with the same seed, instead of different seeds. Here's a more direct contrast, since you would expect code written in a function to work the same as code written inline. This works as expected: --------------------------------------------------- ls = [] def factory(a): def cb(x): return a + x return cb for i in range(10): ls.append(factory(i)) for fn in ls: fn(5) --------------------------------------------------- But this doesn't --------------------------------------------------- ls = [] for i in range(10): def cb(x): return i + x ls.append(cb) for fn in ls: fn(5) --------------------------------------------------- Which, from an first-blush perspective is like "aren't they the same, except cb is defined in a different function in the first example?" Hence why it broke my brain when I encountered it.
Converting to a set only makes sense if you're going to do multiple membership checks. Building the set has its own performance overhead, so you need to make sure it's worth paying that up front cost.
I would actually argue that "Not logging errors correctly" is wrong for printing exceptions. Instead create a logger and use logger.exception("msg") ... it effectively calls traceback.print_exc() but now you have the option of turning off that debugging.
I hate that python does that thing you described in the fifth example. It also does it if you work with dicts. Ich you want to duplicate a dict, you have to use .copy(). Can you tell me why this is not like this by default? What's the benefit of having multiple dict names pointing at the same dict?
Python by default does not pass things by value but instead by reference (except for some basic primitives). This idea is fundamental to the language design and, in my opinion, gels quite well with the intention of the language. By doing this they also avoid complicating the language by removing a layer of decision making in most situations but of course that has a few unfortunate side effects. For me the biggest benefit is a sort of efficiency this provides. Nothing is ever copied without my explicit say so. I like that.
@@JackofSome Thanks for the reply. Why am I then able to just duplicate a list or a value without the .copy() command? To me this seems a bit inconsistent.
@@xXneorlinXx Also performance, no? The data would need to be cloned before executing a function, which would slow down the execution with huge data structures...? Passing a pointer is cheap.
While the last error is no doubt important to look out for, the example given wasn't the best. You can literally add lists, or use the extend method, instead of needlessly iterating through list elements and adding them individually. This might throw off people and think they need need to make needlessly complicated functions like the one shown.
It's a demonstration done with the idea that the people watching have some reasonable familiarity with the language. Literally no one will look at this and go "gee I better stop using extend from now on".
BTW, to make the example of Mutable Defaults a bit prettier, you could've just said *if not aggregate_list:* instead of *if isinstance(aggregate_list, type(None)):*
