Functions within functions, closures, and variable scopes in Python 

ball

@@trag1czny ball

I'd love to see a series on typing too.

Your first question is answered on stackoverflow /q/2284860

I think the reason you DON'T know about this though, is that if you encounter anything like this in real life i doubt the immediate reaction would be "hmm, interesting". Likely closer to "Hmmm, this needs to be fixed" 😅 But for the sake of demonstration, it certainly is cool to see how the inner workings actually pan out

Fun fact: even noddy old PHP has added “use”-clauses so it can do this kind of thing. Maybe the concept is more useful than you realize?

​@@lawrencedoliveiro9104 Maybe. Can you give me an example where nested functions that shadow variable scope is better than linearising your functions and having distinct variable names? I don't think I've ever seen any, and I struggle to dream one up. Maybe in something like decorators? But even then messing around with scope seems unnecessary...

@@QuantumHistorian Shadowing is incidental, rather than being something you deliberately do. But lexical binding is certainly extremely useful in conjunction with functions as first-class objects. If you look up my HarfPy binding for HarfBuzz, for example, the way I define the wrappers for defining callbacks makes heavy use of closures. I estimated that doing it this way saved me about 200 lines of repetitive code.

@@lawrencedoliveiro9104 Thanks for actually giving an example, I'll look into it!

@@QuantumHistorian That same module also includes a def_struct_class function, which creates, not just a function closure, but a whole class closure.

As extension modules or via ctypes?

@@lawrencedoliveiro9104 I'm really not sure, I think that's something that could be addressed. Some type of overview maybe.

Extension modules are where you write special C code to wrap the functionality of other C code as Python objects. ctypes is a standard Python module which lets you write the wrapper in pure Python. This usually makes for much less code than writing extension modules.

Functions are first-class objects, same as any other variable. You can define a function called F, then bind it to G, and use it through G as you normally would with F. Try creating a function that takes a function as an argument and runs it on the other passed arguments - you'll get the hang of it easily.

The compiler doesnt care, but I did have to turn my linter off for the video so it didn't highlight that section.

This is both wonderful and horrifying.

I mean the first one, they will produce equal lists, throw exceptions in the same situations, create the objects in the same order, etc, as long as you aren't doing crazy introspection stuff. They are not exactly equivalent, and they produce different bytecode.

Comments within comments?

Nice

You have misunderstood. Since x is defined in function f1 it will look for x there, or if it wasn't defined there it will look for global, it will only peel outwards. This has to be the case or you would be afraid calling a function anywhere because it might rename global or local variables. This would be a nightmare. It can happen that code affects variables in outer functions, but that is either when specifying it or when you have something like an array where it is refrencing a memory location. Here is an example of the latter: import numpy as np y = np.ones(3) def f1(): # y+=1 gives UnboundLocalError: local variable 'y' referenced before assignment data = y # Reference to array from global variable y print(y) [1,1,1] def f2(): data2 = data data2 +=1 # Changes the global variable y and nonlocal variable data data2 = 1 # Doesn't change the global variable y data +=1 print(y) # [2,2,2]! f2() print(y) # [3,3,3]!!! f1() print(y) # [3,3,3] ################ Now in f2 it doesn't matter what you call data2, as long at isn't the same as the variable your referencing (you get an error). So no y=y or data=data, but data=y or y=data will reference global variable or the non local variables.

Are you looking for the keyword: nonlocal ?

Did you watch the video? Nonlocal is explained in detail with examples.

__slots__ does save memory. Not much, like 100-ish bytes per instantiated object. But if you create LOTS of those objects, the differences add up, sometimes significantly. That said, if you handle thousands or millions of objects, those objects likely hold immutable datapoints to be processed. Consider not writing your own custom classes, but use NamedTuple or Dataclasses with slots=True

There’s a name for that in mathematics: “λ-calculus”.

So apparently I've seen this video before. And yet I still added it to my watch later playlist and had it there for weeks now

I mean, as with many of the examples in the video, please don't do this 😉 buuuut yes even these examples follow the same rules!

@@mCoding yes, I love to surprise my friends by this little piece of code and then explain them closures and variable lookup rules :)

@zohnannor Thanks for your example! It may be a short example, but I have stared at it for 5 minutes to understand it 😂 For those who cannot understand the 2nd line in short time, here is the explanation (Thanks James for his video explanation): - [lambda: x for x in range(10)] can be slightly rewritten for clarity by [(lambda: x) for x in range(10)]. - The list comprehension can then be rewritten as: def gen(): for x in range(10): yield lambda: x list(gen()) - The list comprehension can further be rewritten as: def gen(): for x in range(10): def inner(): return x yield inner list(gen()) - x in inner() refers to the x in the enclosing for-loop, as decided in compile time, but the value lookup is at runtime. - x is first 0, and then 1, and then ..., and finally 9, because of range(10). - list(gen()) is a list of inner's, each with a x in their closure, pointing to the same x in the enclosing for-loop, but the value has not been looked up. - When the code runs f() for f in list(gen()), it assigns inner to f, and runs inner(). - The value of x is now looked up, and the latest value of x is 9. And this apply to all 9 inner's. - Thus, [9, 9, 9, 9, 9, 9, 9, 9, 9]. I hope I did not miss anything. Thanks again for your example, zohnannor! And thanks to James for his video!

​@@NicolasChanCSY yes, pretty much like that! I also show this code after, as an explanation (notice `id(x)`): >>> [(lambda: id(x))() for x in range(10)] [2117705466064, 2117705466096, 2117705466128, 2117705466160, 2117705466192, 2117705466224, 2117705466256, 2117705466288, 2117705466320, 2117705466352] >>> [f() for f in [lambda: id(x) for x in range(10)]] [2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352, 2117705466352] and then, this: >>> l = [lambda: x for x in range(10)] >>> l[0].__closure__[0] # 0 index >>> l[1].__closure__[0] # 1 index, same address >>> l[0].__closure__[0].cell_contents 9 >>> And also, I think it's a good thing to not only show some confusing, "broken" code as it may seem, but also a way to circumvent the shown behavior, in this case, it's this: >>> [(lambda x=x: x)() for x in range(10)] # notice `x=x`, `x` becomes local variable [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> [(lambda x: x)(x) for x in range(10)] # notice `x` being passed as an argument [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> 🙂

@@zohnannor Excellent follow-up code to show the inner working (pun intended)!

Inner is a function. In python, functions are objects just like ints and strings. Just like I can return my_int_varable I can also return my_function. In this case, the name of my function variable is "inner".

@@mCoding thanks,I got it!

I don't understand why you ever want to use Python for production code whatsoever...

Au contraire, lexical binding is one of the fundamental concepts that every programming language should have. Yes, I have taken major advantage of it in production code. Check out

DBussy, my pure-Python binding for libdbus.

It constructs proxy-interface classes for D-Bus RPC interfaces on the fly, making heavy use of lexical binding and closures to define the proxy methods. If you think you can come up with some simpler way of doing it, by all means post your code as well. Then we can compare.

@@lawrencedoliveiro9104 bro, probably no one told you that it is bad to be proud of writing a spaghetti code.

Function factories.

In Python, functions and classes are first-class objects. That means you can operate with them at run-time just like you can with other types like numbers, strings etc. This idea did not originate with Python, but with certain advanced languages from the early days of computing, like LISP. It was recognized to be very useful, which is why Python is designed with it as a core idea.

@@lawrencedoliveiro9104 i know. I program in python but never needed to use closures for anything. I am sure it is probably powerfull ... but i cannot get my head around it when there are easier more straighforward ways of doing things... at least for me

@@DuarteMolha Function factory for code reuse.

@@lawrencedoliveiro9104 factory pattern? Not sure i undertand. The factory parttern is usualy given as an example of oo programing. If you mean more exotic code than maybe. I have never seen a reason to use a closure. But sure ... I am talking about my own experience. I think the examples on this video do show how it can be a complex beast

@@DuarteMolha Look at my HarfPy module. It makes heavy use of both function factories and class factories.

There is no syntax for this in Python, you would need to pick another variable name besides x 😀

Great question! The most common reason is to define some kind of higher order function. A common example would be decorators like @dataclasses.dataclass or @functools.wraps

Yes.

Sup anas

Im a member of the "I just opened RU-vid, accidently" Gang

This should be a PEP 😅

It's like shooting yourself in the foot by knowing the internals of a language!?

The cell would still be emtpy if False was replaced by any condition that happened to be falsey at runtime. This is a very common bug in practice, though the rest of example 7 is of course just an exercise to test your understanding of the rules.

Off screen I have a main function calling the code with True then with False.