My day job is to write these cst visitors and transformers for our static analysis platform, and upgrading our infrastructure to python 3.10 was an absolute game changer for me.
@@pranavnyavanandi9710 to add to what Tushar is saying: static analysis is basically inferring facts about a program without running it, by looking at the source code or sometimes binary (analyzing the instructions) This is in contrast to dynamic analysis / run-time analysis, which looks at the characteristics of a program while it is being run. (Analyzing the behavior)
@Jamie Walkerdine functional tools? Python has the biggest repository of plugins, modules, etc. You can literally do anything with python, from making a full stack website to a neural network, pattern recognition, data science, ETL, etc. Perhaps you mean functional tools as in functional programming, in which case I dont understand your point at all since structural pattern matching is a feature (one of many) of functional programming.
@Jamie Walkerdine Pattern matching is very useful. In fact, it's pretty much the only way to differentiate between different variants of a sum type. Of course, python doesn't have sum types, so... it doesn't add super much. It does mean you can more easily deal with deeply nested structures, but let's be honest - that is pretty rare.
For a semester at my Uni, We have been thought a functional programming language called OCaml. I cannot express how much I liked the concept of structural pattern matching in OCaml back then. Being able to make a recursive call for the rest of a list by using pattern matching was really satisfying.
I watched this ages ago, thought it was neat, but didn't have any use for it. Now I'm working on a large and complex codebase and when I realized I needed to be able to find all the instances of a specific structural pattern, I came back here for a refresher. This is great; exactly what I need to solve my problem!
@@mCoding Lol new one coming tomorrow in fact. I've been taking time away from writing/shoot/animating to read about Reinforcement Learning.. and why it doesn't work well (yet) in production. Not rushing this one. Would like to make a series that's useful and not totally redundant with what's out there.
The other thing about the match statement that's useful is that it allows you to both check the structure of something, and pull values out of it. In the jumble of if statements, there's a couple places where you check something, then pull a value out of the structure, then check against that, and so on. It's useful to match against the structure, and be able to pull out a value and use it straight away
That's true, like checking if (x == 1) and pulling the answer from the statement like true or false You can also use if (is_true:= (x == 1)) and use the new variable to check against other conditions Probably not the most readable code.
The performance benefit of switch isn't the only reason it's valuable. In my opinion, the far more important part is that it promotes DRY. You aren't writing "x ==" a zillion times, and there's only one place to change if you need to change the value being switched on. This, I think, is a very Pythonic benefit, and one that's worth the additional level of indentation.
@@calyodelphi124 100% that. It has the exact benefit of color coding. You know exactly what you're dealing with the moment you lay your eyes on one of those colors. I have no idea why people have been gaslighted into believing that switch is bad. It sounds very Apple-y "you don't need that feature" to me.
Thank you for making a video on this, I've had to explain the whole match vs switch cases countless times in discord help servers. Appreciate showing a great example with csts as well, anyone who has written an interpreter/compiler before knows how valuable structural pattern matching is.
Hello. I'm a novice programmer, I know basics of C and am decent in Python. That said, can you tell me what really is the core difference between a compiled vs. an interpreted language? Because, in a compiled language, the code directly gets converted into machine readable code - which is then executed by the processor **line by line**. In interpreted languages like python, the source code is first converted into something called "bytecode" at once - that is, _compiled_ into bytecode, after which, the bytecode is executed again *line by line*. There really seems to be not much difference. Except that, the conversion of code to bytecode in case of interpreted languages makes the code portable, since the 'Virtual Machine' as I have heard, then converts the bytecode to machine code specific to that machine. Is it that the bytecode is converted line by line into machine code and then immediately executed before moving on to the next line of bytecode, is that the difference? Thanks.
A big difference is that python bytecode is not executed directly by the processor. A Python source file is just a high level way of generating instructions for another program: the python interpreter. And a C source file is just a high level way of generating instructions for the CPU. If you create a C source file or change a C program's source code, you need to recompile it (that is, you need to (re)generate the binary): nobody is going to do that for you. On the other hand, when you create or change a Python file, you don't need to compile anything (in the traditional sense of the term) because the thing that "talks" to the CPU (this is, the python interpreter) has not changed at all. c source -> ... -> instructions for the CPU python source -> ... -> instructions for the Python interpreter -> ... -> instructions for the CPU I think this is the key to your doubt: the python interpreter DOES NOT convert bytecode into machine readable code. Instead, this bytecode contains instructions for the virtual machine. The virtual machine just follows those instructions, executes them (side note: a line of python code does not translate directly intro one instruction for the virtual machine, so there is no such thing as a *line by line* interpretation). Try to think an extremely simple example. A C program that enters a while-true loop and prompts the user for one of 2 characters: d or t. If the user enters "t", the program prints the time on screen. If the user enters "d" the program prints the date on screen. Now the next step is to introduce a small modification on your program in such a way that instead of waiting for the user to input anything, it has the ability to process files with strings of "d"s and "t"s. Your program has become a (rather useless...) interpreter for a new kind of (rather limited...) programming language. Eventually, you can make your C program a little bit more useful by having it accept things like "1000000 t" and not force the user to actually type a million times the letter "t" on a file. Of course, your program gains complexity, it has to parse the file and discover its structure. A nice performance boost: once your program figures out what to do with a line like "100 t" in a file, it would be nice to save it for future use if the input file does not change (think `__pycache__` and bytecode....). In some way, you start "compiling" these files into simpler strings of "d"s and "t"s. But do not forget: these "d"s and "t"s are far far far away from being instructions the CPU can understand. hope this helps! it's a difficult thing to discuss in a youtube comment thread :D
Honestly, I really appreciate the sarcasm at the beginning. Having been in a position of having to explain new features to users, I *really* understand the pain of having someone completely misunderstand, or worse still is to misuse, a new feature set. I also really appreciate the dictionary example. I've used that pattern in multiple languages for abstracting a branch of code that hoists functions from another scope into the main operation of an application. Lastly, while I understood what pattern matching meant, it was really nice to see a concrete example in Python. In C#, I used to love pattern-matching for If-Else chains that relied on variable declarations, since C# allows the pattern match to initialize a new variable of that static type. Python had no use for such a concept, because of dynamic typing, but the obvious benefit is that it is an extremely expressive statement that is (relatively) easy to read. Always great stuff here. Keep up the great work.
cst is a pretty niche application, matching JSON, YAML or XML trees is probably a more common use case however the issue I see with this approach compared to the if-chain is that you don't get specific errors, you only get a "match or not" and that will make debugging harder, you'll have to look for the problem in the whole structure. I think it's better to use a third party parser and validator if possible.
Thank you for the high quality educational content ! about the switch_dict_example() method at 1:50 , line 10, it seems to me that the built in "get" function's default argument doesn't have the name "default" and so we should write : do_next = _switch_dict.get(x, default) instead of : do_next = _switch_dict.get(x, default= default) the default parameter should be put positionally otherwise we get the following error : "get() takes no keyword arguments"
Thanks for catching this! I even checked the docs, but in the docs the parameter name is "default" and it is not mentioned that the arguments are positional only, oops!
Yeah, it's really annoying that many built-in functions take positional-only arguments, even when users might reasonably want to explicitly use a keyword. Someone should write a PEP.
I actually like the match statement as a switch statement better than elif's Yes, it is one more layer of indentation. But it syntactically separates the thing to be tested from the things to be tested against. It can be hard to tell at a glance which value each elif is testing against. (a simple '==' isn't too bad so long as the number of clauses is low, but when a block has multiple matching values, it becomes kind of ugly) With match, it pulls the value tested against separately, make it much clearer at a glance what values do what.
I guess time will tell if people use it this way. I will be happy to change my mind if the community agrees it is a better switch. Apparently some of those integer-only optimizations might actually be on the way, that would be cool!
You always amaze me: so much content, well explained, in such a short video! About the "jump table" dictionary: of course, you are not limited to integer keys, and I would claim that it does make more sense with strings. For when you want to be able to add functions at runtime, say, for a kind of plugin system.
Where I am usually running into use cases for match/case is parsing file formats, where cases are along the lines of "contains three tokens: two keywords and one value". E.g. a proprietary format where a line might be "add key user.name". Such things might make for simpler examples, and demonstrate a case of (a) code getting both clearer AND shorter and (b) extracting values from the matched structure.
Wile I totally understand the value of using the switch statement, I think the imperative if alternative could have benefited from some early returns, not using the arrow anti pattern. Great video otherwise!
Good point! I actually considered whether to also compare against the early-return version that would be less indented, but I decided not to since it would require additional refactoring. In my opinion, the early return one is still less readable than the match, but let me know if you think otherwise!
You almost make it sound like the match statement is for parsing syntax trees specifically, which isn't true. It's a good example, but there are many many more situations where pattern matching is useful.
dont get me wrong, i like my lecturer, but i can easily learn more from this/youtube videos than sitting at a desk in a lecture with white lights making me tired. my lecturer and me talked over match case but you gave me more clarity on the topic thanks.
2:35 On the other hand, adding a lookup for callbacks rather than enumerating them in code is sometimes a good way to achieve some forms of abstraction / extensibility that can't be achieved by simply listing out each case (same reason one would use a dictionary rather than individual variables)
@@iMaxBlazer Is using a dictionary of functions instead of huge elif statement really such a bad practice? It always seemed more readable and much more editable to me.
hard same, I pretty much always do this when possible. python is multi-paradigm, the fact that functions are first-class objects and we can easily use elegant FP patterns like this is awesome.
@@iMaxBlazer were we in a language that makes use of code locality (cache misses and branch prediction misses can be avoided on typical use cases), it could be a huge hit to performance, so only use it when you explicitly need it as part of a library if performance is a concern (just saying be aware of its potential pitfalls)
Thanks for this very insightful video. Not only the match case is a bunch of if statements in disassembly, but it's also a very unoptimized one. It is bound to be the slowest of all three cases.
Thanks for the video! In truth, adding case to replace the long and less readable chain of 'if' would make a lot of sense to me, more specifically from a performance point of view (even if that's not optimized now, it leaves the possibility for later, it would be much harder with the 'if' pattern). But yeah, it's obvious that they've gone way further and implemented Rust-like patterns. It really improves the clarity of code.
Imo structural pattern matching is most useful in functional programming. This, in conjunction with recursive functions is what makes languages like Scala so interesting. It is then so much easier to parallelize and thus we get tools such as Kafka, Spark, etc.
I'd be careful not to confuse functional languages with languages with algebraic data types. While there's a huge amount of overlap between the two, it's the presence of ADTs where having pattern matching shines. Languages like Erlang that lack ADTs use tagged tuples to similar effect, so the end result is the same.
The point of this feature isn't performance, but readability and boilerplate elimination. Python 3.11 gives you a speed boost of ~40%, OTOH. So no, not Python in a nutshell. Python's speed is held back by its C API, and that's hard to fix without breaking stuff.
There's also my favourite aspect of match: exhaustive Enum checks. If you're not using Enums instead of strings (you know what I'm talking about), you should, and you get exhaustiveness checks for free by using match instead of if/else.
As someone without a formal CS background, I am quite lost as to the utility of the match statement, what the cst is, or why this code is more or less readable than the other. Yet still, I watched.
The cst and ast are both objects that help with working with program source code. Without them working with program source code is hard. Think about how hard it would be to find all the places x is assigned to in a string of python code. You could search through the string for 'x=' but then that would find "yx=2" as well, or even " 'x=2' " which is not an assignment but a string expression. If you have the cst or ast you wouldn't run into these problems. Both the cst and ast are tree representations of the source code. They are trees because of the nested nature of source code. Like you can have a number in a function call in an expression in a variable assignment. Ex: x = 3 * f(2) The cst and ast both make it easy to find the assignment to x. All you have to do is look through all the nodes and see if they are an assignment node and if so see if they are an assignment to the variable x. the ast for the above code would look kinda like assignment | expression | multiplication | \ number:3 function call:f | number: 2 The difference between the cst and ast is that the ast just cares about what was written while the cst also cares about how it was written. Things like the extra whitespace in "x= 2" would be ignored by the ast but included in the cst. This is why in the video he needed to use the cst, because what he was looking for was not just any tuple but a tuple written in a way signaling it was likely meant to be something different.
Honestly it's great that python added in their language one of the best feature of rust. Although what makes it so powerful in rust is the strong type system, which python lacks, so idk how it compare to the rust match
We can hope! I honestly don't see why not in some specific cases except that Python doesn't usually place much emphasis on changing things for performance reasons. Would love to be proven wrong in 3.11 or 3.12!
Honestly, I think the `if` version is clearer and easier to communicate, and should be the default option for those kinds of problems. In the `if` version, what is being checked is almost self-evident. For the structural matching, you'll end up second-guessing yourself or, in your example, drill down the structure so much that the tree becomes unwieldly and hard to read. If only Python had a terser syntax for structures, this problem could be alleviated (example: JSONPath, etc.). Also, with an `if`, you can add complex logic that depends on multiple criteria (sometimes including non-structural ones), which in the `match` version is something that (AFAICT) cannot be done (by the nature of such functionality), so why bother at all? The way I see it, `match` is mostly useful for very simple cases, or for value extraction.
The feature most languages lack is sum types primarily. Then pattern matching makes the most sense. I can't look at "match" in python differently than just as a flashy new syntax sugar that will make some new idioms emerge.
If I can pattern match on a type, I can tilt my head a little to the side (because Python's dynamically typed) and see type constructors for sum types. If Python were statically typed, the lack of algebraic types would be a real problem. Go, for instance, would suck so much less if it had sum types: imagine how much less clumsy its error handling alone would be...
i donno how yu were able to understand this complex, but u r a true gem or anyone who understand this, altough ,as per the understanding part i think i got your point but , i know deep inside i cannot implicate this like you to show up to anyone, gr8 hatsoff
Depending on how you are using dictionary switch statements i don't agree that it's less readable. I think especially in the polymorphic like case it's much more readable, and easier to extend
2:00 regarding the use of a dictionary as a switch case, I think that when properly written, it can be more readable than a big if/elif/.../else block. You need the function names to be very declarative and define the dictionary inside the function. This would of course not be done for speed, since defining the dict inside the function is dumb for anything that needs to be called somehwat often, but I think that for an ammount of cases that is not too big nor too small, the following is really readable: def switch(x): switch_dict = { 0: do_0, 1: do_1, 2: do_2, 3: do_3, } switch_dict.get(x, default)() as long as the cases are not just integers, but explicit things, I think that my_condition: do_somehting is nicer to read than elif x==my_condition: do_something().
you can avoid re-creating the dict by assigning it as a member of the function directly after writing the function, so it still 'belongs' to the function def switch(x): return switch.dict.get(x, default)() switch.dict = { 0: do_0, 1: do_1, 2: do_2, 3: do_3, } also added a tail call return in case python ever optimises tail calls also now that it's a publicly accessible member of switch it could still be potentially extended by other parts of the code without having to replace the original function or mess with its dunders
I spend a lot of time writing Python code and I would say that I have a pretty good understanding of python < 3.10, but this just left me with a million unanswered questions: 1) Where did you define _? it looks to me like you are creating an instance of cst.Tuple, passing it a list as keyword argument "elements", and the list contains a variable "_" as the first two elements, unpacked and not unpacked. But you don't define "_" anywhere. After all this is not a function definition right?!. 2) How does the comparison work? does the cst.Tuple instance that you create take care of the comparison between "node" and the cst.Tuple(...) structure you created? or is this also some new feature in Python 3.10?. 3) How can I create my own classes that support this type of structure matching? This is a great example of a very neat new feature, and I appreciate the video a lot, but I would really love a much more detailed video that really does a "deep-dive" into the mechanics that make all of this work under the hood.
Great questions! I made a video about the intricacies of the match statement that I think would clear things up for you ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE--gf4z3g71Mg.html
@@mCoding If anything, add a small break when going to the next example/paragraph, it will help digest the information. But really, that's what the pause button is for. I find your videos challenging enough that I have to go back more than a few times as the topics are not always easy to grasp. Talking slower probably wouldn't help as you'd need to artificially achieve this, it's best to do what feels natural.
That's because most people are just craftsmen, amateurs who picked up their knowledge through playing around and so their videos are easy and reflect the shallowness of their knowledge. mCoding is obviously a university-educated individual who actually remembers everything he learned, and so has true, properly deep knowledge in CS and math. Therefore, you have to actually think about what he's saying a lot of the time, especially if you're not super familiar with it. There is nothing wrong with this, and nothing to fix. In my opinion, his videos have the best possible balance between rigor and accessibility, given the subject matter. You're just not used to rigor.
@@incremental_failure Don't listen to the foxes about their sour grapes. You basically cannot learn these topics with proper rigor without education unless you're a super self-motivated prodigy.
Good video, but I wish it had also tested against basic data types like nested dicts. From what I understand _ is only use able inside the case expression. But if you're not familiar with the node parser api, it can see like mayne it has some other comparison method, and that under the hood its just doing == and the comparison is done inside the node objects.
The proposal lists C#, Elixir, Erlang, F#, Grace, Haskell, Mathematica, OCaml, Ruby, Rust, Scala, Swift, and Thorn as inspirations for the match in Python, so yes!
Hi there I’m new to python and found that your explanations are quite in dept and easy to follow for beginners. I was wondering would you be able to make a short python crash course that covers the basics to get started with thanks
@@mCoding yes, to me a switch-case conveys that you have a specific intent, which if-elif-elif-else doesn't immediately do. I usually use them for things such as factories where I might need to create instances differently for different options, or for parsing text-based commands. It's also much more readable to me.
This might be a silly question, but I can’t seem te find how the words ‘match’ and ‘case’ get a color (aka get detected as a standard python word) in my code. I got python 3.10 and updated all my packages on atom, and if I manually indent and put the case match statement in the code will work. It’s just not detected and subsequently highlighted and auto indented, which really deters me from using it. This might be a stupid question, but I couldn’t find any info online about it and I’m looking at a working example right now. Anybody have a clue?
I'm not familiar with python ecosystem, but that sounds like syntax highlither/parser hasn't been updated yet. It's provided by Atom or one of its packages/addons for python support, not a part of python installation itself.
Yeah this sounds like an issue that would likely be fixed by updating atom or updating your python plugin for atom. It's possible the plugin doesn't support 3.10 yet, but its been a while since it was released so i would hope 3.10 is supported!
The "switch statement" use case was wanted for years. The pattern existed since K&R and it is a shame it took 30 years to finally include it in Python . The switch-case pattern is much much easier to read and maintain than the ugly if - elif - elif - else.
If your code has got this many if-else statements like in the example, you probably did not write good code anyway. There are plenty of ways to overcome this with classes or even with simple dictionaries. Edit: mCoding explains it pretty well at 1:56
I think this pattern matching could be useful for natural language parsing and matching patterns to react on it, also it looks cleaner than if - elif - else. What bothers me, though, is that I can't define every pattern as some kind of object, e.g. foo_pattern = 1 | 'one' | 'I' won't work. if "1" in [1, 'one' , 'I'] does work and I can define a variable with that list inside.
I really appreciate the explanation. Always clear, although this doesn't look like the best real life example. I personally find it easier to read the second way as it explains in a more logic format what its trying to do, despite more code and more error-prone. Just by this video, pattern matching feels like regex, where you always need to explain what it does !!
The problem is you contradict yourself: originally you say that it's not a simple switch statement because that can be represented by a series of if-elifs--but then you show how the ast/cst can also be written by if-elifs. So it doesn't seem that the match statement is solving any issue that hasn't already been solved, but is only trying to make an alternative, possible cleaner way of getting around if-elifs. But using a match statement as a switch is also cleaner than a series of if-elifs--so your entire point is futile!
That the match statement does not add any new functionality to the language is not a contradiction. Python is turing complete, and therefore nothing added to the language does anything that could not already be done. The reason new features are added to the language is to make it easier to do things that could already be done, or to increase performance, readability, etc.
No one even uses switch cases the way they're meant to be used. The fallthrough is there by default for a reason. If you're just going to put break on everything, then just use ifs (the performance "benefit" is negligible for most use cases)
No, it's there because it was an easy way to implement things back in the '70s and enabled some clever coding, but subsequent to that, it's been nothing but an albatross around the neck of most people using switch statements as you rarely want to pass through, it's a pain to explain to beginners, and a common source of bugs. Every code base that I've had to deal with that uses them ends up with comments like "/* pass through */" all over the place to indicate the pass through was intentional. Sure, that's anecdotal, but I wouldn't be surprised if it's common, and that would be a strong indicator that an explicit "passthrough" keyword would've been a better option than making it the default.
I have used a dictionary function dispatch structure in a job where they frequently changed the requirements. It made it simple to quickly alter the code flow using a simple properties file. Yes, that is a horrible use case. But when you have real clients you have to do whatever is necessary to accomodate their foibles.
As someone who cut his teeth on digital humanities, where navigating and matching against complex semantic markup trees was the order of the day, this now makes sense to me.
How would you describe "digital humanities" to a layperson such as myself who has never come across that term before? It sounds quite interesting. Not as a career choice but just out of general curiosity (this is coming from a guy who reads papers on Pubmed [a medical database] because I have random thoughts like "how does capsaicin [the oil in chilies that makes them spicy] work?"). By the way the way capsaicin works is it interacts with a "protein channel" (a big ole hole in nerves that regulates the intake of electrolytes [sodium, potassium, and calcium] by getting wider or narrower) called TRPV1 or "Transient Receptor Potential Vanilloid 1". Capsaicin causes the TRPV1 channel to open up wider, thereby increasing the flow of electrolytes into the nerves which in turn causes the nerves to fire off signals to the brain that makes you experience a warming/burning feeling! ...You didn't really need to know that last part but perhaps it was interesting! By the way I am not a medical expert just a computer nerd with a penchant for eating spicy peppers no matter the amount of water loss in sweating!
Thank you, I haven't had the chance to sit with this new addition yet. I suspect it will be specifically handy when coupled with dataclasses, to provide more powerful pattern matching capabilities, no?
Python 3.10 learner here. In a nutshell you're saying don't use 'switch' instead of 'if elif else'; switch has a use, and this isn't it, correct? Apart from abstract and concrete syntax trees, are there any other good uses for switch?
You can use it as a switch statement if you like, but it's going to be a tad more verbose. This is just trying to make the point that it does much more than that. In my case, it's going to make the code for a whole bunch of finite state machines in my codebase a lot simpler to deal with.
oh yeah right, bots provide comments in that format. The link I provided links to a video which explains why Python's += operator can act counterintuitively
@@delta3244 Thanks, didn't know about that. Also a quick shoutout to mcoding for not greedily using every opportunity to plug his own content (like this one or the hint saying the last video is not a prerequisite)!
@skrt You are welcome. I guess it worked out in this case, but I didn't realize plugging my own content was a bad thing though! I would assume if you liked this video you would like my others too!
Your example is interesting, but where CST is using? I never heard about it in the first place, not that anyone write code in string to do something with it later.
If you've ever used a code formatter like black or the one built into your IDE, this is exactly what your code formatter needs to do. You can also use it for refactoring, e.g. rename all instances of the name x in this function to y. Think code that modifies other code!
@@mCoding Me either! But I've been making my way through Programming Rust 2nd Edition. Pretty dense but my work keep mentioned rust so I thought I should learn it.
It's up to the compiler to decide if it will use a jump table or not, in both Java and in C. If your cases are compile-time known nearly consecutive integers or enum values, it typically will turn into a jump table.
I still want an optimized switch-case in Python, especially now that I use python on microcontrollers. A lot of low level applications are best implemented using a basic jump table. Using any kind of jump based on function calls can be too costly. Plus I also prefer switch-case over if-else-if in cases like this. switch ( v ): case 0: # comment code break case 1: # comment case 5: # comment case 7: # comment code break Easier to read and especially easier to comment than, else if ( v == 1 || v == 5 || v == 7 ): Not saying we should optimize the match-case, just have a separate switch-case. I wish this new trend of deciding for us how we should use everything, from devices, to applications, to programming languages, would end. Especially from these dictators that seem very narrow minded.
I'm liking Case and Elif better. This Match, reminded me of the collage of voices in the beginning of the video. However, there may be advantages I have not considered... It must be a cleaner way. I am thinking an assinged index with Mathematical Induction for a solution.
