Python itertools - The key to mastering iteration

Подписаться 228 тыс.

Просмотров 26 тыс.

50% 1

The key to iteration in Python
A key feature of Python is it's incredible support for lazy iteration. Defining and consuming lazy sequences in Python are easy. You can even define an infinite sequence like powers of 2 or prime numbers! To make working with iterables in Python even easier, Python provides some basic building blocks to help you compose iteration primitives in different ways, which is what the itertools module is for. This module comes with 21 different primitives (and many more recipes for how to combine them), which we go over in this video. Will you use them?
― mCoding with James Murphy (mcoding.io)
Docs: docs.python.org/3/library/ite...
Source code: github.com/mCodingLLC/VideosS...
SUPPORT ME ⭐
---------------------------------------------------
Sign up on Patreon to get your donor role and early access to videos!
/ mcoding
Feeling generous but don't have a Patreon? Donate via PayPal! (No sign up needed.)
www.paypal.com/donate/?hosted...
Want to donate crypto? Check out the rest of my supported donations on my website!
mcoding.io/donate
Top patrons and donors: Laura M, Jameson, Dragos C, Vahnekie, Neel R, Matt R, Johan A, Casey G, Mark M, Mutual Information, Pi
BE ACTIVE IN MY COMMUNITY 😄
---------------------------------------------------
Discord: / discord
Github: github.com/mCodingLLC/
Reddit: / mcoding
Facebook: / james.mcoding
CHAPTERS
---------------------------------------------------
0:00 Intro
1:47 An initial warning
2:30 Lookahead motivating multi-accumulate
3:09 ALL 21 ITERTOOLS - itertools.count
3:41 itertools.cycle
4:01 itertools.repeat
4:17 itertools.accumulate
4:58 itertools.batched
5:28 itertools.chain
5:58 itertools.chain.from_iterable
6:18 itertools.compress
6:42 itertools.dropwhile
7:32 itertools.filterfalse
8:18 itertools.groupby
10:08 itertools.islice
11:04 itertools.pairwise
11:47 itertools.starmap
12:43 itertools.takewhile
13:02 itertools.tee
14:22 itertools.zip_longest
14:48 itertools.product
15:43 itertools.permutations
16:15 itertools.combinations
16:55 itertools.combinations_with_replacement
17:22 multi_accumulate example
19:43 Thanks

Наука

Опубликовано:

30 июн 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 102

@mCoding 4 дня назад

Errata: 0:00 filter(x) isn't valid; to filter out falsy values, use filter(None, x) 2:33 & 17:22: the example call in multi_accumulate's docstrings yields an additional (1, 1) at the beginning 4:40: the min and max should be arguments to itertools.accumulate, not list

@oida10000 3 дня назад

This first error might be the reason for filter_false. But overall I find a nice synatax sugar, might look better then ! or not . Will there be a follow up with more_itertools?

@QuantumHistorian 5 дней назад

The combinatorics functions are pretty damn useful IMO. Sure, it might "just" be for maths stuff, but the range of computational maths problems that involves them is vast! It's essentially the answer to the question "what are all the ways to put these inputs together?" which is super generic.

@dcmayo 5 дней назад

I use the combinatorial ones a lot, often for testing things. I use all four of them. I have 5 versions of networking device firmware, and I want to make sure they're all compatible with each other, so I iterate over all the pairs of combinations_with_replacement. I have 10 devices networked together, and I want to test throughput for all possible paths. That's combinations, or permutations if I want to test both directions. I have 5 versions of firmware and 3 models of devices, and I want to make sure all firmwares work on all devices, so I iterate over the product of firmwares and models

@scottza 4 дня назад

That's an awesome example use case. Thanks for sharing.

@Liam_The_Great 5 дней назад

"iterable" doesn't sound like a word anymore

@zanus5591 4 дня назад

"irrbl"

@kilianvounckx9904 4 дня назад

It's called semantic satiation. Pretty interesting concept

@crayyy_zee 4 дня назад

@@kilianvounckx9904 TIL

@2raddude 2 дня назад

I don’t even need to watch the video to know this will be accurate 😂😂😂

@agbenfante 3 дня назад

“Try not to get caught up in showing off just how well you know the itertools library” I feel attacked

@314Labs 5 дней назад

at 4:41 I'm assuming "min" and "max" are passed to "accumulate" and not the "list" constructor

@shyohevzion984 4 дня назад

Yep. I noticed that too

@isodoubIet 3 дня назад

Combinations with replacement are very useful for implementing the statistical bootstrap.

@jacanchaplais8083 4 дня назад

I think there's a bug in your multi_accumulate example. You'd want the first argument of itertools.accumulate to be iterator, not iterable, right? For non exhaustable iterables, like range or sequences, you will end up counting the first element twice. Not an issue for min or max, but you'll definitely see an issue if you use the running sum example. iterator will have that initial value removed, though, so using that instead should solve the problem.

@tcbtrvpsrno5985 4 дня назад

Indeed, this is also the case for, the most commonly used, python list also. Changing the "iterable" on line 105 to "iterator" as declared on line 99, would ensure only the rest of the iterating elements are included in the accumulate function (instead of the whole iterable from index 0). Anyway thanks mCoding! This is an insightful video!

@guidodinello1369 5 дней назад

Ive found those functions useful while doing simple grid searching where you test combinations of hyperparameters to tune ml models.

@sebastianjost 4 дня назад

Same here.

@jevandezande 5 дней назад

filterfalse makes sense when you have a function defined elsewhere that you are using (e.g. filter(my_func, it)). To reverse the conditional would require writing a new function or adding a lambda (e.g. filterfalse(lambda x: not my_func(x), it)), which is ugly.

@syrupthesaiyanturtle 5 дней назад

the lambda isn't ugly

@IceArdor 4 дня назад

⁠@@syrupthesaiyanturtleit's an extra function call and requires giving a name to the temporary arguments. Python's functools doesn't have a nice compose function, otherwise we'd have: filter(compose(operator.not, my_func), iterable) But none of that matters because a good portion of the Python itertools library relies on filterfalse being defined, so why not make that function public if they had to implement it anyways.

@syrupthesaiyanturtle 4 дня назад

@@IceArdor why not just add a parameter to the filter function instead of creating a new one entirely?

@crayyy_zee 4 дня назад

@@syrupthesaiyanturtlea new parameter, presumably something called "presume_false" would be uglier

@isodoubIet 3 дня назад

Defining filterfalse is also ugly so that's not really a great argument

@traal 4 дня назад

I use batched all the time at work, for sysadmin type stuff, or API queries that slow down when you give too many search terms. I’ve used the combinatoric functions to solve programming challenges, e.g. Advent of Code

@fabiolean 4 дня назад

I was part of a project that did analysis on proposed firewall rules. Since the rules could be subnet-to-subnet we used the combinatoric functions to ensure we analyzed every possible unique combination of source and destination addresses in the proposed rules.

@Jakub1989YTb 3 дня назад

I was like: pff, itertools. I know those, what new can I learn. But that last example... very clever :-) yet still just applying the basics. Nice.

@_Dearex_ 3 дня назад

thanks for the pairwise tip!

@blitzarsun 4 дня назад

I was hoping you would do itertools. Please do functools too!

@yuvaldolev7969 3 дня назад

Great vid. Is functools next?

@Jagi125 2 дня назад

I've used permutations and combinations in genomics. Probably pretty useful for big data in general.

@replicaacliper 5 дней назад

been waiting on this one

@klmcwhirter 4 дня назад

I disagree "slightly" with your comment about using a for-loop instead of chaining the utility functions. First, always test the performance of the algorithm which you are conceiving. In general, the built-in utilities (especially the ones implemented in C) almost always perform better than something written in Python. But, performance in Python is not always intuitive - especially when chaining things together where boxing / un-boxing occurs. Test! I definitely agree with your statement about "showing off". I usually frame my comment about this as a maintainability problem. Think about the poor soul who will have to troubleshoot / enhance that part of the code 3 years from now. And if I wrote the code, inevitably that poor soul will be me. My rule of thumb is ... if you have to write more lines of comments to explain what is going on than the actual lines of code themselves - something is wrong. Thanks for another great video!

@unusedTV 4 дня назад

Combinatorics are guaranteed to be useful at least once a year - Advent of Code.

@traal 4 дня назад

I came here to say this. 😊

@DrDeuteron 5 дней назад

filterfalse is necessary b/c sometimes your predicate is "bool", and lambda x: not x is blech.

@andreismolensky1507 4 дня назад

You can use operator.not instead.

@luketurner314 5 дней назад

The combinatoric ones are useful for Sudoku variant helper tools

@hpewd День назад

Combinatorial once used here in numerical simulation when sum/product sequences over sets occur. My data is large enough so I use combinatorial generators that create the sets.

@FedericoSpada13 4 дня назад

14:34 I've used it to print the content of a list inside a tkinter table: I zip_longest the list and the rows already present in the table; if the row item is None (not enough rows), I add a new row, if the list item is None (too many rows), I delete the row; finally if both are present, I update the row with the list item. I've found out that this is faster than always deleting all rows and then re-add them: it's better to update the existing ones.

@atrus3823 4 дня назад

My rule of thumb for filter and map is use filter and map, unless I want to define my own function. I always find comprehensions nicer than lambdas. For example, map(str, [1, 2, 3]) to be me is nicer and clearer than (str(a) for a in [1, 2, 3]), but (a*2 for a in [1, 2, 3]) is nicer than map(lambda a: a*2, [1, 2, 3]). Same for filter: filter(str.isupper, ['a', 'B', 'c']) is nicer than (a.isupper() for a in ['a', 'B', 'c']), but (a for a in [1, 2, 3] if a == 2) is nicer than filter(lambda a: a == 2, [1, 2, 3]).

@jesavius 5 дней назад

Nice! But I want MORE! I SAID MORE!!! As in more-itertools, of course.☺

@johnnyq4260 5 дней назад

If you could make this comment recursive it'd be really cool.

@IceArdor 4 дня назад

The itertools recipes are available in the more-itertools package. Rather than copy-pasting the recipes, I just pip install more-itertools and I'm done.

@BrennenRaimer 4 дня назад

I don't understand why the docs give the recipes for more-itertools and even link its docs, but then does not just include them in the standard library itertools? It's so silly!

@BenGroebe 5 дней назад

I used zip_longest recently. I needed to vertically display two lists side by side in a GUI, and there was actually very little chance they'd be the same length. zip_longest with fill="", then ' '.join

@LasradoRohan 4 дня назад

Awesome video as usual. Just one question. At 18:21 wouldn't you have to skip one from the iterable when passing it to accumulate? If so, not doing so will result in functions like sum adding the same element twice.

@joshix833 4 дня назад

I've created a typed_stream library to use the most important lazy functions from itertools as methods on a Stream class. That's far more readable imho than the functions

@aflous 4 дня назад

Link

@joshix833 4 дня назад

@@aflous it's on pypi and on github on my account Joshix-1 with the same name (yt often deletes links)

@joshix833 День назад

It's typed_stream on pypi

@atrus3823 4 дня назад

I prefer a combination of the two dot product functions. The starmap and operator are what make the first version clunky, but the rest is OK, and with a generator comprehension it can be made really nice: sum(x * y for x, y in zip(u, v, strict=True))

@Talon_24 5 дней назад

5:05 There's now batched and 11:24 pairwise in the standard itertools? That's fantastic, i had to write these so many times🎉 14:40 I feel i'm using zip_longest less often than zip, but still regularily

@atrus3823 4 дня назад

Another advantage of product is that it can be a step in an iterator stream, whereas nested for loops can't be.

@marcinpohl3264 4 дня назад

Could you give an example of compress vs filter where one of them offers a clear benefit (readability, performance, memory usage, anything)? They seem so close they're almost interchangeable (are they?)

@zeropointer125 5 дней назад

I would have used those combinatorics functions (and even zip longest) for advent of code challenges. Probably not any real world code

@mvsh 3 дня назад

When your C++ background doesn't let you spell "in" without a "t" 0:04

@alex_lanes 3 дня назад

I use tee to duplicate a generator. Useful for counting how many lines my Cursor SQLite returned without consuming it

@anon_y_mousse 4 дня назад

I think the reason they have functions like filterfalse is because they want to provide a way to negate a filter function without wrapping it in a lambda in cases where you already have a function to use for filtering that you can't edit. Things like standard functions or something from a library would fit. It is annoying that it's named weirdly, though. I think something like filternot would make just as much sense and be two letters easier to type. I cloned the repository for Python and was playing with 3.11, but still haven't installed it. Meanwhile, my distro package is only at 3.9 and I'm leery of breaking things by upgrading it. I may end up dual installing it at some point so I can continually upgrade without worrying about breaking the system packages, but I don't use newer features like that for anything more than tests. Oh well, at least some of these will be inspiration for features I add to my own language and it makes me feel better about not having released it yet.

@eliavrad2845 3 дня назад

18:40 last_element seems useful. Honestly I wished there were a built-in "first" and "last" functions: there are so many times I'm using Jupyter or a console and I just want to check the structure of a random iterable (returned list, dict keys, etc), and a "next(iter())" always feels very clunky and not intuitive.

@Aramizyera23 3 дня назад

starmap & zip use in the example in the title is redundant: map can take multiple iterables. So it could be sum(map(mul, x, y))

@MyrLin8 3 дня назад

the 'better use case' you seem to miss is 'security' ... aka: encryption.

@mCoding 3 дня назад

Could you elaborate? I'm not familiar with any encryption schemes that iterate over permutations or combinations.

@thegoose6900 4 дня назад

Looking at your thumbnail it seems that just putting return x[0]*y[0] + x[1]*y[1] is a lot more readable + shorted, probably even faster because it doesnt have to look up for all those functionw you use

@resatcanerbas3541 4 дня назад

Some of them are quite useful, but as a developer, I need to break my habits first to remember the use case and instead of loops go to the itertools.

@theViceth 3 дня назад

for any benchmark testing, permutations is the best thing available. But saying "I use it a lot" when i literally wrote a single script doesn't feel right. Even if I use it almost daily

@mCoding 3 дня назад

A single script that you use every day sounds like an awesome script!

@jasonhenson7948 4 дня назад

I always find it hard to see how some more niche recipes/functions could be used without knowing the problem first. Regarding zip_longest: it feels like I might want to know if something happened, but not always care what happened.

@unperrier5998 4 дня назад

You could have mentionned that multi_accumulate() and multi_reduce() are monadic and what monad they implement.

@mCoding 4 дня назад

Hmmm true, but would throwing in the definition and explanation of a monad in an already 20 minute video be a good thing or a bad thing?

@unperrier5998 3 дня назад

@@mCoding monads are meant for pure FP anyway, because in FP they don't have blocks and in particular exceptions blocks. They don't really have a place in python. That said it would be an interesting topic for a future video if you'd like to attempt it, because it's not an easy topic to explain clearly. Maybe a series on explainind the common monads and it could be complemented with OO design patterns.

@kventinho 4 дня назад

hi, newbie programmer here. Why do you keep using 'assert' in your functions? What's the point of this keyword in this context?

@NotAUtubeCeleb 5 дней назад

Itertools group by is reminiscent of how old fashion Hadoop works with parallel key, value operations

@mmilerngruppe 4 дня назад

12:21 there is no better place for lambda as that. add = lambda pair: pair[0] + pair[1]

@mCoding 4 дня назад

On the contrary, giving a name to a lambda is considered by some (linters) to be a class 3 felony.

@mmilerngruppe 3 дня назад

@@mCoding okay okay okay, then just rewriting add function would be enough

@MLGJuggernautgaming 4 дня назад

In the thumbnail you’re missing a )

@chair547 5 дней назад

When she iterate on my object till I throw

@QuantumHistorian 5 дней назад

How come all the comments are porn bots? And who's this Scott they're all referring to?

@avasam06 5 дней назад

That's just modern day RU-vid. Dislike and report.

@trustytrojan 5 дней назад

luckily they all got deleted, or maybe youtube is filtering them from my view

@johnnyq4260 5 дней назад

I don't see them. Maybe RU-vid is showing them only to frequent visitors to porn sites?😂

@felixfourcolor 5 дней назад

You would see them if you're early. I reported all of them, and I suppose many other early viewers did, so I guess YT has removed them.

@squishy-tomato 5 дней назад

Yeah - stupid bots responses are only removed after many users report them. A technical response I take 3 minutes writing gets banned automatically within seconds. I think I should submit my CV to youtube, because it doesn't seem like they know what they're doing.

@hanabimock5193 4 дня назад

Python is so Slow

@BlackDroid003 4 дня назад

If you need a list with a given length (eg for some sort of buffer/storage), instead of list(itertools.repeat("X", 4)) You could also use ["X"] * 4 Not sure if there are any notable performance differences, but you need one less import, and its shorter

@IceArdor 4 дня назад

Everything in itertools was written to be lazy because iterables may be have infinite items or may have a finite number of items that exceeds the amount of RAM. repeat("X", 1_000_000_000) versus ["X"] * 1_000_000_000 Additionally, you may not know ahead of time how many times you need to repeat that element, and iterators don't specify a length (and shouldn't be eagerly consumed). This starts to matter once you've adopted fully lazy iterators in your code: excel_column_A_cells = zip(repeat("A"), count(1)) yields A1, A2, A3, ... You couldn't implement this with an explicit list.

@anon_y_mousse 4 дня назад

@@IceArdor Technically it's a generator for tuples `('A',1),('A',2),...`, but it is funny that using lazy evaluation requires writing more code.