How IEnumerable can kill your performance in C# 

I swear this is one of those things resharper has taught me with it's warnings, I rarely see it now because I know better. Great explanation of multiple enumerations.

I been programming with C# for about 15 years and there are parts about it that still mystify me. Your example of obtaining a count via an IEnumerable reminded me of how I learned on my own a similar situation with your example. In my case I was loading over 100k records. EF was new to me and I couldn't understand why my app was taking a performance hit until I discovered the difference between IEnumerable and IQueryable. From then on it forced me to take into consideration the overall purpose of the program and how to use IEnumerable properly. You are very well versed in the programming language, more than me after working with C# for so long. On a side note, back when I was learning programming in 1991 I asked a senior developer of our mainframe why people are sloppy with their code. He told me that it will only get worse because as computers get faster it will compensate for bad coding practices and the end result will be lazy programmers. I came from learning to program on a mainframe environment where every byte counted. We ran accounts payable and payroll for 300 employees. All of it was done on a 72 megabyte hard drive.

This enumeration style is called co-routine for those who didn't know. You basically have a function on hold that can give you the next element right when you need it 😄 Actually this is a crazy efficient way to represent e.g. endless streams like indices from 1 to n, e.g. for n=int.MaxValue this is 2^31-1 * 4 byte. Your PC would simply explode if you'd call ToList() on it because it's 8GB of data. But a co-routine like Enumerable.Range() could do that with just 2 int variables and 8 byte. It really makes a huge difference as you can keep this little chunk of 8 byte in faster cache levels of your CPU and crank on it like crazy. A ToList() too less or too much can make your program run 2 hours instead of 1ms 😅😅😅

A common recurring problem among programmers is not knowing how the code they're using works. At the very least, they should understand what the API commits to doing. Deferred enumeration of IEnumerable is a great feature in C#, but if you're using any API that exposes an IEnumerable object, you should always assume that you need to enumerate at some point, unless your objective is to merely chain subsequent operations to perform on the object. In fact, if you never actually enumerate the sequence, it will never actually execute, and this is also an easy trap to fall into. So my best advice would be to write your API according to what client code should expect. If you're returning a finite object, rather than returning IEnumerable, you should return IReadOnlyCollection or IReadOnlyList (or any read-only interface). That way, client code knows that enumeration has already been performed. If you return IEnumerable, client code should assume that enumeration will be required, and even the implementation should probably avoid enumerating to a terminal operation. JetBrains.Annotations also has the [NoEnumeration] attribute that you can assign to an IEnumerable method parameter to indicate that your method isn't performing a terminal operation over the parameter.

Great illustration of how/why this happens. Something I can send to my peers that get confused as to why their code is hitting an API twice when running around with IEnumerable or IQueryable.

My personal choice is to return an I…Collection, so that the consumer knows that the “inner” code isn’t deferred. Of course, there are situations where an IEnumerable is better, for instance when implementing repositories. But such repositories are mostly consumed from other application specific services.

Whoa, for the past one year I was getting sometimes warnings "Possible multiple enumerations" and never knew what does it mean :V Thank you!

it really is quite clean, there is even a warning (at least in visual studio) CA1851: Possible multiple enumerations of IEnumerable collection it just requires the developer to read the warning and handle it. (or the senior developers elevate this from warning to a compilation error)

You could also adress how yield return's are dangerous in that the source list can be changed between enumerations, e.g. items removed between the .Count() and the output.

You have no idea how much this helped me today! I was looking at a problem where counting an IEnumerable with zero elements in it resulted in a significant delay and I thought I was going crazy! I had no idea that IEnumerable would be lazily evaluated. Thanks for the help! :)

IEnumerable - The fast-food restaurant of programming

the IEnumerable approach is the only sensible one in some situations, if there are too many items to fit in memory. I find myself using a 'batchBy(int n)' approach: it turns IEnumerable to IEnumerable so that you can work on a smaller list, but if things are too big, you can take them in byte sized chunks. It does mean something like 'count' (or other things that require global knowledge) can only be accumulated and discovered at the end of the list.

Materialization is not good option too. What if you file is 200GB size? Or what if there is pseudo infinite enumerable like network data or reading database cursor? So you can't always cast to list because of memory. So yes, watch you code and do what you understand. yield return is not bad if you know what you are doing.

A key phrase, which is maybe more of a LINQ term, that I was also hoping to hear was "deferred execution". Great topic to be privy to, and great video, Nick!

IReadOnlyCollection

A bigger problem with methods that return IEnumerable is when they take parameters like a Stream or any IDisposable.

I was just giving my developers a lesson on this exact topic last week. I wish I could have just pointed them at this video! Thanks so much!

Awesome video as always! It’s worth noting however that the implementation of GetCustomers using Select behaves subtly differently to yield return. The call to GetCustomers itself is deferred until enumeration when using yield return whereas it’s called only once when using the Select, when it is assigned to the customers variable. Still absolutely a performance issue as the iteration over the lines still happens twice but the file is only loaded into memory once in the Select example.

Great stuff Nick. Your impact on my programming had been tremendous!

Great video! I always feel good about myself when I know exactly what the problem is and what your solution is going to be at the start of the video... It doesn't happen often, but when it does, I give myself a gold star :-) Thanks for posting such good content!

This is something I knew about and have been working to pass on to others as well. Thanks for making this video. I will share this with them in the future!

Oh boy, this one hit hard on me Some time ago I've had to import a 15 billion line csv into database and IEnumerable gave me the impression that it would help me but it actually didn't 'cause of the multiple interations. The only solution that occurred to me in that time was to slowly add some values to a list and them import, otherwise it would throw a memory exception. Without this approach, the entire proccess would take almost 3 hours to complete. After some modifications and making it more "listy" it ended up being just some minutes.

When you switch to .Select(), the file is only read once while each line is selected twice; and then you could just append .ToList() to the .Select() to return a list of Customer's without splitting each line twice.

If I hadn't already been using R# for the last 4 years I wouldn't have ever known about this. Great explanation!

I learned this the hard way too, but it was a very important and interesting lesson to learn. Glad you made a video on this because it is a very important feature in .NET that can make or break your application.

Funny you should come out with this video. We recently upgraded to using Rider and have caught a few times we missed this when using Visual Studio. Thanks for the content as usual!

This was exactly the solution I was hoping to see because that is what I have been using for years :) As for the return type, I always return as specific as possible while accepting as generic as possible. Worked well for me so far!

This multiple enumerations text occured to me as well Resharper which is the sole reason I am aware of this behavior. Really good thing to teach to people.

I heavily use co-routines on my applications, specially for large operations, I can read 28k lines from the database, make them as objects, turn them into JSON and send them over to clients without loading 28k things in memory, then making 28k objects, then making a 28k items json array, saves a lot of RAM and avoids high-gen GC.

yay - I was able to find the issue before Nick pointed it out. =) Though, I only looked for it because Nick pointed out there was a problem though...probably would not have cought it in real life.

Thanks for sharing this, didn't know about it, but I personally never use IEnumerable. However looking at colleagues code during code review I can now point this issue out to them when I spot it. Cheers.

yes I knew and it caused me a lot of trouble. Thank you for your explanation, always appreciated.

I knew about that and also felt into the downsides of it. Since then I am cautious when I get an IEnumerable and check my call stack if it is used multiple times (aka enumerated). But also I remember to have read in the official c# best practice guide to use IEnumerable as return type and parameter. (did not looked it up again).

I do actually know of this, and typically do take this into account; I'd be lying if I said I catch myself [or others] every single time :). It's one of those things you miss if you're working fast

The general advice for any caller of iterators that return IEnumerable, is to not mix cursor calls like ForEach, with aggregate functions like Count if both results are in scope of each other, unless the first call casts the result to a collection or array and the second call uses that cast result instead. Not doing that is not just a matter of performance; that's even potentially the least of our worries. The problem is instead that most likely we just introduced a potentially hard-to-find bug if the source data can be changed by a third party between both calls. But if both calls are not related and they are out of scope of each other, do not cast. That's a potentially expensive operation in itself.

I've seen that warning before and had no idea what it meant. That was a great explanation, thanks for the video!

Great explanation of multiple enumerations

I learned this the hard way playing with Entity Framework when it first came out years and years ago. It was not a fun or quick learning experience! Unlike this video, which was both fun and quick 👍

Thanks for this video. I don't use IEnumerable. After that video i'll still so :) but i learn why i'm not.

I didn't know about this before and will be more careful about the usage going forward. Thanks Nick

Just came across this warning a few days ago for the first time in Rider. At first glance, it might seem like something insignificant, but in my case this saved a few seconds of execution (in scale) for some flow.

Interesting. In the languages I work with, doing something like this simply fails because the iterable can only be iterated over once, so you'll find out quickly that you need to cache the results to use them twice.

I love that JetBrains catches possible multiple enumerations BUT OH MY GOD If you don't enumerate the same IReadOnlyList multiple times it will NAG You endlessly to change to parameter to IEnumerable Which is annoying because your parameter already conforms to IReadOnlyList then it will again nag you to change it back when you enumerate one more time

Makes me think it be nice to have an enumerable type that when called statically constructs the IEnumerable once with minimal overhead when the function is called, while using ToList is clear, it would be nice to designate from the function definition without requiring to produce a list or other container explicitly. Lazy loading and enumerable reconstruction is deceptive when you're used to how containers work especially when this behavior is built into the language.

Strangely there's no OOB class or ext method to do this better. ToList would create a list that reallocates as it grows. What would be cool is a linked list of exponentially growing blocks - takes O(n) memory and time to build, but more efficient than just a linked list. And a common rule is: "iterate multiple times - use IReadOnlyCollection"

I questioned the system when I used a Where clause on a list and then when I modified a value on that list it suddenly got added/removed based on the where clause. When I never updated the IEnumerable collection to reflect the changes. After trial and error I debugged it and write a before and after count and my value change somehow magically reflected in the collection, I got an inkling of how this enumeration worked after watching this video I've now discovered that Count and iterating through it, reexecutes the Where clause everytime.

8:45: "Know that the warning might be there but there might not be multiple enumeration in every single one of those occasions." Enumeration is the process of _going over the elements of the enumerable,_ not of creating it. When you have a List, LINQ's Count() can call the Count property directly and not have to enumerate the list to count it. But if you had multiple foreach loops or LINQ queries, that would indeed be multiple enumeration even if it's of a List or an array. As the video says, the warning is confusing. The problem isn't multiple enumeration; it's multiple _generation,_ which can be a problem for more reasons than just the performance hit. If the generation of the enumerable is non-deterministic for whatever reason, (maybe you have a call to a random number generator in there, or you're querying a database twice and someone else INSERTs something into it in between your two calls,) you can end up enumerating two different sequences of values when you intuitively thought you'd be enumerating the same values twice, which can cause bugs in your code.

Great and very important point Nick

To be fair, this is why I hate using Var. It can hide these issues. Where specifying type makes sure you know from the outset that it has still yet to be resolved.

I recently switched to Rider and have build a library from scratch, and Rider gave me the warning. First time I ever stumbled upon this. And now you made a video about it :D

Ive know this for awhile and I always thought it was a problem with .Count(). I have creatively coded around .Count() in many projects.

loving the Barking joke at the start 😂

Thank you for tip, quite interesting. One question.. Does your courses come with certificate at conclusion?

Very useful video Nick, thanks again

I havent watch vid yet Nick "Epic" Chapsas is a Legend

Great video. Thank you for the information.

If you are creating a list internally, why not just return a list instead of IEnumerable? What are the advantages of returning IEnumerable?

Wow, that's amazing! The more subscribers you have - the more basic stuff you'll have to explain :( The next thing, I suppose, will be the difference between ++i and i++ and how Interlocked.Increment(ref i) works with async/await code :)

I feel like you should also show the rider-specific "why are you showing me this warning" link they usually build into the alt-enter menu for warnings like these (when they exist)

Its called Lazy initialization ? Can I keep this feature, when I create my own class, which is inherited from IEnumerable?

I usually IReadonlyCollection or IReadOnlyList, unless really need lazy evaluation

Resharper taught me, now i understand it

I know about this feature and always take it into account. But I know too many developers who don't know about it. It's easier to list those who know Most often I see the multiple enumeration warning on projects where people use Visual Studio. I think it's partly Microsoft's fault that they still don't warn people about the multiple enumeration possibility, so people don't care

Great video, but I think it is also important to mention that calling ToList() should be done only if underlying implementation is not enumerated. For example, in your example when your IEnumerable was returning List (instead of yield), a call to ToList() would copy the same list, which increases memory usage. And for new developers, they could start thinking after the video that ToList() should always be done if they are using a method that returns IEnumerable.

Yes, I knew about this problem because Rider told me :)

thanks man u r the best!

good to know, did not know before

I really want a Nick Chapsas for Rust!

does ObservableCollection have the same behavior since it implements IEnumerable in its base source ?

I don’t have resharper and don’t know about this until now…damn

I could be wrong but I get the impression that most people tend to materialize `IEnumerable` instantly and that there's a lack of understanding on how `IEnumerable` works. Also, that was me some years ago...

This video made me subscribe 😁

my first work day as developer / programmer today 🎉

today lesson. thanks :)

Where is Chap Nicksas? 😞

Can anybody telll me what Nick is using to show stuff like "x: "Nick Chapsas, 29"" and "splitline: string[2]" when he debugs? It seems to show the count as he debugs as well.

Thanks 👍

What is the better to use in such cases? ToList() or ToArray()?

I really like your videos. This one was good too but the title was not really correct. Your presentation makes it look like the problem is IEnumerable. Its the correct way to return something when you don't want to limit the implementation of the method to a specific collection type. In fact, the problem is the Select() and knowing when to use Select().ToList() . In fact, you didn't even use ToList() instead you used a clunky list.Add() method. Also Count() is to be avoided in some cases. Length and Count are preferred (but hopefully Count() uses Length and Count behind the scenes, but it still requires a method call and an if statement). Another thing to realize is that by returning a LinQ QueryCollection type instead of IEnumerable you can avoid the double iteration? In that case, your Title would have been more appropriate. Whatever, I enjoyed the video anyway. Your presentations are always top notch in quality !

Yay, new video 👍

What happens if I want to return an empty list? You previously said it was better to use Enumerable.Empty, but I cannot do that if the method returns directly a List, right?

How do you gain all this knowledge regarding perfomance?

lol, never used yield, always did materialization to list intuitively

Why does no one have this problem with `Iterator`/`IntoIterator` in Rust? 🤔

one interviewer asked me to compare IQueryable and IEnumerable :)

why is it that he can write a method without any enclosing class?

I never really liked the fact that certain libraries declare their return types as IEnumerable when they're in fact already Lists or similar. But it's also sometimes a problem because tools like ReSharper will suggest you change the return type if you're not doing anything that isn't possible for IEnumerable. But then again, maybe that's just a sign you don't need to enumerate in the underlying code. Goddamn, now I'm rambling.

7 years c# programmer, just knew about it

why would you ever need to return ienumerable anyway

"Hello Worldish" topic, and every jumior knows it (at least every juior I've worked with)

It is very bad practice to save IEnumerable at variable. If you need to save IEnumerable need to save it as array. By my opinion array is most basically type instead of list.

I can't say I have ever seen a yield return of an IEnumerable in 15 years of C#. Return of an IEnumerator in coroutines, yes, but not this. Must be a business software thing.

Hello Nick, from everybody

This is another example of Linq hiding bad coding. If you don't use Linq here, it's a lot more obvious that you have to iterate the enumerable to get a count, since you don't have a Count property/method.

IEnumerable can kill your performance in C#, but if you want to be 100% sure, combine IEnumerables with databases and Lazy loading features. Devastating.

If you have 1M rows in your file, is better the first approach.

Nick the great, in C we were warned calling length of array in a for loop, it will end up with O(n square) instead of order N

I like your mustache

Good stuff. Curious, at the end of the video you said "I probably wouldn't return IEnumerable, I would probably return the Type". Can you expand on that? Also, in your example you are getting a count. Wouldn't it be better to use ICollection instead?

Oh LINQ and IEnumerables, such a blessing and a curse. Very easy to abuse and misuse.