Profiling and optimizing your Python code | Python tricks 

The final solution is also not linear, but n*log(n), since python sort is not linear.

It will be nice to see perf difference between his final code to your idea..

i think conversion from set to list will take more time.

The goal is to find the duplicates, not to remove them

Was going to post this as well - converting the list into a Counter (e.g. a special dict type from the collections module) and running a comprehension to get back everything that had a count above 1 does seem to be the cleanest way to get to the solution, and I am pleased it is also the fastest!

@@Alister222222 - I tried this and was 0.13 seconds faster or 44% faster than the Zip method

Yeah I can imagine :-) First time here. Coming back to remember yourself that errr... yeah ... why on earth does it take so long and frustrates me ... how am I ever going to find out wth this code is slow. Is it me or is it some crazy circumstance that goes on in my libraries that I use. While waiting for your code to finnish you can actually study the problems and get them fixed. But I do think not by hand just by profilers

Hey🌏🌏🏕️

+Samuel Muñoz thanks ! Yes, the Bio Psy lectures are something new for me. Most of the videos are about Python and/or OpenSesame

+Grimlor glad to hear it!

Good to hear!

That's correct: triple duplicates are not caught with this method. And thank you!

yes, I was thinking the same thing, the final code was so fast because it only checked its 1st neighbor, taking into account that there were only 1 duplicate.

+post fix Thank you! No, I'm afraid that I do not have any Python books myself. But there are plenty of good free Python books out there, such as Byte of Python.

thank you for your answer, dank je wel :-).... you should wirte one about the tricks which you show us here... and here is the link for the Byte of Python for all others: www.gitbook.com/book/swaroopch/byte-of-python/details

My reply is a bit late, but I got this data from here: osf.io/r73y9/

Actually got it to work. It seems that you'll have to encode the Unicode strings to byte strings, and use io.BytesIO, instead of io.StringIO.

NO it's in O(n*ln(n)) because of the Sort()

With Big O notation we are really ony concerend with the term with the highest power. An algorithm on the order of O(3x² + 2x + 11) is usually reduced down to to O(3x²). I've seen books drop the coefficient as well but that has a fairly large impact on the accuracy of the expression in my opinion. So in terms of Big-O, an algorithm on the order of a quadratic equation is usually considered to be on the order of its highest term. If you think about comparing two algorithms, one operating at O(3x² + 2x + 11) to one that runs at O(3x²), let's see how different they really are: So given the following equations: f(x) = 3x² + 2x + 11 g(x) = 3x² Let's see how they correspond given a single input (n=1) f(1) = 16 g(1) = 3 The ratio between these two results is 5.33 and would go to show that quadratic and exponential are not swappable in this context Now lets scale it to 100 inputs, n=100 f(100) = 30211 g(100) = 30000 Now they are operating at a ratio of 1.007. Not identical, but damn near close dependng on the precision needed. In terms of making algorithms efficient with computers 100 inputs is not considered much anyways. Now let's scale it to 1,000,00 inputs f(1,000,000) = 3000002000011 g(1,000,000) = 3000000000000 Ratio of 1.00000066 The difference in comparing these two with without the extra terms is often negligible when comparing them to algorithms on the order of a different exponential power. Run the same exeriment with comparing f(x²) and g(x³), with and without extra quadatic terms and you can see that dropping the lower terms, though not exact, is definitely enough to compare the efficiency of algorithms. So as the size of the inputs grows, paritculary towards quantities where optimization is necessary, we are usually dealing with such vast amounts of data that including the lower terms of the quadratic formula in our assessment of an algorithms efficiency does not necessarily provide extra insight. ps: i'm fully aware this isn't the case in every domain, but it is for the most part how it is done and definitely applies to the kinds of problems in this video.

Lol these comments. It's a loop in a loop which is n * n so n^2 tadah

@@stephenaiesi6073 > I've seen books drop the coefficient as well but that has a fairly large impact on the accuracy of the expression in my opinion. Big O notation has a formal mathematical definition. A function f(x) is said to be O(g(x)) if |f(x)| =x_0, where A and x_0 are some constant values. Hence, when considering big O notation, it really doesn't matter if you drop the 3 or not. If f(x) = O(3x^2) then all we are saying is that there exists some x_0 such that for all x>=x_0, |f(x)|

Stephen Aiesi Thanks mate. This is a really good explanation

That's correct. Triplicates will end up twice in the list of duplicates, which may not be what you want. An easy trick to get around that would be to use a set comprehension (ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-uTUV2eONqSQ.html), rather than a list comprehension. Because sets by definition consist of unique items.