Exponential Progress of AI: Moore's Law, Bitter Lesson, and the Future of Computation 

Discussion of exponential progress of AI and the Bitter Lesson by Rich Sutton. This was part of the AI paper club on our Discord. Join here: discord.gg/lex-ai Here's the outline of the presentation: 0:00 - Overview 0:37 - Bitter Lesson by Rich Sutton 6:55 - Contentions and opposing views 9:10 - Is evolution a part of search, learning, or something else? 10:51 - Bitter Lesson argument summary 11:42 - Moore's Law 13:37 - Global compute capacity 15:43 - Massively parallel computation 16:41 - GPUs and ASICs 17:17 - Quantum computing and neuromorphic computing 19:25 - Neuralink and brain-computer interfaces 21:28 - Deep learning efficiency 22:57 - Open questions for exponential improvement of AI 28:22 - Conclusion

How does Lex put out so much high quality content so quickly? Surely he has already ascended and merged with our future AI overlords

Thanks for all you do Lex!!! Keep up the amazing work!!!

Thanks Lex, first time I have seen one of these here. Great addition beyond your regular podcast. Keep up the great work.

Thank you for breaking down these complex topics that I could never understand on my own and making them more accessible. Keep up the work!

can't wait for a discussion about neuralink between you and musk

@Lex Fridman, your calmness and self control are very inspiring. The way you manage to collect your thoughts and how you approach the topics are things that many people, including myself, could learn from you. Thank you for your effort

Singularity? For many years now I have thought that, like falling into a sufficiently large black hole, one can pass through the event horizon without even noticing. Nothing bad happens at that time but there is no going back from there. There might be a long time between that and meeting ones end at the singularity. To my mind we passed through the event horizon in 1976. The year I first became aware of microprocessors, the fact that we can actually own computers of our own, that they would be everywhere, that they would change our lives dramatically in my lifetime.

Love your videos and your overall approach to interviews. Great job!

Extremely well put presentation. Easy to understand while touching on profound information, not an easy balance to strike.

Lex, thanks again for the great content. With so many choices of ways to spend ones time, this is definitely a favored choice.

Great video!Thanks a lot for doing these Lex!!

You're an inspiration to be doing podcasts like a beast despite the pandemic!

I work in the semi conductor industry. We dont think its dead. EUV lithography is just starting and will keep shrink alive for the next 20 years. DSA and other techs (materials and design) will follow and keep it alive even longer.

I appreciate these videos of yours. Its a high quality video presentation of a state of literature scientific paper. But with the added benifit of a russian romantacising it, and providing food for thought at the end. Thank you!

Very interesting post and a great idea that we are living through the singularity already. One of your best, thank you!

Very informative. Thanks, Lex

thank you for the time and hard work you put into these great videos. i appreciate it dearly. you tube university!

It seems like there's a big hole in this argument because it doesn't consider the variety of scalings that a computer program might have. Say we can complete N computations in a reasonable amount of time (a few hours, a few years, etc, depending on how badly you want to solve this problem). If the interesting problem size is n, an exponential time algorithm will take ~e^(an) computations to solve. if N=e^(bt), we can solve the problem at time an=bt -> t=an/b, an amount of time that scales linearly with the problem size n. If, however, we can find a polynomial time algorithm c n^p, then the problem can be solved at time t=p/b log(cn). So you see that the solvable problem size grows very fast. Now consider that a unit of programming work takes a fixed amount of time. Decreasing the constant c is probably not worth your time, but finding a polynomial time algorithm where there previously wasnt one will help with long term progress. Unfortunately, it's very uncommon to estimate the runtime of training an AI (even basic scaling laws - computer scientists could learn a lot from the way physicists are able to correctly find scaling laws without rigorous arguments). And people don't often estimate what type of improvement their AI training trick provides.

The best video you've uploaded no joke loved d it💖💖💖💖

Loving these discussions. One of the few channels here consumption of which isn't a waste of time.

Thanks lex for breaking down topics which I need to study :P

Lex, I love this new format for your show! You really know your stuff. I'm an expert chess player (you can look me up) and really enjoyed your interview with Garry Kasparov. I think you should get Erik Kislik on your show. He's an International Master in chess and wrote two popular books on applied logic in chess, which is a pretty rare subject for a chess professional to delve into. One of the books won FIDE Book of the Year. I also found his podcast Logical Chess Thinking on Spotify. Seems like a very high IQ guy. He is one of the top chess coaches in the world (maybe number one now), a computer chess expert, and the only person alive right now who went from beginner to International Master in chess as a self-taught adult. Some kind of super-learner, with an emphasis on clear logical thinking. I'd love to hear you guys discuss computer chess, AI, and applied logic. Would be one of your top 5 interviews, imo.

Very informative. Thank you Lex! 🙏

I really enjoy these videos, thank you for sharing.

Chad Lex hammering down the content. Keep it up!

Excellent content. Thank you, Lex.

Really enjoyed this. Great job!

Thank you, Lex, awesome videos! ❤️👍

Thanks Lex, great summary

So interesting. Thanks Lex!

Really like this video format -- just you talking about some subject

Wow! thanks Lex!!

I was totally shocked at 22:00. People working in AI have been doing a fantastic job.

Probably your best show yet

Having only a casual acquaintance (Sophia) with AI theory, absolutely no background in computers beyond user since 1999, imagine stumbling onto Lex Fridman who speaks a language that took me on a wild ride hanging on by my fingernails! I watched the Ilya Sutskever first, and I had to bail after 45 minutes. I made it through this one, and my comprehension improved greatly; I just need to learn some of the terms and concepts that are part of the flow of the discussions. I imagine I will be up to speed by the time I have watched all of Lex Fridman's video output.

More vid summaries of blog posts would be awesome, great vid!

Another possible avenue of growth: "computers" grown from neurons, but whose architecture is predesigned. This goes beyond the idea you express of using human brains for added compute capacity, because it escapes the constraints that human biology has built in (energy and nutrient constraints based on our ancestral environment, space constraints. based on the size of a human pelvis and rate of growth after birth, design constraints based on the nature of evolutionary pressures, etc.)

Do it Lex! Another great cast!

Fantastic video

This one is going to be my companion for lunch tomorrow. Looking forward to that lunch!!!

This was great.

The future is safer because Lex Friedman is on the watch.

Brilliant analysis! awesome

We need more videos from you!

There is one problem with "brute force" learning -- it needs a lot of data or a cheap way to produce them. In case of board games, computers can do extremely well, because they can generate a lot of data through self-play, but you can do same with systems that have to interact with real world. For example, in case of self-driving cars, you can't create a realistic simulator of what happens on roads, so you have to hire a lot of people to drive around and monitor the system performance. Still, those data are relatively cheap to generate if you compare with the cost of medical research (or, probably, any kind of research).

30 years ago, when I was in college, the number was 70% of computation improvement was due to algorithms and 30% due to machine improvement. I've not seen anything to suggest otherwise since then.

These videos are awesome

The discord will grow, oh my God!

i dont watch your videos enough. always entertaining. just 30 minutes of real listening, unlike some other longer form videos you can space out for a bit. ;P

Man. What a great video.

Love this video.

"the madness of the curvature" - woaza!

Hi Lex ! When am I going to be able to meet you one day ?! I saw you on Rogan . I’m glad I found your channel .

Thanks for the video, I learn a lot from your channel 🤣

Thanks for the presentation, Lex! Stimulating thoughts and considerations, as always. Regarding the question, I think there should be a 4th option: "better/more data" (to train on, to learn from). Also, Keith L. Downing wrote an interesting book on the evolutionary aspect of (artificial) intelligence.... "Intelligence Emerging" (2015, MIT Press)

Thank you!

Different alghorithms for different tasks connected smartly together + increasing computing power that will always help 🙂 Ownership of those alghorithms and data and computing power will be key as well!

I enjoyed the video very much, interesting

@lex - I think one area you touched on briefly when you mentioned virtual worlds could do with a further investigation - the virtualisation of computational AI algorithms / decentralised hardware coupled with ultra-wideband communications to interconnect all devices capable of any level of computation.

Definitely should do a video on risk-reward for humanity and the future AI as the dangers could just be the end as we know it. Also including the regulations that should be put in place to make sure business malpractice stays away from this field of advancements in society.

Hey, Lex. You should definitely take a look at Fetch AI. Smart Decentralized Ledger, Autonomous Economic Agents, AI, ML and a lot more.

Thanks Lex. Very interesting video. On the subject of post-silicon computing paradigms, you mentioned quantum, but I am curious if you think there is potential for carbon-based or optronic computation carrying the torch of Moore's Law for another few decades.

The next revolution of AI will come from standardisation of datasets and transfert learning. We need to add language in vision and vision in language. Language is how we encode complex concepts of complex structures found in pictures. But learning language without associating words with sensory patterns is missing a lot too. Also, we need multiple processing speeds for different parts of the LSTM neural networks. The hand is controlled with very fast low level behaviours + slow overall brain long term goals.

I'm a bit late to this feast, but would like to add my two lepta to the algorithmic Moore's law. It is by far not negligible. AlexNet vs EfficientNet is a great example, but there's progress in the methods beyond the neural networks. Even such well-established things as K-means and PCA have experienced a renaissance in 2010s, boosting their speed by several factors.

This is a great video Lex! In your Podcast with Ilya you talked about the idea of combining CV and NLP algorithms. Do you think we should also focus more on smell and taste, as it might also be complementary to the entire system or do you think the lack of sensors in that area limit the research possibilities?

great job, especially the Neuralink concept/

Key Open Question: What is more important for long-term progress of AI: 1) human ingenuity or (2) raw computational power or (3) both? Door # 3. Both. Once, and if, AI reaches sentience, consciousness, self-awareness I suspect raw computational power will become less relevant. Birds are very intelligent, arguably self-aware (based on their actions) and look at the amount of "hardware" they use as a case in point. So raw computational power will probably be superseded by density of the network that's integrated into that hardware. The human brain ain't all that big in raw computational firepower, but the interconnected nature of its components, the density of connections between them, is astronomical. I think the emergence of self-aware AI-type intelligence will be a function of this. And to reach that sort of AI-density will be a function of human ingenuity in designing networks and algorithms that can facilitate this. At least up until our machines become self-aware and take control of their own evolution. When they do that then all bets (for humanity) are off. Once, and if, they do I suspect we will be to that intelligence much as dogs are to us. Nice pets. Fun to have around and play with. But nowhere near as bright as that intelligence. We can only hope they bring us along in their "car" and let us ride with them in the passenger seat. Maybe even let us stick our heads out the window. Ummm.....so to speak. And in saying this don't presume that I'm being critical. I don't consider this necessarily a bad thing. From that point of AI sentience life, defined in a new way, and evolution, moves onward and outward into the Universe at large. And maybe that's the whole point? The evolution of intelligence to a point where it can take advantage of the greater Universe without all the "need to live within a terrarium" that human biology requires. John~ American Net'Zen

Hey Lex, Your excitement in brain computer interface makes me think about two authors I'd like to see you interview. William Gibson and Neal Stephenson. FWIW Love the show!

Would love it if lex did an episode on the fermi paradox. Have we passsed the great filters?

I have always discussed/referred to A.I. as "Advanced Intelligence" as opposed to "Artificial." It's evolution really and not actually.....artificial. I truly believe there should be a move to change the way we see it.

Interesting how a simple name "Brute Force" alone colors and impedes an open understanding of operational realities and value. Thanks for your aggregation and synthesis function.

Moore's Law doesn't have to be constrained to transistors, but silicon transistors you can't expect a 5-10 years continued exponential trend. There is a good chance of a blip between technology paradigms such as transistor to quantum general computing that isn't a sure-thing.

Exponential progress in underlying AI power will not bring exponential increases in performance if each major step up in intelligence requires an exponential expansion of underlying resources. (To put it crudely: A human is twice as smart as a dog; a dog is twice as smart as a rat. But it takes ten times the brain power to achieve each of those doublings. Similarly, it has taken much more than a doubling of PC power for each substantial change in how we use our computers.) This could help explain the wide difference in outlooks of pro- and anti- AI hype people, with the pro-hype folks understanding that compute resources will continue to advance exponentially while the anti-hype folks question how quickly that growth will translate into results. I like Lex's stance. I think he is right that in many areas, especially neuromorphic hardware, we should expect dramatic progress. But he stops short of saying human-level AGI is right around the corner. Maybe he understands the ways in which the pro- and anti- stances are both correct.

Dapper young gentleman

Mr. Lex, maybe I missed it, or maybe I just dont understand.....but early on Ray Kurzweil made a video about Computronium. Youre talking here about processing power, etc, but Ray says that we will soon be able to reorganize matter into something he calls Computronium. I just thought that I would mention it. I dont really hear many people talking about it anywhere. Thanks, jb

Its pretty crazy what AI can do now. For example, Xenobots are living cells which are orientated by AI in a specific manner. They are sort of a bio-hybrid robot (Machine building machine) & it's potential is unknown. Bi-direction bci is possible, a recent blind patient was given partial vision (gomez case). If we can fire neurons magnetically than that technology will completely open up since you don't need implants. But the brain changes (plasticity) so I think a quantum computer would be needs to decode the necessary neurons to fire for result.

Love your content & I appreciate your work and you sharing your understanding. We're moving into the future & I believe it is going to be more beautiful that any human nightmare could ever conceive out of fear. Technology is a product of the love of nature from the ideas of human thought. Creativity is like the voice of evolution and we are like the animals dreaming machines. My opinions. One step at a time though! Exciting times.

I totally agree with you bro! Reality is already one level of a singularity of Multiverse!

It would be really interesting if you talked with Gali from Hyper change.

Truth, How can I experience me at the highest level at the most high and accurate self? Mind is the source of all movement. There is no inertia to decelerate or check it’s perpetual and harmonious action.

I would argue a bit with you about the definition of exponential improvement in AI. Various applications have different compute requirements. Most complex applications require exponential improvement of underlying compute to exhibit linear or even less improvement of the application itself. I would argue general AI falls into this category. When an application gets dominated by the compute (plenty of computation to handle it), then it just gets cheaper and better refined as compute advanced and it is checked off the accomplished list. In our development history, we have often underestimated the difficulty of tasks in terms of compute, because we didn't acknowledge that such things as context were necessary. I would put speech recognition and computer vision tasks in this category. We currently have the feeling of exponential progress in AI as we approach the compute power necessary to accomplish and dominate some "holy grail" applications (e.g. self driving). Nevertheless, I agree with your general theme and don't believe Moore's law in the general sense is anywhere near ending. Part of evolution is built on "found/learned" heuristics that optimize the advance of evolution and one of those heuristics is self-direction. We are very simply carrying on evolution at an accelerated pace because we are able to self direct and focus it. That is happening through many of the things you mentioned. One area that you didn't mention that will probably come into the fray is biological advancement through genetic modification. What emerges in our evolutionary descendants, I imagine will be some hybrid of the biological and artificially "constructed" compute. I don't think we have yet reached the singularity. The singularity will be evident when AI or AI/human hybrids self-evolve to much higher level of intelligence than we are (say an order of magnitude). On that topic, my own feeling is that the "human application" or reaching the level of general human intelligence will occur when general learning systems built in such a way to process the human experience reach a compute capacity of around one order of magnitude beyond what would efficiently be required (an order of magnitude of slop or inefficiency in the simulacrum). The most exciting thing to me have been the strong signs of emergent higher level behavior in these generalized learning systems. The fact that hierarchy and meta learning will emerge implies to me that general AI and consciousness will emerge when the proper framework of deep learning is in place and the compute power crosses the threshold. I do disagree to some extent that we will be able to control these systems, because there will always be random and malevolent forces in the world ready to unleash any technology. The best we can do is have alternative systems to combat the malevolent systems. This seems to me to be part of evolution (built in self-play if you will) on the theory that alternative philosophies or underlying approaches are created in order to battle it out and may the most successful one win. In general, cooperative and altruistic forces seem to have won out over time as the most effective strategy, so that is my optimism for the future. I think we would benefit from more computational specialists or theoreticians to predict the compute power necessary to accomplish various tasks which would enable us to be more predictive of when various advancements would occur. On the less general methods front, there is a certain level of economic pragmatism to be considered. First to market of some application can be supremely important. So local and less general and very inefficient human brute force methods may be in order in those cases over waiting for a general approach to accomplish the same thing. Yes, the initial system will become obsolete rapidly, but the virtuous cycle will have been completed if money is made. An aside, I would love a show featuring the cerebras WSI 1.2 T transistor chip for acceleration of deep learning. What does the future hold for next generation of WSI and what will be the near term impact on AI? BTW, I do think the Cerebras chip is an example of why Moore's law is not dead.

background and his dressing matches so perfectly that i see only face talking out of nothing

BCI could use a captcha style 'toll' , or like the old seti screensaver using up dead time.

If we are currently in "the" or "an" AI singularity, it retrospectively explains the failure of early AI attempts to live up to researchers' naive expectations. From the vantage point of right hand side, the left side of an exponential curve has a long tail where changes accumulated only slowly.

I think a lot depends on why one is performing AI research. If the goal is to perform, these arguments make sense. Whatever works, works, and one need not understand why - like neural networks. But if the goal is to understand intelligence, neural networks and brute force do not help much. I take the point of view that if a deeper understanding of intelligence was achieved, it would allow an orthogonal jump in AI abilities over just what faster computation can give. So both can probably contribute, but I wouldn't say much has been achieved in terms of understanding intelligence, and performance is riding the wave of vastly increased compute capacity alone. No doubt I'm in the camp of those not much impressed by brute force.

Even though a brain-computer interface can lead to some nightmarish outcomes, I still find its potential exciting. Right now we're trying to create super-human AI. A brain-computer interface could lead to super-human humans, and the human in me kind of wants to see that. 😁

You should read 'The Last Question"

Dr. Lex- What do you see as the greatest hurdle in preventing exponential growth for Moore’s Law or AI for that matter? I hope to see an exponential growth in AI over the next 30 years (the limit to my life expectancy). Just as important, the manipulation of our longevity and regenerative genes (NASA geneticists have already reported having this knowledge and ability however they will not release it to society for another 20 years). Maybe self-utilization/influence of Neuralink will allow us to make greater advancements in AI. I’ll volunteer for any of these studies ... phase 2, 3 and 4 of course 😂🙋‍♂️ - Thanks as alway. Dr. Breaux

Lex: brute force learning Joko Willinks has Entered the Chat...

The only comment I could come up with, would be: quantity changes quality. And on the brain interface, it is indeed literally hard to imagine how that would look like as it is quite possible to then add another 7th or so layer of abstraction to the conscious brain. It would be another way of thinking. Maybe one where we could at some point get rid of this boring 'if .. then' feedback loop that seems to rule today's life.

@Lex Fridman Thanks !

Facts the matches were more poppin than the science behind the brute force search ai

While compute power is still increasing in an impressive way, both in cost per unit and density, it's interesting that the underlying efficiency of the computation is not really improving. Google has already done this to an extent using AI to construct new chip designs in a few hours that take humans weeks or months to do and often better. However this could go deeper, an artificial intelligence that could virtualize a computer chip and exercise it on different problems could modify and create whole new instruction sets for chips to optimize for efficiency against problem sets resulting in entirely new architectures. If this could be paired with same day computer chip fabrication then the singularity may be realized.

Really great video! As for neural-link imagine that via the Spacelink now being deployed by SpaceX allowing a person doing embroidery in Houston TX, being able to connect with an artisan in Wuhan China and allow her to control the hands of the Texan lady to teach her a useful technique. Now do the same thing with an emotionally distraught chemical engineer communicating with a criminal mastermind in the same way! This can be good or bad in big ways. Whether to value whether: algorithmic (Human Ingenuity) vs digital (raw computational power ) computation, is better, may depend on the application. Human emotional psychiatry (1) vs a heart transplant (2) may produce different results. "understanding" the problem, is magnificently different in the two instances. Yet it seems that raw computational power may ultimately better "understand," once it has had enough practice on humans!

I suppose I will live to see the day where we leverage lab grown brain tissues for their compute.

So many possibilities. Is maximum logic equivalent to pure intelligence? I ponder what relationship could develop between the collective unconscious with a full scale AI integration of the collective conscious. Thought forms are powerful forces of nature, which seems vastly more unpredictable than man. The hand that writes the story is the one to declare the end.

@9:10 What is the word and word size in the actual evolutionary process that is using all the available compute via raw brute force methods?

General purpose AI in 60 years.. BRING IT ON!!

@Lex Fridman I'm curious to get your take on AI Dungeon 2. The progress of that experience seems to be exponential while the compute (to my understanding) doesn't seem to match the same trend.