Moore's Law is Not Dead (Jim Keller) | AI Podcast Clips 

Btw. For people who don't know. This is the guy who designed AMD's Ryzen architecture.

"The number of people predicting the death of Moore's law doubles every two years." -Peter Lee, Microsoft

This guy does not break eye contact and continues conversation without skipping a beat. I don't think I could do this in a dream.

Jim is like the Rambo of integrated circuits.

Whenever I begin to get confident in my own intelligence I like to listen to guys like Jim Keller talk so I can be pounded back to reality.

Meanwhile, the JavaScript developers are making sure that any hardware progress is nullified by adding more NPM dependencies

Imagine if Joe Rogan interviewed Jim Keller? Their gap in knowledge would be so great that they would form a black hole in between them.

Lol, this got confrontational, but in a totally academic kind of way, then got instantly resolved. Good chat.

Listening to Jim Keller talk is THE most mind blowing experience. Lex you do a great job asking questions and keeping up.

Amazing video! Thank you for making this and the others you have done!

tremendous Questions Lex. I enjoy your channel and will continue to watch and learn.

Man the level of intelligence this man is so high it made him so humble, like what I do might not even matter as much in the grand scheme of things. 👏🏻

Last part of that interview was actually chilling. Absolutely incredible interview

great stuff from both sides... I love the open way of Lex and Jim is in a space in its own... great contribution

I love this! Two extremely intelligent people talking about complex things in a way that even dumbasses like me can follow! Cheers guys!

Keller really puts it into perspective when it comes to layering from a simple switch to an infinity of equations. Impressive.

Jim is such a humble and intelligent guy. He seems like he would make an incredible teacher/mentor if he wanted to go that route someday.

Great sign of intelligence on both sides to disagree and then agree again within a few minutes. No need to even finish most of the thoughts, just dance through it in seconds.

May's Law states, in reference to Moore's Law: Software efficiency halves every 18 months, compensating Moore's Law. David May (born 24 February 1951) is a British computer scientist. He is a Professor in the Department of Computer Science at the University of Bristol and founder of XMOS Semiconductor, serving until February 2014 as the chief technology officer. May was lead architect for the transputer. As of 2017, he holds 56 patents, all in microprocessors and multi-processing. When Inmos was formed in 1978, May joined to work on microcomputer architecture, becoming lead architect of the transputer and designer of the associated programming language Occam. This extended his earlier work and was also influenced by Tony Hoare, who was at the time working on CSP and acting as a consultant to Inmos. The prototype of the transputer was called the Simple 42 and was completed in 1982. The first production transputers, the T212 and T414, followed in 1985; the T800 floating point transputer in 1987. May initiated the design of one of the first VLSI packet switches, the C104, together with the communications system of the T9000 transputer. Working closely with Tony Hoare and the Programming Research Group at Oxford University, May introduced formal verification techniques into the design of the T800 floating point unit and the T9000 transputer. These were some of the earliest uses of formal verification in microprocessor design, involving specifications, correctness preserving transformations and model checking, giving rise to the initial version of the FDR checker developed at Oxford. In 1995, May joined the University of Bristol as a professor of computer science. He was head of the computer science department from 1995 to 2006. He continues to be a professor at Bristol while supporting XMOS, a University spin-out he co-founded in 2005. Before XMOS he was involved in Picochip, where he wrote the original instruction set.

This is a great topic to cover. Please post more like this in the future. Excellent!!!

Really enjoyed this. Thanks.

Best interview ever!

06:50 building a brick building with more Moore in mind 16:38 a difference in quantity is a difference in kind 17:54 the prob with early optimization

"Galavanting through the nether realms of possibility" lol

A Cascade of diminishing return curves: moorception...

This was truly remarkable. Thanks for cutting it down to 26mins now I think I might go watch the full one

Spicy discussion. Liked it very much!

I listened to the video, I was quiet impressed. Not just a chip architect. Loved the discussion of levels of abstraction among other things. But I disagree with 2 points. (I'm an algorithms guy, my ph.d. is in operations research). a) as Lex said, there's a lot of complexity in not just human behavior, but time-varying road conditions. We're talking the Curse of Dimensionality here and it's not clear to me that a brute-force camera-data-only training approach converges on an answer. Rain or hail or sleet can change the driving conditions (traction) quantitatively within 10 seconds. Shade or a bank or a tunnel inlet/outlet can have very different traction. Static and dynamic friction. If we're talking L5 or even L4 capabilities, to my mind that doesn't mean idealized Arizona-driving conditions. It means L4 capability everywhere on Earth. And every video I have seen if full of people making wild predictions while ignoring just how dynamic and chaotic driving can be. Like 60 mph blizzard conditions with chains, one of which was accidentally cut in the wrong place. Was that in the data set? Maybe it will be eventually. But we're very far away. (My adviser says we'll never get there, which is an interesting statement. I disagree. Then again he's one of the smartest people I've ever met). n! is nasty if n is the number of possible objects in the world. b) I actually found while optimizing Matlab code that there are many situations in which a Search problem can be formulated as a maximization problem. And many times it's actually faster within such a scripting language. But my point is that there's definitely a relationship. E.g. "Search for an inflection point" equates to "argmin_x d/dt F(x)". It's a min versus argmin or max versus argmax type of distinction. Highly related. And they must needs have the same Big-O. c) There's a massive conservation of power constraint which is presently ignored in how people think about "AI". Intelligence means not being able to solve a particular problem, but potentially any novel problem. When people are assuming that if we can play checkers, chess, backgammon, sorry, connect 4 and a million other games better than human, then they're assuming that every game can be played at better than human capability. Maybe so, but there are a finite number of games. There are not a finite number of ways in which intelligence is applied. Let's talk cardinals of infinity shall we? We have bandwidth limitations on training and finite training data. No finite number of dots gets you every dot between 0 and 1 on the real line. We have object recognition but we have no scene understanding. No posited models / relationships / dynamics. Perhaps by the time a neural network masters language it will be able to find relationships between many dynamic entities in a dynamic world in real-time. Btw, I asked my adviser in grad school what he thought would be the most remarkable / impressive thing to come in the next few decades ahead. He said materials / manufacturing technologies, which surprised me. So this basically agrees with what Jim Keller said. We're clearly entering into a new frontier when we start being able to construct things (like SpaceX rocket nozzles) on an atomic scale via 3D printing. And what Jim said about the potential for reducing feature dimensions is amazing. Last of all, I wanted to make the point that most every chip is predominantly a 2D device with a couple dozen layers from what I understand. At some point we're going to break that mold and start working with truly 3D designs. And the same goes for neural networks. E.g. treating LIDAR data as a 2D image is pretty silly. We're making enormous performance sacrifices for treating a sparse thing data set as approximately dense. Many other interesting points on data quantization, significant figures, determinism, and the number 42 ;). I think people are badly failing, however, to anticipate what disruption novel technologies are going to have on our way of life, however. Elon Musk gets it. But if energy becomes free (or an ax+b situation where a is tiny and b is amortized to 0...), that fundamentally changes the entire civilization. The conversation about the evolving relationship between humans and technology has many, many implications beyond self-driving cars. It relates to how our bodies become more robotic. How we start working alongside robots. How we have advisory services / counselors / mentors / teachers and eventually bosses that are 'AI'. (I hate this term, it's an oxymoron). About telepresence via robot. About agency. About cybercrime and warfare. About ethics. About responsibility. About passive versus active roles in life. About turning the world into a zoo on the first day we decide we need to setup a MIMO control system to govern a climate gone insane.

This is so good.

Spectacular podcast. Period.

this guy is awesome....... wish he had more content out

Really great. Gonna check out the audio podcast.

Super informative guest 🤯

Great insight

"There are hundreds of billions galaxies...seems repetitive at best" loll

It's encouraging to hear that Moore's Law is still on track from an actual chip architect. Excellent interview.

This was a damn good watch.

given search space versus found search space ... such an elegant way to describe the distinction

Moore’s Law is always alive, dead, and going to die all at once. It’s the Schrödinger’s cat of laws.

What an interview

This makes me so excited to be studying electrical engineering

I did like learning from Keller’s perspective ... the heuristics of progress

@lex Friedman would be great to interview Jim Keller re RISC vs CISC and if RISC will prevail. Also interested which one will be more applicable to the field of AI?

These two are not always speaking the same language. Lex is imperturbable.

I love how he explains himself . What a mind

"Angry bird apps might be the whole point". Hopefully not.

Keller: "...a change in quantity is a change in kind". Mathematician: "A profound change in system dynamics from a change in scale is called _emergent behavior_"

Love it. The way Jim talks makes Michio sound like he works with crayons. As important as communication is, jargon literacy is required after a point.

Nice Video!

Lex, I'm sure you get a lot of advice from people so here's mine: you should release your short clips first or along with your long form interview. You can draw people in with the short and get them to listen to the long if interested. I always find that I see your long interviews and not sure I have the time for it but when I do they are always great. Maybe I'm wrong but I know I'd listen more this way.

Please: a link to the Rich Sutton paper?

Jim did not mention about thermal issues associated with further miniaturization. It may be a critical issue as heat gets more concentrated in smaller footprints. Less electrons to do the switching and rising ambient temperature make it more challenging to make reliable devices. Will it be the end of silicon and time for GaN?

Thanks to TSMC. If we were to rely on Intel we would have 14+++++ on 2025.

I have sort of an understanding of the convo but I just can’t lol ..I’m a huge fan lex u r role model

I loved when he asked Lex if he knows how CNN layers work, how they can detect ears, eyes of a cat. Sometimes you have to know your interviewer as well

His thoughts on value of abstraction in chip design reminded me of ideas from Yuval Noah Harari's book Sapiens. The ability to abstract may well be one of the most important steps in the evolution from apes brains to the human brain.

I wish I grew an appreciation for the beauty and art in science earlier in life.

Lex, don't argue with a guy that knows 100x more things about engineering computers that you do!

Loved listening to Mr. Keller challenge his host.

Hello sir, i saw you with Andrew NG. When that podcast is coming

So with modifications and abstract conceptual ideas to extend Moore's Law, Moores Law becomes a self-perpetuating sort of Mandelbrot Set only limited by time and technological capability to manifest it ?

Read Bernard Lonergan's Insight: 'Thoroughly understand what it is to understand, and not only will you understand the broad lines of all there is to be understood but also you will possess a fixed base, and invariant pattern, opening upon all further developments of understanding.'

Theorist meet practitioner.

Very interesting

25:22 So, a neural network is taking a complex data set and extracting a pattern from it. Now this pattern doesn't explain the whole of the data set or maybe even purpose of the data set. The neural network in our brain doing the same thing to the Universe. So, can our brain explain the purpose of the universe rather than just extracting patterns from it and calling them laws of physics.

He's the Chuck Norris of CPU design

22:05 Yes but they literally weren't in a closet all day. Cellphones are very easy to keep at hand all day.

I keep looking at the screw that is holding the whiteboard to the wall as if it was random debris on my monitor. I tried wiping it off nearly a dozen times, yet, it's still there... Guess, I got screwed...

25:00 hitchhikers guide to the galaxy anyone?

Lex schooled on this video 🤣

If I understood half of what I think I did, that was fascinating!

That last point was incredible. We're developing machines right now that are making calculations that we don't even fully understand.

16:40 "A difference in quantity is a difference in kind...". "Quantity has a quality all its own"...

This guy is a rockstar!

Super interesting

I find this fascinating. In 1980, when I was trying to get an overview of how chips were designed and produced, Moore's Law was predicted to end around 1995. A quarter century after that deadline, and with feature sizes now some 100 times smaller and transistors around the size of a smallish virus particle, your guest says Moore's law, perhaps in a somewhat different mode, is continuing apace. And it strikes me that the public need to understand the role played by the basic science in their way of living. As Keller said with respect to semiconductor manufacturing, "There's equipment, there's optics, there's chemistry, there's physics, there's material science, there's metallurgy... literally thousands of technologies involved." Meanwhile some segments of the population claim that science and expertise are unnecessary. It might help them to know that without, for example, the highly mathematical study of relativity and quantum mechanics, as pursued over the past 120 years, the airplane they travel on would never leave the ground. The cell phone they organize their life with could not possibly exist. The internet they get their news from could never have been developed. The MRI machine that helped save their life last year would never have been dreamed of. I see concrete technological achievements as playing a unique role in proving beyond a reasonable doubt that our lives depend on science and mathematics, including (among many others) medical, food and energy science. How skeptical can one remain about science while cruising in a passenger jet 10 km above the surface of the earth?

I love it. My field :-)

So basically there is the spatial variable along with a time variable for transistor production. It's almost like we live in a time space continuum or something.

10:11 Don't look at me mate, I ain't got a clue

The actual law is that the time to double is around two years. ie we look at the time to double, not the multiplication factor after two years, even tho there is an equivalence there.

I know a lot about this, but you know what, I'm a fucking moron when it comes to explaining or asking questions. Lex is brilliant!

I really enjoyed this talk. Is there a talk on quantum computing. No matter how much I try to understand quantum and I think "i got it 1 or 0 or anything in between". Then I realize i don't get it.

Nice talk. I miss smart people since postgraduate.

this guy is pretty smart, he should code.

This was a tough interview

Nice to know we can keep going. As to the universe being "wastefully large", well I think that's a value judgement. Why not think of it as a lovely large blank canvas?

I wander if it’s possible to create an asic for AI algorithms

Wow I was really surprised by Jim’s intelligence!

17:00 Lenin's favorite quote.

To tighten the vice of the microphone to your desk isn't a good idea at least not at this where you are sitting. Thank you for sharing

Jim Keller the John Wick of AMD without him AMD won't be where it is today

Moore's Law is the opposite of Fusion is will be gone in 30 years and always will be.

Optimization is absolutely a search.

If you let AI decide how to design a city properly, I think it would look very different from what we know

Lex getting rekt at 13:30 =)

Jim keller has eyes of steel! My man. But to the more important question what is it all for?, well good faith. On the behalf of evolutionary creation an expansion in human life. Control, but not tyranny!

Moore's Law was that "Transistor density doubles every 18 months". You can't just redefine it to say Moore's Law isn't dead, under the original definition it has been dead for well over a decade, when Intel stopped going from 386->486->Pentium->P2->P3->P4 and instead switched to core i5 etc was the point when Moore's Law died. Intel had to change it's marketing and branding to obfuscate the fact they could keep up with Moore's Law and keep a differentiated market place.

I have never heard of Jim Keller before he clearly knows his shit though. This is really interesting

Transistors are already not doubling every two years, the "innovations" part Jim is saying is the important thing. 3D stacking, chiplet design, L2 cache distribution, 4-way SMT, these are what pushes the industry forward now. I am excited to see Jim's Royal Core design. That looks to be Intel's last stand before prioritizing foundry

fascinating discuss. Moore's Law predicted the growth in the number of devices on a computer chip, a successive doubling per unit time. Gordon Moore said once that predicting exponential growth of anything into the future is risky, because after some number of generations you exceed the global capacity to support any construct at that scale. In today's cloud world, the growth of computing power has accelerated beyond what Moore predicted, mostly because the number of servers being installed in cloud data centers is growing so fast.