Transformers: The best idea in AI | Andrej Karpathy and Lex Fridman 

Full podcast episode: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-cdiD-9MMpb0.html Lex Fridman podcast channel: ru-vid.com Guest bio: Andrej Karpathy is a legendary AI researcher, engineer, and educator. He's the former director of AI at Tesla, a founding member of OpenAI, and an educator at Stanford.

Andrej speaks at 1.5x speed and Lex, as always, at 3/4x. Yet, somehow they understand each other.

Why Lex doesn't invite actual inventor of Transformers, e.g. Ashish Vaswani? All these people like Sam Altman, Andrej Karpathy are reaping the harvest of the invention by that paper "Attention is all we need", yet they are not invited even once to Lex talks.

Damn. That last sentence. Transformers are so resilient that they haven't been touched in the past *FIVE YEARS* of AI! I don't think that idea can ever be overstated given how fast this thing is accelerating...

Optimus Prime would be proud.

Amazing how one paper can change the course of humanity. I like that kind of return on investment, let’s get more weird ambitious.

I double checked if i was listening at 1.25x speed when Andrej was speaking

Need a phd in AI before I can understand what he is saying

6:30 😂 imagine how fast this sounds to lex

2:18 Meme your way to greatness

The attention name was surrounding there in the past on other different architectures. It was common to see recurrent bidirectional neural networks with "attention" on the encoder side. That's why the name "attention is all you need" comes from. That because it basically deletes the need of a recurrent or sequentially architecture.

It's amazing to have a podcast where the host can hold their own with Kanye West in a manic state and also have serious conversations about state-of-the-art deep learning architectures. Lex is one of one.

Dr. Ashish Vaswani is a pioneer and nobody is talking about him. He is a scientist from Google Brain and the first author of the paper that introduced TANSFORMERS, and that is the backbone of all other recent models.

My professor Dr Sageeve Oore gave a very good intuition about residual connection. He told me that residual connections allow a network to learn the simplest possible function. No matter how many complex layer we start by learning a linear function and the complex layers add in non linearity as needed to learn true function. A fascinating advantage to this connection is that it provides great generalisation. ( Dont know why, I just felt the need to share this)

The way this guy thinks and speaks reminds me of Vitalik Buterin. What do they have in common? High intelligence is not the only factor here.

Karpathy has some great insights. Transformers seem to solve the NN architecture problem without hyper parameter tuning. The "next" for transformers is going to be neurosymbolic computing i.e. integrating logic with neural processing. Right now transformers have trouble with deep reasoning. Its remarkable that reasoning processing automatically arises in transformers based on pretext structure. I believe there is a deeper concept of AI waiting to be discovered. If the mechanism for auto-generated logic pathways in transformers could be discovered, then that could be scaled up to produce general AI.

Self-attention. Transforming. It's all about giving the AI more parameters to optimize what are important internal representations of the interconnections between data itself. We've supplied first order interconnections. What about second order? Third... or is that expected to be covered in the sliding window technique itself? It would seem the more early representations we can add the greater we can couple to "the data" complex/nuance. At the other end, the more we couple to the output, the closer to alignment we can achieve. But input/output are fuzzy concepts in a sliding window technique. There is no temporal component to the information. The information is represented by large "thinking spaces" of word connections. It's somewhere between a CNN like technique to parse certain subsections of the entire thing at once, to a fully connected space between all the inputs. That said sliding is convenient as it removes the hard limit of what can be generated and makes for an easy to understand parameter we can increase at fairly small cost our increase our ability to generate long form representations exhibiting deeper level nuance/accuracy. The ability to just change the size of the window and have the network adjust seems a fairly nice way to flexibly scale the models, though there is a "cost" to moving around IE: network stability meaning you can only scale up or down so much at a time to maintain the most knowledge incurred from previous trainings. Anyway, the key ingredient is, we purposefully encode the spatial information (to the words theme-selves) to the depth we desire. Or at least that's a possible extension. The next question of course is which areas of representation can we supply more data that easily encodes within the mathematics of information we think is important to be represented in the information (that isn't covered by the processes of the system itself (having the same thing represented in multiple ways (IE: the Data + the system) ) is a path to overly-complicated systems in terms of 'growth/addendums". The easiest path is to just represent in the data itself. And patch it. But you can do stages of processing/filtering along multiple fronts and incorporate them into a larger model more easily, as long as the encodings are compatible (which I imagine will most greatly affect the growth of these systems/swapability though standardized ). Ideally this is information that is further self-represented within the data itself. FTT are a great approximations we can use to bridge continuous vs discrete knowledge. Though calculating it on word encodings feels a poor fit, we could break the "data signal" into an individual chosen subset of wavelengths. Note this doesn't not help in the next word prediction "component" of the data representation, but is a past knowledge based encoding that can be used in unison with the spatial/self-attention and parser encoding to represent the info (I'm actually not sure of the balance between spatial and self-attention except that the importance of the token in the generation of each word to the previous word (along with a possibly a higher order of inter-connections between the tokens) is contained within the input stream). If it is higher order than FFT's may already be represented and I've talked myself in a circle. I wonder what results dropouts tied to categorization would yield on the swap-ability of different components between systems? Or the ability to turn various bits/n/bobs on/off in a way tied to the data? I think that's how one can understand the partial derivative reverse flow loss functions as well, by turning off all but one path at a time to split the parts considered, but that depends on the loss function being used. I imagine categorization of subsections of data to then spit off into distinct areas would allow for finer control on representations of subsystems to increase scorability on specific test without affecting other testing areas as much. Could be antithetical to AGI style understanding, but it allow for field specific interpretation of information in a sense. Heck. What if we encoded each word as their dictionary definitions?

In a rarity for this channel, the title approaches clickbait. I will cede the point that it's 100% accurate to the conversation, but I'm bitterly disappointed that I didn't get to hear expert opinion about benevolent, shapeshifting AI platforms protecting sentient organics from the Megatrons of the universe.

“Attention is all you need” is great. It’s like a book title that you can’t forget.

1:56 “I don’t think anyone used that kind of title before, right?” Well, maybe not as a title, but I can’t imagine that the authors of the paper were unaware of the lyric “Love is all you need” from The Beatles’ 1967 song “All You Need is Love.”

5:05 he lost me after series of blocks

Incredibly interesting and thought provoking. But still disappointed this wasn't about Optimus Prime

Andrej's influence on the development of the field is so underrated. He's not only actively contributing academically (i.e. through research and co-founding OpenAI), but he also communicates ideas so well to the public (for free, by the way) that, he not only helps others contribute academically to the field, but also encourages many people to get into it simply because he manages to take an overwhelmingly complex (at least for me it used to be) topic such as the Transformer and strips it down to something that can be (more easily) digested. Or maybe that's just me - as my professor in my undergrad came no where near to an explanation of Transformers that it as good and intuitive as Andrej's videos do (don't get me wrong, [most] professors know their stuff very well, but Andrej is just on a whole other level).

Yes the TF is great, however: 1- Next word prediction only gets the most popular ideas, not the right ones. 2- Hilton wants to improve on Back Prop. 3- Hallucinations in = Hallucinations out. Attribute that to Plato. 4- Will it get lost on real world data - the info in matter? 5- How bad is the compute wall? Factorial? N**4 does not consider parameters and data that must increase. 6- How will the measurement problem and non locality in physics affect AI? 7- does the entropy of physics eat your models alive if you dont engineer them perfectly?

- Understanding the Transformer architecture (0:28) - Recognizing the convergence of different neural network architectures towards Transformers for multiple sensory modalities (0:38) - Appreciating the Transformer's efficiency on modern hardware (0:57) - Reflecting on the paper's title and its meme-like quality (1:58) - Considering the expressive, optimizable, and efficient nature of Transformers (2:42) - Discussing the learning process of short algorithms in Transformers and the stability of the architecture (4:56) - Contemplating future discoveries and improvements in Transformers (7:38)

Introduce myself my name is Ariful Islam leeton im software developers And developer open AI

Great interview. Engaging and dynamic. Thank you.

Best idea seems to be to make sure the robot needs to be plugged in so it can't chase you

The model YOLO (you only look once) is also an example of a "meme title"

I know one of the author's of Attention is All you Need at a personal level. Even that person is rather cagey about talking about its impact on the field. I would agree - I don't think they collectively knew just what this was going to do to the industry.

I read the paper and was wondering Transformer is another kind of a LLM for generative tasks as they mentioned it as a model and also compared with other models at the last of the paper but finally after watching this explanation by Andrej i understood it is a kind of an architecture that learns the relationship between each sequence

We as a field are stuck in this local maxima called Transformers (for now!)

I looked up some of the words he said. I still have no idea what he is talking about.

Pay this man his money (rounders)

Ah, so that's why they called their bot Optimus.

well, this clip was great until it ended abruptly in mid-sentence. PLEASE! I signed up for clips, but I am frustrated to have missed a punchline here from Karpathy explaining how GPT architecture has been held back for 5 years. I have SO little time to now scan the full version! But at least you linked it. Your sponsor has left me sleepless.

And he's known for explaining stuff in simple terms...

I had to check a few times if my video speed was not set to 1.5x

Question: without the optimization algorithms designing the hardware manufacturing is there reason to believe that the fundamental nature of these mechanisms reflect the inherent medium of computation? Nope. I guess not. They're watching it.

Bro made me double check if my playback speed was still at 1x

Saving that one to see if I can understand wth Andrej is talking in a year.

What is the best idea in AI and why is it transformers?

The weird thing about transformers is that it's just so random. There is huge space of potential architectures, but it's not clear why transformers are so good

Transformers is not so good a movie

DR ILYA IS TRANSFORMING CULTURE SOCIETY & WORLD ! ?

"English! Do you speak it?!"

Great short video. What are the some of the limitations of the transformer architecture you can think of?

Recurrence all you need.

Most people can Walk academically-But-ILYA IS “RUNNING” INTELLECTUALLY ! ?

Recall Football; SOMETIMES ONE NEEDS TO GO BACK A LITTLE TO RUN ALL THE WAY WAY FORWARD ! ?

My fav Lex clip byfar

Dr ILYA; consider that it’s not Altman- it’s “FAULT-MAN” ! ?

5:04 while the effort is appreciated, I don't think the blocks analogy simplifies things enough to help the layman

What absolute rubbish!! Transformers are either Autobots or Decepticons. I don’t know what he was talking about but get a clue dude!

Someone should tell Altman; “IF YOU CANT RUN WITH BIG DOGS STAY ON PORCH”

What is meant by "making the.evaluation much bigger"? I do not understand "evaluation" in this context.

"Don't touch the transformer"... good advise regardless of what kind of transformer you're taking about.

@1:15 Well they were probably AWARE since they named the paper : "Attention is all you need" 🤭

4:41 I think his brain hallucinated here. Because sequential and serial is the same thing. I think he meant to say "in parallel".

I concur

Altman TALKS-But-ILYA IS THE ONLY ONE “SAYING” ANYTHING ! ?

If one gave Dr ILYA an Enima you would get an ALTMAN ! ?

Did anyone learn about transformers recently and can recommend a video that made it easy for them? I'm quite new to ML and would appreciate recommendations.

Minuto 6:31 tu cara cuando tu amiga te dice que hizo 25mil fotogramas para Ghibli en una semana 😄

If I understand correctly, transformers = good. N'est pa?

Best explanation of the essence of Transformer architecture. I think the title is a red herring because it makes it more difficult to understand. You need much more than Attention, you need all the smart tweaks. And it keep making my mind think of Megatron from the movies, and I’m not sure what if any is the relationship. I like the generalized differentiable program, as the best description of a Transformer Model today. But that could change. The description is from Yuan LeCun in 2017-19 time period. Jennifer

Attention is all you need? No, 🎼🎶All You Need Is Love 🎶

How many of you saw the entire video and understood nothing?? 😂😂😂

So the last 5 years in AI are "just" dataset size changes while keeping the original Trasformer Architecture (almost) ?

Does any one have a simple definition for “general differentiable computer”?

Coolness

I loved them as well but after the first 2 films it really got boring

This is truly amazing. Many many thanks 😊

When he says the transformer is a general-purpose differential computer does he mean in the sense that it is Turing complete?

Do a podcast with him and George at the same time

How many people thought they were going to talk about Megatron? Lmao

It’s self attention ⚠️

...i am dissapointed it's not about optimus prime

Lex is the translator between genius ideas and us normal folk

I like your funny words magic man

Any guess why Vadswani is ignored?

Idk this

Maye the force

and the greatest thing about it is that you do not need it

Lex Fridman, you seem bored and uninterested. Holding your head up with your hand. You have Andrej in front of you, be professional. ;-)

Long time no see badmephisto!

Attention is truly all you need.

I don't think so 😕

Robots in disguise....

What?

The paper is called "Attention Is All You Need", and IMHO attention is what made transformers so successful, not its application in the transformer architecture.

This guy is semi-Elon’s brain but articulate

"Once you understand the way broadly, you can see it in all things." -- This could be the name for the AGI paper, whoever will write that. Just give a credit for "AGI technology enthusiast TechPiller in a RU-vid comment". :)

This makes me slightly freaked out that we don’t really understand what we’re developing….

Is this guy a robot...

I studied neural networks in the 1990's as part of my undergrad degree in AI. I did a paper on simulated annealing... anyway there's been a lot of "progress" since then, but also unfortunately a lot of drivel written and bollocks spoken. I might trust an ANN bot to hoover my carpet, within reason, but I wouldn't trust it to make me a cup of tea. It's not clear to me I'll ever be able to either. Something important is missing...

i know absolutely nothing about transformer architecture or ai. only recently did i find these clips and they are very interesting to watch. i immedietely recognize the arrangement and flow in the transformer model architecture as a fractal design present in nature already. it was an instant reaction i had to oberserving the transformer model architecture. this "recognition" of the design is likely only due to human pattern recognitions trying to convince me i am seeing similar shapes, but interesting none the less when you look at it this way. not sure if this random youtube comment helps anyone or anyone will see it. im speaking from a purely basic understanding and google searchs lol fun video to watch.

Optimizing NNs based on just the output layer always never really made sense to me, it's really cool knowing the transformer has solved that!

Any top AI expert that is not a member of the tribe?

I think human brain has 2 mode of working : an efficient mode (non Turing complete) and an expensive one (Turing complete) As a simplified illustration, there are 2 ways of doing a simple multiplication : memorising it, or running the multiplication and counting on your finger. Our brain is capable of both, and transformers are probably only able to do it the non Turing complete way. It stills produce impressive results, which teaches something ABT how much seemingly complex reasoning we can produce just by memorizing / interpolating data. But it has a fundamental limitation compared to the turing complete way of reasoning (which is why chat gpt fails pretty fast with simple math problems) (This is just my take on the subject and I obviously have a big respect/admiration for what karpathy achieved with chat GPT and am excited to see the way it will evolve)