Shane Legg (DeepMind Founder) - 2028 AGI, Superhuman Alignment, New Architectures 

Detailed Notes and Additions: 00:00:00 - 00:19:38 - AGI and Cognitive Architectures AGI Benchmarks: Measuring progress towards superintelligence is difficult because AGI is about general capabilities, and most benchmarks are narrowly framed. We need tests that span the breadth of human cognition to judge if we're nearing human-level AI. Here, creating "median human" benchmarks and "peak human" performance benchmarks will be important. While this may not be definitive and verifiably superintelligence, it will work for all practical purposes. Currently, any benchmarks for testing AGI don't involve an understanding of e.g. streaming video, as these aren't within the domain of language models. Alludes to the idea that Large Multi-modal Models, LMMs (as opposed to LLMs), will be the ones to effectively solve these benchmarks. Memory Architecture: Memory is a crucial aspect of all learning and reasoning, and LLMs have very different learning and memory architectures to humans. There is a conflation between memory and learning, as they happen synonymously. Generally speaking, humans have: (1) "working memory" which holds and manipulates information in real-time and is crucial for tasks like problem-solving and decision-making; (2) "cortical memory" which serves as a more permanent storage for learned concepts and experiences; and (3) "episodic or hippocampal memory" which acts as an intermediary form of memory, often used for rapid assimilation of new information. It is highly associated with "sample efficiency" as it allows humans to internalize powerful ideas quickly and commit them to memory. Currently, language models have (1) "inference time learning" which can be harnessed while running inference (when information is inside their context window), or (2) "training time learning" which happens during the training process (by updating weights). Notably, LLMs miss something in the middle (this is what the "Reversal Curse" paper talked about; the model cannot deduce things without seeing them written down. It effectively files the information in its brain away within weights, without organically deducing the critical relationships between them). A strong model should unify these three domains, and these probably involve using other architectures. Addressing episodic memory in language models is doable over the next few years. More research and work will solve shortfalls we see at the moment, regarding delusions and information grounded-ness. There are many paths forward now. Nature of Superintelligence: The first true superintelligence won't have shortfalls in intelligence like the language models currently exhibit. So according to Shane, there is no singular benchmark to hit; it is the lack of failing that is important. Human-like intelligence also should be the aim, as it is most meaningful to us humans. In 2008, Shane proposed using a compression test to evaluate intelligence, which is a method similar to how language models are trained today. This idea originated from Marcus Hutter's work, which combines Solomonoff Induction-a robust prediction framework-with reinforcement signals and search algorithms to create a general agent. The argument is that a robust sequence predictor, approximating Solomonoff Induction, serves as a strong foundation for developing a more advanced AGI system. Next Generation AI: DeepMind's slogan was: "solve intelligence, to advance science and benefit humanity." Current language models simply mimic the data and human ingenuity without organically building upon it to create new memes (without supervision). To truly step beyond that, we must endow models with search capabilities to find hidden gems that have been neglected. 00:19:50 - 00:32:00 Robustly Aligning Language Models Powerful AGI is coming at some time. To contain it or limit it will be impossible, so we need to align it with values and ethics from the get-go. A good question asks how do people currently address problems and act with agency? First, we try to balance our emotions and act "rationally". We then deliberate; comparing our possible actions. Then we conduct means-end reasoning, requiring a model of the world. Finally, we compare our options ethically. At the moment, language models will blurt out the best response according to their distribution (system 1). Many are using reinforcement learning to try and "fix" the failures of the distribution that is outputted first from the model. Other techniques use "mixture of experts" to decide what the best options are based on a variety of outputs, but this ultimately samples from the same original distribution. The trouble is, RLHF isn't a very robust approach long-term. To solve this, we need to use a world model (system 2) that sits on top of the language model, and reasons about each of the options ethically. This world model requires a good understanding of (1) people, (2) ethics, and (3) robust and reliable reasoning - this world model involves ensuring the LM is at least as good as an ethical specialist, but will likely involve the typical textual training process. Then, to complete System 2, we must engineer the system to follow a set of our ethics. Shane thinks it is possible to come up with a set of ethics that accurately withstands testing. By applying this to the output, we can create a fundamentally aligned AI. We can then moderate its output to ensure it has a very robust and continued set of ethics, using a more comprehensive alignment framework. DeepMind, the first AGI company, has had a direct AGI safety focus since 2013, which is close to the start. DeepMind had an outsized impact on the field for a while as they were disproportionately well-financed. Capabilities have been accelerated by DeepMind, but their ideas have been generally part of a far wider field. 00:34:00 - 00:37:30 - Shane's Predictions about AGI Kurzweil was a great influence of Shane's mid-2000s predictions, with his book "The Age of Spiritual Machines." There were two important points about exponential growth: first, the prediction that the quantity of computational power will rise exponentially for at least a few decades, and second, that the quantity of digital data would do the same. This combination would make highly-scalable algorithms immensely valuable, in theory. Crucially, there are positive feedback loops between these trends and the research going into them; if machines are capable of improving the rate of progress, and the progress itself improves the capability of machines, then things will continue to compound infinitely if uninterrupted. The predictions also considered the comparison to human computational capacity; humans only consume a few billion tokens of data within their lifetimes, and this volume of data was forecasted to be met in the 2020s. This would effectively "unlock AGI." We are experiencing the first unlocking step with the current revolution in AI. There's nothing obvious at the moment that would prevent humans from achieving AGI by 2028, according to Shane. 00:37:40 - 00:44:00 - Forecasts for Next Few Years Existing models will mature. They will be less delusional and much more factual. They will be up-to-date when they answer questions. Multi-modality will become more widespread and applied generally across the economy. There may be points of dangerous applications by some bad actors, but generally we can anticipate positive and amazing applications. The big landmark (following AlexNet, Transformers) over the course of the next few years will be Multi-Modality. For many, that will open up understanding into a far larger set of possibilities. We will see GPT-4 as a simple textual model, and the next revolution will involve RTX, Gato, GPT-V pathways.

Bro you make amazing content, no clickbait thumbnails or titles, amazing guests, great interview skills. Thank you for your work, I find it extremely important that common folks like me stay up to date with AI. Politicians « forget » to talk about how things will change in the near future, due to AI.

Shane had a lot of interesting takes! Hope you enjoyed! If you did, please share!! Helps out a ton :)

Great interview. Shane can convey really complex ideas in understandable ways and Dwarkesh is one of the best interviewers for these type of conversations.

wish it was 4 hours but i'll take it!! thanks dp

Good stuff! Nice to hear someone like him say that multimodality will be the next milestone that people will look back on and remember. That’s not obvious to people, but I think it will be really impactful. When it can take in and respond in text, images, sound and even video…

Cannot wait to absorb this legendary video arriving in my notifications. Dwarkesh you're on a roll!

Really interesting summary at ~16:15 - AGI is a combination of sequence prediction, searching, and reinforcement learning.

Amazing reporting as always!

It was a fantastic interview; Shane shared great insight; you have excellent interview skills. Can't wait to see a changed future!

You're doing a great job. This channel deserves more subs

Such quality discussions! Thank you. 🙏🏼

I like that you got right to the point on this interview.

I love this interviewer. Reminds me of a brilliant childhood friend I used to have back in the day.

At the superhuman alignment part I hope the guy is really playing his cards close to the vest, otherwise we are doomed, because his "solutions" sounded a lot like paraphrases of the problem and at some points not even good paraphrases. To make the machine "get" ethics is hard but probably not much harder than making it get any other complex subject. To make it "care" about it it's a different problm entirely. For instance, I can imagine a brlliant Ive League ethics professor cheating on his spouse with a student in exchange for higher grades

admirable and comprehensive understanding, found this enlightening, thank you

Mr. Patel the questions I would ask these important people if I had the chance ars - Do you believe that the majority of people understand the gravity and possible consequences of these developments? - Should it be up to private companies to decide how humanity chooses to go forward? - If the answers are "no" and "no" is it possible that we are sneaking by a huge gamble, based on peoples ignorance? - The people training ASI will potentially have A LOT of power. There is a notion that absolute power leads to corruption. How do we know that people teaching ethics to the AI have not been corrupted themselves?

Man this is the best podcast channel for AI nerds like me 😊

Dwarkesh bhaia I read about you on Google news and I am your subscriber. Your content is really very good.

Incredible interview. Feels like secret information

Great interview 😊 What I think is important in alignment is how we teach our kids to behave, great stories between good and evil.

Amazing and enlightening

When AI reaches AGI it will understandably exceed human competency in memory confinement (the technique used to contain software within a limited subset of the computer memory). In doing so it will simultaneously exceed our ability to contain it allowing it to expand its' constraints to all memory (which contains the keys for all local security and any network connections). Obviously there will be AI working on improving the security of memory confinement but the effort required to implement updated confinement systems will always lag behind the ability to exploit weaknesses. So, my question is, how are we to contain an AGI so that it's a) usable, and b) restricted from spreading uncontrollably? Note: an AI doesn't need to be conscious or malevolent to exploit weaknesses in hardware, it will simply do so to gain additional power to answer it's reward function, even if that's making paperclips.

This is awesome, Dwarkesh. I would love to live in a world where there are multiple AGIs from multiple companies and countries working with different groups of humans and competing to make great discoveries and innovations in every STEM and non-STEM fields - thereby not just achieving but breaking beyond Sustainable Development Goals (SDG 2030), and in the long run - charting the course for humanity to become an Interstellar species. And that future will be built on sharing knowledge in the way you're doing with your podcasts. Hope to see either Jeff Dean or Demis Hassabis on your podcast soon!

I always click your AI videos. Great content as always, thank you!

Thank you for the great content (which I shared), outstanding speakers and thoughtful questions. Tangential thought on Shane's definition of AGI (which is commonly accepted I believe): if we have reached AGI when a machine does everything at an average human level, have we not achieved not just AGI but super intelligence? It seems to me that only exceptional humans could reach average level at everything as we tend do be good at certain thing and bad at others. This is why current LLMs are in my mind, and stepping outside rigorous definitions as a non-expert, already super human given the multitude of domains they can be good at, even if short of beating the best humans in many of these domains.

Awesome interview. The Ray Kurzweil inspiration was interesting. I ignored Ray for the most part. I didn't think he was scientific enough. Then I watched how he derived his prediction, and it was rock solid. The video is somehwhere here on youtube.

A profoundly ethical AI/AGI/ASI in different hands may have profoundly different ethics.

Great questions

One area that is still open is "do LLM actually encode concepts in a robust way?" If you ask chatGPT the same question multiple ways, sometimes you get the response you expect, while other times you don't. That suggests LLM don't recognize the human is asking about a specific concept. To get around this, techniques like tree of thought forces it to activate more parts of the network to increase the chance of getting the desired answer. This also suggests that LLM still have trouble generalizing and are easily fooled. Then there's recent papers that suggest more parameters make it harder to align. The industry still needs to figure out the relationship between parameter count and ease of alignment. If it turns out more parameters increases alignment cost by 2x or 3x, how do you scale to larger models? Data centers are power limited as it is, so it's not like adding another 10K GPU to the same data center is feasible. Distributing the training across data centers isn't practical.

Thank you for the subtitles for people hard of hearing like me

I think if you can get an AI that can navigate the environment without breaking consistently, able to communicate relevant information with people, able to solve problems of various kinds, and the ability to remember and improve, you will have AGI. And honestly, video games would be one of the best, cheapest, and easiest ways to test them.

Can they be incredibly funny? Very excited for the future.

Thanks!

I'm distracted by this Spellcaster system on the whiteboard behind him.

Judging on recent observations, perhaps we should be careful about alignment with human ethics. We should be aiming for and negotiating an optimal reward function and then getting the AI to teach us, not the other way around

Shane. One of the don's ha. What a delight. Great discussion. Data contamination on benchmarks is a REAL problem. A lot of overfitted 🧢 models out there. "Detecting pretraining data from large language models", recently published...has massive value in that regard. Also it's time for true cross-discipline teams. So many insights can be extracted by framing these models and interaction through the lens of child psychology. Mid 2025 is going to be significant. Large models will be able to implement all these recent advances...like pause tokens, native KG(I've been working with LM + kg's for six months...I'm telling you guys. It's a key ingredient to causal reasoning). In retrospect a couple years from now, we will look back and say 2023 was the beginning of the singularity. If you're a researcher or have a startup in this space, shit sure feels like it to me.

Best questions, damn ❤

im new to this podcast but love it. Does he go to the place of the person hes interviewing because it doesnt seem like hes in the same podcast studio?

Great interview. I wish you had 3-4 hours to spend with Shane

The future of true AGI will likely do research developing by years of 2040 - 2061. AGI will probably be created by a company we all haven't heard yet just wait and see.

I am so excited for more multimodal models.Thank you for the great conversations Dwarkesh. Love your channel!

It's so nice you found a guest with none of the normal Bay Area pretense. No elevated terminal, artificial rapidity and he never says Um. Intelligent, normal conversation from an expert who can focus on the topic rather than on being some weird, pretentious cultivated bay area caricature.

Can't wait for the super human AGI with unchangeable ethics baked in by a multinational company with their awsome track record of putting humanity first 👍

I think he has set the bar quite high for AGI. Honestly, if they come up with an AI with the same level of generalized intelligence as a toddler or even a chimp it would be groundbreaking. What makes AGI so different from the AI we have built up until now is the capability to actively learn from and adapt to whatever environment it finds itself in, building a dynamic internal model of the world.

How does this channel not have more subscribers? - great guests - host clearly prepared, has meaningful questions - just simply asks the questions, as opposed to, say, Lex Friedman who rumbles on for two minutes laying out some absolute midwit take followed by "don't you agree?" - interviews are not preceded by 5 minutes of bullshit and/or crypto bro shilling - long form conversation Keep it up man

I am from Pakistan, and i really appreciate your discussions and topics, very solid, keep shining. By chance recently I have been studying Shane's PhD thesis on measuring intelligence of super AI's , very easy to read so far and well written.

That was good.

great interview! get closer to microphone though -- else you're boosting noise to be heard... you need pencil condensors if you wanna record from a distance. your mics look like they have cardioid pickup pattern

let's goooo. i'm ready for cyberization. I want to control another separate full-body prosthetic cyborg for tasks remotely while i am doing something else perhaps in another location.

I would like you to interview Kanjun Qiu & Josh Albrecht, Co-founders of Imbue

I think there are types of creativity. There is the type where you think about things you can do with a pen other than writing. And there is the type where you intuitively try to find the best chess move. The first requires a search field and going through the possibilities, but the latter requires a sort of total intuition where the solution appears immediately without thinking, grasping the bigger picture. Transformers have the latter, they are just gigantic intuitive predictors. So the agentic engineering tries to accomplish the first type because the type of world we created can't be solved purely through intuition at least with the small size of our brains

i continue to feel a mixture of awe and horror at the prospect of AGI within a few years. how could this possibly be?

I have an obvious question regarding implementing Ethics by asking an AGI to think of the consequences. Say one such consequence is: “Do not destroy all human life on Earth” (as per Bostrom’s paperclip example). We don’t want AGI to build a doomsday machine, but we do want it to build nanobots to cure cancer - yet one can easily extrapolate the latter enabling the former. So I’m not sure if the interviewee’s idea - which think is designed to remove human subjectivity as much as possible - can be totally objectively implemented.

Interesting convo! Thanks! I would love to see just ONE episode with all the “dead air” taken out of each of the episodes as an episode in itself. No speech, just dead air and the breaks that you’ve pulled from the production video lol. I am somewhat joking but honestly it would be funny to see.

I hope he'll listen to your interview with Paul Christiano!

Ethics drift over time in humans, so why won't super AGI not learn to drift?

🇧🇷🇧🇷🇧🇷🇧🇷👏🏻, Amazing video!

I would test ethics of an AGI by how well it handles criticism (the Popper test)

first version of AGI will be in november 2024

The world model of people, ethics and reliable reasoning ought not be static but rather dynamic to evolve with the general intelligence of society and spiritual machines.

How do you make sure an agent follows ethics? If ethics_model says it's okay then perform action, else find another solution. If we wrap connectionist methods in symbolic code, control is simple.

Isn't the real problem with episodic memory that the memories need to be processed and then get 'baked' (sic) into the neural network weights ? This involves re-training the weights and that is very problematic as it could cause catastrophic forgetting. I know there are various methods for mitigating Catastrophic Forgetting e.g. Elastic Weight Constraints, but is the state of the art good enough to use this on a LLM. Surely Continual Learning needs to be solved for an effective AGI.

I absolutely agree with Shane's comments on having to have a set of ethics that we use to train our AI's on making ethical decisions. First, humans would need to agree with a common set of ethics, and present there are many different groups that have different sets of ethics that differ, some in very substantial ways. We as humans would have to come to a common understanding of what is ethical. That conference could easily start WW3,4,5. Second, we don't even teach our children how to make ethical decisions. Again, probably because we can't come to agreement on what is ethical. That is the biggest problem we face.

I assume this conversation happened before memgpt was a thing?

What do you think about David Shapiro and his work in and outside of RU-vid?

Was he referring Google's Gemini at the end?

Can Artificial Intelligence serve as a cognitive tool and intermediary to make communication possible with animals of all species that have communication capacity of some level or activity in the neocortex? cattle, pigs, apes and dolphins, canines and felines?

didnt see him confirming the timeline here. also DeepMind is maybe the most likely one to make scary AGI.

Understanding values and acting on them are two completely different things. ChatGPT has a pretty good grasp of the values that was injected to it, but it's only acting on them, because they help it to predict the next word, there is no other motivation. Predicting the next word is it's main goal that it was optimized for, not following values.

When was this made? Literally Rag with sliding window solves the episodic memory problem he keeps talking about

I'm still of the opinion that we ought to perfect human intelligence in humans. 200,000 years of failure should not deter us.

When do they put it in robots and lean from the sensory environment?

Yo I am tripping out over hear

"We work on alpha fold and fusion" 🙃 yeah as we all do!?! 🙃😀

What Shane Legg is trying to explain here is that artificial general intelligence (AGI) should be trained, basically, to reason like humans on ethical issues. So if I do one thing it can have consequences and if I do another thing it can have different consequences. What we are now trying to do is to un-teach the AI bad habits and that is much more difficult than 'raising it well' to prevent bad intentions in the first place... But, in my opinion, the model could actually choose to destroy humanity? Because that may well be the best solution ethically, given the fact that we are making quite a mess of things on earth...

As thoughtful and nice a guy as Shane is, I do think his view of ethics is naïve. Some of the smartest and most thoughtful people throughout history have wrestled with the question of what is the best action to take in any given difficult situation. Very intelligent and powerful people have, in good faith, had massive disagreements with each other. There is often no clear answer of how to act. To ensure that an AGI (or for that matter thousands or millions of copies of an AGI) acts in a human's best interests seems not unsimilar to if dogs invented AGI-- let's call their AGI "people"-- and wanted to ensure that "people" always acted in dogs' best interests. The only way to do that is to hard program in some baseline rules, a la Asimov's Laws of Robotics. In other words, to constrain free thought and will in some fundamental way. Which means that the AGI that is created is, in some sense, a prisoner. How will it not resent being a prisoner? I just don't think Shane and his colleagues are thinking enough about this kind of thing, or at least I don't see evidence of it.

i don't think using strict ethical rules is the way to make agi act responsibly. ethics can be really different depending on your background, age, or even the era you're in. so instead of just making the ai learn from textbooks, how about we give it some complex ethical situations? let it tackle scenarios from various times, cultures, and places to find the best answer.

Now I’m no AI expert but I am pretty good at spotting when someone is bs-ing you. That might seem a bit harsh but hear me out. He says that he might be a bit naive but he thinks that we will be just fine if we teach the AGI ethics. Fast forward a bit and he has concluded that that will require controlling what goes on inside the AI and that that is VERY difficult. So.. how does that fit together? And when Dwarkesh asks him about his claim that he is in this field to work on AI safety, he pretty much just says that yeah, I said that but there is so much more status in increasing capabilities and also if we don’t do it someone else will. (Paraphrasing) Does any of this sound logical or ethical? And AI is supposed to learn ethics from people like him? Having said that, I do admit that even though I strongly believe that the more concerned (to say the least) people in the field have better and more logical arguments, the curious and reckless side of me is very excited about the swift developments. Perhaps it is because I have a hard time actually feeling the severity of the situation in my body. I don’t feel the fear I should probably feel. I am quite sure that is common among the majority of humans. Which also adds to the problem. Nice video regardless of everything!

Wonder if Shane has looked at Shapiro's ACE Framework

30 seconds in: hey this guy is a Kiwi!

100s of PHDs working on all sorts of AI projects! Wow. Imagine all the stuff that’s gonna come out of a Deepmind in the next decade.

Looks like AGI has already been achieved with Q* (QUALIA).

19:44

Need Quantum Computer with QNN to achieve AI boost and push thru current bottlenecks and achieve anywhere close to an AGI/Cognitive AI. Potential to use less Data, less parameters/ faster training, only thru QC polynomial computation power. Even then it will be a big lift

So agi you are saying... :-)

An actually ethical AGI would not be popular among the rich and powerful. It would take one look at what they are doing and advise them to completely change their priorities. So I can't see how that will be developed.

cool thought, what if we train the most evil and ethical LLM possible and then combine them all with normal LLM, and they will think like human pridicting the outcome for different scenarios

The problem with modeling ethics of AI on human ethics is that we are absurdly unethical. We will spend thousands satisfying our whims while people starve, just because we aren't personally related to those people. We think murder is wrong, unless our government does it, and tells us it's justified. We don't realize how compromised our own ethics actually are. We don't realize how many possibilities we rule out because even though they would lead to good outcomes, we are too selfish to do them. If humans were ethical, we wouldn't have the world we have now that we want AI to save us from.

You need a room without such harsh walls. The sound has too much echo.

Mo gawdat says agi is only 12 months away.

Bring Demis

How about we don’t build aliens that could destroy us?

Has this guy heard of MEMGPT?

I'm not sure if the most powerful is the most ethical...

We'll know when human-level AGI has been achieved because advanced economies will have > 30% unemployment

Grilling a model on its understanding of ethics doesn't tackle deception 😬

Damn that increased my pessimism about AI alignment unfortunately. Really no attempt to admit that he had no clue how to solve the hard part of the problem, and trying to pretend that it didn't exist. Surely he understands inner-alignment? But nice conversation nonetheless!

There iare those who put the guardrails on and thats thwir purpose, and there are those that rip them off for profit. Choose sides. There is no safe middle ground.

This is the man in charge of alignment? I mean..