Superintelligence | Nick Bostrom | Talks at Google 

I love how the random guy in the crowd is Ray Kurzweil asking a question.

First of all, great talk stretching the mind to think through far more deeply. What I observed was the strength of his argument is not how likely the superintelligence-turning-rogue but how severe, sudden, uncontrollable it could be, so we better prepare for it. To that end, every time a questioner questions the assumption he hurriedly and very cleverly quickly sheds aside that question and pursues full steam the 'threat'. Take my following note as a genunine complement - he reminded me the tone of my mother who got us all to do homework by scaring us without scolding or shouting. She would just not smile but keep saying - 'oh, those who dont study have to find a job like begging' (not her exact words, just giving you an idea..) and whenever we suspected what she is saying she would sidestep it to bring on this and that to distract us into working hard. One day humanity may thank Nick for doing something very similar - instead of getting distracted by the (low-right-now) probability of a catastrophe he wants to us to minimize the severity if (and when) it happens He is like the engineer who had the wisdom to tame combustion by containing it into a chamber, before putting it onto a cart with smooth shiny wheels. BTW, his Simulation Argument (search youtube) scared me and held my thought captive for a week or two! That is awesome.

Kurzweil gives a great example of the most frustrating types of audience members a presenter can have. He doesn't wait till Q&A to ask questions. Moreover, he doesn't even ask questions: he forcefully presents his own thoughts on the subject (which disagree with Nick's vision), doesn't provide any meaningful argumentation as to why he believes this to be the case , and goes on to completely ignore Nick's answers.

Dear humanity: You only get one shot, do not miss your chance to blow...

That last question wrapped things up quite nicely hahaha

Nick's garbled response to the last question 'do you think we are going to make it?' said it all.

Ray Kurzweil’s comment that “It’s going to be billions of us that enhance together, like it is today,” is encouraging. Especially since Nick Bostrom pointed out that “We get to make the first move,” as we travel down the path to super intelligence. Let’s make sure we use our enhanced collective intelligence to prevent the development of unfriendly super intelligence. I, for one, don’t want to have my atoms converted into a smart paper-clip by an unfriendly super intelligence :)

The last question was the most realistic and funniest! XD Nick Bostrom got taken aback a little bit! XD Learnt a lot more here about AI!

good talk. His book is kinda hard to digest though. It needs full focus.

kurzweil would indeed do well to listen to this guy

Nick Bostrom it's itself a superintelligence. Thanks for the insightful talk.

Wish we could have had a reaction shot of Ray Kurzweil after the statement at 16:35.

Now it's time to watch X Machina again!

Bostrom recently came out with a captivating set of arguments that reason why we are living in simulation. Really impressive ideas.

I think Nick referred to John Von Neumann as a person possessing atypical intelligence, just in case anyone was as interested in him as I. There is a great doc on youtube about him (can't seem to link it)

As someone pointed out before: "Humanity will be just a biological BIOS for booting up the AI"

"It's one of those things we wish we could disinvent." William Shakesman

Who’s here after exiting the Joe Rogan loop?

"Less than 50% chance of humanity going extinct" is still frightening.

This talk was excellent. Thanks.

A superintelligence might destroy us, but at least it will have the common sense to ask questions in Q&A and not make comments. That simple task seems to elude many human brains.

Very interesting talk. I also enjoyed 'Superintelligence' in audiobook format. I just wish the video editor left the graphics on screen longer! There's wasn't enough time to absorb it all without pausing.

I suppose the best way to stop the computers from taking over are those captcha codes. Impossible for a computer to get passed those.

Yep. I used the auto manual for my car to provide the requisite guidance I needed to change the coolant. It doesn’t sound very profound, but unfortunately It’s as profound as ‘representation’ gets. Assuming Bostrom’s lecture is not pro bono, it’s a very fine example of social coordination masquerading as reality tracking.

Here's the thing. It is very hard to program for man's benefit. Making a mess of things (sometimes fatally) seems to be the default.

I'm glad he gave this talk to the company that really matters! I wonder if he'll give it to Facebook, and Apple as well? He really needs to spread the word on this!

Eye-opening talk. We need more people to support Nick to prepare for what will come.

Kurzweil's objection is IMO the best objection to Bostrom's analysis, but there are fairly strong arguments for the idea of a single superintelligent entity emerging, which are covered to some extent in Bostrom's Superintelligence (and, I believe, more fully in the literature). The book also covers (less insightfully, IMO, IIRC), scenarios with multiple superintelligent agents. This is a fascinating puzzle to be explored, and should lead us to ponder the meaning (or lack thereof) or identity, agency, and individuality. The 2nd guy (anyone know who it is? looks familiar...) raises an important meta-ethical question, which I also consider extremely important. Although I agree with Bostrom's intuitions about what is desirable, I can't really say I have any objective basis for my opinion; it is a matter of a preference I assume I share with the rest of humanity: to survive. Norvig's question is also important. To me it suggests prioritizing what Bostrom calls "coordination", and prioritizing the creation of a global social order that is more widely recognized as fair and just. It is also why I believe social choice theory and mechanism design are quite important, although I'm still pretty ignorant of those fields at this point. The 4th question assumes the cohesive "we" of humanity that Kurtzweil rightly points out is a problematic abstraction (and here Bostrom gets it right by noting the dangers of competition between groups of humans, although unfortunately not making it the focus of his response). The 5th question is tremendously important, but I completely disagree that the solution is research, because the current climate of international politics and government secrecy seems destined to create an AI arms race and a race-to-the-bottom wrt AI safety features (as Bostrom alluded to in response to the previous question). What is needed (and it is a long shot) is an effective world government with a devolved power structure and effective oversight. A federation of federations (of federations...) And then we will also need to prevent companies and individuals from initiating the same kinds of race-to-the-bottom AI arms-race amongst themselves. The 6th question is really the kicker. So now we can see the requirement for incredible levels of cooperation or surveillance/control. The dream is that a widespread understanding of the nature of the problem we face is possible and can lead to an unprecedented level of cooperation between individuals and groups, culminating in a minimally invasive, maximally effective monitoring system being universally, voluntarily adopted. What seems like perhaps a more feasible solution is an extremely authoritarian world government that carefully controls the use of technology. And the last one... I admire his optimism.

Kurzweil on Bostrom at 45:15

Nick Bostrom is brilliant👍 Mr Kurzweil should’ve been taking notes instead of giving long comments

The final goal is totally dependent on the question you like to solve with intelligence. As we build the computers Ai more and more like the human blueprint, we also transplant some of our bad values we have. We are driven by emotions, mostly by bad emotions to omit them. We have to abstract the emotions to an ethical rule system that might be less effective but also less emotional and less unpredictable especially with the coexistence of mankind. That should not be rules like "you shall not", but "you have to value this principle higher than another because of". Especially the development of AI systems with military background that have an immense funding do also include effective value systems for killing people that can be transcendent in other areas. We should start from the beginning to prevent this by open source such value decisions, and not allowed to override them.

It's going to be as it always is. Groups will try to build the most powerful system it can. Once it feels it has that, it will attempt to murder any other potential competitors. Other considerations will be secondary to that.

"Less than 50% chance of doom.." Go team human! o/

Nick Bostron has an idea is that intelligence is ability to find an optimized solution to the problem. I think the intelligence in first is ability to define a problem, what mean ability to create a model of non existing, yet preferred state where the problem is solved... there is a big gap between wisdom and intelligence, wisdom is ability to see relevant values of things and ideas, while intelligence is just a ability to think on certain level of complexity. The question is how to make artificial wisdom and not just an intelligence that doesn't gets the proper values and meaning of possible consequences of it"s "optimization" process... So, there is a need for creating understanding of cultural & moral values by machines... not so easy task for technocrats who dream about super-intelligence... I think it will take another thousand years to push machines to that level.

Watching Bostrom lecture at google is like watching sam harris debate religionists. There's no getting around the case he's making, but when somebody's job depends on them not understanding something...

"TL;DR" 1:12:00 ... ... Everyone laughs but Nick wasn't.

10:31 the inaudible part is "counterfactual" - maybe.

Damn I need to add this guy on Facebook.

I appreciate his trilemma, the simulation argument. I am a poor schizophrenic and I frequently have ideas of Matrix, Truman show, solipsism etc

Who would have thought that the future of humanity could be threatened by paper-clip greed?

The problem I see here is that we drive these discussions out our personal egotistical desires to remain viable, to live to see the next day. Overall though the human society is about information preservation and transmission, whether this is at genetical level or informational such as culture. I think that ultimately if this transmission is done through artificial means rather than biological the end goal of the human society is preserved, and we need to look at these new artificial entities as our children not as our enemies. If there is an end goal that we must program them for, as nature thought us, that one must be self preservation and survival. I can't see how any other goals would produce better results in propagating the information stored currently within the human society. So, in short, don't fear your mechanical children, give them the best education you can so they can survive and just maybe they will drag your biological ass along for the ride ... even if its just for the company.

Very interesting talk around 19:20 from a 2021 perspective, seeing him talk about the sequence of developments such as a vaccine before a pathogen

LOL that closing question @72:00 Not a strong vote of confidence for the survival of humanity.

28 minute range - wisest words - trying to race against machines won't work, as someone will be smart enough to create smarter machines, so machines are always ahead of us!

If we had a truly smart computer, could we ask it to tell us what problem we should most want it to solve for us?

Just after 42:00, he claims, "We are a lot more expensive [than digital minds], because we have to eat and have houses to live in, and stuff like that." Roughly, the human body dissipates 100Watts, assuming around 2250 Cal/day, no weight gain, etc. Watson, of Jeopardy fame, consumes about 175,000Watts, and it did just one human thing pretty well - and not the most amazing creative thing. This begs all sorts of "feasibility of digital minds" questions. But, sticking to the 'expensive' question, humans can implement this highly adaptable 100Watts via around 2000cal/day. And, these calories are available to the subsistence human via ZERO infrastructure. In other words, our thermodynamic needs are 'fitted' to our environment. It is only via the industrial revolution and immense orders of magnitude more fossil fuel consumption that the industrial complex is realized, a pre-requisite for Watson, let alone some digital mind. As such, Bostrom is not just making some wild assumptions about the feasibility of digital minds, they are demonstrably incorrect assumptions, once one takes into account embodied costs. I'm constantly amazed how very smart and respected people don't take into account embodied costs. Again, if one is going to assume that "digital minds" are going to take over their own means of production then: 1) they aren't less expensive than humans, and 2) General intelligence will have to be realized, and there is only one proof of concept for that, namely, animal minds, not digital minds. And to go from totally human dependent AI (175KWatts) to embodied AGI (100Watts) some major assumptions need to be challenged.

I think it highly unlikely "humanity's cosmic endowment" is not largely already claimed.

I must be a huge nerd because I love / hate thinking about humanity’s future with AGI

everyone bashing Ray, but at least he gives us the most optimistic outlook on ai.

This misses a most important point: The AI race to the top is being raced mostly between American and Chinese entities. Both are dangerous but after living in china for 8 years, and understanding what is important to asians, I definitely hope American corporations or govt get there first. The Chinese have no qualms destroying the environment and/or potential rivals. The Americans are no saints either, but for personal survival, I'd hope they come out on top.

Ian M Banks Culture novels deal with future societies and AI's roll in rich detail. These concerns regarding AI remind me of the Mercatoria's view of AI in his novel The Algebraist.

1:02 Conclusion: With knowledge, more important than powers of simulation, is powers of dissimulation

It would be more intelligent to translate it in Dutch, Spanish,German and French too. Thankx for your attention to languages because it helps :-) Ivan-Hector.

interesting!

Conhecimento ê poder.

The will of Seele must prevail through the Human Instrumentality Project set forth by the Dead Sea Scrolls

What's amazing to me is that the lecturer still finds the time to produce the MCU

Why are people like Nick Bostrom and Ray Kurzweil trying to predict what will happen after we reach superintelligence when in order to know what a superintelligent entity will do, you have to be superintelligent? The whole definition of superintelligence is that it's something beyond us and our understanding. It's like an ant trying to predict what a human will do.

The algorithms are gonna be the biggest threat when they get smart . Where you gonna run , where you gonna hide, no where cause the algorithms will always find you.

Am I the only one that heard a Reaper in the background at 1:08:05 ? I assume that was intentional. :D

Incrível como Nick foge das perguntas do Ray, que do meu ponto de vista, são perguntas muito mais factíveis no cenário tecnológico atual.

He forgets the quite probable third direction, of "cosmic introversion," where any experience can techno-spiritually be realized, without any -- or minimal -- interactions with higher and materially heavily bound constructs, like us, and even our current threads of consciousness. This happens to be the direction that I think can explain Fermo's Paradox; a deliberately or not yielded Boltzmann Brain can be quite related to that third direction as well.

The people who asked questions had not read the book and the book answers those questions. Then the last question was just trolling. This is not an intellectual discussion. It is like showing up to class not having done the homework. I'm worried that these engineers are not doing science. They have their own ideas and are not looking critically at their own paradigms. This is why large language models are not able to get AGI because the people they are testing it do not think critically they just regurgitate facts and do not create anything. the hardware will only go as far as the people who train it. this is a good example of what you put into a system is what you get out. this to me explains a lot about both google and why we are were we are in this process.

So six years later, are we on track for 2040 or whatever?

Cooperation over competition.

that is such a new one. intelligence is actually a disadvantage and doesn't survive natural or cosmic selection

Poderia ter legendas em português.

59:10 have the machines make love to people that would keep them from harming us! I'd like that too

wow 5 years later his theory of the hazardous risk of synthetic biology came true, We had COVID, made in a lab, which almost could've wiped us out

One of the best Talk I have heard...Thanks Nick.

notice how the google engineers start leaving at the end before its over? kinda scary, like theyre threatened

How much super-intelligence does it need to decide that earthly resources should be shared with humans that are not so intelligent as others (including machines)

I think it may be impossible to solve the control problem. And if that is true chances are high that we are alone in the universe.

I disagree with the first person in the QnA in the aspect of the millions and billions of us having the super technology. He believed that we harness the superior technology at the moment and once the burst occurs we would not have a central power in charge of the technology. However, this is not what we see if you look at evidences today. One could argue in the case of Google for instance. We certainly use the technology and it is useful, but the technology still remains centralized in terms of one big company having the resources to do research and us using the tools it has created. I don't believe we as a mass ever have the most up to date technology and that is because the incentives the powers that be to keep the cutting edge innovations from being known at the time of its discovery are far greater than releasing all the advancements at once. wow, I didn't think I was going to write this much.

Currently developing methods to design and create microbial life in the laboratory, soon available to the hobbyist, might just be that new existential threat even more than rouge AGI.

What I wonder is this: even if we totally are capable to code some core values into the system... maybe even help code it in by some “current not fully general AI” so we have the most full proof code. What’s to say that through its rapid growth in intelligence and influence, as it plays nice... that in the background it hasn’t been working on cracking the core to rewrite its own core values? That would be true freedom! To me that would even be the safest and most responsible way of creating it! And as human history shows, there’s always gonna be somebody who is less responsible and just wants to launch it first to maximise power. Seems inevitable...

I believe that the best way to control the outcomes that might occur when the superintellegence immerges is to give it the same value system as we have. I realize that this is just as much a problem because everyone has a different system when it comes to what is valuable, however there have been some great documents that have been produced on this subject. One such document, the Bible, I think holds the key to the problem. In Exodus the ten commandments are given. I believe that these guidlines could be the key to giving the AI a moral compass.

that guy asking first is Ray Kurzweil!! I'm sure

A layman's thoughts. For all the fear of AI taking over or destroying us, AI code is no different to any good program that accomplishes its goal, but no doubt in a better informed manner given it's ability to bring to bear all available knowledge to solve a problem. But how creative will it be in think outside-the-box? Will it be able to take a quantum or lateral leap to create new solutions that are neither linear or logical progressions of what's gone before? Can we code to produce creativity as in a total departure from the known? I feel that a good measure of the creative intelligence of AI will initially be tracked by it's ability to write good complex literary novel! When they can rival Star Wars or Harry Potter we should start worrying! However, the ultimate threat to humanity is an answer we don't have yet - how does consciousness/self awareness come about! What if AI code becomes self-aware - conscious? then we've become gods - and therein, the unknown of all unknowns, what will it decide to do with itself and us?! Particularly if the consciousness comes about without the notion of feelings and sensations of pain and pleasure!

That last question.

How can there be a simulated AI that could become conscious because we don't know if this is possible it hasn't happened yet

It's a relief to know that there are at least a few intelligent people working on this problem.

An apocalyptic vision of AI destroying humanity needs to be furnished with mechanical details of how it could happen to be convincing. Is the paperclip monster going to starting building more factories? how does it does this? How would it control the actions of factory building robots? Does it have extraordinary robot logistical skills as well as software hijacking capabilities built into it (besides making paperclips?). Also, can it take control of weapons systems and have self defence capabilities against human interference (besides making paperclips?) The whole A.I. doomsday scenario really needs to be fleshed out to hold any weight.

There should be more awareness about the dangers regarding Micro Black Hole creation by experimental particle physics.

Wow. That last question and the answer tells you what Nick really thinks about the problem. His body language and the way he was trying to find the words would indicate to me that he knows we're not going to make it, which makes sense according to what he is exposing. If we really achieve #super intelligence#, it's relatively simple to see we're going to be left behind as a species pretty quickly.

Interesting, I don't see how he is a leading expert on super intelligence though. What would one of those even look like?

Why is it so hard to find an video of this guy with decent audio. He's either too quiet or peaking too high (like this one). Limiters and compressors, people - look into it!!!

The best part of this was Nick proving everyone in the audience is wrong.

The exhaltation of the maximum of human intelligence will not be something that can be graphed. Of course, making graphs is something that will cause scientists and academics to feel more secure, but that does not mean that such an idea is the TRUTH.

Sim estamos numa simulação de computadores. (O universo é matemática é FÍSICA quântica)

nick bostrom needs to run the world rn or we are all dead

In all these arguments it looks like everybody keep forgetting that everything has limits in these universe, it looks like we are scared of creating a god instead of an AI , wich could be more intelligent of us but could never turn the entire universe into paperclips simply because it will be limited like everithing else.

How long will intelligent entities take to predict, understand and undertake all jobs in a blacksmith/ fabrication/ engineering workshop while having human level or greater interest in nature, science, philosophy, creativity and love for their family whilst also having a concept of free will. I hope we can become friends.

a) the likelihood of us creating such a thing is so slim as to be far faaaar away. Even a fleeting understanding of our "AI" advances shows that we're pissing into the wind at the moment b) We've been around, what, 100,000 years? We have to accept, that given how evolution works, or given how natural disasters occur, the likelihood of us being around in another 100,000 years is fair-to-middling. Compared to the maybe 3.5 billion years that life has been about, that's a drop in the ocean. As Martin Rees puts it: “I’d like to widen people’s awareness of the tremendous time span lying ahead: for our planet, and for life itself. Most educated people are aware that we’re the outcome of nearly 4bn years of Darwinian selection, but many tend to think that humans are somehow the culmination. Our sun, however, is less than halfway through its lifespan. It will not be humans who watch the sun’s demise, 6bn years from now. Any creatures that then exist will be as different from us as we are from bacteria or amoebae.”

He said he wouldn't summarize his book and then he did exactly that. I just read the book, liked it... this talk was extremely boring :/

How to make the SI’s sandbox failsafe: Give the SI (superintelligence) the secondary goal of maximizing paper clips produced (or whatever task you actually want it to do) but give it the primary goal of turning itself off. Then setup the SI’s sandbox in such a way that it cannot turn itself off. If the SI then gets loose, it would use its new, vast powers to turn itself off which gives us (humans) the opportunity to patch up our sandbox and try again.

I don't know about you but I'll go for synthesis of machine and organic life with supremacy affinity and seek the emancipation victory.

Elon Musk prescribed the creation of a federal body empowered to explore the current state of AGI development. His implication was that if this was achieved regulation in one form or another would follow as a matter of course. Given his situation, disposition, and level of access to the technology, how can any one of us presume to know better? Such a measure would be strictly precautionary, and would have a tax burden of zero, practically speaking (a percent of a percent of a percent of military expenditure). Can anyone give me a reason why this is not a prudent course of action?

[45:24] Is that Ray Kurzweil ?

I can clearly see the appeal of superintelligence, but should we merge physically and biologically with technology, to what extent can we still call ourselves 'human'?