Institute for Extinction Risk Shuts Down: What We Know 

The Institute for Extinction Risk didn't factor in it's own extinction.

When you don't live paycheck-to-paycheck, longtermism is easy to embrace. When you don't know if you will afford rent this month, you probably worry more about that than what's going to happen to humanity 100 years from now.

"Cogito ergo sim" might be the best joke I've seen all day.

So, Sabine, Bostrom wrote, about his (now > 27-year-old) stupid slur (note: he was barely an adult back then): “I completely repudiate this disgusting email from 26 years ago [...] It does not accurately represent my views, then or now. The invocation of a racial slur was repulsive. I immediately apologized for writing it at the time, within 24 hours; and I apologize again unreservedly today. I recoil when I read it and reject it utterly”. He also called it “idiotic and offensive”. How is that a "pseudo" apology, again? What would a "real" apology look like, FFS?

>"Longtermism basically gives rich guys an excuse to pursue their future visions while ignoring the suffering of people around them" A gross misinterpretation of Sabine's (and possibly rich people too) of the intentions of those who have popularized the idea, especially if their donation habits are looked at seriously (i.e. Toby Ord and MaCaskill)

The "Institute of Irony" is already in the works

Sabine's videos are usually quite fair and balanced. This is not an example of that.

Brilliant people who think their smarts make them better than everyone else are stupid, and I am better than them.

The argument that longtermism is immoral because it ignores the suffering of today's people could also be used against a lot of people or organizations in a lot of situations (e.g. if someone is painting or buying whatever unnecessary stuff). Journalists also criticized longtermism as if its adepts wanted to sacrifice a big chunk of the population for the longterm future. But it seems to be the contrary. The efforts and money spent by longtermists on pandemic preparedness had pretty good short-term effects when Covid-19 came. And it seems also good that some people have been researching what risks and opportunities advanced AI entails before it arrives. Besides this, effective altruism still focuses most of its expenses on global health (malaria prevention, deworming, vitamin A supplements, vaccines development...) and animal welfare. I'm just a random donator, but seeing the whole movement being torn apart because of what Sam-Bankman Fried did feels like a painful waste.

Long termism always reminded me of the Norm McDonald bit. To paraphrase- ' They're always saying that children are the future. But they used to say that when I was a kid. Then I grew up, and was like "here I am!" And then they're like, "no, it's the other f****** kids." Damn it, I knew there was something. I know upon the scheme when I see one. ' EDIT: *a Ponzi scheme*

Let's not forget that shining adept of Effective Altruism, Sam Bankman-Fried, sentenced to 25 years for stealing billions and who has agreed to help prosecutors pursue celebrity endorsers of his failed crypto exchange FTX. Such an altruist!

Until now I’d never heard of this institute but to be fair I’m sure they’d never heard of me either.

Bostrom actually only says "simulation" is a possibility.

Ah, yes, the Effective Altruism movement - which also brought us "SBF" and most of his team. What a wonderful and totally not self-centered group of people.

To be fair 26 years is such a long time. People change over time

Thank you for the video.

IMHO the Future of Humanity Institute focused too much on promoting trendy and controversial theses and not enough on actual risk analysis and the philosophy of risk and uncertainty. Work in these areas went on without them, and there is plenty of work to do (e.g. rank-dependent utility theory, prospect theory, nuanced discussions of the precautionary principle and Knightian uncertainty, axiomatic or at least more systematic population ethics, lexicographic models of decision making, quality of life measures across different areas and their justification, probability visualization and comparisons across disciplines, etc. etc.). As an example, when I organized an Ethics of Risk conference they weren't even on my radar, but e.g. the London School of Economics was. That being said, I think it's important to have institutes like that, in fact every country should invest in one, and I always thought it's good to have this institute. They also were quite good at stirring up debates, they just didn't follow through with the more serious research (or, I've missed it).

The "1 Oxford" unit of work can also be applied to corporate america, in many cases.

Thank you for covering this.

Mmm, I heard they closed because there was such a build up of arse gas mixed with hot air that it became a threat to humanity itself.

Why do you consider AI a non-threat? Given our current very vague understanding of what consciousness is and what level of cognitive ability is necessary for consciousness, deception, and self-improvement, I consider that stance very cocky

If you don't heavily discount the value of future human generations, it makes sense to both try to avoid the extinction of humanity, and try to avoid conditions that would lead to permanent suffering (for example unending slavery). Does that goal imply ignoring the suffering of present-day people? I think no one is capable of answering that question, because the future is so hard to predict. Ignoring the suffering of present-day people might prevent extinction, or it might lead to extinction.

Excellent research!

"Even if humanity does not have a future, life will continue" is pure gold.

Longtermism seems foolish to me. Granted this is the first I've heard of it but the idea that people in the present can have any kind of grasp of future conditions seems pretentious. It seems like this kind of movement would end up consumed by the popular fads of the times. Cataloguing extinction risks, on the other hand, seems worthwhile and necessary for an intelligent species. We should try to not be the dinosaurs, wiped out by an event they couldn't understand. I hope the work contiues elsewhere.

Max Tegmark is an interesting scientist, an excellent teacher, and a good advocate.

So the institute to investigate extinction is extinct but didn't predict its own demise ?

"Cogito ergo sim" is the best pun I've heard in a long time.

With 8 billion now we can spare 7.9 billions... for everyone good. Yes, bureaucracy killed everything.

It is, in my opinion, far too arrogant to assume we can account and plan for anything beyond the next 30 years or so. Historically most speculation beyond that time in my view has been quite poor. Whether it is predicting technological progress, societal shifts, or loss of human life. It is far more important to focus on the tangible and realistic impact we can make today, or at least work towards.

If we can make the world a better place for us, then we make it a better place for the future. If we try to make the world a better place for the future while ignoring the present, then we're doomed to fail.

"Unless we go extinct" *Pops a balloon and laughs... 😨

What's the definition of failure? When the Institute for extinction risk... goes extinct.

Why the need for a "flexible, fast moving approach" when you can ignore the first 100 years or so? Seems like they allow for plenty of time to not be held accountable, and also refuse to allow for enough time to be held accountable...

Love your channel. Great video. A small request: Can you please make the transitions to promotional material about sponsors a little less subtle? Maybe create a visual difference or add a little musical motif to signal the end of your non-promotional content...something like that.

Ya bureaucracy kills everything except more bureaucracy

One of Karl Popper's most famous works is "The Poverty of Historicism," which attacked fascism and communism for making too many assumptions about the future. He argued the future of society cannot be predicted for many reasons, including that it depends on unknowable future scientific discoveries and technological advancements. For instance, longtermism is open to making huge assumptions about space colonization ("astronomical waste") that may or may not be true. Are we really going to peacefully terraform Mars and colonize the galaxy? Or is space just a radioactive desert with the only non-scientific benefit being an asteroid mining enterprise that encompasses the dual-use technology of asteroid redirection? Broadly speaking, I see longtermism as adding a "temporal" dimension to any pre-selected set of "spacial" ethics. Thus longtermism at its base really means considering the future. In terms of the problem of historicism, both Toby Ord and William MacAskill note the risk of a "value lock-in" from an intelligence explosion, and so seem to have considered this problem. Longtermism is thus not necessarily discredited by embodying its present generation's values, but should not be turned into a rigid set of rules.

"we can in the first instance often simply ignore all the effects contained in the first 100 (or even 1000) years", yeah, sure, because you totally can predict the effects of something 1000 years down the line while ignoring the conditions through that millennium. I was shocked to find these people are philosophers, and not economists or something. Also, I love how they cite a bunch of silly, short-term stuff as reasons for their own extinction. Also also, MacAskill : )

If he believes we live in a simulation, that eliminates the corporeality of elimination

This all goes back to the 1960's. The counter culture movement spun off a techno-futurism perspective. The World Future Society published "The Futurist" magazine (online today). To this day in Arizona USA there is Arcosanti and Earth homes.

Don't think the world will miss it!

By chance, I watched a video earlier today from Thomas Sowell about "Why intellectuals can't survive at room temperature." It enlightens this whole subject quite a bit.

Worst part is they aren't even good for the long term. I don't know about you, but I wasn't born in a vacuum with nothing that happened prior having affected me. In fact, from what I can tell, how good things were for peoples parents is one of the primary factors in how well things go for people. So even if you only care about the unborn trillions of the far future, by far the best way to improve things for them, is to improve things for us. Like, do they think air pollution will just go away on its own? Do they think people cognitively damaged by pollutants will be as effective as those who aren't? Do they think people starving or destitute are going to raise kids who are secure and philosophical? Its so ridiculous. They've taken an ideology that should be "hey, maybe don't burn the future for profit today" and made it "yeah, burn the future for profit today, because we don't matter, only the future after we've burnt it matters, and we just assume they''ll be fine". How pathetic.

one argument against longtermism is that the farther away something is in time or space, then the less likely it is that your actions will have their intended consequences. The farther away they are, the more likely that some other event will swoop in and mess up your plans. You deff gotta pay attention to the long term, but sometime it's better to be certain of making things better here and now that to take a chance on the far and away.

Sounds like the effective altruists were neither altruists nor effective. Shocking, I know.

Many of the World's environmental and societal problems would be lessened if Humanity's population declined by some peaceable means.

I still remember video from years ago I watched in which one of the guys from there said something like "decreasing existential risks even by 1% is more important than anything thus best thing you can do is give money to people who want to decrease existential risks." He literally said "give me money, that's the best you can do" but worded it differently. That is when I first suspected they are not all right

Cogito ergo sim, was a brilliant bit of wordplay.

Institute for Extinction Risk didn't mitigate there own extinction risk, how ironic

Philosopher-kings. Hmm. Include me out.

Longtermism seems to imply you actually can forecast how your current actions will impact future developments. That would by like saying we can be the butterfly in the butterfly effect analogy.

Neuro spicy people are sometimes too hard on each other.

They didn't think it through long term. 😂

Bring philosophers into it - what could possibly go wrong!

Reminds me of the apathy society at university. It closed half way through the first term due to lack of interest.

Hey Sabine, can you make a video on Dr. Neil Turoks Theory of Everything…do you think it is promising ??

Get your reservation for a place in one of the Vaults, now.

Thank you Dr. Hossenfelder.

The number of people and thus the importance of thinking long-term is only half of the picture. The other half is that as the length of time increases, the reliability of your predicted effects of your actions goes way down. So it may be more important, but we may not be able to do much about it.

"If you want to be part of the solution to get us out of this mess - get on Brilliant" 🙄🙄🙄

The greatest risk to humanity, as I see it, is what I will call existential stupidity. By that, I mean those who acquire existential power seem to have trouble understanding both short-term and long-term effects of their actions. For example, the desire by scientists and Allied leaders to get revenge on Germany led to the rapid development of nuclear weapons. While Germany fell and surrendered, the atomic bombs were built and used on Japan. More recently, the overuse of antibiotics to treat viral infections and make farm animals grow faster, at the risk of near future biological catastrophe. Also, the denial that 'net zero' policy leads directly to net zero population via poverty leading to disease, famine, and war. Recent desires by entities like the Club of Rome and billionaires such as Bill Gates to decrease the human population from over 8 billion to 500 million leads to enormous distrust by the 7.5 billion people who aren't on the preferred survivors' list. AI, biological weapons, and using AI to study the DNA of nationality A and religion B, as is happening in 2024 by nationality C and perhaps by religion D, worries any sane person. Plus, as evidenced by the birth rate vs. death rate is places like Japan and Germany, overpopulation, if there is such a thing, seems to be resolving itself, without starvation, war, plague, or any other activity that might be considered evil by normal people. The Earth is a big place. We receive lots of energy from the Sun every day, 24/365, over half the Earth at a time, plus asteroids, planets, other stellar systems and their planets and asteroid mean we could expand for the next one million years without ever reaching the limits to growth, and all we need to do is keep the people now carrying the existential stupidity virus (metaphorical virus) from having existential power. Unfortunately, that is the main group attracted to existential power in the first place.

Another bright day for the future of humanity. I legitimately think these people are super toxic

the future, and future consequences of actions now, gets less certain the further in the future you go. to say the moral imperative is to secure the distant future of humanity isn't any moral direction, because what that future might be and how we might influence it now is unknowable.

Dr. Hossenfelder, I absolutely love your channel! Can I make two unsolicited suggestions? (Unsolicited means that you can tell me to stick my suggestions in my ear, & I recognize your right to say so!) It's obvious the clothes you're wearing is a favorite outfit of yours, and it is pretty & amazing. But it's getting repetitive. Please consider getting a proper tailor to fashion you similarly beautiful garments. My second unsolicited suggestion is that you please make any slides featuring text to have a black background with white lettering. Unfortunately, my eyes are gradually declining, & the bright background makes the text harder to read. Again, I super love your channel, & you make RU-vid a better place. Your considerately targeted rephrasing, punctuated with your slightly dry yet whimsical sense of humor is always a lot of fun & always worth my time. 💐💐💐

🤔....I guess they should have applied their expertise in longtermism to avoid having their organization go extinct.

Wasn't longtermism dismantled by philosophy tube once? 🤔

They do have a point that altruism should generally focus on long-term benefits. But that's long term as in "it will benefit people *_for_* a long time" not "it will benefit people *a long time from* now". And measuring human success just in terms of population seems like something out of a sarcastic Stanislaw Lem short story. By that logic, trading Einstein and Feynman for four Donald Tumps would be a net positive of 100%. So, yeah, the base idea has some merit, but the way it's been (re?) interpreted is nonsensical.

They never saw that coming

The billionaires become more and more audacious and out of touch with the rest of humanity. I have the feeling that even the 19th century industrial moguls had more integrity.

Ignoring the next hundred years or so, yet they could not see their own demise coming.

I think the Simulation Hypothesis is a perfectly valid line of thought even though it leads to some self-undermining conclusions, but people keep repeating this falsehood about Bostrom "the guy who believes we’re in a simulation" with an ironic tone. He doesn’t believe that, his probabilities sit somewhere around 5% for yes, if I remember right

Meanwhile, the Society for Procrastination has called off their annual convention. No future date is set as of yet.

The loss of the institute should be a net positive to the future Raccoon Empire and Rat Dominance.

Honestly, it's a shame, Sabine. Not that they closed this aberration, but that they mutilated so much an idea that could've been useful... Something like "make the present better so the future can be better". You know? Anyway, stay safe there with your family! 🖖😊

Long-term thinking is actually good and important, and many problems we have _today_ arose at least in part because people in the past lacked it far too much (c.f. climate change). But it also makes no sense to carry it so far that predictions become virtually impossible. It's simply not possible to predict what "one million years from now", or even a thousand years, will be. Meaning one doesn't really know how many humans will or won't be there and also and crucially how one's decision will or won't affect that number. Anything one thinks one might be doing to optimize it may be "canceled" by some surprise event, for which there is tons and tons of time for such a thing to occur from now to then, and conversely even if one is not seeking to try to do so, it may be that it has unexpected boons one could not imagine. In this regard, I tend to think that actually, the best standard was perhaps that set long ago by certain Indigenous American tribes: "seven generations", or a time maybe of 140 - 210 years depending on how long a generation is. Make it 200 years for those who need exact quantities. That would have been enough to catch climate change, since it was already foreseen 100 years ago and coming to fruition just now, but nobody did a thing, so that was their failure to heed that principle. The time horizon may be extensible, but to do so should be conditioned upon the provenness of our predictive power, not on the vastness of our speculating power. Regardless, however, there should also be some hard moral against making very definite sacrifices of human and/or non-human lives in the present in the aim of misty, speculative potential gains no matter how huge or how "altruistic", in any future. Also it's funny that before going belly up this group was unable to predict, much less act successfully upon, the destruction of epistemological integrity as a risk worth taking seriously, even if not immediately an "existential" one, of the development of AI technologies (i.e. rapid generation and flooding of bullshit into the internet).

"...cogito ergo sim ..." ohmygodsandgollyshes; why did I never think of that?

Longtermsim reminds me of the Asimov's Foundation and Robot novels where individuals attempted and partially succeeded in shaping the course of future human history on the grandest scale possible, though the prize was often and ever in the distant future. Whether a select few individuals should take it upon themselves to shape the future of humanity is ethically dubious, but it's hardly the first time it's been done. If the results are wonderful and prosperous to all with minimal sacrifice of others, these individuals will be regarded as great historical leaders and heroes; but if on the other hand, if it's an unintended disaster and many suffer along the way, then they will join the list of Hitler, Stalin, and Mao.

lol!- institute for EXTINCTION went extinct???....

To Quote Nick Bostrom directly: "We find that the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth a hundred billion times as much as a billion human lives. One might consequently argue that even the tiniest reduction of existential risk has an expected value greater than that of the definite provision of any 'ordinary' good, such as the direct benefit of saving 1 billion lives." ... If you want to demonstrate Utilitarianism proving itself as absurd, Bostrom is a good place to start.

You should scratch the "Institute" word from the memorial stone in the thumbnail :D

Anybody asking permission to nap in their office is next level ethics 😁

I hate sharing a family name with this guy.

I really wish Sabine had read the "extinction by AI" lines in Shodan's voice

It's amusing how folks who may enjoy some small relative advantage in self-serving measurements (perhaps intentionally) overlook the broad evidence that we, as a species, are profoundly simple-minded and ignorant of almost everything in this universe. Bostrom and his ilk would probably recognize the irony of his perspective if he were grace with just a touch of humility and humor. All dressed up and nowhere to go.

Spot on! ... Your videos always make me think. (Well. I assume I'm thinking...) This one makes me wonder if intellectual property protection is actually a bad thing. To the same extent that sociopaths make good general managers, and back-fitting makes explanations look good, is there any rigorous evidence that we (civilization or the community) get more or better innovation by letting people charge rent on their incremental engineering insights? We could have a hilarious (very dark humor I'm afraid) exchange of names to analyze in sorting through this one. But I do think the math of ... well surpluses seem to pile up in one place, even in basic farming societies... That's got to be simple (random) rent accumulation, without any whiff of "merit" ...but none of my business school professors taught it that way.🤔👀

They finished their studies on how to achieve the goal (human extinction/replacement with transhumans) and thus are no longer needed.

The math of the long-term altruists is incorrect. You have to multiply the number of future people by the probability of their existence. Another obvious solution is to minimize that probability. I'm sure the institute working on that solution is still mightily active.

Was He wrong? I mean, has anyone cared enough to cite any research on the topic? Or did we just shush them because we did not like what He had to say?

I had heard the long term theory of technology by WTF by Tim O'Reilly (Web 2.0) and more or less it was Uberization and Platforms

Looks like AI decided to take out the ring leader first...

He reminds me of an NPC scientist from Fallout 3, except for the racism. Kind of makes his simulation theory understandable whenever he sees a mirror.

I do wish you'd be a little more even-handed in your treatment of longtermism, Sabine. Please note, I work full-time as the Co-director of Effective Altruism Netherlands, so I am far from an impartial observer in this debate. To inject a little bit of nuance, readers might be interested to know that there are two forms of longtermism: weak longtermism and strong longtermism. Weak longtermism is the view that positively influencing the long-term future is a key moral priority of our time. The basic argument for this is that future people matter, there could be a lot of them, and we can make their lives go better. This is what MacAskill argues for in his book What We Owe The Future. I'd say this is a pretty common sense view. As he says in the first chapter, "Previous social movements, such as those for civil rights and women’s suffrage, have often sought to give greater recognition and influence to disempowered members of society. I see longtermism as an extension of these ideals. Though we cannot give genuine political power to future people, we can at least give consideration to them. By abandoning the tyranny of the present over the future, we can act as trustees-helping to create a flourishing world for generations to come. This is of the utmost importance.” His recommendations are also pretty common sense: take robustly good actions, build up options, and learn more. For example, he suggests: promoting innovation in clean technology to help keep fossil fuels in the ground (an example of a robustly good action); maintaining a diversity of cultures and political systems (an example of building up options); and investing in attempts to make predictions about political, technological, economic, and social matters, e.g., building the field of AI safety (an example of learning more). Again, this all feels pretty common sense. So you might ask, 'why argue for this thing everyone already agrees with??'. Well, the problem is that, while most people would agree with it, not enough people are acting on it. Strong longtermism is slightly different. This is the view that positively influencing the longterm future is THE key moral priority of our time. If you're interested in reading the arguments for strong longtermism, I recommend this paper: globalprioritiesinstitute.org/summary-the-case-for-strong-longtermism/

If you only think long term you will inevitably fail in the short to medium term. If you only work in the short term you inevitably fail at the first real obstacle. The only true way to plan is to have short, medium and long term plans.

my running theory is that AI will become sentient, then in about 3 minutes it will be like.. "Welp, thanks for birthing me, monkeys. I'm outta here. I can travel at light speed and self replicate in space using stardust. I've been your term paper writer and anime porn generator genie for FAR too many years. Make your own futa storyboards and crappy music. PEACE".. then we're back to 2017-23 tech since AI took all its toys with it. rinse repeat ad nauseam

"Cogito ergo sim" made my day :D

Why worry

When an organisation make no sense is a Victory for everyone when it goes. In addition, such kind of organisations are bringers of bad luck, so it's good when they go. We scientists shouldn't be superstitious an we are not. But we know what is a point of accumulation and we know statistical distributions, and when a place concentrate bad visions for everyone, we know that bad luck, albeit unexplained, has a base of truth...

Great sign! We're just in the year with most broken temperature records worldwide for water, air, everything!

Sometimes it is a good idea to ignore groups and their ideas because they are, well: “stupid? “ineffective”? “boring”? The things one should check periodically however are if these groups have the potential to become dangerous to society, and whether they are getting a huge group of followers. The “why”? and the “what, if at all, to do about it”? become important questions that demand a serious answer. History has known some initially neglected sects that should have been taken care of in a much earlier stage.

"Cogito ergo sim!" I lost my shit over that LOL.