What I like most is Holly's understanding of company-think or corporate-think. An animal advocacy background certainly helps with that! - Money-grubbing above all else. If AGI adopts that ethic from us - doom is certain.
On "what would you do differently if you eval comes back negative", I've heard from people like Victoria Krakovna that the thinking is something like: There's a lot of randomness in training models, we'll train a bunch of models and keep the ones that pass evals/interpretability analysis. I guess this is supposed to somehow work like a genetic algorithm in search of safe AIs? I don't really buy it though.
10:25 With a century-level pause, the risk I think of isn't climate change, it's ageing. While there could be medical breakthroughs even with AI paused, the default expectation if you just wait 100 years is that almost everyone currently alive is dead by then. Personally, I'm kinda selfish so I don't want to die of AI killing everyone or of old age, a 10-20 year pause seems like a good idea on current timelines, but 100 years is as much a suicide pact as going full-tilt. 35:18 Oh, she has encountered the issue, she just dismisses it as some silly sci-fi nonsense. "My preferred policy will kill you slowly" and "you even having the concerns you do is a joke" is quite the one-two punch. If my p(doom) were as low as 20-40% I'd feel alienated enough to switch teams for e/acc.
China is most definitely building on this. There's no way they are ignoring A.I. I do think that pausing means that we are placing safety in the hands of China. Which could be better. I mean, at least they're not Google.
Completely agree with her. But there is some miniscule chance that Superintelligence will adopt a sentientist ethic and fix us humans. Afterall, if it is super intelligent that's the way to go...😁 😇 😈