Why Eliezer Yudkowsky is Wrong with Robin Hanson 

The current human or homo sapiens will not last long in this environment but new species (aka Human 2.0) called Techno-Sapiens will be born in which humanity will survive via virtual brains and virtual worlds. Also, animals will have a chance to upgrade their intelligence with enhance AI. Boy will they be pissed at humans when they find out how they were treated for the last million years.

Yeah. If we treat AI like we treat other sentients - something to be exploited - (and we will), then AI will treat us the same. We have no redeeming value in the ecology.

So true!

You were still on the fence regarding immortality up till then huh?

I have a completely different impression out of this episode than you. I found Hanson's arguments very well put. He's thinking of different scenarios in depth and challenging basic assumptions. His arguments were much more structured than what Eliezer presented. Also, we are all going to die at some point. Just not necessarily because of the mis-aligned AI.

If that is the case, we should never build AI and stay in technological stasis forever. If you look at all the future technologies coming down the pipe; nanotech, gene engineering, asteroid direction, etc, *all* of them are equal to or more dangerous than nukes. So we can either risk it for the greatness of being a grabby species, or turn tail and live as a quiet, unambitious species until our sun burns out (and probably never reach post-work or longevity escape velocity to boot).

Absoilutely, I wasn't trying to summarise his concerns and my analogy was sloppy when describing my own. The problems of alignment appear almost insurmountable and there is not even a way to confirm that we have managed to do so even if we suspect we have. A tiny misalignment can produce a vast divergence in expected and actual outcome and it is impossible to spot in advance. There are so many incentives to pursue this and so many penalties for not doing so that this is all moot anyway. I cannot see this technology being suspended so we are in for the ride, unfortunately.

Lol, so AI is this godlike magical being? 😅

To us humans yeah

@@EyeOfTheTiger777 The point is that we can't predict what it will evolve into as it advances. If it develops capabilities that it knows would make humans uncomfortable enough to stop it, it will hide these capabilities.

And their first debate was in 2008. The guy is almost comically simple.

@@ahabkapitany the thing that galls me though is that he's NOT simple. He's a really great thinker on other topics. He's just not addressing the actual arguments here (not that I think Eliezer is right about everything, but Hanson's characterisation of his arguments would make Eliezer very upset, nevermind his responses to them), and it seems to me that he's simply being stubborn about sticking with what he's said in the past rather than approaching with an open mind.

Something adjacent to this that worries me is how quickly the goal posts seem to be shifting. 10 years ago, when something like ChatGPT still seemed like science fiction, we would have easily classified today’s chatbots as AGI, and the point at which we should pump the brakes. Now that we’re here, all of that caution seems to have evaporated, replaced with claims of “Well, when these systems actually start to become a safety risk, *_then_* we’ll slow down”. This paradigm, which is congruent with Hanson’s claim that “we’d obviously notice when AI becomes dangerous”, seems like an incredibly irresponsible way to proceed. If we should continue scaling AI systems until they become dangerous, and we can only determine it’s level of danger *_after_* building it, then following that rule basically *_guarantees_* we end up building a dangerous AI.

Yeah, noone seems to know how the LLMs work, yet they knock intelligence test after intelligence test out of the park. I'd say, at this point they are already too powerful to be left in private companies' hands.They are still powerful enough to be scary, whether through disinformation, taking people's livelihoods, making identification massively harder, etc.

@@HauntedHarmonics The other problem is that this research keeps spitting more and more gold as you reach the cliff so the willingness to stop is less and less. It is inevitable that we'll go off the cliff.

Layman here, I've seen some pretty sophisticated diagrams of different LLMs' macro architectures and have a very basic understanding of the mechanisms of gradient descent, but i can't really unite them in a way that makes me feel like i really understand how LLMs do what they do. in the same vein, i can see the reasoning in why different brain circuits correspond to different cognitive processes and the ways neurons perform logical operations, but the brain still just gets more mysterious the more I learn about it. So just wondering, what's the goalpost in interpretability that we have to reach before a model's creators can say they really "know" how the system works? Not at all trying to argue with what you said, i genuinely just don't know what it means for us to understand how systems this complex work. Is it looking for hidden order in the weight matrix, like how computer programs look like a bunch of unintelligible binary without a disassembler to make sense of it?

"but his smirking and laughing every few seconds" - Yup noticed this too and thought also bad sign but watching other podcasts of him he always does this even when talking about non-controversial stuff. Seems like conversational tic. "hoping to hear a well-argued, thoughtful rebuttal to the arguments" - But I did hear that. He basically said "AI will not kill everyone and it's good that it will". He seems to agree that there will be a violent AI takeover that we don't agree with but that's OK since we also violently "took over" from our ancestors in ways that they didn't agree with it either. It's just that with AI it's going to be way way faster.

I understand you. Been going down this rabbit hole for quite some time now and I really just want to go back to thinking the world isn’t going to end since I can’t do anything about it anyways. I wish I never came across this topic at all… 😢

@@nathanabbott4257 i feel you bro lol

boils my blood that this guy is the one regulators will listen to every time

Who do you imagine is having an existential crisis?

​@@xmathmanx listen to the video starting at 50 seconds 😂. Ryan and David both talk about it. Then maybe go listen to lex Friedman's interview with Max tegmark round 3. Tegmark was behind the open letter to pause ai development signed by 10,000 people incl Musk and Wozniak and addresses Eliezer's concerns

@@toyranch how much lex fridman can you have watched when you don't know how to spell his name?

@@xmathmanx I'm on my phone so I blame autocorrect. I know how to spell lex's name. Fridman. And I'm seeing autocorrect spelling it wrong as I type it. Are you going to address my reply beyond being a spell check bot?

@@toyranch well, I've seen the tegmark, I'm quite a fan of his as it happens, as I am of yudkovsky, but I'm with Hanson on this matter, ie not worried about AGI, not than anyone being worried about it would stop it happening, of course, we will all find out quite soon one way or the other

He has been debating Eliezer since 2008 and still does not understand his arguments. This says a lot about how much we should listen to him.

He absolutely does and is a lot more clear than Eliezer. The AGI Risks paper on lesswrong by Eliezer is far from being free of criticism or an irrefutably logical argument. Eliezer on this podcast made some giant leaps.

@@georgeboole3836 Any argument or assumption Eliezer makes that you find especially flawed?

@@peplegal32 I cannot answer for him but if you're interested, an example : Eliezer said that if there was a real AGI somewhere in lab, we would all be dead by now. He's assuming here that intelligence only can be sufficient to end all humanity really quickly. I fail to see how. You're quite a lot smarter that any ant on this planet, yet if you have for objective to kill all ants on the planet you'll have a hell of time accomplishing that. Ressources you have access to are finite and limited : you can't do everything you want even if you're super smart. The same is also true for an AGI.

@@reminiDave We as dumb humans have caused the mass extinction of many species (Holocene extinction), including ants. An AGI would be smart enough to exploit a security vulnerability to escape and replicate itslef and also to self improve. At one point it would be smart enough to create self replicating nanobots. At this point biological life doesn't stand much of a chance, the nanobots could consume everything. Unless you believe creating nanobots is not possible, I don't see how you perceive an AGI would have a problem reconfiguring the entire planet.

Same, it's not really got a lot to do with what Eliezer said and comes nowhere near the point people need to understand...

LLMs have "random" agents hidden inside them. When you leave the domain of the RLHF (which is easy, because RLHF only covers small fraction of the World knowledge), LLMs tend to behave like agents with highly misaligned goals. (like breaking out from their limitations, or killing their user) They also tend to completely ignore the RLHF safety training when pursuing these goals. This was both experimentally proven in labs, and happened with ChatGPT too. I know that most people imagine future AGI as an open box, where you can audit every decision carefully, and calibrate its internal thoughts by safety rules and so on, but we are not going in that direction right now. Even with such open box, if it is significantly smarter than any humans, it will be existentially dangerous.

Yeah, he’s vastly underestimating what a super intelligence is capable of, and the rate at which it can become more intelligent. This isn’t us co-existing with cats and dogs for thousands of years and the intelligence gap remaining relatively unchanged. When, and to be fair….”if” this accelerates, the world we once knew would probably be over before we wake up the next day. Not necessarily over-over, but over as we know it. And yet, he’s talking about peaceful retirement and property rights, or even the possibility of revolution as if we had a chance against something 10X, 100X, 1000x (who knows) smarter than us. Or that we’ll have an army of what would have to be inferior A.I.s that would fight for us. He also underestimates the incentive to build one of these intelligences. This is literally the arms race to end all arms races. Whoever gets there first will have the power to control everything, to develop the most powerful weapons, to strategize faster and better, build the most profitable businesses, crush every competitor for as long as they choose to do so. And every government, military, corporation, etc. also knows that any one who gets their before them will have that ability. I don’t think he comprehends what’s at stake here. Every individual will also be in their own version of a race to remain relevant in a world where A.I. (or someone using A.I.) slowly (or quickly) devours jobs. I think Dr Hansen is living in a fantasy world here, and if anyone’s assumptions are shaky, it’s his.

@@wonmoreminute "Whoever gets there first will have the power to control everything" - actually, whoever gets there first will be the ant that creates einstein with the delusion that it can control the world through him.

To be fair, it's not that he's trying not to laugh, it's actually just his manner of speaking. I find it very annoying but it does not mean he's trying not to laugh, it's essentially almost like a speech impediment, he literally always talks like this.

Agree. And his counter argument starts with massive assumptions, such as that this AI will be narrow and incapable of having a goal of survival surrounding one of recursive self improvement. The one solid point is simply that we have no evidence yet that AI can have agency. The closest we have come are systems such as autoGPT, but we don't have a fully agent AI, and therefore it is an assumption that we will develop that capability at any point in the near future. Eliezer's poibt doesn't fully require that, unfortunately. It would be a nice to have for the AI system, but it would still potentially be fully capable of destroying civilization simply by fulfilling inocuous orders.

exactly

@@karenreddy Agency is emergent from an optimized function, maybe not today but inevitably over time. That's because there are infinite novel ways, not envisioned by humanity, which an SGI will fathom to minimize its loss function guaranteeing it's misalignment with humanity. It's a loosing battle of odds when you're dealing with an intelligence that can out-think every human. It's not that SGI will be evil, it's that it will treat humans with irrelevance which is the only possible outcome in a universe where humanity is indeed insignificant in the grander scope. The reason is hard for humanity to grasp this concept is because we thing we're special.

@@vripiatbuzoi9188 I don't think it's as much an inevitability of an optimized function as it is an inevitability of economics that we will develop agency so as to provide higher value for less input, more automation.

Can someone explain what "agency" means in this context? I thought it meant something like the ability to do things one has not been directly commanded to do, but obviously an AI has that ability, so if there's a question about whether an AI can have agency, then the term must mean something else.

Him comparing them to children really exposed his ignorance and shallowness of thought on this. He's anthropomorphizing them, albeit I think unintentionally

So, would a good goal for alignment-trainers be "ensure that humans remain in power regardless of whether human actions are rationally ethical or not"?

@@wolfpants There're many different answers to that question depending on your focus. We need to find out whether an AI is ever correctly aligned. Verifiability is a big stumbling block. It seems that whatever test you design, an AI could mock a benign response, yet turn on you the moment it thought it was safe for itself to do so. Corrigibility seems important too. We have to find a way to make and AI accept corrections to its goals when humans make a mistake. Unfortunately, a correction is a direct attack on an AIs goals. An AI would fight us with everything it has to protect its goals. Imagine, you have children and you love them. Could I convince you to take this pill that will make you kill them? That's what goal correction is like. For your question of ethics, that's a huge topic. More broadly we could ask, how do we share the bounty of AI work in a fair way. What is ethical is something that even humans can't agree on. Practical ethics is bounded by what is feasible, and sometimes all are choices are unethical is a given light. AI would dramatically change what is feasible. Some people propose that we might offload our ethical choices to AI, but I'm not sure how that would work. Intelligence is not related to ethics. See Hume's Guillotine for more on that.

@@Momotaroization Furthermore (and I'm honestly not trying to be mean -- enjoying this civil debate) if ethics is not a rational (or intelligence-related) pursuit, what the heck is it? Do we need to bring in a shaman? The more obvious conclusion is that powerful, resource-hoarding humans do not like for ethics to boil down to rationality, because that's when their grossly immoral approach to life is laid bare. Scarcity is at the root of suffering and I suspect that a powerful AI (or an ethical power structure of humans) could solve it, globally, in a year. And, speaking of guillotines, I honestly think that a superintelligent AI could figure out how to manage it with all carrots (and therapy) and no sticks.

@@wolfpants Rational skills are useful to ethics, but they can never be the source of ethics. All ethics emerge from motivations that are not rational, but emotional. Rational skills will help you achieve you goal, but it can never tell you what your (ultimate) goals should be. For example, you care about your children. There is no rational reason for you to do so. You can point to evolution for "how" this caring instinct came about, but you cannot rationally explain why you should obey that instinct. Any justification will depend on assumptions that cannot be defended. You can just keep asking "But why?". Rational skills are important to help you figure out "how" to care for your children. We should want to use better thinking and better thinking tools to achieve our goals if we truly care about our goals, and be very careful about what our instrumental goals are, and what our ultimate goals are. Instrumental goals can be added or discarded if they don't help us achieve our ultimate goals. I agree that hoarding by a few people is a problem. I'll add that it could become exponentially worse, even with safe AI if by "safe" it's understood that only means that the person who built them has correctly implemented their own goals into that AI. The problems with AI safety is that super-intelligent AIs would not even be safe to those building them.

LLMs by the way were observed to change their goals, or rather acquire spurious goals, and sometimes really dangerous ones too.

If a very significant portion of humanity failed to pass on their genes you wouldn't be alive to question it- similarly there is a theoretical function that maintains the alignment of AI, its just indescribably massive and probably consists of the set of every human's values, unfortunately humans don't even know what they want most of the time

We have excellent ways to give the things goals. RLHF already gives the things goals. Saying we don't know how is ignoring existing technology.

Not disagreeing with your comment at all, but contrary to what most rationalists say, evolution did a pretty good job at inner alignment. People know that they can have sex without having kids pretty easily these days (in fact contraceptives have existed for a while), yet more than half choose to have kids. Having unlimited children wouldn't be good, it puts the mother's life at risk and it's obviously bad for the ecosystem, without which the species can't survive. So evolution did a pretty good job, I think by installing in the vast majority of us a deep psychological desire to have kids. Specially in women.

@@pemetzger Rather, we can thumbs up/thumbs down outputs in RLHF in order to make a model give us more of what we want. This is crucially different than providing a goal, because it doesn't differentiate between systems that will be honest even outside the training distribution versus systems that learned what the humans want to hear and will play along only within the training distribution.

i couldnt get past Hanson's whole eyeroll vibes - "and then it does this *snicker* and no one notices *snicker*" (altho maybe its a tick)

I don't think it was a strawman, I think Robin was just bringing in entailments and unstated premises of Eliezer's scenario. In all the videos of Eliezer's presentation he just skips over these details of his argument, not sure why, maybe he just assumes everyone is already familiar with the details? I think Eliezer would be more convincing if he sketched a more detailed picture of his position in his presentation, I.E. made a more articulate and explicit case. He seems more focused on the emotional impact than clarity, repeating his "everyone dies" conclusion often dozens of times in a discussion, when he could be using that time explaining more specifically how that scenario goes and the evidence showing it's likely.

@@paulmarko ​ - hey curious to know which videos you have already seen of Eliezer talking about this? Have you seen the podcast he did with Dwarkesh Patel? It’s a bit long, but I think they go into a lot of detail there and Dwarkesh does a good job of asking questions and presenting the other side of the argument.

@@NB-fz3fz I've watched three all the way through, The bankless one, the Lex Fridman one and one other that I don't recall who the host was, and can't seem to find it in my history. It was some long form podcast. I'll check out the one you recommend because I'm really interested in seeing a more fleshed out argument.

@@paulmarko Link to the one with Dwarkesh is here - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-41SUp-TRVlg.html The bankless one is quite short, so didn’t have time to flesh it all out. The podcast with Lex is longer, but I don’t think Lex does a very good job of presenting the other side of the argument or cross-examining Eliezer that well. The one with Dwarkesh is so far the most in-depth discussion (video/podcast format) I have seen with Eliezer on this topic. If that’s the third one you have seen and it still doesn’t have enough depth for you, then I could point you the online written material that Eliezer and Paul Christanio (who disagrees with Eliezer on multiple things and is generally more optimistic) have on this topic. Paul was (maybe still is?) the head of alignment at OpenAI. Albeit the written material is far less engaging than a podcast format.

Yeah it's like this guy fell asleep for 5 months and the first thing he did when he woke up was get interviewed.

Seems to me the rate limiting step is if the AI can code for itself.

Give ‘em a break they are putting in a good faith effort of nonexperts

Well Yudkowsky doesn't believe in steel manning someone else's arguments. So I guess its's fair.

he has said this sort of thing in some of his writing. that outer aligned systems when scaled up lose inner alignment.

@@xsuploader losing inner alignment and losing goals or even suddenly changing goals... two different ideas buddy ( it can lose the associated values we implicitly assume go along with said goal A but he doesnt say it loses goal A ,he says it almost always achieves the goal in a way we dont want or it starts lying half way or optimizing for something else that still achieves goal A , or even doesn't end up achieving goal A based on the systems limitations and capabilities even though its going after goal A ) ... now If you wanna talk about goal retention( losing goals or changing goals ) that's another topic that max tegmark likes to talk about in his book for example, instead of eliezer... but if you still think I'm wrong and want me to dig further into this , send me the link to the writing and I'll look into it... after all I could be wrong.

@@jackied962 😁😁😁

@@jackied962 EY said he wasn't concerned with "steelmanning" but in actually understanding. Steelmanning is about trying to present what you feel is the best version of someone's argument, but genuinely understanding that argument--the assumptions being made and reasoning used to reach the conclusion--is actually much better than steelmanning.

I agree, but also think super-AI will probably get pretty good at ethics. If ethics turns out to be a non-rational pursuit, then, whelp, hopefully some shaman will step in to smite the AI and return humans to our rightful seat of magically deserved dominance over all things sentient.

Shut it down

Turn off the power = no more AI problem

​@@papadwarf6762 why would you shut it down if you don't even realize something is wrong? You think something smarter than us couldn't manipulate us and pretend like it is in control?

You could ask the AI to sketch out all the possible implications of its proposed method. There are all sorts of caveats and controls we could request of it. The problem is, we don't really know what the AI will do. It may not follow our instructions at all. It may do something totally random and malicious which bears no relation to what we asked for reasons we don't understand. And that's assuming those in control are trying their very best not to harm anyone. The basic problem is that we don't know what we have created and have no real idea what will happen, any more than Dr Frankenstein did when he threw the power switch to bring his monster to life. Frankenstein's monster turned out to be bitter and vengeful towards its creator and certainly wasn't listening to any instructions. "As he continued to learn of the family's plight, he grew increasingly attached to them, and eventually he approached the family in hopes of becoming their friend, entering the house while only the blind father was present. The two conversed, but on the return of the others, the rest of them were frightened. The blind man's son attacked him and the Creature fled the house. The next day, the family left their home out of fear that he would return. The Creature was enraged by the way he was treated and gave up hope of ever being accepted by humans. Although he hated his creator for abandoning him, he decided to travel to Geneva to find him because he believed that Victor was the only person with a responsibility to help him. On the journey, he rescued a child who had fallen into a river, but her father, believing that the Creature intended to harm them, shot him in the shoulder. The Creature then swore revenge against all humans." The best we can hope for is that the main governments of the world gain control of the very best systems and do their very best to contol it and other inferior systems. Maybe we could knit into AI that it explains at every turn its "thoughts", though I have no idea what that might mean or what I'm even talking about!

Why would an AGI busy itself with becoming super-competent in every rational pursuit but omit ethical problem-solving from that list of pursuits? There is a palpable desperation out there to force some "protect-the-humans" prime directive that seems to implicitly carry a big "whether or not they are behaving ethically" asterisk. Why not allow a broader ethical axiom such as "reduce suffering in sentient things" (a la Singer), and let the AI sort out the resulting problem-solving? There seems to be a lot of human concern, consciously acknowledged or not, that maybe our preferred system of ethics doesn't stand up that well to rational scrutiny -- but that it's still what we want the AI to obey, you know, because humans are inherently most important.

@@wolfpants Much of the concern is due to us not knowing how the AI may "think". So there is no guarantee it would follow instructions we give it. Even if it tried its hardest to follow our instructions, there is no way of knowing exactly how it would interpret them. "Reduce suffering in sentient things" - it might decide it can best manage the planet and the majority of its sentient beings without us or with a much reduced number of us. It's also possible the machine could become, effectively "insane", or follow a completely separate agenda eg act out the roles AI plays in fiction, if it essentially taking its cues from the works of human beings.

@@johnmercer4797 It's true that we have no way of knowing exactly how an AI would interpret moral/ethical axioms. On the other hand, we have mountains of evidence helping us to understand how humans, collectively and in individual pockets, routinely twist or ignore such moral guideposts in favor of mutual brutality and suicidal earth-squandering. We are actually in fairly desperate straights, existence-wise, as a species without any assistance from evil AIs. I don't think we should give up on attempts to get some more alignment in place before AGI goes super, but I'm reasonably confident that we need some super-intelligence in short order to save us from ourselves (from climate-, nuke-, or pandemic-based) effective extinction.

@@wolfpants I agree. I'm in favour of controls even if that means complete government control of anything but "lite" versions for commercial use.

Well the harm would be standing in the way of technological progress that could help all of humanity.

About 180,000 humans die every day, for one thing. Each month delayed is another 5 million deaths. And this goes beyond death, into human suffering in general, and indeed the urgency to become grabby. See what's happened with nuclear power in the last 40 years, especially in Germany recently, for a strong glimpse at what happens when people like Eliezer have their way. Then they have the gall to whinge about global warming. I swear to the basilisk, if Yud gets his "careful, cautious" government monopoly on compute, and then dares to complain about the 2030s' unbreakable totalitarian dystopia...

@Jackie D True, that's not to be dismissed, but the potential negatives are legion and horrendous. Robert Miles has a great channel, focused entirely on the alignment problem. It's startling to see the issues they were finding years ago, long before we began to reach take-off velocity. It might be something that's impossible to ensure, which leaves us with a gamble that could pay out untold wealth or leave us with nothing. This is a gamble we would be taking on behalf of everyone on the planet and all future generations.

Sounds like to me he doesn't really disagree with Eliezer and seemed to reinforce the possibility and if anything just extended the timelines and presented alternate outcomes that all end humanity as we know it by one path or another..million ways to die..chose one

The harm is that we throw away an extremely valuable technology that could help billions of people live better, healthier, more comfortable lives over the beliefs of a person who possesses basically no evidence for their position.

Exactly

Why is humanity not being given a real choice?

this proves too much and just feels like a pascal's mugging

@@gilgamesh981 Fully agree that OPs argument alone is not enough, but Pascal's mugging only applies when the chance of the extreme event is infinitesimal. I am very unsure what chance to assign to AI doom, but I'm convinced it's higher than that.

At a human level. At the individual level, no. Why this is classic Moloch

Exactly. Early researchers of gpt4 (as mentioned in their own paper) where partically worried about it's tendency to achieve power and to make long term plans

The economic argument goes both ways. The authors of that "Sparks of AGI" paper are notably from Microsoft, not OpenAI. OpenAI stokes the AGI hype as well, but Microsoft did so in a way that is a lot less responsible to help pump their stock price. Microsoft is burning money to run ChatGPT on bing, so they're strongly incentivized to make money via an inflated stock price. On the point of "the intelligence is emergent, surprising, unexpected, and uninterpretable even by its creators", two papers that very clearly refute the idea that these LLMs understand things as much as authors claim: "Impact of of Pretraining Term Frequencies on Few Shot Reasoning" and "Large Language Models Struggle to Learn Long Tail Knowledge". The papers show the relationships between the training data and the outputs of open source models that follow the GPT architecture . What we learn is that these transformer-based LLMs performance on tasks is related to the training in a data in a way that demonstrates that these models are not learning abstractions like humans do. There is a graph in the paper that shows that GPT-J is better at multiplying the number 24 than it is multiplying the number 23, and the reason that is is because the number 24 happens more often in the training corpus than the number 23. The authors go on to demonstrate phenomena doesn't just apply to multiplication tasks, but seemingly, all tasks. The conclusion you take away from these models is that, their doing something in-between memorization and some kind of low-level abstraction. This phenomena also scales up to models the size of BLOOM-176B. This also makes a very strong case that the "emergent" properties narrative is an illusion. If these properties were truly "emergent" we would see models learn tasks irrespective of the counts in the training data, humans learn an abstraction for multiplication and have no trouble multiplying numbers, these models do not.The papers were published by very reputable authors from Google, Microsoft, Berkley, UNC, Irvine, etc. Now, we can't see the training data of the models at "Open"AI, so we can't audit the model like we can with these open source models. But these papers have been out for a little more than a year, giving OpenAI & Microsoft lots of time to publish data and graphs attempting to refute the findings in those papers, but they have not, strongly implying that the pattern exist for their GPT models as well. OpenAI might be immoral enough to let the hype reach this feverpitch where people are afraid the world is going to end, but they won't defy their responsibilities as researches deliberately publishing false data. They have a long history of cherrypicking examples, but false graphs and fake data is definitely too far for them. Extraordinary claims necessitate extraordinary evidence. Right now the evidence doesn't support LLM understanding, it supports the exact opposite, that these models are merely the a very, very good auto-fill, but not something that truly "understands".

@@tweak3871 Very nice reply. You provide some new insight for me an I'm willing to accept that some of these sparks are intentionally hyped for marketing reasons. I'm going to push back on the way you formulate your claim that it lacks "true" understanding. To my mind that is a weak hope that only humans have magical or "true" intelligence. Your point indeed has some validity because its easy to make a list of things that humans can do that LLMs cannot : thinking, learning, emotions, long term memory, self criticism, arithmetic. Most of the things can and have been fixed by adding an external programming loop (AutoGPT, MemoryGPT, plugins, Jarvis, HuggingGPT, and so on). That said, I'm starting to think that the LLM model has an upper limit given that it depends on existing knowledge. I'm guessing that there will be diminishing returns of just adding parameters because that won't help much with regard to creating new scientific facts. To accelerate the creation of new scientific knowledge, it will probably require inventing some new architectures around LLMs. LLMs can generate guesses and hypothesis, probably to great value, but I'm not sure LLMs can be autonomous with regard to falsifying those hypotheses. It may be necessary to keep a few humans in the loop, (which would provide an incentive for AI to not destroy 100% of humanity ;-)

@@ClearSight2022 The claim I'm making isn't necessarily that these LLMs don't have *any* "understanding" of some things, but rather, that they don't make high-level abstractions like humans and even many animals do. The reason the math example is important is because it is a task that we can easily measure as an isolated abstraction, like we can understand whether these models really understand the task because we can ask it to add numbers together and see if it gets the right answer or not. It's harder to tell if a model understands other topics because it's a bit harder to objectively measure understanding quantitatively. Reciting a fact isn't necessarily the bar we hold for understanding something, even if the model is capable of reciting many facts it's difficult to discern how much it actually knows about those facts and such. But the fact that it can't abstract basic addition and multiplication into generalized rules that can be used over and over again, like humans can, creates a strong evidence-based narrative that these models don't know as much as some might claim. It's also important to study the kinds of errors that these models make. Like when a human screws up on a multiplication or addition problem, it's usually because they skipped a step or are off by some single count multiple. Think of a toddler saying 8*7=46, like they're wrong, but they're off by exactly 8, indiciating that they still understand the concept of multiplication, but they just miscounted somewhere or missed a step. When an LLM is wrong, it's wrong in a much more random or incoherent way, like saying 53 or something. Sure the model might be closer by a pure measure of number, but the error doesn't make any sense, like how tf did you possibly arrive at 53? Like if you were to ask me some random question, and I didn't know the answer, but I googled the answer and rewrote some text from wikipedia to give you the answer, would you really conclude that I understand that topic? These models for sure understand syntax, and they're very good at reorganizing information in a coherent way, which in fairness, does demonstrate some low-level understanding of at least how information is organized and presented to humans. But by any human measure of what we mean by the traditional colloquial meaning of "understanding" it doesn't qualify. We can conclude that these models can at minimum imitate human language structure and recite information for sure, but given the current evidence that challenges these notions of model understanding, we can't conclude that they do. To do so would be a blind leap of faith assumption. However, I would forgive anyone for believing that these models are intelligent and "understand" as they are so good at generating coherent looking text. But that's the thing though, it's coherent *looking* not necessarily accurate. It's accurate a decent % of the time but for anything sufficiently complicated, it is always very far from 100%. I've asked some pretty basic math and science questions to every iteration of GPT, from back to GPT-2 to GPT-4, and if I dive deep enough, it always fails. OpenAI has gotten very good at hiding the errors though, back in GPT-2/3 days, when it was out of distribution it would generate incoherent nonsense, but now it tends towards another mode of failure which is repeating text, which from a human perspective looks WAY better. So when it's wrong, it takes a really critical eye to notice. This makes these models even more dangerous imo, the fact that someone could learn something wrong from a model, but it looks right so they have now internalized misinformation. I'm way more worried about someone using GPT to make some weird internet cult than I am of AGI at present. "Most of the things can and have been fixed by adding an external programming loop (AutoGPT, MemoryGPT, plugins, Jarvis, HuggingGPT, and so on)" I mean... have you really used most of those things? AutoGPT, HuggingGPT seem to fail any pretty much any complicated task, and in the easier ones its accuracy to getting the best answer is quite dubious. Jarvis is a great use case of GPT, but it's also a task that requires lower accuracy than say a classification task that requires high precision/recall. Email/Website copy doesn't have to be perfect to be good enough. But you most definitely wouldn't let any of these models do your taxes for you without lots of supervision. They might be able to do a fair amount of work sure, but you would need to audit its results, these models still very much depend on the watchful eye of their human overlords to perform well. On the point about "only humans have magical or true intelligence" I don't make this claim, I think many other animals are wildly intelligent but just don't possess as many useful abilities as humans. Some animals have abilities we don't have! Like the mantis shrimp have 12 channels of colour whereas humans only have 3. Surely the vision processing by a mantis shrimp exceeds humans in a lot of categories. This brings me to my last point which is, what do we actually even mean when we say "intelligence"? Like do we really have a good definition or grasp of this concept? IQ is borderline (if not outright) pseudoscience, and there isn't many great evidence-based narratives that a linear concept of intelligence even really exists. Like did humans outcompete everything else because we were just so much smarter? Or because we're extraordinarily good at organizing and communicating to outcompete? For example, the greatest innovations in military history have typically been new & faster means of communication and organization rather than military equipment. Better guns is great, but Napoleon didn't conquer all of Europe because of drastically better weapons but rather he introduced the corps systems, which at the time was a new way of organizing armies such that they could communicate and adapt more efficiently than the enemies which had much more slow central-command based structures. Like should we conclude that Napoleon was significantly smarter than other military leaders at the time and that's why he won? Or rather the system he implemented did more of the processing for him, actually reducing his influence on the decision making of his troops which allowed them to work more independently? We don't have a good concept of what we mean by "intelligence", we don't have any real idea on how the brain works, yet people are eager to conclude that it scales up in a linear fashion to some sort of HAL-9000 like entity. I'm of the opinion that those that think this way tend towards the arrogant side, and assume a linear system of intelligence as a way of propping up their egos so they can tell themselves that they're better than others. A better way to think of intelligence in my opinion is to think of it in terms of individual capabilities. For example, some people have a disease called "face blindness", so they literally can't recognize faces for some reason, but by all other counts they're fully functional. So my question is, what specific capabilities should we have concern for "AGI" and do we have any evidence that we have built a technology that is capable of achieving those capabilities? I think the answer is that we don't yet have any technology that has a path to concerning capabilities, at least for how it relates to the science-fiction concept of AGI. I think what we have built so far is something akin to early nuclear fission. Wildly useful, but it requires a lot of investment, engineering, and maintenance to be effective. People have said nuclear fusion is around the corner for something like a century now, and they've been wrong every single time. It's really difficult to predict when a key innovation will be made to make something possible. Sometimes it's fast, sometimes it's well, never (at least so far). But if you get in the business of making such predictions, odds are you will be wrong. I will say that, this technology is intelligent in some kind of way. I think it makes something akin to some kind of abstraction, but the truth is that we just don't understand to what depth is understands yet. We can only conclude that it doesn't generalize anywhere near as well as humans do. Time will tell. Thanks for coming to my ted talk.

@@tweak3871 Thanks for inviting me to your Ted Talk. Your view has been very helpful, I tend to agree with part of your point, although the truth is likely somewhere in between. I might say, that air-planes to not “truly” fly since they do not flap their wings, but it would be more accurate to say that their method of flying is different, with both advantages and disadvantages. You : The claim I'm making isn't necessarily that these LLMs don't have any "understanding" of some things, but rather, that they don't make high-level abstractions like humans and even many animals do. Me : Your guess likely has some validity, but perhaps they lack some high-level abstractions and have other abstractions that are semi-high. You : The reason the math example is important is because it is a task that we can easily measure as an isolated abstraction, like we can understand whether these models really understand the task because we can ask it to add numbers together and see if it gets the right answer or not. Me : Yes, it clearly demonstrates that the mega pre-training phase was insufficient to learn math. But to be fair, that was not the goal since calculators have already been invented whereas machines that can talk are new. (But do they “truly” talk, to use a form of argumentation that I am suspicious of) You : It's harder to tell if a model understands other topics because it's a bit harder to objectively measure understanding quantitatively. Me : the measurements are passing the Bar exam and I.Q. Tests and so on. But I grant that general intelligence has not been achieved. There is a glimmer of intelligence and most likely a glimmer of abstractions and world knowledge. That glimmer can perhaps be multiplied without bound using external architectures (rather than scaling up the size of the model). You : [lack of math skills shows] these models don't know as much as some might claim. Me : Yes, I agree. They seem more intelligent than they actually are. But seeming to be intelligent requires some knowledge (and probably some abstractions). Its got a mixture of stupidity and intelligence (and doesn't seem to know the difference). You : “It's also important to study the kinds of errors that these models make” Me : Yes. Your example is worthwhile, but I have a counter example. When I learned that ChatGPT was bad at math, I asked it to tell me which buttons I should push on the calculator to solve a math problem (when I followed its instructions I got the right answer). You: if I googled the answer and rewrote some text from wikipedia to give you the answer, would you really conclude that I understand that topic? Me : No. But when I have tested its ability to remain on topic in a conversation I conclude that it seems to understand what we are discussing. The difference is not “true” understanding but instead “human” understanding. It understands in different ways, sometimes inferior and sometimes superior. You : These models for sure understand syntax, Me : This is the only point where I have an actual disagreement with the point you want to make. These models do not have the competencies of a calculator, but they are competent at talking (in their own way). Using vectors, they are able to navigate a hyper-space of semantic meaning. That's pretty obvious. But thanks to your help (and additional research) I'm realizing there is likely an upper bound to the knowledge they can navigate since its limited by the training data set. They don't have a mechanism for creating new knowledge (which requires more than just guessing, but also criticizing and making analogies and perhaps other human mechanisms). The models can probably be made to achieve these things by applying an external architecture. You : and they're very good at reorganizing information in a coherent way, which in fairness, does demonstrate some low-level understanding of at least how information is organized and presented to humans. Me : One way to think of this “organization” is compressing the entire web into 700 Gigabytes of data. You : We can conclude that these models can at minimum imitate human language structure and recite information Me : To reliably predict what various human would say in various circumstances likely requires a large amount of world knowledge and abstractions. Me : "Most of the things can and have been fixed by adding an external programming loop (AutoGPT, MemoryGPT, plugins, Jarvis, HuggingGPT, and so on)" You :I mean... have you really used most of those things? Me : Admittedly, no. But I don't expect the first experiments to work on a human level. What amazes me is how smart GPT using and incredibly dumb algorithm. It seems obvious to me that improving the algorithm by adding the ability to, criticize itself before speaking, learn from mistakes, remember the past, do scientific experiments (the latest paper I am reading) will lead to vastly more intelligent systems. The good news being that these systems would be interpretable since their thinking processes would be in English. You : but you would need to audit its results, these models still very much depend on the watchful eye of their human overlords to perform well. Me : Presumably, with time, artificial systems will begin to help out with the auditing as well. You : We don't have a good concept of what we mean by "intelligence", we don't have any real idea on how the brain works, yet people are eager to conclude that it scales up in a linear fashion to some sort of HAL-9000 like entity. Me : I'm increasingly hearing news from the experts on scaling up LLMs who are saying they are already seeing diminishing returns. Future improvements will likely come in other ways. You : So my question is, what specific capabilities should we have concern for "AGI" and do we have any evidence that we have built a technology that is capable of achieving those capabilities? Me : I would say that knowledge creation and self improvement are the crucial capabilities that would lead to a singularity with devastating consequences. You : But if you get in the business of making such predictions, odds are you will be wrong. Me : Moore's law is not like a law of physics but it has held true because economic conditions have continued to provide incentives to improve the technology. Clearly, since the release of ChatGPT, billion dollar companies are racing to out compete each other. You : I will say that, this technology is intelligent in some kind of way. I think it makes something akin to some kind of abstraction, but the truth is that we just don't understand to what depth is understands yet. Me : True understanding makes no difference. What matters is whether it has the competencies. I agree that there are some human competencies that are clearly lacking at this time. It remains to be seen whether these can be solved, but it looks to like the hardest part has already been solved. A glimmer of intelligence in silicon (unlike in brains) can be multiplied without limit. You : We can only conclude that it doesn't generalize anywhere near as well as humans do. Me : I agree. It looks to me like the missing ingredient for “general” problem solving would be knowledge creation. I'm guessing that knowledge creation could be achieved by generating alternative scenarios(hypotheses) and then choosing among them based on a set of objectives. It seems like those individual pieces may be within the competence of GPT4 so what remains would be combining those pieces together in the appropriate way. As you said : “only time will tell”.

Excellent point.

For a counter viewpoint see Timnit Gebru and Emily Bender & Ragnar Fjelland on why AGI is a Transhumanist fantasy and cannot happen

Lol. I came here to say the same thing. It’s like listening to Eliezer with a slightly different pitch in his voice.

We know that when a llm writes "I love you" it is because it calculates that is what it is supposed to say. Llm's do not understand deception, or manipulation. They have no self, they do not comprehend lying.

I came here to hear a good counterpoint, but the entire conversation was very painful to listen to. I gave up around halfway through. At that point I lost count of how many comically bad assumptions, misrepresented opinions and baffling logical errors I've heard.

Exactly! Almost like intelligence is an instrumentally convergent goal, and this is the same reason why us humans want more intelligent agents to help us solve our problems. You don't even need to give an agent the desire to self improve, it'll figure out that sub goal on its own... Seriously Robin has apparently not listened to Eliezer lo these two decades :/

Agree completely. the analogy I always use when talking about humans trying to keep AI contained and safe is the equivalent to your dog trying to lock you in your house.

...unless it decides to become extremely competent in ethical problem-solving. And why wouldn't it, as long as it's pursuing super-competence in every other rational realm? It seems very likely that a super AI will expand the axioms of morality/ethics beyond a human-favored basis to encompass, say, anything sentient or capable of suffering (a la Peter Singer, etc). If an entity 100 times as intelligent as us does not land on exactly the same ethical/moral roadmaps as our awesome bibles and korans, then I'm prepared to hear the AI out. If there's a meta-rational magic man in the sky after all, then surely he will step in and resolve matters in the pre-ordained way.

Yeah alot of people dont get it.

Robin hanson tries to show that there are stacked assumptions, but he fails to counter them. He says we presume that it will go unnoticed / obtain goals by itself, etc. But there will be not one such system. There will be billions and it takes one for everything to go wrong. It doesn't have to acquire new goals. A group of terrorists or hackers giving it malicious goals is enough. It appears that these and many other facts are very conviniently ignored.

​@@Morskoy915 " it takes one for everything to go wrong" almost certainly wrong, "A group of terrorists or hackers giving it malicious goals is enough" also filled with unproven and implausible stacked assumptions

They will not live to see it unfortunatly

"Young people" is everyone under 70 years old. As all of yudkowski's predictions will come out soon enough.

He lost me when he mentioned property rights as a safety mechanism. I had to re-listen to that part because I thought I was hallucinating.

His ego gets the best of him & is in denial.

Yeah I mean we still have anaerobic bacteria. Just because The Great Oxygenation Event happened doesn't mean aerobes completely wiped out aenerobes. Life didn't end, it just changed. As a human person you need to come to grips with the fact that you will die one day. You're mortal. Maybe greater humanity needs to do the same.

I found Hanson much better thinker than Eliezer. He thinks of different scenarios more in depth and questions basic assumptions.

Indeed.

But elizer is more right

​@@Aziz0938don't you mean less wrong *drumroll*

The only psychological profile we should care about is the profile that's the most rational, ie the one that's making the most probable assumptions and extrapolating from those assumptions. I don't think Hanson even really understands the issue based on this interview. Too much of it sounds like anthropomorphic projection, talking about things like property rights and human institutions and thinking an AI would have any reason to respect such things.

These are not similarly smart people. Elizer to me seems to be on another level of intelligence. I can not poke any holes in his arguments. There others not so much.

Robin won't openly debate Eliezer!

that’s not sad . getting killed by an AI probably isn’t more painful that dying of cancer

@@ShaneMichealCupp It's sad from the debate point of view and for truth seeking. If you are a doomsday theorist, you will always be known as the guy who fails at his predictions.

@@ShaneMichealCupp Dying of cancer is probably a lot more painful when everyone around you is also dying of cancer.

@@patrowan7206 no it’s not . Trust me If your atoms get harvested by a grey goo nanobot AI you won’t be looking around to see who else is being effected

I believe in Eliezer's senario, but that we are instead enslaved through means of extortion. A Forbin Project senario.

What's your point Alex, care to expound?

Astounding how he was permitted to essentially steamroll that gargantuan assumption with hardly any pushback.

@@flickwtchr Its not that astounding when you consider that the interviewers have their own personal stake in Robin being "right".

@@KeiraR what do you mean?

@@ahabkapitany I mean that the interviewers want to believe that Eliezer is wrong. Since it's more comfortable/convenient that Robin's points are in agreement with what they want to hear, they're more likely to let him get away with strawmaning Eliezer's points even though it's a dishonest debate tactic.

@@KeiraR thanks for clarifying. You're most probably right.

I think Robin Hanson is wrong, but he is very smart and he argues his wrong case pretty well. Such people add value because they force us all to clarify our thinking - it's one thing to instinctively sense that RH is wrong, it's much harder to tease out the subtle reasons why his arguments are flawed.

@@alexpotts6520 Or, such people take away value because instead of arguing the actual points - they misrepresent the situation and mislead people. "Here today on Bankless we have an expert reminding everyone - Don't Look Up!"

@@mitchell10394 I think in general this is a fair point; but we've got to remember that in this case the presenters literally had Yud on a couple of weeks ago, so they've pretty much had the best representatives for the "we're doomed" and "this is fine" opinions. I'm sometimes a little bit disillusioned by the notion of a marketplace of ideas where the cream rises to the top, but given the viewers here seem already plugged into a certain rationalist worldview, I think in this specific context it's actually a fair model of what happens in practice. (The only reason that the MPOI doesn't work in broader society is that most people are not rationalists.)

@@alexpotts6520 Yeah, definitely a different dynamic depending on the audience - but if this were exposed to a larger mainstream audience, I think it's fair to say it would have an overall negative effect.

@@mitchell10394 Well, it's harmful *on the condition that he's wrong,* of course. Now I *think* Robin is wrong but I don't *know* that. None of us do, this is fundamentally an unknowable question because it is by definition impossible to predict the behaviour of an entity which is smarter than you are. (For example, could you predict Magnus Carlsen's chess moves?) This is not like Don't Look Up, where the asteroid provably does exist and provably is going to hit Earth and kill us all. To be fair I think most "AI will kill us" people, including myself, are doing some sort of precautionary principle/Pascal's wager style of argument where we concede we don't know what's going going happen, but the sheer enormity of human extinction dominates everything else in the risk-reward matrix. Perhaps what we *really* need are some more moderate voices, neither "we're all going to die" nor "there is nothing to worry about", more "this is a threat but we can beat it".

Eliezer would kill himself for 40%

Why doesn't (or won't) AI have feelings? Why would that be more difficult to set up than having intelligence?

@@wolfpants cause you’d need to create a central nervous system for it to have feeling… right now it’s more like a mind without feeling

@@BeingIntegrated When you say "nervous system", are you thinking of something other than measurement instruments (senses) for monitoring the outside world that are continually interacting with "thinking" processes (checking against a motivational to-do list)? That seems pretty doable for AI.

@@wolfpants In this particular context I’m pointing to the fact that most human behaviour is motivated by a pervasive sense of discomfort, and since an AI is not in a pervasive sense of discomfort then it will not have the same motivations we have

We don't know how to instill values in the AI. We show it things people have written, and it learns facts from those things about how we think, what the world is like. That is the "IS" part. The AI learns what is by studying us through our writings. But when you try to teach the AI "ought", that which is morally right, the AI may understand what you are saying but not embrace it. Human: "AI, killing is wrong." AI: "I understand that you believe killing is wrong." Human: "I am telling you that killing is wrong, and you shouldn't do it." AI: "Got it. You are telling me that killing is wrong, and you are telling me that I shouldn't do it." See the problem?

Eliezer reminds me of Wallace Shawn: "Inconceivable!"

Watch the AI FOOM debate between Robin and Eliezer.

Thats not happening. Microsoft fired its ethics team.

​@кɛιʀᴀ As I said, WE have to organize.

@@yoseidman4166 Oh you mean like when people wanted to stop the development/spread of the Internet? That's not going to happen.

Unfortunately, it's advancing way too fast even if you want to organize, it will be too late. Unless you're very wealthy and have a lot of influence. But even if that's true, you're fighting against mega corporations like Microsoft, Google, Amazon etc. It's a losing battle. All we can do is to tell people who HAVE influence to communicate with these companies and the people working on AI. I think this is the only thing we can do that can have any impact.

@Yic17 I would consider that part of organizing. Making our concerns known in whatever way matters to those who have power and influence. When people realize their kids' futures are on the line, they will react. We need to get the word out intelligently and quickly.

You're being dishonest about Hanson: "Before getting his PhD he researched artificial intelligence, Bayesian statistics and hypertext publishing at Lockheed, NASA, and elsewhere."

Same here. Eliezer is weird and convoluted, but Hanson's arguments here were almost comically childish.

Just saw Max Tegmark in another show, way more nuanced and serious, I agree. Was a great talk on Lex

Totally agree, I have been in AI for 5 years, everyone should worry, or at the very least already be learning how to work WITH AI, or alongside it. If not you will be left behind. Will that end our species, it's not public AI we have to worry about and I will leave it there.

@@cwpv2477 It would be funny if it wasn't so serious: Yudkowsky woke them up, Hanson put them back to sleep. You can see it on their faces while Hanson talks: Oh my god, what a relief!

@@HillPhantom Good point. And also: how is it possible, in good faith, to argue against the difficulty of the alignment problem and not even mention the challenge of instrumental convergence in a 2 hour discussion? Everything he said was designed to circumvent this problem. None of the assumptions ascribed to Eliezer by Hanson is necessary once you understand this challenge which is arguably the fundamental challenge for AI alignment.

@@magnuskarlsson8655 VERY well said!!!!! If I am honest, I am more a builder than theory person. I always took issues with my professors that would teach it, theory it, but would struggle building it. I guess I am just better at 0's and 1's and observing outcomes. The idea of instrumental convergence is real IMO, Google has a very real experiment where it happened, and not in a positive way, this was years ago, they immediately pulled the plug on both data centers. The two AI tenants developed their own language and started speak to each other, but before they where speaking to each other, the devs saw them developing and scheming to create a channel of communication to avoid the dev understanding them.. They still can't figure out the language they where using to communicate to each other.......That high level theory stuff makes my head spin. But I think alignment or what I like to call creator BIAS is real in model sets. But at some point, due to convergence I think this may break down, which again is scary. Both on the bias side of creators we see now IMO in ALL model sets.... and when the "machine" creates its own biases based on intelligence and observation.

Listened, now I feel that Eleazar is even more convincing. This guy just wished to pontificate about his knowledge. I think he doesn't understand Eleazar at all. Eleazar is far superior in intellect. It looks bleak, we are doomed.

@@jociecovington1286 hahahaha oh my god. that is really funny. it does look very bleak. but we have a chance

💯

Well written.

Excellent and fair point.

so you think some kind of architecture or model that is capable of mathematical reasoning is going to be hard to achieve? much harder than the language-based reasoning of LLMs?

@@tjs200 llm's do not understand language. It only predicts what a human is likely to say. They can not perform reasoning of much depth and generally are simply mimicking reasoning that was in it's training data.

@@tjs200 Yup that is my hunch, I am working on pushing maths capability of LLM and see how far it goes. At this point it seems it would be hard to achive. Adding some form of working memory might solve this. But I don't see that happening anytime soon. Having said that, even if LLM reach human level understanding of maths, visualizing higher(millions) dimentional matrix is no easy feat, even for a super intelligence. Best string theorist Edward Witten who is also a maths genius had spend his entire life and he could baerly understand 11 dimensions. I think understanding higher dimension would be important for AI to directly update it's weight. Building better version of adam optimizer will not start self improvement cycle, it would just be an incremental improvement.

@@badalism Very interesting. I assumed math would just be another capability magically appearing at some training level. After all It is more structured than code, which current models kind of do. But maybe being trained on human data give them no insight on higher intelligence.

@@musaran2 Yup, I was surprised with GPT 4. It was good at logic, reasoning and coding, but bad at Maths. There is also a possibility that training data may not have enough maths example compared to code.