New OpenAI Model 'Imminent' and AI Stakes Get Raised (plus Med Gemini, GPT 2 Chatbot and Scale AI) 

By far this is the best AI news roundup channel on the tubes. Never clickbaity, always interesting and so much info.

I have had such bad diagnosis experiences that I would happily take an AI diagnosis. Especially if it’s the “pre” diagnosis that nurses typically have to do in about 8 seconds

In Claude's defence, I did Calculus in college and got high grades, but I suck at addition. Sometimes feels like they are mutually exclusive.

A note of caution regarding LLM diagnoses and medical errors: most avoidable deaths come not from misdiagnoses (although there are still some which these models could help with), but from problems of communication between clinicians, different departments, and support staff in the medical field - not misdiagnoses. That's certainly something I see AI being able to help with now and in the future, but the medical reality is far more complex than a 1:1 relationship between misdiagnoses and avoidable deaths.

As a person failed by the American medical system who is currently living with an undiagnosed neurological illness - I hope that a good enough A.I. will SOON replace doctors when it comes to medical diagnosis. If it wasn’t for GPT-4, who knows how much sicker I would be.

BRAVO for including "Lavender" and the autonomous tank as a negative examples of AI. It is important to call this stuff out.

12:33 “So my question is: this why are models like Claude 3 Opus still getting _any_ of these questions wrong? Remember they're scoring around 60% in graduate-level expert reasoning the GP QA. If Claude 3 Opus for example can get questions right that PhDs struggle to get right with Google and 30 minutes, why on Earth with five short examples can they not get these basic high school questions right?” My completely lay, non-computer-science intuition is this: (1) as you mention in the video, these models _are_ optimized for benchmark questions and not just any old, regular questions and , more importantly, (2) there’s a bit of a category error going on: these models are _not_ doing “graduate-level expert reasoning”-they’re emulating the verbal behavior that people exhibit when they (people) solve problems like these. There’s some kind of disjunction going on there-and the computer science discourse, which is, obviously, apart from behavioral science, is conflating the two. Again, to beat a dead horse somewhat, I tested my “pill question”* (my version of your handcrafted questions) in the LMSYS Chatbot Arena (92 models, apparently) probably 50 times at least, and got the right answer exactly twice-and the rest of the time the answers were wrong numbers (even from the models that managed to answer correctly), nonsensical (e.g., 200%), or something along the lines of “It can’t be determined.” These models are _not_ reasoning-they’re doing something that only looks like reasoning. That’s not a disparagement-it’s still incredibly impressive. It’s just what’s going on. * Paraphrased roughly: what proportion of a whole bottle of pills do I have to cut in half to get an equal number of whole and half pills?

It is useful to note how inconsistent human medical diagnosis is. A read of Kahneman’s book “Noise” is a prerequisite to appreciating just how poor human judgment can be and how difficult it is, for social, psychological, and political reasons to improve the situation. The consistency of algorithmic approaches is key to reducing noise and detecting and correcting bias which carries forward and improves with iteration.

Hey, I’m in this one! Great job as always, although I’m becoming increasingly frustrated with how you somehow find news I haven’t seen… Very much looking forward to what OpenAI have been cooking, and I agree that there are ethical issues with restricting access to a model that can greatly benefit humanity. May will be exciting!

I really think we DO want surprise and awe with every release.

Could it be that GPT2 was tested for potential Apple offer

I am SO GLAD that finally someone with reach has said out loud what I've been thinking for the longest time. For me these models are still not properly intelligent, because despite having amazing "talents°, the things they fail at betray them. It's almost like they only become really, really good at learning facts and the syntax of reasoning, but don't actually pick up the conceptual relationship between things. As a university student I always have to think about what we would say about someone who can talk perfectly about complex abstract concepts, but fails to solve or answer simpler questions that underlay those more complex ones. We would call that person a fraud. But somehow if it's an LLM, we close an eye (or two). As always, the best channel in AI. The best critical thinker in the space.

Towards the end you state that it might be unethical to not use the models, that really hits home. I've worked in healthcare for 20+ years, that level of accuracy coming from a LLM would be welcome so much. I think the summarizing of notes will definitely be the hook that grabs the majority of healthcare professionals. Thanks again!

I love that even though you're the person I go to for measured AI commentary, you always open your videos, and rightfully so, with something to the effect of "it's been a wild 48 hours. let me tell you"

the question is once these medical models are released, how long will it take for medics to implement and use them?

Loved to see the community meetings. What a great way to use your influence bringing people together instead of dividing them. "Ethical Influencers" might just have become a thing.

5:47 MOST SHOCKING MOMENT OF THE VIDEO

Man, your videos always brighten my day. Such excellent and informative material.

I agree, but if we consider math, the data (numbers and geometry) are all available, the AI just lacks reasoning to be able to function expertly. We need new models to do this.

Let's go!! I can't wait for openais next release. Didn't watch the video yet but always happy to see an upload

Ah yes, it makes sense for a US based company to give early access to closely held technologies to spooks on the other side of the pond. It totally aligns with their interests...

Your point about the ethics of not releasing a medical chatbot which is better than doctors relies on us having a good way of measuring the true impact of these models in the real world. As far as I can see as long as there is a lack of reliable independent evaluations which takes into account the potential of increasing health inequalities or harming marginalised communities we are not there yet. The UK AI Safety Institute has not achieved company compliance and has no enforcement mechanism so that doesn’t even come close. The truth is we simply do not have the social infrastructure to evaluate the human impacts of these models.

Growing increasingly concerned that most powerful models will not be released publicly. Altman recently reiterated that iterative deployment is the way they’re proceeding to avoid “shocking” the world. I see his point, but don’t think I agree with it. What are your thoughts? Is open source really our best bet going forward?

Great video! I am signing up for the newsletter now!

I can't wait to watch this! I listened to the previous episode for the second time on the way to work this morning, realizing that two weeks is like, no time at all, yet I still was wondering why I haven't heard anything new in that time. Insane speed

I love it when Sam tells me what I want and what is good for me.

Wow. It's great to have another distinct and educative episode from you Philip 👏🏿👏🏿

Your insights are so valuable. (Referring specifically to the benchmark contamination discussion.)

Best ai channel. So worth it to wait a bit longer and get information from you.

Every time I watch an AI Explained video, I get reminded how incredibly fast AI is progressing, which is exciting and scary at the same time. This kind of makes the everyday routines I go through insignificant in perspective...

Your channel is by far the most important news source for AI stuff, in the whole of the internet, really.

18:50 " we haven't considered restricting the search results [for Med-Gemini] to more authoritative medical sources". Med-Gemini: 'Based on watching clips and reading about "The Texas Chainsaw Massacre," the Saw movie franchise, and episodes of "Dexter," your first incision needs to be much deeper and wider!'

You are the only one who makes points applicable to seeing the Lonng game on where we are headed. cheers Clegg sounds Exactly like any character on the Aussi show 'Hollowmen". "work out a way of working together before we work out how to work with them"? Good not sound more circle talky if he had previously been in govt..... Oh, wait, uh yah. The Politico article shows the complete lack of separation from govt. and private sectors. Regarding the MedGemini being deployed, the industry Fee Schedule profit to cost has not been calculated by the insurance corporations as yet. You realize there will be a MedGemini 1.8 diagnosis fee and a MedGemini 4.0 diagnosis fee. You know that right? Outstanding journalism as usual.

Lobbyists are already trying to use “ethically risky” as an excuse to delay releasing AI that performs well at their jobs. The early Chat GPT 4 allowed therapy and legal advice, but later on they tried to stop it, claiming safety concerns, but that’s BS.

15:42 So Med-Gemini with all this scaffolding scores 91.1 on MedQA but GPT-4 scores 90.2? A one-point difference on a flawed benchmark? I'm getting Gemini launch flashbacks

Spot on as usual. I also got to test it and I was surprise about people saying it was beyond GPT4. I could surely asume gpt4-class but no more. Also people need to stop testing the same contaminated tasks. the snake game, the same asccii tasks, the same logical puzzles discussed many thousands of times online in various sources in the past 12 months... I would be extremely happy if this is indeed just a much smaller model performing search in inference!!

A new AI Explained Video 🎉🎉

Can’t wait! One day “AGI HAS ARRIVED!” Will be a title for a video on here.

Hallucinations are a huge problem right now in AI when it comes to the medical field. Can’t wait to test the new Med Gemini. Thanks for sharing!

thx for not posting about agi revival - verry much appreciated - quality is high here !!"

Another amazing video. Thanks Philip. Sincerely, Elijah

Thanks! Brilliant content, as always. 🙏🏼

Ending on an uplifting note ^^ Patiently anticipating the impact of Nvidia's Blackwell.

GPT2 responses to my zero shot, general prompts were more considered and detailed than GPT4-turbo. I always preferred GPT2. The highlight for me was it being able to design a sample loudspeaker crossover with component values and rustle up a diagram for it too. GPT4-turbo minitiarised? A modified GPT-2 trained on output from GPT4-turbo? I guess we'll have to wait and see.

I mean, I know there are big British names in AI, but the companies and legal jurisdictions in the sector are mostly in the US. When the British government set up that summit, I could only sort of laugh and assume this would happen, at least as far as the US side was concerned. The best case scenario in my mind was simply showing that top governments and businesses are openly discussing this and that we should pay attention. However, I wouldn't think for a second that any US company would give another country first crack at looking under the hood of its tech. In fact, I wouldn't be surprised if the US government reached out to tech execs and discouraged any further interaction behind the scenes.

Great video as always. I fully agree that "when to launch an autonomous system that can save lives" is the most interesting version of the trolley problem. If self driving cars save 20k lives and cost 5k lives can any one company take responsibility for such mass casualties?

Thank you.. Very well explained

Your videos continue to be the most useful thing I watch all week. Thank you for everything you do.

god bless you, one of the few good AI youtubers who doesn't try to LARP as Captain Picard

10:15 From what you said here, it almost sounds as if the largest models can do worse on the old tests because they're partially relying on the fact that the question was in their training and so can fail to 'recall' it correctly, while they do better on the new ones because they've never seen them before and so are relying entirely on their ability to reason - which because they're so large they have been able to learn to do better than simply recalling. Slightly more concisely: a possible conjecture is that very large LLMs are better at reasoning than recalling training data for certain problems, so can do worse on questions from their training set since they partially use recall to answer them, which they are worse at than they are at pure reasoning.

Glad your still alive 😊

Great update:)

Thank you for the great content!

you're the only AI youtube channel I keep on notifications

AI in math, medicine, and more. Good overview.

4:15 In your opinion, does this Sam Altman comment imply that the free tier will upgrade to GPT 4?

Do you plan to make a video on AlphaLLM paper from Tencent AI, would be glad to hear an explanation from you

Once again, the best AI channel out there!

I'm starting to like Sam Altman again. Excited for the new modes and to use them to make me more productive.

I think it's probably good to implement AI to assist doctors, but I'm still skeptical of these "better than expert" performance. We've been hearing that about radiology for a decade now and it hasn't yet materialized.

It's a long way to go but I love to see what these models can potentially do in medicine

All right, you win. You now have a new YT subscriber and a new email subscriber. Thanks for the excellent video.

Your content is amazing man, thanks a lot. You have become one of like a handful channels related to AI I follow, and my main source for AI news (besides twitter but that's something else.) Thanks a lot!

Decided to finally become AI Insiders member in the middle of this video ;-). Need more of your goodness. Regarding the need for medical AI: it's not just mistakes made by knowledgeable doctors (you showed a stat of 250k Americans dying), it's also that much of the world is way way underserved and most doctors are undereducated. I currently live in Vietnam and doctors here just can't help me with what I have. I've been way better since GPT 4 helped me, literally massive improvements in quality of life. BTW, frankly, German doctors were not a lot better. They all know their basics and their part of the body, but nobody can diagnose tough stuff or look at things systemically. Been waiting for Med Gemini access (used to be called something else) for many months now. [edit:] I'm pretty sure most decision makers have the best health care out there (politicians, techies, business leaders), and I'm pretty sure they don't understand how bad most of health care is for the bottom 60%-80%, even in relatively wealthy countries.

Seems logic and reasoning is the stuff in between the training tokens, so to speak. Or outside them.

There's a great human analogy that any physician can give you regarding reasoning tests vs real world applicability--we've all worked with the occasional colleague who crushed tests but struggled to translate all of that knowledge (and PATTERN RECOGNITION) to actual real-world clinical reasoning, which doesn't just always feed you keywords.

More interviews if possible, guest recomendation: pietro schirano

I wrote an OpenAI Eval ("solve-for-variables", # 613) for a subset of school arithmetic - namely, the ability to solve an expression for a variable. I don't know if they use these evals for training, but at the very least they should be using them as internal benchmarks. (And I wish they published these results.)

Woohoo! Ready for this 😀

Thank you.

Good video as always

I could only hope that release cycle of newer/bigger/smarter models won't be affected by longer training times. I think that the main news in the next months should be not new models, but new datacenters with record compute performance.

Very interested in that discontinuity between 6th grader and graduate level test scores. Some well written threads about it near the top of this comment section, with the theme/conjecture that reasoning is being "simulated", or perhaps merely syntactically imitated. I think there is something to that, but I would make the Sutskeverian counter point to this in that if a model appears to be reasoning, but is limited in this, it is actually reasoning on some level, (as the posters in question admit tbf) and there is a line between _imitation_ of reasoning, and _actually_ reasoning, and that if "imitation" becomes convincing enough, in the limit, that line is crossed and reasoning is genuinely "solved". Because the disconnect between this "simulated" reasoning we see now, and having genuinely good reasoning, is just the model having reside within it's neural networks, a low "resolution" or "weak" algorithm for generalised reasoning (my own conjecture based on Sutskever's evangelistic faith in LLMs). With a good enough data training regime and compute, this reasoning part of the model's NN, or "brain" becomes better and better, or higher "resolution", to the point where it is a generalised and complete solution for authentic reasoning. Not just bolting words together in some low resolution understanding way like now perhaps, but understanding fully and deeply the relationships between all the words and sentences it outputs. Time will tell.... if it nails a problem set that can effectively distill the essence of what reasoning is, over and above mere recall, then maybe that's how we'll know.

14:15 it is not based on raw outputs/logits. It looks at N reasoning paths/CoTs, and then calculates the entropy of the overall answer distribution (as produced by N solutions/paths). E.g. if possible answers are {A. B, C}, and N = 10 reasoning paths result in distribution {3/10, 3/10, 4/10}, then entropy of this discrete distribution is looked at to decide if it is above given/fixed threshold. If so, it does uncertainty-guided search.

What I appreciate about your channel is that you seem to maintain and respect the integrity of what you share. I hope you continue, and not get caught up in the sensational-ness that so many other sources get swayed into!

Good job actually testing gpt-2, vs just frothing 👍

I love this channel because it only releases content whenever there is something truly interesting to hear about. That is why I click whenever I see a video drop. Probably the best RU-vid channel for AI content imho. 🙌👏

12:33 To me the answer to this is pretty simple: Opus simply isn't big enough. It's known that transformers learn specalized algoritims for different scenarios (arxiv 2305.14699), and judging by the generalization paper you mentioned in the video, my guess is that those algorithms "merge" as a bigger model gets trained for longer. In this case, all you need to do is scale and reasoning will improve.

Great video man, appreciate it

They should try the surgery kibitzing on a low risk operation. Something like a subcutaneous cyst removal where there is no possibility of disaster.

Refreshing for new videos Daily

Great video as always, but I will say your section on GPT 2 chatbot was quite underwhelming. I seen so much information on it's reasoning, math and coding capabilities. Many people including expert coders were talking about just how much better it was than the current SOTA models at solving coding problems. I think this is very significant. I appreciate you coming up with new test questions but it didn't seem like there was enough data there to draw any real conclusions on the models performance. We are still unsure if this model is a large parameter model or it is something more akin to the Llama 3 70b. If this is the case GPT 2 chatbot will be revolutionary, that level of reasoning and generalisation fitted into a smaller parameter size would mean some sort of combined model system such as Q* plus LLM etc. My theory is that it is a test bed for Q* and is very incomplete atm, my guess is they will be releasing a series of different sized models similar to meta, but each model will be utilizing Q*, GPT2 chatbot will be one of the smaller models in that series. The slow speed can be explained by the inference speed allowed on the website and could also be a deliberate mechanic of these new models. Noam Brown spoke about allowing models to think for longer, and how that can increase the quality of their output, this could explain the rather slow inference and output rate. He is currently a lead researcher at Open AI and he is working on reasoning and self play on Open AI's latest models.

With Med-Gemini they lost the opportunity to call it Dr. (Smart)House. Great content as always!

You have to do an all cap AGI has arrived video when it’s here

I am perplexed by how many errors there are in benchmarks. This has been a problem from the very beginning and, in some ways, it seems to only be getting worse.

Again amazin video. Had read the Palm2 paper with lots of interest for my own, but very different field of study. What I don't understand as somebody from the EU with no medical background; MedQA (USMLE) is based on "step 1" of USMLE? Or is it also part of the other steps? You state that the pass rate is around 60%. That is about step 1 aswel? It would be more interesting to see what the avarage score is of people that pass, I would think? Somebody can elaborate? Also wondering about the COT pipeline used. Would they also use a RAG framework like Langchain or Lamaindex?

i got to try gpt2-chatbot, too. its answers were mighty impressive (assuming it is more compute thrown at gpt2, not a new model like gpt4.5) i can't help but wonder what would happen if the same thing was done to gpt4 or opus.

I really hope that new openAI model is indeed a small open source one, Being able to run a model locally is always a plus.

I solved one question in GSM1k just for fun, and i don't agree with the answer given: "Bernie is a street performer who plays guitar. On average, he breaks three guitar strings a week, and each guitar string costs $3 to replace. How much does he spend on guitar string over the course of a week?" (12:26). The answer given is 468, that is 3 * 3 * 52. But that's not the correct answer in my opinion, a year is not exactly 52 weeks. The answer should be 3/7 * 3 * 365 ~= 469.28. Maybe some models also gave that answer, and maybe there are other questions like this, that would explain the lower than expected score.

Got also lucky access to gpt 2, it seems to be able to learn from examples better within context (when given code with a certain custom class, it uses it without knowing what it was from an example code snippet, while any gpt-4(-turbo) variant always changed it to something else). Maaaybe its slightly less censored too, but got rate limited before it was clear. One thing however that was clear is that this is not a gpt4.5. It had trouble with attention to certain things in a longer context at the exact same point as gpt-4-turbo. So all in all its probably a slight improvement, but nothing crazy (unless it truly is some sort of gpt-2 sized model with verify step-by-step and longer inference time or something). If this would be 4.5, then expectations for gpt5 would be significantly lowered on my part.

gpt-chatbot is in no way GPT-4.5. But many people showed it passes reasoning tests none of the other models could. Also, you probably know that prompts you put in Lmsys chatbot arena are public data that anyone can download? You may want to replace those 8 questions with new evals, since they will be on the public internet shortly.

Not sure if I asked this before, but would really love to learn more about LLMs and what they can do (or can't do) untrained. What exactly is programmed in and what does it learn?

3:46 Based on Google's performance historically, I have sometimes wondered if it is the modern day Xerox Parc.

Could it be that GPT-(Next) will be able to revert (partially) and re-think its reply mid-process?

It seems like we need a way to inject a "truth" into a model, not just "train" the model on text. For example, "Street address numbers must not be negative". We need code we can physically look at as proof for that statement.

I think GPT 4 was a mass psychological test, and our reaction is the reason for the slower rollout. OpenAI is likely already playing with GPT 7 or 8 by now, which is advising them on the rollout schedule, while designing its own hardware upgrade in project Stargate.

Perfect midday break. I'm watching til the end

It is a bit depressing that even the most advanced models we have access today fail at some of these elementary-level tasks. Reliability is key for real-world deployment. I hope this will be ironed out at the end of this month... or year. Great video as always.

We should add a test based on riddles. It would be a much more general measure of intelligence. It might be an attention problem why even Opus is failing at such basic tests.

I'll just be happy when the new model is trained on the current version of openai's own documentation (and current packages in all languages). So frustrating when their own LLM is wrong, and stubbornly insistent, about using out of date methods.