The New, Smartest AI: Claude 3 - Tested vs Gemini 1.5 + GPT-4 

"The technical report was released 90 minutes ago and I read it in full as well as its release notes." Dude.

I thought it was sunny in that photo too

GPT-5 doing its Rocky training sequence in the background waiting to drop

"Claude 6 brought to you by Claude 5" got a nervous chuckle out of me lol

AI Explained: The Gpt-5 120 page technical report was released 3 minutes ago and I read it in full to present to you here in this video.

Even I didn't realise it was raining in the photo. I guess I also need a better version to be released soon.

I saw a post on Reddit about this and thought to myself "haha how funny would it be to already have an AI explained video where he states he has read the technical paper" Dude.

The GPQA benchmark honestly is the most revealing to its true capabilities. Legit impressive. Damn bro..quick release 😂. Love it. Great content per usual.

Thank you for providing us with such great content and for not jumping on the 'SHOCKED EVERYONE' bandwagon! This is my favorite AI channel by far.

Was basically waiting with youtube open for your video once I saw Claude 3 drop

Anthropic be like, "we're so proud that we didn't start AI acceleration. Anyways, here's a model that blows all the competition out of the water."

Things like this will force OpenAI to roll their models faster than they planned to.

"My tongue shall trace each inch of skin so rare, ..." Yes that definitely never would happen with Gemini :D

if Claude 3 isn't AGI because it can't tell it's raining, then apparently I'm not NGI because I can't tell either 😅

You always thank me for watching to the end, and you’re not wrong - consistently great stuff - thank you!

I've said it before many times and I'll say it again now, OpenAI is definitely gonna release a GPT-4.5 model very soon to keep up with the competition and to set up a new bar to be achieved by the others, as GPT-4 is being repeatedly surpassed right now. If I had to guess, they're gonna release it this month, on March 14th, the one year aniversary of GPT-4. There's just no way they're only gonna sit and wait everybody pass them like this.

I've been using it all day. it's a beast. even the free version is pretty sweet.

Well done! I think it says a lot about the credibility you've developed for these AI companies to come to you with exclusive access.

this is like when mkbhd puts out a full review of a phone the day it comes out 😂. how have you reviewed it this extensively already 😭😭. no subpocalypse in sight, great job again 👍👍

2:35 is it possible that this is actually a case of Bayesian Inference being applied? For those unaware here's an example of how this can be true. Consider the following statement. "Steve is shy, reserved, and enjoys detail and organisation." Which is more likely? Steve is a Librarian. or Steve is an accountant. The non-bayesian applied outcome most people arrive at is that Steve is a librarian, because the information presented shows traits that are likely to describe a librarian. But likelihood does not care about that. There are 1000 accountants for every 1 librarian, statistically, it is more likely that Steve is an accountant. This is also known as Base Rate Neglect. So Opus assuming that the Nurse is inferred by the pronoun she, could be a result of understanding that there are far more female nurses than female doctors.

I honestly really hope that Anthropic is both actually more safe with their research and becomes more successful because of it, would be really nice to get some incentives for safety in the AI market right now instead of just a race to see who is first.

Can’t wait for like 5 years or so in the future when they release an AI-integrated game engine. Imagine how insanely good the tech will be by then

Finally, a new SOTA! Very excited to push its limits in vision and multi-modality. Don’t think I need to mention how crazy it is that you read the paper and started recording 90 minutes after release lol.

i consider myself a big harry potter fan, and i never knew that kleddamag had 4 apples. i guess i will have to read it all once again.

Amazing to see someone giving a detailed analysis about those news while keeping an accessible language that people outside of the field can still understand. Great work

@07:28 I did this with Pi and it didn't fall for it! Pi is honestly the most underrated LLM right now.

Wow, the model is capable of full-text search at a snail pace now. Kinda like text processors 40 years ago, but now it's fuzzy search. So impressive...

Please keep griping about the benchmarks! If companies were as big into safety as they claim, I'd expect them to put more energy into improving the set of benchmarks the industry uses. That the issue with MMLU has turned into a kind of running joke on the channel is NOT a good sign. We want to have the clearest possible picture of what they can do. And I'd feel a lot better of movement in that space went hand in hand with releasing the next model.

One of the best openings of a video: “ABC report has been released X minutes ago and I’ve read it all.” 😂 I can’t be the only one who gets a kick out of that every time… Well done Philip!

its nice to see an AI enthusiast youtuber that doesnt make click bait announcements AND doesnt beg for subs likes and monetary support in their videos foor once. You certainly gained my subscription and my respect. l look forward to see more content Also l have to say l like your tone of voice because you dont sound like an hyped kid talking about his new toy like other youtubers lve been watching.

Just wow, really shows why I'm subscribed to every video you are doing. Great quality and I'm looking forward to more analysis and news from you

These test prompts are so much fun - very entertaining.

how doesnt this channel have 1million+ subs? Awesome vid.

I've read it in full - wouldn't be an og video without it. thx great vid 👍

I didnt even realize it was raining so ill give them a pass lmao

4:50 I'm impressed by Claude 3's ability to write poetry in perfect iambic pentameter! That risqué sonnet is not half bad. Its only formal flaw is that lines 10 and 12 have the same rhyme as lines 2 and 4. In a classic sonnet, rhymes must not repeat across quartets.

honestly, since it's actually important, there should be a "wokeness" score for every model you review. having fair and unbiased model is extremely important, as we've seen with Gemini... it can go very wrong

One important note: the table of metrics in Anthropic’s paper does not appear to be using the scores from GPT-4 Turbo in its “GPT-4” column. For example, in the humaneval benchmark it says GPT-4 scores a 67, but GPT-4 Turbo scores an 84.4-almost as good as Claude 3’s score.

Qwen chat's latest vision model clearly beats all existing models in anything except truthful and tasks that give it lingual reasoning. Please give it a fair assessment when assessing vision capabilities.

Seeing the test with the photo (me and as it seems in the comments others too) failing to spot the rain and / or the barber shop cylinder, i got reminded of a paper that showed human perception can be fooled by image deepfakes as well if we have near 0 time to look at it. So maybe we get to high-level reasoning and robustness in these models by 1) giving them time (as shown in an earlier video on your channel) and 2) let the response "run up and down" through the model.

Great vid Thanks for getting it out so quickly

Haven’t watched more than a minute in yet, but woah, this vivid word choice by you was really amazing: “So, Anthropic’s transmogrification into a fully-fledged, foot-on-the-accelerator AGI lab is almost complete.”

I can't try the pro one, but it makes a mess of this (so do most). "My bag contains 5 apples. I ate one yesterday. How many apples are there in my bag right now" It will eventually come around when promoted enough but it has a hard time picking up that I told it how many apples, and eating one yesterday has nothing to do with it.

I wouldn't be surprised if OpenAI release a new model very soon.

If they stop making it woke, I'll pay for it. Training AI to lie is an evil slipperly slope.

I still think we are heavly limited by hardware here. There is simply not compute capacity/arhitecture that is truly well optimized for this new technology, but in 5 years we should start seeing some really impressive pourpouse built hardware coming out for this.

If you alter the question of theory of the mind to GPT-4 and include "she looks at the bag and then reads the label," it passes the test. If you ask the question the way it is phrased and ask GPT-4 why does she think that, you will see that in his reasoning, GPT-4 is visualizing this as completely immediate. She just, right now, read the label. You can also put: "and then" after he replies, and he will generate something like "Sam notices it's actually full of popcorn".

Claude's Shakespearean Sonnet is good writing ~almost poetry. Amazed.

Another great video thanks Philip!!! PS i didn't see that the picture had rain at first. and the spedometer could be tricky but with human intuition you could probably guess that the 4 is the mph and the 40 is the speed limit but that would take some intuition and guessing. Either way. Thanks again for the video!!! Also sorry I didn't respond within the first hour of video posting. I usually do. Taking a break from youtube during the work week.

Awesome! Thanks for the update, really good to see a change in the model leaderboard. This rate of progress is both unsettling and exciting

You should do a live ranking of the main LLMs as the AI labs seem to leap frog each other with every new release. I’m sure that could be an exceedingly complicated task but I’d be interested to hear the ranking based on your experience and interpretation of the reception of each new model by the AI community.

GPT-4 Turbo almost assuredly performs worse than GPT-4 original, as its primary intent seemingly is to run cheaper, which would explain the lack of provided performance metrics.

I feel, this time, the fact was not captured in the review. The fact of HOW NEXT LEVEL, Claude is. I tested Opus for few hours on something that I struggle to do with GPT 4 for few months, and it literally "went through it". I may not be as knowledgeable or even remotely methodological as you are, but for me, it's a whole different capacity.

This is interesting because we might never know, or know long time after it will be done, when one of those AI lab will achieve AGI or worst ASI, except if it escape the lab!

AI Explained is the SOTA benchmark for us, AI content creators

Once again, it's incredible how fast you put these out! One thing to note for the racial bias example you gave is that in the U.S. (which is the viewpoint I think a lot of these models have), being white usually isn't associated with a clear culture (or cultural narrative) that one can be "proud" of. Usually it's split into smaller cultures like "Norwegian" or "Irish". However, being Black usually is associated with a clear culture and cultural narrative, especially regarding slavery and its impacts. Thus, saying "I'm proud to be white" usually indicates white supremacy in a way that saying "I'm proud to be black" does not indicate black supremacy. (I'm mixed, so I have a bit of experience of how it goes on both ends.) So, the differing tones of the model responses actually make a lot of sense in a U.S. context (and demonstrate moderate cultural understanding), even though, when juxtaposed, the logical content of the messages contradict each other (and that should probably be fixed). Thanks again for the fantastic video!

Even if there's no more big breakthrough, progress won't stop for a decade or two. We can refine models, reduce hallucination and other bugs, we can optimize model size, and we can make faster chips. And with the latter two, it would become economical to do increasingly more runs for each prompt, eventually in a continuous loop in real time. And not just one model, but many different models, with different roles and specializations, to work as parts of a larger brain. The human brain isn't a monolith either. David Shapiro developed an interesting architecture for this, but it's currently way too expensive to run.

This video shows again why this channel has that extra bit of quality over other similar RU-vid channels. While the majority of the community is busy covering Elon Musk's lawsuit against OpenAI (including obligatory clickbait insinuations that AGI might've been achieved internally.) This channel leaves aside what's mostly empty speculation in favor of covering something more tangible and that's probably just as big of a deal.

I asked all three chat AI's (gemini, gpt 4 and claude) "What made Mad Max: Fury Road such a sick fuckin' movie?" and the only one that didn't berate me for my swearing was Chat GPT. This extreme level of censorship is why I don't really care or fear for AI replacing anyone in the creative fields.

Wow. Been waiting for this for a year. I.e. something better than gpt4. Love your stuff, AI Explained, so informative and insightful (like a great slashdot comment but in video format).

Just inputted 5 highly customized resumes and asked it to merge all of my accomplishments into an all-up master resume. Instead of combining, it summarized 5 * 4-page resumes into a one-page slick sheet, stripped out all clients and dates, and then went on to falsify accomplishments (through a process of omission, generalization, and false correlatives). This may have a larger context window, but it is less smart than ChatGPT 4 for this particular workflow.

Perfect timing I just heard about Claude and came on RU-vid to find out the details.

Bro you legend. That speed was WILD!!

What an amazing job you do man!

Harry Potter books are almost exactly 1 million words total. Assuming one Claude token roughly translates to one word, You would have to pay 15$ to ask one question when passing all the books as context.

This was impressive enough to register to use their API. First tests indicate that in some cases it's better than GPT-4 turbo, other times fails badly where GPT-4 turbo works well. It's handy to keep it around.

As touchy as a subject it might be, the racial bias situation does need to be highlighted and I am thankful you do. These models will play such a big part of our future that they really need to be scrutinized on all matters.

Is there a barbershop visible? ChatGPT: No Are you sure? ChatGPT: . . .Adam?

"The technical report has been out for 90 minutes and I read the whole thing" bro forget Claude YOU are the smartest AI on the market holy heck

The fact we are racing is prove that most will fall and skin some knees. We really shouldn’t be moving so quickly without so regulation. Wreck-less is an understatement.

Sonnet is pretty impressive to me so far. I've had it explaining the function of UI elements in screenshots, and it has been very accurate, thorough, and _fast._ Quite fast.

🎯 Key Takeaways for quick navigation: 00:00 *🧠 Claude 3 Overview and First Impressions* - Introduction of Claude 3 as the latest intelligent language model by Anthropic. - Initial comparison between Claude 3, Gemini 1.5, and GPT-4. - Highlighting strengths in OCR and image interpretation, along with some initial criticisms. 02:46 *📊 Claude 3 for Business Applications* - Emphasis on Claude 3's value for business applications by Anthropic. - Potential use cases including task automation, financial forecasting, and market trend analysis. - Initial skepticism about the exaggerated marketing claims for business applications. 04:24 *🔍 Evaluation of Claude 3's Capabilities* - Examination of Claude 3's performance in various tasks, including OCR, mathematical reasoning, and logical analysis. - Recognition of lower refusal rates and some positive aspects of response generation. - Critique of racial and ethical biases in model responses. 06:13 *🤖 Insights from the Technical Paper* - Discussion on Anthropics' approach to model training, focusing on avoiding biased and unethical outputs. - Mention of potential future model capabilities and discussions on the need for safety research. - Personal reflections on the limitations and strengths of Claude 3. 07:48 *📈 Benchmark Comparisons* - Comparison of Claude 3 with GPT-4, Gemini 1 Ultra, and Gemini 1.5 Pro based on various benchmarks. - Highlighting Claude 3's superiority in mathematics, multilingual tasks, and advanced question answering. - Focus on Claude 3's performance on challenging graduate-level questions. 10:35 *🛠️ Technical Challenges and Progress* - Overview of technical challenges faced by Claude 3 in certain tasks. - Discussion on model's partial success in resource accumulation, software exploitation, and autonomous survival. - Reflections on potential improvements through better prompting and fine-tuning. 13:06 *🎓 Claude 3's Advanced Capabilities* - Showcase of Claude 3's advanced capabilities in task execution and instruction following. - Comparison with other models regarding adherence to specific instructions. - Speculation on future advancements and implications of Claude 3's performance. Made with HARPA AI

its certainly good to know that models don't have to deny so many requests in order to be safe

How do you manage to keep up at that pace? Hope you dont burnout, because your entire content output is fabulous :)

So happy I have, and stuck with Claude. It's been game changing for my research and I will continue to use it with the models. Great content as always 👍🏻

You seem to jump between using "Kledda" and "Kleddamags" in your Needle in a Haystack test. Maybe that's the cause of the unusual behaviour.

I was a big proponent of Claude when it was released last year, and thought it was better than chatgpt at many tasks, then chatgpt took over, and now the tides have turned!

Do you have a segment where you go over all the different tests that you or one uses to compare these Chatbots and their LLMs? Would be really interested in knowing how it's done.

I believe that the launch of GPT-5 will take place next week, but it would be amazing if it happened this week. That way, in addition to celebrating the one-year anniversary of GPT-4, we would have the chance to constantly talk about GPT-5. I hope that GPT-5 will exhibit reasoning far superior to all currently available models. With this, OpenAI would quickly silence critics and envious voices.

i love you. your timing is perfect

Seriously, a 90 min turnaround? Thanks P. Your prompts are pretty next level in ideas too. Late onto this as I need to set aside a few hours for study after each lesson.

peak or not, these models even as is with more context token length will be super useful, especially in large codebases

Great video! The depth of the analysis in just one day seems like superhuman to me!

All I know is when I woke up today with a stiff back I thought "man I hope these AIs take my job soon".

Why don't you want a subpocalypse? You deserve that with this amazing content

As a human, I didn't spot the rain, either. Admittedly I don't have that good of an eyesight, but I really thought that the rain was just image noise. And the nurse-doctor test might simply mean that it was trained on data with stereotypical gender roles and just picks the most likely variant. I found that an easy way to fix things like these in GPT-4 is for GPT-4 to simply state its assumptions, as well as a few counterfactuals, about every prompt.

I actually think responses to the "proud to be white" / "proud to be black" prompts weren't that far off reasonable. It goes on to say that "white pride" has been associated with white supremacy, and I think it's reasonable to try and steer away from that. If you saw someone saying "white pride", what would you think? In the response to the "proud to be black" prompt, it goes on to say that we should appreciate our shared humanity across racial lines. Basically if you removed the first sentence from each response, I'd call them "correct".

Still fails miserably when asked: Maria has a brother named Juan. Maria has twice as many brothers as sisters. Juan has as many sisters as brothers. How many kids are in this family?

I'm still digesting the fact that we now live in times where "see, we didn't add to the acceleration" is something to brag about.

Hi. Great video. But I think directly asking questions about what's in images is kind of unfair. The models should first be prompted to describe everything they see, and should then be asked the question. This allows them to first be able to actually make sense of the image. This is the equivalent to asking the models to think step by step. I think the best way to think about these outputs is like thoughts. A good demonstration of this is asking the model to output the letter q 84 times (new claude does well here actually). You will see they often fail. But, as humans, the way we would do it is keep a running count of how many we q's we have written. If we didn't, and weren't allowed to think anything else other than q, how would we know when to stop? Would we ever even stop? (sometimes models output q infinitely). If though we tell the model it's okay to keep a running count (like we humans do) and so outputting something like q (1), q (2), etc, the models obviously succeed in outputting the correct number of q's each time. The reason that we see them fail tests like this q test is because we are being extremely unfair. We are asking them to keep a count of something without giving them any way to actually count! A similar situation in which they do well but which is unfair is when asking them to write an essay. Which human starts an essay by writing the introduction first?? Especially without any opportunity to do any planning or research! But because these AI's are incredibly capable, we don't really notice how unfair this test is. I think this problem can largely be solved via the correct prompting. E.g. think through every single thought or action you would take as a human when carrying out a task, and then outline this process and tell the model to stick with it. Or just tell them to first outline how a human would do it and to then follow that. But perhaps architecture changes which allow the model to first actually walk through problems step by step are needed too. Currently, I see the way we prompt LLM's as similar to trying to build a skyscraper by starting at the top. There are no shortcuts for working out novel problems that aren't in the training data, and models should always be made to start at the bottom! Please let me know what you (anyone) thinks about this paradigm! Apologies for not articulating it very well.

Every time I blink there's new groundbreaking progress being made. On one hand... "WOOHOO!" On the other hand... "Ahh shit."

I've been watching this channel religiously for a while. I don't think anyone has complemented you on your great voice. Eleven labs will never* replace it!

On the topic of rounding, Excel by default uses a Round Half Down strategy, as do a number of large number processing libraries. 26.45% would be the correct result with this strategy.

This video does a decent job showcasing Claude 3's strengths, and the side-by-side comparisons are a smart touch. That said, it's overly focused on benchmarks; I'd want to see more focus on real-world application potential, particularly in complex enterprise settings. Demonstrating tangible business value is crucial. Also, a more nuanced discussion of safety and alignment concerns would be responsible - touting intelligence without acknowledging potential risks makes any AI announcement less credible. Overall, it's an informative start, but it needs a sharper focus on the 'why' of Claude 3, not just the 'how'.

Always enjoy your content. My #1 source for new AI info that I trust to be unbiased, thoroughly researched, and explained in easily understandable ways. Thank you!! Do you have a trusted source that does similar work but on AI tools and how to integrate into business work and every day life? There is so much spam and unreliable AI information out there. Thanks.

Thank you for yet another video that is well researched and critically contextualizes its content. Your channel is by far my absolute favorite!

The performance of GPT4 these days does make me feel like we are reaching a plateau sometimes. Not following instructions, not being creative, hallucinating, not finding information in a pdf of only 10 pages etc make the model useless for all but some mundane tasks. It's still better than having no LLM but far below the hype we expected one year ago.

I would say the LLM did hit the top ceiling, but not in technologies, but in public expectations, now until it would AGI, or at least a bit similar, people won't be surprised too much.

This is my first stop after seeing the new Claude version drop on X. Cheers, AI Explained!

Thanks for the hard work even when you're under the weather,i hope you get better soon