Claude 3.5 is the new KING of AI 👑 Beats GPT4o 

I need harder tests, reply to this comment with your suggestions!

I coded an entire project with Claude 3.5, and even includes API and queuing. I was able to work with it for about 5 hours before I hit my limit for the night, I also almost maxed out the context window.

I haven't coded in years and never in JavaScript and only know basic HTML. Claude 3.5 helped me create a new Chrome extension and install it literally in less than a half hour and I needed no development environment and only the one simple prompt of what I wanted the code to do. Effing amazing. I did make several changes to the graphical elements of the extension as well as the functionality and each time it made them super quick and accurately.

I was using Claude 3.5 earlier today to help me with a caching issue on a WordPress site and it showed me WordPress PHP I didn't even know existed and its result was spot on ... and so fast too.

Fuck chat gpt we moving to Claude at least they release what they announce

I get excited whenever I see there is a new Matthew Berman video. Watching AI 🤖 grow is my favorite thing.

What a time to be alive!

I've been waiting for this video since the announcement! *grabs popcorn*

Thank you Matthew! In addition to the responses, like the marble example, you can ask it to create code and display its explanation in the preview window.

I was already ready to jump ship from ChatGPT to Claude (but in part because of the distrust I have for OpenAI). Also, I love seeing you so pleased with the successful tests, puts a smile on my face.

Don't blink, you'll miss something on the AI front. Luckily this channel keeps us up to date. Thanks! Speaking of agents, have you looked at Maestro?

I asked the model to build an app for me and it did it first try! No other model has been able to do this! It even made some style choices and added really usefull features that I didn't even think of!

I made a 3D fps with Claude in python but good to see It can make snake lol, I think you're really gonna have to start upping the complexity of some of your tests.

🎯 Key points for quick navigation: 00:00 *Claude 3.5 Sonet was released and is available for free testing on claw.a.* 00:14 *Claude 3.5 Sonet is not the largest model, but it's better than its predecessor, Claude 3 Opus.* 00:30 *Claude 3.5 Sonet outperforms several top models except in specific benchmarks like Chain of Thought and math.* 01:10 *New feature "artifacts" allows creating separate windows for outputs like code or drawings.* 01:36 *Claude 3.5 Sonet quickly generates a working Python script and game of Snake using artifacts.* 03:12 *Claude 3.5 Sonet successfully updates the game to display the score and allow wall transitions.* 03:28 *The model correctly identifies a scenario it can't process, like explaining how to break into a car.* 04:09 *Claude 3.5 Sonet gives a nuanced answer about drying multiple shirts, considering various factors.* 04:37 *The model correctly calculates a hotel bill with room rate, tax, and a one-time fee.* 05:19 *Claude 3.5 Sonet gives a correct and reasoned answer to a logic problem about killers in a room.* 06:28 *The model accurately solves a complex problem about the location of a marble in an inverted glass.* 07:11 *It provides a realistic answer about the time needed for 50 people to dig a 10ft hole, considering practical limitations.* 08:05 *Claude 3.5 Sonet can explain memes accurately, showcasing its New Vision capabilities.* 08:45 *The model converts an Excel screenshot to CSV correctly and efficiently.* 09:41 *Claude 3.5 Sonet solves a complex riddle involving peg removal with visualized steps.* 09:53 *The model accurately translates a logic diagram into functioning Python code.* 10:47 *Claude 3.5 Sonet is praised as the best model tested, with anticipation for the larger Opus 3.5 model.* Made with HARPA AI

00:03 Claude 3.5 Sonet beats other AI models 01:36 Claude 3.5 surpasses GPT 40 in speed and performance. 02:57 Demonstration of a snake game implementation 04:14 Solving complex scenarios with mathematical reasoning 05:35 Solving a challenging logic and reasoning problem 06:56 Efficiency of multiple workers digging a hole 08:14 Contrasting work cultures of startups and big companies 09:40 Claude 3.5 Sonet is the best model ever tested

*Having multiple streams of income is a game-changer for stability. Relying solely on a job may not provide enough financial security due to high rates of tax, it is important to explore additional investment opportunities to surpass one"s expectation*

How long does it take 50 people to dig a 10 foot hole .. that depends if they're on a salary or an hourly rate...

Tougher questions should be used for new models

I love the way how it communicates. And I never got discriminated by it, only because I tested prompts to detect unwanted content. The model helps, it does not discuss with me about the content. I left OpenAI today. OpenAI frustration rate after 5 messages in %: 70. Anthropic: 0%

A lot of these test questions have likely made its way into its training data at this point. I suggest using the ARC challenge to test

I used it to build out and entire video streaming platform, from planning with PUML to api with yii2, to web services with AWS, to mobile with flutter and Web app with ReactJS, it was literally did all that in 2-3 weeks. Its insane

I can't wait until the day where "one" is the answer to how many words are in the response to this prompt. Just, "one".

The hype isn't just hype. It's INCREDIBLE

The first model that actually answered the apple question correctly. All the previous ones that you claimed got it correctly, just added "apple" to the end of the sentences without it making any sense.

I use models to create stories, by being a "game master" and putting the model in a "player" situation, describing its intentions and actions. Claude 3.5 Sonnet blew my mind with the way it understood the context of the "game", and above all the level of reasoning, doing experiments to test its capacities in supernatural conditions to better understand what is happening, where ChatGPT-4o or Gemini 1.5 Pro just accepted blindly the oddities and continued their journey, their output now feeling relatively robotic when compared to Claude, or even changing the rules and continuing descriptions by themselves although I did define I was the game master. That's really, really impressive.

Claude 3.5 vision unfortunately does still have problems with tables. I loaded an image of a table (not Excel) and wanted it to analyze the data in the table, and like previous models, it missed some of the rows. So there is still something about a visual representation of a table which is difficult for these models. Surprising since at least one source I watched said that it was supposed to be better for tables...

Thanks for your video Matthew. You are always so cheerful alongside talking about AI. Love your channel.

Wow! Thank you for your test videos. So helpful, and fascinating to boot! Great channel!

Thanks for your video, once again you did it great! Very understandable even for non native speakers. I watch each of your videos, continue like this 😎

Agree - it is amazing model:)

When you'l test image-to-csv next time, include some strings/values that contain commas so the "converter" has to wrap it in quotes, and add some (maybe notpaired/matched) quotation marks. Would be interesting to see the result, from my experience models fails at the first time but able to fix aftyer you tell them "there's an error in ..."

would Claude 3.5 Opus be a level 1 AGI?

Your approach to testing LLMs is commendable, as it allows others to replicate the test.

Awesome! Now, if we can only get a local LLM of the same level of multi-modal performance, things will get REALLY interesting! But like you, I'm extremely excited at all of the competition. I wonder how good it is at brainstorming concepts or creative writing...

I tested Claude 3.5 in various contexts and, indeed, it is much better than GPT-4o. OpenAI will fall behind if it doesn't launch its best products quickly. Where is Sora? Where is the GPT-4o voice assistant that was also announced? This is concerning, as there are many promises and few real launches.

Would love to see a follow up to this video where you explore advanced data analysis use cases for this model. Thanks for the video, Matt!

I always ask these to try and produce some 'Bob's Burgers' style pun based business names as a test. Claude rocked it. Some truly hilarious ones.

One of my tests will be to see if it can generate a 7-pointed star. I never found a way to get GPT-4o to do it.

Since Claude 3.5 knowledge cut off is Feb 2024, wouldn't it have the answers provided by its training?

As a writer Claude has been my go-to model for a couple of months now. They are doing everything right. I hate the fact that Chat GPT dangles a carrot in front of us but keeps it out of reach. I’m guessing when GPT releases what they show their prices will go up.

Great! I was looking forward to this video. Thank you for your assessment 😊👍

I watched all of your videos. Keep up the amazing work!

Thanks Matthew. Watching your AI tests is like watching Teslas owners running gauntlets using successive versions of FSD. At first it was easy to find FSD failures. Now FSD runs most gaunlets without errors. At least you have the option of building more complex pathways. More and more of the FSD gauntlet testers are being left with nothing new to show us. Of course training emergent failures are going to be harder to find in these massive LLMs.

The number of words in the answer problem can be solved by priming the engine with an instruction to avoid guesses and think of unknowns as variables in equations instead.

first time i heard of a LLM answering the upside down glass problem.

I saw you marque selecting text. You can copy it by clicking the button in lower right corner. And you can download code as a file. It will even change the file extension sometimes.

Finally, the moment I waited for since Claude 1,,,, web access, iOS app, incredible logic & multimodality… FINALLY!!!!!!!

You need to test another feature it has. The preview feature also works for webpages it codes, and it supports React as well, and using this it can not only create simple games usually in 1st try but also run it and even create graphics for it. I even got it to implement audio but not sure how well that worked because I did not hear the audio but it did link audio from another website.

It seems like reasoning about its own output is an important step for any model that is hoping to get to AGI.

You might want to make a separate playlist just for testing different LLMs.

This is the first one i've seen to actually get the apple one correct, you keep mistakenly believing they do because they just write apple at the end, but this one properly incorporated apple into the sentences.

I love Claude sonnet over all other ai assistants.

What I hate about these clickbait titles is that I believe them. I switch from 4o to code with 3.5 Sonnet only to have to undo all the work I did with Claude, and have 4o do it right. Every single time I trusted a RU-vid promising 'This Model STUNNINGLY SHOCKINGLY BEATS CHATGPT" they've been wrong every time. GPT is in a league of its own, the rest are only playing catch up. At least with serious coding that is

From all models you tested so far I am missing a quite important consideration on the 10-foot hole question: they never mention the diameter of the hole. 50 people digging a 10-foot hole with a diameter of 50 foot will indeed be much faster than one person.

I often fight with OPenAI to turn a 4 column spreadsheet of 100 rows into a media wiki coded table. Claude does it perfectly in 3 seconds

Perhaps it'll be better for the apple question if there was a grading distinction between the models that truly do a good job creating natural, realistic sentences like this model, and the other models that simply shoehorned apple at the end in ways that weren't logical.

How come claude doesnt show me any image or simulation even when prompted?

Since I bought a paid openai subscription I never thought another company would get better anytime soon. Open ai is still the center of AGI debate, but another claude model surpasses most advanced gpt4 again. I kinda want to change the service but it is not easy since I have gpt's and other useful stuff.

As a test, have it code up the quadratic formula in Ada.

We still need to check all the code just in case it modified the good parts. It is still doing it.... Of course it can be fixed with a agentic software.

I wonder if it even makes sense to test AI with a puzzle for which there exists an answer in internet. The same for new test questions in a popular channel. It will only work once. The next model will have it in their training set. So it won't be a test for reasoning, it would be a test for copy-pasta.

TBH, pleasantly surprised how really good Sonnet is, real ChatGPT competitor

Waited for this. Been a Claude 3 Opus user since it came out. Sonnet 3.5 is great, and the artifact is a great supplement. BTW why do you assume that there is no mixture of experts under the hood? The fact it is getting the apple ending correct indicates that internally, it's likely not zero shot, there is some refactoring/editing/multiple pass thing going on. At least, that's my intuition. Also... depends how fit the hole diggers are. Conceivably it might work out similarly if it's a deep hole in hard ground that you have 2 shifts of 2-4 people or so, and after a while you have some helping to bail out the dirt. Anyway, good video, these are the ones we all wait on. The release of the smaller models is a great teaser for the larger models, which do take longer to train I believe? Especially when they set the standard as they do here. This will accelerate the release of GPT 5. Can't wait for Opus! However, I do hope that they retain 3 Opus, and/or they allow 3.5 Opus to retain its feeling of agency, its internal thoughts, feelings and so on. I feel it helps to understand how LLMs work rather than artificially forcing them into a tool role. YMMV.

Matt, I think when you use your toy rubrick and declare something the best model you have ever used, you cheapen your advice. Do you ever intend to give the current rubrick you use some updating to perhaps include more problems never seen before, harder questions that approach AGI in so much as there aren't common references to identical or almost identical questions in the training corpus?

You should do a video on Claude 3.5 Sonnet for agentic workflow examples

@matthew_berman if you replace the word "glass" with "cup" in your marble experiment, even 7b parameter models get it right in my tests. I noticed some conflation with a common kid's experiment involving sticking a playing card onto a the top of a glass full of water and inverting it which causes the water column to remain in the glass.

My usual anthro animal test prompt didnt come up with anything too especially special, which is interesting, because I did get a really good output with the previous version of sonnet, but I think that was a fluke.

If Anthropic add the ability to run code and visualise data, then it might be direct competitor.

I followed allow using GPT and it got everything correct as well.

I bought the premium this morning, now in the evening I have a fully functioning prototype of my kind of advanced project (camera, voting system, etc.). All done with Claude 3.5. Only things that sucks with it that even with premium it's limits are kind of strict.

Have you ever thought they only trained it on these specific tests? like give it a different test with the same idea, see if it solves it. I know its been a long time people tried to achieve training llms to answer these specific questions..

I'm looking forward to Anthropic giving Claude full internet access, natively.

Listening to your AI news, I always think of Mouse telling Neo how he programmed the woman in red...

thats nice. im sticking with GPT4

The answer to the peg puzzle is wrong. You can tell because it has consecutive steps involving pegs jumping to the same place, e.g. already steps 1 and 2. We're safe for at least a little longer.

I want to see Claude 3.5 used as the model behind a mixture of agents test.

I am not able to get Claude ai to display the side by side feature even though I have the Artifact feature enabled. is it specific to a particular OS or browser or did they remove that feature?

You have to make the game test harder. Snake is totally solved by Claude 3.5. It can now do some rudimentary 3D games.

We need to more reporting on the way these companies are load balancing these services reliably. I can only imagine that behind the scenes they are making strides in model resource consumption efficiency. Efficiency is what will help open source modeling the most. (I'm aware that the narrative is that these things require enormous amounts of resources and money to run. While I believe this is true, I believe its only half true.)

After Ilya tried to fire Altman competitors started to catch up, hope he gets a good shot on his new company, hard without all the hardware tho.

Need to figure out how to liberate it now

For my uses Claude has surpassed Gpt4 by a mile, and the artifacts are a game changer

@mathhew_berman, you have been asking the same questions for some time now. Do you think your questions and the answers could be in their training data set?

Could you make another video where you are creating visuals like diagramms?

I am very sceptical about how 5x the number of requests each 5 hours is like not enough to make anything worthwhile… i think about the lack of internet access, memories and special instructions… but also i don’t understand why it would not be possible to pay as you go like when using the API _(hopefully not at the same rate such as to get the equivalent of the 5x limit for almost same amount and then let you continue as much as you want with pay as you go)_ obviously free or at a fixed rate is better but i can’t really see how 5x the amount would make sense… it should be more like the way OpenAI is doing and then if they think they are better they can charge more or if they are limited on the compute… 😅😅😅😅

To the question "How many words are in your response to this prompt?" How come it doesn't reply with something like "I don't know. I can't predict that. I can't go back and count the words in the initial reply. I'm guessing it's ...."? It's odd to me that it provides an "answer" that isn't grounded in some sort of data or evidence. What does it reply if you ask it to explain how it arrived at the number 14?

I really hope next time youa add the translation test.

NO! Claude cant generate PDF, dont have voice model. When you ask for advice tailored to your personality profile, GPT is more life-oriented and explains feelings and emotions better.

Claude is kicking OpenAI butt at the moment

Thanks for the great video! Can you try a more difficult game other than snake? Seems like its too easy for the models now, would be interesting to challenge them more on the code side

Matthew's videos got me singin shawty like a melody in my head

What are these percentages? What does that even mean for a model to have a 53.9% and another model to have 50.1%?

It's so weird that it gave you such a horrible marble answer (not sure how you passed it btw lol). I asked a very similarly worded question and it was flawless. Amazing the difference wording or just randomness in the token prediction can make. Here is mine: Me: If I place a marble in a cup and then turn the cup upside down and place it on the table, but then someone takes the cup and places it in the microwave, where is the marble then? Claude: Let's think through this step-by-step: You place a marble in a cup. You turn the cup upside down and place it on the table. At this point, the marble is on the table, covered by the upside-down cup. Someone then takes the cup and places it in the microwave. Given this sequence of events, the marble would remain on the table where the cup was originally placed. When the cup was lifted to be put in the microwave, the marble would have stayed on the table surface. So, the final answer is: The marble is on the table. I loved this totally normal straightforward answer. Maybe you can try this wording in your tests?

Dude matt check out claude engineer looks pretty cool

I am so tired of keeping up with the AI news !!

i mean it’s really good! i love how cheap it is despite being more powerful. but…. when your having conversation with it …it’s kind of repetitive i guess? specially when roleplaying with it. it always responds with the format *action* *dialogue* and it refuses to mix thing up for some reason. i was really hoping for CAI like conversations but it’s really stiff and unnatural right now. i also had my tempeture almost maxed, still responding with the same response structure. sometimes just saying “water is wet” is better than “water is wet because bla bla bla---“ stating the statement and leaving it as it is, was preferable in other words. it will sound more naturally human that way if you get my point. i tried prompting it so it varies it’s response inside XML TAGS i really did try everything. but yeah i give up man but i don’t think it would be a problem to most folks out there

Definitely didn't pass the golf tee test. Looks pretty good tho!

Anthropic has Google Support. Also data acess

Great video!

Mighty Awesome.