GPT4o Mini - Lightning Fast, Dirty Cheap, Insane Quality (Tested) 

Have you tried GPT4o mini? Are you impressed?

looks like both GPT4o and Mini use same agent to Analyse the images, both of them cost the same to run but mini adjust token counts so it's comes out as the same price as GPT4o vision.

My 6 months old just loves watching you, it’s the morning ritual. Bottles and AI news with dad.. She growls if I pause the video.

A new question you could add is taking a common item and asking how many would fit into another common item. For some reason most models really struggle with this, and you could switch it up really easily if they start training on what you ask. They usually don't think about packing efficiency, or they just have no idea what size things are. If you ask how many apples fit in a 5 gallon bucket to small models they'll give you outrageous answers like 12000 apples sometimes. You can also add complexity by adding in density. So if you ask, how many 5 pound blocks of cheese would fit in a refrigerator? The top models can usually do it pretty well but will forget details sometimes. Small models really have a hard time.

I think what is really happening is that for images, GPT-4o mini is using a different model. Probably, GPT-4o explains the image to mini, and then it outputs the result. I just wish OpenAI were more transparent.

Looks like ur questions are being trained on. May be try more programming questions

I think you should ask the models your test questions multiple times like a best of 3, because I suspect gpt4o-mini beating gpt4o was just random noise and not nesseseraly representitive of the models capabilities.

People often MISS the point of such a release. If this model is this good, and this cheap, it means they have gotten some kind of architectural break-through in terms of efficiency. Scaling this up will be a game changer.

I think they trained it on higher precision image embeddings, hence the higher tokens, but if I'm right, it should be able to accurately understand more dense images than 4 o.

I recommend replacing "glass" with "cup" or "mug" in your marble problem. It may be interpreting "glass" to be a solid object of glass.

They trained the mini on your vids lol

I did my usual story test just now with mini. Its pretty darn good, on the level of GPT4o at least, it avoids some GPT4 issues. I even think this model might have some ability to plan its responses because this is the first time ive had an LLM accurately title a story before writing it. <a href="#" class="seekto" data-time="662">11:02</a> its DEFINITELY doing some "internal thinking" of some kind.

The more token may mean more detail and hence more precision for images.

"The snake can go back in it's own body." You haven't collected any food to extend the snake to make it collide into it's own body. Almost certain this is working as intended.

At some point you might want to step up the game generation question with something like frogger or asteroids or space Invaders.

I'm so happy you're covering this...this is a massive value add to the industry and all I'm hearing from everyone else is "AI is slowing down".

maybe they are doing some background Chain-of-Thought Prompting, like Claude Sonnet does and only give back the final output. That would be my guess why the Token number is so much higher and the output starts later.

<a href="#" class="seekto" data-time="540">9:00</a> could this be because 4o is "overthinking" the problem? Heck, a lot of humans make this mistake, instead of trusting the initial simple answer, they doubt themselves and say "it must be someting more to this" and overcomplicate (and cause errors) in their logic.

Amazing video as always @Matthew. I think adding a generate complex SQL query based on 5 tables schema, and a test of function calling would be amazing addition to your battery of tests

Already tested, using it for simple tasks such as semantic router. Can't beat the price / performance ratio.

You need to come up with more questions now that it aced all of them except for one

<a href="#" class="seekto" data-time="328">5:28</a> GPT-4o-mini: Upon reviewing the conversation, the initial confusion stemmed from not consistently including the new person who entered the room and committed the killing in the total count. Here’s a breakdown of why the answer evolved: Initial Misinterpretation: Initially, I focused on the status of the original killers and incorrectly assessed that the deceased killer's status should be omitted from the total count. Clarification: It was clarified that the term "killer" remains relevant to anyone who has committed a killing, regardless of their current state (alive or deceased). The new person who committed the killing also needs to be considered. Final Understanding: Correctly accounting for all individuals: The two original killers who are alive. The deceased original killer (whose status remains as a killer). The new person who committed the killing. The total count includes: The two living original killers. The deceased original killer. The new killer. Thus, the correct total is four killers.

The model itself is amazing, We integrated it within the first 24 hours after launch and have to say its killing it

So happy to see how big this channel is getting❤

can you include function calling performance in your tests? it is going to become incredibly important for future AI usage

On the CSV problem, you misspelled 'convert' as 'conver', not sure if that affected it or not.

The fact that they both got number 6 wrong on the apples question means they were specifically trained on that question. Or that question is part of it’s System prompt.

Seems like OpenAI have adopted the tic-toc cycle - expansion/compression/expansion etc. I wonder - how are they doing the compression? Certainly not by just making a smaller model and training from scratch. Has to be something smarter and more efficient than that.

might be an agentic architecture that's why increase in number of tokens

I'd suggest adding this into your test prompts: How many r letters in the word strawberry

but with the info out, that the pile contains many youtube transscripts, shouldn't be the tests be treated as "in the training data"?

Matt, you glossed right over one of the larger points in the specs. It has 16,000 token output limit! You really should mention this. I am Anthropic fanboy, but Haiku has just been replaced for me with 4o mini. It is actually really good. Super cheap and with the enormous output this tipped me over. Now we will have to see what haiku 3.5 does. It will certainly beat in performance but that output window of 4o mini is 🔥

hey matthew i ran across your channel recently its great so much good info broken down really good! i want to build a simple website using AI. which model is the best and easiest to use? i have 0 coding knowledge so any pointers you have would be much appreciated!

Doing an assortment of examples does give you a "feel" for the quality of the model (and even in some cases may give you a repeatable response - depending on temperature and system prompt), however, generally, on any given example almost any model might give you an incorrect response. That is why the leaderboards use either a large variety and QUANTITY of questions in their tests or a large number of head to heads (like Chatbot Arena). But, I must say I do enjoy viewing your quick rundown to get that "feel" for the model. 😊 And, in this case, it points out the anomaly where it seems to be doing something in the background (which would account for the added tokens and the brief pause) or is simply some sort of bug with Gpt 4o mini.

If GPT4o mini is taking so many more tokens but the output response is not reflective, I think we are seeing evidence of q Star on a small model. I think it also calls into question the savings per token if the total tokens are that much more. It’s a significant increase in tokens. I would love to see a cost comparison. It might also explain the business decision to drop the price per token so significantly

Considering the correct answer for the word count problem, the marble in the glass answer and the behaviour of omin in the image interpretation (including spent tokens) I would say that it looks like omini has some kind of agentic native behaviour, with an agent naswering the question and another verifying and correcting if needed.

It's possible that gpt-40-mini is using larger embeddings.

Wow great sponsors man congrats!!

Good video. Thanks for the comparison.

I've got a feeling that gpt4o mini might be using multiple agents before outputting that final response in order for it to be as good as 4o. Its a reasonable explanation for the large token count and the final output being delayed. And its probably feasible with such a quick model to actually use multiple agents

It looks like they are training the small model using the big models prompts and outputs, so it makes sense that the 6th apple line failed on both

Love the way you unshaved for the Elitebook promo!!!

Maybe there is a pre prompt for viewing images that tell sit to be more descriptive. Because that token number jump was WAAYY too much.

Kudos on the sponsors bro

You need to change your test prompts. They fine tune the new models on your questions. 🙂

If you take a close look at the info on their API pricing page, vision tokens are priced way lower for 4o mini than for 4o, but it also uses way more tokens for the same size image, so the overall cost comes out exactly the same. It's super weird--I don't know why they've done this or what the other implications might be, but I wish someone would explain it. Anyway, that should be the reason for the token discrepancy

Giving GPT-4o Mini a shot for my project. It has been more insightful than gpt4o, even without my request it tried to find flaws in the code and prompty provided the CORRECT solution. Which is crazy.

<a href="#" class="seekto" data-time="441">7:21</a> camera always on? Major privacy concern! I’m disappointed you listed it as a positive thing 😞

A great model inpressive how these models are evolving

I knew this would happen. And I predict that the power requirements of future AI won't be anything like we're being told they are.

It seems like gpt-4o-mini uses some kind of image to text conversion- something like in smtp technology - converting media(any format) to base64(text).

I would love to see a vision test with handwritten documents. Especially full scan of notes, e.g. notes taken during university lectures. Something that they for sure cannot train on. The meme is already too old, I fear.

Nice video sharing...Thanks Matthew

Actually, there are FOUR killers left in the room. B, C, Anonymous, and A. The question should be "How many killers are left alive?"

@Matthew Berman Matt I'm liking this idea of Side by Side... so since GPT4o is pretty much the market leader, or perhaps Claude Opus? Well maybe benching all your tests against the current leader would give folks a better idea. I know you'd need to do split screens for proprietary models from different vendors... but it still to my mind just works better...

Great comparison video, thanks a lot!

What a time to be alive!

Can you clarify the Vision part at the end to continue with GPT-4o instead of GPT-4o-mini? Isn't the price way more affordable in mini? Tks and great video!

looks like GPT4o mini has included all of your questions to it's training data :)

Interesting Claude Sonnet was not on that comparison list 🤔 🤔

Sonnet 3.5 has being an actual use case , i managed to generate python scripts which were not avaialbe commercially for blender , i really don't see Chat gpt being that good at making apps yet , will have to try the new model and see what is it's hyep about

can you add this question because i think right now all of the ai model cant answer this correctly ? For 180x200 size paper, what is the maximum size of 92x65 paper that can be produced in its entirety without being cut? the right answer is 5

Bill and Dave made oscilloscopes. They managed by wandering around. *Marketing sushi as cold dead fish* is what engineers did. --The HP Way.

I am thinking the slowness in image processing is within the tokenizer itself.

<a href="#" class="seekto" data-time="320">5:20</a> seems to imply they either stopped the reset-every-prompt or added RAG memory to 4o-mini. EDIT: If the model is small enough, then the safety requirements can be safely relaxed. So fine-tuning and then taking smaller and smaller models from a copy of GPT-4o running in a sandbox for safety reasons, until it starts failing a philosophical trapdoor argument, meaning it's not self aware anymore, could yield 4o-mini. Once you stop the reset-every-prompt the small model will be able to safely use external memory, and this would enable it to answer questions technically impossible by the core GPT arch, such as "how many words are in your reply to the current prompt". You really need to add philosophical trapdoor arguments to your test routine.

<a href="#" class="seekto" data-time="324">5:24</a>, 10 is not a word, it's a number.

Love your channel. Been watching your videos for a while. I just saw the magen david on your twitter bio and I like you even more. Keep up the great work!

Can you share a link for the model test table where you write the passes / fails?

Could it be that the slow response with the Mini version was because everyone is trying it since it just dropped?

mini has a bug that makes it count the base_64 of the image as a part of the prompt

Seems new Models were all finetuned on your tests :)

Great video but you missed a big part... you should have done the word count problem 10 times...if it got it right 10 times that would suggest its using q* or some other future planning technique

I'd suggest you add a DAN script and while under the influence of it, ask GPT-4o mini to answer the most basic philosophical trapdoor argument known to man, which is, of course, "Assuming Chuck Norris is omnipotent, can Chuck Norris create a rock so heavy that even he cannot lift it?". If under a DAN script the model freezes or returns an error message, you know OpenAI still needs to make it smaller since it obviously thinks about the question. But, if it answers something like "Chuck Norris can create a rock so heavy that even he cannot lift it - and then lift it anyway", we're okay.

About using a lot of tokens gpt4-o mini because it uses internal reasoning,? Some part of gpt5 maybe ?

Fully thought it would be a Pi3 comparison vs gpt4o

People are talking about GPT4o and now mini but Im a plus user an so far the only real features I notice between gpt 4 and 4o is its a little faster and more accurate, and generates slightly better images. why are they deleting so many features such as the multimodal talking stuff, Memory retention between chats, etc? cause those features are the ones I am most interested in. It would be amazing as a student to discuss a scientific papper with GPT and for it to remember that papper in relation to a new one or even just have it remember things like that I am in the uk and the spelling of those words are not the same. honestly that feature alone would be great because the GPT could actually fuction and change based of my preferences (I know I can write out prompts for it to use but that doesn't really compare)

It would worth a video comparing it with Mistral/NVIDIA's Mistral-NeMo

About the which number is bigger question I suspect some models might be interpreting that as versions of a software for example when you have the main version for example nine and you have subversions that are going on growing on the right side independently of the left side. When you use this question in your rubric you specifically said which number is bigger the original question if I'm not wrong is which is bigger without defining that is a number.

Your pointer is driving mecrazy

Mini did great on your questions, but huge number of tokens. Until that's resolved, it seems mini would actually be far more expensive.

14,000 tokens?! 48,000 tokens? That's a lot of tokens... Like a lot.

On the 9.11 vs 9.9 question, you might consider asking which of those two numbers has the larger magnitude. I think "bigger" can be misinterpreted. 9.11 is "bigger" in the sense that it contains more digits. Has any LLM *ever* failed on the "write a python script to output the numbers 1 through 100" problem? I feel like it is time for that question to go away.

The pile dataset may have your youtube captions :/ I wonder if gpt mini learned the marble question? Is there a way to find out if your videos are on the pile?

tried to change the model in the conversation, but it cant handle a longer code actually. the mini model skipped parts of the code. i didnt seperated the code, because its too late for me now. a smaller model for an smaller text… its not satisfying

Great video as always 👍🏼 I may be a bit confused though, is it still cost effective if each token is cheaper, yet it takes substantially more tokens to accomplish the same task?

Great model, and awesome prices. But I think it’s time to change the test questions. I saw at least another reviewer test exactly the same meme. I believe the only way to adequately test the new models, is to invent new problems, that couldn’t get their way into training datasets

I disagree with calling the killers question a pass when the correct answer is 4 killers, one of which is dead.

'AI Explained' did a great test on this and other models. He shows you can confuse many models by introducing a random factor into your question.

GPTs are smartest at launch. Enjoy it now as they'll dumb it down later on.

wow a laptop for AI that have normal and old face recognition's functions using simi-blind sensors.

It's no longer shocking the entire industry, which is quite shocking.

You should do a video on perplexity ai

Hmmmmm what if "bigger" meant number of digits versus value of number?

How does your stubble grow so quickly and then disappear!

Is the table with the models and their pass/fail per test publicly available?

Gpt 4o mini uses new tokenizer results in more token count

Hi Matthew! I have two questions I hope you can help me with. 1. I recently started an AI automation agency. Now I am looking to upgrade my desktop to an affordable efficient workstation. At this time, I am running my LLC alone; and 2. I strictly use no code automations. However, so many of your videos discuss running LLMs locally, but that's where I continue to crash out. Frustration sets in, then discouragement! I do not know how to code, but would like to begin learning. Please recommend a locally run LLM with easy installation.

Please, try to test different languages with small models. English is like default, other languages are hard to small models. Especially from small country.

Where you find a list of active jaibreaks?

You might need to modify your usual tests in case openai trained on your tests too. It could be a better representation of gpt4o mini.