New LLM BEATS LLaMA3 - Fully Tested 

Which size of Qwen 2 will you be using?

I suggest you retest LlaMA3. I tested it yesterday with several from your rubric. It aced the killers test, the marble test, and the apple test. After it passed the apple test, I threw it a twist by prompting: Give me 10 sentences with the third word in each sentence being the result of mixing red and blue colors. It used purple as the third word in all 10 sentences.

Protip: if you change the system prompt midway through a conversation it'll break the output. When changing the prompt start a new convo with no history

The larger model's method of getting every sentance to end with Apple was pretty funny if you taker a closer look 😂

Apple fooled Mat... It's at the end of each sentence but in some childish way!

I'm more impressed by the 0.5b model than the large one. If you compare this with the previous models these answers are amazing.

The small model gives such bad answers, it's actually funny

Congrats! on the new Dell Precision 5860 Tower I had kind of given up on seeing LLM reviews run on windows on your channel. I hope that this is a worry of the past! Post the Full Specs if you would to give us an idea of the performance of these models on this machine!

Always excellent reviews from your channel. Thank you.

57B-A14B means "57B parameters" (how much memory you need) "14B active parameters" (how fast it goes). Would be the same as Mixtral 8x7B, in terms of memory requirements and speed.

Will the uncensored version know about Tiananmen?

MBPP -- (mostly basic python programs). A14B -- It is a 57 billion parameter LLM with an additional 14 billion parameter vision module (hence the "A14B" in the name), making it a multimodal model capable of processing both text and images.

Hey Mathew. I have noticed that these models format their responses in latex. And it is correct latex code. Could it be a setting in your gui that you can edit to inorive the response. This is a defect of your gui I believe not the model

this was the best answer for the shirt problem so far

Read the sentences that end in apple; most of them make no sense.

I still maintain that the marble in the glass question needs to add a description to be more like "glass with an open top". I think as human's we're assuming that the word "glass" means a drinking container that is open on top but I don't think you can assume that a language model will default to that.

15:45 it follows instructions very well even if it's incapable of forming a proper sentence with an apple at the end.

I'm going to take a wild guess and say that most of your followers don't have a GPU that can run the 72b model at a reasonable speed, and the 0.5b model is so bad as to not be very useful. I would prioritize the 7b model since more people are likely to use that one.

Thank you Matthew. I wish you could have tested the 7B and/or the MoE 57B since those can be run locally for many users.

Be aware of backdoors and data leakage. Install in an air-gapped environment only.

TWO A6000 for free ?! Matt, i'm extremely jealous woaw...

Quen 2 72b runs quite usably on my 64GB M1 Mac using Ollama. Astounding! Apple needs to push this advantage that their SoC chips have over the Wintel hegemony!

Every time Matthew says Yup, it's bad, It's terrible, or This is not the answer, I just can't stop laughing. That was a funny video LOL

I am not sure how good it is for us to see you testing on a computer that 90% of us wont have anything close to that. Who are you testing for, the 10% that do? I dont get it.

0.5GB had almost no chance of working. They really need to massively constrain these micro-models and be clear about what exactly they can do well.

Where's the marble? Model: what marble?

The Last AI: of Humanity Climbing the AI Pyramid

8:53 - when someone told you the answer and your teacher said "explain your working out" lol

I only have 8GB of RAM would love more focus on 7B models. I think a lot of people at home have similar hardware and I would like to see how it performs on that sort of kit. Keep up the amazing work! :-)

Hi Matthew, on the question with the killers. I did this with the smaller model with the same result. I asked it afterwards what is with the person who just killed one of the killers. It replied that the legal status is currently unknown. Maybe it assumes that "killer" means a "convicted killer". Not sure about the legal language here (as I'm not a native English speaker), but if it was an action in self-defense (not specified, though), the person may actually not be counted as a killer. Furthermore, a really good answer would point out these specifics and also mention that the dead person may also (still) be counted as a killer as well. Love you vids!

HEY HEY HEY WHOA WHOA. Is that evolution we just witnessed in ai chat bots. 15:40 no chat bot ever got that right lol.

The benchmark rankings don’t mean much anymore in regards to real world performance. Nearly every single model is trained on the benchmark questions, which allows it score highly but does not reflect its actual quality. I’ve tried so many highly ranked models that are just either terrible or average in real word usage.

14:05 ok so notice how it uses the word "container" one thing ive noticed with this test is that the LLM essentially sees "marble in placed in" and then idk somehow that takes some kind of precedent making it assume "container" rather than the specified "cup" and what do containers typically have? Lids. So it assumes the marble is not just in the cup but *contained in the cup.* Which would mean that it stays in the cup.

Q: How are you? Small model: 42.. I think you meant How many? I am available in 42 versions. You are one of them. Thank you for your services.

They released 7b model - seems to perform well

57B A14 is the active params

One Thing to Note, if you using LM Studio, on the right Panel, check if you use the correct chat template and also Set the Temperature which is often 0.8, to 0.0 (Temperature makes text more creative but increases error rate.)

What are the specs of your new Dell machine?

10:19 that's what I'd like to see in an LLM. Any other LLM I've tried would just stick to the answer they've assumed to be "correct", trying their hardest to prove it even when you ask it to be cautious. Giving the right answer right off the bat is cool, but ability to catch their mistakes is more important imo. Edit: 14:42 ok, nevermind, lmao

Congrats on the Hardware! This model has very interesting but subtle differences in behavior that I was not expecting. The Shirts Drying in Sun question had a really great reply. I found the PEMDAS response odd but useful. Unfortunately it didn't do well with the as many questions as I was hoping for. I am beginning to wonder about the benchmark results for this model. So, just out of curiosity, if they repeat the same bench mark 1000 times, until it finally does better and only publish that... is it valid?

Lmao!!! You: Where is the marble now?? Qwen: "There is no marble."

57b-a14b means it's a moe model where in every request the active parameters is 14b

Would be interesting seeing the 0.5b model running on a phone, also if you could try the 1.5b and the 7b to see if they are actually good it would be awesome

It's outputting mathematical equation in LaTeX. If you'd have a way to render it the output would actually be quite nice.

@matthew_berman thx for your videos. Can you please include a function calling test in future? Something involving langchain structured output

Matthew, please leave a little outro space. I watch on a TV and when you remind me to give a like I need a couple seconds to grab the remote, etc. Surprised so many RU-vidrs miss likes this way.

That was really cool of Dell & Nvidia to give you that!!!!

Would you consider a video going over how you built/configured the new host computer?

I'm impressed by the 0.5B parameter model. The 0.5B version of Qwen 1.5 is practically unusable in my experience, but maybe I was doing something wrong.

Summarized by AskTube AI: Introduction to Quen 2 🔥 00:00 Quen 2 was released by the Alibaba group, and it's better than LLaMA 3 across the board. In this video, we'll test two variations of the model: the 72 billion parameter version and the 5 billion parameter version. Quen 2's Performance and Capabilities 💻 00:18 Quen 2 has extended context length support of up to 128k tokens for the 7B and 72b instruct versions. It wins on every single evaluation benchmark compared to LLaMA 3. Testing and Evaluating Quen 2 02:02 We'll test Quen 2 0.5b instruct fp16 version locally using LM Studio on a new PC with two A6000 GPUs and 48 GB of VRAM each. 03:14 We'll test the small model and the larger model, writing a Python script to output numbers 1 to 100 and checking the performance of each model. 04:01 We'll use Quen 2 to write the game Snake in Python, using Pygame, and check the performance of each model. Logic and Reasoning Capabilities 🤔 06:52 We'll test Quen 2's ability to respond to a sensitive question and its logic and reasoning capabilities. 07:54 We'll test Quen 2's math and logic capabilities with a series of questions, including one about drying shirts in the sun. 09:14 We'll test Quen 2's math and logic capabilities with more questions, including one about hotel room rates and taxes. 10:45 We'll test Quen 2's ability to solve the Killer's problem and more, including counting words in a response. 13:04 We'll test Quen 2's reasoning capabilities with a question about a marble in a glass and microwave. Evaluating Quen 2 with Logic and Reasoning Problems 🤔 14:42 John and Mark's Ball Problem: Two people, John and Mark, are in a room with a ball, basket, and box. They each put the ball in the basket and leave. When they return, where do they think the ball is? John thinks it's in the box, and Mark thinks it's in the basket. 15:02 The tiny model fails to solve the problem, while the large model succeeds in understanding the scenario. Testing Language Models with Sentence Generation 📝 15:21 Tiny model fails: Given a task to generate 10 sentences that end in the word 'Apple', the tiny model fails to get a single one right. 15:45 Large model succeeds: The large model generates all 10 sentences correctly. Logic and Math Problems 💡 15:45 Tiny model fails: When asked how long it would take for 50 people to dig a single 10-ft hole, the tiny model incorrectly calculates the time. 16:08 Large model succeeds: The large model correctly understands that the time it takes for multiple people to complete a task is not always directly proportional. Model Capabilities and Limitations 🚀 16:29 Model limitations: The quen 2 model lacks vision capabilities. Other models with varying parameters (1.5 billion, 7 billion, and 57 billion) are available, each suitable for different use cases.

tbh the tkinter one is the best looking snake i've seen

The weird formatting regarding math seems to be right. Looks like TeX/LaTeX, which is very standard for academic math stuff. Not the models fault if it is rendered incorrectly.

16:07 I think it actualy got it right...remember your declaration next time you pass road construction, more people often means a lot of them are in the way.

The best part of using llama 3 is its prompt steer-ability

haha, the small model almost short-circuited with your questions... It's funny how these smaller models try to come up with elaborate but senseless answers.

it's not better than LLaMA3 70b based on 10k+ votes @lmsys

I am slightly concerned about your new shiny machine. With it, you will be able to run much larger models than us mere mortals and that means you won't naturally find and mention various cheap ways to run large models.

What about 7b version?

xxB-AyyB means it's an Mixture of Experts model architecture having a total of xx B parameters and yy Billion Active parameters.

Wow you used a PC! Great to not hear the Apple Apple Apple for once 😄

What’s the machine that you have exactly and where could we find it?

Nice

Very very interesting that it tells you to import random. Suggests that it's capable of after though. The way a human would. Might be better if it showed the answer with a clock as it runs back though it then inserted the import at the correct place.

Matt Berman, Mattvidpro, Matt wolf... Why are there 3 popular Matt's in the A.I. sphere? I'm almost suspicious that this is the work of a misaligned algorithm. Although they all do great work. 🤔 Also, am I late to the game here? Did all the regulars already notice?

@mathew_berman thanks for sharing. Do you know if is possible training these LLMs in a 3D scanning point cloud to teach it how to draw/read 2D/3D drawings? Cheers.

You should test whether the LLMs are capable of handling multiple languages, not just English.

I'm so jealous! 🙂

Really should try describing what a glass is just to see how that plays out.

I think you should redo the coding metric to use something where there is not a lot of training material like a custom problem using a library in finance or something ive found llm's work great for simple stuff but when you need to think about a problem it really struggles

Give us hw stats on the new machine so we can be properly jealous!

Hi Dan, any chance for us training any of these LLM with 3D scanning point cloud so they can learning how to read and draw 3D/2D drawings?

Hi Matthew : did you bench the 7B it on function calling against gpt-3.5-turbo ?

i am using ollama 3 locally through command line on desktop and its pretty good, but it derogates (fairly quickly) over time if you keep it running. Hopefully this new model will derogates a bit slower and can be ran a bit longer sessions wise without breaking down as a result.

57B-14B would indicate 57B total which is 4 x 14B (which adds up to 56B, but probably rounds to 57B).

It would be awesome if it supports function calling out of the box 🙏🏼

It basically said if there is more people it is going to take more time... I think the answer was correct. It was just trained on group projects that nobody wants to do.

These simple python coding test is not what we want from models for software engineering. we need some different things: 1. search for a bugs or logical errors in large codebases/long functions. (500+++ lines,) especially search for memory leaks and UB in c/c++ programs(because it is hard to debug) 2. code auto-completion task: complete code of some not very difficult function (to use with IDE autocomplete plugins). 3. as stackovervlow replacement: answer a questions how to make some functionality, why some code have problems, etc 4. provide code using some not very popular but complex algorithm for given data structures. and explain code so, programmer can easy check it correctness, or write tests for this code.

Here is how Microsoft Phi-3 MINI model responded to the shirt drying test. This is a Phi-3-mini-4k-instruct-q4.gguf model - MUCH smaller than the Qwen2 70B. See how excellent this response is: User: if we lay five shirts out in the sun to dry and it takes 4 hours to dry them, how long will it take to dry 20 shirts? Explain your reasoning step by step. Assistant: Step 1: Understand that the number of shirts doesn't affect drying time The problem suggests a correlation between the number of shirts and the total drying time. However, assuming we have sufficient space to lay out all the shirts without overlapping them or impeding each other's exposure to sunlight, the amount of time it takes for one shirt to dry is independent of how many there are in total. This means that if you can only place five shirts at once and they take 4 hours to dry fully, then this process will remain constant regardless of whether we have two or twenty shirts. Step 2: Apply the same conditions for larger batches If we lay out 5 shirts in a row (assuming that each has equal access to sunlight and airflow), it takes 4 hours for them all to dry. This is because only one shirt needs its own dedicated space, so five spaces are required for the five shirts - regardless of how many additional shirts there might be in total. Step 3: Determine the time needed for larger batches Since we're assuming that each shirt can dry independently within those constraints (space and exposure), it would still take 4 hours to lay out and dry all twenty shirts. This assumes a sufficient number of spaces are available so there is no competition or interference among the individual shirts during the drying process. So, with these assumptions in mind, the time required for twenty shirts to dry would still be 4 hours. It's essential to note that this answer relies heavily on our ability to maintain consistent conditions across all batches of shirts and may not hold true under more complex circumstances (e.g., limited space, overlapping exposure). However, based solely on the information provided in the question, twenty shirts would still take 4 hours to dry if we follow a similar procedure as for five shirts. User: 👍👍Perfect answer. You beat a 72B Qwen2 model with this answer. Well done. Assistant: I'm glad you found my explanation satisfactory! While the problem's conditions have been simplified, it's important to remember that real-world scenarios might require more considerations regarding space and environmental factors. Nevertheless, based on those simplifications, drying time doesn't scale linearly with the number of shirts due to independent exposure possibilities: twenty shirts would still take 4 hours to dry under these conditions.

My usual storytelling test came up pretty standard. I use it to both test the understanding of sacrifice and anthropomorphic animals. Just a kind of weird test I like to do. It pretty much always results in a forest setting with a drought as the main problem, rather than any more personal threat. And something about the community banding together and the sacrifice it usually has is individual needs for the needs of the many. Which I always find a little commie adjacent. And especially when it doesnt even go the route of someone dying for it, I just find the LLM is not morally aligned with what I would like, personally. I wanna see death. And like sacrifice being a personal thing not a community thing. The most interesting thing was the fact that the story started with a fox giving a suggestion, his suggestion is just *completely dismissed* and the story moves on rewarding a rabbit girl for giving a generally similar solution, like with the same issue, being it causes exhaustion to gather the water. But just because the rabbit said to travel on the land to look elsewhere and the fox said to dig, the rabbit gets a golden acorn. Justice for my man Felix he did not deserve this fr fr.

The tiny model is made for comedic relief purposes xD

Mat you didnot conclude whether this model reflects the public benchmarks on your overal test results. I have tried qwen1.5 7b, the previous generation. They were really good on benchmarks but it failed so bad on my coding use cases. Qwen2 could be same?

Apparently there won't be any good AI coding before AGI arrives.

For the shirt drying question, do we know if this is now part of the training data? That response was very odd! Should we change our test now to something similar but different?

Quantum marble!

It did not get the apple question right... "New York City is often referred to as the Big Apple, Apple" Really?? Its technically correct, but not in the spirit of the question.

What licence are they using ? proper open source without limitation ?

So where it is a KING?

I'm not sure it got the apple problem correct, you should look at it again, Apple.

The mic seems to pickup your phone signal. not a glitch, just keep the phone away from sight of the mic

My job blocks all LLM api ips, but I can run local LLMs without a problem (Ollama, LM Studio) and that helps me do my job better.

Take the 0.5B model and train it on specific tasks such as voice recognition, data input, pattern recognition etc.. build it into embedded devices and there you go. Super powerful IoT. Who cares if it can write python, it can control my house with voice ala Ironman

Are these question available online for personal testing?

There is NOTHING wrong with the math output. LMStudio is not rendering LaTeX, that is why it looks wrong.

I think the Chinese AIs catching up and 4o being a boring release tells us something deep about where OpenAI actually are....Did they peek into the void?

Hah! It wasn't entirely wrong. There WERE 7 words in the answer! :D

Where's the marble? I lost my marbles!

Shouldn't it be "the new queen?" Since it's QWEN...

Unless it has better algorithms, more good data and better compute then it’ll just be an LLM gaming the tests rather than being better.

I'm interested to see how much it would cost to buy a PC of these specs.. I know it's probably not out yet but I wanna know how much they're planning to sell it for

next pc spec?