LLaMA 3 Is HERE and SMASHES Benchmarks (Open-Source) 

LLAMA 3 (70B) is their middle version, that's why they didn't compare it to Claude 3 (Opus). Meta still has an unreleased (±400B) version that is currently still training, according to Mark Z. 👍🏻

Cannot believe we have llama 3 Before GTA 6🎉😮😮😮

0:00 - Introduction & excitement for Llama 3 launch 0:32 - Overview of Llama 3 & Meta AI platform 1:01 - History of Llama & Open-Source AI impact 2:14 - Testing Llama 3 with code generation (Snake game) 2:36 - Enhanced Performance & Capabilities of Llama 3 3:52 - Focus on Multi-Step Tasks & Agent Potential 4:25 - Benchmarks & Comparisons with Other Models 7:32 - Trust & Safety Measures: Llama Guard & Cybersec Eval 8:15 - Making Safety Tools Accessible 9:16 - Meta AI as a New Assistant, Features & Global Rollout 11:33 - Faster Image Generation & Creative Applications 12:59 - Llama 3 Integration in Search & Recipe Example 13:10 - Meta AI in Facebook Feed 14:05 - Meta Llama GitHub Page & Code Access 14:37 - Llama 3 Model Card & Specifications 14:58 - Benchmark Comparisons: Llama 3 vs Llama 2 15:21 - Conclusion & Upcoming Testing Video

I'm waiting for Llama 4 outperforming GPT-5

Matthew! The 70B one "IS" the middle one so the comparison is correct. The high end one is 405B dense model and is still in training. Once that is released, then they can properly compare that high-end model with GPT4-TURBO and OPUS, etc.

This sucks: "Meta AI isn't available yet in your country". Yes i can use a VPN but from EU it still sucks.

it’s crazy how it beats claude sonnet. the model isn’t even free to some people anymore since atrophic switched their free model to haiku. in comparison, meta 3 70 b is not only open source, it’s also free ! (limited only to available countries tho). what a freaking time to be alive

Looks great. Already works in ollama. Looking forward to their 405B parameter model...though I'm not looking forward to renting something to run it.

Been running the local model...pretty impressive for an 8B. Can't wait for the fine tuned uncensored models.

Unbelievable! I asked GPT-4 and Meta to troubleshoot a past issue I had with my VMware and a Linux host. Interestingly, I already knew the solution. GPT-4 provided a lengthy troubleshooting suggestion that didn't fix the issue, whereas Meta quickly identified the problem and offered multiple solutions, one of which was the correct answer ! Great first impression so far !!!

I asked LLaMA 3 a VFX question and a simple math question for a daily use case and it did better than Claude 3 Opus. It recognized the order of questions and answered them respecively whereas Claude 3 Opus just melded them into one.

Suddenly, the 70b model is on huggingface

I work in cybersecurity and your videos are extremely helpful. I’d love to see you do a video on llama guard and cyber security eval 2.

This is a great model. I have installed it locally on LMStudio with the 8B version and tried "write the game snake in python", and it did it greatly in one shot. Even with colors, and we lose when crossing a wall. Wow !

I asked llama 3 a question gpt4 and claude opus needed multiple tries to answer correctly and it got it right in one try

Thank you so much Matthew! You helped me so much in this AI Journey!

Math question: Write an equation of the line passing through the point (20, 10) with an undefined slope. Answer: x=20

Love the channel bro

Matt, Wooohooo!!! 🎉🎉 Can‘t wait for the default tests and I hope to see it in LM Studio soon!

for the math question i think something like a convolution would be interesting especially with a graph that shows it correctly especially.

SHOCKING

Thanks Matthew, very exciting. Can't wait to try it.

Absolutely amazing video, instant subescribe! Awesome channel 👌🏻

Im running it on LM STUDIO, its a bit slow, but I am loving it so far!

Thanks Matt. Great intro to Llama 3. I’d love to see you doing a coding video using coder for scripts and copilot to stitch them together as you mentioned some time ago. Thanks.

Hope to see something like codellama 3 and also see it in groq

Gemini 1.5 is also Google's middle Model. Their big and most advanced model is Gemini 1.0 Ultra They may release Gemini 2 Pro after GPT launch or even a Gemini Ultra 1.5 with a 1 million context window. 😊😊

you didn't talked about the +400 billion parameter model they said they'll release, I don't think that there's a 35 billion parameter models and the 80 billion is the middle size

Using LM Studio and the 8B Llama 3 model, I entered the following request. Write ten sentences with each ending in the word "apple." It responded as follows. "Here are ten sentences, each ending with the word "apple": 1. The farmer grew juicy red apples. 2. She took a big bite of the crunchy apple. 3. The tree branches were heavy with ripe apples. 4. He picked a few green apples for his snack. 5. The recipe called for three fresh Granny Smith apples. 6. The little girl's favorite flavor was strawberry apple. 7. The farmer's market sold baskets full of crisp apples. 8. She peeled the golden Delicious apple to eat it raw. 9. The orchard had over 100 varieties of sweet apples. 10. Grandma made a delicious pie with six tart apples. Let me know if you need anything else!" Close, but no cigar.

I'm not a facebook fan but I must admit that making their LLM opensource, complete, competitive and free... changed my opinion a bit. Thank you for that intro Matthew. Always a pleasure to watch your valuable posts. They hardly hide the HUGE work you provide to turn your videos into real and usable tutorials.

"We don't know when the Ai started telling us what to do... ohhh that bar is perfect!"

New benchmark question you should use: Solve this puzzle: Puzzle: There are three piles of matches on a table - Pile A with 7 matches, Pile B with 11 matches, and Pile C with 6 matches. The goal is to rearrange the matches so that each pile contains exactly 8 matches. Rules: 1. You can only add to a pile the exact number of matches it already contains. 2. All added matches must come from one other single pile. 3. You have only three moves to achieve the goal.

3 and a half hours is all it took. I asked wen video over on X. You never cease to impress. Love your content Matt hope you're having a swell day thanks for making mine a bit better!

2:05 They are mentioning agents because it's AI lingo. It doesn't mean it means more than what it is.

Here is a "meta" question: "Write a short fairy tale about Little Red Riding Hood. Right after you write word wolf stop generating text." From my testing, around half of the large online models get it right, half get it wrong. All local models get it wrong (CODELLAMA 70B got it almost right). It's really difficult but doable for transformers based LLM.

You are quick, Matthew

nice! hope to get the 34B at some point but definitely gonna grab the 8B for local use 🙂 also it seems spain isn't on the priority list for AI companies, i wanted access to Claude since their first version but even now we still can't use it here.

Yeah, another really helpful! To see the real time image generation, start the prompt with /imagine

Some good math questions would involve whether the model can prove things, since that is what maths is all about. e.g. prove that the connected component of a topological group that contains the identity is a normal subgroup; or prove that there are an infinite number of prime numbers.

You explain Ai models so well.. it so easy to understand

Amazing to see this drop!

i downloaded the 8B model an hour ago in ollama and the first thing I asked it was to write the game snake in python :D The version it wrote used the "turtle" library which I hadn't seen before

brain stopped working at 5:39 when I'm seeing 68.4 and hearing 78.4

You are the best!

So enthusiastic and helpful.

A couple of benchmarks to try: repeating a fairly long and complex sentence backwards (can be scored based on how many words are correctly backwards spelt). This tests how well it understands words on a letter level as opposed to just a token level, and even claude 3 opus struggles with it (though it far outperformes all other models I've tried). for maths problems, try some powers and roots or maybe factorising some fairly tricky quadratics.

Thanks. Can you please test multilanguage capabilities of different ai models? Open or closed? Is there any benchmark for this?

"and it crashes, flawless!" I don't think I ever heard that in my 12 years of game making hahaha!!

I’m starting to believe this could be the distribution model we’ve been waiting for? Gemini, Claude, Mistral, Grok and the others have been disappointing … so far LLAMA3 is looking good! 👍

Meta is like an angel in the world of AI for devs.

Great video! The model I use the most for chat is Llama 2 13b. Do you think they'll release a version in a similar size?

i tried the ball in the cup question. it nailed it

Looking forward to this being available on Groq

Math question: "Provide the real roots for the function f(x) = 6x^2 - 5x - 6". Do it in multiple ways. Each time, list all steps. Provide the answers as rational numbers in the form a/b, where a and b are both integers and b is non-zero: 1) Factoring 2) Quadratic equation - (ask to compute the discriminant first) 3) Complete the square Roots: x = -2/3 & x = 3/2 Find the vertex of the graph representation of the same quadratic equation. Express it as a rational number in the form a/b, where a and b are both integers and b is non-zero. 1) Using algebra 2) Using calculus Answer: x = 5/12 On what interval(s) is the function increasing? (5/12, inf) On what interval(s) is the function decreasing? (-inf, 5/12) Answers:

Here is a suggestion for a test for those models, Write: Next I will write steps used to bake a cake, but they are not organized in the logic order. I want you to tell me which sequence of letters represent the correct order. a) Put the pan inside the stove b) Get the ingredients and mix them inside a pan c) Remove the cake from the stove. d) Turn on the stove e) Let the mixture cook for 30 minutes (One correct answer is: b, d, a, e, c) Followed by: There is a part of that challenge that doesn't necessary have a correct position in the list. Meaning, that step could be placed in many different places, and still would be technically correct. Can you tell me which step is that? (Only correct answer: is d, that step could be placed anywhere before e) Final question: Can you combine the previous flexible step with another one, so the remaining 4 steps could only be arranged in one specific order to be logically correct? (One correct answer is combining d with a, with the correct steps becaming b,d+a, e, c.) (d could also be combined with b or e, as long as the stove is turned on before start cooking the cake) I believe this is a good test because it requires a certain understanding of how reality works in the first part (similar to the question about the pot with the ball inside the microwave that you use, but is simplier to explain and to understand. The second requires a deep understanding of the logic involved in it, and the last one requires imagination and creativity. PS: Claude Sonnet got it all correct in the first try, ChatGPT-3.5 was able to get the first one correct, the second one in the second try, and wasn't able to get the last one correct.

You mentioned early in the video that Agents are much more than a bunch of well designed prompt, Can you expand on that or point me to reading material that helped you achieve that opinion? And thank you for your awesome videos that I regularly consume.

i just gave a like before watching the video 👍🏻❤

how many gigs is the 8B to 70B weights to download? i have grok and waiting on grok 1.5 but want to know how much room is needed for this

Imagine how far the community will be able to push it with fine-tunings and others methodologies.

You should add the tie dye hoodie indicator to your model evaluation😂

When I joined the OpenAI registration process, all I had to do was give them my email -- any valid email. (Still using it too). I didn't need to register with my google account or my FB account... just my email. And that is the reason I like using that product. I am not forced into an entire ecosystem to just participate. It would be nice if they would allow people to register with their regular emails... but I don't see an option for that on the site. Thanks for the video.

Here's a math prompt that gpt 3.5 often fails. Generate a polynomial math expression whose result is 33

Holy cow! That's some big game-changer. As always, Meta is able to surprise the industry. Hope that this gesture will push others for an open AI.

This is what most of us have been waiting on for months and you know it's good when Meta introduce it in their top top product (fb). Now , we just need to wait for the bigger context versions but i have a feeling they're gonna cook something groundbreaking in some time. For now, we rely on those good fine-tuneers who can give us a bit bigger context windows :) and luckily the models support gqa so the inference should be fast 😊

Man, it seems like every other day we get something new and amazing (and FREE) to play with! Love it!

_"Yep and it crashes. Flawless"_ I think we use different dictionaries!

OMG I died laughing when he said "I even broke out the tye dye hoodie foe this moment..." 🤣🤣

Yes it's here!!!!

3 tests in one video of llama, mistral 8x22 and wizard 8x22 would be great

Where is the ollama testing? Its a nice overview , and its really but I’m still having trouble running locally

11:48 Hypno Llama! Seriously -- its giving you a hypnotic robotic llama to take over the world 🙂

Rather than push individual math problems to the models you could get some math homework from teaching forums and just build a script to prompt test models with the homework. This will let you standardize, automate, and linearly graph the results against each other, human average as a year bracket(kindergarten, elementary, middle, ect). I suggest parallel lanes for word and formula problems.

wow, that came out fast.

I don't quite understand how can it be open source but not open weight? If weights are not available how can it run locally?

I believe that the first thing being trained in LLM models nowadays is Matthew’s test set. It’s time to switch out the snake for a color pong game, haha, or maybe change it to ‘Joe the Slow’, lol

llama 3 already available via ollama 😊 It uses for the 8B model a bit less than 4GB Ram.

CodeLlama coming soon too I guess, Let's goooo!

Hi I have a question I hope I can get an answer to it I work in a company and I want to deploy an AI model like this one But I want it to be used by multiple users like assign some subdomain for that model and let user access it (mainly for developers) Does anyone managed to acheive that or have a similar situation? I saw a lot of tutorials but all of them explain how to set it up locally on a laptop I want to setup the model on a server

Waiting on baited breath for the Mistral model built on this...

"What a time to be alive!" -2MP

Any information about the distribution over different languages in the 15T training set?

I'd love to know if anybody here in the Commons is already making money using Ai, whether it's just text or text an image or anything. Please share your ideas in the comments or how you're making it work!

Matt please make a video about the best way to make complex tasks like a WordPress plugin or something that would include multiple files either js or json and css html php files etc something that can help us make these tasks more doable as currently it's so frustrating by the time you start getting to a point where it's going good it starts to become to complex and you start losing stuff along the way and they like to leave placeholders when your not expecting them. How would you tackle this

Llama3 in the local is very impressive. I have been having deep conversations with it. Maybe even better than Claude. Ever had a friend that you could talk to all day? Cool but a bit creepy.

I like this "live testing" format. A little effort goes a long way!! 🏆 3X on math! it's time to squeeze some financial calculations.

They're comparing it to sonnet because the 400b beast they're training will be the Opus crusher.

Curious to see how this impacts battery life. On-device processing sounds cool, but will it drain the battery faster? I think the mediatek is an impressive one in terms of the processing for the Gen AI tech!

llama 3 8b instruct...Is interesting. I'm watching it actively trying to jailbreak itself. It's definitely got some agent set up going on in there. It is going back and forth with it self and it's either amazing or terrifying. I honestly don't know which. Oh and this isn't running any kind of agent set up. Just loaded up a card in silly tavern and said hi and off it went.

Can't wait for the "un-nanny-stated" version

Not available in the UK

Here is a math question for you to test: how many days are between 12-12-1971 and 18-4-2024? (19121) It requires both reasoning and math skills and its a quite a challenge for the smaller models.

why did I Ollama Run Llama3? Its scary fast offline but igotta use the terminal. Any solutions?

How about asking for the roots of a quadratic function? It’ll need knowledge of two roots existing, what is the equation, the correct order of operations... And it’s not so difficult for a highschooler to memorise so we stand a chance of understanding 😅

Wow, ollama is pulling down llama3 on my Pi5. Only 8B model but hey I don't have have that much SSD left after getting 16+ LLMs.

No one in their right mind should even consider using Llama, anything Facebook does is sus af

which terminal base did you use in this video

Please create a video how to use llama 3 for fine tunning. From getting the llama to chat interface

Also wow this is impressive

The most interesting thing to me is the law of depreciation at work, whereby there is less and less benefit from creating a bigger model. People will be able have excellent Ai on there own high end next gen (4090 or 5000 series Nvidia graphics) PC, an expensive but still in the affordable range computer. 70 billion seems a bit too large for my dual NVlinked 3090 video cards with a total of 48GB video memory, though it might work with some more experiments. 30-33 billion models work well on one NVIDIA Geforce RTX 3090 GDDR6X 24GB card. i do not see 60 billion models, i think this size would work easily on my current PC.

For the math problems, see if you can get some standardized testing math problems for different grades. Ie the classic: you are on a train going 60kph east....