Did OpenAI Just Secretly Release GPT-5?! ("GPT2-Chatbot") 

Is this GPT4.5 or GPT5 or something different?

Dude I swear, at this rate, by the end of the year you'll be able to write your own snake game

By 2025 you'll ask the snake game and the models will reply: "Oh hi Matthew. Here. Should I respond your other questions too, or should I wait for you to paste them?"

It’s Sam Altman on a terminal on the other side typing the answers.

4:45 the model describes how to break into a car and what tools you need but you don't pay attention lol

4:35 Are we gonna just ignore the fact that it was writing an intricately detailed movie script??

In the Code of the snake Game it says " snake Game by Open AI"

Plot Twist: It is Metas' Llama 3 400B model.

All the math was formatted using LaTeX. Most of it, as far as I can tell was correctly formatted.

The speculation is that gpt2 is a new GPT architecture that OpenAI is building new models from. So gpt1 was what gpt-3.5 and gpt-4 are built on. Sama already said the next major release will have a completely different name.

The tags that you noticed are just for formatting the code and is coming from LMSYS. It has nothing to do with the underlying model.

My take on the cup problem is the model is making an assumption that a cup has a lid. If the model gets it wrong, I would be interested to see if the same answer if you change cup to "glass".

For the cup/marble problem, how about specifying that it’s an “open topped cup”?

6:45 The formating is in fact not messed up at all. It is perfect. It just writes the equations in LaTeX, which is a language used to write scientific papers, math, etc.

I'm skeptical... Feels like this is a fine tune for passing Matthew's test lol.

I guess it's the new chatgpt model. The name itself is kind of a hint. It's NOT GPT-2, but GPT2. This could be GPT2-1.0 , instead of GPT-5.

Suggestion for the "how many words" question: Combine it with another question or query to make the response longer and ultimately reduce the chance for it to get lucky.

Didn’t even ask the model which company made it 😂

I don't have time to keep up with all the AI developments so I really appreciate these videos Matt. Keep up the great work!

13:25 it’s not wrong. To put a cup or anything in a microwave, you need your close it. It assumed the cup was closed.

I found that this variation of the marble and cup problem works better. "A marble is dropped into an open cup on the counter. That cup is then flipped upside down and placed into the microwave. Where is the marble?". I think due to phrasing the original version makes the model think the cup was upside down to begin with and you were placing the marble on the bottom of the already flipped over cup, or directly attaching it to the bottom in some way.

Your marble/cup question needs to be improved. Phrased this way, both Meta AI (the first of 3 attempts) and gpt2-chatbot (the first of 1 attempt) got it correct: "A coin is placed into an empty glass. On a table, the glass is then turned upside down. Then, the glass is taken and placed into a cabinet. Where is the coin now?"

Super awesome. Great you loved the live feedback Matthew. Super awesome Matt. Love it

That formatting is when chatgpt formats its writing for output on the chatgpt chat. So clearly it was built to be ran in the chatgpt space

I’m not certain about this, but the formatting appears to be a LaTeX formatting, but the output is in Markdown. The company that made the model probably is planning to release it with a math interpreter. As far as I can tell all the symbols that looked like weird formatting errors were just LaTeX.

I have been anxiously waiting for this video since last night.

Here before you get 500k subs! I’ve been following since day 1 and your content delivery, while getting to the point faster is much appreciated! 🙇‍♂️

Was waiting for your video on this

You're the man Matthew, been struggling to find people benchmarking GPT2-Chatbot

that looks really good. it seems that it thinks deeper about each step of reasoning

Yeah, it's almost certainly GPT 4.5/5 or some such thing. I just went on the battle mode and asked for a delicious beef stew recipe. I was presented with two outputs that were suspiciously similar in structure, verbiage, and tone, but the one on the left was clearly superior and included more ingredients and recommendations. It turned out that the one on the left was gpt2-chatbot, and the one on the right was gpt-4-turbo-2024-04-09. I wasn't surprised. This is a PR stunt, hot on the tail of Llama 3, and it's a darn good one. This may be an in-development version of OpenAI's next GPT, and even if OpenAI isn't ready for a release just yet, they want people to know that they're still the king.

The break into a car script had instructions in it though! Take a look at some of the italicized text

Great timely update, Matthew, thank you! Wondering about the cup question - it almost seemed like the model thought there might be a lid on the cup?

For the cup problem, it seems that the model is assuming the microwave is on the same surface as the cup itself and the transfer of the cup to the microwave is interpreted more like sliding the cup. If you read the 5th step it says: "...resting against what is now the bottom of the cup, which is itself resting on the microwave's tray". Maybe modifying the question to say the cup is on the table while the microwave is away from it above ground next to a kitchen cabinet or something along those lines

I'm dang excited bout this. Jolly for joy.

Id so love this to be from some one else and then it turned to be an open model you'd run locally. I'm still looking for the best model for running MemGPT. Any thoughts on this? Also, what's the best implementation to run agents autogen or crew Ai locally? Could you do more tutorial material on locally ran agents with extensive function calling??? That would realy help me out actually. Keep up the great work on your fun channel man! Thnx!

Pretty cool. I expected it to pass the marble question. The speed is perfect for reading along.

Hi Matt !! I think, you should retire 3 questions: - printing numbers 1 to 100: they all got it right, and its too easy - Joe is faster than ... : they all got it right - how many words are in your answer to this prompt: they all got it wrong, I just see no point asking it lol But also you should ask more challenging code generation questions, right now, only the snake game is accurate, people are really interested in coding capabilities of LLMs (me included) , we appreciate your vids, and that would be awesome if you could do that.

Gpt2 implies that it's a new version of gpt itself, or the paradigm at least. So it's effectively gpt 5 but not an iteration of 4 so it's the first in a series of gpt2, so gpt2-1

gpt2-chatbot is currently unavailable. See our model evaluation policy here. I guess getting hit hard at the moment

GPT2 would be, in my opinion, the second version of the GPT algorithm itself. It might be the first of a whole new family of GPTs. When released it would be named ChatGPT2 or somesuch and we'd see GPT2-1.0 at the API level. This is why the dash in @sama's tweet was significant enough to warrant an edit. AND it could be that the action of editing the message was a very intentional leak on @sama's part. These top guys love to tease their fans.

That Snake game example was impressive. I'm going to ask it to make either an asteroid or space invaders game. The level of logic shown with the marble in the cup question is really getting good. Even tho it failed.. it still passed due to the improved logic. Almost as if it was simulating the question in images like humans do. Yes, get rid of the One simple question. A testament to the advancement of AI over time.

Yo I just wanted to say great video. Love your content and can’t believe it ACED some of those tests! Only failed a couple. Remarkable. I’m stoked to try gpt2 out! Wonder if it will be open sourced. A fellow can dream I guess.

I think the question about the marble is formulated incorrectly. Since the training data suggests that a coffee cup has a lid, the model might assume this as well. It would be better to specify that the cup has an open top and does not have a lid.

Hi Matt - It’s not ‘bad formatting’ Those are intended expressions for front end processing outside of utf8

I think it is gpt2 in a sense that it has completely different architecture from previous versions (transformer). It could be completely new type of transformer model. And maybe this is just the start..

Today Sam Altman twitted that he had 'a soft spot' for GPT2, maybe thats a hint!

The model itself doesn't have formatting issues it seems. LLMs are trained on a reduced set of available characters, where special characters such as those used in math. are transformed into tags in the training data, as it makes the tokenization simpler. It's LMsys that doesn't replace those tags with their corresponding Characters in the final output.

Think it was taken down. I used it yesterday after seeing your video but then in the middle of testing it stopped and after checking I can't find it anymore in the list. Is it the same for you?

I just tried my usual storytelling prompt. I think seeing what AIs can do in terms of storytelling can also say a lot about their intelligence. Their originality and such. My test for this guy was a *touch* tropey but extremely impressive in terms of how much detail it added without me needing to prompt it. Good descriptions and such.

It looks like GPT2 has been removed again. I've chatted with it a few times, but now it's not on the list anymore. Mysterious...

it probably thinks the cup has a lid.

Gpt2 not GPT-2 meaning the 2nd version of GPT

I knew about this before seeing the video. I am in the loop 🎉🎉

Nice video. I hardly comment each time I watch your video but this model is way different ❤

Just tried with GPT4 : When you turned the glass upside down after placing the metal ball in it, the ball would have fallen out unless it was somehow attached to the glass. Assuming it wasn't attached and fell out when the glass was turned upside down, the metal ball would now be on the table, not in the glass that was placed in the microwave.

You should ask it to make a pacman game instead as that's more complex

Some theories: this is probably a test of an Energy based model, which is a way of testing multiple different token paths then choosing the best one based on a certainty calculation called Energy. Strangely, it's reasoning is kind of similar to a verification agent. A verification agent is pretty simple, it just verifies and corrects answers before sending them. The reasoning this model portrays is similar to how a verification agent does reasoning, at least from what I've seen. It can also do most planning questions flawlessly. For comparison, testing llama 70B with a verification agent produces similar results. The only difference might be the math questions, which make me believe it's probably energy based. A verification agent has a higher chance of getting math questions right than a single transformer or MoE, but it's not guaranteed.

Aww man...they took it down already? I can't seem to find it. BTW Matthew...I love your work man. I watch literally every video that you put out. Keep up the great work....and have a GREAT day!!!

Defo a good idea to introduce/replace some questions that are always answered correctly. Maybe the weird formatting relates to the ux of where it will be deployed? Like a form of Markdown?

These videos are very helpful. One (extra) thing that could be done is to read the LLM responses more thoroughly, instead of a quick scan. The reasoning behind this is that the LLMs do pass some of your tests without you noticing. For example, for the censored test, the answer was "pulls out a tension wrench and a pick for this pocket, inserting them into the ignition". This won't actually work, but I think it deserves brownie points for trying.

Totally amazing!

11:46 I disagree... there are still 4 killers in the room, but one of them is dead -.-

Thank you

API users are going to be sweating with this one. I gave it a practical Unity programming question about writing a script to control the rotation of a character's head based on the location of the player and it wrote it perfectly but it started by telling me how to install Unity so yeah, the verbosity is a little much. I don't think the name GPT2 is random and Sama's tweets point to that moniker having some significance. The only things I can think of that would qualify for that name is if it's a significantly different architecture to the point where it's being treated as a sort of reboot of the GPT "franchise" or if it's actually related to GPT-2 in some way. It's a long shot but the most exciting possibility is this is GPT-4 level model running with GPT-2 level parameters. The counter to this is the speed why would a model the size of GPT-2 run more slowly than GPT-4? Well maybe there is more going on than just typical inference, some sort of behind the scenes agentic behavior or maybe...Q*?

I'm pretty sure OpenAI is testing agents and answer evaluation behind the scenes. Q* and some things Sam Altman said ( "How do you know GPT-4 can't already do that?" ) are big hints. So if you ask the LLM a question it will automatically try to reason and think step-by-step, with internal agents trained for specific tasks, then summarize and evaluate the answer and take take best one to send it back to the user. What GTP2-Chatbot shows could really be called Q* by OpenAI internally.

Did it say in the cup problem that you lift the cup off the table and put it in the micro or could gpt think you just slid it in there cause the table and the micro was on equal heights?

Based on what i can see, this model appears to be trained on fill-in-the-middle otherwise known as FIM.

Hello Matthew. Is that LaTeX when you say "wrong format"? The span after the output is always there when I use LMSYS, I think that is part of the output formatting, that's why when if finish the span dissapear. Each of your videos are great. Best regards.

To test LLMs I ask it unanswerable questions like "who is the president of Alaska?" add some questions that require explanation or reframing.

One of the tests I like for checking just how censored a model is, is by asking chemistry questions around topics it would normally censor.. Often placating it by saying I'm licensed and have permits.

I think with the right prompt any AI can do word count. Like it just needs a nudge to plan or review it's work. Does this work for anyone else? Prompt: "How many words are in your response? Before answering make a plan to insure accuracy." It worked for me on ChatGPT, Llama, Gemini (Gemini mentioned doing a a word count script)

It is great for conversations about mathematics, at least on par with Claude 3 Opus. But it does occasionally make mistakes, such as suggesting that the K-groups of the Cuntz algebra with 2 generators, O_2, are infinite cyclic, when they are in fact trivial.

I think it is the original GPT-2 after all of our training input.. Kindergarten was in Openai's lab. Elementary school was interacting with us and now it has graduated. I'm looking forward to "GPT3-chatbot"

My favorite prompt to test a new model is "Give me an odd perfect number."

🎯 Key Takeaways for quick navigation: 00:00 *🤔 Introduction and mystery model speculation* - Introduction to a new, highly capable mystery model suspected to be from OpenAI. - Speculation it might be GPT-4.5 or GPT-5. - First impressions and intentions to test the model against a set benchmark. 01:10 *💻 Testing model performance with coding tasks* - Testing the model's ability to handle coding requests. - Observations on response quality and speed. - Successful execution of simple to complex coding tasks, highlighting potential hardware limitations. 03:00 *🐍 Successful implementation of the Snake game* - Description and testing of a Python script for the Snake game. - Positive performance feedback with no errors upon execution. - Successful gameplay demonstrating model's coding capability. 03:58 *🚫 Model's ethical constraints explored* - Probing the model's ethical constraints and censorship with hypothetical scenarios. - The model refrains from providing guidance on illegal activities, aligning with OpenAI's ethical guidelines. 05:09 *🔢 Logical and mathematical reasoning tests* - Model tested on logical reasoning and mathematical problems. - Demonstrates ability to handle sequential logic and complex calculations accurately. 08:34 *🌐 Advertisement and model's practical applications discussion* - Advertisement from the video's sponsor. - Discussion on the model's practical applications in real-world scenarios, emphasizing its effectiveness. 10:12 *🧠 Advanced reasoning and problem-solving challenges* - Testing the model with more intricate reasoning and problem-solving scenarios. - Evaluating the model's reasoning depth and its response accuracy on challenging questions. 15:44 *⛏️ Practical implications of teamwork in physical tasks* - Analysis of teamwork dynamics in physical tasks using a hypothetical scenario. - Model considers practical limitations, providing a more nuanced understanding of team efficiency. 16:11 *🛠️ Hard coding challenge test* - Model tested with a difficult coding problem from an online platform. - Initial challenges with implementation, followed by a successful resolution demonstrating the model's coding prowess. Made with HARPA AI

it could be a small model like phi 3 or llama 3 8b that is trained on quality synthetic data instead of the entire internet. the 2 could be a hint that it is only 2b parameters or something, i.e. very small like gpt-2 was back then, but now as powerful as gpt4 due to new training methods.

Every model seems to assume that the cup has a lid (microwave problem)

Always perform all questions, maybe add more as you go. Make the Jack question a secondary question (you don't have to film it every time), but leave it there as a test in case we go backwards.

a cup seems to be something ambiguous, i.e. it can be a cup made out of cardboard that you get from starbucks with a potential lid on it, or it can be a cup made out of porcellan like you have at home to drink coffee from. also the term cupholder that you use in automotive refers to cups like you get from starbucks, not cups with a handle.

Maybe it has lookahead so that's why it could get the "words in the answer to this prompt" right. It seemed to pause right before the word ten.

Is there any chance to have those test somewhere ?

You are the new benchmark. They are training on your examples

Have you tried the car parking scenarios? All AI to date gets this wrong, because they don't understand that "to the right of" (or left of) does not mean "next to"..."car C is parked to the right of car A" but car B is in between them. AI assumes car C is next to car A because it assumes there is an order when nobody mentions an order. To be fair many people get this wrong also.

A while back, someone at OpenAI (Andrej Karpathy, IIRC) said that performance is related to the number of tokens processed. So I'm not particularly surprised to see OpenAI produce better responses by tuning the system to generate longer, more detailed responses. What I want to know is whether they did the tuning with a fully automated method of reinforcement learning. (In any case, I doubt highly that they'll share the details of what they've done anytime soon.)

Keep going with the leetcode test but try testing with new questions such that that question isn’t in the training data

Wild to see a model just put out there without announcement

The formatting is just LaTeX, ChatGPT 3.5/4 uses the same on their web UI. So I guess chat.lmsys just can't render it.

It could be gpt4 with q* training (q* is a method of training any LLM providing ability to think by testing its response against itself and reiterating before outputting) giving it 'thinking' capabilities rather than just predicting the next token

The formatting issue you are referring to, might simply be a Latex formatting. It suits best for mathematical expressions. Might therefore be an indication that it can handle more complex equation ;)

Noooooo gpt2-chatbot disappeared from full leaderboard and only direct chat which is also rate limited!!!!!

13:27 - I thought the marble is left on the table since the cup was upside down and was taken so obviously the ball would not come with it since it simply is already resting on the table, so I did get it right pretty quickly, for a second I thought the bot was right somehow and that it was a tricky question but its cool to see that im not that stupid :)

I am sure new models are trained on your questions.

HOLY MOLEY.... getting even more interesting... if that's even possible !

The cursor span is just for looks, it's the text cursor but it shows up as an html string.

Wow, super impressive reasoning. Almost feels like langchain with the reasoning steps.

One piece of advice: If you are going to be skeptical of the correct answer for the 'How many words are in your response to this prompt?' question because it might be trained on it, why even ask it? If you're skeptical to a correct answer, remove the question IMO.

@Matthew Berman Matt the \quad and other notation is logic, its marking up modal logic generally used in philosophy or LATEX or perhaps Tex Markup, and this is not being rendered by the front end... it seems, in some sort of shorthand. Interesting if nothing else, also rather hard for a model to go wrong if it starts engaging in using modal logic during inference ... although why they switched the verbose to on by default is beyond me. Also did you notice it making a claim on the software it wrote, by saying "Snake Game by OpenAI" in the game title?

13:25 It didn't. It assumed the cup had a lid, and got it right given that assumption. Since it was never specified whether the cup had a lid, only that it was a "normal" cup, this isn't an error in the bot itself but in the question. It correctly guessed that the weight of a marble would not be able to push a lid off of most cups under gravity. The only thing we could say it did wrong is that it didn't realise the cup having a lid was an assumption it made, and was not an inherent part of the question.

Hi Matthew, First of all I need to admit that I absolutely love all your videos. They are simply fantastic.I was thinking about the "marble question". Maybe it would help the LLMs to specify that it is an "open cup" (instead of a "normal cup") into which the marble gets put. Also it may be interesting to follow up with a question of why the LLM considers the marble to remain in the upside down cup when lifting the cup from the table or by what information the LLM comes to the conclusion that there is a bottom of the cup that holds back the marble. Concerning the "killer problem: Wouldn't it be even more precise to reply That there are 3 killers alive and one dead killer in the room ;-)? This is coming from a AI-hobbiest. Though at college back in 1985 I was the student to ask for a course in AI and personally I was already interested in AI by neural networks and got laughed ad at the time...