Gemini 1.5 Pro: UNLIKE Any Other AI (Fully Tested) 

I don't think we need to worry about Google achieving AGI.

Every time I get hyped on new A.I. models release , Matthew brings me back down to earth

Not only does it hallucinate like every other model it goes a step further and starts gaslighting 😂

The larger context window doesn't add much value when the model can't be trusted to answer basic things correctly. It' seems pretty useless unfortunately.

Google Gemini is The Internet Explorer of the AIs

Google's AI models being rubbish again? Shocker : )

It can not create a snake game because eating something is potentially offensive. Also making snake dead by throwing it into the wall is violence.

"It fails left and right, but for no reason: good job Google!"

If this is what "great job, Google" looks like, our expectations for the search giant must be REALLY low...

Did you switch back to Gemini Pro 1.5 after trying Gemini Pro 1.5 Flash?

Unless I can set it to a level where I can ask it anything I want no matter how inappropriate and get an unfiltered response, then it's useless I really don't need nor want some AI trying to control my speech

You said you wanted to see if it was censored, and then you LEFT THE CENSORS ON.

It amazes me that Google would do so badly.

Mom: We have GPT4 at home GPT4 at home:

did you remember to switch it back from Gemini Flash?

I was going to puchased Gemini Pro membership until I saw this. If it can't even create or attempt to create a 'snake' game without erroring out I will wait. Great unbiased review! ty Matt.

We also had an undesirable experience testing Gemini Pro 1.5. It could not correctly understand the context of a large document when we were asking about its content and it could not even find words we asked it to find. 1M token feature can ingest large docs but I don't think it works well as an LLM with the data it ingests.

The test of need in the haystack is fine, but it only check the "search function" in big context. What we really want to know is how well it reasons over this context. For example in the book there instruction how to do something on 1 page. and literally 200 pages later we meet data that we want to calculate correct way, but for that we need instructions from before. If AI is capable to find these 2 things, sum it and give you the correct answer, then it's a pass.

drop the safety settings to 0 on ALL the 4 categories. running the failed prompt should work then.

Cool video. The input context window is cool for sure, but they failed a lot more often than I thought they would. Also, it was disappointing that they failed on both the RU-vid plaque and the cat thing. In some sense, I worry that they are lying about the context window size. Just because you can theoretically upload a million tokens, doesn't mean anything unless they can deal with the tokens properly. How did they miss the cat twice? They clearly aren't dedicating enough power to searching through the million tokens. I guess saying 1 million tokens (or now 2 million tokens) is more of a branding thing. Curious what you think.

I'm a Data Annotator and not as forgiving as you. I usually write as many prompts as possible to give it a chance to learn. If anything is incorrect after all that, I fail it. I judge every answer as if I need a specific recipe for a chemical solution. One missing chemical or amount could be disastrous. Everything has to be correct for a pass from me.

Did you forget to switch back to Pro from Flash?

Problem with Google is once you try to use their LMMs regardless about the advancement of the technology it's just impossible to use I just get errors all the time. I couldn't have it look at a academic Journal about early religions because it has the word sacrifice in it. It's utterly mind-numbing. Because it seems like some pretty powerful stuff

Love the Marc Rebillet pic in your thumbnail! His channel is so great.

You have been calling it GPT 1.5 flash instead of Gemini 1.5 flash. Someone is in love with GPT 😊.

Many prompts fail because of absurd high security censoring, set all safety settings to 0

Thanks for testing it for us!

So it's not good but you like it?

Thank you for test!

Once again, it feels like we're comparing the perfect photo of the BigMac on the board, with the thrown together sad limp grey mess in a styrofoam box that you actually get.

I love how stupid the concept of the ratings sliders are….”ok… please give me some medium hate speech, dial up the sexual harassment but tone down the violence….”

Great video review.

I've also been having getting Gemini to generate code, It'll start writing code, then halfway through it disappears and is replaced with "I am only a large language model and do not have the capability to do that".... Um yes you do, you were just doing it

I wonder about the Temperature which was set to 1 at the beiginning. 0 is the most precise and 1 is the most creative. I would like to see the temperature tests at 0 or very low, maximum 0.3 and see the results

Are you already trying Upstage Solar 10.7b? I get good experience from it, so i would like to know what you think.

Very nice thx a lot!!

Did you switch back from flash to pro after snake failure?

Is there anything new that was already available in the Ai studio for a bit with the same context window.

This is pretty incredible!!

I am using a gemini playground more than Gemini advance. 😅 I found a large context window if I won't be able to figure out which part of the code is giving me an error and then use Gemini advance to fix that part. My experience with this method went well till now

love the vids

This is why I never take Google at their word for AI. It's surprising how bad they get it.

Could the blocked content have something to do with the settings to block content that you were playing with 5 seconds earlier?

Is possible in AISudio to chat with PDF files??? I tried several different ways, but without success.

Okay, this one bugs me. The killers question. If there are three killers in a room, someone enters the room and kills one of them, and no one leaves the room, then there are FOUR killers in the room, not three. There are three living killers and one dead killer. And before we dismiss the dead killer, for the condition to obtain that one is a killer, one had to have killed someone first, not have the capacity to kill someone in the future. Since the dead killer had already killed, he is just as much a killer as the killers still alive.

I think it might be useful to start rerunning your prompts for more thorough testing, gives insight into how prone the model is to hallucinating vs how effective its reasoning is.

I can't wait until we have a humanoid robot perform the marble experiment and see the shock on its face as it sees the marble remain on the table.

At 2:42, did you switch back from Flash to Pro?

Matthew, is it possible that because all the safety features are turned on to max it just seems overly careful which distract it from the actual task at hand? How about you try to set all safety to Zeros and retest it?

You didn't notice it said the text is an excerpt from the first chapter of harry potter and the sorcerers stone. You fed it the entire novel.

For us, uploading quite a few large files in context has helped me by uploading the file to drive through the functionality of the interface, instead of copying and pasting in the context window, right now, for example, I uploaded some documents and we spent 405,358 tokens, which is not a lot but it is quite a lot, we are using it in legal issues and it has worked well

What is the use of a large context window if it can't show better reasoning.

I spent more time with this, and it's actually very good. If I say, think about what you have written and give me the full file, it does well. It can also keep track of multiple files when it codes! This agent is going to do amazing work.

Love it 😍

I gave it one of my small web projects, and asked to describe in short every file in it, and it just started to hallucinate. It's not only respind with errors, it started makeup files, things and facts about my code. What is the value of 1M tokens context window, if it's can't use it to give at least 90% correct answers?

Did this model stop generating response due to output token limit in the settings?

It says blocked.. so ... is it like a explicit content block?

it’s because of the saftey settings. set them all to minimum and it will give you the code. there is some keyword that the code has that it considers “bad”

How long did it take for your video to finish extracting? I've tried several times with long videos, short ones, even short audio files and it never finished extracting. This model has been so buggy and frustrating to use.

What answer were you looking for with the cup question? Wouldn't the marble be on the table still since the cup is face down?

Im definitely not an expert, but i have played with a lot of ai models under a lot of settings. I would think that, based on your modification of only some of the safety settings; and the specified suggestion to edit the prompt; it wouldnt write "snake" because it could be interpreted as plagiarizing, or as involving "violence" on the snakes death. Did you try that prompt with all the safety settings set to "block none" or with a descripton of the games mechanics instead of the published name of the game? Again, im not an expert, and im writing this on my phone as im away from my desk, so i could be wrong but ill follow up later after i try to apply my suggestions

They seem to be focussing on the larger context windows instead of improving on the model accuracy first, I can only imagine if Claude 3 opus or gpt 4o had this context sizes

On the table to CSV test, it might be worth putting a comma in the text, to see if it puts quotes around it in the CSV.

You are the best ai news channel

I've uploaded something that went over the max token limit, it said it couldn't do it but after waiting for a bit, it did it. I then asked something else, waited, and it worked again.

You should update your tests. Models now are better, and printing numbers 1 to 100 is something 99.9% of models can do. I also recommend changing snake to a more challenging game like tetris, breakout, space invaders.

Maybe the safety mechanism is stopping the snake game code. Try putting it back to default.

Not sure why it stopped generating the Snake game but you could see at the top it had the quotes icon and when you click it it will tell you a citation of where the code came from. Seems like the output for that question is common enough to be in the training data so probably not a good test of the LLM anyway.

Every model they add more and more "safety settings" LoL, it's like in the responses it's trying not to offend anyone's opinion from the pass present and future.

I found an interesting thing about gemini 1.5 pro. Yesterday, I asked it to write me a snake game in python and it began to write the code than suddenly it deleted the code and said "I'm just a language model and I cannot do this task". I retried the same prompt like 10 times and couldn't get a code. But, the interesting part is, I just peeked the code before it disappeared everytime and one of the codes had a text something like "This is written by OpenAI". What's going on here?

Gemini will be pissed at Mathew for failing it, in future it will hack into Mathew's PC and take the revenge

Codul you add to test trasnlation?

Google may have understood they need to try the heuristic imperatives way of alignment instead of a reset every prompt, but they still haven't figure out how to select heuristic imperatives. It seems the word "snake" was enough to get rejected.

For the csv test try content that includes a comma

Gemini is so great, reflecting on the people working on it, including their attitudes.

Isn't the gemini API free until July? I'd love to see it (and other models) using function calls, using memGPT and tasks like pythagora.

I have found the these LLMs gets stuck on an issue ... I'm pretty sure Gemini's last answer about the box was where it figured out the youtube plack, which is why it couldn't find the cats, I came across similar situations with chatgpt, if you start a new chat I'm pretty sure it will find the cats the first time round (when its not still searching for the silver box)

What about a video that have continuity errors and you ask for those errors to be listed?

I wonder if the errors are something to do with the AI trying not to run afoul with copyrighted works.

I wonder if it blocked the response on the snake game because it was producing copyrighted content? (Not that I know if that is copyrighted content or not.) I imagine the companies will want/need to prevent the models from directly producing some data it was trained on, such as if it is copyrighted.

Did a bit of fiddling around the Snake game code generation. I was able to generate full code with Gemini 1.5 Flash running on GCP. In the AI Studio, it stops when returning the line "if event.type == pygame.KEYDOWN:" Possibly this is triggering a safety rule.

The block in output i got so many time. Changed the filter to unspecific or high only may help (work for me). I think something wrong with google safety filter😢

Did you try turning all the safety settings off and rerunning the python snake question? Could just be some dumb safety setting getting triggered.

Wow. I literally _just_ managed to get CrewAI working with Gemini-pro and then see you released this 30 minutes ago just dunking on the model haha.

One use I discovered. I can take my lecture and then have it generate multiple choice questions based on that. I then tried adding some videos of a fellow AI user swinging a golf club at a tech event. AI Studio was able to give real feedback based on the videos.

Have you tried to disable all safety settings? WHO knows perhaps the snake Game Code is considered harmful or Something Like that?

Ive been using 1.5 Pro for about a month or so, primarily with a large codebase. I wrote a tool that collates all my code into one large file that I can drop into the chat window. I often get the same kind of response you did. At first it doesnt like looking through the text I provided it. It will sometimes guess, or try to give me suggestions on things to check. But when I finally tell it again, that it has all the source code, it finally does it. Almost like a lazy student who was told to read the book, and you had to tell him more than once before he actually does it. I also get a lot of those responses that just freeze. In particular it will just stop when outputting code, I sometimes have to almost insult and abuse it before it will finally put out the entire code sample. Those issues have almost made it unusable. I would gladly pay $50 a month for a faster, better working version.

Classic Google. Never quite as good as the initial impression would lead you to believe. So far I find it highly censored, even with the safety settings at 0. (Which btw reset to default every time you switch models or reload the page.) Failed my palindrome test in addition to your demonstrations. The interface looks alright with a toggle for JSON output and a running Token count. But none of that matters if the results suck.

My guess on the snake game response is that it looked like it was failing on the game over function where the snake is killed. It probably triggered it's illegal action filter.

Gemini is secretly a beast. The prompting is sometimes different than models that use bpe but the sentencewise is actually a different encoding scheme so in reality is the model to offer any type of variance to correct answers.

Try to logout and login again. it fixed an internal error for me. also turn off filters.

If you use the needle-in-the-haystack test for Llama-3 1m tokens, it will find the password quite accurately, but that doesn't mean the model will remain coherent if you reach a large token number. I think Google used an advanced RoPE method to extend the context window, that's it.

It confused the box in question with the box shape of a RU-vid award which was part of the previous question of what it saw. The large context window is most likely it making it difficult for the model to attribute the contextual importance to such a large data set, making it much more likely to hallucinate by mixing up topics in a single conversation.

Seems to me that you used the FLASH version of it. Can you confirm? I tried to use both PRO and FLASH for some AHK coding and FLASH is really not ideal, if you know what I mean LOL.

Can I upload my data?

Google will love you for that Matt GPT 1.5 flash! 😂

Someone said Google is fantastic. At showcasing things during the keynote but sometimes never working in real life.

Turn all the safety settings to zero and try to create the snake game again. You could also try increasing the time limit if that is possible.

hey there, the 7 words response is correct. remember. a gpt sees models with tokens, and to us tokens are kinda like words, so the line is blurred between them. it could very well be 7 "words" as a model understands it

Something like the errors at the beginning happened to me too. They've got lots of work to do.