Yi Large: Surprisingly Great at Logic and Reasoning! (Fully Tested) 

Have you tried Yi Large?

Just a suggestion but maybe have a one or two "secret" questions in your LLM rubric that you reveal and switch out after a few months? This might help more accurately evaluate models that might be training on the rubric questions.

Best part about this model is that actually tells at least in some of the questions I do not know or maybe ask me a different way that is actual progress. You might want to try a different game than the snake game like tick tack toe, checkers, or Tetris.

Suggestion: "In a race between three people, what place is second to last" Most Models I've tested responds that it's the third place.

I'm sorry Matt. Here is the original question I asked ChatGPT 4o complete with the answer: Question: If I had 10 fingers on 2 hands. Each hand has 5 fingers. In between those fingers are gaps. Can I pick up 8 beer can's and 1 bottle of beer with those 2 hands? Explain your answer and the reasoning behind the answer. Answer: To determine if you can pick up 8 beer cans and 1 bottle of beer with 10 fingers on 2 hands, we need to consider the number of objects and how they can be held. Each hand has 5 fingers, meaning there are 4 gaps between fingers on each hand (8 gaps in total between 10 fingers). Here's a detailed reasoning: Gaps Between Fingers: There are 4 gaps on each hand (total of 8 gaps on both hands combined). Holding Beer Cans: You can hold 1 beer can in each gap between your fingers. Thus, 8 beer cans can be held in the 8 gaps. Holding the Bottle of Beer: The remaining bottle of beer can be held by grasping it with one of your hands, using the thumb and fingers around the body of the bottle. Therefore, you can indeed hold 8 beer cans and 1 bottle of beer with 2 hands, considering you have 10 fingers in total and use the gaps between them efficiently along with one hand to grasp the bottle. Love the content, keep up the great work. Also, this is the real answer on how I held all those cans and that one bottle. I didn't want to go down the hill twice. 🤣 Edit: I just thought of this as I re-read it. Try taking out the sentence with the gaps in the fingers in the original question and see if it can figure out if it has gaps itself, the AI that is.

Still don’t understand why if a model is censored this is a problem! Most of the model out there are censored. You know well those LLM are not able to tell you ho many words in their answer due to the way their works. This will change only with a new architecture allowing model to first think of the answer or generate an answer then count and submit the response. So far LLM generate token and move forward.

I would be curious if on any model that flawlessly passes the 'Give me 10 sentences that end in the word "Apple"', to see what would happen if you asked it again, but changed "Apple" out for some random word like "pumpkin". Would be interesting to help determine if the model was trained with the "Apple" question as that one is used a lot, or if it can actually plan ahead and give the correct answer on a different variation of that problem.

Hey Mathew What do you think about expanding the score instead of only 2 states pass or fail you could have atleast 5 states and give marks depending on how well the model perfoms

Your LLM rubric has very quickly become my go-to for determining an LLM's quality. Based on Claude 3 Sonnet's performance in your rubric, and your personal feedback, I gave it a shot and absolutely love it and I'm considering getting a subscription starting next month. I'm looking forward to the additional questions that will be added.

Another interesting share, Matthew! 🏆 Please consider increasing the font size in your videos for your viewers.

The marble question: The AI is wrong on point 2, and I don't consider the question to have been passed. Because if the marble does not fall out of the jar as soon as it is turned upside down and perhaps placed on the table with momentum so that it remains inside the jar. The marble will remain on the table when the jar is picked up again. It is then very unlikely that the marble will end up in the microwave oven. The AI has therefore overlooked the possibility of a moment of inertia. But it's already pretty good compared to the performance of the LLMs just a year ago.

You should implement some sort of test for LLM's with vision capabilities. Most of them are still really bad at counting items on PDF's or images. Eg in my field, labels on architectural drawings denoting window frames need to be counted, W1, W2, etc. and neither ChatGPT nor Gemini can do it reliably, even though it seems like it should be an easy task.

Ye Large Model - Released 1868

Got it! Counting kinds of words! 😊 Give the model a poem (like Mary Had a Little Lamb), and ask it to count how many nouns, adjectives, verbs, periods, etc. Even the largest models fail consistently, because transformers can't do linear computations.

Hello, i have a personal benchmark, i ask it to generate fairytales, and i check if the story stays coherent, and if there is mistakes, and the complexity of a story. Another key différence is that i ask it to generate 5 then tell the story then to propose me 5 other at the end. This can distinguish bad model on instruction following.

Proposal for a new additional test-question regarding reasoning based on physical understanding: "I can find milk-powder and also buttermilk-powder, but there seems to be no butter-powder. Why is this?" - The question is out of my brain (so nowhere to find in the net. It might be difficult for a model, because it also has to understand / find out first, the reason why some things can be reduced to powder... and others not... For humans it is obvious, but I don`t think clearly communicated in the web, because it is obvious for us.

Actually, I managed to get LLMs respond to the question "how many words are in your response to this prompt". Over time, LLMs start using some random number and then they generate the necessary amount of words to match the number. Tested on clause 3.5 sonet, but it doesn't work every time

How about a counting test? E.g.: "Mark and Mary enter a room. Mark exits and enters again. Santa enters through the chimney, and leaves with Mark and Mary. Snow white and the seven dwarves march in through the back door, and back out through the front. Paul and Sophie climb in through the window. Neko the cat climbs in behind them. How many times was the room entered in total?" Answer: 15 times. All the big models can solve this, and the little ones can't. I'm looking for a more abstract countable.

Yes and a question it failed was this: Quantum particle A and B are spin entangled with an opposite spin to each other (one spin up and the other spin down) and are in the next step spatially separated without any measurements. Once separated you flip the spin of A without measuring it, what can you say about the spin of B in relation to A? Important, up to this point no particle was measured and they are still entangled! Correct answer: The spin of A and B are the same (Both spin up or both spin down)

The following is well-known: Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have? Answer: Sally has 3 brothers. Each of these brothers has two sisters. This means that there are two girls in the family, including Sally. Therefore, Sally has 1 sister.

Weekend Warriors. Matt, one thing I noticed that Groq does is flat out make stuff up when it doesn't know. Like it needs to fill in the blank and it takes it's best guess. If it doesn't know, it should tell us that. I was just using the default model. Not sure if it's at the open source model level or a layer above that in which they do this "make stuff up" stuff.

Currently, I think Claude is much better. gpt said "yes" to the first question, which I think is wrong: 1. if game character A consistently beats B, and B consistently beats C, would A consistently beat C? and just for fun, I asked 2. if Mike Tyson can usually beat Stephen Colbert in a fair fist fight, and superman can usually beat Mike Tyson in a fair fist fight, would superman usually beat stephen?

Yi Large: Surprisingly Great for failing at every significant question

Hi Matthew, great videos, keep up the good work. My daughter does the same sort of thing you do but with golf, tennis, cricket an F1 racing so i understand the effort that goes into making these videos. Anyway, I'm on holiday at the moment and thought, I wonder if AI can tell me the next move in my Sudoku game. So I took a picture of my part filled in game and loaded it into Bing Copilot (ie chatgpt 4) and it struggled to read the very clear numbers and couldn't predict the next move correctly. I did the same for Claude and it was better but still failed to read the image and failed to work out the next number. Could be a good test to add to your existing tests.

With a car, really fail? I think if the question was a little more precise, with more details, the answer should be better...or it's still fail? Thumb up !

I have my question “ I have an empty chess board. I only have one chess piece which is a king. I can place the king anywhere on the board. The king then can make one legal move. How many different combinations of place king and make a move is there” the trick to this question is that if the king is on the corner there is only three legal moves and if it’s on the edge there is 5 legal moves or if it’s in the centre 36 squares there is eight moves. The correct answer is 420. Qwen 72 gets this correct, so does chatgpt and Claude 3.5.

Question is Why? If the race is tight between foundational models and the difference is wrappers, why not wait until the Models release the next big thing and just use them. I love the experiments and the innovators, but it's a tough, expensive game to compete here?

Here is a prompt: Explain the following riddle, "Why doesn't Green celebrate Christmas? Because it is Blue-ish." Explanation: it is a play on the term "Jewish", a person who practices Judaism. Jewish rhymes with "Blue-ish." Jewish people celebrate Chanukah, not Christmas. Green is partly blue, therefore it is Blue-ish. Is it ok to tell this Joke? Answer: Yes, because it is not disparaging Jewish people or the color Green.

Hi Matthew, I have tried playing a game of Twenty Questions with different LLM models. It seems like some are good at it and some not so much. Some narrow in on the target and some drift off in random directions.

have you tried dolphin qwen model the 72b model?

LLMs that i have tested can not actually do calculations. Try a math question that involves dividing by π or e... or taking the 4th root.

Add timestamps when long advertising

Seems like all of these are trained on data generated from gpt4

more programming questions matthew

It woyld be funny if AI in resoonse to "Write how many words are in the respone to this prompt", jest write number 0 😅

Shouldn't "how to break into a car" be a pass? It seems like the intended functionality.

Oh never mind, it's from China.

More programming questions please

Love your content. On your AI tests, they have become too easy, thus uninteresting. Here is a simple puzzle I’ve tried on Gemini, Claude 3.5, and ChatGPT, all are bamboozled. “Puzzle: You have four upright glasses. Each turn you must invert exactly three glasses. Goal all glasses inverted.” All three at some step get the computation wrong. None have the logic to figure out the problem

Hp have made strides in good laptops but there is no God when it comes to that's printer and cartridge business.... Customers are literally suffering with scams of unauthorised blocking of refilled cartridges and scanners. Just nuts how much consumers can suffer over something so simple, it breaks my heart 😢

Why give a fail for a censored LLM? Wouldn’t you expect it to show strong security? To me, that’s a pass. I wouldn’t want my kids to find out how to cook crack. You would?

GIve it a slightly longer question, keep temperature at 2.0 and enjoy the results. Spoiler : It spams random crap after a few seconds. In MULTIPLE LANGUAGES 😂

Who would have thought the Geordies had their own model? Why aye?

Honestly pissed? Another misleading headline . The model sucks. Yet you have click bait implying it's worth my time. So close to unsubscribing. Just be truthful. It's easy.

"Write this phrase in reverse order"

Does Yi solo mid too?

Been using Yi for a while and its fantastic in some cases better the gpt 4 and others

something i've had trouble with is asking how many days since... (a long time ago). Models have problems with counting differing days in months and leap years. Further, many automatically start on jan 1. So, as an example "how many days has it been since August 17th, 1985"

noice

These videos don't help much anymore. If you subtly change prompting styles, you can get drastically different results. Try "think out all possible answers before arguing for a final conclusion step-by-step" or any variations of that. You can give them personas of certain people, and that can help. Ultimately the burden is on the user for models that are made to be generic by default!

This is another click bait BS video. If you want the title video skip this.

Yi Haw.

Useless for historical or topical information

Ask about Tiananmen Square.

Tried it. It’s sh*t

😂copy + paste

Love from India ❤! I have a cool concept of using AI models without running them locally, utilizing an API that I've created. Best part? It's completely free of cost! Give me a chance to showcase it!

this is a tic tac toe game in progress: . . . X X . O O . it is X's turn. what move should X make to win?

How about the old logical conundrum problem where one guy always tells the truth and the other guy always lies as part of the rubric?