RouteLLM Tutorial - GPT4o Quality but 80% CHEAPER (More Important Than Anyone Realizes) 

I'm putting the final touches on my business idea presentation that I'm going to give away, which is partially inspired by RouteLLM. Can you guess what it is? 🤔 Subscribe to my newsletter for your chance to win the Asus Vivobook Copilot+ PC: gleam.io/H4TdG/asus-vivobook-copilot-pc (North America only)

💥 Matthew Berman is so clear and concise. He has that natural talent for explaining things in a way that everybody understand, with emphasis on all phrases, clear diction, intonation, that hooks the listener. People like David Shapiro, Matt Wolf, Matthew Berman, who speaks only the necessary, and value every phrase. This is great. 🎉❤❤❤

WOW. This is crazy. I had a similar idea last week and mentioned to my team this morning 10AM UK time. I’ve just seen this video and it blows my mind that our thoughts were so aligned. And yes. I’m not sure if people really got it together, and Agentic on device system + Routing LLM technology is the future. I believe models will become smaller and smaller. Crazy how this part of tech is advancing so fast. It is as scary as it’s exiting. I learn a lot from your videos that forces me to read papers and try some of the tech for myself. I really appreciate your content.

Is the context preserved across models for subsequent queries? E.g. Where is country x? - goes to the weak model What is good to eat there?- goes to the strong model Does the strong model have the context of country X?

Awesome video, with GPT4o mini though idk how much I'll be routing lol

Matt: you need to discuss how the router logic works. I am nervous that it will not be smart enough for my niche business use cases

I can see this being useful in scenarios where you have local Phi-3 models, then escalating to GPT4-Mini. If there was some way of declaring what each SLM is best suited for, then RouteLLM could kick it to the expert that knows best. Great video, as always Matthew! Cheers from Australia!

Thank you sir for your videos, your clear explanations and the amount of tools you give us visibility to. Please don't stop. God bless!!

We were already working on the business idea you mention. I assume more will be now too. just makes sense. Many clients can't afford to just pay $20/user/month for everyone in their enterprise when they don't really know if there will be value yet (ROI). The challenge with RouteLLM is that it doesn't quite do everything you suggest because you only get weak and strong. So local.. or 4o... What if I want Local for really private, and then 4o-mini for simple but the llama3.1 might struggle? or I want a poem, and Anthropic is better. The model limit is the challenge - great solution to start saving $$, but right now its like 4o and 40-mini are the only logical choices for an enterprise and I'd suggest a full solution needs a little more subtle routing options.. Great stuff.

I did not expect llamas to be so important in my life, a person I know has a llama as a pet too

Hi, thanks for the great video. I love watching them. The idea about RouteLLM is great, but I would say it lacks control. The control needs to be reliable and avoid hallucinations, which is a huge problem with LLMs. For me, it is not clear about the criteria. On what basis is he trying to route ? For production, I think it is better to develop an agent whose task is to understand the requirements and, with some instructions or even fine tunning some LLM model, actually perform the routing. LLMs are not ready for general-purpose use. If you need an AI agent to perform a specific job, then make sure you have a very good specification, good prompt engineering, use function calling, and do a lot of testing. I am using the LamaIndex framework, and I want to do the same LLM routing thing, but using my own agent or agents, which should provide me with more control over verification and decision making. I did some some agents with Lamaindex and it is working OK so far. I can mix models and for output verification, using always top tier LLM model.

Sam Altman already mentioned that, in addition to trying to improve the reasoning ability of the models, another aspect they were working on was a mechanism capable of discerning the complexity of a task to allocate more or fewer resources accordingly. Currently, LLMs always use all resources regardless of the task, which is wasteful.

Great video! With GPT4o mini idk how much I'll be routing though lol

This could work really well with locally run models, especially if using specialised models for given tasks, having multiple models, each one good in their own respected area and much lighter on the hardware because of not being a jack of all trade and could potentially be a game changer for running much stronger A.I. on our local hardware, especially if there is a way to have it where it does a good job at picking the right model for the given task you ask of it and honestly, storage space is a lot cheaper than vram is, so if A.I. models can be switched in and out of memory on the fly, having a lot of A.I. models on your hard drive that are good in their own area isn't going to be that big of a deal but could boost the quality of A.I. massively at a local level while not needing a crazy amount of vram. Mind you, all this only works if A.I. models can be switched in and out of memory on the fly and do so quite quickly that from an end user, it all seems like the same A.I. model, this also would only work if you have a master general A.I. model that can do well at delegating to other specialised A.I. models, after all, having a few hundred GB's of A.I. models isn't going to be that big of a deal with how cheap storage is, and it sure would be a lot cheaper and faster to run if they can be switched in and out on the fly.

Matthew, can you do a deep dive on how to choose hardware for home use? Can we use Linux? And also, what hardware choice might we make if we wait? Should we wait? Should we buy now to avoid a signed GPU? Etc.

That is a task not a prompt 😮it is useful for complex tasks, setting up a system prompt and classifying the task level. Great video 👏

I was actually thinking to use it the other way around. Route LLM to have a strong cheap model and a weak even cheaper model and then add it to MOA together with strong frontier models to aggregate the responses so you could potentially build something that is slightly better than gpt4-o while still reducing your cost a bit. Another step would be to also chain it together with memgpt to have a longterm memory. And then use the endpoint in aider as the ultimate coding assistant 😅

Matt, do you know how this (or your "business idea") will deal with context? Most chat uses for LLMs involve a conversation. I'm not sure with this sort of simplistic routing that the only way to deal with context is to keep using the model the conversation has started with, although you could keep a separate running context that you pass back to whichever model is being used - I just don't know how well that would work, though. BTW, your "business idea" is what many people are currently working on (or something similar), even some LLM providers may be doing this behind the scenes. There is a distinct lag and additional token cost with more complex queries on GPT-4o mini that may also suggest a similar approach.

did you know that gpt-4o and mini can run python code? I have tried it and it worked, but only with python, hopefully the other LLM will soon follow with this ability and maybe add more programming languages that can be run directly in the prompt.

you dont need to run llama3 with ollama in terminal, ollama app took over the 11434 port already, as long as you invoke local models with the matching model name, ollama will do the work

If Llama 3.1 is looking at "9.11" > "9.9" as a String data type and not as a Float data type, 9.11 is larger because it contains more characters.

Another excellent video. I wonder if this can judge multiple models, and if not, when they might add it. You might want to choose between a smart expensive model, a fast model, a cheap model, and several specialized fine-tuned models. I've tried to do my own LLM routing in my agents, but mine is naive compared to this thing. Mine just asks the weaker LLM which LLM should do a task prompt, and it's often wrong, even when I supply a bunch of multi-shot examples.

🙌 How nice would be if gpt4 had the speed of groq. Thanks for the video Matthew

Thank you, I'll try this out soon! ❤

hopping langchain to implement this...because if frameworks (for agents) don't support this, it will be hard for developers to implement it

Thank you! Matthew. I like your video very much.

another game change project, excited for the next video

Wow Amazing...thank you Matthew :)

How does it decide which model to go for. Knowing the mechanism would be useful. Also possibly having more than 2 models would be great. For example if I tailor a sales message, it’s possible to do it with a 7B model but with a 70B model as far as I experienced it’s much better. I guess everyone needs to test if for their usecases. Overall if the selecting mechanism works correctly it’s great tech and highly relevant. (I don’t need gpt4o for example to tell me that with rstrip(„/„) I remove the last /. I’m sure mixtral etc can do that) If it goes wrong just in few times in production, where people are in an interaction with the bot, then it would not be worth it since the quality of the product cannot be guaranteed. Anyways, it’s all in development, thanks for the video!!

That's the way to go! Great and thanks for sharing!

Really great stuff, Matthew - thank you! I noticed the prompt was to keep the cost of the call to the MF router api down to 11.5c - does this mean the router llm uses tokens per cost or does that run locally?

Matt, you need to get with Groq to improve their process. They MUST get rid of the just make things up behavior when it doesn't have data. That is REALLY annoying. I have to ask it now - did you make that up?

Great getting started thanks. But how to instruct Route LLM the conditions for choosing the strong or weak model ?

By the way, why do this at all, I simply did ICL, with llama3:8b, and it gives me {key:value} in response whether the query requires complex workload or lite workload and then I case switch. Simple, now I have removed llama3:8b to gpt-4o-mini. We need to get over this increase security crap, all these apis provided by large companies are safe and you have written guarantee.

Thanks!

Thanks a lot Matthew 😮❤

Ok. Matt… I can sense your excitement, and recognize your potential. I am grateful for your work so I will return the favour. “Don’t let your hard work become a fundamental lesson in business foolishness” Take a breath before speaking openly. Seek wise and trustworthy leadership and counsel. I could go on. I don’t have to. Good luck Jeremy

I tried using GPT-4o to solve a seemingly simple problem, but it failed. I'm not the greatest prompt engineer so the failure might be on my part, but I spent several hours trying to refine the prompt and never solved the problem. 👉Question: If GPT4o can't solve it, is there much point in spending hours going from one to another trying to find an AI that can do it? Or is it more like: "If GPT-4o can't, no one can!"? The problem, in case you are interested: My problem is to translate the shape of a object (e.g., a boardgame miniature) into a string of numbers that represent the shape and then characterize other properties of the miniature (e.g., height, width, distribution of colors...). Procedure: I upload a photo of a miniature laying on graph paper. Each cell of the graph paper is numbered. The AI's job is to determine which cells were covered by the object, then list those cell numbers. That list would be the numerical representation of the shape. GPT-4o cannot consistently give correct answers for different miniatures. Perhaps this problem requires too much genuine understanding of visual data and, thus, I may need to wait until we have something closer to actual AGI? Or is there some AI that is better at handing the "meaning" of visual data better than GPT-4o?

There are times when I find the answer of 3.5 better than 4o. Because 4o sometimes tries too hard to be helpful and it gets annoying. Also it seems 4o can’t just give part of an answer… It has to repeat the complete answer.

Great video and love your great businesses idea. Thanks a lot! Does routeLLM support visual input or only text input?

No, you made a mistake, Apple wont route it to chaTGPT they will route it to a bigger version of the same model in same Silicon slice in the cloud. ChatGPT is only used if you want to use it.

I've been doing something similar with an 7B model as front end, if it can't answer the question, it forwarded it to a few bigger models, and then cook a response with all the answers, I call it TPMFMOEX because it's catchy (the poor man's fake mixture of experts). Thanks for your videos!

My new LLM stack: AIOS, LLM Router, Mixture of Agents, Agency swarm, Tool Chest, Open Agents. I am working on improvements. Also, a reason and logic engine, a long-term memory engine, and a continuous learning engine

Is it possible to use LM studio instead of ollama? Would be nice to see a tutorial how to set it up with cursor.

Great video!

I've actually tried this for a Production use-case and it did not work as expected. Simple questions as "What is the name of our Galaxy" were routed to GPT4o. Also there is a delay problem here because it takes a while for the RouteLLM model to run and decide the route.

How can I use this with an AI coding assistant in VS Code? Specifically that the ai assistant will use local llm for most tasks and cloud llm for higher difficulty.

A model that feeds a model that feeds a model that feeds yet another model. What could go wrong.

Suppose I was using something like Perplexica and OAI and a local LLM like Llama3, what API would it call to at that point? Hypothetically speaking, of course. Also cool delivery.

Not sure how useful this really is. When you're writing production agents, you need to send appropriate prompts to the weak/strong models to make sure you get consistency.

Say If AI LLMs are like workers, then why would a company hire a PHD in "everything" for some simple task? When you think about it, this is how human organizations work.

okay, this is fine for single questions, but what about context?

I dont have an High end CPU nor GPU, so I am not sure if going local is doable for me, I am always confuse looking at what I need to go local, would be nice to have a video that would clarify a bit what you need as a minimum gear to run an acceptable AI. What should people aim in term of PC to enjoy local AI? I know things change fast, can we hope to have models so small one day to run on an old 2010 gear???

I wanr the best not the cheapest, therefore better if you show how to integrate gpt-4o(mini) or claude in to mixture of agents as main llm. MoA is really a fantastic framework. I am sorry but i don't see any benefit in this routellm for me. If I can use groq or good open source .models, i don't really want to save any cost there..we need the best of best...therefore improvement in MoA would be nice like custom system prompt or ability to use top frontier models

Mate great content.. Any LLM or platform that I can point it to or give it a website to analyze its content with some specifics mentioned in the prompt, instead of me compiling the whole website copy into some document first (tedious) as a prompt?

Does this support which llms can be experts of what? If not, I can only see it being useful if I had a very expensive model and a very cheap model.

Any idea why they use Python3.11 and not the lastest version (Python 3.12)?

thankyou

Have you seen leaked tests? Seems llama 3.1 70b is better than gpt-o and small llama 3.1 8b is smarter than "old" llama 3 70b! Insane.

How is it compared to GPT4o-mini?

Does this work within agent work flows, where the general instruction set is extensive, even though the specific instance of the task may not be complex

Thanks!!!

the best use case for me would be if I can use it purely locally and have a small model like llama3 for most of the tasks and a vision llm. so it needs to know what choose. can this be used for that? thanks

Do a video on Mistral NeMo

Would it work for rag use cases, where the prompt can be way larger?

This seems like a hack. The LLM itself should be able to do this internally eventually. Perhaps that is what project strawberry is about.

but what LLM is the router using?

Have you played with Fabric by Daniel Messler?

it's a bad idea, and will fail, the best way is YOU are the one who should know how to route your SPECIFIC prompts to what LLM based on intensive testing, not a random routeLLM to decide for you...

GPToMini support for RouteLLM? API calls LLMs are still to expensive. What are the hardware requirement for building a serious weak model local inference.

Does it still makes sense after 4o mini?

If the snake game didn't' get verified, then you have no idea if you got a correct answer. Right. So, let's say that it failed the snake code. Well then, the prompts answer wasn't very good, was it? Also, during the apple sentence test always give the AI a chance to correct itself. If AI is working properly it should learn from it's faults.

Loudermilk

Can it work 3 tier? If I had a phi model locally, gpt4o mini, and Claude 3.5. I think it would make for an awesome dirt cheap setup.

ok, this is massive! how can this be integrated in OUI ? make it a pseudo LLM model?

Is this not what aider does? My aider uses gpt4o-mini as the weak model

Without a shared context window this routerLLM is useless 🤷🏼

Why do you dress like that...

I had high hopes for Gemini Advanced, but it was such a letdown. Unintuitive, glitchy, and the results were just awful. Don't waste your time. #disappointed

A.I. should be almost free or free with Ads soon.

Ok . 3rd here

Openrouter Auto is faaaar better. In fact Openrouter is awesome!!!

No, you made a mistake, Apple wont route it to chaTGPT they will route it to a bigger version of the same model in same Silicon slice in the cloud. ChatGPT is only used if you want to use it.