What is an LLM Router? 

You could probably modify the code as well so that you have both a "debug" context and a "production" context so that cheaper LLMs could be used when the final output doesn't require the most expensive tokens.

@toadlguy Thanks. You have pointed me to solutions several times. We were trying to figure out how to get prompt plus user choice data. There are plenty of sites that run your prompt against 2 LLMs and ask which you prefer. That is some really valuable data... Now I don't need to - and that would have blocked me. My current algorithm involves looking at the size and complexity of the prompt. Most of my prompting is highly structured and templatized - I expect LLMRouter will pick Chat 4o most of the time. But later today I will find out :) Now I suspect need to find a way to generalize the use of tokenizers and different embeddings and such... Do you have a quick and easy solution to this, too? Thanks for all the help.

@toadlguy It was a nice thought... The application I am making uses prompts that contain RDF - state/node/conditional edge... names, descriptions, pass messages, fail messages... The prompt then acts as a state engine outputting the proper messaging based on the non-templatized stuff passed in. This is ChatGPT 4o all the way.

@JohnBoen do post how it went, for you test it

@anantgupta3285 Sure. I get different results - undesired results - when I try using lesser models. My initial thoughts: * Templatized agent prompts work differently in different LLMs. * It would be quite a bit of work to revise things so I could dynamically select the LLM - this is not something I considered when I started playing with LangChain... Not gonna work for this project. However - I am writing an auto-tester to execute my standard test suite against new LLMs...I will plug it into that... However... ChatGPT 4o $20 plan supports 8k tokens. I assume 750 words for 1000 tokens... 5600 words in the context window. Realistically this limits my story engine to about 10 275-word pages of text. A couple of chapters before it loses track of the story metadata and rules... A team account would get me 32k tokens perhaps 50 pages... I think I might just need to use Google's million token context window...

Maybe it can help you iterate faster. Help you manually test your prompt + models quicker and see which of the cheapest models are suitable for each task.

@hqcart1 Initial thoughts... After poking around with example prompts... Highly structured and templatized prompts seem to always suggest frontier models. Not too useful for me either.

@@tvwithtiffani no, all it does is add another layer of complexity and latency to a judge (route LLM) that is weak in nature, so it will guess which model to use based on unknown criteria. a manual route is way better without complex systems and reliable results.

@@hqcart1 oh well don't use it then 🤷🏾‍♀️and it's not unknown criteria. This video said it uses its llm arena data.

@@tvwithtiffani exactky, thats the unknown

The router would be a very small and fast model. The cheaper model would also be a smaller model. Since cheaper models are smaller, they respond much faster. Latency would only go up a tiny bit for the responses that get routed to the most expensive model. Overall, you'd see a massive improvement in latency. Consider cost to be a proxy for compute time spent. They say they save 85% of the cost while maintaining 95% of the benchmark score. So estimate a 85% latency reduction (not counting the fixed networking latency). This doesn't actually play out exactly as expensive models are more parallel as well but you get the idea.

I would think so. Perhaps that’s another parameter to be optimised for: latency required for the type of query. Eg if conversational voice tokens input would route to a low latency model like upcoming 4o voice mode.

Haha, nice man, yeah I’ve been doing the same thing too, I also use the cheap guys for filtering. It’s very cool to bump into someone else who does the same thing!

This is a good question. It may do ok out of the box, but you could certainly train one of their models to handle other languages.

They used their own data from the arena to produce the framework. So theoretically it should be more versed in which queries are ok for each of the Models you select to MoE(xperts)/MoA(gents) with.

@@tvwithtiffani so same concept just a bit more polished :)

Am I right that this could be combined with MoA to enable you to then optimise for cost / performance and accuracy?

@@BorisHrzenjak I think so. More like mixture of models tho because Agents and experts are typically preset with a specific task or set of tasks for agents and a specific set of preset knowledge for each the Experts in a mixture.

​@@BorisHrzenjakSambaNova call it CoE I guess

This is different than LiteLLM it is dynamically changing between the two model choices

I linked their GitHub with the code etc. in the description

Yeah I do feel like that about a bunch of these things. This I would look at as more of a proxy you go through.

Not very accurate I have built a tool for my org that works on a custom built sql agent that could use a cheap model for over half of the questions being asked. I am building a "router" to check complexity and context need.

the good thing is it out there an open sourced so others can work on improving it for lots of use cases.

😢

Clearly not for (most) of production cases. But it be useful in dev as a heuristic