LLMs are already so cheap its basically free like a few pennies per million tokens for most open source models and even the best of the best proprietary models cost like 3$ per million tokens
Couldn't agree more on usefulness of chatbots. RAG is awesome features. But with the growing size of context window for recent LLM (Mistral-NeMo has a window of 128k tokens for example), RAG isn't that useful now. It greatly depends of the size of your knowledge database
there's actually a new paper about using both RAG and context window, by routing queries to RAG or long context depending on self-reflection, p cool u can check it out arxiv.org/abs/2407.16833
No. RAG is still very needed for LLMs to produce practical output for some practical use cases. 128k context window is much SMALLER than you think. Sure, it can fit around a 50-150 page document depending on the content, but some use cases easily surpass that 128k context window. And I wouldn't just jam every context information I have inside the context window, even though for some cases it can easily fit all of them. Pre-filtering context information to pass unto the LLM is a great practice in general so that the model can have very relatively sharp and concise results. You have to remember that every token you pass in the context window actually has an affect on the output's probability distribution. Having a pretty well defined and cookie-cutter path for the desired text output's probability distribution is very, very important.
@@macchiato_1881 100%. Quality goes down in new tokens the further down the context window you are. Also LLMs have the similar memory issues as humans, where they remember the beginning and end the best, with gaps in the middle. Maybe the big context window will still work for needle-in-a-haystack, but for quality new tokens e.g. code? Not so much.
I think RAG could be used to get rid of hallucinations. Like having all of the Wikipedia (just for example, probably not the best source of information) pages inside of a knowledge base and giving factual answers only after getting documents from the base. There is no way you could fit the whole wikipedia into any model's context.
RAG or GRAG or some iteration of it is still beating every other development pattern on every benchmark. topic/query specific enriched context is key for getting the most out of LLMs responses for a lot of use cases where you want less stochastic results.
I dont think LLMs are going to replace RAG. RAGs are going to be a long term solution for context retrieval. Even if the context window is large enough for processing thousends of books, it would still be expensive and the LLM looses precission with growing input. The LLM should be able to focus on its specific task and not be overlorded with lots of expensive context. I wouldn't also call it a "hacky" way, its just another type of database.
Well the end goal would be to get rid of RAG and do it natively. That's the long term solution. All the things you mentioned are presumptions based on the limitations of current transformer architectures. Lost in the middle has already been pretty much solved, and if we find an architecture that scales linearly or at least subquadratically, then big context wouldn't be so expensive either. This is what all the mamba hype was about
@@Askejm Your brain does not do it. I need to think pretty long sometimes to remember stuff. That means there is a multi agentic framework running in my brain searching for information. I dont have more than 500 tokens in my short-term-context when I am talking, so current llms are already surpassing that part. Maybe IF it scales well there might be a chance to use one LLM for it. But for solving tasks you still want to use smaller context windows cause the context would be full of unnecessary stuff otherwise. I think there is a high chance that would reduce the llms task solving ability.
@@HanzDavid96 Your brain is not comparable to a computer, though. Your brain runs at like, 100-200 Hz, and comparing that to a H100 is just not possible. That is fundamentally different. Also, with a good architecture the AI wouldn't care if its context is cluttered. That's a human presumption. It would just be able to use the information it needs, which they already do in some capacity. I definitely think the next paradigm shift in LLMs will make RAG largely obsolete, but there's no telling when it will happen.
LLMs are like cars, if it stands in the middle of the deep forest we can point at it and laugh at how it's stupid and how it's better to just walk through the forest. RAG and tools (as in tool-calling for llms) are the infrastructure comparable to roads. Many people don't realize that once the "car" gets on the proper "road", it is all of sudden very efficient at what it does. We don't faster cars (e.g. GPT-5), infrastructure is all we need right now.
at this point we are creating diffrent parts of a brain. this is littrally how our brain works. amazing! keep up the content you now alot about the topic and i can really find out what is the latest hot news
But is it though? The conceptual model might be similar, but the model is not the brain. I'm pretty sure that actual brain memory works in a complicated way (i.e. short term and long term, forgetting, transfer between the memory types, the actual retrieval is unlikely to be just vector lookup and so on).
@@xomiachunaThe actual brain is much weirder. Every nerve in the path from a limb to the brain does some data processing. Every transfer of information from one part to another interferes into everything in the middle.
It's not how our brain works. For "sampling" large scale information, the brain has all kinds of complicated electromagnetic synchronisation patterns (alpha/beta/gamma/... waves, which may also be influenced by the form of the skull!), which are much different from how a semiconductor works.
This channel is a circejerk for people who already know enough about the topic to not need these videos. I'm a reasonably smart layman with an interest in AI and could learn nothing from this video. Too much jargon. It's the reason why this channel has so few subscribers. It should have millions, but the information is not packaged into easily understandable bits.
Esstally RAG helps LLMs have data access to whatever contextualized information you have at hand, and helps it bring more meaningful data out of it cheaper and faster.
I wish I could use AI to do something productive with out having to learn rocket surgery. This sounds interesting but way beyond a layman's understanding of chat AIs.
I think the future of llms will be a system where an llm primarily traverses a rag like systems and uses thag to build rules, it keeps building rules until all requirements are fullfilled and then sends back the result. Models like these would be poor ceratives but insanly factual and logistically strong
I coulddd and there are so much more I wanted to talk about too, but I think it'll get too long for a "conceptual" video about RAG... thanks for letting me know tho
so LLMs will rewrite complex practical questions into simplified, general form... Losing the entire complex question in the process... Seriously, what's the practical use case of this? AI already just spouts nonsense about generalist topics, this just removes the ability to ask specialized questions.
Seems like you missed a ton of the video? The rewriting stage is just potential adjustment that can be added in one stage (the indexing stage) that would only be introduced if the person making the model wanted it. As one example, maybe you know your application is served to users that don’t phrase questions well. The overall point of RAG is to help the LLM to be LESS of a generalist by introducing contextual knowledge to draw on. Optimizing the indexing stage is just one potential way to improve this specialization
As I understand, simplified form of a question would be used just in a document searching stage. In the end, LLM will have both retrieved (by using simplified query) context and the question in an original form. But that simplified question may just work better with vector search in knowledge base.
It’s funny, I just got fed up with coding on my research which is in optimizing RAG lol. I must say hugginface’s retriever which is BERT based is great, but for the LLM I am using Mistral-7B. Combining these components together and doing end-to-end fine tuning is a challenge! I wish there proper containers which would just fit in. Although in my case I am creating k prompts to pass through mistral and then marginalizing the output over the documents. This sort of way to backdrop is somewhat contrived but it seems to work haha. Let’s see where my shit goes. Kinda lost right now
