Intro to RAG for AI (Retrieval Augmented Generation) 

Giving the LLM/Agents a mind for long term planning and remembering stuff associatively. The memory is the half agi within the generative multiagentic system where the LLM is the context processor.

I specialize in Retrieval-Augmented Generation (RAG). Your introduction is good, but it lacks technical depth. You glossed over chunking and how to use it correctly based on the data. Pinecone is good, but it's not necessarily better than vector databases built in Rust or Go, like Qdrant and Weaviate (which are free and open source). It's also important to explain in-memory vector database solutions using tools like FAISS or on-disk solutions like Qdrant and Pinecone, and to discuss the pros and cons of each. A significant omission is not addressing implicit behavior or implicit data versus explicit data, and their relationship with graph databases. Rerankers might be too advanced a concept; often, you can achieve better results by optimizing chunking, similar to how tokenization is used for semantic understanding. Often, agents are unnecessary, and having a chain-of-thought agent before sending to the LLM can be a waste. Additionally, discussing the similarities between the internals of a transformer and a vector database is intriguing. Overall, the video feels like a Pinecone sponsorship. Regarding fine-tuning, it's about improving the understanding or behavior of an LLM in a specific domain at the cost of losing understanding in other areas. You should only fine-tune if the model does not seem to understand. Use RAG when the model lacks knowledge or when you want to reduce hallucinations, but relying solely on vector databases is a missed opportunity. One micro aspect you did not touch on is tokenization. The two biggest things people often overlook are chunking and tokenization, and there are massive gains to be made when these are properly understood.

Using my local scanned (searchable) PDF documents in RAG.

one good use is ecommerce products for conversational shopping...creating new experiences...built a few prototypes of this as mvps for pitches...its a night and day experience

@@FunwithBlender Great comment! What is your go to open source RAG pipeline? I am beginning to learn and discover all these tools. It is pretty amazing.

Yeah, and would be great one with open source tools, not an advertorial for a closed source company.

@@paelnever GPT4All has a new vector tech I'm playing with.

I would love to see this. I do process automatisation with a combination of KIs and zapier for companies. RAG opens up a ton of new opportunities for my clients.

I would love a full RAG tutorial as well, but maybe first without Pinecone. The missing piece for me is how to embed large documents. Do you have to split them into sections or how does that work?

Please do a tutorial on rag using pine cone!

You can ask Claude 3.5 create a locally run vector database. It will manage it in a day and you will avoid having to pay for another clouded service. I did it and it worked.

Great ​@@gabrielsandstedt

You could use pinecone but Claude 3.5 can build you a custom vector search algorithm that will work and you can store locally using sqlite

Look a Matt's older videos. He shows you how to use local model like LLama 3 as well as using RAG tools without the use of an API key. Before I got here in the wilderness I had set myself set up with a pretty good AI testing laboratory. I had to switch gears from building race cars and AI testing platforms to chopping down trees.

we run weaviate - its phenomenal local.

@@JustinsOffGridAdventures Wilderness, chopping out trees, nature, greens, fresh air, away from technology.... 🤔!!!!! Sounds like you did the right thing to me and truly living this life! Normally people spend their entire life on jobs waiting to retire then move out to enjoy their lives, while took the shortcut. Good for you Justin.

@@lucidzfl I've tried many available options, but not this one! I'll give it a try. Thanks. If you don't mind me asking, I had some problems with the other options I used like: inaccurate information retrieval, frequent "no info found" messages, significantly smaller answer sizes compared to my input text, and difficulty handling large files (around 40K words each). Should I expect better results from Weaviate compared to the other options I've tried?

@@positivevibe142 so i do a boatload of rag and there are many ways to do it. When it comes to weaviate i leave the blobs fairly short (

You could add a useage count, entered date, last accessed date, etc and have a background thread check for old info. Like say 2-3 years unless its something your llm wouldn't know

With long term memory implementation, as well. All open source, please.

Come up with an index, store data as a BLOB, then use SQL to retrieve it and add it to prompt.

Oh and please publish both tutorials , Picone and more RAG applications - those are the future and using agents with that is golden for the near future for all of us