Gemma 2 - Local RAG with Ollama and LangChain 

AutoGen is cool. I will make some videos about it at some point. I often use LangChain because it is easy and quick. I do like LlamaIndex as well and need to find some time to make vids about it as well.

Interesting I tried with and with out the prompt format and both seemed ok. I will try it more with some other ideas this weekend .

@@samwitteveenai good to know! Personally for RAG applications I'm always unsure on using a chat template or not. Typically, I'm setting the task and data to look at as system prompt, and then have the standard user/assistant roles following the template. But then, Gemma models don't support the system role at all so... 😅

For Ollama I am not sure if it operates different when you call it as an API. I need to look into. I saw the Ollama people were in the early access group with me so I figured they would have figured it out, but everything was very last minute for all of us this time round, so they may have gotten quant versions much later.

@@samwitteveenai Ollama has chat and generate API endpoints. For the chat endpoint you're expected to pass a list of messages with roles, and that'll be formatted according to the model's chat template. For the generate endpoint, it's just text that'll be passed to the LLM. Which one will answer your questions best tho, is worth looking into 😃

And the generate has an option raw that when you set it to true it does not further apply any template. It takes the input as is. Useful when you want to apply the template by yourself.

Actually I made the Mesopotamia version but the streaming wasn’t working with their Chat UI. I need to look into it more.

Yes. Dark mode in the whole video, please.

Dark mode has better contrast for many visually impaired viewers. Dark mode please.

You don’t need any matching between the embedding model and the text generation model. They are both handling separate parts of the process. The embeddings generated by the embedder are never given to the TextGen LLM.they are used with the vector store and retrieval. I am using a MacMini with 32GB of Ram so not sure how well it work with low Ram etc.

@@samwitteveenai Yes, one of the neat things about RAG in general is that the interface is "English" (or any language the LLM supports) 😂

i've thought it is possible to switch it off completely. There is a big choice of vector or hybrid local databases these days, including postgres and mongo.

I didn’t know about the telemetry on Chroma will look into. Thanks for letting me know

I wonder if you could strip out the telemetry data with a prompt on the LLM?

Which Vectordb do you suggest?

@@stanTrX I was hoping to get a better one suggested. I would prefer to use MariaDB, as I have it as a relational DB anyway on my server. Though, I've never seen any RAG examples with it.

I would also like to know this.

I was using a M2 MacMini with 32gb of RAM. It works on my MBA though its slower there.

Great question. Yes every LLM or more specifically every Fine tune of a LLM has a structure that the training data was given to it for the training. When we want to do inference we have to match that structure to to get the best results. In the past if we didn't match the structure we would get garbage out, nowadays the models are getting so good even when we don't match it they can return decent results. The structure normally has some special tokens that tell the model things like was this line said by the user or the assistant, when a section starts or ends. This structure can be very different from model to model especially if made by different companies etc. That why the Llama models have a different prompt template than Gemma etc. You can usually find the prompt templates listed next to models on the HuggingFace in the model card for each model. Hope that helps.

@@samwitteveenai Thank you so much!

Hey thanks for pointing this out. I have just added it