Whats the best Chunk Size for LLM Embeddings 

I read the thumbnail as "Talk With Your Dogs" and was impressed with just how multi-modal Ollama has become.

Hi Matt, I really like your experimental approach to chunk size, so useful. I'm looking forward to installing bun (a step into the unknown for me 😱) and trying out your code! Thanks for your videos, they are a pleasure to watch 🙂

Loved your video and especially the experiment you ran! Everything is so new and changing so fast with LLMs that experiments like yours are very valuable. I would love to see you perform a more in-depth experiment on this and/or anything else you think would be worthwhile! It would be interesting to see a program based on utilizing two representations of the same document with different chunk sizes like one high and one low (e.g. 100, 5) and using one or the other based on what you need from it.

Great video - I dig your style. Just back from Spring break adventures and needed something to get pumped about diving in the AI madness on Monday. well done Sir

Brilliant results thank you.

Thank you for addressing this - great video! Thanks for providing an example for analyzing the chunk size and overlap. I've wanted to try this exact thing but wasn't sure on a good way to programatically assess the quality of the LLM's response. I'm surprised that the chunk size makes so much of a difference in the end results, and I'd LOVE to see your analysis of the myriad vector dbs out there!

excellent / pro tutos ❤ , clean english too and suite explainations thank you

I love all your videos so please continue posting them. On chunking, I recently built a RAG app to do QnA on our product docs where I have loaded all the product documentation in vector DB. In my case I converted these docs in to json array where each json object talk about one independent requirement for given capability. So there is no fixed chunk size for me, I stored each requirement in one chunk irrespective of its size an I have not used any overlapping. I was surprised by the accuracy of the results with this approach with nomin-embed-text embedding.

Very interesting, as usual

Thank you good sir!

Nicely done. I struggle with chunking on my projects and sometimes I think the performance varies from day to day. I am curious how chunk size varies RAG performance on smaller models like Mistral 7B vs the big ones like GPT 3.5Turbo?

Thanks!

Que vídeo espetacular! Adorei! Forte abraço, diretamente do Brasil!

Hey not sure if this is your thing or not (no worries if not totally cool!) but could you like concatenate your videos about embedding together into a super tutorial or maybe set up a playlist from your embedding videos? Super super good info in your videos man

What K value for retrieval did you see had the best results?

9:35 One interesting twist would be to use a rather short chunk size to look up the embedding, then privide the text of the couple of previous and next chunks as well to the LLM. That would give the LLM a longer context to use for providing the answer while still being able to find the closest matching embedding. I'm not too surprised about the results, except perhaps about overlap not making much of a difference. I would think having enough overlap to at least have complete sentences would be ideal.

Ciao Matt, Thanks for the insight. Been playing with Langchain+Ollama(nomic-embed-txt)+Groq(free now & speed) - Chunking on a pdf article with 20 sub-topics. Found out chunk size = 1024 & 512 w/o overlap sufficed. Finish off using gpt-3.5 (free) to merge my outputs for each topic do produce rather good summary for own use on a cpu laptop for now. On large pdf single topic - my goto approach is ask llm to generate list of questions for me - then will peruse those questions - in this way allows for better enjoyable reading & understanding of the pdf. What's your view on this? Keep up your videos. Cheers.

Music is inspiring me to try chunking text -:)

Stating the obvious a little but there are more than 2 variables to control for here. The video mentions another big one: the specific application. Also another huge variable is the specific embedding model you use and what capabilities it was trained with. Also how are you embedding? top match top 5 matches, anything above a similarity threshold. How do you measure similarity cosine, euclidian, manhattan, ...? Are you embedding raw chunks or generating hypothetical embeddings? I love any material that help answer any part of this. 😍

Several months ago I have made a very similar test. I found that chunks of 150 words with 20 words overlap were the best for my case. But now I am using token chunks instead of word chunks. Approximately, those 150 words are equivalent to 300 tokens.

Remember when the MCU was great, and how as good as a movie may have been, the post-credits stingers were always the best part? Thank you for filling the void. 10:31 (alternatively, "dude...")

Could You test the code repo chunking?

very nice video. I use pgvector, for me, chroma db did not suport all my files

I'm barely a novice on any of this but intuitively I'm wondering why the length of your longest question doesn't factor into choosing your minimum embedding length. Along the same intuitive line, wouldn't you want to pad or augment your answers so that questions always start at the beginning of a chunk? If your questions and combinations upon your questions are the most likely search criteria which you are likely to receive from users, wouldn't guaranteeing that each question is totally encapsulated in a chunk facilitate the speed and accuracy of the resulting searches?

Vector dbs can't wait my favourite is milvus for local dev

Chunk from Goonies?

I do have a question: wouldn't chunking by sentence(s) make sense?

Hi! Really interesting to hear 100 words only is kind the best... Still, this is based on use case right? Let say the question is: How to do xyz step by step? It's impossible to be answered in 100 words right? And if the list of how to split into chunks ( even with overlap ), it will lose the context above right? I mean... Let say there's 10 list item. The first 3 list item know the context since it's at the start ( 1st chunk ). In the second chunk it has 3 more list item, and maybe it doesn't know the start of the sentence which is the key. This is the problem right? Or embedding somehow can get the connection between the list and the main intention? Thanks

What about semantic chunking?

And what about using an LLM to do the embedding ?? Seems like it would know what's the best place to cut the text !

What's the difference between RAG and Embeddings?

I'm starting to think I'm going to have to learn ts (and bun).

the sms length of 160 characters was based on research that suggested that was long enough for most things people say,,, i think we overestimate how much information we're adding most times that we say more than that,,, a few words is enough to enter a completely different world of meaning, we forget how just a sentence or two is enough to bring us to almost anywhere

Hi the ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-6QAlbThWomc.html video isn't available any more. Is there a new URL? Thank you

Why Docs need to have all the same size. Cant it be dynamic?

ciao Matt, where is the code? :-)

Like granted for using Bun

What about a more content driven approach, such as using a traditional NLP library like spaCy to first break your text up into sentences? (You can remove all punctuation and let spaCy decide the breaks, or I suppose you could go an evil regular expression route.) Also, what about hierarchical embeddings? E.g., use the sentence embeddings, but also have paragraph embeddings, section embeddings, and so on until you have document embeddings at the very top?