Would be good if you could add a section where you show the query performance WITHOUT the NER filtering so we can understand the benefits that the additional NER complexity brings.
thanks, sir for sharing, it's such informative content, but there is a question "hmm, instead of using pinecone can we use the elastic search ? and why not apply "removing stopwords step and keyword extraction before working on the NER engine? Looking forward to hearing back from you, thanks "
@James Briggs First of all I would like thank you very much for your wonderful content. Is it possible to use the Llama-2 model for NER search for legal documents instead of "dslim/bert-base-NER"?
Amount of text you can embed into a single embedding is far more with OpenAI vs. sentence-transformers. Performance for shorter chunks of text is comparable to a average fine-tuned sentence transformer, and if you fine-tune the sentence transformer well you might even get better performance - but this is just shorted chunks of text (sentence - to - paragraph length)
Great video, again. Would have been nice to see what the benefit of NER in the search actually was. i.e. would vanilla semanitc search have yielded similar results anyway. Time for me to run some experiments :)
Sometimes yes sometimes no, I think it’s very use-case dependant, in this case we saw some better results when testing it but I didn’t show that in the video oops 😬
