What are you going to create with this tool? 💼Consulting: calendly.com/engineerprompt/consulting-call 🦾 Discord: discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: ko-fi.com/promptengineering |🔴 Join Patreon: Patreon.com/PromptEngineering ▶ Subscribe: www.youtube.com/@engineerprompt?sub_confirmation=1
I have listened to this 4 times because I think you have presented the information fantastically. Unfortunately, however, this means I had to endure 32 adverts, including some really long one of 3 mins or more. I am sure this is earning you good money, and I am letting them run to help with that for you, but really, if you can change the setting so people like me can support you without having to wait 10 minutes every time we watch your video. Much appreciated.
This history feature is great. However, i would be great to have a chat history like with the chatGPT-user interface .. so that one can retrieve any of the former historical chats I had with localGPT. Just an idea :)
I just had an idea: What about saving the session chat as a pdf with a title (example "Session 64 about something") ... and then uploading this document at a new session and tell localChatGPT to use the pdf "Session 64 about somethin" as chat history) and that i want to continue this chat with the history as context?
Hello brother! Thanks again for the video. I Have noticed one thing, i don't know if you have the explanation. I have ingested a medical book with 700 pages and ran the program in a rented CUDA 48GB VRAM machine using the Llama2 13B chat GPTQ model. I make a question about some disease that i know is explained in the book, and the first answer always starts with "According to the provided text, ...." and the answer is always correct. However, after this first answer the subsequent answers about the same disease, let's say, what are the symptoms, program seems to be responding with its own original LLM knowledge, because it doesn't say anymore "According to the provided text" and the answers are correct (doesn't seem to be hallucination) but some information from the answer are not in the book, so i am sure that is not from the book. Do you have any explanation for that?
Try with a well-known disease with symptoms everyone is familiar with, cross-reference and try to trap it, and you'll eventually figure out if it's a hallucination.
@@Data_Core_ yes I have already tried that and I cross checked inside the book. The answers are coherent but not the same written in the book. I mean, looking in Google I see that those answers are correct, but some parts are not present in the book, which means that is coming from its own knowledge base, not from the book. That's my guess.
Thanks for your video my friend! One question :if i ingest documents in a certain language (portuguese, in my case), would the system handle the prompts in Portuguese? Llama2 can handle it? Thanks!
I noticed you are using a chunk_size of 1000 when splitting documents, but the instruct embeddings only work up to 512 characters. Won't your document embeddings be only based on about half of each document?
I think you are referring to the size of embeddings which is different than the input text that I can receive. So it will take a chunk of 1000 chars and convert it into a 512 dimensional space. Hope this helps.
@@engineerpromptHi, I wish I was. The embedding dimension is 768 and the sequence length is 512. I hope I'm wrong because I'm also using this embedding model for a project. Look at the HF MTEB leaderboard. There are also a post out there where someone asks the dev what it would take to raise the sequence length.
This is a great video. Thanks a ton! I set up on my M2, with 24G RAM . However during inference I cannot see the GPUs being put to task. The P-CPUs are pressed instead. I have passed device_type as mps. Any hints? Thanks!
Thank you very much for your nice work! 1. Related to the order of context, history, and question in template and input_variables, is it required to be the same order? Different order can give any effects? 2. For example, I use LLM "Baichuan2-13B-Chat", but I cannot find its prompt template. Thus, what exact template should I use?
Thanks for the repo! But I found running llama-2 model locally is extremely slow in my Mac Mini, and I noticed that the total time of running a query in this video is so slow too. Anyone knows why prompt eval time takes so long? 😅😅😅
Tks for the video and efforts for making it this far on local GPT project, I have question here, what about if I need to add new document to source documents files which are already embeded (ingested) and stored in chromadb. How to embed the new document and add it to the VD base without ingesting again all existing docs, in other words could we avoid ingesting previous embeded files every time we add a document and instead do the embwding for only the new doc and add it to the vectore data base.
@@engineerprompt Sorry if my questions sounds lame. Is FAISS a vector database? I remember in your earlier videos you used FAISS which was later pickled and used if same PDF's were used for our search. that way we didn't have to vectorize the whole doc again. How different is Chromadb from FAISS?
@arjundev4908 yes, FAISS is a vector store like chroma. You can use either. I like to use FAISS if I have a smaller dataset and chroma for larger datasets. Hope this helps
I usually put new documents to the SOURCE_... folder . After running ingest the new embeddings are added to the vector store. After that I remove the new documents and put them in a ALREADY_PROCESSED folder. I am not sure ... but i believe this is exactly what you were asking for.
Is the VRAM supposed to fill up as you enter more queries? Using 7b models it starts at 6gb vram then it works itself up until it can no longer operate.
I have watched all the videos of the project localGPT. Thank you so much for all your efforts and contributions. I wanna know if are you planning to upload any videos to set up llama on my local system with memory. I am able to set up llama-2 13B on my system, but the things I can only perform one-time conversations and it does not have the memory to perform interactive conversations. And, why your code are not using any GPU? what shall I do if i wanna use GPU? It is using only CPU
I am working on it but i want like chatgpt i want to create one time embedding and based on user input it reaponse according to my embedding ...the problem is that it give reaponse butit create embedding every time ...can you help me.?
Hi thanks for all. You use Chroma as db vector' database is there any reason ? why not Faiss or Qdrand ? As embedding model the new gte-large seems to be a good idea no ?
Hi, Also, I'm doing tests when using chat history (ConversationBufferWindowMemory) + local data retrieval + llm (Baichuan2-13B-Chat) to get answer. I have two tests. Note that, all questions are related to the local data I feed to llm. Without chat history ConversationBufferWindowMemory with k = 0, the answer is quite ok. It means, llm can gives, at least, related information from my question. With chat history ConversationBufferWindowMemory with k = 4, when I pass the next question, really "different" from previous question, but I got the same answer as the previous question. It happens several times. Anyone has the same problem with me and any solution I can fix that?
De momento se está trabajando en eso. Hay que entender que los modelos de lenguaje no saben lo que dicen, sino que infieren sus respuestas según tus preguntas (en el caso de los modelos ajustados como chat). Imaginalo como un autocompletado con esteroides.
