PrivateGPT 4.0 Windows Install Guide (Chat to Docs) Ollama & Mistral LLM Support! 

You're welcome! I am glad to hear the video assisted! Thanks so much for reaching out.

Glad it was helpful! Thanks for the feedback, much appreciated.

Thanks for the sub and the great feedback. Appreciated!

You're very welcome! Thanks for the feedback.

Thanks for the feedback, glad the video was helpful.

You're welcome! Glad to hear you are up and running. Thanks for the feedback!

Thanks for the sub! Glad the video helped out.

You're welcome! Thanks for the sub.

Glad it was helpful! Thanks for taking the time to reach out.

Great to hear! Glad you are up and running. Thanks for the feedback.

You're welcome! Glad the video got you up and running.

You're welcome! Thanks for the feedback. Glad you are up and running.

Great to hear! Thanks for the feedback much appreciated.

You're welcome! Thanks for reaching out. Glad the video helped.

You’re welcome 😊 Glad the video assisted. Thank you so much for the feedback.

Glad it helped! Thanks for the feedback.

Welcome aboard! Thanks for the sub and feedback.

Glad it helped, Good to hear your are up and running.

Pleasure! Glad the video assisted. Thanks for the feedback!

Great to hear! Thanks for the feedback.

You're welcome! Glad video assisted and you are up and running. Thanks for reaching out.

@@stuffaboutstuff4045 Could you please also make a video about how to add gpu support on privateGPT?

You're welcome! Thanks for making contact.

Hi, thank you so much for the feedback, let me look into it and I will revert soon! Thanks.

Hi, I use Anaconda for most of my AI environments when working on Windows. I find it easy to work with and install required SW etc. Thanks for reaching out.

Hi, thank you for the feedback! Noted on the video idea. Glad you are up and running.

Hi, glad you are up and running. You can change the Ollama model. Download and install the model you want to use and change your config files. Check the Ollama example shown on the below link. docs.privategpt.dev/manual/advanced-setup/llm-backends

Hi, sorry for the late reply. You can check out this page to change your LLM. Let me know if you came right with this. Thanks for reaching out! 🔗docs.privategpt.dev/manual/advanced-setup/llm-backends🔗

Thanks for reaching out. Glad to hear you are up and running.

Hi, yes its a bit slower against a local LLM dependent on the GPU available in the machine. Did you try and use Open AI or one of the online providers, if you want to have it super fast. If confidentiality is not your main concern maybe give it a go. If you want to remove just uninstall all SW and delete the project folder you built PrivateGPT in and you should be fine. Thanks for reaching out.

Thanks for the feedback. Glad you are up and running and the video assisted.

Hi, can you confirm you are running this in your Anaconda PowerShell terminal, Check the steps I use from about 9:20 in the video. Let me know if you are up and running.

Hi, sure you can. You can install llama3 on Ollama. You would need to change the config files. The link below should assist until I can update this video. Thanks for the feedback and the video idea. docs.privategpt.dev/manual/advanced-setup/llm-backends

@@stuffaboutstuff4045 Great thanks for the prompt reply and the link. Looking forward to your new video as well!! You make very easy for beginner like me! Really appreciate your work

@@stuffaboutstuff4045 if you don't mind, allow me to ask a question, I am plannign to adopt the ollama approach but i don't know of what part of the video should i turn to the command PGPT_PROFILES=ollama make run. Great thanks!

Sure its possible with PowerShell scripts. Let me check it out and revert.

Hi, if you build with Ollama it will offload to the GPU automatically (Nvidia or AMD). It does not hammer it to its full potential I have seen. This utilization will get better each evolution of the project. Let me know if you got the GPU to kick in when offloading.

Hi, did you get this resolved. Yes, I also created a pgpt folder in the root of the drive. Just to confirm are you running Python 3.11.xx? Let me know if you came right with this.

I wrote also a comment, I am complaining about the same issue. I have also the version 2.0 also from him and as if it wasnt uploading slow enough, but in version 2.0 in any case the upload was considerably faster. I had during the upload 80% load on my 32 Thread cpu but now in 4.0 the cpu is just boring itself with 5%, which explains the slower upload. The parsing nodes are genereting the embeddings much slower. Since I have more than 10000 pdf files, it is unacceptable to wait endless during the uplad. Now I am waiting since 40 minutes only for 2 huge files with around 3000 pages, which took with the old one only 20 mins totally. I have no idea, how long it will take till end and we are talking only about 2 x files. The other 9998 Files will not even be uploaded in one year if the problem will not be solved. I am disappointed to lose time with 4.0.

Hi, thanks for reaching out. The new version allows you to use numerous LLM Backends. This video shows how to use Ollama just to make the install easier for most and its now the recommended option. The new version can still be built exactly like the previous, if you had better performance using local GPU and LlamaCPP you can still enable this as profile. If you really want high speed processing you can send it to Open AI or one of the Open AI like options. Have a look at the backends you can enable for this version in the link below. Let me know if you come right.. docs.privategpt.dev/manual/advanced-setup/llm-backends

Now that I installed gitbash from the makeforwindows website it works. However, I now run into this error when running make run: Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C: gpt\private-gpt\private_gpt\__main__.py", line 5, in from private_gpt.main import app File "C: gpt\private-gpt\private_gpt\main.py", line 4, in from private_gpt.launcher import create_app File "C: gpt\private-gpt\private_gpt\launcher.py", line 12, in from private_gpt.server.chat.chat_router import chat_router File "C: gpt\private-gpt\private_gpt\server\chat\chat_router.py", line 7, in from private_gpt.open_ai.openai_models import ( File "C: gpt\private-gpt\private_gpt\open_ai\openai_models.py", line 9, in from private_gpt.server.chunks.chunks_service import Chunk File "C: gpt\private-gpt\private_gpt\server\chunks\chunks_service.py", line 10, in from private_gpt.components.llm.llm_component import LLMComponent File "C: gpt\private-gpt\private_gpt\components\llm\llm_component.py", line 9, in from transformers import AutoTokenizer # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\dmm\anaconda3\envs\privategpt\Lib\site-packages\transformers\__init__.py", line 26, in from . import dependency_versions_check ImportError: cannot import name 'dependency_versions_check' from partially initialized module 'transformers' (most likely due to a circular import) (C:\Users\dmm\anaconda3\envs\privategpt\Lib\site-packages\transformers\__init__.py) make: *** [run] Error 1 Any Idea how I can resolve this?

Hi, can you confirm loading all the required SW including all the Make steps I perform from 3:35 into the video. Let me know if you were able to resolve this. Also confirm you are running everything in the same terminals and admin mode where needed. Make sure you use Python within 3.11.xx in your Anaconda Environment.

Agreed, just increases the build difficulty as bit. 👨‍💻 Thanks for reaching out.

when i open anaconda prompt shows this: Failed to create temp directory "C:\Users\Neo Samurai\AppData\Local\Temp\conda-\"

Same issue Here i cannot pass this neither v2 or this version

Hi, just checking if you came right with this? When you open your Anaconda Prompt or Anaconda PowerShell prompt they must open and load and show (base). Is this not showing in both Anaconda Prompt or Anaconda PowerShell prompt? Did you try and open both in admin mode? It seems there is a problem with the anaconda install on the machine.

Hi, are you using Ollama as backend? If not follow the steps in the Ollama video on the channel and just hook up your install to that. Otherwise test this on a hosted LLM like OpenAI. You should not struggle if you follow the steps exactly in the 2 videos. Let me know if you are up and running.

Hi, from what I can gather you will get this error if you prompt documents but no documents are loaded. Can you ensure you uploaded documents into PrivateGPT and selected them prior to prompt. Let me know if you come right with this. Thanks for reaching out! If problem persits check out these links and check if it helps, github.com/zylon-ai/private-gpt/issues/1334 , github.com/zylon-ai/private-gpt/issues/1566

From the git issues page, this resolved the issue for me. "This error occurs when using the Query Docs feature with no documents ingested. After the error occurs the first time, switching back to LLM Chat does not resolve the error -- the model needs to be restarted." Enter Ctrl-C in Powershell Prompt to stop the server and of course 'make run' to re-start.

Hi, can you confirm you are installing pipx in a normal admin mode command prompt. Just to check if you followed the steps from 6:30 in the video onwards. If still not working can you confirm you have Python 3.11.xx installed with the pip package that ships with. Let me know if you came right with this. Thanks for reaching out.

Hi, you can just run Anaconda PowerShell prompt again, activate the environment you created. Make sure you are in the project folder. Set the env variable you want to use and execute make run. Check the steps performed in the Anaconda PowerShell from 9:24 in the video. Let me know if you are up and running. Thanks for reaching out.

Hi, You would need Python 3.11.xx. The code currently checks if the installed Python version is in that range. I got build errors with 3.12 installed in the environment. Let me know if you are up and running.

Great, glad you are up and running.

Hi, did you come right with this. There are some good comments on this video on speeding up the install including working with large docs that slow down the system. Check the link below, maybe this can assist. Also check the terminal when this happens for any hints on what might be hanging up. docs.privategpt.dev/manual/document-management/ingestion#ingestion-speed

@@stuffaboutstuff4045 Thanks, how can I contact you? I noted you are South African through the accent!

Hi, does it not launch at all and stops with this error. Its seems its the webserver that needs to start. When you launch it I know it can display uvicorn.error message but when you open the browser you will see the site up and everything works. If you get this, uvicorn.error - Uvicorn running on http : // 0. 0. 0. 0 : 8001 then it works. But by the comment it sound like you have the whole module missing. PrivateGPT is a complicated build, but the steps in video are valid, I would suggest retracing the required SW and versions required like Python etc. and the setup steps just to make double sure no steps were missed. I also find more success running the terminals in admin mode to avoid issues. Let me know if you came right with this and thanks for making contact.

I also faced the same issue because I forgot to start the env. Check yours too

Hi, when you launch it in the Anaconda PowerShell Prompt, just go back to that terminal when done and press "Control + C". This will shut it down. You can save the starting profile as a PowerShell script and start it or as bat if you use cmd. Thanks for making contact, let me know if you came right with this.

Thanks for the idea. If you want you can try and get it up and running. You can install the 8b model if you use Ollama. (ollama run llama3:8b) The link below has the example configs that would need to change. Thanks for reaching out and for the feedback, much appreciated. docs.privategpt.dev/manual/advanced-setup/llm-backends

Hi, just checking if you got this built on Ubuntu. If not you can follow the steps for Linux using the link below. Thanks for reaching out. docs.privategpt.dev/installation/getting-started/installation

@@stuffaboutstuff4045 thanks a lot!!

I suspect it might caused by Gnu's path, although I did include it in the env variables...

Hi, did you manage to resolve this issue. Please check from about 9:15 into the video. Are you completing these steps in an Admin Anaconda PowerShell with the environment activate and from the correct folder. Let me know if you came right. Thanks..

Hi, If you are using Ollama you can just update your config file in your PrivateGPT folder to point to the model downloaded in Ollama. if you want multiple models on Ollama its fine. I use my Ollama to feed numerous AI frontends with multiple LLMs running. Check the link below for the defaults: Default Mistral 7b. On your Ollama box: Install the models to be used, the (default for PRIVATEGPT settings-ollama.yaml) is configured to use mistral 7b LLM (~4GB) and nomic-embed-text Embeddings (~275MB). Commands to run in CMD: ollama pull mistral ollama pull nomic-embed-text ollama serve docs.privategpt.dev/installation/getting-started/installation

Thanks for the input, agreed I started to zoom in on the command prompts in newer vids. Thanks for reaching out and I hope the video helped.

Hi, did you install all the required SW I install at the start.

Close both Anaconda Prompts and restart the process. Dont rebuild your project though. GPT4 says its a connection issue when creating and sometimes a computer restart sorts the issue. Worked for me.

open your tomfile and update the tom file with the correct Build version then update the lock file

Hi, you will have to activate your conda environment. Make sure you are in the project folder and launch Anaconda PowerShell again. Check the steps from 9:24 in the video. Let me know if you are up and running.

@@stuffaboutstuff4045 Wow, Thanks for your answer! Just amazing!

so I ran it over night and ollama is still not performing well with ingestion. It definitely under utilizes the hardware for ingestion. Right now a lot of the local LLMs don't seem to leverage the hardware as well when it comes to ingestion. That is an improvement I would like to see in general. Not just of ollama or privateGPT. The ability to ingest faster through better hardware utilization/improved processing and storing ingest files long term on the drive along with the ability to query the drive and load relevant chunks into the vRAM would significantly expand the depth and breadth of what these tools can be used for. vRAM is never going to offer enough and constantly training models won't work either.

Hi thanks for the update. Had a bit of a scare with the update available moments after publishing this vid 😊. Thanks for the confirmation, I also checked and the install instructions remain intact. Appreciate the feedback. PS. I totally agree with the performance comment made.

Hi, thanks for the feedback. Let me look into this.

Hi, just checking in if you resolved this issue? Just want to confirm that you are following the steps I use in the video to install poetry, please check from 6:28 in the video. I use a command prompt in admin mode to complete all these steps. From 7:36 we back in Anaconda and Anaconda PowerShell prompts. Also confirm you are using Python 3.11.xx for the Anaconda environment otherwise you will get a bunch of build errors and failures. Let me know and thanks for reaching out.

@@stuffaboutstuff4045 im pretty sure i was doing those steps but im using webui now instead cuz its easier to do

Hi, thanks for the idea, they release new version so quickly I will check how I can incorporate in the next one. Thanks for reaching out.

Hi, it is not the fastest with Ollama, the upside is its relatively easy to get working. Should confidentiality not be an issue using the Open AI profile will increase speed exponentially. You could also build this local if you have a proper GPU but expect a more complicated install to follow. Thanks for reaching out.

Hi, yes if you load all the required SW. This ensure you dont get errors if you build the other non Ollama options..

Hi, can you have a look at this post and see if it helps. Let me know if you come right. github.com/zylon-ai/private-gpt/issues/1701

Hi, VS Studio components are used in the background for compilation and build of the programs. Hope the video helped and your PrivateGPT is up and running.

Hi, I personally have not worked with massive datasets. I know some in the comments have. You might want to check out the link for bulk and batch ingestion. docs.privategpt.dev/manual/document-management/ingestion#bulk-local-ingestion

I think you can once you have everything installed, at least that works for me

Hi, as noted below, that correct. Once installed you can disconnect the machine if you have the LLM local. Let me know if you come right.

@@stuffaboutstuff4045 Hi thanks for the reply. I am struggling a little understanding this. Do I have to download a portable version for everything or just a portable VSC? Meaning if I want the privategpt to work on another machine from the thumbdrive, do I just need to transfer the VSC files or must I transfer everything, such as git, anaconda, python etc?

Hi, thanks for reaching out. The new version allows you to use numerous LLM Backends. This video shows how to use Ollama just to make the install easier for most and its now the recommended option. The new version can still be built exactly like the previous, if you had better performance using local GPU and LlamaCPP you can still enable this as profile. If you really want high speed processing you can send it to Open AI or one of the Open AI like options. Have a look at the backends you can enable for this version in the link below. Let me know if you come right.. docs.privategpt.dev/manual/advanced-setup/llm-backends

@@stuffaboutstuff4045 Thanks for advice, if change anything in the backend, it comes to error, despite according to the official manual and your explanation. If I setup ofr both ollama then it works but as mentioned the file upload is extreme slow. Now I found a solution by installing from scratch according to version 2.0 with llmacpp with huggingface embeddings, whereas I changed the ingest_mode from single to parallel now it works much faster. There should be more options in order to increase the speed by increasing the bash size or worker counts. Since they did not work before, i will not change and corrupt the installation as long as you can provide a manual how to increase the embedding speed to maximum most probably with help of gpu like in chat. The GPU support in chat works good but during embedding the GPU is not being used

@@stuffaboutstuff4045 after changing to parallel the cpu utilization is at 100% and that explains the faster embedding. Since I have one of the fastest consumer cpus the result is now finally satisfying.

@@firatguven6592 Awesome, glad you are running at acceptable speeds.

​@stuffaboutstuff4045 in addition to that, i could change some paramters in settings.yaml with help of LLM. These are, batch size to 32 or 64, dimension from 384 to 512, device to cuda and ingest_mode: parallel, which gave the most improvement. Now the embeddings are really fast. Thank you very much. I would like to test once also the mode sagemaker, since I could not succeed that mode working. I will try it later again.

Hi, look carefully, I am in a normal Anaconda prompt at that stage and the next commands need to go into Anaconda PowerShell. 👨‍💻 Thanks for reaching out, hope the video helped..

Hi, yes you can, the packages and flow will be similar to the video but obviously following the Linux steps. You can check out what's involved in building on Linux by checking out the below link. Thanks for reaching out, let me know if you come right. 🔗docs.privategpt.dev/installation/getting-started/installation

Thanks for your kindly response. But now I'm facing a new problem while try to run the private_gpt module: "TypeError: BertModel.__init__() got an unexpected keyword argument 'safe_serialization'" Please convey me how to resolve this error?

Hi, if you are using Ollama you can install it and test it out, I am currently downloading it to test. 8B Model can be installed on Ollama using - ollama run llama3:8b or you can install the 70B Model -ollama run llama3:70b. Let me know if you get it working.

Hi, agreed, its has to be Python 3.11.xx.

Hi, Thanks for the idea, let me look into that.

its localhost noob

Hi, thanks for the feedback, Noted.

Hi, what error do you get? Let me know and maybe I can help you out.. Thanks!

​@@stuffaboutstuff4045its showing Error Collection make_this_parameterizable_per_api_call not found

Hi, can you confirm what LLM you sending it to? Ollama local like in the video? Are you getting no responses when you ingests docs and on the LLM Chat? Both not working? Anything happening in the terminal when it process in the Web UI? Let me know and we can hopefully get you up and running.

@@stuffaboutstuff4045 I have difficulty with PROFILES="ollama" (LLM: ollama | Model: mistral). I followed the same steps indicated in the video. LLM Chat (no file context) doesn't work, it gives me blank responses; and Query files doesn't work either, it also gives me blank responses. The error I get in the terminal is the following: [WARNING ] llama_index.core.chat_engine.types - Encountered exception writing response to history: timed out

@@stuffaboutstuff4045 I have solved the problem by modifying the response time in the 'setting-ollama.yaml' file from 120s to 240s. Thanks for the well-explained tutorial, keep it up.

@@stuffaboutstuff4045 I have solved the problem by modifying the response time in the 'setting-ollama.yaml' file from 120s to 240s. Thanks for the well-explained tutorial, keep it up.

Hi, you can have a look at these threads, you would need to have LLM support for the language. I hope they point you in the right direction. github.com/zylon-ai/private-gpt/issues/28 github.com/zylon-ai/private-gpt/discussions/764

@@stuffaboutstuff4045 🙏 thx

Glad you are up and running. Thanks for sharing.

I am sorry, that was the error for querying file. the error I get when trying to use the LLM Chat is "NoneType' object has no attribute 'split"

Hi, this is usually a document ingestion or documents not selected issue. Can you check out the below link and restart the model. Let me know if you came right with this. github.com/zylon-ai/private-gpt/issues/1566

Hi, just checking if you resolved this issue? Looks like it cannot talk to the backend you sending the requests to. Are you using Ollama on a different server? If so make sure you open its webserver to support more than localhost connections.

@@stuffaboutstuff4045 I've changed it to totally local with $env:PGPT_PROFILES="local" and everything seems to work fine there, just not with anything else but that is also alright for me as I am still tinkering around with this. Can't thank you enough for this video and how easy it is to follow anyway.

Hi, can you confirm that when you set the environment variable you are doing this in a Anaconda PowerShell prompt in admin mode. Make sure your environment is activated in the PowerShell terminal, these are the steps from 9:17 into the video. Let me know if you come right. Thanks for reaching out.

@@stuffaboutstuff4045 yes, I opened a new anaconda powershell prompt and ran it as admin. I am thinking of starting everything over again... What would you advise me to do ? uninstall everything and refollw the steps of your video ? Thanks for your answear !

Hi, did you come right with this? Just checking, you have the required SW, running everything in correct terminals (CMD, Anaconda, Anaconda PowerShell etc. I usually also ensure I run in admin mode terminal to avoid some issues. If you run make can you confirm its in the machines path? After adding to path ensure you open a new prompt windows so it loads the path. Let me know if the above helps..

@@stuffaboutstuff4045 Apparently the problem was windows 11. I repeated the exact same steps on a Win 10 system and it worked perfectly.

Hi, this issue would usually arise when the system does not see documents selected or ingested. Have a look at the below two posts. Let me know if resolved. github.com/ollama/ollama/issues/3052 github.com/zylon-ai/private-gpt/issues/1334

Hi, just checking if you came right with this. Sound like something is going wrong when you set PGPT Profile and execute make run. Can you confirm you are doing this in admin mode Anaconda PowerShell prompt. Ensure environment is active first and check steps 9:20 into the video. Let me know if resolved, thanks for reaching out.

@@stuffaboutstuff4045 I'm having the same problem access denied to mistral

Hi, can you confirm you are setting the environment variable and using make run in a admin mode Anaconda PowerShell prompt. Check 9:16 onwards in the video. If error persist please confirm if your are using Ollama like I do in the video or building for local Lllama. Thanks for reaching out!

@@stuffaboutstuff4045 I've followed all of those steps correctly but I have found that after I run, poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant", setuptools never completes installation no matter how long i wait. Also even though I have completely uninstalled python and then reinstalled only python 3.11.0, when I check the version in Anaconda Prompt is returns 3.11.9

Hi, just checking if you managed to resolve this? Did you follow all the make steps I use from about 5:25 in the video. Also just check the steps from about 8:05, I create my folder for the software in the root of the drive i.e. c:\pgpt. Just check those steps and confirm SW install location. Lastly make sure you load the $env variables in Admin mode Anaconda PowerShell prompt. Let me know if you were able to resolve..

Hi, if you follow the instructions and config for Ollama, Ollama will handle the GPU offload. Otherwise you have to build it for full local for Llama-CPP support. Let me know if you come right..