Run Any Chatbot FREE Locally on Your Computer 

Taxpayers and venture capitalists probably? Who knows though. I am blown away by how tech savvy some people are. Even with only a little bit of knowledge and with AI some pretty powerful workflows are built and shipped. Exciting stuff.

I never thought it was hard. LM studio makes it so easy even my 11-year-old daughter can do it

@@CM-zl2jw Probably not its a gui they not getting tax money to print hello world but if they are maybe its better then sending it to ukraine orgies thanks..

You are complaining that a LANGUAGE model can't do math... You are definitely special 😂

Yea, i asked it to calculate the coefficient of a warp 9 warp drive and it couldnt even do that.@freedomoffgrid

@freedomoffgrid No models can do basic math, they have to use tools. Even the might GPT 4 has serious problems with math.

@freedomoffgrid limitations in the technology it self brother..

Better how?

I can feel your excitement. I was about to do the same but think I will use discipline and read the comments first 😂🎉. Self control!!

@@CM-zl2jw haha too smart! So far testing has been pretty good 👍

How did you get it working, every time i enter a message, i get error occured: failed to fetch.

I was wondering the same if mixtral was running well because of his beast machine or not.

A vague rule to go by is make sure you have a few gigabytes of ram more than the file size of the model itself, but the larger the file the better GPU you also need. @@TheMiczu

how did you get it to run on the GPU ?@@qster

@@ProofBenny it will automatically use it, no need to change settings

The current, pioneering and rather primitive LLM models have an average size of a few to tens of gigabytes. In this size they can pack most of the information, works of literature, paintings or films known to mankind. This means that they FANTASTICLY compress data. What is the result of this? That the data captured from a single user, even if filmed with sound 24/7, will be compressed to such a minuscule size that a precocious 14.4k modem would suffice to transmit it just when the Internet access appears. Besides, the model may be canny enough to find ways to connect to that Internet, attach user data to any file and send it that way to the rulers of our lives. Privacy needs serious work.

Or oobabooga with infinitely more control and multimodal. It’s the Ui that everyone creating models uses.

Do any of these allow for agents or the equivalent of copilot or autogpt?

From the video the main difference with lmstudio is Jan is open source, but am not sure if it can run on older processors or machines like lmstudio is starting to do..

@@ChrisS-oo6fl Yeah, but it's a pile of code-vomit on Github, which is exactly why normal peeps like me are NOT using it...

LM studio can also run Autogen @@mayagayamsee various tutorials on youtube :)

I adore YTers who quote what % of CPU or ram is being used without saying. Because that means they think we're family and know every detail of each other. And will be pleased when I show up to crash in their guest room.

you can try stable diffusion to generate image locally on your computer

@@Krisdomain That would be awesome! Thanks! Is it as easy to set up as JanAI or is it difficult?

@@BionicAnimations It is easy as I could do that. However, the images I generated were not nice, maybe I used poor models :(

There are great tutorial to install, I enjoy the Automatic1111 UI, it's prettier to use, but if you want quality images to be generated locally, there is some studying required and follow up installs. Keywords for your search: ControlNet , Negative Prompting, Seed number. This should give you an idea to where you are heading.

@@stribijev Hmm... I've usually seen really good pics with Stable Diffusion. How did you learn how to make them?

I got the same error message

Same ​@@Dayo61

@@Dayo61 Same, and no help , nothing........no answers to fix it

There are local models trained for way higher token limits upto 20k.

@@USBEN. What are you talking about? Self hosting an AI needs tolkens? What?

@@americanswan token limit =word context limit it can take in. Like 4096 slider in the video but upto 20k words

@@americanswan These models are restricted in what amount of content they can "remember". This content is measured in "tokens". GPT-4 has a context window of 128k tokens (which is a lot), which some people say means it could remember and talk about a 300 page book for example. So to find out if there is the right model for your needs, you would have to find out how many tokens the text in your PDFs has. As far as I know, OpenAI has a tool where you can paste text and it tells you how many tokens it has. Then you would have to find a model that has enough tokens in it's "context window" so it could remember the text in your PDFs. If every PDF only has one page with text it would be a lot easier than if it has a 100 pages (since a higher token limit also means higher demands for your computer). The next problem is that in "Jan" there doesn't seem to be an option to input your documents, so you would have to find a similar tool that allows that. At this moment in time I don't think you will find a satisfying solution (if your PDFs have a lot of text). But many people are looking for solutions for exactly your problem (especially since this would be very valuable for a lot of companies). So I'm optimistic something will come up in the next months or so.

@@USBEN.can you point to best working models with higher context in your opinion?

Train or upload docs for reference

I only know of ChatPDF (at the top if Googled). It's done online, so not ran locally, but it's free!

LMStudio is closed source though.

That is right, you can see Matt uses his own API key to enable GPT models.

This is because there is no "downloadable" version of gpt. Mixtral, on the opposite, doesn't have an online version (provided by the developers) so if you want to use it you must download it and run it on your computer.

@@stribijev Yeah, that’s when it became clear the title is misleading. I think the video title should be changed to “Run Any Open Source Chatbot FREE Locally on Your Computer”

@@r0bophonic Right. Actually, it is no news, the LLM Studio has been out there for quite a while.

Me too, LM studio will also inform you if your hardware is capable of running the model.

I would love a comparison between the two, I was actually going to ask Matt for that

Do you have any tips for what the recommended settings are for GPU layers and CPU offsets? My laptop is not very robust, sadly, but I did just upgrade to 32gb of ram. That did not fix my high cpu usage when running LM studios, however. It still has moments where it spikes up to 100%

@@jennab176 depends if your laptop even has an independent GPU. Many of the mid to lower tier laptops with just use the CPU’s integrated graphics processing, which at that point modification of GPU layers won’t improve token processing speed. It really depends on your specific hardware configuration. I bet if you post your specs here the community can help you optimize your settings.

When using the API the personal data is not used for training in theory.

Not used in training. I think that there is also an opt-out toggle in paid version of regulat chatgpt.

@@InnocentiusLacrimosayes. True. You can opt out of sharing data on chatGPT but then it doesn’t save the chat history.

If it's open source and run locally you're fine. If it's using any API then your data is being sent across the net and is not private