Running a Hugging Face LLM on your laptop 

Thanks!

except for the part that i have got no clue what the required software to start is. what os to run on nor how to run the code in step 2. the farthest i got was git bash for the transformer and thats about it. i just want to run some local ai, why does it have to be so obtuse??

@@norwegiansmores811 It's running in a Jupyter notebook - jupyter.org/ but you can run the code anywhere that can run Python code e.g. a Python REPL or a Python script.

@norwegianmores811 I think the easiest way to run local AI as of June 2024 is now this - lmstudio.ai/

Thanks - very kind of you :D

Glad it was helpful :)

Thanks, glad you enjoyed it!

Random idea, but a video on how to run an embeddings model on a laptop would be really cool 😀 Could even combine embeddings + text2text for more specific answers. Or even t5_3b + selenium to create something similar to bing chat. Anyway, wish you luck on RU-vid 😊

Sorry, I didn't see this reply! I've got a notebook with that idea sketched out, so I'll create a video for that soon. On holiday at the moment, but will do it when I get back home!

Thanks for your kind words! Let me see if I can figure out a good way to implement your suggestion 🙂

utils should be referring to this file - github.com/mneedham/LearnDataWithMark/blob/main/llm-own-laptop/notebooks/utils.py - so in theory that's independent of WiFi connectivity. If it can't find that module you could copy/paste those functions into the notebook and use them like that.

So long as you have a behemoth of a machine. 16GB Ram = 100% memory usage 😭😂

I don't think anyone can afford an expensive laptop lol

Do you know how to run it on live servers!! How to get?

It is needed to download the model!

The api key is used to authenticate account for huggingface

Yeh you can could combine it with any database to do RAG.

Winning!

I'm using the latest version of Jupyter Lab and I have it set to dark mode with pretty much every one of the views hidden so that I can use as much of the screen as I can. Not sure if that answered your question, so feel free to follow up!

Oh I'm not sure about that error - I haven't seen that one before. Since I made this video I've been playing around with another tool called Ollama which I found easier to use. It might be worth giving that a try to see if that works for you? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NFgEgqua-fg.html

You can. You can use a Python REPL or even the iPython CLI. Or you could put it all in a Python script and run that.

Yeh I think it should work with both of those.

Does it work after you see that message?

@@learndatawithmark getting the same error and hang when running in colab or locally. seems like the pipeline("my query") never resolves

Same thing happens to me! Hanging and giving random replies! I am using mac m3

Good tip! I get that error somewhat randomly but never quite figured out the combination steps that result in it happening!

It depends. If you are passing your HF API token because you're using the HF inference endpoint then your data is getting sent to the HF API. If you're passing it because you're downloading a model that requires token auth then your data will only be local to where you run the model that you download.

It's reading from the ~/.cache directory. So it constructs a file path based on that directory & the model id

Yes you should be able to use a similar approach. There's a good guide on image to text over here - huggingface.co/tasks/image-text-to-text

I think it's the same thing under the hood - no need to change from your approach!

I'm using a Jupyter notebook in the video

This is Jupyter Lab - jupyter.org/

I use a Mac M1 with 64GB RAM. I think it's a 2021 edition. I've found in general that the quantised models work better on my machine - either using Ollama or llama.cpp. ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-YDj_ScvBpKU.html ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NFgEgqua-fg.html

Do you mean this one? huggingface.co/

If it's running too slowly then maybe it'd make sense to try out some of the quantised models instead. Those ones are smaller and better suited for running on consumer hardware. I quite like Ollama and I've made a few videos on that. This is probably the best place to start - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NFgEgqua-fg.html

My impression is that most of the models being created are assuming that you have insanely good GPUs to run them on! Since I created this video, there's been a lot of work done by a guy called TheBloke on Hugging Face to 'quantise' the models, which effectively means that the amount of resources required is reduced, but the quality of the model is slightly reduced too. I've found those models work a lot better on my laptop. The Bloke is using a format called GGUF, which is kind of a defact format for LLM models. I made a video showing how to run one of his models on my machine - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-7BH4C6-HP14.html. That video uses a tool called Ollama which works on Linux/Mac - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-NFgEgqua-fg.html There is also another library called CTransformers which lets you choose whether to run models on the GPU or CPU. I've found the 7B parameter quantised models work reasonably well even on the CPU. I should probably create a video about that I guess! But in the mean time, this is the link - github.com/marella/ctransformers

@@learndatawithmark thanks!

Can you share a script with all the code you ran and I'll try to reproduce?

use 3.9 its more stable

your python installation is missing the transformer module 'pip install transformers' will do the trick.

Haha, thanks. It took me a little while to figure out how to do it!

Yes it should work without wifi - but you will need a connection to the internet to download the model.

Hard to know exactly why - maybe connectivity with Hugging Face or maybe your internet or maybe the download tool?! You could try going to Hugging Face directly and click through to files and download them directly to see if it helps.

@@learndatawithmark You got to rerun that part couple times and make sure you have a fast connection. For example, pytorch_model.bin is 6.71gb that would take some time to download

It will already be there so if you try to use it again there won't be any need to download it

You can do this using CTransformers like I did in this video - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-S2thmwdrYrI.html I think you might even be able to do it with HuggingFace transformers, but I haven't tried it myself.

@@learndatawithmark if one day you make a video on this, I would like to see it, in fact what I would have liked was to discuss with the model directly with python without going through any interface and to give it a personality with json like we have could do it with webui (but without webui) I tried various methods and honestly I find so little explanation. I had the idea of ​​making my own bot as I saw in "wifu" mode in the sense that it is totally customizable and we give it a personality with a long term memory. The basic idea was to have a small model just for me. I'm just frustrated to see bots that don't even remember talking to us 2 seconds before. xD

@@The_Little_One_Of_Darkness it sounds like you want to keep the history of the chat messages between you and the LLM? I showed how to do this in memory on this video using Ollama, but it can be adapted to another approach - ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-MiJQ_zlnBeo.html. I can across a tool called MemGPT which I think attempts to solve this problem, but I haven't tried it yet - memgpt.ai/

Probably works for this one but sometimes there will be multiple different versions of the same model and it'll take up all your free space if you do that!

I'm using a Jupyter Notebook, but the code would work in any Python environment or script jupyter.org/

@@learndatawithmark ah thanks, that’s interesting - I’ve never heard of that

Yeh I think with a bunch of the models you're able to run them on the Hugging Face website on the right hand side of the page. And then in general there are many services that offer APIs that you can call. The approach describe in this video is only for if you don't want to use those services.

I guess I need to set the HUGGING_FACE_API_KEY variable to my token beforehand.

You would need to set that environment variable before running your Python environment otherwise yeh it'll be none