Using Ollama to Run Local LLMs on the Raspberry Pi 5 

I just recieved my pi, and I'm about to do the same thing.. Are you doing anything else on it?

Nice, I'd like to compare to see how much faster an nVME would run these models.

If you want to do a test, let me know. I could run the same model and query as you, and we could compare notes. My guess is that processing time has more to do with CPU and RAM. but not 100% sure. Having said that large (1TB+) nvme makes storing models on the Pi convenient. Also boot times are rather expeditious. When the Pi5 was announced, I knew right away that I wanted to to add an nvme via the PCI express connector. Worth the money, IMO. @@IanWootten

I don't follow? What VM? ssh into what?

@@BradleyPitts666 WM ... windows Manager.

​ @BradleyPitts666 meh ... youtube not showing my previous (phone written) answer again, so I can't edit it, and I can't see/edit my previous answer ... so this might be a near identical answer to another answer, sorry. I blame RU-vid :P The Edit is that I disabled even more services, and marginally faster answer. So : WM is the Windows Manager. It uses resources (processor time and memory) while it runs, not a lot, but it's not marginal. So disabling the WM with 'sudo systemctl disable lightdm' and rebooting is beneficial for this particular usecase. Technically ,just calling 'systemctl stop lightdm' would work too, but by disabling and rebooting you make sure any services lightdm started really aren't running in the background. You can then use ollama on the command line. If you want to use it from your main system without hooking the rpi to a monitor and plug a keyboard in it you can enable sshd (the ssh daemon, which isn't enabled by default in the pi-os image afaik) and then ssh to it, and then use ollama there (THAT uses a marginal amount of memory though). I also disabled bluetooth, sound.target and graphical.target, snapd (though I only stop that one, as I need it for nvim), pipewire and pipewire-pulse (those two are disabled using systemctl --user disable pipewire.socket and systemctl --user disable pipewire-pulse.socket). Without any models loaded, at idle, I only have 154MB of memory used. With that configuration tinyllama on the question 'why is the sky blue' I get 13.02 t/s on my rpi5, so nearly 1/3rd faster than with all the unneeded services

Nice. I love your Hal replica. Was that done with a Raspberry Pi?

@@IanWoottenYes, a 3B+. I'm working on a JARVIS that uses ChatGPT API and I'm interested in preforming the AI function locally. That's why I'm looking into Ollama.

That would be great to try out if I could get my hands on one.

Well, how about a pi-cluster where one node runs ollama and one runs a screen reader ssh'd into the ollama node? Could add another layer and have another node running NLP for the screen reader node, or a series of nodes connected to animatronics and sensors.

You can run meta whisper alongside your model from what I read. t-t-s and s-t-t

Just using the microsd. I'd imagine speeds would be a fair bit better from USB or nvme.

Not really - a model file is downloaded to the machine using Ollama and brought into memory.

Not necessary, but may be faster. All the experiments here I was just using a microsd.

​@@IanWootten That's just amazing to me. I have a Pi3, but am planning on upgrading to a pi5. After I saw your video, I downloaded ollama onto my windows pc. It only has 4 GB RAM, but I will still able to run several models!

You mean the ollama one? I'm on there from time to time.

@@IanWootten Just posted this video there. Glad to know you're part of the community.

You probably could, but it wouldn't give a quick enough response for something like a conversation.

I am working on something similar, but using a Pi4 for STT & TTS (and animatronics) and a dedicated Pi5 for running the LLM with Ollama like Ian demonstrates. They are on the same network and use MQTT for communication protocol. This is for robotics project.@@IanWootten

I've been trying to do this, but its impossibly hard to get tts setup.

@@isuckatthat Yes not easy. I was trying to implement speech with Google Wavenet using RU-vid Data Slayer example. I put the key reference in pi's user.profile as export. Script runs okay now creating the mp3 files but no speech so pretty much gave up as other fish to fry.

​@@isuckatthathave you tried espeak? It would give robotic quality output but uses very little processing and works fine on a Pi

Yes, I've mentioned this in my other videos and have in my blog on this too.

@@IanWootten ah, sorry hadn't seen that. Anyway thanks again for the video! I've subscribed to your channel as looks great 🙂

No doubt they will do. But, the Pi 4 was released 4 years ago, so you might have to wait a while.

That's wishful thinking. You might as well try to figure out how to run an ADLINK Pocket AI on a Pi 5.

Maybe one of the smaller models, but it'll run a lot slower than here

All the text model responses are in realtime. I've only made edits when using llava since there was a 5 min delay between hitting enter and it responding...

A Coral module is not suited for this. It lacks the available Ram to really partake in helping an LLM run. What you really need is something like an external GPU, something like one of those ADLINK Pocket AI GPUs to hook up to the system, BUT it only has 4GB Vram.

Could do, but I don't think Ollama would be able to leverage it, plus it's not out yet.

Happy to give it a try if there's one going spare!

EXACTLY!

A compute module is a RPi in a slightly different form. So I think it would behave the same.

Others have mentioned it in the comments too. It is a much smaller model, but there are many others to choose from (albeit possibly slower).

Less than 400KB. Might try a jetson nano if I get my hands on one.

To satisfy my curiosity - to see whether it's technically possible on such a low powered, cheap machine.

thanks for the genuine response :D Yes I can see that drive now. @@IanWootten

There's lot of videos of people running it on their PC, but if you use it all the time it will hog your PC all the time. There's several reasons you might want a dedicated host.

The response for that one model/prompt may have been, but there are plenty of others to choose from.

Not tried it out yet.

Possibly if you already have a machine. This might work out if you don't. Power consumption is next to nothing on the Pi too.

The coral TPU isn't suited for this, it lacks the available Ram to do any good with an LLM. What you'd need is one of those ADLINK Pocket AI GPUs but it only has 4GB Vram.

@@madmax2069 makes sense.