Replace Github Copilot with a Local LLM 

On macs the RAM is shared

Is it possible to do a partial GPU offload, or is it everything or nothing ?

Yup. On LM Studio PC you set the number of layers, the higher the number the more GPU resources used. On ARM Macs yes, RAM is shared, and your option is to enable Metal or not.

​@@matthewgrdinic1238 I always enable Metal 🤘

and also for that reason, you pay for copilot (and give your data) or you pay for VRAM... Not really free. Return on investment might be a bit faster on discreate GPU... That you can upgrade...

Glad it was helpful! I believe these tools will become common place in the next few years, it's fun to be here where it all starts.

I agree

Same. I trialed GitHub Copilot for a while but ended up ditching it. I found I was spending time debugging generated code rather than actually understanding problems and solutions. And debugging someone else’s code, for me, is NOT my favourite part of software engineering! 😂

When you really understand how copilot works, and how to actually BEGIN writing and structuring the code, in order to trigger the right completion, then you start speeding things up. At first, I was just like you, but then I got the hang of it, and man, copilot is awesome.

@@cnmoro55 yeaaah, it's amazing when it starts "getting" you.

Had not heard of Neovim before, looks like an outstanding project!

Would you pay $1900 for a spell check co-processor? I know you can do plenty of other things with it, but that's still the price of a 24G RTX 4090. That's a lot of cloud credit.

@@howardjones543 Yeah the hardware needs are expensive atm, but hopefully (and I would think most definitely) that will start to become less of an issue. I think for the people that already have the hardware today though, the choice is very obvious.

Well said, but I wouldn't pay $2000 on a GPU to use LLM only on that computer

@@skejeton Sure but you can play Starfield and encode video for significantly less. THIS is the application that requires all that VRAM - typical game requirements, even for heavy games, don't get near 24GB. The implication at the end of the video is that you would save $10/month with this free local LLM, but that's bending the truth a bit. If things like these new NPU-equipped processors and different models can remove the need for these gigantic GPUs, then it might be interesting.

@@howardjones543 the benefit of freedom tho (and privacy?)

Thank you so much for the sub, it very much appreciated! It's late so I can't recall if I mentioned in this video, but somewhat shockingly the MacBook is actually faster then the 4090 with LLMs. Granted this isn't with the Tensor RT LLM framework (which should give a 2x bump), but for me, shows how lots of RAM and dedicated, modern hardware make Inference not only possible, but surprisingly easy. This bodes well for the future of local AI, and I'm super excited to see the PC space evolve.

Is there any way perhaps we can configure that vscode extension to point to a hosted runtime of the same llms? There are a couple of hosted llm providers that seem to be serving llms of our choice for very cheap prices

Not mentioned in the video, but Continue defaults to hosted LLM's. This video was to show how the local side works, but it's not required.

Ha, same here. I’ve been comfortable with 16/32 go for my Mac’s. But now suddenly I can justify a 128gb monster.

Yessss, would love this. Lightweight app for devices, access to AI in VSCode or image gen locally without being on the physical machine.

That's what I would do and have a TPM or two in it. Used ones come down in price pretty quickly.

You could implement a REST API for a home server. If it only runs on the local network and doesn't talk to the internet at all, maybe on an isolated VLAN, there shouldn't be any security issues to worry about.

Totally possible. I have Win11, Ollama running Dolphin Mixtral, ollama-webui running in a Docker locally (looks just like ChatGPT), a tunnel set up through Cloud flare to a subdomain in my own DNS. It's available externally, securely. All because work blocks Open ai😄 but I can access my own local LLM anywhere. API could be exposed in the same way

A great point - I've got a related comment on the same theme I need to reply to but in short: I *happen* to have this beefy laptop but it's not at all required for local inference! The biggest constraint right now is memory, but their are plenty of (much) smaller models that fit just fine in more modest hardware. Long-term AI's here to stay, and AMD and Intel are already adding AI specific hardware to their chips. Yes dedicated hardware will be faster, but in 2 years local inference on CPU will be more than fast enough.

Mixtral not mistral.

An excellent idea, and the answer is yes! Now I haven't heard of or tried language-specific models myself, but one project I'm super keep to see released is Nvidia's "Chat With RTX". The basic idea being training a model on data *you* provide - for example a book on your favorite programming language. www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/

Codeium all the way for me, outperforms CoPilot AND it's free.

Well said.

Yes there are alternatives which let you use a local inference server with many Llama CPP has one built in too But I'd recommend another alternative Some use ollama as the inference endpoint You could even use others like fireworks or a custom endpoint

Apologies for not listing this more clearly - 36

Yeah. Run locally a 70b model as copilot xD

Running the 33b deepseek-coder model locally was slow enough. (The 6.7b model is REALLY good.)@@merlinwarage

@@merlinwarage If you have a silicon mac, totally possible.

@@camsand6109 it's possible but just not practical. Do you like waiting?

I know right lol. Downloading the smallest 2-bit version now.... Well it runs (36 gig M3, ~16 tokens / second), but...huh. The results are total nonsense. I realize this is the 2bit version but still. I guess even tough the model loads it still needs loads of extra RAM, or the low quantization is more punishing than I've seen in other models. The good news: you can try it here: catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/codellama-70b I will say: from a few tests I ran I still prefer Dolphin Mixtral - didn't expect that but it's a *crazy* good model.

This is an important question. With copilot, it is aware of stuff that I have written in other files for the same project. Dont know if it is just doing so by storing keystrokes, or if it knows to look at the entire project

yes, in vscode copilot you just need to open all the files, which you wnt to have indexed by the copilot. closed files won't be indexed@@ilearncode7365

Ollama + Codellama

There is a chance that copilot subscription cost would increase from the current 10$ to maybe 15 or 20 or more. Each user costs 20$ for Microsoft apparently with the limited features it has now.

You can do that, but calculate the price of the server (for 4-5 developers you will need an 4090 at least, A100 for 10+ devs) + electricity cost vs the copilot's $10-19/month per user plan. Even with the $19 business plan you can use copilot for ~2 years with a 5 person team for the price of a server. ~4 years with the personal subscription.

@@merlinwarage but I have to expose my code to Microsoft and can’t train own codebases. But yes I would expect more like 200 devs on a single A100 so it pays off. Most use it quite sparingly

You bet - 1. Download LM Studio and download a code-centric model that fits on your hardware (Mixtral Instruct or Code Llama are great places to start). 2. In VS code install the Continue Extension. 3. Click Continues extension tab on the left side of VSCode's interface, then click the little gear icon (at the bottom of the screen). 4. Add this code within the JSON's models block: { "title": "Mixtral", "provider": "lmstudio", "model": "CodeLlama7b" } You're ready to go, though please let me know if you run into any issues!

A great question: On PC we can separate and scale between the two, on Mac it's all or nothing. At least on the Mac then, a response from Open Oraca that generated at 28 tokens / sec pinned the GPU for the duration of that specific 3.2 second response time. I'd imagine that;s the case for all responses.

Jetson AGX Orin 64GB wins, lol

Unfortunately I haven't tried this yet, but I'd imagine just as you can point locally networked computers to say, a web server, this would be no different.

With Continue, you need to manually choose which files to upload with each request. I guess copilot does this automatically

A great point, and one I haven't considered deeply yet. At first blush then so long as the code's part of a virtuous feedback loop I'm ok with it. As well, Microsoft has a vested interest in the quality of AI beyond just siphoning data, so again, for now....I'm ok with it.

For 7b models you can use any GPU with at least 8-10GB VRAM. For 13b - 30b models 10-16GB VRAM, For 40-70b models you'll need 16-24GB VRAM. The system RAM doesn't really matter in this case.

@@merlinwarage igpu shares system ram

@@merlinwarage For the $1500 for a 4090 you can buy a LOT of DDR5 memory and make that available to the GPU - Windows does it automagically. It's not anywhere as fast as GDDR6 (like, not even close) but you can get 64GB spare memory quite easily. I'd be curious about how usable (or unusable) that would be.

Why don't you just install and try?

@@kotekutalia I will when I have time I was just curious if someone out there knew

VRAM is the key and yes, that should be more than enough to run Llama 7b.

It is up to you, the driver, to make that happen.

On the Windows side LM Studio let's you run on CPU, "offload" to the GPU, or a combination of both. On CPU only, Llama 7b on a 2 year old Intel 10850H laptop pulled 5 tokens / second (tokens roughly equate to words). Humans generally read between 5 and 10 words per second, so even this "worse-case" is nearly real-time. And that's just now - Intel and AMD are already starting to add dedicated AI hardware to their new chips. Faster helps bus is not needed, and basically any future hardware is going to be far more performant. Copilot is must faster now but, that won't always be the case.

imagine paying microsoft to let you help them train their AI that they intend on replacing you with.

Yeah, but you'll be able to run GTA6 when in comes out on PC.

@@tbird81 🤣

I've been dealing with insurance companies lately. When faced with composing yet another email to yet another repair shop, I finally gave in and had chat to write one for me. Not only was the result more concise and clearly stated than what I was writing by hand, it was far faster. It's little things like that are slowly winning me over and more importantly, freeing up time to do more meaningful work.

@@matthewgrdinic1238 my commiserations that you need to deal with insurance companies at all … some of my first career projects were with insurance companies, so I feel your pain :) Yep agree, I am a big adopter of AI in my workflow too, and it’s been invaluable for web work, particularly dealing with css ! That’s why I’m really keen on setting up a local model, so I can hopefully train it over time to be great at zig development as well. The challenge here is that we are building a new language and std lib which evolves daily, so prior art is neither available or helpful. Exciting times

The smallest language model takes like 1MB of space, so you're blatantly wrong. Note that you said "smallest" and not "best" or even "usable".

Hate to break it to you, it's not perfect.

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO ARGHHHHHHHHHHHHHHHHHHHHH@@tbird81

No…if you already have the tools, it’s a good alternative. If you don’t already have them, then pay for Copilot lol

@@bcpr Yeah that kind of a title and description would make more sense. The way he described it in the video: "superior to paid services [...] results are absolutely worth it [...] local LLM is free (as opposed to paying $20/month)", makes it sound like he's saying that people should consider a $2000/$4000 upgrade over a $20/month plan.

I have 4090 which I bought for VR gaming. Now I know I can use it for local LLMs with good enough output for coding or other tasks, even better. Would I go and buy 4090 to be able to use local LLMs only, hell no.

You bet! Create a web page that contains a blue circle. The ball should be centered on the screen horizontally, and placed at the top of the page. Update this page so that the ball drops to the bottom of the page using a realistic gravity equation written in JavaScript. When the ball reaches the bottom of the screen, it stops. Update the logic so that when the ball reaches the bottom of the page it bounces up realistically like a rubber ball. On each bounce it looses some momentum until it stops completely. Do keep in mind the only model I've found to be truly great at this specific task is Dolphin Mixtral Q3. That is, it basically one-shots everything on the first pass. It also helps to, in LM Studio, to check the box in advanced settings that keeps the entire model in memory Finally and to be clear, the more of a conversation you have with the AI the better. That is, it's totally ok to have it clarify its reasoning, tell it something doesn't work and ask it to fix it - it's very much a case of well, chatting with it.