LocalAI LLM Single vs Multi GPU Testing scaling to 6x 4060TI 16GB GPUS 

I appreciate that!

Welcome aboard!

No, but I def could put one together. Thanks for the idea!

I agree it's about LLM qaunt/size much more than performance. I don't have any AMD cards to test with, but would if I did. I do have several generations of Nvidia cards I might run through some tests with.

The RX 7900 XT (not the XTX) should be about on par with the RTX 3080 TI to 3090 (around 23 tk/s) info sourced from a Reddit post (can't send links on YT sadly). For a point of comparison my RX 6700 XT hovers around tk/s. Hope this helps, and isn't too late of an info :)

One number is missing, thanks it helps for me! ​@@blackhorseteck8381

Thanks! It did cost a bit, but not as much as my other node you folks haven't seen yet - coming soon!

@@RoboTFAI what single card would you recommend, considering I'm looking for the 75% - not state of the art mega bucks.... but able to keep up. 4080? will 4070 have hope? (specifically thinking huge context)

Hmm always depends on use case, and models you want to run - and if you have lots of RAM to load into if not enough VRAM. I don't have a 4070/4080 to run tests with.....but I could run some tests of 4060's vs 4500's (more enterprise level that can be had fairly cheap these days for what they are)

@@RoboTFAI you don't think a 4080 would have been cheaper than 6x 4060's? ... or hindsight?

I am sure it would have been if looking for large VRAM single card with speed, however that is not my specific use case with these cards. I use these to run multiple smaller LLM models (7b/13b) in parallel and not concerned about having them move as fast as possible just decently. Agent based/bot workflows/etc and playing around. For much larger models I use in my daily work, I have another GPU node with several A4500's (20GB VRAM) for running 70b+ models at speed. That's why I run this all in Kubernetes so multiple services can use multiple LLM models, across different GPU's as needed in parallel. And yes I am crazy....

Yea I run mostly 70b+ models in my work/playing for more serious things. It however depends on the quant, and how much context you set the model up for. Newer Llama 3 models with huge context will eat your VRAM real fast. If you mix offloading to VRAM, and RAM (slower) you could run some very large models.

Sure I do have the specs listed on the app, but happy to do a follow up video on the lab machines I use for these type tests and the hardware involved for doing that many GPUs on a single node.

Thanks for replying! I look forward to watch your new videos, cheers!

Yes! Most software for running LLM's/training/etc support splitting the model between multiple GPUs, the newest version of LocalAI (llama.cpp under the hood) even support splitting models across multiple machines over the network.

I could throw together a quick test with some of the 4060's and codestral 22b at different quants levels. Tokens per second isn't just about that hardware, but also model (and model architecture MoE/etc), and context size, etc, etc.

Certainly can show some tests with bigger models, if you have specifics let me know! For this test 13b was just enough to fill one card and show results as split. Happy to do models vs model comparisons also.

I go over some of the specs of my GPU nodes in other videos, but plan to do an overview video on each node soon!

@@RoboTFAI that would be great thanks for the response :)