Downgrading My GPU For More Performace 

As long as I know, llama-cpp can use tensor cores

He's a Pro, don't tell him he's wrong 😂

you just turinged him on

😂

holy molly bro, 45? any link or tip to get one that cheap? ty and hope u enjoy it x)

​@@yungdaggerdikkk Newegg has them at about that price

Running Stable Diffusion does it run out of vram at 12gb or at 24gb? The tech docs claim the system is 2 systems of Cuda and vram etc...

I am looking into buying a Z840 also, how are you able to keep the P40s cool enough?

@@jaffmoney1219 Air ducting and cramming the PCIe zone intakes to 100%. If you buy the HP branded P40s supposedly their BIOS will tell the motherboard to ramp the fans automatically. I'm using a pair supposedly from PNY so I don't know.

@@KiraSlith Hello! Can you make a short video on how it works for you from the side of hardware and a language model such as LLAMA? If you can’t or don’t want to make a video, you can briefly describe here your hardware configuration, and what is better to take for this? I'm looking at an old platform 2011-v3 18-22 core cpu, gaming motherboard from asus or asrock with 128/256gb ddr4 ecs ram. At first I wanted to buy a modern video card RTX 30xx / 40xx line, but then I came across Tesla server accelerators, which have a large amount of VRAM 16/24/32 GB which we have about 150/250/400 euros Unfortunately, there is somehow little information, and if you come across videos on RU-vid, then people start stable diffusion, which gives very deplorable results even at tesla V 100, which the RTX3060 bypasses. Thanks in advance!

​@@strikerstrikerson8570 Sure, when it comes down for maintenance next. It's currently training a model. If you want new cards only and don't have a fat wallet to spend from, you're stuck with Consumer cards either way. Otherwise, what you want depends entirely on what your primary goal is. Apologies in advance for the sizable wall of text you're about to read, but it's necessary to understand how to actually pick a card. I'll start by breaking it down by task demand: - image recognition and voice synthesis models want fast CUDA cores but still benefit from higher core counts, and the larger the input or output, the more VRAM they need. - Image generation and voice recognition models also want fast CUDA cores, but their VRAM demands expand exponentially faster. - LLMs want enough VRAM to fit the whole model uncompressed and lots of CUDA cores. They aren't as affected by core speed but still benefit. - Model training always requires lots of VRAM and CUDA cores to complete in a reasonable amount of time. Doesn't really matter what the model you're training does. Some models bottleneck harder than others (though the harshest bottleneck is always VRAM capacity), but ALL CUDA Compute capable GPUs (basically anything made after 2016) are able to run all models to some degree. So I'll break it down by their degree of capability, within their same generation and product tier. - Tesla cards have the most CUDA cores and VRAM, but have the slowest cores and require your own high CFM cooling solution to keep them from roasting themselves to death. They're reliably the 2nd cheapest card option for their performance used and the only really "good" option for training models. - Tesla 100 variants trade VRAM capactiy for faster HBM2 memory, but don't benefit much from that faster memory outside enterprise environments with remote storage. They're usually the 2nd most expensive card in spite. - Quadro cards strike a solid balance between Tesla and Consumer. Fewer CUDA than Tesla but more than Consumer. Faster CUDA cores than Tesla but slower than Consumer. More VRAM than consumer, but usually less than Tesla. Thanks to "RTX Experience" providing solid gaming on these cards too, they're the true "Jack of all trades" option and appropriately end up with a used price right in the middle. - Quadro "G" variants (eg GP100) trade their VRAM advantage over consumer for HBM2 VRAM at absurd clock speeds, giving them a unique advantage in Image generation (and video editing). They're also reliably the most expensive card in their tier. - Consumer cards are the best used option for the price if you want bulk image generation, voice synthesis, and voice recognition. They're slow with LLMs, and if you try to feed them a particularly big model (30b or more) will bottleneck even more harshly on their lacking VRAM (be it capacity or speed) and potential to bottleneck even further paging out to significantly slower system RAM.

Stuffed a z440 mobo into a 3u case, will be putting 2x p40s in here shortly.

Dude you’re loaded 😁$

Definitely not overkill if it's for professional use

Yes, as long as it supports above 4g decoding

i'm glad you were able to pick up a p40 and not the m40 since pascal arch can run 4bit modes which is most llm models but llm's changes so rapidly i can't even keep up myself but i have been running the docker container for github.com/Atinoda/text-generation-webui-docker . but yes this is a deep rabbit hole i feel your pain

Easiest out-of-box apps for running local LLMs are GPT4All and AnythingLLM. Huggingface requires lots of hugging to not sink into rabbit holes :) The apps like I mention keep things simple. Both have active Discord channels that are helpful too.

Remember how much it was at the time?

😂

With them having the choice of worst wood, no thanks

That's right, the Tesla M40 and P40 use an EPS (aka "8-pin CPU") cable, which can thankfully be resolved using an adapter cable. Just a note, the 6-pin PCI power to 8-pin EPS cables some chinese sellers offer should ONLY be used with a dedicated cable run from the PSU to avoid cable meltdowns! Thankfully this isn't an issue if you're using a HP Z840 (which also conveniently solves the airflow issue too), or a custom modular PSU with plenty of PCI power connections, but it can quickly become an issue for something like a Dell T7920.

That's exactly my question.

Cost:Performance...

Thank you, I have been looking for this info

Does it use the whole 25Gb VRam, because it's basically 4 cores put together, is the Vram working as 1?

Mining has been proven to not cause any more significant wear than regular duty cycles.. In fact, in some situations the mining rig would be a cleaner and safer environment than in a PC case, on the floor in some persons home with toddlers sloshing their chocky milk around, for example 😂

M40 and P40 GPU are single CPU

I think you are talking about K80 gpu.

Owner of P40 and 3090 in the same PC. No problems whatsoever, just install Studio driver

My M40 idled at ~30 watts, P40 is closer to 20

look for an HP z840, but buy a GPU separately because you are probably going to pay way more if included.

Yes

For training or FP16 inference get the P100, it got decent FP16 performance, the P40 is horrible at that, it was specialised for INT8 inference@@b_28_vaidande_ayush93

I have a video on this topic with using tdarr

All tesla cards can play games, the problem with those is the cooling because there is no heatsink fan, you have to either buy your own 3D printed shroud or have a server that shoots air across the chassis