How to Choose an NVIDIA GPU for Deep Learning in 2023: Ada, Ampere, GeForce, NVIDIA RTX Compared 

How did you find one for $200? That's amazing! I'll give it some more time to see if i can find one around there soon to upgrade from my RX 470.

Can you deliver some numerical data on speedup? In terms of training speed, I mean

@FiveFishAudio does it supports cuDNN and what do you think about it for a lap (the same one)

@@gnanashekar Yes,RTX 3060 supports cuDNN. I compiled OpenCV with CUDA 11.7, and cuDNN 8.7.0, running on Linux Ubuntu 22.04.02

@@lp67O Depends on model complexity and size of your dataset, but yes it speeds up a lot compared to CPU only or my old GTX 970

Yes it has 12 GB of ram, but they cheaped out on it, the ram is only rated for 192-bits. It's why if you look at the 3070 it has 8GB of 256-bit ram, but the ram is faster and more efficient making up for the smaller number and evidently being better than the 3060's 12GB. People complained about the older ram in the 3060 so they released a new 3060ti that has 8GB of the newer and faster ram. Honestly both cards are good, some say the 8GB one will preform better, others claim the older 12GB one works better on their machine.

@@Jake___W I'm not talking about the 3060 Ti having only 8GB - I mean the plain old 3060 in some versions only has 8GB. Say what you will, the 3060 12GB for ~ $280 is good value for money for machine learning. You can have the speediest card in the world, but if your model won't fit in available RAM you are out of luck as well as memory.

@@datapro007 slower speed for me personaly is okay. I eventualy choose RTX 3060 12GB DDR6 over RTX 3060Ti 8GB DDR6X. so many reviewer says don't buy RTX 3060 in 2021, and recommend to buy Ti instead. but it's seem their recommendation is changed to favor 12GB now, lots of games now need more than 8GB, and also for AI stuff, more VRAM Size is important as a matter of go or no go, not just slow or fast.

@@marhensa is i5 12 h 16gb ram intel iris good enough

Oh thanks, they do exist!! I was curious about that one. I tried to obtain one, but had failed in my attempt.

I did some testing on RTX 3x graphics card by training Classifier with and without Mixed precision (on Tensorflow). And I found that mixed precision actually slowed down training on RTX 3x cards (2.5x slower). I think it's because the RTX 3x uses tensor float 32 to calculate, so there are no need to use float 8 or float 16, the operations that convert float 32 to float 16 slowed down the computation. Or maybe I did something wrong.

is this even real life? I'm in westcoast Canada and nowhere near around that price range we could get 3090 for. by any chance you could share a link of some of these 3090s?

@@Gogies777 keep looking!! Can confirmed I just bought 5 3090s at 700$ cad

Where did you catch the Canadians to trade?

Were they stolen? 🤔 Which brand? I mean, the average price for a RTX 3090 is about $1600 USD today. The cheapest I could find was $1050 USD. That's about 1430 Canadian dollars for the cheapest! Brand new for half the price of the cheapest brand available sounds unlikely, since it would be a loss for the retailer.

Just got a used 3060 for $300 :)

buy 4090, 24 gb is the minimal amateur size when you actually can do stuff with limitations.. 40 and 80 gb are pro memory sizes.

@@denysivanov3364 Thank you !

Are your 3090s second hand? I cannot find any 3090 for that price even now.

i agree, the memory usage/dependency will change... do you think the 4070ti is a good purchase or shell out more cash for more cuda cores on the 4080/4090 or wait? 4070ti is my budget but im not in a rush

@@xphix9900 It depends on your use case. If you are buying a GPU for DL then go for RTX 4090 (if your budget allows) solemnly for the VRAM of 24 Gb. You don't want to be in a situation where you face the VRAM bottleneck whenever you are running a bigger network training. My suggestion is wait and go for 24GB card whenever your budget allows.

@@aayushgarg5437 thank you sir, appreciate the advice, and i agree!

@@aayushgarg5437 also just to get something running for now to use NERF, would you have a suggestion?

@@xphix9900 I don't know much about neural 3D rendering. Apologies.

Water-cooling is seldom worth the time/cash investment unless wanting to go for maximum performance while staying on ambient cooling. I'd say it's potentially a 5% performance improvement for 15-25% more cost (relative to the GPU base price). The only application I'm aware where this is useful other than satiate ones need for tinkering is when trying to squeeze maximum performance out of a single GPU for gaming purposes for instance. I don't see the value for ML purposes unless going for absolute performance or if noise is a main concern, since water-cooling tends to be less noisy.

You are welcome, from the states!

It is fully compatible with cuda

same here. I was thinking about getting a 4090 but now I think it'd be nice to save some money

@@danieldaniel625 for what it's worth, I ended up getting a used 3090. I paid around 800 eur + 40 for buyer protection on the platform I used. Overall, I'm happy with everything!

@@xntumrfo9ivrnwf Lucky. I went this route and mine was DOA.

4070 Ti with 12 GB would be enough for any Deep learning ?

What compatibility issues have you seen with 40 series? I have not yet encountered issues.

For a lot of these professional applications, performance is directly proportional to VRAM bandwidth and the TFLOPs are well enough to not matter at all, as it's bandwidth-limited. More VRAM is indeed very desirable. I hope we will soon see cards with 96GB GDDR6W, and the leaked H100 PCIe with 120GB HBM.

Yeah I need at least 48 GBs of vram, any less and I'm like ehhhhhhhhh

i'd be looking at my pciex16 slots and if they share lanes. usually port 1 has direct separate lanes to the processor. port 2 and 3 are usually connected or share lanes, for instance pciex16 port 3 is directly connected to processor and port 2 runs through port 3. How i'd use it? port 1 would have my rendering card like a RTX 4080. Port 3 would an A5000 because its tremendously fast, now port 2 i'd probably install a usb 3.2 card. then make a external rack of say 4 NVidia k80's all doubled up and plugged into port 2's usb to pciex16 slot because it will be like giving my A5000 an extra 96gb of ram for like $700. So if you needed a crazy amount of ram to load terabytes of data sets this is what i'd be looking at and my electric bill. At least making the ram boost external gets heat away from my more expensive hardware and i can simply plug in however many stacks i need and power up what i need for the task. Its more of a modular approach to cutting costs with electric and heat. hopefully that gives you a better idea of board architecture and how you'd want to look at it. There is software like octane render that can link all that ram up and make it work together. So looking at it like well i'll just run out and buy the BEST isn't always bright. For instance i can buy 1 4090 but i can buy 2 4080's for the same price and i'd gain 8gb ram and 4k cuda cores and double the speed of processing. Its all perspective and knowing what you'll need and why, compatibility for use might be a major issue too! You might not want a card more for rendering at all as its a mining rig setup or solely a ai computational setup, but i'd say keep in mind that having a rendering card is beyond useful for video editing and a multitude of other tasks that these cards won't be. If you're go doing something insane with lots of heat, shell out the money for a more expensive and heavier board then, anything up too the $500 range on a mobo is going to be a much heavier built board, which means better soldering metals and thicker plating and oh boy does it matter when you're shoving over 5,000 watts through it. Now if its separate entirely from your main pc, like a server build then definitely take a look at the micro boards with multiple processors. Those can actually be setup between a stack and the main controller pc (your rendering or personal use pc). Having the extra processor lanes allows you to direct traffic of the information as you can have 4 processors on one board that are moving information in/out giving a higher efficiency of data transfer through the network you've now built. If you got enough money and understand the architecture, you'll realize you can do anything and the market is a blessing. Other important things to look at would be your processor's ram and its latency for much higher speeds and the ability to use nvme memory cards directly installed to those pciex16 lanes for even faster computing(it makes a massive difference).

1x pcie does not work in my case, i need at least 8x or the training time will x3 You need to go with epyc, threadripper or xeon to support them. X299, x99 or x399 mobo will be the cheapest.route. e5 v4 cpu is really chip with dual cpu x99 chinese mobo

Newer architecture gets work done faster with less power draw meaning that even with the lower core count, the 4070 will out perform the 3080Ti. I'd recommend the 4070 SUPER as it comes with a higher activated core count and an even higher clock.

Cost of 4060Ti is much higher than 3060Ti, but the numbers of Cuda core is not worth... I think can consider get a second hand 12GB 3060Ti.

That is an interesting system, it looks like you could get support for PyTorch through ROCm, though you would be dealing with Linux at that point. If your willing to deal with somewhat more complex installs, and maybe some features not working, it could work out.

Nice to hear, I always thought that would be a decent entry point GPU.

No, that's a really old toy and have no image display.(compute compality 6.1)

@@dongyangli3985 sorry I don't understand why I need image display since I have already one in my system.

RTX 3090 seems the best value by far. Large 24GB VRAM which is 93% as fast as the 3090 Ti/4090, but 100W/22% less wasted power and half the cost of a 4090.

4090 provides very good value, forget about 3090. 4090 also much better for transformers inference.

@Zach absolutely not. Especially since VRAM capacity and bandwidth are the same. 3090 is the better option.

@@ProjectPhysX 4090 is faster than a100 for inference, especially for transformers.

do you have experience with multi gpu training? is it easy to put them to work together as 1? I dont have experience with it, but since in pytorch you have to move the tensors around im not sure how easy will be to train a network in several gous at once even though pytorch offer managemetn for it themselves

Yes, absolutely, 16GB ram and 6k CUDA cores. It would be a fine card for desktop machine learning.

motherboard ram can't be utilized for gpu work loads

@@generalshepherd457 With M1 Apple silicon we can, coz the RAM is in the SOC and not on the motherboard.

@@DJpiya1 sounds skethcy. would not trust tim crook.

The M1/M2 does share GPU and system RAM, I've observed this on my own Mac.

Oh wow, you are totally awesome! Thank you so much!

3090 all day every day, in my opinion. As far as nvlink... Not sure how well that works for pytorch etc. (I think the professional cards are the ones worth the drivers to take advantage of that). Definitely do more research about linking the GPUs. Make sure you have a good PSU (1600w) if u have multiple 3090s in a box.

Good points bigdreams!

@@bigdreams5554 if nvlink can’t work should I switch to something else? 3090ti, 4080, etc? Are the amd gpus even worth considering?

no NV Link or P2P over PCI-E support on 4080/4090. The last consumer card that support NV Link and P2P is 3090. If your model need VRAM, 3090 x2 is better. If your model fits within 24GB, single 4090 is probably close to dual 3090 performance.

"Given" ?😭

3060 12gb is more than enough to play with SD, unless you want to train large models

K80s have 1/3 the CUDA Cores

Absolutely ture. And 3090 may be a good choice.