my AI model box

Подписаться 222 тыс.

Просмотров 24 тыс.

50% 1

Setting up AI models on the DAS and speed comparisons - visual studio / virtual machine tests.
Temperature/fan on your Mac: www.tunabellysoftware.com/tgp... (affiliate link)
Run Windows on a Mac: prf.hn/click/camref:1100libNI (affiliate)
Use COUPON: ZISKIND10
🛒 Gear Links 🛒
* 🍏💥 New MacBook Air M1 Deal: amzn.to/3S59ID8
* 💻🔄 Renewed MacBook Air M1 Deal: amzn.to/45K1Gmk
* 🎧⚡ Great 40Gbps T4 enclosure: amzn.to/3JNwBGW
* 🛠️🚀 My nvme ssd: amzn.to/3YLEySo
* 📦🎮 My gear: www.amazon.com/shop/alexziskind
🎥 Related Videos 🎥
* 🌗 RAM torture test on Mac - • TRUTH about RAM vs SSD...
* 🛠️ Host the PERFECT Prompt - • Hosting the PERFECT Pr...
* 🛠️ Set up Conda on Mac - • python environment set...
* 🛠️ Set up Node on Mac - • Install Node and NVM o...
* 🤖 INSANE Machine Learning on Neural Engine - • INSANE Machine Learnin...
* 💰 This is what spending more on a MacBook Pro gets you - • Spend MORE on a MacBoo...
* 🛠️ Developer productivity Playlist - • Developer Productivity
🔗 AI for Coding Playlist: 📚 - • AI
Repo
github.com/open-webui/open-webui
Docs
docs.openwebui.com/
Docker Single Command
docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main
- - - - - - - - -
❤️ SUBSCRIBE TO MY RU-vid CHANNEL 📺
Click here to subscribe: / @azisk
- - - - - - - - -
Join this channel to get access to perks:
/ @azisk
- - - - - - - - -
📱 ALEX ON X: / digitalix
#machinelearning #llm #softwaredevelopment

Наука

Опубликовано:

18 май 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 89

@aliBoumedyen 28 дней назад

Never bored with this crazy experiments 💜

@JiBe128 27 дней назад

Thanks for your videos ! Love them. It would be very nice to get your review on a Sinology NAS, I am thinking of buying 1 of those.

@falklumo 3 дня назад

There are USB4 cases for M2 SSDs with sick performance which work like TB4 devices on a mac. I bought some high quality yet affordable ones from Alibaba with Aluminum case acting as heatsink.

@mahesh5452 14 дней назад

Great stuff

@HMexperience 27 дней назад

My exact same experience. My new laptop is way too small for AI models. I can only do a few 8B p. models before my SSD is full. Cloud based models will not go away anytime soon they are better and they are fast despite being in the cloud.

@froggy5967 27 дней назад

Easy Alex. Just get a 8TB M4 Ultra next time 😂

@rithikkumar7683 28 дней назад

Please make a video which model a software dev should have and others model can be may have, because not all can have resources for this , thanks

@le_bouvier 28 дней назад

Get one of the Ugreen NAS. If you get teh 6 bay you get 6 3.5 Drive Bays, 2 m.2 Slots (aside from the OS drive) Finally it has Thunderbolt 4 ports in addition to the 10 Gbit ports

@AZisk 28 дней назад

ordered :)

@eriglac 28 дней назад

oh yeah, totally. i would run it on TB if you can afford to buy TB drives. for a poor grad student like myself, it’ll just have to be a makeshift NAS and external drives.

@zezhenxu9113 27 дней назад

Do not buy ugreen nas, they suck

@AZisk 27 дней назад

@@zezhenxu9113Ive heard “they suck” about every piece of gear I use from one person or another. What are your reasons?

@eriglac 27 дней назад

They suck because I'm green with envy that I can't afford them. I don't know if ugreen with envy too. Ugreen should have an Envy line of products but I think hachpee got that covered. Thank goodness it's not Compaq or eMachines haha. Sorry I couldn't help it. Seriously though, wish I can afford that ugreen NAS. Have to make do with proxmox and truenas.

@sveinjohansen6271 27 дней назад

Just wait for the 400b model coming soon hehe

@kilobitz8639 28 дней назад

Great video.

@AZisk 28 дней назад

Glad you enjoyed it

@trenxnet 28 дней назад

🤣🤣 I had the same problem and configured a NAS with some n100 mini pcs, then it wasn't enought so I got a new PC with a 4090 and like 16TB storage. LLMs are the perfect excuse to need storage.

27 дней назад

I wonder if a external storage with network connectivity would be fast enough. You could match it with a VPN like Tailscale and have your models available anywhere.

@AZisk 27 дней назад

i’ll let you know when i get my NAS :) although might need to upgrade my network first

@EricHarmon67 22 дня назад

Would you suggest the Samsung SSDs with or without the heat sinks for that particular setup?

@RomPereira 28 дней назад

Proxmox + truenas on an inexpensive mini pc (intel n305, if not thunderbolt) with 2.5 Gbit ethernet with thunderbolt or USB 3.2 port eith this DAS box.

@AZisk 28 дней назад

i thought about doing this, but then just ordered the new ugreen nas instead :)

@RedDragon72q 27 дней назад

you can buy the SD card adapter that allows an SD drive to be inserted sideways. I did that and put a 2T SD in there and with that card the the 2Tb SSD in my M3 I have a ton of room for models on the SD card. Love it.

@AZisk 27 дней назад

what model do you have?

@RedDragon72q 27 дней назад

@@AZisk M3 Pro 16 with the Max chip and 64 GB 2TB. I bought this to hide the SD card. BASEQI UHS-II Aluminum microSD Adapter for 2021 M1 MacBook Pro 14 & 16” (Silver) Model USHii-420A

@DanielHarrisCodes 26 дней назад

@@RedDragon72qWhat are the speeds like on it? I got a Transcend JetDrive for my M1 Pro MacBook and TBH haven’t really used the storage for anything. It’s too slow for most things but it’s there if I need it for storing large files. I keep a backup on my Parallels VM on there but it’s too slow to actually run from

@RedDragon72q 26 дней назад

@@DanielHarrisCodes standard speeds for an SD card, maybe a bit slower on read for some reason. I keep long term files and models on it. Loading the model takes a bit longer but once it it loaded you're all good.

@carloseduardoalmeida6469 23 дня назад

Hey Alex, great content! Would love to see some practical examples of what you have been using LLAMA for. Don’t know if I got it right, but what are the advantages over using web ChatGPT, for example?

@falklumo 3 дня назад

You can loop it into your own app.

@AlmorTech 27 дней назад

No way, how big SSD is big enough for you, monster! 😅

@Winnetou17 27 дней назад

LoL I can't believe it! 4 SSDs in RAID 0 for gigantic speed, only to be bottlenecked by the 10 Gbps USB transfer rates :)) If that wasn't a bottleneck, those 4 drives, if they were decent PCI-E 3.0 ones, can go over 10 GB/s (that is gigaBYTES). Fast PCI-E 5.0 ones could probably go over 30 GB/s (I remember Corsair has a 10 GB/s SSD, so 4 of them + a bit of overhead should be able to do 30 GB/s). Anyway, the thing that triggered me was that Apple's SSD is much faster at 4:19 ... I really doubt it is. Compared to 4-RAID 0 normal SSDs, that is. Also the breakdown at 1:37 is pure gold. Thanks Apple! Edit: ok, wanted to check something and rewatched a bit. No mention of that USB 3.2 what type it is, but from the end tests on the Windows VM, reaching over 3 GB/s, makes me think it's actually a 20 Gbps (USB 3.2 gen 2x2 F*** the USB comitee for these absolute i-diotic names). Still, 20 / 8 means only 2.5 GB/s theoretical, more like 2.0 GB/s practical so where's the 3.3 GB/s coming from ? Not sure. Also realized that the SSDs are Samsung 980 (not Pro), which is PCI-E 3.0, so around 3 GB/s each (it even says it on the box at 2:46 ). So the mention at 3:27 "It's only USB 3.2 But you don't need than 'cuz the fastest drive in there is gonna be uuhm 1 GB/s" is VERY wrong.

@razorgarf 27 дней назад

why so many different AI models though, would be interesting to know what sets them apart

@smaad7 7 дней назад

is there a way to manipulate llm on thw cloud and dont have them on the machine storage ?

@eriglac 28 дней назад

haha. omg alex, seriously put your stuff on a NAS or an external drive. i put everything either on NAS, Dropbox (if i need to share with my lab), or on external drive (spinning disks). have you considered doing a hackathon for those near you?

@AZisk 27 дней назад

i have a dropbox subscription. i’m sick and tired of the costs associated with it, and the lack of immediate availability of my data. NAS is next

@DS-pk4eh 24 дня назад

Just download more storage (and RAM)?

@abduislam23 27 дней назад

So using this solution, I should not care about space customization while making purchasing decisions?

@ElbayMalik 28 дней назад

What is your old time machine? Could you show us?

@AZisk 28 дней назад

yes, i’m considering making a vid

@tibbydudeza 27 дней назад

Quick question - how many tokens per second do you get on say 8B and 70B local LLM on the Mac ???. I want to buy a server dedicated to LLM but adding an NVidia GPU to my PC is not what I had in mind - currently have a Radeon RX 6600XT - it spins up and makes a loud noise when using Ollama.

@falklumo 3 дня назад

Very fast for 8b and about a few/s for 70b on a M1 Max 64GB. Fast enough - but only with the default 2k context. With the derived llama models allowing large context windows and a 250k context, llama 70b almost comes to a halt, a token every 10s or so… Note that an Nvidia GPU meant for gaming probably can’t run 70b at all because of a lack of VRam.

@terencedodge3249 28 дней назад

So much fun…

@tutacat 24 дня назад

Why keep below 34b than 7b. Or just keep the quantized version, you can delete or store in 16bit/8bit

@OlegShulyakov 27 дней назад

Some day you’ll just buy Synology

@max75025 27 дней назад

why ollama not LMStudio?

@williamsquires3070 27 дней назад

Now put a sign on the black box that says, “do not feed the A.I.” 😀

@AZisk 27 дней назад

🤣

@ericy91745 28 дней назад

Why not use services like Backblaze to increase your cold storage space? Yes, you don’t get the convience of local redundancy, But it’s cold storage! If local HDD fails, get the copy online.

@AZisk 28 дней назад

Ideally I should, but I don't like paying monthly storage fees.

@BelarusianInUk 27 дней назад

For your sd raid 0 you are limited by usb3.

@gadaao 27 дней назад

وماذا عن كمية الشحنة الكهربائية داخلها كيف نعرف

@edvardasjuodakis7644 27 дней назад

Why not to just remote desk into a desktop?

@Scarrus666 28 дней назад

That's a lot of money for "only" computing.

@tutacat 24 дня назад

Fine tuning doesn't mean software development.

@_jerieljan 27 дней назад

If you're eating that much storage, then yeah, you should really be offloading them when not in use to a NAS or external media. It's not like you'll use all these models and whatever quantization or version they have at all times, right?

@mattisrensen9162 27 дней назад

Why use a das when you can use a nas, so you can also stream films and series + run your vms

@AZisk 27 дней назад

already ordered

@dtesta 28 дней назад

Wait wait wait! Hold up! So you are using usb 3.2? So maximum 20gbit, giving you like maximum 2500mb/second. Slower than what ONE of those nvme drives can do! What exactly do you think you gain by putting them in a stripe raid???

@DS-pk4eh 24 дня назад

Probably the total capacity of all 4. Maybe a bit better than just JBOD.

@dtesta 24 дня назад

@@DS-pk4eh With JBOD, he would not lose ALL data if one drive fails. The stripe raid give no benefit at all in this setup. Stripe raid is for maximising throughput at the expense of seek-time, as all drives needs to seek for one read. Does not hurt as much on SSDs of course, but still hurts.

@itzhexen0 28 дней назад

Wow, look at that shit.

@AZisk 28 дней назад

Check it out!

@AndreasMolnar-Dev 28 дней назад

Why didn't you get a dedicated AI server?

@AZisk 28 дней назад

if i build out a server like that, i’ll want to spec it out with nvidia stuff, and i’m waiting to see what the 50xx series do

@adrimathlener8008 28 дней назад

remember Bill Gates: Here’s the legend: at a computer trade show in 1981, Bill Gates supposedly uttered this statement, in defense of the just-introduced IBM PC’s 640KB usable RAM limit: “640K ought to be enough for anybody.”

@sativagirl1885 27 дней назад

Alex, you need to show #AI who is THE BOSS (you). Put each LLM on a 2TB ext. USB so they don't conspire to take your fame & fortune and go to Las Vegas to gamble with other #AI

@aeonlancer 28 дней назад

I guess professional video editors are the piggest ones

@asksearchknock 28 дней назад

RAID 0 is not raid… the clue is in the name 😂😂😂😂

@AZisk 28 дней назад

lol. i suppose we can just call it AID :)

@asksearchknock 17 дней назад

@@AZisk I have at one time or another used: Risky Arrangement Inviting Disaster Really Awful Idea for Data Reckless Architecture Ignoring Durability Reliably Arranging Imminent Deletion