Low Power Cluster - Small, Efficient, BUT Powerful!

Подписаться 233 тыс.

Просмотров 174 тыс.

50% 1

I built a lower power, efficient, and near silent server cluster! Although this cluster is small and efficient, it's still powerful enough to run a high availability Kubernetes cluster with many services running in High Availability mode! There are so many options with running a small cluster like this, the possibilities are endless!
Video Notes: technotim.live/posts/low-powe...
A HUGE thanks to Datree for sponsoring this video!
Combat misconfigurations. Empower engineers.
www.datree.io
Disclosures:
- I purchased everything with my own money
Intel NUC 11 - amzn.to/43TK8nS
Intel NUC 12 - amzn.to/3MZjC6C
SAMSUNG 980 PRO SSD 2TB PCIe NVMe - amzn.to/41vUOrk
SAMSUNG 870 EVO SATA - amzn.to/3KQMriU
G.Skill RipJaws DDR4 SO-DIMM Series 64GB - amzn.to/3AlstI8
Mk1 Manufacturing - www.mk1manufacturing.com/cart...
See the kit here:
kit.co/TechnoTim/efficient-lo...
Want to set these up with Proxmox? No problem!
• Build a Low Power, Eff...
Looking to se up a cluster yourself? Check out the video and the docs!
• The FASTEST Way to run...
Thanks (Red Shirt) Jeff Geerling for the clip!
/ @jeffgeerling
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a RU-vid member: / @technotim
Merch Shop: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
2nd channel: / @technotimtalks
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - How to Build a Low Power Cluster
00:18 - Hardware Specs
01:20 - Rack Mounting Intel NUCs
02:17 - Remote Control & Power Control
02:58 - Provisioning the Cluster
03:28 - Why Kubernetes?
03:39 - Automated Install of Kubernetes
04:20 - Kubernetes Policy Management (Today's Sponsor, Datree)
05:46 - Installing a High Availability Kubernetes Cluster
06:51 - Testing High Availability with NGINX
08:57 - Introducing Chaos
10:04 - What Can We Do with a Cluster?
10:35 - How Much Power Does This Cluster Used?
11:20 - What Do I think of a Low Power but Powerful Cluster?
Thank you for watching!

Наука

Опубликовано:

22 июл 2024

Ссылка:

Скачать:

Готовим ссылку...

Добавить в:

Мой плейлист

Посмотреть позже

Комментарии : 145

@JeffGeerling Год назад

I like the clean rack mount most of all. I just finished a project with two NUCs and agree on their handy utility! (And I'm glad Red Shirt Jeff has been pestering you lately... finally I can get some work done!)

@sjwright2 Год назад

Nice cameo

@TechnoTim Год назад

Thanks not Red Shirt Jeff!

@gowinfanless Год назад

Hi Jeff,I got you here!!

@2422930 Год назад

Yup the OG Jeff is here ❤

@error-id Год назад

wow see Jeff here 🎉

@TerraMagnus Год назад

Literally how Chick-Fil-A stores run their edge clusters in-store. K8s on NUCs.

@deckardstp Год назад

Thanks for the video. I run a DN2820FYK Celeron now for more than 9 years. It was my entrypoint to Linux on a x64 platform. Evaluated numerous stuff on it. Learned about Docker and so on. NUCs are super reliable from my point of view and worth every penny over time.

@twon4212 Год назад

Crazy you did this! Exactly what i did about 8 months ago. I had a dell R720, readynas, 2 switchs (with POE) AND BOY WAS MY POWER METER SPINNING. Got to the point where I didnt see the need to have such a power hungry server in the house and moved it to a data center where the space and bandwith was far cheaper than my costs to have it at home. But having this 3 node cluster was perfect for my needs and i could still do some low end development on the home lab if need be. Great video man looking forward to the next one!

@l0gic23 Год назад

I'm so glad to see you getting into low(er) power boxes. Excited to watch this!

@fatalsg92 Год назад

This is really good! I was just thinking of deploying a full functioning project locally because I don't have money to maintain cloud services fees while still in development! I think you just saved me my friend.

@RyanJones26 Год назад

My cluster is 3 NUCs for workers and 3 hp mini PCs for the control plane. Been using this setup for a while. Very stable

@rankenfile Год назад

The Mk1 hardware looks pretty nice. Glad to see you having lots of fun in the lab!

@repairstudio4940 Год назад

Found this channel researching NUCs...absolutely awesome content. Liked and Subbed! 🎉

@CohanRobinson Год назад

I'm super glad I found your channel. This is the pure kind of nerdery I need in my life. Love it.

@therevoman Год назад

Nice work! I love those little 4"x4" boxes and have a few in my lab

@FrancescoCarucci Год назад

Great video, being learning a lot from you, thanks for your work. That storinator is using more power than my entire homelab :D

@andrerademacher9701 Год назад

Great video 👍 This makes me wanting to setup a kubernetes cluster myself 😊

@rashie Год назад

👍👍- phenomenal! Thanks Tim!

@dosdaysareover7781 Год назад

Cool to see I’m not the only one clustering mini PC’s. 👍 those small form factor devices are quite powerful lately.

@supernerd6983 Год назад

The unexpected Geerling cameo had me laughing out loud!

@gowinfanless Год назад

such a professional video review,envy of your hardware!

@MrJackPL Год назад

I love your videos even when it is not my game sometimes:) ❤ Keep on being awesome !

@brahyamalmonteruiz9984 Год назад

Love all the low power pc content!

@local-admin Год назад

Love the NUC line. Definitely a real tb port.

@johnyferreira8733 Год назад

That’s pretty awesome. As you can see from my profile picture. I use the little Lenovo Thinkcentre and I use them as proxmox cluster. The SFF desktop below them is my OPNSense box. As you mentioned, they’re not as power sipping as the pi but they’re still very low powered compared to full size desktop or rack mounted servers. I use docker for my containers just because I find it simpler but I’ve been meaning to start using Kubernetes. Again really cool video and it’s cool to see these micro sized computers serve a purpose and in my case given a second life. Keep up the good work and can’t wait for your next video. Cheers!

@seethruhead7119 Год назад

I am thinking to run 3 R86S but as a proxmox cluster the 2 10gig interfaces are really interesting, you can use couple of ways 1x for client, 1x for ceph network 2x for ceph ring network and on top of proxmox cluster you can of course run k8s nodes as well so it seems best of both worlds

@pgm1245 Год назад

Relating to the remote KVM part: there are NUCs with full vPro functionality with remote KVM over VNC. They also got a web interface to power cycle etc. - and there are also command line tools to start them remotely

@MarceloAlcantaraX 10 месяцев назад

The problem is to find them - cannot find them anywhere on the internet (just simply nuc ...)

@Dough296 Год назад

Nice work ! At home I use 6 Lenovo Tiny as Proxmox Cluster to run a Docker Swarm, a K3S Cluster, ...

@TweakMDS Год назад

Love this video, it's exactly into my hobby sphere. A NUC is great because of a small footprint. If you have a rack (and it's somewhere where a bit more noise doesn't matter), you'll get a lot more back for your buck with an actual server or 3. Typically, the dell/lenovo/hp machines are quite affordable, especially if you buy used/refurbished. I'm running 2 lenovo's with a 6-core I5, 64GB of RAM and 2 500GB SSD's each. For a homelab, anything "tiny mini micro" is great, and suits a K8s cluster quite well, but for me the real strength of a homelab comes from building experience in setup + teardown. From that perspective, getting 3 machines to form a Proxmox cluster would make a lot more sense. On this setup with virtualization, you could easily fit 4 or 5 clusters with a controlplane and 3 nodes per k8s cluster, and have a bit more room to play with. In any case - for anyone wanting to build more hosting / devops experience, I highly recommend finding a small machine, cram as much RAM into it as you can, and start setting stuff up. Right now I'm working on an "end to end terraform solution" where I setup 4 VM's on proxmox with cloud init on an ubuntu template, initiate k0s as a distro with 1 controlplane and 3 nodes, and then also automate all of the final details, like storage using nfs to my nas, metallb and ingress. Challenges always remain, but it's fun to perfect (and then toss out and start over with something else).

@transatlant1c Год назад

Correct in all aspects, I just think he wanted to do something different and jump straight to the k8s side of things. Spot on though!

@chrisa.1740 Год назад

Great video! I just had to comment about the cost though... holy crap that rack mount drawer is the same price as one of the NUC's! That's an insane cost for what is essentially a folded steel drawer.

@SneakyJoeRu Год назад

I'm not sure if I should be proud or afraid that I start to understand what you're talking about and envy your setup after joining devops 2 years ago...

@GundamExia88 Год назад

I love the NUCs, they are awesome

@henrysowell Год назад

Tim! Great video. I’m going to have to explore this. I’ve been thinking of selling my 8 x Raspi 4B (8gb) cluster and doing something like this. By the way, for your last Cloud Init video, I put together a ansible playbook based on the video to automate the process. It was awesome, used it to make templates with a single click across five proxmox nodes. Thank you for all you’re doing! 👏

@LtdJorge Год назад

With 8GB Pis going for 220€, you can build more than a 3 node NUC cluster, lol

@TonyGonzales Год назад

I either ran into you or your doppelganger just now in downtown Houston. Couldn't put my finger on who I was reminded of before the convo ended but wanted to say thanks after my memory was sufficiently jogged. Have appreciated your work for some time now. ✌️

@TechnoTim Год назад

Hey, and thank you! Not me I am way up north but would appreciate that you came here to tell me!

@TonyGonzales Год назад

@@TechnoTim haha well your twinner is here and in shorts.

@JohnMatthew1 Год назад

Great video Tim, thank you. I use K3sup to provision, but I'm very interested in your Ansible version.

@testdasi Год назад

Tim's K3s Ansible playbook really simplifies setting up an HA Kubernetes cluster. On Ubuntu, it might be faster to set up microk8s but after trying both, I'm like k3s slightly more because of lower idle CPU usage. And even with stateful apps (which is... uhm... pretty much everything?), you can still achieve almost-HA with a single replica in a HA cluster + replicated storage (e.g. OpenEBS Jiva or GlusterFS). Just need to set tolerations to something like 10 seconds. So if the node (that carries the pod) is down, 10 seconds later, a new pod will automatically spins up on another online node. That's HA enough for me.

@NonyaDamnbusiness Год назад

I liked the Jeff Geerling "Red Shirt Jeff" cameo. I'm good with docker and have been using it for a few years but man, I gotta get off my butt and get into kubernetes.

@kingcamofficial Год назад

Really looking forward to your next live on Thursday because i work Tuesday until 8 unless your going live later on in the day

@DavidAshwell Год назад

Those Thunderbolt ports will directly do 10gig+ Ethernet host to host (iPerf around 14). 3 hosts, 2 ports each so you can daisy chain. Takes some configuration, but it's a cheap way to get fast networking.

@salg8502 Год назад

Great video. If I can suggest wherever your reading from to be lower. Looks like your reading from the ceiling or something

@RaidOwl Год назад

Drag race? Your NUC k3s cluster vs mine 💪🏼

@TechnoTim Год назад

Let's go! 🚀

@swollenaor Год назад

I would love to see a livestream contest, judged by Tom from lawrence system.

@Bianchi77 Год назад

Nice info, thanks for sharing :)

@lamar9525 Год назад

Great video, I have a question with the NUC's. What would you do if you want to add more storage as in external hard drives?

@torgrimt Год назад

any plan for a followup on this? Interested in seeing how you go on with all the services you mentioned. How do you setup the storage and running traefik as ingress. Would be cool to see some homeservices for instance zigbee2mqtt with an dongle. You can add a tag to the one who has it connected, and let k3s move the container around.

@THEMithrandir09 Год назад

Planning to do this in the future too, but I think I'm going to wait for mainstream ARM to come a little further to get the power usage down even further. Though maybe I should do a cost/benefits analysis, maybe switching to NUCs now and then again later saves the hardware costs in power costs.

@AndrewFrink Год назад

Assuming 20w each for the nucs, and 5w each for some future arm thing, and $0.20/kWh. That works out to a difference of 45w. 0.045*24*365 = 394.2kwh per year, at a cost increase of about $79 vs the theoretical arm setup. That's about 1 raspberrypi in a case with a PSU per year. Some cstate tuning, and maybe scheduling a shutdown would likely close the gap. I'm not sure which cpufreq governor Ubuntu server ships with by default, but switching from performance to ondemand would also save power. If Tim's text overlay of 9watts average is correct, it's closer to 140kwh savings, or about $28. These mobile CPUs in the usff form factors really are very energy efficient.

@LampJustin Год назад

Since those nucs have 2 Thunderbolt ports, would you please try and use them as network connections? If you connect them to each other you'll only need to modprobe the thunderbolt-net driver I think. (Be sure to use real Thunderbolt cables!) Maybe the speed will be really good! TB does 40Gbit/s after all, so if you get like 10G that would be amazing!

@johnsondu1896 6 месяцев назад

Just wondering if I can use different spec of computer for each node.

@jaygreentree4394 Год назад

"Red shirt" Jeff is a bit too much chaos.

@serhioromano Год назад

This is a perfect setup I think. A bit expensive for me. I've bought NUC i5 8 Gen. It supports virtualization but not those fancy features. I think home server in NUC is an optimal solution.

@GuillaumeDeplancke Год назад

Did you look at charmed kubernetes options with JuJu? It even integrates quite nice with MAAS 😃

@UntouchedWagons Год назад

What addons do you have in VS Code to help with Kubernetes?

@chriswiggins3896 Год назад

What can you do with longhorn? Can you setup something like ceph over the 2.5G nics, or install 10G usb dongle nics for more speed?

@ETVELTALLERDELOSVIDEOS Год назад

Hello, I have a question about clusters, maybe will be useful in order to work in video edition with software like Davinci Resolve?Can you do an example of that in case it is possible? cause video edition in Davinci Fusion requieres a lot of machine and maybe if one person can use 2 or 3 cheap pcs instead of a monster pc maybe would be a cheap solution

@Carlos-Rodrigues Год назад

Got rid of my 7 boys Proxmox cluster. Now using only 3 AMD 4800U servers. Yes it's 15W each, yes it's 16CPUs each, yes it has 2.5Gb LAN. Only issue is that it has only 1 slot for NVME and no SSD disk. But it's a bomb and my proxmoxies say thank you daddy every time I see these 30 VMs dancing 🎉🎉🎉❤

@TechnoTim Год назад

😂

@marcosaurelio8591 Год назад

Gostaria que tivesse mais conectores SATA para fazer um NAS storage.

@Denchozzz_ 3 месяца назад

Hello, I've just made a corosync and pacemaker cluster between two debian 12 machines and I want to install CasaOS on the cluster so the load is balanced between the machines. Is it possible to install CasaOS on the cluster and if it is, how?

@dzmelinux7769 4 месяца назад

Just a question everybody seems to be let out (or I don't have the right understanding of HA) what about if a pod/service runs with a database and a node goes down? How does Kubernet know who is holding the latest database entry? Thanks 👍

@FractureCLinic47 Год назад

Hi Tim, not sure if you've done a video on it, but do you manage your Unifi network with either Terraform or Ansible as part of your home lab setup?

@jenniferw8963 10 месяцев назад

3 x Lenovo Tiny m920q high availability, zfs replicated, proxmox cluster. They each have pice slot and I have Mellanox ConnectX-3 dual SFP+ installed in each. They each run 5.5 idle without the dual sfp+. With the dual SFP+ they idle around 11 watts. Mine each have 32gb ram, and currently 512GB NVME.. Will be upgrading the nvme soon enough. No ECC but I am okay with that.

@Merrlin 2 месяца назад

What cable is that supplying power & data to the SSD? It’s very interesting, I haven’t seen that before

@francisbernard6838 Год назад

Heya, thx for the monitor! Cool vids as well!

@TechnoTim Год назад

Thanks man! Enjoy! That monitor makes an appearance in many of my videos!

@amcape5654 10 месяцев назад

Could you run proxmox on each node and make each node a separate k3s cluster using VMs?

@_Aki__ Год назад

How does kubernetes handle P and E cores in modern CPUs? I've been keeping my homelab on 10th gen intel due to never getting a solid answer to this.

@smalltimer4370 10 месяцев назад

I have a question, would it be possible to create a VM that would otherwise use cores from several cluster devices?

@user-od6gi4lz8t 8 дней назад

Intel vPro is probably integrated in those NUC so you could probably use AMT that for KVM 😊

@thomasboos2470 Год назад

Is there a reason not to use Intel ME (vPro) and buy expensive KVM stuff? The Intel NUCs and many more mini PCs have already Intel ME build in, including the capability to remotely mount FD and CD Images.

@hongtanke Год назад

@TechnoTim how is this managing E / P cores with K3s/ K8s

@zbigniewkrajewski7536 Год назад

Tim, dont mess with Red Shirt Jeff... :D

@godgutten Год назад

where did you get that large kvm?

@i_Kruti Год назад

9:00 😂🤣😂🤣😂🤣😂🤣😂🤣THE BEST and HILARIOUS part.....!!!

@i_Kruti Год назад

TIM be aware of RED_SHIRT_Jeff .....🤫🤫🤫

@Matthew-pg7mu Год назад

I’m confused, what storage provider are you using?

@mathieuleclerc4136 Год назад

Hey Tim so where is you master node, not in one of your NUC, right?

@computersales Год назад

I'm amused by your notion of wanting low power while I'm thinking about how sweet a R940 would be to have as a Truenas box. 🤪

@businessoftechnology Год назад

Vs RPi? You forgot to mention the architecture. It seems there are many containers that do not run on ARM processors.

@MPECSInc Год назад

No ECC? How come?

@majorgear1021 Год назад

This looks nice for a lab play with. For power saving, I find proxmox on PC better as I can run the same services on fewer machines. I can also put 10Gbe interfaces, hba cards, gpu’s for ML , and more in them.

@DakotaTheKid97 Год назад

I'm not sure if the NUC's are compatible, but there's lots of Intel processors with AMT that allow out of band management on the same interface. I do this on my OptiPlex desktops using the MeshCommander desktop application. This might be something to look into if you'd like to save the PiKVM for something else!

@TechnoTim Год назад

Thank you! When I bought the first NUC, I thought they came with AMT, which they didn't and the reason why I went PiKVM!

@cryptic_dingo Год назад

@@TechnoTim 1st: Another excellent video, thanks Tim. 2nd: I'm not 100% sure what I'd do with it, but I want it 3rd: Some Intel NUC models have vPro (at a steep price increase)

@Randoneering Год назад

Tim. You need to stop making me have tech envy. I am trying to stop myself from buying a system76 meerkat or an Intel NUC for using at my desk.

@jesusmachado9087 4 месяца назад

My thing with nucs is the ram!

@AndreaGhensi Год назад

Since NUCs have vPro technology, have you tried to use MeshCommander to access AMT/KVMoIP instead of using the pikvm?

@TechnoTim Год назад

Not all NUCs have VPRO / AMT. I found out the hardware, but it was a happy little accident because now I can use PiKVM to control everything!

@diginomad6016 Год назад

09:00 chaos 😂❤🎉 Hope @jeffgeerling saw this

@vincentaguilar6249 Год назад

RED SHIRT JEFF CAMEO

@llortaton2834 Год назад

Don't let red shirt Jeff near the server!

@JzJad Год назад

Let's be real the tidy cables never last long for us lol

@Tom-kl9jf Год назад

red shirt jeff cameo!!!! 😂

@davocc2405 Год назад

Moving off VMs like this: we shall refer to this as LITERALISATION.

@RainySeasonMemory Год назад

Not having ecc support is a deal breaker for me, the worst thing about it is that you may not know if there is a problem when memory corruption happened.

@JKos-cz9lj 11 месяцев назад

Does Intel NUCs support ECC Sodimms?

@TechnoTim 11 месяцев назад

It does not

@linrobert8208 Год назад

Nice idea. BTW, it's much better to use new intel 12th gen low-power cpu like N100. A small pc just look like intel NUC with N100 inside, just cost about 100 US$, without mem and ssd.

@tifflabs 7 месяцев назад

That's exactly the path I'm going to take. I have 3 NUCs, a new one from Beelink and two really old ones from eBay which I'm looking to replace. The ones he linked are great but I can get an N100 from CyberGeek for half the price, more RAM and a TB SSD.

@ws_stelzi79 Год назад

I see The Chaos isn't properly credited! 🤭

@notreallyme425 Год назад

3:21 LastPass? With all your servers and clusters I’m really surprised you don’t self host Bitwarden/Vaultwarden or similar self hosted password manager.

@lagrz Год назад

should look into consul / nomad stack for something lighter weight than kubernetes

@ZippyDooDa435 Год назад

That is what I run at home on my NUCs, love it

@Danielo515 9 месяцев назад

Do they send worldwide?

@TechnoTim 9 месяцев назад

Amazon should!

@Danielo515 9 месяцев назад

@@TechnoTim I mean the rack mount provider

@kampyzako7482 11 месяцев назад

why not harvester?

@donaldsimpson9911 Год назад

Who let Red Shirt Jeff out?

@ChristopheRobinson14 Год назад

Cool but kind of expensive

@MrHics 11 месяцев назад

Bahaha redshirt Jeff being a chaos monkey

@maxjackson.7533 Год назад

Are these devices POE powered?

@ZippyDooDa435 Год назад

Nope

@michaels8336 Год назад

No but I can see how you think they were, the power output in the UniFi screens was via their PDU and not a switch.

@marcin6386 Год назад

You can set and configure any devices. The main problem is the money to buy hardware 😢 and at the end there need to be some purpose to build those setups. Ideally it would serve business that would generate money.

@stevemulcahy5014 Год назад

The experience could also be the difference between being on helpdesk and going up to devops salary. My homelab experience was way more valuable in getting me from $70k to $125k than work experience alone.

@marcin6386 Год назад

@@stevemulcahy5014 oh yeah I learn a ton when I'm playing with my homelab too!

@TheMrDrMs Год назад

This seemed to have rapidly age like milk as Intel announced no more nucs. *But* there are other great options still!

@druxpack8531 Год назад

No it hasn’t. NUCS will be kicking around for years to come as the warranty will still be honored, much like what has happened to Intel Optane. Also, mini PCs are here to stay, so just swap in any mini pc and reasons remain the same.

@bradleystannard3492 Год назад

You mean you went with K3s because my linkedin comments were a compelling enough argument…

@picudo Год назад

I don't know if 3K USD worth of these machines are being "efficient" , surely not for the money. 3,000 USD you can get same power for years on Hetzner and not get stuck with soon to be dated hardware.

@MuffFlux 6 месяцев назад

Why does every RU-vidr who shows off their cluster on RU-vid never talk about what they actually practically use them for (other than generic answers that everyone knows/says about what you "could" use them for)?

@TechnoTim 5 месяцев назад

Check out the rest of my channel, I have about 100 use cases