Meet the new SBC Linux Cluster King! 

"All's well that ends in "Not A Fire"" - a motto to live by.

Jeff, Thank you for your honesty in who sent you what and how things were paid for. That’s very refreshing in this space and shows you value your audience

My wallet wishes I had never discovered your channel ... keep up the great coverage of your SBC adventures, you play in the same space I'm interested in.

“I don’t need a cluster” “What would I use it for” “It’s just gonna be running some bullshit” I so really want one

Man i really hope that Raspberry Pi prices comes down to a reasonable level, so that us enthusiast can do stuff like these

Yo, Only Fans reveal at 11:12!

That rockship kicks butt, I can tell, I have a NanoPC-T6 powered by that baby, it ROCKS!

Thank you for showing the debugging steps with the cluster, including how to see which pod had stalled and finding error logs using "describe". You can watch 100 happy path tutorials without ever seeing these things.

As soon as you said pod was pending and you were using nfs I said to my self "nfs-common". Great video!

The RK1 boards are crazy fast! I have two of these TuringPi 2 boards, running 2 RK1s and 2 CM4s each. A sort of P-module, E-module configuration. 92GB of RAM and 82 GHz of CPU. I use Hashicorp Nomad to orchestrate tasks using a combination of the Docker and Exec drivers. Ceph to manage the persistent volumes. Soon I'll be able to retire my power-hungry my Dell R710s in favor of a system that draws < 200W. Couldn't be happier with the results!

Really love the screen time during installation and configuration so didactic!!

Break out the soldering iron and swap the direction those power leads exit the PSU board. No more fan-blade jeopardy, no more stressing of the board connector.

Excellent! Thanks for sharing. Looking forward to the new base board with 10GbE.

Thanks for all your hard work, Jeff.

I have one of these, and it is spectacular! I couldn't afford to get 4x of the 32gb nodes, but I have two of em. They are heckin nice.

this stuff does not make sense when for this price you can buy 3-5 mini pc with 8 core ryzen cpus. you don't have to deal with arm specific software and the performance are 10 times faster.

Really seems like the Turing Pi board could benefit from having a switch chip. Even if you still only had 1 Gb or 2.5 Gb connectivity to the external network, having a 40/80Gb switching fabric between nodes would make something like Ceph internal replication or any node to node comms in the cluster much better. Of course, having a 10 Gb SFP+ would also be ideal, since so much 10 Gb fiber gear is plummeting in price these days, and might obsolete the need for finding another big chip and traces on the board, so that might the smarter play.

Letting red shirt Jeff in to the studio seems reckless.

Thanks for the mini-rack link. I've been looking for something like this for a while, thinking I was just not looking in the right place. Great for a mini home lab.

I've been using the turingpi 2 with 3x RK1's for a few weeks, and really enjoying it. Ive been managing it using the Ansible for DevOps and Ansible for Kubernetes books you put out, so many thanks for that!

These cluster boards keep getting cooler and cooler. Not sure what i would do with one but it would be cool to play with.

Love these tiny videos. As someone who does Van Life traveling full time in my van, I’m always looking for ways to run NAS and cluster solutions in a compact “portable” way. Especially love it when you can run on 12v, 24v, or 48v DC.

This rack is a master piece of engineering for home datacenters. 🙂 A little UPS would be indeed amazing. Whoever creates it, shut up and take my money! 😀

I just like how it looks, a compact blinkenlights thingy. So cool.

The 32GB Turing is still less than the price to upgrade to 32GB Ram in a Mac.

Red shirt seems calm in this video.

This thing is sooo cool. I stopped running a homelab or server for my house because of power consumption. This setup is making me rethink that. I could do crazy stuff with one of these boards.

Free tip: Don't shake and wave you hands when you are holding something we are trying to see. Very common on RU-vid but you can beat it. I did for teaching.

The Rackmate T1 in this video is awesome!

That Quad board look like such a nice cluster training tool!

I wonder how it compares to a Milkv pioneer, double the cores, same memory, not a cluster but a "desktop", it does have 10G eth (x2) and NVME. It's less than double the price, for double the cores and the same amount of ram but probably faster as it is all in one place. No NPU on the pioneer, but it does have pci-express ports so you can add a large video card if the drivers support it, and if you upgrade the power supply. I absolutely loved the 10 inch rack idea, this is new to me. I have a (official status pending) RISC-V lab in Costa Rica, with a bunch of licheepi4a, vision five I&II, beagleV, mangopi, etc, etc. I think the milkv pioneer is mini-itx, so it would be cool to fit in all of the lab in this rack. It's a bit pricey, but would look and function way nice than what I have right now.

Practically, you don't need an SBC cluster. Single CPU x86 machine will be faster, and more important, easier to support. So performance comparison between different SBC cluster does not have a practical meaning. But it is still fun!

Brought to you by the Pi Cluster King!

I did not know about 10" rack systems. Very cool.

I don't have a use for a cluster system, but they're entertaining. Make it work, do some brutal testing with it. Really brutal testing.

I wonder when failsafe systems and ECC memory will be popular🤔

That little mini rack thingy is rather neat. I wonder if one could make a small easy to move server with a rack for compute, a rack for storage, a switch, and if there is such thing, a UPS.

Sick video, Jeff as always. Love to see some RGB on that rack. 😊

You read my mind when you talk about a UPS for a mini rack. That would be siiiick

The mini UPS sounds like a cool idea. I'm thinking, make it be a shared power supply as well, so integrate a switch mode power supply in there that keeps the batteries charged, and have a bunch of 12v outputs with individual reset-able fuses, plus maybe some 5v outputs. Include a micro-controller in it with some temperature and voltage sensors to keep the charge levels and temperature in check, with an output to a character LCD and a USB port for UPS management.

PI's are getting insane dude! 😲

I’m always highly impressed when someone gets Drupal to do anything sensible at all, let alone a usable deployment of it.

I ordered new Turing Pi 2.5 with those RK1s with 32GB RAM each. I can't wait! But yeah... elephant is big, and even dual 2,5G would be great. 1Gigs for this is too small bandwith and just because of that I was postponing my ordering since like a year... I finally decided that it might be ok, but I still wish it has faster network.

I saw a talk a few years ago at kubecon where the speaker was the guy who first named kubectl and he said it was "kube control". And that fits my narrative, so I'm sticking with it

Simply AMAZING !!!

Fever dreaming out loud for engagement: I want to stick one of these cluster boards in a 2x 3.5" bay NAS case for a 3 node apps cluster and the 1 node that has the sata slots being the bulk storage. Alternately, would love to see a laptop form factor that could take one of those RK1s (like the MNT reform, or similar).

i just suscribed, lets be real, your content is really interesting and i enjoy it :)

14:00 you could even solder in the power leads on the other side of the pcb

Awesome detailed look. The 1Gbps is definitely showing its age. The price is kinda crazy also. Hope they can increase production to make it cheaper

These RK3588 SoCs are quite amazing to behold anyway, it seems. From a regular user's perspective, I'd still love to see an ITX board for desktop use but the board you used here is interesting to watch too.

Thanks. Good info.

That's a cool computer mini rack 👍

I found with k8s that setting up calico or flannel (can’t remember which one) messed with mDNS on the cluster, it would change the source port of the response and avahi would drop it. It would add some iptable entries and would always put its own rules first even if you manually inserted your own rules before. No idea if you use those in your script but noticed you’re using mDNS figured it was worth shouting about

if you want the led activity indicators from your cluster board, I would try to make a passive light pipe solution with some pieces of clear petg filament, perhaps with a little bit of gaff tape or electrical tape or something to keep the light in the pipes seperate. diy fiber optic.

I got plenty 10" racks at work and we are using the TP-Link TL-SG1210MPE switch for it (: PoE, SFP, managed.... downside is the external power supply... fits perfectly in a 10" shelf

It's such a minor thing, but "sudo su" always bugs me. It feels like when the switch from using su to using sudo went mainstream people said "but how can I use su now?" and found a hacky way of doing it without looking at the documentation. "sudo -i" gets you an interactive shell using the target user's default shell and runs the login files for that shell. "sudo -s" uses the invoking user's shell. Either feels cleaner imo. There's many ways to cut a cake, and there's rarely a right and a wrong, but I wanted to put "sudo -i" out there in case others maybe think it sounds neater too.

Felt like I unlocked a core memory when Jeff slid that Pi Blade out. Forgot I'm still waiting on mine. Soon™

with the mini UPS thing, could use some of those USB-PD triggerboards and a compatible battery bank. course might want to collab with one of the eletrical engineer youtubers to work out how to make it do passthrough+charge cause some batterybanks turn off when charged up

If you are going to standardize on 12 VDC devices, you could put a battery in the bottom of the rack. Then you can feed the rest of the rack from there. Could even get a DC power bank and use that to connect to the AC or DC source of your choice and keep the rack powered even in transport.

Creative video, thank you :)

I have to say that the 10 Inch Rack just makes sense for small builds where space is very limited. hell I've debated about going that path for somethings after you've mentioned 'myelectronics' a couple of times on top of you deploying their cases for some of the builds

"Worst case, we'll just have a little smoke come out here." and then a little later "all's well that ends in not a fire". Sounds like a good day at the office.

@JeffGeerling In reply to your comment on 17:46 about the availability of 10" racks in Europe: I got lucky getting a 30cm deep closed 12U rack from a now defunct Polish company (COVID stock/supply issues I guess) I only could get the MyElectronics 2U enclosure in there by putting in some custom spacers to get the ITX case closer to the glass front door. Otherwise the cables in the back would just not fit. Any deeper rack would not have fitted in my utility closet. I once revved out the diesel my driving instructor used when I was still learning for my driverslicense, some 300 meters from where MyElectronics is currently based in Alkmaar. I now have one more reason to chuckle about that event, 'some' American guy made realise that they were based there.. :)

18:30 well you can get some small form factor ups from eaton, well there are just desktop ones and i think you will have to make a bracket of some sort to fix them into place but it schould be doable (maybe they even include mounting kits for 10 inch, its some time since i installed one of them so i can´t remember 100% if it had that or note but from memory the size schould fit)

Idea for a cluster: take Framework Laptop mainboards (the ones with USB 4 for PCIe tunneling and 10Gbps+ networking) and bolt them into these enclosures. I think you might even be able to fit a full 4 of them in there. the keystone slots at the back up top could be USB-C Keystones for power and for the big opening you could 3d print an IO shield so you could put in other keystones for IO. Fully x86, BLAZING fast IO for that small of a cluster, and possibly pretty energy efficient.

That’s pretty cool - I want For price comparison- there are plenty of used epyc cpu + mobo + ecc ram combos on offer for less money.. more grunt, more power bills

BTW I literally saw the person who wrote kubectl at a talk in person and he said that it is pronounced "cube control".

That rack would be great for basic networking and a media pc.

I love to see how tech gets smaller and more usefull, and how open source software reigns among certain communities, but boy do I feel 'lesser' nowadays, in the late 2000's I grew up daily driving and tinkering with linux (mainly mandriva at the time), got into programming a bit early still do some, but I'm mainly a digital artist to pay the bills, and I feel so out of the loop, when I hear about tech like kubernetes, having no clue what it is about ahaha

I remember when data rates went from 110 baud to 300 baud. That speed increase was amazing for the time. And years later the first 14,400 baud modem came to market at $14,400 each. These young folks think 1Gbps is slow. They do not know what slow is. It did teach us patience.

I wouldn't even want a real ups for it, but more just a batter bank to supply 12v to the itx picopsu and maybe a switch, though might need 52v for a poe switch

Server on the go? I think this would make an awesome little travel server. Your main mini itx pc, your cluster server, and a travel router would probably fit really well and it looks like it could fit in a little carrying case.

big QUESTION, how would a minecraft server perform on this?

you can add 4pin fan header to v 2.4 boards but you have to solder a coonecot and a chip onto the board.

I, (like so many others) are going back to Mint! We've had it with Snap Packages and Ubumtu slowing down. So we'd like to see you using Debian more.

well, now I need that mini rack

I am eagerly awaiting my Turing RK1 modules. My plan is to build an ARM based CI/CD cluster and I am curious to see if the gigabit connection ends up being a bottleneck.

about 20 years ago the cluster system working on 1G bit ethernet was too slow, which we could get off, our HW teams had a 10G switch backplane and looking at 100G or maybe a 1T switching backplane (but we were working on an edge router for a city 500,000 users per cluster) full HA with fall over in less than 1mS. but looking at the internet today we may have needed 10T bit

I've done a fair bit of cluster computing, on big clusters like Stampede at the University of Texas, so I'm familiar with the desire of cluster creators to have fast shared filespace and high speed interconnect fabrics etc - *but* - at least for all the real projects (as opposed to demos or benchmarks) that I've used clusters for in real life, I've always been able to structure my code so that high speed interconnects and fast shared filespace are just are not needed. Even for applications where there was big data rather than just a lot of CPU being used. My observation is that these extreme hardware facilities are mostly used by people who don't make the effort to optimise their application for parallelism and who try to shoehorn code that was really designed with uniprocessing in mind, into a multiprocessing environment without taking on the necessary restructuring. Now I'm retired, when I need to throw CPU at a problem I just farm it out to a few dozen regular Pi's and a couple of spare x86 portables I have around the house, running over Wifi, with one regular NFS mount to supply shared files across the lightweight cluster. In fact - although I do have proper cluster MPI software installed, I generally can get by with just kicking off tasks using "ssh". So I mention all this to suggest that perhaps the features you describe and the cost of the system you're using is perhaps a little overkill, that could be avoided by a little more coding effort. An interesting approach for a subsequent video might be in terms of computation done per dollar rather than just what is the fastest shinyest new hardware available?

OK, have to say this jokingly. Damn you Jeff, costing me money. I use bought the T1 and the 6 Pi ITX board. I am going to use that case for a talk I am going to give at the Southeast Linux Fest in June. I love this because I was using mini milk crates for my SBC lab that I am trying to replace big bare metal with.

6:24 is the important moment. If you want to use two of this boards in a cluster, 10 Gbit/s Ethernet is what you need. It was mentioned twice in the beginning (a little redundant) and it is what stops me from buying such a board: Ethernet is too slow. The connections between the Compute Modules must be very fast and the connection with the external world should be as well. Everything else is great. Storage, compute power, all great.

Thanks 👍

It wouldn’t be cheap but if someone wanted to they could use the asmedia pci-e 3 switch chips and break out the pci-e to a 2.5gbe nic and an m.2 slot to make a higher speed connectivity version of a cluster board, going up to 10gbe would add a lot of $$.

for deep stuff like the RK1 in that rack, it'd be nice if they included rear supports like what the shelf had.

As an electronic engineer, sometimes I just wish I had more time so I could also be a RU-vidr, allowing me to work on silly projects for weeks, like building a little 10" UPS for this cute little rack ^^'

Great vid - the sound is very different, and it looks like you green screened your eclipse photo/workspace - is this in your new space ?

The mini rack needs a 10" PDU that has a 110v input power. And as Red Shirt Jeff pointed out, a 10" UPS would be a nice option. EDIT: A Ubiquiti EdgeSwitch 8 PoE (ES-8-150W) would fit nicely in the 10" rack.

Another question - do you think that the tower cabinet could handle two of the Mini-ITX cases? [I have two Turing Pi 2 boards from the campaign, along with eight RK1s, ready to go... and have been instigating Mini-ITX cases for a while now...]

Have been working with the TuringPi2 and RK1 Boards for a few months and recently also received a few different mPCIe devices, unfortunately with very bad results. Only 1 in 5 boards would show up. I do hope you look into testing mPCIe devices with the TuringPI2 similar to what you have done with the CM4 as it seems to be either a bit of a jungle compatibility wise or alternatively the faulty connection on the TuringPi2 side. only board I got working was a IOCREST 4 Port SATA with a 88SE9215 chip, Tried a few realtek ethernet adapters and a different SATA adapter.

I have the TP 2.4 with 3 CM4's and an RK1, would be great to see a series based on using ansible and k3s to add services like plex, owncloud, etc. I have it up and running k3s and want to add mulitple server applications using symfony and maybe how to use argoCD etc.

for $1500, could easily get a 16 core 32 thread and with 128GB dd5 and a great motherboard, case and power for less than the cost of the max version.

At the moment I am a little fed up with clusters. But this one is nice

Nice t-shirt!

I switched from .local to .lan to avoid mdns to kick in. This avoided lots of issues in dns.

I actually forgot to put in issue/pr over a year ago for the nfs-common issue as I used your playbook to deploy to Ubuntu arm systems previous, my bad!

18:48 cool idea - how about a few lipos with a pi doing power management..

Jeff, you should design your own desk mat!! :-D

4:55 - Yep. Just what I thought. That red linux shirt was a sign...

Jeff, have you seen the Mixtile Blade 3 with the cluster box. Looks like a nice clustering box

My biggest issue with working with Pi's and Pi alternatives has been lack of conformity to standards such as ITX boards, or even conformity to Pi's I/O and size so that you can just use Pi cases or options. Something like that ITX Board, as well as the rackable case is a huge leap forward. Yes, we can 3d print things, but the more options we have, the better. I love SBCs, but prior to now them not having more standard footprints means that I literally sit a pi-hole with a POE adapter on top of a POE switch in a rack. It made no sense to have a full 1u or bigger rack mount for 1 pi. Then I have the second pihold on a separate part of the network, just sitting next to my NAS in a different room using a power adapter and network cable as I had power available there. I also have numberous Libre Computers sitting around, just strewn everywhere because where they needed to be, I couldn't put a rack. Most of these locations have other computers and if we had more options for say, PCI brackets as well with power and network cabling through the PCI bracket, I'd mount them in the PC cases. But for now, they are just sitting in 3d printed cases. So things like this are a great idea and leap forward. I'm excited to see more options coming forward as I mainly work in IT Support, not systems administration (which I can do and am good at), not Devops, not with 3d modeling, etc. I can implement and combine solutions if they've been built and sold, or I can implement very simple solutions. The more options I have, the higher probability of Pi and Pi alternatives have of getting into my workflows instead of tiny computers like the Lenovo M90n, or the other Lenovo Tiny models. I definitely don't need their horsepower, but they've just been a better option ultimately when I can get them at work's surplus sales for say $40-$150.