Homelab Storage Cluster, How Well Can I Reuse Existing Hardware? 

Your content has always been good, but as of late I really need to say... You've been consistently producing higher and higher quality content! Your explanations are informative, engaging, smooth and paced well even for new users! Thank you for sharing your knowledge with the world

Glad you went with real distributed filesystems. As the author of mergerfs I've seen people try to use mergerfs + NFS/SMB/etc. to build out a "poor man's remote filesystem" and while it can work OK if you don't have sufficient hardware to dedicate to just a filesystem it certainly is an abuse of said technologies.

This is an incredible video I love seeing fellow creators giving all of us new ideas and new ways to do things, thanks so much for the video and you have a new sub!!

Wonderful explanation and details. You should have told us newbies what SMR was. I'd love you to go over my [very small/not awesome] server setup as I'm wanting to add a power vault to my r330 and upgrade the r330 in the future. Love your content!!

Interesting Problem. Thank you for going through this with us.

Excellent as always

Years ago I really wanted to move to those 5TB 2.5" drives. Less space. Less noise. Less power use. Unfortunately, the reliability was not great in my experience and the cost went up over time rather than down.

We had same dilemma, but went completely different route at the end: iscsi SAN. Multiple fat disk servers, each having multiple md raid arrays, each exported as separate iscsi device. All that is imported into slim iscsi initiator head server and bound together into a pool through LVM, sliced up into volumes for different shares and exported to servers and clients through NFS and Samba. It's not clustering, but it's relatively easy to add a box, export raid arrays through iscsi and import it from head server and expand the existing pool.

Great video.

very interesting challenge .. especially with SMR hdds. 45Drives has a 3 yr old video using OpenCAS to accelerate Ceph HDD with Flash (optane I believe). Maybe a little SSD in the mix could bring the performance up? Keep the great content coming!

Great video, very usefull. Thanks.

Good video. I'm worried though about down the road, when the pool starts filling up past ~80% and drives start dying. SMR drives may seem to work, until they don't. They're just horrible tech that nobody asked for, when CMR drives fill the role perfectly fine. MHO, your client would be better off tossing these 5TB SMR drives as a ticking time bomb and rebuilding with SSDs, or buying a SAS disk shelf and rebuilding with 3.5-inch NAS-rated or SAS disks for long-term. But at any rate, they're going to need regular reliable BACKUPS.

Thank you for sharing

Nice. Thank you

This is LITERALLY my exact issue I'm having with my HP Z440 Unraid build... bought a rear PCI-E bracket to attach an HDD too and see how that goes.

The addiction is real. Ar first 1tb was way more than i would ever need... Now at 3tb total and it's almost completely full... I think I just keep adding more stuff until eventually it's full, no matter how big it is.

I never can make my mind up on this stuff. I just know I want a lot of space, decent redundancy, and gotta go fast.

You could use zfs as the base (and some cheap optane for zil if sync write with proper speed is needed) and then put longhorn on top of that as shared fs across all servers. It's not as sophisticated as ceph but works well for me. That said: glusterfs also worked fine for me but now that kubernetes 1.25 deprecated it and truenas dropped it I don't really think I should keep using it anymore in the future. What do you think?

What does the clients backup solution look like? Maybe do away with raid entirely as it makes very little sense for a home environment, do a JBOD, maybe with mergerfs on top for convenience and just restore data from the backup if a drive dies, no resilvering needed.. Or are we talking the classic homelab strategy of "RAID is sexy, backups are not, let's rely on RAID to keep my collection of pirated movies safe"?

similar servers, 900tb SAS Enterprise mixed with 1tb ssd's, truNAS Scale

I also experimented with ceph on Dell R720 and Proxmox a year ago. I have SATA SSD drives and need high random read performance. Also tested nvme drives. But performance was always very poor: i measure it in io operations per second, and it always was a factor of 10 worse than local zfs. Network should not be a problem as i also have Infiniband 40G. ( some magical internet posts also declare that RDMA should be possible. I could use it in beeGFS, but in ceph there was no documented way to get it running ). So my solution is to use zfs, and replication. beeGFS was really fast - but there is no parity possible. Or you you think that there could be a way to get ceph on ssd faster?

A video I would like to see is: how to setup a small 3 node cluster that serves: (a) 18+ tb of object store for many tiny files accessed over zerotier network with local file caching of 1tb at each end remote client .. posix LAN style file access over high latency wan, (b) high speed object, file, and block store for compute cluster. Issue is devops cluster and remote file server are really hard to manage. Cant use smb or nfs for wan. Massive binaries for chip development tools make scaling our docker images across cluster hard. Object store typically not great for posix style file access over wan. Coherency of decentralized file cache a problem. Zerotier expensive if each vm / container in cluster has own zero-tier managed ip address. Difficulties managing ip and single sign on address space for both zero-tier wan and local high speed lan for inter vm/docker/.. communications for CI/CD workers. - basically insanely difficult to convert old style lan for software developer company and decentralized the acess where all workers are work from home.

Do you reckon if the slow performance is mostly due to the 10gbit network and sync or is ceph just a bad solution in general for small deployments like this? I've been thinking about moving to a ceph cluster myself, though at 100gbit networking speeds, but I would only have a small number of OSDs as well and I'm worried about performance. I could move my HDDs to zfs and use zfs-ha to keep my storage highly available in the network, which is really all I'm after.

Linstor can be option too

How is TrueNAS with ZFS a clustered solution?

think about getting customer to think about optane and/or pci-s ssd too, even those on raid0 - it could change the equation on performance but probably won't help rebuild times - maybe have some hot spares? a followup and/or special on ultimate nas that still is somewhat budget would be appropriate followup - since prices are normalizing on all this old enterprise gear i think more people will be looking for high performant but still economical solutions - there has to be a sweet spot out there that balances out the hw and sw for an optimum solution but there are a ton of variables so finding exactly what will work best is not easy for sure. #tylenol christ

Best practice: XCP-ng and TrueNAS Core for Business Users and Proxmox and TrueNAS Scale for Home Users. No proprietary storage solutions, if they use closed source software.

Good luck with week long rebuilds and multiple vdevs, lol. Especially considering how 24h+ rebuilds already accelerated dRaid development and adoption.

ceph CACHE TIERING?

Tell your friend to deal with the pirated porn himself 😂

Cha ge all smr 5tb hard drives

I hope you do not take it as a personal attack. You are in the media area. So I think you may consider JerryRigEverything or Jo Rogan style for your hair. Again this is not to offend. I'm a fan of your content.

Good video, i think you should just shave your head, it would look a lot over long term. :) GL man,

Bad "solution" choice. Research before trying to implement.

Only idiots use SSDs for storage. 🤦‍♂️