I would almost be more interested in a fan chiller plate OR maybe a little raspberry pi mounting solution... Still with your branding/ design, but with a more practical use... MAYBE even one with a small patch panel insert if we can find one for cheap enough.
Any product with dimensionality or that requires manufacturing likely won't come out of my shop for quite a while. I'm a one-man-band with decent tools, but I'm also very expensive per hour ;-)
There's something for everyone in here. Low TDP VM hosts that cost less than $400. A 64-Core Epyc build that was $20K when fully built. High end NAS, low end NAS, 1Gb PoE, 100Gb Fiber...
@@CraftComputing Your videos served as base for when I graduated as a network engineer from Uni and started working, and after a career change and opening my own business, I still reference to your content for upgrading and modifying my network.
I loved your lab tour. I live in the northeast where electricity definitely does not cost $0.07/KWh, more like $0.20. I have FiOS “gigabit” which provides 950 down / 880 up. I’ve had it for three years and it has not once failed, though our power here is extremely reliable. Take from that what you will :)
Currently have to pay the equivalent of US$0.47/KWh here in the UK. Really makes me want to replace my home server with something more power efficient 😅
Question: How is your wear on the consumer Silicon Power SSDs, given that they're in a ZFS array? Also - I think you have the definitions of homelab and home prod environment mixed up, lol.
Should have just used that Brocade switch if 100 gigabit is a waste. I'm running two of the ICX6610s. One with 24 PoE+ ports and one with 24 SFP ports. 40 gig between them and one gets a 40 gig connection to my server. The one with PoE+ is used for my home, like WiFi and cameras. Connected to my UDM-Pro with SFP+, and connected to my desktop with SFP+. The switch with all the SFP ports is for my neighbors who pay me for internet. I live in a campground and their WiFi doesn't work well (campers are almost like Faraday cages). Everyone gets an OM3 line and a media converter than also injects PoE to a Ubiquiti access point. This spring I will be up to 11 (maybe 12) customers. 23:42 I feel the same way about Comcast. I'm on 600/35 from them. It's getting really really old. It's asinine how their network is mostly XGSPON but that last few hundred feet HAS to be coax, unless you're willing to pay thousands. And it's stupid how their business fiber is. They only offer crazy plans for that. A dedicated line, static IP, SLA that makes them fix an outage in a certain amount of time. I'd be fine with PON, DHCP, and a more residential SLA.
May I suggest you try an air purifier with a particle filter? It could substantially reduce the amount of dust, at least it works really well for me! I use a relatively cheap one IKEA sells.
It's a 600sqft garage that houses wood working and laser engraving tools. It's going to be a bit dusty. Easier and more cost efficient to blow out the servers once every 6 months.
@@CraftComputing Every six months? My thought was seeing that dust, that it might keep you busy with every 2-3 months. But hey, if 6 is good enough, I'm fine with it. Considering you busy doing other dusty works there, seeing it in this particular time, it's not that bad.
@@CraftComputing No, no. I didn't mentioned anything about being offended, or anything like that. I just told you what thought was in my mind. It was a perception on periods of dust cleaning. You answered as no performance hit on it. Great. No balloons popped in your hand, and I am lucky, I don't have a rack sized infrastructure. :P Dust still a factor, haha.
You're incredibly lucky! I pay between $0.32/kwH and $0.36/kW, so I have to be really picky about what equipment I need to keep running and what I have to shut down right when I'm done using it.
For your UPS batteries, you might look into 12V LiFePO4 cells. They have integrated BMS's that work with lead-acid chargers and whatnot while still having longer runtimes and much-much better deep-cycle tolerance than lead-acid batteries. They're a bit more expensive than lead-acid packs, but you don't need to replace the whole UPS unit to benefit from the improved chemistry. I replaced the 4 lead-acid packs in my Eaton 9130 rack UPS, and they've been rock-solid for several years, despite frequent power blips and outages in my area.
I considered it, but the UPS is only there to prevent brown-outs and surges, not really for any kind of uptime insurance. For $240, I should get another 5-6 years out of the Pb battery packs, and that's plenty.
I did this with my APC SMT1500rm2u, and eventually upgraded to a SMT2200. The only thing to be aware of with APCs is they have an unusually high float voltage, and it will shorten the life of the LiFePO4 cells. Luckily with the SMT and older APC units you can enter a programming mode with an FTDI serial adapter and adjust the float voltage to within spec for the new batteries.
That t-shirt is very, very, very distracting jesus. I love it. I'll buy one just to enter meetings with it and people to only look at it and reading it over and over and over again which i have been doing 🤣🤣
Did I hear that correctly? You only get 35Mbps upload with Comcast? Good god, I suddenly appreciate my symmetrical 940Mbps fiber even more. At least you have enough to do a remote 4K stream from Plex/Jellyfin, but damn.
Your single year worth of server rack costs is the same as 2 months of my ENTIRE home's usage in the UK. Mental how cheap your power is. We're sitting at around 27p (.43c)/kwh
Nice Rack Jeff, needs a little cleaning, but considering its in your garage it looks positively pristine. You are motivating me to get this sort of thing for my attic (don't have a garage or basement available) and get to learning things.
I still cannot believe the idle power consumption, unless most of what we just saw is powered off 99% of the time. That is considering my R730xd with dual xeonv4 is already consuming around 200W alone with all the hard drives installed. That is without networking equipment, without AC, without console, just the server alone. Having that said, electricity is around 35cent/kWh here. Fortunately I do have a big solar farm, so summers are not a problem. During the winters however I'm hurting and still.trying to figure out a way to at least be able to use the heat the rack produces as heating for the house without also importing the noise level, which failed so far.
If it makes you feel any better, my power company is a small customer-owned non-profit. The big power company that dominates the rest of the area is still $0.22/kWh.
our provider uses a smart meter to change what you are billed based on time of day. from 7PM to 7AM its 8.7ct (CAD), then depending on the time of year the other mid peak and on peak rates are 12.2ct and 18.2ct respectively
Love the 100gbit, completely overkill but hey. Honestly I’d run it if I could ^^ The WAN upload is garbage, I know DSL can have a distance issues because of signal degradation. I don’t know about cable but if your download speeds are that high it smells like ISP tomfoolery… And I have to say it, please clean the rack a little :)
This is an issue with DOCSIS low-split where providers reserve only 5-42MHz for upload and use 54-860MHz for download and live cable TV. To support higher upload speeds, the ISP needs to switch to mid-split or high-split.
This 12:00 is the reason why I decided to go with just a regular 19 inch monitor + keyboard + mouse and then rack mounted it. It takes up more space in my server rack. I am looking for a swinging monitor rack mount that would accommodate a regular monitor but can fold up and stick inside the rack. No luck so far.
Great to see how things are evolving with the home lab! I managed to set up 100Gb networking at home too and agree it’s pointless. I can get around 50Gb real-world file transfers using ksmbd (the kernel-side implementation of SMB which supports RDMA using RoCE) on Linux for the file server and using Windows 11 Pro for Workstations which supports client-side RDMA using RoCE (all using PCIe Gen 4 x4 NVMe SSDs on each end with fast file systems like XFS on the server rather than ZFS and NTFS on Windows). Would recommend looking into this as I get much better speeds than using regular Samba and lower CPU usage. Configuration isn’t too hard as well - ksmbd can basically just be enabled on most modern Linux flavours and is essentially a drop-in replacement for Samba. And on the Windows side (as long as it is the Workstations version or some other enterprise versions I think work as well) it all should just work with default settings (at least with Mellanox cards) if things are configured correctly on the server side. Because this uses RoCE you can just use regular Ethernet networking without needing to play around with Infiniband. But, as you say, you basically never need these sorts of speeds with anything you do in the real world as there are usually lots of other bottlenecks. Would therefore only recommend max 25Gb or 40Gb if using older cheap Mellanox ConnectX-3 cards, for home use.
Hi could you do a Video. On how to run mostly Docker Containers. There are a lot of possibilities and I'm a little bit lost on what is the best way. For example. Do i build a powerfull server instal Linux and then just run them? Do i install Proxmox and then run them in a Container? Do I use multiple Containers? Should i use Unraid for the other benifits and only the minimal lost in performance? Should i run a kubernetis cluster with multiple small Servers? My example would be plex / wordpress / gaming server.
That is what I'm going. I'm running Unraid on an old PC of mine. But now I'm thinking about what should my setup be when I want to upgrade. What dedecated Hardware and usecase should i look into. For example I have a lot of issues with my VM for Veeam on Unraid. Do I run a seperate Server for that or do i run proxmox. But this all depends on my biggest usecase Docker and the optimale Setup i want to grow into. :) Thanks for the reply.
Hi. I just leave this here : Your 350Wh would cost you 1441 Euros @ my old kWh price and on the new price it will require 1747 Euros to run this rack for a whole year... A single kWh costs me 0.57 Euros or 57 Euro cents and i rent a house so theres no chance to install solar panels with batteries :D
So you're limited to 1000/35. In Australia, NBN Co has in the last 12 months or so released 1000/50 plans and has recently dropped the price for it too. Having such high speeds are just amazing. Also, Yes, Starlink is definitely great for backup purposes or even a primary over a 4G connection in rural and remote areas.
Good to see your rigs, interesting Aircon for the rack . I have my rack in the basement. I use it to dry clothes. The temp doesn't get above 20c. Interesting power load. I'm at 400w with 4 servers (2 xcp-ng, soon to be proxmox, pfsense and truenas scale 24 drives) The power cost is 35p (UK price) something like $0.04 So I built a Victron system. That can power the house if there's a blackout. But the main point was to access cheap electricity overnight at 7p. So I charge up the 82kWh battery (3 phase 230v) then run the house off grid for the day. It is also the whole house UPS system.
Here in Seattle I had the choice between Comcast "1Gig" and Centurylink 1G Fiber. Centurylink 1G Fiber is the winner hands down with 1Gbps up and down. Real world speeds are 800 Mbps down and 900 Mbps up. I attribute the 900 Mbps up to most people not uploading much of anything but comes in handy when transfering backups to S3.
Ok first of all, I am not sure why you need all of that for a "home lab". to be honest I will never get the definition of the word "home lab" because most home labbers find themselves using consumer equipment while still getting second hand enterprise grade equipment for no reason and end up using their entire infrastructure for testing and not production. you seem to be mixing enterprise, non-enterprise and test equipment. this is terrible for production. also for tiny environments, the equipment you have is unnecessary. you can literally virtualize almost all of that what you have. also never put test machines on the rack, it doesn't make sense. I have been running servers at my house since the year 2005 and only in 2012 is when I changed from desktop pcs into rack servers. I still have my active directory environment from 2005. now I am hosting tiny websites for two customers using one of my virtual machines on my server system. I always "use" the infrastructure and never use it for testing. if I find myself testing, the entire network goes down and in enterprise, this is not acceptable. this has made me buy more servers. I have to keep asking myself. what equipment do enterprises use? does this system qualify to be enterprise grade? I would never use consumer equipment like non xeon, no name motherboards or consumer hard drives for equipment running 24/7 because they weren't designed for that purpose. however at the same time, your equipment also proves that technology changes, that to keep things more efficient, you could just use modern consumer desktop hardware to just virtualize your test infrastructure because the enterprise equipment out there is old, inefficient and only good for reliability over efficiency. I would still never use them to run 24/7 unless its something non critical.
green-white, green, orange-white, blue, blue-white, orange, brown-white, brown. That shirt irks me greatly. B is for plebs: The US government requires TIA-568A, and only allows B "if necessary to accommodate certain 8-pin cabling systems". Also worth noting that ANSI recommends A for horizontal cables (the vast majority of CAT cabling). So yeah.
This is my first time seeing the 45 Drives home lab server in a rack. I hate that it doesn't match the width of normal gear, and I'm the kind of person who wouldn't buy it for that reason alone.
thanks for this tour! One question - how do you manage your Mikrotik devices? My understanding is that there is no centralised management dashboard, like Unify (or TP-Link/Omada) has. Configuring each device separately, of are they support some form of stacking to be managed e.g. through Winbox? (I'm looking at building my home network with CRS328-24P-4S+RM and possibly CRS310-8G+2S+IN, but their AP range is not looking great)
I just got my first ever server rack : an old Dell Poweredge R710 dual Xeon E5649 that is actually running on Proxmox 8.1.4. I must admit that it as been flawless and more than enough for my needs. I mainly use it as a game server host (running multiples game servers at the same times) for me and my friends. I love it! I think I'm gonna build a server rack with a nas. I mean, once you taste it, you want more I guess... I also plan to max it out before starting to spend money on something newer. I want to put dual x5690 (which they are very cheap on ebay), 144GB of RAM (yes, I will use it fully) and fill up the 8 slot SAS with nas hdd for backups. (The VMs are actually running on a nvme plugged on a PCI-E adaptor, even tho it's old pci2.0 it still reach up to a 1gb/s wich is great for my heavily modded fabric minecraft server). But in fact, I'm gonna fill up the PCI ports with even more nvme that I'm gonna software raid to reach even more.
I didn't got about 10 gig switch. You said it passthrough between 100 gig network and 1 gig network. But your 100 gig switch connected to 1 gig switch directly. Soo may be you simply don't need 10 gig switch any more? You also have 3 more sfp+ ports in your 1 gig switch left. It will save you some power.
You would do well to learn to do your own SC fiber patch cable, it is surprisingly easy and the tools have become very cheap. As for network speed, 40gbe networking (or IP over infiniband) is far far more affordable than microtik prosumer switches. Only real downsides is they use MTO/MTP cables which are not economical to DIY. If you want speed, you need network cards that support RDMA on both ends, which most consumer grade cards do not.
I have two racks in my house. The NAS is using 1kW and my other one has almost all my networking hardware at close to 200W. 350W with everything sounds insane!
Damn, I just have a raspberry pi server with a USB DAS as a NAS... Runs tons of services with only 4 GB of RAM. And I thought that it idling at ~18 W was a lot...
At least you get 35 up with comcast. I get a whopping 600 down and 10 up. I literally get better upload on my cell phone.... I don't understand why ISPs insist on doing this. I would be happy to even have 30. I had 50/50 FIOS in a different area, and it was great. I don't see a lot of use cases above that for me. But 10 is just maddeningly pathetic.
I'm pretty sure you know this, but as the switch does not support RDMA, the best you can do to maximize the use of the 100G links are jumbo frames for now.
Hello Love the channel, I would like to see a build of DIY NAS based mostly on what you can find at an EWASTE facility, i found a lot of good hardware, but a guide would be greatly appreciate, the deal are amazing there also wiggle room for haggling on price...
Hey Jeff.... As a Canadian..I assume you've already had it checked....but as you do reside in the good ol' USA...I wanted to ask if you have had the mole on your neck checked out. This video, and it's "unique" perspective is the first time I noticed it
😊 America land of the Great. American ISPs : Best we can do is one gig down 20 meg up it's your only option so you have to take it. (For the majority of Americans) Thanks, I hate it.
Most people are doing more downloads not uploads, I just tested mine, 52gb upload and 498 download, which is less than I'm paying for but could be either my router or modem, not my workstation, plenty fast enough at this point, but getting ready to set up a server and nas
I wish someone would make a super shallow (maybe 9" deep) rack mounted NAS for use in 12" deep wall mounted racks. Being 5-6 rack spaces tall would be fine as long as it was shallow.
unfortunately i also have cable internet about the same speeds as you. Hopefully they expand the fibre network. You can get 25gb over fibre in Switzerland
Damn, that's cheap power At the top of NYS on the Canadian border next to the Adirondacks we pay 2.5X your price and it's all hydro power with lots of farms being turned into solar farms (but that power is all going far far South)
"Still uses lead acid batteries". That's still kind of the norm for UPSes, the lithium based batteries for a UPS are kind of a new thing and are still pretty niche.
A.) I envy you’re electricity rates😭. B.) For a garage with woodworking usage damn that’s not a lot of dust but do get some of the comments you could at least cleaned it for the sake of this video as it’s technically a showcase vid.
Don't you have to have the same CPU's when you want to transfer files over to get the max speed? That's what it looked like when LTT was doing a video. Putting the same CPU seem to make the speed max out hardware side.
Is a homelab suitable and safe for small companies hosting their own Odoo servers? Odoo is an open-source ERP where I can host my own ecommerce site and accounting system etc.
I keep getting thrown off by you referring to things you did earlier this year... yet this is a 2024 video. I already did the change over of years in my mind so this video is difficult to watch.
Have you ever considered building a dedicated Ai home server? I have a r430 I want to add an accelrator card to but have not really found any build videos.
Still trying to understand why and how someone uses such a set up. People show what they have, maybe how they built it, but not why any viewer should consider doing one themselves.
do you have more information on the drive bay inserts for that those black 2U servers..... i just picked up one of those boxes recently but it has a faceplate on it. would be nice to have access there
I both love and can't stand that t-shirt. I have memorised that with the white and colour around the other way (orange-white, orange etc) and it gnaws at my brain seeing it printed the other way.
Cool stuff. As usual i am as casual a computer user as they come. By that i mean i do not even need the consumer i9s lol. Still like listening to people talk about tech lol
Duuude I love most of your videos but this one: why are you not cleaning out the dust ?? that rig looks like something I get assigned to troubleshoot and I get to go : Not until they get rid of the dust.
My parents just got Starlink. Previously they had a 0.7mbps connection. They only stream tv/movies and surf the web. It has been an amazing improvement for them. The other option sucked so, so much that even with the occasional blip, it's still worlds better.
