After i found your channel (from the home lab video) i got hooked on your content. Im a Software Developer from Brazil with a passion for tecnology and Japanese cars and here i was thinking it was impossible to find both of these contents in one place, yet here you are with a home server AND a video about the EJ engine! You are probably a busy person but if you could give us more content on what you do on your free time that would be amazing, thank you for existing!
hi Jeff, i love your content so far, i would love if you could go into more detail about the home building details, how you planned out your house, infrastructure, concrete vs other materials. I think the subject would be interesting for some of your viewers. Best regards, Dan
This is useful not for only homelabs, but for setting up university labs on a limited department budget ... thanks for this, been looking at how to justify the purchase of a UPS for an observatory's automated systems. The invertor usage was one reason I'd used before, but I never considered voltage distortion could be caused by random debris on power lines some ways down the line. Our equipment is in fact extremely sensitive to this, so that is helpful information and may explain some rare problems.
More thought than I want to put into this part of my homelab. I do monitor this through Solar Assistant, but my feed goes into a single 1500va APC UPS that does a pretty good job, it seems, but my homelab is no where near the size of yours. Mine maxes out at 500watts with the loads I put on it. I might add an ATS, but right now, if I'm down on off-grid solar, I physically move the UPS cable from the 120v out to the House 120v, which is also close to 75% from solar as well. My home office, is also powered by solar from an extension cable run from the load out on an Epever charge controller, to the battery-in on an Eco-Worthy, Charge controller/ATS/UPS that works pretty well. That system is powered by 900 watts of panels on the roof of a tool shed, and a home built 1.3kW recycled 18650 battery. We should probably start a group of Homelabbers powering with renewable power sources. I no longer feel guilty about running my servers 24/7.
Incredibly informative, well done. This was all explained very simple to deliver a somewhat debated topic amongst the IT crowd. How to balance the costs, pros and cons of different solutions.
Hey Jeff! Love the explanation videos so far. I am in Australia and the *UPS fire* comments ring true for me. I had 2x miscellaneous 3000VA ups units that got wiped out by a being in the path of a leaking toilet flexi hose. The firies got called because they let off the magic smoke and I was in hospital at the time so it left the family to deal with the "electrical fire". I now rock two Cyberpower OLS3000E with Online (Double Conversion) for the obvious advantages that those have. Note to self- don't sit UPS units on the ground ever again. I moved most of my kit to the DC as I am starting a small business and now only have a half rack for my home lab. One UPS is for the rack and the other I have connected to my main PC/rig.
So true on the UPS fires, we had a brand new APC go up in flames within 30 minutes of plugging it in (this was 20 years ago). Nearly took out the server cabinet it was in. Luckily our manager heard the arcing and was able to unplug it and smother the fire.
It may sound counterintuitive.. but undervoltage events can be a total hazard for some devices. At my old work place, we had really instable power, where one day 1 phase of a local transformer had failed but didn't shut off the other 2 phases (the installation was 70 years old). This meant that in our offices, some devices shut down as they were on the failed phase (inconvenient no damage), but other devices kept running on the other 2 phases... however its line voltage had dropped to 90-120V instead of 240V! We had some 240V-only lead-acid battery chargers running that went up in smoke. I suspect those chargers tried to deliver its nominal output power despite the MUCH low line voltage => a LOT more input current => P=I^2 x R => poof. Fortunately, chargers and batteries can be replaced.. but talking about fire hazards.. Now I suspect many computer PSUs are protected with UVLOs and refuse to run with a too low line voltage, but be careful as not all equipment may be as well protected.
Thanks very much for sharing. I saw another comment asking if you'd do a tour of the fire system - I second that. I also appreciate the bit about 240v - I am about to run 240 to my lab and it took me far too long to understand what exactly single phase 240 is, and that 208 isn't really an option for residential. Looking forward to whatever you come up with next!
Jeff, your home datacenter is amazing. If you have a chance in the future, could you talk a little bit about your hypervisor configuration and do you use any smart KVM solution there?
Thank you for taking the time to share your home lab setup. I first saw your lab posted on reddit so I was very excited when I saw a video of the same lab pop up in my youtube recommendations. Looking forward to your next video!
I’ve not heard an American advising others to bring 240v into the server room for increased efficiency. It’s good advice which is almost always overlooked.
Yea, there is really nothing but upside to doing it , even in a residential setting. In real production datacenters they are almost always 208v due to the 3 phase power.
I'm very interested in the cooling video as I suspect that you are recylcing the heat generated by the servers as much as possible within your house? I mean this thing was literally considered while building the house so it'd make sense. My homelab - a far cry from yours - outputs at least enough power to heat a terrarium :D
Jeff i really enjoy your videos, keep it uppp!! Your passion and knowledge gave me some motivation to study, which i lacked of in the past few months. Greetings from Italy.
Cool video. Yeah, here in the US we have pretty consistent energy flows, and so the double conversion UPSs really aren't necessary, but in say, Latin America, it's a totally different story.
One thing I might suggest is to get a flood sensor and put it in your mechanical room. While rare, plumbing can fail and quickly fill a room. My flood sensor saved my bacon last Friday when a plastic valve on my water softener blew (possibly due to a momentary over-pressure event from the city's supply) and my alarm system notified me and I shut off the water to the house within one minute. Thankfully only about 10 gallons sprayed out on the floor but it missed all the expensive stuff in my server rack 😌
Indeed there is both a flood sensor plus a large drain in the floor. I’m also using a Flo from Mohen which alerts me if a fixture is on too long or a leak pops up. It is always good to have backup for problems like that as water can do so much damage.
I love that I stumbled onto your channel. Out of curiosity, what is your amp service from your utility. 200A? Or were you lucky enough to get something higher like a double feed for 400A?
My cheap APC UPS are all capable of detecting low voltage without passing it through. You don’t need a line interactive or double conversion to get protection. Most equipment will happily run at 90V, just not long term.
Yea, most of the UPS will trip with voltage that low, so then you are on battery with the inverter. I agree that most modern stuff is pretty resilient to low voltage - The one thing I have seen is strangely enough is some cheap ATX power supplies that die at anything below 100V. Given they are switching power supplies they should be more than capable of handling a low voltage like that... of course I did say cheap so that is that. Of course if you are running 240V, those voltage dips make even less difference.
UPS - Don't pull UPS down to 10% if using lead-acid chemistry batteries. This shortens life quickly. Stop at 50%. APC's UPS's are great with crappy power but also great at cooking batteries. Mounting, many are low for the weight as well as if the batteries leak, no servers under to get stuff.
That is good advice - I did ask an APC engineer about this topic a few years ago and he suggested that when the UPS says 0% it is actually close to 25% (based on cell voltage)... this going down to 10% isn't really taking the batteries down to 10% of their real capacity. Of course he could have been making this up, as it is not documented anywhere I could find.
@@jeffsponaugle6339 It's partially true but for a different reason. When under load, the batteries are being pulled on way too hard for their design, SLA is not designed for high currents and have a lot of internal resistance which a biproduct besides heat is Vd (Volt Drop). The battery management sees the lower voltage thinking cells are done before they are actually done thus the early shut-off. What sucks in the APC is the battery sensing circuit drifts with age and you will find them pushing higher and higher float or bulk charge voltages and thats why you get cookers. There are some guys who reprogram them, others replace the crap componeants, I myself, dropped all common UPS's and built my own with LF inverters that support UPS/Bypass function.
Very interesting. I would have assumed the UPS logic would include derating logic based on current flow to adjust the measure cell level, but perhaps that would not be accurate over the long term. I did not know that the battery sense drifted... that would explain the observation that these old APCs seem to cook batteries. I'll have to look into the reprogramming/recalibration!
