@@512TheWolf512 Open source with an optional official support contract, has always seemed to be ideal to me. You get the benefit of multiple sources validating code, and the ability to pay for a disaster. The official provider cant stop people from making their own guides so if there is enough documentation online, you're smart enough and have enough time (The last one is basically never in a corporate environment) then you're golden. The owners get the added bonus of less staff due to the community's forks, fixing and updating code for you.
@@michallv Reminds WinRAR. Never once bought it for myself at home, but back when I was in high school, I remember talking to IT people and them saying that the school district needed a quote for a price to buy WinRAR.
It's also great for businesses that want to test run a software before committing the full price, companies will do everything possible to stick to "if it ain't broken don't fix it", even when it means having to deal with extremely outdated and annoying (but technically functional) software, but if they can test it for free before spending big bucks to upgrade, they are more likely to do it.
Idk if Linus or the staff will see this, but I had an idea for a video I thought would be useful (especially for myself). Have you thought about a video on PC maintenance? Idk if it has already been done, but something like "Here's what you should be doing for your PC health every month, 6 months, year, etc." Cleaning fans, reapplying thermal paste, updating BIOS/drivers, checking stressed connections, etc.
I was an adjunct professor teaching a class on microprocessor architectures. One of the students asked me about how to fix her machine that would run for a while and then give her a BSOD. (Blue Screen Of Death). I told her "I bet you have a cat." She was quite surprised, then asked "How did you know?" It was time for her to clean the insides of her PC, especially the CPU heatsink. She was expecting a software issue...
I've been running an HA cluster at home for a year or two now. The downside is Proxmox doesn't understand "Hey! We have a power outage and there's only 5 minutes left of the UPS." It really doesn't shut down nicely---the cluster will keep migrating VMs as you try to do clean power-off in the dark. For server-down maintenance, it's fantastic though.
Yes, because other than stated in this video, the purpose of this tech is not to help you with hardware failures, it's for load balancing. Won't do shit to keep your system running. It's useless for most people, who do not run a datacenter.
NUT is your best friend, i have it running on a PI hooked to a UPS, when it looses power it monitors runtime, if it gets too low, SAFE shutdown all VM's and then host.
0:01 That chair needs to be replaced before it sheds all over the office. Faux leather gets EVERYWHERE when it starts flaking off and once it starts flaking, it accelerates fast.
This was my senior thesis for university! I designed a distributed fault coincidence avoidance solution using Proxmox VE with DRBD as VM backing storage. It bounced the virtual machine across the machine cluster randomly to reduce the odds of a fault coinciding with the critical process. It technically outperforms VSphere FT (But is not technically a full tolerance solution so it's not necessarily comparable.)
That's so cool. Can I get the link to your paper or your LinkedIn? Lmao this seems so random, but I am also a graduate searching for good thesis topics to study.
It's funny to watch LTT slowly work through the last 50 years of server & DC innovation, as they grow and run into every issue that originally spawned those innovations to begin with. Eventually they might actually arrive in the present best practices.
The weird thing the most times on LTT nearly every video there is a windows machine that is in the blue upgrading screen when you actually want to use it. I think this is a bigger problem especially with the recent crowd strike failure taking out the worlds computers.
The thing about docker containers that can resolve the issue that Jake mentioned: A: run a Virtual Machine just to host these docker containers and the that VM will migrate around the hosts as needed. B: run that container in Kubernetes. Then you can configure load balancing, scaling and other cool features.
Came here to make the Kubernetes comment. If you run everything in containers, you get all this shit for free with very little effort, especially if you run managed k8s like Rancher or one of the cloud managed k8s services.
I think yayIHaveAUserName mentioned this in the disussion forum you linked, but unplugging your server and having whatever VMs are on there going down actually means that those VMs have to be rebooted on other servers. So technically...those VMs are failing, they're just being rebooted automatically. This matters if you're running a program that does not save state before it crashes because you might lose all your progress. It might also matter because you could mess up your filesystem if there were important write operations happening at the time of the crash. Very cool technology, but the video title is not 100% achieved, in my opinion. Also, the clustering section goes really quickly over fault-tolerance (i.e. "quorum"), but I don't feel like it was very well motivated other than just saying having two computers is not safe. Unless I misunderstood, the piece that seems like its missing is that this clustering program seems to be trying to handle Byzantine fault tolerance, where a computer could have a malicious user giving false data, which although is out of scope with your video, is the reason 2 computers with one fault is not safe for knowing what is the current, valid state of the system. Otherwise, why not trust the other computer to have the correct data? Simple redundancy would let you trust the one working computer with the source of truth.
Three things: 1) I'm already running this at home with three OASLOA Mini PCs (which sports an Intel N95 processor (4-core/4-thread)) in each node, along with 16 GB of RAM, and a 512 NVMe SSD. The system itself has dual GbE NICs, so I was able to use one of them for the clustering backend, and then present the other interface as the front end. (Each node, at the time, was only like $154.) 2) My 3-node Proxmox HA cluster was actually set up in December 2023, specifically with Windows AD DC, DNS, and Pi-Hole in mind, but then ended up changing to AdGuard Home, after getting lots of DNS overlimit warnings/errors. (Sidebar: I just migrated my Ubuntu VM from one node to another over GbE. It had to move/copy 10.8 GiB of RAM over, so that took the most time. Downtime was in the sub-300ms range. Total time was about 160 seconds.) 3) 100 Gbps isn't *that* expensive anymore. The most expensive part will likely be the switch, if you're using a switch. (There are lower cost switches, in terms of absolute price, but if you can and are willing to spend quite a bit more, you can end up with a much bigger switch where you'd be able to put a LOT more systems on 100 Gbps network vs. getting a cheaper switch, but with fewer number of ports overall. I run a 36-port 100 Gbps Infiniband switch in my basement. I have, I think, either 6 or 7 systems hooked up to it right now, but I can hook up to 29-30 more, if I need to.) On a $/Gbps basis, 100 Gbps ends up being cheaper, overall.
A video sponsored by intel about a PC that can't fail with its current reliability issues is so funny. I know these deals are sometimes a long process but wow intel did not gain much out of this particular video's sponsorship.
Having spent as much time as I have in my career troubleshooting DRBD issues, getting calls in the middle of the night about dreaded split brain, etc. I would gladly trade some level of performance in order to not use DRBD. Realistically you should have a cluster for storage and a cluster for compute, and therefore not have to worry about using something like DRBD to keep things in sync. With that being said, it's nice to finally see LMG moving closer to enterprise level infrastructure!
I was tasked with setting up a redundant storage cluster for an existing vm computer cluster. Literally got stuck for months back and forth with drbd support trying to get it working. They have a small team amd their support is good. But I seemingly managed to stumble over every bug and problem with their software just setting it up. Gave up in the end and decided to try with Microsoft hypervisor...
I just wanted to comment to say that Jake you’re looking good, my dude! You mentioned before about losing weight and it’s clear you’ve lost some more :) keep up the amazing work!
From experience 1) that is not enough RAM, I'd recommend at least twice as much. If you have memory heavy workloads, I'd recommend even more. 2) You should really use a dedicated NIC for management and coro sync. 3) If you use network storage, you want jumbo frames and therefore a dedicated network. 4) if you have heavy guest traffic, you should definitely use a dedicated NIC for that as well.
I know LTT has never been very 'enterprise-y', but I find it hilarious that the example 'perfect use case' of a hyper-converged HA cluster of hosts is DNS and Active Directory DCs. Some of the most fault-tolerant systems, which are specifically designed in a way for multiple VMs to be used.
Yeah, I wouldn't even necessarily include the DCs in the failover. Leaves more resources for other VMs. Just make sure you have enough of them and don't have them running on the same node. Linus seemed so excited for AD, I wonder why they did not implement it yet. It's not that hard compared to other stuff they did
@@timcappell71 He had said it in some video, but I don't quite remember it right, but I think it was due to not having a dedicated IT team to manged it. Of course, they can manged it themselves, but it would take time from other stuff.
@@michaelkreitzer1369no need to. Have a VRRP/keepalived IP in front of multiple servers. DNS is literally the worst possible example. It has good caching and replication built in, with it being (mostly) UDP you won’t even have a TCP stream disrupted if it changes machines during a failover. It’s the one thing that just works in a replicated fashion. (Also I guess AD also offers some HA replicated setup…)
"beginners should start with Ceph" said no one ever. Ceph has gotten easier to setup and maintain over the years, but that doesn't make it less of a complex distributed application with surprisingly strict requirements on certain aspects of the setup like network stability. It's easy to get it to e.g. dead lock all your vms because mons failed over due to some packetloss in an overloaded switch. Been there, debugged that.
Having run some large Ceph clusters (multi PB), I usually find as long as the network is stable, and performant enough, Ceph is rock solid. Proxmox does a great job of deploying Ceph for you. DRDB on the other hand, I ran out of performance pretty quickly on some database servers.
I've had a three node CEPH cluster at home for about 9 years. My only Linux experience before that was using ZFS for a file server. I've never used Proxmox or 45Drives' Houston UI, but it seems like they make CEPH a lot simpler. Just don't expect very many IOPS out of CEPH.
13:29 are you telling me LMG has been around for as long as it has, with as many employees as it has, with as many severs as it has... and you haven't been using AD up to this point...?
Dude, I was just about to ask the same question. And aswell, they just NOW started a virtualisation Cluster, huuuuh? I work in it consulting and at a certain size, everyone of our customers has a virtualisation cluster.
At a certain scale it's not worth the overhead/cost and if I'm being honest, if you have a Greenfield estate I would just bang everyone on azure directory and side step on-prem AD. Current company is around 800 users with a decent chunk of infra and no AD in sight. Currently looking to implement it for a product as it will provide a better user experience to the workers if a certain bit of kit.
Ltt never ceases to amaze 😂 they have over 100 employees, 100g netowrking, multiple sites, but have yet to implement basic enterprise infrastructure like AD!
I wouldn’t even bother with on-prem AD at this point, they seem to operate already mostly in the cloud and with SaaS apps. Would just be easier to point local servers and authentication to a cloud service like EntraID, Okta, etc.
I was already planning a very expensive upgrade to my home lab that will take a while to save up for and you show me this really cool stuff that I would also love to do. Please stop.
Before everyone’s like “sponsored by Intel on a reliability video”, this isn’t blonde Linus, this was probably filmed like 30 years ago As for that unplugging PC bit, one of my old teammates was part of Live Migration in AWS, which is that “carry over an instance from server to server without noticing a difference”
Who cares? They are a big enough channel to eat the cost of not running the video given current events or to at least delay the video. If you screw over an entire 2 generations of customers you deserve to be clowned regardless of context.
This video was shot (at least partially) on July 18th or 19th.. Linus had the "bald spot" caused by Bell testing the hair bleaching.. And he told on the wan show that he has to go around like that for a couple of days... So... Intel was still fine back then. Well, better than now at least.
@watercannonscollaboration2281 Except you can see the blond patch in back, so we know exactly what day it was done: July 25th (one day under 3 weeks ago from this video) at some time before the WAN Show.
Good for two virtual desktops basically doing nothing. I wonder if there was a couple of hundred task worker / devs or heavy users doing there things. Not so sure.
@@glaucorodrigues6400 in reality, those couple of hundred Devs would be spread across the Hypervisor hosts already, so you'd only get "up-to" 2 minutes of disruption on 1/4th, 1/5th etc of them (depending how many hypervisor hosts you have, probably a lot more than 4 for a couple hundred heavy users)
Yeah that kinda made me laugh - "We made a thing that can migrate VMs real fast... But we wait 2 minutes to do it"... Kinda undermines the whole point of the system...
@@glaucorodrigues6400 If you have a couple hundred task workers on 4 machines... you are already expecting to have a bad day. While VM's are voodoo... thin provision at your own risk I guess.
This is great, your PC blue-screens and proxmox moves the VM to another server where it will continue to blue-screen. It's perfect. VMware have had this functionality for a decade, they even have HA VMs where the memory is constantly synced giving an instant failover that is several hundred times faster.
Yeah, most of the time a BSOD has got nothing to do with hardware failure an thus this entire stupidly expensive sponsor/PR-money video is useless as a method of avoiding bluescreens.
Huge props to the team on this one, this can't have been an easy video to plan/film and keep entertaining. It's extremely exciting and awesome technology but very difficult to show, Good Job!
I have a Proxmox cluster at home that runs off of 3 mini PCs: AMD Ryzen 7 5800Us, 32GB of RAM, 2.5GB ethernet, and shared storage via NFS on my NAS. Those mini PCs give me 48 vCores and 96GB of RAM in my cluster, which is plenty for my home lab. Also was relatively cheap, got those all running for about $1000 (NAS not included, would be about $1000 more for my RAID1 36TB NAS). Also because the Ryzen 7 5800Us are laptop chips, this thing sips power and has great efficiency.
@@BoraHorzaGobuchul it looks like if I mention any links or brands I get deleted, but if you search "5800u mini PC dual LAN" you'll likely find something very similar to what I have.
I have a somewhat mini version of this hardware in a way with two laptops of Ryzen PRO 2300U, 8 GB RAM, 512 GB drive, plus a NAS with 20 TB usable (1 Gbps network tho). I'm now considering getting a third of such a laptop and configuring it into a cluster too... (currently only one of them has Proxmox and the other just Debian) But not sure if it's worth it with such cheap machines, plus they could use some RAM upgrade perhaps. But hey, at least they were cheap (~50$) and have built-in battery backup...
I have the same but with 4 intel N100's half the price, 1/4 the power usage, but also 1/4 the performance. But still, it's plenty to run everything I need. however, I'm gonna look at those ryzen 9000 series laptop chips.
I mean I've known about live migration for years and it still impresses me :P It's particularly cool seeing it done open source with custom HCI nodes, I am curious why they didn't go with OpenStack though.
@@almc8445 IIRC they have experience with Proxmox VE. Also Proxmox seems way easier to manage on this small scale, while still providing all necessary features.
2004 actually, so 20 by now. But yes, what I thought as well. It has come as far as being removed or at least deprecated by many newer DC/Hypervisors/-scalers in one way or another, as it's main flaw is, that it only protects you against things failing below the VM level. If the OS or the App fails, you're still just as screwed. A more modern design, e.g. putting DNS in multiple autoscaling containers or even just VMs on similar hardware cluster, would also allow you to 1. have a failure in any level of application or below, 2. scale and distribute load with demand, and 3. if you do it right, even allow you to be protected against failure do to changes/updates, as you can do various forms of rolling, blue green, or more complex etc. deployments, and still have both older and newer version available to fall back to. With a single VM with live migration, you only option then would be to restore to a snapshot, assuming you have one recent enough. Still causes downtime and potentially significant data loss though, which a modern system wouldn't need to risk. That's also why the container doesn't have live migration capability unlike the VM in their example as well. It assumes you have moved beyond that, if you have implemented containers into your architecture. Nothing theoretically preventing you from implementing the same things on a container level, just nobody cares anymore. Also more resource hungry to implement, as you need constant memory duplication synchronised with I/O activity, if you really want it to work for failure and not just planned migrations. It wouldn't work seamlessly if your memory was restored to a state before you send a network or disk request after all, as it would cause potentially catastrophic inconsistencies.
@@almc8445 why would they prefer OpenStack? Proxmox is much more integrated and easier to learn/understand. OpenStack might have been a good option if they were doing this for clients.
8:40 "If you have a server that doesn't have IPMI I don't even know if that's a server, really" And that's why I've gone to the effort of adding it to my servers. Before I got my hands on it, it was a used Dell optiplex. Then I stuffed pikvm into it and it became a server. It really is the best to be able to adjust bios settings and install different operating systems without having to get on a plane and go to the machine.
Very cool tech. I’ve used HA (High Availability - this tech in the VMWare world) for years and it’s amazing. Even more amazing is FT (Fault Tolerance) where a complete mirror VM is already running on a second host and no “migrating” occurs. Packets just redirect and it’s very magical. Shared storage is usually involved though, so this storage sync method is very cool if you don’t have shared storage. One nitpick though. Your thumbnail implied this would help with blue screens, but any issue at the software/OS level would not be prevented with this setup at all. If you add watchdog tools to reboot or migrate a VM if a heartbeat is lost, that’s completely different than the host becoming unavailable from a hardware issue and doesn’t need this “fancy” of a setup to just address a software glitch. If this setup has a heartbeat watchdog feature, I don’t see it mentioned here, but again, software glitches like a blue screen are an entirely different problem with entirely different solutions than hardware failures.
saying drbd is open source is *technically* true, but if you have ever had the displeasure of trying to actually build their code, you will find out just how far you can stretch the term "open source"
@@FlyboyHelosim "Where is the documentation!?" Right here "Where is the documentation that helps me make any sense of the documentation!? That was me trying to figure out how to make some kernel level changes to make it so the touchscreen on my laptop didn't permanently turn off whenever the laptop went to sleep. I had found some forum post that said " make these changes" but didn't say where I needed to make those changes. I tried looking through the documentation, and I felt like I needed a masters degree in Linux to understand any of it. I gave up and reinstalled Windows so I could use the working driver.
@@dekkonot sure, but sometimes getting something to compile is like trying to trial and error summoning a demon with black magic. It's esoteric, the error messages make no sense, and no one is willing to write the programming version of the freaking Ars Goetia so I can do it!
@@toon908 Remember in that 2 weeks they also had to build, test and commission 3 more servers, deploy them and configure the cluster, then film the demos and *then* edit the video. Pretty quick actually.
For my work, the production environment runs on Solaris clusters for HA. It was interesting to the differences between the configuration and management tools and interfaces. One thing I would note is that one still needs a good UPS system across all that hardware infrastructure or all the HA fail over won't mean anything with a power outage. I know you all have those big Eaton UPS systems there.
@@Frank-li8uj They are Potentially also affected by the same issues, though it's not confirmed yet from what I've heard (or not). It depends on if the chips where made at the same Fab and so on...
They did not explained why quorum is important in cluster environments it is made to handle conflict's when offline node still may think that is working and up, but if it can't see majority of nodes, it can't be true source, thus it needs to be decided by majority what to consider true/online source and what is broken. so, if you can't get majority without failed nodes both broken and working parts would become of same importance. every failover cluster can use tie breaker machines like proxmox can use rpi or other machine not running full proxmox just with small program installed, windows server failover can use smb share running on some other server etc.
No, that's not true. I've got Floatplane, and they usually put videos on both platforms at the same time-ish. Only for scrapyard wars on Floatplane we've got a few days of "premiere time". The vast majority of the videos are uploaded at the same moment.
If you look carefully, this was shot at most a few days before the wan show with the hair dying - he has a dyed patch on the back, which was done as a test
finally hearing them discuss active directory and diskless systems coming up in future videos has me geeked , I've been requesting this since they started discussing vms ! even tried to audition a script for pxe network booting but they didn't take me up on the offer 😢 can't wait to see their take on it though, been a long time coming
That drive redundancy is 100% necessary with those SSDs. I deployed around 50 of those and I've had 6 fail within the last 6 months. Absolute garbage drives.
FWIW, it looks like someone messed up the links in the description. The link to the intel Xeon goes to the website for the crucial RAM. Everything else seems to be correct thought. Neat video! I will literally never be able use one of these, but it's cool to see how it's done.
Probably RAM or Power supply. Check your reliability history to try diagnose what is causing it, it will tell you weather it was a hardware issue and give you some info that might help diagnose the source.
I think it's very cool that I learned all of this in some college classes and literally got a degree for putting this kind of technology to use. I appreciate the refresher course Linus!
I'll be using my first commenter privilege today. Because of LTT I'm now in ECE engineering (The closest branch I could find to work in pc's) and plan to do vlsi and hopefully will end up in chip design on either the gpu or cpu side. I do watch the main channel but WAN is my jam. (P.s please release WAN on spotify earlier cause 2 days is too long) I love the wan show too damn much. Much love! Sumanth
live migration is super cool, a while back I tried it on a small 2 host proxmox cluster and I was just AMAZED to see it migrate a VM from one host to another with only a few 100 ms of actual downtime
Some small additions: Make sure the nodes all have the same CPU feature set available or make sure your VM use the least common denominator of CPU feature sets or you won't be happy because if your VM runs on a node with e.g. AVX-512 and also utilizes that but then the node goes down, the VM won't work after a migration if the other nodes don't support AVX-512. Also for your VMs, you can overprovision CPUs, but you can NOT overprovision RAM.
Sapphire Rapids is pretty reliable since it’s based on the old Alder Lake micro architecture, its performance is inferior to EPYC but it’s easier to get your hands on entry level XEON.
I already do this but without distributed storage. I use HA in Proxmox, and it will copy over the VM's disk every 15 minutes. This sounds awful, but it only copies over the changed bits, so it's only a few seconds on a 1gbit connection. Then if I migrate it does the same as in the video, copy over the changed bits from disk, from ram, and then resume on other host. If the host goes down unexpectedly it will not resume from the current state, but it will boot from what it was maximum of 15 minutes ago, which is fine for my use case.
@@nicesmile3125 nah, they did a test spot on the back of his head before bleaching the rest. So this was maybe the day before he bleached the rest of it
As some one that has designed and built plenty Hyper-V cluster and dabbled with VmWare and vSphere... This is extremely cool, just because they are allowing you to build an Enterprise level hosting environment for free
As a rule of thumb, any component that is a few years old is more-or-less absolutely reliable. I've still got a system running on a Ryzen 1700; though that was solid when it released, let alone 5 or whatever-years on from now.
It's not the age of the component, it's the run time. Some people will only notice the degradation in their years old Intel 14th gen CPUs in a few years if they only use it lightly and don't update their bios
My grandpa had the most secure computer in the world a prebuilt windows 7 machine that was never connected to the internet. He played soltaire, I played pinball was endless fun back in the day.
I build this setup with 10 year old servers, 1gbe between them and Proxmox and linstor a few years ago. This served the production traffic for a not so small professional webhosting scenario. It needed some finetuning but linstor works quite well with not so fast bandwith.
So if we are basically just bouncing an OS between storage/hardware, how does this avoid operating system level failures? Referring specifically to the "Infinite days without a bluescreen" thumbnail. If something were to fail on the software level and cause the OS to BSOD, just moving that faulting software to another set of hardware wouldn't do anything, you would just have 2 bluescreening computers.
69 is overrated. It looks fun but it's hard enjoying it while you're concentrating on making it enjoyable for the other party. I find that doing those in sequence as opposed to in parallel is much preferable
"all at a modest 300W TDP"? Are you freaking kidding me? C'mon guys, yes it's more efficient then a bloody 14900KS, but this thing is designed to run 24/7, you can get much more efficient 32 Core Processors. Just not from Intel lol.
@@insu_na This lmao. AMDs server CPUs were dog thrash early on. They're just now getting to the point where you could consider them for your next deployment. AMD is legendarily awful in the server space.
@@Pipsispite They've been undefeated in performance for many years now. Sure idle wattage may not be great but datacenters don't leave their hardware idle most of the time
I'm somewhat impressed, yet also not surprised that you're not running a DC. With the employee count going up, it only makes sense to start centralising AAA.
Now I'm tempted to do this to a bunch of Raspberry Pis for "minor services" I'm running at my house. Plex server (a Pi5 is more than powerful enough to replace the old Intel Skylake compact desktop doing that now,) HomeAssistant, PiHole, and other similar tasks spread across a few different mostly-old-desktops-repurposed-as-servers. Although I'd really want to add 2.5GbE to them. One PCIe 2.5GbE "hat" plus one USB 3.0 2.5GbE each should do it; with the onboard GbE as the management interface.
Just wanted to say I love the way Jake describes stuff. I have almost no idea about any of this stuff, but listening to Jake talk about it, I just kind of 'get the idea' enough to follow along. Sometimes hearing people talk about super technical stuff outside of my sphere, I get extremely lost very quickly. Not saying I understood everything, but I understood *enough* to follow along with the video. Y'know?
Much simpler method, but following the philosophy of the video: 1) Install your preferred Linux distro (Proxmox it's nothing more than Linux with a fancy interface to run virtualization, containers, ...). During step 1, delete your existing Windows installation. And, problem solved in just one step! Much simpler, doesn't need a server or a fancy network, ...
I expected a video that was more to the tune of setting up a RAID array, installing ECC memory, downclocking and installing huge tower coolers without fans. But this works too I suppose :D
Fault tolerance in Cloud Computing is basically the same thing. We just use multiple servers which we don't own and have no physical access to. We rent them in a way which reduces the costs a lot. Seeing this being done on physical hardware was a great experience and a good demonstration of how it is done. Knowing the theory behind it and seeing it happen was fun. It is kind of uninteresting when you use an interface to do it and have no contact with the actual hardware. Gonna do this one day when I create my own server.
