Please do some interesting tasks like NAS or PiHole or VM OpnSeense/pfsense as MOST users who would watch this beginners video will need those steps. Basically something useful besides install and UI overview
Hey Lawrence, check this crazy idea…. Most IT people have a laptop and a desk rig. So the idea is, how about we install proxmox on the laptop to get (macOS/Linux/windows/etc/on tap). Then we have a windows vm for xen-orchestra to manage the desktop wrkstn running xcp-ng with oses and projects running oses 24/7. Or maybe we can do xcp-ng on both, and we dual-booth the laptop for when we need orchestra???
With the free version of ESXi gone and perpetual vSphere liceneses migrating to a subscription, this topic will be really important for those of us who are using ESXi and looking for a replacement
Been using XCP-ng in our DC for years now. We are running between 10-20 physical servers and some 50-60 virtual machines. Its been rock solid and once you get used to the Orchestra you will love it.
I'm looking at this as a VMware alt for maybe running 12,000+ VMs across two data centres - 200+ physical servers. We are starting a POC. You think it could do the job ?
@@sagan666 12k VMs on 200 servers sounds like a huge setup. ~60 VMs/server sounds like massive over committing or beastly servers. ;) I am sure that with the right implementation it can be handled by xcp-ng, but it will require a lot of testing. I wish you good luck with the project.
I was a Citrix Xen Server junkie for many years. So many virtual machines, it was my starting point for multiple VM hosting. And this Citrix went no more free licencing for home users. So I kicked over to XCP-NG and then Xen Orchestra(docker) and the next chapter started. It is a great hypervisor but personally when shit goes south there is not enough documentation to help resolve issues, I had some issues with data corruption and spent too much time trying to resolve.
Yeah that's been my experience with xcpng as well... Not enough information/documentation readily available to effectively deal with complex problems yourself. I've found PVE to be generally more reliable and easier to fix with better reference material. The fact that I don't have to go to war with the "appliance" OS package manager to install troubleshooting tools and other things is also a big plus.
Work with the Citrix XEN implemetation with more the 100 VM's and i hate the implementation that Citrix has for XEN. When i look at Xen Orchestra it looks much better although i have no experience with that. I compare it mostly with VMWare ESX as that is the main thing i know for over 10 years for both work in large environments and also as my homelab as it only needs restarts when i do mayor upgrades. I do not run inbetween hotfixes if nothing is broken in my homelab. So uptimes are over a year in most occasions for my homelab. Since the forecast of VMware now it's Broadcom that owns then i am looking what to switch to if something changes at VMware in the near future and it is no longer a valid option.
I have been finding quite the opposite, but I come from VMware (and esxi), VirtualBox and a smattering of Hyper-V. Using the same iron in my homelab for testing, I had XCP-NG and v-USBs (USB hardware pass through, used with Amateur Radio, HDMI capture and editing, and other uses) set up for several VMs in just a few hours. I spent nearly a week with ProxMox and never had them set up well enough to be stable. I think it may be more of 'what are you used to', or trained on.
@@dreeastwood2500 I have a USB Etherrnet adapter, and am willing to try to test for you. I have multiple other USB devices, sound cards, Amateur Radios, radio scanners, etc in XCP-NG pass throughs that work well but was never able to get them to work with ProxMox...
@@dreeastwood2500 Just spun up a Debian 11 VM and tested 2 USB Ethernet adapters connected to a XCP-NG Host, passing through to the VM. Both were detected and usable by the VM as if they were directly connected. I have one mac in a static on my DHCP, and it pulled the correct IP and settings, the other is straight DHCP and also worked as expected. I haven't passed through any PCIe hardware, but that is supposedly doable also.
@@kevinshumaker3753 I appreciate your response. Thanks this makes me so much for comfortable in switching my setup. Will be migrating in the next month or so now.
Wow, it's not common to hear an opinion that is balanced and fair on the mighty Internet. Thanks a lot for the video, your feedback will be heard ;) we also want to make home labbers happy, and I totally get your point of view. Also, XO lite will partially solve one of the biggest point, that's why we started to work on it few years ago. Eager to show you our roadmap and progress at some point!
Would love to see you guys hire a full time community manager at some point. Some of those techbeard takedowns of newbies in the XCP-ng and Vates forums are... *rough*, and alienate potential users who prefer the "home mockup on junk->take seriously at next NOC review" path, versus those coming from the enterprise trying to get to more open, or cut costs on the heavier enterprise metal.
@@deano1699 "Some of those techbeard takedowns of newbies in the XCP-ng and Vates forums are... *rough*, and alienate potential users" It's especially bad when the CEO of the company and the project does this. Beyond the lack of some of the technical features/capabilities of xcp-ng (which Proxmox is able to do), and the willful political alienation of (some) new users -- I put it in a head-to-head competition between xcp-ng and Proxmox and Proxmox ultimately won, first on technical capability, and then second since their CEO wasn't involved in any "takedowns of newbies". I will never understand why the CEO of this company and project, think that it would ever be a good idea for the CEO to do something like that. (TrueNAS Core/Scale, was a late entrant into that competition, and it also lost very early on as well. There were some things that it could do better than Proxmox (GPU passthrough), but ad simile to xcp-ng (in the forums) -- the TrueNAS forums crowd is only marginally better in that the CEO of the company and the project isn't involved like that, in the forums.)
Still the best backup system on the market today! Ive tried them all. With UI updates, storage/remotes re-, working, and lxc containers could really give proxmox homelabers something to think about! I'll be spinning up a proxmox again just for container mgmt. Although running a nested docker host VM has been smooth as butta. I'll never give up my XCP-ng and XOA from Vates tho!
@@mt_kegan512 I guess that it depends on what you use it for. I was "drag racing" Proxmox and xcp-ng (and TrueNAS, by extension) in a three-way head-to-head competition and both xcp-ng and TrueNAS failed the technical qualification tests. Unfortunately, when I asked or reported the issue to the respective teams, both Olivier Lambert and the TrueNAS teams were equally belligerent and/or argumentative. As a result, the failure on the technical front, plus the attitude of the developers for both of these projects -- and especially the attitude of the developers for both of these projects -- I ended up using Proxmox instead. But again, it really depends on what it is that you are doing with the software and the systems.
I've messed with XCP-NG a few times, and I always end up going back to Proxmox. I much prefer the Proxmox VM console. I can't believe the XCP-NG console is locked into the browser window and can't be opened separately. The backup features are really nice, but running Windows VMs is kind of a pain with the integration features. VirtIO works much better in that regard. I do think XCP-NG is better than Proxmox when you have multiple hosts, but that's not something that I do.
You are expressing exactly my feelings when I tried to move from Vcenter to XCP-ng the first "few" times... because I rolled back 3 times to Vmware. The "intuitive" interface is not really intuitive at all - for technical people. It reminds me the first time someone from the Windows world tries to use the MacOS or OS/X paradigm of Apple UI: what is seen as intuitive by some is not really that for others. Like Ubiqiti interface: it looks very nice, but when you need to do something, you has to fend pages and pages and pages of drilling down to what you need to find first. Not my style at all and not productive either.
The beauty of proxmox is that it's basically Debian+KVM+GUI, very low overhead, nothing exotic required, and all of the KVM knowledge plays nice with it, so if you do need to do highly automated stuff, you can still use it like any other KVM host.
is Proxmox a type 1 or type 2 Hypervisor?. Sounds like I need to give this a try. Been using XEN-SERVER (Citrix Hypervisor now) in client environments and XCP-NG in lab (but willing to put XCP In production environments too). But Proxmox sounds intriguing at least for a home lab , would you agree?
Hi falazarte, can you let us know what issues were the big huge ones that turned you another direction? I have two XCP-NG hypervisors set up for high availability (in a pool) that were just set up as a test, and it's time to wipe that out since it's not easy breaking a pool apart without a wipe out (as I'm finding out), and either load XCP-NG again (this time just as stand-alone hypervisors not linked together), or move to a different open-source Hypervisor product that doesn't have a bunch of features withheld in a pay-wall. What would you recommend? Thanks Kindly for any help.
I don't see how using an external controller is better than proxmox approach. When you deploy proxmox cluster, you can also access it from a single UI. Pick any node and use it as a single point of access. You even have a backup: if something goes wrong, just use a different node.
Having run Citrix then XCP-NG for many years, I recently moved multiple clusters to KVM. This was in part because XCP-NG uses a horrible virtualization agent on Windows and Linux guests that is clunky and has more than it's fair share of bugs. When you move to virtIO it really does make a difference to guest performance, much more so than XCP. The whole uninstallation of XCP-NG agents was a nightmare!
Removal of the agent software was the worst part of the whole platform. It left nasty registry entries that cause BSOD on Windows Server guests. We wrote scripts to remove 99% of the offending entries, however each migrated host still required manual intervention and had to be checked. Then once migrated Windows Update Software Distribution had to be deleted and re-created to prevent Windows applying XEN guest agent updates which would then cause BSOD on next reboot. When you are migrating thousands of VMs it very quickly becomes a big mess. Never again.
@@DarrenMossAU Not only that, Every update it will change the vNIC drive and it will remove the static IP...on a server!!!! That was insane and it caused a lot of problems to us. We have to stop Windows updates.
@@falazarte There was a registry hack to fix that problem... which of course resulted in another reboot. Then once you set the IP (again) it should be consistent. Messy and painful for application servers because services would fail on boot, requiring another boot once the IP was reset. Glad we moved away from that.
@@DarrenMossAU That's crazy that had to be done...We are very happy with Proxmox so far. Trying to eventyully move to a more rebust hypervisors but for sure KVM based. No more XEN.
Bailed from XCP-ng because its upgrade cadence was too infrequent. Was waiting forever for TPM support... Compiling Xen orchestra to manage multiple hosts, a bit of a pain. Managing multiple machines on Proxmox also has its down sides (quorum requirements prevented me from allowing servers to remain off and save power...) but still a manageable experience if you have 4 or less hosts, (I have home lab, not a high availability environment.)
As a SMB I really think about giving it a shot. Its only 5-ish virtualized servers (hosts) in our rack, but clustering them all in one management interface seems pretty tempting. Even the $85 tier could be an option if I did not get the source compiled.
I'm relatively new in the hypervisor scene, tested XCP-NG and Xen Orchestra vs Proxmox and found that the free community edition of Xen Orchestra handled all of my immediate needs. Super excited to get your input
i have run proxmox for a few years and have everything set up and working perfectly, have many times considered trying something else and have both looked at xcp-ng and other vm. but after watching your video xcp falls away as it was way too messy.. so thanks for this video and probably prevented me from doing something I would regret.
Did you check out the restore health check? Where it automatically backs up, then restores to check and make sure the backup is viable? Once I spin one of these up, cant wait to try that. Tom Lawrence shows it off often.
Its an important feature, my professional start in IT in the early 90's and one of my most impressive saves to date involved a hardware failure in a Netware 286 server, backups that had never been tested, the old IT person who was more into free vacations kicked back from vendors than keeping things working, and me, a HS dropout with no formal education but the heart of a hacker and drive to solve problems. @@LAWRENCESYSTEMS
Great explanation of the features and getting started. I personally don't have a need for this setup but if I ever do, I will know what video to watch to get the right info.
From VMware in the 00's to Hyper-V in the teens, to XCP-ng in the 20's. With regards to TCO, XCP-ng is the hands down winner. Love that Xen Orchestra is available from source with all the features necessary for the Enterprise. Zero regrets after nearly 4 years. Also considered Proxmox when moving away from Hyper-V, but it just didn't feel like a good fit for the enterprise at the time.
As Mac/Linux user Proxmox is best choice. No additional resources (docker,vm) for WEB-UI and no need for desktop app (which is windows only).. also it's great with IAC tools like Terraform, Ansible e.t.c for spawning multiple VM's out of box using Proxmox API. So XCP-ng does not fit any of my needs.
I've used and am using both. XCP-NG is simpler and great for an all in one solution for VM's. Proxmox has greater utility but is much more complex for things like HA failover.
You should try getting GPU passthrough actually working on a windows VM. For some reason I can't ever get this to work. I want to have all of my machines virtualized but gaming on a windows VM is broken.
I love Xen, I run multiple big clusters in my last job....but it is just that SMAPIv1 that really drags this thing down, e.g. you can't have disks bigger than 2TB, live extension of disks is also not possible, etc. :(
I don't really understand why you would mention truenas in that mix as it is vastly different type of product... Foremost a NAS sollution with some virtualization capabilities (which admittently have gotten a lot better ofc with scale). But I've tried xcp-ng several times in the past, even recently after having used proxmox a lot over the last years.. XCP-ng is not my thing... I absolutely hate the UI of the interface,, the undescriptive error messages and basically a lot of smaller and bigger things which I fail to even remember now... But good luck with your system.
Ive used citrix and xcp-ng long before proxmox never was a fan. I'll stick with Proxmox. Its so much cleaner and easier to use/update/fix. It is also way better documented.
We run in production. You do need 3 servers to use correctly and with more VMs it makes sense. I think we have around 50~ VMs. My personal server in the data center is also xcp-ng
On the intuitive bit - it's a more "20 years ago" or "European" thinking process (not that those two things are directly comparable). You see the same thing from other products from EU companies a lot of the time, especially lined up against similar US-led projects/companies.
@@masterTigress96 Europe just rocks steady, about 15-30 years behind the times at any given moment. There are definitely "blips" of them getting out in front... but it's rarely reflected in technology itself... much more likely to happen around policy stuff. Which is also relevant: Euro-devs will actually think past their wallets when developing software, in a way few Americans will. When you don't have to worry about losing your healthcare, housing, or educational options based on income level, you don't spend every waking hour trying to monetize every last bit of code, I guess. Anyway, they just tend to focus less on creature comforts and user experience until after they've nailed the code efficiency/security/stability side of things. And since the latter is a constantly-moving target, they just... never get around to making the code usable by anyone but fellow devs. Sort of a "if you want to administer my router os, then prove to me you're a legit network admin by doing it all on the commandline, starting from default block all traffic"... And then they'll do this for home routers as well, and... well.. there's a reason why there aren't as many successful European companies in tech/networking/consumer products. And sadly, it's not just a software problem. I'm absolutely done buying italian radio equipment... and they INVENTED the stuff! It's just nuts how much effort is putting into making things unnecessarily difficult. At least starting from "totally usable but insecure" gets you enough customers to hire on people to fix all your zero day bugs, ya know? ;)
The limitation with the number of VLANs did not meet my needs to run hypervisor for network services (including pfsense community and a range of other network server VMs) on the host. I needed to link aggregate 2 x 10GBE SFP++ ports across to a Network Switch (48x1GBE POE + 4x10GBE SFP++) and then VLAN trunk the aggregated link within the hypervisor to fan out multiple vlans for pfsense to firewall between the vlans as well as the internet. I couldn't do that with xcp-ng even using admin shell. The other 2 10gbe on the switch serve my daily driver and a intel xeon running truenas scale. I am migrating those vms to proxmox to see how that goes. I'll keep xcp-ng on an old i7 for ad-hoc learning/research using 2 x 4x1gbe to get 8 of the vlans to it as well as the mainboard gbe for oobm.
"Proxmox. Get your shit together. Give us a proper host backup." Maybe I am missing it, what does xcp-ng backup functionally do that proxmox backup (to either vzdump or pbs) can't do? Love the video, trying out xcp-ng after watching it.
I loved running XCP-ng on my home lab for the two years I had it, but some of the more weird issues like the meh virtualization drivers and larger than 2TB virtual drives pushed me towards something a bit more modern and ambitious. Since i was already trying to learn DevOps amd Kubernetes, Harvester seemed compelling, and has proven to be my favorite Hypervisor so far, despite how new the OS is. While not necessarily easier than XCP-NG, its been brilliant so far. However, i run several dozen VMs and k8s ckusters across 8 physical servers so I wouldnt consider my usecase normal by ANY means...
I’m moving from xcp-ng to proxmox because you have to be a Linux expert to do things like USB passthru and passing a physical optical drive to a VM on xcp-ng.
I wish they have more GUI features. I love proxmox for that. For homelab, Only time I had to use shell is for graphics card passthrough. Otherwise, proxmox is great with GUI, specially with storage management.
How did you run two servers in one pool? Please make a video on this topic. I have four servers (Dell R710, R720, R730 and R740). I installed xcp-ng on all of them and depoly them in one XOC, but there are 4 pools and 4 hosts.
I've had two issues with XCP-ng. First while managing multiple VMs, the navigation from one to another requires many clicks. When you have many VMs, the default view only shows 10 and paginates often requiring me to use the search feature. While search is powerful, you have to redo it every time you enter the VM list. There is no way to go directly from one console to another. Xen Orchestra just becomes a major pain when managing more than a handful of VMs. Also PCI pass through support is nonexistent in Xen Orchestra, so you must use the command line to setup. My second issue with XCP-ng is that it is Centos based. While they've setup their own repository to manage updates, any extra libraries that you might need on your host (dom0) you have to install the RHEL extras.
You can customize views by clicking the person icon just under "Sign Out" and I rarely need to look at the console of many VM's but if I do I just open as many tabs as I need.
Use the tagging feature when you have more than 10 VMs. It's wonderful. I too sincerely wish they had easy PCI passthrough. That and native container creation would literally make XCP-ng perfect. I'm willing to overlook those things however for their FAR SUPERIOR backup modules!
After running a home server for 10 years first on esxi, then xcp-ng and orchestra (sources) - i have now moved to synology because of the turn key hyperbackup. I found that esxi does have better performance than qemu and luckily hyperbackup allows you to backup the free esxi version. But if I have to go back to some other backup solution, it will be back to xcp-ng and orchestra. As that stuff just works. My only gripe.... how the heck do you clear logs! I know I could probably try and get access to the redis server and delete from their, but why cant I just have a delete button for the backup logs???
I want to create a shared heirarchical distributed file system (? Ceph) + Access speed tiered memory pools .... I want to create on from cloud to/and from a single machine to ..... I was thinking of a peer to peer ai assisted unified data/processors/sensors mesh.
I could never kekW . I'm too used to Proxmox, every single button,option,feature in Proxmox is just what i need and even when something is broken , i can fix it (i never said easily , but i can fix it lmao)
I where using xcp/xen for years in our organization, and long time ago started to migrate to proxmox, I prefer it much better than xcp, features and intuitive ui nothing to compare
You touched on the answer as to why the backups are called remotes. Since the XO instance can be hosted pretty much anyway it may not have access to your underlying NFS mounts on the hypervisors. In that way you might have separate backup storage someplace else as well. If your XO was someplace without access to your NFS then adding the same mount for backups wouldn't have worked. (The remote mounts NFS inside the XO VM and not on the HVs). Overall nice review though!
1) You should try to share data between the VM Host WITHOUT going through the network. :) An example use case of this is your storage server hosts your media files and you want to run Plex Media Server. Why go through the network stack if you can just communicate and share data DIRECTLY with the host, WITHOUT having to go through the network stack? Try it. 2) Also, the other thing that you try is simulate a failure on your Xen Orchestrator Appliance (XOA). The scenario is that your single instance of XOA is down and you need to create a new VM. Try that. 3) Create containers or deploy Turnkey Linux LXC containers via the Turnkey Linux LXC "application" templates (e.g. Turnkey NextCloud). Try that as well.
For small project with small amount of VM is fine. I would not recommend when you have 200 and more VM's. It has a lot of headache. Easy lock you self in upgrade process and you have shutdown VM's and so on. When migrate VM you are not sure if this will migrate 100%. It randomly could happen that one of hypervisor unable catch VM. And VM will be in shutdown state if you a lucky, in worse case lost disk. And some of caveats are mentioned in documentation some, most of them not. If you have idea put a lot of hypervisors to one pool forget, you will not able then make changes on VM fix for example dynamic memory settings that was dropped in 8.1 version. Even backups not working upgrade time. So solution maybe could be have pool with 2 or 3 hypervisors not more. Or just choose more production ready solution. 🤦♂
@@DarkNightSonata For prod environment I would not use XCP-NG. I can't say what would be other, with other free hypervisors could be the same with other things. Now I thinking to try PROXMOX but here could other gotchas.
Maybe...if it supports ZFS, which seems it does, but does it have the coding, tools, gui to manage zfs?? But, I do believe KVM will be the one that will win/remain in the next 5-7 years...
Just downloaded the 8.3 beta the other day and checked it out. I like the updated UI for sure but like 90% of the features are “still in development”. I’ll do a video about it when it’s ready for sure.
I guess you never used VMware vcenter or ovirt/rhev or even ovm (Oracle bullshit but also based on xen). Proxmox was example of having this kind of architecture
@@RaidOwl ok I see what you where saying I did that but I had a two set up but xen ended up feeling a bit dated from a hyper-V visor standpoint loved the web interface and alerting but the Hypervisor its self felt lacking xen orch was a great tool
I went from a pre-XVP-NG Xen server to Proxmox. Tried XCP-NG and ended up going back to Proxmox. I was put off at the time by the constant pushing of the paid version of Xen Orchestra. I understand there is now a "lite" version of XO but i have not tried it but yet.
