OPNSense High Availability - 1 VM, 1 IP! 

Thanks. All the VMs and physical machines should still have the same MACs though. Suspect it could be ARP related as the ports for LAN Trunk and WAN do change. I'll do some more digging. Either way, I can deal with a few seconds of outage for the bonus of this setup.

Awesome, thanks for sharing!

My pleasure! Glad it was useful 😃

@@substandard649 thanks, yeah I'm really glad they've finally created HA! If I had a udm I would probably go the same route, quite expensive though for me to buy 2 off the bat

@@Jims-Garage you can't put a price on a good nights sleep Jim 😀 And that hair transplant you'll inevitably need will cost way more!

Haha! It hasn't been simple, lots of work has gone into this behind the scenes.

Stop scaring me.. Mine is on backorder.

@@emanuelpersson3168 ha, don't worry. You don't need to go mad like I have 😂

@@Jims-Garage The end game for me is to go down that route just like you. But i don't think i will ever be able to... My dream is to learn Kubernetes and to get a "Proxmox HA CEPH" cluster and in that a "K3s HA Cluster".

@@emanuelpersson3168 awesome, well hopefully I've done enough to document my trials and tribulations and help you along the way!

@@PlatyBZH that's reassuring to hear, thanks for commenting

Nice, that should work well.

Thank you. Just need to keep plugging away

@@Jims-Garage Was it an automated strike for something like "fag packet maths"? I don't think the septics' content filters speak proper English...

@@organon69 thanks for that, it's a good suggestion and something that I considered. Ultimately I wanted to try what I believe to be the easiest option first,. especially given my cluster is identical. Fortunately this seems to work well albeit it's not perfect.

@@Jims-Garage Totally get it. Get "The Now" working, noodle on "The Next". One thing to watch if you consider an OPNsense-driven HA setup is how your ISP device allows DMZ/IP Passthrough to the firewall. Generally they allow a single IP (which would ostensibly be the CARP-based VIP) but sometimes don't like MAC-change shenanigans for the same IP. That is, CARP VIPs aren't discrete MACs - the VIP is an additional IP on the same int/MAC - so in an HA failover scenario the ARP behaviour on the ISP device needs to not freak out that the MAC behind that "DMZ" IP has changed all of a sudden. That dynamic alone may make you stick with the setup you walked through in the vid.

I prefer VMs for security and simplicity, although I've covered LXCs in the past and have used them. The strike was for Plex. Apparently that's against their policy (for me at least).

It's running all 3 at around 150W which is a huge improvement over my old setup. These run my workloads but I also have a TrueNAS NAS attached to the network for long term storage.

@@amosgiture not sure what that means, but I did state that it was live.

The original switch was bought around 5 years ago when I also had a UDM Pro. Cheapest option I could think of was to add the USW-Agg.

Yes, it's a thunderbolt ring network.

@@WilsonVelez hey, please check out my earlier MS-01 videos, I believe it's linked on there, cannot remember off hand. To be honest any basic switch will do for that part.

@@Jims-Garage Yeah, my apologies, after writing the comment I noticed your "Recommended Hardware" link. Again, thank you for your videos.

@russellmm My guess is that is vpro related and a workaround is likely to disable vpro in the bios.

@@johnwalshaw yes, it is related but there is no way to turn that off in the MS-01 BIOS that I am aware of. Minisforum does not have the best BIOS support.

@russellmm On my 3xLenovo P340 towers running Proxmox, in addition to the 2x10Gbps I use for primary, I use the onboard 1Gbps vpro nic. It is configured as a linux bridge. From memory, the vpro and host IP required it to be native vlan and tagged (trunked) vlans for everything else works fine. I also use this as a secondary path for CEPH. I have not tested PCIe passthrough of a vPRO NIC. I think vPRO is configured as static and not DHCP in this case. I checked my notes but not sure where I documented all this. I was very happy with the serial over IP feature and reccommend this.

​@@russellmm It is, mine came with vPro off. I tried it, it sucks, so I turné it off again. Unfortunately no time to tell you exactly, where it is, but it's there.

@@MrakCZ i'll check again, thanks

@@SharkBait_ZA I wanted to avoid double NAT and I only have a single IP.

@@Jims-Garage Sorry, I forgot about that. My setup has public IPs, so only single NAT for me. 🙂

No, there's no double NAT here.

It might be helpful for you to say why you think that this would add an extra level of NAT, as it's likely just a misunderstanding. All this does is add a switch between the incoming WAN connection and the routers, so a packet from WAN hits the switch and whichever node is currently acting as the router receives the packet. The other 2 aren't listening for it and don't respond. As far as the devices (both ISP on WAN side and on LAN side) using the router are concerned, this is exactly the same as having just one machine permanently acting as the router.

@@oli1505 no, this video is about OPNSense. Sophos is still good though

@Jims-Garage so u're using both? That would be an interesting video of how that's working.. I'd also appreciate another sophos video. 🤟 There is not much out there. It's hard to get things done without any practice. So general best practice videos how things should be designed/work together would also be nice 😁

@@oli1505 no, I moved completely to OPNSense. Long story but it was to do with my new internet (I explained it in a video). Long story short, I could go back to Sophos now but I'm enjoying OPNSense at the moment.

@@Jims-Garage ohh I guess I missed that one.. I'm gonna watch it 👍

It is, but even with the retries it was able to hit 2.5GB/s. My understanding is that the performance I see is typical of Ceph as it's not designed with raw performance in mind.

@@Jims-Garage i think an opnsense update is more like 4k iops than sequential writes what 2.5GB/s seems to be

@sku2007 i run ceph and guest vlans on shared 2x10Gbps LACP LAG for each host and not aware of any issues. I would think tb links would outperform, but maybe it's a driver issue?