Virtualizing OPNsense on Proxmox as Your Primary Router 

Thanks! I have more real world examples coming up soon! In fact, most of my guides are based on real world examples (I like to base them on real examples that I have done for my own home network either currently or in the past and sometimes I create examples in a lab environment to try new things and to verify the process works properly).

My thoughts exactly 👍

This is great, easy to follow. I'm a complete noob and got the parameters for opnsense set up on my proxmox. Step by step i go slow, but things are looking good! 🎉🎉

Thanks!

I’m glad that was helpful to show all those steps! I’m going to doing the same when showing how to set up a basic 3 node Proxmox cluster soon.

Thanks for the info! Someone already mentioned several of those points and I also made an addendum follow up video pointing out a few settings you could do differently. I’m now running a Proxmox cluster and I like using the live migration feature for my OPNsense VM so I can reboot the Proxmox system for updates without taking the network down (I demonstrated this in my cluster video). It works amazingly well. I was surprised not doesn’t even drop the existing connections. It only adds a slight delay when pinging when the cutover occurs. I don’t think live migrations has to be a rare use case for home labs, haha. Because I have multi-homed my NAS and use dedicated 10G backend networks, I don’t have much of a need to transfer large amounts of data across VLANs so leaving everything as bridges is fine for my use case even though there is a performance hit. It’s not bottlenecking my network (most of my devices are 1G so even bridges can handle that no problem). My internet is about 1.2Gbps down and about 25Mbps up and I can still use Zenarmor with a bridge with no hit to throughput. I’ve measured I can get up to 2.3Gbps on a bridge with my hardware with Zenarmor so I’d only have issues above 2.5G interfaces (which isn’t a problem as I’ve mentioned since my high throughput devices are on the same networks or connected to a dedicated, isolated storage network). With that said it’s good for others to be aware of all of those things you mentioned! It’s good to know the caveats and/or optimal settings depending on the use cases.

Thanks! Glad you liked it! I think it’s helpful to explain the options instead of just picking them. I tend to go in more detail in written guides on my website. I have to be a little more terse in videos to try to stay on topic and keep the length shorter.

Sweet! I love it when it's perfect timing for my subscribers (and others). Someone else said it was also perfect timing earlier today.

Thanks! I hope I covered enough to help people along. It’s a lot of info to cover (and there could be even more but I tried to keep the length somewhat reasonable). Takes a lot of time to produce content in general, let alone during your limited free time. Haha.

That's great! I'm glad the timing worked out. Sometimes I'm just in time for some users and too late for others. haha. I thought I would mention PCI passthrough in the video even though I didn't do it in the video to keep things a bit simple but I also tried to ensure that the instructions should still work if you plan to use a Proxmox cluster. Things get more complicated when doing PCI passthrough with a cluster. I have yet to try all that out as well. Bridges are safer and you will only notice performance issues with 10G interfaces or faster. You can still get 5-6Gbps with the VP6650 I used in the video so it's still faster than the 2.5G interfaces (and really you should try to not route 10G NAS and other traffic when possible to reduce the load on the firewall by having a separate 10G network).

Maybe use the managed switch and create a WAN subnet using a VLAN 🤔 connect the WAN cable to the switch and then any Proxmox node can access the Internet VLAN for a virtual bridge ?!? Just a thought. Might be more complex using the newest SDN feature on Proxmox . . . Guess it's time to experiment around a bit . . . Great work 👍🏻👍🏻

Thanks! I hope it has enough info to get started because there is a lot of information to cover. I tried to keep it focused on the topic at hand.

Thanks! I try to explain how and why you need to do certain things without getting too deep into the weeds. I like to think it’s like teaching you how to operate all the controls in a vehicle rather than how everything works under the hood. Of course the more you know under the hood, the more things you can do.

Great to hear! Glad it gave you reassurance! Make sure you have a good backup plan if you only have one Proxmox server (but even if you have a bare metal installation, it’s good to have a backup plan). If all is configured properly the virtualized instance should function essentially the same as bare metal as you have discovered!

No problem! Glad it was helpful!

Thanks! I'm glad you appreciate it! I hope to dig more into Proxmox clustering with OPNsense and how I think I'm going to go about it on my home network so that I can do live migrations (it will be very awesome to have the ability to move my main router/firewall over to a different physical machine with only a split second blip in downtime for my network!). I don't care about high availability/failover as much as being able to live migrate the VMs (because with VMs it's easy to restore from a backup from my PBS system, which is another nice piece of software). The configuration and requirements for live migrations is less intense which I think will suite my needs perfectly.

You’re welcome! I plan to into configuring a Proxmox cluster soon and show how you can live migrate OPNsense to different nodes with very minimal downtime.

Nice! If you set up a cluster with 2 nodes, you need to make sure you have a 3rd device as a “Q” device (a 3rd voting member) so you can have quorum. You need an odd number of devices so you can reliably know which nodes are available.

@@homenetworkguy Good to know!

Following my instructions or you already have an OPNsense VM in your datacenter? Either way, that’s awesome!

@@homenetworkguy In production for +-1,5 years. I can also acces ipmi with a vpn that is not running on the server ;-).

Yeah you can if you don’t plan to use the untagged parent interface. Since I use a different interface for the LAN for untagged traffic, I don’t need a second untagged interface and just only need to use VLANs on that second interface for just tagged traffic.

I will note one potential gotcha that I encountered when testing out some things. If you want the VLAN interfaces to use a MTU that is higher than the default 1500 used by all interfaces (to enable jumbo frames with MTU of 9000, for instance), you will need to have the parent interface assigned and enabled so that you can set the MTU value on the parent interface. This is likely a rare scenario since typically jumbo frames are used on isolated networks with higher speed interfaces (10Gbps+) rather that for routing traffic across 2 networks with larger frame/packet sizes.

Thanks! I intended on creating a diagram in this video (and some others as well), but I had a lot on my plate and wanted to get it out there. I’d like to spend more time doing polishing the videos further, but it would take me 2-3 months instead of 2-3 weeks per video. Haha. (I only do this in my ‘spare’ time). If I can get caught up on some things I’ll try to do better about including more diagrams in the future even if they are not super fancy.

Yep you still need to have virtualization features enabled in the BIOS but if you don’t also enable IOMMU on Proxmox, only raw device pasthrough is available.

Thanks for those tips! I should’ve looked up Discard to better understand if it was necessary or not. Funny thing is that the pfSense documentation shows to do it that way for disabling Secure Boot (docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html#booting-uefi). I figured their docs would also work fine for OPNsense for recommendations for VMs. Since I always use OPNsense in a VM for demo/testing purposes I didn’t care about optimal settings as much but if I use it as my main router/firewall, it becomes more important! Hardware offloading is disabled by default in OPNsense which is why I never typically mention doing it. I think for pfSense it may be enabled by default.

​@@homenetworkguy Yeah that's funny but one can't know everything. The PVE docs (I apparently can't link things without the comment being deleted) say this > pre-enroll-keys specifies if the efidisk should come pre-loaded with distribution-specific and Microsoft Standard Secure Boot keys. It also enables Secure Boot by default (though it can still be disabled in the OVMF menu within the VM). To elaborate on the discard as far as I understand it. On most linux OSs there's a weekly "fstrim" timer which calls "fstrim" which gives unused chunks back to the underlying storage. Assuming the virtual disk is on thin-allocated storage and "Discard" is enabled, of course. I believe windows also needs the "SSD emulation" option. I'm not sure how pfSense/OPNsense/FreeBSD handle trimming. I'm very far from an expert with BSD. Trim seems to be disabled in my OPNsense VM according to "tunefs -p" but I'd recommend to enable "Discard" for every disk on thin-allocated storage.

Yes, I appreciate when others let me know about details such as this so I can continue to learn as well. To clarify from what I looked into this morning-- without discard enabled, the initial VM storage doesn't take up the full 64 GB when I looked at the disk usage. It's sitting at 3GB and I have a few CTs set up as well. However, I'm assuming discard will help free up space on the host when data is deleted from within the VM. It's good to know that it doesn't fully allocate the 64 GB even if discard is left disabled. I'm not sure how trim is handled in OPNsense either.. I think I've seen others talk about it at some point but not sure if it is something that needs to be enabled to make it function properly.

Thanks! I’m planning to expand upon this and show clustering in Proxmox. I will demonstrate how to manually live migrate the VM to another Proxmox node as well.

Great that you found it helpful in creating your first homelab!

You’re welcome! Have fun!

What kind of pain? In planning to mostly keep OPNsense on one on the nodes so I can live migrate it. I’m not going to do any of the high availability features nor mess with shared storage or Ceph to keep it simple as possible. I just want to be able to move VMs between nodes if I take a node down for maintenance or if it fails. I’m not concerned with automation failover scenarios which is another reason (among other reasons) I haven’t implemented high availability with OPNsense itself.

@@homenetworkguy If anything at all goes wrong with your host infrastructure, either physically or you make with a mistake your config then you lose your connectivity. In an enterprise environment which is strictly change controlled then I am happy with virtual firewalls, but in a home environment unless you have similar controls, built and proofed in an dev environment and then rolled out to production, invariably you will make a mistake, mess up a VLAN assignment, trunk, host or the OPsense VM and then you are dead in the water as you will have no connectivity across your VLANs and no internet connectivity. That was my experience and attempting to get my environment back up and running at 4am in the morning and was not fun. It looks like those who have this working as a solid solution have a much better at home based change control than me. Love your video's BTW and thank you for this video in particular.

Yeah, I understand the need for tight control for configuration management in the enterprise, but home networks typically aren't nearly as complex so it should be easier to manage. I don't make major architecture changes very often but I plan for some down time when I do. Also Proxmox clusters can be relatively simple and not be configured with all of the high availability features. At the bare minimum, you can simply group systems together so you can manage them all from a single UI and you can migrate VMs between them. That's mostly what I would be interested in because it's quicker than backing up VM, shutting it down, and restoring the VM on a different independent Proxmox node (if not using clustering). There is a less than 1 second cutover from what I have seen from others which is pretty sweet. Since you mentioned DRS, you might be more familiar with the VMware world which perhaps may be more complex to configure/manage clusters (I don't have personal experience in that area). I'm going to give a Proxmox cluster a shot soon, but I could always keep an extra box with a bare metal installation to swap out if need be. Wouldn't hurt to have a hardware backup!

DETAILS, details! “It’s hard, complicated and error prone!” (Only for “some”). I did run my main pfsense, plus 2 more for HA, under ESXi, for a few years and there was NO SUCH PAIN! The main reason that I run pfsense on a dedicated machine, is because I found cheap used quad core mini PCs that work perfect. The “people” that utter vague claims like this, usually don’t know the stuff well!

I mentioned that you need a backup plan if you only run a single node since it will take your network down which I mentioned near the beginning about how I prefer bare metal because I’m considering using a Proxmox cluster so I will feel more comfortable about virtualizing OPNsense for my primary router/firewall. I plan to show my cluster configuration in the future. It will be pretty awesome to be able to live migrate my primary router/firewall with less than 1 second downtime!

@@homenetworkguy that would be nice, would love to see this. Especially how to do this when your provider allows only one device with the public ip

@@homenetworkguy I always double router... I keep the ISP provided router in front with family wifi. Then have a proxmox/opnsense router behind, so I have my own network I can freely break without affecting the family. Which is good because sometimes I break it a lot 😅 I've heard double router can cause problems but so far I've never faced a single issue caused by double router so not sure what that's about

@@CowCow-o5m I also play around with OPNsense VMs on a separate lab network for the same reasons. I try to keep the main network stable for my family and also because I work from home (and my wife does some work from home too). Having a separate lab network is nice because I can play around with stuff so I can make guides/videos and I don’t get tech support tickets if something breaks. Haha. But I will move to a virtualized OPNsense once I set up a Proxmox cluster because it will provide me with more redundancy so I will feel more comfortable virtualizing the main router. It will allow me to migrate to different hardware much more easily since I tinker with different mini-PCs and other hardware on a regular basis.

@@CowCow-o5m yep when you have a family, you are basicly on -call 18 hours a day 7 days a weeek, the more complicated, you have as part of the family networking, the more likely it wil break, and of course it always breaks when you are the busiest. If do this i'm going to use opensense on a virtual network that doesn't leave the system then i can have more bandwith between VMs and I can play around with rate limiting, and other firewall features.

Yeah… it’s a mouthful of IT jargon.

Good point I hadn’t considered yet. I haven’t created my cluster yet but plan too soon. I can easily back everything up to my PBS system and restore it back on the cluster.

I looked into this further. The primary mode where you create the cluster can have VMs/CTs running but any new nodes that you are adding to the cluster must be empty to avoid naming conflicts between nodes. Makes sense. I was worried I would have to start over with a clean slate to create a cluster. I have backups on PBS so it’s easy enough to start over if need be.

You're welcome! Appreciate it!

Yes but it’s almost has value to do so especially if it’s running on the same Proxmox server. VMs are very easy to backup and restore and you can take advantage of deduplicated snapshots with Proxmox Backup Server as well to get you back up and running quickly if something goes wrong. Could make a HA video because it’s interesting to learn even though I wouldn’t personally use it especially since I only have 1 public IPv4 address.

@@homenetworkguy I did some research and it should be possible and it's more ways to do it. And 1 public IP is enough, HA is good for HW failure too. It's my future plan, after I end with this vlans etc. sht I would like to learn and understand properly.

I like dedicating one interface to manage my Proxmox server so I can plug directly into it, if need be. I’m using the 10G interface for all of the VLANs on my network. You don’t have to do it this way. You can use a single interface for everything. I have the network interfaces to spare so it’s easy enough to use it that way. This also allows you to separate tagged and untagged network traffic as well which is recommended by OPNsense because there is the possibility of allowing traffic intended for the parent interface to the associated VLANs on the same interface (if you’re not careful with how you write the firewall rules and it may also require a network switch which have a specific flaw).

You’re welcome!

Yeah you have to weigh the pros and cons. I definitely wouldn’t use the MS-01 as a dedicated OPNsense box. Systems like these are too powerful not to use virtualization to make full use of the hardware. Not all of the services in OPNsense take full advantage of all the cores. In fact some of them may fight for the same couple of CPU cores (the Zenarmor team has noted as much to me).

Yeah, basically I had allow all rules on every network. I had to decide if I wanted the video to be 40 minutes or 1.5 hours, etc to show a full build (which I have done twice already.. I may do a 3rd in the future as I slowly work to improve overall production quality, etc).

Thanks! I’m planning to show I will do this in a cluster. With the limited research I’ve done, you would want to ensure the bridge names are the same on both nodes so the 2 machines would need to be configured similarly in that regard. Also if you’re not using shared storage, you would need to restore from a backup (and there might be a step to “manually migrate” the VM to a different node by messing with the config files since the VM wasn’t migrated while the node was still alive- not sure about that one yet until I try it out and/or do more research).

I completely replaced my old Proxmox server with this Protectli and it runs everything even better than my old server which was a Ryzen 7 1700. I only use 4 cores for the OPNsense VM since I noticed it doesn't tend to use much more than that. I have most of my hosted network services running on the box such as Plex, Nextcloud, Caddy reverse proxy, Vaultwarden, UniFi Controller, Grandstream GWN Manager, RustDesk, Uptime Kuma, Homepage dashboards, etc. I haven't even reached full capacity yet. It runs in a Proxmox cluster so I have stuff running on different nodes for various purposes (one is mostly dedicated to Home Assistant while the 3rd node hosts all of my apps/VMs I use on my LAB network).

Haha, I take it that you really liked the video. I have one that goes one step further with creating a basic Proxmox cluster that’s dropping soon. It’s very cool to be able to live migrate your router/firewall without the network connection noticeably dropping.

@@homenetworkguy Yes, I watched it too!!! you are awesome man, I hope you know you are doing an amazing service to the community!!! God bless you and prosper you!

Depending how you are configuring your devices, it’s sometimes best to not have everything plugged into your existing network because there could be IP address conflicts or you could end up with 2 DHCP servers running on the same network, etc. I believe I have the system I’m configuring OPNsense with plugged directly into the management interface of Proxmox but I manually set a static IP address on that system. You should be able to access the Proxmox web interface as well as the OPNsense web interface if you are using the same bridge for the OPNsense VM.

I'm not sure I've seen that happen but you can set up Proxmox to start OPNsense first and then make all of your other VMs start only after the OPNsense VM has started. You can even add a short delay to ensure OPNsense is up and running before anything else on Proxmox starts. This could potentially help with your problem, but I'm not sure why it doesn't detect the network is up and running.

@@homenetworkguy unfortunately I have tried this and it does not make a difference which one starts first. No matter whether OPNsense starts first or the VMs, services have to be reloaded. I'm going to have to watch your video and make sure I follow it step by step because if you aren't familiar with this, there is something I have to be doing wrong

Yeah you could dedicate one interface for the WAN but the other interface will have to be the Proxmox management interface, the LAN interface and if you want any additional VLANs. You could experience some bottlenecks using a single interface. The configuration will be a bit different than what I demonstrate but the concepts should be the same (you would just use the same bridge interface for all internal networks instead of separate interface(s)).

If you need maximum performance, you use can use passthrough. Otherwise bridging will be fine if it doesn’t hinder throughput (depends on the speed of the CPU on your system). If you are using a cluster, you would need to be careful when using passthrough especially if using different hardware. There is a resource mapping option for the cluster but I haven’t tested it to see how well that works when live migrating VMs (and haven’t tested full high availability either). I mainly keep my cluster simple and manually do live migrations when I need to reboot one of the nodes that has my OPNsense VM.

You’re welcome! Thanks for the support! I appreciate it!

Yeah since I’m using the default 192.168.1.1/24 for both the Proxmox interface and the LAN network of OPNsense, I didn’t have to make any adjustments later to make them be in the same network.

It depends on what you are trying to accomplish. Performance is best with passthrough but you can’t use the interfaces for anything else. With bridges you can have other VMs and CTs be on the same network by sharing the same bridged interface. It’s very flexible but there is a performance penalty. Since I’m planning to cluster it makes it easier to migrate VMs between nodes.

You’re welcome, thanks! Yeah I’m sure you could, but it’s possible to create the LAGG on either Proxmox or OPNsense so I’m not quite sure which would be the best approach (you need to it on on side or the other but not both).

@@homenetworkguy Thank you for the response. I tried it from OPNsense, it doesn't work for me. Let me try it from Proxmox, and let you know.

Yeah it’s possible you would need to passthrough the physical interface for it to work in OPNsense so it can have direct access to the network interfaces. But if you do it in Proxmox, I’m thinking you could use the LAGG interface of Proxmox in the OPNsense VM and treat it like a normal single physical interface in OPNsense. I haven’t tried that out so I’m not sure how all that would work. Hah

@@homenetworkguy Yes, I think it will work if I pass through the NIC to OPNsense VM. But I am using a Realtek card, so I am trying it through Proxmox. And also, I am setting up VLAN and want to use this VLAN in other VMs and CTs. Let me try the LAGG in Proxmox and see how it goes.

You’re using the same bridge in Proxmox as the LAN interface in OPNsense? As long as you don’t pass through that interface which is used for Proxmox management and you have the LAN configured properly in the OPNsense VM, it should have access to the Internet just like any other device on the LAN network.

Realized my error; reinstalled proxmox / opnsense and all is well now. I guess this all is the learning process.

@@cyrilpinto418 Nice! Glad you got it resolved. Sometimes missing a minor detail can cause a problem.

I have not encountered that issue after updating or rebooting Proxmox or the OPNsense VM. It’s hard to say what happened with knowing more details because if everything is on the same network using the same bridge for the LAN/Proxmox management interface, you should be golden.

Haha, thanks! Took a bit of effort some to get it made but my favorite videos are real world examples pulling multiple concepts together.

@@homenetworkguy man this was perfect and honestly I appreciate the content it’s helpful for poeple who want to try this and the examples and explanations is perfect for beginners . Will be showing my friend as well who’s trying this to

Cool! I hope it goes well if you do!

I have 4 Protectli boxes (since I have some sponsored hardware) and nothing has died yet with 24/7 operation. The oldest Protectli is about 3 years old. I run the hardware in my server closet which runs a few degrees hotter than room temperature so the operating environment isn’t very hot. Something to consider when running fanless mini-PCs because the hardware might not last as long if it’s in a hotter room (do not run it in an attic for example in a hot summer, for example). I will say that I always have my systems connected to a UPS and rarely have any hardware die unless is getting very old (which is to be expected). Most of the time my hardware in general becomes essentially ‘obsolete’ before I replace it.

You’re welcome! As for your question, I’m not sure of how many interfaces you have in the colocated server. Of it’s only one, then you cannot use passthrough because that means only the OPNsense VM can use that interface. You will have to use the default bridge interface in Proxmox. With only one interface it’s going to be tricker to set up a WAN/LAN interface but it’s possible using VLANs. If you follow the basic principles in the video, you will be able to use the default bridge for both the Proxmox management and the OPNsense LAN interface. You simply just assign the same bridge to other VMs so they can be on the same network.

You’re welcome!

I just used another PC from the one I did the recording (I have a couple of mini-PCs I use for demo purposes). If you only have 1 PC/laptop and 1 Proxmox server, you’ll have to temporarily connect your PC/laptop to the Proxmox server to configure it. Once you’re done and have OPNsense installed, you can connect it back to your network. I’m assuming in the video you’re using the default LAN network for both the Proxmox management and the OPNsense LAN interface. That network interface is 192.168.1.1/24 (which means usable IP addresses between 192.168.1.2-192.168.254).

​@homenetworkguy: Are you giving "pve-test" the new address of 192.168.1.50? My current network uses a 10.27.27.x scheme, but I don't know how to locate my new proxmox node when I remove it from the network. 😕

You can assign the Proxmox static IP to be whatever you want so you can make it 10.27.27.100/24 if you like (make sure it’s outside your DHCP range to avoid potential IP address conflicts). If you plan to put OPNsense and make it your primary virtualized router and you still want to use that network address, you’ll have to change the default LAN IP addresses or create another interface with the appropriate IP address ranges. I tend to keep the default LAN network of 192.168.1.1/24 since it’s keeps things simple (but I add other VLANs too, of course, with different IP ranges).

Just figured out that it's the ip address for the "stand alone" pc used to configure the proxmox device that needs the new static ip in the proxmox device's network scheme. 🤯 Thanks for all the info (and patience)!

Ohh yeah. You can use any PC you want to administrate your devices. I was just using a ZimaBoard because it's much smaller than setting a full tower PC on top of my desk to show all of the connections, haha.

@@homenetworkguy Ah, thanks - I was quite confused and honestly didn't know if this board also did something else required. But yes, I tried it with my PC and it seems to be working (can't tell until I did a proper configuration).

Haha yeah I just wanted a small PC to use to set everything up and to show where a PC could be connected on the network. Glad you got your network set up!

Thanks!

I would have to look into this more. One thing that I find annoying to deal with in Docker is its networking. Deployments are easy but then you have to mess with the networking aspect. Simple things aren’t too bad but what what if you want containers to be on different VLANs as you mentioned? I’ve always just put them on the same network in the past but that was before I started segmenting my network. I avoid Docker since I use LXCs (don’t need an extra container layer) so I haven’t tried setting up apps on different VLANs. Also a VM might be more desirable than an LXC for Docker (at least when I tried a while ago, restoring backups of LXCs which use Docker was problematic for me).

@@homenetworkguy I find setting up a Docker container much easier than using LXC, but maybe that's because I know more about it. I think having one LXC with Docker and multiple Docker containers is less overhead than having multiple LXCs. I'd love to read about your findings!

Setting up containers on Docker is easy but the networking aspect is something you have to work through. I’m not sure without researching it further on how to run containers of different VLANs (I don’t know if MACVlan or IPVlan modes are what they sound like they should be used for.. almost seems like it’s for containers to internally communicate on different virtual networks). The nice thing about LXCs (without Docker) is that I can allocate exactly the right amount of host resources that I want as well as install whatever I want inside the LXC (without needing to create custom Docker images for example). It’s very simple to put LXCs on different networks. I also like to utilize the ufw firewall in the LXCs so I can easily block all unused ports on each LXC (I know Docker only exposes certain ports for containers but it also interferes with the firewall on the host machine so you have to do workarounds to be able to use ufw firewall or iptables without Docker interference). I typically setup SSH access for all of the LXCs so I can get into them if I need to do anything. I think you can do that with Docker as well but not sure if it’s as straightforward depending on the networking mode used (I believe I recall logging in more easily to a terminal window using Portainer long ago when I was using it). I also like being able to back up individual services that are in LXCs rather than the whole Docker instance because I only have a few critical LXCs that I backup offsite. With individual LXCs, I can move them around to different Proxmox nodes easily. Ultimately, it’s a matter of preference. There are pros and cons to either approach but I’ve come to like using LXCs better. A lot of people like Docker and I understand its appeal especially in deploying web apps that have a lot of dependencies including setting up databases, web servers, etc.

Thanks! I've had a few requests for Kea DHCP. I'll get around to it eventually since it will be the new way forward but currently I do not believe it is considered feature complete so I do not see myself personally migrating any time soon (but I will likely do a video on it before I make the transition on my own home network).

​@@homenetworkguyI switched right after update with Kea support and it's not that hard to set up. And it's working without problem. Only con I see is no hostnames, only theirs IP adresses (in DNS server, monitored communications etc.).

It supports snapshots so you don’t have to pause or restart your CTs/VMs when you do backups (you can do snapshots with ext4 but you have to select LVM thin and not LVM for this to work). You can also take advantage of built in LZ4 compression which could not only save space but speed up read operations (I believe). Caching certain operations in RAM may help improve performance but I haven’t compared that directly. ZFS can still detect bitrot with a single drive because of the checksums but it wouldn’t be able to correct it without having redundancy. I don’t use deduplication with ZFS even on my TrueNAS system since it requires too much system resources.

@homenetworkguy thank you for answering. One more question if you don't mind. I'm planning to get VP6650 I was just thinking, how can I design the storage wisely give that nvme is for VMs/CTs 2.5 SSD1 - Host 2.5 SSD2 - RAID1? What if I upgrade in the future and it fails, the upgrade will also replicated to it so it will not work? Please advise what is the best storage design for it. Thank you in advance!

Yeah you could mirror the SATA drives (RAID1) for the host OS and use the NVMe for CTs/VMs. That's a good way to set it up and it's similar to how I used to have my 4U rackmount Proxmox server before I migrated it to the VP6650. There's not really a good way to recover from a failed Proxmox update but those sorts of failures are pretty rare. I had an issue long ago but it was when migrating from v6 to v7. Not sure if it was self-inflicted because I was new to using Proxmox back then. If you keep some of your configuration files under the /etc/pve folder, it will help you with a new installation because you can recreate your configuration more quickly. In theory the Proxmox host is supposed to be minimally modified so that it's easy to reinstall or move to a new system (the idea being that your CTs/VMs contain most of the configured apps/services). However, in practice, you still need to backup some of your config files to save time if something fails. I recently set up a Proxmox cluster so if I have a hardware failure, I can remove that node, and add a new one more easily because much of the configuration is at the cluster level (still a good idea to backup the network config because each node needs to have the interfaces configured appropriately).

It’s either that or choose “other”. I think it affects the options that are available for the VM configuration since some options aren’t available for certain OS’s. Not sure if it makes a difference for FreeBSD based VMs or not.

Under the Services > ISC DHCPv4 > Leases page, you will see a list of all devices and IP addresses of the clients using DHCP. You won’t be able to see any devices that are using static IP addresses but you should be able to see everything else.

you can also use nmap to scan your networks

Are you plugged directly into the Proxmox management network interface? Or connected to a network switch? You will need a static IP on your laptop only if you’re plugged directly into the Proxmox management interface. Otherwise you can use DHCP if you’re on the same network as the Proxmox management interface.

@@homenetworkguy im plugged directly into the interface. Followed your guide.

Did you configure the subnet of the static IP to be 255.255.255.0? Also make sure it’s not accidentally the same as the Proxmox IP address as well. You could try different interfaces on your Proxmox box in case you have a different one configured than the one you’re plugged into.

​@homenetworkguy could I contact you on a discord or something alike to grt a bit more help. I'm really stuck and can't seem to figure out what is going wrong

I do have a Discord account. I don’t always hop on it but you could use that. Keep in mind that it’s becoming a bit more difficult to keep up with everyone’s messages. I still have a couple week backlog left in my email (I caught up on a couple weeks worth of email last night).

It allows the guest virtual machine to have virtual CPUs process the network traffic which can help improve throughput. According to the following link, it is recommended to set the multiqueue value only when anticipating a lot of network traffic since it increases the CPU load of the host/guest as network traffic increases: forum.proxmox.com/threads/multiqueue-inside-of-vm.66321/

@@homenetworkguy thx for your fast help 🙏

You can create a rule on each interface to allow all access (protocol any, source any, destination any) for testing purposes.

@@homenetworkguy okay let me try it

No boot device before or after installation? Sounds like a boot order issue possibly?

@@homenetworkguy it appears I have been using the incorrect ISO. I would download the image from Open sense straight into my Proxmox and just realized that this was some kind of a zip file. I unzipped and now uploading manually but this method might take an entire day to complete. Stay Tuned!

Ohh yeah, you will need the DVD ISO image and have it unzipped before importing into Proxmox.

@@homenetworkguy Thank you again, Second Question if I set this up as virtualized just for learning can I keep it strictly isolated to my proxmox vms and not have it manage my main network/wifi. And what would be the best setup for that scenario?

Yeah for sure. I have a couple of OPNsense VMs I use for demos/testing, etc. The main thing you need to be careful of is not putting the WAN interface of the OPNsense VM on one of your primary networks while also having LAN interfaces on the OPNsense VM using the same IP addresses because the LAN interfaces will take priority over the WAN interface. It’s hard to explain but I’ll give an example. If you put the WAN interface of your OPNsense VM on the 182.168.1.1 network, the WAN of the OPNsense VM will be assigned something like 192.168.1.100. But if you also have a LAN interface in the OPNsense VM with 192.168.1.1/24, your WAN interface in the OPNsense VM will not be able to communicate with your primary network because the gateway address of the WAN interface will be 192.168.1.1 which happens to be the LAN interface IP address. One other gotcha is you will likely want to enable query forwarding under Unbound DNS if you are running into DNS issues. I’ve found that running a recursive DNS resolver behind my primary OPNsense box doesn’t work (probably since I am using DNS over TLS on my primary OPNsense so it can’t recursively resolve to the root DNS servers).

Sure, I think that would be possible! What you can do is create a bridge in Proxmox that is not assigned to any physical interface. Make sure you select that interface as the WAN when you install OPNsense (hint: if you make it the first network interface for the VM, it will be called vtnet0 inside the VM if you’re not doing PCIe passthrough). Then later you can update that bridge in Proxmox to use a physical interface. You can easily remap network interfaces later. That’s the beauty of virtualization. You can even do this while the VM is running (but caution is advised)!

@@homenetworkguy perfect, wasn't sure vmbr doesn't need any physical NIC assigned :O learned something new today :D

You can also assign VMs/CTs to that same bridge and everything would be on that same virtual network. This is pretty neat if you want a fully virtualized lab network within Proxmox.

@@homenetworkguy ah fully virtual SDN is something I'm yet to get into :D

@@homenetworkguy yup I can confirm works exactly as expected, for fully virtualized opnsense install: fake WAN in prox, fake LAN in prox, then make a random VM with Windows or whatever, assign real LAN, fake LAN (static IP in Windows at opnsense LAN range), that way it's easily possible to RDP into this machine and from its web interface tinker with opensense web UI, fun stuff :D

I notice it uses about 20-22W but I had a couple network interfaces plugged in and I have a second disk (SSD) which would add to the base wattage. However I think that’s a good basic use case for real world wattage. It has faster single threaded performance than my Ryzen 7 1700 Proxmox server but at 1/4th the idle power consumption. It uses about twice as much power as their 4 port models but it’s also much more powerful too. I have the VP2410 and VP2420 and the two systems combined uses nearly the same power at idle as the VP6650.

You’re welcome!

It's hard to say where the config is going wrong without seeing any of it. Perhaps you could take a look at my website which the videos are based off of for more details since there may more explanations that will help you understand it better. It does take some time to wrap your mind around firewall rules when you are new to them (at least it did for me): homenetworkguy.com/how-to/set-up-a-fully-functioning-home-network-using-opnsense/

I believe this was addressed in newer versions of Proxmox. I know many had issues with the N5105 and the N6005 but I’ve used Proxmox with the N6005 without issue several months ago.

@@homenetworkguy that’s great thank you for replying. I plan to change over to proxmox. Can I use a Ethernet adapter (2.5gb) to use for proxmox/setup and setup opnsense. So I can set up my 4 built in ports as follows: WAN, LAN 1 , LAN 2 and LAN 3

You could I suppose but keep in mind if you use bridges, you can share the same port with your Proxmox host/VMs/CTs as demonstrated in the video. You don’t necessarily have to dedicate all the ports to OPNsense (you may need to use passthrough on the N5100 to achieve 2.5Gbps but faster hardware can handle 2.5Gbps even with bridges just fine).

@@homenetworkguy I think I understand it now. I have 3 Ethernet cables from my opnsense, LAN 1= server (unraid) LAN 2= WiFi access point upstairs and LAN 3 for lounge. And last port is my WAN. So if I understand correctly I can e.g use LAN 1 to install/setup proxmox and opnsense and then have my ports work in the same way?

Yes if you use the default vmbr0 bridge that Proxmox sets up during the installation. That’s the great thing about bridges but there is a performance impact depending on your CPU and the speed of the network interface. I’ve discovered that bridging performance in Proxmox is greatly impacted by single threaded performance of the CPU.

Can you install a hypervisor on the Sonicwall? New prices seem like $3500? I’m assuming you’re referring to used hardware prices. You could also do this guide with a $200-300 mini PC which has 4 network interfaces. It depends on what you need. The VP6650 is faster (single threaded performance) than my old Ryzen 7 1700 Proxmox server at 1/4th the power consumption. I could easily replace my huge 4U server with the Protectli if I wanted but I’ll probably just cluster a few of my systems at some point.

@@homenetworkguy each to his own my guy. Great video and I'm sure it'll be very informative and helpful to a lot of people.

Thanks! It seems like the video is being well received by those interested in the topic. Also, I was genuinely curious in my previous comment if you can run a hypervisor like Proxmox on it and get the device plus a year support for $1300? I wasn’t implying the Protectli box is superior to the Sonicwall but rather it’s an apples to oranges comparison (one is a general purpose computer while the other is a firewall appliance). For a home network, having a general purpose low power mini PC is great for virtualization servers, etc.

I usually run bare metal but I know a lot of people like to virtualize for various reasons. Do you have any documented examples of what you are referring about compromising the hypervisor on a virtualized firewall? I’d be interested in reading up on it.

@@homenetworkguy The IP packet arrives at an interface on the server and is analysed by the server (OSI Layer 2 & 3 analysis) and forwarded to the VM. These steps take place on the server before the packet arrives at the VM. Only the IP tables of the server forward the packet to the VM. This means that the IPTables including the kernel are before the firewall. Draw the path for each OSI layer once on a piece of paper and write who is responsible at each point.

I understand what you are saying. I am just curious how many documented cases of compromise due to virtualizing the firewall. So many people do it that I’m surprised more people say “don’t do it!”

@@homenetworkguy Security is not a question of the frequency of events! The host server is not protected and is therefore directly connected to the "bad" Internet. Why use a firewall then?

​@@marcodoehler4089 Because the OPNsense VM uses interfaces that are connected to bridges on the physical Proxmox VE interfaces, Proxmox VE doesn't analyze anything. It will only receive Ethernet frames (layer 2 only), the bridge will look up the destination MAC address (of the OPNsense virtual interface) and simply forward it. Iptables (or soon nftables) on Proxmox VE will not be used for this at all, unless you want to block traffic to and from the OPNsense VM from the host. If you do not set an IP address on any of the bridge interfaces to which the OPNsense VM virtual interfaces are attached, there is no way to communicate with the host.

Yes! No, just kidding. I think it was distracting because sometimes with my edits I have to jump around out of chronological order. Also it sometimes takes a few days for me to get time to get all the recordings complete. But I realize that blurring it out makes it too distracting for some users.

Thanks for the feedback!

Yeah I wasn’t impressed with the UI when I first started using it but over time it has grown on me. I enjoy using the product. I’ve only had a few mishaps over the years and most of it was probably user error on my part. Haha

They have hardware you can purchase and they offer business license support as well if you need professional support. It’s goes far beyond high school lab testing. I’d much rather have a solid product with a simple logo than a terrible product with the most beautiful branding. 😄

@@homenetworkguy what product does not offer professional support??? and good luck to them with that ugly logo, hope they get enough to pay for it. NO company with a brain will use this in DEV environment talk less production. Sure it can work but logos cost like $50 and up

Thanks! I hope they help you along the way! I have been evolving my network for many years (more so in the last 6 years).

@@homenetworkguy , do you have any thoughts on IPFire ? For example, can I use it to achieve something similar to your "opnsense for beginner" video/post ?

I have thought about learning more about other firewalls (OpenWRT, IPFire, etc) once I have exhausted the main topics I want to cover in OPNsense but after writing on my website for nearly 6 years (and more recently, RU-vid videos), I still haven't exhausted everything I'd like to learn about. haha. I think IPFire could be a good Linux based alternative. There are a lot of similar features but also some things it doesn't offer via plugins. I would like to test out the performance of it because it's possible Linux could perform better than FreeBSD depending on driver support, etc.

First of all, 1 of the 5 ports will need to be connected to Proxmox and that port will have ALL VLANs assigned as a trunk port so the traffic can pass through to Proxmox. A trunk port can have as many VLANs as you want to pass through to other switches, routers, wireless access points, and servers (all of which need to be VLAN aware devices). Each non-trunk port can only be assigned to a single VLAN. In Proxmox you can create a virtual bridge (with or without VLAN tags) that you can use as a virtual network within the Proxmox server or Proxmox cluster- if you wish to have CTs/VMs on their own virtualized network (this can be helpful for lab networks, etc).

@@homenetworkguy the last sentence is what I want to do; create virtualized CTs in 3 Vlans, 1 each for Caddy, Apps, and Arr stack. I created a Vlan aware Bridge that wasn’t connected to any NIC, and all was working fine, but the setup is causing OpnSense to crash / restart. I’m now thinking of using my last remaining Nic, making a VLAN aware bridge, creating 7-8 VLANs and trunking them to my Mikrotik Hex which will only have 4 available ports, to be used to separate physical devices such as office, Iot, guest etc. The only 3 VLANs to be used for Caddy, Apps, and the Stack. Am totally lost here.

Without IDS/IPS, it shouldn’t be a problem but it can’t do IDS/IPS on OPNsense at 8 Gbps because not all of those services are fully optimized to take advantage of all the cores on the CPU. You may potentially have better luck with other operating systems. I haven’t tried other firewalls such as IPFire yet. It’s Linux based so it may perform better. I should try it before I start using the VP6650 in my future Proxmox cluster.

@@homenetworkguy would love to see it. i wish there was a chart that just said you need x for y feature somewhere, the information is always vague or refers to buying some enterprise level hardware, pretty sure my wife would not be happy to that purchase. vs something smaller one could build out ^_^

I’ve thought about creating a chart/table for the hardware I have personally tested to help others determine how much hardware they need for certain services in OPNsense. I wasn’t able to test all of the older boxes I have quite as thoroughly but it’s getting easier for me to set up test cases since I have more sponsored hardware and other hardware that I purchased available for testing.

You’re welcome!

It is your primary router but just virtualized. You can plug your modem/ONT directly into the interface used as WAN on Proxmox just like you would on a bare metal installation plugging into the WAN interface. Proxmox is not doing any of the routing or firewalling for your network-- OPNsense in the VM is doing that task. This is the nature of virtualization. Proxmox is not "in front" of the OPNsense router. Rather, Proxmox is simply hosting the router/firewall software in a virtual machine (all routed network traffic flows through that VM just like a bare metal installation). I am currently using a bare metal installation of OPNsense, but I will probably move to a virtualized installation (in a Proxmox cluster) so that I can have more flexiblity to "move" my router to different hardware without doing a separate bare metal installation. I can just migrate it over to a different machine. Since I test out various hardware, that flexibility will be great to have. As far as security is concerned, the main security risk with virtualization vs bare metal is escaping the VM sandbox. If an attacker can break out of the VM, they can get on the host system. Those sorts of attacks are very rare. Other than that, the security is generally pretty much the same. I understand virtualization is not for everyone. I have guides that show both bare metal and virtualized instances of OPNsense.

@@homenetworkguy did you notice that after installing opnsense and setting it up as the main proxmox router that pve>system>network,DNS,certificates etc have to be changed to match the new network?

Thanks! Sorry about that. Sometimes it’s easier to remote into another machine but sometimes I also use SPICE for my Proxmox VMs which doesn’t have the second mouse cursor/delay.

You should be able to dedicate one interface to the WAN and the second to the management interface of Proxmox and the LAN network of OPNsense. You can even add VLANs on the second interface as well (but you’d need to add the VLANs to the network switch as well). Technically you could it all from a single interface using VLANs but the config is a little bit more involved. It’s easier to configure separate interfaces and also reduces the potential for bottlenecks in throughput. You need to make sure that Proxmox has an IP address in the same network as the LAN on OPNsense (which defaults to 192.168.1.0/24).

@@homenetworkguy Thanks so much for the reply. I ended up figuring out my issue... my proxmox box needed to have its gateway set to the router. I had set it to opnsense, and from what I've learned that was causing asymmetric routing.

Nice! The gateway is the interface IP of each network- IP address which is used to route the data to other networks essentially so for the default LAN that is 192.168.1.1, as you likely are now aware. Glad you got it working!

You would have to use a bridge for the LAN interface similar to how I demonstrated in the video. It would be the same interface you use to manage your Proxmox server. You can’t use PCI passthrough on that LAN interface and also use it as the management interface for Proxmox because that interface will be dedicated to the OPNsense VM if using passthrough.

Love it when the content is release just in time!