Network design. I started my homelab / selfhost journey about a year ago. Network design was the topic that scared me most. To challenge myself, and to learn about it, I bought myself a decent firewall box with 4 x 2.5G NICs. I installed OPNsense on it, following various guides. I setup my 3 LAN ports as a network bridge to connect my PC, NAS and server. I set the filtering to be applied between these different NICs, as to learn more about the behavior of the different services. If I want to access anything on my server from my PC, there needs to be a rule allowing it. All other trafic is blocked. This setup works great so far an I’m really happy with it.

Here is where I ran into problems. I installed Proxmox on my server and am in the process of migrating all my services from my NAS over there. I thought that all trafic from a VM in Proxmox would go this route: first VM --> OPNsense --> other VM. Then, I could apply the appropriate firewall rules. This however, doesnt seem to be the case. From what I’ve learned, VMs in Proxmox can communicate freely with each other by default. I don’t want this.

From my research, I found different ideas and opposing solutions. This is where I could use some guidance.

  1. Use VLANs to segregate the VMs from each other. Each VLAN gets a different subnet.
  2. Use the Proxmox firewall to prevent communication between VMs. I’d rather avoid this, so I don’t have to apply firewall rules twice. I could also install another OPNsense VM and use that, but same thing.
  3. Give up on filtering traffic between my PC, NAS and server. I trust all those devices, so it wouldn’t be the end of the world. I just wanted the most secure setup I could do with my current knowledge.

Is there any way to just force the VM traffic through my OPNsense firewall? I thought this would be easy, but couldn’t find anything or just very confusing ideas.

I also have a second question. I followed TechnoTim to setup Treafik and use my local DNS and wildcard certificates. Now, I can reach my services using service.local.example.com, which I think is neat. However, in order to do this, it was suggested to use one docker network called proxy. Each service would be assigned this network and Traefik uses lables to setup the routes. ’ Would’t this allow all those services to communciate freely? Normally, each container has it’s own network and docker uses iptables to isolate them from each other. Is this still the way to go? I’m a bit overwhelmed by all those options.

Is my setup overkill? I’d love to hear what you guys think! Thank you so much!

  • @DeltaTangoLima@reddrefuge.comEnglish
    4•10 months ago

    VLANs are absolutely the key here. I run 4 SSIDs, each with its own VLAN. You haven’t mentioned what switch hardware you’re using, but I’m assuming it’s VLAN-capable.

    The (high-level) way I’d approach this would be to first assign a VLAN for each purpose. In your case, sounds like three VLANs for the different WLAN classes (people; IoT; guest) and at least another for infrastructure (maybe two - I have my Proxmox VMs in their own VLAN, separate to physical infra).

    VLANS

    Sounds like 5 VLANs. For the purposes of this, I’ll assign them thusly:

    1. vlan10: people, 192.168.10.0/24
    2. vlan20: physical infrastructure, 192.168.20.0/24
    3. vlan30: Proxmox/virtual infra, 192.168.30.0/24
    4. vlan40: IoT, 192.168.40.0/24
    5. vlan50: guest, 192.168.50.0/24

    That’ll give you 254 usable IP addresses in each VLAN. I’m assuming that’ll be enough. ;)

    SWITCH

    On your switch, define a couple of trunk ports tagging appropriate VLANs for their purpose:

    1. One for your Nighthawk, tagging VLANs 10, 20, 40 and 50 (don’t need 30 - Proxmox/VMs don’t use wireless)
    2. One for your Proxmox LAN interface, tagging all VLANs (you ultimately want to route all traffic through OPNsense)

    If you had additional wired access points for your wireless network, you’d create additional trunk ports for those per item 1. If you have additional Proxmox servers in your cluster, ditto for item 2 above.

    WIRELESS

    I’m not that familiar with OpenWRT, but I assume you can create some sort of rules that lands clients into VLANs of your choice, and tags the traffic that way. That how it is on my Aruba APs.

    For example, anything connecting to the IoT SSID would be tagged with vlan40. Guest with vlan50, and so on.

    PROXMOX

    1. Create a Linux Bridge interface for the LAN interface, bridging the physical interface connected to SWITCH item 2, above
    2. Create Linux VLAN interfaces on the bridge interface, for each VLAN (per my screenshot example)

    You haven’t mentioned internet/WAN but, if you’re going to use OPNsense as your primary firewall/router in/out of your home network, you’d also create a Linux Bridge interface to the physical interface connecting your internet

    OPNSENSE

    This is the headfuck stage (at least, it was for me at first). Simply put, you need to attach the Proxmox interfaces to your OPNsense VM, and create VLAN interfaces inside OPNsense, for each VLAN.

    I’m not going to attempt to explain it in reduced, comment form - no way I could do it justice. This guide helped me immensely in getting mine working.

    If you have any issues after attempting this, just sing out mate, and I’ll try and help out. Only ask is that we try and deal with it in comment form here where practical, for when Googlers in the future land here in the Fediverse.

    • @filister@lemmy.worldEnglish
      2•10 months ago

      Thanks a lot for the detailed reply, and the time you took to answer an Internet stranger. Much appreciated.