Help/advice needed - OpenWrt as proxmox VM on multi nic router hardware

icos66 · August 20, 2024, 4:09pm

Hi,

to prepare for my change from DSL to fibre I decided to migrate from my existing router to more powerful hardware (mini PC with N100 cpu and 6x2.5GbE). I want to run proxmox on it and use OpenWrt (maybe switch to OPNsense later) in a VM. Additional VMs shall host other services later. Since I am new to OpenWrt but run a proxmox based server for some time I am unsure, if my setup is correct/optimal.

So far I have:

each phyical port (eth0 -5) is associated with its own virtual bridge within proxmox (vmbr0 to 5) to allow additional VMs to attach to each port as needed.
the Openwrt VM has six NICs (virtIO), each associated with one virtual bridge that corresponds to six devices (eth0 - 5) within OpenWrt.
a LAN bridge in OpenWrt spans eth3-5 and is assigned to interface 'lan'
interface 'wan' is assigned to eth0 (but unused, since the fibre is not yet functional)
the remaining devices eth1 and eth2 shall correspond to two additional interfaces/zones (dmz and dmz2) that will have additional proxmox VMs and/or external devices connected.
WLAN will be handled by another AP in the lan network later (former router probably)

Questions/problems:

Is OpenWrt the right way to go for this kind of setup?
I would be glad to stay away from VLAN configurations to reduce complexity (I saw some tutorials for this). Since I have enough ports this should work, shouldn't it?
I am a bit concerned about the nested interfaces virtual bridges - will I run into performance issues?
How to setup DHCP server in OpenWrt to serve IPs on three different subnets (lan, dmz and dmz2)? Or is there a smarter way (e.g. assigning dedicated IP ranges to those subnets from a /24 net)?
How do I setup routing between those subnets incl. firewalling? So far, I could not get traffic from lan to dmz or dmz2.

Thanks for any help and thoughts on this setup!
Cord

/etc/config/network:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd6b:a293:25f2::/48'

config interface 'lan'
        option device 'br-lan'
        option proto 'dhcp'

config interface 'wan'
        option device 'eth0'
        option proto 'pppoe'
        option ipv6 'auto'

config interface 'wan6'
        option device 'eth0'
        option proto 'pppoe'
        option ipv6 'auto'

config device
        option name 'eth5'

config device
        option type 'bridge'
        option name 'br-lan'
        list ports 'eth3'
        list ports 'eth4'
        list ports 'eth5'

config device
        option name 'eth4'

config device
        option name 'eth1'

config device
        option name 'eth3'

config device
        option name 'eth0'

config device
        option name 'eth2'

config interface 'dmz2'
        option proto 'static'
        option device 'eth2'

config interface 'dmz'
        option proto 'static'
        option device 'eth1'

/etc/config/firewall:

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'dmz'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'dmz'

config forwarding
        option src 'dmz'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'dmz'

config zone
        option name 'dmz2'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'dmz2'

config forwarding
        option src 'dmz2'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'dmz2'

brada4 · August 20, 2024, 4:12pm

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/firewall

brada4 · August 20, 2024, 4:14pm

default eth0 was lan and eth1 wan. You need to match MAC addresses inside and outside VM "just in case"

icos66 · August 20, 2024, 4:56pm

Thanks for your fast reply. This is the ubus cmd output:

{
        "kernel": "5.15.162",
        "hostname": "OpenWrt",
        "system": "Intel(R) N100",
        "model": "QEMU Standard PC (i440FX + PIIX, 1996)",
        "board_name": "qemu-standard-pc-i440fx-piix-1996",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "x86/64",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}

network and firewall is in my post at the end.

brada4 · August 20, 2024, 5:15pm

You have to add lan->dmz forward rules like you did dmz->wan

icos66 · August 21, 2024, 8:00am

Thanks, but that's set.

icos66 · August 21, 2024, 8:17am

The MAC addresses match inside OpenWrt match the ones from the virtual devices in proxmox for the VM. The physical ports match the MAC for the virtual bridges in proxmox. Looks ok to me but I am not really sure...

What puzzles me is the MAC of br-lan that matches eth1 (aka dmz). That seems wrong. But I am unsure which MAC of the three eth devices should appear for br-lan (if any at all)?

EDIT: Swapping the default eth0/1 for lan/wan looks fine to me: wan - eth0 / lan - eth3-5 (via br-lan). Or do you spot anything?

Cord

brada4 · August 21, 2024, 8:20am

You need ip addresses on those interfaces?

icos66 · August 21, 2024, 8:29am

Sorry, don't understand your question.
I need dhcp serving dmz and dmz2. Actually, I added this in the config this morning.

my /etc/config/dhcp:

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option cachesize '1000'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'hybrid'
        option ra 'hybrid'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        option ignore '1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'dmz'
        option interface 'dmz'
        option start '100'
        option limit '5'
        option leasetime '1h'

config dhcp 'dmz2'
        option interface 'dmz2'
        option start '200'
        option limit '5'
        option leasetime '2h'

I reduced the lease time for testing, but my test device on dmz is still not reachable.

I also checked that proxmox has no firewall on the virtual device for dmz.

icos66 · August 21, 2024, 10:02am

Some additonal information: the MAC from br-lan changed after a reboot. Seems to be the first of the devices in the set (I had included eth1 yesterday shortly for tests, but did not reboot).

I can now see a dhcp lease in /tmp new for my test device (192.168.202.2 on dmz2 network) - good. It's also possible to ping it from OpenWrt.

Since I need to experiment in my current LAN infrastructure the new router PC is a dhcp client from my existing router (192.168.222.1/24). That's why dhcp is off on br-lan from OpenWrt.

brada4 · August 21, 2024, 11:31am

Your config/network has no ip-s thus no routing

icos66 · August 21, 2024, 1:18pm

OK, I got routing to work with static addresses on the dmz interfaces and dhcp serving the networks. With additional routes on my old router I am now able to reach devices in dmz zone from lan zone.

This is working on IP4s which is fine for the moment.

Do you have any tipp on how to configure DNS to enable proper resolving?
Or any improvements for the general setup?

Thanks for your help!

brada4 · August 21, 2024, 3:12pm

Just add static dns entries for fixed hosts?

icos66 · August 21, 2024, 3:35pm

sure, will do for fixed hosts. But I also want to experiment with additional VMs in dmzs and then static entries aren't my first choice.

brada4 · August 21, 2024, 4:58pm

Client-provided name from dhcp is set in dnsmasq-s dns