Setup mwan3 with NAT + firewall

Hello,
I have started setup OpenWRT on my secondary router Mikrotik hEX S ("RouterBOARD") that is connected to the primary ISP router
Actually I have 3 routers in my network, but starting from scratch I will focus here on ISP router (Fritz!Box 6490) and OpenWRT router.

My ISP provides 2 different WAN connections.
WAN1 is "classic" routing, means dynamic public IPv4 and with NATed LAN network.
WAN2 is static IPv4 with bridge-mode on defined ethernet port.

I followed this tutorial for setting up multiple WAN interfaces, however there are some challenges.

  1. This network setup with multiple routers should not result in double NAT. Therefore I prefer a configuration with disabled NAT and additional routing tables as described here.
    The challenge is that disabled NAT can only work with WANa connected to ISP router's downstream. For WANb connected to ISP router's bridge port NAT must be enabled (in my understanding).

  2. The challenges described in 1. impacts firewall rules.
    And if I add another network segment DMZ the complexity increases.

Question:
Could you please advise for a OpenWRT configuration that reflects the network setup with ISP router and OpenWRT router?

THX

Make a second wanb firewall zone without nat.

This sound like a suitable solution for the issue:
WAN -> no NAT
WANb -> NAT

However, I'm stuggling with the network device configuration.
This OpenWRT router will have 4 network devices:
wan
wanb
lan
dmz

wan is currently configured as DHCP, connected to ISP's LAN subnet 192.168.1.0/24. If required this interface can be configured with static IP.
wanb is configured with static IP that is provided by my ISP.
lan subnet is 172.16.1.0/24, IP 172.16.1.1
dmz subnet is 172.16.9.0/24, IP 172.16.9.1

This is the routing table:

# ip r
94.xxx.xxx.224/30 dev br-wanb proto static scope link metric 10 linkdown 
172.16.1.0/24 dev br-lan proto kernel scope link src 172.16.1.1 
172.16.9.0/24 dev br-dmz proto kernel scope link src 172.16.9.1 linkdown 
192.168.1.0/24 dev br-wan proto static scope link metric 20

My understanding is that I should have 2 default gateways:

default via 94.xxx.xxx.224
default via 192.168.1.1

Question:
For any interface I could check "Use default gateway".
In which scenario is this check required?
And if it's required, for which interface(s)?

…obviously continuation of Router cascade and double NAT problem

monty, it is helpful, to stay inside a single thread, even though you may have different subquestions of the same problem. It helps other people to get the big picture, if 1 problem correlates with 1 thread.

I post you're referring to was the starting point.
You pointed to mwan3, and I implemented this in the meantime.
But now the configuration challenges continues...

You should

In the scenario that you want to use the internet from that provider.

For both obviously.

This whole bridge wan thing is recipe for problems.

Thanks for your reply.
Why do you thing the "br-wan thing" is causing problems?
Is this related to usage/configuration of mwan3?

I have identified the root cause for missing default gateways:

  1. "Use default gateway" must be checked
  2. Gateway IP must be correct

After enabling "Use default gateway" for interfaces wan and wanb and setting the correct gateway IP for interface wanb the routing table is displayed as expected:
root@clancy:~# ip r

default via 94.xxx.xxx.225 dev br-wanb proto static metric 10 linkdown 
default via 192.168.1.1 dev br-wan proto static metric 20 
94.xxx.xxx.224/30 dev br-wanb proto static scope link metric 10 linkdown 
172.16.1.0/24 dev br-lan proto kernel scope link src 172.16.1.1 
172.16.9.0/24 dev br-dmz proto kernel scope link src 172.16.9.1 linkdown 
172.16.10.0/30 dev br-homenet proto kernel scope link src 172.16.10.1 linkdown 
192.168.1.0/24 dev br-wan proto static scope link metric 20

So, this issue is solved now.

However, there are connection issues now. This means, ping and traceroute are not working:

root@clancy:~# ping -c 3 openwrt.org
ping: bad address 'openwrt.org'
root@clancy:~# traceroute openwrt.org
traceroute: bad address 'openwrt.org'

Can you please advise how to trouble-shoot this issue?

Because there is no need to have a wan interface bridged.

It is not related to mwan3.

Use either a public dns resolver, or if you use the ISP ones, make sure they use the proper uplink.

I have fixed the connection issues in the meantime.

root@clancy:~# ping -c 3 google.com
PING google.com (142.250.185.110): 56 data bytes
64 bytes from 142.250.185.110: seq=0 ttl=116 time=27.551 ms
64 bytes from 142.250.185.110: seq=1 ttl=116 time=24.810 ms
64 bytes from 142.250.185.110: seq=2 ttl=116 time=13.125 ms

--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 13.125/21.828/27.551 ms


root@clancy:~# traceroute google.com
traceroute to google.com (142.250.185.238), 30 hops max, 38 byte packets
 1  192.168.1.1 (192.168.1.1)  0.587 ms  0.359 ms  0.334 ms
 2  *  *  *
 3  ip-081-210-148-070.um21.pools.vodafone-ip.de (81.210.148.70)  21.615 ms  22.707 ms  13.769 ms
 4  de-str01c-rc1-ae-21-0.aorta.net (84.116.190.241)  15.824 ms  15.800 ms  14.826 ms
 5  de-fra01b-rc2-ae-4-0.aorta.net (84.116.140.201)  21.782 ms  24.790 ms  17.884 ms
 6  de-bfe18a-rt01-lag-1.aorta.net (84.116.190.34)  15.718 ms  15.937 ms  14.988 ms
 7  74.125.32.52 (74.125.32.52)  14.847 ms  74.125.48.122 (74.125.48.122)  17.934 ms  18.894 ms
 8  *  *  *
 9  142.250.236.30 (142.250.236.30)  45.527 ms  142.250.210.208 (142.250.210.208)  18.682 ms  172.253.64.118 (172.253.64.118)  15.760 ms
10  108.170.251.144 (108.170.251.144)  17.343 ms  108.170.252.18 (108.170.252.18)  24.767 ms  108.170.252.19 (108.170.252.19)  33.874 ms
11  209.85.242.79 (209.85.242.79)  34.848 ms  31.959 ms  108.170.228.9 (108.170.228.9)  16.037 ms
12  108.170.238.61 (108.170.238.61)  25.826 ms  209.85.252.215 (209.85.252.215)  14.886 ms  209.85.241.230 (209.85.241.230)  16.965 ms
13  108.170.252.65 (108.170.252.65)  68.157 ms  16.983 ms  24.984 ms
14  142.250.236.57 (142.250.236.57)  14.798 ms  15.977 ms  172.253.50.151 (172.253.50.151)  16.034 ms
15  fra16s53-in-f14.1e100.net (142.250.185.238)  17.814 ms  19.059 ms  23.052 ms

Currently there's only interface wan (WAN -> no NAT) connected, and any connection check from the router to WAN works as expected.
However, for any client connected to lan interface connection checks (ping, traceroute, nslookup) fails.
I assume that this issue is related to existing firewall rules, therefore I paste my firewall rules below; please note that some rules are disabled mainly because the HTTP proxy server does not exist, yet. And I didn't modify and default rule in tab "Traffic Rules".

root@clancy:~# uci export firewall
package firewall

config defaults
	option forward 'REJECT'
	option synflood_protect '1'
	option input 'REJECT'
	option output 'ACCEPT'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option family 'ipv4'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option family 'ipv4'
	option log '1'
	list network 'wan'

config zone
	option name 'wanb'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option family 'ipv4'
	option mtu_fix '1'
	option log '1'
	list network 'wanb'

config zone
	option name 'dmz'
	option input 'REJECT'
	option output 'REJECT'
	option forward 'REJECT'
	option family 'ipv4'
	list network 'dmz'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option family 'ipv4'
	option target 'ACCEPT'
	list icmp_type 'echo-request'
	list src_ip '192.168.1.1'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled '0'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled '0'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled '0'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	option enabled '0'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'LAN: Allow access to NTP server'
	option src 'lan'
	option src_port '123'
	option dest_port '123'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'LAN: Allow access to SSH server'
	list proto 'tcp'
	option src 'lan'
	option target 'ACCEPT'
	option family 'ipv4'
	option src_port '22'
	option dest_port '22'

config rule
	option name 'LAN: Allow access to DHCP server'
	list proto 'udp'
	option src 'lan'
	option target 'ACCEPT'
	option family 'ipv4'
	option src_port '67-68'
	option dest_port '67-68'

config rule
	option name 'LAN: Allow access to DNS server'
	option src_port '53'
	option dest_port '53'
	option target 'ACCEPT'
	option src 'lan'
	option family 'ipv4'
	list proto 'tcp udp'

config rule
	option name 'LAN: Allow access to SSH server in DMZ'
	list proto 'tcp'
	option dest 'dmz'
	option src 'lan'
	option target 'ACCEPT'
	option family 'ipv4'
	option src_port '22'
	option dest_port '22'

config rule
	option name 'LAN: Allow ping to DMZ'
	list proto 'icmp'
	option src 'lan'
	option dest 'dmz'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'DMZ: Allow access to DHCP server'
	list proto 'udp'
	option src 'dmz'
	option target 'ACCEPT'
	option family 'ipv4'
	option src_port '67-68'
	option dest_port '67-68'

config rule
	option name 'DMZ: Block access to private networks and allow access to internet on ports 53, 123, 465, 587'
	option dest '*'
	list dest_ip '!10.0.0.0/8'
	list dest_ip '!172.16.0.0/12'
	list dest_ip '!192.168.0.0/16'
	option target 'ACCEPT'
	option family 'ipv4'
	option src 'dmz'
	list proto 'tcp'
	list proto 'udp'
	option src_port '53 123 465 587'
	option dest_port '53 123 465 587'

config forwarding
	option src 'lan'
	option dest 'dmz'

config forwarding
	option src 'lan'
	option dest 'wanb'

config redirect
	option name 'Allow access to Reverse Proxy server HTTP'
	option dest 'dmz'
	option target 'DNAT'
	list proto 'tcp'
	option src 'wanb'
	option src_dip '94.79.184.226'
	option src_dport '80'
	option dest_ip '172.16.9.10'
	option dest_port '80'

config redirect
	option name 'Allow access to Reverse Proxy server HTTPS'
	option dest 'dmz'
	option target 'DNAT'
	list proto 'tcp'
	option src 'wanb'
	option src_dip '94.79.184.226'
	option src_dport '443'
	option dest_ip '172.16.9.10'
	option dest_port '443'

config rule
	option name 'LAN: Allow access to HTTP(S) server'
	list proto 'tcp'
	option src 'lan'
	option dest_port '80 443'
	option target 'ACCEPT'
	option family 'ipv4'
	option dest '*'
	list dest_ip '172.21.10.10'
	option enabled '0'

config rule
	option name 'DMZ: Allow access to HTTP(S) server'
	list proto 'tcp'
	option src 'dmz'
	option dest_port '80 443'
	option target 'ACCEPT'
	option family 'ipv4'
	option dest '*'
	list dest_ip '172.21.10.10'
	option enabled '0'

config rule
	option name 'LAN: Block all'
	option family 'ipv4'
	option src 'lan'
	option dest 'wan'
	option target 'REJECT'
	option enabled '0'

config rule
	option name 'DMZ: Block all'
	option src 'dmz'
	option dest 'wan'
	option target 'REJECT'
	option family 'ipv4'

config rule
	option name 'LAN: Block all'
	option family 'ipv4'
	option src 'lan'
	option dest 'wanb'
	option target 'REJECT'
	option enabled '0'

config rule
	option name 'DMZ: Block all'
	option src 'dmz'
	option dest 'wanb'
	option target 'REJECT'
	option family 'ipv4'

If you disable nat, the isp router must have a static route to the lan subnet of OpenWrt via the wan ip address.

I have setup a static route in ISP router as instructed.

Could you please explain why this static route is required for egress traffic from lan subnet of OpenWRT?
My understanding was that this static route in ISP router is only required to ensure communication from ISP router's LAN to any subnet "behind" this router. And this would be ingress traffic from OpenWRT point of view.

And would it make sense to setup 1 static route in ISP router for /20 subnet, e.g. network mask 255.255.240.0, instead of multiple static routes for each /24 subnet?

That is half of the traffic, but if you don't have it how will you get the responses to what you send from the lan of OpenWrt?
Also this doesn't always work properly. Sometimes routers from ISPs don't masquerade all outgoing traffic, but only the traffic originating from their directly connected interfaces.

If the gateway is the same, then yes.

Many thanks for your support... this network issue is solved now.

I'll continue with setup of DMZ zone and fine-tune the firewall rules.
Let's wait and see if further issues arise after this...