Double Nat Issue (LTE modem + OpenWrt router)

Lynx · October 22, 2022, 11:25pm

Continuing the discussion from Preventing double NAT:

For this example the WAN router's LAN is 192.168.0.1/24 and the OpenWrt router's LAN is 192.168.1.1/24.

In the WAN router first create a DHCP reservation so that the OpenWrt router has a consistent IP address such as 192.168.0.20.
If you chose an IP different from the existing leased IP, restart the OpenWrt router or at least restart the network so it picks up the new IP.
Then add a route for the second LAN (192.168.1.0/24) via 192.168.0.20 which is the reserved IP of the OpenWrt router on the first router's LAN.
These changes should not otherwise affect the operation of the WAN router at all even if the OpenWrt router is not connected.
Then go to the OpenWrt router firewall setting and uncheck "masquerade" on the WAN and restart the firewall. No more double NAT.

Does the above well-written guide from @mk24 apply to the situation I now have:

192.168.0.1 (OpenWrt Zyxel NR7101 LTE router)
192.168.1.1 (main OpenWrt router)
ethernet cable connecting from LAN port of NR7101 to WAN port of OpenWrt router.

I want to emulate a poor man's bridge mode like as described here:

Assuming @mk24's guide applies to my situation then I can follow it all but can't seem to get it to work properly. When creating the static route as per:

Then add a route for the second LAN (192.168.1.0/24) via 192.168.0.20 which is the reserved IP of the OpenWrt router on the first router's LAN.

What form should this take in LuCi:

What do I set interface and source IP to be? When I set up the route I don't see anything get entered in the routing table unless I set source IP to one of the options.

Then go to the OpenWrt router firewall setting and uncheck "masquerade" on the WAN and restart the firewall. No more double NAT.

When I do this I lose internet connectivity.

psherman · October 22, 2022, 11:33pm

I'm assuming the 192.168.0.0/24 subnet belongs to the upstream router -- that is to say that the one with LTE is the way that the entire network obtains internet connectivity. Is that correct?

Typically the best way to achieve this is to connect the downstream router via its WAN port. So it would look like this:

Zyxel NR7101 LAN port > main OpenWrt router WAN port.
On the Zyxel, you will then add a static route 192.168.1.0/24 via < wan_ip_address_of_main_OpenWrt_router >
- where the WAN IP address of the main router is in the 192.168.0.0/24 network.
The main router does not need a static route
On the main router, disable masquerading on the wan firewall zone.
If you consider the NR7101 LAN to be trusted and if it has hosts that need to reach other hosts that sit behind the main router, you can enable wan > lan forwarding on the main router... this is optional.

That should be all you need to do.

Lynx · October 22, 2022, 11:36pm

Exactly.

So it seems you concur with the applicability of @mk24's approach for my scenario, but as stupid as this is I can't figure out how to add the static route correctly as per the above. Can you give me any tips here?

I'm not sure what to set:

route type
interface
source IP

I presume 'table' can be default.

psherman · October 22, 2022, 11:40pm

make the interface and route type unspecified.
Is the WAN address of the 2nd router 192.168.0.2?

How are you testing (and determining that it isn't working)?

Lynx · October 22, 2022, 11:41pm

Yep - I set that as static lease on Zyxel NR7101.

Route type must be one of:

When I check 'ip route show table default' then I don't see any entry.

And when I disable "masquerade" on the WAN it breaks.

psherman · October 22, 2022, 11:42pm

select anycast.

Lynx · October 22, 2022, 11:46pm

config route
        option target '192.168.1.0/24'
        option gateway '192.168.0.2'
        option table 'default'
        option type 'anycast'

Look correct?

OK now I have internet access but cannot access the Zyxel from my LAN clients.

psherman · October 22, 2022, 11:48pm

These lines can even be omitted! No harm in them being there, though.

To clarify, you cannot reach the Zyxel from the 192.168.1.0/24 network?

Let's see the network and firewall files from each device (please make it clear which is which).

Lynx · October 23, 2022, 12:01am

Actually it's not working sorry. When I disable "masquerade" on the WAN it breaks.

I'll post network and firewall on both?

psherman · October 23, 2022, 12:04am

The whole point of disabling masquerade is to enable symmetric routing. If it breaks the internet, that means something isn't right (obviously). We'll get it working.

yes, please.

Lynx · October 23, 2022, 12:07am

OpenWrt router:

-- network:

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'XX'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option proto 'dhcp'
        option ip4table 'wan'
        option peerdns '0'
        list dns '185.228.168.168'
        list dns '185.228.169.168'
        option device 'wan'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config interface 'vpn'
        option proto 'wireguard'
        option private_key 'XX'
        list addresses 'XX'
        option ip4table 'vpn'
        option fwmark '0x3'

config wireguard_vpn
        option public_key 'XX'
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '::/0'
        option endpoint_port 'XX'
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        option description 'XX'
        option endpoint_host 'XX'

config rule
        option priority '14000'
        option lookup 'wan'
        option src '192.168.1.134/32'
        option in 'lan'
        option disabled '1'

config rule
        option priority '14000'
        option lookup 'wan'
        option src '192.168.2.8/32'
        option in 'guest'

config rule
        option priority '14000'
        option lookup 'wan'
        option src '192.168.2.9/32'
        option in 'guest'

config rule
        option priority '14000'
        option lookup 'wan'
        option in 'guest'
        option src '192.168.2.10/32'

config rule
        option priority '14000'
        option lookup 'wan'
        option in 'lan'
        option dest '192.168.0.1/32'

config rule
        option priority '15000'
        option in 'lan'
        option lookup 'vpn'

config rule
        option in 'guest'
        option lookup 'vpn'
        option priority '15000'

config rule
        option priority '40000'
        option in 'lan'
        option lookup 'wan'

config device
        option type 'bridge'
        option name 'br-guest'

config interface 'guest'
        option proto 'static'
        option device 'br-guest'
        option ipaddr '192.168.2.1'
        option netmask '255.255.255.0'
        option ip4table 'br-guest'
        option type 'bridge'
        option defaultroute '0'

config rule
        option priority '40000'
        option in 'guest'
        option lookup 'wan'

config device
        option name 'wan'

config device
        option name 'wan'

-- firewall:

root@OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'guest'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option input 'REJECT'
        list network 'guest'

config zone
        option name 'wan'
        option output 'ACCEPT'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'
        list network 'vpn'
        option input 'REJECT'
        option forward 'REJECT'
        option masq '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option dest 'wan'
        option src '*'
        option start_time '21:00:00'
        option stop_time '17:00:00'
        option target 'REJECT'
        option name 'Restrict-OpenWrt-GitHub'
        list proto 'all'
        list dest_ip '139.59.210.197'
        list dest_ip '192.30.252.0/22'
        list dest_ip '185.199.108.0/22'
        list dest_ip '140.82.112.0/20'
        list dest_ip '143.55.64.0/20'
        list dest_ip '20.201.28.151/32'
        list dest_ip '20.205.243.166/32'
        list dest_ip '102.133.202.242/32'
        list dest_ip '20.248.137.48/32'
        list dest_ip '20.207.73.82/32'
        list dest_ip '20.27.177.113/32'
        list dest_ip '20.200.245.247/32'
        list dest_ip '20.233.54.53/32'
        option weekdays 'Mon Tue Wed Thu Fri'
        option enabled '0'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option dest_ip '1.1.1.1'
        option src_dport '53'
        option src 'guest'
        option name 'Intercept-GUEST-DNS-Cloudflare'
        list src_mac 'XX'
        list src_mac 'XX'
        list src_mac 'XX'

config redirect
        option target 'DNAT'
        option src 'guest'
        option src_dport '53'
        option name 'Intercept-GUEST-DNS'

config redirect
        option target 'DNAT'
        option src 'lan'
        option dest_ip '1.1.1.1'
        option src_dport '53'
        option name 'Intercept-LAN-DNS-Cloudflare'
        list src_mac 'XX'
        option enabled '0'

config redirect
        option target 'DNAT'
        option src_dport '53'
        option src 'lan'
        option name 'Intercept-LAN-DNS'

config forwarding
        option src 'guest'
        option dest 'wan'

config rule
        option dest_port '53'
        option target 'ACCEPT'
        option src 'guest'
        option name 'Allow-GUEST-DNS'

config rule
        list proto 'udp'
        option dest_port '67'
        option target 'ACCEPT'
        option src 'guest'
        option name 'Allow-GUEST-DHCP'

Zyxel

-- network:

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '1'
        option ula_prefix 'XX'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.0.1'

config interface 'wan'
        option proto 'qmi'
        option device '/dev/cdc-wdm0'
        option apn 'XX'
        option auth 'pap'
        option username 'XX'
        option password 'XX'
        option pdptype 'ipv4'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config route
        option target '192.168.1.0/24'
        option gateway '192.168.0.2'
        option table 'default'
        option type 'anycast'

-- firewall:

root@OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option mtu_fix '1'
        option forward 'REJECT'
        option masq '1'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config forwarding
        option src 'lan'
        option dest 'wan'

n.b. masquearing enabled in LuCi so I can write this.

psherman · October 23, 2022, 12:10am

Is the VPN active? In the case where you are using a VPN, this would require masquerading... but if all traffic is tunneled, it doesn't matter if you're double-NAT (or more). If only some of the traffic is tunneled, you'll need PBR and you must put the VPN in a different zone so that you can turn masquerading off on the wan zone.

Lynx · October 23, 2022, 12:10am

Yes! VPN active in the OpenWrt router.

psherman · October 23, 2022, 12:11am

Is all traffic running through the VPN (nominally)? Or is it policy driven?

Lynx · October 23, 2022, 12:11am

PBR.. so should I split out wan / vpn and enable masquearing only for vpn?

psherman · October 23, 2022, 12:15am

Yes. Create a new zone for the VPN. It can mirror the settings of the wan zone (forward and input = reject, output = accept, masquerading enabled).

Then remove the vpn network from the wan zone and you should be able to turn off masquerading on the wan zone.

You'll need to adjust your routing policies to work with the new zone.

I'd recommend testing with the vpn disabled so that you can verify connectivity is working as expected in general.

Lynx · October 23, 2022, 12:15am

Is this the best way? It was a lot simpler with my B818-263 in bridge mode.

p.s. thanks so much for your help here!

psherman · October 23, 2022, 12:16am

'best' depends on the objectives... we haven't discussed exactly what you are trying to achieve. But putting the VPN in a unique zone is not really much different than stuffing it into the wan zone... but it gives you more granular control.

Lynx · October 23, 2022, 12:17am

In terms of objective just trying to 'do the right thing' and sort of emulate what I had before with B818-263 in bridge mode. I looked at 'trelay' but that looks complicated so I figured this solution described in this thread is good enough?

Lynx · October 23, 2022, 12:20am

Actually situation unchanged:

Now I need to keep 'wan' masquerading active to retain internet connectivity.

Or maybe I need some refreshes?