NAT reflection only working when doing tcpdump on br-lan

freshter · January 6, 2023, 8:24am

Hi all,

Long time lurker, first time poster here.

I've been using openwrt my raspberry pi CM4 for a while now and I've recently upgraded to 21.02.5.

In the firewall I have some "port forward" rules sending e.g. port 80 and port 443 on the wan interface to a server on my lan.

Previously, connecting on the public IP address from within the lan worked fine and correctly redirected the traffic to the internal address so I had no problem connecting to it from within the network. Since the upgrade however this seems to have stopped working (without the rules having changed). For completeness, I did start using docker on openwrt, but removing it and all the interfaces/zones for it does not seem to have any effect.

Since it didn't work, I started troubleshooting and launched a tcpdump on the br-lan interface to see what's going on, and surely, while tcpdump is running on the br-lan interface I can properly connect to it. Note that this only happens when running on br-lan. If I run on the actual physical interface (eth0) it does not work.

Does that ring a bell to anyone?

EDIT:
I'm running tcpdump like this

tcpdump -n -i br-lan host 10.0.2.1 and port 443

Here's the firewall configuration (/etc/config/firewall)

root@pirouter:~# cat /etc/config/firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list device 'tun+'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	list network 'WAN'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled '0'

config include
	option path '/etc/firewall.user'

config redirect
	option target 'DNAT'
	option name 'http'
	list proto 'tcp'
	option src 'wan'
	option src_dport '80'
	option dest 'lan'
	option dest_ip '10.0.2.1'
	option dest_port '80'
	list reflection_zone 'lan'

config redirect
	option target 'DNAT'
	option name 'https'
	list proto 'tcp'
	option src 'wan'
	option src_dport '443'
	option dest 'lan'
	option dest_ip '10.0.2.1'
	option dest_port '443'
	list reflection_zone 'lan'

config redirect
	option target 'DNAT'
	option name 'ssh'
	list proto 'tcp'
	option src 'wan'
	option src_dport '22'
	option dest 'lan'
	option dest_ip '10.0.2.1'
	option dest_port '22'
	list reflection_zone 'lan'

config redirect
	option target 'DNAT'
	option name 'et'
	list proto 'udp'
	option src 'wan'
	option src_dport '27960'
	option dest 'lan'
	option dest_ip '10.0.2.1'
	option dest_port '27960'
	list reflection_zone 'lan'

config redirect
	option target 'DNAT'
	option name 'znc'
	list proto 'tcp'
	option src 'wan'
	option src_dport '6697'
	option dest 'lan'
	option dest_ip '10.0.2.1'
	option dest_port '6697'
	list reflection_zone 'lan'

config rule 'ovpn'
	option name 'Allow-OpenVPN'
	option src 'wan'
	option dest_port '1194'
	option proto 'udp'
	option target 'ACCEPT'

config redirect
	option dest 'lan'
	option target 'DNAT'
	option name 'torrent'
	list proto 'tcp'
	option src 'wan'
	option src_dport '51413'
	option dest_ip '10.0.2.1'
	option dest_port '51413'

config redirect
	option target 'DNAT'
	option name 'intercept-dns'
	option src 'lan'
	option src_dport '53'

#config zone 'docker'
#	option input 'ACCEPT'
#	option output 'ACCEPT'
#	option forward 'ACCEPT'
#	option name 'docker'
#	list network 'docker'

config rule
	option name 'allow adguard web'
	list proto 'tcp'
	option src 'lan'
	option dest_port '8080'
	option target 'ACCEPT'

and (redacted) network configuration (/etc/config/network)

root@pirouter:~# cat /etc/config/network 

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdba:717d:a9cc::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '10.0.2.254'
	list dns '10.0.2.253'
	list dns '10.0.2.252'
	list dns_search 'local'
	option ip6assign '60'

config interface 'WAN'
	option proto 'pppoe'
	option device 'eth1'
	option username 'foobar'
	option password 'faa'
	option ipv6 'auto'

#config interface 'docker'
#	option device 'docker0'
#	option proto 'none'
#	option auto '0'
#
#config device
#	option type 'bridge'
#	option name 'docker0'

trendy · January 6, 2023, 9:00am

What is the output of iptables-save -c -t nat ?

freshter · January 6, 2023, 9:23am

Here it is (redacted public address)

# Generated by iptables-save v1.8.7 on Fri Jan  6 09:22:51 2023
*nat
:PREROUTING ACCEPT [13338:2385669]
:INPUT ACCEPT [5394:369391]
:OUTPUT ACCEPT [3872:246044]
:POSTROUTING ACCEPT [714:37162]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
[19284:2763735] -A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
[0:0] -A PREROUTING -i tun+ -m comment --comment "!fw3" -j zone_lan_prerouting
[16791:2530583] -A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
[2485:229732] -A PREROUTING -i pppoe-WAN -m comment --comment "!fw3" -j zone_wan_prerouting
[9943:885310] -A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
[0:0] -A POSTROUTING -o tun+ -m comment --comment "!fw3" -j zone_lan_postrouting
[717:37342] -A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
[9226:847968] -A POSTROUTING -o pppoe-WAN -m comment --comment "!fw3" -j zone_wan_postrouting
[717:37342] -A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j SNAT --to-source 10.0.2.254
[3:180] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 443 -m comment --comment "!fw3: https (reflection)" -j SNAT --to-source 10.0.2.254
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 10.0.2.254
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p udp -m udp --dport 27960 -m comment --comment "!fw3: et (reflection)" -j SNAT --to-source 10.0.2.254
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 6697 -m comment --comment "!fw3: znc (reflection)" -j SNAT --to-source 10.0.2.254
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 51413 -m comment --comment "!fw3: torrent (reflection)" -j SNAT --to-source 10.0.2.254
[16791:2530583] -A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
[0:0] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http (reflection)" -j DNAT --to-destination 10.0.2.1:80
[225:13500] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 443 -m comment --comment "!fw3: https (reflection)" -j DNAT --to-destination 10.0.2.1:443
[0:0] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 10.0.2.1:22
[0:0] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p udp -m udp --dport 27960 -m comment --comment "!fw3: et (reflection)" -j DNAT --to-destination 10.0.2.1:27960
[0:0] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 6697 -m comment --comment "!fw3: znc (reflection)" -j DNAT --to-destination 10.0.2.1:6697
[132:7920] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 51413 -m comment --comment "!fw3: torrent (reflection)" -j DNAT --to-destination 10.0.2.1:51413
[0:0] -A zone_lan_prerouting -p tcp -m tcp --dport 53 -m comment --comment "!fw3: intercept-dns" -j REDIRECT --to-ports 53
[4878:320241] -A zone_lan_prerouting -p udp -m udp --dport 53 -m comment --comment "!fw3: intercept-dns" -j REDIRECT --to-ports 53
[9226:847968] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
[9226:847968] -A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
[2485:229732] -A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
[16:832] -A zone_wan_prerouting -p tcp -m tcp --dport 80 -m comment --comment "!fw3: http" -j DNAT --to-destination 10.0.2.1:80
[49:2840] -A zone_wan_prerouting -p tcp -m tcp --dport 443 -m comment --comment "!fw3: https" -j DNAT --to-destination 10.0.2.1:443
[90:5248] -A zone_wan_prerouting -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh" -j DNAT --to-destination 10.0.2.1:22
[331:14417] -A zone_wan_prerouting -p udp -m udp --dport 27960 -m comment --comment "!fw3: et" -j DNAT --to-destination 10.0.2.1:27960
[0:0] -A zone_wan_prerouting -p tcp -m tcp --dport 6697 -m comment --comment "!fw3: znc" -j DNAT --to-destination 10.0.2.1:6697
[225:13068] -A zone_wan_prerouting -p tcp -m tcp --dport 51413 -m comment --comment "!fw3: torrent" -j DNAT --to-destination 10.0.2.1:51413
COMMIT
# Completed on Fri Jan  6 09:22:51 2023

trendy · January 6, 2023, 10:05am

As far as iptables are concerned the reflection rules are there and have some hits. You can try to reset the counters and try again without the tcpdump running to verify that the counters will increase.

freshter · January 6, 2023, 11:18am

Focusing on port 22 now (it's less chatty), after iptables -Z -t nat I see the counters increase only on prerouting, not postrouting:

root@pirouter:~# iptables-save -c -t nat | grep "dport 22"
[0:0] -A zone_lan_postrouting -s 10.0.2.0/24 -d 10.0.2.1/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j SNAT --to-source 10.0.2.254
[3:180] -A zone_lan_prerouting -s 10.0.2.0/24 -d A.B.174.111/32 -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh (reflection)" -j DNAT --to-destination 10.0.2.1:22
[0:0] -A zone_wan_prerouting -p tcp -m tcp --dport 22 -m comment --comment "!fw3: ssh" -j DNAT --to-destination 10.0.2.1:22

freshter · January 6, 2023, 2:53pm

Update after a discussion on IRC:
it seems when I run tcpdump -p, thus not putting the br-lan interface in promiscuous mode, the nat reflection does not work.

So the fact that the interface gets put in promiscuous mode makes the NAT reflection work. Hope that's a clue for someone.

pavelgl · January 6, 2023, 3:06pm

The rules seem correct, so look for a workaround.
Enable masquerading on the lan zone or create a generic SNAT rule to see if it makes a difference.

iptables -t nat -I zone_lan_postrouting -d 10.0.2.1/32 -j SNAT --to-source 10.0.2.254

freshter · January 6, 2023, 3:32pm

Nope, unfortunately no difference. Shouldn't I make an issue on github?

trendy · January 6, 2023, 3:46pm

It works fine for me even without promiscuous mode.
You can add option promisc '1' in br-lan device to enable it if it helps.

freshter · January 6, 2023, 5:37pm

I guess I'll first try the same thing on spare, freshly installed pi. If it works fine there I can track where I've gone wrong.

freshter · January 8, 2023, 10:12am

So, it seems docker is interfering after all. On a fresh install it works fine until I install the dockerd package. Even if I disable the service with /etc/init.d/dockerd disable it still does not work. I have to actually remove the package with opkg remove dockerd

trendy · January 8, 2023, 12:38pm

Are the firewall rules added by docker removed when you disable it?

freshter · January 8, 2023, 12:45pm

Even when I disable all rules, interfaces and zones created by dockerd it still happens.

it's caused by the sysctl file created by the dockerd package:

# Do not edit, changes to this file will be lost on upgrades
# /etc/sysctl.conf can be used to customize sysctl settings

# enable bridge firewalling for docker
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1

Only when I disable those it works. Still looking how I can fix that.

freshter · January 9, 2023, 6:30pm

Still not working unless I disable the bridge firewalling. I'd like to keep that enabled so docker can keep firewalling traffic on the bridges.

I've started troubleshooting by inserting rules at the very top of the iptables chains with these commands:

iptables -I INPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering filter INPUT: "
iptables -I OUTPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering filter OUTPUT: "
iptables -I FORWARD -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering filter FORWARD: "

iptables -t mangle -I INPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering mangle INPUT: "
iptables -t mangle -I OUTPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering mangle OUTPUT: "
iptables -t mangle -I FORWARD -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering mangle FORWARD: "
iptables -t mangle -I PREROUTING -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering mangle PREROUTING: "
iptables -t mangle -I POSTROUTING -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering mangle POSTROUTING: "

iptables -t nat -I INPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering nat INPUT: "
iptables -t nat -I OUTPUT -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering nat OUTPUT: "
iptables -t nat -I PREROUTING -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering nat PREROUTING: "
iptables -t nat -I POSTROUTING -p tcp -m tcp --dport 28 -j LOG --log-prefix "entering nat POSTROUTING: "

So I'm connection from host 10.0.2.31 towards the public address on tcp port 28. This has a redirect to the internal host 10.0.2.30.

However, when looking at the logs I only see these rules hitting:

Mon Jan  9 18:09:41 2023 kern.warn kernel: [86788.180876] entering mangle PREROUTING: IN=br-lan OUT= PHYSIN=eth0 MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=A.B.114.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32838 DF PROTO=TCP SPT=47436 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:09:41 2023 kern.warn kernel: [86788.203118] entering nat PREROUTING: IN=br-lan OUT= PHYSIN=eth0 MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=A.B.114.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=32838 DF PROTO=TCP SPT=47436 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0

So it only hits the mangle table prerouting chain and the nat table prerouting chain and then processing stops. I know it hits the proper NAT prerouting rule because I also tried adding a log statement just before and just after that rule. Only the one just before gets logged.

Here's the log when bridge firewalling is disabled:

Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.365945] entering mangle PREROUTING: IN=br-lan OUT= MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=A.B.114.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.387120] entering nat PREROUTING: IN=br-lan OUT= MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=A.B.114.39 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.408152] entering mangle FORWARD: IN=br-lan OUT=br-lan MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=10.0.2.30 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.429271] entering filter FORWARD: IN=br-lan OUT=br-lan MAC=e4:5f:01:16:45:24:a6:5f:ed:98:38:df:08:00 SRC=10.0.2.31 DST=10.0.2.30 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.450332] entering mangle POSTROUTING: IN= OUT=br-lan SRC=10.0.2.31 DST=10.0.2.30 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0
Mon Jan  9 18:20:26 2023 kern.warn kernel: [87433.467268] entering nat POSTROUTING: IN= OUT=br-lan SRC=10.0.2.31 DST=10.0.2.30 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=59327 DF PROTO=TCP SPT=34346 DPT=28 WINDOW=64240 RES=0x00 SYN URGP=0

So for some reason instead of ending up in the forward chains, the packet processing just stops.

I'm running out of ideas at this point. Anyone else got any ideas?

pmw · April 1, 2023, 8:31pm

I had the same issue on OpenWRT 22.03.2 on x86/64. It was also triggered by the sysctl changes made by dockerd and momentarily "fixed" by putting br-lan in promiscuous mode. Reverting net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to 0 fixed NAT reflection with no change to Docker's behavior that I can see.

I am also wondering whether disabling these sysctls contrary to what Docker wants is OK. I found that this is actually recommended by Oracle Linux's documentation but I'm not sure if the reasons they have for recommending it apply here as well.

Another thing to consider is disabling nf-call-iptables globally with the sysctl and enabling it only for Docker's bridges through /sys/class/net/docker0/bridge/nf_call_iptables, etc. - this seems more like a proper solution but I am not sure.

gehrq2 · February 15, 2024, 12:53pm

same problem on OpenWrt SNAPSHOT r25038-20240201, not using docker

lleachii · February 15, 2024, 1:37pm

As noted above, the rule is OK. Here's an example rule:

config redirect             
        option target 'DNAT'  
        option src 'wan'
        option dest 'lan'
        option src_dport '51413'        
        option dest_port '51413'           
        option dest_ip 'xxx.xxx.xxx.xxx'
        option name 'BitTorrent_dsktp'
        list proto 'tcp'          
        list proto 'udp

Are you're saying the BitTorrent doesn't work until you run tcpdump on br-lan?
How are you testing that the port is open?