Fw4 MASQUERADE (un)expected behavior?

wokkocher · August 13, 2024, 7:34pm

Hi all. I stumbled over some (for me) weird behavior. I wanted to create some firewall rules and they behaved different than expected in IPv4 and IPv6. For some reason I wanted to SNAT all traffic which goes out on my interface "wgcircle". I know this should be avoided for IPv6, but I have some reasons to do so. As far as I know there are different ways to do so.

1.) Create a seperate firewall zone and and configure option masq '1' and option masq6 '1' on it.
2.) Do selective SNAT by a firewall rule.

Just for testing I did the 2nd way:

config nat 'nat_cirle_both'
	option src 'wg_zone'
	option target 'MASQUERADE'
	option name 'WG Circle NAT both'
	option device 'wgcircle'
	list proto 'all'

However, despite the grafically description of IPv4 and IPv6 this rule did only work for IPv4. Source (not SNATed) IPs were 192.168.12.3 and dead:beef::ca11:3. SNATed IPs should be 192.168.123.1 and d00d:badc:ab1e::1

root@A:/home/dagama# tcpdump -i wgcircle
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wgcircle, link-type RAW (Raw IP), snapshot length 262144 bytes
21:11:05.756607 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1515, seq 1, length 64
21:11:05.801231 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1515, seq 1, length 64
21:11:06.754157 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1516, seq 1, length 64
21:11:06.798580 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1516, seq 1, length 64
21:11:07.840234 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1517, seq 1, length 64
21:11:07.881127 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1517, seq 1, length 64
21:11:36.261500 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1518, seq 1, length 64
21:11:37.282313 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1519, seq 1, length 64
21:11:38.264975 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1520, seq 1, length 64
21:11:39.283943 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1521, seq 1, length 64
21:11:40.273306 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1522, seq 1, length 64
21:11:41.289804 IP6 dead:beef::ca11:3 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1523, seq 1, length 64

If I set two seperate firewall rules for IPv4 and IPv6 instead then the result is like expected:

config nat 'nat_cirle'
	option src 'wg_zone'
	option target 'MASQUERADE'
	option name 'WG Circle NAT'
	option device 'wgcircle'
	option family 'ipv4'
	list proto 'all'

config nat 'nat66_cirle'
	option src 'wg_zone'
	option target 'MASQUERADE'
	option name 'WG Circle NAT66'
	option device 'wgcircle'
	option family 'ipv6'
	list proto 'all'

root@A:/home/dagama# tcpdump -i wgcircle
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wgcircle, link-type RAW (Raw IP), snapshot length 262144 bytes
21:13:06.631924 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1524, seq 1, length 64
21:13:06.673868 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1524, seq 1, length 64
21:13:07.653366 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1525, seq 1, length 64
21:13:07.708570 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1525, seq 1, length 64
21:13:08.674219 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1526, seq 1, length 64
21:13:08.717588 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1526, seq 1, length 64
21:13:09.647632 IP 192.168.123.1 > 192.168.89.1: ICMP echo request, id 1527, seq 1, length 64
21:13:09.687356 IP 192.168.89.1 > 192.168.123.1: ICMP echo reply, id 1527, seq 1, length 64
21:13:19.273348 IP6 d00d:badc:ab1e::1 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1528, seq 1, length 64
21:13:19.324282 IP6 somehost.somedomain.dynv6.net > d00d:badc:ab1e::1: ICMP6, echo reply, id 1528, seq 1, length 64
21:13:20.298049 IP6 d00d:badc:ab1e::1 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1529, seq 1, length 64
21:13:20.339751 IP6 somehost.somedomain.dynv6.net > d00d:badc:ab1e::1: ICMP6, echo reply, id 1529, seq 1, length 64
21:13:21.284698 IP6 d00d:badc:ab1e::1 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1530, seq 1, length 64
21:13:21.335579 IP6 somehost.somedomain.dynv6.net > d00d:badc:ab1e::1: ICMP6, echo reply, id 1530, seq 1, length 64
21:13:22.300383 IP6 d00d:badc:ab1e::1 > somehost.somedomain.dynv6.net: ICMP6, echo request, id 1531, seq 1, length 64
21:13:22.345165 IP6 somehost.somedomain.dynv6.net > d00d:badc:ab1e::1: ICMP6, echo reply, id 1531, seq 1, length 64

So my question is: is this expected behavior? Is it a display bug in LuCI? Am I thinking wrong?

Thank you in advance for all that help I will hopefully get !

brada4 · August 13, 2024, 8:46pm

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

wokkocher · August 13, 2024, 9:51pm

Sure, why not, if it helps . I just wanted to post the most important parts from my point of view.

root@A:/home/dagama# ubus call system board
{
        "kernel": "5.15.162",
        "hostname": "A",
        "system": "Qualcomm Atheros QCA9533 ver 2 rev 0",
        "model": "GL.iNet GL-AR300M (NOR)",
        "board_name": "glinet,gl-ar300m-nor",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "ath79/nand",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}

root@A:/home/dagama# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'ddaa::/16'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '64'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix '60'
        option sourcefilter '0'

config interface 'wwan'
        option proto 'dhcp'

config interface 'wgcafe'
        option proto 'wireguard'
        option private_key 'REDACTED'
        option listen_port 'REDACTED'
        list addresses '192.168.12.1/24'
        list addresses 'dead:beef::b055/61'

config wireguard_wgcafe
        option public_key 'REDACTED'
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        option preshared_key 'REDACTED'
        list allowed_ips '192.168.12.3/32'
        list allowed_ips 'dead:beef::ca11:3/128'
        option description 'Pixel 6a Max'

config interface 'wgcircle'
        option proto 'wireguard'
        option listen_port 'REDACTED'
        option private_key 'REDACTED'
        list addresses 'dd00:badc:ab1e::1/128'
        list addresses '192.168.123.1/32'
        option sourcefilter '0'

config wireguard_wgcircle
        option public_key 'REDACTED'
        option preshared_key 'REDACTED'
        list allowed_ips '192.168.123.2/32'
        list allowed_ips 'dd00:badc:ab1e::2/128'
        list allowed_ips '192.168.89.0/24'
        list allowed_ips 'aaaa:bbbb:cccc::/48' # contains (somehost.somedomain.dynv6.net)
        option route_allowed_ips '1'
        option persistent_keepalive '25'
        option endpoint_host 'REDACTED'
        option endpoint_port 'REDACTED'

root@A:/home/dagama# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/ahb/18100000.wmac'
        option channel '1'
        option band '2g'
        option htmode 'HT20'
        option cell_density '0'
        option disabled '1'

config wifi-iface 'wifinet1'
        option device 'radio0'
        option mode 'ap'
        option ssid 'BBC'
        option encryption 'sae-mixed'
        option key 'REDACTED'
        option network 'lan'
        option disabled '1'

root@A:/home/dagama# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '0'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

root@A:/home/dagama# cat /etc/config/firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'
        list network 'wwan'

config zone
        option name 'wg_zone'
        option input 'ACCEPT'
        option forward 'ACCEPT'
        option output 'ACCEPT'
        list network 'wgcafe'
        list network 'wgcircle'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Wireguard Cafe IPv4'
        option family 'ipv4'
        list proto 'udp'
        option src 'wan'
        option dest_port 'REDACTED'
        option target 'ACCEPT'

config rule
        option dest_port 'REDACTED
        option src 'wan'
        option name 'Wireguard Cafe IPv6'
        option family 'ipv6'
        option target 'ACCEPT'
        list proto 'udp'

config forwarding
        option dest 'lan'
        option src 'wg_zone'

config forwarding
        option dest 'wan'
        option src 'wg_zone'

config forwarding
        option dest 'wg_zone'
        option src 'lan'

config forwarding
        option dest 'wg_zone'
        option src 'wan'

config rule
        option name 'Wireguard Circle IPv4'
        option family 'ipv4'
        list proto 'udp'
        option src 'wan'
        option dest_port 'REDACTED'
        option target 'ACCEPT'

config rule
        option family 'ipv6'
        list proto 'udp'
        option src 'wan'
        option dest_port 'REDACTED'
        option target 'ACCEPT'
        option name 'Wireguard Circle IPv6'

config nat 'nat66'
        option family 'ipv6'
        option src 'wan'
        option target 'MASQUERADE'
        option name 'WG Cafe NAT66'
        list proto 'all'
        option src_ip 'dead:beef::/61'
        option enabled '0'

config nat 'nat_cirle'
        option src 'wg_zone'
        option target 'MASQUERADE'
        option name 'WG Circle NAT'
        option device 'wgcircle'
        option family 'ipv4'
        list proto 'all'

config nat 'nat66_cirle'
        option src 'wg_zone'
        option target 'MASQUERADE'
        option name 'WG Circle NAT66'
        option device 'wgcircle'
        option family 'ipv6'
        list proto 'all'

config include 'npt6'
        option path '/etc/nftables.d/npt6.sh'

root@A:/home/dagama# cat /etc/nftables.d/npt6.sh
#!/bin/sh
LAN_PFX="dead:beef::/61"
. /lib/functions/network.sh
network_flush_cache
network_find_wan6 WAN_IF
network_get_device WAN_DEV "${WAN_IF}"
network_get_prefix6 WAN_PFX "${WAN_IF}"
WBITS=${WAN_PFX##*/}
LBITS=${LAN_PFX##*/}
if [ "$WBITS" -le "$LBITS" ] ; then
#NPT
    if [ "$(uci -q get firewall.nat66.enabled)" != "0" ] ; then
        uci set firewall.nat66.enabled="0"
        uci commit firewall
        service firewall restart
    fi
    WAN_PFX_PART=${WAN_PFX%%/*}/$LBITS
    nft add rule inet fw4 srcnat oifname "${WAN_DEV}" snat ip6 prefix to ip6 saddr map { "${LAN_PFX}" : "${WAN_PFX_PART}" }
else :;
#NAT
    if [ "$(uci -q get firewall.nat66.enabled)" = "0" ] ; then
        uci set firewall.nat66.enabled="1"
        uci commit firewall
        service firewall restart
    fi
fi

brada4 · August 14, 2024, 6:26am

Change wan forward to drop, there is no packets you are about to forward or willing to receive to addresses that are not your public ones.
Probably you dont want NAT (making all to go out from single address) on top of npt (re-mapping just prefixes)

wokkocher · August 14, 2024, 7:54am

Yes, I will do that. This is an ancient fragment from an old SIP configuartion.

AFAIK the NPT applies only to packages which go out through wan and only if I have a global prefix big enough to do some mapping (minimal /61, better more). Elsewise I do NAT for ULAs instead of NPT.
I added this as you asked for my complete configuration. It should not apply to packages going out through the interface wgcircle and those should receive NAT. As I do not see any adress being process by NPT in my tcpdump I think this is configured correct. Thanks anyway, I appreciate any hints.

So my question is still the same: why is IPv6 not covered by the configuration in my post #1 if I do not configure it explicitly? The screenshot says the opposite.

egc · August 14, 2024, 11:05am

I am not sure what you want to NAT, but it looks like you have two WG interfaces setup as a WG server?

It looks like you want to also handle IPv6 traffic from the connected WG clients

For this you have added IPv6 ULA addresses to the WG clients.
this should be sufficient to handle internal traffic provided you have route allowed IPs enabled (you have) and/or use a /64 mask for the IPv6 list address (you do not but it is highly recommended)

But to let your WG clients have IPv6 internet access you have to NAT66 the WG ULA address going out of the WAN
Masquerading is nothing more than dynamically SNAT on the IP address of the out interface
You can turn on masq6 on the WAN interface but then you are NATting all traffic so better is a selective NAT of only the WG subnet, for my own WG server it looks like:

config nat 'nat6'
	option family 'ipv6'
	option proto 'all'
	option src 'wan'
	option src_ip 'fddb:b40f:f9bc:4ba5::0/64'
	option target 'MASQUERADE'

Note that I use the WAN zone and not the wg_zone, I think using the wg_zone is what makes it unpredictable in your case

See: https://openwrt.org/docs/guide-user/firewall/fw3_configurations/fw3_nat#selective_nat

Note NPT6 is also possible

Note 2: you also need to disable source routing but I saw you already did that

wokkocher · August 14, 2024, 8:19pm

Indeed, I have 2 wireguard instances on the same router. The first one (wgcafe) is in use for roadwarriors like my android phone or my travel router. The second one (wgcircle) is in use to connect to an internal network out of my reach. Therefore I do have only one pair of IPv4 and IPv6 adresses and all traffic to this network must be SNATed. Elsewise the allowed_ips on the other end would not match and all packages would be droped. So I have to SNAT before packages go through the wgcircle interface.

On the other hand those packages would never go through my wan interface. Because of that I still think, NPT for a specific ULA prefix (--> selective) configured for interface wan and should not mix up with NAT/NAT66 for interface wgcircle.

I have a working configuration if I set two separate rules. It is just: I dont want to. Situations like this are interesting. Either I can understand this behavior and learn something new or I can confirm something weird and discover a bug.
To verify my NPT and NAT do not mix up I disabled NPT completely and also removed all other NAT rules. The behavior stays the same.
I could also set up a new small router from scratch and do only selective NAT on wan and dump packages on the other side of the wan and see what happens. If it is the same on a fresh configuartion, this should be considered a bug, am I right? If nobody else has a better idea I will try do this in the next days. Currently I do not have a lot of free time because of my private life...