NAT6 + mwan3: should devices keep leasing GUA?

Hikari · May 5, 2021, 2:07am

Hello friends!

I had made my first attempt to setup NAT6 on my router and it kinda works, but I have a few doubts.

The reason for choosing NAT6 is because both my ISP provide only a single /64 prefix (each) and I'm rly tired of fighting them to properly follow IPv6 standard (which doesn't mean I won't restart the fight in the near future), and because I want mwan3 to properly manage load balancing and priorization on IPv6 as it does on IPv4.

So, plz, avoid posts demanding me to not use NAT6, I had thought about it for over a year and the time for guessing is in the past.

This is a diagram of my LAN

I know NAT6 is working because, before the setup, LixVM was unable to reach Internet with IPv6 due to VMRouterWRT not having any global prefix to lease a GUA to it. LixVM was able to ping LAN. Now it pings Internet too.

This is current settings of RyzenWRT


network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option local_source 'lan'
	option mmx_mask '0x3F00'
	option ula_prefix 'ddfa::/48'

config interface 'lan'
	option type 'bridge'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '192.168.49.1'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ip6ifaceid '::1'

config interface 'want'
	option ifname 'eth3'
	option proto 'pppoe'
	option username 'cliente@cliente'
	option password 'cliente'
	option peerdns '1'
	option dns '208.67.222.222,208.67.220.220'
	option ipv6 '1'
	option delegate '0'
	option metric '10'
	
config interface 'want6'
	option ifname '@want'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option peerdns '0'
	option metric '20'
	option reqprefix '56'

config interface 'wan_mrd'
	option proto 'dhcp'
	option ifname 'eth2'
	option metric '30'
	option peerdns '1'
	option dns '208.67.222.222,208.67.220.220'
	option ipv6 '1'

config interface 'wan_mrd6'
	option proto 'dhcpv6'
	option ifname 'eth2'
	option reqaddress 'try'
	option metric '40'
	option peerdns '0'
	option reqprefix '56'


dhcp

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '0'
	option localservice '0'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '110'
	option limit '60'
	option leasetime '4h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	option ra_slaac '1'
	option ra_default '2'
	option ra_management '1'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	list dhcp_option '6,192.168.49.4'
	list dns 'fdfa::253'		# force to not use
	list dns '192.168.49.4'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'


firewall

config defaults
	option syn_flood	1
	option input		ACCEPT
	option output		ACCEPT
	option forward		REJECT
# Uncomment this line to disable ipv6 rules
#	option disable_ipv6	1

config zone
	option name		lan
	list   network		'lan'
	option input		ACCEPT
	option output		ACCEPT
	option forward		ACCEPT

config zone
	option name		wan
	option input		REJECT
	option output		ACCEPT
	option forward		REJECT
	option masq			1
	option mtu_fix		1
	option masq6			1	# NAT6, disable if not
	option masq6_privacy	1	# NAT6, disable if not
	list   network		'want'
	list   network		'want6'
	list   network		'wan_mrd'
	list   network		'wan_mrd6'

config forwarding
	option src		lan
	option dest		wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
	option name		Allow-DHCP-Renew
	option src		wan
	option proto		udp
	option dest_port	68
	option target		ACCEPT
	option family		ipv4

# Allow IPv4 ping
config rule
	option name		Allow-Ping
	option src		wan
	option proto		icmp
	option icmp_type	echo-request
	option family		ipv4
	option target		ACCEPT

config rule
	option name		Allow-IGMP
	option src		wan
	option proto		igmp
	option family		ipv4
	option target		ACCEPT

# Allow DHCPv6 replies
# see https://dev.openwrt.org/ticket/10381
config rule
	option name		Allow-DHCPv6
	option src		wan
	option proto		udp
	option src_ip		fc00::/6
	option dest_ip		fc00::/6
	option dest_port	546
	option family		ipv6
	option target		ACCEPT

config rule
	option name		Allow-MLD
	option src		wan
	option proto		icmp
	option src_ip		fe80::/10
	list icmp_type		'130/0'
	list icmp_type		'131/0'
	list icmp_type		'132/0'
	list icmp_type		'143/0'
	option family		ipv6
	option target		ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Input
	option src		wan
	option proto	icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	list icmp_type		router-solicitation
	list icmp_type		neighbour-solicitation
	list icmp_type		router-advertisement
	list icmp_type		neighbour-advertisement
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Forward
	option src		wan
	option dest		*
	option proto		icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT
	option enabled		0 # NAT6, reenable if not

config rule
	option name		Allow-IPSec-ESP
	option src		wan
	option dest		lan
	option proto		esp
	option target		ACCEPT

config rule
	option name		Allow-ISAKMP
	option src		wan
	option dest		lan
	option dest_port	500
	option proto		udp
	option target		ACCEPT

# allow interoperability with traceroute classic
# note that traceroute uses a fixed port range, and depends on getting
# back ICMP Unreachables.  if we're operating in DROP mode, it won't
# work so we explicitly REJECT packets on these ports.
config rule
	option name		Support-UDP-Traceroute
	option src		wan
	option dest_port	33434:33689
	option proto		udp
	option family		ipv4
	option target		REJECT
	option enabled		false

config rule
	option target 'ACCEPT'
	option src 'wan'
	option proto 'udp'
	option dest_port '547'
	option name 'Allow DHCPv6 (546-to-547)'
	option family 'ipv6'
	option src_port '546'
 
config rule
        option target 'ACCEPT'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option name 'Allow DHCPv6 (547-to-546)'
        option family 'ipv6'
        option src_port '547'

# include a file with users custom iptables rules
config include
	option path /etc/firewall.user

config include 'nat6'
	option path '/etc/firewall.nat6'
	option reload '1'

Notice I changed ULA prefix from fdfa:: to ddfa::
I had also rebooted both RyzenWRT and VMRouterWRT after the setup, to be sure everything is renewed.

My doubt is that VMRouterWRT keeps receiving ISPs GUA:

# ip -6 route show
default from 2804:14c:xxxx:xxxx::31 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from 2804:14c:xxxx:xxxx::/64 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from 2804:1b2:yyyy:yyyy::31 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from 2804:1b2:yyyy:yyyy::/64 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from ddfa::31 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from ddfa::/64 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
default from ddfa:0:0:4::/62 via fe80::a236:9fff:feaa:4b68 dev eth0 proto static metric 512 pref medium
2804:14c:xxxx:xxxx::/64 dev eth0 proto static metric 256 pref medium
unreachable 2804:14c:xxxx:xxxx::/64 dev lo proto static metric 2147483647 error 4294967183 pref medium
2804:1b2:yyyy:yyyy::/64 dev eth0 proto static metric 256 pref medium
unreachable 2804:1b2:yyyy:yyyy::/64 dev lo proto static metric 2147483647 error 4294967183 pref medium
ddfa::/64 dev eth0 proto static metric 256 pref medium
unreachable ddfa::/64 dev lo proto static metric 2147483647 error 4294967183 pref medium
ddfa:0:0:4::/64 dev br-lan proto static metric 1024 pref medium
unreachable ddfa:0:0:4::/62 dev lo proto static metric 2147483647 error 4294967183 pref medium
ddfb::/64 dev br-lan proto static metric 1024 pref medium
unreachable ddfb::/48 dev lo proto static metric 2147483647 error 4294967183 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium

Notice it's not receiving fdfa:: which means it isn't (shouldn't) still using outdated leases. Same behavior goes for other Lix and Win10 devices, but I haven't rebooted them.

Shouldn't NAT6 disable that and make devices receive only ddfa:: addresses?

At least VMRouterWRT is properly using ddfa:0:0:4::/62 prefix to route IPv6 Internet for LixVM. But if devices receive 3 prefixes addresses, won't they leak mwan3? They could be able to choose for a 2804:: address and reach Internet directly without using mwan3 load balancing.

Another odd behavior: once LixVM had whatismyipaddress.com report its IPv6 addr as 2804:14c:xxxx:xxxx:462:4487:2dff:12. But suffix 12 is Main's suffix. VMRouterWRT's is 31.

Then I tried it again and it reported RyzenWRT's address, outside of its delegated prefix, which means it was using NAT6 this time. Might this be an evidence that NAT6 is working but it's also being leaked?

trendy · May 5, 2021, 10:00am

ddfa is not ULA. ULAs are fc00::/7 and if you want to be RFC compliant, the fd00::/8 is defined for /48.

This is something you should troubleshoot with RyzenWRT.

Not by itself. You should make sure only ULA addresses are distributed. Then you must nat them on the wan router.

Hikari · May 5, 2021, 1:08pm

That's what https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6 says, to replace f for d:

We must also be cautions for devices OSs to keep "choosing" IPv6 over IPv4. If they understand that there's only ULA, won't they give up on IPv6? I really don't know how to configure NAT6 properly.

Well I tried to troubleshoot and got stuck, then I came ask for help to understand why GUA is still being leased.

And how to make sure? Did I miss something on the tutorial?

It'd be great if I could easily set a config to enable and disable the lease of GUA, then I could test both cases.

trendy · May 5, 2021, 3:29pm

It says to do that if you face an issue with clients not using the IPv6 ULA.

The important thing is to have the default route advertised. ULA is by default advertised without the default gateway, if you recall from the other topic. That way it definitely won't use ipv6 to access the internet.

In OpenWrt there is option ip6class for that.

Hikari · May 5, 2021, 3:38pm

So, we must set ra_default to 1 or 2 (1 to advertise default route when the router has a route and 2 to always advertise it)?

And will OSs use ULA instead of IPv4 if it's advertised with a default route?

Let me read about it and I'll edit here.

trendy · May 5, 2021, 3:53pm

1 will do, as you have the default route on the router already.

Well, you'll have to figure that out.
I find it better idea to use in such cases something reserved for documentation, like 2001:db8::/32, what do you think @vgaetera ?

vgaetera · May 6, 2021, 5:34pm

@Hikari, consider to keep the GUA prefixes if you plan to expose any services to public.
This can help you easier reach LAN hosts from WAN without DNAT redirects.

@trendy, there may be operational implications in general case.
Although it currently works well enough with OpenWrt.

trendy · May 7, 2021, 7:47am

My understanding is that it is up to the operators to block it, so it could serve well in this context.

jamesmacwhite · May 7, 2021, 9:38am

Just throwing in as I use this setup currently.

If you have at least one GUA prefix you can still have this delegated across your LAN. The reason for changing the ULA prefix from fc00::/7 is if you don't have any GUA prefix at all. Generally most devices will prefer IPv4 over IPv6 ULA only, so that's why you may want to change it, otherwise you'll find most clients still use IPv4 than IPv6, somewhat negating doing NAT6 to begin with.

As @vgaetera has said, having a GUA prefix also allows you to allow inbound IPv6 traffic in the usual way through the firewall, given NAT6 is mostly around outbound traffic.

Hikari · May 11, 2021, 9:26pm

I'm unable to understand what it means

It seems to be related to restricting which IPv6 prefixes this interface accepts receiving connections.

Thanks. ATM, I can just use IPv4 to reach home, as long as ISPs keep providing valid IPv4 addresses. I also haven't find any noip service that supports IPv6 anyway.

I understand that ULA and GUA are used in parallel, also with link-local. ULA doesn't require authorittive entities to assign a prefix to us and is to connect to LAN and intranet devices, while GUA is used to reach Internet.

If we have a static GUA /56 prefix then ULA loses its use because we can just use the received GUA on our LAN too. In practice, we use the LAN domain hosts which are resolved to addresses.

Hikari · May 11, 2021, 9:53pm

Sorry again for the delay. I have been using for some days NAT6 with dd:: prefix together with both GUA prefix.

According to Pihole, 50% of the queries it receives are answered using its cache, 19% by internet DNS and 10% are LAN domain queries. 50% are IPv4 queries, 31% are IPv6, so over half IP queries are for IPv6. This means that devices have been using IPv6.

Just after enabling NAT6, some myip queries I did were still using GUA PD address, but after that, all the few times I verified I was always getting router WAN addresses. This means that even with GUA enabled, router is translating packages to its WAN addresses.

IDK if devices are using dd:: "ULA" and router is translating that, or they are using GUA and even so router is translating also GUA to WAN address. Is it possible to run curl and choose the route it must use? With that I could make some nice myip tests. Maybe disable routes?

I'd like to disable GUA leasing and use only ULA, both fd:: and dd::, so I could test if it works in these settings too. Anybody knows how to do it? I never had to enable IPv6 leasing so I have no idea how to disable it.

There wasn't any situation that I noticed a failed connection and it was using IPv6. I'm developing a small monitoring system, but I haven't developed a myip for IPv6 because I chose to monitor only GUA PD, because (at the time) each time I tested myip IPv6 I was getting a different interface ID. And when I developed the availability monitor, I developed it only on IPv4 because I understood that availability is related to ISP link and not IP protocol so I considered that IPv4 and IPv6 would be equivalent.

I was clearly using misconceptions. If I disable GUA leasing, PD is irrelevant. And there were a few times I got unavailable connection and mwan3 reported IPv4 interface was down and IPv6 was up. I'm starting to get bored on this so I'll hardly continue the development for now.

It's indeed sad to use NAT6 because it takes down all "awesome" (irony) IPv6 features for LAN and also makes PD irrelevant. But, on the end of the day, IPv6 Internet works the same way, and it's easier to config 1 unique prefix. At least for now I don't need to reach home using IPv6. In fact, enabling NAT6 only on the main router provided IPv6 internet connectivity to my VM networks without having to change anything on their routers, this is incredible!

Another path I could use is to keep GUA together with ULA for main VLAN and use ULA-only for other VLANs. But for that I must make sure that GUA isn't troubling mwan3 to load balance and fail over IPv6.

vgaetera · May 12, 2021, 3:42am

https://openwrt.org/docs/guide-user/network/ipv6/ipv6_extras#disabling_gua_prefix

Hikari · May 15, 2021, 1:28pm

After some time working with no GUA and with ULA under d top prefix, I can confirm it's working nice.

Both Win10 and Ubuntu choose to use IPv6 for Google.com and whatismyip recognizes Win10 IPv6 addr always as router's WAN. LAN domains are also being resolved for IPv6 and working.

I'm having Comodo Firewall blocking ping over IPv6, when it's disabled then ping works. I haven't found how to fix that but it's not troubling for now.

Oddly, my VM OpenWRT is prefering IPv4 for google.com, but ip -6 works, while ping to LAN fails.

I remember that, when I setup NAT6, VMs started working over IPv6, and now it works on Win10 but not on Ubuntu. IDK what broke it. It's unable to reach Internet and LAN over IPv6. Oddly, both Win10 and Ubuntu keep choosing IPv6 for LAN domains, which forces me to use IPv4, but they choose IPv4 for google.com and it works.