6in4 and ISP IPv6 with mwan3 load balancing not working

[cliffordanup]

I have two ISP's configured in mwan3 for load balancing as wan and wanb interfaces for IPv4. For IPv6, only one of my ISP's is even giving me an IPv6 address and that is my wanb ISP with /64 Prefix.

I created a new 6in4 tunnel and attached it to my wan6 interface from the first ISP who do not give me any IPv6 address. The 6in4 tunnel says up, but it shows 0 packets received and several packets sent.

Since the 6in4 tunnel is unable to handle IPv6 traffic as expected, I am now using only the wanb6 interface in mwan3 configuration for IPv6 traffic. The intention however is to have wan6 (6in4) and wanb6 (ISP IPv6 /64 Prefix) interfaces in load balancing configuration in mwan3 for IPv6 traffic.

Please help me first by understanding why my 6in4 tunnel is not able to receive any incoming traffic and then I believe I should be able to get it added in load balancing configuration for IPv6 Traffic in mwan3.

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan.10'
        option proto 'static'
        option ipaddr '192.168.1.100'
        option netmask '255.255.255.0'
        option delegate '0'
        option ip6assign '64'
        option ip6ifaceid '::1'
        option defaultroute '0'
        option ip6hint 'a'

config interface 'wan'
        option device 'wan'
        option proto 'static'
        option ipaddr 'x.x.x.x'
        option netmask 'x.x.x.x'
        option gateway 'x.x.x.x'
        option metric '1'
        option delegate '0'

config device
        option type '8021q'
        option ifname 'br-lan'
        option vid '10'
        option name 'br-lan.10'

config device
        option type '8021q'
        option ifname 'br-lan'
        option vid '20'
        option name 'br-lan.20'

config device
        option type '8021q'
        option ifname 'br-lan'
        option vid '30'
        option name 'br-lan.30'

config bridge-vlan
        option device 'br-lan'
        option vlan '10'
        list ports 'lan4:u*'

config bridge-vlan
        option device 'br-lan'
        option vlan '20'
        list ports 'lan2:u*'

config bridge-vlan
        option device 'br-lan'
        option vlan '30'
        list ports 'lan3:u*'

config interface 'guest'
        option proto 'static'
        option device 'br-lan.20'
        option ipaddr '192.168.10.1'
        option netmask '255.255.255.0'
        option defaultroute '0'
        option delegate '0'

config interface 'vpn'
        option proto 'static'
        option device 'br-lan.30'
        option ipaddr '192.168.20.1'
        option netmask '255.255.255.0'
        option defaultroute '0'
        option delegate '0'

config interface 'openvpnclient'
        option proto 'none'
        option device 'tun0'
        option metric '5'

config interface 'wanb'
        option proto 'pppoe'
        option device 'lan1'
        option username '1234567890'
        option password '1234567890'
        option ipv6 '0'
        option metric '2'

config interface 'wanb6'
        option proto 'dhcpv6'
        option device '@wanb'
        option reqaddress 'try'
        option reqprefix 'auto'
        option metric '4'

config interface 'wan6'
        option proto '6in4'
        option peeraddr '216.218.221.6'
        option ip6addr '2001:470:18:ef1::2/64'
        list ip6prefix '2001:470:faea::/48'
        option tunnelid 'abcdef'
        option username 'abcdef'
        option password 'abcdef'
        option mtu '1480'
        option metric '3'

root@OpenWrt:~# 

root@OpenWrt:~# cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'openvpnclient'
        list network 'wanb'
        list network 'wanb6'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'guest'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'guest'

config zone
        option name 'vpn'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'vpn'

config forwarding
        option src 'lan'
        option dest 'guest'

config forwarding
        option src 'guest'
        option dest 'wan'

config forwarding
        option src 'vpn'
        option dest 'wan'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'Xbox_UDP_4500'
        list proto 'udp'
        option src 'wan'
        option src_dport '4500'
        option dest_ip '192.168.10.254'
        option dest_port '4500'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'Xbox_UDP_3544'
        list proto 'udp'
        option src 'wan'
        option src_dport '3544'
        option dest_ip '192.168.10.254'
        option dest_port '3544'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'Xbox_UDP_500'
        list proto 'udp'
        option src 'wan'
        option src_dport '500'
        option dest_ip '192.168.10.254'
        option dest_port '500'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'Xbox_TCP_88'
        list proto 'tcp'
        option src 'wan'
        option src_dport '88'
        option dest_ip '192.168.10.254'
        option dest_port '88'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'Xbox_All_3074'
        option src 'wan'
        option src_dport '3074'
        option dest_ip '192.168.10.254'
        option dest_port '3074'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'RDP'
        list proto 'tcp'
        option src 'wan'
        option src_dport '3389'
        option dest_ip '192.168.1.110'
        option dest_port '3389'

config rule
        option name 'Allow-DHCP-DNS-Guest'
        option src 'guest'
        option dest_port '53 67 68'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCP-DNS-VPN'
        option src 'vpn'
        option dest_port '53 67 68'
        option target 'ACCEPT'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/firewall.include'

config rule
        option name 'Allow-protocol-41'
        option src 'wan'
        option proto '41'
        option target 'ACCEPT'

config rule
        option name 'Allow-protocol-59'
        option src 'wan'
        option proto '59'
        option target 'ACCEPT'
        option extra '-m length --length 40'

root@OpenWrt:~# 

root@OpenWrt:~# cat /etc/config/mwan3

config globals 'globals'
        option mmx_mask '0x3F00'

config interface 'wan'
        option enabled '1'
        list track_ip '1.0.0.1'
        list track_ip '1.1.1.1'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '2'

config interface 'wan6'
        option enabled '1'
        list track_ip '2606:4700:4700::1001'
        list track_ip '2606:4700:4700::1111'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '2'
        option initial_state 'online'
        option track_method 'ping'
        option count '1'
        option size '56'
        option max_ttl '60'
        option timeout '4'
        option interval '10'
        option failure_interval '5'
        option recovery_interval '5'
        option down '5'
        option up '5'

config interface 'wanb'
        option enabled '1'
        list track_ip '1.0.0.1'
        list track_ip '1.1.1.1'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option family 'ipv4'
        option reliability '2'
        option initial_state 'online'
        option track_method 'ping'
        option count '1'
        option size '56'
        option max_ttl '60'
        option timeout '4'
        option interval '10'
        option failure_interval '5'
        option recovery_interval '5'
        option down '5'
        option up '5'

config interface 'wanb6'
        option enabled '1'
        list track_ip '2606:4700:4700::1001'
        list track_ip '2606:4700:4700::1111'
        list track_ip '2620:0:ccd::2'
        list track_ip '2620:0:ccc::2'
        option family 'ipv6'
        option reliability '2'
        option initial_state 'online'
        option track_method 'ping'
        option count '1'
        option size '56'
        option max_ttl '60'
        option timeout '4'
        option interval '10'
        option failure_interval '5'
        option recovery_interval '5'
        option down '5'
        option up '5'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config member 'wanb_m1_w2'
        option interface 'wanb'
        option metric '1'
        option weight '2'

config member 'wanb_m1_w3'
        option interface 'wanb'
        option metric '1'
        option weight '3'

config member 'wanb_m2_w2'
        option interface 'wanb'
        option metric '2'
        option weight '2'

config member 'wan6_m1_w3'
        option interface 'wan6'
        option metric '1'
        option weight '3'

config member 'wan6_m2_w3'
        option interface 'wan6'
        option metric '2'
        option weight '3'

config member 'wanb6_m1_w2'
        option interface 'wanb6'
        option metric '1'
        option weight '2'

config member 'wanb6_m1_w3'
        option interface 'wanb6'
        option metric '1'
        option weight '3'

config member 'wanb6_m2_w2'
        option interface 'wanb6'
        option metric '2'
        option weight '2'

config policy 'wan_only'
        list use_member 'wan_m1_w3'
        list use_member 'wan6_m1_w3'

config policy 'wanb_only'
        list use_member 'wanb_m1_w2'
        list use_member 'wanb6_m1_w2'

config policy 'balanced'
        option last_resort 'default'
        list use_member 'wan_m1_w3'
        list use_member 'wanb_m1_w3'

config policy 'wan_wanb'
        list use_member 'wan_m1_w3'
        list use_member 'wanb_m2_w2'
        list use_member 'wan6_m1_w3'
        list use_member 'wanb6_m2_w2'

config policy 'wanb_wan'
        list use_member 'wan_m2_w3'
        list use_member 'wanb_m1_w2'
        list use_member 'wan6_m2_w3'
        list use_member 'wanb6_m1_w2'

config rule 'vpn_rule'
        option family 'ipv4'
        option proto 'all'
        option src_ip '192.168.20.0/24'
        option dest_ip '0.0.0.0/0'
        option sticky '0'
        option use_policy 'openvpn_only'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'balanced'
        option family 'ipv4'

config rule 'default_rule_v4'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'
        option family 'ipv4'
        option proto 'all'
        option sticky '0'

config policy 'wanb6_only'
        list use_member 'wanb6_m1_w3'
        option last_resort 'default'

config rule 'default_rule_v6'
        option family 'ipv6'
        option proto 'all'
        option sticky '0'
        option use_policy 'wanb6_only'
        option dest_ip '::/0'

config interface 'openvpnclient'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        list track_ip '10.100.0.1'
        list track_ip '1.1.1.1'
        list track_ip '8.8.8.8'
        option track_method 'ping'
        option reliability '2'
        option count '1'
        option size '56'
        option max_ttl '60'
        option timeout '4'
        option interval '10'
        option failure_interval '5'
        option recovery_interval '5'
        option down '5'
        option up '5'

config member 'openvpnclient_m1_w3'
        option interface 'openvpnclient'
        option metric '1'
        option weight '3'

config policy 'openvpn_only'
        list use_member 'openvpnclient_m1_w3'
        option last_resort 'blackhole'

config policy 'wan6_only'
        list use_member 'wan6_m1_w3'
        option last_resort 'default'

config policy 'ipv6_only'
        list use_member 'wan6_m1_w3'
        list use_member 'wanb6_m1_w3'
        option last_resort 'default'

root@OpenWrt:~# 

[patrakov]

mwan3 does not really support IPv6. Well, it does, but it requires a non-standard setup with IPv6 NPT, which is not supported by OpenWrt.

You will need a custom script that sets up NPT for you. Devices in LAN will no longer have public IPv6 addresses, only ULAs, and OpenWrt will translate the prefixes. This is inevitable, as the end hosts have no way to know which source IP to use, so that have to use a temporary one, and the party which knows the correct source IP (OpenWrt) will substitute it.

Here is the script. Save it as /etc/firewall.nat6 and make executable:

#!/bin/sh

source /lib/functions/network.sh

# IPv6 NPT (horrible workaround for the lack of such functionality in OpenWrt)

ip6tables -t nat -F PREROUTING
ip6tables -t nat -F POSTROUTING
ULA=$(uci get network.globals.ula_prefix)
for IFACE in $(uci show mwan3 | sed -n '/=interface/s/^mwan3\.\(.*\)=interface/\1/ p') ; do
  network_get_device DEVICE "$IFACE" || continue
  network_get_prefix6 PREFIX "$IFACE" || continue
  BASE_IFACE=${IFACE%_4}
  BASE_IFACE=${BASE_IFACE%_6}
  network_get_metric METRIC "$BASE_IFACE"
  BITS=${PREFIX##*/}
  if [ "$BITS" -le 48 ] ; then BITS=48 ; fi
  ULA_PART=${ULA%%/*}/$BITS
  FIRST_IP=${PREFIX%%:/*}:1
  echo "Adding a fallback route for $IFACE"
  ADJUSTED_METRIC=$(( METRIC + 20000 ))
  EXISTING_FALLBACK=$(ip -6 route list default from :: dev "$DEVICE" metric "$ADJUSTED_METRIC" )
  if [ -n "$EXISTING_FALLBACK" ] ; then
    echo "$EXISTING_FALLBACK" | sed 's/^/ip route del /' | sh
  fi
  WANTED_ROUTE=$(ip -6 route list default dev "$DEVICE" metric 512 | head -n 1 | sed -e 's/from [0-9a-f:/]* //' -e 's/metric 512 //')
  if [ -n "$WANTED_ROUTE" ] ; then
    ip route add $WANTED_ROUTE dev $DEVICE from :: metric "$ADJUSTED_METRIC"
  fi
  if ip route get "$FIRST_IP" 2>/dev/null | grep "dev lo" >/dev/null ; then
    echo "Interface $IFACE (on $DEVICE) does not seem to need NPTv6"
    continue
  fi
  echo "Mapping $ULA_PART <-> $PREFIX for $IFACE (on $DEVICE)"
  ip6tables -t nat -A PREROUTING -d $FIRST_IP -j REDIRECT
  ip6tables -t nat -A PREROUTING -d $PREFIX -j NETMAP --to $ULA_PART
  ip6tables -t nat -A POSTROUTING -s $ULA_PART -m conntrack --ctorigdst $PREFIX -j NETMAP --to $PREFIX
  ip6tables -t nat -A POSTROUTING -s $ULA_PART -o $DEVICE -j NETMAP --to $PREFIX
  ip6tables -t nat -A POSTROUTING -s $ULA -o $DEVICE -j MASQUERADE
done

Note: this script requires that each IPv6 WAN has source-based routing, which is false by default for 6in4 tunnels with static IP addresses. Just add this to the tunnel interface configuration:

    option sourcefilter '1'

You would also need the iptables-mod-nat-extra package due to NETMAP.

What this script does is attempt to translate the available public prefixes 1:1 to the beginning of the ULA range, and do a horrible NAT on addresses outside of the mapped area. Also, it helps OpenWrt with selection of the default IPv6 route in situations where source-based routing creates a chicken-and-egg situation ("I cannot choose a source IP, because that would require a valid route to the target, while all routes require that the specified source IP matches").

Then, in /etc/config/firewall, add this:

config include
	option path '/etc/firewall.nat6'

Then, in /etc/config/network, add this to each LAN so that public prefixes are suppressed (because the end devices don't have enough information to select the correct source IP otherwise):

config interface 'lan'
	list ip6class 'local'

In /etc/config/mwan3, create rules and policies as usual (the example below is copy-pasted from my home router, minus the parts beyond the basics):

config globals 'globals'
	option mmx_mask '0x3F00'
	option logging '1'
	option loglevel 'notice'
	list rt_table_lookup '220'

config interface 'wan'
	option enabled '1'
	list track_ip '8.8.4.4'
	list track_ip '8.8.8.8'
	option family 'ipv4'
	option initial_state 'online'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '2'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'

config interface 'wan_6'
	option enabled '1'
	list track_ip '2606:4700:4700::1001'
	list track_ip '2606:4700:4700::1111'
	option family 'ipv6'
	option initial_state 'online'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '2'
	list flush_conntrack 'connected'
	list flush_conntrack 'disconnected'

config interface 'lte_4'
	option enabled '1'
	option initial_state 'offline'
	option family 'ipv4'
	list track_ip '8.8.8.8'
	list track_ip '8.8.4.4'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '2'

config interface 'lte_6'
	option enabled '1'
	option initial_state 'offline'
	option family 'ipv6'
	list track_ip '2606:4700:4700::1001'
	list track_ip '2606:4700:4700::1111'
	option track_method 'ping'
	option reliability '1'
	option count '1'
	option size '56'
	option max_ttl '60'
	option timeout '4'
	option interval '10'
	option failure_interval '5'
	option recovery_interval '5'
	option down '5'
	option up '2'

config rule 'pfx_deleg'
	option family 'ipv6'
	option proto 'all'
        # Substitute with your own ULA prefix
	option dest_ip 'fd7b:7d49:75e3::/48'
	option sticky '0'
	option use_policy 'default'

config rule 'default_rule_v4'
	option dest_ip '0.0.0.0/0'
	option family 'ipv4'
	option proto 'all'
	option sticky '0'
	option use_policy 'failover'

config rule 'default_rule_v6'
	option dest_ip '::/0'
	option family 'ipv6'
	option proto 'all'
	option sticky '0'
	option use_policy 'failover'

config member 'wan_m80'
	option interface 'wan'
	option metric '80'

config member 'wan_6_m80'
	option interface 'wan_6'
	option metric '80'

config member 'lte_4_m90'
	option interface 'lte_4'
	option metric '90'

config member 'lte_6_m90'
	option interface 'lte_6'
	option metric '90'

config policy 'failover'
	option last_resort 'unreachable'
	list use_member 'wan_m80'
	list use_member 'wan_6_m80'
	list use_member 'lte_4_m90'
	list use_member 'lte_6_m90'

config policy 'wan_only'
	list use_member 'wan_m80'
	list use_member 'wan_6_m80'
	option last_resort 'unreachable'

config policy 'lte_only'
	option last_resort 'unreachable'
	list use_member 'lte_4_m90'
	list use_member 'lte_6_m90'

And not, this is not how it is supposed to work from the user experience standpoint. I started a huge thread about doing IPv6 fail-over without NPT, and @justinappler even identified a very good RFC (https://datatracker.ietf.org/doc/rfc8475/) that explains how it should be done. Yet, this is not how mwan3 works. I think they have painted themselves into a corner by committing to support the situation that different uplinks could be chosen based on the destination IP - in which case that RFC is inapplicable.

---

Or, I have a better idea: just switch off IPv6. It is not ready in the multi-WAN case, and security auditors will love you. Besides, a /64 is not enough for anything except the simplest possible home network.

[lleachii]

You know in recent versions (i.e. those running nft), you can do a masq6 config in the web GUI.

I realized your post was 5 months prior to mine.

[patrakov]

Yes, and I also know that it does not do the same as my script. That's why I don't use this option. There was a feature request to implement this natively, but it went nowhere because we could not agree on the new options capable of expressing it.

What the masq6 option does: translate all addresses for outgoing IPv6 traffic to a single one, drop unsolicited incoming traffic.

What my script does:

• Translate the outgoing IPv6 packet addresses from the ULA range to the corresponding IPv6 address from the public GUA range (delegated prefix), changing only the prefix and keeping the suffix, assuming that it fits (i.e., that the global prefix is large enough)
• Translate the outgoing IPv6 packet addresses that are "too far" in the ULA range (e.g., the guest WiFi if your ISP only assigns /64) to the public IPv6 of the outgoing interface
• Translate the unsolicited incoming traffic from the public GUA range (delegated prefix) to the ULA range, keeping the suffix

This preserves the reachability of internal devices via both uplinks and isolates them from the upstream prefix changes during the fail-over.