Multiwan with IPv6

With the mwan3 package, using IPv6 is not very nice: https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3#ipv6_support

The documented solution requires NAT6 and that requires copy and pasting a script from the wiki (which does not seem to be packaged) which looks like it causes extra work on upgrades.

And then, there is the next problem that the client does not know its public IPv6 address anymore. For me, IPv6 is more important than IPv4.

Providing multiple IPv6 uplinks to a (Linux) client with OpenWrt works out of the box with Router Advertisements which can inform a client about multiple public network prefixes. OpenWrt provides the correct routing out of the box so that the different source addresses are routed to the different uplinks. The only remaining problem is the source address selection algorithm at the clients that mainly uses the longest matching prefix.

To solve this, I am using a custom proxy server running at the client computer (not at the OpenWrt router) that manually selects different public source addresses to use multiple uplinks. One positive aspect of this is that this can be enabled per application.

To see that it works, one needs a speedtest that uses multiple connections at the same time. One speedtest that does this is https://breitbandmessung.de/test

Does anyone use something similar? Are there other concepts for load balancing IPv6 traffic?

Main provider - native ipv6 (w/o NAT6) announced to the clients
Additional providers - NAT6

I have no solution for DHCPv6 but you could use radvd or bird with radv to announce RA to clients. You have to disable RA for Dnsmasq then.

And yes half bakes random scripts in the wiki pages are annoying :confused:

@csharper2005 Do I understand it correctly that your clients know one public address (so they know how they are reachable) and the router does a load balancing for outbound traffic?

Are you using the script from the Wiki for your Nat6?

Yeah, that's right. Script is no longer necessary with OpenWrt 22.03.0. I use masq6 1 option

Instead of masquerade, I was able to rewrite the prefix of the addresses. This must be repeated when the addresses change and I did not automate that yet. Interestingly, I had to restart mwan3 before it worked but I am not yet sure about the cause for this.

ubus call network.interface dump | jq --raw-output '
  def clamp(minValue; maxValue):
    [minValue, .] | max |
    [maxValue, .] | min ;

  def toHexString:
    . as $input |
    if $input < 16 then ("0123456789abcdef" | split("") | .[$input] )
    else ($input / 16 | floor | toHexString) + ($input % 16 | toHexString) end
  ;

  def toSuffixMask:
    . as $maskLength |
    [range(0; 8)] |
    map(
      # calculate remaining zeros
      $maskLength - . * 16 |
      clamp(0; 16) |
      # calculate set bits
      16 - . |
      # generate mask part
      pow(2; .) - 1 |
      toHexString
    ) |
    join(":");

  .interface |
  map({
    name: .device,
    prefixList:
      (."ipv6-prefix") |
      map({
        address: .address,
        mask: .mask
      })
  }) |
  map(select((.prefixList | length) == 1)) |
  map({ name: .name } + (.prefixList | .[])) |
  map(
     "    oifname \"\(.name)\" snat ip6 to ip6 saddr and \(.mask | toSuffixMask) or \(.address)"
  ) |
  join("\n") |
  (
    "delete table inet ip6mangle;\n\n" +
    "table inet ip6mangle {\n" +
    "  chain srcnat {\n" +
    "    type nat hook postrouting priority srcnat; policy accept;\n" +
    . + "\n" +
    "  }\n" +
    "}"
  )
' | nft -f /dev/stdin

There is only one sane, non-hacky solution for multihoming with IPv6.
Get your own ASN and GUA IPv6 prefix, announce it to multiple transit providers over BGP, balance it the way you want.
There are costs, but these days it can be done for like 15$ per year, if you are OK with using tunnels for IPv6 connectivity.

Is this possible for inbound traffic, too?

This sounds like increased latency.

AS-prepends, BGP communities... takes some tweaking.
https://www.reddit.com/r/networking/comments/w84kg2/bgp_load_sharingegress_balancing/
Me personally, I only BGP multihome for reliability reasons, active-passive configuration

Perhaps your ISPs can provide you with BGP sessions (not very likely for consumer plans though)
Tunnels in your country can provide with a low latency as well.
If you are in Germany, test latency to these servers.
image

This does not sound like something that works out of the box without trouble

Does this need BGP at all? Outages are very uncommon and I would have no problem with a IP address change in this case.

With tunneling, I could rent a server and use a VPN to use the public addresses of the server. This avoids getting a ASN and using BGP. The round trip time to one of the servers I rented is comparable to one of the mentioned servers (16 ms).

While I think that playing with BGP would be good for learning purposes, I don't think that it is a good fit for multiwan using consumer internet connections.

I am experiencing a problem with mwan3 and I think that it has problems if uplink IPs change while the interface is up. From my testing, it looks like it's enough to run mwan3 ifup [interface] to make it detect the new IP although this triggers some error messages.

This is what I'm using now:

mkdir -p /usr/share/ipv6-masq
cat > /usr/share/ipv6-masq/update-rules.sh <<"EOF"
#! /bin/sh
ubus call network.interface dump | jq --raw-output '
  def clamp(minValue; maxValue):
    [minValue, .] | max |
    [maxValue, .] | min ;

  def toHexString:
    . as $input |
    if $input < 16 then ("0123456789abcdef" | split("") | .[$input] )
    else ($input / 16 | floor | toHexString) + ($input % 16 | toHexString) end
  ;

  def toSuffixMask:
    . as $maskLength |
    [range(0; 8)] |
    map(
      # calculate remaining zeros
      $maskLength - . * 16 |
      clamp(0; 16) |
      # calculate set bits
      16 - . |
      # generate mask part
      pow(2; .) - 1 |
      toHexString
    ) |
    join(":");

  .interface |
  map({
    name: .device,
    prefixList:
      (."ipv6-prefix") |
      (if . == null then [] else . end) | # this list is sometimes null
      map({
        address: .address,
        mask: .mask,
        addressWithMask: "\(.address)/\(.mask)"
      })
  }) |
  (map(.prefixList) | flatten | map(.addressWithMask)) as $prefixList |
  map(select((.prefixList | length) == 1)) |
  map({ name: .name } + (.prefixList | .[])) |
  map(
    (
      $prefixList - [.addressWithMask] |
      # skip rules if there are no other addresses to rewrite
      # this happens if there is only one bound interface
      select((. | length) > 0) |
      "{ " + join(", ") + " }"
    ) as $saddrList |
    "    ip6 saddr \($saddrList) oifname \"\(.name)\" snat ip6 to ip6 saddr and \(.mask | toSuffixMask) or \(.address)"
  ) |
  join("\n") |
  (
    "# make sure that the empty table exists\n" + 
    "table inet ip6mangle {\n" +
    "}\n" +
    "\n" +
    "delete table inet ip6mangle;\n\n" +
    "table inet ip6mangle {\n" +
    "  chain srcnat {\n" +
    "    type nat hook postrouting priority srcnat; policy accept;\n" +
    . + "\n" +
    "  }\n" +
    "}"
  )
' | nft -f /dev/stdin
EOF
chmod u+x /usr/share/ipv6-masq/update-rules.sh
cat > /etc/hotplug.d/iface/21-ipv6-masq <<"EOF"
/usr/share/ipv6-masq/update-rules.sh
EOF
cat > /etc/odhcp6c.user.d/mwan-ipv6-masq <<"EOF"
/usr/share/ipv6-masq/update-rules.sh
EOF

This automatically adjusts the generated firewall rules when the addresses change. Moreover, it handles some cases that occurred during testing like null instead of a empty prefix list and only one uplink that needs no rules instead of rules with an empty source address filter.

I'm using that now and it works reliable.

I'm not sure if that prefix rewriting would be a useful feature for the fw4.