IPv6 WAN fail-over without IPv6 NAT

Feature Requests
1

The current recommendation for WAN fail-over is to use mwan3. However, for IPv6, it needs NAT or NPT. This is understandable: mwan3 deals not only with fail-over, but also with load-balancing, and for load-balancing the decision regarding the choice of the source address must be delegated to the router on a per-connection basis, and therefore the originating host in the LAN must use a dummy temporary source address that will be replaced as needed.

However, none of that should be necessary if there is no policy routing involved, i.e. if either all the traffic goes out through WANA or all traffic goes through WANB. Indeed, the prefix delegated from either WANA or from WANB should be directly announced on the LAN, and reannounced with zero validity period during the fail-over if that WAN breaks.

The current "solution" by OpenWrt is to announce all delegated prefixes on the LAN, which also does not make sense in the context of the fail-over to an expensive or slow connection.

I admit that I cannot formulate the feature request more precisely, but the end goal is: simplify the IPv6 WAN fail-over for those who don't care about load balancing, and do not require IPv6 NAT in this setup.

What exactly needs to be done for this is unfortunately unknown. I think of these factors:

  • Periodic IPv6 connectivity checking as a core feature for all interfaces that have a route to ::/0
  • Not announcing delegated prefixes for interfaces that are "up" on paper but don't actually work
  • Not announcing delegated prefixes for interfaces that are "up" but should not be used (e.g. the LTE prefix when the fiber connection is up)
1 Like
2

What about sending 2 advertisements, one for each prefix, with a different preference?

3

I don't know whether all operating systems use the preference in the source selection process. What would be the advantage of announcing this extra useless prefix information? Besides, that still leaves us with the remaining parts - namely, periodic checks, and somehow making sure that prefixes that belong to a WAN that went down or stopped working are immediately depreferred.

4

Well, you asked for ideas, so there was one. :slight_smile:
If the OS doesn't use the preference in the source selection process, NAT6 is the only way.
The remaining parts are simply some checking mechanism, i.e ping, dns query, http get, to verify the availability of the uplink and adjust the RA.

5

Well, I still disagree that NAT6 is the only way. If there is nothing except one option to choose from (and that's the use case), then it will be chosen.

6

Let's wait for some other suggestions then.

7

I have tried (in GNS3) the suggestion to use two router advertisements. To do that, I set up a virtual OpenWrt router with two WANs and one LAN, and some other OpenWrt routers upstream, that would announce different prefixes.

As it turns out, odhcpd does not support sending two router advertisements. Another available tool that supports sending router announcements is bird2, but I also did not have any success asking it to send two router announcements (one medium priority and one high), each containing one prefix - except by running two instances manually, with different configuration files. Of course, when trying bird2, I have set RA and dhcpv6 to disabled for odhcpd.

As a client, Puppy Linux was used, with modprobe ipv6 being the only command to configure IPv6. I.e., just the kernel default SLAAC client.

Result: I was able to catch the situation where, after nothing but a reboot and modprobe ipv6, it was consistently using the source address coming from a prefix announced with the medium priority. So, the announcement of the unneeded prefix is actively harmful, and this experiment just confirms my viewpoint: only one prefix, the one that is supposed to be used by the clients at each particular moment, should be announced with non-zero lifetime.

2 Likes
8

I'm speaking from ignorance but can't this be done using cake-autorate stall detection to mark interface down on stall plus netifd wan failover pbr?

9

I think I don't fully understand your proposal. Namely, the word "mark" in "mark interface down". Obviously you don't mean ifdown $interface, because this would leave it with zero chances to recover. So we would need to somehow make sure that the non-working WAN interface is up (unless it went down by itself) but not used for routing of anything except liveness probes. For IPv4, this means setting the metric on the default route high enough so that it is not used unless a process binds to that specific interface. For IPv6, it means just not sending any packets with the source IPs associated with this interface, or from prefixes delegated through there. And the easiest way to do this would be to stop announcing these unneeded prefixes on the LANs.

Regarding your reference to pbr, I suppose you are referring to the "Route LAN to VPN with failover to WAN" example - I tried to understand it, but without context in the form of routes (not rules) it doesn't make much sense. It is only telling the router to use a separate routing table for packets entering from the LAN.

But why would we need that? The whole idea is that, in the scenario under discussion, all packets except the liveness probes live by the same rules, namely, "at each moment, only one WAN interface is relevant". For IPv6, in the default OpenWrt setup, all WAN interfaces are relevant, because all their delegated prefixes are announced on the LAN - and that's what this misfeature-removal request is about.

But even the liveness probes for the non-default interface could be managed with a single routing table (IPv4: set up different metrics and bind to the correct interface, IPv6: bind to the correct source IP and then let the default source-based routing in OpenWrt do its job).

As for the liveness probes (and thus the reference to the cake-autorate code) - well, simple pings of multiple targets over the interface, with a timeout, are sufficient. mwan3 contains 99% of the correct logic - if for X targets Y last pings failed, declare the interface as non-working. The only flaw there (not relevant for my setup) is the reliance on the static target IPs, no way to say "gateway" or "dns" - and the old "multiwan" package had even that.

10

Yeah I meant ifdown then ifup. And indeed the netifd pbr like for VPN failover except in this case failover to LTE on failing FIBRE. Isn't there a way that could work?

When I used VPN I had this all set such that when VPN went down everything got routed through to WAN directly. So the principle seems very similar.

11
12

Do you mind to share the bird conf you have tested?

13

You are missing a very important point: the correct fail-over setup, with PBR or not, functions very differently for IPv6 and IPv4. That's why two topics (this one for IPv6 and Connectivity checking and simple fail-over as core features for IPv4). And by the way, all IPv6 examples on https://openwrt.org/docs/guide-user/network/routing/examples/pbr_netifd look wrong, because the source IP question is essential for IPv6 (but taken care of automatically for IPv4) and is not even commented upon.

Let me explain the difference in more detail.

IPv4 is, realistically, always NATed in home and small-business networks. Hosts in LAN use their private IP address as a source to initiate connections to the outside world. The router is then responsible for making the correct routing decision, and for replacing this dummy internal IP with the IP address of the outgoing interface. That's where you can plug routes with different metrics, or, for complex setups where the routing decision depends on factors other than the destination IP and the set of interfaces that are up, policy-based routing. But both forum topics are about fail-over, and therefore exclude such complex setups. The linked thread (Set different Upstream for specific source IP) specifically asks for routing based on the (private) source IP, i.e. for simultaneous non-zero traffic on both WAN connections, and is therefore out of scope. For that task, policy-based routing is needed. For pure fail-over, it's overkill.

For IPv6, with multiple WANs, there are three possible setups.

First, let's consider the default OpenWRT IPv6 setup without any extra packages or configuration, which does not provide fail-over, and for me it is unclear where such "lump everything together" semantics would be useful at all. In the IPv6 world, all WANs provide delegated prefixes. OpenWrt creates router advertisement packets that tell the hosts in LAN all WAN delegated prefixes and the ULA prefix, and therefore these hosts will assign themselves one IP from each range. OpenWrt will then apply its default policy-based routing magic to look at the source IP of each packet, and route it into the corresponding WAN. The problem is, if both the fiber and the LTE are up, then hosts in LAN have absolutely no idea which of the source addresses to prefer - and that, unlike with IPv4, is their responsibility. So the worst-case result is: I can ping my hosts from some VPS, both on the address from the LTE range, and from the range assigned by the fiber ISP, but due to bad luck, they always use LTE for outgoing connections. For them, it's just 2405:8d40:c81:81c2::x vs 2001:fd8:f09f:c24b::x. Can you tell (without invoking whois) which of these addresses is better? No, and there is no knob to tell them about that. And that's why the forum link that you provided (Set different Upstream for specific source IP) is irrelevant: it is for the IPv4 case, where such choice is done on the router.

For IPv6, the second possible setup (the current recommended fail-over setup with mwan3) requires IPv6 NAT or NPT, to isolate the hosts in LAN from the process of choosing the correct source address. In this setup, hosts in LAN have only one IPv6 address, from the ULA range. The router then applies the link-selection logic (i.e. routing), and translates the address into the globally-routable space suitable for the chosen link. It works, but in communities outside of this forum (e.g. Hacker News) this gets a fair "yuck" type of response just because of the NAT, and also comes with downsides such as these hosts preferring IPv4 when communicating with dual-stack servers (because IPv4 is preferable to ULA according to RFC 3484). Note that this yucky setup is inevitable if two WANs are to be used simultaneously, e.g. via load-balancing or "normal" destination-based routing.

The third setup, which is the subject of this feature request, is currently not implemented. It is identical to the first (default) setup, with the only difference being that prefixes for the irrelevant WANs (backup WAN if the primary WAN works, or the primary WAN if it only pretends to be up) are not included in the router advertisement packets. There is no NAT. And, because hosts in LAN have only one non-ULA IPv6 address, they never make the wrong choice of the source address. Then the default source-based routing magic in OpenWrt routes these packets to the correct link.

14

Sure. Here is /etc/bird.conf:

log syslog all;
debug protocols all;

router id 192.168.1.1;

ipv6 table radv_routes_a;

protocol device {
        scan time 10;           # Scan the interfaces often
};

protocol static {
    ipv6 { table radv_routes_a; };
    route ::/0 unreachable;
}

protocol radv {
    propagate routes yes;
    ipv6 { table radv_routes_a; export all; };
    interface "br-lan" {
        prefix 2001:db8:0:6::/64 {skip no; };
        prefix ::/0 { skip yes; };
    };
};

The /etc/bird_b.conf file is:

log syslog all;
debug protocols all;

router id 192.168.1.2;

ipv6 table radv_routes_a;

protocol device {
        scan time 10;           # Scan the interfaces often
};

protocol static {
    ipv6 { table radv_routes_a; };
    route ::/0 unreachable;
}

protocol radv {
    propagate routes yes;
    ipv6 { table radv_routes_a; export all; };
    interface "br-lan" {
        prefix 2001:db8:0:a::/64 {skip no; };
        prefix ::/0 { skip yes; };
        default preference high;
        route preference high;
    };
};

The processes (both started manually) are:

 3399 root      2104 S    /usr/sbin/bird -c /etc/bird.conf -P /var/run/bird.pid
 3517 root      2088 S    /usr/sbin/bird -c /etc/bird_b.conf -P /var/run/bird_b.pid -s /var/run/bird_b.ctl

OpenWrt has been reconfigured to assign only a /64 of the public prefixes to the LAN.

15

Do I understand you correct that you had no luck define both prefixes in the same radv proto instance? I'm not quiet sure if I understand why you used two bird instances...

16

I don't get why the netifd pbr failover approach can't be made to work with at least the second option. Seems to make sense to me. Having one or another route in dependence upon connection specifics implies gateway doesn't it? Packet goes from LAN device to router and router decides what to do in dependence upon the upstream connections.

17

It's not that I had no luck, it's impossible. One radv section generates one router advertisement packet, big or small. The router preference field is global to the router advertisement packet, so there can be only one. So, to have different router preferences, we need two packets - which means two radv sections. And this only works with two bird processes.

18

With the second option (i.e. with a NAT) it might be possible to make netfd-based pbr work. But look at the title: "without IPv6 NAT".

19

But why? What's the benefit?

20

IPv6 NAT is always bad. It is a non-standard setup, to which some applications have not adapted - e.g. some SIP phones do not know that they have to use STUN to get their public IP, even on IPv6. Also, with this setup, the hosts will have one IPv4 address and one IPv6 ULA address, thus preferring IPv4 for external communications.

1 Like