Multi-homed IPv6 with failover but with a twist

Installing and Using OpenWrt
1

TL;DR:

Cannot find a way to multi-home my two IPv6 connections because Comcast is using DHCPv6 PD and T-Mobile requires relaying NDP to/from its router's own servers and won't give out prefixes. If I can make dual homing work, I'd also like to set priorities so clients use Comcast unless it's unavailable.

Long version with explanations:

So apologies if this has been asked before. I'm seeing recommendations under "Your topic is similar to" that are not quite what I'm asking about, but:

I have a main (Comcast) Internet connection, with a back-up service from T-Mobile (Home Internet backup)

For IPv4, setting up failover was (relatively) easy, mwan3 does it, well, it's NAT, so relatively straightforward.

For IPv6, I have everything set up for Comcast, because that's my primary connection. I have not yet set up IPv6 for T-Mobile.

Obviously I'd like IPv6 to failover too. But there's at least two issues.

1. Best Practices

Issue one is what's the right approach (dual homed IPv6 I mean)? From what I'm reading about Route Advertisements, you can set priorities so presumably (if it worked) I could let every client have both a Comcast and T-Mobile prefix, give the higher priority to Comcast, and that should work. I've also heard of what seems like an uglier solution involving NPT which presumably mwan3 could handle, which might be the solution if the priorities thing is a no-go.

2. Setting up two incompatible IPv6 implementations

Issue two (the twist) is can these two networks co-exist?

  • Comcast does IPv6 properly, it gives me prefixes via DHCP6 and OpenWRT handles that without issues, with OpenWRT handling prefixes.
  • T-Mobile's gateway OTOH does not give out any prefixes except via its own RAs, so (see other posts) you need to set up OpenWRT as a RA/SLAAC/DHCP6 relay to make this work. From what I can tell OpenWRT really does just do simple relaying if this is set up, it doesn't attempt to manage the prefix or send custom RAs.

From what I can determine these are incompatible? And I assume I can't set the priority on T-Mobile's relayed advertisements either, even assuming I can set them on OpenWRT's Comcast ones?

Any ideas? Or has this all been covered? Or are we not there yet? Can OpenWRT modify relayed RAs and if so would it help to use a separate OpenWRT for T-Mobile's IPv6? I have at least two other pieces of hardware I can put OpenWRT on.

(Thanks, I should add, for this amazing piece of software. Took me a while to get my head around it, but I don't think I'm going back to stock firmware after this!)

2

One minor thing I should add is while this sounds like an odd configuration, I suspect it'll become common, given T-Mobile is specifically offering their thing as an Internet Back-up service, which exactly the kind of thing you want to multi home, and combine with a more normal higher quality service. So the only bit that's just "This is you with a weird use case" is that I'm one of the few people I know who cares about IPv6 working.

3

Hello,

The "best practices" is a tough question. Just a few years ago, I was told that it is an active research area.

For myself, I formulated the following answer:

  • The best practice is reflected in RFC8475
    • The key is not to play with priorities (it doesn't work) but to exclude the advertisements of the currently-unneeded prefix altogether
  • OpenWrt does not implement RFC8475, but, for the case when the primary WAN is PPPoE-based, one can get close with a custom script
    • The relaying of RAs for the non-PD ISP is not a problem; in fact, that's what OpenWrt does with my USB modem successfully
    • If you want to adapt my solution to a non-PPPoE link, you would need a custom probing script instead of relying on the interface to go down
  • The second-best practice, which works for non-PPPoE links too, is MWAN3 + a custom script that performs NPTv6, so that devices in the LAN only use stable ULAs and not addresses from ISP prefixes
    • That's what I used in the past, and I'd say it is more flexible: for example, it's pretty easy to have more subnets than /64 prefixes available, and just NAT the ones that don't fit.
    • This is also more resilient to surprise renumberings done by the ISPs
    • IPv6 support in mwan3 relies on NAT6 or NPTv6, but OpenWrt does not support NPTv6; it's you who has to provide this missing part
    • Documentation: 6in4 and ISP IPv6 with mwan3 load balancing not working - #2 by patrakov (contains a script that adds NPTv6)
    • NDP relay can no longer be used because of the NPT, but in most cases you can still get effectively the same result by installing ndppd and configuring it to claim all addresses in the 2000::/4 prefix. This obviously only works if the T-Mobile modem does not protect itself against duplicate IPs.

In a tricky situation like yours, I would probably try to avoid the issue altogether:

  • Ignore IPv6 prefixes from both ISPs
  • Register an account on https://hoppy.network/, they will become your new virtual ISP
    • They provide one static public IPv4 address and a /56 worth of IPv6 via WireGuard, which is very easy to set up on OpenWrt
  • Now deal only with the IPv4 route fail-over to the Hoppy VPN server - easy to do with mwan3
    • Fail-back is the tricky part and needs to be tested
  • Delete all IPv6 rules from mwan3, let the LAN get the prefix from Hoppy

The advantage of this setup is that even IPv4 connections would not break if your WAN fails, due to the seamless fail-over provided by WireGuard.

I would gladly help with any of the three proposed setups via a video meeting.

1 Like
4

If its T-mobile, more then likely they don't use IPV6. I've had T-mobile home internet and they don't use ipv6.

5

Having been there... the problem is TMo with not issuing PD's - not impossible to work around, but you basically have to allow them to assign the IPv6...

I had TMHI as a remote work GW, so I wouldn't have to use my home cable internet for work stuff...

Doesn't help that TMo Gatway is using 464XLAT, so their device will assign you a private IPv4 address for the WAN interface - so firewall rules can be funny...

That, along with DNS shenanigans on the Tmo side - basic connectivity is ok with their GW, but going dual-WAN is a challenge, and if you talk to their customer care team, they've got limited depth on the level of assistance they can provide.

1 Like
6

Thanks, a lot to think on. The hoppy.network one is nice though obviously has limitations, but I guess I don't need to use it for the bulk of my outgoing traffic. mwan3 triggering things also seems like something to explore...

7

That's not true, at least not with my connection (and apparently the experience of others.) My lan4 interface (which is connected to the T-Mobile hub) is able to ping external IPv6 addresses (such as Google.) Indeed I believe internally their entire network is IPv6.

It's just T-Mo doesn't provide devices with a bridge-mode type thing. Instead you get a simple /64 available to anything that connects directly to their (for want of a better word) "modem". Hence you need to configure both LAN and WAN as "relay" if you want IPv6 connectivity behind an OpenWRT router.

8

Their network is all IPv6 on the Telco side - that's a good thing, IMHO...

the challenge is how to route and subnet, as they do not issue a PD to the attached clients on their GW - you'll get an IPv6 address on your device, and it's functional..

Bingo... you're starting to get it... and no, they don't issue an /64...

9

I honestly thought each home network gets a /64, routing must be a PITA if every NDP packet has to go over the cellular network as well as their modem/gateway's internal network...

10

Mixed bag - with TMo, you'll still have link-local IPv6, so internal traffic stays just that...

IPv6 public - that has to go and assigned thru them...

11

OK, so having thought on this all night, I don't get a /64 all to myself but is the reverse true? Can I assume that T-Mobile will tell everything on my network to be under a single /64 (at least for a specific time period, ignoring the usual ISP changes?

I'm leaning towards something similar to @patrakov 's first option it looks as if it requires both ISPs issue a PD (if I'm reading the script correctly) but maybe there's a way to forge a "This prefix is dead, ie lifetime=0" packet for the T-Mobile one. But obviously that's only practical if T-Mobile will always only ever put my network's devices under one prefix.

Alas if that works I will still have to mess with /etc/config/dhcp because I'll need to switch lan's RA and DHCP6 from server to relay mode and back. So whatever solution is picked will be ugly.

12

No, it doesn't require a PD.

If your ISP does not delegate a prefix, here is a two-step workaround:

  • Add an option extendprefix 1 to the relevant WAN;
  • Install ndppd and configure it to answer NDP for absolutely everything (2000::/4) on the relevant WAN link.
13

I'll need to switch lan's RA and DHCP6 from server to relay mode and back

With option extendprefix 1 and ndppd, no, you don't. It converts the only /64 prefix to a delegated one from the viewpoint of OpenWrt, so you can always run odhcpd in server mode.

1 Like
14

So spent the night thinking on this. First of all, THANK YOU for the awesome and comprehensive information. It's a shame the priority thing doesn't work in practice, that feels like it would have been simpler.

OpenWrt does not implement RFC8475, but, for the case when the primary WAN is PPPoE-based, one can get close with a custom script

  • The relaying of RAs for the non-PD ISP is not a problem; in fact, that's what OpenWrt does with my USB modem successfully
  • If you want to adapt my solution to a non-PPPoE link, you would need a custom probing script instead of relying on the interface to go down

This seems like the right solution at the moment but the relayed RAs seems like it might complicate things. I'll need to change /etc/network/dhcp each time I switch (not a big deal, but lan->ra and lan->dhcpv6 would need to change from server to relay) and I need to find a way to generate an RA that makes the T-Mobile route lifetime=0 when Comcast comes back online. I assume OpenWRT doesn't have a built-in way to do that?

Neither being PPPoE isn't an issue, as I can use mwan3's notify script as a replacement (mostly: technically Comcast's IPv4 could go down but not IPv6, or vice-versa, but I suspect that's far rarer than both being down at the same time.)

The second-best practice, which works for non-PPPoE links too, is MWAN3 + a custom script that performs NPTv6, so that devices in the LAN only use stable ULAs and not addresses from ISP prefixes

I'm not sure how practical this is so still thinking on it. As @sfx2000 notes, my assumption I was getting a /64 from T-Mobile isn't entirely accurate, there may be nodes that needs to be reachable outside my home network that have the same prefix.

Register an account on https://hoppy.network/, they will become your new virtual ISP

I absolutely love this idea and may sign up anyway, but it solves a slightly different problem. There's also Route64 (which is free but no IPv4.) No experience of either, Hoppy might be a better choice simply because it's paid and so there's a predictable commitment.

15

I think we need to split the problem into the "convert the non-cooperative ISP to a pseudo-delegated prefix" and "implement a fail-over over two (pseudo-)delegated prefixes" parts.

Look: no matter whether you choose the best or second-best practice solution, you effectively need a delegated prefix from both providers. So I suggest focusing on ndppd and extendprefix as the first step, and discussing the next steps later.

In about 1.5 hours, I will be free for a meeting - is it OK to PM the Jitsi Meet link to you so that we can implement this together?

1 Like
16

That would be very strange. The relevant RFC (https://datatracker.ietf.org/doc/html/rfc7278) specifically mentions that on mobile links, the entire /64 prefix belongs to the subscriber. Android, when sharing IPv6-only connections, assumes this, so I believe we can assume this too.

17

Unfortunately I'm at work (and my availability afterwards is awkward!), but I'll take a look at ndppd and extendprefix and see what they might do to make this easier.

EDIT: Good news, this is working. I get a prefix reported for wanb6, and I can use the T-Mobile address (2607:...) on my PC to ping Google.

Actions performed:

  • Installed ndppd (did no configuration yet. I assume it doesn't do anything on an interface if DHCP6 PD worked?)
  • Edited /etc/config/network and added 'option extendprefix '1'" to the entry for wanb6.

So at this point I guess something to ifup and ifdown the associated IPv6 interface in mwan3 is all that's needed unless I'm missing something.

18

I can certainly see issues with it, not least what happens if you're connected to two networks at once (as mobile devices often are, cellular and Wi-Fi) I mean, yeah, they can minimize these issues but for multiple reasons it seems unnecessarily suboptimal (presumably TMo is having to send NDP packets to all the gateways on a particular /64 and they all have to relay to one another over the cellular network? That's really ugly!)

19

Without configuration (and I asked for a very specific and evil configuration - "claim on wanb for everyone that asks that the whole Internet belongs to me and is directly reachable one NDP packet away"), ndppd does not do anything.

So, maybe, the cellular modem already routes the whole prefix to OpenWrt? In that case, ndppd is unnecessary.

20

Heh. Well, hard to tell, this one might even be gateway specific. Or it might just be the fact I'm pinging FROM a node behind the router is being cached by T-Mobile's gateway as implicit neighbor information (that'd be my guess.) So if I want incoming connections to work I probably need to configure ndppd. Or I can just get everything that needs incoming connections to work to ping something external once a minute, what could possibly go wrong? :wink:

(I'm worried your recommended configuration might be a problem assuming @sfx2000 is right about shared /64s, as I'd presumably start getting traffic for other T-Mobile customers, and those other customers would lose Internet access. Which again suggests T-Mo is playing a dangerous game if that's really how they've configured things. The more I think about it, the less likely it seems to be true.)

EDIT: Reading the example configuration file (https://github.com/DanielAdolfsson/ndppd/blob/master/ndppd.conf-dist), it looks like ndppd can be made to run more sanely than that. There's an iface and an auto statement where ndppd will at least send a local neighbor solicitation. So... I assume something like:

proxy lan4 {
    promiscuous true
    rule 2607::/16 {
        iface lan
    }
}

will do the trick. The T-Mobile gateway is connected to lan4.

Does this sound sane? (Before I risk crippling T-Mobile's network with an overgreedy NDPPD...)