Dual ISP IPv6 configuration help

rmappleby · April 11, 2025, 4:50pm

I'm not sure if what I'm about to ask for even makes sense, so perfectly happy to be educated, or diverted onto better approaches if that makes more sense.

What I'm trying to do is put one machine on my LAN on the static IPv6 prefix from one of my two upstream ISPs, and all the other machines on my LAN on the dynamic IPv6 prefix from my other upstream ISP. Fundamentally I'm trying to avoid the (multi-home) problems of any interface having more than one GUA.

So, is there any way that I can configure OpenWRT to implement this? A sort of blacklist/whitelist based on MAC addresses perhaps? I'm happy to manually configure the networking on the "solo" machine if necessary.

slh · April 11, 2025, 6:51pm

That doesn't work (aside from manual client-side configuration), either the prefix is active on your LAN (with all its auto-configuration) or it isn't.

dlakelan · April 11, 2025, 7:01pm

You can do this just fine with VLANs though. Put the one machine on its own VLAN and put the static prefix there. Put the other machines on a separate VLAN with the dynamic prefix.

patrakov · April 11, 2025, 8:44pm

Please configure two separate LANs. You can do this either through VLANs or using separate bridges, or a combination of both. Then, for each of the LAN, set the ip6class property to the name of the WAN from which it should assign IPv6 addresses.

This is exposed on the Advanced tab of the LAN interface configuration.

rmappleby · April 11, 2025, 10:43pm

Thanks everyone. Really appreciate the help with this - I will set up separate lans with VLANs. Judging from @patrakov screenshot I can spread my ULA across both VLANs to keep internal addressability/connectivty working smoothly too, which is a big help. Will try this out on Sunday and report back!

rmappleby · April 19, 2025, 10:33am

Just to conclude this:
I separated my br.lan into two new devices using VLAN filtering. I then switched my main "lan" interface onto one of those, leaving the other to be used with my second ISP as "lan-new". While doing this I also took the opportunity to extend those across my switches with trunks etc.

Back on the router I got IPv4 working really easily, with private subnets assigned to each VLAN instance, and OpenWRT managing routing between them. All outbound IPv4 traffic went with my default route, which followed the IPv4 priority, and out onto my default (original) ISP as expected.

I then installed the PBR app, which allowed me to create a simple rule for the IPv4 traffic (so that anything from the "lan-new" subnet not destined for a private IP address range goes with my non-default ISP) and that was it for IPv4. Done and working.

IPv6 was more problematic, and although I have a working solution now, I'm not sure why I needed to do it the way I did. Bottom line is I followed @patrakov approach with prefix filters on the interfaces, which appeared to cause prefixes to be selectively assigned correctly, and dhcp6 and RA to configure addresses etc.

But, routing didn't work. From within lan-new I couldn't reach any IPv6 addresses outside, and from outside I couldn't access any IPv6 address within lan-new, despite both being in the same firewall zone. That was equally true for ULA or GUA addresses.

In the end, after a lot of fighting, since I only intend this new subnet to be used by one or two machines, I tried manually creating an absolutely minimal static netplan config on my test ubuntu machine:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      optional: true
      accept-ra: no
      addresses:
        - 192.168.251.212/24
        - "2001:xxx:xxxx:xx00::212/64"
        - "fdb4:0:0:1::212/64"
      gateway4: 192.168.251.1
      gateway6: "2001:xxx:xxxx:xx00::1"
      nameservers:
        search: [my.lan]
        addresses:
          - 192.168.251.1
          - "2001:xxx:xxxx:xx00::1"
          - "fdb4:0:0:1::1"

I added a couple of DNS records for 192.168.251.212 and fdb4:0:0:1::212 to OpenWRT for convenience, and everything works perfectly.

So I suspect somewhere in the morass of IPv6 settings to automatically configure DHCP6 RA NDP etc etc, I misconfigured something. I'd love to understand what, and if anyone can point me to a good guide for setting up multiple routable IPv6 subnets on a recent version of OpenWRT, I'd be delighted to read and understand.

But for me, this now works well enough for my immediate needs. Hope this might help someone else.

egc · April 19, 2025, 10:42am

Without having seen your settings just a wild guess.

Disabling IPv6 source routing is often needed to get a default IPv6 routing via e.g. an alternate interface like VPN, Luci > Network > Interfaces > wan6 > Advanced tab: Disable/untick : IPv6 Source Routing
/etc/config/network add under wan6: option sourcefilter '0'

rmappleby · April 19, 2025, 10:51am

Thanks @egc I will take a look at that next, as it would be good to get it working automatically.

rmappleby · April 28, 2025, 8:18pm

Ok, I really wish I'd not started trying to make the IPv6 auto configuration work. I've spent over a week now, and I have failed miserably. This is now a matter of principle: I am determined to understand it. However, I'm well out of my depth and need some help from someone that really understands configuring IPv6, preferably in a multihomed environment.

Just to recap, I'm trying to move to a solution where I have a "bulk" wan, and an "overlay" wan. The bulk wan will eventually have CGNATed IPv4, and dynamic IPv6. The "overlay" wan will be accessed by an L2TP tunnel run over the "bulk" wan, and have a single static IPv4 and a static /48 IPv6 block. The "overlay" is metered (albeit generously), so it makes sense to run just the services that require static addresses through that, and everything else through the "bulk" wan.

I've tried to break the problem down into steps so I can tackle them one at a time:

Split the router's br-lan in two using VLAN filtering:
I've created br-lan.1 and br-lan.10, made br-lan.1 the device for my original lan, and created a lan_server interface and assigned br-lan.10 as it's device.
I adjusted the VLAN ingress/egress rules so my router uses physical Lan2 port as a tagged trunk to my managed switch, and has untagged access to lan and lan_server on physical ports Lan3 and Lan4 respectively. My switch runs VLAN1 as default, and I've added VLAN10, and exposed an untagged VLAN10 on one port of the switch too. Finally I assigned both lan and lan_server interfaces to the routers LAN firewall zone.
Set up separate IPv4 subnets on each interface.
I carried over 192.168.252.0/22 for my lan interface, and added 192.168.251.0/24 for the lan_server interface. I have two upstream IPv4 addresses, supplied by wan (PPPoE) and aaisp (the L2TP tunnel). wan has a gateway priority of 0 and default gateway is set. aaisp has gateway priority of 50 and default gateway is not set. So the router flows IPv4 traffic upstream via the wan (PPPoE) interface. At this point I have IPv4 connectivity between the subnets. Each can ping the other, and both can ping devices on the internet, via the PPPoE connection (ie the "bulk" wan). The devices configure on pseudo-static IP addresses, with hostnames given to them during auto configuration based on information I've put into the "static leases" in LUCI. As mentioned in previous posts, I will use PBR to make lan_server IPv4 addresses use the aaisp L2TP connection (ie, the "overlay") in due course. But first...
Set up separate IPv6 subnets on each interface.
I have 2 sources of IPv6 connectivity - wan6 (HE 6in4 tunnel) with a /48 delegated prefix, and aaisp6 (DHCPv6 off the L2TP tunnel) which currently has a /60 delegated prefix. My plan was to delegate a /64 prefix from wan6 to lan, and from aaisp6 to lan_server, using prefix filtering on the interfaces. Both would also receive a /64 from the ULA. So I hoped each interface would essentially have its own upstream IPv6 connection for internet connectivity, with internal connectivity handled with ULAs. I had hoped that separating the prefixes would avoid any device getting two GUA addresses, and therefore avoid the problems where you attempt to flow traffic from one prefix out over the other prefix (and get all your packets dropped).

And this is where I have hit problems. I can see the prefixes being delegated in LUCI. But when I plug freshly installed, unmodified Ubuntu server 25.04 instances (running on RPIs) into the lan and lan_server I'm not getting the IPv6 connectivity set up that I was expecting.

Firstly, the default Netplan in Ubuntu Server 25.04 brings up dynamic IPv4 and IPv6 connections, but doesn't seem to collect DNS Search domains, so DNS resolution only happens if I supply the FQDN:

network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true

So ping laser fails DNS resolution, while ping laser.site1.lan works correctly. If I supply the DNS Search Domains to the Ubuntu clients in their Netplan yaml:

network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true
      nameservers:
        search: [site1.lan, site2.lan]

then I can see that ping laser now works. So a specific question: how do I enable this DNS configuration to be pushed to the clients automatically?

If I clean boot the router and the two Ubuntu servers, I don't see the IPv6 connection on lan_server being set up correctly. Specifically, the device plugged into the lan_server interface appears to have an address and a route on the prefix that I thought I had only delegated to the lan interface. Worse, this appears to be used by default, which breaks IPv6 connectivity - trying to ping -6 ipv6.google.com results in complete packet loss, as does ping -6 laser.site1.lan etc. Fortunately IPv4 still seems to be correctly configured, so I can at least SSH into it for debugging etc.

Looking at netplan status seems to imply that this IP address & route came from Router Advertisements. Frankly, I'm struggling to debug this. Anyone able to help me out please, both to fix it, and more importantly, help me understand it?!

Thanks to anyone reading this far! What configuration files should I share?

rmappleby · April 30, 2025, 1:24pm

It appears that what I reported in my last post was incorrect. The two Ubuntu servers are brought up exactly as I expected, and seem to work perfectly either for some (short) period of time, or possibly for one interaction with the router, after which the server on lan_server (ie, VLAN10) appears to gain an IP address and a route on a prefix that is not delegated to that interface. I assume that has to be coming from OpenWRT as some kind of NDP or RA packet, presumably because I've misconfigured something.

This console trace is from the ubuntu server attached to the lan_server interface, after that was freshly restarted and the server rebooted. 2001:XXX:XXXX:XXXX::0/64 is the prefix delegated from my overlay ISP (via my L2TP tunnel), and 2001:YYY:YYYY::0/64 is the prefix delegated to the other interface (lan) from my Hurricane Electric (6in4) tunnel, which acts as my bulk IPv6 ISP.

We can see 6 IP addresses defined on the ethernet interface. The first three are provisioned by DHCP, the next two by SLAAC (judging by the MAC & hostname), and the last is the NDP-created link-local address used to self-configure basic connectivity, namely:

IPv4 address for eth0: 192.168.251.212
IPv6 address for eth0: fdb4:0:0:1::212
IPv6 address for eth0: 2001:XXX:XXXX:XXXX::212
IPv6 address for eth0: fdb4::1:ba27:ebff:fe73:a5ba
IPv6 address for eth0: 2001:XXX:XXXX:XXXX:ba27:ebff:fe73:a5ba
IPv6 address for eth0: fe80::ba27:ebff:fe73:a5ba/64

We can see that the prefixes available are fdb4::1/64 and 2001:XXX:XXXX:XXXX/64, which matches my ULA and my overlay ISP delegated prefix, and agrees with the setting of the IPv6 prefix filters on this (lan_server) interface.


 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

 System information as of Tue Apr 29 18:40:32 BST 2025

  System load:           1.42
  Usage of /:            8.0% of 28.69GB
  Memory usage:          25%
  Swap usage:            0%
  Temperature:           48.3 C
  Processes:             145
  Users logged in:       0
  IPv4 address for eth0: 192.168.251.212
  IPv6 address for eth0: fdb4:0:0:1::212
  IPv6 address for eth0: 2001:XXX:XXXX:XXXX::212
  IPv6 address for eth0: fdb4::1:ba27:ebff:fe73:a5ba
  IPv6 address for eth0: 2001:XXX:XXXX:XXXX:ba27:ebff:fe73:a5ba

 * Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s
   just raised the bar for easy, resilient and secure K8s cluster deployment.

   https://ubuntu.com/engage/secure-kubernetes-at-the-edge


Last login: Tue Apr 29 18:34:57 2025 from 192.168.252.119
user@ubuntu-pi1:~$ sudo cat /etc/netplan/50*.yaml; echo "---"; netplan status; echo "---"; resolvectl;
[sudo] password for user: 
network:
  version: 2
  ethernets:
    eth0:
      optional: true
      dhcp4: true
      nameservers:
        search: [site1.lan, site2.lan]

---
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: site1.lan
                   site2.lan

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128

●  2: eth0 ethernet UP (networkd: eth0)
      MAC Address: b8:27:eb:73:a5:ba (Microchip Technology, Inc. (formerly SMSC))
        Addresses: 192.168.251.212/24 (dynamic, dhcp)
                   fdb4:0:0:1::212/128 (dynamic, dhcp)
                   2001:XXX:XXXX:XXXX::212/128 (dynamic, dhcp)
                   fdb4::1:ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   2001:XXX:XXXX:XXXX:ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   fe80::ba27:ebff:fe73:a5ba/64 (link)
    DNS Addresses: 192.168.251.1
                   fdb4:0:0:1::1
                   2001:XXX:XXXX:XXXX::1
       DNS Search: site1.lan
                   site2.lan
           Routes: default via 192.168.251.1 from 192.168.251.212 metric 100 (dhcp)
                   192.168.251.0/24 from 192.168.251.212 metric 100 (link)
                   192.168.251.1 from 192.168.251.212 metric 100 (dhcp, link)
                   2001:XXX:XXXX:XXXX::/64 metric 100 (ra)
                   fdb4:0:0:1::/64 metric 100 (ra)
                   fe80::/64 metric 256
                   default via fe80::9683:c4ff:fea2:e8df metric 100 (ra)

1 inactive interfaces hidden. Use "--all" to show all.
---
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.251.1
       DNS Servers: 192.168.251.1 fdb4:0:0:1::1 2001:XXX:XXXX:XXXX::1
        DNS Domain: site1.lan site2.lan
     Default Route: yes

Link 3 (wlan0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no

user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ # At this point the server seems to be configured correctly ... see routes 
user@ubuntu-pi1:~$ # to ipv6.google.com and to my "other" server on my "lan" interface.
user@ubuntu-pi1:~$ # If I were to ping either of these now, they would work - once.
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ ip -6 r get to 2a00:1450:4009:815::200e
2a00:1450:4009:815::200e from :: via fe80::9683:c4ff:fea2:e8df dev eth0 proto ra src 2001:XXX:XXXX:XXXX::212 metric 100 pref medium
user@ubuntu-pi1:~$ ip -6 r get to fdb4::22
fdb4::22 from :: via fe80::9683:c4ff:fea2:e8df dev eth0 proto ra src fdb4:0:0:1::212 metric 100 pref medium
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ # Now wait a few minutes ...
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ date
Tue Apr 29 18:41:53 BST 2025
user@ubuntu-pi1:~$ date
Tue Apr 29 18:51:36 BST 2025
user@ubuntu-pi1:~$ # 
user@ubuntu-pi1:~$ # Now look at the IP addresses and routes ...
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ sudo netplan status; echo "---"; resolvectl;
     Online state: online
    DNS Addresses: 127.0.0.53 (stub)
       DNS Search: site1.lan
                   site2.lan

●  1: lo ethernet UNKNOWN/UP (unmanaged)
      MAC Address: 00:00:00:00:00:00
        Addresses: 127.0.0.1/8
                   ::1/128

●  2: eth0 ethernet UP (networkd: eth0)
      MAC Address: b8:27:eb:73:a5:ba (Microchip Technology, Inc. (formerly SMSC))
        Addresses: 192.168.251.212/24 (dynamic, dhcp)
                   2001:YYY:YYYY:0:ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   fdb4::ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   fdb4:0:0:1::212/128 (dynamic, dhcp)
                   2001:XXX:XXXX:XXXX::212/128 (dynamic, dhcp)
                   fdb4::1:ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   2001:XXX:XXXX:XXXX:ba27:ebff:fe73:a5ba/64 (dynamic, ra)
                   fe80::ba27:ebff:fe73:a5ba/64 (link)
    DNS Addresses: 192.168.251.1
                   fdb4:0:0:1::1
                   2001:XXX:XXXX:XXXX::1
                   2001:YYY:YYYY::1
       DNS Search: site1.lan
                   site2.lan
           Routes: default via 192.168.251.1 from 192.168.251.212 metric 100 (dhcp)
                   192.168.251.0/24 from 192.168.251.212 metric 100 (link)
                   192.168.251.1 from 192.168.251.212 metric 100 (dhcp, link)
                   2001:YYY:YYYY::/64 metric 100 (ra)
                   2001:XXX:XXXX:XXXX::/64 metric 100 (ra)
                   fdb4::/64 metric 100 (ra)
                   fdb4:0:0:1::/64 metric 100 (ra)
                   fe80::/64 metric 256
                   default via fe80::9683:c4ff:fea2:e8df metric 100 (ra)

1 inactive interfaces hidden. Use "--all" to show all.
---
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.251.1
       DNS Servers: 192.168.251.1 fdb4:0:0:1::1 2001:XXX:XXXX:XXXX::1 2001:YYY:YYYY::1
        DNS Domain: site1.lan site2.lan
     Default Route: yes

Link 3 (wlan0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
     Default Route: no
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ # Again, ask for specific routes to ipv6.google.com and my other server.
user@ubuntu-pi1:~$ # Note the changes. 
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ ip -6 r get to 2a00:1450:4009:815::200e
2a00:1450:4009:815::200e from :: via fe80::9683:c4ff:fea2:e8df dev eth0 proto ra src 2001:YYY:YYYY:0:ba27:ebff:fe73:a5ba metric 100 pref medium
user@ubuntu-pi1:~$ ip -6 r get to fdb4::22
fdb4::22 from :: dev eth0 proto ra src fdb4::ba27:ebff:fe73:a5ba metric 100 pref medium
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ # Try to actually ping - fails
user@ubuntu-pi1:~$ #
user@ubuntu-pi1:~$ ping -6 -c 1 2a00:1450:4009:815::200e
PING 2a00:1450:4009:815::200e (2a00:1450:4009:815::200e) 56 data bytes

--- 2a00:1450:4009:815::200e ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

user@ubuntu-pi1:~$ ping -6 -c 1 fdb4::22
PING fdb4::22 (fdb4::22) 56 data bytes
From fdb4::ba27:ebff:fe73:a5ba icmp_seq=1 Destination unreachable: Address unreachable

--- fdb4::22 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Looking at the output from that second netplan status command, we can see that 2001:YYY:YYYY:0:ba27:ebff:fe73:a5ba/64 (dynamic, ra) has been provisioned on the interface, allegedly by an "ra". Ditto a DNS server and a route using that prefix.

I feel like I must be missing something really obvious here. Can anyone help?