IPv6 routing does not work on LAN, but does work from router

Hello, I have installed OpenWRT r25239-a3a33f02ce on the Linksys E8450. This has been working as desired for about a month, but this morning at about 7:15am local time on Mar 23rd (according to my network monitoring), IPv6 stopped working for all devices on all VLANs.

The router is able to reach IPv6 endpoints:

root@yumi:~# wget -O/dev/null -6 https://example.com
Downloading 'https://example.com'
Connecting to 2606:2800:220:1:248:1893:25c8:1946:443
Writing to '/dev/null'
/dev/null            100% |*******************************|  1256   0:00:00 ETA
Download completed (1256 bytes)

However, devices on the network are not able to do so:

{raxod502@kelsier} ~ % curl -v -6 https://example.com
*   Trying 2606:2800:220:1:248:1893:25c8:1946:443...
* connect to 2606:2800:220:1:248:1893:25c8:1946 port 443 failed: No route to host
* Failed to connect to example.com port 443 after 3078 ms: No route to host
* Closing connection 0
curl: (7) Failed to connect to example.com port 443 after 3078 ms: No route to host

The following IPv6 routing table is defined for a device on the local network:

{raxod502@kelsier} ~ % ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2607:f598:b15a:5c1::/64 dev wlp166s0 proto ra metric 600 pref medium
2607:f598:b15a:5c0::/60 via fe80::8069:1aff:fe23:8d77 dev wlp166s0 proto ra metric 600 pref medium
fd11:2198:bc83:1::/64 dev wlp166s0 proto ra metric 600 pref medium
fd11:2198:bc83::/48 via fe80::8069:1aff:fe23:8d77 dev wlp166s0 proto ra metric 600 pref medium
fe80::/64 dev wlp166s0 proto kernel metric 1024 pref medium
default via fe80::8069:1aff:fe23:8d77 dev wlp166s0 proto ra metric 20600 pref medium

I configured an internal network and guest network:

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	list ipaddr '192.168.1.1/24'
	list dns '9.9.9.9'
	list dns '149.112.112.112'
	list dns '2620:fe::fe'
	list dns '2620:fe::9'
	option ip6assign '64'
	option delegate '0'
	option ip6hint '0'

config interface 'guest'
	option proto 'static'
	option device 'br-guest'
	list ipaddr '192.168.3.1/24'
	list dns '9.9.9.9'
	list dns '149.112.112.112'
	list dns '2620:fe::fe'
	list dns '2620:fe::9'
	option delegate '0'
	option ip6assign '64'
	option ip6hint '1'

The following information is shown under the IPv6 Upstream section in LuCI:

Protocol: DHCPv6 client
Prefix Delegated: 2607:f598:b15a:5c0::/60
Address: 2607:f598:b150:252:0:a:0:1f/128
Address: 2607:f598:b150:252:8269:1aff:fe23:8d75/64
Gateway: fe80::227:e3ff:fe78:93db
DNS 1: 2607:f598:0:1::2
DNS 2: 2607:f598:0:1::3
DNS 3: 2001:4860:4860::8888
Connected: 27d 18h 47m 18s
Device: Switch port: "wan"
MAC address: 80:69:1A:23:8D:75

Based on this my understanding is that my ISP has delegated a /60 prefix to me. My intent is to use one /64 sub-block for the internal network, and another /64 sub-block to the guest network, which would utilize one-eighth of the available address space. This is why I set unique ip6hint values for each network.

IPv4 works correctly on all devices on all VLANs. Before this morning, IPv6 also worked correctly on all devices on all VLANs; now, it doesn't work on any of them, but IPv4 is still fine. I am not aware of any configuration changes that were applied this morning; in fact, I was asleep at the time.

I uploaded the system log, which covers the time at which routing stopped working: https://gist.github.com/raxod502/b761df9c36e9bf9e39e27b284e65ad2d

What should I look into or learn about in order to understand what could be wrong? Is there any other diagnostic or configuration information I should provide here? (I checked the wiki, but couldn't find any information on suggested information to include in a forum post.)

Update:

At about 3:05am on Mar 24th local time (according to my network monitoring), with no configuration changes on my part, IPv6 routing started working again for all devices on all VLANs. Here is the system log from this morning: https://gist.github.com/raxod502/c9136a106a807e35fa8ba517394fbaf7

I still have no idea what is going on. The fact that the system recovered on its own suggests to me that perhaps rebooting the router could have remediated the issue, but I'd prefer to understand why it became broken in the first place, especially given that IPv6 routing clearly worked on the router itself, and only appeared to be an issue with OpenWRT network configuration.

The issue has recurred today at about 7:20am PT on Jun 16th with much the same symptoms, and IPv6 networking is currently offline for all devices. As before, the router itself is perfectly well able to reach IPv6 hosts so this must be an issue with the local network.

{raxod502@cephandrius} ~ % curl -v -6 https://example.com
*   Trying 2606:2800:21f:cb07:6820:80da:af6b:8b2c:443...
* Immediate connect fail for 2606:2800:21f:cb07:6820:80da:af6b:8b2c: Network is unreachable
* Closing connection 0
curl: (7) Couldn't connect to server
{raxod502@cephandrius} ~ % ip -6 route
::1 dev lo proto kernel metric 256 pref medium
fd11:2198:bc83::2f6 dev enx70886b8d150d proto kernel metric 100 pref medium
fd11:2198:bc83::96d dev wlp0s20f3 proto kernel metric 600 pref medium
fd11:2198:bc83::/64 dev enx70886b8d150d proto ra metric 100 pref medium
fd11:2198:bc83::/64 dev wlp0s20f3 proto ra metric 600 pref medium
fd11:2198:bc83::/48 via fe80::8269:1aff:fe23:8d76 dev enx70886b8d150d proto ra metric 100 pref medium
fd11:2198:bc83::/48 via fe80::8269:1aff:fe23:8d76 dev wlp0s20f3 proto ra metric 600 pref medium
fe80::/64 dev enx70886b8d150d proto kernel metric 1024 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium

I didn't make any configuration changes that line up with the outage. This time, I have rebooted the router (since I hypothesized last time that it would fix the problem), but it did not produce any change.

If anyone has suggestions I would love to hear them as I would very much enjoy having working IPv6 networking.

Edit: Here's the full output from uci show, which should cover all of my OpenWRT settings: https://gist.github.com/raxod502/372cd7922c02c33656ad06fcb51799ae

I watched that video from the linked timestamp, but I'm not sure I understand how it can help me resolve my problem.

The author seems to be primarily talking about issues arising from running a dedicated IPv6-only home network, which isn't what I'm doing - I use IPv4/IPv6 dual-stack. Furthermore they are flagging issues caused by impaired IPv6 support from upstream ISPs, but the fact that outbound IPv6 works fine from my router implies that the ISP is supporting the protocol just fine, and this is specifically an OpenWRT issue.

Did I perhaps miss the point?

IPv6 is a PITA and not necessairly worth it.

Sorry, I did not know the timestamp would be included. I meant for the whole thing to be watched.

I'll fix it.

Thanks. I watched the rest of the video. I'm still not sure I understand the point, though. Are you suggesting that the best way to use OpenWRT is to completely block IPv6 traffic? I don't want to do that, and it would surprise me if OpenWRT is not intended to support IPv6.

Or are you pointing to the video because the author is primarily reporting an issue that occurred with his network when his ISP changed the IPv6 prefix delegation assigned to his gateway? Are you hypothesizing that the same thing might have happened to me, leading to the outage?

In the video, the author mentioned that having SLAAC configured on his network (rather than static address assignments) would have prevented any issues, and SLAAC is indeed enabled in my configuration. Is there a different configuration that would be better?

There are countless threads on getting IPv6 working and many have using OpenWrt.
OpenWrt is fully capable of using IPv6.

I presume any changes on the ISP side can wreak havock with IPv6.
Also,
didn't the guy in the video say home use is problematic? That most sites don't use IPv6 and it still has to be translated to IPv4 on their end? I have not watched it in a while.

You can search for all those threads and get all the information you need to get it done but expect maintance.
@psherman is the best I know of with vlans but I'm not sure he is a fan of IPv6.

IPv6 is here to stay, there are no alternatives - and it works.

Established US ISPs and big-tech might sit on enough IPv4 addresses for now, but the situation elsewhere in the world is quite different. Be it newer cable- or fibre ISPs simply not getting enough IPv4 address space for their customers (cgNAT is not cool, IPv6 is a life saver in that situation) or even hosting providers charging extra for IPv4 addresses for their servers (while you can't omit IPv4 for the public at large yet, the question arises when your server only needs to cater to a more limited audience, where you may be able to insist on (incoming-) IPv6 only, rather than paying extra each month).

Well, maybe that is the issue:
If you are in the US our ISPs don't make IPv6 easy; I think they go out of their way to make it hard to use.. I suppose the situation is different outside, based on your reply.

You can search for all those threads and get all the information you need to get it done

Yes, of course I did so before opening a new thread, per standard forum etiquette. I wasn't able to find any relevant information to my specific problem in those threads, although I read through hundreds of comments across numerous threads. Was there a particular discussion you had in mind?

To be clear, I'm not asking for someone to set up my entire network for me. I already spent many hours reading the documentation and configuring my VLANs to work correctly, including with IPv6. What I'm looking for are suggestions on how to troubleshoot the unexpected outage, e.g. tools and procedures that could identify which component is at fault or which aspect of the configuration is problematic.

didn't the guy in the video say home use is problematic?

I really don't understand what I should take away from that statement. As @slh said, IPv6 is here to stay, there are no alternatives - and it works.

My goal here is to correct the configuration or software issues that are causing IPv6 routing errors, not to completely change my network architecture. If there's some specific aspect to my use case that makes IPv6 inappropriate, please feel free to let me know what it is, but I don't think a statement like "home use is problematic" is enough for me to go on.

I presume any changes on the ISP side can wreak havock with IPv6 [...] maybe that is the issue

You've implied several times that this might be an ISP issue. Could you say more about what diagnostic information from my post led you to this conclusion? As I mentioned before, the IPv6 internet is fully reachable from my router, so I have no reason to believe this is anything other than an OpenWRT (configuration) problem.

Well, we know, for a fact, nothing was changed on your router.

What does that leave?
Just the process of elimination.

Dump this so we can have a look.

nothing was changed on your router.

What does that leave?
Just the process of elimination.

Technology is complicated. I've many times assumed "it must be X, because nothing else changed". Then it turns out that there was an unknown system daemon that performed an unattended upgrade, or somebody's TLS certificate expired, or a third-party server made some change, or some device on the network started speaking a different protocol, or a time zone changed, or some log file filled up, or a metastable system component finally fell out of sync by pure chance, or goodness knows what else.

I'm more than happy to report a problem to my ISP, but all the evidence I have right now indicates that the IPv6 service they provide me is still fully operational. Standard operating procedure for bisecting between router versus ISP issues is to connect directly to the uplink instead of going through the LAN. Since I can wget public sites over IPv6 from the router host, this seems to clearly point to the router as opposed to the ISP.

copy the output of the following commands and post it here

Sure, no problem:

root@yumi:~# ubus call system board
{
	"kernel": "6.1.78",
	"hostname": "yumi",
	"system": "ARMv8 Processor rev 4",
	"model": "Linksys E8450 (UBI)",
	"board_name": "linksys,e8450-ubi",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r25239-a3a33f02ce",
		"target": "mediatek/mt7622",
		"description": "OpenWrt SNAPSHOT r25239-a3a33f02ce"
	}
}

root@yumi:~# cat /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd11:2198:bc83::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	list ipaddr '192.168.1.1/24'
	list dns '9.9.9.9'
	list dns '149.112.112.112'
	list dns '2620:fe::fe'
	list dns '2620:fe::9'
	option ip6assign '64'
	option delegate '0'
	option ip6hint '0'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'

config device
	option type 'bridge'
	option name 'br-guest'

config interface 'guest'
	option proto 'static'
	option device 'br-guest'
	list ipaddr '192.168.3.1/24'
	list dns '9.9.9.9'
	list dns '149.112.112.112'
	list dns '2620:fe::fe'
	list dns '2620:fe::9'
	option delegate '0'
	option ip6assign '64'
	option ip6hint '1'

root@yumi:~# cat /etc/config/wireless

config wifi-device 'radio0'
	option type 'mac80211'
	option phy 'wl0'
	option channel '1'
	option band '2g'
	option htmode 'HT20'
	option country 'US'
	option cell_density '0'

config wifi-iface 'default_radio0'
	option device 'radio0'
	option network 'lan'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option key 'redacted'
	option ifname 'lan-2g'

config wifi-device 'radio1'
	option type 'mac80211'
	option phy 'wl1'
	option band '5g'
	option country 'US'
	option cell_density '0'
	option channel 'auto'

config wifi-iface 'default_radio1'
	option device 'radio1'
	option network 'lan'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option key 'redacted'
	option ifname 'lan-5g'

config wifi-iface 'wifinet2'
	option device 'radio0'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option isolate '1'
	option ifname 'guest-2g'
	option key 'redacted'
	option network 'guest'

config wifi-iface 'wifinet3'
	option device 'radio1'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option isolate '1'
	option ifname 'guest-5g'
	option key 'redacted'
	option network 'guest'

config wifi-iface 'wifinet5'
	option device 'radio0'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option key 'redacted'
	option network 'lan'
	option disabled '1'

config wifi-iface 'wifinet6'
	option device 'radio1'
	option mode 'ap'
	option ssid 'redacted'
	option encryption 'psk2'
	option key 'redacted'
	option network 'lan'
	option disabled '1'

root@yumi:~# cat /etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'guest'
	option interface 'guest'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option dhcpv6 'server'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

config host
	option name 'redacted'
	list mac 'REDACTED'
	option ip 'redacted'

root@yumi:~# cat /etc/config/firewall

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'guest'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'guest'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Allow-Guest-DNS'
	option src 'guest'
	option dest_port '53'
	option target 'ACCEPT'

config rule
	option name 'Allow-Guest-DHCP'
	list proto 'udp'
	option src 'guest'
	option dest_port '67'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-DHCPv6'
	option family 'ipv6'
	list proto 'udp'
	option src 'guest'
	option dest_port '547'
	option target 'ACCEPT'

config forwarding
	option src 'guest'
	option dest 'wan'

Other than two APs turned off I don't know IPv6 well enough to refine your setup.

When you got everything set up and were done tinkering, did you archive a known good save?

I do not have a configuration backup, although I can take one through LuCI the next time the system is in a healthy state.

The two disabled APs are intentional; I added them four days prior (about 11:30am PT on Jun 12th) as copies of my primary LAN APs under a separate SSID, so that I can temporarily grant a visitor unrestricted access to LAN devices for special use cases, but keep them disabled the rest of the time to prevent unauthorized usage when I am not present.

Okay, I've opened Wireshark and checked to see what IPv6 packets show up when activating the wireless interface on my device. I can see that it is sending the RS packet fine and indeed receiving an RA response that is correctly delivering a prefix of fd11:2198:bc83::/64. Based on that response I would expect my device to populate a default route and gateway, but it does not do so. The default route just goes to the loopback device for the IPv6 route table.

% ip route get to 93.184.215.14    
93.184.215.14 via 192.168.1.1 dev enx70886b8d150d src 192.168.1.226 uid 1000

% ip route get to 2606:2800:21f:cb07:6820:80da:af6b:8b2c
RTNETLINK answers: Network is unreachable

I can see my router link-local address in the routing table and it is set to use that address as a gateway for fd11:2198:bc83::/48. Furthermore those routes behave as expected since I can reach other hosts on my local network via IPv6. But for some reason the default WAN route is not established.

On the router itself I can see from https://192.168.1.1/cgi-bin/luci/admin/status/routes that there is a default route there:

device  target                    source
wan6	::/0	                  2607:f598:b150:252::/64
wan6	2607:f598:b150:252::/64
lan	    fd11:2198:bc83::/64

Which naturally explains why routing is working from the router. But something is going wrong in the advertisement of the default route to devices on LAN.

I can manually create default routes on Linux by providing the router link-local IPv6 address to these commands:

% sudo ip -6 route add default dev enx70886b8d150d via fe80::8269:1aff:fe23:8d76 metric 100
% sudo ip -6 route add default dev wlp0s20f3 via fe80::8269:1aff:fe23:8d76 metric 600

Then I can get a response from the kernel when looking up a route, at least:

% ip route get to 2606:2800:21f:cb07:6820:80da:af6b:8b2c
2606:2800:21f:cb07:6820:80da:af6b:8b2c from :: via fe80::8269:1aff:fe23:8d76 dev enx70886b8d150d src fd11:2198:bc83:0:7d9f:b179:f564:ae0e metric 100 pref medium

But, connections still do not work:

% curl -v -6 https://example.com          
*   Trying 2606:2800:21f:cb07:6820:80da:af6b:8b2c:443...
* connect to 2606:2800:21f:cb07:6820:80da:af6b:8b2c port 443 failed: Network is unreachable
* Failed to connect to example.com port 443 after 1 ms: Network is unreachable
* Closing connection 0
curl: (7) Failed to connect to example.com port 443 after 1 ms: Network is unreachable

Presumably the default route is not being established automatically because it would not work if it were there. The question is why is it not working.

Update: Some more progress. I tested a default route using fd11:2198:bc83::1 rather than fe80::8269:1aff:fe23:8d76 as gateway, just to see what would happen. Same result. This makes sense as both addresses refer to the same router.

Then I thought to run traceroute and found this:

% traceroute -6 example.com
traceroute to example.com (2606:2800:21f:cb07:6820:80da:af6b:8b2c), 30 hops max, 80 byte packets
 1  yumi.lan (fd11:2198:bc83::1)  4.601 ms !N  5.000 ms !N  4.985 ms !N

Why traceroute terminates at the gateway? Well, Wireshark says that the router is returning ICMP Destination Unreachable (no route to destination) in response to the UDP packets. Then I looked at the OpenWRT routing table... and indeed:

root@yumi:~# ip -6 route
default from 2607:f598:b150:252::/64 via fe80::227:e3ff:fe78:93db dev wan  metric 384 
2607:f598:b150:252::/64 dev wan  metric 256 
unreachable 2607:f598:b150:252::/64 dev lo  metric 2147483647 
fd11:2198:bc83::/64 dev br-lan  metric 1024 
fd11:2198:bc83:1::/64 dev br-guest  metric 1024 
unreachable fd11:2198:bc83::/48 dev lo  metric 2147483647 
fe80::/64 dev eth0  metric 256 
fe80::/64 dev br-lan  metric 256 
fe80::/64 dev lan-2g  metric 256 
fe80::/64 dev guest-2g  metric 256 
fe80::/64 dev br-guest  metric 256 
fe80::/64 dev wan  metric 256 
fe80::/64 dev lan-5g  metric 256 
fe80::/64 dev guest-5g  metric 256 
anycast 2607:f598:b150:252:: dev wan  metric 0 
anycast fd11:2198:bc83:: dev br-lan  metric 0 
anycast fd11:2198:bc83:1:: dev br-guest  metric 0 
anycast fe80:: dev eth0  metric 0 
anycast fe80:: dev br-lan  metric 0 
anycast fe80:: dev br-guest  metric 0 
anycast fe80:: dev lan-2g  metric 0 
anycast fe80:: dev wan  metric 0 
anycast fe80:: dev guest-2g  metric 0 
anycast fe80:: dev lan-5g  metric 0 
anycast fe80:: dev guest-5g  metric 0 
multicast ff00::/8 dev eth0  metric 256 
multicast ff00::/8 dev br-lan  metric 256 
multicast ff00::/8 dev br-guest  metric 256 
multicast ff00::/8 dev lan-2g  metric 256 
multicast ff00::/8 dev guest-2g  metric 256 
multicast ff00::/8 dev wan  metric 256 
multicast ff00::/8 dev lan-5g  metric 256 
multicast ff00::/8 dev guest-5g  metric 256 

Why in heavens has the default route here got a source IP restriction? Of course the router itself has a WAN IP in the 2607:f598:b150:252::/64 range so it can route out of the network, but anything else on the network will not be able to make use of this route.

Compare to the simpler, and obviously correct, IPv4 routing table:

root@yumi:~# ip -4 route
default via 199.188.193.129 dev wan  src 199.188.193.167 
192.168.1.0/24 dev br-lan scope link  src 192.168.1.1 
192.168.3.0/24 dev br-guest scope link  src 192.168.3.1 
199.188.193.128/25 dev wan scope link  src 199.188.193.167 

I have no idea how this configuration got generated by OpenWRT, I guess I need to look through the routing/firewall settings... not that those have changed in months, so I'm not sure how this was working before or why it stopped now.

Thanks for the shoutout about VLANs.
As far as IPv6 -- I am a fan (I view it as a very important development and the true future of the internet), but I don't actually currently have much experience with it, so I'm not able to offer much in the way of help on IPv6 related issues.

Okay, more investigation.

I've looked at /etc/config/network and found this directive:

config globals 'globals'
	option ula_prefix 'fd11:2198:bc83::/48'

Which makes sense as my VLANs are on fd11:2198:bc83::/64 and fd11:2198:bc83:1::/64 respectively.

But for some reason the default IPv6 route that OpenWrt configured is using a source-based policy to only route packets from 2607:f598:b150:252::/64, as if it somehow expects my VLANs to not be using ULAs (i.e., addresses in the fd00::/8 range), even though OpenWrt is what configured them to be ULAs in the first place.

In forum post Understanding IPv6 routes in web UI I see that the OP has a different routing table setup. Two blocks assigned to their LAN, one ULA and one GUA?

I see also in a hitherto-unexplored config section that the wan6 interface has an option IPv6 source routing enabled, with the comment Automatically handle multiple uplink interfaces using source-based policy routing. I didn't enable this, it came with OpenWrt on initial configuration. When disabling that config option, the unexpected source filter goes away in the route table rules. Still, no correct routing.

Ok, I look at WireGuard again while enabling my wireless interface. The RA packet uses ICMPv6 option "Route Information", which I presume is what tells Linux to create the default route. The option includes info on the range fd11:2198:bc83::/48, and indeed I do see this in my device route table:

fd11:2198:bc83::/48 via fe80::8269:1aff:fe23:8d76 dev enx70886b8d150d proto ra metric 100 pref medium
fd11:2198:bc83::/48 via fe80::8269:1aff:fe23:8d76 dev wlp0s20f3 proto ra metric 600 pref medium

So why is OpenWrt not responding with RA establishing a default route, and instead only suggesting that packets can route within/between its VLANs...?

I see in https://datatracker.ietf.org/doc/html/rfc4191#section-2.2 that there is explicit support in the protocol for communicating a default route to clients, but OpenWrt is not making use of this apparently.

Ok, back to the test I ran earlier, now that the OpenWrt route table is fixed to not reject all outbound traffic using source-based policy. Now if I add default routes manually, then trying to curl an IPv6 endpoint hangs, rather than rejecting immediately. Progress? The traceroute does this:

{raxod502@cephandrius} ~ % traceroute -6 example.com
traceroute to example.com (2606:2800:21f:cb07:6820:80da:af6b:8b2c), 30 hops max, 80 byte packets
 1  yumi.lan (fd11:2198:bc83::1)  2.039 ms  1.957 ms  1.912 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

No ICMP is coming back from the router. I guess my traffic is just going into a blackhole.

Relevant settings, maybe:

  • Network > Interfaces > Interfaces > lan > DHCP Server > IPv6 RA Settings > Default router: Set by default to "automatic", which will not announce as default router unless a default IPv6 route is available. Default route appears to be available on the router itself, so that seems not relevant.
  • Network > Interfaces > Global network options > IPv6 ULA-Prefix: Aha! I was wondering where that came from since I didn't set it. This prefix is randomly generated at first install, it says.
  • Network > Firewall > wan > Advanced Settings > IPv6 Masquerading: Disabled. Maybe needed?

Is it possible that OpenWrt is just yeeting my packets out to the public internet without translating them from ULA? How on earth would that be supposed to work? It would certainly explain why they reach the router and then drop off, though. Both the blackholing behavior here and the source-based routing filter line up properly with the assumption that no address translation is happening.

I mean, it would certainly make sense if my devices were addressed based on the IPv6 subnet that is delegated to me from the ISP, but that is not what OpenWrt is doing, at least by default it is using ULA, so are the defaults for some reason configured to be a broken mix between the two? Where it numbers with ULA but then does not enable the needed address translation?

Looking at https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6 and thinking about the best practices for an IPv6-network, I would expect that the issue here is that GUA address blocks are not being handed out to my VLANs, and only the ULAs are being used, and that is the source of the problems. (Presumably I can then re-enable source-based IPv6 routing.)

Maybe prefix delegation just needs to be enabled somewhere that it is not?

Aha! Ok, I pulled up some old screenshots of OpenWrt config screens and I noticed that for working VLANs, there are 2 CIDR ranges delegated, one ULA and one GUA. For broken VLANs there is only 1 range, the ULA. Right now I have ULA only assigned to both, which explains things being broken.

Btw, looking at the old screenshots, I can confirm that this is not the ISP changing the prefix they are delegating to me, that was 2607:f598:b150:252:8269:fe23:8d75/64 months ago just as it is now.

Ah. I think I understand the issue. I only get a /64 from my ISP, therefore VLANs cannot be configured at all, because IPv6 "basically" doesn't support subnets smaller than /64. See https://serverfault.com/a/875443.

I think IPv6 was broken on the guest network most likely, but I do not have any critical devices on there, so I didn't notice. Then something flip-flopped internally to OpenWrt and now the internal network is broken but the guest network was working.

I guess I'll request a larger address block from my ISP, and in the meantime, collapse my SSIDs into the same VLAN, and set up firewall rules based on MAC address to isolate devices that should not talk to each other. Lame, but I don't know a better workaround.

Update: Then again, I see https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6 which links to https://openwrt.org/docs/guide-user/network/ipv6/configuration#ipv6_relay on the wiki and suggests that VLANs could be configured in DHCPv6 relay mode if receiving only /64 from an ISP. Maybe a better option.

I disable prefix delegation on wan6 and prefix assignment on lan, then reconfigure wan6, lan, and guest all to use relay mode for RA and DHCPv6. Okay, this gets IPv6 GUAs assigned to my devices on VLANs, finally. But requests to IPv6 hosts still hang.

So, what now? (It'd be really great if the wiki explained clearly what configuration is necessary to set up a basic IPv6 network, since the defaults don't appear to work even with extensive fiddling.)

https://github.com/openwrt/openwrt/issues/9881 does not look good... does IPv6 subnetting of a /64 work in OpenWrt?

Edit: Wtf! https://github.com/openwrt/openwrt/issues/9881#issuecomment-1711820408 is buried, but it is correct, removing the ULA prefix from settings causes connectivity to immediately work! Seems like a bug in OpenWrt then?

Exactly

You can NPT6 or selectively NAT66 your ULA but that is not the way to go.

There must be a good reason IPv6 ULA is on by default but I do not know, I always disable it to avoid confusion.

Make sure you request a /60 IPv6 subnet on WAN6, WAN6 usually itself does not need a /64 so on WAN6 add:
Request IPv6-prefix of length 60 (or try 56)
On advanced tab enable Delegate IPv6 prefixes and set prefix length to disabled
This is my setup requesting a /56 prefix

config interface 'wan6'

	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix '56'
	option peerdns '0'
	option norelease '1'
1 Like