IPv6 upstream disappears after 30 minutes

fischerman · January 19, 2023, 12:48pm

I'm having trouble with my IPv6 WAN connection. When I start the device I see a "IPv6 Upstream" and IPv6 is working on end devices.

After about 30 minutes the "IPv6 Upstream" disappears. End devices now receive "no route to destination" from the router.

However, the WAN_6 interface is still up.

Rebooting the device brings back IPv6. What might be causing this issue? I don't see anything in the logs.

Here is the config

# uci export network; uci export dhcp
package network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd4a:ba87:0b21::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'
	list dns '8.8.8.8'
	list dns '1.1.1.1'

config device
	option name 'eth0'

config interface 'wan'
	option proto 'pppoe'
	option username '...'
	option password '...'
	option mtu '1492'
	option device 'eth0.2'
	option ipv6 'auto'

config device
	option type '8021q'
	option ifname 'eth0'
	option vid '2'
	option name 'eth0.2'

config device
	option name 'pppoe-wan'
	option type 'tunnel'

config device
	option name 'pppoe-wan'
	option type 'tunnel'

config device
	option type '8021q'
	option ifname 'br-lan'
	option vid '8'
	option name 'pferde'
	option ipv6 '0'

config interface 'Pferde'
	option proto 'static'
	option device 'pferde'
	option ipaddr '192.168.3.1'
	option netmask '255.255.255.0'

package dhcp

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

config dhcp 'Pferde'
	option interface 'Pferde'
	option start '100'
	option limit '150'
	option leasetime '12h'

Target Platform	bcm27xx/bcm2711
Firmware Version	OpenWrt 22.03.3 r20028-43d71ad93e / LuCI openwrt-22.03 branch git-22.361.69894-438c598
Kernel Version	5.10.161

I'm dealing with other problems as well, but I think they are unrelated: DNS responds on wrong ports

trendy · January 19, 2023, 1:59pm

uci export network ?

fischerman · January 19, 2023, 2:05pm

See above.

trendy · January 19, 2023, 2:21pm

Sorry, I wanted to write firewall.

mk24 · January 19, 2023, 2:23pm

Does IPv4 still work, i.e. from the router can you ping -4 some site?
It kind of looks like the ISP dropped / didn't renew your lease, though I'm not sure how that works in a dual stack.
There is a note in the wiki about needing to open ports 546 and 547 through wan_6 so that lease renewal can be processed. wan_6 should be added to the wan firewall zone, by default there is wan6 which is not the same.

fischerman · January 19, 2023, 3:46pm

It is always 30 minutes. I honestly don't understand this wan6 vs wan_6.

# uci export firewall
package firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'Pferde'
	list network 'lan'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'
	option name 'Allow-DHCPv6 (547-to-546)'

config rule
	option name 'Allow DHCPv6 (546-to-547)'
	list proto 'udp'
	option src 'wan'
	option src_port '546'
	option dest_port '547'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

fischerman · January 19, 2023, 3:49pm

IPv4 continues to work fine. Ping 6 from router to internet stops working after 30 minutes.

mk24 · January 19, 2023, 4:08pm

wan6 is part of the default configuration for cases where the ISP is pure DHCP. wan_6 is spawned by pppoe to handle the v6 part of the dual stack. The two names are not the same so you need to add wan_6 directly to the firewall so your IPv6 connection is properly firewalled. This should cause "WAN_6" to show as red the same as "WAN" on the LuCI interface summary page. Note that it is only LuCI upcasing the names when it displays them (a feature I really don't like). Continue to use lowercase internally.

trendy · January 19, 2023, 4:20pm

This one is not needed and wrong, as it accepts from dhcp6 client from wan to send to dhcp6 server on OpenWrt. Did you add it or it was there?
Try also what @mk24 mentioned.

fischerman · January 19, 2023, 4:30pm

I only had one rule earlier. But in the docs there are two. So I added it (https://openwrt.org/docs/guide-user/network/ipv6/configuration#ppp-based_protocols_and_option_ipv6). Are the docs wrong?

I added wan_6 to zone wan. It takes 30 minutes to test. Will let you know in 17 minutes

BTW I looked at the WAN traffic (in front of the router) and there was nothing on 'udp.port == 546' after 30 minutes (but on reboot).

fischerman · January 19, 2023, 4:52pm

Adding wan_6 to wan didn't help. IPv6 still disappears.

Around that time there was nothing on the WAN.

Compare that to after boot.

Is 30 minutes a well known timeout for something? Seems very specific. What surprises me is that I'm reading zip in the logs.

mk24 · January 19, 2023, 5:55pm

ifstatus wan_6 should have life numbers that count down. The initial lifetime is received from the ISP. Then your client is supposed to ask for a renewal. This is much like IPv4.

trendy · January 19, 2023, 6:27pm

Yes. I'll update them.

If your lease time is 30 minutes, at 15 minutes the client will send a renewal.
In my case I have vltime 300 = 5 minutes, so every 2,5 minutes there is a renewal.

19:19:07.262395 IP6 (flowlabel 0x4eebc, hlim 1, next-header UDP (17) payload length: 133) fe80::ec8d:adfe:f539:c497.546 > ff02::1:2.547: [udp sum ok] dhcp6 renew (xid=135603 (elapsed-time 0) (option-request SIP-servers-domain SIP-servers-address DNS-server DNS-search-list SNTP-servers NTP-server AFTR-Name opt_67 opt_94 opt_95 opt_96) (client-ID hwaddr type 1 001e101f0000) (server-ID hwaddr type 1 6c3b6bee0d46) (Client-FQDN) (IA_PD IAID:1 T1:0 T2:0 (IA_PD-prefix 2001:0001:0002:7500::/56 pltime:0 vltime:0)))
19:19:07.265934 IP6 (hlim 64, next-header UDP (17) payload length: 121) fe80::30.547 > fe80::ec8d:adfe:f539:c497.546: [udp sum ok] dhcp6 reply (xid=135603 (client-ID hwaddr type 1 001e101f0000) (server-ID hwaddr type 1 6c3b6bee0d46) (DNS-server 2001:4860:4860::8888 2001:4860:4860::8844) (IA_PD IAID:1 T1:150 T2:240 (IA_PD-prefix 2001:0001:0002:7500::/56 pltime:270 vltime:300)))
19:21:37.278389 IP6 (flowlabel 0x4eebc, hlim 1, next-header UDP (17) payload length: 133) fe80::ec8d:adfe:f539:c497.546 > ff02::1:2.547: [udp sum ok] dhcp6 renew (xid=6be773 (elapsed-time 0) (option-request SIP-servers-domain SIP-servers-address DNS-server DNS-search-list SNTP-servers NTP-server AFTR-Name opt_67 opt_94 opt_95 opt_96) (client-ID hwaddr type 1 001e101f0000) (server-ID hwaddr type 1 6c3b6bee0d46) (Client-FQDN) (IA_PD IAID:1 T1:0 T2:0 (IA_PD-prefix 2001:0001:0002:7500::/56 pltime:0 vltime:0)))
19:21:37.301770 IP6 (hlim 64, next-header UDP (17) payload length: 121) fe80::30.547 > fe80::ec8d:adfe:f539:c497.546: [udp sum ok] dhcp6 reply (xid=6be773 (client-ID hwaddr type 1 001e101f0000) (server-ID hwaddr type 1 6c3b6bee0d46) (DNS-server 2001:4860:4860::8888 2001:4860:4860::8844) (IA_PD IAID:1 T1:150 T2:240 (IA_PD-prefix 2001:0001:0002:7500::/56 pltime:270 vltime:300)))
^C

fischerman · April 5, 2023, 2:32pm

The problem is that the routes expire:

	"route": [
		{
			"target": "::",
			"mask": 0,
			"nexthop": "fe80::...",
			"metric": 512,
			"valid": 1697,
			"source": "2a02:...::/56"
		},
		{
			"target": "::",
			"mask": 0,
			"nexthop": "fe80::...",
			"metric": 512,
			"valid": 1697,
			"source": "2a02:.../64"
		}
	],

After 30 minutes (when valid hits 0), the routes disappear. After the initial DHCPv6 request, I don't see any renew requests on the wire.

"route": [
		
],

When you say client, you mean OpenWRT, right? It seems that my OpenWRT doesn't send out a renew request. Are there any configuration options I should be looking at? Any other ideas?

trendy · April 5, 2023, 5:17pm

Yes.

No, it is required by the protocol definition to work like this. Check the logs for dropped packets.

fischerman · April 6, 2023, 9:02am

After the interface is up and beyond when the routes disappear, the log (using logread) is completely empty (except for clients connecting, ssh and stuff). Where should I be looking for logs?

Also I've never seen a single log message from odhcp6c, not even when the interface comes up.

logread | grep odhcp6c
# nothing

According to this thread there should be log messages: https://github.com/openwrt/openwrt/issues/6474

Can I change the log level of odhcp6c? I only found an option to change the log level for odhcpd.

trendy · April 6, 2023, 11:27am

Get logs and it would be also useful to capture dhcp6 as well.

fischerman · April 6, 2023, 3:11pm

I don't see any renew messages in the capture unless I trigger them by hand using killall -SIGUSR1 udhcpc. However, even that doesn't bring back IPv6.

When I restart the interface I get a prefix with a very high validity >86334 and routes with a lower validity 1800 (i.e. 30 minutes). When I renew manually using killall -SIGUSR1 udhcpc, then the validity of the prefix increases but the validity of the routes stay the same (until they eventually expire). Running killall -SIGUSR1 udhcpc afterwards doesn't bring back the routes. My suspicion is that odhcp6c only renews its lease based on the prefix but not the routes.

Is it normal for the routes to have a lower validity? What defines the lifetime of the routes?

trendy · April 6, 2023, 10:15pm

It's odhcp6c not udhcpc, the latter is for DHCPv4.

I have a higher router validity than routes.

The Router Advertisement. Better capture that too using filter

ip6[40] == 133 or ip6[40] == 134

fischerman · April 11, 2023, 7:31am

Sorry, copy&paste error when I was writing the post. I did use odhcp6c. But DHCP isn't the problem as this is route related.

I captured Router Advertisement. I get an initial RA after RS. But after that, no unsolicited RA's reach my router. So the problems seems to be on the ISP side (again). After I knew what to look for I found similar issues. Seems to be not that uncommon.

I'm waiting to hear back from my ISP. My neighbors use an ISP-provided FritzBox. They don't experience the problem. I'm wondering what a FritzBox does different. Maybe they send RS periodically (violating RFC4861 6.3.7.).