IPv6 UDP (OpenVPN) packets from LAN PD IPv6 address to WAN not forwarding

I've set up an OpenVPN client on my OpenWrt (23.05.3) router (Netgear WAX206).

The OpenVPN server (which I run myself on a VPS) is dual stack, and its ("default"/preferable) DNS hostname entry also resolves both A and AAAA records:

  • <vpn-server-ipv4>
  • <vpn-server-ipv6>

The OpenVPN client, upon fresh start, seems to prefer <vpn-server-ipv6> which should be fine. However I've noticed a strange issue with UDP oVPN packets from the client not being forwarded out the WAN interface after the tunnel is established, predictably leading to connections/traffic failing and eventually OpenVPN's ping timeout kicking in. Eventually the OpenVPN client gives up on <vpn-server-ipv6>, re-establishes the tunnel using <vpn-server-ipv4> for the link. After that point things work correctly.

I could of course restrict the client to use <vpn-server-ipv4> address but since this seems to have uncovered a forwarding issue with IPv6, I'd like to get to its root cause and fix it if possible.

Router address allocation

My ISP (Comcast/Xfinity) assigns the router's WAN interface:

  • One /21 IPv4 address: <wan-ipv4>
  • One /128 IPv6 address: <wan-ipv6>
  • One /60 PD block: <common prefix>:xxx0::0/60>

Out of that /60 PD, I delegate one distinct /64 each to the br-lan, devices and guest interfaces/networks:

  • <common prefix>:xxx0::0/64 to br-lan
    router interface on <common prefix>:xxx0::1

  • <common prefix>:xxx1::0/64 to devices
    router interface on <common prefix>:xxx1::1

  • <common prefix>:xxx2::0/64 to guest
    router interface on <common prefix>:xxx2::1

Devices on those networks have IPv6 connectivity with the Internet without issues. This includes OpenVPN traffic to the same server when the (non-router) device is the VPN client.

Steps that reproduce the problem

  • Configure the router OpenVPN client to use <vpn-server-ipv6>:

    config openvpn 'myvpn'
    	option nobind '1'
    	option float '1'
    	option client '1'
    	option reneg_sec '0'
    	option dev 'tun'
    	option verb '3'
    	option persist_tun '1'
    	option persist_key '1'
    	option remote_cert_tls 'server'
    	list remote '<vpn-server-ipv6>'
    	option ca '/etc/openvpn/root_ca_cert.pem'
    	option cert '/etc/openvpn/client_cert.pem'
    	option key '/etc/openvpn/client_key.pem'
    	option cipher 'AES-256-GCM'
    	option proto 'udp'
    	list pull_filter 'ignore redirect-gateway'
    	list pull_filter 'ignore dhcp-option'
    	option enabled '1'
    	option ping '30'
    

    (I've added the two pull_filters because although the server pushes a default route and DNS, I don't want this particular client - the router - to use those.)

  • Start the VPN client on the router. During this traffic flows on the WAN as expected.

    # tcpdump -ni wan host <vpn-server-ipv6>
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on wan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    13:09:15.045697 IP6 <common prefix>:xxx0::1.35001 > <vpn-server-ipv6>.1194: UDP, length 14
    13:09:15.057996 IP6 <vpn-server-ipv6>.1194 > <common prefix>:xxx0::1.35001 UDP, length 26
    ...
    

    Note here that, presumably due to the nobind, openvpn is using an all-addresses socket. This doesn't stop the bidirectional IPv6 traffic needed to establish the tunnel as shown by the packet trace above.

    # netstat -nlpu |grep openvpn
    udp        0      0 :::35001                :::*                                6266/openvpn
    

    For whatever reason, it always chooses br-lan's interface address as the source address to communicate with the server. This is fine during the setup phase, and also seems to cause the conntrack entry to be added:

    # grep 1194 /proc/net/nf_conntrack

    ipv6 10 udp 17 167 src=<common prefix>:xxx0:0000:0000:0000:0001 dst=<vpn-server-ipv6> sport=35001 dport=1194 packets=9 bytes=3162 src=<vpn-server-ipv6> dst=<common prefix>:xxx0:0000:0000:0000:0001 sport=1194 dport=35001 packets=11 bytes=5843 [ASSURED] mark=0 zone=0 use=2

    (Edit: I think the reason br-lan's address is chosen is because the delegated prefix (<common prefix>:xxx0::0/60) has a longer match with <vpn-server-ipv6> than <wan-ipv6> does. ip -6 r shows two default IPv6 routes, one each for <common prefix>:xxx0::0/60 and <vpn-server-ipv6>, so longest match wins. Anyway that "should" not break anything since traffic has already flowed successfully using this addresses/ports association and conntrack has picked it up.)

  • The tunnel interface is up:

    # ip -4 a s dev tun0
    67: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN qlen 500
        inet 10.42.1.6 peer 10.42.1.5/32 scope global tun0
           valid_lft forever preferred_lft forever
    
  • Ping the router's tun0 address from the VPN server:

    # ping 10.42.1.6
    PING 10.42.1.6 (10.42.1.6) 56(84) bytes of data.
    ^C
    --- 10.42.1.6 ping statistics ---
    5 packets transmitted, 0 received, 100% packet loss, time 4113ms
    
  • The echo packets were arriving on the router and being responded to:

    # tcpdump -ni tun0 icmp
    tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on tun0, link-type RAW (Raw IP), snapshot length 262144 bytes
    13:26:56.328377 IP 10.42.1.1 > 10.42.1.6: ICMP echo request, id 8, seq 1, length 64
    13:26:56.328568 IP 10.42.1.6 > 10.42.1.1: ICMP echo reply, id 8, seq 1, length 64
    13:26:57.368835 IP 10.42.1.1 > 10.42.1.6: ICMP echo request, id 8, seq 2, length 64
    13:26:57.368970 IP 10.42.1.6 > 10.42.1.1: ICMP echo reply, id 8, seq 2, length 64
    13:26:58.392902 IP 10.42.1.1 > 10.42.1.6: ICMP echo request, id 8, seq 3, length 64
    13:26:58.393040 IP 10.42.1.6 > 10.42.1.1: ICMP echo reply, id 8, seq 3, length 64
    13:26:59.416879 IP 10.42.1.1 > 10.42.1.6: ICMP echo request, id 8, seq 4, length 64
    13:26:59.417015 IP 10.42.1.6 > 10.42.1.1: ICMP echo reply, id 8, seq 4, length 64
    13:27:00.440928 IP 10.42.1.1 > 10.42.1.6: ICMP echo request, id 8, seq 5, length 64
    13:27:00.441065 IP 10.42.1.6 > 10.42.1.1: ICMP echo reply, id 8, seq 5, length 64
    
  • But evidently the VPN link traffic was not making it out of the WAN interface, despite the same association tuple having worked during VPN setup and having the conntrack entry above. Only incoming VPN transport is seen on the WAN:

    # tcpdump -ni wan host <vpn-server-ipv6> tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on wan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    13:26:56.327978 IP6 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
    listening on wan, link-type EN10MB (Ethernet), snapshot length 262144 bytes
    13:26:56.327978 IP6 <vpn-server-ipv6>.1194 > <common prefix>:xxx0::1.35001: UDP, length 108
    13:26:57.368463 IP6 <vpn-server-ipv6>.1194 > <common prefix>:xxx0::1.35001: UDP, length 108
    13:26:58.392526 IP6 <vpn-server-ipv6>.1194 > <common prefix>:xxx0::1.35001: UDP, length 108
    ...
    

This problem goes away if I change the remote server entry in /etc/config/openvpn to <vpn-server-ipv4>.

It also goes away if I disable nobind and bind to <wan-ipv6, but that's even less desirable/robust in case of the WAN address being changed from the ISP side etc.


So, any insight/pointers would be very much appreciated here.

It seems that I had source routing (filtering?) enabled on the WAN6 interface. Disabling it:

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix '56'
	option sourcefilter '0'

fixes this.

I notice that now the IPv6 traffic out of the router is using its <wan-ipv6> /128 address, instead of the PD.

For better understanding and posterity, I'll leave this thread open as long as discourse lets it to see if someone could add any pointers to "upstream" material: Why did this problem occur?

Had more or less the same problem with WireGuard I also had to disable source routing

I want to say the difference in behavior between tunnel setup (worked) and regular traffic (didn't) when source routing was enabled is probably related to openvpn changing socket options after setup? But only a guess at this point.