Howto make OpenWrt RFC6724 compliant?

RFC6724 describes two algorithms, one for source address selection and one for destination address selection.

By default, the following policy table should be applied for destination-address selection when multiple alternatives are possible:

Prefix        Precedence Label
::1/128               50     0
::/0                  40     1
::ffff:0:0/96         35     4
2002::/16             30     2
2001::/32              5     5
fc00::/7               3    13
::/96                  1     3
fec0::/10              1    11
3ffe::/16              1    12

In this context, IPv4 addresses are matched with the ::ffff:0:0/96 prefix.

Nevertheless, on an OpenWrt 21.02.0-rc1, as well as on a 19.07.6, with the device having dual-stack Internet connectivity, trying to ping google.com results in OpenWrt using IPv4 by default, where it should be using IPv6.

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (142.250.179.142): 56 data bytes
64 bytes from 142.250.179.142: seq=0 ttl=120 time=15.062 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 15.062/15.062/15.062 ms

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 -6 google.com
PING google.com (2a00:1450:400e:801::200e): 56 data bytes
64 bytes from 2a00:1450:400e:801::200e: seq=0 ttl=119 time=13.799 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 13.799/13.799/13.799 ms

A packet capture running in parrallel confirms the first ping command triggered a DNS query for both IPv4 & IPv6 entries and got a answer to both of them.
The second ping only triggers a query for the IPv6 record, as expected, and shows OpenWrt has working IPv6 connectivity.

┌──[SSH://root@openwrt]──[~]────────
# tcpdump -ni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
21:50:10.073341 IP 127.0.0.1.58926 > 127.0.0.1.53: 43035+ A? google.com. (28)
21:50:10.075040 IP 127.0.0.1.53 > 127.0.0.1.58926: 43035 1/0/0 A 142.250.179.142 (44)
21:50:10.075850 IP 127.0.0.1.58926 > 127.0.0.1.53: 45993+ AAAA? google.com. (28)
21:50:10.077293 IP 127.0.0.1.53 > 127.0.0.1.58926: 45993 1/0/0 AAAA 2a00:1450:400e:801::200e (56)
21:50:12.241738 IP 127.0.0.1.54021 > 127.0.0.1.53: 3640+ AAAA? google.com. (28)
21:50:12.243066 IP 127.0.0.1.53 > 127.0.0.1.54021: 3640 1/0/0 AAAA 2a00:1450:400e:801::200e (56)
^C
6 packets captured
12 packets received by filter
0 packets dropped by kernel

The same behavior is also displayed when doing an opkg update: DNS query shows both A & AAAA records being returned, but OpenWRT defaults once again to IPv4 to connect to the mirror.

# tcpdump -ni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
22:04:53.133567 IP 127.0.0.1.34433 > 127.0.0.1.53: 1424+ A? downloads.openwrt.org. (39)
22:04:53.133704 IP 127.0.0.1.34433 > 127.0.0.1.53: 3717+ AAAA? downloads.openwrt.org. (39)
22:04:53.134797 IP 127.0.0.1.53 > 127.0.0.1.34433: 1424 2/0/0 CNAME mirror-02.infra.openwrt.org., A 168.119.138.211 (96)
22:04:53.136103 IP 127.0.0.1.53 > 127.0.0.1.34433: 3717 2/0/0 CNAME mirror-02.infra.openwrt.org., AAAA 2a01:4f8:251:321::2 (108)

Even if I completely purge all IPv4 IP addresses from the different interfaces, which incidently also purges the IPv4 routing table, OpenWRT still uses IPv4 as default stack, which results in failure to communicate with the remote host, which would be reachable if OpenWrt conformed to RFC6724.

┌──[SSH://root@openwrt]──[~]────────
# for i in $(ip addr | grep inet | grep -v inet6 | grep -v 127.0.0.1 | cut -d " " -f 6); do ip addr del $i dev $(ip addr show | grep $i | cut -d " " -f 11); done

┌──[SSH://root@openwrt]──[~]────────
# ip addr | grep inet | grep -v inet6
    inet 127.0.0.1/8 scope host lo

┌──[SSH://root@openwrt]──[~]────────
# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (172.217.17.46): 56 data bytes
ping: sendto: Network unreachable

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com -6
PING google.com (2a00:1450:400e:808::200e): 56 data bytes
64 bytes from 2a00:1450:400e:808::200e: seq=0 ttl=119 time=13.324 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 13.324/13.324/13.324 ms

┌──[SSH://root@openwrt]──[~]────────
# ip addr del 127.0.0.1 dev lo

┌──[SSH://root@openwrt]──[~]────────
# ip addr | grep inet | grep -v inet6

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (172.217.17.110): 56 data bytes
ping: sendto: Network unreachable

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com -6
PING google.com (2a00:1450:400e:80c::200e): 56 data bytes
64 bytes from 2a00:1450:400e:80c::200e: seq=0 ttl=119 time=13.721 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 13.721/13.721/13.721 ms

With the corresponding packet capture on DNS queries:

┌──[SSH://root@openwrt]──[~]────────
# tcpdump -ni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
09:57:54.897536 IP6 ::1.43623 > ::1.53: 60427+ A? google.com. (28)
09:57:54.899395 IP6 ::1.43623 > ::1.53: 996+ AAAA? google.com. (28)
09:57:54.899822 IP6 ::1.53 > ::1.43623: 60427 1/0/0 A 172.217.168.206 (44)
09:57:54.902244 IP6 ::1.53 > ::1.43623: 996 1/0/0 AAAA 2a00:1450:400e:80d::200e (56)
09:57:56.892449 IP6 ::1.49613 > ::1.53: 50860+ AAAA? google.com. (28)
09:57:56.894695 IP6 ::1.53 > ::1.49613: 50860 1/0/0 AAAA 2a00:1450:400e:80d::200e (56)

I tried configuring a custom policy by creating /etc/gai.conf and filling it.
This wasn't documented anywhere in the OpenWRT documentation but seems to be the default location to configure the policy on linux systems, so I thought it'd be worth a shot...
Unfortunately, this didn't lead to any behaviour change.

┌──[SSH://root@openwrt]──[~]────────
# cat /etc/gai.conf
label  ::1/128       0
label  ::/0          1
label  2002::/16     2
label ::/96          3
label ::ffff:0:0/96  4
precedence  ::1/128       50
precedence  ::/0          40
precedence  2002::/16     30
precedence ::/96          20
precedence ::ffff:0:0/96  10

I digged in the OpenWRT documentation (wiki & forum), but couln't find a way to display openwrt default policy nor a way to change it.

Any idea as to wether or not this is possible in current builds ?

That's not the case in my system:

root@magiatiko / > ping -c 1 google.com
PING google.com(prg03s06-in-x0e.1e100.net (2a00:1450:4014:80d::200e)) 56 data bytes
64 bytes from prg03s06-in-x0e.1e100.net (2a00:1450:4014:80d::200e): icmp_seq=1 ttl=57 time=5.14 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.137/5.137/5.137/0.000 ms

root@magiatiko / > tcpdump -ni lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
08:26:21.262583 IP 127.0.0.1.58534 > 127.0.0.1.53: 42102+ A? google.com. (28)
08:26:21.262672 IP6 ::1.58534 > ::1.53: 42102+ A? google.com. (28)
08:26:21.262707 IP 127.0.0.1.58534 > 127.0.0.1.53: 42768+ AAAA? google.com. (28)
08:26:21.262731 IP6 ::1.58534 > ::1.53: 42768+ AAAA? google.com. (28)
08:26:21.262736 IP 127.0.0.1.53 > 127.0.0.1.58534: 42102 1/0/0 A 172.217.23.238 (44)
08:26:21.262814 IP 127.0.0.1.53 > 127.0.0.1.58534: 42768 1/0/0 AAAA 2a00:1450:4014:80d::200e (56)
2 Likes
/etc/gai.conf
/etc/nsswitch.conf

Those configs apply to glibc, but not musl which OpenWrt is build on:

# ldd $(readlink -f $(type -p ping))
	/lib/ld-musl-x86_64.so.1 (0x7f0739ed2000)
...

The priority may also depend on specific app:

# wget -O /dev/null https://openwrt.org/
Downloading 'https://openwrt.org/'
Connecting to 2a03:b0c0:3:d0::1af1:1:443
...
1 Like

Thanks for taking the time to take a look at this.

TLDR current issue status:

  • ICMP probes: defaults to IPv4
  • TCP outbound connections: defaults to IPv6 (even when it should not)

Long version:

Based on your feedback, I checked again the 3 devices I have at hand, all of them sharing a very similar configuration:

  1. GL.iNet GL-B1300 - ipq40xx/generic - arm_cortex-a7_neon-vfpv4 - OpenWrt 21.02.0-rc1 r16046 - custom firewall config
  2. NetGear WNDR3700 - ar71xx/generic - mips_24kc - OpenWrt 19.07.6 r11278 - stock firewall config
  3. NetGear WNDR4300 - ar71xx/nand - mips_24kc - OpenWrt 19.07.6 r11278 - stock firewall config

After re-checking the B1300 custom ip6tables config, I noticed had no rule allowing outgoing IPv6 tcp traffic, hence the device falling back to IPv4. Duh!... Changing this solved the issue for the outbound TCP connections which now use IPv6 overall by default (using opkg update or uclient-fetch). Re-cheking it, outbound TCP connection were defaulting to IPv6 on devices 2 & 3.

But with the ping command, the issue persist on all three devices. (it briefly seemed OK on device 3, but when I forced a ping both in IPv4 and in IPv6, the subsequent attemps always used IPv4).

┌──[SSH://root@openwrt]──[~]────────
# uclient-fetch -O /dev/null https://google.com/
Downloading 'https://google.com/'
Connecting to 2a00:1450:400e:80c::200e:443
Redirected to / on www.google.com
Writing to '/dev/null'

Download completed (13808 bytes)

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com -4
PING google.com (172.217.168.238): 56 data bytes
64 bytes from 172.217.168.238: seq=0 ttl=118 time=12.527 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 12.527/12.527/12.527 ms

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com -6
PING google.com (2a00:1450:400e:80d::200e): 56 data bytes
64 bytes from 2a00:1450:400e:80d::200e: seq=0 ttl=117 time=13.343 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 13.343/13.343/13.343 ms

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (172.217.168.238): 56 data bytes
64 bytes from 172.217.168.238: seq=0 ttl=118 time=12.886 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 12.886/12.886/12.886 ms

@trendy Can you confirm it's never the case even after forcing an IPv4 ping at least once ?

# ping -c 1 google.com -4
# ping -c 1 google.com -6
# ping -c 1 google.com

@vgaetera Thanks for pointing out why /etc/gai.conf had no effect :slight_smile:
So there is currently no mechanism in OpenWRT to configure the destination-address selection policy ?

I did a last test trying to connect to an internal device whose DNS record returns both an IPv4 address and an IPv6 ULA address (prefix fc00::/7). As per the RFC6724, in this case, IPv4 should have precedence over IPv6, but OpenWrt seems to behave as per the depreciated FRC3484.
In this particular case, it's the behaviour I wanted to configure, so my specific issue is "solved". But this should be configurable if one seeks to adapt this behaviour...

# uclient-fetch -O /dev/null http://internal-host.localnetwork.lan/
Downloading 'http://internal-host.localnetwork.lan/'
Connecting to fd56:6d3f:456a:7ef9:55::1:80
Writing to '/dev/null'
Download completed (522 bytes)

I still can't wrap my head about the different behaviour between ICMP & TCP connections... I doesn't make much sense that would be app-specific.

2 Likes

I'm afraid this requires to study the source code for musl/getaddrinfo.
Actually, there are similar know issues for incorrect IP priority choice.
E.g. collectd prefers IPv6 for ICMP when IPv6 connectivity is missing:
Collectd network plugin getaddrinfo failed System error - #7 by vgaetera

1 Like
root@magiatiko / > ping -4 -c 1 google.com
ping -c 1 google.com
PING google.com (216.58.201.78) 56(84) bytes of data.
64 bytes from prg03s01-in-f78.1e100.net (216.58.201.78): icmp_seq=1 ttl=57 time=5.11 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.109/5.109/5.109/0.000 ms
root@magiatiko / > ping -6 -c 1 google.com
PING google.com(prg03s01-in-x0e.1e100.net (2a00:1450:4014:800::200e)) 56 data bytes
64 bytes from prg03s01-in-x0e.1e100.net (2a00:1450:4014:800::200e): icmp_seq=1 ttl=57 time=5.55 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.550/5.550/5.550/0.000 ms
root@magiatiko / > ping -c 1 google.com
PING google.com(prg03s01-in-x0e.1e100.net (2a00:1450:4014:800::200e)) 56 data bytes
64 bytes from prg03s01-in-x0e.1e100.net (2a00:1450:4014:800::200e): icmp_seq=1 ttl=57 time=5.27 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 5.272/5.272/5.272/0.000 ms

root@magiatiko / > ping -V
ping from iputils 20210202
1 Like

I think there has been a recent fix here:

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=7fea9d9f5dd282a7049d77cc6b75e0a703ead26c

1 Like

TLDR current issue status:

  • busybox ping command always defaults to IPv4 when both A & AAAA records are returned by DNS; will be solved by a recent commit on busybox
  • iputils ping & outgoing TCP connections behave as per the policy defined in deprecated RFC3084 instead of the new policy defined in RFC6724; as mentioned by @vgaetera this seems to be an issue with musl/getaddrinfo implementation. I'll submit a bug report for that issue.

Long version:

@trendy Here's where the difference comes from !

┌──[SSH://root@openwrt]──[~]────────
# ping -v
BusyBox v1.33.0 () multi-call binary.

<--- output omitted for brevity --->

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (172.217.168.206): 56 data bytes
64 bytes from 172.217.168.206: seq=0 ttl=120 time=15.077 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 15.077/15.077/15.077 ms

┌──[SSH://root@openwrt]──[~]────────
# opkg install iputils-ping
Installing iputils-ping (20210202-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc1/packages/arm_cortex-a7_neon-vfpv4/packages/iputils-ping_20210202-1_arm_cortex-a7_neon-vfpv4.ipk
Configuring iputils-ping.

┌──[SSH://root@openwrt]──[~]────────
# ping -V
ping from iputils 20210202

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com(ams15s33-in-x0e.1e100.net (2a00:1450:400e:80a::200e)) 56 data bytes
64 bytes from ams15s33-in-x0e.1e100.net (2a00:1450:400e:80a::200e): icmp_seq=1 ttl=119 time=12.7 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 12.656/12.656/12.656/0.000 ms

┌──[SSH://root@openwrt]──[~]────────
# opkg remove iputils-ping
Removing package iputils-ping from root...

┌──[SSH://root@openwrt]──[~]────────
# ping -V
ping: unrecognized option: V
BusyBox v1.33.0 () multi-call binary.

<--- output omitted for brevity --->

┌──[SSH://root@openwrt]──[~]────────
# ping -c 1 google.com
PING google.com (172.217.168.206): 56 data bytes
64 bytes from 172.217.168.206: seq=0 ttl=120 time=13.890 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 13.890/13.890/13.890 ms

So the incoherent ping behavior is indeed a busybox related issue. We're making progress :slight_smile:

I'll fill in a bug report for the lack of compliance with RFC6724:

  • OpenWrt implements the deprecated RFC 3484
  • OpenWrt lacks a mechanism for allowing administrators to provide policy that can override the default behavior

I suppose it's not necessary to report anything for the busybox ip utilities implementation given the commit mentioned by @ParanoidZoid, which will hopefully sooner or later make its way into the official builds.

:+1: Big thanks to everybody involved in clarifying the issue!

3 Likes

Bug report available here:
https://bugs.openwrt.org/index.php?do=details&task_id=3813

2 Likes