MacOS 10.15.1 is ignoring RA route lifetime for IPv6 (OpenWrt 18.06.4)?

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi folks. I'm seeing odd IPv6 behaviour on my iMac, which is not the same as other clients on my LAN, so this appears to be macOS specific and not an OpenWRT issue, but I'm hoping to get some advice.

Scenario:
Set a constant ping running (ping6 google.com)
No output for several minutes, then 9 or 10 successful pings, then no output for several minutes again. Lather, rinse, repeat.

Diagnostics: (See pastebin folder as I can only add two links - noob!)
Wireshark capture filter for "icmp or icmp6" on my Mac's Ethernet port en0. (Captured pcapng file 33kBytes)
Capture output of "ifconfig -a inet6" and "netstat -f inet6 -rn" when there's no ping response, and again when the pings are coming back.

iMac-2017:Desktop michthom$ diff -u ifconfig_netstat_missing_route.txt ifconfig_netstat_route_present.txt
--- ifconfig_netstat_missing_route.txt	2019-11-06 23:38:31.000000000 +0000
+++ ifconfig_netstat_route_present.txt	2019-11-07 00:31:02.000000000 +0000
@@ -80,7 +80,7 @@
 fe80::1%lo0                             link#1                          UHLI            lo0       
 fe80::%en0/64                           link#4                          UCI             en0       
 fe80::1412:cb7b:1ccd:13c2%en0           98:10:e8:f3:d7:ef               UHLI            lo0       
-fe80::42f2:1ff:fe0b:c552%en0            link#4                          UHLWIir         en0       
+fe80::42f2:1ff:fe0b:c552%en0            40:f2:1:b:c5:52                 UHLWIir         en0       
 fe80::%en1/64                           link#5                          UCI             en1       
 fe80::2a:2c75:ee9c:47fc%en1             64:70:33:d4:90:8e               UHLWIi          en1       
 fe80::458:c576:bce3:72e5%en1            74:8d:8:5:b2:41                 UHLWIi          en1

Analysis:
While there's no route, no pings are sent.
When the RA timer expires, OpenWRT sends a new RA packet, which triggers the Mac to create the route, via the OpenWRT MAC address.

ICMPv6 Option (Route Information : Medium 2a00:23c4:6d99:8800::/56)
    Type: Route Information (24)
    Length: 3 (24 bytes)
    Prefix Length: 56
    Flag: 0x00, Route Preference: Medium
        ...0 0... = Route Preference: Medium (0)
        000. .000 = Reserved: 0
    Route Lifetime: 8589751
    Prefix: 2a00:23c4:6d99:8800::

I then see successful pings being sent and replied received.
Approximately 10 seconds later the pings stop because the route reverts to the "link#4"

Assumption:
macOS is being irritating, accepting the RA but messing up the route lifetime (8589751), and dropping it way too fast?

Question:
Apart from disabling IPv6, do I have any options in OpenWRT to force the Mac to be a better network citizen? Or am I misreading the whole situation?

1 Like
2

I've submitted a feedback report to Apple under FB7431553 - doubt it'll even get noticed!

1 Like
3

What version of macOS are you using? Catalina?
I have a similar setup and I don't see the problem in my MacBook pro.

4

In fact, I had similar problems with Windows (not macOS) and, in it's time, I remember to change the configuration of the lan DHCP. I don't remember exactly what I did but it seems to work good. Maybe this can help:

config dhcp 'lan'
option dhcpv4 'server'
option dhcpv6 'server'
option domain 'lan'
option leasetime '24h'
option interface 'lan'
option ra 'server'
option ra_management '1'
option ra_useleasetime '1'
option dhcpv4_forcereconf '1'
option force '1'
option limit '251'
option ra_preference 'high'
option start '2'
5

Thanks for the pointer - that may be just what I need, and I'll test later today then report back. ra_useleasetime feels quite hopeful, ra_preference more mysterious. I'm running Catalina 10.15.1.

Currently /etc/config/dhcp contains only these DHCP options (amongst other blocks):

config dhcp 'lan'
	option interface 'lan'
	option leasetime '12h'
	option dhcpv6 'server'
	option ra 'server'
	option ra_management '1'
	option start '150'
	option limit '100'
6

option ra_useleasetime '1'
is not the answer in my case, but
option ra_preference 'high'
DOES seem to have improved matters considerably. I'll monitor for a bit and see if it's a true fix.

1 Like
7

I'm happy that the ra_preference 'high' setting seems to have fixed the route flapping.
I'm not yet fully understanding why that affected my iMac but not my Raspberry Pi devices? The IPv6 network has been stable overnight and through any IPv6 test traffic I've used.
Thanks to @NoTengoBattery for your swift and useful contribution!

8

I'm in the same position as you. I don't understand a thing but I was pretty sure that those settings would make a difference. Only a Windows laptop went nuts in my case, all other clients were working just fine

Anyway, you should probably mark your question as "Solved". Because I was more guessing than answering, your own post is your solution.

9

Bah humbug. Spoke too soon. The old behaviour is back, even with the new settings on the OpenWRT end still in place. Any other theories would be welcomed...

10

Does ifstatus lan show anything around preferred and valid lifetimes? From my limited understanding it sounds like macOS is only processing the valid lifetime and considers the route stale as soon as it got it.. perhaps try testing a range of different route lifetimes (eg. 1min, 5, 10mins ) and compare what the route table looks like from macOS devices and any others you have for comparison (eg watch -n 1 route

11

(facepalm) It seems I managed to commit the uci change to increase the route preference to high, but forgot to actually restart the dhcp process previously... now done, so I'm back to seeing if that worked first...

Using a Wireshark capture filter of "icmp or icmp6", and looking at only the IPv6 Router Advertisements with a display filter of "icmpv6.type==134" I now see:

    ICMPv6 Option (Prefix information : 2a00:23c4:6d9a:2d00::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
        Valid Lifetime: 43200
        Preferred Lifetime: 43200
        Reserved
        Prefix: 2a00:23c4:6d9a:2d00::
    ICMPv6 Option (Prefix information : fde6:9508:a013::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
        Valid Lifetime: 43200
        Preferred Lifetime: 43200
        Reserved
        Prefix: fde6:9508:a013::
    ICMPv6 Option (Route Information : High 2a00:23c4:6d9a:2d00::/56)
        Type: Route Information (24)
        Length: 3 (24 bytes)
        Prefix Length: 56
        Flag: 0x08, Route Preference: High
        Route Lifetime: 8589796
        Prefix: 2a00:23c4:6d9a:2d00::
    ICMPv6 Option (Route Information : High fde6:9508:a013::/48)
        Type: Route Information (24)
        Length: 3 (24 bytes)
        Prefix Length: 48
        Flag: 0x08, Route Preference: High
        Route Lifetime: Infinity (4294967295)
        Prefix: fde6:9508:a013::

The Prefix Information lifetimes look sensible (43200 = 12 hours for both Valid and Preferred Lifetime, matching the DHCP lease time setting). But the Route Information options (preference now set to High, yay!) carry lifetimes of 99+ days and "infinity" respectively... I had hoped that the ra_useleasetime option would also set the route lifetime, but that clearly isn't how this works.

I will try an experiment with ra_reachabletime next, and report back.

1 Like
12

Well, ra_reachabletime seems to have no effect on the Route Lifetime settings within each Route Information block, but it has altered the overall RA message options.

When I was having problems, the top of the IPv6 RA message included:

    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0x9448 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0xc8, Managed address configuration, Other configuration, Prf (Default
Router Preference): High
    Router lifetime (s): 1800
    Reachable time (ms): 0
    Retrans timer (ms): 0

But with the recent updates I'm now seeing:

Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xfe63 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0xc8, Managed address configuration, Other configuration, Prf (Default Router Preference): High
    Router lifetime (s): 1800
    Reachable time (ms): 360000
    Retrans timer (ms): 0

Which seems more sensible, but I guess that changing the ra_reachabletime to 1800000 (to match the Router lifetime) might be more sensible? I think I intended to set it to 3600000 before and missed a zero!

13

Having read RFC4861 or tried to, the ra_reachabletime shouldn't be relevant here and setting it to the same as the router lifetime is likely a bad move. When set to zero it should be benign, simply saying that the router isn't making any promises on behalf of clients:

      Reachable Time 32-bit unsigned integer.  The time, in
                     milliseconds, that a node assumes a neighbor is
                     reachable after having received a reachability
                     confirmation.  Used by the Neighbor Unreachability
                     Detection algorithm (see Section 7.3).  A value of
                     zero means unspecified (by this router).

So I'm still puzzled as to why MacOS dropped its default IPv6 route in the first place, and recreated it for only 10 seconds when it got a route advertisement from the OpenWRT box.

14

Stable interoperable software needs to be

Be strict in what you send, be generous in what you accept.

If none of your other devices are having trouble, it’s likely a bug in macOS.

15

Hi @Sparks - I'm convinced this is a macOS (and possibly iOS) oddity as it seem not to be affecting other clients on my LAN. Today's testing showed the Mac was stable while it stayed awake (12 hours with no problems at all) but after I let it sleep the old problem returned after it woke, even with the suggestions above still in place).

I originally reported here in case I was missing an OpenWRT directive that would sort the Mac out - I'm now wondering if it's a Catalina 10.15.1 bug that I can't work around with OpenWRT, or something else entirely.

Despite what I asserted above, nettop shows the default route is still in place, even when ping packets are not being sent, which I find very odd. Under that is the Google address, which blinks in and out every few seconds:

default -> en0 -> fe80::42f2:1ff:fe0b:c552%en0
  2a00:1450:4009:809::200e

I don't wish to waste everyone's time, so will quit spamming this thread, but I'll come back if I find a permanent fix or if anyone can suggest a better way to diagnose this than eyeballing the nettop output - Ideally I'd rather get a log of what routes are being added / removed from the routing table with a timestamp, but I don't know how to achieve that.

16

No you’re fine- while I don’t yet have to deal with ipv6 (yet) I’m sure this will be helpful to a few people in future (maybe even me)..

Regarding monitoring/logging give this a try:

nettop -m route -s 5 -l 17280 | tee -a logfile.txt

It’s supposed to only show deltas/changes , maybe I’ve mucked up the order , but it should give you an idea. Looks like you can also filter out columns you don’t need, and even export to csv (-L) if you prefer. 17280 is just the number of 5 second intervals in a day - change along with sleep interval as needed.

And feel free to keep updating this thread, whatever the outcome is.

17

Sadly, there's no ifstatus command on macOS, nor an installable Homebrew recipe for it.
nettop can be switched to "delta" mode but that just means the various counter values since the last output, not the changes in the routing table itself.
So I've thrown this script abomination together:

#! /bin/bash

netstat -rn -f inet6 > previous_netstat_inet6.txt

while true
do
  echo -e "\n------------------------------------------"
  date
  echo ""
  netstat -rn -f inet6 > current_netstat_inet6.txt
  diff -u previous_netstat_inet6.txt current_netstat_inet6.txt
  mv current_netstat_inet6.txt previous_netstat_inet6.txt
  sleep 1
done

That at least shows (per second) the routes removed and added back in. I'll see what I get when I encounter the issue again.

Searching for "MacOS ipv6 sleep" does turn up a few hits for similar issues, e.g.

https://discussions.apple.com/thread/7604644

and


These reports are going back some YEARS so this may be something Apple know about but don't care to fix, or else I need to get stubborn and find some allies to storm the Cupertino Castle...

1 Like
18

Well, it's reproducible at least...

If I send my Mac to sleep and then wake it up (not immediately, some amount of time seems to be required to trigger this), a ping6 that was working before the sleep happened, stops receiving responses. While the ping is running but isn't working, I see a series of routes being alternately added and removed every few seconds:

-fe80::42f2:1ff:fe0b:c552%en0            link#4                          UHLWIir         en0

and then a second or so later

+fe80::42f2:1ff:fe0b:c552%en0            link#4                          UHLWIir         en0

That repeats until the Mac receives an ICMP Router Advertisement from the OpenWRT router, at
which point the route changes subtly (and in a single update cycle)

-fe80::42f2:1ff:fe0b:c552%en0            link#4                          UHLWIir         en0       
+fe80::42f2:1ff:fe0b:c552%en0            40:f2:1:b:c5:52                 UHLWIir         en0

40:f2:1:b:c5:52 is the MAC address of my OpenWRT LAN interface. That makes sense. Now I get a series of pings through (usually 9 or 10) and then the route drops away again, replaced by the link#4 version.

-fe80::42f2:1ff:fe0b:c552%en0            40:f2:1:b:c5:52                 UHLWIir         en0       
+fe80::42f2:1ff:fe0b:c552%en0            link#4                          UHLWIir         en0

So what is link#4?

$ netstat -ni | head -1 ; netstat -ni | fgrep Link#4
Name       Mtu   Network       Address            Ipkts Ierrs    Opkts Oerrs  Coll
en0   1500  <Link#4>    98:10:e8:f3:d7:ef 91700234     0 74093664     0     0

No big surprise, it's my wired Ethernet port en0

So - why is the Mac dumping a perfectly good route it only just learned, through a locally-connected gateway that works, in favour of a route using a gateway on the SAME Ethernet segment, but that it can't actually get packets out of?

If it helps, here's the full dissection of the Router Advertisement:

Internet Control Message Protocol v6
    Type: Router Advertisement (134)
    Code: 0
    Checksum: 0xb815 [correct]
    [Checksum Status: Good]
    Cur hop limit: 64
    Flags: 0xc8, Managed address configuration, Other configuration, Prf (Default Router Preference): High
        1... .... = Managed address configuration: Set
        .1.. .... = Other configuration: Set
        ..0. .... = Home Agent: Not set
        ...0 1... = Prf (Default Router Preference): High (1)
        .... .0.. = Proxy: Not set
        .... ..0. = Reserved: 0
    Router lifetime (s): 1800
    Reachable time (ms): 360000
    Retrans timer (ms): 0
    ICMPv6 Option (Source link-layer address : 40:f2:01:0b:c5:52)
        Type: Source link-layer address (1)
        Length: 1 (8 bytes)
        Link-layer address: Sagemcom_0b:c5:52 (40:f2:01:0b:c5:52)
    ICMPv6 Option (MTU : 1476)
        Type: MTU (5)
        Length: 1 (8 bytes)
        Reserved
        MTU: 1476
    ICMPv6 Option (Prefix information : 2a00:23c4:6d9a:3700::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 43200
        Preferred Lifetime: 43200
        Reserved
        Prefix: 2a00:23c4:6d9a:3700::
    ICMPv6 Option (Prefix information : fde6:9508:a013::/64)
        Type: Prefix information (3)
        Length: 4 (32 bytes)
        Prefix Length: 64
        Flag: 0xc0, On-link flag(L), Autonomous address-configuration flag(A)
            1... .... = On-link flag(L): Set
            .1.. .... = Autonomous address-configuration flag(A): Set
            ..0. .... = Router address flag(R): Not set
            ...0 0000 = Reserved: 0
        Valid Lifetime: 43200
        Preferred Lifetime: 43200
        Reserved
        Prefix: fde6:9508:a013::
    ICMPv6 Option (Recursive DNS Server fde6:9508:a013::1)
        Type: Recursive DNS Server (25)
        Length: 3 (24 bytes)
        Reserved
        Lifetime: 6000
        Recursive DNS Servers: fde6:9508:a013::1
    ICMPv6 Option (Advertisement Interval : 600000)
        Type: Advertisement Interval (7)
        Length: 1 (8 bytes)
        Reserved
        Advertisement Interval: 600000
19

It looks like macOS is preferring the wrong route. That why the 'high' setting for the RA preference indeed helped. This is kinda weird and maybe it have to do with... I don't know... The order of the interfaces in Preferences?

1 Like
20

Hi folks. No progress towards a fix, but (somewhat surprisingly) Apple have actually responded to my feedback report and asked for an instrumented reproduction - they supplied a net-diagnose script which collected info including a packet capture while I repeated the steps above. 939MBytes of log data across the test (!) uploaded tonight - will see if they have a response in the coming days.
I should have noted earlier that (as a workaround) if I manually disconnect and reconnect the Ethernet cable, it runs stably again until the next deep sleep.
Equally I'm sure that running sudo ifconfig en0 down; sleep 10; sudo ifconfig en0 up would also do the trick.

2 Likes