Thanks - that's interesting. I'll bear in mind when reviewing the logs and timings, although I don't think I would necessarily know when the mesh reconfigures (the Plume app does identify network optimisations but these are few and far between, every few weeks, and certainly don't match the kernel warnings I get regularly each day). Another question for Plume.
Thanks
I just observed similar symptoms on my r7500v2 configured as an AP only. Repeated kernel warning messages every couple of minutes. I don't use a mesh network but...
The AP only configuration I'm using has been stable in the past (I did not observe these symptoms - at least at this frequency); however, I just put this device back into service after a period of disuse for about a month and a half.
One recent change I made is to the switch config in which I configured one of the lan ports to forward my "trunk" network (two vlans) to a second AP. I tested this change and it functions like I want, but that AP is not on or even physically connected when these symptoms started.
EDIT: a warning message:
Tue Jul 7 08:57:51 2020 kern.warn kernel: [121402.650525] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:XX:59, vlan:0)
my vlan tags are 2 and 3...
I just tried enabling STP for both the br-lan iface (one of the vlans) and also on second br iface (the other vlan for wifi "guests") but 2-3 warning messages continue to show up in my logs every couple of minuets. As I use my logs, I don't want these messages spamming them. Aside form these warnings, I don't seem to have network issues.
Since my network config is different than the OPs I think I should start a new thread to post my config - but that may take a day or two.
Please copy the output of the following commands and post it here using the "Preformatted text </> " button:
Remember to redact passwords, MAC addresses and any public IP addresses you may have:
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall
k, guess I'll do this here and now (if the OP requests it, I'll move it to a new thread).
The following logread output while doing a tcpdump is probably most helpful. Note there are no clients on my lan (everything is off) and I own no device with the the mac address 33:33:ff:4c:08:59 (but the last 6 digits do match my eth1 mac...)
This a ipv6 misconfiguration on my AP (i.e. something I need to turn off given my router should handle ipv6)?
EDIT: "Use builtin IPv6-management" is currently checked for br-lan and the guest bridge. I'll try with that unchecked to see if it makes any difference. "IPv6 assignment length" is currently disabled on both bridges.
r7500v2 # tcpdump -i br-lan -evn not host XXX.XXX.45.25 and ether host XX:XX:XX:XX:08:59&
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
r7500v2 # logread -f
Tue Jul 7 11:06:12 2020 user.notice root: wol.sh: XXX.XXX.45.26 is alive
11:06:25.772097 XX:XX:XX:XX:08:59 > 33:33:ff:4c:08:59, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff02::IPv6:XXXX:XXXX: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff4c:859
Tue Jul 7 11:06:25 2020 kern.warn kernel: [129117.176760] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
11:06:29.691913 XX:XX:XX:XX:08:59 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2
Tue Jul 7 11:06:29 2020 kern.warn kernel: [129121.096581] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
11:06:31.612075 XX:XX:XX:XX:08:59 > XX:XX:XX:XX:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff05::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff05::2
Tue Jul 7 11:06:31 2020 kern.warn kernel: [129123.016714] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
11:08:32.411965 XX:XX:XX:XX:08:59 > XX:XX:XX:XX:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff05::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff05::2
Tue Jul 7 11:08:32 2020 kern.warn kernel: [129243.816088] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
11:08:32.732064 XX:XX:XX:XX:08:59 > XX:XX:XX:XX:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2
Tue Jul 7 11:08:32 2020 kern.warn kernel: [129244.136164] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
11:08:36.412237 XX:XX:XX:XX:08:59 > XX:XX:XX:XX:08:59, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80::IPv6:XXXX:XXXX:859 > ff02::IPv6:XXXX:XXXX: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::1:ff4c:859
Tue Jul 7 11:08:36 2020 kern.warn kernel: [129247.816317] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
r7500v2 # 6 packets captured
7 packets received by filter
0 packets dropped by kernel
/etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr 'XXX.XXX.0.1'
option netmask 'XXX.XXX.0.0'
config globals 'globals'
option ula_prefix 'IPv6:XXXX:XXXX::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth1.1 eth0.3'
option proto 'static'
option ipaddr 'XXX.XXX.45.25'
option netmask 'XXX.XXX.255.0'
option gateway 'XXX.XXX.45.26'
option igmp_snooping '1'
option stp '1'
list dns 'XXX.XXX.222.220'
config switch
option name 'switch0'
option reset '3'
option enable_vlan '3'
config switch_vlan
option device 'switch0'
option vlan '3'
option ports '0t 1t 2 3 4 5t 6'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '0t 1t 5t'
config interface 'guestWLAN'
option ifname 'eth0.2'
option proto 'static'
option ipaddr 'XXX.XXX.44.25'
option netmask 'XXX.XXX.255.0'
option igmp_snooping '1'
option type 'bridge'
option stp '1'
/etc/config/wireless
config wifi-device 'radio0'
option type 'mac80211'
option hwmode '11a'
option path 'soc/1b500000.pci/pci0000:00/0000:00:00.0/0000:01:00.0'
option htmode 'VHT80'
option country 'US'
option legacy_rates '0'
option channel '36'
config wifi-iface 'default_radio0'
option device 'radio0'
option network 'lan'
option mode 'ap'
option ssid 'SSIDX5'
option key 'PPPPPPPP'
option wpa_group_rekey '86400'
option encryption 'psk2+ccmp'
option max_inactivity '3600'
option disassoc_low_ack '0'
option ieee80211w '1'
config wifi-iface 'wifinet0'
option device 'radio0'
option mode 'ap'
option ssid 'SSIDguest5'
option network 'guestWLAN'
option encryption 'psk2+ccmp'
option key 'PPPPPPPP'
option wpa_group_rekey '86400'
option max_inactivity '3600'
option disassoc_low_ack '0'
option ieee80211w '1'
config wifi-device 'radio1'
option type 'mac80211'
option hwmode '11g'
option path 'soc/1b700000.pci/pci0001:00/0001:00:00.0/0001:01:00.0'
option country 'US'
option legacy_rates '0'
option htmode 'HT20'
option channel '11'
config wifi-iface 'default_radio1'
option device 'radio1'
option network 'lan'
option mode 'ap'
option ssid 'SSIDX24'
option encryption 'psk2+ccmp'
option key 'PPPPPPPP'
option wpa_group_rekey '86400'
option max_inactivity '3600'
option disassoc_low_ack '0'
option ieee80211w '1'
config wifi-iface 'wifinet1'
option device 'radio1'
option mode 'ap'
option ssid 'SSIDguest24'
option network 'guestWLAN'
option encryption 'psk2+ccmp'
option key 'PPPPPPPP'
option wpa_group_rekey '86400'
option max_inactivity '3600'
option disassoc_low_ack '0'
option ieee80211w '1'
firewall, dhcp, dnsmasq services are off and disabled for this AP only config...
Out of curiosity, where did you read about using value 3 there?
And where is vlan1 defined?
Ha, i didn't. That's a leftover "misunderstanding" on my part from when I first set up the AP/vlan's. I think it should be 1. I can change it. Is that contributing to my symptoms?
EDIT: I have no vlan tag defined as '1'. Note my trunk line from my router comes into my AP on the WAN port if that matters...
router -> trunk (vlan 2 & 3) -> AP
i.e. my "network" only has two devices excluding ~9 wireless clients connected at the moment
One additional observation. I have seen these warning messages before - I think when I reboot my router. During or just after the router reboot, I might see a few in the logs and then they stop.
Prior to today, such occurrences were infrequent and usually associated with a network change as mentioned above.
results from a google for "ipv6 ra mac 33:33" makes me think this is originating from my router and is related to ipv6 ra "neighbor discovery". The logs indicate its on "vlan:0" - is that the untagged part of my trunk line?
anyway I'd really like this to stop... any suggestions welcom
EDIT: I rebooted the AP after making some changes mentioned in EDITs above.
dmesg after the reboot shows
...
[ 30.672553] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 30.678045] br-lan: port 4(wlan0) entered blocking state
[ 30.680327] IPv6: br-lan: IPv6 duplicate address fe80::b2b9:8aff:fe4c:859 used by XX:XX:XX:XX:08:59 detected!
...
and some other unusual entries related to the guest br. The repetitive "own source address" have not started yet - that took some time to develop last time.
After:
r7500v2 # ip -6 addr show dev br-lan
9: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet6 fe80::b2b9:8aff:fe4c:859/64 scope link dadfailed tentative
valid_lft forever preferred_lft forever
r7500v2 # ip -6 addr show dev br-guestWLAN
7: br-guestWLAN: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet6 fe80::b2b9:8aff:fe4c:85a/64 scope link
valid_lft forever preferred_lft forever
r7500v2 # ip -6 addr show dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet6 fe80::b2b9:8aff:fe4c:859/64 scope link
valid_lft forever preferred_lft forever
r7500v2 # ip -6 addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
inet6 fe80::b2b9:8aff:fe4c:85a/64 scope link
valid_lft forever preferred_lft forever
I googled "dad failed" and came up with a redhat bug report here that describes similar symptoms that I see.
I'm still not sure if this is a bug, a mis-configuration on my part, or both. At least I think I'm getting closer.
Seems that something else is using that IP or you have a loop. You have redacted the mac address of the device, but with reversing the slaac address it is obvious which one it is.
The redhat bug report is about openvswitch.
Can you describe the layout of your network? Routers, switches, access points etc.
Thank you for responding. My apologies for editing prior posts - I do that as a form of "note taking" so I can reference it later.
As described above here, my layout (at the moment) is one router and one AP. The AP connects to the router via its WAN port. The "link local" ipv6 address for eth1 (the LAN port) and br-lan are the same (see here). I'm unsure if this duplicate address is contributing to this issue but it seems likely at this point.
I have run this configuration for over a year without seeing this issue. I do have some old logs from the openwrt AP that I plan to review later today to see if the duplicate link local address was present in the past. There have been some changes to my network - I upgraded the router (a DIY ubuntu box) from 18.04 to 20.04 about 2 months ago but I have keep the same configuration there.
I do have a second, non openwrt, AP that is configured to be a drop in replacement for the openwrt r7500v2 AP currently in use (i.e. unplug the r7500v2 from the router, plug in the second AP, reboot the router and I'm good to go). Alternatively, I can "extend" my network layout by connecting this second AP to port 1 of the r7500v2 - see /etc/config/network above here.
I have used this second AP for the past 40+ days with the router (but no r7500v2) and have not observed this issue on this AP. Unfortunately, this AP is a bit different so this may not be helpful.
Then start by disconnecting the Access Point. If the problem stops, you've found your culprit.
Although as you mention later:
which makes me believe that there is something wrong with the primary AP.
That's what I think currently. Give me a day or two to look over my past logs - I might find something there. Also, after rebooting my r7500v2, I have yet to see the repetitive "own address as source address" return. I have a few ideas about how to trigger that, but I'm not sure.
Lastly, the log when this was happening indicate ipv6 ra/rs packets were on vlan:0. Do you know if vlan:0 refers to the untagged part of my trunk line?
No idea, but that sounds plausible.
It is vlan:0, not vlan0, just to be accurate.
thanks for that. Also I, corrected a couple typo's above regarding WAN/LAN and eth0/eth1. To be clear, the br-lan and the LAN interface eth1 have duplicate link-local ipv6 address.
Reviewing some old logs, I see this has always been the case and there is always a warning about it the logs when I run the r7500v2 as an AP only.
I have also run the r7500v2 as a GW/AP (i.e. router). As a GW/AP, br-lan has an ULA ipv6 address, its "normal" ipv6 address and a duplicate link local ipv6 address on eth1 and br-lan. There are no warning in the logs about a duplicate link local address as a GW/AP - I don't understand why... but i still think the duplicate address is contributing to the issue for an AP only config.
I suspect the potential for this issue has always been present (for my AP only config) - I may have just created the right conditions to trigger it recently.
I need to "improve" my ipv6 networking knowledge if I'm going to resolve this.
fun, fun, fun
I noticed that the guest bridge and eth0 also have duplicate ipv6 link local address (lla) but there is no warning about duplicates for that in dmesg.
So... if I reorder
to
config interface 'lan'
option type 'bridge'
option ifname 'eth0.3 eth1.1'
I
[ 38.572173] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 41.052007] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 43.373689] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 45.933042] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
...
[ 592.502130] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 592.502260] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 716.336264] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 717.775918] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
[ 720.495920] br-lan: received packet on eth0.3 with own address as source address (addr:XX:XX:XX:XX:08:59, vlan:0)
...
so yea! it is duplicate ipv6 lla's that cause this. But WTF!
When the default behavior for creating a bridge is to duplicate the MAC of the first interface defined in the config and then create a duplicate ipv6 lla from it - this feels, shall we say, less than ideal.
So now, how do I solve this? I'm going to try adding
net.ipv6.conf.eth0.disable_ipv6 = 1
net.ipv6.conf.eth1.disable_ipv6 = 1
to /etc/sysctl.conf and hope that I eliminate the duplicate ipv6 lla but still have ipv6 for the bridges (and any lan, or wifi clients).
This did not stop duplicate lla's on eth0 or eth1...
EDIT this post indicates hotplug may be a way to workaround this. Note also this same post links to FS#2126 regarding openwrt lla inadequacies.
@Ellah1 I'm not entirely sure the "cause" for my symptoms is the same as yours. I'm also not sure that my symptoms are actually the result of a mis-configuration on my router (or the AP) and perhaps these symptoms could be resolved by correcting any such mis-configuration.
Creating a hotplug script to change the bridge ipv6 lla's does not work. After a short period of no log spamming, the repetitive "'..own address as source..." messages returned.
EDIT: I plan to try using hotplug to set unique MAC address. Based on the kernel messages I saw during my other attempts, this seems related my eth1 interface. Clearly, I don't really understand why this happens.
Regardless, I'm out of time for playing with this for now and it may be awhile before I can get back to it.
Hi @anon98444528,
You've really given it a good shot and sorry you've not been able to get a working solution.
I've been working with the Plume support team and ran my AP's in router mode (instead of bridge mode) for two weeks and I'm now waiting for them to release their logs to me so I can investigate further. I've now reverted my AP config and am still getting up to 100 log entries a day myself.
I'm pretty convinced the logs are harmless as previously suggested, and I haven't noticed any negative performance issues, but it's still pretty annoying not getting root cause and having to filter the warnings out of the logs..........
When I get the logs I'll update again. Good luck your side too.
Thanks for that... but it also got me thinking. I've only had an issue with this once when the br-lan bridge derived it's MAC from eth1 (the LAN). All other times that br-lan derives it's MAC from eth1, I don't have an issue (including now). In this case,
ip -6 addr show dev br-lan
showed the br-lan ipv6 lla as ...dadfailed tentative...
So instead of trying to change the ipv6 lla's I've just removed all of them from the r7500v2 AP interfaces except for the lo interface (details below). I still have ipv6 (but this needs a long term test as clients can cache an ipv6 address - something that has caused me grief in the past). Now, even if br-lan derives its MAC from eth0 I don't see any "own source as address" messages (a configuration that, for me, causes these messages from boot).
Again I consider this a kluge workaround and not really a solution. I still don't know what the problem is, but I am symptom free for the hour or two that I have tested. Also, I am not using STP on the bridges.
To remove the ipv6 lla's from the AP interfaces (including the wireless interfaces) I put the following script in /etc/hotplug.d/net (don't use the iface directory):
#!/bin/sh
#env > /tmp/envs_log.log
if [ "$ACTION" != "add" ]; then
exit
fi
logger="/usr/bin/logger"
scope="link"
ip -6 addr flush scope "${scope}" dev eth0
ip -6 addr flush scope "${scope}" dev eth1
ip -6 addr flush scope "${scope}" dev wlan1
ip -6 addr flush scope "${scope}" dev wlan1-1
ip -6 addr flush scope "${scope}" dev wlan0
ip -6 addr flush scope "${scope}" dev wlan0-1
ip -6 addr flush scope "${scope}" dev br-lan
ip -6 addr flush scope "${scope}" dev br-guestWLAN
echo "/etc/hotplug.d/net/99-rmlla: lla's removed" | $logger
I give the script execute permissions via
chmod ugo+x /etc/hotplug.d/net/99-rmlla
It took me a while to find this thread. But once I did, it helped me resolve a very frustrating glitch that bedeviled me for at least a month. I am reviving the thread to provide a summary for anyone who, in the future, may stumble upon it.
I have a topology very similar to @Ellah1's:
OpenWrt-based -> Luxul AMS-2624P -> Plume SuperPods x6
Turris Omnia managed switch hardwired back haul
My symptoms were very parallel, including seeing 33:33:ff prefixed MAC addresses. Further, every time a nasty "own address" packet showed up, my router (or possibly my switch) went catatonic for a minimum of 3 minutes.
As best as I can tell from this thread, the issue comes down to:
(Is that a fair characterization?)
Using LuCI, I turned off IPv6 support any and everywhere I found it. That failed to fix the problem. It was only when I added a firewall rule to drop absolutely all IPv6 packets that my network was restored to health.
(For me, banning IPv6 traffic is a viable solution because my ISP does not support IPv6.)
IPv6 uses multicast instead of broadcast. Instead of broadcasting an arp-who-has, it will send a multicast neighbor solicitation to a specific address.