Devices getting disconnected every few minutes

jarapablo · September 14, 2023, 3:34pm

Hey Guys, I first of all, I have to accept that I have to do more troubleshooting on this and maybe narrow down the issue better, but I wanted to pick everyone's brain on this.
I have OpenWRT installed on my Linksys WRT3200ACM router, I have the WiFi disabled. I have an eero mesh WiFi network (3 devices), where only one of the eeros is plugged in directly to the router.
The problem that I am experiencing is that the eeros are getting "disconnected" every 5-10 minutes (thus, all my wifi network goes down). They remain down for a while and then they come back up. I am not sure if they are losing the IP or there is any other weirdness happening.
Things I have tried:

This behavior doesn't happen when I connect my eeros to my other router (not OpenWRT)
I have checked the system and Kernel logs (on LuCI) but I haven't been able to find anything
I have rebooted a couple of times, no change on behavior

Things I still have to try:

I need to hardwire my laptop and monitor to see if I also lose connection when the eeros get dropped (it could be only a problem with the eero?)
I need to connect my eero to a different ethernet port on the router (but I am not sure that this would make any difference)
I disabled my DDNS service to see if that made any difference (it didn't)

Ideas:

I am wondering if there is some kind of policy on my router that is blocking the port when it sees more than one MAC Address on a single port?
The router is just broken?
No other ideas .

I appreciate your help on this troubleshooting. I honestly don't know what could be going on

psherman · September 14, 2023, 4:53pm

Because your wifi system is not OpenWrt, you may need to look at that system and request help from the support channels (manufacturer or user forums) that they offer.

However, we can look at your OpenWrt configs to see if there are any issues that should be addressed.

Please copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

jarapablo · September 15, 2023, 3:34am

Thank you @psherman, I appreciate the help.

Just a quick update. It seems that the whole network gets down on the router and it's not only affecting the eeros because I decided to hardwire my computer to another port on the router today and it also got disconnected (I believe the WiFi also went down but wasn't able to catch it).

Here is the outputs that you requested:

# ubus call system board
{
        "kernel": "5.10.146",
        "hostname": "OpenWrt",
        "system": "ARMv7 Processor rev 1 (v7l)",
        "model": "Linksys WRT3200ACM",
        "board_name": "linksys,wrt3200acm",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.2",
                "revision": "r19803-9a599fee93",
                "target": "mvebu/cortexa9",
                "description": "OpenWrt 22.03.2 r19803-9a599fee93"
        }
}

# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd54:d25e:ccf3::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '10.104.XX.XXX'

config device
        option name 'wan'
        option macaddr '26:f5:a2:2f:XX:XX'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:01.0/0000:01:00.0'
        option channel '36'
        option band '5g'
        option htmode 'VHT80'
        option disabled '1'
        option country 'CA'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ssid 'OpenWrt'
        option encryption 'none'
        option macaddr '24:f5:a2:2f:f2:7a'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'soc/soc:pcie/pci0000:00/0000:00:02.0/0000:02:00.0'
        option channel '1'
        option band '2g'
        option htmode 'HT20'
        option country 'CA'
        option cell_density '0'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option macaddr '24:f5:a2:2f:f2:79'
        option ssid 'Test1'
        option encryption 'psk2'
        option key 'juanitaymartina'

config wifi-device 'radio2'
        option type 'mac80211'
        option path 'platform/soc/soc:internal-regs/f10d8000.sdhci/mmc_host/mmc0/mmc0:0001/mmc0:0001:1'
        option channel '34'
        option band '5g'
        option htmode 'VHT80'
        option cell_density '0'
        option disabled '1'

config wifi-iface 'default_radio2'
        option device 'radio2'
        option network 'lan'
        option mode 'ap'
        option ssid 'Test1'
        option encryption 'psk2'
        option key 'thisisnotmypassword'
        option disabled '1'

# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        option start '10'
        option limit '50'
        option leasetime '48h'
        list dhcp_option '6,8.8.8.8,1.1.1.1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config host
        option name 'RaspberryPi-4'
        option dns '1'
        option mac 'd8:3a:dd:48:dd:66'
        option ip '10.104.X.X'

config host
        option name 'raw-mini'
        option dns '1'
        option mac '00:8e:f2:cb:94:44'
        option ip '10.104.X.X'

config host
        option name 'BrotherPrinter'
        option dns '1'
        option mac 'd8:80:83:b0:a4:09'
        option ip '10.104.X.X'

# cat /etc/config/firewall
config defaults
        option syn_flood        1
        option input            ACCEPT
        option output           ACCEPT
        option forward          REJECT
# Uncomment this line to disable ipv6 rules
#       option disable_ipv6     1

config zone
        option name             lan
        list   network          'lan'
        option input            ACCEPT
        option output           ACCEPT
        option forward          ACCEPT

config zone
        option name             wan
        list   network          'wan'
        list   network          'wan6'
        option input            REJECT
        option output           ACCEPT
        option forward          REJECT
        option masq             1
        option mtu_fix          1

config forwarding
        option src              lan
        option dest             wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
        option name             Allow-DHCP-Renew
        option src              wan
        option proto            udp
        option dest_port        68
        option target           ACCEPT
        option family           ipv4

# Allow IPv4 ping
config rule
        option name             Allow-Ping
        option src              wan
        option proto            icmp
        option icmp_type        echo-request
        option family           ipv4
        option target           ACCEPT

config rule
        option name             Allow-IGMP
        option src              wan
        option proto            igmp
        option family           ipv4
        option target           ACCEPT

# Allow DHCPv6 replies
# see https://github.com/openwrt/openwrt/issues/5066
config rule
        option name             Allow-DHCPv6
        option src              wan
        option proto            udp
        option dest_port        546
        option family           ipv6
        option target           ACCEPT

config rule
        option name             Allow-MLD
        option src              wan
        option proto            icmp
        option src_ip           fe80::/10
        list icmp_type          '130/0'
        list icmp_type          '131/0'
        list icmp_type          '132/0'
        list icmp_type          '143/0'
        option family           ipv6
        option target           ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
        option name             Allow-ICMPv6-Input
        option src              wan
        option proto    icmp
        list icmp_type          echo-request
        list icmp_type          echo-reply
        list icmp_type          destination-unreachable
        list icmp_type          packet-too-big
        list icmp_type          time-exceeded
        list icmp_type          bad-header
        list icmp_type          unknown-header-type
        list icmp_type          router-solicitation
        list icmp_type          neighbour-solicitation
        list icmp_type          router-advertisement
        list icmp_type          neighbour-advertisement
        option limit            1000/sec
        option family           ipv6
        option target           ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
        option name             Allow-ICMPv6-Forward
        option src              wan
        option dest             *
        option proto            icmp
        list icmp_type          echo-request
        list icmp_type          echo-reply
        list icmp_type          destination-unreachable
        list icmp_type          packet-too-big
        list icmp_type          time-exceeded
        list icmp_type          bad-header
        list icmp_type          unknown-header-type
        option limit            1000/sec
        option family           ipv6
        option target           ACCEPT

config rule
        option name             Allow-IPSec-ESP
        option src              wan
        option dest             lan
        option proto            esp
        option target           ACCEPT

config rule
        option name             Allow-ISAKMP
        option src              wan
        option dest             lan
        option dest_port        500
        option proto            udp
        option target           ACCEPT


### EXAMPLE CONFIG SECTIONS
# do not allow a specific ip to access wan
#config rule
#       option src              lan
#       option src_ip   192.168.45.2
#       option dest             wan
#       option proto    tcp
#       option target   REJECT

# block a specific mac on wan
#config rule
#       option dest             wan
#       option src_mac  00:11:22:33:44:66
#       option target   REJECT

# block incoming ICMP traffic on a zone
#config rule
#       option src              lan
#       option proto    ICMP
#       option target   DROP

# port redirect port coming in on wan to lan
#config redirect
#       option src                      wan
#       option src_dport        80
#       option dest                     lan
#       option dest_ip          192.168.16.235
#       option dest_port        80
#       option proto            tcp

# port redirect of remapped ssh port (22001) on wan
#config redirect
#       option src              wan
#       option src_dport        22001
#       option dest             lan
#       option dest_port        22
#       option proto            tcp

### FULL CONFIG SECTIONS
#config rule
#       option src              lan
#       option src_ip   192.168.45.2
#       option src_mac  00:11:22:33:44:55
#       option src_port 80
#       option dest             wan
#       option dest_ip  194.25.2.129
#       option dest_port        120
#       option proto    tcp
#       option target   REJECT

#config redirect
#       option src              lan
#       option src_ip   192.168.45.2
#       option src_mac  00:11:22:33:44:55
#       option src_port         1024
#       option src_dport        80
#       option dest_ip  194.25.2.129
#       option dest_port        120
#       option proto    tcp

I just found something interesting on the logs. Not sure of what it means (researching now) but it could give some good insights of why this may be happening?

Thu Sep 14 20:07:25 2023 kern.info kernel: [ 8202.246943] mv88e6085 f1072004.mdio-mii:00 lan2: Link is Up - 1Gbps/Full - flow control rx/tx
Thu Sep 14 20:07:25 2023 kern.info kernel: [ 8202.255619] br-lan: port 2(lan2) entered blocking state
Thu Sep 14 20:07:25 2023 kern.info kernel: [ 8202.260877] br-lan: port 2(lan2) entered forwarding state
Thu Sep 14 20:07:25 2023 daemon.notice netifd: Network device 'lan2' link is up
Thu Sep 14 20:07:25 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.104.15.19 a0:ce:c8:dc:8a:21
Thu Sep 14 20:07:25 2023 daemon.info dnsmasq-dhcp[1]: DHCPNAK(br-lan) 10.104.15.19 a0:ce:c8:dc:8a:21 address in use
Thu Sep 14 20:07:29 2023 daemon.info dnsmasq-dhcp[1]: DHCPDISCOVER(br-lan) a0:ce:c8:dc:8a:21
Thu Sep 14 20:07:29 2023 daemon.info dnsmasq-dhcp[1]: DHCPOFFER(br-lan) 10.104.15.34 a0:ce:c8:dc:8a:21
Thu Sep 14 20:07:29 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.104.15.34 a0:ce:c8:dc:8a:21
Thu Sep 14 20:07:29 2023 daemon.warn dnsmasq-dhcp[1]: Ignoring domain X.X.net for DHCP host name WIN-1XXXX
Thu Sep 14 20:07:29 2023 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.104.X.X a0:ce:c8:dc:8a:21 WIN-1XXXX
Thu Sep 14 20:07:29 2023 daemon.warn dnsmasq[1]: possible DNS-rebind attack detected: dns.msftncsi.com
Thu Sep 14 20:07:29 2023 daemon.warn dnsmasq[2]: possible DNS-rebind attack detected: dns.msftncsi.com
Thu Sep 14 20:07:30 2023 daemon.warn dnsmasq[1]: possible DNS-rebind attack detected: dns.msftncsi.com
Thu Sep 14 20:07:30 2023 daemon.warn dnsmasq[1]: possible DNS-rebind attack detected: dns.msftncsi.com
Thu Sep 14 20:07:30 2023 daemon.warn dnsmasq[3]: possible DNS-rebind attack detected: dns.msftncsi.com

Thank you for your help! Any insights/ideas are appreciated

psherman · September 15, 2023, 3:41am

If the entire network is going down, you should start by determining the failure mode.

Specifically, is it only access to the internet that is interrupted, or are you unable to reach network resources (other hosts on your network, the router itself, etc)

jarapablo · September 15, 2023, 3:44am

Thank you @psherman !

I am unable to reach everything on the network, including the router. Trying to figure out why the whole thing is going down. But the router is not actually restarting.
May I ask you what you mean by "determine the failure mode"? I am hoping to find some kind of logs that can actually give me some light.

psherman · September 15, 2023, 3:50am

If only the internet was the issue, the failure mode would be external. What you are actually experiencing is internal.

This suggests an issue with your equipment or connections, not likely the router itself.

You need to look for a device (or set of devices) that could cause issues which will bring down a network such as a broadcast storm or a switching loop.

Do you happen to have a usb-c docking hub with Ethernet?

What about any devices that may be connected to your wired and wireless networks at the same time (think iot devices like a peloton bike, sonos music system, etc).

Your mesh wifi system could be to blame here, too. Are all nodes wirelessly connected to each other with just one connected to the main router by wire?

jarapablo · September 15, 2023, 3:59am

The answer to this is YES. This is the exact way that they are connected.

To be honest, I am not sure if the mesh (eero) is to blame. What makes the whole thing super weird is that everything works fine on my ISP provided router. But as soon as I move everything to the OpenWRT router, the problems start happening (goes down every 5-10 minutes or so).
Lastly (to make it even more odd), if I just connect my laptop to the OpenWRT router (via WiFi or wired), I can work indefinitely with no issues (about 1 hour so far).

Any way to see find any logs on OpenWRT that could give me an idea of what's going on?

psherman · September 15, 2023, 4:03am

Try plugging a computer directly into the openwrt router. Run a persistent ping to the router’s address. Then, when there is an outage, immediately unplug the connection to the WiFi mesh system. See if the pings start succeeding again within a few seconds of disconnecting the WiFi system.

Meanwhile, what about the other questions I asked - anything in that list that you have in your network?

jarapablo · September 15, 2023, 4:09am

I don't have this

Well, yes, I have some of these. You have a good point, one of them could be causing OpenWRT to crash, but, it shouldn't... My ISP router can handle it... I will still try to capture some traffic with wireshark. This may give me an idea.

Ping starts succeeding again after a few minutes without disconnecting anything. And then it goes down again. But you have a good point, I may just try to start disconnecting one thing at a time to start narrowing it down.

However, I will go back to my question. Isn't there a way to get some insights from OpenWRT?

psherman · September 15, 2023, 4:13am

I do not believe that your router is at fault here. It seems like a layer 2 (switching) type issue since all devices are unable to talk to each other. The router is not involved in l2 connectivity, thus it cannot directly be the issue.

However, address conflicts could be relevant. As could what are known as switching loops (which can be cause by an actual loop of Ethernet - a wire connecting between two ports on a switch, or other things like sonos devices - I had this take down my network), or the WiFi device having a faulty spanning tree protocol implementation.

jarapablo · September 15, 2023, 4:19am

I have to disagree. The router has a WAN port and 4 LAN ports. So there is L2 connectivity in it. And TBH, I believe the router is the one that is not able to handle "something" that is happening with my network.
Again, everything works OK with my ISP Provided router (same ports). But when I move everything over to my OpenWRT, it starts crashing.

psherman · September 15, 2023, 4:21am

An issue with the hardware cannot be ruled out yet, but the router itself (as in the actual routing engine, etc) cannot be involved here because that happens at l3. You will not likely see anything in the logs because of that, either.

jarapablo · September 15, 2023, 4:28am

Interesting. I thought there was a way to see some L2 logs (Port up/down/disabled/blocked) on OpenWRT. It has pretty advanced L2 features (bridge, VLANs, etc)... But I guess it makes sense that the main purpose is being L3.
I will probably spend a couple more hours and report back if I find anything.