Random wireless latency spikes and timeouts on the Xiaomi Mi Router 4A Gigabit Edition

chronos · January 29, 2023, 10:29am

My Xiaomi Mi 4A Gigabit router has been suffering from very random and unpredictable latency and packet loss bursts. On some days (sometimes a week) it works perfectly well, on other days while pinging the router the latency would spike up to 2000 ms and timeout on some packets continuously for ~2 mins before returning to normal. The logs rarely show "Hardware restart was requested" but not always. I am not sure when this regression began but old community built OpenWRT images (19.07 and older) didn't have this issue. At first I thought this might be caused by interference but upon doing channel analysis I saw that the channel I was using was free. Furthermore, whenever there's a current burst of latency and packet loss, restarting the wireless interface immediately resolves the issue (even when the channel is manually set). When using Ethernet, the issue is non-existent. It also happens even when there is almost no load on the router. I have not been able to reproduce this issue, and there appears to be no pattern as to when it happens (different times of day, severity, periodicity, etc.)

Ping monitoring to (192.168.1.1) using Ping Tracer:

I have tried doing the following:

Using 2.4 GHz and 5 GHz bands
Manually selecting the channel
Using a different client (the issue happens to all clients simultaneously)
Increasing logging verbosity for hostapd
Re-flashing and wiping configuration
Minimal install and configuration
Using different versions and point releases from 21.02 until 22.03.3

ThiloteE · January 29, 2023, 12:12pm

According to OpenWRT wiki, this is a mt7621 + MediaTek MT7603E, MediaTek MT7612E device.

You could check out other forum threads and github issues with mt7603 and mt7612:

the latency would spike up to 2000 ms and timeout on some packets continuously for ~2 mins

At first glance, this kinda sounds like bufferbloat though or at least like something is severely stuck. Have you tried latest master snapshots?

lleachii · January 29, 2023, 12:16pm

Can you show this log entry?

So since you said it was a regression - what [old] version did?

chronos · January 29, 2023, 1:27pm

Thanks for the reply,

Unfortunately, the router has restarted (power outage) and reset the logs. The issue hasn't happened again yet (it's been 2 days). I will post the logs along with ping latency as soon as the issue appears again. However I can tell you that 99% of the time the issue appears there is absolutely nothing in the logs in the same time frame.

Before the router gained official support with version 21.02.0 I had been using a custom build of 19.07.4 by community member db260179. Unfortunately, since I can not reproduce this bug, I am unable to pinpoint the exact version that introduced it.

lleachii · January 29, 2023, 1:39pm

@db260179 tagged them.

chronos · January 29, 2023, 1:47pm

Thanks for digging these up. These certainly seem related, but not exactly the same. For me the issue solves itself on its own and it even happens on the 5 GHz band.

Bufferbloat is more likely to happen for internet traffic with a small pipe. In this case, I was merely pinging the router locally. Also, the issue appears even when there's barely any traffic or load. I have not tried the latest snapshot. I am currently running the latest OpenWrt 22.03.3 release. Is there a related fix for this in the latest master?

ThiloteE · January 29, 2023, 3:16pm

I don't know, but master snapshot is ~ 2000 commits ahead of 22.03 branch, so who knows?

chronos · January 30, 2023, 7:31am

I have tried to reproduce the issue with iperf3 to no avail. It's clear from the graph that the issue in this case is just bufferbloat and there are no dropped packets (no red lines). I will try master snapshot and post an update.

Latency Graph:

iperf3 result (client side):

user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -t 60 -i 10
Connecting to host 192.168.1.1, port 5201
[  4] local 172.26.253.58 port 50722 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec   106 MBytes  88.6 Mbits/sec    0    481 KBytes
[  4]  10.00-20.00  sec   104 MBytes  87.5 Mbits/sec    0    481 KBytes
[  4]  20.00-30.00  sec   105 MBytes  88.1 Mbits/sec    0    481 KBytes
[  4]  30.00-40.00  sec   105 MBytes  88.4 Mbits/sec    0    481 KBytes
[  4]  40.00-50.00  sec   105 MBytes  88.4 Mbits/sec    0    481 KBytes
[  4]  50.00-60.00  sec   111 MBytes  93.0 Mbits/sec    0    481 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec   637 MBytes  89.0 Mbits/sec    0             sender
[  4]   0.00-60.00  sec   634 MBytes  88.7 Mbits/sec                  receiver

iperf Done.
user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -P 4 -t 60 -i 10
Connecting to host 192.168.1.1, port 5201
[  4] local 172.26.253.58 port 50726 connected to 192.168.1.1 port 5201
[  6] local 172.26.253.58 port 50728 connected to 192.168.1.1 port 5201
[  8] local 172.26.253.58 port 50730 connected to 192.168.1.1 port 5201
[ 10] local 172.26.253.58 port 50732 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  27.8 MBytes  23.3 Mbits/sec    0    460 KBytes
[  6]   0.00-10.00  sec  30.1 MBytes  25.2 Mbits/sec    0    481 KBytes
[  8]   0.00-10.00  sec  28.3 MBytes  23.7 Mbits/sec    0    460 KBytes
[ 10]   0.00-10.00  sec  31.0 MBytes  26.0 Mbits/sec    0    461 KBytes
[SUM]   0.00-10.00  sec   117 MBytes  98.3 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  10.00-20.00  sec  28.3 MBytes  23.8 Mbits/sec    0    460 KBytes
[  6]  10.00-20.00  sec  28.4 MBytes  23.8 Mbits/sec    0    481 KBytes
[  8]  10.00-20.00  sec  28.2 MBytes  23.7 Mbits/sec    0    460 KBytes
[ 10]  10.00-20.00  sec  28.1 MBytes  23.6 Mbits/sec    0    461 KBytes
[SUM]  10.00-20.00  sec   113 MBytes  94.8 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  20.00-30.00  sec  29.7 MBytes  24.9 Mbits/sec    0    460 KBytes
[  6]  20.00-30.00  sec  29.7 MBytes  25.0 Mbits/sec    0    481 KBytes
[  8]  20.00-30.00  sec  29.6 MBytes  24.8 Mbits/sec    0    460 KBytes
[ 10]  20.00-30.00  sec  29.5 MBytes  24.8 Mbits/sec    0    461 KBytes
[SUM]  20.00-30.00  sec   119 MBytes  99.4 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-40.00  sec  31.1 MBytes  26.1 Mbits/sec    0    693 KBytes
[  6]  30.00-40.00  sec  29.6 MBytes  24.9 Mbits/sec    0    481 KBytes
[  8]  30.00-40.00  sec  30.9 MBytes  26.0 Mbits/sec    0    690 KBytes
[ 10]  30.00-40.00  sec  29.6 MBytes  24.8 Mbits/sec    0    461 KBytes
[SUM]  30.00-40.00  sec   121 MBytes   102 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  40.00-50.00  sec  22.9 MBytes  19.2 Mbits/sec    0    693 KBytes
[  6]  40.00-50.00  sec  22.9 MBytes  19.2 Mbits/sec    0    481 KBytes
[  8]  40.00-50.00  sec  23.0 MBytes  19.3 Mbits/sec    0    690 KBytes
[ 10]  40.00-50.00  sec  22.8 MBytes  19.1 Mbits/sec    0    461 KBytes
[SUM]  40.00-50.00  sec  91.6 MBytes  76.8 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  50.00-60.00  sec  27.4 MBytes  23.0 Mbits/sec    0    693 KBytes
[  6]  50.00-60.00  sec  27.4 MBytes  23.0 Mbits/sec    0    481 KBytes
[  8]  50.00-60.00  sec  27.4 MBytes  23.0 Mbits/sec    0    690 KBytes
[ 10]  50.00-60.00  sec  27.3 MBytes  22.9 Mbits/sec    0    461 KBytes
[SUM]  50.00-60.00  sec   109 MBytes  91.8 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec   167 MBytes  23.4 Mbits/sec    0             sender
[  4]   0.00-60.00  sec   164 MBytes  22.9 Mbits/sec                  receiver
[  6]   0.00-60.00  sec   168 MBytes  23.5 Mbits/sec    0             sender
[  6]   0.00-60.00  sec   166 MBytes  23.2 Mbits/sec                  receiver
[  8]   0.00-60.00  sec   167 MBytes  23.4 Mbits/sec    0             sender
[  8]   0.00-60.00  sec   164 MBytes  22.9 Mbits/sec                  receiver
[ 10]   0.00-60.00  sec   168 MBytes  23.5 Mbits/sec    0             sender
[ 10]   0.00-60.00  sec   166 MBytes  23.2 Mbits/sec                  receiver
[SUM]   0.00-60.00  sec   671 MBytes  93.8 Mbits/sec    0             sender
[SUM]   0.00-60.00  sec   660 MBytes  92.3 Mbits/sec                  receiver

iperf Done.
user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -P 8 -t 60 -i 10
Connecting to host 192.168.1.1, port 5201
[  4] local 172.26.253.58 port 50736 connected to 192.168.1.1 port 5201
[  6] local 172.26.253.58 port 50738 connected to 192.168.1.1 port 5201
[  8] local 172.26.253.58 port 50740 connected to 192.168.1.1 port 5201
[ 10] local 172.26.253.58 port 50742 connected to 192.168.1.1 port 5201
[ 12] local 172.26.253.58 port 50744 connected to 192.168.1.1 port 5201
[ 14] local 172.26.253.58 port 50746 connected to 192.168.1.1 port 5201
[ 16] local 172.26.253.58 port 50748 connected to 192.168.1.1 port 5201
[ 18] local 172.26.253.58 port 50750 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  14.9 MBytes  12.5 Mbits/sec    0    481 KBytes
[  6]   0.00-10.00  sec  17.9 MBytes  15.1 Mbits/sec    0    461 KBytes
[  8]   0.00-10.00  sec  15.1 MBytes  12.7 Mbits/sec    0    461 KBytes
[ 10]   0.00-10.00  sec  17.5 MBytes  14.7 Mbits/sec    0    481 KBytes
[ 12]   0.00-10.00  sec  11.6 MBytes  9.72 Mbits/sec    2    460 KBytes
[ 14]   0.00-10.00  sec  20.6 MBytes  17.3 Mbits/sec    0    482 KBytes
[ 16]   0.00-10.00  sec  17.8 MBytes  15.0 Mbits/sec    0    460 KBytes
[ 18]   0.00-10.00  sec  20.8 MBytes  17.5 Mbits/sec    0    482 KBytes
[SUM]   0.00-10.00  sec   136 MBytes   114 Mbits/sec    2
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  10.00-20.00  sec  14.2 MBytes  11.9 Mbits/sec    0    481 KBytes
[  6]  10.00-20.00  sec  14.1 MBytes  11.9 Mbits/sec    1    461 KBytes
[  8]  10.00-20.00  sec  14.1 MBytes  11.9 Mbits/sec    0    461 KBytes
[ 10]  10.00-20.00  sec  14.1 MBytes  11.9 Mbits/sec    0    481 KBytes
[ 12]  10.00-20.00  sec  14.8 MBytes  12.4 Mbits/sec    0    560 KBytes
[ 14]  10.00-20.00  sec  14.1 MBytes  11.9 Mbits/sec    0    482 KBytes
[ 16]  10.00-20.00  sec  15.3 MBytes  12.9 Mbits/sec    1    690 KBytes
[ 18]  10.00-20.00  sec  14.1 MBytes  11.8 Mbits/sec    0    482 KBytes
[SUM]  10.00-20.00  sec   115 MBytes  96.4 Mbits/sec    2
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  20.00-30.00  sec  14.7 MBytes  12.4 Mbits/sec    0    481 KBytes
[  6]  20.00-30.00  sec  14.8 MBytes  12.4 Mbits/sec    0    461 KBytes
[  8]  20.00-30.00  sec  14.7 MBytes  12.4 Mbits/sec    0    461 KBytes
[ 10]  20.00-30.00  sec  14.7 MBytes  12.4 Mbits/sec    0    481 KBytes
[ 12]  20.00-30.00  sec  14.8 MBytes  12.4 Mbits/sec    0    560 KBytes
[ 14]  20.00-30.00  sec  14.8 MBytes  12.4 Mbits/sec    0    482 KBytes
[ 16]  20.00-30.00  sec  14.8 MBytes  12.4 Mbits/sec    0    690 KBytes
[ 18]  20.00-30.00  sec  14.7 MBytes  12.4 Mbits/sec    0    482 KBytes
[SUM]  20.00-30.00  sec   118 MBytes  99.0 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-40.00  sec  13.5 MBytes  11.4 Mbits/sec    0    481 KBytes
[  6]  30.00-40.00  sec  13.6 MBytes  11.4 Mbits/sec    0    461 KBytes
[  8]  30.00-40.00  sec  13.5 MBytes  11.4 Mbits/sec    0    461 KBytes
[ 10]  30.00-40.00  sec  13.5 MBytes  11.3 Mbits/sec    0    481 KBytes
[ 12]  30.00-40.00  sec  13.5 MBytes  11.3 Mbits/sec    0    560 KBytes
[ 14]  30.00-40.00  sec  13.5 MBytes  11.3 Mbits/sec    0    482 KBytes
[ 16]  30.00-40.00  sec  13.5 MBytes  11.4 Mbits/sec    0    690 KBytes
[ 18]  30.00-40.00  sec  13.5 MBytes  11.4 Mbits/sec    0    482 KBytes
[SUM]  30.00-40.00  sec   108 MBytes  90.8 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  40.00-50.00  sec  11.9 MBytes  9.96 Mbits/sec    0    481 KBytes
[  6]  40.00-50.00  sec  11.8 MBytes  9.91 Mbits/sec    0    461 KBytes
[  8]  40.00-50.00  sec  11.9 MBytes  9.96 Mbits/sec    0    461 KBytes
[ 10]  40.00-50.00  sec  11.8 MBytes  9.91 Mbits/sec    0    481 KBytes
[ 12]  40.00-50.00  sec  11.8 MBytes  9.91 Mbits/sec    0    560 KBytes
[ 14]  40.00-50.00  sec  11.9 MBytes  9.96 Mbits/sec    0    482 KBytes
[ 16]  40.00-50.00  sec  11.9 MBytes  9.96 Mbits/sec    0    690 KBytes
[ 18]  40.00-50.00  sec  11.8 MBytes  9.91 Mbits/sec    0    482 KBytes
[SUM]  40.00-50.00  sec  94.8 MBytes  79.5 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  50.00-60.00  sec  13.8 MBytes  11.6 Mbits/sec    0    481 KBytes
[  6]  50.00-60.00  sec  15.1 MBytes  12.7 Mbits/sec    0    694 KBytes
[  8]  50.00-60.00  sec  13.8 MBytes  11.6 Mbits/sec    0    461 KBytes
[ 10]  50.00-60.00  sec  13.7 MBytes  11.5 Mbits/sec    0    481 KBytes
[ 12]  50.00-60.00  sec  13.8 MBytes  11.6 Mbits/sec    0    560 KBytes
[ 14]  50.00-60.00  sec  13.7 MBytes  11.5 Mbits/sec    0    482 KBytes
[ 16]  50.00-60.00  sec  13.8 MBytes  11.6 Mbits/sec    0    690 KBytes
[ 18]  50.00-60.00  sec  13.8 MBytes  11.6 Mbits/sec    0    482 KBytes
[SUM]  50.00-60.00  sec   111 MBytes  93.4 Mbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  83.0 MBytes  11.6 Mbits/sec    0             sender
[  4]   0.00-60.00  sec  80.8 MBytes  11.3 Mbits/sec                  receiver
[  6]   0.00-60.00  sec  87.4 MBytes  12.2 Mbits/sec    1             sender
[  6]   0.00-60.00  sec  84.0 MBytes  11.7 Mbits/sec                  receiver
[  8]   0.00-60.00  sec  83.2 MBytes  11.6 Mbits/sec    0             sender
[  8]   0.00-60.00  sec  81.1 MBytes  11.3 Mbits/sec                  receiver
[ 10]   0.00-60.00  sec  85.4 MBytes  11.9 Mbits/sec    0             sender
[ 10]   0.00-60.00  sec  83.3 MBytes  11.6 Mbits/sec                  receiver
[ 12]   0.00-60.00  sec  80.2 MBytes  11.2 Mbits/sec    2             sender
[ 12]   0.00-60.00  sec  77.6 MBytes  10.9 Mbits/sec                  receiver
[ 14]   0.00-60.00  sec  88.5 MBytes  12.4 Mbits/sec    0             sender
[ 14]   0.00-60.00  sec  86.4 MBytes  12.1 Mbits/sec                  receiver
[ 16]   0.00-60.00  sec  87.1 MBytes  12.2 Mbits/sec    1             sender
[ 16]   0.00-60.00  sec  83.8 MBytes  11.7 Mbits/sec                  receiver
[ 18]   0.00-60.00  sec  88.8 MBytes  12.4 Mbits/sec    0             sender
[ 18]   0.00-60.00  sec  86.6 MBytes  12.1 Mbits/sec                  receiver
[SUM]   0.00-60.00  sec   684 MBytes  95.6 Mbits/sec    4             sender
[SUM]   0.00-60.00  sec   664 MBytes  92.8 Mbits/sec                  receiver

iperf Done.
user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -t 60 -i 10 -R
Connecting to host 192.168.1.1, port 5201
Reverse mode, remote host 192.168.1.1 is sending
[  4] local 172.26.253.58 port 50754 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   250 MBytes   210 Mbits/sec
[  4]  10.00-20.00  sec   259 MBytes   218 Mbits/sec
[  4]  20.00-30.00  sec   259 MBytes   217 Mbits/sec
[  4]  30.00-40.00  sec   255 MBytes   214 Mbits/sec
[  4]  40.00-50.00  sec   238 MBytes   200 Mbits/sec
[  4]  50.00-60.00  sec   255 MBytes   214 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  1.48 GBytes   212 Mbits/sec    1             sender
[  4]   0.00-60.00  sec  1.48 GBytes   212 Mbits/sec                  receiver

iperf Done.
user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -P 4 -t 60 -i 10 -R
Connecting to host 192.168.1.1, port 5201
Reverse mode, remote host 192.168.1.1 is sending
[  4] local 172.26.253.58 port 50758 connected to 192.168.1.1 port 5201
[  6] local 172.26.253.58 port 50760 connected to 192.168.1.1 port 5201
[  8] local 172.26.253.58 port 50762 connected to 192.168.1.1 port 5201
[ 10] local 172.26.253.58 port 50764 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  59.4 MBytes  49.8 Mbits/sec
[  6]   0.00-10.00  sec  71.4 MBytes  59.9 Mbits/sec
[  8]   0.00-10.00  sec  54.2 MBytes  45.5 Mbits/sec
[ 10]   0.00-10.00  sec  56.8 MBytes  47.6 Mbits/sec
[SUM]   0.00-10.00  sec   242 MBytes   203 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  10.00-20.00  sec  64.1 MBytes  53.8 Mbits/sec
[  6]  10.00-20.00  sec  66.7 MBytes  55.9 Mbits/sec
[  8]  10.00-20.00  sec  66.9 MBytes  56.1 Mbits/sec
[ 10]  10.00-20.00  sec  61.7 MBytes  51.7 Mbits/sec
[SUM]  10.00-20.00  sec   259 MBytes   218 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  20.00-30.00  sec  61.3 MBytes  51.4 Mbits/sec
[  6]  20.00-30.00  sec  60.1 MBytes  50.4 Mbits/sec
[  8]  20.00-30.00  sec  59.3 MBytes  49.8 Mbits/sec
[ 10]  20.00-30.00  sec  59.0 MBytes  49.5 Mbits/sec
[SUM]  20.00-30.00  sec   240 MBytes   201 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-40.00  sec  57.7 MBytes  48.4 Mbits/sec
[  6]  30.00-40.00  sec  60.2 MBytes  50.5 Mbits/sec
[  8]  30.00-40.00  sec  60.7 MBytes  50.9 Mbits/sec
[ 10]  30.00-40.00  sec  57.0 MBytes  47.8 Mbits/sec
[SUM]  30.00-40.00  sec   236 MBytes   198 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  40.00-50.00  sec  50.1 MBytes  42.0 Mbits/sec
[  6]  40.00-50.00  sec  52.0 MBytes  43.6 Mbits/sec
[  8]  40.00-50.00  sec  51.1 MBytes  42.9 Mbits/sec
[ 10]  40.00-50.00  sec  46.7 MBytes  39.1 Mbits/sec
[SUM]  40.00-50.00  sec   200 MBytes   168 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  50.00-60.00  sec  56.6 MBytes  47.5 Mbits/sec
[  6]  50.00-60.00  sec  61.9 MBytes  51.9 Mbits/sec
[  8]  50.00-60.00  sec  62.2 MBytes  52.2 Mbits/sec
[ 10]  50.00-60.00  sec  54.7 MBytes  45.9 Mbits/sec
[SUM]  50.00-60.00  sec   235 MBytes   197 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec   349 MBytes  48.8 Mbits/sec    5             sender
[  4]   0.00-60.00  sec   349 MBytes  48.8 Mbits/sec                  receiver
[  6]   0.00-60.00  sec   372 MBytes  52.1 Mbits/sec    4             sender
[  6]   0.00-60.00  sec   372 MBytes  52.1 Mbits/sec                  receiver
[  8]   0.00-60.00  sec   355 MBytes  49.7 Mbits/sec    4             sender
[  8]   0.00-60.00  sec   355 MBytes  49.6 Mbits/sec                  receiver
[ 10]   0.00-60.00  sec   336 MBytes  47.0 Mbits/sec    5             sender
[ 10]   0.00-60.00  sec   336 MBytes  47.0 Mbits/sec                  receiver
[SUM]   0.00-60.00  sec  1.38 GBytes   198 Mbits/sec   18             sender
[SUM]   0.00-60.00  sec  1.38 GBytes   197 Mbits/sec                  receiver

iperf Done.
user@computer:~$ iperf3 -c 192.168.1.1 -p 5201 -P 8 -t 60 -i 10 -R
Connecting to host 192.168.1.1, port 5201
Reverse mode, remote host 192.168.1.1 is sending
[  4] local 172.26.253.58 port 50768 connected to 192.168.1.1 port 5201
[  6] local 172.26.253.58 port 50770 connected to 192.168.1.1 port 5201
[  8] local 172.26.253.58 port 50772 connected to 192.168.1.1 port 5201
[ 10] local 172.26.253.58 port 50774 connected to 192.168.1.1 port 5201
[ 12] local 172.26.253.58 port 50776 connected to 192.168.1.1 port 5201
[ 14] local 172.26.253.58 port 50778 connected to 192.168.1.1 port 5201
[ 16] local 172.26.253.58 port 50780 connected to 192.168.1.1 port 5201
[ 18] local 172.26.253.58 port 50782 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  30.1 MBytes  25.2 Mbits/sec
[  6]   0.00-10.00  sec  26.8 MBytes  22.5 Mbits/sec
[  8]   0.00-10.00  sec  30.3 MBytes  25.4 Mbits/sec
[ 10]   0.00-10.00  sec  32.6 MBytes  27.4 Mbits/sec
[ 12]   0.00-10.00  sec  20.7 MBytes  17.3 Mbits/sec
[ 14]   0.00-10.00  sec  30.6 MBytes  25.6 Mbits/sec
[ 16]   0.00-10.00  sec  26.5 MBytes  22.2 Mbits/sec
[ 18]   0.00-10.00  sec  28.9 MBytes  24.2 Mbits/sec
[SUM]   0.00-10.00  sec   226 MBytes   190 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  10.00-20.00  sec  31.8 MBytes  26.6 Mbits/sec
[  6]  10.00-20.00  sec  19.2 MBytes  16.1 Mbits/sec
[  8]  10.00-20.00  sec  31.0 MBytes  26.0 Mbits/sec
[ 10]  10.00-20.00  sec  37.6 MBytes  31.6 Mbits/sec
[ 12]  10.00-20.00  sec  28.8 MBytes  24.1 Mbits/sec
[ 14]  10.00-20.00  sec  31.6 MBytes  26.5 Mbits/sec
[ 16]  10.00-20.00  sec  28.9 MBytes  24.2 Mbits/sec
[ 18]  10.00-20.00  sec  34.2 MBytes  28.7 Mbits/sec
[SUM]  10.00-20.00  sec   243 MBytes   204 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  20.00-30.00  sec  26.9 MBytes  22.5 Mbits/sec
[  6]  20.00-30.00  sec  17.2 MBytes  14.4 Mbits/sec
[  8]  20.00-30.00  sec  31.0 MBytes  26.0 Mbits/sec
[ 10]  20.00-30.00  sec  26.8 MBytes  22.5 Mbits/sec
[ 12]  20.00-30.00  sec  26.9 MBytes  22.6 Mbits/sec
[ 14]  20.00-30.00  sec  25.1 MBytes  21.0 Mbits/sec
[ 16]  20.00-30.00  sec  27.2 MBytes  22.8 Mbits/sec
[ 18]  20.00-30.00  sec  26.5 MBytes  22.2 Mbits/sec
[SUM]  20.00-30.00  sec   208 MBytes   174 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  30.00-40.00  sec  31.2 MBytes  26.2 Mbits/sec
[  6]  30.00-40.00  sec  21.2 MBytes  17.8 Mbits/sec
[  8]  30.00-40.00  sec  31.7 MBytes  26.6 Mbits/sec
[ 10]  30.00-40.00  sec  31.3 MBytes  26.3 Mbits/sec
[ 12]  30.00-40.00  sec  33.1 MBytes  27.7 Mbits/sec
[ 14]  30.00-40.00  sec  31.1 MBytes  26.1 Mbits/sec
[ 16]  30.00-40.00  sec  30.6 MBytes  25.6 Mbits/sec
[ 18]  30.00-40.00  sec  30.9 MBytes  25.9 Mbits/sec
[SUM]  30.00-40.00  sec   241 MBytes   202 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  40.00-50.00  sec  30.4 MBytes  25.5 Mbits/sec
[  6]  40.00-50.00  sec  14.6 MBytes  12.3 Mbits/sec
[  8]  40.00-50.00  sec  23.1 MBytes  19.4 Mbits/sec
[ 10]  40.00-50.00  sec  24.0 MBytes  20.1 Mbits/sec
[ 12]  40.00-50.00  sec  29.4 MBytes  24.6 Mbits/sec
[ 14]  40.00-50.00  sec  29.5 MBytes  24.8 Mbits/sec
[ 16]  40.00-50.00  sec  23.8 MBytes  20.0 Mbits/sec
[ 18]  40.00-50.00  sec  26.4 MBytes  22.2 Mbits/sec
[SUM]  40.00-50.00  sec   201 MBytes   169 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  4]  50.00-60.00  sec  28.0 MBytes  23.5 Mbits/sec
[  6]  50.00-60.00  sec  16.9 MBytes  14.2 Mbits/sec
[  8]  50.00-60.00  sec  26.2 MBytes  22.0 Mbits/sec
[ 10]  50.00-60.00  sec  24.1 MBytes  20.2 Mbits/sec
[ 12]  50.00-60.00  sec  27.8 MBytes  23.3 Mbits/sec
[ 14]  50.00-60.00  sec  23.2 MBytes  19.5 Mbits/sec
[ 16]  50.00-60.00  sec  22.3 MBytes  18.7 Mbits/sec
[ 18]  50.00-60.00  sec  26.8 MBytes  22.5 Mbits/sec
[SUM]  50.00-60.00  sec   195 MBytes   164 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec   179 MBytes  25.0 Mbits/sec   38             sender
[  4]   0.00-60.00  sec   178 MBytes  24.9 Mbits/sec                  receiver
[  6]   0.00-60.00  sec   116 MBytes  16.2 Mbits/sec   38             sender
[  6]   0.00-60.00  sec   116 MBytes  16.2 Mbits/sec                  receiver
[  8]   0.00-60.00  sec   174 MBytes  24.3 Mbits/sec   26             sender
[  8]   0.00-60.00  sec   173 MBytes  24.3 Mbits/sec                  receiver
[ 10]   0.00-60.00  sec   177 MBytes  24.8 Mbits/sec   32             sender
[ 10]   0.00-60.00  sec   177 MBytes  24.7 Mbits/sec                  receiver
[ 12]   0.00-60.00  sec   168 MBytes  23.4 Mbits/sec   26             sender
[ 12]   0.00-60.00  sec   167 MBytes  23.3 Mbits/sec                  receiver
[ 14]   0.00-60.00  sec   172 MBytes  24.0 Mbits/sec   29             sender
[ 14]   0.00-60.00  sec   171 MBytes  23.9 Mbits/sec                  receiver
[ 16]   0.00-60.00  sec   160 MBytes  22.4 Mbits/sec   20             sender
[ 16]   0.00-60.00  sec   160 MBytes  22.3 Mbits/sec                  receiver
[ 18]   0.00-60.00  sec   174 MBytes  24.4 Mbits/sec   32             sender
[ 18]   0.00-60.00  sec   174 MBytes  24.3 Mbits/sec                  receiver
[SUM]   0.00-60.00  sec  1.29 GBytes   185 Mbits/sec  241             sender
[SUM]   0.00-60.00  sec  1.29 GBytes   184 Mbits/sec                  receiver

iperf Done.

db260179 · January 30, 2023, 9:34am

Try my custom repo and feed back - https://gitlab.com/db260179/openwrt-base/-/tree/openwrt-22.03-mt7621/docker

Few things changed from the current openwrt 22.03 branch - Updated MT7603e/7612e firmware updated from the mtk-wifi kernel driver and due to the way the MCU mechanism works when dealing with firmware timeouts (was introduced in 2018 and been causing headaches ever since), have increased the TX to 45 tries instead of just 10.

I've had timeout spikes and notice since the firmware update seems to be fine now - there was a firmware update done in the official 22.03 on January 2022, i think that is the cause of the issues?

Anyway feedback.

I don't do custom builds for people, just install docker and run the run-build.sh script, easy to do.

Thanks

chronos · January 31, 2023, 5:38am

Thanks! I successfully built and flashed version 0e2fdba0dd. Since the issue isn't reproducible, I will need to run it for a couple of days to make sure the issue is resolved and post an update.

chronos · January 31, 2023, 3:46pm

Unfortunately, after ~10 hour of uptime the issue appeared again. The system log, again, show nothing in that time frame. However, the kernel log seems to have some odd entries (especially towards the end).

Latency Graph:

Kernel Log:

[    0.000000] Linux version 5.10.165 (builder@buildhost) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r20077-0e2fdba0dd) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Mon Jan 30 09:26:18 2023
[    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
[    0.000000] MIPS: machine is Xiaomi Mi Router 4A Gigabit Edition
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] VPE topology {2,2} total 4
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] On node 0 totalpages: 32768
[    0.000000]   Normal zone: 288 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32768 pages, LIFO batch:7
[    0.000000] percpu: Embedded 15 pages/cpu s30480 r8192 d22768 u61440
[    0.000000] pcpu-alloc: s30480 r8192 d22768 u61440 alloc=15*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32480
[    0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Writing ErrCtl register=00006000
[    0.000000] Readback ErrCtl register=00006000
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 119344K/131072K available (7176K kernel code, 630K rwdata, 832K rodata, 1268K init, 243K bss, 11728K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 256
[    0.000000] CPU Clock: 880MHz
[    0.000000] clocksource: GIC: mask: 0xffffffffffffffff max_cycles: 0xcaf478abb4, max_idle_ns: 440795247997 ns
[    0.000013] sched_clock: 64 bits at 880MHz, resolution 1ns, wraps every 4398046511103ns
[    0.007940] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 4343773742 ns
[    0.016999] Calibrating delay loop... 586.13 BogoMIPS (lpj=2930688)
[    0.083163] pid_max: default: 32768 minimum: 301
[    0.087931] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.095134] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.105802] rcu: Hierarchical SRCU implementation.
[    0.110847] dyndbg: Ignore empty _ddebug table in a CONFIG_DYNAMIC_DEBUG_CORE build
[    0.118956] smp: Bringing up secondary CPUs ...
[    0.124252] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.124263] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.124276] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.124351] CPU1 revision is: 0001992f (MIPS 1004Kc)
[    0.178752] Synchronize counters for CPU 1: done.
[    0.211218] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.211227] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.211236] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.211287] CPU2 revision is: 0001992f (MIPS 1004Kc)
[    0.270177] Synchronize counters for CPU 2: done.
[    0.300765] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.300775] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.300784] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.300837] CPU3 revision is: 0001992f (MIPS 1004Kc)
[    0.355364] Synchronize counters for CPU 3: done.
[    0.385232] smp: Brought up 1 node, 4 CPUs
[    0.393485] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.403312] futex hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.410327] pinctrl core: initialized pinctrl subsystem
[    0.418009] NET: Registered protocol family 16
[    0.423723] thermal_sys: Registered thermal governor 'step_wise'
[    0.424605] cpuidle: using governor teo
[    0.441035] rt2880-pinmux pinctrl: there is not valid maps for state default
[    0.454897] FPU Affinity set after 11720 emulations
[    0.481899] clocksource: Switched to clocksource GIC
[    0.488791] NET: Registered protocol family 2
[    0.493510] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.502406] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
[    0.510730] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.518373] TCP bind hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.525347] TCP: Hash tables configured (established 1024 bind 1024)
[    0.531784] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.538264] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.545426] NET: Registered protocol family 1
[    0.549722] PCI: CLS 0 bytes, default 32
[    0.556098] workingset: timestamp_bits=14 max_order=15 bucket_order=1
[    0.566948] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.572727] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.583813] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.593145] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.598984] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.604892] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.611305] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.622116] printk: console [ttyS0] disabled
[    0.626418] 1e000c00.uartlite: ttyS0 at MMIO 0x1e000c00 (irq = 19, base_baud = 3125000) is a 16550A
[    0.635419] printk: console [ttyS0] enabled
[    0.643687] printk: bootconsole [early0] disabled
[    0.656297] spi-mt7621 1e000b00.spi: sys_freq: 220000000
[    0.663261] spi-nor spi0.0: gd25q128 (16384 Kbytes)
[    0.668209] 8 fixed-partitions partitions found on MTD device spi0.0
[    0.674624] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    0.681955] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    0.690083] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    0.697427] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    0.704973] Creating 8 MTD partitions on "spi0.0":
[    0.709757] 0x000000000000-0x000000030000 : "u-boot"
[    0.715718] 0x000000030000-0x000000040000 : "u-boot-env"
[    0.722012] 0x000000040000-0x000000050000 : "Bdata"
[    0.727938] 0x000000050000-0x000000060000 : "factory"
[    0.734106] 0x000000060000-0x000000070000 : "crash"
[    0.739940] 0x000000070000-0x000000080000 : "cfg_bak"
[    0.746018] 0x000000080000-0x000000180000 : "overlay"
[    0.752139] 0x000000180000-0x000001000000 : "firmware"
[    0.758523] 2 uimage-fw partitions found on MTD device firmware
[    0.764502] Creating 2 MTD partitions on "firmware":
[    0.769461] 0x000000000000-0x0000002954b9 : "kernel"
[    0.774458] mtd: partition "kernel" doesn't end on an erase/write block -- force read-only
[    0.783828] 0x0000002954b9-0x000000e80000 : "rootfs"
[    0.788794] mtd: partition "rootfs" doesn't start on an erase/write block boundary -- force read-only
[    0.799012] mtd: setting mtd9 (rootfs) as root device
[    0.804254] 1 squashfs-split partitions found on MTD device rootfs
[    0.810423] 0x000000630000-0x000000e80000 : "rootfs_data"
[    0.863255] mt7530 mdio-bus:1f: MT7530 adapts as multi-chip module
[    0.874086] mtk_soc_eth 1e100000.ethernet eth0: mediatek frame engine at 0xbe100000, irq 21
[    0.883550] mtk_soc_eth 1e100000.ethernet wan: mediatek frame engine at 0xbe100000, irq 21
[    0.892887] i2c /dev entries driver
[    0.898947] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
[    0.905701] mt7621-pci 1e140000.pcie:   No bus range found for /pcie@1e140000, using [bus 00-ff]
[    0.914508] mt7621-pci 1e140000.pcie:      MEM 0x0060000000..0x006fffffff -> 0x0000000000
[    0.922688] mt7621-pci 1e140000.pcie:       IO 0x001e160000..0x001e16ffff -> 0x0000000000
[    0.930924] mt7621-pci 1e140000.pcie: Parsing DT failed
[    0.938599] NET: Registered protocol family 10
[    0.945005] Segment Routing with IPv6
[    0.948761] NET: Registered protocol family 17
[    0.953339] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    0.966646] 8021q: 802.1Q VLAN Support v1.8
[    0.974329] mt7530 mdio-bus:1f: MT7530 adapts as multi-chip module
[    1.023537] mt7530 mdio-bus:1f lan2 (uninitialized): PHY [mt7530-0:02] driver [MediaTek MT7530 PHY] (irq=26)
[    1.035796] mt7530 mdio-bus:1f lan1 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7530 PHY] (irq=27)
[    1.048863] mt7530 mdio-bus:1f: configuring for fixed/rgmii link mode
[    1.059222] DSA: tree 0 setup
[    1.062518] rt2880-pinmux pinctrl: pcie is already enabled
[    1.068066] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
[    1.074780] mt7621-pci 1e140000.pcie:   No bus range found for /pcie@1e140000, using [bus 00-ff]
[    1.083575] mt7621-pci 1e140000.pcie:      MEM 0x0060000000..0x006fffffff -> 0x0000000000
[    1.091726] mt7621-pci 1e140000.pcie:       IO 0x001e160000..0x001e16ffff -> 0x0000000000
[    1.099988] mt7621-pci-phy 1e149000.pcie-phy: PHY for 0xbe149000 (dual port = 1)
[    1.107742] mt7621-pci-phy 1e14a000.pcie-phy: PHY for 0xbe14a000 (dual port = 0)
[    1.115394] mt7621-pci 1e140000.pcie: failed to parse bus ranges property: -22
[    1.222817] mt7621-pci-phy 1e149000.pcie-phy: Xtal is 40MHz
[    1.228385] mt7621-pci-phy 1e14a000.pcie-phy: Xtal is 40MHz
[    1.334090] mt7621-pci 1e140000.pcie: pcie2 no card, disable it (RST & CLK)
[    1.341026] mt7621-pci 1e140000.pcie: PCIE0 enabled
[    1.345904] mt7621-pci 1e140000.pcie: PCIE1 enabled
[    1.350766] mt7621-pci 1e140000.pcie: PCI coherence region base: 0x60000000, mask/settings: 0xf0000002
[    1.360215] mt7621-pci 1e140000.pcie: PCI host bridge to bus 0000:00
[    1.366581] pci_bus 0000:00: root bus resource [io  0x1e160000-0x1e16ffff]
[    1.373449] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
[    1.380296] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.385779] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff] (bus address [0x00000000-0x0fffffff])
[    1.395976] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
[    1.402004] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.408247] pci 0000:00:00.0: reg 0x14: [mem 0x60400000-0x6040ffff]
[    1.414568] pci 0000:00:00.0: supports D1
[    1.418561] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.424739] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
[    1.430758] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.437033] pci 0000:00:01.0: reg 0x14: [mem 0x60410000-0x6041ffff]
[    1.443355] pci 0000:00:01.0: supports D1
[    1.447350] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
[    1.454631] pci 0000:01:00.0: [14c3:7662] type 00 class 0x028000
[    1.460663] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    1.467724] pci 0000:01:00.0: reg 0x10: [mem size 0x00100000 64bit]
[    1.474020] pci 0000:01:00.0: reg 0x30: initial BAR value 0x00000000 invalid
[    1.481038] pci 0000:01:00.0: reg 0x30: [mem size 0x00010000 pref]
[    1.487314] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    1.494757] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    1.499980] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
[    1.506072] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    1.512851] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
[    1.520048] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.526915] pci 0000:02:00.0: [14c3:7603] type 00 class 0x028000
[    1.532955] pci 0000:02:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    1.539977] pci 0000:02:00.0: reg 0x10: [mem size 0x00100000]
[    1.545855] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
[    1.553263] pci 0000:00:01.0: PCI bridge to [bus 02-ff]
[    1.558483] pci 0000:00:01.0:   bridge window [io  0x0000-0x0fff]
[    1.564575] pci 0000:00:01.0:   bridge window [mem 0x60200000-0x602fffff]
[    1.571336] pci 0000:00:01.0:   bridge window [mem 0x60300000-0x603fffff pref]
[    1.578548] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.585215] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
[    1.591802] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
[    1.598751] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
[    1.605354] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
[    1.612306] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    1.619066] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    1.626276] pci 0000:00:01.0: BAR 8: assigned [mem 0x60200000-0x602fffff]
[    1.633056] pci 0000:00:01.0: BAR 9: assigned [mem 0x60300000-0x603fffff pref]
[    1.640247] pci 0000:00:00.0: BAR 1: assigned [mem 0x60400000-0x6040ffff]
[    1.647037] pci 0000:00:01.0: BAR 1: assigned [mem 0x60410000-0x6041ffff]
[    1.653819] pci 0000:00:00.0: BAR 7: assigned [io  0x1e160000-0x1e160fff]
[    1.660578] pci 0000:00:01.0: BAR 7: assigned [io  0x1e161000-0x1e161fff]
[    1.667363] pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x600fffff 64bit]
[    1.674672] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    1.681885] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.686830] pci 0000:00:00.0:   bridge window [io  0x1e160000-0x1e160fff]
[    1.693606] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    1.700364] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
[    1.707586] pci 0000:02:00.0: BAR 0: assigned [mem 0x60200000-0x602fffff]
[    1.714366] pci 0000:00:01.0: PCI bridge to [bus 02]
[    1.719310] pci 0000:00:01.0:   bridge window [io  0x1e161000-0x1e161fff]
[    1.726084] pci 0000:00:01.0:   bridge window [mem 0x60200000-0x602fffff]
[    1.732860] pci 0000:00:01.0:   bridge window [mem 0x60300000-0x603fffff pref]
[    1.744929] mt7530 mdio-bus:1f: Link is Up - 1Gbps/Full - flow control off
[    1.755131] VFS: Mounted root (squashfs filesystem) readonly on device 31:9.
[    1.766317] Freeing unused kernel memory: 1268K
[    1.770846] This architecture does not have kernel memory protection.
[    1.777417] Run /sbin/init as init process
[    1.781494]   with arguments:
[    1.781499]     /sbin/init
[    1.781504]   with environment:
[    1.781510]     HOME=/
[    1.781514]     TERM=linux
[    2.338777] init: Console is alive
[    2.342716] init: - watchdog -
[    3.023326] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.140408] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.162286] init: - preinit -
[    4.104394] random: jshn: uninitialized urandom read (4 bytes read)
[    4.257652] random: jshn: uninitialized urandom read (4 bytes read)
[    4.295379] random: jshn: uninitialized urandom read (4 bytes read)
[    4.539741] mtk_soc_eth 1e100000.ethernet eth0: configuring for fixed/rgmii link mode
[    4.548442] mtk_soc_eth 1e100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[    4.552246] mt7530 mdio-bus:1f lan1: configuring for phy/gmii link mode
[    4.563683] 8021q: adding VLAN 0 to HW filter on device lan1
[    4.572471] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.577673] mt7530 mdio-bus:1f lan1: Link is Up - 1Gbps/Full - flow control off
[    7.585058] IPv6: ADDRCONF(NETDEV_CHANGE): lan1: link becomes ready
[    8.910477] jffs2: notice: (563) jffs2_build_xattr_subsystem: complete building xattr subsystem, 45 of xdatum (43 unchecked, 2 orphan) and 71 of xref (2 dead, 0 orphan) found.
[    8.930611] mount_root: switching to jffs2 overlay
[    8.940596] overlayfs: upper fs does not support tmpfile.
[    8.963986] urandom-seed: Seeding with /etc/urandom.seed
[    9.146949] mt7530 mdio-bus:1f lan1: Link is Down
[    9.166812] procd: - early -
[    9.169936] procd: - watchdog -
[    9.822063] procd: - watchdog -
[    9.826729] procd: - ubus -
[    9.914256] random: ubusd: uninitialized urandom read (4 bytes read)
[    9.923197] random: ubusd: uninitialized urandom read (4 bytes read)
[    9.930132] random: ubusd: uninitialized urandom read (4 bytes read)
[    9.948209] procd: - init -
[   10.181987] random: crng init done
[   10.185421] random: 23 urandom warning(s) missed due to ratelimiting
[   10.807701] kmodloader: loading kernel modules from /etc/modules.d/*
[   10.925982] urngd: v1.0.2 started.
[   11.108716] GACT probability on
[   11.113449] Mirror/redirect action on
[   11.131561] u32 classifier
[   11.134371]     input device check on
[   11.138015]     Actions configured
[   11.165342] Loading modules backported from Linux version v5.15.81-0-ge4a7232c917c
[   11.172991] Backport generated by backports.git v5.15.81-1-0-ge1867d55
[   11.228447] xt_time: kernel timezone is -0000
[   11.464808] mt7621-pci 1e140000.pcie: bus=2 slot=1 irq=23
[   11.470274] pci 0000:00:01.0: enabling device (0006 -> 0007)
[   11.475971] mt7603e 0000:02:00.0: enabling device (0000 -> 0002)
[   11.482170] mt7603e 0000:02:00.0: ASIC revision: 76030010
[   12.513254] mt7603e 0000:02:00.0: Firmware Version: CL173766_R
[   12.519092] mt7603e 0000:02:00.0: Build Time: 20171110105932
[   12.561899] mt7603e 0000:02:00.0: firmware init done
[   12.730534] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   12.744157] mt7621-pci 1e140000.pcie: bus=1 slot=0 irq=22
[   12.749594] pci 0000:00:00.0: enabling device (0006 -> 0007)
[   12.755298] mt76x2e 0000:01:00.0: enabling device (0000 -> 0002)
[   12.761524] mt76x2e 0000:01:00.0: ASIC revision: 76120044
[   13.532232] mt76x2e 0000:01:00.0: ROM patch build: 20141115060606a
[   13.543006] mt76x2e 0000:01:00.0: Firmware Version: 0.0.00
[   13.548521] mt76x2e 0000:01:00.0: Build: 1
[   13.552676] mt76x2e 0000:01:00.0: Build Time: 201706151016____
[   13.582016] mt76x2e 0000:01:00.0: Firmware running!
[   13.588759] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   13.659634] PPP generic driver version 2.4.2
[   13.666077] NET: Registered protocol family 24
[   13.679889] kmodloader: done loading kernel modules from /etc/modules.d/*
[   22.223924] mtk_soc_eth 1e100000.ethernet eth0: Link is Down
[   22.239225] mtk_soc_eth 1e100000.ethernet eth0: configuring for fixed/rgmii link mode
[   22.247448] mtk_soc_eth 1e100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   22.255140] mt7530 mdio-bus:1f lan1: configuring for phy/gmii link mode
[   22.262797] 8021q: adding VLAN 0 to HW filter on device lan1
[   22.271151] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   22.278856] br-lan: port 1(lan1) entered blocking state
[   22.284243] br-lan: port 1(lan1) entered disabled state
[   22.290441] device lan1 entered promiscuous mode
[   22.295299] device eth0 entered promiscuous mode
[   22.327472] mt7530 mdio-bus:1f lan2: configuring for phy/gmii link mode
[   22.334772] 8021q: adding VLAN 0 to HW filter on device lan2
[   22.344637] br-lan: port 2(lan2) entered blocking state
[   22.349925] br-lan: port 2(lan2) entered disabled state
[   22.356883] device lan2 entered promiscuous mode
[   22.371034] mtk_soc_eth 1e100000.ethernet wan: PHY [mdio-bus:04] driver [MediaTek MT7530 PHY] (irq=POLL)
[   22.380665] mtk_soc_eth 1e100000.ethernet wan: configuring for phy/rgmii-rxid link mode
[   24.452212] mtk_soc_eth 1e100000.ethernet wan: Link is Up - 100Mbps/Full - flow control off
[   24.907056] mt7530 mdio-bus:1f lan1: Link is Up - 1Gbps/Full - flow control off
[   25.960722] br-lan: port 3(wlan0) entered blocking state
[   25.966198] br-lan: port 3(wlan0) entered disabled state
[   25.972475] device wlan0 entered promiscuous mode
[   25.977659] br-lan: port 3(wlan0) entered blocking state
[   25.983320] br-lan: port 3(wlan0) entered forwarding state
[   25.989713] br-lan: port 3(wlan0) entered disabled state
[   25.995867] IPv6: ADDRCONF(NETDEV_CHANGE): wan: link becomes ready
[   26.003286] br-lan: port 1(lan1) entered blocking state
[   26.008557] br-lan: port 1(lan1) entered forwarding state
[   26.014737] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   26.023519] IPv6: ADDRCONF(NETDEV_CHANGE): wan.35: link becomes ready
[   26.531027] br-lan: port 4(wlan1) entered blocking state
[   26.536440] br-lan: port 4(wlan1) entered disabled state
[   26.542853] device wlan1 entered promiscuous mode
[   26.548128] br-lan: port 4(wlan1) entered blocking state
[   26.553541] br-lan: port 4(wlan1) entered forwarding state
[   27.012392] br-lan: port 4(wlan1) entered disabled state
[   27.374271] pppoe-wan: renamed from ppp0
[   32.569064] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   32.576055] br-lan: port 4(wlan1) entered blocking state
[   32.581440] br-lan: port 4(wlan1) entered forwarding state
[   37.153263] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   37.159921] br-lan: port 3(wlan0) entered blocking state
[   37.165673] br-lan: port 3(wlan0) entered forwarding state

@db260179

db260179 · January 31, 2023, 3:48pm

Try the openwrt master snapshot image, if the same possible faulty router switch.

chronos · February 11, 2023, 4:57pm

Unfortunately, the snapshot version (r21962-d649a8dcef) also has the same issue. However, this time it was fine for 10 days straight until it happened again today (around four times so far)

How would this account for 10 days without the bug. Additionally, I have to note that Ethernet is completely unaffected by this issue.

The closest thing I got to reproducing this issue is doing a channel analysis on the router. Which leads me to believe that there might be an external trigger causing the AP to go off channel. I have tried uci set wireless.radio0.noscan='1' but it did not help.

db260179 · February 13, 2023, 9:10am

Being that its a wireless issue, external interference will cause this kind of issue - regardless of channel interference.

Make sure to set the correct country code and do periodic channel scan - i use wifi-schedule in luci and do a monday,wednesday and saturday schedule.

I've also have the model without the shielding, so added aluminum tape to the top underside of the case to help with keeping interfence away.

Rorito · February 13, 2023, 10:32pm

I have the same router… I have the same problems as you mention… It is not a software problem, my connection is lost in OpenWrt and also the official firmware either from China or the Global firmware… I think it is a factory or batch defect… Right now I'm using it as a Switch... The problem is more noticeable in 2.4ghz and with few devices it feels stable but when more are connected it starts to fail and the Wi-Fi signal even disappears... Mine doesn't have the metal shields...

chronos · February 14, 2023, 7:44am

Thank you guys for helping me with troubleshooting this cursed router. I really appreciate it.

That's what I thought initially. However, I noticed that upon restarting the WiFi interface the problem always instantly disappears and the router operates well for a good time after that. I have also tried moving my laptop next to the router when I notice the spikes but it didn't help. Is it possible that the RF interference causes the device to misbehave even if the interference source stops later? Also, my model happens to have shielding.

I've been doing a channel scan every time I see the spikes. 90% of the time my channel doesn't overlap. I have noticed auto channel selection to be pretty much useless so I set my channel manually. I also don't use DFS channels.

Do you use it for making channel scans or just to turn the WiFi on/off?

I agree, 2.4 GHz is worse for me as well. However, 2.4 GHz spectrum is extremely noisy and there are no free channels available here. I have noticed the issue mostly happens after some time of usage, almost behaving like a memory leak (no memory leak however). I will try to do a cron job to restart the interface everyday and report my findings here.

chronos · February 22, 2023, 3:00pm

Restarting the WiFi interface daily doesn't seem to matter.

Rorito · March 13, 2023, 2:44am

I wrapped the router with several layers of aluminum foil and did not experience any disconnections or data loss over wifi... This router definitely has design flaws; the earlier versions had protected processors, but the newer ones do not. They've made the device unusable just to save costs... Wrapping it completely with aluminum has improved it, but I don't know if there will be consequences in the long run because it won't be able to cool the air... I also don't know the long-term consequences.

Rorito · March 13, 2023, 5:52am

I just did more tests... When the network is not in use, it's fairly stable, but when there's a lot of traffic on the network, my iPhone and PC connected to the same WiFi (2.4ghz)

and using (iperf), it starts to fail and becomes unstable... I think these routers come defective, besides I see that it even lacks a green transistor and, above all, the shielded processors... I hope not to buy anything Xiaomi again.

Rorito · March 13, 2023, 7:40am

I did a complete reset of the router and ran some tests, and I noticed a lot of stability. I had some Luci apps installed, and perhaps that was causing instability. Definitely wrapping it in aluminum has helped because before the wifi signal would even crash, but now it's quite stable. I think having multiple apps installed can reduce stability. Right now, I have only installed Luci SQM. I tested 5ghz and it is very stable