AQL and the ath10k is *lovely*

when I started this bug I was using -ct-shortbuffers

1 Like

Continuing from AQL and the ath10k is lovely:

The speed also drops drastically as there are more clients - no matter if they are active or not ... Ping is smaller but not stable.

But this is not this... Six clients and block all in short time.

   39.043137] ath10k_ahb: added a new wireless network interface: wlan0 [phy0]
[38781.533558] usb usb2-port1: Cannot enable. Maybe the USB cable is bad?
[78111.449574] ath10k_ahb a000000.wifi: Invalid peer id 283 peer stats buffer
[91846.621269] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91846.621487] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91846.627560] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91846.634848] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91846.642297] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 4
[91846.649408] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 4
[91846.656658] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 4
[91846.663985] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 4
[91846.671276] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91846.678559] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 463 tid 0
[91916.808358] ath10k_warn: 102 callbacks suppressed
[91916.808521] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 3
[91916.812232] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 3
[91916.819333] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 3
[91916.826620] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 3
[91916.834339] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 0
[91916.841251] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 0
[91916.848540] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 0
[91916.855757] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 0
[91916.863122] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 3
[91916.870410] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 287 tid 0
[92166.985614] ath10k_warn: 319 callbacks suppressed
[92166.985788] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92166.989482] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92166.996563] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92167.003831] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92167.011352] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 4
[92167.018442] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 4
[92167.025731] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 4
[92167.033023] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 4
[92167.040281] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92167.047600] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 291 tid 0
[92247.485593] ath10k_warn: 110 callbacks suppressed
[92247.485661] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92247.489474] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92247.496707] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92247.503944] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92247.511703] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 0
[92247.518463] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 0
[92247.525853] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 0
[92247.533109] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 0
[92247.540333] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92247.547691] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 296 tid 4
[92356.123155] ath10k_warn: 510 callbacks suppressed
[92356.123324] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.127033] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.134129] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.141429] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.149120] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.155987] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.163350] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.170637] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.177848] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92356.185233] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 298 tid 0
[92418.158329] ath10k_warn: 82 callbacks suppressed
[92418.158497] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 299 tid 0
[92418.162199] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 299 tid 0
[92418.169221] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 299 tid 0
[92418.176509] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 299 tid 0
[92492.669923] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92492.670144] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92492.676213] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92492.683725] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 3
[92492.690832] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 3
[92492.698125] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 3
[92492.705344] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 3
[92492.712706] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92492.719994] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92492.727276] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 300 tid 0
[92510.540171] ath10k_warn: 65 callbacks suppressed
[92510.540343] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.544042] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.551064] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.558348] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.566015] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.572986] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.580270] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.587562] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.594916] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92510.602662] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 301 tid 0
[92576.659154] ath10k_warn: 43 callbacks suppressed
[92576.659321] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.663020] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.670044] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.677336] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.685038] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.691962] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.699252] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.706452] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.713721] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0
[92576.721031] ath10k_ahb a000000.wifi: failed to lookup txq for peer_id 302 tid 0

I will test @nbd patches new (from number 330 to 336 and 337) now...

I've always had the errors above on my R7800, my dmesg is full of them (on mainline driver and firmware, not ct). They seem to be harmless, but maybe they do point to issues that are not immediately noticeable.

ipq40xx (ea6350v3) may behave differently as ipq806x....
It throws errors and there is blocking of wireless users....

Brief testing of the new patches- it's better in terms of latency, but the speed is just as crappy

Hi @nbd, I noticed that the ATF/AQL codes manages the airtime by each station's Access Categories (AC) queues. Wouldn't this results in certain AC queues using excessive airtime? If a station has loads of packet in each of the 4 AC queues, would this result in one station using more than it's fair share of airtime?

I would think that airtime should be accounted for per station, regardless of AC and for each station, the packets in each of AC queues getting sent out by weighted priority in round-robin fashion, e.g. VO > VI > BE > BK, where each AC queue will at least get one of their packets sent when it's the station's turn to transmit, provided it has enough deficit.

Hope I'm making sense above.

Oh, you noticed that. I was always demonstrating how bad things could still get if you actually used all four queues, as part of the unfunded phase II of the make-wifi-fast project. There is another, more devastating queue, the multicast queue, which at the time had unlimited depth (it's been 6+ years, maybe this was addressed), where the standard had always said to rate limit and "spread out somehow".

What queue gets used is not as simple as weighting them via priority in sw, you have to have them "in the hardware" to win the airtime election ( https://en.wikipedia.org/wiki/IEEE_802.11e-2005 ) which is over a very precise 10us guard interval.

So great! to do this, we end up with 4 queues in hw, 4 ready to go, at a loss in "smartness" as to what to schedule next. A compromise might have been only filling two - at most - hw queues at a time.

In practice, until recently (see qosify), the other access classes were largely unused. Multicast was (and IMHO remains) the biggest problem backward compatible wifi has. I like to think that fq_codel derived, BE scheduling only, doing packet aggregation well, is the right approach for APs, but that clients could benefit from the VI class especially. We showed excellent VOIP MOS with just BE in "ending the anomaly".

This does not mean I'm still right, particularly for wifi-6. I still hath no knowledge of a working gang scheduler on that front tho.

1 Like

Hmm ... wireless traffic is indeed a complicated subject.

From what I understand of the ath10k driver code, it looks like it will push frames to the firmware according to how mac80211 controls it via the transmission queues ATF/AQL algo. So if mac80211 pushes frames in order of the AC priority, I would think the ath10k firmware will similarly push it out in the same relative order. I'm just guessing here, as I have no idea how the firmware operates.

Edit: Re-reading what you wrote made me realised that it is more complicated than that. With a single AC, there may be multiple streams of traffic, where fq_codel will help. So in additional to ATF/AQL, there's also fq_codel to split packets into streams to ensure all streams have their fair share of traffic. This is indeed complicated.

2 Likes

FWIW, I do see a change with commit 6d49a25 (I still need to load an image the that includes the latest patch 7bf5233a "improve the quantum doohickey" to see if that affects me so don't jump on this).

The throughput streaming from the AP to a fast client is now almost half what I've observed before (~270 mbps comparted to ~500 mbps - even when it is the only client on the 5 GHz band btw). Interestingly, streaming from the same client to the AP is unchanged at about 500 mbps.

EDIT Restarting the client (it's a mac) fixed the slowdown. AP uptime is almost 5 days. Once I get an updated image on, I'll report back if this happens again.

I don't think it should be like that - the transfer sucks, it's only good near the router.
Isn't it done in such a way that the closer the clients are to the router, the more weight they are assigned, and thus they take up more bandwidth (even if they don't use it) - shouldn't it be the other way round?

I don't care about access times of 25-28 ms (earlier often 30-32), when transfer from average 18-20Mbps dropped to ~1,5-2Mbps (that is 10x measuring with the simplest speedtest). Iperf (after TCP) used to show 22-25Mbps in this location, but now it shows 15-17Mbps (even so, the speedtest should not differ too much from iperf). Either the transfer is messed up, or iperf is miscalculating the bandwidth.

PS. At least no errors yet (and there were already six clients connected).

EDIT: Maybe... Link

Ya, if your testing kernel 5.15 there may be other wifi issues so testing AQL/ATF updates may be difficult (but still worthwhile). Note ansuel has updated the patches for 5.15 here and would love some feedback. If your not already using those, you might want give them a try.

EDIT: I've loaded a new image with the latest AQL/ATF patches on master included. No issues so far.

1 Like

I just ran a few flent rtt_fair4be tests (a 5 GHz wifi client to 3 hosts on my lan and a netem qdisc on two of the lan hosts) both with aql enabled and with it disabled. The results are nice to see as they demonstrate AQL making a positive difference.

With AQL enabled:
Average ping: 98 ms +/- 1 ms (6 observations)

After echo 0 > /sys/kernel/debug/ieee80211/phy0/aql_enable to "disable" AQL:
Average ping: 130 ms +/- 6 ms (4 observations)

Note: this relates to mt76, but I see it somehow linked to this topic.

Unifi NanoHD running latest r19937 snapshot. Since latest @nbd patches, I've seen huge increase periodic latency.

Testing environment:

  • 12 devices 2.4 GHz and 5 GHz mixed idling.
  • Macbook Pro (Silicon) 2x2 MIMO connected to AP on 5GHz 3 metres away from it (MCS9).
  • RBPi4 running netserver for flent.
  • AP connected to RBPi4 using 1 Gbit LAN.

Laptop connection:
image

Basic ping testing:

64 bytes from 192.168.1.2: icmp_seq=345 ttl=64 time=3023.586 ms
64 bytes from 192.168.1.2: icmp_seq=346 ttl=64 time=2018.087 ms
64 bytes from 192.168.1.2: icmp_seq=347 ttl=64 time=1012.499 ms
64 bytes from 192.168.1.2: icmp_seq=348 ttl=64 time=3024.955 ms
64 bytes from 192.168.1.2: icmp_seq=349 ttl=64 time=2019.174 ms
64 bytes from 192.168.1.2: icmp_seq=350 ttl=64 time=1013.527 ms
64 bytes from 192.168.1.2: icmp_seq=351 ttl=64 time=7.727 ms
64 bytes from 192.168.1.2: icmp_seq=352 ttl=64 time=7.046 ms
64 bytes from 192.168.1.2: icmp_seq=353 ttl=64 time=5.449 ms
64 bytes from 192.168.1.2: icmp_seq=354 ttl=64 time=6.738 ms
64 bytes from 192.168.1.2: icmp_seq=355 ttl=64 time=7.335 ms
64 bytes from 192.168.1.2: icmp_seq=356 ttl=64 time=5.413 ms
64 bytes from 192.168.1.2: icmp_seq=357 ttl=64 time=6.813 ms
64 bytes from 192.168.1.2: icmp_seq=358 ttl=64 time=6.826 ms
64 bytes from 192.168.1.2: icmp_seq=359 ttl=64 time=9.901 ms
64 bytes from 192.168.1.2: icmp_seq=360 ttl=64 time=6.664 ms
64 bytes from 192.168.1.2: icmp_seq=361 ttl=64 time=5.515 ms
64 bytes from 192.168.1.2: icmp_seq=362 ttl=64 time=5.273 ms
64 bytes from 192.168.1.2: icmp_seq=363 ttl=64 time=6.578 ms
64 bytes from 192.168.1.2: icmp_seq=364 ttl=64 time=3.909 ms
64 bytes from 192.168.1.2: icmp_seq=365 ttl=64 time=6.970 ms
64 bytes from 192.168.1.2: icmp_seq=366 ttl=64 time=7.150 ms
64 bytes from 192.168.1.2: icmp_seq=367 ttl=64 time=6.911 ms
64 bytes from 192.168.1.2: icmp_seq=368 ttl=64 time=6.882 ms
64 bytes from 192.168.1.2: icmp_seq=369 ttl=64 time=6.812 ms
64 bytes from 192.168.1.2: icmp_seq=370 ttl=64 time=5.465 ms
64 bytes from 192.168.1.2: icmp_seq=371 ttl=64 time=5.622 ms
64 bytes from 192.168.1.2: icmp_seq=372 ttl=64 time=6.928 ms
64 bytes from 192.168.1.2: icmp_seq=373 ttl=64 time=5.459 ms
64 bytes from 192.168.1.2: icmp_seq=374 ttl=64 time=6.327 ms
64 bytes from 192.168.1.2: icmp_seq=375 ttl=64 time=1007.482 ms
64 bytes from 192.168.1.2: icmp_seq=376 ttl=64 time=3019.966 ms
64 bytes from 192.168.1.2: icmp_seq=377 ttl=64 time=2014.703 ms
64 bytes from 192.168.1.2: icmp_seq=378 ttl=64 time=1011.934 ms
64 bytes from 192.168.1.2: icmp_seq=379 ttl=64 time=7.749 ms
64 bytes from 192.168.1.2: icmp_seq=380 ttl=64 time=6.970 ms
64 bytes from 192.168.1.2: icmp_seq=381 ttl=64 time=7.015 ms
64 bytes from 192.168.1.2: icmp_seq=382 ttl=64 time=7.245 ms
64 bytes from 192.168.1.2: icmp_seq=383 ttl=64 time=6.960 ms

Waveform testing:

flent rrul_be testing:
image

flent rrul_be run data points (click me to download)

Hope it helps somehow, back to the old version.

1 Like

FWIW I am trying to test the "reverse" of the flent rtt_fair test above (i.e. flent on lan client using "servers" on wifi clients. No netem required here - this is what I get with 3 clients, two of which in somewhat wifi challenged locations.

                                             avg       median          # data pts
 Ping (ms) ICMP1 56//1  :        40.55        41.50 ms              350
 Ping (ms) ICMP2 59     :       308.40       144.00 ms              350
 Ping (ms) ICMP3 56//2  :        40.34        41.60 ms              350
 Ping (ms) ICMP4 nmba   :        30.32        21.70 ms              350
 Ping (ms) avg          :        104.90          N/A ms              350
 TCP download BE1 56//1 :        39.61        38.95 Mbits/s         350
 TCP download BE2 59    :         0.37         0.36 Mbits/s         350
 TCP download BK1 56//2 :        36.61        35.87 Mbits/s         350
 TCP download BK2 nmba  :         9.65         9.21 Mbits/s         350
 TCP download avg       :        21.56          N/A Mbits/s         350
 TCP download fairness  :         0.62          N/A Mbits/s         350
 TCP download sum       :        86.24          N/A Mbits/s         350
 TCP upload BE1 56//1   :        41.27        41.77 Mbits/s         350
 TCP upload BE2 59      :         0.07         0.65 Mbits/s         350
 TCP upload BK1 56//2   :        44.10        44.34 Mbits/s         350
 TCP upload BK2 nmba    :         9.47         9.68 Mbits/s         350
 TCP upload avg         :        23.73          N/A Mbits/s         350
 TCP upload fairness    :         0.60          N/A Mbits/s         350
 TCP upload sum         :        94.91          N/A Mbits/s         350

A single client netperf to server "59" from the lan client does ~80 mbps for wifi client to AP, ~100 mbps for AP to wifi client. This is an example of how "speed" kills my wifi.

nmba is the mac book air which I find problematic to use for testing. For example, running irtt as a server on nmba and connecting to it from a lan client gives different results than running irtt as a client on nmba connecting to a lan server. I'm still working through issues like this so I wouldn't pay too much attention to these results just yet. I'd avoid using the mac if I could.

FYI the above also have a patch to enable ATF on the r7500v2 enabled. I don't think this is making much difference but I'll repeat without that as soon as I can reboot the AP.

1 Like

Did you run the same type of tests before trying out that build with the new patches? Periodic latency could be due to your Wifi card doing periodic background scans looking for other access points. Could you increase the length of your test to 5 minutes (300 seconds)? Would be interesting to see whether this is truly periodic behaviour with a set interval.

PS: @dtaht wrote about the issue of background scans:

1 Like

Yes, I did. But just for the giggles, I just did it again. A few clients active now:

  • 3 computers, one using Parsec connected via WiFi upstairs, one using an active SSH session and another one surfing the web.
  • 2 tables active streaming YouTube.
  • 1 phone surfing Insta.

BTW, very aware of @dtaht's article. I disabled awdl0 interface and Location in my macOS prior to the test.

So without further delay here you have very same tests on 21.02.1.

Laptop connection:
image

awdl0: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
	options=400<CHANNEL_IO>
	ether 0a:78:a3:27:d0:0a
	nd6 options=201<PERFORMNUD,DAD>
	media: autoselect
	status: inactive

Basic ping test:

64 bytes from 192.168.1.2: icmp_seq=0 ttl=64 time=4.653 ms
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=4.732 ms
64 bytes from 192.168.1.2: icmp_seq=2 ttl=64 time=4.686 ms
64 bytes from 192.168.1.2: icmp_seq=3 ttl=64 time=2.513 ms
64 bytes from 192.168.1.2: icmp_seq=4 ttl=64 time=4.746 ms
64 bytes from 192.168.1.2: icmp_seq=5 ttl=64 time=4.732 ms
64 bytes from 192.168.1.2: icmp_seq=6 ttl=64 time=4.873 ms
64 bytes from 192.168.1.2: icmp_seq=7 ttl=64 time=4.773 ms
64 bytes from 192.168.1.2: icmp_seq=8 ttl=64 time=4.656 ms
64 bytes from 192.168.1.2: icmp_seq=9 ttl=64 time=4.707 ms
64 bytes from 192.168.1.2: icmp_seq=10 ttl=64 time=4.834 ms
64 bytes from 192.168.1.2: icmp_seq=11 ttl=64 time=4.710 ms
64 bytes from 192.168.1.2: icmp_seq=12 ttl=64 time=4.762 ms
64 bytes from 192.168.1.2: icmp_seq=13 ttl=64 time=8.866 ms
64 bytes from 192.168.1.2: icmp_seq=14 ttl=64 time=5.626 ms
64 bytes from 192.168.1.2: icmp_seq=15 ttl=64 time=4.823 ms
64 bytes from 192.168.1.2: icmp_seq=16 ttl=64 time=4.348 ms
64 bytes from 192.168.1.2: icmp_seq=17 ttl=64 time=4.740 ms
64 bytes from 192.168.1.2: icmp_seq=18 ttl=64 time=4.214 ms
64 bytes from 192.168.1.2: icmp_seq=19 ttl=64 time=4.028 ms
64 bytes from 192.168.1.2: icmp_seq=20 ttl=64 time=4.770 ms
64 bytes from 192.168.1.2: icmp_seq=21 ttl=64 time=2.345 ms
64 bytes from 192.168.1.2: icmp_seq=22 ttl=64 time=4.932 ms
64 bytes from 192.168.1.2: icmp_seq=23 ttl=64 time=51.487 ms
64 bytes from 192.168.1.2: icmp_seq=24 ttl=64 time=3.765 ms
64 bytes from 192.168.1.2: icmp_seq=25 ttl=64 time=2.843 ms
64 bytes from 192.168.1.2: icmp_seq=26 ttl=64 time=6.010 ms
64 bytes from 192.168.1.2: icmp_seq=27 ttl=64 time=6.184 ms
64 bytes from 192.168.1.2: icmp_seq=28 ttl=64 time=5.708 ms
64 bytes from 192.168.1.2: icmp_seq=29 ttl=64 time=98.807 ms
64 bytes from 192.168.1.2: icmp_seq=30 ttl=64 time=3.304 ms
64 bytes from 192.168.1.2: icmp_seq=31 ttl=64 time=6.213 ms
64 bytes from 192.168.1.2: icmp_seq=32 ttl=64 time=6.107 ms
64 bytes from 192.168.1.2: icmp_seq=33 ttl=64 time=5.326 ms
64 bytes from 192.168.1.2: icmp_seq=34 ttl=64 time=5.833 ms
64 bytes from 192.168.1.2: icmp_seq=35 ttl=64 time=4.779 ms
64 bytes from 192.168.1.2: icmp_seq=36 ttl=64 time=6.005 ms
64 bytes from 192.168.1.2: icmp_seq=37 ttl=64 time=2.852 ms
64 bytes from 192.168.1.2: icmp_seq=38 ttl=64 time=4.339 ms
64 bytes from 192.168.1.2: icmp_seq=39 ttl=64 time=6.190 ms
64 bytes from 192.168.1.2: icmp_seq=40 ttl=64 time=6.128 ms
64 bytes from 192.168.1.2: icmp_seq=41 ttl=64 time=4.679 ms
64 bytes from 192.168.1.2: icmp_seq=42 ttl=64 time=6.317 ms
64 bytes from 192.168.1.2: icmp_seq=43 ttl=64 time=6.079 ms
64 bytes from 192.168.1.2: icmp_seq=44 ttl=64 time=6.284 ms

Waveform test (my connection is 1000/50, SQMed to 312/43):

flent rrul_be test:
image
** Note: you can clearly see WiFi half-duplex nature in these tests, love it.

flent rrul_be run data points (click me to download)

Here you go! Big difference. :wink:

+1; great test, now start moving the clients into positions of different signal quality (or check signal quality and what MCS values they respectively achieve). After all the goal of airtime fairness is clearly not equal rates to all stations, but equal share of airtime, so throughput should scale with relative link speeds.

2 Likes

@amteza, could you please make a build where you remove patches 334 and 335 from package/kernel/mac80211/patches/subsys and try again?

The iOS Wi-Fi issue - Wi-Fi symbol shows device connected, but no internet connectivity pending manual disconnect and reconnect - resurfaced.

Namely with snapshot:

22.03-SNAPSHOT - r19464-a4390ea283

no issues.

But with snapshot:

22.03-SNAPSHOT r19466-08e1812900

issues back.

Therefore I think at least from this commit:

Things were good, but from:

Things went bad again. I am blaming something in these airtime fairness commits:

  • aab535d
  • 08e1812

@ka2107 and @vochong what have your findings been?


@amteza I am not sure of the significance of this, but @richb-hanover-priv did some testing on 22.03-RC4 and found a big latency increase from 8ms to 28ms 90% using WiFi on RT3200 here:

Compare (wired):

with (over Wi-Fi):

image

It's frustrating because we have managed to get bufferbloat-related latency resolved even for variable rate LTE connections (by adjusting CAKE bandwidth on the fly based on RTT measurements) but then Wi-Fi introduces its own latency, undoing our work for Wi-Fi clients.

So now I'm not sure where things work well for the RT3200. My big issue has been iOS connectivity issues which seem to have been worsened with the recent airtime fairness commits.

Hello,

My r7800 just crashed (uptime was about 3 days) when I woke up early this morning and used my phone to check messages. Basically almost no traffic at this early morning (4am) and the crash happened. Could someone please take a look at the crash dumps below. Thanks!

NAME="OpenWrt"
VERSION="SNAPSHOT"
BUILD_ID="r19916+18-326e109f24"
OPENWRT_BOARD="ipq806x/generic"
OPENWRT_ARCH="arm_cortex-a15_neon-vfpv4"

<1>[211787.522720] 8<--- cut here ---
<1>[211787.522758] Unable to handle kernel paging request at virtual address c0b6348c
<1>[211787.524681] pgd = fafc0e29
<1>[211787.531961] [c0b6348c] *pgd=42a1941e(bad)
<0>[211787.534751] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[211787.538911] Modules linked in: nss_ifb ecm ath10k_pci ath10k_core ath wireguard nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet mac80211 libchacha20poly1305 ipt_REJECT curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY sch_cake ppp_async poly1305_arm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_flow_table nf_conntrack_netlink libcurve25519_generic libcrc32c iptable_nat iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat chacha_neon fuse sch_tbf
<4>[211787.539672]  sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs nfs_ssc msdos bonding ifb ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel tun xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512
<4>[211787.609333]  twofish_generic twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv md5 kpp echainiv ecb des_generic libdes cmac authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys trusted tpm
<4>[211787.760086] CPU: 0 PID: 30299 Comm: kworker/0:1 Not tainted 5.10.120 #0
<4>[211787.782315] Hardware name: Generic DT based system
<4>[211787.789010] Workqueue: events dbs_work_handler
<4>[211787.793598] PC is at 0xc0b6348c
<4>[211787.798124] LR is at krait_mux_set_parent+0x60/0x64
<4>[211787.801583] pc : [<c0b6348c>]    lr : [<c0698678>]    psr: 60000033
<4>[211787.806533] sp : c7303d90  ip : 00000000  fp : c1ea9280
<4>[211787.812867] r10: c15b9b18  r9 : 00000000  r8 : c7303dd4
<4>[211787.818164] r7 : 00000002  r6 : ffffffff  r5 : 00000001  r4 : c15f8a58
<4>[211787.823460] r3 : c0b6348d  r2 : c0d9b31c  r1 : 20000013  r0 : 000346dc
<4>[211787.829800] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[211787.836396] Control: 10c5787d  Table: 481d406a  DAC: 00000051
<0>[211787.844034] Process kworker/0:1 (pid: 30299, stack limit = 0x724fb079)
<0>[211787.849676] Stack: (0xc7303d90 to 0xc7304000)
<0>[211787.856195] 3d80:                                     c15f8a64 c0d803a4 ffffffff c069a038
<0>[211787.860726] 3da0: 00000000 c0d803a4 ffffffff c0341ab4 c15b9b00 c0d803a4 c15f93c0 00000002
<0>[211787.868972] 3dc0: 2faf0800 c1ea9400 00000000 c0688a2c c159ae00 c15fa6c0 2faf0800 23c34600
<0>[211787.877218] 3de0: c159ae00 23c34600 c15f93c0 00000000 c1591180 c068ce98 c15f92a8 c1591180
<0>[211787.885464] 3e00: c0698b24 c14d13c0 2faf0800 c1ea9400 00000000 c068cedc c15f93c0 00000000
<0>[211787.893711] 3e20: 23c34600 c1591180 c1ea9480 c1ea9400 00000000 c068d108 23c34600 23c34600
<0>[211787.901957] 3e40: 00000000 ffffffff 23c34600 c0d82604 c15f93c0 c1ea0880 23c34600 dd98a010
<0>[211787.910205] 3e60: 2faf0800 c1ea9480 c1ea9400 c068d288 c1ca3c00 23c34600 dd98a010 2faf0800
<0>[211787.918451] 3e80: c1ea9480 c07b0f0c 00000000 c0dd2ca8 c1ea9438 c1ea94b8 00000000 23c34600
<0>[211787.926696] 3ea0: c7302000 c1ebc400 00000000 c0dd2c70 00000001 000927c0 00000000 00000000
<0>[211787.934944] 3ec0: c7302000 c07b6164 c1ebc400 000c3500 000927c0 000000a1 c1ebc400 c1ea9700
<0>[211787.943191] 3ee0: c1ea9780 c1ea9700 c1ea0f00 c1ea9780 00000000 c07b96fc c1ea9738 00000000
<0>[211787.951437] 3f00: c1ea9704 c0d90460 00000000 00000000 00000000 c07ba3e4 c1ea9738 c272cf80
<0>[211787.959682] 3f20: dd990980 dd993b00 00000000 c03388f4 00000008 dd990998 c272cf80 c272cf94
<0>[211787.967930] 3f40: dd990980 00000008 dd990998 c0d03d00 dd990b40 c0338bdc c0d9bab8 c0d0c164
<0>[211787.976176] 3f60: c272cf80 c8a831c0 c8150900 00000000 c7302000 c0338b68 c272cf80 c839fec4
<0>[211787.984422] 3f80: c8a831e4 c033eac0 00000000 c8150900 c033e964 00000000 00000000 00000000
<0>[211787.992669] 3fa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[211788.000916] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[211788.009162] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[211788.017407] [<c0698678>] (krait_mux_set_parent) from [<c069a038>] (krait_notifier_cb+0x58/0xb8)
<0>[211788.025653] [<c069a038>] (krait_notifier_cb) from [<c0341ab4>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[211788.034592] [<c0341ab4>] (srcu_notifier_call_chain) from [<c0688a2c>] (__clk_notify+0x70/0x94)
<0>[211788.043704] [<c0688a2c>] (__clk_notify) from [<c068ce98>] (clk_change_rate+0xfc/0x2b8)
<0>[211788.052121] [<c068ce98>] (clk_change_rate) from [<c068cedc>] (clk_change_rate+0x140/0x2b8)
<0>[211788.060107] [<c068cedc>] (clk_change_rate) from [<c068d108>] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[211788.068441] [<c068d108>] (clk_core_set_rate_nolock) from [<c068d288>] (clk_set_rate+0x3c/0x170)
<0>[211788.077471] [<c068d288>] (clk_set_rate) from [<c07b0f0c>] (dev_pm_opp_set_rate+0x348/0x674)
<0>[211788.086498] [<c07b0f0c>] (dev_pm_opp_set_rate) from [<c07b6164>] (__cpufreq_driver_target+0x1a0/0x5b4)
<0>[211788.094921] [<c07b6164>] (__cpufreq_driver_target) from [<c07b96fc>] (od_dbs_update+0xcc/0x1a0)
<0>[211788.104030] [<c07b96fc>] (od_dbs_update) from [<c07ba3e4>] (dbs_work_handler+0x38/0x74)
<0>[211788.113060] [<c07ba3e4>] (dbs_work_handler) from [<c03388f4>] (process_one_work+0x1fc/0x470)
<0>[211788.121133] [<c03388f4>] (process_one_work) from [<c0338bdc>] (worker_thread+0x74/0x5d4)
<0>[211788.129550] [<c0338bdc>] (worker_thread) from [<c033eac0>] (kthread+0x15c/0x160)
<0>[211788.137707] [<c033eac0>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<0>[211788.145163] Exception stack(0xc7303fb0 to 0xc7303ff8)
<0>[211788.152288] 3fa0:                                     00000000 00000000 00000000 00000000
<0>[211788.157512] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[211788.165753] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[211788.174000] Code: 6962 5f6f 7274 6d69 (6200) 6f69 
<4>[211788.180841] ---[ end trace b770df971b364141 ]---
<1>[211787.522720] 8<--- cut here ---
<1>[211787.522758] Unable to handle kernel paging request at virtual address c0b6348c
<1>[211787.524681] pgd = fafc0e29
<1>[211787.531961] [c0b6348c] *pgd=42a1941e(bad)
<0>[211787.534751] Internal error: Oops: 8000000d [#1] SMP ARM
<4>[211787.538911] Modules linked in: nss_ifb ecm ath10k_pci ath10k_core ath wireguard nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet mac80211 libchacha20poly1305 ipt_REJECT curve25519_neon cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_conntrack xt_comment xt_cgroup xt_addrtype xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_DSCP xt_CT xt_CLASSIFY sch_cake ppp_async poly1305_arm nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_flow_table nf_conntrack_netlink libcurve25519_generic libcrc32c iptable_nat iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat chacha_neon fuse sch_tbf
<4>[211787.539672]  sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact qca_nss_qdisc qca_nss_pppoe pppoe pppox ppp_generic slhc ledtrig_usbport cryptodev xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_NPT ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 nfsv4 nfsv3 nfs nfs_ssc msdos bonding ifb ip6_udp_tunnel udp_tunnel sit qca_nss_drv qca_nss_gmac oid_registry tunnel4 ip_tunnel tun xfrm_user xfrm_ipcomp af_key xfrm_algo vfat fat lockd sunrpc grace hfsplus hfs cdrom dns_resolver nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 nls_cp1250 wp512
<4>[211787.609333]  twofish_generic twofish_common tgr192 tea serpent_generic khazad cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis xts crypto_user algif_skcipher algif_rng algif_hash algif_aead af_alg sha1_generic seqiv md5 kpp echainiv ecb des_generic libdes cmac authenc uas usb_storage leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_ipq806x_usb ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd ramoops reed_solomon pstore gpio_button_hotplug f2fs ext4 mbcache jbd2 exfat dm_mirror dm_region_hash dm_log dm_crypt dm_mod dax crc32c_generic crc32_generic cbc encrypted_keys trusted tpm
<4>[211787.760086] CPU: 0 PID: 30299 Comm: kworker/0:1 Not tainted 5.10.120 #0
<4>[211787.782315] Hardware name: Generic DT based system
<4>[211787.789010] Workqueue: events dbs_work_handler
<4>[211787.793598] PC is at 0xc0b6348c
<4>[211787.798124] LR is at krait_mux_set_parent+0x60/0x64
<4>[211787.801583] pc : [<c0b6348c>]    lr : [<c0698678>]    psr: 60000033
<4>[211787.806533] sp : c7303d90  ip : 00000000  fp : c1ea9280
<4>[211787.812867] r10: c15b9b18  r9 : 00000000  r8 : c7303dd4
<4>[211787.818164] r7 : 00000002  r6 : ffffffff  r5 : 00000001  r4 : c15f8a58
<4>[211787.823460] r3 : c0b6348d  r2 : c0d9b31c  r1 : 20000013  r0 : 000346dc
<4>[211787.829800] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA Thumb  Segment none
<4>[211787.836396] Control: 10c5787d  Table: 481d406a  DAC: 00000051
<0>[211787.844034] Process kworker/0:1 (pid: 30299, stack limit = 0x724fb079)
<0>[211787.849676] Stack: (0xc7303d90 to 0xc7304000)
<0>[211787.856195] 3d80:                                     c15f8a64 c0d803a4 ffffffff c069a038
<0>[211787.860726] 3da0: 00000000 c0d803a4 ffffffff c0341ab4 c15b9b00 c0d803a4 c15f93c0 00000002
<0>[211787.868972] 3dc0: 2faf0800 c1ea9400 00000000 c0688a2c c159ae00 c15fa6c0 2faf0800 23c34600
<0>[211787.877218] 3de0: c159ae00 23c34600 c15f93c0 00000000 c1591180 c068ce98 c15f92a8 c1591180
<0>[211787.885464] 3e00: c0698b24 c14d13c0 2faf0800 c1ea9400 00000000 c068cedc c15f93c0 00000000
<0>[211787.893711] 3e20: 23c34600 c1591180 c1ea9480 c1ea9400 00000000 c068d108 23c34600 23c34600
<0>[211787.901957] 3e40: 00000000 ffffffff 23c34600 c0d82604 c15f93c0 c1ea0880 23c34600 dd98a010
<0>[211787.910205] 3e60: 2faf0800 c1ea9480 c1ea9400 c068d288 c1ca3c00 23c34600 dd98a010 2faf0800
<0>[211787.918451] 3e80: c1ea9480 c07b0f0c 00000000 c0dd2ca8 c1ea9438 c1ea94b8 00000000 23c34600
<0>[211787.926696] 3ea0: c7302000 c1ebc400 00000000 c0dd2c70 00000001 000927c0 00000000 00000000
<0>[211787.934944] 3ec0: c7302000 c07b6164 c1ebc400 000c3500 000927c0 000000a1 c1ebc400 c1ea9700
<0>[211787.943191] 3ee0: c1ea9780 c1ea9700 c1ea0f00 c1ea9780 00000000 c07b96fc c1ea9738 00000000
<0>[211787.951437] 3f00: c1ea9704 c0d90460 00000000 00000000 00000000 c07ba3e4 c1ea9738 c272cf80
<0>[211787.959682] 3f20: dd990980 dd993b00 00000000 c03388f4 00000008 dd990998 c272cf80 c272cf94
<0>[211787.967930] 3f40: dd990980 00000008 dd990998 c0d03d00 dd990b40 c0338bdc c0d9bab8 c0d0c164
<0>[211787.976176] 3f60: c272cf80 c8a831c0 c8150900 00000000 c7302000 c0338b68 c272cf80 c839fec4
<0>[211787.984422] 3f80: c8a831e4 c033eac0 00000000 c8150900 c033e964 00000000 00000000 00000000
<0>[211787.992669] 3fa0: 00000000 00000000 00000000 c0300148 00000000 00000000 00000000 00000000
<0>[211788.000916] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[211788.009162] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
<0>[211788.017407] [<c0698678>] (krait_mux_set_parent) from [<c069a038>] (krait_notifier_cb+0x58/0xb8)
<0>[211788.025653] [<c069a038>] (krait_notifier_cb) from [<c0341ab4>] (srcu_notifier_call_chain+0x7c/0xf4)
<0>[211788.034592] [<c0341ab4>] (srcu_notifier_call_chain) from [<c0688a2c>] (__clk_notify+0x70/0x94)
<0>[211788.043704] [<c0688a2c>] (__clk_notify) from [<c068ce98>] (clk_change_rate+0xfc/0x2b8)
<0>[211788.052121] [<c068ce98>] (clk_change_rate) from [<c068cedc>] (clk_change_rate+0x140/0x2b8)
<0>[211788.060107] [<c068cedc>] (clk_change_rate) from [<c068d108>] (clk_core_set_rate_nolock+0xb4/0x1f8)
<0>[211788.068441] [<c068d108>] (clk_core_set_rate_nolock) from [<c068d288>] (clk_set_rate+0x3c/0x170)
<0>[211788.077471] [<c068d288>] (clk_set_rate) from [<c07b0f0c>] (dev_pm_opp_set_rate+0x348/0x674)
<0>[211788.086498] [<c07b0f0c>] (dev_pm_opp_set_rate) from [<c07b6164>] (__cpufreq_driver_target+0x1a0/0x5b4)
<0>[211788.094921] [<c07b6164>] (__cpufreq_driver_target) from [<c07b96fc>] (od_dbs_update+0xcc/0x1a0)
<0>[211788.104030] [<c07b96fc>] (od_dbs_update) from [<c07ba3e4>] (dbs_work_handler+0x38/0x74)
<0>[211788.113060] [<c07ba3e4>] (dbs_work_handler) from [<c03388f4>] (process_one_work+0x1fc/0x470)
<0>[211788.121133] [<c03388f4>] (process_one_work) from [<c0338bdc>] (worker_thread+0x74/0x5d4)
<0>[211788.129550] [<c0338bdc>] (worker_thread) from [<c033eac0>] (kthread+0x15c/0x160)
<0>[211788.137707] [<c033eac0>] (kthread) from [<c0300148>] (ret_from_fork+0x14/0x2c)
<0>[211788.145163] Exception stack(0xc7303fb0 to 0xc7303ff8)
<0>[211788.152288] 3fa0:                                     00000000 00000000 00000000 00000000
<0>[211788.157512] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
<0>[211788.165753] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
<0>[211788.174000] Code: 6962 5f6f 7274 6d69 (6200) 6f69 
<4>[211788.180841] ---[ end trace b770df971b364141 ]---
<0>[211788.193774] Kernel panic - not syncing: Fatal exception
<2>[211788.193816] CPU1: stopping
<4>[211788.198153] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D           5.10.120 #0
<4>[211788.200670] Hardware name: Generic DT based system
<4>[211788.208325] [<c030e46c>] (unwind_backtrace) from [<c030a204>] (show_stack+0x14/0x20)
<4>[211788.213098] [<c030a204>] (show_stack) from [<c0632ea8>] (dump_stack+0x94/0xa8)
<4>[211788.221081] [<c0632ea8>] (dump_stack) from [<c030d190>] (do_handle_IPI+0x140/0x184)
<4>[211788.228196] [<c030d190>] (do_handle_IPI) from [<c030d1f0>] (ipi_handler+0x1c/0x2c)
<4>[211788.236184] [<c030d1f0>] (ipi_handler) from [<c037184c>] (__handle_domain_irq+0x90/0xf4)
<4>[211788.243564] [<c037184c>] (__handle_domain_irq) from [<c064c154>] (gic_handle_irq+0x90/0xb8)
<4>[211788.251896] [<c064c154>] (gic_handle_irq) from [<c0300b8c>] (__irq_svc+0x6c/0x90)
<4>[211788.260393] Exception stack(0xc146df18 to 0xc146df60)
<4>[211788.267775] df00:                                                       00000000 0000c09e
<4>[211788.272918] df20: 1cd58000 dd99fd80 00000000 c809dbc0 c1c69040 00000000 dd99f030 0000c09e
<4>[211788.281163] df40: 00000000 0000c09e 365d2ec0 c146df68 c07bd41c c07bd43c 60000013 ffffffff
<4>[211788.289399] [<c0300b8c>] (__irq_svc) from [<c07bd43c>] (cpuidle_enter_state+0x180/0x380)
<4>[211788.297643] [<c07bd43c>] (cpuidle_enter_state) from [<c07bd68c>] (cpuidle_enter+0x3c/0x5c)
<4>[211788.305888] [<c07bd68c>] (cpuidle_enter) from [<c034e678>] (do_idle+0x208/0x2a4)
<4>[211788.314046] [<c034e678>] (do_idle) from [<c034e9d0>] (cpu_startup_entry+0x1c/0x20)
<4>[211788.321685] [<c034e9d0>] (cpu_startup_entry) from [<423015ac>] (0x423015ac)

Your crash is likely caused by the Krait CPU clock bug. Doesn’t look like it’s related to mac80211.

1 Like