AQL and the ath10k is *lovely*

For Developers
775

Forgot to do a proper Internet + SQM testing (312/47 Mbps), see below 'rrul_be r20089 mt76':

result
result1280×960 383 KB

Approx. ≈5 ms variation latency under load. Very good "meh" job! :wink:

Apple Network Quality test for reference.

@reaper$ ➜  ~ networkQuality -v
==== SUMMARY ====
Upload capacity: 34.297 Mbps
Download capacity: 274.261 Mbps
Upload flows: 16
Download flows: 12
Responsiveness: High (3310 RPM)
Base RTT: 9
Start: 16/7/2022, 1:26:40 pm
End: 16/7/2022, 1:26:51 pm
OS Version: Version 12.4 (Build 21F79)

And for comparison, upping download SQM limit to 800/47 Mbps, so WiFi is choking, see below 'rrul_be r20089 mt76':

result
result1280×960 416 KB

Another very good 'meh' job. :slight_smile: ≈20 ms variation latency.

Apple Network Quality test for reference.

@reaper$ ➜  ~ networkQuality -v
==== SUMMARY ====
Upload capacity: 29.917 Mbps
Download capacity: 331.873 Mbps
Upload flows: 16
Download flows: 20
Responsiveness: High (1063 RPM)
Base RTT: 16
Start: 16/7/2022, 1:27:54 pm
End: 16/7/2022, 1:28:08 pm
OS Version: Version 12.4 (Build 21F79)
776

This is because Location Services are on. Just realised it a few moments ago thanks to a link were it mentions it that @ka2107 shared with me. In my machines they are always off as I use them to play from time to time in GeForce NOW.

777

Once things are this good, it is helpful to zoom out to see bigger spikes. The all, rather than all_scaled, graph does this. Also, could you post the .flent.gz files for future reference? I do hope with some stuff waiting in the wings, we can improve this still further.

I am disturbed by how much location services interrupts flows, and wonder how many users leave it on, not knowing it's causing them other trouble. What is the actual width of the pulse? It looks to me to be nearly 2 seconds.... the irtt plot we are using for starlink can help here.

778

Sorry if this is unrelated, but I just noticed that for an AP connected to my main RT3200 router on OpenWrt 22.03-SNAPSHOT r19541-ec9f82fa18 via 5Ghz over WDS, if I connect to AP over 2.4Ghz the ssh speed is very slow (as in I type and it takes a while for characters to appear) on idle WiFi load, whereas it becomes very reactive when I run a speed test.

image
image1392×643 74.3 KB

Is this something to do with what is being discussed on this thread?

779

hard to say. If you have applied nbd's latest patch to deal with multicast, I'd first suggest you turn off your wifi interface power save on the device and the AP if they are on. If that doesn't help then idk, maybe.

780

I have up until this:

I have a bunch of smart plugs that send out multicast packets on 2.4.

In any case when connecting to AP over 2.4, then ssh from laptop to main router is periodically painfully slow, but internet speeds seem normal. When connected to AP over 5, then ssh from laptop to main router is normal.

781

This seems like a new symptom and I suppose it's possible that its related. You could try disabling AQL and see if the symptom goes away.

1 Like
782

Since this is a WDS setup, should I disable AQL on main router and AP's or just AP through which I connect?

783

yes, meaning try one, then the other, then both and see what happens.

Note just disabling AQL might not do anything as the bug might be more closely related to ATF. Disabling AQL is easier to test first.

784

OK - issue goes away and I get normal ssh when I issue:

echo 0 > /sys/kernel/debug/ieee80211/phy0/aql_enable

on the AP only.

785

Give it some time to see if the symptom comes back (maybe as much as few days). Others have observed (before the latest fixes) that the latency symptoms can take time develop or come back after turning off AQL. Regardless, ping @nbd with some details on the AP (device, what's it running, does it have the latest multicast fix patch, etc.)

1 Like
786

What explains why I can see internet speeds like 15Mbit/s, but then ssh speed to main router painfully slow. Is that because the ssh connection is being given too low priority or something?

787

It's a good question, but first try to isolate the issue. Assuming it gets fixed, the answer will be in the fix.

788

Seems non-related. We share a very similar scenario, 2 WDS clients connecting to a main WDS AP. In my case connecting to my 2.4 GHz network doesn't affect ssh latency at all.

789

I have a slightly wacky arrangement in that I have main router provide guest WiFi WDS AP on 2.4 and normal WiFi WDS AP on 5.

Both APs connect to both WDS APs on main router and provide 2.4 AP for guest WiFi and both 2.4 and 5 APs for normal WiFi.

Connecting to normal WiFi on 2.4 on AP seems to give problem (at least in presence of Netflix related traffic on guest WiFi and multicast from about ten smart plugs also on guest WiFi).

The ssh is painfully slow like entering 'logread' takes ages to respond. As you can see from screenshot above iperf3 shows normal rates but lags behind on SSH.

790

Sure can. I redid all of them.

rrul_be-r20089-m76-SQM_312-47_Mbps-cake_layer4
rrul_be-r20089-m76-SQM_312-47_Mbps-cake_layer41280×960 236 KB

rrul_be-r20089-m76-SQM_800-47_Mbps-cake_layer4
rrul_be-r20089-m76-SQM_800-47_Mbps-cake_layer41280×960 298 KB

rrul_be-r20089-m76-to_router
rrul_be-r20089-m76-to_router1280×960 383 KB

PS. I'm trying to build an image with the latest patches for my RE450v2 (ath10k-ct), but I'm having some challenges, please bear with me.

791

@Lynx, what kind of device is your AP, and if it's an ath10k device, which driver is it running?

792

I just have three of these:

One acts as main router and the other two connect to main router on separate WDS APs for guest WiFi (WDS AP on 2.4Ghz) and normal WiFi (WDS AP on 5Ghz). They extend guest WiFi on 2.4Ghz and normal WiFi on 2.4 and 5Ghz.

I have a single SSID for guest WiFi and a single SSID for normal WiFi and FT is enabled for both across the three devices.

All works fine to provide WiFi throughout large three floor home, but somewhere along the line in terms of snapshot upgrades I noticed very slow ssh until I would reconnect my laptop to the normal WiFi and now I realise that it is because my laptop is connected over normal WiFi via 2.4Ghz and for some reason this exhibits significant delay / poor latency despite decent looking overall transfer rates. Reconnecting means switching over to 5Ghz and problem is then gone.

Looks like this:

image
image1314×905 23.9 KB

2 Likes
793

is your ssh marking the dscp field? Landing in another queue might be doing weird things.

1 Like
794

Only if that's the default behaviour. I haven't changed the default and don't bother with any of the DSCP marking stuff as I'm a little dubious about it all, at least for my particulars. Isn't life too short for endless tinkering around with DSCP markings?

For my 4G internet connection default CAKE with the autorate stuff works a treat.