AQL and the ath10k is *lovely*

_FailSafe · 2020-06-13T12:40:41.868Z

I’m glad to see your feedback on this. I have also found that 10ms seems to provide a much better balance for me than 5ms, at least on 5ghz. I found my throughput took a respectable hit at 5ms, with no perceivable change in overall latency, even if a graph shows otherwise. At 10ms, I could achieve higher throughput and still no perceivable impact on other stations.

I honestly care very little about 2.4ghz anymore as the saturation of that frequency range in my neighborhood makes it basically unusable. In fact, I only have my IoT devices and Apple Watches on 2.4ghz, out of necessity. Neither throughput, nor latency, are a concern for me on those devices.

Anyway, it would be great to get more feedback from others here as to 10ms vs 5ms. Or maybe there’s a target inside of that range that’s more ideal? 7? 8?

And I concur with your observations on the traffic patterns of the ath10k. AP to client does exhibit some weird drops in speed, even if it recovers after a few seconds. (I have an r7800, FWIW)

moeller0 · 2020-06-13T12:43:15.149Z

_FailSafe:

This behavior you just described—is it in context of the @dtaht’s 982-do-codel-right patch, or without that patch?

This is without the patch, my testing so far was with ath9K with air time fairness patches. But this is an issue of the macbook as sender and I assume that is independent of the APs airtime fairness settings as these, as far as I understand, only affect the APs own sending. But I would be delighted to be wrong here!

rickk · 2020-06-13T14:16:37.978Z

Any better or worse at 10ms?
Also, are you testing on 2.4 or 5ghz?

I'm testing on 5 Ghz. 2.4Ghz is active but like you, only for IoT and Apple Watches.
I had no problems whatsoever with 5ms so far, but switched to 10ms in the name of science.

Did some tests with one station running 6down lan->wlan every 15 seconds and the other running 6down and rrul between the r7800->macbook, but it's really hard to control all variables between tests and draw concusions.

The ones attached should at least be comparable. Left is 5ms, right is 10ms.

by default 2020-06-13 at 15.50.482566×960 630 KB

by default 2020-06-13 at 15.57.112560×966 893 KB

Didn't do much real-world testing yet. Seems ok at first glance.

If anything, I'd say the difference between 5 and 10ms isn't huge.

rickk · 2020-06-13T20:54:30.235Z

I kinda got attached to my steady <10ms pings on the 5ms patch, which were gone on 10ms. Pings were visually more unstable. So now I'm giving 8ms a shot

I also played around with aql_threshold. Lowering this cuts throughput hard when another station is active (cycling a tcp 6down on and off every 15s), at least on a high rate connection. In the graph below the first ~45 seconds are on 16000, the rest is on (the default) 24000. You see the there's little difference when the other station is not active, but once the other station starts downloading throughput drops to ~25% on 16k, ~50% on 24k.
This effect was typical and easy to repeat/reverse. 24k seems like a good compromise for my situation.

netperf-6down-20200618-2218-MXM-5Ghz-c8-aqlt16-24k-lug2gw-loaded-bank

_FailSafe · 2020-06-13T21:26:49.496Z

I just refreshed my r7800 build and built current (kernel 5.4.45) 5ms, 10ms, and 20ms targets. Here are some of the results I am seeing. My macbook was positioned at the same location (same distance) between all tests.

rrul_down11934×1391 116 KB

rrul_up11934×1391 102 KB

rrul_icmp_cdf1953×1391 145 KB

rrul_ping_cdf1953×1391 338 KB

Something really nasty keeps happening at 5ms target though. The location I'm testing from is probably 20-25 feet from my r7800, but with a floor and a couple walls in between. My 5ms ping plots show crazy latency spikes that I do not see at all on 10ms or 20ms targets:

rrul_ping1953×1391 747 KB

(Note the solid colored lines which are 5ms vs the dashed lines for 10ms)

EDIT Also, I am running my build with the ath10k-ct-smallbuffers driver and the latest ath10k-ct beta firmware.

dtaht · 2020-06-16T05:04:40.750Z

Thank you very much for those tests. That's weird. There is some fateful interaction deeper in the stack with targets that low. (which is basically what we are circling around here, all of us) Could you tar up and stick the flent data somewhere?

Another good test is flent -x --socket-stats --step-size=.04 -t whatever_is_under_test --te=upload_streams=1 tcp_nup # upload_streams=4, upload_streams=16

This captures the actual tcp statistics. It's even better with ecn on and a packet capture taken
at the same time on the client or server.....

dtaht · 2020-06-16T05:16:35.235Z

rickk:

If anything, I'd say the difference between 5 and 10ms isn't huge.

Partially because that test measures the wrong thing. That is because "rrul" testing the performance of 3 of 4 hardware queues which are filling up occupying airtime independently. A better comparison test for 5ms vs 10ms target is the rrul_be test, or the tcp_nup test I just mentioned above.

This thread has got really long, and I so appreciate everyone leaping on it. Somewhere in the past part of this thread we talked about how using packets, rather than bytes, in the high and low watermarks was probably a source of weirdness. A way to test this is for someone to try the regular -ct code, not the smallbuffers version, on the lower target. @tohojo who has his fingers in too many pies and doesn't even have this chip seemed to have some insight on that.

elsewhere (not on this thread, I think) I talked about how unbounded retries were messing wifi up in the case of interference. The only way know how to look at this is with aircaps.

and over here I talked about how testing for only a minute bit me badly: http://blog.cerowrt.org/post/disabling_channel_scans/ so -l 300 on a given test, and taking a long walk might help...

so far, to me, the net sum of testing was that 10ms seemed a safe default for the -ct-smallbuffers version, and worse case we should try to push that into the next mainline release. Actually getting
something even better and more consistent than that would be great. I did just get a new
x86_64 box that I can start putting custom kernels on.

dtaht · 2020-06-16T05:19:28.810Z

rickk:

This effect was typical and easy to repeat/reverse. 24k seems like a good compromise for my situation.

my take on aql was that it needed to be an algorithm, not a constant.

dtaht · 2020-06-16T05:26:30.382Z

In terms of trying to explain how we might think about the interactions between aql, napi, hi/low watermarks, bql, etc, I kind of find it useful to refer to this paper:

https://sci-hub.tw/10.1016/j.comnet.2020.107136

In that the pie folk finally realized that draining the rx ring and filling the tx ring in a real OS as opposed to a simulation... was a bulk operation, NOT a fluid model, and it broke pie's default rate estimator. They switched pie to timestamps which sort of fixed the problem (codel has always used timestamps, and while we always knew pie's rate estimator was broken...) AND did a really nice analysis of BQL and interrupt handling for two different ethernet cards.

If you can grok that...

Ath10k is even more complicated than that. I so wish we had access to firmware.

moeller0 · 2020-06-16T11:16:47.720Z

dtaht:

That is because "rrul" testing the performance of all 4 hardware queues which are filling up occupying airtime independently.

Mmmh, curious, rrul uses CS0, CS1, CS5, and EF. As far as I can tell in WMM that will map to AC_BE, AC_BK, AC_VI, AC_VI, so rrul should only exercise 3 of the 4 queues (which essentially does not invalidate your argument), no?

I guess, I need to run a ruul test over wifi and the check which queues accumulated more packets....

moeller0 · 2020-06-16T11:18:30.427Z

dtaht:

They switched pie to timestamps which sort of fixed the problem (codel has always used timestamps, and while we always knew pie's rate estimator was broken...)

Ha, so much for the PIE-camp's claim that PIE is "better" than codel since without needing timestamps it could be more easily implemented in silicon....

dtaht · 2020-06-16T14:48:40.879Z

you are correct, rrul only tests 3 of the 4 queues. My bad - both in the original spec and implementation, and too late to fix now. (rrulV2 anyone?). I also edited my comment above.

3 queues is still 2 too many to correctly test the target 5ms mod. However the side effects of using the extra hardware queues do seem to suggest that we fiddle with targets (or preferably, intervals) based on how many hw queues are in use.

moeller0 · 2020-06-16T17:00:01.770Z

dtaht:

My bad - both in the original spec and implementation, and too late to fix now. (rrulV2 anyone?)

Well, there is rrul_cs8, which by using all 8 CSs should map two flows in each AC. This nicely demonstrates how anti-social greedy flows in AC_VO behave....

huaracheguarache · 2020-08-09T12:55:02.715Z

An AQL airtime estimation patch has caused some issues for me. The patch assumes an average aggregate length of 16 which doesn't seem to work well at the very edge of my network.

Notice that the latency stays at around 100 ms under load with the patch:

aql_with_patch1302×321 99.1 KB

Without the patch I'm getting a far better value of around just under 40 ms:

aql_without_patch1302×321 89.5 KB

_FailSafe · 2020-08-09T15:24:11.843Z

Interesting find. I had noticed some [new] spotty connectivity patterns on some of my devices near the edge of my network as well, but didn't associate it with this patch until you pointed it out.

Did you end up manually removing these patches and rebuilding?

Curious to hear if others are noticing issues as well.

huaracheguarache · 2020-08-09T16:41:04.266Z

Yeah, I reverted the patch locally and compiled a new image.

_FailSafe · 2020-08-09T17:47:17.608Z

The behavior pattern I am noticing (while not as scientific as a flent test) is this:
I SSH to one of my internal servers from a wirelessly connected iPhone/iPad (VHT80 connected) and start something like a watch -n1 ... command. It refreshes every second for a few seconds, then it "pauses" and takes several seconds (anywhere from 2-5+ secs) to refresh. It might pick back up with refreshes for a second or two before a long delay again. This cycle repeats endlessly.

This started fairly recently, so I reverted the same patch thinking it might have been related, but no dice. There appears to be something else going on in my case. I'm not losing sleep over it at this point, but hopefully I (or someone else) can get to the bottom of it soon.

huaracheguarache · 2020-08-09T20:54:36.241Z

Hmm, I haven't experienced the issue you describe. If you can reliably reproduce it you could try to bisect to find the commit that causes it.

nbd · 2020-08-11T08:18:53.084Z

Hi @huaracheguarache,

Thanks for reporting this.
For high throughput links, my patch is necessary for decent aggregation performance, because otherwise the hardware queues will run empty too often.
Regarding your setup that produces these latency spikes, do you simply have too many stations producing traffic (not filling the individual queues enough to aggregate packets), or are the links using lower data rates because of weak signal strength?
Unfortunately I can't make the patch use actual A-MPDU length, because most drivers can't report it properly. So I have to find another way to work around this.

huaracheguarache · 2020-08-11T08:51:25.326Z

In my case I'm only testing one station at the edge of my network where the signal strength is poor which causes low data rates. There have been some issues with VoIP performance in these areas of my house lately. I suspect this is related to the patch.