Give it some time to see if the symptom comes back (maybe as much as few days). Others have observed (before the latest fixes) that the latency symptoms can take time develop or come back after turning off AQL. Regardless, ping @nbd with some details on the AP (device, what's it running, does it have the latest multicast fix patch, etc.)
1 Like
What explains why I can see internet speeds like 15Mbit/s, but then ssh speed to main router painfully slow. Is that because the ssh connection is being given too low priority or something?
It's a good question, but first try to isolate the issue. Assuming it gets fixed, the answer will be in the fix.
Seems non-related. We share a very similar scenario, 2 WDS clients connecting to a main WDS AP. In my case connecting to my 2.4 GHz network doesn't affect ssh latency at all.
I have a slightly wacky arrangement in that I have main router provide guest WiFi WDS AP on 2.4 and normal WiFi WDS AP on 5.
Both APs connect to both WDS APs on main router and provide 2.4 AP for guest WiFi and both 2.4 and 5 APs for normal WiFi.
Connecting to normal WiFi on 2.4 on AP seems to give problem (at least in presence of Netflix related traffic on guest WiFi and multicast from about ten smart plugs also on guest WiFi).
The ssh is painfully slow like entering 'logread' takes ages to respond. As you can see from screenshot above iperf3 shows normal rates but lags behind on SSH.
Sure can. I redid all of them.
PS. I'm trying to build an image with the latest patches for my RE450v2 (ath10k-ct), but I'm having some challenges, please bear with me.
I just have three of these:
One acts as main router and the other two connect to main router on separate WDS APs for guest WiFi (WDS AP on 2.4Ghz) and normal WiFi (WDS AP on 5Ghz). They extend guest WiFi on 2.4Ghz and normal WiFi on 2.4 and 5Ghz.
I have a single SSID for guest WiFi and a single SSID for normal WiFi and FT is enabled for both across the three devices.
All works fine to provide WiFi throughout large three floor home, but somewhere along the line in terms of snapshot upgrades I noticed very slow ssh until I would reconnect my laptop to the normal WiFi and now I realise that it is because my laptop is connected over normal WiFi via 2.4Ghz and for some reason this exhibits significant delay / poor latency despite decent looking overall transfer rates. Reconnecting means switching over to 5Ghz and problem is then gone.
Looks like this:
2 Likes
is your ssh marking the dscp field? Landing in another queue might be doing weird things.
1 Like
Only if that's the default behaviour. I haven't changed the default and don't bother with any of the DSCP marking stuff as I'm a little dubious about it all, at least for my particulars. Isn't life too short for endless tinkering around with DSCP markings?
For my 4G internet connection default CAKE with the autorate stuff works a treat.
Follow-up question. I gather that attention was recently afforded to multicast data. What about traffic like this:
root@OpenWrt:~# tcpdump -vpni br-guest
tcpdump: listening on br-guest, link-type EN10MB (Ethernet), capture size 262144 bytes
22:06:08.043450 IP (tos 0x0, ttl 255, id 53539, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
22:06:08.514987 IP (tos 0x0, ttl 255, id 53529, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.191.49154 > 255.255.255.255.6666: UDP, length 175
22:06:08.529170 IP (tos 0x0, ttl 255, id 53525, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.156.49154 > 255.255.255.255.6666: UDP, length 175
22:06:09.398553 IP (tos 0x0, ttl 255, id 34152, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.109.49154 > 255.255.255.255.6667: UDP, length 188
22:06:09.846746 IP (tos 0x0, ttl 255, id 34157, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.200.49154 > 255.255.255.255.6667: UDP, length 188
22:06:10.085035 IP (tos 0x0, ttl 255, id 34150, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.134.49154 > 255.255.255.255.6667: UDP, length 188
22:06:10.085365 IP (tos 0x0, ttl 255, id 34150, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.134.49154 > 255.255.255.255.6667: UDP, length 188
22:06:10.269147 IP (tos 0x0, ttl 255, id 52923, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.179.49155 > 255.255.255.255.6667: UDP, length 188
22:06:10.527005 IP (tos 0x0, ttl 255, id 53528, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.119.49154 > 255.255.255.255.6666: UDP, length 175
22:06:10.564785 IP (tos 0x0, ttl 255, id 34171, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.128.49154 > 255.255.255.255.6667: UDP, length 188
22:06:11.043718 IP (tos 0x0, ttl 255, id 53540, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
22:06:11.513351 IP (tos 0x0, ttl 255, id 53530, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.191.49154 > 255.255.255.255.6666: UDP, length 175
22:06:11.530135 IP (tos 0x0, ttl 255, id 53526, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.156.49154 > 255.255.255.255.6666: UDP, length 175
22:06:12.074089 IP (tos 0x0, ttl 255, id 47449, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.118.49154 > 255.255.255.255.6667: UDP, length 188
22:06:12.102683 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.179 tell 192.168.2.179, length 28
22:06:12.374825 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.109 tell 192.168.2.109, length 28
22:06:12.776482 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.134 tell 192.168.2.134, length 28
22:06:12.903406 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.200 tell 192.168.2.200, length 28
22:06:13.520926 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.128 tell 192.168.2.128, length 28
22:06:13.529649 IP (tos 0x0, ttl 255, id 53529, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.119.49154 > 255.255.255.255.6666: UDP, length 175
22:06:14.042985 IP (tos 0x0, ttl 255, id 53541, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
22:06:14.395631 IP (tos 0x0, ttl 255, id 34153, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.109.49154 > 255.255.255.255.6667: UDP, length 188
22:06:14.486113 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.118 tell 192.168.2.118, length 28
22:06:14.514464 IP (tos 0x0, ttl 255, id 53531, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.191.49154 > 255.255.255.255.6666: UDP, length 175
22:06:14.524313 IP (tos 0x0, ttl 255, id 53527, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.156.49154 > 255.255.255.255.6666: UDP, length 175
22:06:14.846485 IP (tos 0x0, ttl 255, id 34158, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.200.49154 > 255.255.255.255.6667: UDP, length 188
22:06:15.085166 IP (tos 0x0, ttl 255, id 34151, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.134.49154 > 255.255.255.255.6667: UDP, length 188
22:06:15.270983 IP (tos 0x0, ttl 255, id 52924, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.179.49155 > 255.255.255.255.6667: UDP, length 188
22:06:15.564898 IP (tos 0x0, ttl 255, id 34172, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.128.49154 > 255.255.255.255.6667: UDP, length 188
22:06:16.527908 IP (tos 0x0, ttl 255, id 53530, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.119.49154 > 255.255.255.255.6666: UDP, length 175
22:06:17.043297 IP (tos 0x0, ttl 255, id 53542, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
22:06:17.073189 IP (tos 0x0, ttl 255, id 47450, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.118.49154 > 255.255.255.255.6667: UDP, length 188
22:06:17.514675 IP (tos 0x0, ttl 255, id 53532, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.191.49154 > 255.255.255.255.6666: UDP, length 175
22:06:17.528499 IP (tos 0x0, ttl 255, id 53528, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.156.49154 > 255.255.255.255.6666: UDP, length 175
22:06:19.399187 IP (tos 0x0, ttl 255, id 34154, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.109.49154 > 255.255.255.255.6667: UDP, length 188
22:06:19.528582 IP (tos 0x0, ttl 255, id 53531, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.119.49154 > 255.255.255.255.6666: UDP, length 175
22:06:19.848940 IP (tos 0x0, ttl 255, id 34159, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.200.49154 > 255.255.255.255.6667: UDP, length 188
22:06:20.048174 IP (tos 0x0, ttl 255, id 53543, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
22:06:20.085719 IP (tos 0x0, ttl 255, id 34152, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.134.49154 > 255.255.255.255.6667: UDP, length 188
22:06:20.269806 IP (tos 0x0, ttl 255, id 52925, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.179.49155 > 255.255.255.255.6667: UDP, length 188
22:06:20.510184 IP (tos 0x0, ttl 255, id 53533, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.191.49154 > 255.255.255.255.6666: UDP, length 175
22:06:20.525346 IP (tos 0x0, ttl 255, id 53529, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.156.49154 > 255.255.255.255.6666: UDP, length 175
22:06:20.562477 IP (tos 0x0, ttl 255, id 34173, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.128.49154 > 255.255.255.255.6667: UDP, length 188
22:06:22.073323 IP (tos 0x0, ttl 255, id 47451, offset 0, flags [none], proto UDP (17), length 216)
192.168.2.118.49154 > 255.255.255.255.6667: UDP, length 188
22:06:22.099811 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.179 tell 192.168.2.179, length 28
22:06:22.375276 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.109 tell 192.168.2.109, length 28
22:06:22.523823 IP (tos 0x0, ttl 255, id 53532, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.119.49154 > 255.255.255.255.6666: UDP, length 175
22:06:22.771964 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.134 tell 192.168.2.134, length 28
22:06:22.905432 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.2.200 tell 192.168.2.200, length 28
22:06:23.043737 IP (tos 0x0, ttl 255, id 53544, offset 0, flags [none], proto UDP (17), length 203)
192.168.2.192.49154 > 255.255.255.255.6666: UDP, length 175
Could such IoT device traffic be the cause of lag on 2.4Ghz?
ssh last I looked set the tos imm bit for interactive traffic by itself. Thus it will typically land in the VO queue from a linux client.
Yes, multicast is a hidden bane of many wifi networks. It's almost always turned off in large public networks. We've found ways to elide common multicast things like arp, and nd, but mdns and a few other protocols use it heavily. I routinely increase the multicast rate to 12mbit or higher nowadays. The unfortunately limits range, and a righter answer would have been (where it baked into the standard) to probe for the right rate, or use one that was the lowest observed rate achieved for all the stations on the network. In the standard was also the concept of rate limiting multicast so it would only eat, say 20ms of airtime per 100ms, max. (to be clear, in the first case I was talking about encoding and broadcasting multicast at the 12mbit rate, in the second, limiting the number of packets that wlll be sent to under 20ms of airtime per 100ms)
The important detail that has cropped up in this string of tests over the last few weeks is the damage being done by location services. I had no idea, and would like a comparison plot with them on and off.
1 Like
I had a quick look in my local network (captured via wireshark on my mac):
SSH traffic using port 22:
macosx sends: AF21; TOS 0x48 -> TOS immediate bit set
ubuntu22 LTS sends: 4; TOS 0x10 -> TOS immediate bit not set
But, AF21 maps into AC_BK by default WMM mapping rules, while 4 at least stays in AC_BE:
UP DSCP AC PHBs(decDSCP)
Range0 0-7 BE CS0(0)
Range1 8-15 BK CS1(8), AF11(10), AF12(12), AF13(14)
Range2 16-23 BK CS2(16), AF21(18), AF22(20), AF23(22)
Range3 24-31 BE CS3(24), AF31(26), AF32(28), AF33(30)
Range4 32-39 VI CS4(32), AF41(34), AF42(36), AF43(38)
Range5 40-47 VI CS5(40), VA(44), EF(46)
Range6 48-55 VO CS6(48)
Range7 56-63 VO CS7(56)
Recent OpenWrt mapping rules keep both in AC_BE:
UP DSCP AC PHBs(decDSCP)
Ex0 BE BE(0) BE/CS0(0)
Range0 2-16 BE CS1(8)**, AF11(10), AF12(12), AF13(14), CS2(16)
Range1 1-1 BK LE(1)
Range2 -
Range3 18-22 BE AF21(18), AF22(20), AF23(22)
Range4 24-38 VI CS3(24), AF31(26), AF32(28), AF33(30), CS4(32), AF41(34), AF42(36), AF43(38)
Range5 40-40 VI CS5(40)
Range6 44-46 VO VA(44), EF(46)
Range7 48-56 VO CS6(48), CS7(56)
Testing mosh showed CS0 in both directions....
Will need to do a similar test for a VoIP call...
1 Like
mosh used to be ecn enabled on some OSes, over ipv4 at least. It was originally marked AF42 until (ironically) - a machine on MIT's campus was claimed to block that eventually (after many seconds).
1 Like
Yepp, that is what I see ECT(0) in both directions (so between macosx monterey and ubuntu 22 LTS, both x86_64), while SSH is Not-ECT.
The only quick example I can provide today. This is during a ping to my router clicking the WiFi icon which triggers a WiFi scan. I timestamped ICMP requests for your viewing pleasure. 
macOS does periodic WiFi scans, so this is what's expected each time.
1 Like
@dtaht Location services tests, WLAN to WAN.
Location services on:
Location services off:
PS. I need some help compiling a sysupgrade for a RE450v2. How can I reduce the size of it?
3 Likes
Not really the right thread for that. Ripping out unnecessary packages - or putting in all the packages you need so they are properly compressed as part of the image, is my first thought.
Thx for showing what location services "looks like" against the cosmic background bufferbloat radiation. It looks to me as though your wifi is not saturated in either direction, but we do see things get pretty fuzzy while that is going on. Open question is what that looks like on ath10k, 9k...
Otherwise things are working pretty good. It's difficult to recall all the different bugs we've encountered along the way, and it's my hope we get more testers in general soon. Going out and pursuing the bug-filers of so many "wifi is flaky" bugs, perhaps, or just waiting for the next release....
3 Likes
Ref: AQL and the ath10k is *lovely* - #737 by ka2107
5 GHz Band, AP and Client in different room with brick wall between them
Belkin RT3200 Router running "OpenWrt 22.03-SNAPSHOT r19575-506432a783 / LuCI openwrt-22.03 branch git-22.204.42822-9a18337" and Netperf netserver + IRTT server
Flent command: flent tcp_nup --test-parameter=upload_streams=1 -l 60 -H <Router_IP> -v
macOS 12.5 Monterey - All Location Services Disabled
Flent Data File: mac_to_router_5ghz_tcp_1up_Location_disabled.flent
% /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I
agrCtlRSSI: -69
agrExtRSSI: 0
agrCtlNoise: -101
agrExtNoise: 0
state: running
op mode: station
lastTxRate: 52
maxRate: 144
lastAssocStatus: 0
802.11 auth: open
link auth: wpa2-psk
BSSID:
SSID: XXXX_5GHz
MCS: 3
guardInterval: 800
NSS: 2
channel: 36
% networkQuality -v
==== SUMMARY ====
Upload capacity: 21.584 Mbps
Download capacity: 36.202 Mbps
Upload flows: 16
Download flows: 12
Responsiveness: Low (169 RPM)
Base RTT: 39
Start: 24/Jul/2022, 05:28:23
End: 24/Jul/2022, 05:28:38
OS Version: Version 12.5 (Build 21G72)
macOS 12.5 Monterey - 'Find My Mac' and 'Network & Wireless' Location Services Enabled
Flent Data File: mac_to_router_5ghz_tcp_1up_Location_enabled.flent
% /System/Library/PrivateFrameworks/Apple80211.framework/Versions/Current/Resources/airport -I
agrCtlRSSI: -70
agrExtRSSI: 0
agrCtlNoise: -100
agrExtNoise: 0
state: running
op mode: station
lastTxRate: 52
maxRate: 144
lastAssocStatus: 0
802.11 auth: open
link auth: wpa2-psk
BSSID:
SSID: XXXX_5GHz
MCS: 3
guardInterval: 800
NSS: 2
channel: 36
% networkQuality -v
==== SUMMARY ====
Upload capacity: 16.488 Mbps
Download capacity: 34.957 Mbps
Upload flows: 12
Download flows: 12
Responsiveness: Low (171 RPM)
Base RTT: 39
Start: 24/Jul/2022, 07:01:23
End: 24/Jul/2022, 07:01:34
OS Version: Version 12.5 (Build 21G72)
Ping when upload direction is fully loaded is too high. Is there anything that the AP can do to reduce latency when Client-to-AP stream is loaded?
1 Like