Lynx
October 8, 2022, 8:37am
1
I have 3x RT3200's, with one acting as a router (192.168.1.1) and the other two as WDS linked access points (192.168.1.2 and 192.168.1.3).
What would explain lots of retries when running iperf3 as a server on my main router (192.168.1.1) and connecting from one of the WDS linked access points (192.168.1.2):
root@OpenWrt:~# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[ 5] local 192.168.1.2 port 35252 connected to 192.168.1.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 13.2 MBytes 110 Mbits/sec 90 70.7 KBytes
[ 5] 1.00-2.00 sec 22.2 MBytes 186 Mbits/sec 91 106 KBytes
[ 5] 2.00-3.00 sec 34.4 MBytes 288 Mbits/sec 122 29.7 KBytes
[ 5] 3.00-4.00 sec 10.1 MBytes 85.0 Mbits/sec 93 7.07 KBytes
[ 5] 4.00-5.00 sec 13.1 MBytes 110 Mbits/sec 93 17.0 KBytes
[ 5] 5.00-6.00 sec 15.4 MBytes 129 Mbits/sec 70 15.6 KBytes
[ 5] 6.00-7.00 sec 44.5 MBytes 373 Mbits/sec 129 154 KBytes
[ 5] 7.00-8.00 sec 42.8 MBytes 359 Mbits/sec 69 116 KBytes
[ 5] 8.00-9.00 sec 47.5 MBytes 398 Mbits/sec 57 80.6 KBytes
[ 5] 9.00-10.00 sec 20.1 MBytes 169 Mbits/sec 136 100 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 263 MBytes 221 Mbits/sec 950 sender
[ 5] 0.00-10.00 sec 262 MBytes 220 Mbits/sec receiver
iperf Done.
If I flip the client and server so that I run the server on my WDS access point (192.168.1.2) and connect from my main router (192.168.1.1) I see:
root@OpenWrt:~# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 40136 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 100 MBytes 840 Mbits/sec 0 1.95 MBytes
[ 5] 1.00-2.00 sec 78.2 MBytes 656 Mbits/sec 0 1.95 MBytes
[ 5] 2.00-3.00 sec 78.0 MBytes 654 Mbits/sec 0 1.95 MBytes
[ 5] 3.00-4.00 sec 87.0 MBytes 728 Mbits/sec 0 1.95 MBytes
[ 5] 4.00-5.00 sec 114 MBytes 952 Mbits/sec 0 1.95 MBytes
[ 5] 5.00-6.00 sec 108 MBytes 913 Mbits/sec 0 1.95 MBytes
[ 5] 6.00-7.00 sec 118 MBytes 993 Mbits/sec 0 1.95 MBytes
[ 5] 7.00-8.00 sec 89.6 MBytes 752 Mbits/sec 0 1.95 MBytes
[ 5] 8.00-9.00 sec 95.9 MBytes 804 Mbits/sec 0 1.95 MBytes
[ 5] 9.00-10.00 sec 102 MBytes 858 Mbits/sec 0 1.95 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 972 MBytes 815 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 969 MBytes 813 Mbits/sec receiver
iperf Done.
Here is how the WDS connected AP (192.168.1.2) appears from the main router (192.168.1.1) using 'iwinfo wlan1 assoc':
root@OpenWrt:~# iwinfo wlan1 assoc
XX -41 dBm / unknown (SNR -41) 0 ms ago
RX: 2401.9 MBit/s, HE-MCS 11, 80MHz, HE-NSS 4, HE-GI 0, HE-DCM 0 1893281 Pkts.
TX: 2268.5 MBit/s, HE-MCS 11, 80MHz, HE-NSS 4, HE-GI 1, HE-DCM 0 3878164 Pkts.
expected throughput: unknown
What is going on here? @patrakov perhaps this is one for you?
It is not retries, but TCP retransmissions. Ideally we should get a wireless pcap from the third router, and ask @dtaht to look at it.
To rule out the obvious (e.g. kmod-tcp-bbr being installed by mistake on one of the routers) - can you provide the output of this on both ends?
cat /proc/sys/net/ipv4/tcp_available_congestion_control
cat /proc/sys/net/ipv4/tcp_congestion_control
Also let's use modern tools instead of the obsolete iwinfo:
iw dev wlan1 station dump # on the AP
iw dev wlan1 link # on the client
Lynx
October 8, 2022, 9:01am
3
From main router (192.168.1.1):
root@OpenWrt:~# cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic
root@OpenWrt:~# cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
From WDS AP (192.168.1.2):
root@OpenWrt:~# cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic
n_controlroot@OpenWrt:~# cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
Then on main router (192.168.1.1):
root@OpenWrt:~# iw dev wlan1.sta4 station dump
Station [192.168.1.2_MAC] (on wlan1.sta4)
inactive time: 0 ms
rx bytes: 1973296693
rx packets: 1903656
tx bytes: 5138617203
tx packets: 3900088
tx retries: 0
tx failed: 0
rx drop misc: 14593
signal: -41 [-47, -47, -46, -45] dBm
signal avg: -41 [-47, -46, -46, -45] dBm
tx bitrate: 2268.5 MBit/s 80MHz HE-MCS 11 HE-NSS 4 HE-GI 1 HE-DCM 0
tx duration: 1854401275 us
rx bitrate: 2401.9 MBit/s 80MHz HE-MCS 11 HE-NSS 4 HE-GI 0 HE-DCM 0
rx duration: 187693163 us
airtime weight: 256
authorized: yes
authenticated: yes
associated: yes
preamble: long
WMM/WME: yes
MFP: no
TDLS peer: no
DTIM period: 0
beacon interval:0
connected time: 140859 seconds
associated at [boottime]: 270.634s
associated at: 1665078718356 ms
current time: 1665219577035 ms
And on WDS AP (192.168.1.2):
root@OpenWrt:~# iw dev wlan1 link
Connected to [192.168.1.1_MAC] (on wlan1)
SSID: [SSID]
freq: 5180
RX: 1258461086 bytes (6661947 packets)
TX: 2115506926 bytes (2187101 packets)
signal: -40 dBm
rx bitrate: 2401.9 MBit/s 80MHz HE-MCS 11 HE-NSS 4 HE-GI 0 HE-DCM 0
tx bitrate: 2268.5 MBit/s 80MHz HE-MCS 11 HE-NSS 4 HE-GI 1 HE-DCM 0
bss flags: short-slot-time
dtim period: 2
beacon int: 100
Going back to my first post, any idea why the client-server switch makes things completely different?
Could this be another glitch in the old airtime fairness stuff that seems to have caused various issues?
No idea.
One more thing to try would be to set up temporarily a separate wireless network, with the roles of the client and the AP switched (a non-DFS channel is required for this network to coexist with the main one), and retest.
Lynx
October 8, 2022, 9:24am
5
Interestingly it's the same way round with my laptop. When my main router is the iperf3 server I see very slow rates, but when my laptop is the iperf3 server things look better.
Here is iperf3 server on main router receiving connection from laptop:
Accepted connection from 192.168.1.134, port 60994
[ 5] local 192.168.1.1 port 5201 connected to 192.168.1.134 port 60995
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 312 KBytes 2.56 Mbits/sec
[ 5] 1.00-2.00 sec 2.75 MBytes 23.1 Mbits/sec
[ 5] 2.00-3.00 sec 8.04 MBytes 67.4 Mbits/sec
[ 5] 3.00-4.00 sec 12.4 MBytes 104 Mbits/sec
[ 5] 4.00-5.00 sec 11.3 MBytes 94.8 Mbits/sec
[ 5] 5.00-6.00 sec 12.3 MBytes 103 Mbits/sec
[ 5] 6.00-7.00 sec 6.97 MBytes 58.4 Mbits/sec
[ 5] 7.00-8.00 sec 5.60 MBytes 46.9 Mbits/sec
[ 5] 8.00-9.00 sec 5.54 MBytes 46.5 Mbits/sec
[ 5] 9.00-10.00 sec 11.6 MBytes 97.4 Mbits/sec
[ 5] 10.00-10.34 sec 4.05 MBytes 99.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.34 sec 80.8 MBytes 65.5 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201 (test #6)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
And here is main router connecting to iperf3 server on laptop:
root@OpenWrt:~# iperf3 -c 192.168.1.134
Connecting to host 192.168.1.134, port 5201
[ 5] local 192.168.1.1 port 58084 connected to 192.168.1.134 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 23.3 MBytes 196 Mbits/sec 0 496 KBytes
[ 5] 1.00-2.00 sec 17.0 MBytes 142 Mbits/sec 6 242 KBytes
[ 5] 2.00-3.00 sec 21.9 MBytes 184 Mbits/sec 0 242 KBytes
[ 5] 3.00-4.00 sec 16.6 MBytes 139 Mbits/sec 10 170 KBytes
[ 5] 4.00-5.00 sec 18.6 MBytes 156 Mbits/sec 0 214 KBytes
[ 5] 5.00-6.00 sec 22.9 MBytes 192 Mbits/sec 0 214 KBytes
[ 5] 6.00-7.00 sec 18.8 MBytes 158 Mbits/sec 0 214 KBytes
[ 5] 7.00-8.00 sec 16.8 MBytes 141 Mbits/sec 2 170 KBytes
[ 5] 8.00-9.00 sec 13.5 MBytes 113 Mbits/sec 3 132 KBytes
[ 5] 9.00-10.00 sec 19.7 MBytes 165 Mbits/sec 0 212 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 189 MBytes 159 Mbits/sec 21 sender
[ 5] 0.00-10.00 sec 187 MBytes 157 Mbits/sec receiver
@bluewavenet any clue what's going on here (comparing above with first post)?
What if you disable 802.11ax (thus downgrading to ac) on the AP?
1 Like
Lynx
October 8, 2022, 9:28am
7
Should I do this on just the main router AP or on all the main router AP and WDS AP?
On the main router AP would be enough - we just need to make sure that the link over which iperf3 operates is not ax.
1 Like
Lynx
October 8, 2022, 9:37am
9
I see many fewer retransmissions in switching from AX to AC, albeit this time round the transfer rates aren't as bad as in the first post:
root@OpenWrt:~# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[ 5] local 192.168.1.2 port 42168 connected to 192.168.1.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 66.0 MBytes 554 Mbits/sec 297 177 KBytes
[ 5] 1.00-2.00 sec 44.8 MBytes 375 Mbits/sec 118 173 KBytes
[ 5] 2.00-3.00 sec 44.9 MBytes 376 Mbits/sec 213 157 KBytes
[ 5] 3.00-4.00 sec 49.8 MBytes 417 Mbits/sec 129 144 KBytes
[ 5] 4.00-5.00 sec 52.0 MBytes 436 Mbits/sec 55 189 KBytes
[ 5] 5.00-6.00 sec 51.9 MBytes 435 Mbits/sec 110 181 KBytes
[ 5] 6.00-7.00 sec 53.6 MBytes 450 Mbits/sec 141 191 KBytes
[ 5] 7.00-8.00 sec 54.6 MBytes 458 Mbits/sec 88 181 KBytes
[ 5] 8.00-9.00 sec 42.2 MBytes 354 Mbits/sec 52 185 KBytes
[ 5] 9.00-10.00 sec 44.9 MBytes 377 Mbits/sec 83 185 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 505 MBytes 423 Mbits/sec 1286 sender
[ 5] 0.00-10.00 sec 502 MBytes 421 Mbits/sec receiver
iperf Done.
root@OpenWrt:~# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[ 5] local 192.168.1.2 port 48600 connected to 192.168.1.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 42.3 MBytes 355 Mbits/sec 26 109 KBytes
[ 5] 1.00-2.00 sec 32.4 MBytes 271 Mbits/sec 20 112 KBytes
[ 5] 2.00-3.00 sec 41.5 MBytes 348 Mbits/sec 40 202 KBytes
[ 5] 3.00-4.00 sec 38.7 MBytes 325 Mbits/sec 59 70.7 KBytes
[ 5] 4.00-5.00 sec 48.2 MBytes 404 Mbits/sec 30 163 KBytes
[ 5] 5.00-6.00 sec 41.2 MBytes 346 Mbits/sec 39 158 KBytes
[ 5] 6.00-7.00 sec 42.8 MBytes 359 Mbits/sec 43 146 KBytes
[ 5] 7.00-8.00 sec 36.4 MBytes 305 Mbits/sec 34 79.2 KBytes
[ 5] 8.00-9.00 sec 23.5 MBytes 197 Mbits/sec 21 84.8 KBytes
[ 5] 9.00-10.00 sec 29.1 MBytes 244 Mbits/sec 24 83.4 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 376 MBytes 315 Mbits/sec 336 sender
[ 5] 0.00-10.00 sec 375 MBytes 315 Mbits/sec receiver
iperf Done.
What about the other direction?
Lynx
October 8, 2022, 9:40am
11
You mean when I run iperf3 server on my WDS AP and connect from main router? That has always given full transfer rates and no retransmissions.
Yes, I mean this, and I wanted to reconfirm.
Still, the test with a second SSID, where the old WDS client becomes an AP, and the old AP becomes a WDS client, would be interesting. This way we will be able to learn whether it is physical position or the client/AP role that matters.
Lynx
October 8, 2022, 9:42am
13
Here is one run with AX then one run with AC in that direction:
root@OpenWrt:~# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 34388 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 103 MBytes 863 Mbits/sec 0 1.85 MBytes
[ 5] 1.00-2.00 sec 76.8 MBytes 644 Mbits/sec 0 1.85 MBytes
[ 5] 2.00-3.00 sec 75.6 MBytes 634 Mbits/sec 0 1.85 MBytes
[ 5] 3.00-4.00 sec 77.0 MBytes 646 Mbits/sec 0 1.85 MBytes
[ 5] 4.00-5.00 sec 98.5 MBytes 827 Mbits/sec 0 1.95 MBytes
[ 5] 5.00-6.00 sec 116 MBytes 974 Mbits/sec 0 1.95 MBytes
[ 5] 6.00-7.00 sec 117 MBytes 981 Mbits/sec 0 1.95 MBytes
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 1.95 MBytes
[ 5] 8.00-9.00 sec 107 MBytes 903 Mbits/sec 0 1.95 MBytes
[ 5] 9.00-10.00 sec 103 MBytes 865 Mbits/sec 0 1.95 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 987 MBytes 828 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 985 MBytes 826 Mbits/sec receiver
iperf Done.
root@OpenWrt:~# iperf3 -c 192.168.1.2
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 45378 connected to 192.168.1.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 112 MBytes 939 Mbits/sec 0 1.91 MBytes
[ 5] 1.00-2.00 sec 99.6 MBytes 834 Mbits/sec 0 1.91 MBytes
[ 5] 2.00-3.00 sec 79.4 MBytes 667 Mbits/sec 0 1.91 MBytes
[ 5] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec 0 1.91 MBytes
[ 5] 4.00-5.00 sec 78.0 MBytes 654 Mbits/sec 0 1.91 MBytes
[ 5] 5.00-6.00 sec 76.9 MBytes 645 Mbits/sec 0 1.91 MBytes
[ 5] 6.00-7.00 sec 81.9 MBytes 685 Mbits/sec 0 1.91 MBytes
[ 5] 7.00-8.00 sec 120 MBytes 1.00 Gbits/sec 0 1.91 MBytes
[ 5] 8.00-9.00 sec 91.7 MBytes 772 Mbits/sec 0 1.91 MBytes
[ 5] 9.00-10.00 sec 95.8 MBytes 803 Mbits/sec 0 1.91 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 912 MBytes 765 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 908 MBytes 762 Mbits/sec receiver
iperf Done.
And for good measure here is another run from WDS AP to iperf3 server on main router showing the problem:
root@OpenWrt:~# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[ 5] local 192.168.1.2 port 36660 connected to 192.168.1.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 33.8 MBytes 283 Mbits/sec 109 117 KBytes
[ 5] 1.00-2.00 sec 36.4 MBytes 305 Mbits/sec 163 156 KBytes
[ 5] 2.00-3.00 sec 35.1 MBytes 295 Mbits/sec 116 150 KBytes
[ 5] 3.00-4.00 sec 49.8 MBytes 418 Mbits/sec 71 192 KBytes
[ 5] 4.00-5.00 sec 36.2 MBytes 304 Mbits/sec 149 123 KBytes
[ 5] 5.00-6.00 sec 46.7 MBytes 392 Mbits/sec 80 175 KBytes
[ 5] 6.00-7.00 sec 39.8 MBytes 334 Mbits/sec 158 134 KBytes
[ 5] 7.00-8.00 sec 41.3 MBytes 347 Mbits/sec 152 4.24 KBytes
[ 5] 8.00-9.00 sec 38.3 MBytes 321 Mbits/sec 47 42.4 KBytes
[ 5] 9.00-10.00 sec 16.3 MBytes 137 Mbits/sec 107 89.1 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 374 MBytes 314 Mbits/sec 1152 sender
[ 5] 0.00-10.00 sec 373 MBytes 313 Mbits/sec receiver
Lynx
October 8, 2022, 1:07pm
14
@patrakov someone on IRC has been helping me test more and when running iperf3 server on 192.168.1.2 and a bidir iperf3 test from 192.168.1.1 I see:
root@OpenWrt:~# iperf3 -c 192.168.1.2 --bidir
Connecting to host 192.168.1.2, port 5201
[ 5] local 192.168.1.1 port 51050 connected to 192.168.1.2 port 5201
[ 7] local 192.168.1.1 port 51066 connected to 192.168.1.2 port 5201
[ ID][Role] Interval Transfer Bitrate Retr Cwnd
[ 5][TX-C] 0.00-1.00 sec 98.2 MBytes 821 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 0.00-1.00 sec 16.3 MBytes 136 Mbits/sec
[ 5][TX-C] 1.00-2.00 sec 90.9 MBytes 765 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 1.00-2.00 sec 15.5 MBytes 131 Mbits/sec
[ 5][TX-C] 2.00-3.00 sec 101 MBytes 847 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 2.00-3.00 sec 13.3 MBytes 112 Mbits/sec
[ 5][TX-C] 3.00-4.00 sec 106 MBytes 889 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 3.00-4.00 sec 11.7 MBytes 97.9 Mbits/sec
[ 5][TX-C] 4.00-5.00 sec 97.0 MBytes 813 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 4.00-5.00 sec 12.9 MBytes 108 Mbits/sec
[ 5][TX-C] 5.00-6.00 sec 91.0 MBytes 762 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 5.00-6.00 sec 14.0 MBytes 118 Mbits/sec
[ 5][TX-C] 6.00-7.00 sec 77.0 MBytes 648 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 6.00-7.00 sec 16.3 MBytes 137 Mbits/sec
[ 5][TX-C] 7.00-8.00 sec 106 MBytes 888 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 7.00-8.00 sec 13.1 MBytes 110 Mbits/sec
[ 5][TX-C] 8.00-9.00 sec 97.2 MBytes 818 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 8.00-9.00 sec 13.7 MBytes 115 Mbits/sec
[ 5][TX-C] 9.00-10.01 sec 95.9 MBytes 800 Mbits/sec 0 1.94 MBytes
[ 7][RX-C] 9.00-10.01 sec 12.5 MBytes 104 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-10.01 sec 960 MBytes 805 Mbits/sec 0 sender
[ 5][TX-C] 0.00-10.01 sec 959 MBytes 803 Mbits/sec receiver
[ 7][RX-C] 0.00-10.01 sec 140 MBytes 118 Mbits/sec 296 sender
[ 7][RX-C] 0.00-10.01 sec 139 MBytes 117 Mbits/sec receiver
iperf Done.
No not really, except it might imply a cpu bottleneck on the router when running the server perhaps.
Lynx
October 8, 2022, 7:30pm
16
Thanks for your insight here. I posted on the RT3200 thread and it could be a driver issue:
I just noticed this yesterday.. or at least the high retransmission rate when transmitting from a 4mac STA interface -> 4mac AP interface. If I switch to OpenWrt master or use the mac80211+hostapd packages from master on 22.03 I no longer see this behavior. So something likely in mac80211 between 22.03 and master seems to have fixed it.
What version are you running?
I haven't had a chance to narrow it down to a specific patch but if I had to guess maybe this is what fixes it: https://git.open…
1 Like