iPerf3 reported bitrate is lower when the server is running on the OpenWRT router itself than on connected devices

khs · November 12, 2021, 2:11am

I found very interesting results testing the speed of OpenWRT routers using iPerf3

Set-up:

All devices are wired through Gigabit ethernet. iPerf3 server was run with command iperf3 -s on the "router", "dump AP", and my notebook computer temporarily connected for testing. The desktop PC was the client with command iperf3 -c <IP address>

Results:

$ iperf3 -c 192.168.0.1
Connecting to host 192.168.0.1, port 5201
[  5] local 192.168.0.211 port 56272 connected to 192.168.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  45.0 MBytes   377 Mbits/sec    0    969 KBytes       
[  5]   1.00-2.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   2.00-3.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   3.00-4.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   4.00-5.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   5.00-6.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   6.00-7.00   sec  36.2 MBytes   304 Mbits/sec    0    969 KBytes       
[  5]   7.00-8.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   8.00-9.00   sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
[  5]   9.00-10.00  sec  35.0 MBytes   294 Mbits/sec    0    969 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   361 MBytes   303 Mbits/sec    0             sender
[  5]   0.00-10.09  sec   359 MBytes   299 Mbits/sec                  receiver

iperf Done.
$ iperf3 -c 192.168.0.175
Connecting to host 192.168.0.175, port 5201
[  5] local 192.168.0.211 port 54612 connected to 192.168.0.175 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   115 MBytes   963 Mbits/sec    0    210 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   935 Mbits/sec    0    210 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec    0    210 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   933 Mbits/sec    0    210 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec    0    210 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   940 Mbits/sec    0    210 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   940 Mbits/sec    0    210 KBytes       
[  5]   7.00-8.00   sec   113 MBytes   945 Mbits/sec    0    210 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    210 KBytes       
[  5]   9.00-10.00  sec   113 MBytes   945 Mbits/sec    0    210 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.
$ iperf3 -c 192.168.0.2
Connecting to host 192.168.0.2, port 5201
[  5] local 192.168.0.211 port 52910 connected to 192.168.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   108 MBytes   906 Mbits/sec    0    352 KBytes       
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec    0    352 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   931 Mbits/sec    0    352 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   931 Mbits/sec    0    352 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   937 Mbits/sec    0    423 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   937 Mbits/sec    0    423 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    423 KBytes       
[  5]   7.00-8.00   sec  99.8 MBytes   837 Mbits/sec    0    443 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    0    467 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    467 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec    0             sender
[  5]   0.00-9.93   sec  1.07 GBytes   927 Mbits/sec                  receiver

iperf Done.

The results are very interesting. Gigabit speed was not reached when iperf-ing the router itself. However, the problem is not due to the connections as near-gigabit speed was achieved when iperf-ing the notebook, which is connected to the same router as the desktop. I don't think it is particularly related to the default configuration in OpenWRT as well, as near-gigabit speed was achieved when iperf-ing the "dump AP" which is also running OpenWRT.

Speedtest.net, when run from the desktop, also reported around ~900Mbps speed. (I'm on a 1000Mbps fibre connection)

Any ideas/pointers in diagnosing such a problem? Thank you.

dave14305 · November 12, 2021, 2:15am

Watch the router's CPU usage during the tests. The iperf3 server will consume CPU that takes away from the router's ability to handle the higher throughput. Measuring the throughput impacts the throughput.

anon89577378 · November 12, 2021, 2:26am

Try iperf3 -c 192.168.0.1 -u -w 64K -p 5201 -b 1000M

TCP packets have higher overhead than UDP.

khs · November 12, 2021, 2:38am

I run iperf3 continous for 60 seconds and here's the monitoring from netdata:

Note: the netdata monitoring data is streamed continuously from the router to my desktop, and I'm viewing the dashboard from the desktop so there shouldn't be extra work related to serving the dashboard page on the router.

CPU usage is round 50%, which is not bad. However, when I look at the per core statistics, I can see that one of the cores is 100%.

I think that's the reason why the iPerf3 performance is capped. Thank you.

khs · November 12, 2021, 2:40am

$ iperf3 -c 192.168.0.1 -u -w 64K -p 5201 -b 1000M
Connecting to host 192.168.0.1, port 5201
[  5] local 192.168.0.211 port 54121 connected to 192.168.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   113 MBytes   950 Mbits/sec  81996  
[  5]   1.00-2.00   sec   113 MBytes   952 Mbits/sec  82150  
[  5]   2.00-3.00   sec   114 MBytes   953 Mbits/sec  82295  
[  5]   3.00-4.00   sec   114 MBytes   955 Mbits/sec  82414  
[  5]   4.00-5.00   sec   114 MBytes   955 Mbits/sec  82472  
[  5]   5.00-6.00   sec   113 MBytes   952 Mbits/sec  82180  
[  5]   6.00-7.00   sec   113 MBytes   944 Mbits/sec  81485  
[  5]   7.00-8.00   sec   113 MBytes   945 Mbits/sec  81540  
[  5]   8.00-9.00   sec   114 MBytes   955 Mbits/sec  82424  
[  5]   9.00-10.00  sec   114 MBytes   959 Mbits/sec  82767  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.11 GBytes   952 Mbits/sec  0.000 ms  0/821723 (0%)  sender
[  5]   0.00-10.00  sec   281 MBytes   235 Mbits/sec  0.036 ms  615801/819143 (75%)  receiver

iperf Done.

Yes I got Gigabit speed when running this with UDP.

anon89577378 · November 12, 2021, 2:41am

You'll get better TCP rates -

iperf3 -c 192.168.0.1 -w 64K -p 5201 -b 1000M

khs · November 12, 2021, 2:50am

$ iperf3 -c 192.168.0.1 -w 64K -p 5201 -b 1000M
Connecting to host 192.168.0.1, port 5201
[  5] local 192.168.0.211 port 44872 connected to 192.168.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  34.1 MBytes   286 Mbits/sec    0   67.9 KBytes       
[  5]   1.00-2.00   sec  32.9 MBytes   276 Mbits/sec    0   67.9 KBytes       
[  5]   2.00-3.00   sec  34.0 MBytes   285 Mbits/sec    0   67.9 KBytes       
[  5]   3.00-4.00   sec  34.1 MBytes   286 Mbits/sec    0   67.9 KBytes       
[  5]   4.00-5.00   sec  35.1 MBytes   294 Mbits/sec    0   67.9 KBytes       
[  5]   5.00-6.00   sec  34.5 MBytes   290 Mbits/sec    0   67.9 KBytes       
[  5]   6.00-7.00   sec  37.7 MBytes   316 Mbits/sec    0   67.9 KBytes       
[  5]   7.00-8.00   sec  36.0 MBytes   302 Mbits/sec    0   67.9 KBytes       
[  5]   8.00-9.00   sec  35.3 MBytes   296 Mbits/sec    0   67.9 KBytes       
[  5]   9.00-10.00  sec  34.3 MBytes   288 Mbits/sec    0   67.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   348 MBytes   292 Mbits/sec    0             sender
[  5]   0.00-9.98   sec   348 MBytes   292 Mbits/sec                  receiver

iperf Done.

The rates were not better.
But I think routers CPUs are not able to handle any kind of "real" Gigabit connections/operations. I can get away with hardware offloading on the router though.
That ~300Mbps is consistent with the speed I'm getting without hardware offloading. So I think that's a valid result for TCP.
For UDP, I agree that the overhead is lower. But the router also can't handle that much and essentially dropped 75% of the packets.

But it is also interesting to me that single core performance on the Qualcomm IPQ4018 (the CPU of the ASUS RT-AC58U) is stronger than that of MediaTek MT7621AT (the CPU of the of the Linksys), if iperf3 is totally single-threaded.

anon89577378 · November 12, 2021, 3:22am

The TCP rates doubled for me on my single core (Archer C7 v2), with SQM, and no software/hardware offloading.

I was thinking the 880 Mhz on the Linksys would get better numbers than my 720 Mhz.

khs · November 12, 2021, 9:05am

I also have an Archer C7 v2 (as the second dumb AP). However, I'm not running mainline OpenWRT but gwlim's Openwrt firmware with SFE and FlowOffload on it. I couldn't running iPerf3 server on it after installing via opkg. This might be due to the custom firmware. I may switch back to official OpenWRT one day but not very inclined to do so as now it runs perfectly as a dumb AP.

root@OpenWrt:~# iperf3 -s
Error relocating /usr/bin/iperf3: __select_time64: symbol not found
Error relocating /usr/bin/iperf3: __localtime64: symbol not found
Error relocating /usr/bin/iperf3: __fstat_time64: symbol not found
Error relocating /usr/bin/iperf3: __getrusage_time64: symbol not found
Error relocating /usr/bin/iperf3: __gmtime64: symbol not found
Error relocating /usr/bin/iperf3: __time64: symbol not found
Error relocating /usr/bin/iperf3: __clock_gettime64: symbol not found

I'll still use my Linksys EA7500v2 as my main router as it is the only one with officially supported hardware offloading. Maybe the CPU was just poorly optimized for iPerf3 benchmarks. I'm lucky that I can get a bufferbloat rating of A with the 1Gbps fibre connection, so I can just get away with hardware offloading only without SQM. Otherwise I'll need to upgrade to more powerful boards (like Raspberry Pi 4 or FriendlyELEC NanoPi R4S).

system · November 22, 2021, 9:06am

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.