Iperf3 - Perfomance "problem" C2600

Hi,

At home I have 2 OpenWRT devices switched to a Gigabit Switch, working like a charm.
The firts device is my router (192.168.1.1) x86 Celeron Machine J1900 CPU.
The second device is my AP (192.168.1.3) Archer C2600.

When I test the network speed with iperf from my J1900 to my C2600 I get:

root@Router-J1900:~# iperf3 -c 192.168.1.3 -p 5201
Connecting to host 192.168.1.3, port 5201
[  5] local 192.168.1.1 port 52224 connected to 192.168.1.3 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   106 MBytes   890 Mbits/sec    0    334 KBytes
[  5]   1.00-2.00   sec   107 MBytes   897 Mbits/sec    0    553 KBytes
[  5]   2.00-3.00   sec   109 MBytes   913 Mbits/sec    0    857 KBytes
[  5]   3.00-4.00   sec   111 MBytes   931 Mbits/sec    0   1.07 MBytes
[  5]   4.00-5.00   sec   111 MBytes   935 Mbits/sec    0   1.25 MBytes
[  5]   5.00-6.00   sec   112 MBytes   936 Mbits/sec    0   1.25 MBytes
[  5]   6.00-7.00   sec   112 MBytes   939 Mbits/sec    0   1.25 MBytes
[  5]   7.00-8.00   sec   111 MBytes   934 Mbits/sec    0   1.25 MBytes
[  5]   8.00-9.00   sec   112 MBytes   938 Mbits/sec    0   1.25 MBytes
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec    0   1.25 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   925 Mbits/sec    0             sender
[  5]   0.00-10.05  sec  1.08 GBytes   920 Mbits/sec                  receiver

iperf Done.

Looks great, no problem!!

But the same upside down:

root@C2600:~# iperf3 -c 192.168.1.1 -p 5201
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.3 port 56902 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  48.3 MBytes   400 Mbits/sec    0    136 KBytes
[  5]   1.01-2.01   sec  47.4 MBytes   398 Mbits/sec    0    178 KBytes
[  5]   2.01-3.01   sec  47.1 MBytes   398 Mbits/sec    0    199 KBytes
[  5]   3.01-4.02   sec  48.4 MBytes   399 Mbits/sec    0    214 KBytes
[  5]   4.02-5.00   sec  46.4 MBytes   399 Mbits/sec    0    232 KBytes
[  5]   5.00-6.01   sec  48.3 MBytes   401 Mbits/sec    0    246 KBytes
[  5]   6.01-7.02   sec  48.8 MBytes   403 Mbits/sec    0    263 KBytes
[  5]   7.02-8.00   sec  46.2 MBytes   397 Mbits/sec    0    308 KBytes
[  5]   8.00-9.01   sec  48.1 MBytes   402 Mbits/sec    0    421 KBytes
[  5]   9.01-10.02  sec  48.8 MBytes   402 Mbits/sec    0    421 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec   478 MBytes   400 Mbits/sec    0             sender
[  5]   0.00-10.06  sec   478 MBytes   398 Mbits/sec                  receiver

iperf Done.

It looks really poor!!
Any idea why are the results so different?

Many Thanks,

It may be the testbed using TCP is not fair. Try UDP.

iperf3 -c 192.168.1.1 -u -V

Although it seems counter-intuitive, this is probably not an issue. It is curious, but not necessarily totally surprising, that the performance is so asymmetric.

As it turns out, routers (especially consumer grade) are really good at routing packets, but are very poor at generating or terminating them. This is because there is dedicated hardware for the routing functions, but the general purpose CPU (which is responsible for generating/terminating packets in an iPerf test) is generally not powerful enough to keep up with line-rate packets.

To really test your router's throughput, you'd want to run iPerf between a computer on the WAN side of the network and one on the LAN side. If you have (or can experiment with) a double-NAT configuration, it is a fairly easy test to run.

EDIT: Also, it is safe to assume that packets running through the switch (typically LAN side) will run at line rate (i.e. ~940 Mbps) assuming a high enough power general purpose computer (i.e. any normal mac/win/linux box in the last 5-10 years) and a gigabit connection with good cables between each computer and the router's LAN ports (switched). Easy to test, too.
If you're testing across two or more VLANs, you'd be testing the routing speed since it would still require some CPU and routing HW to move packets between the VLANs.

2 Likes

Woowww!!! Worse than before!!

root@C2600:~# iperf3 -c 192.168.1.1 -u -V
iperf 3.6
Linux C2600 4.14.101 #0 SMP Mon Feb 18 15:11:32 2019 armv7l
Control connection MSS 1448
Setting UDP block size to 1448
Time: Fri, 22 Feb 2019 08:14:58 GMT
Connecting to host 192.168.1.1, port 5201
      Cookie: d7nzxte72j3vyl5gxiwxncjnjanhj534en6l
[  5] local 192.168.1.3 port 37598 connected to 192.168.1.1 port 5201
Starting Test: protocol: UDP, 1 streams, 1448 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   129 KBytes  1.05 Mbits/sec  91
[  5]   1.00-2.00   sec   127 KBytes  1.04 Mbits/sec  90
[  5]   2.00-3.00   sec   129 KBytes  1.05 Mbits/sec  91
[  5]   3.00-4.00   sec   129 KBytes  1.05 Mbits/sec  91
[  5]   4.00-5.00   sec   127 KBytes  1.04 Mbits/sec  90
[  5]   5.00-6.00   sec   129 KBytes  1.05 Mbits/sec  91
[  5]   6.00-7.00   sec   127 KBytes  1.04 Mbits/sec  90
[  5]   7.00-8.00   sec   129 KBytes  1.05 Mbits/sec  91
[  5]   8.00-9.00   sec   127 KBytes  1.04 Mbits/sec  90
[  5]   9.00-10.00  sec   129 KBytes  1.05 Mbits/sec  91
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  1.25 MBytes  1.05 Mbits/sec  0.000 ms  0/906 (0%)  sender
[  5]   0.00-10.05  sec  1.25 MBytes  1.04 Mbits/sec  0.083 ms  0/906 (0%)  receiver
CPU Utilization: local/sender 16.5% (3.3%u/13.2%s), remote/receiver 0.3% (0.3%u/0.0%s)

iperf Done.
root@C2600:~#

Thanks @psherman

I have read iperf3 is not able to work in multicore, only uses one core. C2600 is a dual core 1.4 Ghz router.
Maybe that's the point? Maybe iperf3 can't handle it?

There are known ethernet performance issues on the ipq8 platform, not much you can do atm unless you have hardware docs and know your way around the driver source code.

1 Like

Thanks @diizzy , I'll wait for better times on ipq8 platform.

Agreed but if you run the same test from 192.168.1.1 you will see the same result. The idea is to have the same result in both direction. You can tune the test to get higher bandwidth results.

1 Like