Iperf 2.3.8-rc1 and bufferbloat testing

Iperf2 has added a ton of new features intended around testing for bufferbloat better, and they just put out an rc1 candidate which will hopefully go final in a few weeks. BUT! It has not been well tested on various low end architectures yet. I am not really sure how I could spin up a test on various platforms of that rc before it goes final, so here's a patch (is there a better way to do this?) to build it. I've prebuilt packages for a few arches I use as well. These are here:

https://www.rjmcmahon.com/iperf/testing/2.1.8-rc1/openwrt/
src and a windows version here: https://sourceforge.net/projects/iperf2/files/

diff --git a/net/iperf/Makefile b/net/iperf/Makefile
index d071d0c03..ada775880 100644
--- a/net/iperf/Makefile
+++ b/net/iperf/Makefile
@@ -8,11 +8,11 @@
 include $(TOPDIR)/rules.mk

 PKG_NAME:=iperf
-PKG_VERSION:=2.1.3
+PKG_VERSION:=2.1.8-rc1
 PKG_RELEASE:=$(AUTORELEASE)

 PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
-PKG_HASH:=dfe2197e2842fe9c9d9677bf1cb20a5a9ccfcb9a9de79f9927c39f73204ba003
+PKG_HASH:=c1adeb999d7385e99e60f0db5b8ef03644976749ed858f4738ec34b9e57f0d33
 PKG_SOURCE_URL:=@SF/iperf2

 PKG_MAINTAINER:=Felix Fietkau <nbd@nbd.name>
@@ -45,6 +45,7 @@ define Package/iperf/config
 endef

 CONFIGURE_ARGS += \
+       --enable-fastsampling \
        $(call autoconf_bool,CONFIG_IPERF_ENABLE_MULTICAST,multicast) \
        $(call autoconf_bool,CONFIG_IPV6,ipv6)

I'll describe the new stuff in the next post on this thread

4 Likes

In particular there is a new low rate, high accuracy (we hope) "bounceback" test.

[rjmcmahon@bobcat iperf2-code]$ iperf -v
iperf version 2.1.8-rc (6 August 2022) pthreads

[rjmcmahon@bobcat iperf2-code]$ iperf -s --permit-key=mytest  -P 1 --hide-ips
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
Permit key is 'mytest' (timeout in 20.0 seconds)
------------------------------------------------------------

[rjmcmahon@ryzen3950 iperf2-code]$ iperf -c hostname -i 1 --bounceback --permit-key=mytest --hide-ips
------------------------------------------------------------
Client connecting to (**hidden**), TCP port 5001
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[mytest(1)] local *.*.*.96 port 38048 connected with *.*.*.123 port 5001 (bb len/hold=100/0) (icwnd/mss/irtt=14/1448/15038)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[mytest(1)] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.333/9.247/15.596/2.493 ms    0   14K/11667 us    88 rps
[mytest(1)] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.164/9.340/13.791/1.417 ms    0   14K/10374 us    98 rps
[mytest(1)] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.301/9.321/14.310/2.010 ms    0   14K/10796 us    88 rps
[mytest(1)] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.242/9.432/14.401/2.128 ms    0   14K/10720 us    88 rps
[mytest(1)] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.576/9.558/14.442/2.196 ms    0   14K/10821 us    86 rps
[mytest(1)] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.310/9.514/14.615/2.151 ms    0   14K/10806 us    88 rps
[mytest(1)] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.944/9.310/14.490/2.233 ms    0   14K/10548 us    91 rps
[mytest(1)] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.584/9.468/19.286/3.254 ms    0   14K/10794 us    86 rps
[mytest(1)] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.987/9.378/14.322/2.139 ms    0   14K/10533 us    90 rps
[mytest(1)] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.449/9.930/14.722/2.151 ms    0   14K/10832 us    87 rps
[mytest(1)] 0.00-10.03 sec  19.5 KBytes  15.9 Kbits/sec    100=11.189/9.247/19.286/2.190 ms    0   14K/11083 us    89 rps
[  1] 0.00-10.03 sec BB8(f)-PDF: bin(w=100us):cnt(100)=93:1,94:9,95:5,96:11,97:9,98:2,99:5,100:6,101:9,102:3,103:3,104:1,105:1,106:2,107:1,115:1,129:1,130:1,132:1,137:1,138:2,139:3,141:4,142:2,143:4,144:3,145:4,147:2,148:1,156:1,193:1 (5.00/95.00/99.7%=94/147/193,Outliers=0,obl/obu=0/0)
1 Like

Maybe you could elaborate a bit what kind of packets are exchanged here and what the main take-away is?

Confused, Iperf3 have been around for ages.

Or isn't Iperf3 the same as Iperf2, but newer?

iperf and iperf2 are the same. Maintained by Broadcom.

iperf3 is a completely rewritten version ( by academics) of the same concepts, but in particular lacks the high speed threading that iperf2 has, and not the same feature set. a comparison of features is here: https://iperf2.sourceforge.io/IperfCompare.html

OpenWrt bundles both.

Iperf2's man page: https://iperf2.sourceforge.io/iperf-manpage.html - in particular see the doc on bounceback.

3 Likes

Iperf2's macos binary: https://drive.google.com/file/d/1cBQNsrAiuZjOA398LWwB_sNLV-JrRt8k/view?usp=sharing

Just in case anyone wants to test it and don't have tools to compile it in macOS.

2 Likes

Unfortunatly there is a lot of confusion around iperf 2 and iperf 3. Iperf 3 really should have used a different name. Iperf 2 is the continuation of the original iperf code base. We have a WiFi bent to our use cases but also test with 100Gb/s NICs, etc. The later features are around latency and responsiveness. A comparison table

IPerf 2 (this program) is different from the iperf3 found at https://github.com/esnet/iperf Each can be used to measure network performance, however, they DO NOT interoperate. They are completely independent implementations with different strengths, different options, and different capabilities. Both are under active development

3 Likes

Iperf doesn't really send packets. It issues socket writes and the OS takes care of the packetizing. For UDP the writes and the packets do match if the write size is less than the MTU.

There is an industry push for measuring network responsiveness under "working conditions" or "working loads." From a test & measurement perspective, these are difficult to define. We do a poor engineer's attempt with the bounceback-congest option. (There is also pyflows python wrappers that are in an alpha stage.)

The bounceback test is a socket write of length given by -l (defaults to 100 bytes) which is received by the server. The server then does a 100 byte write in return. The flow writeA->readB->writeB->readA. To minimize impact, the periodic bounceback defaults to one second. There are 10 bouncebacks per second (by default) which allows for standard central limit theorem type stats of mean, min, max and standard deviation. These results are given both in latency (time) as well as "responses per second" shown as RPS.

Those that want advanced use cases including one-way delay (OWD) measurements will want to synchronize the clocks with something like a GPS atomic clock reference and PTP or IEEE1588 for clock distribution.

1 Like

I'm wondering if APs had support for bounceback so IoT WiFi devices could report both sensor information along with WiFi responsiveness information to the cloud service. Having iperf 2 server support with openwrt could help with this, particularly if the server were advertised to the local network. Then a time series with responsiveness data could be provided to those trying to manage WiFi networks and would have minimal impact on the network.

Most SOCs in common wireless router (including modern high-end ones) aren't fast enough to run iperf or iperf3 on the router at full wired/ wireless speed (you are looking at over 700-1200 MBit/s for 802.11ax). Both tools are available for OpenWrt and can be run on the router, just that they usually won't provide meaningful results due to the (too slow) SOC performance.

some of the beauty of the bounceback idea is that it is lightweight and can aid in diagnosing if the problem is on the local wifi network or elsewhere on the path.

The iperf bounceback test does not overload the network capacity so-to-speak. It's a responsiveness test and not a capacity test. Most networks today are now limited by responsiveness and not so much link capacity. The goal is to continuously measure responsiveness as a time series (without impact, i.e. mitigate "observe is to disturb") and report issues ahead of user complaints.

Here's an example on a 10G link, the average consumed throughput is 16Kbit/sec. Same for a run to my server on the real internet over a last mile, internet access link. The RPS metric indicates the responsiveness which can be extrapolated to others competing for the WiFi TXOP arbitrations.

[root@ryzen3950 iperf2-code]# iperf -c 192.168.1.87 -i 1 --bounceback -e
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 1322315 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 39156 connected with 192.168.1.87 port 5001 (bb len/hold=100/0) (sock=3) (icwnd/mss/irtt=14/1448/432) (ct=0.55 ms) on 2022-08-08 16:21:35 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.198/0.112/0.609/0.149 ms    0   14K/199 us    5035 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.184/0.090/0.412/0.098 ms    0   14K/156 us    5379 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.448/0.236/1.081/0.233 ms    0   14K/332 us    2208 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.317/0.152/0.972/0.253 ms    0   14K/283 us    3116 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.194/0.095/0.378/0.078 ms    0   14K/191 us    5109 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.472/0.170/1.326/0.314 ms    0   14K/333 us    2100 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.438/0.163/1.158/0.278 ms    0   14K/339 us    2262 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.460/0.304/0.973/0.187 ms    0   14K/379 us    2155 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.450/0.282/0.989/0.200 ms    0   14K/390 us    2207 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=0.469/0.319/1.079/0.219 ms    0   14K/404 us    2110 rps
[  1] 0.00-10.01 sec  19.5 KBytes  16.0 Kbits/sec    100=0.363/0.090/1.326/0.236 ms    0   14K/476 us    2730 rps
[  1] 0.00-10.01 sec BB8(f)-PDF: bin(w=100us):cnt(100)=1:3,2:18,3:22,4:13,5:35,6:1,7:1,10:3,11:2,12:1,14:1 (5.00/95.00/99.7%=2/10/14,Outliers=0,obl/obu=0/0)

[root@ryzen3950 iperf2-code]# iperf -c <hostame> -i 1 --bounceback -e --hide-ips
------------------------------------------------------------
Client connecting to (**hidden**), TCP port 5001 with pid 1322678 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local *.*.*.96%enp4s0 port 38184 connected with *.*.*.123 port 5001 (bb len/hold=100/0) (sock=3) (icwnd/mss/irtt=14/1448/10503) (ct=10.66 ms) on 2022-08-08 16:24:49 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.535/9.189/14.642/2.416 ms    0   14K/10772 us    86 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.064/9.418/14.457/2.288 ms    0   14K/10601 us    90 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.178/9.113/14.746/2.194 ms    0   14K/10682 us    89 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.388/9.584/14.568/2.100 ms    0   14K/10819 us    87 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.472/9.911/14.807/2.032 ms    0   14K/10905 us    87 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.579/9.816/14.982/2.176 ms    0   14K/11083 us    86 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.968/9.394/14.364/1.803 ms    0   14K/10681 us    91 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=11.145/9.649/16.134/2.224 ms    0   14K/10725 us    89 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.646/9.373/14.586/1.870 ms    0   14K/10373 us    93 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.705/9.017/14.219/1.887 ms    0   14K/10519 us    93 rps
[  1] 0.00-10.03 sec  19.5 KBytes  16.0 Kbits/sec    100=11.168/9.017/16.134/2.034 ms    0   14K/10931 us    89 rps
[  1] 0.00-10.03 sec BB8(f)-PDF: bin(w=100us):cnt(100)=91:1,92:2,93:1,94:6,95:7,96:2,97:6,98:5,99:7,100:9,101:5,102:4,103:5,104:2,105:1,106:1,107:2,108:1,113:2,119:1,122:1,123:1,127:1,128:1,137:1,139:2,140:1,141:1,142:3,143:4,144:6,145:1,146:2,147:1,148:1,149:1,150:1,162:1 (5.00/95.00/99.7%=94/147/162,Outliers=0,obl/obu=0/0)



I just gave it a go, ran iperf -s -P 1 in my mt76 AP and launched it in my macOS station connected to the AP:

@reaper$ ➜  ~ iperf -c nanohd-downstairs.lan -i 1 --bounceback
------------------------------------------------------------
Client connecting to nanohd-downstairs.lan, TCP port 5001
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs)
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.151 port 59603 connected with 192.168.1.2 port 5001 (bb len/hold=100/0) (icwnd/mss/irtt=16380/1448/7000)
[ ID] Interval            Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.0000-1.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=4.851/2.613/19.561/5.240 ms    0 17794K/2000 us    205 rps
[  1] 1.0000-2.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.254/2.770/3.879/0.445 ms    0 19208K/3000 us    305 rps
[  1] 2.0000-3.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.488/2.294/5.934/1.175 ms    0 20622K/3000 us    281 rps
[  1] 3.0000-4.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.880/2.760/10.288/2.300 ms    0 22036K/3000 us    257 rps
[  1] 4.0000-5.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.250/2.836/5.431/0.836 ms    0 23450K/3000 us    307 rps
[  1] 5.0000-6.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.594/2.258/10.002/2.307 ms    0 24864K/3000 us    277 rps
[  1] 6.0000-7.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.625/2.781/6.911/1.275 ms    0 26278K/3000 us    275 rps
[  1] 7.0000-8.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.106/2.715/5.017/0.687 ms    0 27693K/3000 us    321 rps
[  1] 8.0000-9.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.395/2.157/9.611/2.222 ms    0 29107K/2000 us    294 rps
[  1] 9.0000-10.0000 sec  1.95 KBytes  16.0 Kbits/sec    10=3.833/2.798/10.195/2.288 ms    0 30521K/3000 us    260 rps
[  1] 0.0000-10.0284 sec  19.5 KBytes  16.0 Kbits/sec    100=3.628/2.157/19.561/2.238 ms    0 30522K/22000 us    274 rps
[  1] 0.0000-10.0284 sec BB8-PDF: bin(w=100us):cnt(100)=22:2,23:4,26:2,27:3,28:13,29:17,30:18,31:6,32:3,33:4,34:3,35:2,37:1,38:2,39:2,40:5,45:2,51:1,53:1,55:2,60:1,70:1,97:1,101:1,102:1,103:1,196:1 (5.00/95.00/99.7%=23/97/196,Outliers=1,obl/obu=0/0)

Thanks for this. I did some fixes to the code timing calculations. It also defaults to one second interval so no need for -i 1.

I get a similar result on my WiFi network. I do notice the Mac OS X congestion window (cwnd) seems off. Need to look into that.

bm932125@C02YQ4QNLVCG iperf2-code % iperf -c 192.168.1.87 -e --bounceback
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 8310 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=0 usecs)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.74%en0 port 54122 connected with 192.168.1.87 port 5001 (bb len/hold=100/0) (sock=4) (icwnd/mss/irtt=16380/1448/3000) (ct=2.76 ms) on 2022-08-09 21:09:33 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=3.878/2.477/7.887/1.944 ms    0 17794K/3000 us    257 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=2.670/2.222/3.468/0.509 ms    0 19208K/3000 us    374 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=3.649/2.810/4.621/0.580 ms    0 20622K/3000 us    274 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=4.171/2.904/7.359/1.368 ms    0 22036K/2000 us    239 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=3.595/2.137/8.936/2.223 ms    0 23450K/3000 us    278 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=2.650/2.314/3.170/0.233 ms    0 24864K/3000 us    377 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=2.519/2.119/4.375/0.696 ms    0 26278K/2000 us    396 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=3.026/2.155/4.442/0.722 ms    0 27693K/2000 us    330 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=3.841/2.706/9.511/2.094 ms    0 29107K/3000 us    260 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=4.487/2.695/9.741/2.785 ms    0 30521K/3000 us    222 rps
[  1] 0.00-10.03 sec  19.7 KBytes  16.1 Kbits/sec    101=3.506/2.119/9.741/1.719 ms    0 30663K/20000 us    285 rps
[  1] 0.00-10.03 sec BB8(f)-PDF: bin(w=100us):cnt(101)=22:7,23:7,24:5,25:3,26:3,27:9,28:8,29:9,30:4,31:5,32:3,33:3,34:3,35:3,36:1,37:4,38:2,39:1,41:1,42:3,44:2,45:2,46:1,47:2,56:1,63:1,69:1,74:1,79:1,90:1,93:1,96:1,97:1,98:1 (5.00/95.00/99.7%=22/79/98,Outliers=0,obl/obu=0/0)
1 Like

The number I have been trying to explore is... why are these rps's so low? We should be capable of about a 250us RT. Are the other latencies hiding in the wifi driver? the ethernet driver? Another thought is to play with the VO queue.

@dtaht, I've a few hours in front of me I might be able to help with some testing. What do you need exactly? And, how can I help in this occasion?

go outside. Enjoy the starshine. Have a beer. If you MUST run tests also :), try 'em from these more distant vantage points. In the case of these bounceback tests, this will explore the impact of lower rates, interference and wifi retries. VO queue?

The only other thing I can think of to improve no-load bounceback in this context is increasing the ethtool interrupt rate.

You could fire off these again AQL and the ath10k is *lovely* - #830 by amteza at this distance.

Do you have 2 machines to do rtt_fair against?

Hahaha! 3 AM here and not much of a sunny day ahead in Melbourne today. :slight_smile:

I'll see what I can do, I've a couple of machines I can use for a few tests.

Enjoy the starshine, then. And thx for confirming this new feature of this new iperf actually works on the mt76. I don't know how accurate iperf is in regards to flent + netperf on a variety of other potential benchmarks.

1 Like

@rjmcmahon

What I find really remarkable in your 10gbit result was userspace jitter in the 1.3ms range. This on native hw, not a vm?

10=0.472/0.170/1.326/0.314 ms

there are a couple ethtool command to tell it to interrupt more often. You can try realtime privs. I personally run on a realtime linux kernel (ubuntu studio)