Iperf 2.3.8-rc1 and bufferbloat testing

FYI, just did some fixes around TCP_QUICKACK. Also, there are four transactions per one bounceback, two writes and two TCP acks. Quickack is enabled by default.

bounceback-hold can be used to verify that quick ack is working, by comparing the RTT with the bounceback time, and using --bounceback-no-quickack to disable.

I'll play around with 10G a bit. There is a netgear switch too though doubtful this would drive a worst case to over 1 ms.

[rjmcmahon@ryzen3950 iperf2-code]$ src/iperf -c 192.168.1.87 --bounceback -**-bounceback-no-quickack --bounceback-hold 10**
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 1586326 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=10000 usecs)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 39380 connected with 192.168.1.87 port 5001 (bb len/hold=100/10000) (sock=3) (icwnd/mss/irtt=14/1448/202) (ct=0.31 ms) on 2022-08-10 11:44:09 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.620/10.357/11.063/0.299 ms    0   14K/3726 us    94 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.858/10.572/11.105/0.220 ms    0   14K/8959 us    92 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.929/10.555/11.425/0.293 ms    0   14K/**10400** us    91 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.854/10.468/11.308/0.337 ms    0   14K/10765 us    92 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.890/10.532/11.170/0.233 ms    0   14K/10828 us    91 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.893/10.558/11.122/0.188 ms    0   14K/10855 us    91 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.793/10.545/11.378/0.275 ms    0   14K/10812 us    92 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.792/10.526/11.128/0.229 ms    0   14K/10746 us    92 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.766/10.456/11.333/0.309 ms    0   14K/10704 us    92 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.927/10.589/11.425/0.292 ms    0   14K/10859 us    91 rps
[  1] 0.00-10.02 sec  19.5 KBytes  16.0 Kbits/sec    100=10.832/10.357/11.425/0.274 ms    0   14K/10889 us    92 rps
[  1] 0.00-10.02 sec BB8(f)-PDF: bin(w=100us):cnt(100)=104:2,105:6,106:21,107:12,108:8,109:5,110:8,111:22,112:9,113:1,114:4,115:2 (5.00/95.00/99.7%=105/114/115,Outliers=0,obl/obu=0/0)


[rjmcmahon@ryzen3950 iperf2-code]$ src/iperf -c 192.168.1.87 --bounceback **--bounceback-hold 10**
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 1586369 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=10000 usecs & tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 39382 connected with 192.168.1.87 port 5001 (bb w/quickack len/hold=100/10000) (sock=3) (qack) (icwnd/mss/irtt=14/1448/295) (ct=0.34 ms) on 2022-08-10 11:44:36 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.822/10.548/11.346/0.282 ms    0   14K/**360 us**    92 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.929/10.506/11.320/0.345 ms    0   14K/425 us    91 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.818/10.470/11.148/0.274 ms    0   14K/510 us    92 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.891/10.567/11.321/0.286 ms    0   14K/658 us    91 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.906/10.442/11.273/0.261 ms    0   14K/670 us    91 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.866/10.509/11.190/0.271 ms    0   14K/629 us    92 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.810/10.435/11.164/0.276 ms    0   14K/547 us    92 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.711/10.435/11.047/0.237 ms    0   14K/492 us    93 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.419/10.373/10.569/0.054 ms    0   14K/234 us    95 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.407/10.326/10.544/0.056 ms    0   14K/161 us    96 rps
[  1] 0.00-10.02 sec  19.5 KBytes  16.0 Kbits/sec    100=10.758/10.326/11.346/0.302 ms    0   14K/192 us    92 rps
[  1] 0.00-10.02 sec BB8(f)-PDF: bin(w=100us):cnt(100)=104:7,105:18,106:19,107:9,108:3,109:4,110:10,111:16,112:7,113:4,114:3 (5.00/95.00/99.7%=104/113/114,Outliers=0,obl/obu=0/0)

forgot to mention, -z and --realtime is supported if it's supported on the device. This will set the traffic thread to SCHED_RR

-z, --realtime
Request real-time scheduler, if supported.

/* -------------------------------------------------------------------
 * Support for realtime scheduling of threads
 * ------------------------------------------------------------------- */
#if HAVE_SCHED_SETSCHEDULER
#include <sched.h>
#endif
#ifdef HAVE_MLOCKALL
#include <sys/mman.h>
#endif
void thread_setscheduler(struct thread_Settings *thread) {
#if HAVE_SCHED_SETSCHEDULER
    if (isRealtime(thread)) {
	struct sched_param sp;
	sp.sched_priority = sched_get_priority_max(SCHED_RR);
	// SCHED_OTHER, SCHED_FIFO, SCHED_RR
	if (sched_setscheduler(0, SCHED_RR, &sp) < 0)  {
	    perror("Client set scheduler");
#ifdef HAVE_MLOCKALL
	} else if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0) {
	    // lock the threads memory
	    perror ("mlockall");
#endif // MLOCK
	}
    }
#endif // SCHED
}

Here's one way delay (OWD) using UDP in each direction. Clock synced to 100 ns using ptp4l and GPS disciplined OCXO as the reference.

[root@rjm-nas rjmcmahon]# iperf -s -i 1 -e -u -B 192.168.1.87%enp2s0 --realtime
------------------------------------------------------------
Server listening on UDP port 5001 with pid 15024
Binding to local address 192.168.1.87 and iface enp2s0
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.87%enp2s0 port 5001 connected with 192.168.1.96 port 59750 (trip-times) (sock=3) (peer 2.1.8-rc) on 2022-08-10 12:59:19 (PDT)
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-1.00 sec   131 KBytes  1.07 Mbits/sec   0.023 ms 0/91 (0%) 0.168/0.067/0.303/0.041 ms 92 pps 91/0(0) pkts 794
[  1] 1.00-2.00 sec   128 KBytes  1.05 Mbits/sec   0.016 ms 0/89 (0%) 0.155/0.057/0.314/0.035 ms 89 pps 89/0(0) pkts 842
[  1] 2.00-3.00 sec   128 KBytes  1.05 Mbits/sec   0.014 ms 0/89 (0%) 0.130/0.042/0.163/0.019 ms 89 pps 89/0(0) pkts 1007
[  1] 3.00-4.00 sec   128 KBytes  1.05 Mbits/sec   0.008 ms 0/89 (0%) 0.156/0.049/0.291/0.030 ms 89 pps 89/0(0) pkts 837
[  1] 4.00-5.00 sec   128 KBytes  1.05 Mbits/sec   0.025 ms 0/89 (0%) 0.166/0.084/0.292/0.030 ms 89 pps 89/0(0) pkts 786
[  1] 5.00-6.00 sec   128 KBytes  1.05 Mbits/sec   0.022 ms 0/89 (0%) 0.168/0.088/0.297/0.027 ms 89 pps 89/0(0) pkts 777
[  1] 6.00-7.00 sec   129 KBytes  1.06 Mbits/sec   0.032 ms 0/90 (0%) 0.158/0.088/0.302/0.032 ms 89 pps 90/0(0) pkts 836
[  1] 7.00-8.00 sec   128 KBytes  1.05 Mbits/sec   0.016 ms 0/89 (0%) 0.170/0.078/0.304/0.031 ms 89 pps 89/0(0) pkts 768
[  1] 8.00-9.00 sec   128 KBytes  1.05 Mbits/sec   0.030 ms 0/89 (0%) 0.158/0.092/0.296/0.029 ms 89 pps 89/0(0) pkts 827
[  1] 9.00-10.00 sec   128 KBytes  1.05 Mbits/sec   0.022 ms 0/89 (0%) 0.146/0.063/0.212/0.030 ms 89 pps 89/0(0) pkts 898
[  1] 0.00-10.02 sec  1.25 MBytes  1.05 Mbits/sec   0.019 ms 0/895 (0%) 0.158/0.042/0.314/0.033 ms 89 pps 895/0(0) pkts 833


[root@ryzen3950 iperf2-code]# iperf -s -i 1 -e -B 192.168.1.96%enp4s0 -u --realtime
------------------------------------------------------------
Server listening on UDP port 5001 with pid 1600916
Binding to local address 192.168.1.96 and iface enp4s0
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 5001 connected with 192.168.1.84 port 51101 (trip-times) (sock=3) (peer 2.1.8-rc) on 2022-08-10 13:27:53 (PDT)
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-1.00 sec   131 KBytes  1.07 Mbits/sec   0.181 ms 0/91 (0%) 0.482/0.059/0.851/0.229 ms 92 pps 91/0(0) pkts 278
[  1] 1.00-2.00 sec   128 KBytes  1.05 Mbits/sec   0.053 ms 0/89 (0%) 0.232/0.109/0.772/0.205 ms 89 pps 89/0(0) pkts 565
[  1] 2.00-3.00 sec   128 KBytes  1.05 Mbits/sec   0.005 ms 0/89 (0%) 0.152/0.103/0.645/0.126 ms 89 pps 89/0(0) pkts 864
[  1] 3.00-4.00 sec   128 KBytes  1.05 Mbits/sec   0.216 ms 0/89 (0%) 0.315/0.099/0.878/0.233 ms 89 pps 89/0(0) pkts 415
[  1] 4.00-5.00 sec   128 KBytes  1.05 Mbits/sec   0.112 ms 0/89 (0%) 0.434/0.097/0.806/0.239 ms 89 pps 89/0(0) pkts 302
[  1] 5.00-6.00 sec   128 KBytes  1.05 Mbits/sec   0.133 ms 0/89 (0%) 0.454/0.106/0.868/0.248 ms 89 pps 89/0(0) pkts 288
[  1] 6.00-7.00 sec   129 KBytes  1.06 Mbits/sec   0.245 ms 0/90 (0%) 0.478/0.098/0.919/0.227 ms 89 pps 90/0(0) pkts 277
[  1] 7.00-8.00 sec   128 KBytes  1.05 Mbits/sec   0.209 ms 0/89 (0%) 0.550/0.088/0.884/0.200 ms 89 pps 89/0(0) pkts 238
[  1] 8.00-9.00 sec   128 KBytes  1.05 Mbits/sec   0.202 ms 0/89 (0%) 0.481/0.100/0.869/0.225 ms 89 pps 89/0(0) pkts 272
[  1] 9.00-10.00 sec   128 KBytes  1.05 Mbits/sec   0.047 ms 0/89 (0%) 0.261/0.089/0.851/0.226 ms 89 pps 89/0(0) pkts 502
[  1] 0.00-10.02 sec  1.25 MBytes  1.05 Mbits/sec   0.042 ms 0/895 (0%) 0.383/0.059/0.919/0.251 ms 89 pps 895/0(0) pkts 343


04:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)




@dtaht This thread has seemed cold. Have you completed your work? Is the rc now in mainline? Curious how this has proceeded. Thank you!

Hi, @clutch2sft,

I summited a PR for the package to be included in the mainline long ago. It was approved, and this is part of the master now.

Kind regards.

2 Likes

I hope more use it now!

1 Like

BTW, I wonder if you plan to provide iperf2 servers running in your flent servers. Do you plan to do so?

Is there an app to use iperf2 on iOS? It would be very useful to test the latency over Wi-Fi!

yes, but I haven't done it yet. I was mostly planning a new round with a new name for the cluster when linux 6.2 landed.

1 Like

Not to my knowledge, I updated the brew formula too at the same time that OpenWrt, just in case you have a macOS computer.

1 Like

Okay thanks anyway. Yes running it on macOS or OpenWrt is useful but since macs can have Ethernet cable, a simple client on iOS would be more useful for Wi-Fi testing. There are some iperf3 apps on the Store but not iPerf2 unfortunately.