Iperf 2.3.8-rc1 and bufferbloat testing

FYI, just did some fixes around TCP_QUICKACK. Also, there are four transactions per one bounceback, two writes and two TCP acks. Quickack is enabled by default.

bounceback-hold can be used to verify that quick ack is working, by comparing the RTT with the bounceback time, and using --bounceback-no-quickack to disable.

I'll play around with 10G a bit. There is a netgear switch too though doubtful this would drive a worst case to over 1 ms.

[rjmcmahon@ryzen3950 iperf2-code]$ src/iperf -c 192.168.1.87 --bounceback -**-bounceback-no-quickack --bounceback-hold 10**
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 1586326 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=10000 usecs)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 39380 connected with 192.168.1.87 port 5001 (bb len/hold=100/10000) (sock=3) (icwnd/mss/irtt=14/1448/202) (ct=0.31 ms) on 2022-08-10 11:44:09 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.620/10.357/11.063/0.299 ms    0   14K/3726 us    94 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.858/10.572/11.105/0.220 ms    0   14K/8959 us    92 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.929/10.555/11.425/0.293 ms    0   14K/**10400** us    91 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.854/10.468/11.308/0.337 ms    0   14K/10765 us    92 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.890/10.532/11.170/0.233 ms    0   14K/10828 us    91 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.893/10.558/11.122/0.188 ms    0   14K/10855 us    91 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.793/10.545/11.378/0.275 ms    0   14K/10812 us    92 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.792/10.526/11.128/0.229 ms    0   14K/10746 us    92 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.766/10.456/11.333/0.309 ms    0   14K/10704 us    92 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.927/10.589/11.425/0.292 ms    0   14K/10859 us    91 rps
[  1] 0.00-10.02 sec  19.5 KBytes  16.0 Kbits/sec    100=10.832/10.357/11.425/0.274 ms    0   14K/10889 us    92 rps
[  1] 0.00-10.02 sec BB8(f)-PDF: bin(w=100us):cnt(100)=104:2,105:6,106:21,107:12,108:8,109:5,110:8,111:22,112:9,113:1,114:4,115:2 (5.00/95.00/99.7%=105/114/115,Outliers=0,obl/obu=0/0)


[rjmcmahon@ryzen3950 iperf2-code]$ src/iperf -c 192.168.1.87 --bounceback **--bounceback-hold 10**
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001 with pid 1586369 (1 flows)
Write buffer size:  100 Byte
Bursting:  100 Byte writes 10 times every 1.00 second(s)
Bounce-back test (size= 100 Byte) (server hold req=10000 usecs & tcp_quickack)
TOS set to 0x0 and nodelay (Nagle off)
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 39382 connected with 192.168.1.87 port 5001 (bb w/quickack len/hold=100/10000) (sock=3) (qack) (icwnd/mss/irtt=14/1448/295) (ct=0.34 ms) on 2022-08-10 11:44:36 (PDT)
[ ID] Interval        Transfer    Bandwidth         BB cnt=avg/min/max/stdev         Rtry  Cwnd/RTT    RPS
[  1] 0.00-1.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.822/10.548/11.346/0.282 ms    0   14K/**360 us**    92 rps
[  1] 1.00-2.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.929/10.506/11.320/0.345 ms    0   14K/425 us    91 rps
[  1] 2.00-3.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.818/10.470/11.148/0.274 ms    0   14K/510 us    92 rps
[  1] 3.00-4.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.891/10.567/11.321/0.286 ms    0   14K/658 us    91 rps
[  1] 4.00-5.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.906/10.442/11.273/0.261 ms    0   14K/670 us    91 rps
[  1] 5.00-6.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.866/10.509/11.190/0.271 ms    0   14K/629 us    92 rps
[  1] 6.00-7.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.810/10.435/11.164/0.276 ms    0   14K/547 us    92 rps
[  1] 7.00-8.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.711/10.435/11.047/0.237 ms    0   14K/492 us    93 rps
[  1] 8.00-9.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.419/10.373/10.569/0.054 ms    0   14K/234 us    95 rps
[  1] 9.00-10.00 sec  1.95 KBytes  16.0 Kbits/sec    10=10.407/10.326/10.544/0.056 ms    0   14K/161 us    96 rps
[  1] 0.00-10.02 sec  19.5 KBytes  16.0 Kbits/sec    100=10.758/10.326/11.346/0.302 ms    0   14K/192 us    92 rps
[  1] 0.00-10.02 sec BB8(f)-PDF: bin(w=100us):cnt(100)=104:7,105:18,106:19,107:9,108:3,109:4,110:10,111:16,112:7,113:4,114:3 (5.00/95.00/99.7%=104/113/114,Outliers=0,obl/obu=0/0)

forgot to mention, -z and --realtime is supported if it's supported on the device. This will set the traffic thread to SCHED_RR

-z, --realtime
Request real-time scheduler, if supported.

/* -------------------------------------------------------------------
 * Support for realtime scheduling of threads
 * ------------------------------------------------------------------- */
#if HAVE_SCHED_SETSCHEDULER
#include <sched.h>
#endif
#ifdef HAVE_MLOCKALL
#include <sys/mman.h>
#endif
void thread_setscheduler(struct thread_Settings *thread) {
#if HAVE_SCHED_SETSCHEDULER
    if (isRealtime(thread)) {
	struct sched_param sp;
	sp.sched_priority = sched_get_priority_max(SCHED_RR);
	// SCHED_OTHER, SCHED_FIFO, SCHED_RR
	if (sched_setscheduler(0, SCHED_RR, &sp) < 0)  {
	    perror("Client set scheduler");
#ifdef HAVE_MLOCKALL
	} else if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0) {
	    // lock the threads memory
	    perror ("mlockall");
#endif // MLOCK
	}
    }
#endif // SCHED
}

Here's one way delay (OWD) using UDP in each direction. Clock synced to 100 ns using ptp4l and GPS disciplined OCXO as the reference.

[root@rjm-nas rjmcmahon]# iperf -s -i 1 -e -u -B 192.168.1.87%enp2s0 --realtime
------------------------------------------------------------
Server listening on UDP port 5001 with pid 15024
Binding to local address 192.168.1.87 and iface enp2s0
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.87%enp2s0 port 5001 connected with 192.168.1.96 port 59750 (trip-times) (sock=3) (peer 2.1.8-rc) on 2022-08-10 12:59:19 (PDT)
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-1.00 sec   131 KBytes  1.07 Mbits/sec   0.023 ms 0/91 (0%) 0.168/0.067/0.303/0.041 ms 92 pps 91/0(0) pkts 794
[  1] 1.00-2.00 sec   128 KBytes  1.05 Mbits/sec   0.016 ms 0/89 (0%) 0.155/0.057/0.314/0.035 ms 89 pps 89/0(0) pkts 842
[  1] 2.00-3.00 sec   128 KBytes  1.05 Mbits/sec   0.014 ms 0/89 (0%) 0.130/0.042/0.163/0.019 ms 89 pps 89/0(0) pkts 1007
[  1] 3.00-4.00 sec   128 KBytes  1.05 Mbits/sec   0.008 ms 0/89 (0%) 0.156/0.049/0.291/0.030 ms 89 pps 89/0(0) pkts 837
[  1] 4.00-5.00 sec   128 KBytes  1.05 Mbits/sec   0.025 ms 0/89 (0%) 0.166/0.084/0.292/0.030 ms 89 pps 89/0(0) pkts 786
[  1] 5.00-6.00 sec   128 KBytes  1.05 Mbits/sec   0.022 ms 0/89 (0%) 0.168/0.088/0.297/0.027 ms 89 pps 89/0(0) pkts 777
[  1] 6.00-7.00 sec   129 KBytes  1.06 Mbits/sec   0.032 ms 0/90 (0%) 0.158/0.088/0.302/0.032 ms 89 pps 90/0(0) pkts 836
[  1] 7.00-8.00 sec   128 KBytes  1.05 Mbits/sec   0.016 ms 0/89 (0%) 0.170/0.078/0.304/0.031 ms 89 pps 89/0(0) pkts 768
[  1] 8.00-9.00 sec   128 KBytes  1.05 Mbits/sec   0.030 ms 0/89 (0%) 0.158/0.092/0.296/0.029 ms 89 pps 89/0(0) pkts 827
[  1] 9.00-10.00 sec   128 KBytes  1.05 Mbits/sec   0.022 ms 0/89 (0%) 0.146/0.063/0.212/0.030 ms 89 pps 89/0(0) pkts 898
[  1] 0.00-10.02 sec  1.25 MBytes  1.05 Mbits/sec   0.019 ms 0/895 (0%) 0.158/0.042/0.314/0.033 ms 89 pps 895/0(0) pkts 833


[root@ryzen3950 iperf2-code]# iperf -s -i 1 -e -B 192.168.1.96%enp4s0 -u --realtime
------------------------------------------------------------
Server listening on UDP port 5001 with pid 1600916
Binding to local address 192.168.1.96 and iface enp4s0
Read buffer size: 1.44 KByte (Dist bin width= 183 Byte)
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  1] local 192.168.1.96%enp4s0 port 5001 connected with 192.168.1.84 port 51101 (trip-times) (sock=3) (peer 2.1.8-rc) on 2022-08-10 13:27:53 (PDT)
[ ID] Interval        Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  Rx/inP  NetPwr
[  1] 0.00-1.00 sec   131 KBytes  1.07 Mbits/sec   0.181 ms 0/91 (0%) 0.482/0.059/0.851/0.229 ms 92 pps 91/0(0) pkts 278
[  1] 1.00-2.00 sec   128 KBytes  1.05 Mbits/sec   0.053 ms 0/89 (0%) 0.232/0.109/0.772/0.205 ms 89 pps 89/0(0) pkts 565
[  1] 2.00-3.00 sec   128 KBytes  1.05 Mbits/sec   0.005 ms 0/89 (0%) 0.152/0.103/0.645/0.126 ms 89 pps 89/0(0) pkts 864
[  1] 3.00-4.00 sec   128 KBytes  1.05 Mbits/sec   0.216 ms 0/89 (0%) 0.315/0.099/0.878/0.233 ms 89 pps 89/0(0) pkts 415
[  1] 4.00-5.00 sec   128 KBytes  1.05 Mbits/sec   0.112 ms 0/89 (0%) 0.434/0.097/0.806/0.239 ms 89 pps 89/0(0) pkts 302
[  1] 5.00-6.00 sec   128 KBytes  1.05 Mbits/sec   0.133 ms 0/89 (0%) 0.454/0.106/0.868/0.248 ms 89 pps 89/0(0) pkts 288
[  1] 6.00-7.00 sec   129 KBytes  1.06 Mbits/sec   0.245 ms 0/90 (0%) 0.478/0.098/0.919/0.227 ms 89 pps 90/0(0) pkts 277
[  1] 7.00-8.00 sec   128 KBytes  1.05 Mbits/sec   0.209 ms 0/89 (0%) 0.550/0.088/0.884/0.200 ms 89 pps 89/0(0) pkts 238
[  1] 8.00-9.00 sec   128 KBytes  1.05 Mbits/sec   0.202 ms 0/89 (0%) 0.481/0.100/0.869/0.225 ms 89 pps 89/0(0) pkts 272
[  1] 9.00-10.00 sec   128 KBytes  1.05 Mbits/sec   0.047 ms 0/89 (0%) 0.261/0.089/0.851/0.226 ms 89 pps 89/0(0) pkts 502
[  1] 0.00-10.02 sec  1.25 MBytes  1.05 Mbits/sec   0.042 ms 0/895 (0%) 0.383/0.059/0.919/0.251 ms 89 pps 895/0(0) pkts 343


04:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)