Some LAN<->WAN iperf3 tests, made with 2 intel gigabit computers, running last linux mint in usb pens.
Only had time to test some of the options, so in all the bellow tests, Routing/NAT Offloading options were:
Software flow offloading = ENABLE (Didn't tried Disable).
Hardware flow offloading = DISABLE (Didn't tried Enable).
I made different test with:
Packet Stearing Enable/Disable (Had to reboot after changing Enable or Disable to take effect).
Made big speed difference only when iperf3 --bidir switch was used.
Irqbalance Enable/Disable (I could not see any noticable difference with Enabled or Disabled).
Test #1:
Packet Stearing Enable; irqbalance Enable
See iperf3 --bidir option takes 100% of one core, but speed always ~930Mbits.
- - - - - - - - - - - - - - - - - - - - - - - - -
**iperf3 -c 192.168.0.110 -t 30**
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-30.00 sec 3.27 GBytes 935 Mbits/sec 0 sender
[ 5] 0.00-30.00 sec 3.26 GBytes 934 Mbits/sec receiver
- - - - - - - - - - - - - - - - - - - - - - - - -
0[|||||||||||||||||||||||||||||||| 58.9%] Tasks: 25, 9 thr, 78 kthr; 1 running
1[||| 4.9%] Load average: 0.31 0.10 0.03
2[| 0.7%] Uptime: 05:42:08
3[|| 1.3%]
Mem[||||||||||||||||||||||||||| 154M/408M]
Swp[ 0K/0K]
PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command
11309 root 20 0 2284 2008 1112 R 1.8 0.5 0:25.16 htop
10 root 20 0 0 0 0 S 0.9 0.0 3:53.90 ksoftirqd/0
11300 root 20 0 1064 732 660 S 0.9 0.2 0:00.75 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.
1 root 20 0 1664 1120 864 S 0.0 0.3 0:01.58 /sbin/procd
-----------------------------------------------------------------------------------------
- - - - - - - - - - - - - - - - - - - - - - - - -
**iperf3 -c 192.168.0.110 --bidir -t 30**
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-30.00 sec 3.25 GBytes 931 Mbits/sec 123 sender
[ 5][TX-C] 0.00-30.00 sec 3.25 GBytes 930 Mbits/sec receiver
[ 7][RX-C] 0.00-30.00 sec 3.25 GBytes 931 Mbits/sec 112 sender
[ 7][RX-C] 0.00-30.00 sec 3.25 GBytes 930 Mbits/sec receiver
- - - - - - - - - - - - - - - - - - - - - - - - -
0[||||||||||||||||||||||||||||||||||||||||||||||||99.2%] Tasks: 25, 9 thr, 78 kthr; 2 running
1[|||||||||| 18.5%] Load average: 0.01 0.04 0.00
2[ 0.0%] Uptime: 05:48:35
3[| 1.4%]
Mem[||||||||||||||||||||||||||| 155M/408M]
Swp[ 0K/0K]
PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command
10 root 20 0 0 0 0 S 81.8 0.0 4:24.54 ksoftirqd/0
16 root 20 0 0 0 0 S 1.8 0.0 0:01.05 ksoftirqd/1
11309 root 20 0 2284 2008 1112 R 1.8 0.5 0:30.01 htop
11300 root 20 0 1064 732 660 S 0.9 0.2 0:00.90 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.
1 root 20 0 1664 1120 864 S 0.0 0.3 0:01.58 /sbin/procd
-----------------------------------------------------------------------------------------
**cat /proc/interrupts**
CPU0 CPU1 CPU2 CPU3
11: 390582 199648 366107 185775 GIC-0 20 Level arch_timer
13: 0 0 0 0 GIC-0 39 Level arch_mem_timer
16: 2 0 0 0 GIC-0 354 Edge smp2p
17: 0 0 0 0 GIC-0 216 Level 4a9000.thermal-sensor
18: 0 0 0 0 GIC-0 239 Level bam_dma
21: 0 0 0 0 GIC-0 270 Level bam_dma
22: 6 0 0 0 GIC-0 340 Level msm_serial0
23: 45752 0 0 0 GIC-0 178 Level bam_dma
24: 0 0 0 0 GIC-0 35 Edge wdt_bark
25: 0 0 0 0 GIC-0 357 Edge q6v5 wdog
29: 5 0 0 0 GIC-0 348 Edge ce0
30: 65855 0 0 0 GIC-0 347 Edge ce1
31: 17163 0 0 0 GIC-0 346 Edge ce2
32: 1047 0 0 0 GIC-0 343 Edge ce3
34: 1 0 0 0 GIC-0 443 Edge ce5
36: 1204 0 0 0 GIC-0 72 Edge ce7
38: 0 0 0 0 GIC-0 334 Edge ce9
39: 1 0 0 0 GIC-0 333 Edge ce10
40: 0 0 0 0 GIC-0 69 Edge ce11
47: 0 0 0 0 GIC-0 323 Edge reo2ost-exception
48: 157 0 0 0 GIC-0 322 Edge wbm2host-rx-release
49: 28 0 0 0 GIC-0 321 Edge reo2host-status
50: 1717 0 0 0 GIC-0 320 Edge reo2host-destination-ring4
51: 13414 0 0 0 GIC-0 271 Edge reo2host-destination-ring3
52: 14596 0 0 0 GIC-0 268 Edge reo2host-destination-ring2
53: 908 0 0 0 GIC-0 267 Edge reo2host-destination-ring1
57: 1350 0 0 0 GIC-0 263 Edge ppdu-end-interrupts-mac3
58: 0 0 0 0 GIC-0 262 Edge ppdu-end-interrupts-mac2
59: 44944 0 0 0 GIC-0 261 Edge ppdu-end-interrupts-mac1
60: 1 0 0 0 GIC-0 260 Edge rxdma2host-monitor-status-ring-mac3
61: 0 0 0 0 GIC-0 256 Edge rxdma2host-monitor-status-ring-mac2
62: 1 0 0 0 GIC-0 255 Edge rxdma2host-monitor-status-ring-mac1
63: 3 0 0 0 GIC-0 235 Edge host2rxdma-host-buf-ring-mac3
64: 0 0 0 0 GIC-0 215 Edge host2rxdma-host-buf-ring-mac2
65: 1 0 0 0 GIC-0 212 Edge host2rxdma-host-buf-ring-mac1
66: 0 0 0 0 GIC-0 211 Edge rxdma2host-destination-ring-mac3
67: 0 0 0 0 GIC-0 210 Edge rxdma2host-destination-ring-mac2
68: 0 0 0 0 GIC-0 209 Edge rxdma2host-destination-ring-mac1
73: 10138 0 0 0 GIC-0 191 Edge wbm2host-tx-completions-ring3
74: 5377 0 0 0 GIC-0 190 Edge wbm2host-tx-completions-ring2
75: 15792 0 0 0 GIC-0 189 Edge wbm2host-tx-completions-ring1
77: 11836166 0 0 0 GIC-0 377 Level edma_txcmpl
78: 0 0 0 0 GIC-0 385 Level edma_rxfill
79: 17602596 0 0 0 GIC-0 393 Level edma_rxdesc
80: 0 0 0 0 GIC-0 376 Level edma_misc
81: 0 0 0 0 MSI 0 Edge PCIe PME, aerdrv
82: 0 0 0 0 smp2p 0 Edge q6v5 fatal
83: 1 0 0 0 smp2p 1 Edge q6v5 ready
84: 0 0 0 0 smp2p 2 Edge q6v5 handover
85: 0 0 0 0 smp2p 3 Edge q6v5 stop
86: 0 0 0 0 msmgpio 34 Edge keys
87: 31 0 0 0 MSI 524288 Edge ath10k_pci
88: 63 0 0 0 GIC-0 353 Edge glink-native
IPI0: 98593 159529 149809 144011 Rescheduling interrupts
IPI1: 5707 7876083 2946458 3798672 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 0 0 Timer broadcast interrupts
IPI5: 4849 176 243 218 IRQ work interrupts
IPI6: 0 0 0 0 CPU wake-up interrupts
Err: 0
Test #2:
Packet Stearing Enable; irqbalance Disable
See iperf3 --bidir option takes 100% of one core, but speed always ~930Mbits.
I couldn't see difference from test#1.
EDIT: Looking again at --bidir tests, here ksoftirqd/0 uses 107% core0 and in test#1 it only shows ~80%.
- - - - - - - - - - - - - - - - - - - - - - - - -
root@X2:~# service irqbalance status
inactive
**iperf3 -c 192.168.0.110 --bidir -t 30**
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-30.00 sec 3.25 GBytes 931 Mbits/sec 0 sender
[ 5][TX-C] 0.00-30.00 sec 3.25 GBytes 930 Mbits/sec receiver
[ 7][RX-C] 0.00-30.00 sec 3.26 GBytes 932 Mbits/sec 0 sender
[ 7][RX-C] 0.00-30.00 sec 3.25 GBytes 931 Mbits/sec receiver
- - - - - - - - - - - - - - - - - - - - - - - - -
0[|||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 24, 8 thr, 78 kthr; 2 running
1[|| 1.9%] Load average: 0.67 0.25 0.09
2[||||||| 13.1%] Uptime: 00:14:28
3[| 0.7%]
Mem[|||||||||||||||||||||| 126M/408M]
Swp[ 0K/0K]
PID USER PRI NI VIRT RES SHR S CPU%-MEM% TIME+ Command
10 root 20 0 0 0 0 R 107. 0.0 1:56.26 ksoftirqd/0
66 root 20 0 0 0 0 I 0.7 0.0 0:00.05 kworker/u8:2-events_power_efficient
3475 root 20 0 1064 740 668 S 0.7 0.2 0:00.37 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 192.
3535 root 20 0 2068 1768 1080 R 0.7 0.4 0:04.09 htop
-------------------------------------------------------------------------------------------------------------
root@X2:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
11: 32558 35175 45261 21368 GIC-0 20 Level arch_timer
13: 0 0 0 0 GIC-0 39 Level arch_mem_timer
16: 2 0 0 0 GIC-0 354 Edge smp2p
17: 0 0 0 0 GIC-0 216 Level 4a9000.thermal-sensor
18: 0 0 0 0 GIC-0 239 Level bam_dma
21: 0 0 0 0 GIC-0 270 Level bam_dma
22: 4 0 0 0 GIC-0 340 Level msm_serial0
23: 42231 0 0 0 GIC-0 178 Level bam_dma
24: 0 0 0 0 GIC-0 35 Edge wdt_bark
25: 0 0 0 0 GIC-0 357 Edge q6v5 wdog
29: 5 0 0 0 GIC-0 348 Edge ce0
30: 6 0 0 0 GIC-0 347 Edge ce1
31: 10 0 0 0 GIC-0 346 Edge ce2
32: 1 0 0 0 GIC-0 343 Edge ce3
34: 1 0 0 0 GIC-0 443 Edge ce5
36: 0 0 0 0 GIC-0 72 Edge ce7
38: 0 0 0 0 GIC-0 334 Edge ce9
39: 1 0 0 0 GIC-0 333 Edge ce10
40: 0 0 0 0 GIC-0 69 Edge ce11
47: 0 0 0 0 GIC-0 323 Edge reo2ost-exception
48: 0 0 0 0 GIC-0 322 Edge wbm2host-rx-release
49: 0 0 0 0 GIC-0 321 Edge reo2host-status
50: 0 0 0 0 GIC-0 320 Edge reo2host-destination-ring4
51: 0 0 0 0 GIC-0 271 Edge reo2host-destination-ring3
52: 0 0 0 0 GIC-0 268 Edge reo2host-destination-ring2
53: 0 0 0 0 GIC-0 267 Edge reo2host-destination-ring1
57: 1 0 0 0 GIC-0 263 Edge ppdu-end-interrupts-mac3
58: 0 0 0 0 GIC-0 262 Edge ppdu-end-interrupts-mac2
59: 1 0 0 0 GIC-0 261 Edge ppdu-end-interrupts-mac1
60: 1 0 0 0 GIC-0 260 Edge rxdma2host-monitor-status-ring-mac3
61: 0 0 0 0 GIC-0 256 Edge rxdma2host-monitor-status-ring-mac2
62: 1 0 0 0 GIC-0 255 Edge rxdma2host-monitor-status-ring-mac1
63: 3 0 0 0 GIC-0 235 Edge host2rxdma-host-buf-ring-mac3
64: 0 0 0 0 GIC-0 215 Edge host2rxdma-host-buf-ring-mac2
65: 1 0 0 0 GIC-0 212 Edge host2rxdma-host-buf-ring-mac1
66: 0 0 0 0 GIC-0 211 Edge rxdma2host-destination-ring-mac3
67: 0 0 0 0 GIC-0 210 Edge rxdma2host-destination-ring-mac2
68: 0 0 0 0 GIC-0 209 Edge rxdma2host-destination-ring-mac1
73: 0 0 0 0 GIC-0 191 Edge wbm2host-tx-completions-ring3
74: 0 0 0 0 GIC-0 190 Edge wbm2host-tx-completions-ring2
75: 0 0 0 0 GIC-0 189 Edge wbm2host-tx-completions-ring1
77: 8544235 0 0 0 GIC-0 377 Level edma_txcmpl
78: 0 0 0 0 GIC-0 385 Level edma_rxfill
79: 15310377 0 0 0 GIC-0 393 Level edma_rxdesc
80: 0 0 0 0 GIC-0 376 Level edma_misc
81: 0 0 0 0 MSI 0 Edge PCIe PME, aerdrv
82: 0 0 0 0 smp2p 0 Edge q6v5 fatal
83: 1 0 0 0 smp2p 1 Edge q6v5 ready
84: 0 0 0 0 smp2p 2 Edge q6v5 handover
85: 0 0 0 0 smp2p 3 Edge q6v5 stop
86: 0 0 0 0 msmgpio 34 Edge keys
87: 31 0 0 0 MSI 524288 Edge ath10k_pci
88: 63 0 0 0 GIC-0 353 Edge glink-native
IPI0: 11404 15697 17646 18826 Rescheduling interrupts
IPI1: 746 3840260 6381339 1314527 Function call interrupts
IPI2: 0 0 0 0 CPU stop interrupts
IPI3: 0 0 0 0 CPU stop (for crash dump) interrupts
IPI4: 0 0 0 0 Timer broadcast interrupts
IPI5: 221 59 68 72 IRQ work interrupts
IPI6: 0 0 0 0 CPU wake-up interrupts
Err: 0
Test#3:
Packet Stearing Disable; irqbalance Disable
iperf3 --bidir option takes 100% of one core,
but now with Packet Stearing off, speed is not ~930Mbits.
I rebooted after disabling Packet Stearing and tested this 3 times, always same speeds.
mint@mint:~$ iperf3 -c 192.168.0.110 --bidir -t 60
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval Transfer Bitrate Retr
[ 5][TX-C] 0.00-60.00 sec 6.46 GBytes 925 Mbits/sec 118 sender
[ 5][TX-C] 0.00-60.00 sec 6.46 GBytes 924 Mbits/sec receiver
[ 7][RX-C] 0.00-60.00 sec 3.11 GBytes 445 Mbits/sec 762 sender
[ 7][RX-C] 0.00-60.00 sec 3.10 GBytes 444 Mbits/sec receiver
iperf Done.
Regarding the retries, I tried with several cables and kept a CAT6 2 meters cable.
In bidir test sometimes there are retries other times there are none, with same cable.
Note: Didn't test WIFI<->WAN because I don't have a fast card, I'm waiting for ax210 do be delivered.