A few SQM benchmarks

[gigabit314]

In part because of the thread SQM QoS performance on MT7621, I've made some informal benchmarks with several devices I have on hand. In each case, a laptop acted as an iperf3 client connected to each router's LAN port, and another machine on the WAN network acted as the iperf3 server. For the two routers with a single Ethernet port, the WAN interface was a USB 3.0 gigabit Ethernet adapter (AX88179-based).

In these tests, when SQM was enabled, it was on the WAN interface with 800Mbps/800Mbps limits, except the first WD test which was set to 500Mbps/500Mbps. There was no tuning done---each one was tested with the default settings, other than getting the eth0/LAN and eth1/WAN interfaces configured and the luci SQM app installed.

OpenWrt 19.07.3 was used in each case, except for the Atomic Pi, for which I used a snapshot from August 10. The iperf3 runs were done without --reverse, and then done with --reverse, so traffic was essentially only flowing one way at a given time.

SQM enabled, Software flow offloading disabled:
Device                   Mbps not reverse     Mbps reverse
WD My Net N750           137                  235
Linksys EA8500           506                  486
Raspberry Pi 4 Model B   644                  648
Atomic Pi                765                  589

SQM disabled, Software flow offloading disabled:
Device                   Mbps not reverse     Mbps reverse
WD My Net N750           319                  359
Linksys EA8500           693                  844
Raspberry Pi 4 Model B   665                  699
Atomic Pi                922                  669

SQM disabled, Software flow offloading enabled:
Device                   Mbps not reverse     Mbps reverse
WD My Net N750           719                  835
Linksys EA8500           867                  925
Raspberry Pi 4 Model B   725                  940
Atomic Pi                922                  899

I'd expected the Raspberry Pi or the Atomic Pi to be able to handle SQM at full line rate, but in my test, neither could quite do that. I also ran the same 4 routers through testing without SQM enabled, with and without software flow offloading.

[Edited August 11: add SQM-disabled results and clarification that iperf3 runs were single-direction at a time.]

[geminis3]

I think you'll need X86 hardware to do SQM at gigabit rates

[moeller0]

Great project, thanks for doing this and sharing the results.

Mmmh, did you test bidirectional traffic or did you test the two directions sequentially?
Could you also split out the results per direction?
And could you also add numbers with SQM disabled just to confirm the baseline, please?

For the multi core machines it can become essential to check how interrupt/network processing is distributed across the cores. There are some peculiar configurations where all/most processing happens on a single core, while the others idle.
And also worth noting, SQM/traffic shaping does not play to well with frequency/voltage scaling so for testing it might be helpful to disable all of that otherwise desirable features.

[ejl]

Regarding Raspi 4 check this forum topic. Roughly from the topic I remember, that USB-ethernet adapter made a significant change pushing RPi to gigabit rates, but the tests done in the topic did not measure with SQM, just "raw routing power".

RPi4 routing performance numbers

I got myself an RPi 4 since I've been recommending it to people, and had the following experience routing iperf3 via ipv6 in a test network: [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 50.0 MBytes 419 Mbits/sec [ 5] 1.00-2.00 sec 60.6 MBytes 508 Mbits/sec [ 5] 2.00-3.00 sec 60.4 MBytes 506 Mbits/sec [ 5] 3.00-4.00 sec 60.6 MBytes 509 Mbits/sec [ 5] 4.00-5.00 sec 61.…

[dlakelan]

Yes, my tests showed those Alix USB Ethernet devices required far more cpu power than the realtek based UE300 from tplink. the RPi using the UE300 did gigabit routing AND shaping with HFSC based shaping scheme. I would not expect cake to be dramatically different.

the ue300 is cheap if you are looking for speed try switching.

[fantom-x]

@gigabit314

Raspberry Pi 4 Model B 644 648

Did you make sure that each eth port is running on its own CPU core?

[gigabit314]

The Atomic Pi is actually a 64-bit x86 machine, but only a 4-core Atom.

[gigabit314]

I did not do any tuning to associate network interfaces with particular CPU cores. The setup was essentially out-of-the-box on each unit.

[gigabit314]

@dlakelan

Yes, my tests showed those Alix USB Ethernet devices required far more cpu power than the realtek based UE300 from tplink. the RPi using the UE300 did gigabit routing AND shaping with HFSC based shaping scheme. I would not expect cake to be dramatically different.

The AX88179 adapter happens to be what I have on hand. I actually have an RTL8153-based USB 3.0 gigabit adapter, too, but unfortunately, it's USB Type C.

[gigabit314]

I still have all 4 units available for running other kinds of tests. For example, I can try pinning IRQ handlers to particular CPU cores, or try other CPU frequency scaling governors. Bidirectional traffic and also many parallel streams (8? 64?) in iperf3 sound interesting.

There were a few cases where results were not as reproducible as I would like. The My Net N750 with software flow off-loading enabled scored 719 Mbps the first time, but only 599 Mbps the second time.

[moeller0]

@gigabit314

also many parallel streams (8? 64?)

Mmh, using a few streams can even out/hedge against individual TCP streams saturating the link, which on long fat links actually can happen. But beyond say 16-32 you will see finishing returns from doing that, and you will start testing other features like stochastic flow hashing ... So for pure throughput tests on gigabit links, I would guess <=32 should be sufficient, but I have no real hard data to back up this number.

[bjonglez]

Interesting results.

However, there seems to be a mistake in the results of the first post? The first two tests have identical description (SQM disabled, software flow offloading disabled).

Since most of your models have multiple CPU cores, you should definitely try multiple parallel connections/streams (8 seems like a good tradeoff). OpenWrt tries to balance traffic on several cores, both for RX (RPS, Receive Packet Steering) and TX (XPS, Transmit Packet Steering). The script that configures that is here.

[gigabit314]

@bjonglez

Interesting results.

However, there seems to be a mistake in the results of the first post? The first two tests have identical description (SQM disabled, software flow offloading disabled).

Thanks for the reply. I'm dismayed that in my last edit to add the SQM-disabled results, I made an error just as you spotted. It seems I can no longer edit my post, either. The first set of results were SQM-enabled, software flow-offloading disabled.

Hopefully, this weekend I can run another set of tests. I'll go with 8 streams.

[bjonglez]

Ah, it makes sense. I've just fixed your original post.

[JonP]

@gigabit314

The AX88179 adapter happens to be what I have on hand. I actually have an RTL8153-based USB 3.0 gigabit adapter, too, but unfortunately, it's USB Type C.

I for one, would like to see your RPi4 results using a UE300, as well as the group with different traffic patterns.

[gigabit314]

I've done a bit more work on automating benchmark runs. The script I'm trying is shown below. It creates a new run directory, reboots the router, copies config files from the router into the run directory to record the conditions, and runs normal, reverse, and bidirectional iperf3 tests, 60s each, with 1, 2, 4, 8, 16, and 32 parallel streams. I have a separate python script to summarize the conditions and results into a table.

This shell script is run on a Linux laptop on the LAN network of the router under test. The iperf3 server is another Linux machine on the WAN network.

To prepare the router, I configure it in LuCI. For setting up something like smp_affinity for IRQs, I add a few lines to the rc.local and let the router reboot in the script take care of running them. This avoids interactively setting something like that and not having it recorded in the run directory.

set -x

ROUTER_IP=192.168.1.1
IPERF3_SERVER_IP=192.168.6.126
REBOOT=yes

OUTPUT_DIR="runs/$(date +%s)"

# from https://github.com/maximmenshikov/iperf (fixes JSON for bidirectional runs)
IPERF3=~/bin/iperf3-patched

if [ $REBOOT == "yes" ] ; then
  ssh ${ROUTER_IP} "reboot"

  while ping -c1 -q ${ROUTER_IP} > /dev/null 2>&1
  do
    echo waiting for ${ROUTER_IP} to reboot....
    sleep 1
  done

  until ping -c1 -q ${IPERF3_SERVER_IP} > /dev/null 2>&1
  do
    echo waiting for ${IPERF3_SERVER_IP}....
    sleep 1
  done
fi

mkdir -p "${OUTPUT_DIR}/results"
scp -pqr ${ROUTER_IP}:/etc/config "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/board.json "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/openwrt_release "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/rc.local "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/sqm/sqm.conf "${OUTPUT_DIR}"

ssh ${ROUTER_IP} "cat /proc/cpuinfo" > "${OUTPUT_DIR}/cpuinfo"
ssh ${ROUTER_IP} "cat /proc/interrupts" > "${OUTPUT_DIR}/interrupts-before"
ssh ${ROUTER_IP} "[ -f /sys/class/thermal/thermal_zone0/temp ] && cat /sys/class/thermal/thermal_zone0/temp" \
  > "${OUTPUT_DIR}/thermal_zone0-temp-before"

RUN_TIME=60

for PARALLEL_STREAMS in 1 2 4 8 16 32 ; do
  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-client-sends.json"

  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --reverse \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-server-sends.json"

  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --bidir \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-bidir.json"
done

ssh ${ROUTER_IP} "cat /proc/interrupts" > "${OUTPUT_DIR}/interrupts-after"
ssh ${ROUTER_IP} "[ -f /sys/class/thermal/thermal_zone0/temp ] && cat /sys/class/thermal/thermal_zone0/temp" \
  > "${OUTPUT_DIR}/thermal_zone0-temp-after"

[gigabit314]

Here is a summary table of my results. Pasting it into another tool where it can be interactively filtered to just certain conditions is nice. All rates are Mbps calculated as iperf3's reported bits per second / 1e6. There is a repeated run for the WD router.

I'm not totally sure how to interpret the bidirectional iperf3 results, as there are 4 bits_per_second values shown in the end sums. I'm showing sum_sent for the sender: true as Sending, sum_received for the sender: false as Receiving, and the sum of these two as Total.

There are a lot of observations to be made, but one that jumps out for me is that the SQM-enabled bidirectional receiving rate for both ARM-based routers was quite slow.

The Pi 4, with SQM disabled and software flow offloading, was the only case able to hit the gigabit limit in the reverse direction. However, the Atomic Pi was able to record the highest bidirectional total in the testing so far.

Router,SQM,Soft Flow Offloading,SMP Affinity,Direction,Streams,Sending,Receiving,Total
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,150.2,149.9,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,141.2,140.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,137.7,137.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,136.2,135.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,134.8,134.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,133.6,132.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,220.8,220.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,190.5,190.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,176.2,176.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,159.4,159.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,144.2,144.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,130.0,129.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,112.5,50.1,162.5
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,110.7,45.1,155.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,104.0,45.0,149.1
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,101.3,45.5,146.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,107.0,35.9,142.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,112.4,25.5,137.9
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,591.0,590.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,631.6,631.3,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,717.6,717.2,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,718.3,717.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,690.1,689.3,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,632.8,631.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,732.8,732.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,815.8,815.1,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,790.6,789.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,779.6,778.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,727.9,726.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,645.1,644.0,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,308.9,422.2,731.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,291.2,457.7,749.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,291.5,445.6,737.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,329.3,387.9,717.2
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,275.1,380.9,656.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,258.1,387.9,646.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,266.4,266.0,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,264.2,263.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,264.9,264.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,258.6,257.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,255.0,253.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,251.0,249.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,330.2,329.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,329.9,329.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,323.1,322.5,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,307.4,306.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,286.0,284.6,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,277.1,275.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,119.9,184.3,304.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,106.2,196.6,302.8
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,89.4,208.1,297.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,99.0,184.4,283.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,105.0,170.5,275.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,106.1,164.6,270.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,149.1,148.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,139.8,139.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,135.9,135.5,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,133.9,133.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,133.2,132.5,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,131.9,131.1,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,229.2,229.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,196.3,196.1,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,186.1,185.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,161.4,161.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,145.7,145.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,131.6,131.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,108.9,53.8,162.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,109.8,42.9,152.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,105.1,42.0,147.2
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,99.7,46.6,146.3
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,105.8,35.5,141.3
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,115.0,19.9,134.9
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,529.4,529.2,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,524.4,523.8,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,575.3,574.3,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,502.0,499.3,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,439.4,434.9,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,387.8,379.9,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,484.0,483.7,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,448.8,448.0,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,424.4,423.7,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,402.8,402.1,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,333.2,332.2,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,327.5,325.6,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,517.7,7.6,525.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,552.5,5.8,558.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,544.6,4.6,549.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,497.3,3.7,501.0
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,416.3,4.0,420.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,368.4,3.6,372.0
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,863.3,862.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,870.8,870.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,865.6,864.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,846.7,843.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,814.2,810.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,771.8,768.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,926.5,926.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,921.0,920.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,895.8,894.3,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,913.2,910.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,856.7,852.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,717.8,713.4,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,661.3,249.1,910.4
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,614.1,286.2,900.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,642.5,231.8,874.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,513.7,324.6,838.2
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,406.8,354.3,761.1
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,397.1,338.0,735.1
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,656.1,655.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,647.0,646.2,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,624.1,622.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,583.9,580.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,475.5,471.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,436.8,432.7,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,832.7,832.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,811.6,811.2,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,761.8,760.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,695.9,692.9,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,541.0,536.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,453.4,449.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,346.9,343.4,690.4
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,319.7,349.6,669.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,238.9,392.9,631.8
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,181.7,351.3,533.0
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,194.3,260.6,454.9
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,208.5,234.5,443.0
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,648.9,648.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,653.1,652.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,651.5,650.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,608.3,605.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,548.4,542.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,508.3,499.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,652.1,651.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,688.0,687.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,584.7,584.0,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,503.4,502.8,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,474.1,473.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,461.1,460.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,485.2,168.8,654.0
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,543.0,95.3,638.3
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,481.2,126.4,607.6
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,427.1,123.3,550.4
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,435.1,50.6,485.7
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,477.6,8.9,486.5
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,752.8,752.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,752.3,751.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,748.3,747.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,734.4,733.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,724.2,722.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,674.3,673.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,941.6,941.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,896.8,896.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,867.9,867.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,896.0,894.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,804.5,802.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,761.0,758.8,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,623.6,160.1,783.7
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,341.7,515.4,857.1
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,408.9,396.3,805.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,391.9,385.4,777.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,389.2,318.6,707.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,413.5,262.1,675.6
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,681.8,681.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,678.2,677.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,676.2,675.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,658.4,657.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,638.9,637.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,511.7,510.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,826.1,825.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,709.1,708.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,745.6,744.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,681.7,680.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,627.2,625.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,596.6,594.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,527.4,172.4,699.8
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,313.7,417.4,731.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,398.1,268.0,666.1
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,374.1,233.8,607.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,317.6,239.8,557.4
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,326.0,208.5,534.5
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,1,763.5,763.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,2,760.8,760.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,4,767.3,766.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,8,760.4,759.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,16,746.1,744.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,32,716.3,714.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,1,941.4,941.0,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,2,942.1,941.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,4,942.0,940.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,8,942.7,941.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,16,942.8,941.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,32,942.4,940.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,1,338.8,671.2,1009.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,2,425.2,471.6,896.7
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,4,422.9,481.4,904.3
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,8,435.3,416.9,852.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,16,418.5,361.1,779.6
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,32,448.7,280.7,729.5
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,768.0,767.5,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,765.6,765.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,765.6,764.7,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,765.7,764.8,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,747.5,746.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,581.1,580.0,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,585.8,585.3,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,554.2,553.7,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,533.7,533.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,512.5,511.9,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,498.0,497.2,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,489.1,488.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,698.3,221.1,919.5
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,716.1,179.9,896.0
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,668.7,195.6,864.3
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,576.2,228.8,805.0
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,402.9,274.4,677.3
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,388.4,176.8,565.1
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,921.1,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,921.4,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,922.0,920.8,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,922.0,920.8,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,921.8,920.7,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,921.6,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,897.7,897.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,898.8,898.0,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,899.2,898.0,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,899.6,898.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,900.8,899.1,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,902.7,900.2,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,457.0,814.3,1271.3
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,625.3,704.5,1329.9
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,735.4,621.9,1357.4
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,852.6,490.8,1343.4
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,886.6,379.3,1265.9
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,856.6,302.6,1159.2
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,921.0,920.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,921.2,920.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,921.2,919.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,921.1,919.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,920.9,919.7,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,917.2,915.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,663.3,662.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,661.4,660.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,658.4,656.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,649.4,647.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,630.5,628.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,610.3,607.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,319.0,501.0,820.0
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,394.4,454.3,848.7
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,591.4,330.2,921.6
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,781.7,189.5,971.2
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,741.2,183.2,924.4
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,601.6,148.5,750.1

[gigabit314]

Someone asked about setting smp_affinity for the ASIX USB 3.0 gigabit Ethernet adapter. I got an error when trying to assign the USB xhci_hcd smp_affinity, so I settled for this in /etc/rc.local. Perhaps I should try moving these interrupts to CPU1 and CPU2 instead, so that they both avoid CPU0 where the xhci_hcd interrupt ends up.

#CPU0 - eth0 189
echo "1" >/proc/irq/31/smp_affinity
#CPU1 - eth0 190
echo "2" >/proc/irq/32/smp_affinity

Here is how /proc/interrupts looked at the end of the Pi 4 run. It does seem to help, on the order of 10%.

           CPU0       CPU1       CPU2       CPU3       
  3:     193942       3388      10035       3321     GICv2  30 Level     arch_timer
 11:        651          0          0          0     GICv2  65 Level     fe00b880.mailbox
 14:          2          0          0          0     GICv2 153 Level     uart-pl011
 17:         45          0          0          0     GICv2 114 Level     DMA IRQ
 24:          1          0          0          0     GICv2  66 Level     VCHIQ doorbell
 25:       7867          0          0          0     GICv2 158 Level     mmc1, mmc0
 31:    5300912          0          0          0     GICv2 189 Level     eth0
 32:          2   13432486          0          0     GICv2 190 Level     eth0
 38:          0          0          0          0     GICv2 175 Level     PCIe PME, aerdrv
 39:   21892839          0          0          0  BRCM STB PCIe MSI 524288 Edge      xhci_hcd
IPI0:      4198     121454     450517       1689       Rescheduling interrupts
IPI1:        35        408        358        297       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:     21932      22447      56546        223       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

For comparison, here is /proc/interrupts at the end of one of the Atomic Pi runs.

            CPU0       CPU1       CPU2       CPU3       
   0:         22          0          0          0   IO-APIC    2-edge      timer
   4:          0          0         12          0   IO-APIC    4-edge      ttyS0
   8:          0          0          0          0   IO-APIC    8-fasteoi   rtc0
   9:          0          0          0          0   IO-APIC    9-fasteoi   acpi
 116:          0          0          0        459   PCI-MSI 32768-edge      i915
 118:   23674708          0          0          0   PCI-MSI 327680-edge      xhci_hcd
 119:          0          0          0   23935736   PCI-MSI 524288-edge      eth0
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:     277526     279505     278958     277853   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 IWI:          0          0          0          0   IRQ work interrupts
 RTR:          0          0          0          0   APIC ICR read retries
 RES:        401       1021       2140       2425   Rescheduling interrupts
 CAL:        608        321        987        875   Function call interrupts
 TLB:          0          0          0          0   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 DFR:          0          0          0          0   Deferred Error APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:          4          4          4          4   Machine check polls
 ERR:          0
 MIS:          0
 PIN:          0          0          0          0   Posted-interrupt notification event
 NPI:          0          0          0          0   Nested posted-interrupt event
 PIW:          0          0          0          0   Posted-interrupt wakeup event

[gigabit314]

@JonP

@gigabit314

The AX88179 adapter happens to be what I have on hand. I actually have an RTL8153-based USB 3.0 gigabit adapter, too, but unfortunately, it's USB Type C.

I for one, would like to see your RPi4 results using a UE300, as well as the group with different traffic patterns.

I wish I had one, and at $13 it is tempting to buy one just for testing it, but I don't need it for anything else. Maybe another forum member has one, and they could run a test.

[gigabit314]

In case it's useful, here is the python script for summarizing all the results.

import json
import os


class BenchmarkResult:
    def __init__(self, path, model=None):
        self.path = path

        with open(os.path.join(self.path, "board.json")) as f:
            s = f.read()
        self.model = json.loads(s).get("model", {}).get("name", "unknown model")

        with open(os.path.join(self.path, "config", "sqm")) as f:
            s = f.read()
        self.sqm = "option enabled '1'" in s

        with open(os.path.join(self.path, "config", "firewall")) as f:
            s = f.read()
        self.flow_offloading = "option flow_offloading '1'" in s

        with open(os.path.join(self.path, "rc.local")) as f:
            s = f.read()
        self.smp_affinity = "smp_affinity" in s

        self.results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "client-sends" in fn.name
        ]
        self.reverse_results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "server-sends" in fn.name
        ]
        self.bidir_results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "bidir" in fn.name
        ]

    def _read_result(self, fn):
        with open(fn) as f:
            s = f.read()
        d = json.loads(s)
        return d

    def _result_lines(self, dir):
        data = {
            "Normal Direction": self.results,
            "Reverse": self.reverse_results,
            "Bidirectional": self.bidir_results,
        }
        return tuple(
            ",".join(
                (
                    self.model,
                    "SQM ON" if self.sqm else "SQM OFF",
                    "Soft Flow Offloading ON"
                    if self.flow_offloading
                    else "Soft Flow Offloading OFF",
                    "SMP Affinity SET" if self.smp_affinity else "SMP Affinity NOT SET",
                    dir,
                    f'{result["start"]["test_start"]["num_streams"]}',
                    f'{result["end"]["sums"][0]["sum_sent"]["bits_per_second"] / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else f'{result["end"]["sum_sent"]["bits_per_second"] / 1e6:.1f}',
                    f'{result["end"]["sums"][1]["sum_received"]["bits_per_second"] / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else f'{result["end"]["sum_received"]["bits_per_second"] / 1e6:.1f}',
                    f'{(result["end"]["sums"][0]["sum_sent"]["bits_per_second"] + result["end"]["sums"][1]["sum_received"]["bits_per_second"]) / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else "NA",
                )
            )
            for result in data[dir]
        )

    def csv(self):
        return "\n".join(
            sum(
                (
                    self._result_lines(dir)
                    for dir in ("Normal Direction", "Reverse", "Bidirectional")
                ),
                start=tuple(),
            )
        )


def main():
    results = [BenchmarkResult(path=e.path) for e in os.scandir("runs")]
    for result in results:
        print(result.csv())


if __name__ == "__main__":
    main()