A few SQM benchmarks

Mmh, using a few streams can even out/hedge against individual TCP streams saturating the link, which on long fat links actually can happen. But beyond say 16-32 you will see finishing returns from doing that, and you will start testing other features like stochastic flow hashing ... So for pure throughput tests on gigabit links, I would guess <=32 should be sufficient, but I have no real hard data to back up this number.

Interesting results.

However, there seems to be a mistake in the results of the first post? The first two tests have identical description (SQM disabled, software flow offloading disabled).

Since most of your models have multiple CPU cores, you should definitely try multiple parallel connections/streams (8 seems like a good tradeoff). OpenWrt tries to balance traffic on several cores, both for RX (RPS, Receive Packet Steering) and TX (XPS, Transmit Packet Steering). The script that configures that is here.

Thanks for the reply. I'm dismayed that in my last edit to add the SQM-disabled results, I made an error just as you spotted. It seems I can no longer edit my post, either. The first set of results were SQM-enabled, software flow-offloading disabled.

Hopefully, this weekend I can run another set of tests. I'll go with 8 streams.

Ah, it makes sense. I've just fixed your original post.

I for one, would like to see your RPi4 results using a UE300, as well as the group with different traffic patterns.

I've done a bit more work on automating benchmark runs. The script I'm trying is shown below. It creates a new run directory, reboots the router, copies config files from the router into the run directory to record the conditions, and runs normal, reverse, and bidirectional iperf3 tests, 60s each, with 1, 2, 4, 8, 16, and 32 parallel streams. I have a separate python script to summarize the conditions and results into a table.

This shell script is run on a Linux laptop on the LAN network of the router under test. The iperf3 server is another Linux machine on the WAN network.

To prepare the router, I configure it in LuCI. For setting up something like smp_affinity for IRQs, I add a few lines to the rc.local and let the router reboot in the script take care of running them. This avoids interactively setting something like that and not having it recorded in the run directory.

set -x

ROUTER_IP=192.168.1.1
IPERF3_SERVER_IP=192.168.6.126
REBOOT=yes

OUTPUT_DIR="runs/$(date +%s)"

# from https://github.com/maximmenshikov/iperf (fixes JSON for bidirectional runs)
IPERF3=~/bin/iperf3-patched

if [ $REBOOT == "yes" ] ; then
  ssh ${ROUTER_IP} "reboot"

  while ping -c1 -q ${ROUTER_IP} > /dev/null 2>&1
  do
    echo waiting for ${ROUTER_IP} to reboot....
    sleep 1
  done

  until ping -c1 -q ${IPERF3_SERVER_IP} > /dev/null 2>&1
  do
    echo waiting for ${IPERF3_SERVER_IP}....
    sleep 1
  done
fi

mkdir -p "${OUTPUT_DIR}/results"
scp -pqr ${ROUTER_IP}:/etc/config "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/board.json "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/openwrt_release "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/rc.local "${OUTPUT_DIR}"
scp -pqr ${ROUTER_IP}:/etc/sqm/sqm.conf "${OUTPUT_DIR}"

ssh ${ROUTER_IP} "cat /proc/cpuinfo" > "${OUTPUT_DIR}/cpuinfo"
ssh ${ROUTER_IP} "cat /proc/interrupts" > "${OUTPUT_DIR}/interrupts-before"
ssh ${ROUTER_IP} "[ -f /sys/class/thermal/thermal_zone0/temp ] && cat /sys/class/thermal/thermal_zone0/temp" \
  > "${OUTPUT_DIR}/thermal_zone0-temp-before"

RUN_TIME=60

for PARALLEL_STREAMS in 1 2 4 8 16 32 ; do
  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-client-sends.json"

  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --reverse \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-server-sends.json"

  ${IPERF3} \
    --client ${IPERF3_SERVER_IP} \
    --time $RUN_TIME \
    --parallel ${PARALLEL_STREAMS} \
    --interval 5 \
    --bidir \
    --json \
    --logfile "${OUTPUT_DIR}/results/$(date +%s)-${PARALLEL_STREAMS}-bidir.json"
done

ssh ${ROUTER_IP} "cat /proc/interrupts" > "${OUTPUT_DIR}/interrupts-after"
ssh ${ROUTER_IP} "[ -f /sys/class/thermal/thermal_zone0/temp ] && cat /sys/class/thermal/thermal_zone0/temp" \
  > "${OUTPUT_DIR}/thermal_zone0-temp-after"

Here is a summary table of my results. Pasting it into another tool where it can be interactively filtered to just certain conditions is nice. All rates are Mbps calculated as iperf3's reported bits per second / 1e6. There is a repeated run for the WD router.

I'm not totally sure how to interpret the bidirectional iperf3 results, as there are 4 bits_per_second values shown in the end sums. I'm showing sum_sent for the sender: true as Sending, sum_received for the sender: false as Receiving, and the sum of these two as Total.

There are a lot of observations to be made, but one that jumps out for me is that the SQM-enabled bidirectional receiving rate for both ARM-based routers was quite slow.

The Pi 4, with SQM disabled and software flow offloading, was the only case able to hit the gigabit limit in the reverse direction. However, the Atomic Pi was able to record the highest bidirectional total in the testing so far.

Router,SQM,Soft Flow Offloading,SMP Affinity,Direction,Streams,Sending,Receiving,Total
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,150.2,149.9,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,141.2,140.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,137.7,137.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,136.2,135.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,134.8,134.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,133.6,132.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,220.8,220.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,190.5,190.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,176.2,176.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,159.4,159.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,144.2,144.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,130.0,129.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,112.5,50.1,162.5
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,110.7,45.1,155.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,104.0,45.0,149.1
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,101.3,45.5,146.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,107.0,35.9,142.8
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,112.4,25.5,137.9
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,591.0,590.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,631.6,631.3,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,717.6,717.2,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,718.3,717.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,690.1,689.3,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,632.8,631.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,732.8,732.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,815.8,815.1,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,790.6,789.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,779.6,778.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,727.9,726.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,645.1,644.0,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,308.9,422.2,731.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,291.2,457.7,749.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,291.5,445.6,737.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,329.3,387.9,717.2
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,275.1,380.9,656.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,258.1,387.9,646.0
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,266.4,266.0,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,264.2,263.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,264.9,264.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,258.6,257.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,255.0,253.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,251.0,249.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,330.2,329.8,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,329.9,329.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,323.1,322.5,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,307.4,306.4,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,286.0,284.6,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,277.1,275.9,NA
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,119.9,184.3,304.1
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,106.2,196.6,302.8
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,89.4,208.1,297.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,99.0,184.4,283.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,105.0,170.5,275.5
Western Digital My Net N750,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,106.1,164.6,270.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,149.1,148.7,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,139.8,139.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,135.9,135.5,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,133.9,133.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,133.2,132.5,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,131.9,131.1,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,229.2,229.0,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,196.3,196.1,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,186.1,185.8,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,161.4,161.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,145.7,145.3,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,131.6,131.2,NA
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,108.9,53.8,162.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,109.8,42.9,152.7
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,105.1,42.0,147.2
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,99.7,46.6,146.3
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,105.8,35.5,141.3
Western Digital My Net N750,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,115.0,19.9,134.9
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,529.4,529.2,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,524.4,523.8,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,575.3,574.3,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,502.0,499.3,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,439.4,434.9,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,387.8,379.9,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,484.0,483.7,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,448.8,448.0,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,424.4,423.7,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,402.8,402.1,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,333.2,332.2,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,327.5,325.6,NA
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,517.7,7.6,525.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,552.5,5.8,558.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,544.6,4.6,549.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,497.3,3.7,501.0
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,416.3,4.0,420.3
Linksys EA8500 WiFi Router,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,368.4,3.6,372.0
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,863.3,862.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,870.8,870.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,865.6,864.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,846.7,843.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,814.2,810.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,771.8,768.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,926.5,926.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,921.0,920.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,895.8,894.3,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,913.2,910.0,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,856.7,852.1,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,717.8,713.4,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,661.3,249.1,910.4
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,614.1,286.2,900.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,642.5,231.8,874.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,513.7,324.6,838.2
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,406.8,354.3,761.1
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,397.1,338.0,735.1
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,656.1,655.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,647.0,646.2,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,624.1,622.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,583.9,580.8,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,475.5,471.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,436.8,432.7,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,832.7,832.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,811.6,811.2,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,761.8,760.6,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,695.9,692.9,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,541.0,536.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,453.4,449.5,NA
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,346.9,343.4,690.4
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,319.7,349.6,669.3
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,238.9,392.9,631.8
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,181.7,351.3,533.0
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,194.3,260.6,454.9
Linksys EA8500 WiFi Router,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,208.5,234.5,443.0
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,648.9,648.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,653.1,652.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,651.5,650.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,608.3,605.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,548.4,542.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,508.3,499.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,652.1,651.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,688.0,687.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,584.7,584.0,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,503.4,502.8,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,474.1,473.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,461.1,460.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,485.2,168.8,654.0
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,543.0,95.3,638.3
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,481.2,126.4,607.6
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,427.1,123.3,550.4
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,435.1,50.6,485.7
Raspberry Pi 4 Model B Rev 1.1,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,477.6,8.9,486.5
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,752.8,752.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,752.3,751.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,748.3,747.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,734.4,733.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,724.2,722.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,674.3,673.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,941.6,941.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,896.8,896.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,867.9,867.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,896.0,894.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,804.5,802.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,761.0,758.8,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,623.6,160.1,783.7
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,341.7,515.4,857.1
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,408.9,396.3,805.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,391.9,385.4,777.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,389.2,318.6,707.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,413.5,262.1,675.6
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,681.8,681.4,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,678.2,677.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,676.2,675.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,658.4,657.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,638.9,637.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,511.7,510.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,826.1,825.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,709.1,708.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,745.6,744.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,681.7,680.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,627.2,625.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,596.6,594.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,527.4,172.4,699.8
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,313.7,417.4,731.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,398.1,268.0,666.1
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,374.1,233.8,607.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,317.6,239.8,557.4
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,326.0,208.5,534.5
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,1,763.5,763.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,2,760.8,760.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,4,767.3,766.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,8,760.4,759.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,16,746.1,744.6,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Normal Direction,32,716.3,714.9,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,1,941.4,941.0,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,2,942.1,941.3,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,4,942.0,940.7,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,8,942.7,941.2,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,16,942.8,941.1,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Reverse,32,942.4,940.5,NA
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,1,338.8,671.2,1009.9
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,2,425.2,471.6,896.7
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,4,422.9,481.4,904.3
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,8,435.3,416.9,852.2
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,16,418.5,361.1,779.6
Raspberry Pi 4 Model B Rev 1.1,SQM OFF,Soft Flow Offloading ON,SMP Affinity SET,Bidirectional,32,448.7,280.7,729.5
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,768.0,767.5,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,765.6,765.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,765.6,764.7,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,765.7,764.8,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,747.5,746.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,581.1,580.0,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,585.8,585.3,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,554.2,553.7,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,533.7,533.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,512.5,511.9,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,498.0,497.2,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,489.1,488.1,NA
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,698.3,221.1,919.5
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,716.1,179.9,896.0
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,668.7,195.6,864.3
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,576.2,228.8,805.0
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,402.9,274.4,677.3
AAEON MF-001,SQM ON,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,388.4,176.8,565.1
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,1,921.1,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,2,921.4,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,4,922.0,920.8,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,8,922.0,920.8,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,16,921.8,920.7,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Normal Direction,32,921.6,920.6,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,1,897.7,897.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,2,898.8,898.0,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,4,899.2,898.0,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,8,899.6,898.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,16,900.8,899.1,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Reverse,32,902.7,900.2,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,1,457.0,814.3,1271.3
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,2,625.3,704.5,1329.9
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,4,735.4,621.9,1357.4
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,8,852.6,490.8,1343.4
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,16,886.6,379.3,1265.9
AAEON MF-001,SQM OFF,Soft Flow Offloading ON,SMP Affinity NOT SET,Bidirectional,32,856.6,302.6,1159.2
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,1,921.0,920.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,2,921.2,920.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,4,921.2,919.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,8,921.1,919.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,16,920.9,919.7,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Normal Direction,32,917.2,915.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,1,663.3,662.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,2,661.4,660.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,4,658.4,656.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,8,649.4,647.5,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,16,630.5,628.3,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Reverse,32,610.3,607.9,NA
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,1,319.0,501.0,820.0
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,2,394.4,454.3,848.7
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,4,591.4,330.2,921.6
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,8,781.7,189.5,971.2
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,16,741.2,183.2,924.4
AAEON MF-001,SQM OFF,Soft Flow Offloading OFF,SMP Affinity NOT SET,Bidirectional,32,601.6,148.5,750.1

Someone asked about setting smp_affinity for the ASIX USB 3.0 gigabit Ethernet adapter. I got an error when trying to assign the USB xhci_hcd smp_affinity, so I settled for this in /etc/rc.local. Perhaps I should try moving these interrupts to CPU1 and CPU2 instead, so that they both avoid CPU0 where the xhci_hcd interrupt ends up.

#CPU0 - eth0 189
echo "1" >/proc/irq/31/smp_affinity
#CPU1 - eth0 190
echo "2" >/proc/irq/32/smp_affinity

Here is how /proc/interrupts looked at the end of the Pi 4 run. It does seem to help, on the order of 10%.

           CPU0       CPU1       CPU2       CPU3       
  3:     193942       3388      10035       3321     GICv2  30 Level     arch_timer
 11:        651          0          0          0     GICv2  65 Level     fe00b880.mailbox
 14:          2          0          0          0     GICv2 153 Level     uart-pl011
 17:         45          0          0          0     GICv2 114 Level     DMA IRQ
 24:          1          0          0          0     GICv2  66 Level     VCHIQ doorbell
 25:       7867          0          0          0     GICv2 158 Level     mmc1, mmc0
 31:    5300912          0          0          0     GICv2 189 Level     eth0
 32:          2   13432486          0          0     GICv2 190 Level     eth0
 38:          0          0          0          0     GICv2 175 Level     PCIe PME, aerdrv
 39:   21892839          0          0          0  BRCM STB PCIe MSI 524288 Edge      xhci_hcd
IPI0:      4198     121454     450517       1689       Rescheduling interrupts
IPI1:        35        408        358        297       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:     21932      22447      56546        223       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
Err:          0

For comparison, here is /proc/interrupts at the end of one of the Atomic Pi runs.

            CPU0       CPU1       CPU2       CPU3       
   0:         22          0          0          0   IO-APIC    2-edge      timer
   4:          0          0         12          0   IO-APIC    4-edge      ttyS0
   8:          0          0          0          0   IO-APIC    8-fasteoi   rtc0
   9:          0          0          0          0   IO-APIC    9-fasteoi   acpi
 116:          0          0          0        459   PCI-MSI 32768-edge      i915
 118:   23674708          0          0          0   PCI-MSI 327680-edge      xhci_hcd
 119:          0          0          0   23935736   PCI-MSI 524288-edge      eth0
 NMI:          0          0          0          0   Non-maskable interrupts
 LOC:     277526     279505     278958     277853   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 PMI:          0          0          0          0   Performance monitoring interrupts
 IWI:          0          0          0          0   IRQ work interrupts
 RTR:          0          0          0          0   APIC ICR read retries
 RES:        401       1021       2140       2425   Rescheduling interrupts
 CAL:        608        321        987        875   Function call interrupts
 TLB:          0          0          0          0   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 DFR:          0          0          0          0   Deferred Error APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:          4          4          4          4   Machine check polls
 ERR:          0
 MIS:          0
 PIN:          0          0          0          0   Posted-interrupt notification event
 NPI:          0          0          0          0   Nested posted-interrupt event
 PIW:          0          0          0          0   Posted-interrupt wakeup event

I wish I had one, and at $13 it is tempting to buy one just for testing it, but I don't need it for anything else. Maybe another forum member has one, and they could run a test.

In case it's useful, here is the python script for summarizing all the results.

import json
import os


class BenchmarkResult:
    def __init__(self, path, model=None):
        self.path = path

        with open(os.path.join(self.path, "board.json")) as f:
            s = f.read()
        self.model = json.loads(s).get("model", {}).get("name", "unknown model")

        with open(os.path.join(self.path, "config", "sqm")) as f:
            s = f.read()
        self.sqm = "option enabled '1'" in s

        with open(os.path.join(self.path, "config", "firewall")) as f:
            s = f.read()
        self.flow_offloading = "option flow_offloading '1'" in s

        with open(os.path.join(self.path, "rc.local")) as f:
            s = f.read()
        self.smp_affinity = "smp_affinity" in s

        self.results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "client-sends" in fn.name
        ]
        self.reverse_results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "server-sends" in fn.name
        ]
        self.bidir_results = [
            self._read_result(fn.path)
            for fn in os.scandir(os.path.join(self.path, "results"))
            if "bidir" in fn.name
        ]

    def _read_result(self, fn):
        with open(fn) as f:
            s = f.read()
        d = json.loads(s)
        return d

    def _result_lines(self, dir):
        data = {
            "Normal Direction": self.results,
            "Reverse": self.reverse_results,
            "Bidirectional": self.bidir_results,
        }
        return tuple(
            ",".join(
                (
                    self.model,
                    "SQM ON" if self.sqm else "SQM OFF",
                    "Soft Flow Offloading ON"
                    if self.flow_offloading
                    else "Soft Flow Offloading OFF",
                    "SMP Affinity SET" if self.smp_affinity else "SMP Affinity NOT SET",
                    dir,
                    f'{result["start"]["test_start"]["num_streams"]}',
                    f'{result["end"]["sums"][0]["sum_sent"]["bits_per_second"] / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else f'{result["end"]["sum_sent"]["bits_per_second"] / 1e6:.1f}',
                    f'{result["end"]["sums"][1]["sum_received"]["bits_per_second"] / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else f'{result["end"]["sum_received"]["bits_per_second"] / 1e6:.1f}',
                    f'{(result["end"]["sums"][0]["sum_sent"]["bits_per_second"] + result["end"]["sums"][1]["sum_received"]["bits_per_second"]) / 1e6:.1f}'
                    if dir == "Bidirectional"
                    else "NA",
                )
            )
            for result in data[dir]
        )

    def csv(self):
        return "\n".join(
            sum(
                (
                    self._result_lines(dir)
                    for dir in ("Normal Direction", "Reverse", "Bidirectional")
                ),
                start=tuple(),
            )
        )


def main():
    results = [BenchmarkResult(path=e.path) for e in os.scandir("runs")]
    for result in results:
        print(result.csv())


if __name__ == "__main__":
    main()
1 Like

Thanks for sharing these, will try it out on a MT621A platform I'm trying to tune.

To improve my testing setup, I picked up a 4-port gigabit PCIe NIC (HP NC365T) and installed it in my "lab" PC. It shows up as four network interfaces. My testing reported earlier in this thread depended on a USB NIC which I found could not reach 900Mbps in iperf3 in bidirectional mode.

I created 3 separate systemd-nspawn containers, each owning one of the network interfaces:

  1. 10.0.0.1 (static) with DHCP server, attached to router-under-test WAN port
  2. dynamic IP, attached to router-under-test LAN port 1
  3. 192.168.1.10 (static), attached to router-under-test LAN port 2 (if available)

The container on LAN port 1 acts as a testing client, while the container on the WAN port acts as the testing server. I use the container on LAN port 2 as a best-case baseline, since by default traffic flows between the LAN ports just through the router's switch.

I've been running iperf3 with 1, 4, and 16 parallel connections in normal (UP), reverse (DN), and bidirectional (BI) modes.

I also added the caddy HTTP server in the "server" container and run the hey HTTP load-tester on the testing client, recording the requests-per-second result getting a 4MiB random file with 50 concurrent workers for 60 seconds.

Both the iperf3 and caddy/hey tests are run with SW flow offloading off and on, and with SQM enabled but with 800Mbps/800Mbps rate limits.

Here's what I have so far, with just the Linksys EA8500, running 19.07.4 without any special tuning. The rates are in Mbps or requests per second.

SQM      SW OFFLOAD  ROUTER       1:UP/DN/BI       4:UP/DN/BI       16:UP/DN/BI      caddy-hey-4M
NA       NA          Switch only  942/940/927      933/931/926      928/926/920      28.1
OFF      OFF         EA8500       932/893/361      25/718/236-433   553/540/224-261  13.8
OFF      ON          EA8500       932/932/463      914/864/430-459  869/833/373-403  22.3
800/800  OFF         EA8500       762/495/2.2-760  598/411/2.5-608  486/322/2.4-487   9.4

Like before, I'm seeing one direction of the bidirectional iperf3 test having very low rates (around 2.5 Mbps) with SQM enabled. I'll see if this shows up on the Raspberry Pi 4 again---it seemed to be common to the two ARM-based routers I tested.

If there are any suggestions on my test methods, now would be a great time to share them. I'm about to run the same tests on several other routers.

Intriguing, could you post (again) the output of tc -s qdisc from just before and after the iperf3 test, as well as the iperf output and a redacted (check for passwords and maybe IP addresses) version of the output of dmesg and logread (in both cases it should be sufficient to just post the output starting around the time you did the first tc -s qdisc invocation). Also run top -d 1 on the router during the test and observe the %idel column for signs of CPU overload.
And for ARM CPUs it also is important to rule out voltage/frequency scaling (I believe the performance governor should be selected). And finally receive packet steering might cause issues as it might cause all SQM processing to bunch up on the same CPU resulting potentially in one CPU idling while the other is overloaded.

OK, I've run the single-connection bidirectional iperf3 test again, collecting before and after information.

It appears the OpenWrt default for this router uses the ondemand governor:

# cat /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
ondemand

iperf3 output (the RX-C result was much faster this time than last time, at the expense of the RX-C result):

$ iperf3 -c 10.0.0.1 --bidir --time 60 | tee -a iperf3-output.txt 
Connecting to host 10.0.0.1, port 5201
[  5] local 192.168.1.225 port 36008 connected to 10.0.0.1 port 5201
[  7] local 192.168.1.225 port 36010 connected to 10.0.0.1 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec  74.0 MBytes   621 Mbits/sec    0   3.06 MBytes       
[  7][RX-C]   0.00-1.00   sec  11.7 MBytes  98.2 Mbits/sec                  
[  5][TX-C]   1.00-2.00   sec  90.0 MBytes   755 Mbits/sec  355   2.28 MBytes       
[  7][RX-C]   1.00-2.00   sec   877 KBytes  7.18 Mbits/sec                  
[  5][TX-C]   2.00-3.00   sec  90.0 MBytes   755 Mbits/sec  451   1.71 MBytes       
[  7][RX-C]   2.00-3.00   sec   382 KBytes  3.13 Mbits/sec                  
[  5][TX-C]   3.00-4.00   sec  90.0 MBytes   755 Mbits/sec  159   1.28 MBytes       
[  7][RX-C]   3.00-4.00   sec   967 KBytes  7.92 Mbits/sec                  
[  5][TX-C]   4.00-5.00   sec  88.8 MBytes   744 Mbits/sec  163    987 KBytes       
[  7][RX-C]   4.00-5.00   sec  2.61 MBytes  21.9 Mbits/sec                  
[  5][TX-C]   5.00-6.00   sec  78.8 MBytes   661 Mbits/sec  141    755 KBytes       
[  7][RX-C]   5.00-6.00   sec  9.45 MBytes  79.3 Mbits/sec                  
[  5][TX-C]   6.00-7.00   sec  70.0 MBytes   587 Mbits/sec    9    598 KBytes       
[  7][RX-C]   6.00-7.00   sec  17.6 MBytes   147 Mbits/sec                  
[  5][TX-C]   7.00-8.00   sec  66.2 MBytes   556 Mbits/sec    0    680 KBytes       
[  7][RX-C]   7.00-8.00   sec  20.6 MBytes   173 Mbits/sec                  
[  5][TX-C]   8.00-9.00   sec  71.2 MBytes   598 Mbits/sec    0    755 KBytes       
[  7][RX-C]   8.00-9.00   sec  15.1 MBytes   127 Mbits/sec                  
[  5][TX-C]   9.00-10.00  sec  68.8 MBytes   577 Mbits/sec   16    597 KBytes       
[  7][RX-C]   9.00-10.00  sec  18.0 MBytes   151 Mbits/sec                  
[  5][TX-C]  10.00-11.00  sec  65.0 MBytes   545 Mbits/sec    0    673 KBytes       
[  7][RX-C]  10.00-11.00  sec  21.2 MBytes   177 Mbits/sec                  
[  5][TX-C]  11.00-12.00  sec  71.2 MBytes   598 Mbits/sec    0    752 KBytes       
[  7][RX-C]  11.00-12.00  sec  15.3 MBytes   128 Mbits/sec                  
[  5][TX-C]  12.00-13.00  sec  70.0 MBytes   587 Mbits/sec   14    594 KBytes       
[  7][RX-C]  12.00-13.00  sec  16.1 MBytes   135 Mbits/sec                  
[  5][TX-C]  13.00-14.00  sec  66.2 MBytes   556 Mbits/sec    0    669 KBytes       
[  7][RX-C]  13.00-14.00  sec  20.2 MBytes   170 Mbits/sec                  
[  5][TX-C]  14.00-15.00  sec  72.5 MBytes   608 Mbits/sec    0    749 KBytes       
[  7][RX-C]  14.00-15.00  sec  15.9 MBytes   134 Mbits/sec                  
[  5][TX-C]  15.00-16.00  sec  75.0 MBytes   629 Mbits/sec    0    822 KBytes       
[  7][RX-C]  15.00-16.00  sec  12.1 MBytes   102 Mbits/sec                  
[  5][TX-C]  16.00-17.00  sec  78.8 MBytes   661 Mbits/sec    0    892 KBytes       
[  7][RX-C]  16.00-17.00  sec  9.87 MBytes  82.8 Mbits/sec                  
[  5][TX-C]  17.00-18.00  sec  78.8 MBytes   661 Mbits/sec  126    690 KBytes       
[  7][RX-C]  17.00-18.00  sec  10.1 MBytes  85.0 Mbits/sec                  
[  5][TX-C]  18.00-19.00  sec  72.5 MBytes   608 Mbits/sec    0    778 KBytes       
[  7][RX-C]  18.00-19.00  sec  14.2 MBytes   119 Mbits/sec                  
[  5][TX-C]  19.00-20.00  sec  76.2 MBytes   640 Mbits/sec    0    840 KBytes       
[  7][RX-C]  19.00-20.00  sec  11.4 MBytes  95.5 Mbits/sec                  
[  5][TX-C]  20.00-21.00  sec  77.5 MBytes   650 Mbits/sec   55    636 KBytes       
[  7][RX-C]  20.00-21.00  sec  11.1 MBytes  93.2 Mbits/sec                  
[  5][TX-C]  21.00-22.00  sec  70.0 MBytes   587 Mbits/sec    0    714 KBytes       
[  7][RX-C]  21.00-22.00  sec  16.3 MBytes   137 Mbits/sec                  
[  5][TX-C]  22.00-23.00  sec  72.5 MBytes   608 Mbits/sec    0    790 KBytes       
[  7][RX-C]  22.00-23.00  sec  13.5 MBytes   113 Mbits/sec                  
[  5][TX-C]  23.00-24.00  sec  77.5 MBytes   650 Mbits/sec    0    863 KBytes       
[  7][RX-C]  23.00-24.00  sec  10.7 MBytes  89.4 Mbits/sec                  
[  5][TX-C]  24.00-25.00  sec  81.2 MBytes   682 Mbits/sec    0    932 KBytes       
[  7][RX-C]  24.00-25.00  sec  8.69 MBytes  72.9 Mbits/sec                  
[  5][TX-C]  25.00-26.00  sec  82.5 MBytes   692 Mbits/sec  125    710 KBytes       
[  7][RX-C]  25.00-26.00  sec  7.30 MBytes  61.3 Mbits/sec                  
[  5][TX-C]  26.00-27.00  sec  73.8 MBytes   619 Mbits/sec    0    803 KBytes       
[  7][RX-C]  26.00-27.00  sec  12.4 MBytes   104 Mbits/sec                  
[  5][TX-C]  27.00-28.00  sec  78.8 MBytes   661 Mbits/sec    0    871 KBytes       
[  7][RX-C]  27.00-28.00  sec  10.3 MBytes  86.2 Mbits/sec                  
[  5][TX-C]  28.00-29.00  sec  81.2 MBytes   682 Mbits/sec    0    922 KBytes       
[  7][RX-C]  28.00-29.00  sec  8.68 MBytes  72.8 Mbits/sec                  
[  5][TX-C]  29.00-30.00  sec  82.5 MBytes   692 Mbits/sec  133    689 KBytes       
[  7][RX-C]  29.00-30.00  sec  7.12 MBytes  59.7 Mbits/sec                  
[  5][TX-C]  30.00-31.00  sec  75.0 MBytes   629 Mbits/sec    0    766 KBytes       
[  7][RX-C]  30.00-31.00  sec  12.3 MBytes   103 Mbits/sec                  
[  5][TX-C]  31.00-32.00  sec  76.2 MBytes   640 Mbits/sec    0    841 KBytes       
[  7][RX-C]  31.00-32.00  sec  11.3 MBytes  95.1 Mbits/sec                  
[  5][TX-C]  32.00-33.00  sec  80.0 MBytes   671 Mbits/sec    0    912 KBytes       
[  7][RX-C]  32.00-33.00  sec  9.20 MBytes  77.2 Mbits/sec                  
[  5][TX-C]  33.00-34.00  sec  83.8 MBytes   703 Mbits/sec    0    979 KBytes       
[  7][RX-C]  33.00-34.00  sec  7.26 MBytes  60.9 Mbits/sec                  
[  5][TX-C]  34.00-35.00  sec  73.8 MBytes   619 Mbits/sec  159    778 KBytes       
[  7][RX-C]  34.00-35.00  sec  12.8 MBytes   108 Mbits/sec                  
[  5][TX-C]  35.00-36.00  sec  77.5 MBytes   650 Mbits/sec    0    854 KBytes       
[  7][RX-C]  35.00-36.00  sec  11.1 MBytes  93.2 Mbits/sec                  
[  5][TX-C]  36.00-37.00  sec  80.0 MBytes   671 Mbits/sec    0    906 KBytes       
[  7][RX-C]  36.00-37.00  sec  8.95 MBytes  75.0 Mbits/sec                  
[  5][TX-C]  37.00-38.00  sec  72.5 MBytes   608 Mbits/sec   30    710 KBytes       
[  7][RX-C]  37.00-38.00  sec  14.8 MBytes   124 Mbits/sec                  
[  5][TX-C]  38.00-39.00  sec  73.8 MBytes   619 Mbits/sec    0    785 KBytes       
[  7][RX-C]  38.00-39.00  sec  13.8 MBytes   116 Mbits/sec                  
[  5][TX-C]  39.00-40.00  sec  76.2 MBytes   640 Mbits/sec    0    857 KBytes       
[  7][RX-C]  39.00-40.00  sec  11.1 MBytes  93.2 Mbits/sec                  
[  5][TX-C]  40.00-41.00  sec  81.2 MBytes   682 Mbits/sec    0    925 KBytes       
[  7][RX-C]  40.00-41.00  sec  8.67 MBytes  72.7 Mbits/sec                  
[  5][TX-C]  41.00-42.00  sec  73.8 MBytes   619 Mbits/sec   86    732 KBytes       
[  7][RX-C]  41.00-42.00  sec  14.0 MBytes   118 Mbits/sec                  
[  5][TX-C]  42.00-43.00  sec  75.0 MBytes   629 Mbits/sec    0    810 KBytes       
[  7][RX-C]  42.00-43.00  sec  12.8 MBytes   107 Mbits/sec                  
[  5][TX-C]  43.00-44.00  sec  77.5 MBytes   650 Mbits/sec    0    867 KBytes       
[  7][RX-C]  43.00-44.00  sec  10.3 MBytes  86.2 Mbits/sec                  
[  5][TX-C]  44.00-45.00  sec  81.2 MBytes   682 Mbits/sec    0    935 KBytes       
[  7][RX-C]  44.00-45.00  sec  8.55 MBytes  71.7 Mbits/sec                  
[  5][TX-C]  45.00-46.00  sec  75.0 MBytes   629 Mbits/sec   93    738 KBytes       
[  7][RX-C]  45.00-46.00  sec  13.1 MBytes   110 Mbits/sec                  
[  5][TX-C]  46.00-47.00  sec  73.8 MBytes   619 Mbits/sec    0    813 KBytes       
[  7][RX-C]  46.00-47.00  sec  12.8 MBytes   107 Mbits/sec                  
[  5][TX-C]  47.00-48.00  sec  71.2 MBytes   598 Mbits/sec   21    638 KBytes       
[  7][RX-C]  47.00-48.00  sec  15.7 MBytes   132 Mbits/sec                  
[  5][TX-C]  48.00-49.00  sec  68.8 MBytes   577 Mbits/sec    0    717 KBytes       
[  7][RX-C]  48.00-49.00  sec  17.2 MBytes   144 Mbits/sec                  
[  5][TX-C]  49.00-50.00  sec  73.8 MBytes   619 Mbits/sec    0    793 KBytes       
[  7][RX-C]  49.00-50.00  sec  13.4 MBytes   112 Mbits/sec                  
[  5][TX-C]  50.00-51.00  sec  77.5 MBytes   650 Mbits/sec    0    865 KBytes       
[  7][RX-C]  50.00-51.00  sec  10.4 MBytes  87.3 Mbits/sec                  
[  5][TX-C]  51.00-52.00  sec  81.2 MBytes   682 Mbits/sec    0    935 KBytes       
[  7][RX-C]  51.00-52.00  sec  8.66 MBytes  72.6 Mbits/sec                  
[  5][TX-C]  52.00-53.00  sec  82.5 MBytes   692 Mbits/sec    0    998 KBytes       
[  7][RX-C]  52.00-53.00  sec  6.50 MBytes  54.5 Mbits/sec                  
[  5][TX-C]  53.00-54.00  sec  81.2 MBytes   682 Mbits/sec  129    775 KBytes       
[  7][RX-C]  53.00-54.00  sec  7.76 MBytes  65.1 Mbits/sec                  
[  5][TX-C]  54.00-55.00  sec  77.5 MBytes   650 Mbits/sec    0    864 KBytes       
[  7][RX-C]  54.00-55.00  sec  10.6 MBytes  88.5 Mbits/sec                  
[  5][TX-C]  55.00-56.00  sec  75.0 MBytes   629 Mbits/sec  121    665 KBytes       
[  7][RX-C]  55.00-56.00  sec  13.5 MBytes   113 Mbits/sec                  
[  5][TX-C]  56.00-57.00  sec  70.0 MBytes   587 Mbits/sec    0    741 KBytes       
[  7][RX-C]  56.00-57.00  sec  15.7 MBytes   132 Mbits/sec                  
[  5][TX-C]  57.00-58.00  sec  75.0 MBytes   629 Mbits/sec    0    816 KBytes       
[  7][RX-C]  57.00-58.00  sec  12.5 MBytes   105 Mbits/sec                  
[  5][TX-C]  58.00-59.00  sec  73.8 MBytes   619 Mbits/sec   23    641 KBytes       
[  7][RX-C]  58.00-59.00  sec  13.3 MBytes   112 Mbits/sec                  
[  5][TX-C]  59.00-60.00  sec  70.0 MBytes   587 Mbits/sec    0    718 KBytes       
[  7][RX-C]  59.00-60.00  sec  17.0 MBytes   143 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-60.00  sec  4.47 GBytes   640 Mbits/sec  2409             sender
[  5][TX-C]   0.00-60.00  sec  4.47 GBytes   639 Mbits/sec                  receiver
[  7][RX-C]   0.00-60.00  sec   707 MBytes  98.9 Mbits/sec  444             sender
[  7][RX-C]   0.00-60.00  sec   705 MBytes  98.5 Mbits/sec                  receiver

iperf Done.

tc -s qdisc before:

qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 23414 bytes 144 pkt (dropped 0, overlimits 0 requeues 1) 
 backlog 0b 0p requeues 1
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 23414 bytes 144 pkt (dropped 0, overlimits 0 requeues 1) 
 backlog 0b 0p requeues 1
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 direct_qlen 1000
 Sent 1454 bytes 12 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 1454 bytes 12 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 164 drop_overlimit 0 new_flow_count 4 ecn_mark 0
  new_flows_len 0 old_flows_len 1
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 264 bytes 5 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
 Sent 292 bytes 5 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 292 bytes 5 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 70 drop_overlimit 0 new_flow_count 2 ecn_mark 0
  new_flows_len 1 old_flows_len 0

tc -s qdisc after:

qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root 
 Sent 6784101267 bytes 4820857 pkt (dropped 0, overlimits 0 requeues 289352) 
 backlog 0b 0p requeues 289352
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 6784101267 bytes 4820857 pkt (dropped 0, overlimits 0 requeues 289352) 
 backlog 0b 0p requeues 289352
  maxpacket 1514 drop_overlimit 0 new_flow_count 305497 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 direct_qlen 1000
 Sent 5917911859 bytes 3962089 pkt (dropped 0, overlimits 3669 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 5917911859 bytes 3962089 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 12112 drop_overlimit 0 new_flow_count 54217 ecn_mark 0
  new_flows_len 0 old_flows_len 3
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
 Sent 834903770 bytes 856289 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 direct_qlen 32
 Sent 864891508 bytes 855416 pkt (dropped 532, overlimits 1212 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
 Sent 864891508 bytes 855416 pkt (dropped 532, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 12112 drop_overlimit 0 new_flow_count 17101 ecn_mark 0
  new_flows_len 1 old_flows_len 2

Snapshot of top -d during the run:

Mem: 38104K used, 442348K free, 80K shrd, 2608K buff, 8116K cached
CPU:   0% usr   0% sys   0% nic  21% idle   0% io   0% irq  77% sirq
Load average: 0.55 0.19 0.07 1/61 3661
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
   14     2 root     RW       0   0%  68% [ksoftirqd/1]
    7     2 root     RW       0   0%   7% [ksoftirqd/0]
 3661  3646 root     R     1096   0%   1% top -d 1
   50     2 root     IW       0   0%   1% [kworker/1:1]
 3645   788 root     S      908   0%   1% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T
  692     1 root     S     1828   0%   0% /sbin/rpcd -s /var/run/ubus.sock -t 30
  845     1 root     S     1556   0%   0% /sbin/netifd
    1     0 root     S     1360   0%   0% /sbin/procd
  884     1 root     S     1248   0%   0% /usr/sbin/odhcpd
 3623     1 dnsmasq  S     1176   0%   0% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/r
 2183     1 root     S     1112   0%   0% /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -t 60 -T 30 -k
 3646  3645 root     S     1088   0%   0% -ash
 2880     1 root     S<    1080   0%   0% /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.
 2113   845 root     S     1080   0%   0% udhcpc -p /var/run/udhcpc-eth0.2.pid -s /lib/netifd/dhcp.script -
  661     1 root     S     1032   0%   0% /sbin/logd -S 64
  164     1 root     S     1008   0%   0% /sbin/ubusd
  788     1 root     S      844   0%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T
 2116   845 root     S      840   0%   0% odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
  182     1 root     S      816   0%   0% /sbin/urngd
  165     1 root     S      704   0%   0% /sbin/askfirst /usr/libexec/login.sh
    5     2 root     IW       0   0%   0% [kworker/u4:0]
   58     2 root     IW       0   0%   0% [kworker/0:2]
   17     2 root     IW       0   0%   0% [kworker/0:1]
  350     2 root     IW       0   0%   0% [kworker/u4:2]
    8     2 root     IW       0   0%   0% [rcu_sched]
   13     2 root     SW       0   0%   0% [migration/1]
   10     2 root     SW       0   0%   0% [migration/0]
   59     2 root     SW       0   0%   0% [ubi_bgt0d]
   57     2 root     IW<      0   0%   0% [dsa_ordered]

SQM config:

config queue 'eth1'
        option enabled '1'
        option qdisc 'fq_codel'
        option script 'simple.qos'
        option qdisc_advanced '0'
        option linklayer 'none'
        option interface 'eth0.2'
        option download '800000'
        option upload '800000'
        option debug_logging '0'
        option verbosity '5'

dmesg output collected after the run:

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.195 (builder@buildhost) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r11208-ce6496d796)) #0 SMP Sun Sep 6 16:19:39 2020
[    0.000000] CPU: ARMv7 Processor [512f04d0] revision 0 (ARMv7), cr=10c5787d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
[    0.000000] OF: fdt: Machine model: Linksys EA8500 WiFi Router
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] On node 0 totalpages: 122880
[    0.000000] free_area_init_node: node 0, pgdat c0a27880, node_mem_map ddc39000
[    0.000000]   Normal zone: 960 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 122880 pages, LIFO batch:31
[    0.000000] random: get_random_bytes called from 0xc09008e0 with crng_init=0
[    0.000000] percpu: Embedded 15 pages/cpu s29388 r8192 d23860 u61440
[    0.000000] pcpu-alloc: s29388 r8192 d23860 u61440 alloc=15*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 121920
[    0.000000] Kernel command line: console=ttyMSM0,115200n8 ubi.mtd=16
[    0.000000] Bootloader command line (ignored): console=ttyHSL1,115200n8 init=/sbin/init rootfstype=squashfs root=31:16
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 479428K/491520K available (4978K kernel code, 160K rwdata, 756K rodata, 1024K init, 235K bss, 12092K reserved, 0K cma-reserved, 0K highmem)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xde800000 - 0xff800000   ( 528 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xde000000   ( 480 MB)
[    0.000000]     pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
[    0.000000]     modules : 0xbf000000 - 0xbfe00000   (  14 MB)
[    0.000000]       .text : 0xc0208000 - 0xc07dc880   (5971 kB)
[    0.000000]       .init : 0xc0900000 - 0xc0a00000   (1024 kB)
[    0.000000]       .data : 0xc0a00000 - 0xc0a28180   ( 161 kB)
[    0.000000]        .bss : 0xc0a2a000 - 0xc0a64e18   ( 236 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] clocksource: dg_timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 305801671480 ns
[    0.000008] sched_clock: 32 bits at 6MHz, resolution 160ns, wraps every 343597383600ns
[    0.000023] Switching to timer-based delay loop, resolution 160ns
[    0.000234] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.50 BogoMIPS (lpj=62500)
[    0.000261] pid_max: default: 32768 minimum: 301
[    0.000390] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000409] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000977] CPU: Testing write buffer coherency: ok
[    0.001751] Setting up static identity map for 0x42300000 - 0x42300060
[    0.001915] Hierarchical SRCU implementation.
[    0.002653] smp: Bringing up secondary CPUs ...
[    0.004443] smp: Brought up 1 node, 2 CPUs
[    0.004461] SMP: Total of 2 processors activated (25.00 BogoMIPS).
[    0.004472] CPU: All CPU(s) started in SVC mode.
[    0.014085] VFP support v0.3: implementor 51 architecture 64 part 4d variant 2 rev 0
[    0.014236] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.014262] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.014374] pinctrl core: initialized pinctrl subsystem
[    0.015347] NET: Registered protocol family 16
[    0.015623] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    0.016948] cpuidle: using governor ladder
[    0.017012] cpuidle: using governor menu
[    0.039868] msm_bus_fabric_init_driver
[    0.041301] usbcore: registered new interface driver usbfs
[    0.041482] usbcore: registered new interface driver hub
[    0.041563] usbcore: registered new device driver usb
[    0.041619] pps_core: LinuxPPS API ver. 1 registered
[    0.041631] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    0.041664] PTP clock support registered
[    0.043382] clocksource: Switched to clocksource dg_timer
[    0.046535] NET: Registered protocol family 2
[    0.047135] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    0.047174] TCP bind hash table entries: 4096 (order: 3, 32768 bytes)
[    0.047228] TCP: Hash tables configured (established 4096 bind 4096)
[    0.047320] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    0.047346] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.047520] NET: Registered protocol family 1
[    0.047567] PCI: CLS 0 bytes, default 64
[    0.048740] No memory allocated for crashlog
[    0.049035] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    0.053867] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.053885] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.060430] io scheduler noop registered
[    0.060449] io scheduler deadline registered (default)
[    0.062076] qcom-pcie 1b500000.pci: 1b500000.pci supply vdda not found, using dummy regulator
[    0.062184] qcom-pcie 1b500000.pci: 1b500000.pci supply vdda_phy not found, using dummy regulator
[    0.062279] qcom-pcie 1b500000.pci: 1b500000.pci supply vdda_refclk not found, using dummy regulator
[    0.062984] OF: PCI: host bridge /soc/pci@1b500000 ranges:
[    0.063026] OF: PCI:    IO 0x0fe00000..0x0fefffff -> 0x0fe00000
[    0.063050] OF: PCI:   MEM 0x08000000..0x0fdfffff -> 0x08000000
[    0.171213] qcom-pcie 1b500000.pci: link up
[    0.171382] qcom-pcie 1b500000.pci: PCI host bridge to bus 0000:00
[    0.171407] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.171426] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xfe00000-0xfefffff])
[    0.171442] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fdfffff]
[    0.171459] pci_bus 0000:00: scanning bus
[    0.171511] pci 0000:00:00.0: [17cb:0101] type 01 class 0x060400
[    0.171601] pci 0000:00:00.0: calling 0xc030d3c0
[    0.171676] pci 0000:00:00.0: supports D1
[    0.171686] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    0.171702] pci 0000:00:00.0: PME# disabled
[    0.171958] pci_bus 0000:00: fixups for bus
[    0.171982] PCI: bus0: Fast back to back transfers disabled
[    0.172003] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 0
[    0.172130] pci_bus 0000:01: scanning bus
[    0.172318] pci 0000:01:00.0: [168c:0040] type 00 class 0x028000
[    0.172696] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    0.173196] pci 0000:01:00.0: calling 0xc030d3c0
[    0.174000] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    0.174043] pci 0000:01:00.0: PME# disabled
[    0.174376] pci_bus 0000:01: fixups for bus
[    0.174452] PCI: bus1: Fast back to back transfers disabled
[    0.174467] pci_bus 0000:01: bus scan returning with max=01
[    0.174485] pci 0000:00:00.0: scanning [bus 01-ff] behind bridge, pass 1
[    0.174503] pci_bus 0000:00: bus scan returning with max=ff
[    0.174550] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x081fffff]
[    0.174576] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x081fffff 64bit]
[    0.174711] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    0.174734] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x081fffff]
[    0.174765] pci 0000:00:00.0: calling 0xc055ff28
[    0.174953] pcieport 0000:00:00.0: assign IRQ: got 68
[    0.175261] pcieport 0000:00:00.0: AER enabled with IRQ 36
[    0.175377] pci 0000:01:00.0: calling 0xc055ff28
[    0.175793] qcom-pcie 1b700000.pci: 1b700000.pci supply vdda not found, using dummy regulator
[    0.175907] qcom-pcie 1b700000.pci: 1b700000.pci supply vdda_phy not found, using dummy regulator
[    0.176000] qcom-pcie 1b700000.pci: 1b700000.pci supply vdda_refclk not found, using dummy regulator
[    0.176688] OF: PCI: host bridge /soc/pci@1b700000 ranges:
[    0.176727] OF: PCI:    IO 0x31e00000..0x31efffff -> 0x31e00000
[    0.176751] OF: PCI:   MEM 0x2e000000..0x31dfffff -> 0x2e000000
[    0.286609] qcom-pcie 1b700000.pci: link up
[    0.286756] qcom-pcie 1b700000.pci: PCI host bridge to bus 0001:00
[    0.286776] pci_bus 0001:00: root bus resource [bus 00-ff]
[    0.286792] pci_bus 0001:00: root bus resource [mem 0x2e000000-0x31dfffff]
[    0.286806] pci_bus 0001:00: scanning bus
[    0.286846] pci 0001:00:00.0: [17cb:0101] type 01 class 0x060400
[    0.286917] pci 0001:00:00.0: calling 0xc030d3c0
[    0.286977] pci 0001:00:00.0: supports D1
[    0.286987] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    0.287001] pci 0001:00:00.0: PME# disabled
[    0.287226] pci_bus 0001:00: fixups for bus
[    0.287246] PCI: bus0: Fast back to back transfers disabled
[    0.287264] pci 0001:00:00.0: scanning [bus 01-ff] behind bridge, pass 0
[    0.287376] pci_bus 0001:01: scanning bus
[    0.287552] pci 0001:01:00.0: [168c:0040] type 00 class 0x028000
[    0.287896] pci 0001:01:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    0.288360] pci 0001:01:00.0: calling 0xc030d3c0
[    0.289002] pci 0001:01:00.0: PME# supported from D0 D3hot D3cold
[    0.289043] pci 0001:01:00.0: PME# disabled
[    0.289364] pci_bus 0001:01: fixups for bus
[    0.289435] PCI: bus1: Fast back to back transfers disabled
[    0.289450] pci_bus 0001:01: bus scan returning with max=01
[    0.289465] pci 0001:00:00.0: scanning [bus 01-ff] behind bridge, pass 1
[    0.289483] pci_bus 0001:00: bus scan returning with max=ff
[    0.289519] pci 0001:00:00.0: BAR 8: assigned [mem 0x2e000000-0x2e1fffff]
[    0.289541] pci 0001:01:00.0: BAR 0: assigned [mem 0x2e000000-0x2e1fffff 64bit]
[    0.289665] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    0.289685] pci 0001:00:00.0:   bridge window [mem 0x2e000000-0x2e1fffff]
[    0.289711] pci 0001:00:00.0: calling 0xc055ff28
[    0.289884] pcieport 0001:00:00.0: assign IRQ: got 101
[    0.290150] pcieport 0001:00:00.0: AER enabled with IRQ 69
[    0.290257] pci 0001:01:00.0: calling 0xc055ff28
[    0.290680] qcom-pcie 1b900000.pci: 1b900000.pci supply vdda not found, using dummy regulator
[    0.290783] qcom-pcie 1b900000.pci: 1b900000.pci supply vdda_phy not found, using dummy regulator
[    0.290888] qcom-pcie 1b900000.pci: 1b900000.pci supply vdda_refclk not found, using dummy regulator
[    0.291565] OF: PCI: host bridge /soc/pci@1b900000 ranges:
[    0.291602] OF: PCI:    IO 0x35e00000..0x35efffff -> 0x35e00000
[    0.291624] OF: PCI:   MEM 0x32000000..0x35dfffff -> 0x32000000
[    1.325435] qcom-pcie 1b900000.pci: phy link never came up
[    1.327423] qcom-pcie 1b900000.pci: cannot initialize host
[    1.328007] qcom-pcie: probe of 1b900000.pci failed with error -110
[    1.330216] L2 @ QSB rate. Forcing new rate.
[    1.330432] L2 @ 384000 KHz
[    1.330604] CPU0 @ 800000 KHz
[    1.330617] CPU1 @ QSB rate. Forcing new rate.
[    1.330742] CPU1 @ 384000 KHz
[    1.334735] gsbi 16300000.gsbi: GSBI port protocol: 6 crci: 0
[    1.336665] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    1.339176] msm_serial 16340000.serial: msm_serial: detected port #0
[    1.339273] msm_serial 16340000.serial: uartclk = 7372800
[    1.339356] 16340000.serial: ttyMSM0 at MMIO 0x16340000 (irq = 134, base_baud = 460800) is a MSM
[    1.339396] msm_serial: console setup on port #0
[    2.127097] console [ttyMSM0] enabled
[    2.131833] msm_serial: driver initialized
[    2.140298] loop: module loaded
[    2.142335] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xa1
[    2.142371] nand: AMD/Spansion S34MS01G2
[    2.148949] nand: 128 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
[    2.152940] 18 fixed-partitions partitions found on MTD device qcom_nand.0
[    2.160224] Creating 18 MTD partitions on "qcom_nand.0":
[    2.167076] 0x000000000000-0x000000040000 : "SBL1"
[    2.173796] 0x000000040000-0x000000180000 : "MIBIB"
[    2.179992] 0x000000180000-0x0000002c0000 : "SBL2"
[    2.184788] 0x0000002c0000-0x000000540000 : "SBL3"
[    2.186916] random: fast init done
[    2.196415] 0x000000540000-0x000000660000 : "DDRCONFIG"
[    2.199137] 0x000000660000-0x000000780000 : "SSD"
[    2.203122] 0x000000780000-0x000000a00000 : "TZ"
[    2.210477] 0x000000a00000-0x000000c80000 : "RPM"
[    2.215637] 0x000000c80000-0x000000dc0000 : "art"
[    2.218545] 0x000000dc0000-0x000000ec0000 : "APPSBL"
[    2.221806] 0x000000ec0000-0x000000f00000 : "u_env"
[    2.225633] 0x000000f00000-0x000000f40000 : "s_env"
[    2.230143] 0x000000f40000-0x000000f80000 : "devinfo"
[    2.235043] 0x000000f80000-0x000003780000 : "kernel1"
[    2.309913] 0x000001280000-0x000003780000 : "rootfs1"
[    2.375700] 0x000003780000-0x000005f80000 : "kernel2"
[    2.446733] 0x000003a80000-0x000005f80000 : "rootfs2"
[    2.507207] random: crng init done
[    2.512560] 0x000005f80000-0x000008000000 : "syscfg"
[    2.571787] libphy: GPIO Bitbanged MDIO: probed
[    2.592993] switch0: Atheros AR8337 rev. 2 switch registered on gpio-0
[    3.436137] ar8327: qca,phy-rgmii-en is not specified
[    3.436523] libphy: Fixed MDIO Bus: probed
[    3.441680] ipq806x-gmac-dwmac 37200000.ethernet: PTP uses main clock
[    3.444573] stmmac - user ID: 0x10, Synopsys ID: 0x37
[    3.450677] ipq806x-gmac-dwmac 37200000.ethernet: Ring mode enabled
[    3.455793] ipq806x-gmac-dwmac 37200000.ethernet: DMA HW capability register supported
[    3.461792] ipq806x-gmac-dwmac 37200000.ethernet: Enhanced/Alternate descriptors
[    3.469837] ipq806x-gmac-dwmac 37200000.ethernet: Enabled extended descriptors
[    3.477400] ipq806x-gmac-dwmac 37200000.ethernet: RX Checksum Offload Engine supported
[    3.484427] ipq806x-gmac-dwmac 37200000.ethernet: COE Type 2
[    3.492253] ipq806x-gmac-dwmac 37200000.ethernet: TX Checksum insertion supported
[    3.498143] ipq806x-gmac-dwmac 37200000.ethernet: Wake-Up On Lan supported
[    3.505527] ipq806x-gmac-dwmac 37200000.ethernet (unnamed net_device) (uninitialized): device MAC address aa:aa:aa:bb:bb:bb
[    3.512233] ipq806x-gmac-dwmac 37200000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    3.524960] ipq806x-gmac-dwmac 37400000.ethernet: PTP uses main clock
[    3.532117] stmmac - user ID: 0x10, Synopsys ID: 0x37
[    3.538361] ipq806x-gmac-dwmac 37400000.ethernet: Ring mode enabled
[    3.543298] ipq806x-gmac-dwmac 37400000.ethernet: DMA HW capability register supported
[    3.549437] ipq806x-gmac-dwmac 37400000.ethernet: Enhanced/Alternate descriptors
[    3.557435] ipq806x-gmac-dwmac 37400000.ethernet: Enabled extended descriptors
[    3.564994] ipq806x-gmac-dwmac 37400000.ethernet: RX Checksum Offload Engine supported
[    3.571947] ipq806x-gmac-dwmac 37400000.ethernet: COE Type 2
[    3.579926] ipq806x-gmac-dwmac 37400000.ethernet: TX Checksum insertion supported
[    3.585745] ipq806x-gmac-dwmac 37400000.ethernet: Wake-Up On Lan supported
[    3.593048] ipq806x-gmac-dwmac 37400000.ethernet (unnamed net_device) (uninitialized): device MAC address bb:bb:bb:cc:cc:cc
[    3.599892] ipq806x-gmac-dwmac 37400000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    3.611575] i2c /dev entries driver
[    3.620425] Calibration not found.
[    3.623916] Speed bin: 0
[    3.626189] PVS bin: 1
[    3.630174] cpuidle: enable-method property 'qcom,kpss-acc-v1' found operations
[    3.631124] cpuidle: enable-method property 'qcom,kpss-acc-v1' found operations
[    3.638904] sdhci: Secure Digital Host Controller Interface driver
[    3.645622] sdhci: Copyright(c) Pierre Ossman
[    3.651800] sdhci-pltfm: SDHCI platform and OF driver helper
[    3.657637] NET: Registered protocol family 10
[    3.663363] Segment Routing with IPv6
[    3.666397] NET: Registered protocol family 17
[    3.669998] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    3.675006] 8021q: 802.1Q VLAN Support v1.8
[    3.687564] Registering SWP/SWPB emulation handler
[    3.704033] qcom_rpm 108000.rpm: RPM firmware 3.0.16777342
[    3.715671] s1a: supplied by regulator-dummy
[    3.715763] s1a: Bringing 0uV into 1050000-1050000uV
[    3.719265] s1b: supplied by regulator-dummy
[    3.724107] s1b: Bringing 0uV into 1050000-1050000uV
[    3.728457] s2a: supplied by regulator-dummy
[    3.733204] s2a: Bringing 0uV into 800000-800000uV
[    3.737740] s2b: supplied by regulator-dummy
[    3.742073] s2b: Bringing 0uV into 800000-800000uV
[    3.750428] ubi0: attaching mtd16
[    4.035651] ubi0: scanning is finished
[    4.045341] ubi0: attached mtd16 (name "rootfs2", size 37 MiB)
[    4.045365] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[    4.050058] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[    4.056998] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[    4.063826] ubi0: good PEBs: 296, bad PEBs: 0, corrupted PEBs: 0
[    4.070543] ubi0: user volume: 2, internal volumes: 1, max. volumes count: 128
[    4.076862] ubi0: max/mean erase counter: 3/1, WL threshold: 4096, image sequence number: 2129532265
[    4.083884] ubi0: available PEBs: 0, total reserved PEBs: 296, PEBs reserved for bad PEB handling: 20
[    4.093207] ubi0: background thread "ubi_bgt0d" started, PID 59
[    4.093923] block ubiblock0_0: created from ubi0:0(rootfs)
[    4.108032] ubiblock: device ubiblock0_0 (rootfs) set to be root filesystem
[    4.113516] hctosys: unable to open rtc device (rtc0)
[    4.128991] VFS: Mounted root (squashfs filesystem) readonly on device 254:0.
[    4.131996] Freeing unused kernel memory: 1024K
[    4.708612] init: Console is alive
[    4.708760] init: - watchdog -
[    5.396242] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    5.430283] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    5.431993] SCSI subsystem initialized
[    5.437836] libata version 3.00 loaded.
[    5.439764] ahci 29000000.sata: forcing port_map 0x0 -> 0x1
[    5.439813] ahci 29000000.sata: AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl platform mode
[    5.445000] ahci 29000000.sata: flags: ncq sntf pm led clo only pmp pio slum part ccc apst 
[    5.454147] scsi host0: ahci
[    5.461859] ata1: SATA max UDMA/133 mmio [mem 0x29000000-0x2900017f] port 0x100 irq 26
[    5.795867] ata1: SATA link down (SStatus 0 SControl 300)
[    5.797591] ehci-platform: EHCI generic platform driver
[    5.803200] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    5.805820] ohci-platform: OHCI generic platform driver
[    5.941534] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    5.941633] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
[    5.946416] xhci-hcd xhci-hcd.0.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
[    5.953588] xhci-hcd xhci-hcd.0.auto: irq 138, io mem 0x11000000
[    5.963717] hub 1-0:1.0: USB hub found
[    5.969124] hub 1-0:1.0: 1 port detected
[    5.972954] xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
[    5.976827] xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
[    5.982039] xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0  SuperSpeed
[    5.989746] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    5.996580] hub 2-0:1.0: USB hub found
[    6.004607] hub 2-0:1.0: 1 port detected
[    6.008319] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    6.012162] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 3
[    6.017663] xhci-hcd xhci-hcd.1.auto: hcc params 0x0228f065 hci version 0x100 quirks 0x0000000002010010
[    6.025109] xhci-hcd xhci-hcd.1.auto: irq 139, io mem 0x10000000
[    6.034826] hub 3-0:1.0: USB hub found
[    6.040650] hub 3-0:1.0: 1 port detected
[    6.044463] xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
[    6.048269] xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 4
[    6.053568] xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0  SuperSpeed
[    6.061275] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
[    6.068110] hub 4-0:1.0: USB hub found
[    6.076153] hub 4-0:1.0: 1 port detected
[    6.080785] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    6.085852] init: - preinit -
[    7.972093] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[    7.974446] dwmac1000: Master AXI performs any burst length
[    7.981021] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[    7.986776] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
[    7.996785] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    9.033822] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[    9.033920] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   11.170849] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 155
[   11.221601] UBIFS (ubi0:1): recovery needed
[   11.359095] UBIFS (ubi0:1): recovery completed
[   11.359272] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[   11.362448] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   11.370456] UBIFS (ubi0:1): FS size: 30093312 bytes (28 MiB, 237 LEBs), journal size 1523712 bytes (1 MiB, 12 LEBs)
[   11.380363] UBIFS (ubi0:1): reserved for root: 1421380 bytes (1388 KiB)
[   11.390564] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID C63A4B12-97D5-4A95-988D-129D5AE8F3EB, small LPT model
[   11.415875] mount_root: switching to ubifs overlay
[   11.435875] urandom-seed: Seeding with /etc/urandom.seed
[   11.545958] procd: - early -
[   11.546116] procd: - watchdog -
[   12.262007] procd: - watchdog -
[   12.262529] procd: - ubus -
[   12.355057] procd: - init -
[   12.640277] urngd: v1.0.2 started.
[   12.734852] kmodloader: loading kernel modules from /etc/modules.d/*
[   12.756673] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   12.760000] Mirror/redirect action on
[   12.765340] u32 classifier
[   12.765363]     input device check on
[   12.767292]     Actions configured
[   12.779374] nf_conntrack version 0.5.0 (8192 buckets, 32768 max)
[   12.787155] Loading modules backported from Linux version v4.19.137-0-gc076c79e03c6
[   12.787179] Backport generated by backports.git v4.19.137-1-0-g60c3a249
[   12.794599] ip_tables: (C) 2000-2006 Netfilter Core Team
[   12.827834] xt_time: kernel timezone is -0000
[   12.846087] PPP generic driver version 2.4.2
[   12.846617] NET: Registered protocol family 24
[   12.856408] ath10k_pci 0000:01:00.0: assign IRQ: got 68
[   12.856432] ath10k 4.19 driver, optimized for CT firmware, probing pci device: 0x40.
[   12.856849] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
[   12.863307] ath10k_pci 0000:01:00.0: enabling bus mastering
[   12.863936] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   13.038108] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/fwcfg-pci-0000:01:00.0.txt failed with error -2
[   13.038160] ath10k_pci 0000:01:00.0: Falling back to user helper
[   13.075746] firmware ath10k!fwcfg-pci-0000:01:00.0.txt: firmware_loading_store: map pages failed
[   13.079119] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/ct-firmware-5.bin failed with error -2
[   13.083620] ath10k_pci 0000:01:00.0: Falling back to user helper
[   13.117610] firmware ath10k!QCA99X0!hw2.0!ct-firmware-5.bin: firmware_loading_store: map pages failed
[   13.117858] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/ct-firmware-2.bin failed with error -2
[   13.125910] ath10k_pci 0000:01:00.0: Falling back to user helper
[   13.154257] firmware ath10k!QCA99X0!hw2.0!ct-firmware-2.bin: firmware_loading_store: map pages failed
[   13.154437] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/firmware-6.bin failed with error -2
[   13.162463] ath10k_pci 0000:01:00.0: Falling back to user helper
[   13.191103] firmware ath10k!QCA99X0!hw2.0!firmware-6.bin: firmware_loading_store: map pages failed
[   13.504705] ath10k_pci 0000:01:00.0: qca99x0 hw2.0 target 0x01000000 chip_id 0x003801ff sub 168c:0002
[   13.504756] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[   13.516978] ath10k_pci 0000:01:00.0: firmware ver 10.4b-ct-9980-fW-13-10af6a005 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 4a0fa880
[   13.586224] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 1:2 crc32 08fa09f2
[   14.776839] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16  peers: 48  tid: 96
[   14.776872] ath10k_pci 0000:01:00.0: msdu-desc: 2500  skid: 32
[   14.854795] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186  msdu-desc: 2500  sw-crypt: 0 ct-sta: 0'
[   14.855585] ath10k_pci 0000:01:00.0: wmi print 'free: 31080 iram: 23092 sram: 9596'
[   15.052773] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[   15.202552] ath: EEPROM regdomain: 0x0
[   15.202566] ath: EEPROM indicates default country code should be used
[   15.202576] ath: doing EEPROM country->regdmn map search
[   15.202591] ath: country maps to regdmn code: 0x3a
[   15.202602] ath: Country alpha2 being used: US
[   15.202610] ath: Regpair used: 0x3a
[   15.207102] ath10k_pci 0001:01:00.0: assign IRQ: got 101
[   15.207136] ath10k 4.19 driver, optimized for CT firmware, probing pci device: 0x40.
[   15.207988] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
[   15.214147] ath10k_pci 0001:01:00.0: enabling bus mastering
[   15.214723] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   15.385363] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/fwcfg-pci-0001:01:00.0.txt failed with error -2
[   15.385398] ath10k_pci 0001:01:00.0: Falling back to user helper
[   15.539998] firmware ath10k!fwcfg-pci-0001:01:00.0.txt: firmware_loading_store: map pages failed
[   15.542209] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/ct-firmware-5.bin failed with error -2
[   15.547963] ath10k_pci 0001:01:00.0: Falling back to user helper
[   15.597070] firmware ath10k!QCA99X0!hw2.0!ct-firmware-5.bin: firmware_loading_store: map pages failed
[   15.597256] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/ct-firmware-2.bin failed with error -2
[   15.605365] ath10k_pci 0001:01:00.0: Falling back to user helper
[   15.643800] firmware ath10k!QCA99X0!hw2.0!ct-firmware-2.bin: firmware_loading_store: map pages failed
[   15.643997] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA99X0/hw2.0/firmware-6.bin failed with error -2
[   15.652008] ath10k_pci 0001:01:00.0: Falling back to user helper
[   15.684047] firmware ath10k!QCA99X0!hw2.0!firmware-6.bin: firmware_loading_store: map pages failed
[   15.684213] ath10k_pci 0001:01:00.0: qca99x0 hw2.0 target 0x01000000 chip_id 0x003801ff sub 168c:0002
[   15.691909] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[   15.703797] ath10k_pci 0001:01:00.0: firmware ver 10.4b-ct-9980-fW-13-10af6a005 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 4a0fa880
[   15.773937] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 1:1 crc32 08fa09f2
[   16.960105] ath10k_pci 0001:01:00.0: 10.4 wmi init: vdevs: 16  peers: 48  tid: 96
[   16.960134] ath10k_pci 0001:01:00.0: msdu-desc: 2500  skid: 32
[   17.036098] ath10k_pci 0001:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186  msdu-desc: 2500  sw-crypt: 0 ct-sta: 0'
[   17.036865] ath10k_pci 0001:01:00.0: wmi print 'free: 31080 iram: 23092 sram: 9596'
[   17.249487] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1
[   17.373982] ath: EEPROM regdomain: 0x0
[   17.373994] ath: EEPROM indicates default country code should be used
[   17.374004] ath: doing EEPROM country->regdmn map search
[   17.374019] ath: country maps to regdmn code: 0x3a
[   17.374032] ath: Country alpha2 being used: US
[   17.374041] ath: Regpair used: 0x3a
[   17.384151] kmodloader: done loading kernel modules from /etc/modules.d/*
[   20.751437] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   20.752231] dwmac1000: Master AXI performs any burst length
[   20.760457] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[   20.765875] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
[   20.774913] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   20.783992] br-lan: port 1(eth0.1) entered blocking state
[   20.787496] br-lan: port 1(eth0.1) entered disabled state
[   20.793128] device eth0.1 entered promiscuous mode
[   20.798414] device eth0 entered promiscuous mode
[   20.804756] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   20.815585] IPv6: ADDRCONF(NETDEV_UP): eth0.2: link is not ready
[   21.833787] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   21.833837] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   21.851508] br-lan: port 1(eth0.1) entered blocking state
[   21.851542] br-lan: port 1(eth0.1) entered forwarding state
[   21.856170] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.2: link becomes ready
[   21.874027] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready

logread collected after the run:

[...]
Wed Oct 14 16:10:54 2020 user.notice SQM: Starting SQM script: simple.qos on eth0.2, in: 800000 Kbps, out: 800000 Kbps
Wed Oct 14 16:10:54 2020 daemon.notice procd: /etc/rc.d/S50sqm: SQM: Starting SQM script: simple.qos on eth0.2, in: 800000 Kbps, out: 800000 Kbps
Wed Oct 14 16:10:54 2020 user.notice SQM: Using generic sqm_start_default function.
Wed Oct 14 16:10:54 2020 daemon.notice procd: /etc/rc.d/S50sqm: SQM: Using generic sqm_start_default function.
Wed Oct 14 16:10:55 2020 user.notice SQM: simple.qos was started on eth0.2 successfully
Wed Oct 14 16:10:55 2020 daemon.notice procd: /etc/rc.d/S50sqm: SQM: simple.qos was started on eth0.2 successfully
Wed Oct 14 16:10:55 2020 daemon.err odhcpd[884]: Failed to send to ff02::1%lan@br-lan (Address not available)
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Network device 'eth0' link is up
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.833787] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.833837] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.851508] br-lan: port 1(eth0.1) entered blocking state
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.851542] br-lan: port 1(eth0.1) entered forwarding state
Wed Oct 14 16:10:55 2020 daemon.notice netifd: VLAN 'eth0.1' link is up
Wed Oct 14 16:10:55 2020 daemon.notice netifd: VLAN 'eth0.2' link is up
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'wan' has link connectivity
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'wan' is setting up now
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'wan6' has link connectivity
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'wan6' is setting up now
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.856170] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.2: link becomes ready
Wed Oct 14 16:10:55 2020 daemon.notice netifd: bridge 'br-lan' link is up
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'lan' has link connectivity
Wed Oct 14 16:10:55 2020 kern.info kernel: [   21.874027] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
Wed Oct 14 16:10:55 2020 daemon.notice netifd: wan (2113): udhcpc: started, v1.30.1
Wed Oct 14 16:10:55 2020 daemon.err odhcp6c[2116]: Failed to send RS (Address not available)
Wed Oct 14 16:10:55 2020 user.notice firewall: Reloading firewall due to ifup of lan (br-lan)
Wed Oct 14 16:10:55 2020 daemon.notice netifd: wan (2113): udhcpc: sending discover
Wed Oct 14 16:10:55 2020 daemon.notice netifd: wan (2113): udhcpc: sending select for 10.0.0.189
Wed Oct 14 16:10:55 2020 daemon.notice netifd: wan (2113): udhcpc: lease of 10.0.0.189 obtained, lease time 3600
Wed Oct 14 16:10:55 2020 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/dhcp
Wed Oct 14 16:10:55 2020 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/radvd
Wed Oct 14 16:10:55 2020 user.notice ucitrack: Setting up /etc/config/wireless reload dependency on /etc/config/network
Wed Oct 14 16:10:55 2020 daemon.notice netifd: Interface 'wan' is now up
Wed Oct 14 16:10:55 2020 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/luci-splash
Wed Oct 14 16:10:55 2020 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/qos
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/miniupnpd
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/sqm
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/dhcp reload dependency on /etc/config/odhcpd
Wed Oct 14 16:10:56 2020 user.notice SQM: Stopping SQM on eth0.2
Wed Oct 14 16:10:56 2020 daemon.err odhcp6c[2116]: Failed to send DHCPV6 message to ff02::1:2 (Address not available)
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up non-init /etc/config/fstab reload handler: /sbin/block mount
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/system reload trigger for non-procd /etc/init.d/led
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/system reload dependency on /etc/config/luci_statistics
Wed Oct 14 16:10:56 2020 user.notice ucitrack: Setting up /etc/config/system reload dependency on /etc/config/dhcp
Wed Oct 14 16:10:56 2020 user.notice SQM: Starting SQM script: simple.qos on eth0.2, in: 800000 Kbps, out: 800000 Kbps
Wed Oct 14 16:10:56 2020 user.notice SQM: Using generic sqm_start_default function.
Wed Oct 14 16:10:56 2020 daemon.notice procd: /etc/rc.d/S99bootcount: Boot count sucessfully reset to zero.
Wed Oct 14 16:10:56 2020 daemon.info procd: - init complete -
Wed Oct 14 16:10:57 2020 daemon.err odhcp6c[2116]: Failed to send DHCPV6 message to ff02::1:2 (Address not available)
Wed Oct 14 16:10:57 2020 user.notice SQM: simple.qos was started on eth0.2 successfully
Wed Oct 14 16:10:57 2020 user.notice firewall: Reloading firewall due to ifup of wan (eth0.2)
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[755]: exiting on receipt of SIGTERM
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: started, version 2.80 cachesize 150
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: DNS service limited to local subnets
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify dumpfile
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq-dhcp[3548]: DHCP, IP range 192.168.1.100 -- 192.168.1.249, lease time 12h
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain test
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain onion
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain localhost
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain local
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain invalid
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain bind
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: using local addresses only for domain lan
Wed Oct 14 16:10:59 2020 daemon.warn dnsmasq[3548]: no servers found in /tmp/resolv.conf.auto, will retry
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: read /etc/hosts - 4 addresses
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq-dhcp[3548]: read /etc/ethers - 0 addresses
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3548]: exiting on receipt of SIGTERM
Wed Oct 14 16:10:59 2020 daemon.err procd: Got unexpected signal 1
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: started, version 2.80 cachesize 150
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: DNS service limited to local subnets
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-auth no-DNSSEC no-ID loop-detect inotify dumpfile
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq-dhcp[3623]: DHCP, IP range 192.168.1.100 -- 192.168.1.249, lease time 12h
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain test
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain onion
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain localhost
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain local
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain invalid
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain bind
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: using local addresses only for domain lan
Wed Oct 14 16:10:59 2020 daemon.warn dnsmasq[3623]: no servers found in /tmp/resolv.conf.auto, will retry
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: read /etc/hosts - 4 addresses
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq[3623]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Wed Oct 14 16:10:59 2020 daemon.info dnsmasq-dhcp[3623]: read /etc/ethers - 0 addresses
Wed Oct 14 16:11:06 2020 daemon.info dnsmasq[3623]: read /etc/hosts - 4 addresses
Wed Oct 14 16:11:06 2020 daemon.info dnsmasq[3623]: read /tmp/hosts/odhcpd - 1 addresses
Wed Oct 14 16:11:06 2020 daemon.info dnsmasq[3623]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Wed Oct 14 16:11:06 2020 daemon.info dnsmasq-dhcp[3623]: read /etc/ethers - 0 addresses
Wed Oct 14 16:11:12 2020 daemon.info dnsmasq[3623]: read /etc/hosts - 4 addresses
Wed Oct 14 16:11:12 2020 daemon.info dnsmasq[3623]: read /tmp/hosts/odhcpd - 2 addresses
Wed Oct 14 16:11:12 2020 daemon.info dnsmasq[3623]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses
Wed Oct 14 16:11:12 2020 daemon.info dnsmasq-dhcp[3623]: read /etc/ethers - 0 addresses
Wed Oct 14 16:11:16 2020 daemon.info dnsmasq-dhcp[3623]: DHCPDISCOVER(br-lan) 192.168.1.225 aa:aa:aa:bb:bb:bb
Wed Oct 14 16:11:16 2020 daemon.info dnsmasq-dhcp[3623]: DHCPOFFER(br-lan) 192.168.1.225 aa:aa:aa:bb:bb:bb
Wed Oct 14 16:11:16 2020 daemon.info dnsmasq-dhcp[3623]: DHCPREQUEST(br-lan) 192.168.1.225 aa:aa:aa:bb:bb:bb
Wed Oct 14 16:11:16 2020 daemon.info dnsmasq-dhcp[3623]: DHCPACK(br-lan) 192.168.1.225 aa:aa:aa:bb:bb:bb arch-machine-0-c2d6120c765ec16a
Wed Oct 14 16:12:03 2020 authpriv.info dropbear[3645]: Child connection from 192.168.1.10:53296
Wed Oct 14 16:12:03 2020 authpriv.notice dropbear[3645]: Pubkey auth succeeded for 'root' with key sha1!! 99:3a:ca:0f:e7:c9:92:b7:1b:3a:c8:40:b1:7a:cb:cc:35:39:74:21 from 192.168.1.10:53296

I would recommend to switch to the performance governor or configure the ondemand one to be less twitchy. The issue seems to be that cake is a bursty load, that on average does not keep the CPU busy enough to make the ondemand governor keeping the CPU in high frequency mode, but is latency sensitive enough, that the shaper does not get enough CPU cycles at lower frequencies (but is too short to profit from the increasing frequencies once the CPU scales up again).

Unfortunately, neither HTB nor fq_codel offer as nice statistics output as cake does (I believe that tc -d qdisc might give some more detail for fq_codel).

Mmmh, 21% is not terrible, if that is the absolute minimum, but top -d 1 samples in 1 second intervals, and 21% idle might be any 210milliseconds in that interval and cake is sufficient latency sensitive that not getting enough CPU for 790ms will show either in decreased throughput or increased latency under load or both... that would call for a closer monitoring of CPU load (which in itself will cause a noticeable CPU load)...

Thanks for all the data.

I've tried another iperf3 run with the performance governor.

# cat /sys/devices/system/cpu/cpufreq/policy?/scaling_governor
performance
performance

The iperf3 results aren't significantly different.

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-60.00  sec  4.57 GBytes   654 Mbits/sec  1925             sender
[  5][TX-C]   0.00-60.00  sec  4.57 GBytes   654 Mbits/sec                  receiver
[  7][RX-C]   0.00-60.00  sec   659 MBytes  92.1 Mbits/sec  422             sender
[  7][RX-C]   0.00-60.00  sec   656 MBytes  91.7 Mbits/sec                  receiver

Here's tc -d qdisc after this run:

# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth0.1 root refcnt 2 
qdisc htb 1: dev eth0.2 root refcnt 2 r2q 10 default 0x12 direct_packets_stat 0 ver 3.17 direct_qlen 1000
qdisc fq_codel 120: dev eth0.2 parent 1:12 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 130: dev eth0.2 parent 1:13 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 110: dev eth0.2 parent 1:11 limit 1001p flows 1024 quantum 300 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ---------------- 
qdisc htb 1: dev ifb4eth0.2 root refcnt 2 r2q 10 default 0x10 direct_packets_stat 0 ver 3.17 direct_qlen 32
qdisc fq_codel 110: dev ifb4eth0.2 parent 1:10 limit 1001p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn 

A bit later, and after another run, here are the cpufreq stats:

root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/time_in_state 
384000 956
600000 102
800000 64
1000000 96
1200000 76
1400000 221166
root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/policy1/stats/time_in_state 
384000 838
600000 90
800000 57
1000000 98
1200000 81
1400000 222070

I'm set up to measure current at the 12VDC input. With the performance governor, I'm seeing 389mA at idle and 485mA while running a bidirectional iperf3 test. The ondemand numbers are almost the same, which is not what I would expect.

This is another iperf3 run, still with the performance governor, and comparing the client and server output is interesting. The server reports sending 0 bytes for long stretches of time.

Client output:

$ iperf3 -c 10.0.0.1 --bidir --time 60 | tee iperf3-output.txt
Connecting to host 10.0.0.1, port 5201
[  5] local 192.168.1.225 port 36082 connected to 10.0.0.1 port 5201
[  7] local 192.168.1.225 port 36084 connected to 10.0.0.1 port 5201
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][TX-C]   0.00-1.00   sec  76.3 MBytes   640 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   0.00-1.00   sec  11.2 MBytes  94.3 Mbits/sec                  
[  5][TX-C]   1.00-2.00   sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   1.00-2.00   sec  60.8 KBytes   498 Kbits/sec                  
[  5][TX-C]   2.00-3.00   sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   2.00-3.00   sec   180 KBytes  1.47 Mbits/sec                  
[  5][TX-C]   3.00-4.00   sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   3.00-4.00   sec   810 KBytes  6.64 Mbits/sec                  
[  5][TX-C]   4.00-5.00   sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   4.00-5.00   sec   214 KBytes  1.75 Mbits/sec                  
[  5][TX-C]   5.00-6.00   sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   5.00-6.00   sec   344 KBytes  2.81 Mbits/sec                  
[  5][TX-C]   6.00-7.00   sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   6.00-7.00   sec   270 KBytes  2.21 Mbits/sec                  
[  5][TX-C]   7.00-8.00   sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   7.00-8.00   sec   257 KBytes  2.11 Mbits/sec                  
[  5][TX-C]   8.00-9.00   sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   8.00-9.00   sec   266 KBytes  2.18 Mbits/sec                  
[  5][TX-C]   9.00-10.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]   9.00-10.00  sec   266 KBytes  2.18 Mbits/sec                  
[  5][TX-C]  10.00-11.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  10.00-11.00  sec   264 KBytes  2.17 Mbits/sec                  
[  5][TX-C]  11.00-12.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  11.00-12.00  sec   263 KBytes  2.15 Mbits/sec                  
[  5][TX-C]  12.00-13.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  12.00-13.00  sec   256 KBytes  2.10 Mbits/sec                  
[  5][TX-C]  13.00-14.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  13.00-14.00  sec   252 KBytes  2.06 Mbits/sec                  
[  5][TX-C]  14.00-15.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  14.00-15.00  sec   249 KBytes  2.04 Mbits/sec                  
[  5][TX-C]  15.00-16.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  15.00-16.00  sec   283 KBytes  2.32 Mbits/sec                  
[  5][TX-C]  16.00-17.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  16.00-17.00  sec   269 KBytes  2.20 Mbits/sec                  
[  5][TX-C]  17.00-18.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  17.00-18.00  sec   270 KBytes  2.21 Mbits/sec                  
[  5][TX-C]  18.00-19.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  18.00-19.00  sec   276 KBytes  2.26 Mbits/sec                  
[  5][TX-C]  19.00-20.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  19.00-20.00  sec   256 KBytes  2.10 Mbits/sec                  
[  5][TX-C]  20.00-21.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  20.00-21.00  sec   272 KBytes  2.22 Mbits/sec                  
[  5][TX-C]  21.00-22.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  21.00-22.00  sec   255 KBytes  2.09 Mbits/sec                  
[  5][TX-C]  22.00-23.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  22.00-23.00  sec   262 KBytes  2.14 Mbits/sec                  
[  5][TX-C]  23.00-24.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  23.00-24.00  sec   253 KBytes  2.07 Mbits/sec                  
[  5][TX-C]  24.00-25.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  24.00-25.00  sec   260 KBytes  2.13 Mbits/sec                  
[  5][TX-C]  25.00-26.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  25.00-26.00  sec   250 KBytes  2.05 Mbits/sec                  
[  5][TX-C]  26.00-27.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  26.00-27.00  sec   255 KBytes  2.08 Mbits/sec                  
[  5][TX-C]  27.00-28.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  27.00-28.00  sec   277 KBytes  2.27 Mbits/sec                  
[  5][TX-C]  28.00-29.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  28.00-29.00  sec   279 KBytes  2.28 Mbits/sec                  
[  5][TX-C]  29.00-30.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  29.00-30.00  sec   255 KBytes  2.09 Mbits/sec                  
[  5][TX-C]  30.00-31.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  30.00-31.00  sec   272 KBytes  2.22 Mbits/sec                  
[  5][TX-C]  31.00-32.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  31.00-32.00  sec   272 KBytes  2.22 Mbits/sec                  
[  5][TX-C]  32.00-33.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  32.00-33.00  sec   255 KBytes  2.09 Mbits/sec                  
[  5][TX-C]  33.00-34.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  33.00-34.00  sec   264 KBytes  2.17 Mbits/sec                  
[  5][TX-C]  34.00-35.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  34.00-35.00  sec   257 KBytes  2.11 Mbits/sec                  
[  5][TX-C]  35.00-36.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  35.00-36.00  sec   262 KBytes  2.14 Mbits/sec                  
[  5][TX-C]  36.00-37.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  36.00-37.00  sec   230 KBytes  1.89 Mbits/sec                  
[  5][TX-C]  37.00-38.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  37.00-38.00  sec   269 KBytes  2.20 Mbits/sec                  
[  5][TX-C]  38.00-39.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  38.00-39.00  sec   277 KBytes  2.27 Mbits/sec                  
[  5][TX-C]  39.00-40.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  39.00-40.00  sec   205 KBytes  1.68 Mbits/sec                  
[  5][TX-C]  40.00-41.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  40.00-41.00  sec   199 KBytes  1.63 Mbits/sec                  
[  5][TX-C]  41.00-42.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  41.00-42.00  sec   273 KBytes  2.24 Mbits/sec                  
[  5][TX-C]  42.00-43.00  sec  92.5 MBytes   776 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  42.00-43.00  sec   269 KBytes  2.20 Mbits/sec                  
[  5][TX-C]  43.00-44.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  43.00-44.00  sec   269 KBytes  2.20 Mbits/sec                  
[  5][TX-C]  44.00-45.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  44.00-45.00  sec   257 KBytes  2.11 Mbits/sec                  
[  5][TX-C]  45.00-46.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  45.00-46.00  sec   263 KBytes  2.15 Mbits/sec                  
[  5][TX-C]  46.00-47.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  46.00-47.00  sec   263 KBytes  2.15 Mbits/sec                  
[  5][TX-C]  47.00-48.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  47.00-48.00  sec   256 KBytes  2.10 Mbits/sec                  
[  5][TX-C]  48.00-49.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  48.00-49.00  sec   260 KBytes  2.13 Mbits/sec                  
[  5][TX-C]  49.00-50.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  49.00-50.00  sec   277 KBytes  2.27 Mbits/sec                  
[  5][TX-C]  50.00-51.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  50.00-51.00  sec   240 KBytes  1.97 Mbits/sec                  
[  5][TX-C]  51.00-52.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  51.00-52.00  sec   277 KBytes  2.27 Mbits/sec                  
[  5][TX-C]  52.00-53.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  52.00-53.00  sec   276 KBytes  2.26 Mbits/sec                  
[  5][TX-C]  53.00-54.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  53.00-54.00  sec   266 KBytes  2.18 Mbits/sec                  
[  5][TX-C]  54.00-55.00  sec  92.5 MBytes   776 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  54.00-55.00  sec   269 KBytes  2.20 Mbits/sec                  
[  5][TX-C]  55.00-56.00  sec  90.0 MBytes   755 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  55.00-56.00  sec   264 KBytes  2.17 Mbits/sec                  
[  5][TX-C]  56.00-57.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  56.00-57.00  sec   259 KBytes  2.12 Mbits/sec                  
[  5][TX-C]  57.00-58.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  57.00-58.00  sec   255 KBytes  2.09 Mbits/sec                  
[  5][TX-C]  58.00-59.00  sec  91.2 MBytes   766 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  58.00-59.00  sec   263 KBytes  2.15 Mbits/sec                  
[  5][TX-C]  59.00-60.00  sec  91.2 MBytes   765 Mbits/sec    0   3.01 MBytes       
[  7][RX-C]  59.00-60.00  sec   276 KBytes  2.26 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-60.00  sec  5.32 GBytes   762 Mbits/sec    0             sender
[  5][TX-C]   0.00-60.00  sec  5.32 GBytes   761 Mbits/sec                  receiver
[  7][RX-C]   0.00-60.00  sec  29.5 MBytes  4.12 Mbits/sec  1021             sender
[  7][RX-C]   0.00-60.00  sec  26.6 MBytes  3.72 Mbits/sec                  receiver

iperf Done.

Server output:

Accepted connection from 10.0.0.189, port 36080
[  5] local 10.0.0.1 port 5201 connected to 10.0.0.189 port 36082
[  8] local 10.0.0.1 port 5201 connected to 10.0.0.189 port 36084
[ ID][Role] Interval           Transfer     Bitrate         Retr  Cwnd
[  5][RX-S]   0.00-1.00   sec  75.9 MBytes   637 Mbits/sec                  
[  8][TX-S]   0.00-1.00   sec  13.7 MBytes   115 Mbits/sec   40    276 KBytes       
[  5][RX-S]   1.00-2.00   sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec  100    195 KBytes       
[  5][RX-S]   2.00-3.00   sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec  125   18.4 KBytes       
[  5][RX-S]   3.00-4.00   sec  90.5 MBytes   759 Mbits/sec                  
[  8][TX-S]   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec  145    130 KBytes       
[  5][RX-S]   4.00-5.00   sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec   98   43.8 KBytes       
[  5][RX-S]   5.00-6.00   sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec   31   7.07 KBytes       
[  5][RX-S]   6.00-7.00   sec  90.8 MBytes   762 Mbits/sec                  
[  8][TX-S]   6.00-7.00   sec  1.11 MBytes  9.32 Mbits/sec   14   4.24 KBytes       
[  5][RX-S]   7.00-8.00   sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec   12   5.66 KBytes       
[  5][RX-S]   8.00-9.00   sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    7   11.3 KBytes       
[  5][RX-S]   9.00-10.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec   12   7.07 KBytes       
[  5][RX-S]  10.00-11.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  10.00-11.00  sec  1.12 MBytes  9.38 Mbits/sec    8   7.07 KBytes       
[  5][RX-S]  11.00-12.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  11.00-12.00  sec  0.00 Bytes  0.00 bits/sec    7   11.3 KBytes       
[  5][RX-S]  12.00-13.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  12.00-13.00  sec  0.00 Bytes  0.00 bits/sec    9   8.48 KBytes       
[  5][RX-S]  13.00-14.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  13.00-14.00  sec  0.00 Bytes  0.00 bits/sec   10   5.66 KBytes       
[  5][RX-S]  14.00-15.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  14.00-15.00  sec  1.12 MBytes  9.38 Mbits/sec    6   7.07 KBytes       
[  5][RX-S]  15.00-16.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec   10   5.66 KBytes       
[  5][RX-S]  16.00-17.00  sec  91.0 MBytes   764 Mbits/sec                  
[  8][TX-S]  16.00-17.00  sec  0.00 Bytes  0.00 bits/sec   11   7.07 KBytes       
[  5][RX-S]  17.00-18.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec   10   7.07 KBytes       
[  5][RX-S]  18.00-19.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  18.00-19.00  sec  1.12 MBytes  9.38 Mbits/sec    9   7.07 KBytes       
[  5][RX-S]  19.00-20.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  19.00-20.00  sec  0.00 Bytes  0.00 bits/sec   10   7.07 KBytes       
[  5][RX-S]  20.00-21.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  20.00-21.00  sec  0.00 Bytes  0.00 bits/sec    8   7.07 KBytes       
[  5][RX-S]  21.00-22.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  21.00-22.00  sec  0.00 Bytes  0.00 bits/sec    7   5.66 KBytes       
[  5][RX-S]  22.00-23.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  22.00-23.00  sec  0.00 Bytes  0.00 bits/sec    7   9.90 KBytes       
[  5][RX-S]  23.00-24.00  sec  91.3 MBytes   765 Mbits/sec                  
[  8][TX-S]  23.00-24.00  sec  1.12 MBytes  9.38 Mbits/sec   11   5.66 KBytes       
[  5][RX-S]  24.00-25.00  sec  91.1 MBytes   765 Mbits/sec                  
[  8][TX-S]  24.00-25.00  sec  0.00 Bytes  0.00 bits/sec    7   7.07 KBytes       
[  5][RX-S]  25.00-26.00  sec  91.0 MBytes   764 Mbits/sec                  
[  8][TX-S]  25.00-26.00  sec  0.00 Bytes  0.00 bits/sec    8   5.66 KBytes       
[  5][RX-S]  26.00-27.00  sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]  26.00-27.00  sec  0.00 Bytes  0.00 bits/sec    8   9.90 KBytes       
[  5][RX-S]  27.00-28.00  sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]  27.00-28.00  sec  1.12 MBytes  9.38 Mbits/sec    6   5.66 KBytes       
[  5][RX-S]  28.00-29.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  28.00-29.00  sec  0.00 Bytes  0.00 bits/sec   13   5.66 KBytes       
[  5][RX-S]  29.00-30.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  29.00-30.00  sec  0.00 Bytes  0.00 bits/sec   11   2.83 KBytes       
[  5][RX-S]  30.00-31.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  30.00-31.00  sec  0.00 Bytes  0.00 bits/sec    6   9.90 KBytes       
[  5][RX-S]  31.00-32.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  31.00-32.00  sec  0.00 Bytes  0.00 bits/sec    7   8.48 KBytes       
[  5][RX-S]  32.00-33.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  32.00-33.00  sec  1.12 MBytes  9.38 Mbits/sec   11   9.90 KBytes       
[  5][RX-S]  33.00-34.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  33.00-34.00  sec  0.00 Bytes  0.00 bits/sec    9   8.48 KBytes       
[  5][RX-S]  34.00-35.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  34.00-35.00  sec  0.00 Bytes  0.00 bits/sec    7   8.48 KBytes       
[  5][RX-S]  35.00-36.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  35.00-36.00  sec  0.00 Bytes  0.00 bits/sec    8   7.07 KBytes       
[  5][RX-S]  36.00-37.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  36.00-37.00  sec  1.12 MBytes  9.38 Mbits/sec    8   9.90 KBytes       
[  5][RX-S]  37.00-38.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  37.00-38.00  sec  0.00 Bytes  0.00 bits/sec   10   5.66 KBytes       
[  5][RX-S]  38.00-39.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  38.00-39.00  sec  0.00 Bytes  0.00 bits/sec   11   4.24 KBytes       
[  5][RX-S]  39.00-40.00  sec  90.8 MBytes   762 Mbits/sec                  
[  8][TX-S]  39.00-40.00  sec  0.00 Bytes  0.00 bits/sec    9   7.07 KBytes       
[  5][RX-S]  40.00-41.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  40.00-41.00  sec  0.00 Bytes  0.00 bits/sec    9   8.48 KBytes       
[  5][RX-S]  41.00-42.00  sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]  41.00-42.00  sec  1.12 MBytes  9.38 Mbits/sec    9   8.48 KBytes       
[  5][RX-S]  42.00-43.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  42.00-43.00  sec  0.00 Bytes  0.00 bits/sec   13   7.07 KBytes       
[  5][RX-S]  43.00-44.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  43.00-44.00  sec  0.00 Bytes  0.00 bits/sec    7   8.48 KBytes       
[  5][RX-S]  44.00-45.00  sec  91.1 MBytes   765 Mbits/sec                  
[  8][TX-S]  44.00-45.00  sec  0.00 Bytes  0.00 bits/sec    9   4.24 KBytes       
[  5][RX-S]  45.00-46.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  45.00-46.00  sec  1.12 MBytes  9.38 Mbits/sec    5   8.48 KBytes       
[  5][RX-S]  46.00-47.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  46.00-47.00  sec  0.00 Bytes  0.00 bits/sec   12   7.07 KBytes       
[  5][RX-S]  47.00-48.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  47.00-48.00  sec  0.00 Bytes  0.00 bits/sec    7   7.07 KBytes       
[  5][RX-S]  48.00-49.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  48.00-49.00  sec  0.00 Bytes  0.00 bits/sec    8   7.07 KBytes       
[  5][RX-S]  49.00-50.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  49.00-50.00  sec  1.12 MBytes  9.38 Mbits/sec   10   5.66 KBytes       
[  5][RX-S]  50.00-51.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  50.00-51.00  sec  0.00 Bytes  0.00 bits/sec   10   9.90 KBytes       
[  5][RX-S]  51.00-52.00  sec  91.0 MBytes   763 Mbits/sec                  
[  8][TX-S]  51.00-52.00  sec  0.00 Bytes  0.00 bits/sec   13   8.48 KBytes       
[  5][RX-S]  52.00-53.00  sec  90.9 MBytes   762 Mbits/sec                  
[  8][TX-S]  52.00-53.00  sec  0.00 Bytes  0.00 bits/sec    8   8.48 KBytes       
[  5][RX-S]  53.00-54.00  sec  91.0 MBytes   764 Mbits/sec                  
[  8][TX-S]  53.00-54.00  sec  0.00 Bytes  0.00 bits/sec    9   7.07 KBytes       
[  5][RX-S]  54.00-55.00  sec  90.9 MBytes   763 Mbits/sec                  
[  8][TX-S]  54.00-55.00  sec  1.12 MBytes  9.38 Mbits/sec    7   5.66 KBytes       
[  5][RX-S]  55.00-56.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  55.00-56.00  sec  0.00 Bytes  0.00 bits/sec    9   7.07 KBytes       
[  5][RX-S]  56.00-57.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  56.00-57.00  sec  0.00 Bytes  0.00 bits/sec    5   8.48 KBytes       
[  5][RX-S]  57.00-58.00  sec  91.2 MBytes   765 Mbits/sec                  
[  8][TX-S]  57.00-58.00  sec  0.00 Bytes  0.00 bits/sec    8   8.48 KBytes       
[  5][RX-S]  58.00-59.00  sec  91.1 MBytes   764 Mbits/sec                  
[  8][TX-S]  58.00-59.00  sec  1.12 MBytes  9.38 Mbits/sec    9   7.07 KBytes       
[  5][RX-S]  59.00-60.00  sec  91.0 MBytes   764 Mbits/sec                  
[  8][TX-S]  59.00-60.00  sec  0.00 Bytes  0.00 bits/sec    8   9.90 KBytes       
[  5][RX-S]  60.00-60.00  sec   317 KBytes   800 Mbits/sec                  
[  8][TX-S]  60.00-60.00  sec  0.00 Bytes  0.00 bits/sec    0   9.90 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][RX-S]   0.00-60.00  sec  5.32 GBytes   761 Mbits/sec                  receiver
[  8][TX-S]   0.00-60.00  sec  29.5 MBytes  4.12 Mbits/sec  1021             sender

Here are some new results from the multi-port NIC test setup. There are 5 different router models (plus a no-router, switch-only baseline), each tested with and without SQM (800Mbit/800Mbit) and with and without software flow offloading.

The upload, download, and bidir columns are the average results in Mbps from goben, a golang tool like iperf. The 4p means these were runs with 4 parallel streams. The http columns are for caddy serving a random file of the size given, and the hey tool retrieving the file for 60 seconds, with 50 workers, reporting the average requests per second.

The Atomic Pi (AAEON MF-001 in the table) and Pi 4 were tested with an RTL8153-based USB 3.0 gigabit adapter as a WAN link.

Run Router SQM SoftFlowOffload upload-4p download-4p bidir-up-4p bidir-dn-4p http-1KiB http-100KiB http-10000KiB
1604156365 no router, 8000 mtu both NICs NA OFF 986 984 982 982 28294.50 1203.19 12.04
1604153917 no router NA OFF 940 940 931 926 28228.82 1145.91 11.49
1603936584 AAEON MF-001 OFF ON 932 940 864 849 27849.07 1145.73 11.49
1604011594 AAEON MF-001 OFF OFF 932 939 796 875 27691.97 1145.84 11.49
1604020449 Raspberry Pi 4 Model B Rev 1.1 OFF ON 939 939 693 812 27986.27 1145.75 11.49
1604022831 Raspberry Pi 4 Model B Rev 1.1 OFF OFF 939 939 698 645 28002.81 1146.00 11.49
1604028557 Linksys EA8500 WiFi Router OFF ON 902 902 454 434 27697.64 996.42 9.11
1603934190 AAEON MF-001 ON OFF 764 764 739 744 26892.87 859.82 9.13
1604018070 Raspberry Pi 4 Model B Rev 1.1 ON OFF 764 764 688 572 27849.89 931.12 9.22
1604105001 Western Digital My Net N750 OFF ON 787 763 376 374 28126.49 766.55 7.73
1604030957 Linksys EA8500 WiFi Router OFF OFF 728 711 247 422 26018.58 573.10 5.62
1604065573 Buffalo WZR-HP-G300NH OFF ON 691 679 424 330 27875.83 758.39 7.56
1604026160 Linksys EA8500 WiFi Router ON OFF 608 409 570 0 19108.70 380.10 3.87
1604068022 Buffalo WZR-HP-G300NH OFF OFF 300 303 156 156 16340.22 324.24 3.23
1604107427 Western Digital My Net N750 OFF OFF 284 294 147 135 15386.65 317.91 3.16
1604102579 Western Digital My Net N750 ON OFF 140 148 99 61 8858.54 138.47 2.50

The old Buffalo router was not stable running this benchmark with SQM enabled. The Western Digital router also crashed once when running with SQM enabled. The EA8500 has a puzzling very poor upload rate in the bidirectional test with SQM.

I also collected DC power measurements on the Atomic Pi and Raspberry Pi 4; they are very similar, at about 5.8W while running a load test. The Atomic Pi actually draws a bit less power at idle: 4.4W vs 5.0W.

2 Likes