Terrible results using RPi4 + SQM to control buffer bloat... anyone else with this setup comment?

[amteza]

@darksky

@amteza

I can confirm it works. Connecting my computer directly to the RPi4 I can max out my gigabit connection and my upload bandwidth.

Can you share your SQM settings with me? How about your network setup? Are you using VLANs? Thanks!

Sure thing, disclaimer, I'm using @ldir SQM's Netify experimental branch. All my devices are connected by WiFi and DSCP marking aligns the best possible with WMM. divserv4 is utilised in my script.

My eth1 is a U300 USB NIC connected to my WAN (1000/50 Mbps). eth0 is internal NIC connected to my NanoHD.

Steering and irqbalance are active, no VLAN are defined and RPi4 is overclocked to 2 Ghz.

config queue 'eth1'
	option debug_logging '0'
	option verbosity '5'
	option qdisc 'cake'
	option linklayer 'ethernet'
	option overhead '44'
	option qdisc_advanced '1'
	option qdisc_really_really_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option ingress_ecn 'ECN'
	option egress_ecn 'ECN'
	option interface 'eth0'
	option upload '46000'
	option enabled '1'
	option download '950000'
	option iqdisc_opts 'nat ingress'
	option eqdisc_opts 'nat ack-filter docsis'
	option script 'ctinfo_4layercake.qos'

[darksky]

A bit that's over my head in your reply. Not sure what the Netify branch is (github link?) nor sure about DSCP or divserv4.

[amteza]

FYI, https://github.com/ldir-EDB0/sqm-scripts/tree/sqmqosnfa

What I meant is that I'm not using simply piece_of_cake, that's why I included the "disclaimer" for. I apologise for making it more confusing, I'll go back to my lockdown cave now.

[darksky]

@amteza

FYI, https://github.com/ldir-EDB0/sqm-scripts/tree/sqmqosnfa

Ah thanks. Are those within the sqm-scripts-extra package or something you add manually? What does it offer vs cake/piece_of_cake to the average user wanting to control upstream bufferbloat? You mentioned something about alignment with WMM and some other stuff.

[moeller0]

@darksky

Are those within the sqm-scripts-extra package or something you add manually?

Nope sqm-scripts-extra is essentially abandoned, sorry. Not enough stuff to put there.

[amteza]

@darksky

@amteza

FYI, https://github.com/ldir-EDB0/sqm-scripts/tree/sqmqosnfa

Ah thanks. Are those within the sqm-scripts-extra package or something you add manually? What does it offer vs cake/piece_of_cake to the average user wanting to control upstream bufferbloat? You mentioned something about alignment with WMM and some other stuff.

I didn't know about sqm-scripts-extra. This offers you flexibility to tag packets to specific priority buckets: Background, Best Effort, Video and Voice. In my household with home schooling and working from home due to constant COVID lockdowns this is very relevant. Is not easy to manage people gaming, streaming and video conferencing in parallel. This work from @ldir makes it easy. Netify helps identifying the different types of streams on the fly, works well enough, not perfect, tho'.

When I referred to WMM, I meant that I'm tagging packets aligning DSCP with WMM, so this works as best as possible over WiFi too.

Differentiated services

Wireless Multimedia Extensions

[dlakelan]

The packet tagging and WMM and such are all very interesting etc, but will not be factors related to how much CPU usage there is really. The issue is the use of an ASIX dongle because those drivers seem to cause a lot more interrupt handling. A simple fix is to replace with the UE300 from TP-Link for ~$15 it cuts interrupt overhead by a factor of 10 or more.

[darksky]

OK, I finally got a UE300 (rtl8150) and ran the waveform bufferbloat speed tests comparing it to the Trendnet (asix-ax88179).

The other difference is I am now running the RPi4 @ 2,000 MHz:

/boot/config.txt

# cat /boot/config.txt 
################################################################################
# Bootloader configuration - config.txt
################################################################################

################################################################################
# For overclocking and various other settings, see:
# https://www.raspberrypi.org/documentation/configuration/config-txt/README.md
################################################################################

# OpenWrt config
include distroconfig.txt

[all]
# Place your custom settings here.
over_voltage=6
arm_freq=2000

Conclusion: UE300 achieves approx 25% higher speeds with the more or less the CPU load. Bufferbloat is controlled in each case.

Trendnet

• CPU use: max 99+% on single core
• Latency: 19 ms unloaded, +0 ms download active, +0 ms upload active
• Speed: 670.5 Mbps down (average of two runs 672 and 669)

TP-Link UE300

• CPU use: max 98% on single core (did not saturate)
• Latency: 17 ms unloaded, +2 ms download active, +0 ms upload active
• Speed: 839 Mbps down (average of two runs 826 and 852)

Both tests used fq_codel/simplest_tbf with 850000 (download speed) and 24000 (upload speed). Packet steering was enabled for both.

As a control, no SQM (using Trendnet):

• CPU use: 40% max on single core
• Latency: 32 ms unloaded, +39 ms download active, +0 ms upload active
• Speed: 909 Mbps down and 24.4 up

[dlakelan]

Are you running irqbalance? Because that may help a lot. I can not only shape a gigabit, but I can do it over a squid proxy at the native clock freq. So there should be a lot of headroom on this device.

[darksky]

@dlakelan

Are you running irqbalance?

Yes, it is running. The issue is the SQM is single threaded. See this pic in post #7 of this thread.

[dlakelan]

It's strange because I managed to shape a gigabit with something like 10-15% of one core in the test thread. Now I was using hfsc but I don't expect cake to be 10x as cpu intense

[jrambo99]

@darksky Are you using snapshots?

There seems to be a regression in SQM performance lately:

OpenWrt 21.02.0 - First Stable Release

Everything is fine on my RPI 4, but i see a high CPU usage with SQM that i didn't have before. It's the same config, but now one of the cores is used at 90% with 600 Mbps. I had an snapshot before and i didn't remember any core going abobe 25-30%.

OpenWrt 21.02.0 - First Stable Release

is visible in both 21.02.0 and snapshots later than around r17300... (aka both kernels ~ and probably most targets) possibly; sqm-scripts: bump to v1.5.1 maybe try an older ipk from the rc4/3 repo..

OpenWrt 21.02.0 - First Stable Release

Yes, that fixes the problem. v1.5.0-2 [image] v1.5.1-1 [image] It can't even boost to full bandwith. Btw im ussing your script, that could be the problem too.

I guess this will get fixed at some point, but in the meantime you can try an older version of sqm-scripts as cited, that should get you results more in line with @dlakelan's

[darksky]

@jrambo99

Are you using snapshots?

Yes, built against HEAD. I guess I could downgrade to the commit from Dec of 2020.

History for net/sqm-scripts - openwrt/packages

[moeller0]

@darksky

Yes, built against HEAD. I guess I could downgrade to the commit from Dec of 2020.

If you do, please try to incrementally add the 3 commits between Dec 2020 and Feb 28 2021 to see which of these innocent looking changes is responsible for the observed CPU load.

[moeller0]

@jrambo99

I guess this will get fixed at some point,

Well, for fixing it we first need to locate the offending change and then understand why it fails...
My home router is still on release 19 (and I can not currently change that due to homeschooling/videoconferencing)

[darksky]

@moeller0

If you do, please try to incrementally add the 3 commits between Dec 2020 and Feb 28 2021 to see which of these innocent looking changes is responsible for the observed CPU load.

Well, there is really only one changing commit, the other is just updating the hash. So 1.51 is bad and we know from that other thread that 1.50 is good.

I'm thinking there is a repo hosting these packages? Is there one for rc1, rc2, etc. of 21.02.1?

EDIT:

https://downloads.openwrt.org/releases/21.02.0-rc1/targets/ipq806x/generic/packages/

Not sure where the sqm-scripts package is in that dirtree nor am I sure what deps it will need/if pulling it only would be a good idea... might be easier to just manually edit the files directly assuming those commits are changes to just the scripts/nothing compiled.

# opkg files sqm-scripts
Package sqm-scripts (1.5.1-1) is installed on root and has the following files:
/usr/lib/sqm/simplest.qos
/etc/hotplug.d/iface/11-sqm
/usr/lib/sqm/start-sqm
/usr/lib/sqm/run.sh
/etc/init.d/sqm
/usr/lib/sqm/update-available-qdiscs
/usr/lib/sqm/simple.qos.help
/usr/lib/sqm/piece_of_cake.qos
/usr/lib/sqm/layer_cake.qos.help
/usr/lib/sqm/simple.qos
/usr/lib/sqm/simplest_tbf.qos
/usr/lib/sqm/stop-sqm
/usr/lib/sqm/simplest_tbf.qos.help
/usr/lib/sqm/piece_of_cake.qos.help
/usr/lib/sqm/layer_cake.qos
/usr/lib/sqm/defaults.sh
/usr/lib/sqm/functions.sh
/etc/sqm/sqm.conf
/usr/lib/sqm/simplest.qos.help
/etc/config/sqm

EDIT2: Making this too difficult:

Comparing v1.5.0...v1.5.1 · tohojo/sqm-scripts

[moeller0]

Yes, you can basically manually edit the files in /usr/lib/sqm as well as /etc/hotplug.d/iface/11-sqm to match the different versions of sqm scripts (which really is just a set of scripts to set-up a working sane qdisc combination to get competent AQM into the hand of normal users).

Just make a copy of the existing files on your router before you start editing. Personally on a router with enough space/ram I install both nano and midnight commander to make my life easier in editing files, but I a sure other's prefer the terse elegance of ed or vi instead.

[moeller0]

@darksky

Well, there is really only one changing commit, the other is just updating the hash. So 1.51 is bad and we know from that other thread that 1.50 is good.

Just to be explicit for others (you already figured it out), between version 1.5.0 and 1.5.1 there were only three commits in sqm-scripts, so testing these by manually editing the relevant files seems not too difficult.
I note that myself I am facing some issues with hotplug on my turris omnia (but that it with TOS which is based on OpenWrt 19.07.8) but these materialize in SQM instances not coming up at all or missing after hotplug events and not massive CPU cost increases.

[dlakelan]

Isn't the regression likely to be in the cake qdisc or the kernel rather than the scripts?

[moeller0]

Yes, that would be the most likely way for SQM to break, but I think someone reported selectively downgrading SQM on the just released OpenWrt reduced the CPU load. Not sure about the root cause but it would help if one of the three commits in SQM could be shown to make a difference. Alas, I am in no position to really test that, so need to recruit volunteers...