SQM-autorate configuration and setting on fiber with DS-Lite

[Smitham]

Hello!

I'm using sqm-autorotate because despite having a fiber connection (1Gbps theoretical), speed tests give wildly different results based on time of day.

To better understand my connection, I measured around a week worth of results at different times of day using speedtest-go. There are a few outliers/faulty results depending on time but it's fair to say I expect between 200-300Mbps on average for download and a bit less for upload.

Anyway I was a bit unclear on what to setup for the upload_base_kbits and download_base_kbits so I put the highest value from my speedtest results.

However, after setting up everything, it’s unclear if it is worth it, see the results from Waveform:

• Before
• After

Speed is halved and while upload latency is better, download latency is roughly the same. Is that expected or did I miss something in the settings?

My assumptions was that stopped would be as close to “real speed” and only updated based on current traffic condition (which wasn't an issue at the time I tested).

Here is my sqm-autorotate config:

config network
	option upload_interface 'eth0'
	option download_interface 'ifb4eth0'
	option upload_base_kbits '300100'
	option download_base_kbits '723600'
	option upload_min_percent '20'
	option download_min_percent '20'

config output
	option log_level 'WARN'
	option use_syslog '1'
	option suppress_statistics 'No'
	option stats_file '/var/log/sqm-autorate.csv'
	option speed_hist_file '/var/log/sqm-speedhist.csv'

My sqm config:

config queue 'eth0'
        option enabled '1'
        option interface 'eth0'
        option download '723600'
        option upload '300100'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option qdisc_advanced '1'
        option ingress_ecn 'ECN'
        option egress_ecn 'ECN'
        option linklayer 'ethernet'
        option debug_logging '0'
        option verbosity '5'
        # Overhead for DS-lite over Ethernet:
        # Total: 18 + 40 = 58 bytes
        option overhead '58'
        option squash_dscp '0'
        option squash_ingress '0'
        option qdisc_really_really_advanced '1'
        option iqdisc_opts 'nat dual-dsthost ingress'
        option eqdisc_opts 'nat dual-srchost'
        option linklayer_advanced '1'
        option tcMPU '84'
        option tcMTU '2047'
        option tcTSIZE '512' 

TC command output:

~# tc qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 60020Kbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 58 mpu 84
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1522 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc noqueue 0: dev br-guest-turris root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev phy0-ap0 root refcnt 2
qdisc noqueue 0: dev guest_turris_0 root refcnt 2
qdisc noqueue 0: dev ds-wan root refcnt 2
qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 144720Kbit besteffort dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 58 mpu 84

[brada4]

How could we guess your router cpu?

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button (red circle; this works best in the 'Markdown' composer view in the blue oval):

Remember to redact passwords, VPN keys, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

[Smitham]

Hello, thanks for answering.

Apologies, I was half on the run and forgot about basic information. Here they are; my current configuration works without issue although I am unsure if anything is messing with SQM/CAKE.

System board:

{
	"kernel": "6.6.119",
	"hostname": "turris",
	"system": "ARMv8 Processor rev 4",
	"model": "CZ.NIC Turris Mox Board",
	"board_name": "cznic,turris-mox",
	"rootfs_type": "btrfs",
	"release": {
		"distribution": "TurrisOS",
		"version": "9.0.2",
		"revision": "r29077+93-f415748ba8",
		"target": "mvebu/cortexa53",
		"description": "TurrisOS 9.0.2 f415748ba8e979757595d8619feddc3ec8aca788",
		"builddate": "1765502369"
	}
}

Network:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix '[redacted]'

config interface 'wan'
	option proto 'dslite'
	option peeraddr '[redacted]'
	option peeraddrdomain '[redacted]'
	option tunlink 'wan6'
	option mtu '1460'
	option encaplimit 'ignore'

config interface 'lan'
	option macaddr '[redacted]'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option _turris_mode 'managed'
	option ip6assign '64'
	option device 'br-lan'
	option delegate '1'

config interface 'guest_turris'
	option enabled '1'
	option proto 'static'
	option device 'br-guest-turris'
	option ipaddr '10.111.222.1'
	option netmask '255.255.255.0'
	option ipv6 '0'

config device 'br_guest_turris'
	option name 'br-guest-turris'
	option type 'bridge'
	option bridge_empty '1'
	option igmp_snooping '1'
	option multicast_querier '1'

config device 'br_lan'
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	option igmp_snooping '1'
	option multicast_querier '1'

config interface 'wan6'
	option device 'eth0'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option extendprefix '1'

config device 'dev_wan'
	option name 'eth0'

Wireless:

config wifi-device 'radio0'
	option type 'mac80211'
	option path '[redacted]'
	option band '5g'
	option channel '36'
	option htmode 'VHT80'
	option disabled '0'
	option country '[redacted]'

config wifi-iface 'default_radio0'
	option device 'radio0'
	option network 'lan'
	option mode 'ap'
	option ssid '[redacted]'
	option encryption 'sae'
	option disabled '0'
	option hidden '0'
	option wpa_group_rekey '[redacted]'
	option key '[redacted]'

config wifi-device 'radio1'
	option type 'mac80211'
	option path '[redacted]'
	option band '5g'
	option channel '34'
	option htmode 'VHT80'
	option disabled '1'
	option country '[redacted]'

config wifi-iface 'default_radio1'
	option device 'radio1'
	option network 'lan'
	option mode 'ap'
	option ssid '[redacted]'
	option encryption 'sae'
	option disabled '1'

config wifi-iface 'guest_iface_0'
	option disabled '0'
	option device 'radio0'
	option mode 'ap'
	option ssid '[redacted]'
	option network 'guest_turris'
	option encryption 'sae'
	option wpa_group_rekey '[redacted]'
	option key '[redacted]'
	option ifname 'guest_turris_0'
	option isolate '1'

DHCP:

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option cachesize '1000'
	option authoritative '1'
	option readethers '0'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	option ednspacket_max '1232'
	option filter_aaaa '0'
	option filter_a '0'
	option port '0'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	option ndp 'relay'
	option ra_default '2'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option ra_maxinterval '1800'
	option ra_mininterval '600'
	option ra_lifetime '3600'
	list dhcp_option '6,192.168.1.1'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config dhcp 'wan6'
	option interface 'wan6'
	option master '1'
	option ra 'relay'
	option ndp 'relay'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'
	option piofolder '/tmp/odhcpd-piofolder'


config dhcp 'guest_turris'
	option interface 'guest_turris'
	option ignore '0'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv6 'disabled'
	option ra 'disabled'
	option ndp 'disabled'
	list dhcp_option '6,10.111.222.1'

Firewall:

config defaults
	option syn_flood '1'
	option input 'DROP'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone 'guest_turris'
	option enabled '1'
	option name 'tr_guest'
	option input 'REJECT'
	option forward 'REJECT'
	option output 'ACCEPT'
	list network 'guest_turris'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option sentinel_fwlogs '1'
	option sentinel_minipot '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config forwarding 'guest_turris_forward_wan'
	option enabled '1'
	option name 'guest to wan forward'
	option src 'tr_guest'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Critical'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '100/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Echo'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	option limit '10/sec'
	option limit_burst '20'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Errors'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '50/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward-Critical'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	option limit '100/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward-Echo'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	option limit '10/sec'
	option limit_burst '20'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward-Errors'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '50/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule 'guest_turris_dns_rule'
	option enabled '1'
	option name 'guest dns rule'
	option src 'tr_guest'
	option proto 'tcpudp'
	option dest_port '53'
	option target 'ACCEPT'

config rule 'guest_turris_dhcp_rule'
	option enabled '1'
	option name 'guest dhcp rule'
	option src 'tr_guest'
	option proto 'udp'
	option src_port '67-68'
	option dest_port '67-68'
	option target 'ACCEPT'

config rule
	option name 'Drop-ULA-Leak-to-WAN'
	option src 'lan'
	option dest 'wan'
	option src_ip 'fd14:c1b6:c8e8::/48'
	option family 'ipv6'
	option target 'REJECT'

config rule
	option name 'Drop-ULA-Leak-Guest-to-WAN'
	option src 'tr_guest'
	option dest 'wan'
	option src_ip 'fd14:c1b6:c8e8::/48'
	option family 'ipv6'
	option target 'REJECT'

config rule
	option enabled '1'
	option name 'Rate-Limit-ICMP-Accept'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option limit '1/sec'
	option limit_burst '5'
	option target 'ACCEPT'

config rule
	option enabled '1'
	option name 'Rate-Limit-ICMP-Drop'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'DROP'

config rule
	option enabled '1'
	option name 'Rate-Limit-ICMPv6-Accept'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv6'
	option limit '1/sec'
	option limit_burst '5'
	option target 'ACCEPT'

config rule
	option enabled '1'
	option name 'Rate-Limit-ICMPv6-Drop'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv6'
	option target 'DROP'

config include 'user_settings'
	option enabled '1'
	option fw4_compatible '1'
	option type 'script'
	option path '/etc/firewall.user'

config include 'bcp38'
	option type 'script'
	option path '/usr/lib/bcp38/run.sh'

config include 'sentinel_firewall'
	option type 'script'
	option path '/usr/libexec/sentinel/firewall.sh'

config redirect 'adblock_tr_guest53'
	option name 'Adblock DNS (tr_guest, 53)'
	option src 'tr_guest'
	option proto 'tcp udp'
	option src_dport '53'
	option dest_port '53'
	option target 'DNAT'
	option family 'any'

config rule 'adblock_tr_guest853'
	option name 'Adblock DNS (tr_guest, 853)'
	option src 'tr_guest'
	option proto 'tcp udp'
	option dest_port '853'
	option target 'REJECT'
	option dest '*'

config rule 'adblock_tr_guest5353'
	option name 'Adblock DNS (tr_guest, 5353)'
	option src 'tr_guest'
	option proto 'tcp udp'
	option dest_port '5353'
	option target 'REJECT'
	option dest '*'

Also my custom firewall.user for good measure:

#!/bin/sh
nft -f - << 'EOF'

insert rule inet fw4 input position 0 ct state established,related accept comment "Fast-path: established connections"
insert rule inet fw4 forward position 0 ct state established,related accept comment "Fast-path: established forwards"

add rule inet fw4 input ct state invalid counter drop comment "Drop invalid INPUT"
add rule inet fw4 forward ct state invalid counter drop comment "Drop invalid FORWARD"

add rule inet fw4 input tcp flags & (fin|syn|rst|ack) != syn ct state new counter drop comment "Drop non-SYN NEW connections"

add rule inet fw4 input ip saddr { \
        0.0.0.0/8, \
        # 10.0.0.0/8, \
        100.64.0.0/10, \
        127.0.0.0/8, \
        169.254.0.0/16, \
        # 172.16.0.0/12, \
        192.0.0.0/24, \
        192.0.2.0/24, \
        # 192.168.0.0/16, \
        198.18.0.0/15, \
        198.51.100.0/24, \
        203.0.113.0/24, \
        224.0.0.0/3 \
    } counter drop comment "Drop IP bogons (all interfaces)"

add rule inet fw4 input ip6 saddr fc00::/7 counter drop comment "Drop IPv6 bogons (all interfaces)"

add rule inet fw4 input tcp flags & (fin|syn|rst|psh|ack|urg) == 0x0 counter drop comment "Drop NULL scan"
add rule inet fw4 input tcp flags & (syn|fin) == (syn|fin) counter drop comment "Drop SYN-FIN scan"
add rule inet fw4 input tcp flags & (fin|syn|rst|psh|ack|urg) == (fin|syn|rst|psh|ack|urg) counter drop comment "Drop XMAS scan"

add rule inet fw4 input ip frag-off & 0x1fff != 0 counter drop comment "Drop IPv4 fragments"

add rule inet fw4 input tcp flags syn tcp option maxseg size 1-535 log prefix "MSS_FILTERING: " counter drop comment "Drop suspicious small MSS"

add rule inet fw4 input ct state new limit rate over 60/second burst 100 packets counter drop comment "Rate limit NEW connections"

add set inet fw4 ssh_ratelimit { type ipv4_addr; size 65536; flags dynamic,timeout; timeout 1m; }

add rule inet fw4 input tcp dport 1922 ct state new add @ssh_ratelimit { ip saddr limit rate over 3/minute } log prefix "SSH_BRUTE_FORCE: " drop comment "SSH brute-force protection"

EOF

# Check if nftables commands succeeded
if [ $? -eq 0 ]; then
    logger -t firewall "Custom hardening rules applied successfully"
    echo "Custom firewall hardening rules applied (nftables)"
else
    logger -t firewall -p err "Failed to apply custom hardening rules"
    echo "ERROR: Failed to apply custom hardening rules"
fi

[brada4]

@Smitham

"description": "TurrisOS 9.0.2 f415748ba8e979757595d8619feddc3ec8aca788",

You have to ask turris about specifics.

It appears you are using firmware that is not from the official OpenWrt project.

When using forks/offshoots/vendor-specific builds that are "based on OpenWrt", there may be many differences compared to the official versions (hosted by OpenWrt.org). Some of these customizations may fundamentally change the way that OpenWrt works. You might need help from people with specific/specialized knowledge about the firmware you are using, so it is possible that advice you get here may not be useful.

You may find that the best options are:

1. Install an official version of OpenWrt, if your device is supported (see https://firmware-selector.openwrt.org).
2. Ask for help from the maintainer(s) or user community of the specific firmware that you are using.
3. Provide the source code for the firmware so that users on this forum can understand how your firmware works (OpenWrt forum users are volunteers, so somebody might look at the code if they have time and are interested in your issue).

If you believe that this specific issue is common to generic/official OpenWrt and/or the maintainers of your build have indicated as such, please feel free to clarify.

[Smitham]

Apologies but I am confused by your answer.

My original question is about setting up cake-autorate sqm-autorate from user @Lynx and getting confirmation about my current settings.
Your answer is about how a vendor-specific builds might be different from official versions, which I am well aware off. However, I am unsure how asking Turris would help, as they do not maintain cake-autorate.

Specifically, I am trying to understand why the following bandwidth are selected when my tests confirmed my current connection can handle way more, and I set a higher value in the config files. I cannot rule out a hardware related issue, but this would need to be confirmed first. Furthermore, cake-autorate sqm-autorate not being an "official" package provided by OpenWrt (it's currently downloaded through GitHub), I am not clear how installing an official version of OpenWrt would help solve my original question.

~# tc qdisc | grep cake
qdisc cake 8009: dev eth0 root refcnt 9 bandwidth 60020Kbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 58 mpu 84
qdisc cake 800a: dev ifb4eth0 root refcnt 2 bandwidth 144720Kbit besteffort dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 58 mpu 84

Perhaps I am in the wrong category, and my message should've been posted on a different, more appropriate one?

[brada4]

Yes, go to turris support forum, no help in this one.
You may try cake-autorate thred though to help diagnosing YOUR problem.

[Smitham]

I am sorry but are you always this obtuse?

You either did not read my message or do not wish to understand and admit your mistake.

Searching for "cake-autorate" on this forum yield other thread within this very category ([1], [2], [3]), some with helpful answers. My thread is all but similar. And again this is a software/configuration issue, unless proved otherwise.

I understand this is a free forum and people come here to help others for free. If you do not wish to contribute or wish to remain focus on your unfunded hardware-related remark, please do not answer here.

And please stop marking your own response as the "solution" when they are clearly anything but that.

[dave14305]

cake-autorate and sqm-autorate are separate projects, so you have to be careful which one you’re posting about.

[Smitham]

Indeed, apologies. I got confused as I'm also testing that project on the side.

I am indeed asking about sqm-autorotate as per my first message and the config file I shared. I updated my posts to normalize to the correct project name.

[moeller0]

@Smitham

	"model": "CZ.NIC Turris Mox Board",

Mmmh, so do not expect too much out of these a53 cores with relative low frequency... I would not be amazed if with sqm-autorate running you might not have enough CPU cycles left for traffic shaping beyond ~200 Mbps... we can see whether we can squeeze a bit more out by enabling packet steering for all CPUs and maybe irqbalance. But I guess we should establish the CPU capabilities first before considering how to configure sqm-autorate (which might best be discussed as an issue in sqm-autorate's github repository).

@Smitham

		"version": "9.0.2",

OK, that is pretty close to OpenWrt 24.10... sure some users on this forum consider OpenWrt derivatives non-discussable over here and always point to the modifier's forums, but I am not one of those, and I do not think your issue is really specific to turris' OpenWrt variant.

@Smitham

        # Overhead for DS-lite over Ethernet:
        # Total: 18 + 40 = 58 bytes
        option overhead '58'

This seems the right thought process to deal with ds-lite, but the 18 look a bit suspicious... (real ethernet overhead is larger than 18 bytes, but the question is, does your linklayer actually use real ethernet frames). What kind of fiber do you use, say GPON or XGSPON, or AON (aka plain ethernet).

@brada4

Yes, go to turris support forum, no help in this one.

Please speak for yourself. Perfectly fine for you to focus on "proper" OpenWrt users only, but please do not generalize for all users of this forum. IMHO the issue has little to nothing to do with differences between upstream OpenWrt 24.10 and TurrisOS 9 (which is closely based on OpenWrt 24). And so discussing it here will be useful for "our" users trying sqm-autorate as well.

@Smitham

However, after setting up everything, it’s unclear if it is worth it, see the results from Waveform:

Now this might indicate that either sqm-autorate severely throttled your rate, or and that is my hunch, you are running out of CPU cycles and alas cake will fail in that case with trying to maintain the configured rate but allowing larger latency spikes...

Let's start by disabling sqm-auotorate to establish the limits for sqm on your MOX first.
TO do this I want you to do the following:
0) disable both sqm-autorate and sqm

1. run a test each on (and post the results):
   https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
   https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
   https://www.waveform.com/tools/bufferbloat (an post a link to the results)
2. re-enable sqm-scripts but configure for simpest.qos and fq-codel, setting the shaper rates to the throughput rates measured in the above tests (just pick the reported maximum over all tests).
3. run a test each on (and post the results):
   https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
   https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
   https://www.waveform.com/tools/bufferbloat (an post a link to the results)
4. change sqm from simplest.qos to piece_of_cake.qos and from fq-codel to cake
5. run a test each on (and post the results):
   https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
   https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
   https://www.waveform.com/tools/bufferbloat (an post a link to the results)

The question I want to answer is can sqm actually traffic shape on your link at the desired rates without autorate of any kind or not, and if not what is the achievable rate.
Once we have this data we can consider the next steps...

[Smitham]

Thank you for your kind and detailed answer!

@moeller0

variant.

@Smitham

        # Overhead for DS-lite over Ethernet:
        # Total: 18 + 40 = 58 bytes
        option overhead '58'

This seems the right thought process to deal with ds-lite, but the 18 look a bit suspicious... (real ethernet overhead is larger than 18 bytes, but the question is, does your linklayer actually use real ethernet frames). What kind of fiber do you use, say GPON or XGSPON, or AON (aka plain ethernet).

Thanks for catching that! I had to do a bit of guessing there because I saw a lot of contradicting information online.
In front of my router is a GE-PON-ONU (G-PON) from NTT. According to the manual it supports 1000BASE-T, 100BASE-TX, and 10BASE-T. I have a cable type 6 between the ONU and the router, and I've confirmed that router negotiates 1000Mbps. I am currently subscribed to a 1Gbps connection (best effort).

@moeller0

or and that is my hunch, you are running out of CPU cycles and alas cake will fail in that case

Interesting. Under normal load, sqm-autorate seems to use 14% CPU according to htop; this is by far the most CPU intensive service. I did not check while stressing it though. Let me go through your checklist.

Apologies as I couldn't find a good way to get picture (nor the "Download JPG" button for LibreQoS) so I took screenshots.

For best results I made sure I was the only user in the network during testing, and used a RJ45 cable instead of Wi-Fi for all tests.

Test Without QoS

@moeller0

• run a test each on (and post the results):
  https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
  https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
  https://www.waveform.com/tools/bufferbloat (an post a link to the results)

Cloudflare:

LibreQoS (couldn't find a "Download JPG" button unfortunately):

Waveform without QoS

With QoS (fq-codel)

@moeller0

• re-enable sqm-scripts but configure for simpest.qos and fq-codel, setting the shaper rates to the throughput rates measured in the above tests (just pick the reported maximum over all tests).
• run a test each on (and post the results):
  https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
  https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
  https://www.waveform.com/tools/bufferbloat (an post a link to the results)

Cloudflare:

LibreQoS:

Waveform (fq_codel)

With QoS (cake)

@moeller0

• change sqm from simplest.qos to piece_of_cake.qos and from fq-codel to cake
• run a test each on (and post the results):
  https://speed.cloudflare.com (post a screenshot after the test has finished of the full page)
  https://bufferbloat.libreqos.com (click to expand the "Advanced details" table than click the "Download JPG" button and post as image)
  https://www.waveform.com/tools/bufferbloat (an post a link to the results)

(I also removed all the custom setting I had)

Cloudflare:

LibreQoS:

Waveform (cake)

Results are a bit surprising, specially between LibreQoS and Waveform. I'm guessing the is a bit of luck as well but I am not sure what's the best way to interpret them.

[moeller0]

@Smitham

In front of my router is a GE-PON-ONU (G-PON) from NTT. According to the manual it supports 1000BASE-T, 100BASE-TX, and 10BASE-T. I have a cable type 6 between the ONU and the router, and I've confirmed that router negotiates 1000Mbps. I am currently subscribed to a 1Gbps connection (best effort).

In that case, I believe you are potentially hitting the true ethernet overhead, so 38+40... I assume your ISP does not use a VLAN (as indicated by your config).

@Smitham

Interesting. Under normal load, sqm-autorate seems to use 14% CPU according to htop; this is by far the most CPU intensive service.

Well, the cost of the actual traffic shaping is only visible during load and it will be accounted as softinterrupt/SIRQ that htop by default will not show unless you configure it to show detailled CPU statistics.

Regarding the tests, looking at the no sqm testd the 1000 Mbps seems to be truly a best effort thing... do you ever see close to a gigabit so aroung 900+ Mbps? But then, the actual bufferbloat even without sqm is not all that terrible. Not excellent, but I have seen much worse...

The fq-codel tests imply that your mox in the current configuration will shape only around 200 Mbps reliably and since cake is more demanding, I would guess less with cake.

This assumes that the tests were shaper limited and without the shaper you might have seen around 300 to 400 Mbps throughput.

OK, my hunch is we can improve the shaper performance a bit to either increase the shaper throughput, or keep some CPU reserves for other processing.

First step is to configure htop to show detailled CPU statistics and confirm that during the bidirectional phase of the libreqos test one CPU maxes out, that is is running close to 90 to 100% mostly in SIRQ context. If that is confirmed, the next step is to enable packet steering, I thing in the GUI in the network interface section in the global something tab. Start with the "all CPU" option and rerun a test while looking at htops cpu load bar... the CPU load should be lower than before...

[brada4]

For starters halve upload bandwidth and check latencies again.

[Smitham]

@moeller0

In that case, I believe you are potentially hitting the true ethernet overhead, so 38+40... I assume your ISP does not use a VLAN (as indicated by your config).

Thanks, so I'm changing to 78. ISP does not use VLAN indeed.
Would that number change with XGSPON?

@moeller0

the 1000 Mbps seems to be truly a best effort thing... do you ever see close to a gigabit so aroung 900+ Mbps?

Ha! I wish! I took measures several times a day for a week directly from the router, and best I ever got was around 550Mbps. And that number was more of an outlier than the norm. Average is more around 370Mbps (on a good day).

@moeller0

First step is to configure htop to show detailled CPU statistics and confirm that during the bidirectional phase of the libreqos test one CPU maxes out, that is is running close to 90 to 100% mostly in SIRQ context.

During both upload/download and bidirectional, the CPU maxes out frequently. That would explain things.

@moeller0

If that is confirmed, the next step is to enable packet steering, I thing in the GUI in the network interface section in the global something tab. Start with the "all CPU" option and rerun a test while looking at htops cpu load bar... the CPU load should be lower than before...

Which changed the following option:

	option packet_steering '2'

However it doesn't seem to do anything for the CPU (this is after a network restart):

Perhaps those a53 cores are indeed a bit too slow for such task.

[moeller0]

@Smitham

Would that number change with XGSPON?

No... GEM (GPON) and XGSPON (XGEM) have slightly different overhead (5 vs 8 byte) but in both cases this is smaller than the overhead on your 1 Gbps ethernet link to the ONT/LAN so that is setting the relevant value for the per packet overhead...
Mind you with a 1 Gbps contract and a shaper likely ending up set to around 200 Mbps this will likely have very little effect on your measured rates and bufferbloat, but I still think it is useful to configure this decently, so if you should switch to a more powerful router you do not need to revisit this.

@Smitham

Ha! I wish! I took measures several times a day for a week directly from the router, and best I ever got was around 550Mbps. And that number was more of an outlier than the norm. Average is more around 370Mbps (on a good day).

OK, so likely the capacity sacrifice will not bee that steep, more ~200 versus 370, not 200 vs 1000...

@Smitham

During both upload/download and bidirectional, the CPU maxes out frequently. That would explain things.

Sure, however that is a rather hash test, waveform and cloudflare only ever test unidirectional load. Again, I think it is rational to configure sqm for the true worst case load (given that bidirectional loading, while not super-common, is not super rare either).

@Smitham

Which changed the following option:

option packet_steering '2'

Yes that will help, but only during the bidirectional test phase...

@Smitham

Perhaps those a53 cores are indeed a bit too slow for such task.

If this is taken during the bidirectional phase this looks like packet steering did not engage properly... (also the color scheme looks odd, I would have expected a lot of pink bars, at least for me in my console htop will color SIRQ pink, but in your plot I only see red)?

Could you run:

for file in /sys/class/net/*
do
	for queue in "${file}"/queues/rx-*
	do
		echo "${queue}" RX rps_cpus
		cat "${queue}"/rps_cpus
	done
	for queue in "${file}"/queues/tx-*
	do
		echo "${queue}" TX xps_cpus
		cat "${queue}"/xps_cpus
	done
done

and post the output here? That should let us figure out whether/where the packet steering settings were applied.

[Smitham]

@moeller0

Sure, however that is a rather hash test, waveform and cloudflare only ever test unidirectional load.

So it seems even with Cloudflare/Waveform unidirectional load, the CPU hits 100% or close to it a bit too frequently. Even with SQM off. Those machines are really weak.

@moeller0

If this is taken during the bidirectional phase this looks like packet steering did not engage properly... (also the color scheme looks odd, I would have expected a lot of pink bars, at least for me in my console htop will color SIRQ pink, but in your plot I only see red)?

This was and it's unclear if it engaged but one of the CPU seems to be slacking off.
As for the color, I am using the default colorscheme. I can check if there is a different theme easier to read.

@moeller0

and post the output here? That should let us figure out whether/where the packet steering settings were applied.

Thanks for the script, here you go:

br-guest-turris /sys/class/net/br-guest-turris/queues/rx-0 RX rps_cpus
0
br-guest-turris /sys/class/net/br-guest-turris/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
br-lan /sys/class/net/br-lan/queues/rx-0 RX rps_cpus
0
br-lan /sys/class/net/br-lan/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
ds-wan /sys/class/net/ds-wan/queues/rx-0 RX rps_cpus
0
ds-wan /sys/class/net/ds-wan/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
eth0 /sys/class/net/eth0/queues/rx-0 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-1 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-2 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-3 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-4 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-5 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-6 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/rx-7 RX rps_cpus
0
eth0 /sys/class/net/eth0/queues/tx-0 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-1 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-2 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-3 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-4 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-5 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-6 TX xps_cpus
1
eth0 /sys/class/net/eth0/queues/tx-7 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/rx-0 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-1 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-2 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-3 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-4 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-5 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-6 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/rx-7 RX rps_cpus
0
eth1 /sys/class/net/eth1/queues/tx-0 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-1 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-2 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-3 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-4 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-5 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-6 TX xps_cpus
1
eth1 /sys/class/net/eth1/queues/tx-7 TX xps_cpus
1
guest_turris_0 /sys/class/net/guest_turris_0/queues/rx-0 RX rps_cpus
0
guest_turris_0 /sys/class/net/guest_turris_0/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
ip6tnl0 /sys/class/net/ip6tnl0/queues/rx-0 RX rps_cpus
0
ip6tnl0 /sys/class/net/ip6tnl0/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
lan1 /sys/class/net/lan1/queues/rx-0 RX rps_cpus
0
lan1 /sys/class/net/lan1/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
lan2 /sys/class/net/lan2/queues/rx-0 RX rps_cpus
0
lan2 /sys/class/net/lan2/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
lan3 /sys/class/net/lan3/queues/rx-0 RX rps_cpus
0
lan3 /sys/class/net/lan3/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
lan4 /sys/class/net/lan4/queues/rx-0 RX rps_cpus
0
lan4 /sys/class/net/lan4/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
lo /sys/class/net/lo/queues/rx-0 RX rps_cpus
0
lo /sys/class/net/lo/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
mlan0 /sys/class/net/mlan0/queues/rx-0 RX rps_cpus
0
mlan0 /sys/class/net/mlan0/queues/tx-0 TX xps_cpus
0
mlan0 /sys/class/net/mlan0/queues/tx-1 TX xps_cpus
0
mlan0 /sys/class/net/mlan0/queues/tx-2 TX xps_cpus
0
mlan0 /sys/class/net/mlan0/queues/tx-3 TX xps_cpus
0
phy0-ap0 /sys/class/net/phy0-ap0/queues/rx-0 RX rps_cpus
0
phy0-ap0 /sys/class/net/phy0-ap0/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory
sit0 /sys/class/net/sit0/queues/rx-0 RX rps_cpus
0
sit0 /sys/class/net/sit0/queues/tx-0 TX xps_cpus
cat: read error: No such file or directory

It seems mostly 0 is active here.

[moeller0]

@Smitham

So it seems even with Cloudflare/Waveform unidirectional load, the CPU hits 100% or close to it a bit too frequently. Even with SQM off. Those machines are really weak.

The Arm a53 cores come from the efficiency line, not the performance line... and under even only mildly adverse conditions that tends to show... these are in-order cores that likely need a beefy memory system, and I am not sure the mox offers that.

@Smitham

It seems mostly 0 is active here.

Yeah, and that means it is not active at all... this is supposed to be a numeric representation of a bitmap of the existing CPUs, so 1 means CPU0 only, 2 means CPU1 only, 3 means CPU0 and CPU1.
Try running this to force the use of both CPUs temporarily:

for file in /sys/class/net/*
do
	for rxqueue in "${file}"/queues/rx-*
	do
		if [ -f "${rxqueue}"/rps_cpus ]
		then
			echo "rxqueue:" ${rxqueue}
			echo "previous value" $(cat "${rxqueue}"/rps_cpus)
			echo 3 > "${rxqueue}"/rps_cpus
			echo "current value" $(cat "${rxqueue}"/rps_cpus)
		fi 
	done

	for txqueue in "${file}"/queues/tx-*
	do
		if [ -f "${txqueue}"/xps_cpus ]
		then
			echo "txqueue:" ${txqueue}
			echo "previous value" $(cat "${txqueue}"/xps_cpus)
			echo 3 > "${txqueue}"/xps_cpus
			echo "current value" $(cat "${txqueue}"/xps_cpus)
		fi
	done
done

and then perform the bidirectional libreqos test again and look at the CPU usage bar... during the bidirectional phase we would expect both CPUs to show signs of saturation.

[brada4]

Cake needs a single core performance to chuck all data to the wire.
Just like with ramips disanble (set to zero) ingress shaping and try to get uploads under control. Seems ingress is already nice.

[moeller0]

@brada4

Cake needs a single core performance to chuck all data to the wire.

Yes, but you can (and arguably) should put the ingress and egress instance each on a different cores. I believe this will still matter for cake_mq, but I am not sure that this uses the packet steering machinery.

@brada4

Just like with ramips disanble (set to zero) ingress shaping and try to get uploads under control. Seems ingress is already nice.

That is certainly an option. Personally I am not too keen on that as that puts the ingress direction under the ISP's mercy and the ISP might change things when ever the ISP pleases.

[Smitham]

(Note: I rewrote this message to respond to the correct one from moeller0)

I applied the temporary fix to force CPU usage, confirmed it is indeed 3 everywhere but it is unclear how much it is helping.

It seems the second CPU is getting a bit more load than previously. I was expecting them to move somewhat at the same rate though, which is not the case.

(this is during bidirectional, without SQM)