Validating NSS fq_codel's correctness

Another thought is flows 16 limit 1024. I keep thinking we are out of some on-chip resource. Each flow consumes 64 bytes of extra memory, a packet limit of 1024 uses up 2k each for the packet list, and the default is a whopping 4096 which makes sense without GRO at 10Gbit, but not at 1gbit.

@KONG,

I've "installed" the NSS SQM scripts from your repo on my R7800:

I've also "installed" the selectable NSS.qos setup script option from your repo:

I've enabled SQM this way on a stable 21.02 build from @ACwifidude's repo (OpenWrt 21.02-SNAPSHOT r16328+19-f441be3921).

When I look at the log I see some errors, I'd like to resolve them to get SQM with NSS running based on your latest findings but realize I might be walking on thin ice here. Maybe you can tell from the messages what I've done wrong?

Wed Nov 24 11:35:05 2021 user.notice SQM: Starting SQM script: nss.qos on eth1, in: 490000 Kbps, out: 490000 Kbps
Wed Nov 24 11:35:06 2021 daemon.err modprobe: failed to find a module named act_ipt
Wed Nov 24 11:35:06 2021 daemon.err modprobe: failed to find a module named sch_fq_codel
Wed Nov 24 11:35:06 2021 kern.err kernel: [102179.705566] debugfs: File 'virt_if' in directory 'stats' already present!
Wed Nov 24 11:35:06 2021 kern.info kernel: [102179.705885] Created a NSS virtual interface for dev [nssifb]
Wed Nov 24 11:35:06 2021 kern.info kernel: [102179.711564] NSS IFB data callback registered
Wed Nov 24 11:35:06 2021 kern.info kernel: [102179.717390] NSS IFB transmit callback registered
Wed Nov 24 11:35:06 2021 kern.info kernel: [102179.722088] NSS IFB module loaded.
Wed Nov 24 11:35:06 2021 daemon.err modprobe: failed to find a module named act_ipt
Wed Nov 24 11:35:06 2021 daemon.err modprobe: failed to find a module named sch_fq_codel
Wed Nov 24 11:35:07 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 0 u32 match ip protocol 1 0xff flowid 1:13
Wed Nov 24 11:35:07 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel
Wed Nov 24 11:35:07 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 match ip protocol 1 0xff flowid 1:13
Wed Nov 24 11:35:07 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel
Wed Nov 24 11:35:07 2021 user.notice SQM: WARNING: sqm_start_default: nss.qos lacks an egress() function
Wed Nov 24 11:35:07 2021 kern.warn kernel: [102180.383768] nss_qdisc_init[1993]:NSS qdisc 25ef04a5 (type 1) used along with non-nss qdiscs, or the interface is currently down
Wed Nov 24 11:35:07 2021 kern.info kernel: [102180.393108] 61a7cae9: Found net device [eth0]
Wed Nov 24 11:35:07 2021 kern.info kernel: [102180.403845] 61a7cae9: Net device [eth0] has NSS intf_num [1]
Wed Nov 24 11:35:07 2021 kern.info kernel: [102180.409845] Nexthop successfully set for [eth0] to [nssifb]
Wed Nov 24 11:35:07 2021 user.notice SQM: nss.qos was started on eth0 successfully

I reckon that SQM isn't working (properly) on eth1. It's probably because some kernel modules to be loaded can't be found. That very likely results in the message that the cmd_wrapper is unable to talk to the kernel.

I'm also a bit puzzled that SQM is also started on eth0 since I explicitly set eth1:

root@OpenWrt:/etc/config# cat sqm

config queue 'eth1'
        option interface 'eth1'
        option linklayer 'none'
        option verbosity '5'
        option qdisc 'fq_codel'
        option script 'nss.qos'
        option download '490000'
        option upload '490000'
        option debug_logging '1'
        option enabled '1'

Any clues what I did wrong?

--EDIT--
I've found this:

And indeed, that package is not installed on this build. Since it's a community build, I can't install it anymore. I was able to install those missing packages, the modules are loaded (checked with lsmod. There is one module missing however:

daemon.err modprobe: failed to find a module named sch_fq_codel

I can't find it in the modules directory, these are the ones loaded:

# lsmod | grep sch_
sch_cake               40960  0 
sch_codel              20480  0 
sch_dsmark             20480  0 
sch_fq                 20480  0 
sch_gred               24576  0 
sch_hfsc               28672  0 
sch_htb                28672  0 
sch_ingress            16384  0 
sch_multiq             16384  0 
sch_pie                20480  0 
sch_prio               20480  0 
sch_red                20480  0 
sch_sfq                24576  0 
sch_tbf                20480  0 
sch_teql               16384  0 

Since I see a sch_codel and a sch_fq, could it be a (re)naming issue of module names or is there actually another module sch_fq_codel?

fq_codel is built in, not a module, in openwrt. not really an error here.

I think I found where it comes from, in /usr/lib/sqm/defaults.sh there's a check for a NULL value in the $QDISC variable, if it's NULL, set it to at least fq_codel:

[ -z "$QDISC" ] && QDISC=fq_codel

Later on when loading the needed modules in that same defaults.sh script, this gets forced like so:

# These are the modules that do_modules() will attempt to load
ALL_MODULES="act_ipt sch_$QDISC sch_ingress act_mirred cls_fw cls_flow cls_u32 sch_htb sch_hfsc nss-ifb"

That's likely where the daemon.err modprobe: failed to find a module named sch_fq_codel error message comes from.

But I believe I've also found why NSS fq_codel won't work on my build, because it's missing the nssfq_codel QDISC:

Wed Nov 24 22:05:26 2021 daemon.err modprobe: failed to find a module named sch_nssfq_codel
Wed Nov 24 22:05:26 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name SQM_IFB_f4167 type ifb
Wed Nov 24 22:05:26 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /sbin/tc qdisc replace dev SQM_IFB_f4167 root nssfq_codel
Wed Nov 24 22:05:26 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Usage: ... nssfq_codel target TIME interval TIME [ flows NUMBER ] [ quantum BYTES ][ limit PACKETS ] [ set_default ] [ accel_mode ]
Wed Nov 24 22:05:26 2021 user.notice SQM: ERROR: QDISC nssfq_codel is NOT useable.
Wed Nov 24 22:05:26 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev SQM_IFB_f4167 down
Wed Nov 24 22:05:26 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete SQM_IFB_f4167 type ifb

Where should I find this nssfq_codel QDISC? Is it the actual /usr/lib/sqm/nss.qos file?

eth0 is hardcoded in the driver.

In order to include nss.qos in your custom build, just change sqm-scripts Makefile and pull from my tree:

diff --git a/net/sqm-scripts/Makefile b/net/sqm-scripts/Makefile index 5240cd3a4..5058aa94e 100644 --- a/net/sqm-scripts/Makefile +++ b/net/sqm-scripts/Makefile @@ -8,13 +8,13 @@ include $(TOPDIR)/rules.mk

PKG_NAME:=sqm-scripts
-PKG_SOURCE_VERSION:=5d7e7977e14e147d5bcfa8a5f01636c7ab230fa4
+PKG_SOURCE_VERSION:=c377d7e398baa3cffc764ab19707b57bf358a2d0
PKG_VERSION:=1.5.1
-PKG_RELEASE:=1
+PKG_RELEASE:=nss

PKG_SOURCE_PROTO:=git
-PKG_SOURCE_URL:=https://github.com/tohojo/sqm-scripts
-PKG_MIRROR_HASH:=dad79a899da9e9389d8bdedb0c7b48235dd55f834edf6ed4b4e38c13eff46c19
+PKG_SOURCE_URL:=https://github.com/ricsc/sqm-scripts.git
+PKG_MIRROR_HASH:=d41301ed1e318ea81c6c8f29c1847efdda3663573d12a3e0b855b4b8b8cf0610

PKG_MAINTAINER:=Toke Høiland-Jørgensen toke@toke.dk
PKG_LICENSE:=GPL-2.0-only

1 Like

Thank you @KONG, I've replaced the hardcoded eth0 with ${IFACE}. No messages about eth0 anymore. I've got one error left:

Fri Nov 26 11:40:59 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name SQM_IFB_e8921 type ifb
Fri Nov 26 11:40:59 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /sbin/tc qdisc replace dev SQM_IFB_e8921 root nssfq_codel
Fri Nov 26 11:40:59 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Usage: ... nssfq_codel target TIME interval TIME [ flows NUMBER ] [ quantum BYTES ][ limit PACKETS ] [ set_default ] [ accel_mode ]
Fri Nov 26 11:40:59 2021 user.notice SQM: ERROR: QDISC nssfq_codel is NOT useable.
Fri Nov 26 11:40:59 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev SQM_IFB_e8921 down
Fri Nov 26 11:40:59 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete SQM_IFB_e8921 type ifb

I've selected pppoe-wan as my interface, but the queue is configured for eth1 on my R7800, so I wonder if this is entirely correct?


config queue 'eth1'
        option debug_logging '1'
        option download '450000'
        option upload '450000'
        option verbosity '8'
        option interface 'pppoe-wan'
        option linklayer 'ethernet'
        option overhead '50'
        option qdisc 'nssfq_codel'
        option script 'nss.qos'
        option enabled '1'

In /usr/lib/sqm/functions.sh there's a check function called verify_qdisc(). It prepares the actual qdisc to be replaced with the one you select in Luci, it does so here:

    $TC qdisc replace dev $ifb $root_string $qdisc $args
    res=$?
    if [ "$res" = "0" ] ; then
        sqm_debug "QDISC $qdisc is useable."
    else
        sqm_error "QDISC $qdisc is NOT useable."
    fi
    delete_ifb $ifb
    return $res

This check let's me know whether it actually worked to replace the loaded qdisc with the nss qdisc, but from the "wrong usage" error from the tc command I doubt if all variables are actually in place?

Don't change IFACE it is there on intention, the reason is this:

See char nss_dev_name_array[32] = "eth0";

Device which will handle traffic. That's what I meant by is hardcoded in the driver.

Then In my script:

ingress() {
sqm_debug "ingress"
#cut off ifb4
DEV=echo $DEV | cut -c 5-
printf $DEV > /sys/module/nss_ifb/parameters/nss_dev_name

Should take care of wan type pppoe or regular wan. $DEV contains either ifb4pppoe-wan of ifb4eth0, so we cut off ifb4 and tell the driver that wan is either pppoe-wan or eth0.

1 Like

@kong

#define TX_Q_LIMIT 32

??? What happens on that 128 flow test (with fq_codel queues 1024) if that's, like, 1024? Any other magic looking numbers like that?

(I didn't even know we had this much source code)

I think that's @quarky 's work.
He might be able to give you some information.

The nss ifb driver is based on the Linux ifb driver. The TX Q limit is the same as those defined in the Linux ifb driver. I’ll be honest and say that I do not really know what that does in terms of packet latencies.

I'm still trying to figure out that bifurcation in this bug report with 128 active flows. having insufficient buffering between the hardware driver and the OS might account for it, and given how more correct the nssfq_codel firmware behaved with 16 rather than 1024 queues, would like to see people trying flows 32 and flows 64 there to see when that happens.

From my understanding of how the NSS subsystem works, once a TCP/UDP flow is taken over by NSS, Linux is basically blind to it. Once a packet reaches the LAN port, it’s processed by the NSS firmware. If the packet destination is another LAN port, it sends it out. It will then periodically report stats back to the Linux network stack.

All the buffering are inside the NSS firmware. So I guess to test the correctness of NSS QDISCs, it’ll have to be black box tested.

Gotcha, reverted that to eth0.

What I'm trying to do is see if I can get your most recent NSS fq_codel improvements working by also using your basic script to select NSS fq_codel in Luci, which is simple but elegant and effective; I don't need to reboot to make changes via the web interface.

There's something I'm missing here, I'm busy doing multiple things so my head might be messing with me. Let my try to summarize my situation and I'm taking an educated guess you don't need much words to follow me.

Tracing back what happens in which order, I'll make an attempt:

  1. Service SQM (/etc/init.d/sqm) runs the run.sh script at /usr/lib/sqm/run.sh
  2. This run.sh sources the /etc/sqm/sqm.conf file and the ${SQM_LIB_DIR}/functions.sh for usage of general functions when setting up SQM.
  3. The contents of this /etc/sqm/sqm.conf file on my R7800 is as follows:
root@OpenWrt:/usr/lib/sqm# cat /etc/sqm/sqm.conf
SQM_LIB_DIR=/usr/lib/sqm
SQM_STATE_DIR=/var/run/sqm
SQM_QDISC_STATE_DIR=${SQM_STATE_DIR}/available_qdiscs
SQM_CHECK_QDISCS="nssfq_codel fq_codel codel pie sfq cake"
SQM_SYSLOG=1
  1. The run.sh script gets the config of /etc/config/sqm and then runs "${SQM_LIB_DIR}/start-sqm" which does some checks if every variable and file is in place. If that's true, then it runs the sqm_start_default function in ${SQM_LIB_DIR}functions.sh.
  2. This sqm_start_default function will load the necessary modules (do_modules function) amongst which is the nss-ifb module. The sch_nssfq_codel module can't be found by modprobe however? But is this an actual error? It also runs the verify_qdisc function.
  3. This verify_qdisc function does some checks and will add the IFB link, that succeeds. It then runs the $TC qdisc replace dev $ifb $root_string $qdisc $args command to replace a device. This will fail with nss_fqcodel so the nssfq_codel QDISC is considered not usable.

This is the log I see:

Tue Nov 30 21:43:48 2021 user.notice SQM: Starting SQM script: nss.qos on pppoe-wan, in: 450000 Kbps, out: 450000 Kbps
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: function candidate name: sqm_start
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: TYPE_OUTPUT: sqm_start: not found
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: return value: 1
Tue Nov 30 21:43:48 2021 user.notice SQM: Using generic sqm_start_default function.
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: function candidate name: sqm_prepare_script
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: TYPE_OUTPUT: sqm_prepare_script is a function
Tue Nov 30 21:43:48 2021 user.notice SQM: fn_exists: return value: 0
Tue Nov 30 21:43:48 2021 user.notice SQM: sqm_start_default: starting sqm_prepare_script
Tue Nov 30 21:43:49 2021 daemon.err modprobe: failed to find a module named sch_nssfq_codel
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -X QOS_MARK_eth0
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -X QOS_MARK_eth0
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -N QOS_MARK_eth0
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -N QOS_MARK_eth0
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -j MARK --set-mark 0x2/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -j MARK --set-mark 0x2/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -j MARK --set-mark 0x2/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -j MARK --set-mark 0x2/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class CS1 -j MARK --set-mark 0x3/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class CS1 -j MARK --set-mark 0x3/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class CS1 -j MARK --set-mark 0x3/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class CS1 -j MARK --set-mark 0x3/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class CS6 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class CS6 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class CS6 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class CS6 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class EF -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class EF -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class EF -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class EF -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class AF42 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -m dscp --dscp-class AF42 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class AF42 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -m dscp --dscp-class AF42 -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D QOS_MARK_eth0 -m tos --tos Minimize-Delay -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D QOS_MARK_eth0 -m tos --tos Minimize-Delay -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:49 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A QOS_MARK_eth0 -m tos --tos Minimize-Delay -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A QOS_MARK_eth0 -m tos --tos Minimize-Delay -j MARK --set-mark 0x1/0xff
Tue Nov 30 21:43:50 2021 user.notice SQM: Squashing differentiated services code points (DSCP) from ingress.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D PREROUTING -i eth0 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D PREROUTING -i eth0 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -I PREROUTING -i eth0 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -I PREROUTING -i eth0 -m dscp ! --dscp 0 -j DSCP --set-dscp-class be
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: FAILURE (1): /usr/sbin/iptables -w 1 -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: LAST ERROR: iptables: No chain/target/match by that name.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: FAILURE (1): /usr/sbin/ip6tables -w 1 -t mangle -D POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: LAST ERROR: ip6tables: No chain/target/match by that name.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A POSTROUTING -o eth0 -m mark --mark 0x00/0xff -g QOS_MARK_eth0
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -D OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: iptables: SUCCESS: /usr/sbin/iptables -w 1 -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: invocation silenced by request, FAILURE either expected or acceptable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip6tables: SUCCESS: /usr/sbin/ip6tables -w 1 -t mangle -A OUTPUT -p udp -m multiport --ports 123,53 -j DSCP --set-dscp-class AF42
Tue Nov 30 21:43:50 2021 daemon.err modprobe: failed to find a module named sch_nssfq_codel
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link add name SQM_IFB_1e794 type ifb
Tue Nov 30 21:43:50 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /sbin/tc qdisc replace dev SQM_IFB_1e794 root nssfq_codel
Tue Nov 30 21:43:50 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Usage: ... nssfq_codel target TIME interval TIME [ flows NUMBER ] [ quantum BYTES ][ limit PACKETS ] [ set_default ] [ accel_mode ]
Tue Nov 30 21:43:50 2021 user.notice SQM: ERROR: QDISC nssfq_codel is NOT useable.
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link set dev SQM_IFB_1e794 down
Tue Nov 30 21:43:50 2021 user.notice SQM: cmd_wrapper: ip: SUCCESS: /sbin/ip link delete SQM_IFB_1e794 type ifb

I have the strong idea I'm either missing a driver, a module or an important config setting. I'm not seeing it (yet?). Maybe you can tell me in a few steps what I need to check/add/repair/re-RTFM?

Use the latest:

4f3492aa93ad436b9312ce731d3ad45bf5e8a6e1

Thanks, already using the latest, forgot to mention that... Sorry.

~/tmp/sqm-scripts$ git log | head -20
commit 4f3492aa93ad436b9312ce731d3ad45bf5e8a6e1
Author: ricsc <ddwrtkong@t-online.de>
Date:   Sun Nov 14 17:46:29 2021 +0100

    use get_limit

commit c377d7e398baa3cffc764ab19707b57bf358a2d0
Author: ricsc <ddwrtkong@t-online.de>
Date:   Thu Nov 11 09:46:47 2021 +0100

    fix ingress for nss

commit 2793899b4ba46c0f02dbfaa9c41c1dbcff7f3a7c
Author: ricsc <ddwrtkong@t-online.de>
Date:   Sat Nov 6 01:16:23 2021 +0100

    use get_burst to set reasonable burst values

commit 27fb04f4a49b80e8ccde27c7587ce2a63bd8ac43
Merge: 9b8e985 5d7e797

OK.

There are currently lots of errors that are irrelevant. This line however is important:

Tue Nov 30 21:43:50 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Usage: ... nssfq_codel target TIME interval TIME [ flows NUMBER ] [ quantum BYTES ][ limit PACKETS ] [ set_default ] [ accel_mode ]

if you run command tc -s qdisc, is nssfq_codel showing up on an interface, it should be at least on one ethernet interface. I haven't tested pppoe in a while, but it is possible, that the patches for pppoe hw accel are not working.

it's not showing up on any interface:

root@OpenWrt:/usr/lib/sqm# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 6117739 bytes 57539 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 541278406 bytes 884106 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 17204 drop_overlimit 0 new_flow_count 1486 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.5 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.4 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.6 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 4761633 bytes 51020 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 589 drop_overlimit 0 new_flow_count 381 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

There's a script called update-available-qdiscs which gets run by run.sh. If I run that, this already does a check for usable qdiscs. Although in /etc/sqm/sqm.conf the variable SQM_CHECK_QDISCS lists nssfq_codel, there's something missing I think. The update-available-qdiscs script tests if it is present, basically an empty file in /var/run/sqm/available_qdiscs. When I reboot this is the contents of that directory:

ls -l /var/run/sqm/available_qdiscs/
-rw-r--r--    1 root     root             0 Nov 30 23:20 cake
-rw-r--r--    1 root     root             0 Nov 30 23:20 codel
-rw-r--r--    1 root     root             0 Nov 30 23:20 fq_codel
-rw-r--r--    1 root     root             0 Nov 30 23:20 pie
-rw-r--r--    1 root     root             0 Nov 30 23:20 sfq

When I touch nssfq_codel, I can select nssfq_codel from the Luci web interface but the verify_qdisc still fails with:

hu Dec  2 16:35:03 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (1): /sbin/tc qdisc replace dev SQM_IFB_469b5 root nssfq_codel
Thu Dec  2 16:35:03 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: Usage: ... nssfq_codel target TIME interval TIME [ flows NUMBER ] [ quantum BYTES ][ limit PACKETS ] [ set_default ] [ accel
_mode ]                                    
Thu Dec  2 16:35:03 2021 user.notice SQM: ERROR: QDISC nssfq_codel is NOT useable.

Is a specific qdisc like cake, codel, fq_codel and so on a builtin qdisc in the kernel, or a loadable module or a specifically linked library or perhaps a "driver/firmware" file/package I need to install for nssfq_codel?

Quick test:

tc qdisc add dev wlan1 root handle 1: nsstbl rate 1000kbit burst 1Mb
tc qdisc add dev wlan1 parent 1: handle 10: nssfq_codel limit 4096 flows 1024 quantum 1514 target 5ms interval 100ms set_default

Test result:
tc -s qdisc | grep wlan1

qdisc nsstbl 1: dev wlan1 root refcnt 2 buffer/maxburst 1Mb rate 1Mbit mtu 1514b accel_mode 0
qdisc nssfq_codel 10: dev wlan1 parent 1: target 5ms limit 4096p interval 100ms flows 1024 quantum 1514 set_default accel_mode 0

If this does not work, the build you are using is broken.

Does this mean it's broken?:

tc -s qdisc | grep wlan1
qdisc nsstbl 1: dev wlan1 root refcnt 2 buffer/maxburst 1Mb rate 1Mbit mtu 1514b accel_mode 0 
qdisc nssfq_codel 10: dev wlan1 parent 1: target 5ms limit 4096p interval 100ms flows 1024 quantum 1514 set_default accel_mode 0 
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
qdisc noqueue 0: dev wlan1-2 root refcnt 2 

I've added wlan1 but no data is being transmitted anymore over that device. When I remove it again using tc qdisc del dev wlan1 root data is transmitted again over wlan1.

These are the loaded modules for NSS:

# lsmod | grep nss
bonding               118784  1 qca_nss_pppoe
nss_ifb                16384  0 
ppp_generic            36864  9 ecm,ppp_async,qca_nss_pppoe,pppoe,pppox
pppoe                  20480  4 ecm,qca_nss_pppoe
qca_nss_drv           442368  5 nss_ifb,ecm,mac80211,qca_nss_qdisc,qca_nss_pppoe
qca_nss_gmac           53248  1 qca_nss_drv
qca_nss_pppoe          16384  0 
qca_nss_qdisc          61440  0 

@KONG I believe I have a working build now:

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2                                                                                                                                                                                                       
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)                                                              
 backlog 0b 0p requeues 0
qdisc nsstbl 1: dev eth0 root refcnt 2 buffer/maxburst 56250b rate 450Mbit mtu 1514b accel_mode 0 
 Sent 3866175634 bytes 3664883 pkt (dropped 1940, overlimits 66900 requeues 0) 
 backlog 0b 0p requeues 0
qdisc nssfq_codel 10: dev eth0 parent 1: target 5ms limit 4096p interval 100ms flows 1024 quantum 1514 set_default accel_mode 0 
 Sent 3866178094 bytes 3664901 pkt (dropped 1940, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 maxpacket 1518 drop_overlimit 466 new_flow_count 603886 ecn_mark 0
 new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 16885917 bytes 55610 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 1506 drop_overlimit 0 new_flow_count 99 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-iot root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.5 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth1.1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.4 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.6 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 687839 bytes 9646 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 147 drop_overlimit 0 new_flow_count 71 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc nsstbl 1: dev nssifb root refcnt 2 buffer/maxburst 56250b rate 450Mbit mtu 1514b accel_mode 0 
 Sent 6269594319 bytes 4763483 pkt (dropped 60921, overlimits 108792 requeues 0) 
 backlog 0b 0p requeues 0
qdisc nssfq_codel 10: dev nssifb parent 1: target 5ms limit 4096p interval 100ms flows 1024 quantum 1514 set_default accel_mode 0 
 Sent 6269605187 bytes 4763499 pkt (dropped 60921, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 maxpacket 1518 drop_overlimit 5439 new_flow_count 342065 ecn_mark 0
 new_flows_len 0 old_flows_len 1

The nss qdisc is now actually used:

root@OpenWrt:~# lsmod | grep nss
bonding               118784  1 qca_nss_pppoe
nss_ifb                16384  0 
ppp_generic            36864  9 ecm,ppp_async,qca_nss_pppoe,pppoe,pppox
pppoe                  20480  4 ecm,qca_nss_pppoe
qca_nss_drv           442368  5 nss_ifb,ecm,mac80211,qca_nss_qdisc,qca_nss_pppoe
qca_nss_gmac           53248  1 qca_nss_drv
qca_nss_pppoe          16384  0 
qca_nss_qdisc          61440  5 

This build doesn't include your very latest nss.qos improvements yet, without your improvements, here are some results: http://www.dslreports.com/speedtest/70165048

And for @dtaht the flent reports on this link (500Mbit/s up 500Mbit/s down):

So the above reports are what NSS fq_codel does on a symmetric fiber link with 500Mbit/s set to 450000 on ingress and egress:

config queue 'eth1'
        option debug_logging '1'
        option download '450000'
        option upload '450000'
        option verbosity '8'
        option linklayer 'ethernet'
        option overhead '50'
        option script 'nss.qos'
        option interface 'pppoe-wan'
        option qdisc 'fq_codel'
        option enabled '1'

I still get errors however, don't know how serious these are:

Sun Dec  5 17:22:40 2021 daemon.err modprobe: failed to find a module named act_ipt                                                                                                                                                         
Sun Dec  5 17:22:40 2021 daemon.err modprobe: failed to find a module named sch_fq_codel
...
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: ip: FAILURE (2): /sbin/ip link add name ifb4eth0 type ifb
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: ip: LAST ERROR: RTNETLINK answers: File exists
...
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 0 u32 match ip protocol 1 0xff flowid 1:13
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: tc: FAILURE (2): /sbin/tc filter add dev eth0 parent 1:0 protocol ipv6 prio 1 u32 match ip protocol 1 0xff flowid 1:13
Sun Dec  5 17:22:41 2021 user.notice SQM: ERROR: cmd_wrapper: tc: LAST ERROR: RTNETLINK answers: Not supported We have an error talking to the kernel 

Now, this is with the updated nss.qos from @KONG: http://www.dslreports.com/speedtest/70165177

And the flent reports:

I then also ran a test at Waveform to see of the large bloat on egress is also measured there like it is at DSLreports, but funny enough it doesn't seem to be the same:

From what I can gather, egress and ingress are handled differently by nss.qos. I assume this is by design? From what I can tell there's a large bufferbloat on uploading with 500Mbit/s links. Is there something I can do to improve that?