OK. My guess is that your nftables wasn't setup correctly.
This looks like you truly get the bulk of your ingress packets marked CS1... with WiFi WMM and cake that is not going to be ideal... (now cake does not care all that much, given your shaper rate is high enough, but WiFi will potentially be less fun...)
I got it working! I think the documentation should include āservice firewall restartā. After doing that I received a message āAutomatically including '/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft'ā
That was my problem. Noobish as it is. It might help others.
Thank you for this! This is the icing on the cake!
Edit: After some more playing I think my problem might have been a bad edit I made in cake-qos-simple.nft. Issuing āservice firewall restartā notified me of the error but cake-qos-simple did not.
Also, when I run ā/etc/init.d/cake-qos-simple restartā. I get the following message:
Removing ingress handle on wan.
Removing CAKE on wan.
Removing CAKE on ifb-wan.
Setting IFB interface ifb-wan down.
Removing IFB interface ifb-wan.
Stopped cake-qos-simple.
Setting up ingress handle for dev wan.
Setting up tc filter to restore DSCPs from conntrack on ingress packets on interface wan.
Setting up CAKE on interface wan with bandwidth 12.5Mbit/s and options: diffserv4 dual-srchost nonat wash no-ack-filter docsis.
Setting up CAKE on interface ifb-wan with bandwidth 88Mbit/s and options: diffserv4 dual-dsthost nonat nowash ingress no-ack-filter docsis.
Cannot find device "ifb-wan"
Started cake-qos-simple.
And the following status: You can see ifb-wan isnāt setup.
oot@OpenWrt:~# /etc/init.d/cake-qos-simple status
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc cake 8011: dev wan root refcnt 2 bandwidth 12500Kbit diffserv4 dual-srchost nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64
qdisc ingress ffff: dev wan parent ffff:fff1 ----------------
qdisc noqueue 0: dev br-guest root refcnt 2
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev wg0 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
To restart Iām forced to issue:
/etc/init.d/cake-qos-simple stop && /etc/init.d/cake-qos-simple start
Different approaches...
Quosify:
- uses an efficient eBPF approach to change (ingress) DSCPs based on simple rules before ingress qdisc get hold of the packets.
- can interface with dnsmasq to essentially fill its own ipsets from dnsmasq DNS data, so can filter on domain-names (IIUC it will filter based on IP addresses returned for the domain names, so it is possible to get false positives if different websites share the same IP address, which with the popularity of CDNs is not totally theoretical)
- is currently limited in that it can "filter" on either IP addresses or external port, but not on combinations of both
- can not use internal port numbers
- can not use internal IP addresses (it operates before network address translation happens)
cake-qos-simple:
- uses the tc action that allows to restore the DSCP per connection based on the DSCP stored in the kernel's connection tracking database
- it uses nftables to actually copy DSCPs from egress packets into the conntrack database (which currently is not super elegant due to nftables limitations)
- by using nftables it can easily filter on ip-address:port combinations
- by using nftables it can easily filter internal and external IP addresses and ports
- can operate with DSCPs set by applications or by the router (a policy decision)
So there are plenty of differences between the two...
If at all cake-qos-simple is comparable to DSCP classify as both use the same elegant tc action approach, that @ldir invented. How these two differ I can not really tell you, as I tried none of those myself (both due to lacking a compatible router, my turris omnia is still on OpenWrt21 based TOS6 and hence offers neither of the three approaches and due to lacking the need). Conceptually, I think that the conntrack method is more elegant but qosify likely is more efficient...
2 Likes
Probably we should just add a 'sleep 1' at the end of the stop() routine in the service file. Can you verify that that fixes the restart issue? If so I will push that commit.
Another thing that I like about this approach is that LAN clients can set DSCPs on upload and they will get restored on download. And that's optional - DSCPs can be set by LAN clients and/or router.
Probably we should just add a 'sleep 1' at the end of the stop() routine in the service file. Can you verify that that fixes the restart issue?
No that didnāt solve the issue. I tried peppering sleep throughout the stop function but that didnāt work either. I still get the same message even with this:
stop()
{
printf "Removing ingress handle on ${ul_if}.\n"
tc qdisc del dev ${ul_if} ingress
printf "\nRemoving CAKE on ${ul_if}.\n"
tc qdisc del dev ${ul_if} root
if [[ -z ${dl_if} ]]; then
dl_if=ifb-${ul_if}
printf "\nRemoving CAKE on ${dl_if}.\n"
sleep 1
tc qdisc del dev ${dl_if} root
printf "\nSetting IFB interface ${dl_if} down.\n"
sleep 1
ip link set ${dl_if} down
sleep 1
printf "\nRemoving IFB interface ${dl_if}.\n"
ip link del ${dl_if}
else
printf "\nRemoving CAKE on ${dl_if}.\n"
tc qdisc del dev ${dl_if} root
fi
printf "\nStopped cake-qos-simple.\n"
logger -t cake-qos-simple "Stopped cake-qos-simple."
sleep 1
}
OK I see the same thing myself (I never use 'restart'). Let's see...
Using random sleeps to fix a sequencing/race condition is typically not a super robust solution 
Any clue what's causing this oddity? Calling:
'service cake-qos-simple stop; service cake-qos-simple start' works fine, but 'restart' does not.
And yet isn't this what rc.common does:
Ah, perhaps it's calling stop and start at the same time? @dave14305 perhaps you can rescue us?
Is dl_if defined or empty at the top of the script?
1 Like
Empty for me. But this feels like it's relevant.
Ah yes, ${dl_if} gets set by the stop() and reused for the start() within the same shell context.
Now I'm thinking about the most elegant way to address this.
@plater can you please confirm that just adding:
dl_if=""
to give:
stop()
{
printf "Removing ingress handle on ${ul_if}.\n"
tc qdisc del dev ${ul_if} ingress
printf "\nRemoving CAKE on ${ul_if}.\n"
tc qdisc del dev ${ul_if} root
if [[ -z ${dl_if} ]]; then
dl_if=ifb-${ul_if}
printf "\nRemoving CAKE on ${dl_if}.\n"
tc qdisc del dev ${dl_if} root
printf "\nSetting IFB interface ${dl_if} down.\n"
ip link set ${dl_if} down
printf "\nRemoving IFB interface ${dl_if}.\n"
ip link del ${dl_if}
dl_if=""
else
printf "\nRemoving CAKE on ${dl_if}.\n"
tc qdisc del dev ${dl_if} root
fi
printf "\nStopped cake-qos-simple.\n"
logger -t cake-qos-simple "Stopped cake-qos-simple."
}
fixes this 'restart' issue at your end? It resolves it for me.
If so, I'll commit this. Albeit I've never actually needed to use 'restart'.
1 Like
Yes, that fixes it on my end too. Thank you!
As long as I donāt change the interface names it seems safe to edit cake-qos-simple while itās still running. This is my use for ārestartā. Leaving it up while I edit then issuing a restart when Iām ready to commit the changes. Is there any other reason to avoid ārestartā?
Nah should be fine. And in fairness users are quite legitimate in expecting a simple service restart to work.
Your suggested edit to README:
And the 'service firewall restart' fix:
By the way nftables is super flexible for setting DSCPs. Let me know if you need any help with that. Personally I like setting them using my LAN clients though.
1 Like
Will do. Thank you again!
I'm using a mix of both LAN and application. Windows is great for this. Linux has been more interesting. For example Transmission allows you to set
"peer-socket-tos": "lowcost",
Which sets egress to CS1 but the requests are still sent at (canāt remember exactly) but they fall within CS0. cake-qos-simple correctly then remarks the ingress as CS0. Next step might be to put it in a container but Iām not ready to cross that bridge yet.
1 Like
Iām having trouble with nftables. If I add a rule to cake-qos-simple.nft and issue āfw4 reload/restartā or āservice firewall reload/restartā the output of ānft list rulesetā confirms the new entry.
However, if I remove rules from cake-qos-simple.nft and issue the same reload or restart command the output of ānft list rulesetā shows that the rules haven't been removed.
Iāve also tried āservice network restartā but that doesnāt work either. The only way Iāve found to clear removed entries is to reboot the router.
fw4 stop/restart will only clear the inet fw4 table. It wonāt touch extra tables like inet cake-qos-simple. You could use fw4 flush but thereās probably other ways. Iām not sure why flush table inet cake-qos-simple in the nft file wouldnāt take care of this already.
1 Like
Nor me. I think that's why I put that flush call in there.
@plater is it perhaps when you reload and the nft file contains errors and hence the reload is aborted?
Otherwise I'm not sure why old rules aren't getting wiped. That is presuming they are all part of the inet cake-qos-simple table. If you are defining new table then you can just add a further flush call for that new table.
Unless some adds a new table to the file without flushing firstā¦ 
I look forward to when conntrack restore can be done in nftables too.