CAKE w/ DSCPs - cake-qos-simple

This thread is for discussing cake-qos-simple:

cake-qos-simple offers a simple means to facilitate setting up cake with bidirectional DSCP handling.

In summary cake-qos-simple:

  • sets up cake on upload and download using a service file and hotplug file
  • sets DSCPs on upload using easy to to configure nftables rules
  • stores DSCPs on upload to conntrack (every connection in Linux has an associated conntrack to track upload and download components of any connection)
  • restores DSCPs saved to conntrack on download
  • requires minimal package dependencies
  • is super flexible and lightweight
6 Likes

how to set download and upload limit with this script ?

Thank you for creating this script. After reading through the code, it seems like it could be the best solution for QoS with cake.

But when I tried to use it, I couldn't get it working. Perhaps you could help me with that?

My issues

It seems that the init.d script references the wan interface as "wan", but my wan interface is eth0.

I tried to change all instances of wan to eth0, but I am not sure if all of them are references to the device or a keyword.

And it still didn't work after making that change. The log showed:

RTNETLINK answers: File exists
Error: Exclusivity flag on, cannot modify.
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
Error: Exclusivity flag on, cannot modify.
Error: Exclusivity flag on, cannot modify.

And here are the commands and their outputs when I run them directly:

root@FriendlyWrt:~/cake-qos-simple# ip link add name ifb-wan type ifb

root@FriendlyWrt:~/cake-qos-simple# ip link set ifb-wan up

root@FriendlyWrt:~/cake-qos-simple# tc qdisc add dev eth0 handle ffff: ingress
Error: Exclusivity flag on, cannot modify.

root@FriendlyWrt:~/cake-qos-simple# tc filter add dev eth0 parent ffff: protocol ip matchall action ctinfo dscp 63 128 action mirred egress redirect dev ifb-wan
RTNETLINK answers: Invalid argument
We have an error talking to the kernel

root@FriendlyWrt:~/cake-qos-simple# tc qdisc add dev eth0 root cake bandwidth 10Mbit diffserv4 dual-srchost nonat wash no-ack-filter noatm overhead 0

root@FriendlyWrt:~/cake-qos-simple# tc qdisc add dev ifb-wan root cake bandwidth 10Mbit diffserv4 dual-dsthost nonat nowash ingress no-ack-filter noatm overhead 0
root@FriendlyWrt:~/cake-qos-simple#

Issues I noticed

I also noticed two issues...

The default config causes fw4 syntax errors

This section...

# local MAC addresses to set to bulk (e.g. IoT devices)
define BULK_MACS = {
#	XX,
#	YY
}

... causes syntax errors in fw4:

# fw4 reload
Section @zone[1] (wan) IPv4 fullcone enabled for zone 'wan'
Section @zone[1] (wan) IPv6 fullcone enabled for zone 'wan'
Section @rule[14] (Allow iperf3 on WAN) is disabled, ignoring section
Section @rule[15] (Allow librespeed) is disabled, ignoring section
Automatically including '/root/cake-qos-simple/cake-qos-simple.nft'
In file included from /dev/stdin:398:1-52:
/root/cake-qos-simple/cake-qos-simple.nft:21:6-6: Error: syntax error, unexpected newline
#	XX,
 	   ^
In file included from /dev/stdin:398:1-52:
/root/cake-qos-simple/cake-qos-simple.nft:23:1-1: Error: syntax error, unexpected '}'
}
^

Removing the commented out lines also didn't work. It seems like you can't have an empty set in nftables.

Errors in the cake-qos-simple.nft prevent all other firewall rules from being loaded

I rebooted my router after adding cake-qos-simple and suddenly none of the routing would work anymore.

When I ran nft list ruleset, I saw that none of the fw4 openwrt rules were loaded.

When I ran fw4 reload, I saw the above error. After uncommenting the empty set and the rule that uses it, I was able to load all the rules.

This script completely breaking networking is not ideal. Especially when this happens in the default config.

Perhaps there is a way to load the firewall rules in a way that doesn't prevent all other rules from loading? If a syntax error breaks the DSCP tagging that's not great, but still way better than a sudden stop of all routing and firewalling on the next reboot.

This is especially critical since the user might not even notice the issue initially. There was no visible error anywhere about the syntax issues in the .nft file.

So tracking it back to this is not trivial.

And a few suggestions on how to improve the usability

Add a variable for the wan interface

My suggestion would be to add a WAN_INTERFACE variable at the top of the script to help users configure the correct interface and then use that throughout the script instead of hardcoding wan.

Make the init.d script more userfriendly

I found it very confusing that the init.d script didn't output anything. To me that was a sign that everything was ok.

Only later did I realize that there is a logfile and all output ends up in there.

I would suggest to add echos that describe what the next command is going to be so that it's clear which command the error output belongs to.

Something like echo "Creating ifb-wan interface...".

Add a help section to the init.d script

To make it easier to know which commands are available, it would be great to see a help when you run the init.d script.

Here is how it looks with QoSify:

# /etc/init.d/qosify
Syntax: /etc/init.d/qosify [command]

Available commands:
	start           Start the service
	stop            Stop the service
	restart         Restart the service
	reload          Reload configuration files (or restart if service does not implement reload)
	enable          Enable service autostart
	disable         Disable service autostart
	enabled         Check if service is started on boot
	running         Check if service is running
	status          Service status
	trace           Start with syscall trace
	info            Dump procd service info

Add a way to easily see the status of cake

I'm sorry that I keep comparing this to QoSify, but I think it's currently the main competitor for your script and they really nailed the usability of the package.

You can run...

# qosify-status
===== interface wan: active =====
egress status:
qdisc cake 800c: root refcnt 9 bandwidth 100Mbit diffserv4 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 42 mpu 84
 Sent 10537 bytes 49 pkt (dropped 0, overlimits 1 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1408b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           40 /     751
 min/max overhead-adjusted size:       84 /     793
 average network hdr offset:            2

                   Bulk  Best Effort        Video        Voice
  thresh       6250Kbit      100Mbit       50Mbit       25Mbit
  target            5ms          5ms          5ms          5ms
  interval        100ms        100ms        100ms        100ms
  pk_delay          9us         20us          0us          0us
  av_delay          0us          2us          0us          0us
  sp_delay          0us          2us          0us          0us
  backlog            0b           0b           0b           0b
  pkts                3           46            0            0
  bytes            1278         9259            0            0
  way_inds            0            0            0            0
  way_miss            1           17            0            0
  way_cols            0            0            0            0
  drops               0            0            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            0            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len           725          765            0            0
  quantum           300         1514         1514          762

[...]

And it shows you alll the important info. It seems like it's mainly the output of tc -s qdisc show dev eth0.

So it would be easy to add to the init.d script as a status or info command. That would make it much easier for beginners to debug and use.

Conclusion

I want to stress once more that I think this solution is awesome and could becom the ideal QoS solution on openwrt. So I hope this feedback is not discouraging. :slight_smile:

2 Likes

his normal i think you should be just comment

the last line

Thanks for the detailed feedback @Kimcha. I can easily add those features. I suppose at the moment this is more just a recipe rather than plug and play but I agree we should probably parameterize it all and provide those commands like status etc.

@jow changed the fw4 behaviour in 22.03 such that a syntax error in an nft file (in this case just leaving XX, YY in a set without uncommenting) will flag an error on service firewall restart and not actually restart the firewall. I think it should go further such that on router reboot each file will be checked before the file is included because at the moment it seems a file with a syntax error will bring the whole tree down in router reboot. But that's not my call to make.

I'd have thought just replacing all instances of 'wan' with 'eth0' should work and did you install the packages listed in the readme? Since from memory the error message you quoted above relates to a missing package.

1 Like

hi lynx i has a same router like you i has tester

nonerror

but like Say Kimcha

thé script doesnt work for thé moment

thé firewall make dont appair thé rules

i dl all packages also :slightly_smiling_face:

Amazing, thank you!

Yeah that doesn't seem optimal. Maybe that's why dscpclassify was using a config file instead of a .nft script for the rules.

Yes, I installed all the packages and changed wan to eth0. I even tried to change ifb-wan to ifb-eth0 as I saw that interface name somewhere, but it still didn't work.

How about this:

@moeller0 any advice on the above?

1 Like

hi lynx yes I think this approach is much better

to declare an up and dl

thanks for your contribution i appreciate it :wink:

I'll give it a try and let you know

I don't always have the translator with me ^^

I assume we will need to iron out minor issues. @Kimcha please can you also test?

1 Like

just a question is possible to put

?

cake_ul_rate_Mbps=16.8 # cake upload rate in Mbit/s
cake_dl_rate_Mbps=48.2  # cake download rate in Mbit/s

1 Like

I tested and it works.

1 Like

do you have an error actually i has that

In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:92:29-29: Error: syntax error, unexpected '{', expecting string
        chain dscp_set_bulk {
                            ^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:93:20-23: Error: syntax error, unexpected dscp, expecting string
                ip dscp set cs1
                   ^^^^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:94:9-9: Error: syntax error, unexpected '}'
        }
        ^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:97:35-35: Error: syntax error, unexpected '{', expecting string
        chain dscp_set_besteffort {
                                  ^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:98:20-23: Error: syntax error, unexpected dscp, expecting string
                ip dscp set cs0
                   ^^^^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:99:9-9: Error: syntax error, unexpected '}'
        }
        ^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:102:30-30: Error: syntax error, unexpected '{', expecting string
        chain dscp_set_video {
                             ^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:103:20-23: Error: syntax error, unexpected dscp, expecting string
                ip dscp set cs2
                   ^^^^
In file included from /dev/stdin:233:1-65:
/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft:104:9-9: Error: syntax error, unexpected '}'
        }
        ^

Weird - are you using:

without any changes? Or can you post your .nft file with edits here?

1 Like

is my fault is good now :wink:

Automatically including '/usr/share/nftables.d/ruleset-post/cake-qos-simple.nft'
Automatically including '/usr/share/nftables.d/table-post/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/chain-post/dstnat/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/chain-post/forward/20-miniupnpd.nft'
Automatically including '/usr/share/nftables.d/chain-post/srcnat/20-miniupnpd.nft'

thanks :slight_smile:

OK - next question - do:

service cake-qos-simple
service cake-qos simple enable
service cake-qos-simple restart

work?

1 Like

i put tap one by one ?
like this

root@OpenWrt1er-router:~# service cake-qos-simple
Syntax: /etc/init.d/cake-qos-simple [command]

Available commands:
        start           Start the service
        stop            Stop the service
        restart         Restart the service
        reload          Reload configuration files (or restart if service does not implement reload)
        enable          Enable service autostart
        disable         Disable service autostart
        enabled         Check if service is started on boot

        cake-qos-simple custom commands:

        upload          show stats for upload interface
        download        show stats for download interface
root@OpenWrt1er-router:~#

for work i has make like this

/etc/init.d/cake-qos-simple enabled
then
/etc/init.d/cake-qos-simple start

then i has restart my firewall

the rules in fw4 appair good now

1 Like

Yes, and then please show output of 'tc qdisc ls'.

1 Like

for my test i has just put 4 and 2 :wink: and he work

my ouput i has add tc -s qdisc too

root@OpenWrt1er-router:~# tc qdisc ls
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
qdisc noqueue 0: dev lan1 root refcnt 2
qdisc noqueue 0: dev lan2 root refcnt 2
qdisc noqueue 0: dev lan3 root refcnt 2
qdisc noqueue 0: dev lan4 root refcnt 2
qdisc cake 800b: dev wan root refcnt 2 bandwidth 2Mbit diffserv4 dual-srchost nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 38
qdisc ingress ffff: dev wan parent ffff:fff1 ----------------
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev wlan1 root refcnt 2
qdisc noqueue 0: dev wlan1-1 root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc noqueue 0: dev wlan0-1 root refcnt 2
qdisc cake 800c: dev ifb-wan root refcnt 2 bandwidth 4Mbit diffserv4 dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 38
root@OpenWrt1er-router:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 103631382 bytes 227533 pkt (dropped 0, overlimits 0 requeues 4)
 backlog 0b 0p requeues 4
  maxpacket 6072 drop_overlimit 0 new_flow_count 405 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan2 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan3 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan4 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800b: dev wan root refcnt 2 bandwidth 2Mbit diffserv4 dual-srchost nonat wash no-ack-filter split-gso rtt 100ms noatm overhead 38
 Sent 9109378 bytes 14924 pkt (dropped 647, overlimits 13027 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 49736b of 4Mb
 capacity estimate: 2Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       66 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh        125Kbit        2Mbit        1Mbit      500Kbit
  target          145ms       9.08ms       18.2ms       36.3ms
  interval        291ms        104ms        113ms        131ms
  pk_delay          0us       54.9ms          0us       2.12ms
  av_delay          0us       20.3ms          0us         68us
  sp_delay          0us         10us          0us         30us
  backlog            0b           0b           0b           0b
  pkts                0        15483            0           88
  bytes               0     10018701            0         7606
  way_inds            0            0            0            0
  way_miss            0          116            0           80
  way_cols            0            0            0            0
  drops               0          647            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            2            0            0
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0        11632            0          410
  quantum           300          300          300          300

qdisc ingress ffff: dev wan parent ffff:fff1 ----------------
 Sent 15181117 bytes 18249 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1-1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0-1 root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800c: dev ifb-wan root refcnt 2 bandwidth 4Mbit diffserv4 dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 38
 Sent 14458238 bytes 17486 pkt (dropped 658, overlimits 16991 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 129336b of 4Mb
 capacity estimate: 4Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       84 /    1538
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh        250Kbit        4Mbit        2Mbit        1Mbit
  target         72.7ms          5ms       9.08ms       18.2ms
  interval        168ms        100ms        104ms        113ms
  pk_delay          0us       1.54ms          0us       2.37ms
  av_delay          0us         88us          0us         79us
  sp_delay          0us         11us          0us         22us
  backlog            0b           0b           0b           0b
  pkts                0        18063            0           81
  bytes               0     15439094            0        11974
  way_inds            0           13            0            1
  way_miss            0          657            0           75
  way_cols            0            0            0            0
  drops               0          658            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            1            0            6
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len             0         4542            0          385
  quantum           300          300          300          300

root@OpenWrt1er-router:~#

earlier I had an idea as I play games tell me what you think, to create several rules example like this

hence my mistake

udp . 3074 : goto dscp_set_voice,  # COD1
			tcp . 3074 : goto dscp_set_voice,  # COD2
			tcp . 1935 : goto dscp_set_video,  # TWITCH
			tcp . 80 : goto dscp_set_besteffort,  # BROWSING
			tcp . 443 : goto dscp_set_besteffort,  # BROWSING 
			tcp . 51413 : goto dscp_set_bulk ## BULK

1 Like

Is it tinning correctly on download? You should see the counters with:

service cake-qos-simple download

1 Like