So the 'wash' keyword instructs cake to re-mark the 6bit DSCP field in the IP header to 000000 aka CS0 before sending it out. If a packet has a different DSCP set and cake is configured for one of the diffserN modes, it will still use the original DSCP to sort a packet into one of the priority tiers, and only later re-mark to zero.
So qosify and 'wash" should be fully compatible with each other. On ingress however you need to keep in mind that re-marking all packets to CS0 will have all packets use the same access class AC_BE for all packets instead of using all four default ACs. IMHO that is not really a bad thing as especially AC_Vi and AC_VO have noticeable side effects on total segment throughput.
Users may also want to try out:
It's a very simple means to apply DSCPs using appropriate nftables rules (edit the example .nft file) and/or in LAN clients (e.g. set rules on Windows desktop) and those apply and are stored to conntrack on upload and using a service file tc rules are applied that restore DSCPs from conntrack on download and apply cake on upload and download.
In summary this:
- sets up cake on upload and download using a service file and hotplug file
- sets DSCPs on upload using easy to to configure nftables rules
- stores DSCPs on upload to conntrack (every connection in Linux has an associated conntrack to track upload and download components of any connection)
- restores DSCPs saved to conntrack on download
- requires minimal package dependencies
- is super flexible and lightweight
And thereby facilitates setting up cake with bidirectional DSCP handling.
Naturally as the author I personally think this is the simplest and most elegant solution .
yes thé marking dscp i playing to fps top
Ideally there would be a single line for nftables* to enable the copying of "egress" DSCPs to the conntrac "data-base", just as there is a single line for making tc copy the DSCPs from the conntrac database.
That would then work with whatever method a user wants to use to mark egress packets, be that mark at source application, mark on router via the firewall GUI or use an elaborate nft rule hierarchy. But as far as I understand we are not there yet...
*) Or tc or a modification to cake to do this all by itself, it already peeks into the conntrac database anyways so also extracting the DSCP would not be much work (but probably rejected upstream by being gross and non-generic)
22:27:15.145545 IP (tos 0x0, ttl 64, id 40396, offset 0, flags [none], proto UDP (17), length 117)
192.168.2.160.3074 > 5.200.29.29.41610: UDP, length 89
22:27:15.147488 IP (tos 0x80, ttl 51, id 43986, offset 0, flags [DF], proto UDP (17), length 1130)
5.200.29.29.41610 > 192.168.2.160.3074: UDP, length 1102
ok if I understand well dscpclassify already dscp 0x80 with cake diffserv4 (cs4 for real time interactive in input where I have 0x00 )
is right @moeller0
and happy new year everybody
more hour later i have this out
qdisc cake 801e: dev ifb4wan root refcnt 2 bandwidth 8Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 38 mpu 84
Sent 15035133 bytes 12718 pkt (dropped 472, overlimits 23799 requeues 0)
backlog 0b 0p requeues 0
memory used: 112896b of 4Mb
capacity estimate: 8Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 500Kbit 8Mbit 4Mbit 2Mbit
target 36.3ms 5ms 5ms 9.08ms
interval 131ms 100ms 100ms 104ms
pk_delay 0us 10.9ms 0us 334us
av_delay 0us 6.25ms 0us 6us
sp_delay 0us 9us 0us 6us
backlog 0b 0b 0b 0b
pkts 0 13183 0 7
bytes 0 15641791 0 632
way_inds 0 0 0 0
way_miss 0 128 0 5
way_cols 0 0 0 0
drops 0 472 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 0 4 0 1
bk_flows 0 2 0 0
un_flows 0 0 0 0
max_len 0 3028 0 158
quantum 300 300 300 300
Thank you for creating and sharing this. It looks AWESOME.
It seems like the only thing missing to replace QoSify is DSCP tagging based on the DNS host.
Probably the best long term solution would be to make dnsmasq 2.87 work on open wrt, which supports nftsets (Ipset equivalent).
There is already a thread here, but so far nobody was able to come up with a PR that was accepted.
I wondered if in elan cake qos script it is possible to add this famous bidirectional dscp
Thanks. I've written a few projects like that and others such as 'adblock-oisd' with the aim to provide a simple alternative.
Agreed.
I have the feeling there is a lot of effort that goes into managing DSCPs that's a waste of time because the bandwidth is already well adequate enough that the default 'sparse flow boost' in CAKE handles latency sensitive connections automatically. This has been discussed at length elsewhere on this forum with input from the original cake developers (just search 'sparse flow boost').
So it strikes me as a fruitless exercise for those very high bandwidth connections where you see huge long DSCP lists. Perhaps time spent on such exercises might be better spent with friends and family.
My understanding is that DSCPs should be used very sparingly to deal with situations where bandwidth falls below say 10Mbit/s (think crappy LTE or old-style copper ADSL) and you need to work out which connections to sacrifice during this period of constrained bandwidth. So e.g. you can place latency sensitive applications like Zoom and Teams in voice.
From this perspective, how important is it to classify based on domain?
Personally I like having LAN client set DSCPs based on executable (this is easy to set up in Windows).
And, in any case, it does indeed seem that domain based setting is just a matter of time anyway.
The theoretical problem with DNS based rules is that these boil down to IP addresses in the end, and ever since the advance of content delivery networks IP addresses are not 'diagnostic' for single services (at least for IPv4, not sure how CDNs work with IPv6).
I will alao take the opportunity to repeat my prioritization rant:
For every packet treated better/faster than average, other packets will need to be treated worse. The consequence is that under saturating conditions (and prioritization only really matters under those conditions) if too many packets are prioritized they essentially are not prioritized anymore. In the limit, if all packets are sorted into the highest priority tier the effect on delivery is identical to all packets being sorted into the default tier.
So independent of total bandwidth of a link*, prioritization works best if done as sparingly as possible, so that there are always plenty of packets to treat worse.
In the context of cake one additional thing to keep in mind is that the higher cake tiers are 'soft-limited', so if a tier exceeds its guaranteed bandwidth, the whole tier gets scheduled round robin with the best effort class. The consequence of which is that one better not put (too much) greedy traffic like TCP into the higher tiers... (This is subtle, as round-robin a single greedy flow in say Video with multiple greedy flows in BestEffort will still give the Video flow more throughput and less service delay, as roughly every 2nd packet will be taken from the Video class with its single flow, while in BestEffort all flows are serviced round-robin as well. But Relative better service depends of number of parallel flows in each tier, hence my hedgin 'not too much greedy traffic' above.)
*) What a link's maximum capacity affects is how likely one encounters saturating conditions...
it seems with DSCPCLASSIFY dscp marking are showing when i capture my routers packets with wireshark , but for some reason they don't seem to be shown in cake's diffserv4/8 tins ,very few packets packets are shown in the voice tin after playing csgo for few hours knowing that i marked my game with EF
could this be because of how this script's code is written ?
Seems like not much packets hitting your voice tin?
Whats the output of:
cat /etc/config/sqm
cat /etc/config/dscpclassify
Where did you capture your packets from? lan or wan? And can you please post the whole output of:
tc -s qdisc
When looking at your dscplassify config a few post ago you were prioritizing source and destination port 27000 – 27500. Are you sure this is the case when playing csgo?
config rule
option name 'CSGO'
option proto 'udp'
list src_port '27000-27500'
option class 'ef'
option proto 'tcp'
Try this rule and see if it helps…
config rule
option name 'COD'
option proto 'udp'
list src_port '3074'
list dest_port '30000-45000'
option class 'cs4'
option proto 'udp'
like this ?
Yea, looks good but I would leave list dest_port blank or define a bigger range as cod sometimes uses ports from 30000-60000.
This is my config for COD which seems to be working for me:
/etc/config/dscpclassify
config global 'global'
option class_bulk 'le'
option class_high_throughput 'af13'
option client_hints '1'
option threaded_client_min_bytes '10000'
option threaded_service_min_bytes '1000000'
option wmm '0'
config rule
option name 'Cod1'
option proto 'udp'
option dest_port '3074'
option class 'cs4'
option dest_ip '192.168.1.208'
config rule
option name 'Cod2'
option proto 'udp'
option src_port '3074'
option src_ip '192.168.1.208'
option class 'cs4'
The first rule (Cod1) isn’t actually needed!!!
sqm config:
config queue 'eth1'
option debug_logging '0'
option verbosity '5'
option qdisc_advanced '1'
option ingress_ecn 'ECN'
option egress_ecn 'ECN'
option linklayer 'ethernet'
option overhead '38'
option linklayer_advanced '1'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '84'
option linklayer_adaptation_mechanism 'default'
option qdisc_really_really_advanced '1'
option interface 'eth1'
option enabled '1'
option qdisc 'cake'
option script 'layer_cake_ct.qos'
option download '90000'
option upload '45000'
option iqdisc_opts 'nat dual-dsthost ingress diffserv4'
option eqdisc_opts ' nat dual-srchost ack-filter diffserv4'
option squash_ingress '0'
option squash_dscp '0'
thanks for sharing do you has test
tcpdump host 192.168.1.128 -i wan -v -n
tcpdump host 192.168.1.128 -i ifb-wan -v -n
tcpdump host 192.168.1.128 -i br-lan -v -n
can you show a tc -s qdisc
config rule
option name 'ICMP'
option proto 'icmp'
option class 'cs5' ??? not cs0 ??
option enabled '0'
config set
option name 'xcloud' ### i'm on ps5
option family 'ipv4'
option interval '1'
list element '13.104.0.0/14' # Western Europe
config rule
option name 'DNS'
list proto 'tcp'
list proto 'udp'
list dest_port '53'
list dest_port '853'
list dest_port '5353'
option class 'cs2'
config rule
option name 'Cod1'
option proto 'udp'
option dest_port '3074'
option class 'cs4'
option dest_ip '192.168.2.160'
config rule
option name 'Cod2'
option proto 'udp'
option src_port '3074'
option src_ip '192.168.2.160' ## my ps5
option class 'cs4'
config rule
option name 'ICMP'
option proto 'icmp'
option class 'cs0'
option enabled '1'
config rule
option name 'unmarked tcp src'
option proto 'tcp'
option src_port '!80 !443 !1935-1936'
option src_ip '192.168.2.160'
option class 'cs4'
config rule
option name 'unmarked tcp dst'
option proto 'tcp'
option dst_port '!80 !443 !1935-1936'
option dst_ip '192.168.2.160'
option class 'cs4'
config rule
option name 'unmarked udp src'
option proto 'udp'
option src_port '!80 !443'
option src_ip '192.168.2.160'
option class 'cs4'
config rule
option name 'unmarked tcp dst'
option proto 'udp'
option dst_port '!80 !443'
option dst_ip '192.168.2.160'
option class 'cs4'
config rule
option name 'unmarked tcp src'
option proto 'udp'
option src_port '!80 !443'
option src_ip '192.168.2.160'
option class 'cs4'
config rule
option name 'streamsrc'
option proto 'tcp'
option src_port '1935-1936, 2396, 2935'
option src_ip '192.168.2.160'
option class 'cs3'
config rule
option name 'streamdst'
option proto 'dst'
option dst_port '1935-1936, 2396, 2935'
option dst_ip '192.168.2.160'
option class 'cs3'
is just a suggestion what do you think ?
FWIW that is available on the master branch and I've been running 2.87 on 22.03 using patches from the staging repo of @ldir for months.
I'm not sure what the technical reasons are that are blocking it from being merged into 22.03 but the developer said he doesn't have the time in this PR.
This is the output of tcpdump. One capture from lan (eth0) and one from wan (eth1):
tcpdump udp port 3074 -i eth0 -v -n
16:33:22.002851 IP (tos 0x0, ttl 128, id 54163, offset 0, flags [none], proto UDP (17), length 113)
192.168.1.208.3074 > 5.200.29.138.39280: UDP, length 85
16:33:22.009151 IP (tos 0x80, ttl 54, id 21501, offset 0, flags [DF], proto UDP (17), length 410)
5.200.29.138.39280 > 192.168.1.208.3074: UDP, length 382
tcpdump udp port 3074 -i eth1 -v -n
16:40:28.398588 IP (tos 0x80, ttl 127, id 51325, offset 0, flags [none], proto UDP (17), length 113)
x.x.x.x.3074 > 95.179.217.195.30010: UDP, length 85
16:40:28.399563 IP (tos 0x0, ttl 49, id 53869, offset 0, flags [DF], proto UDP (17), length 550)
95.179.217.195.30010 > x.x.x.x.3074: UDP, length 522
So it seems to be working…
This is the output of tc -s qdisc:
tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 11705171284 bytes 11168275 pkt (dropped 0, overlimits 0 requeues 1784)
backlog 0b 0p requeues 1784
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 5020555951 bytes 4833737 pkt (dropped 0, overlimits 0 requeues 806)
backlog 0b 0p requeues 806
maxpacket 66616 drop_overlimit 0 new_flow_count 509 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 6684615333 bytes 6334538 pkt (dropped 0, overlimits 0 requeues 978)
backlog 0b 0p requeues 978
maxpacket 66616 drop_overlimit 0 new_flow_count 50878 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8028: dev eth1 root refcnt 9 bandwidth 45Mbit diffserv4 dual-srchost nat nowash ack-filter split-gso rtt 100ms noatm overhead 38 mpu 84
Sent 495579848 bytes 2287805 pkt (dropped 7617, overlimits 1172159 requeues 1082)
backlog 0b 0p requeues 1082
memory used: 246648b of 4Mb
capacity estimate: 45Mbit
min/max network layer size: 28 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 2812Kbit 45Mbit 22500Kbit 11250Kbit
target 6.46ms 5ms 5ms 5ms
interval 101ms 100ms 100ms 100ms
pk_delay 19us 202us 24us 101us
av_delay 7us 15us 6us 10us
sp_delay 2us 2us 3us 1us
backlog 0b 0b 0b 0b
pkts 2706 2224923 166 67627
bytes 428090 490462756 12780 5264586
way_inds 0 268659 0 2353
way_miss 37 54989 160 5258
way_cols 0 0 0 0
drops 0 47 0 0
marks 0 2 0 0
ack_drop 0 7570 0 0
sp_flows 1 3 1 1
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 1326 52990 90 1382
quantum 300 1373 686 343
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 6423467586 bytes 9063907 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.10 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.15 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.33 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.50 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev lxcbr0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.44 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev phy0-ap0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev tun1 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 10776 bytes 71 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev docker0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-303c81f631c5 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-dd0139594ff6 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev vethd5d15a1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8029: dev ifb4eth1 root refcnt 2 bandwidth 90Mbit diffserv4 dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 38 mpu 84
Sent 6586528610 bytes 9062896 pkt (dropped 1011, overlimits 8569202 requeues 0)
backlog 0b 0p requeues 0
memory used: 1251072b of 4500000b
capacity estimate: 90Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14
Bulk Best Effort Video Voice
thresh 5625Kbit 90Mbit 45Mbit 22500Kbit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 131us 94us 34us 51us
av_delay 11us 13us 2us 17us
sp_delay 3us 3us 2us 0us
backlog 0b 0b 0b 0b
pkts 4720 5792888 103 3266196
bytes 1999081 6383595859 9270 202291970
way_inds 0 329148 0 0
way_miss 37 69476 103 295
way_cols 0 0 0 0
drops 0 1011 0 0
marks 0 3 0 0
ack_drop 0 0 0 0
sp_flows 1 3 2 1
bk_flows 0 0 0 0
un_flows 0 0 0 0
max_len 1330 39364 90 1382
quantum 300 1514 1373 686
qdisc fq_codel 0: dev tun0 root refcnt 2 limit 10240p flows 1024 quantum 1500 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
Sent 6463 bytes 27 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
I think you don't need as many rules. My approach is just prioritizing what’s really needed and let cake do the rest of the magic. Just my opinion.
root@OpenWrt:~# tcpdump udp port 3074 -i br-lan -v -n
tcpdump: listening on br-lan, link-type EN10MB (Ethernet), capture size 262144 bytes
192.168.2.160.3074 > 72.26.219.145.44150: UDP, length 125
16:32:58.179672 IP (tos 0x80, ttl 50, id 62948, offset 0, flags [DF], proto UDP (17), length 303)
root@OpenWrt:~# tcpdump udp port 3074 -i wan -v -n
tcpdump: listening on wan, link-type EN10MB (Ethernet), capture size 262144 bytes
16:33:04.728524 IP (tos 0x80, ttl 63, id 11822, offset 0, flags [none], proto UDP (17), length 153)
192.168.XXXXXX.3074 > 72.26.219.145.44150: UDP, length 125
16:33:04.736867 IP (tos 0x80, ttl 63, id 31734, offset 0, flags [none], proto UDP (17), length 153)
192..XXXXXX.3074 > 72.26.219.145.44150: UDP, length 125
yes thanks for all hudra
he seems worked like a charmed
Do you think you could create a separete thread introducing cake-qos-simple so that we can discuss it without distracting this thread?
I could create a new thread myself, but I figured it would be better if you "owned it", since you are the creator of this solution.
Sure - see here: