CAKE w/ DSCPs - cake-qos-simple

Community Builds, Projects & Packages
42 
root@OpenWrt:~# /etc/init.d/cake-qos-simple start
Setting up ingress handle for dev eth0.2.

No dl_if specified, so setting up appropriate IFB.

Creating IFB device: ifb-eth0.2.

Setting interface ifb-eth0.2 up.

Setting up tc filter to restore DSCPs from conntrack on ingress packets on interface eth0.2 and redirecting to IFB interface ifb-eth0.2.

Setting up CAKE on interface eth0.2 with bandwidth 0.7Mbit/s and options: diffserv8 dual-srchost nat wash ack-filter atm overhead 32 ether-vlan.

Setting up CAKE on interface ifb-eth0.2 with bandwidth 12Mbit/s and options: diffserv8 dual-dsthost nat nowash ingress no-ack-filter atm overhead 32 ether-vlan.

Started cake-qos-simple.

looks like the script is working as it should
now i should start testing the markings

1 Like
43

You can test with 'tcpdump -i ifb-eth0.2 -v'. And also 'service cake-qos-simple download'.

1 Like
45

My ISP (Comcast) marks ingress packets to CS1. I’ve tried setting ‘wash’ but cake-qos-simple isn’t able to then re-mark them.

I saw that you said it was possible to do with nftables ingress hook in the qosify thread. How would you suggest setting all ingress packets on the wan interface to CS0 before they are acted on by cake so that it would work with cake-qos-simple?

If I can get this sorted, it looks exactly like what I’m looking for!

46

Did you confirm this recently? If not could you do so, please? (Take a packet capture and look what kind of DSCPs you see on ingress, or just run the following on your router:

tcpdump -i pppoe-wan -v -n 'ip and (ip[1] & 0xfc) >> 2 != 0'

which should dump all (IPv4?) packets with non-zero DSCP field to the console)
I know of reports in the past of similar behaviour, I think this might be something Comcast could be willing to investigate, if we can confirm first that this is still on on-going issue?

47

I'd first verify using:

service cake-qos-simple download

which prints out the download cake stats, that packets are hitting the 'bulk' tin.

You can also use 'tcpdump' with '-v' at various points along the network pathway.

If we need to set incoming packets to cs0 then a challenge is that the nftables ingress hook is after the tc ingress hook.

I think what might work would be to explicitly set all outgoing packets to cs0 in this chain:

before the DSCPs are stored to conntrack, and then rely on the tc conntrack restore on ingress, which should restore the explicitly set DSCP that was stored to conntrack.

So I think if you just place:

jump dscp_set_besteffort

on the line before line 46 to give:

jump dscp_set_besteffort
jump classify-dscp
jump store-dscp-in-conntrack

then that might work?

But I'm not sure why this would be necessary because @moeller0 don't outgoing packets default to cs0 anyway absent intervention? In which case I'm not sure why the change above would be necessary because I'd have thought the cs0 packets would already get stored to conntrack on upload and then restored on download?

48

Yes, that is the default for all OSs, at least as far as I know...

@plater it would still be great if you could confirm that you still see CS1 from your ISP, and if you could estimate the fraction of CS1 traffic that would also be great!

49

I'm that case I'd have thought the packets on ingress are already getting reset to cs0 anyway given the conntrack restoration. Since in my nftables script we already set all DSCPs to conntrack on upload and in the tc script we restore from conntrack on download.

I think I can check because my ISP sets to 0x28 or something.

50

I have not tested your approach (my router is still using iptables/OpenWrt21) so I have no idea what happens here, but that should be easy to test, both by looking at cake's per-tin counters and by tcpdumps...

51

Just tested. Yeah so actually for the reasons provided above I think 'cake-qos-simple' already strips out any DSCPs set by ISP:

root@OpenWrt-1:~# tcpdump -i wan -v host google.com
tcpdump: listening on wan, link-type EN10MB (Ethernet), capture size 262144 bytes
12:01:44.835599 IP (tos 0x0, ttl 65, id 61325, offset 0, flags [none], proto ICMP (1), length 60)
    host.lan > maa05s22-in-f14.1e100.net: ICMP echo request, id 1, seq 24, length 40
12:01:45.190966 IP (tos 0x28, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 24, length 40
12:01:45.860049 IP (tos 0x0, ttl 65, id 61326, offset 0, flags [none], proto ICMP (1), length 60)
    host.lan > maa05s22-in-f14.1e100.net: ICMP echo request, id 1, seq 25, length 40
12:01:46.221060 IP (tos 0x28, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 25, length 40
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

root@OpenWrt-1:~# tcpdump -i ifb-wan -v host google.com
tcpdump: listening on ifb-wan, link-type EN10MB (Ethernet), capture size 262144 bytes
12:01:56.213086 IP (tos 0x0, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 28, length 40
12:01:57.235496 IP (tos 0x0, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 29, length 40
12:01:58.245440 IP (tos 0x0, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 30, length 40
12:01:59.271998 IP (tos 0x0, ttl 107, id 0, offset 0, flags [none], proto ICMP (1), length 60)
    maa05s22-in-f14.1e100.net > host.lan: ICMP echo reply, id 1, seq 31, length 40
^C
4 packets captured
8 packets received by filter
0 packets dropped by kernel

So I don't think @plater needs to do anything, albeit he should answer:

and double check that the incoming packets are indeed not getting set to the 'bulk' tin of cake by calling 'service cake-qos-simple download' (because cake-qos-simple should already be restoring cs0 on download anyway).

1 Like
52

I can confirm that ingress packets from Comcast are still marked CS1 (0x20).

2 Likes
53

Thanks, do you have any idea what proportion is marked as such? And any speculation what these packets have in common?

54

There is no non-CS1 packet received at all. I ran this a few minutes and nothing appeared.

tcpdump -i wan -nv '( ip and ip[1] & 0xfc != 32 )' and dst host mywanip

This has been Comcast’s modus operandi for years.

55

Just as a sanity check, if you do

tcpdump -i wan -nv '( ip and ip[1] & 0xfc == 32 )' and dst host mywanip

you get shiploads of packets? What is the precedence of '&' and '==' here?

56

Yes, I see all my traffic when I use = 32.

I have an nft ingress hook for my wan interface to set all incoming packets to cs0 because I only use CAKE on egress, but want WME to not see everything as bulk class.

1 Like
57

I confirmed it last night using Wireshark via sshdump. When I sorted by DSCP all packets were CS1.

Doubting myself I ran your command this morning and traffic began flowing by. It’s definitely still happening when I don’t wash ingress. When I do wash ingress your command doesn’t display any traffic. I also tried it with wash and nowash on egress. It had no effect on ingress.

I saw the PSA by Dave on Reddit a couple years back which I can confirm is still happening. I’m a novice at this stuff but I can probably generate a Wireshark log if proof is needed. Just let me know what options and fields you’d like me to select.

58

But @plater I think it doesn't matter because cake-qos-simple will just overwrite these with cs0 before redirecting to the IFB before cake sees the packets as explained above. Can you confirm this by pasting the output from 'service cake-qos-simple download'?

59

This is the result of watching youtube video via firefox on Linux. Everything is in bulk.

`root@OpenWrt:~# service cake-qos-simple download
qdisc cake 8070: root refcnt 2 bandwidth 88Mbit diffserv4 dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64 
 Sent 21512901 bytes 16063 pkt (dropped 1, overlimits 26590 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 346752b of 4400000b
 capacity estimate: 88Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh       5500Kbit       88Mbit       44Mbit       22Mbit
  target            5ms          5ms          5ms          5ms
  interval        100ms        100ms        100ms        100ms
  pk_delay       16.4ms          0us          0us          0us
  av_delay       12.9ms          0us          0us          0us
  sp_delay       11.9ms          0us          0us          0us
  backlog            0b           0b           0b           0b
  pkts            16064            0            0            0
  bytes        21514300            0            0            0
  way_inds            0            0            0            0
  way_miss           95            0            0            0
  way_cols            0            0            0            0
  drops               1            0            0            0
  marks               0            0            0            0
  ack_drop            0            0            0            0
  sp_flows            0            0            0            0
  bk_flows            1            0            0            0
  un_flows            0            0            0            0
  max_len         16126            0            0            0
  quantum           300         1514         1342          671

Edit: sorry for the formatting

60

That does not look right at all. It should look more like:

root@OpenWrt-1:~# service cake-qos-simple download
qdisc cake 801e: root refcnt 2 bandwidth 20Mbit diffserv4 dual-dsthost nonat nowash ingress no-ack-filter split-gso rtt 100ms noatm overhead 0
 Sent 15790986093 bytes 14528477 pkt (dropped 39883, overlimits 19164891 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 2393376b of 4Mb
 capacity estimate: 7500Kbit
 min/max network layer size:           50 /    1500
 min/max overhead-adjusted size:       50 /    1500
 average network hdr offset:           14

                   Bulk  Best Effort        Video        Voice
  thresh       1250Kbit       20Mbit       10Mbit        5Mbit
  target         14.5ms          5ms          5ms          5ms
  interval        110ms        100ms        100ms        100ms
  pk_delay        170us        658us       1.77ms        1.7ms
  av_delay         21us         51us        564us        330us
  sp_delay          9us          9us         14us         29us
  backlog            0b           0b           0b           0b
  pkts            39400     12615069      1640498       273393
  bytes         3223582  14128506690   1594257925    118293785
  way_inds            0       696154            0          237
  way_miss          104        32309           29         6533
  way_cols            0            0            0            0
  drops               0        39859           22            2
  marks               0           93            0            0
  ack_drop            0            0            0            0
  sp_flows            1            4            1            1
  bk_flows            0            1            0            0
  un_flows            0            0            0            0
  max_len           459        45342         1514         5918
  quantum           300          610          305          300

Is your nftables script set up? Can you post the output of:

nft list ruleset
61

I would be most interested to learn what you get just using the layer_cake script in sqm-scripts to set up cake and disable cake-qos-simple for the test. Looking at the output of tc -s qdisc is fine.
My goal here is to gather enough evidence and then try to point someone at Comcast at the issue, but for that to have even a slight chance this needs to clearly not be a consequence of and local DSCP marking rules....

62

@Lynx I'm looking over my setup with your scripts and will report back later

Things look better with layer_cake script in sqm-scripts. I used the luci app and had to set Squash DSCP (ingress) to 'DO NOT SQUASH' and Ignore DSCP (ingress) to 'Allow' in order to get diffserv3 and nowash on the ingress if that matters.

Does this look correct to you (tested watching youtube again)?

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1518 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64 
 Sent 5079509783 bytes 8758339 pkt (dropped 0, overlimits 0 requeues 13) 
 backlog 0b 0p requeues 13
  maxpacket 2916 drop_overlimit 0 new_flow_count 478 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev lan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan2 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan3 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev lan4 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a1: dev wan root refcnt 2 bandwidth 12500Kbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64 
 Sent 631013 bytes 5730 pkt (dropped 0, overlimits 5111 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 63Kb of 4Mb
 capacity estimate: 12500Kbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh      781248bit    12500Kbit     3125Kbit
  target         23.3ms          5ms       5.81ms
  interval        118ms        100ms        101ms
  pk_delay          8us        398us         12us
  av_delay          0us        124us          0us
  sp_delay          0us         12us          0us
  backlog            0b           0b           0b
  pkts               55         5670            5
  bytes            2970       627719          324
  way_inds            0            0            0
  way_miss            3           42            4
  way_cols            0            0            0
  drops               0            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            1            1            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len            54         3050           90
  quantum           300          381          300

qdisc ingress ffff: dev wan parent ffff:fff1 ---------------- 
 Sent 19535230 bytes 14724 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-guest root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wg0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan1 root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 80a2: dev ifb4wan root refcnt 2 bandwidth 90Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100ms noatm overhead 18 mpu 64 
 Sent 19743139 bytes 14723 pkt (dropped 1, overlimits 25411 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 225792b of 4500000b
 capacity estimate: 90Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       5625Kbit       90Mbit    22500Kbit
  target            5ms          5ms          5ms
  interval        100ms        100ms        100ms
  pk_delay       5.95ms         38us          8us
  av_delay       4.21ms          3us          0us
  sp_delay       1.13ms          3us          0us
  backlog            0b           0b           0b
  pkts            14685           36            3
  bytes        19737424         6934          180
  way_inds            0            0            0
  way_miss           43            3            1
  way_cols            0            0            0
  drops               1            0            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            1            0
  bk_flows            0            0            0
  un_flows            0            0            0
  max_len          4928          198           60
  quantum           300         1514          686
1 Like