CAKE 'wash' option

Is it to be expected that if restoring conntrack on br-guest egress as per:

 tc filter add dev br-guest parent 1: protocol ip handle 2 fw flowid 1:1 action ctinfo dscp 63 128 action mirred egress redirect dev ifb-dl

that the 'wash' option in tc-cake will not be effective in scrubbing the restored DSCPs:

tc qdisc add dev ifb-dl root cake bandwidth 25Mbit diffserv4 dual-dsthost nonat wash ingress no-ack-filter noatm overhead 92

downstream of the br-guest egress, i.e. at the bridge members including wlan0-1?

I ask because this indeed seems to be the case from my testing:

tcpdump -i wlan0-1 -v ip[1]!=0
tcpdump: listening on wlan0-1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:51:46.000993 IP (tos 0x2,ECT(0), ttl 37, id 2679, offset 0, flags [DF], proto TCP (6), length 1065) > LGwebOSTV.lan.48484: Flags [P.], cksum 0xa4c8 (correct), seq 3925:4938, ack 10176, win 577, options [nop,nop,TS val 178386263 ecr 4743160], length 1013

It will egress ifb-dl as CS0 due to wash.

In that case why am I not seeing that as above?

Namely I see non zero DSCPs at wlan0-1 as in my tcpdump, suggesting the wash was not effective - hence my post.

I wonder if it is too late to try to wash at this point (cake on IFB crafted from br-guest egress).

Cake will wash as it’s determining the tin. If it’s tinning it properly then it’s washing too. What happens after it transmits the packet in your setup is unknown. Too many moving parts.

1 Like

Thanks @dave14305 that's helpful to see where it appears in the code.

Copied over from a parallel discussion of the same topic (sorry, did not see this one):

I think the issue here is that TOS 0x2 means the second lowest bit is set -> 00000010
but the two lowest bits where removed from the old type of service byte to create the new ECN field, while the upper 6bit now make up the diffserv field that holds the 6bit diffserv code points (DSCP).
As wireshark hepfully remarks: tos 0x2,ECT(0)
See rfc3168 page 6

      | ECN FIELD |
        ECT   CE         [Obsolete] RFC 2481 names for the ECN bits.
         0     0         Not-ECT
         0     1         ECT(1)
         1     0         ECT(0)
         1     1         CE

ECT(0) means the higher ECN bit is set, but that is the same as TOS 0x2.

Wash, as you know, is careful not to mess with the ECN bits, as cake optionally uses these to signal congestion, so washing these is counter-productive.

1 Like

Follow up question: I presume WireGuard encapsulation will scrub those, but if they are used to optionally signal congestion then I suppose that is yet another reason that the WireGuard maintainer ought to be convinced to provide a toggle for the user to:

  1. preserve ECN; or

  2. preserve DSCP?

BTW maybe WireGuard encapsulation already preserves ECN I haven't tested exactly that. But I presume it'll scrub.

I assume wireguard will preserve the ECN bits (I seem to recall looknig it up in wireguards source code some years ago, but I am not 100% sure it actually was wireguard I looked at).
It seems wireguards principled developer desires wireguard to be secure out of the box and by default, and he considers DSCPs a "side-channel" he is not interested in opening, I guess there is toke's solution around that you posted earlier.

Yes, I think it does. However, I have not actually run wireguard myself so have zero first hand experience. Why? Because I only use VPN to log into my home network from outside, and do to it being more conveniently supported by turris OS I simply used OpenVPN, which has the advantage of also working over TCP. Yes, tunneling TCP through another TCP is generally a bad idea, but I am really only use VPN very lightly so the higher chance of being able to reach it from remote locations outweighs the fact tat it will not be all that useful for big file transfers.

1 Like

i just found this:

So if I read that correctly WireGuard already does 1) so the only issue is convincing the maintainer to also facilitate the optional override to provide 2) for those that want to put CAKE at wan?

Good luck with that ;). I really think the best way forward is to try to adapt Toke's eBPF code to achieve the same, if possible on an IFB

Or set up dual IFBs - that works too, albeit requires some careful fwmarking and tc filtering to handle it all and deal with e.g. DNS hijacking. But after quite a lot of time I think I have it all figured out now.


What I wanted to convey is that I do not think wireguard is going to grow a mode to propagate DSCPs any time soon, and for understandable reasons (I guess we are lucky that ECN is still propagated in spite of allowing to exfiltrate 1-1.5 bits per packet).

1 Like

For those of us that use nftables to wash on wan upload, I presume this:

ip dscp set cs0

will also not overwrite the ECN bits of the TOS value.

Does CAKE use this ECN mechanism by default or does it require specific enabling? Will it work on an LTE 4G connection?

I have not checked that but testing that should not be hard any ECT(0)/ECT(1)/CE marks (as reported by wireshark, so tos 0x1, 0x2, 0x3) need to survive past the set command.

Yes if cake decided to signal congestion for a packet (based on the CoDel component) if will either drop that packet OR, if the IP header carries either ECT(00 or ECT(1), it will mark the packet by setting the ECN bits to the congestion experienced pattern (CE). Cake's BLUE component however will drop packets independent of their ECN bit pattern, it acts as a back-stop for persistently unresponsive flows.
Most servers on the internet will happily negotioate and use ECN signaling if the remote client politely asks for it, so if you want to use ECN you will need to enable it on your ed-hosts (again Linux defaults to accepting ECN but you need to configure it so it actively tries to request ECN usage from servers).

Why does MS Windows have it disabled by default I wonder:

Microsoft Windows [Version 10.0.22000.856]
(c) Microsoft Corporation. All rights reserved.

Querying active state...

TCP Global Parameters
Receive-Side Scaling State          : enabled
Receive Window Auto-Tuning Level    : normal
Add-On Congestion Control Provider  : default
ECN Capability                      : disabled
RFC 1323 Timestamps                 : allowed
Initial RTO                         : 1000
Receive Segment Coalescing State    : enabled
Non Sack Rtt Resiliency             : disabled
Max SYN Retransmissions             : 4
Fast Open                           : enabled
Fast Open Fallback                  : enabled
HyStart                             : enabled
Proportional Rate Reduction         : enabled
Pacing Profile                      : off

Likely dumb question, but the answer will help my understanding, is it possible to override in the router to force ECN capability on, even if end hosts have not had it enabled?

Also I'm a little confused about TOS/ECT. If TOS=0x2 means ECN bit is set, doesn't that mean we should never use 'cs1' to indicate bulk flow(?):

Oh wait, the point is that 0x2 != 0x20 right?

Because early on the roll-out of ECN some hops (I guess with overzealous firewalls) caused problems (like simply silently dropping ECT marked packets). So disabling ECN ist sort of the save option. But by now these problematic hops should be fixed.
Another possibility could be that Microsoft might have drunk the L4S cool-aid and disabled rfc3168 ECN completely to make deployment of L4S simpler. But I would bet on the first alternative...

No, you can manipulate the ECN bitfield, and make packets appear to use ECN (by setting ETC(1) or ET(1)) but unless the endpoints actually negotiated to use ECN these will not force the endpoints to interpret CE markings as congestion signals. This manipulation is great for testing as it allows to create flows that ignore CE marks to test how an AQM deals with unresponsive traffic.

No, the old 8bot TOS field got split into the upper 6bit DSCP field and the lower 2bit ECN field. All software should be changed to treat the content of the old TOS byte as two different enrities, but alas not all software has been changed. Especially wireshark returns the full 8bit value as well as information about ECN and DSCPs id recognized as such IIRC.

Diffserv CS1: 001000 + 00(ECN)
TOS CS1:      001000 00

Note how the TOS value is two bits longer? (This is why the TOS-decimal values of codepoints like CS1 are 4 times the DSCP-decimal value, as you need to left shift the DSCP by two bits, which is equivalent with multiplication by 4).

Yes, pretty much.

hex 0x2 -> dec 2
hex 0x20 -> dec 32

side note: oct 31 equals dec 25, which it is argued is why some CS folk confuse halloween and christmas, SCNR

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.