CAKE w/ DSCPs - cake-qos-simple

You might want to ask any question on the github project's page, lynxthecat has unsubscribed from the forum (I still hope he will come back) ...

Thanks moller, I'm going to try to make my internal network travel each packet through its place and try to understand how everything works perfectly.

Here is a example for Zoom for Windows, Right click on WIndows Powershell and run as Administor.

Zoom

New-NetQosPolicy -Name "Zoom" -AppPathNameMatchCondition "Zoom.exe" -NetworkProfile All -DSCPAction 46 -IPProtocolMatchCondition Both -PolicyStore Localhost

Linux

iptables -t mangle -A POSTROUTING -p tcp -dport 80 -j DSCP -set-dscp-class cs1
iptables -t mangle -A POSTROUTING -p udp -dport 80 -j DSCP -set-dscp-class cs1

Update: the config error was identified in a few places, fix is ready, soak testing in 1 - 2 weeks and then deployment to complete in the early part of July. I'll try to remember to post back here when we're done. Thanks again to this list for the heads up!

8 Likes

Once you’re done, you may want to see about getting the cake man page updated.

2 Likes

Inspired by @shm0 in this thread here, I have incorporated the facility in cake-qos-simple to overwrite the ECN bits on upload and/or download packets before the cake instances see them, e.g. to prevent the cake instances from marking any packets.

It seems there are situations in which it is desirable to prevent cake from marking rather than dropping packets.

Firstly, see discussion from around here:

Secondly, whenever an ISP bleaches ECN bits (which is common for mobile operators), the bleaching occurs prior to cake on download, but after cake on upload - thus cake is blind to the bleaching in the upload direction, and this means that by default cake will mark packets on upload in response to saturation, which is futile because the ISP ultimately bleaches those markings anyway. So in this situation it strikes me as better to proactively scrub the ECN bits on upload before cake sees the packets to prevent cake form ineffectively marking the ECN bits.

@dave14305, @dlakelan and @moeller0 does this approach seem reasonable to you:

https://github.com/lynxthecat/cake-qos-simple/compare/master...overwrite-ecn-bits

Here is a brief summary of my implementation.

For download, to overwrite the ECN bits the following tc arguments are added:

pipe pedit ex munge ip dsfield set ${overwrite_ecn_val_dl} retain 0x3

See here:

https://man7.org/linux/man-pages/man8/tc-pedit.8.html#EXAMPLES

For upload, the situation is more complicated.

Taking inspiration from:

https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.qdisc.filters.html

I have implemented the following:

# create egress handle using prio qdisc
tc qdisc add dev "${ul_if}" handle 1: root prio
# filter the flow using pedit and send the filtered flow to band 1:1
tc filter add dev "${ul_if}" parent 1: protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_ul}" retain 0x3 flowid 1:1
# then apply cake on band 1:1
tc qdisc add dev "${ul_if}" parent 1:1 cake bandwidth "${cake_ul_rate_Mbps}Mbit" ${cake_ul_options}

I tested this on upload and it works - ECN bits are overwritten before cake sees them and cake does not mark any packets on upload. I cannot test download because my ISP bleaches the ECN bits.

Any thoughts?

1 Like

Yes and no. As a desperate measure it seems acceptable, but it really exposes bugs in the involved TCP stacks. These should notice the bleaching and fall-back to not mark their packets at all and fall back to pure drop based congestion signaling. Yet, with broken TCP stacks out there and cake having no toggle for shutting ECN-signaling off, this seems like a reasonable addition...

You should test carefully whether this leaves the DSCP bits alone, it should, but conformation via wireshark seems advisable.

1 Like

Why the extra prio qdisc? You should be able to attach the filter directly to cake, like:

# create egress handle using prio qdisc
tc qdisc add dev "${ul_if}" handle 1: cake bandwidth "${cake_ul_rate_Mbps}Mbit" ${cake_ul_options}
tc filter add dev "${ul_if}" parent 1: protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_ul}" retain 0x3
1 Like

Hi why would any one want to use this over SQM or Qosify? What does it do that the other packages don't or cant. From some of the posts I see that some people have used this on a 21.xx build of openwrt, but firewall 4 has changed things.

1 Like

This uses tc's ability to apply a stored egress DSCP from the conntrack database to ingress packets from the same connection, BEFORE an ingress qdisc sees the packets. So you mark packets on egress as you desire (either on the end host or on the router) and automatically the same marking and associated priority tin selection in cake is applied to the ingress (DSCPs from the internet are typically not reliable or robust so overwriting them with something you selected is pretty nice).
Currently nftables does not really support that with a one-liner command (as OpenWrt's iptables did) so sqm-scripts does not offer that capability.
The advantage over qosify is you can use nftables full power especially you can create marking rules based on internal and internal IP address and port combinations, something qosify does not currently offer.
Whether you prefer one or the other is a matter of what you want to achieve, but these clearly are not fully overlapping sets of features so the existence of each seems somewhat rational.

4 Likes

So installing this on a router where you have a mix of devices would be pointless? EG apple iphones can not set dscp. As far as I know. They use somthing called fastlane from what I read witch removes dscp marcking from packets. Then you have all the other things that do things there way. Sounds like a rite pane in the ass to me.

Not really, as I said the egress marking can happen at the end device, or if you do not trust these devices you can have e.g. firewall rules that assign the egress DSCPs. Even if an end device will wipe the DSCPs cake will still honor these to assign the respective priority tier.
However, as I always repeat prioritization only works well if used sparingly, up prioritize as little as possible, as fir every packet that will be treated better than average other packets will be treated worse, if no such sacrificial packets exists prioritization stops helping.

Yes in such a case you create a forewall rule based on e.g. the IP address of the iphone and the port number of the application and set the DSCP at the firewall...

Yes, but that is the beauty of this approach, you can trust the end-nodes if you want, but you can always override their DSCP choices in the firewall.

1 Like

I can't get a formulation under your proposed lines to work.

With:

        printf "\nSetting up CAKE on interface: '${ul_if}' with bandwidth: '${cake_ul_rate_Mbps}Mbit/s' and options: '${cake_ul_options}'        
        tc qdisc add dev "${ul_if}" handle 1: cake bandwidth "${cake_ul_rate_Mbps}Mbit" ${cake_ul_options}

        if [[ -n "${overwrite_ecn_val_ul}" ]]
        then
                printf "\nSetting up filter to overwrite ecn bits to decimal value: '${overwrite_ecn_val_ul}' on upload.\n"
                tc filter add dev "${ul_if}" parent 1: protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_ul}        
        fi

I see:

Error: Specified qdisc not found.

This formulation runs:

        printf "\nSetting up CAKE on interface: '${ul_if}' with bandwidth: '${cake_ul_rate_Mbps}Mbit/s' and options: '${cake_ul_options}'        
        tc qdisc add dev "${ul_if}" root cake bandwidth "${cake_ul_rate_Mbps}Mbit" ${cake_ul_options}

        if [[ -n "${overwrite_ecn_val_ul}" ]]
        then
                printf "\nSetting up filter to overwrite ecn bits to decimal value: '${overwrite_ecn_val_ul}' on upload.\n"
                tc filter add dev "${ul_if}" root protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_ul}        
        fi

but the filter is not applied before cake sees the packets and cake ends up marking the packets.

What about this with the word root between dev and handle?

1 Like

Yes - that works - @dave14305 to the rescue yet again!

Would it actually make sense to do it both ways like this then? That is, apply the filter to the cake qdisc for download too (rather than before passing to the IFB)? If that would work that would simplify the code yet further.

So I mean like this:

	printf "\nSetting up CAKE on interface: '${ul_if}' with bandwidth: '${cake_ul_rate_Mbps}Mbit/s' and options: '${cake_ul_options}'.\n"
	tc qdisc add dev "${ul_if}" root handle 1: cake bandwidth "${cake_ul_rate_Mbps}Mbit" ${cake_ul_options}

	if [[ -n "${overwrite_ecn_val_ul}" ]]
	then
		printf "\nSetting up filter to overwrite ecn bits to decimal value: '${overwrite_ecn_val_ul}' on upload.\n"
		tc filter add dev "${ul_if}" parent 1: protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_ul}" retain 0x3
	fi

	printf "\nSetting up CAKE on interface: '${dl_if}' with bandwidth: '${cake_dl_rate_Mbps}Mbit/s' and options: '${cake_dl_options}'.\n"
	tc qdisc add dev "${dl_if}" root handle 1: cake bandwidth "${cake_dl_rate_Mbps}Mbit" ${cake_dl_options}

	if [[ -n "${overwrite_ecn_val_dl}" ]]
	then
		printf "\nSetting up filter to overwrite ecn bits to decimal value: '${overwrite_ecn_val_dl}' on download.\n"
		tc filter add dev "${dl_if}" parent 1: protocol ip matchall action pedit ex munge ip dsfield set "${overwrite_ecn_val_dl}" retain 0x3
	fi

Also consider IPv6 by swapping traffic_class for dsfield. Once you have everything else working. :grinning:

That should work - the filter runs before the qdisc (cake calls it as part of its classification logic), so you don't really gain anything from running it as part of the IFB redirect. So just do whatever is simplest from a configuration PoV :slight_smile:

1 Like

Thanks so much @tohojo and @dave14305 - this simplifies things significantly.

@tohojo might it not still be an idea to further include switches for the cake qdisc that enables or disables marking?

I'm also curious about your thoughts about ECN Vs dropping in general. Most mobile operators (including Vodafone UK it seems) bleach the ECN bits. How beneficial would it be if they did not?

I see that you and @moeller0 discussed here:

enabling or disabling marking based on rate.

1 Like

I seem to recall that we discussed this at some point and decided that this was probably too niche a use case to be worth complicating the cake code for; especially since it can be solved fairly simply with a filter, as you have so aptly demonstrated :slight_smile:

Good question! ECN has the nice property that you can get the congestion signal without incurring a retransmission (which is inefficient since you have to transmit the same data twice). However, how much of a benefit this ends up being in practice is one of those "it's complicated" questions. It depends a lot on the characteristics of the network, where the drop/mark occurs on the path, etc. Not to mention the whole discussion around using ECN for more granular congestion signalling.

I generally turn on ECN on my own systems, but TBH I couldn't really tell you how much difference it makes...

This particular discussion is because of another nice property of dropping instead of marking: a dropped packet disappears from your queue immediately, whereas a marked packet still has to be transmitted (or, as @dtaht likes to put it: "packets have mass"). This means that a drop gets you an immediate queue reduction, leaving to lower instantaneous queueing delay. At higher rates this is not so important, because the serialisation delay of a single packet is negligible. But at really low rates it does: at 1Mbps, a 1500 byte packet takes ~13 ms to transmit, so if you're at the point where you need to throttle a flow, you will probably get more benefit from that queue reduction than any efficiency gains from the marking. At least if your queue is before the bottleneck link (so the packet hasn't already taken up the time on the wire). Hence the rate-based toggle of drop/mark.

That's extremely helpful thank you. On the subject of timings, cake-autorate updates its allowed bandwidth with every ping response. Allowed bandwidth is determined by increasing the bandwidth under high load until bufferbloat is detected (enough ping responses within a window are delayed, e.g. 3 out of the last 6) and then evaluating the achieved rate during bufferbloat. Under low load bandwidth is returned to a kind of safe harbour set in accordance with a base rate.

See here:

So that rate of update could be, for example, 20Hz (pings from multiple reflectors are interleaved to give higher granularity).

Is there anything I should be aware of in terms of using tc qdisc change calls to update the cake bandwidth? What happens during such change calls. Can that bandwidth be updated immediately with no ill effects? How about a huge drop from 50Mbit/s to 10Mbit/s?

Any harm in updating very rapidly, at say, 20Hz? I've often wondered if there might be a reason to limit the rate of the bandwidth updates or rate of change, for some reason or another.