SQM settings questions

  1. Layer cake vs piece of cake, what is better?

  2. If I use piece of cake do I need to use the wash command in the upload?

  3. In layers cake, does sqm automatically mark packages or do I need to configure it manually?

  4. Which one is the best? diffserv3, diffserv4 or diffserv8.

  5. As recommended, it is necessary to use the wash command with besteffort in the download, due to incorrect marking of providers. The question is how do I know if my provider incorrectly marks packages and what happens if I use their markings?

  6. The recommendation on the Sqm Details page says the following: An important exception to the above rules is when the bandwidth limit is set by the ISP's traffic shaper, not by the equipment that speaks to the physical line. Let's consider an example. The ISP sells a 15 Mbit/s package and enforces this limit, but lets the ADSL modem connect at whatever speed is appropriate for the line. And the modem “thinks” (as confirmed in its web interface) that 18 Mbps is appropriate. In this case, the ATM Link Layer Adaptation is likely inappropriate, because the ISP's shaper is the only relevant speed limiter, and it does not work at the ATM level. In fact, it is more likely to work at the IP level, which means that none is the appropriate setting. I would like this to be better explained, since it says that it is not necessary to use overhead, however this contradicts a large part of the community and what the Sqm page says.

  7. Somewhere that shows with data a comparison of the different sqm configurations? A dual-dsthost vs triple isolate example.

  8. I see that the recommendation is dual-dsthost and dual-srchost but what is the difference in performance between the two options, for triple isolate.

  9. What does the nat option do exactly?

  10. If I use layers cake and the packages of an application fall into voice for example, which has a 25% limit. Will the application only use this percentage of bandwidth?

  11. Regarding the RTT, which has a default of 100ms, should this be changed to a value close to that of the application I use?

P.S: Feel free to contribute to any question, any knowledge is welcome

Depends on your use-case... the difference is piece_of_cake only uses a single priority tier for all traffic, while layer_cake uses three, four, or eight, but leaves you with the challenge to steer packets into those prioritu tiers... (it uses DSCPs to address the tiers, and ypu need to set the desired DSCPs).

This is not related to piece of/layer cake, but only whether your ISP does weird things with DSCP marked packets coming from your network.

The latter...

It depends... this is a policy question for which there is no general optimal solution, pick the one best suited to your use-cases.

A simple test is to configure diffserv8 and run it for a day with normal use cases. Then look at the output of tc -s qdisc if there is only little/no traffic outside of best effort and background you should be fine. Otherwise you might need to take packet captures on the wan interface and try to figure out what packets your ISP/the internet marks in which way and whether you can accept/tolerate that. Again a policy decision for you to make...

IMHO that section is confused and best ignored, I added an addendum there.
Tl; dr: don't overthink the per packet overhead, if in doubt, (slightly) over estimate it instead...

Again this is a policy decision foremost. Triple-isolate tries to generally do the right thing without needing to know the direction of the shaped link towards the internet. For normal traffic mixes that more or less works as intended. However in these artificial conditions we often use for simple experiments the strict internal fairness configuration using dual-srchost for internet-upload traffic and dual-dsthost for internet-download traffic ends up being more predictable and closer to expectation. But have a look at the cake paper here:

I think this shows some data for the different isolation modes.

Again, see the cake paper linked above...

The nat option allows cake to glean into the conntrack data base the kernel uses to perform network address translation; that way cake can figure out from seeing a packet still outside of the NATed internal network, which internal IP address this is going to be sent at. This is required as especially the ingress/download qdisc typically operates outside of the NAT domain, so does not see the correct internal IPv4 addresses, but if you use dual-dsthost, cake needs to know exactly the internal addresses (all packets arriving at a NAted IPv4 router carry the same destination address, the public IPv4 of the router, and the NAT subsystem rewrites the destination address (and destination port) to the correct internal host). Again this is mentioned in the cake paper.

Let me see if I got it.

So layer cake does not automatically mark packages. The marking is done by the provider in the download and must be done by me in the upload, right?

You said that each option will depend on the use case however the documentation clearly advises you to use the dual-dsthost and dual-srchost option, why then?

Are you editing the SQM Details page? If so, this page needs good maintenance as there is a lot of mixing of content and contradictions. I say this without wanting to offend, but only to better clarify the knowledge to be transmitted.

Yes, layer_cake simply configures cake to offer three different priority tiers by default and uses a fixed set of DSCPs to steer packets into these three tiers. Setting these DSCPs is up to you. For egress/upload one can use the OpenWrt firewall to set DSCPs but for ingress that is trickier. There are two projects that might be helpful here:
qosify: this allows to define rules also for ingress/download traffic, with one big caveat, currently it can not see internal IP addresses and hence you can not do something like:
all UDP packet to internal host and port XXX -> mark EF (which ends up in the highest priority tier)
there is also a project that uses nftables to essentially copy DSCPs marks from egress to the ingressing packets of the same connection, which in most cases does the right thing.
But again theses are just tools you could use, whther/which you use is up to you.

Simply because triple-isolate will result in some deviations from what people expect fr simplistic tests, while the correct useage of the dual-xxxhost options eve works as predicted in those simplistic tests. So recommending the dual-xxxhost set-up results in less support questions :wink:
But feel free to test both and pick the one you prefer, nobody is going to judge :wink:

Well, this is a wiki, so everybody can add stuff. I try to be respectful by not outright deleting stuff there I did not write myself. Just make a proposal how you would like that page to be changed (or even get a wiki account and make those changes yourself). Just noting the page is not perfect, while fair, is not going to change much.

In the part where you answer about layers cake, do you say yes about everything I wrote or just in the part where I say that SQM does not automatically mark packages?

What are your SQM settings?

Yes, I think we agree that I already answered that.

I did not explicitly cover this, sorry. For ingress/download you can either accept whatever comes out of the internet (any party along the full path that is the originally sending computer and any router along the way in any involved network may set/modify the DSCP value). Unless you have agreements with all parties in place it is unlikely that the DSCPs received from the ISP are going to be all the useful (but do not trust me here, take a packet capture and see what reaches your router). This is why qosify, cake-qos-simple and dscpclassify all got created, to help making sure DSCPs on ingress can be used easily.


config queue 'eth1'
	option ingress_ecn 'ECN'
	option egress_ecn 'ECN'
	option itarget 'auto'
	option etarget 'auto'
	option verbosity '5'
	option qdisc 'cake'
	option script 'layer_cake.qos'
	option qdisc_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option qdisc_really_really_advanced '1'
	option eqdisc_opts 'nat dual-srchost memlimit 32mb'
	option linklayer 'ethernet'
	option linklayer_advanced '1'
	option tcMTU '2047'
	option tcTSIZE '128'
	option linklayer_adaptation_mechanism 'default'
	option debug_logging '1'
	option iqdisc_opts 'nat dual-dsthost ingress memlimit 32mb'
	option interface 'pppoe-wan'
	option tcMPU '88'
	option enabled '1'
	option overhead '34'
	option download '105000'
	option upload '45500'

This is on a VDS2/PTM link with a download sync of 116790 and an upload sync of 46719 kbit/s . I note that I use layer_cake/diffserv3 here, but I use neither qosify nor any other option, incoming DSCPs are mostly in BestEffort, which is just as I like it...

Interesting what are these ecn options?

Could you try to answer my new questions. I would like you to edit your first answer if you respond.

In relation to layers cake in the upload, the packages will end up in categories other than besteffort, what does this mean?

These exist for simple.qos/fq_codel, cake will always use ECN independent of these settings.

Well, that is nice of you, but I will take the liberty to respond as I see fit, if at all.

So by default layer_cake.qos uses the diffserv3 mode of cake, with three different priority tins (from low to high):
Bulk; Best Effort; Video.

Bulk is guaranteed 1/16 of the capacity and all left overs
Best Effort, by default will get up to 100% of capacity, but will leave 1/16 for bulk and will leave 1/4 for Video
Video will get 1/4 capacity as high priority access and if the traffic in Video exceeds 1/4 it will be scheduled with Best Effort 1:1, if I recall correctly, again the cake paper might answer your question more in detail.

1 Like

No, in cake priority tiers are not hard but soft limited, packets in Voice that exceed the 25% capacity share get, if I recall correctly, scheduled 1:1 with packets in Best Effort, that is they are not strictly prioritized any more (otherwise Video could fully starve Best Effort, or would need to be hard limited).

That RTT value configures Cobalt's interval, a queue needs to have sojourn times >= RTT/10 for at least RTT ms before that queue will see a drop mark. The exact value is not all that important, the order of magnitude however is. Flows with a true path RTT >> cake's interval RTT will see considerably more RTT bias than usual for cake... the default 100ms work pretty well for normal terrestrial links over the full internet. But if say your moat important traffic terminates in a data center 2ms away reducing cake's RTT setting can be useful. Again a policy question and something you might need to test for your own network and traffic mix (just keep in mind that the exact value is less important than the order of magnitude, e.g. I expect that changing from 100 to 95 will not be noticeable).

I only say this because this may be other people's doubts and would leave the knowledge centralized in an answer.

If packages are not marked, shouldn't they fall into BestEffort? Here they are falling in voice too.

"Not marked" means that the 6bit DSCP bitfield in the IP header is set to all zeros, and yes cake will put those packets into the BestEffort queue/tin.
However, your ISP might send some packets with DSCPs that cake will put into the voice tin. And if you instruct the download cake instance to "wash" it will first evaluate the DSCP to select the appropriate priority tin and will re-mark that packet's DSCP field to zero, so once the packet gets released from cake it will not show is initial DSCP anymore...

Here is a tcpdump command line I run on my router (pppoe-wan is my wan interface):

tcpdump -i pppoe-wan -v -n '(ip and (ip[1] & 0xfc) >> 2 != 0)' or '(ip6 and (ip6[0:2] & 0xfc0) >> 4  != 0)'

this will print a line for any packet received via IP4 or IP6 that has a DSCP different from 0.
Tcpdump will report the DSCP either as tos for IPv4 or as class for IPv6:

19:23:21.805864 IP (tos 0x20, ttl 64, id 25254, offset 0, flags [none], proto ICMP (1), length 28) > ICMP echo reply, id 32899, seq 54334, length 8
19:23:21.832821 IP6 (class 0xc0, hlim 63, next-header ICMPv6 (58) payload length: 104) 2a02:3001::1a4 > 2a01:c23:90bf:db00::1: [icmp6 sum ok] ICMP6, time exceeded in-transit for 2804:14c:41f9:5::1

You can look up the DSCP values from tos/class by looking at the TOS (hex) column here:

1 Like

Thanks for your help, my SQM settings were almost the same as yours, I just changed the overhead to 44 and the mpu to 84, and I set the download speed to 70000 and upload speed to 35000.