Ultimate SQM settings: Layer_cake + DSCP marks

Well, I certainly would test whether I actually need special rules for this at all. BUT I am also a lazy person and hence will accept a bit more slack im my QoS setup than gamers in general, if it saves me the tedious work of designing heuristic rules and (even worse) maintaining those rules (aka check continuously whether the employed heuristics still work as intended).
But I am also conscious that this is a matter of subjective policy and I will not go and tell you to not make detailed rules in your network (as these can perform better than the naive approach I am taking assuming one is willing to pay the maintenance cost). I would just make sure that each rule actually improves things (as each rule will have some cost in complexity and potentially in computation).

I believe this to be an example of beautifully custom tailored QoS rules that probably work/worked very well for the OP, but I would be hesitant to copy and use these verbatim without confirming for each individual rule (or set of related rules) whether they are a) still required (in light of the sparse flow boosting in fq_codsel and cake) an b) whether they still work as designed.

Traditionally CS7 was reserved for critical networking infrastructure, for that to work the devices actually interpreting CS7 need to be super selective from whom to accept CS7 markings in the first place (otherwise any client running "sudo ping -f -Q CS7 your.critical.networking.device" (pseudo-code only) can basically DOS the "management plane"). As far as I can tell cake's diffserv3 and diffserv4 will both lump CS6 and CS7 into the same priority tier. Both the precedence and the diffserv8:

static int cake_config_diffserv8(struct Qdisc *sch)
{
/*      Pruned list of traffic classes for typical applications:
 *      
 *              Network Control          (CS6, CS7)
 *              Minimum Latency          (EF, VA, CS5, CS4)
 *              Interactive Shell        (CS2, TOS1)
 *              Low Latency Transactions (AF2x, TOS4)
 *              Video Streaming          (AF4x, AF3x, CS3)
 *              Bog Standard             (CS0 etc.)
 *              High Throughput          (AF1x, TOS2)
 *              Background Traffic       (CS1)
 *      
 *              Total 8 traffic classes.
 */

static const u8 diffserv8[] = {
        2, 5, 1, 2, 4, 2, 2, 2,                    
        0, 2, 1, 2, 1, 2, 1, 2,
        5, 2, 4, 2, 4, 2, 4, 2,                    
        3, 2, 3, 2, 3, 2, 3, 2,                    
        6, 2, 3, 2, 3, 2, 3, 2,                    
        6, 2, 2, 2, 6, 2, 6, 2,
        7, 2, 2, 2, 2, 2, 2, 2,
        7, 2, 2, 2, 2, 2, 2, 2,
};

and the precendene:

static const u8 precedence[] = {
        0, 0, 0, 0, 0, 0, 0, 0,
        1, 1, 1, 1, 1, 1, 1, 1,      
        2, 2, 2, 2, 2, 2, 2, 2,
        3, 3, 3, 3, 3, 3, 3, 3,
        4, 4, 4, 4, 4, 4, 4, 4,      
        5, 5, 5, 5, 5, 5, 5, 5,
        6, 6, 6, 6, 6, 6, 6, 6,
        7, 7, 7, 7, 7, 7, 7, 7,
};

scheme will honor CS7 (albeit differently).

That is odd, TOS (type of service) was an older recommendation for the usage of a specific set of 8 bits in the IP header. With the introduction of ECN, two of these 8 bits have been rededicated, hence ideally plain 8-bit TOS is obsolete and should not be used anymore.

Possible, but not super likely (as far as I know, it is considered good network hygiene to remap incoming TOS/dscp values if one uses these in one's own network.)

Thanks for your valued reply.
so what is the use of VA (VOICE-ADMIT) class, i'm also sometimes lazy to maintain those rules, but also
i don't copy/paste those rules blindly, i just read about then edit it as needed.
but my real concern is not QOS for gamer's, if you remember @Emtee was focus on game optimize only , not the whole network.
but i'm aiming to build a qos that control everything, i was planing to use ndpi and L7 protocol.
but i don't know why openwrt doesn't gives any interest about NDPI or L7 protocol.
lib-ndpi is already available in the packages list, but why there's no modules for iptables?
L7 is totaly removed, i know that those two features have some limitations or instability, but at least it
should be provided as experimental"use it as your own risk" , if you look at router os or dd-wrt and tomato
they already have L7 working and still many users use it.

I think L7 is older, and unmaintained as a kernel system. It got moved to userspace but that may be a problem for speed

Nftables is the future of packet handling. If you plan to go this deep into custom packet handling I suggest to move to nftables. It has a scheme for sending packets to userspace.

But I don't really suggest this "deep packet inspection" as a great way to go. Rather than inspecting individual packets I think it makes better sense to prioritize whole flows. You can use conntrack connection marks, and a squid proxy.

Suppose for example you have torrents you want to down prioritize. Set up the torrent client to use a narrow range of ports, when you see this port, connmark the packet, then restore the mark on ingress and dscp tag it, and put it in the slow queue.

Or if you can't control the ports, try running torrents off a different IP range, or different VLAN... It makes more sense to segment your network to different functions than to deep packet inspect each packet.

1 Like

Because the l7 modules weren't upated for a long a time.
And more and more services are moving to encrypted connections which makes l7 detection useless.

Here is a working ndpi filter module for openwrt:

2 Likes

The encryption point is a very good one. Squid proxy can help there, it at least knows the domain name you are connecting to and can prioritize https flows based on the name

Squid is good too.
I like the dnsmasq + ipset approach x)
But we already had that discussion in a other thread, don't we? :wink:

so what is the use of VA (VOICE-ADMIT) class

Well some VoIP systems use this dscp for (some of their) packets. Here is Wikipedia's take:
https://en.m.wikipedia.org/wiki/Differentiated_services:

Voice AdmitEdit

The IETF defines Voice Admit behavior in RFC 5865. The Voice Admit PHB has identical characteristics to the Expedited Forwarding PHB. However Voice Admit traffic is also admitted by the network using a Call Admission Control (CAC) procedure. The recommended DSCP for voice admit is 101100B (44 or 2CH).

Yes the ipset approach works too, though it will prioritize all flows to/from a given ip address, so for example maybe youtube streaming and google drive uploads will be prioritized the same because they both wind up on the same local google node. But this may not matter much.

@hisham2630 The biggest reason L7 was used was to prioritize the peer-to-peer traffic or game traffic on random ports. It requires a fair amount of cpu speed, and of course doesn't work at all on encrypted packets (but still consumes all the resources).

Also I think if you are excited about more customized QoS, you should move to HFSC because it allows you to explicitly control both latency and average bandwidth consumption. In my mind there are basically 4 kinds of traffic in terms of latency:

  1. real-time traffic. This includes NTP, VOIP, and gaming
  2. priority traffic. This includes video streams that you want to run smoothly but that have buffers to allow for a reasonable amount of jitter, as well as things like DNS lookups because your flow waits on DNS.
  3. Normal traffic. This includes web surfing etc.
  4. Latency Insensitive. This is anything that is expected to run longer than say 100 seconds. Who cares if it has a 100ms or even 1000ms delay? Torrents, long running http downloads, uploading videos to youtube, whatever.

So, HFSC will let you set up queues where it will trade off latency and bandwidth. Specifically it allows you to set two speeds, a "burst" speed with a given short duration to drain the queues of latency sensitive traffic, and a "stead state" speed that applies after the burst is over. The burst clock starts whenever there is the initiation of contention between queues. So if your torrent is going along fine, and then suddenly there's some game traffic in the game queue, the burst clock starts, and it switches to sending the higher priority traffic at a specified burst speed, which if calculated correctly will drain the latency sensitive queue before the next packet comes in. The torrent may be entirely starved for say 10ms to get the game queue drained... but that's exactly what you want.

I regularly get about 1.5ms of jitter reported on my VOIP calls using this scheme with a real-time queue specially for VOIP (but I can identify my voip PBX by a known ip/ip6 address so it's easy to classify this traffic)

1 Like

I am not a fan of DPI (unless we are talking dots per inch, in that case the more the merrier).
And as others said already l7/DPI is easily evaded by employing encryption, and then you need to use man in the middle attacks to still be able to intercept the traffic, sort of deeper packet inspection....
Also one of the main use cases of these techniques have been torrents, but those have both adapted by hiding even deeper and by trying to be good citizens by using uTP (tangent: the choice of accepting up to 100 of added latency before scaling back was an implementation error IMHO, but the idea of getting out of the way of other traffic is sane and nice).

Another reason to use HFSC. IMHO the cake default of trying to keep all flows reasonably low latency is probably a good default, but for those who do things like regularly updating large archives of software packages (think a Debian mirror) or torrenting stuff all night long, it makes perfect sense to intentionally delay those streams when there is contention. Who cares if a 12 hour long background flow is delayed 1000ms even? And if by delaying it you improve latency for the flows you do care about, then it's a good idea.

EDIT: also @moeller0 it might make sense to make modifications to the SQM scripts to include a simple layer-hfsc script.

TCP cares :wink: TCP congestion control algorithms will get into bad territory if the delay gets too large, I always assumed that means weird oszillations but never bothered to look into the details.

As long as they don't start flooding the link, oscillation is fine in this case. For example if TCP says "gee I'm not getting any ACKs, I'll shut off my flow until I get some" that's exactly what you want. on the other hand if they say "gee I'm not getting any ACKs I'll expand my window and flood the link because I must have a fat pipe" that'd be bad :wink:

Well, mark that flow BK and sqm will do the right thing no need for hfsc...

We did have a hfsc based script for a long time but that was removed due to a long standing bug in the networking subsystem that caused kernel messages for hfsc. IIRC loading the module was sufficient, you did not even have to actually use it.
I believe @tohojo would accept a well made new hfsc script if it comes with a reasonably high confidence of being maintained :wink:
Since I do not use hfsc myself and am already out of time for most things I would like to try in sqm-scripts, you better not keep your breath until I get around to it....

TOS bits are the same bits as the DSCP bits, it's just a different name for a particular set of bits in the packet header. So if you set a DSCP and then you set a TOS, you are in effect changing the DSCP. So basically you probably just changed the DSCP value to something higher priority when you set the TOS.

Specifically the TOS byte is the second byte of the ipv4 header, and the DSCP bits are the first 6 bits of that byte.

Me either. Also I'm not really much for understanding how UCI works etc. I stick to scripts of regular old bash shell or the like, particularly since my routers actually run Debian not OpenWrt, it's only APs where I use OpenWrt. Even there I'm looking to push the envelope, I'm planning to strip the firewall and use nftables on my OpenWrt stuff soon after seeing how nice it is for my Debian boxes. Then I can easily have DSCP tagging on ingress in the APs.

I had a hunch that TCP will start dumping too much data into the link, clogging the upstream sise of the true bottleneck link, but all I am finding right now is papers that show the long RTT flow having less throughput, and I fully agree for a background transfer that is not only acceptable, but actually desired :wink:

Oh at the core of sqm-scripts there are relative simple scripts and a few helper/library scripts to supply commonly used functions. The GUI really is a separate project :wink:

I guess I should note that the sqm-scripts (but not luci-app-sqm) also wok on normal distributions like Arch and Debian...

I think for typical congestion control algorithms in TCP, the long round trip time causes exponential back off and slow-start behavior, so it then rapidly slows down its sending, then it ramps it up slowly, and then may rapidly slow again, ramp... etc so on average it winds up oscillating, but slower than it could have if it were more zeroed in on the proper average... but as you say that's actually ok, even desirable.

EDIT: it's even better if you are setting the max speed on the BK queue to be slightly even-less than you really have. I know this is good practice on the general link, but it seems like good practice on the background queue to go even further down from the max, so that there's a little headroom for higher priority flows to initiate themselves.

thanks for the clarifications, nft is still new also i should change the whole network to use nft, or is it still
possible to use nft just for packet handling?, also by send packets to userspace which tool is used to
analysis those packets?
i can kill downloads or torrents by using connbytes, it's still useful!

I see that ndpi is much better than l7, also it's support decryption. Vel21ripn is still maintained the ndpi-netfilter
module, now it's support adding protocols to file.
this make doesn't work for mips and give errors, i made a make file that compile without errors for openwrt
x64/x86, but it won't compile for mips too?!

  • it would be helpful to include ndpi-netfilter in openwrt package repo, it will save people's and newbies
    time, like lib-ndpi is already compiled?!

squid is good and can do many helpful things from caching to blocking, etc.
but have some problems with SSL and HTTPS ??!

Me too.
i use ipset+dnsmasq is really great feature and useful, it's easy to tag youtube and google traffic's.

Thanks, but i think it's still not widely used like EF or CS6 !

We can use NDPI, i tried to block whatsapp calls and viber and it was working properly!

Some said that it have a slow response and they prefer HTB!

Thanks for this explanation, i read something about use a smaller txqueue_len so it will drain faster!
it's easy to prioritize a VOIP PBX using ipv4/ipv6 but it's not an easy way for P2P or whatsapp calls. they
use random ports + sender/receiver ip.
I ran a test called RTR and found that call jitter is about 2ms, voip is much better with SQM+CAKE/DSCP
clear voice and there's a rare interruption's (i think it's duo to a bad isp's setup).

so how isp's is managed to block whatsapp calls,viber in some countries ?!

i think it would be nice if it's possible to change cake default latency's for flows, i think a delay between
100ms and 400ms is sane!, also this will keep TCP stable(this need some test's and observations).

True, but i'm still testing, TOS is an old style, DSCP with ECN is the new style, i used this table to set both TOS and DSCP https://wiki.innovaphone.com/index.php?title=Howto:Calculate_Values_for_Type_of_Service_(ToS)_from_DiffServ_or_DSCP_Values
When was on one of isp's that i tried they was using TOS and not DSCP!, they was using 0x60 or 0x30
for youtube but i can't remember exactly.

One problem for ECN is intermediate routers should ECN, i think my isp doesn't support ECN. because
When i used RTR test i saw that ECN is off, while i have it on on both sqm and sysctl!
i would be happy if i can select the BK queue max speed!.
Now the problem that i have is when i open a webpage it will stay white then will fully load after 5 sec!
after this 5 sec i saw that download will slowdown and webpage will load completely.

*Any idea's and suggestion to improve this setup are welcome.

All have a good night!