SQM Logic approach for Internet Bandwith management

I have been playing with Luci APP SQM for some time. The first rule I setup was to protect my VOIP and the other one to bound the global bandwidth I have with my ISP (something at 95% of the bandwidth).

All went well until yesterday. I am in the process to segregate all entertainment devices in one VLAN control its bandwidth. So lately I successfully created that VLAN and started segregation one device at a time. I put the XBOX yesterday and activated a rule. Moments later everything went awry. Internet went down, and to get it back I deactivated some of the scripts and had to edit the sqm file in etc/config where apparently something was in it that should not (a rule on an interface I don't have).

Anyhow, I took a step back, deactivated some rules and now I am questioning whether I have to right approach or not.

Assuming I want to put at least 3-4 VLANs with their own access to the internet with some control over the bandwidth
Assuming the global bandwidth rule applied to the WAN/WAN6 interface did great for the latency.

Should the QoS rules to the other VLAN be applied to the interfaces tied to those VLANs or should I have as many WAN interface bridging each VLAN to the internet and apply the rules to those interface?

So the general idea behind SQM is to do the minimum required to get decent (not perfect) performance under load fo the typical home network situation. It turns out that per internal IP-fairness, as in the sing and dance section of https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details solves most of the issues for most users. But it certainly is not the fine grained control typically packaged under the QoS label. Now you can, as you figured out do anything you want on top of sqm, for example using luci-app-ntf-qos/nft-qos for detailed control in the internal network.
Regarding your question, to successfully battle bufferbloat on the wan link, SQM needs to be in full control of the wan link, so bridging multiple networks with independent SQM instances on WAN will only work if the sum of the shaper rates of all instances stays below the true wan rate, a rather wasteful setup. I would recommend to set up a main SQM instance on WAN to deal with your access link's bufferbloat, and then either use additional SQM instances per VLAN/subnet or nft-qos to gain finer control.
But try the per-internal IP-fairness first, it might not be perfect, but it might be good enough to buy you time to design and test your own optimized QoS scheme...

Sorry to hijack the thread @Nainsurvolte. @moeller0, QQ in regard to nft-qos. Will it work marking packets/connections on ingress? My goal will be to keep my iptables interface removing @ldir patches done to support connmark --set-dscpmark because I found it will be a better and more streamlined approach, even when these patches are working phenomenally in my case, tho'.

@amteza see my thread Nftables custom QoS, round 2

1 Like

It turns out that per internal IP-fairness, as in the sing and dance section of https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details solves most of the issues for most users.

Loved that section name, made me laugh alone with my screen,,, Ok I will look at it. I am kind of new at tuning network performance and some OpenWRT wiki are above my knowledge to even understand if it applies to my situation,

Regarding your question, to successfully battle bufferbloat on the wan link, SQM needs to be in full control of the wan link, so bridging multiple networks with independent SQM instances on WAN will only work if the sum of the shaper rates of all instances stays below the true wan rate, a rather wasteful setup.

I get your point and following my last setup to isolate my work environment, I did realize the goal is maybe to guarantee the bandwidth of the few and leave the rest to the mass.

The point of bridging is that for now I belive I may have wrongly create the SQM rules. If I have 4 VLAN, 10, 20, 30, 40 respectivaly for VoIP, IoT, the mass and for work all defined on eth0, then I have created SQM for eth0.10, eth0.20 and eth0.40. I am thus wondering if the SQM should be on some interfaces I would create on eth1 and split the original WAN which is on eth1.2? I guess I am kind of wondering if my actual rules are simply providing QoS on the internal network rather than the internet.

As for the Main SQM istance, its already up and running and it did fix some weird latency spike I got in the first place. Question is on that one, should it be on eth1.2 or eth1 as a whole.?

Thanks,

I don’t know much about the nft approach to this so cannot answer with authority. I have two observations/questions:

  1. Since nft allows you to run packet mangling rules on ingress the current sqm-scripts method of creating an IFB interface (on which a cake instance is applied) has been dropped. This leads to the observation that packet classification rules will need to be run on every ingress and egress packet.

  2. On the ingress path, are the packets pre or post NAT? If pre-NAT then any classification rules that are based on internal IP addresses aren’t going to work.

The evolution (in the sense that evolution is a series of successful mistakes) of the existing IFB/tc act_ctinfo/iptables/set-dscpmark combines a series of limitations & workarounds into something that’s bigger than the sum of its parts. I’m sure that nftables can be used but I do wonder if in the rush to do so other babies have been thrown out with the bathwater of iptables.

Keep in mind a) I’m not an expert and b) I came into this based on the existing framework that sqm-scripts provided ie. IFBs. I understand some of those ‘old skool’ limitations, I do not understand what ‘freedoms’ are offered by the ’new hotness’ of nftables. So what is done under the ‘old skool’ and why:

Ingress:

Packet comes in on wan interface. We’d like to apply shaping to it for flow fairness purposes but cannot (limitation 1), we can only shape on an egress path (we can police on an ingress path but that’s functionally limited). Workaround 1) redirect ingress packets from wan to an IFB interface (ifb4wan). Apply CAKE shaper to egress of the ifb4wan interface, thus we now have a CAKE shaped ingress path. Because CAKE is clever it can look into the conntrack/nat table and apply host fairness across internal hosts. This all happens before iptables gets a handle on it, thus mangle rules to play with DSCPs aren’t run yet. So this is limitation 2) How to apply DSCPs to packets such that CAKE sees them for classification purposes and preferably post NAT so that internal addresses are used for that decision. Workaround 2) We can’t directly. But the conntrack table is something we can access at that point so we could store a desired DSCP into that table (and set a bit flag in it to say that we’ve done so) and restore that DSCP to the packet before CAKE gets to see it. So re-write the IFB redirect to 1) use act_ctinfo to restore any stored DSCP and 2) redirect to the IFB interface, so now CAKE sees DSCPs and can do its own lookup for NAT interface fairness purposes. Job done for ingress at least.

wan ingress -> act_ctinfo (restore DSCP) -> ifb4wan -> CAKE (nat lookup) shaper -> us!

Egress:

Shaping on wan egress is easier since we can just apply CAKE to the interface and it naturally goes on the egress side of wan. Cake has NAT/conntrack smarts to lookup internal hosts for internal host fairness, all is sweetness and light.

Since packets are still in iptables domain (unlike ingress), if we want packets to be classified in a certain way all we need do is write some rules in iptables’ mangle table to change the DSCP and egress CAKE will notice and do the right thing. If at the end of that mangling we then had some tool to store that DSCP into the firewall mark then we can let act_ctinfo in on the story and also ingress CAKE will do the right thing too. That tool is iptables connmark —set-dscpmark. And at that point the job is done, however it’s a bit inefficient since every packet has to go through the mangle tables/routine and we update the conntrack firewall mark every time too (even though mostly it won’t change)

I sortof created a ‘DSCP offload engine’. The easiest tweak is to only go through the classification rules if we haven’t stored a DSCP into the firewall mark yet. That also requires an instance of ‘act_ctinfo’ on the egress interface (before egress cake) where it effectively restores what was decided for the connection by the first run through the mangle rules.

For my own purposes I wanted to implement an automatic de-prioritisation, ie. if a best-effort connection transfers sufficient traffic, consider and treat it like a long-term download/upload. That required a second flag store in the firewall mark and some more slightly convoluted iptables rules but the principle is there.

iptables mangle -> unset go through dscp setting rules -> store with -set-dscp -> act_ctinfo -> cake -> egress wan
-> set -> act_ctinfo -> cake -> egress wan

Kevin

4 Likes

IMHO, the nftables ingress keyword does not obsolete SQM's need for creating a queue for ingress traffic, what it does do is allowing easy-peasy creation of filtering rules for ingress traffic, since, as far as I can tell nftables now runs before SQM's qdisc and hence all fancy filters and conntracking can be used with the relatively easy to grasp nftable commands and no arcane tc onvocations need to be done on the first midnight after a blue moon :wink:

I think, but I habe not conformed/tested that, nftables has access to conntrack information...

Nah, don't hide your own accomplishments, these are pretty impressive and nifty and follow SQM's path of solving ~80% of the issue with minimal hassle :wink: (and 20% fetishists can and will always grow their own rules and configurations :wink: )

That, I dispute, like it or not, you are one of sqm's champions.

Really the one big thing is that nftables can run, before IFB sees a packet and hence all/most of nftables power is available, thing DSCP re-marking and similar shenanigans.

3 Likes

No nft-qos is a package with a GUI that allows either IP-address or MAC addresses based traffic shaping policing giving you fine grained control on a per-host basis. I have never tried it myself, but to reign in an unruly computer or a too greedy program or to carve out sufficient reserve bandwidth for your VoIP phone it seems like a reasonable solution.
(I note I never had a need for that, but I only operate a small family network with pretty tolerant users).

I do not understand this sentence, sorry.

1 Like

So one thing you could also try is to switch to layer_cake.qos and make sue your VoIP traffic carries DSCP marks that elevate it into the high priority tier.
@ldir does your beautiful conntrack method also work for UDP flows? @Nainsurvolte do your VoIP flows actually use UDP or TCP?

Yes. If a flow is IP and hence capable of carrying a DSCP then act_ctinfo will restore it.

1 Like

My puzzlement is more about the fact, that for TCP reverse pairs of ports are used and hence identification of both directions of a conceptual flow is straight forward, while for UDP none such strict rules exist, heck there is not even a guarantee that there is reverse direction traffic at all... So for a VoIP UDP call, how does conntrack track both directions? And the same question I have for 1:m relationships like video conferences. I admit that I never looked at this so your answer probably is RTFM :wink:

According the manufacturer, here is the list of port used :UDP 53, UDP 123, UDP 514, UDP 1194,UDP 3386, UDP 3480, UDP 10000-30000, TCP 110, TCP 53 and TCP 443.

Other than that, can't say exactly which one is used for what as some may be to access the device remotely to get messages and log information and others for the communication. I'm purely speaking logic wise.

Reading a little bit more about the cake, I see your point. I guess I did not understand completely the SQM purpose. I removed all rules for now except the one for the bufferbloat. I'll let it go as such for some time and see if I need anything else. I guess I will also try to understand NAT and NFT as well looking at the comments. I went also through the advanced configuration... and I guess I am not desperate enough for now to play with that, not at least enough to jeopardize a teenager mutiny at home.

Since I have been looking into more of the tids and bits of networking configuration, I see that there are more than one way to achieve your goal and that just confuses me.

Thanks for the help

1 Like

I guess I would try to take some packet captures at the router while performing a few voip calls to settle that question.

Fair enough that is basically SQM's main use/design-case.

Well, I run per-internal-IP-fairness at home and its gurantees, a fair share of the available bandwidth for all concurrently active machines at least is easy to convey and understand. Even teenager should accept that they can not expect more than their fair share (but also not less :wink: )

I would assume a mutiny might be avoided in explaining the new QoS/AQM system before enabling it :wink:

First of all, clarify that I am no saying that act_ctinfo plus iptables plus --set-dscpmark are not a good thing. As you know, I use it everyday for my traffic and it is working wonderfully well. I am very grateful of having it, and I think is a really good idea. :slight_smile:

However, being nft the future and not iptables, and having the availability of marking traffic on ingress even before SQM touches it, it will supposedly allow for better and/or simpler marking and control with SQM. :slight_smile: So, that is why I was wondering about it.

Fair enough, I was referring to using nft to mark traffic on ingress and not using iptables set-dscpmark. In my mind, it might simplify DSCP marking in general.

It does, dramatically. If you can spare the testing effort, the new topic I started should get you where you want to go.

1 Like

You make a very valid point! In my cases the applications do send bi-directional across the address/port pair and that fortunately satisfies the 'classify by egress' mechanism enough. I have also just encountered my 2nd in the wild application that uses DSCP (Unity Intercom) on a bi-directional UDP flow. Currently a bidirectional 96kbps UDP VO flow competing successfully with a 19mbit Bulk flow, CAKE doing its usual magic :slight_smile:

3 Likes

pointer please? I may not be up to the challenge myself at the moment, but happy to lurk and maybe learn a thing or three...

sure it's over here: Nftables custom QoS, round 2