hello everybody i would like configure a qos like this with my router hap ac2 (not developed again but robiemarko has a build and blaze too) can you help me to configure please ? thanks
it's seems very interesting, it's possible ??
This software is explicitly about handling thousands of different users for an ISP. Are you serving thousands of users?
no just a private user 
I actually think it's not even that great a solution. With a large number of users (like they are talking about thousands) it makes more sense to use statistics to reduce the processing overhead. No need to give a single queue to each user, instead one queue for say 30 users makes more sense. With randomized hashing, you will always have maybe 25 to 35 users in each queue, so speeds may vary +- 10% or so but it's not a big deal. You put ONE token bucket that sets the total rate, and then use QFQ with 1000/30 = 33 classes and randomized hashing. Then fq_codel below each of these 33 classes... it's a much more scalable solution.
I'd actually probably recommend the use of a prime number of queues, so 31 in this case, for 10000 users 10000/30 = 333 so a close prime number is 337 etc (prime number table: https://www.mathsisfun.com/prime_numbers.html )
2 Likes
Pretty cool idea.
I note I rather liked QFQ (and it's author), but it was too buggy at the time (2012) to use effectively. And is it actually in openwrt?
No it doesn't. ISPs typically charge by access rates and hence need to enforce speeds per subscriber, so need to queue different customer's traffic already. Typically they use a single queue and no AQM, what libreQoS adds is to run a flow queuing AQM with multiple sub-queues on top of the per-subscriber queue. Now, I remember your proposal about equitably sharing a link for all subscribers (with a potentially bounded priority weighting) which I like a lot, but that is not the problem that libreQoS tries to tackle, as far as I understand.
The thing is, libreQoS demonstrates, that actually doing stochastic FQ per subscriber still scales reasonably well, so no need to cut corners ;). It turns out that neither fq_codel nor cake's fq component are terribly CPU intensive, it really is the traffic shaper that kills performance on typical plastic box router platforms.
1 Like
But they don't need to do this, unless people are going to complain about getting 10-15% more than they are supposed to. All they need to do is ensure that the total number of subscribers on a link doesn't exceed the level where they can't reliably provide at least the contracted rate under real world observed demand. But they have to do that ALREADY. ISPs don't size their upstream for the worst case, they use the statistically say 95%tile case which because not everyone is using the internet at max contracted rate at all times is WAY less than worst case. Like maybe a 1Gbps uplink is used for 100 customers at 100Mbps. Worst case demand is 10Gbps but only if everyone in the neighborhood all hits a speedtest site at once. ISPs take that risk because it's essentially nonexistent. Instead they monitor upstream actual capacity continuously and look at what is the 95% or maybe 99% utilization. When this gets to say 900Mbps for a week in a row time to upgrade the uplink.
It's the same concept for the shaper. You put the token bucket at the top at say 1Gbps, and then you have 1000 subscribers so you QFQ 37 buckets. Each bucket will have on avg around 27 people assigned to it. Below each bucket is an fq_codel. Although there will be some variation in usage, it'll vary between say 20 and 35 or something similar (you can use flow monitoring to get the statistics from observed data). Realistically the "downside" to this is some evening at 1 AM someone will hit a speedtest, and only 3 other people in the neighborhood will be using the internet and they will see speeds of say 400Mbps on their 100Mbps line... I doubt there will be major complaints. The key is to size the upstream so that people hitting speedtests mid day get ~105-115Mbps and then everyone's happy.
What libreqos seems to be doing instead is giving a HTB to each subscriber. It's that HTB which is the bottleneck you mention, and is the part I'm objecting to as both totally unnecessary and an expensive waste of resources to run a Xeon box to meet your shaping needs to do all those borrow calcs etc, plus it shapes "the wrong way"... HTB is going to ensure noone ever goes over the limit (unless you allow borrowing) it doesn't ensure they always can get at least their contracted rate.
It's not, I've asked for it in a bug report but it was put on low priority... It should be kinda trivial to add I'd think, just select it in the kernel config. I'm running QFQ on my desktop machines on my home LAN and it seems to work fine. I think it's fully stabilized since 2012 when you were looking at it.
3 Likes
The config I have is I think 4 classes with weights 1, 2, 4, and 1000 the 4 weight is normal browsing traffic, the 2 weight is NFS traffic, the 1 weight is CS1 bulk traffic like torrents etc and the 1000 weight is interactive traffic like zoom/jitsi/games.
Under this scheme if I ping my router at interactive priority and then dd a big file to my NFS server I get ~1ms increases in ping times and no traffic shaper is required at all (QFQ is a work preserving scheduler only)
2 Likes
I really like how you are describing things here. The way ISPs work nowadays is that they promise
"UP to 100Mbit" and it's up the consumers, lawyers, and userbase to hold their feet to the fire if they get less. If the promise is "No less than 100Mbit, and btw, if you try it at 1AM, odds are good you will do better" - the usage pattern shifts and people adopt themselves to the load.
So far as I know backend bandwidth costs are really really low, and pricing does not matter on a diurnal cycle much.
When I was a kid, same principle applied - you walked into the computer room at 1AM, you got more computer time to yourself than if you came in in the middle of the day, so determined programmers, with complicated code, like... me... worked nights.
So a small ISP COULD offer vastly more bandwidth at night (so long as the latencies are low) and
TELL people they did that...
About the only flaw in this scheme is... um... bittorrent. But even then it ain't much of a problem....
1 Like
I come up with pretty much the same analysis as you do, I just doubt that normal commercial ISPs would find that a good match for their business plan of charging users by capacity. But personally I would probably happily become customer of such an enlightened ISP 
Yes, exactly the traditional approach that puts a hard limit on a links peak information rate, that is what bean counters and ed-users are accustomed to, for better or worse.
I have my reservations whether rarely exercised qdiscs ever fully stabilize, but I have no idea/opinion on whether QFQ falls into that bucket
That indicates at least that your router's ethernet interface is not overbuffered... But I believe you are on a 1Gbps plan and run 1 Gbps ethernet..., no?
ISPs over here are in a silly race to "over-provide" their links (IMHO that is just PR-talk, in reality they simply under-report the provisioned rates, but I digress), e.g. normal DOCSIS 1 Gbps links are actually provisioned such that ~1.1 Gbps goodput can be achieved, which makes ingress traffic handling tricky without a traffic shaper (but the 1 Gbps links are not terribly over-buffered)...
+1; except in the EU (REGULATION (EU) 2015/2120 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL makes pretty clear and strict demands on ISPs what they need to deliver in respect to the contractually promised rates. This helps a bit with the "feet to the fire part", but still sticks to the old price differentiation by top-speed scheme.
+1; I would love such an enlightened ISP.
Yes, but ISPs always complain how expensive the roll-out (and less so the maintenance) of the access network is (which might just be a way to distract from the fact that bandwidth costs reduced a lot).
Yes, that might even be a way for such a small ISP to grow a dedicated customer base...
I guess worst case would be one QFQ class per subscriber, akin to cake's dual-xxxhost isolation modes, in both cases a per use fq_codel set would be helpful.
I was actually referring to on-LAN traffic, just pinging the router and dumping data to the NFS server, both on LAN. Basically QFQ keeps my computer from becoming non-interactive even if I'm hammering data onto the NFS server as fast as possible (ie. copying files or dumping the output of a scientific computation to disk). At the router, I run a multi-tier HFSC because although I'm on ATT's "gigabit" fiber, in reality they like everyone else run an "up to" scheme, and in reality shaping to 750Mbps produces a better result latency wise (and honestly the difference between 750Mbps and 1Gbps is irrelevant because no single website is likely to give you those kinds of speeds anyway)
It does seem to "just work" for me. I do understand there's a risk that things break and people don't notice though.
If an ISP is going to be enlightened, they should also offer diffserv as a service. Send EF or CS4 or CS5 traffic and they put you into a separate queue, with a hard limit of say 10% of the total link, but promise of low latency, 1ms to exit the ISP network... Send CS1 and they put you in a queue with up to 100% of the upstream link, but latency of up to say 100ms. Send CS0 and you're in the regular queue with latency increase of say 10ms.
I looked more carefully, and the way I set it up is the 2 weight is normal traffic, the 1 weight is bulk traffic, the 4 weight is for mild priority traffic, and the 1000 weight is for games/voip.
Right now I'm not using the "mild priority" traffic. NFS and bittorrent all go into weight 1, browsing in weight 2, and voip/games in 1000, the priority 4 queue is not actually in use but was intended to be stuff like DNS lookups or maybe if possible to differentiate a video and audio stream to put video in priority 4 and audio in 1000
1 Like
CS1 is unusable. Switches prioritize it, Wifi deprios it, and places like comcast remark everything they dont' recognise to CS1.
IF you want the new "hotness", please see LE: https://tools.ietf.org/html/rfc8622
So far as I know cake and sqm were the first to implement it. There is some scary stuff in the standard as operators wanted to be able to allow LE to starve, we gave it
a min 5% which seems sane.
As for the other diffserv codepoints.... FQ solves everything... really.
It is certainly useful to mark traffic you want to be treated badly in LE, but anything
else that tries for boost priority is doomed to game theory failure. I've showed time and time again how, in wifi - a good scheduler for JUST the BE tx queues in general outperforms the 4 in the standard.
1 Like
Not at my house. Everything that comes in WAN is remarked CS3, and then up or down-prioritized from there.
Not games or voip, and not video conferences. I have hard realtime requirements on games and voip, and soft-realtime on video conferences. Putting them in their own queues does the right thing. The "game theory" problems only occur if you misconfigure your LAN and let whatever random marks the app puts on the packets persist. In my LAN 100% of everything is remarked to meet my policy.
This is actually one of the misperceptions of Diffserv, it's supposed to be that each diffserv domain has its own policy and that 100% of packets are remarked at diffserv boundaries (that's your router). Instead people treat it as some kind of "set in stone" standard that's "supposed to be end-to-end but you know doesn't really work".
The other thing that is often misconstrued is that there's some kind of global "utility function" that all people can agree upon. Sometimes people act like they believe that even if they obviously wouldn't believe it if put in that way. Clearly though a home with a gamer a zoomer and a person downloading an Ubuntu ISO image will have three very separate utility functions:
- gamer doesn't need more than 500kbps but every packet needs to exit the router in less than 1ms without any drops
- zoomer has ~60ms of buffering in the app, so needs 3000kbps or so, with latency in the range of 20-30ms and zero drops of audio data, a little video data could drop if necessary.
- ISO downloader plans to spend ~15 mins so has requirement for "as much bandwidth as can be had" with latency low enough that acks keep the packets flowing, perhaps 100-300ms would be fine, and can drop anything that needs to be dropped to allow the other users to meet their requirements because it just means a couple retransmits over a 15 min download.
These are just really different requirements.
OK, there is a 99% chance we will never agree on this issue. However...
Could you try adding the LE codepoint to your existing setup instead of CS1 and see if it works? I have higher hopes for it than all the other dscp markings combined.
I'm still disappointed that my 'LE' tin aka diffserv5 for CAKE didn't get more traction and that it gets lumped into Bulk tin for diffserv3/4.
VO - 1/4bw min g'tee - highest priority - voice
VI - 1/2bw min g'tee - video streaming
BE - no min g'tee - normal traffic
BK - 1/16 min g'tee - Bulk - Downloads
LE - 1/64 min g'tee - Least effort - Bit torrent
For me, it's not so much the absolute priorities but more the soft admission priority limits/min guarantees.
1 Like
Well, for me BK is not bulk, but rather background, and IMHO LE is the "newly" coronated DSCP for background. The thing about background is, stuff in there is better latency insensitive, because that is what it is getting jitter and latency-under-load en mass ;). For unfortunately still existing slowish links with say ADSL at 16/2 Mbps, 1/64 for the uplink is 2000/64 = 31.5 Kbps, that is almost too little for BK traffic to make forward progress.
IMHO the bigger issue with cake's priority tiers seems to be that for gaming we would need an additional high priority tier that is hard rate limited (limit needs to be configurable) as some games seem to measure the available bandwidth...
So, it seems I am also advocating for a 5th (or 4th) optional priority tier, only on the high end...
BUT, unlike you I have not implemented that and hence did not test how well my scheme works, while your's is actually real-world tested and does work....
That is not going to work, IMHO, all tiers should have a minimum CIR/guaranteed rate, if only so that it becomes easier to reason about them, but if my math is correct, BE still gets 1 - (1/4) -(1/2) - (1/16) - (1/64) = 0.171875 ~ 1/6 by virtue of that being the left-over once all other minima are served, no?
Correct. 'minimum' is a bad way of expressing it with cake - it's the threshold at which a tin switches from priority to bandwidth mode. It's what enables cake to steal bandwidth from the BE tin. BE tin has nothing to steal from because actually it's the lowest priority tin, and can never exceed its threshold.
Internally the tin order is BE, (LE,) BK, VI, VO, the displayed order (LE,) BK, BE, VI, VO is a lie 
It has a minimum guaranteed bandwidth by virtue that (hopefully) the higher priority tin's minimum thresholds don't add up to the total shaper bandwidth - best effort gets what's left on a best effort basis.