SQM: Fast and Slow device

I have set up 2 routers in the same location.
1st: SQM at 20Mbit Down/2Mbit Up
2nd: SQM at 250Mbit Down/26Mbit Up

For the 1st one, all connected device can get a very quick response time. All device included Wifi tested Bufferbloat at A+

However, for the 2nd router, the Gb wired connected device is good - no bufferboat issue. Some device esp over 5G/N Wifi have bufferbloat. I have to reduce speed for all device by SQM 120Mbit Down/26Mbit Up so that those over slow WIFI standard to have Bufferbloat A+

My questions are:

  1. Do I usually need to take care of the slower Wifi device when using SQM? I suppose as long as the router reserved the some 15% of total bandwidth unuse by a single connection will get optimized quality.......
  2. Is there a way to reduce link speed for a specific device mac address/ip address/SSID?
  3. What is the best/doable solution for my case?


Sounds like your "slowest link" is not the upstream WAN connection, but your slow LAN wifi. That is rather atypical, as generally most people have slower WAN than LAN.

I think that you need to adjust the targeted speed limit settings according to the slower wifi connection inside LAN.

Sounds logical that you need to slow down the WAN download speed to prevent bufferbloat, but you do not need to adjust upload:

  • Uploading from the slow device has the slow wifi link first on the path, so after the traffic reaches the router and the speedy WAN uplink, there is no need for limiting/buffering traffic any more
  • Downloading from speedy WAN to router and then routing downstream via slow wifi causes buffering needs at the router when the WAN download speed is set higher than the slow LAN link accepts.

SQM qdiscs do not really "reserve bandwidth". Generally they just try to minimize latency and use in the calculations the set speed limits as the assumption for the slowest link.

That may well be your optimal solution.

Just a question out of interest, since I am looking for a new router to handle similar internet connection speeds with shaping enabled: What router are you running that does 250/26 Mbit with shaping enabled? And what shaping algorithm do you use? Piece of cake, fq_codel, or something else entirely? Thank you very much in advance for the information!

@hnyman Thanks for your advise. You fully get my situation. Now after I do more experiment on the slowest device, I think I better stay at the 250Mbps settings rather than 125Mbps. After all it looks like my ISP is getting slow in this 2 hours making me feel it is related to Bufferbloat - It is not bad, rating is B.

@Mushoz The router I am using is WRT1900ACS version 2.
My ISP provision 300Mb/30Mb for me.
My SQM settings:
qDisc: Cake
script: Piece of Cake or Layer of Cake (both seems the same)

I tested fq_codel, it limits the speed below 100Mbps, just like the original linksys firmware does.

After reading the release note of 17.01, I retest my wifi with my old ath9k router with my 300Mb/30Mb connection

The settings of ingress/egress is still the same (set at 250Mb Down/ 26Mb Up

This time even the settings is not set below my client's limited speed of 120Mb, the Bufferbloat test still have A ranking. This lead me to think this is the mentioned ath9k bufferbloat optimization..... So I pack my WRT1900ACS and send back now - using the ath9k router is more responsive afterall

script: Piece of Cake or Layer of Cake (both seems the same)

Is there any actual difference between those two? Like one of them is lighter on CPU or the other gets better results?

using the ath9k router is more responsive afterall

Are you using it with cake or fq_codel?

Yes, piece_of_cake only employs cake's besteffort mode, so all packets are processed in the same tin. Layer_cake uses diffserv4 which uses 4 different priority tiers and uses DSCP marks to steer packets into the correct tin. For example layer_cake will route packets marked with CS1 (which got redfined as background class) into a low priority tier. The idea is to put scavenger services we bit-torrent into this tier so cake will throttle those harder to make room for higher priority traffic. I hope that help a bit...

Best Regards

Yes, it does.
You mentioned bittorrent marking as CS1. I guess I have to mark them myself, it's not automatic? I think I would like to utilize that feature.

Do you have maybe any reading material about this stuff?

@r43k3n When I go back to the old ath9k router WZR-HP-AG300H Atheros AR7161@680MHz 128MiB ram, I notice below on SQM on wire:

  1. fq_codel + simple.qos is stable at 153Mbps it increase to this level and flat until the test end.
  2. cake + layer_of_cake reach 180Mbps and then fall to 165Mbps and stable until test end
  3. cake + piece_of_cake go up and down between 170Mbps - 190Mbps, stable at 175Mbps
  4. The speed above is far below the speed I set in the SQM settings: 270bps
  5. Wireless always have A+ bufferbloat, speed is at about 120Mbps (Maximum for this router 5G)

in WRT1900ACS, fq_codel + simple.qos does not exceed 100Mbps. cake + xxx_of_cake can reach maximum I set in SQM settings.

I do not have much knowledge on the results above, I don't know how to modify the queue setup script as well - just from an end user aspect to give you the figures.

`using the ath9k router is more responsive afterall`
 Are you using it with cake or fq_codel?

It does not matter. My point is: My old router can give me more responsiveness on daily usage, such as Google a lot of info with many webpage click, it is faster. I only notice this change after going to LEDE. So when I look at the release note:

Improved Networking Support

  • Improvements to the WiFi stack eliminating bufferbloat on ath9k, mt76 and some ath10k chipsets

This make me realized there are something done to the Wifi stacks.

For what I paid for the new WRT1900ACS, it gives me throughput only, but not responsiveness of the level ath9k now give me. So I choose to stay in ath9k, and will probably get the R7800.

Well, unfortunately it is not straight forward to dscp mark arbitrary application's packets, one's best bet is using a (torrent) client application that explicitly allows to specify the dscp marks. The challenge is finding a client that allows to set DSCP marks effortlessly (deluge seems to offer that, as does vuze and transmission*). And even if you manage to do this on your system all the incoming packet most likely are marked wrongly (since the sender would need to use a reasonably DSCP and all transit points would have to leave the DSCP marks alone, I am highly doubtful that this combination is sufficiently likely to make torrent sorting by inbound DSCPs a viable solution in the real world, but no matter my opinion, it still can and should be tested).
And a quick overview of the challenge with dscp and torrent is I believe found under https://nelsonslog.wordpress.com/2014/06/11/tosdscp-byte-in-ip-packets/

*) the state of tos/dscp suppprt in bit-torrent clients was unsatisfactory the last time I looked, often they try to set the whole 8bits of the old TOS field, even though two of these bits are now reserved for ECN and should not be touched. Also I seem to recall that transmission only allows dscp

I kind miss the ability to set different priorities in sqm-scripts like it was possible in qos-scripts. I'm not sure why they abandoned this feature but it was useful.

one's best bet is using a (torrent) client application that explicitly allows to specify the dscp marks

That would be great but...

all the incoming packet most likely are marked wrongly (since the sender would need to use a reasonably DSCP and all transit points would have to leave the DSCP marks alone

Would it be enough to just mark DSCP with iptables? Usually (most) torrent traffic is generated (I think) on one port selected in bittorrent client settings. So if we set a iptables rule to mark all traffic generated on that port would it be enough to utilize DSCP function of SQM? It's not ideal but it's far more reliable then counting on other people on the internet to have DSCP configured too. I think this would need to be done as default setting on torrent client for users to actually use it.

Update1: I have a dedicated computer in my home that handles almost all torrent traffic. It's a Windows machine (with reserved IP with qBittorrent installed on it on port (let's say) 62888. I my case it would be sufficient to just mark all traffic coming in and out of this machine on that specific port. Wouldn't you agree?

Well, sqm-script's goal is sort of being a generic buffer-debloater "for the rest of us" that does the right thing with minimal configuration requirements. The qos-scripts feature you mention is almost diametrical opposite to that goal :wink: That said, sqm-scripts effectively is infrastructure to handle different actual qos-policy scripts, you can simply create your own scripts with all the bells and whistles you might want (see /usr/lib/sqm/*.qos for examples). What you will not get is a GUI that allows easy tweaking of non-sqm parameters.

Well, for ingress shaping packets see the shaper before iptables had a chance to run, so this will not work, you could use "tc filters" instead, but these are a bit unwieldy (but supposedly fast). But that really depends on being able to reliably select torrent packets without mis-classifying to many non-torrent packets into the same tin. But anything requiring user intervention in the client seems to be not sufficient (given that most torrent clients I have seen allow to configure incoming bandwidth explicitly, so if users really cared they would set an acceptable limit there and be done with most of the problem; but since I do not use torrents routinely, I have no personal experience one way or another).

Oh, sure in that case things get relatively easy (especially if you would simply put all of that machines traffic into the background class). The challenge than still would be that tc filter on the wan interface will see incoming packets before NAT is performed so has no easy way to get to the internal private address of that machine. What I would recommend for your use case with a single torrent host would be to try the instructions at https://forum.turris.cz/t/how-to-use-the-cake-queue-management-system-on-the-turris-omnia/3103 to set the "Advanced option string..." for egress to "nat dual-dsthost" and for ingress to "nat dual-srchost" and see whether that alone might not alleviate your torrent issues sufficiently to make finer-grained QOS configration less desirable/urgent?
Please note to check first whether your cake module actually supports the nat option, by logging into your router and issue "tc qdisc add root cake help". Here is an example output from Reboot (SNAPSHOT, r3109-6b013019f9)
user@router:~# tc qdisc add root cake help Usage: ... cake [ bandwidth RATE | unlimited* | autorate_ingress ] [ rtt TIME | datacentre | lan | metro | regional | internet* | oceanic | satellite | interplanetary ] [ besteffort | precedence | diffserv8 | diffserv4 | diffserv-llt | diffserv3* ] [ flowblind | srchost | dsthost | hosts | flows | dual-srchost | dual-dsthost | triple-isolate* ] [ nat | nonat* ] [ ptm | atm | noatm* ] [ overhead N | conservative | raw* ] [ wash | nowash* ] [ memlimit LIMIT ] (* marks defaults) root@nacktmulle:~#

If this does not show "[ nat | nonat* ]" then my recommendations will not work.... (not all all sqm-scripts with incorrect advanced option strings will fail to start-up correctly, there is a reason why this field is hidden behind to increasingly alarmist check boxes and carries a scary name/description).

Best Regards

Yes, I realised that a few minutes before your post :smiley:

Yes, I could set an upload and download limit on my torrent but here's the thing. The computer with torrent is running 24/7 and at night no one uses the internet so why not let bittorrent utilise the whole connection when available? I use SQM to exactly manage my connection in response to number of users and level of utilization to make sure everyone is given a fair share of available resources.

I actually got the same recommendation a while back from sqm/cake developers when I posted and issue regarding fq_codel. Below you will find an exact quote. I think I will use this configuration and get back to you. Thank you very much for your help.

Basically, what you want is the 'dual-srchost' option applied to your egress interface. Or the default, 'triple-isolate' will work too, though I'm not sure when it became the default.

One question though. Do you know what he means by "triple-isolate"?
Is it maybe this "layer_cake" feature you mentioned before?

Layer_cake uses diffserv4 which uses 4 different priority tiers and uses DSCP marks to steer packets into the correct tin.

[quote="r43k3n, post:13, topic:1236"]
One question though. Do you know what he means by "triple-isolate"?
[/quote]Yes, he does: https://www.mail-archive.com/cake@lists.bufferbloat.net/msg00502.html

"Triple isolate" is the current default mode for Cake in SQM 1.1.2.

You can verify that in console:

root@lede:~# tc -s -d qdisc show
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 800a: dev eth0 root refcnt 2 bandwidth 9Mbit besteffort triple-isolate rtt 100.0ms raw

Triple-isolate promises to do something in-between per-internal-IP and per-internal-IP fairness making it possible to forego having to configure the right "direction" into the shaper. The goal making it easier for the user to get a reasonable, albeit not perfect, fairness guarantee is surely laudable;

So "triple-isolate" was design to make the configuration "hands free" but at a cost of the results. From the quote above I conclude that using dual-srchost and dual-dsthost configuration provides overall better results but requires user input and effort to configure.

Hi r43k3n,

first of I concurr with hnyman ;). What is "better" is a policy question IMHO, so I hesitate to call triple-isolate worse than dual-XXXhost, they just serve different use cases. So if strict per IP fairness is your goal than surely the dual-isolation options fit the bill, if you typically just want no side to hog the link and you do not want to think about the directionality triple-isolate is your friend. I would not be amazed if the enthusiast crowd here would tend to fall into the first camp...

Best Regards

To be fair I haven't been able to find much information about triple-isolate. I basically found out that there is no more information about it because it's very hard to explain to people, what this thing is actually doing, how it works or in which cases is it better or worse. It's recommended to leave it be if you're not sure what you are doing.

I was actually able to find out that some people were more happy with using dual-XXXhost rather then triple-isolate which only confirms my earlier conclusion. I also agree with you that triple-isolate is sufficient for most people and the "enthusiast crowd" will probably go with dual-XXXhost.

@moeller0 - I wanted to ask you about some of the config parts from the link you provided.

option squash_dscp '1' 
option squash_ingress '1' 

Those two options are completely against of everything that you've told me about DSCP. I would like to leave those options set to '0'. Is it OK or does it somehow interfere with dual-XXXhost set-up? Why someone would what to have those options enabled?

Well, to make sure we are on the same page et me quickly describe what each of these options does:

squash_dscp: this will remap the DSCP filed of incoming packets to CS0 (basically it will make the 6 DSCP bits unset, which is the default or besteffort marking)
squash_ingress: this is misnamed a bit it does no squashing at all, it does instruct simple.qos (and layer_cake IIRC) to simply ignore incoming DSCP bits, effectively reducing the ingress direction to a 1-tier shaper, whithout any special priority bands.

The reason for this set up is that iptables which we use for remapping/zeroing the DSCP bits is not available at the point in time the ingress qdisc handles incoming packets. If you do not trust the incoming packets DSCP markings (and there are no real good reasons to trust them without periodically checking them) squash_ingress makes sense. Even after ignoring the incoming DSCP fields it still can be a good idea to squash_dscp, as some switches and wlan adapters (see: https://en.wikipedia.org/wiki/IEEE_802.11e-2005) might automatically try to do priority shenanigans which might or might not be what you want (and especially whether you want untrusted outside sources affect you WLAN performance).
To your question, it is a simple policy question whether you want to trust incoming markings to be reasonable (the set squash_ingress to 0) and whether you also want internal devices make decisions based on the DSCPs (which can be a great help for debugging...).

Best Regards