SQM cake: traffic prioritisation

Okay thanks but I am bemused why I have to lose over half of my bandwidth then to get an acceptable result?

Recall I am syncing at 64Mbps and I have to cap my speed to 25Mbps, I'm sure this is not typical?

Well, do you?

I thing it might make sense to start a binary search for the highest shaper rate that still yields acceptable latency under load increase.... We know 50 Mbps seems to high, while 25 seems okay, so maybe try 37.5 next? If still acceptable try 43.75, if not try 31.25....

But yes, sometimes the requiresdrate "sacrifice" to get low latency under load increases can be significant. That is onevof thecreasons why I believe that every network admin needs to make a policy decision what trade-off he/she is willing to accept.

1 Like

I haven't read all the way through this thread so I've probably missed something. In terms of deliberate capacity loss due to cake in my setup I run egress at 99.5% of upstream vdsl modem rate (ie 20000kbps ends up at 19900kbps configured). Ingress I run at 96.25% of downstream modem rate (ie 80000kbps ends up at 77000kbps)

Note I have ingress mode enabled on the ingress shaper, also 'bridged-ptm ether-vlan mtu 72' to help ensure vdsl framing, overheads and frame sizes respected/calculated. As a sky customer I don't have to get involved with PPPoE (thank goodness) so if PPPoE is involved you'll need the ' pppoe-ptm' keyword instead of 'bridged-ptm'. UK vdsl also has a vlan tag overhead hence the 'ether-vlan' tag.

In terms of induced latency on busy link, I typically see around 3ms in the egress direction and 7ms on the ingress though ingress is usually a bit more spiky.

In the graph below, 3am-9am was my daily backup to onedrive going through and just after 9am to end of day was a large download where I was deciding to actually test one of my backups (it worked)

https://www.thinkbroadband.com/broadband/monitoring/quality/share/5bf57e52bbdcec699e3141cb95622dc901a0fc26-09-10-2020

In my setup backup traffic is categorised as 'bulk' traffic so everyone was oblivious to the fact the link was 'full' and normal browsing, downloads, streaming just worked...the backup traffic gave way to everything else.

3 Likes

I would not use the two ppm variants, as I consider this to be too much work for too little gain. PTM#s 64/65 encoding really is ignorant of packet sizes and hence is cheaper to account simply by reducing the shaper rate to <= 100*64/65 = 98.46 % of the sync rate, instead of going an much approximate calculation for every packet (this is different from ATM/AAL5 which requires per packet adjustments)... Especially, since quite a lot of ISPs use secondary traffic shapers so that the sync is not the relevant limit anyway... ,)

Well, always good to see numbers from real life usage, thanks.
That said, my >= 5 ms per congested direction really is not an approximation but really a fundamental property of how codec-type AQMs work with their default target of 5ms, if faced with saturation of sufficient duration :wink:

1 Like

Hi, are you able to post your full SQM config please?

Sure, beware I use a custom sqm script 'ctinfo_5layercake.qos' which combines a patched version of cake implementing a 'diffserv5' 5 tin classification system and 'ctinfo/connmark dscp' to store DSCPs across connection egress/ingress paths so DSCPs and hence classifications can be restored on packet ingress.

# cat /etc/config/sqm 

config queue
	option debug_logging '0'
	option ingress_ecn 'ECN'
	option interface 'eth0'
	option qdisc_advanced '1'
	option egress_ecn 'ECN'
	option qdisc_really_really_advanced '1'
	option squash_dscp '0'
	option squash_ingress '0'
	option linklayer 'none'
	option enabled '1'
	option script 'ctinfo_5layercake.qos'
	option iqdisc_opts 'dual-dsthost bridged-ptm ether-vlan nat mpu 72 ingress'
	option eqdisc_opts 'dual-srchost bridged-ptm ether-vlan nat mpu 72 ack-filter'
	option verbosity '1'
	option qdisc 'cake'
	option upload '19900'
	option download '77000'

And the modified sqm scripts https://github.com/ldir-EDB0/sqm-scripts/commit/e701873cf6393d056360dbc4f174db602ca02f09

Patches to cake for 'diffserv5' https://git.openwrt.org/?p=openwrt/staging/ldir.git;a=shortlog;h=refs/heads/mine

and some domains filled in by dnsmasq (/etc/config/dhcp)

        list ipset '/zoom.us/Zoom4,Zoom6'                                                                                           
        list ipset '/googlevideo.com/Vid4,Vid6'                                                                                     
        list ipset '/nflxvideo.net/rangeA-netflix.cdn.enbgk.isp.sky.com/Vid4,Vid6'                                                  
        list ipset '/aiv-cdn.net/r.cloudfront.net/aiv-delivery.net/Vid4,Vid6'                                                       
        list ipset '/s.loris.llnwd.net/as-dash-uk-live.bbcfmt.hs.llnwd.net/aod-dash-uk-live.bbcfmt.hs.llnwd.net/aod-dash-uk-live.aka
        list ipset '/vs-dash-uk-live.akamaized.net/Vid4,Vid6'                              
        list ipset '/cdn.bllon.isp.sky.com/live.bidi.net.uk/Vid4,Vid6'                     
        list ipset '/ssl-bbcdotcom.2cnt.net/Vid4,Vid6'                                     
        list ipset '/fbcdn.net/Vid4,Vid6'                                                  
        list ipset '/ttvnw.net/Vid4,Vid6'                                                  
        list ipset '/vevo.com/Vid4,Vid6'                                                   
        list ipset '/millicast.com/Vid4,Vid6'                                              
        list ipset '/xirsys.com/Vid4,Vid6'                                                 
        list ipset '/audio-fa.scdn.cot/Vid4,Vid6'                                          
        list ipset '/deezer.com/Vid4,Vid6'                                                 
        list ipset '/sndcdn.com/Vid4,Vid6'                                                 
        list ipset '/last.fm/Vid4,Vid6'                                                    
        list ipset '/v.redd.it/Vid4,Vid6'                                                                                           
        list ipset '/ttvnw.net/Vid4,Vid6'                                                  
        list ipset '/ms-acdc.office.com/windowsupdate.com/update.microsoft.com/Bulk4,Bulk6'
        list ipset '/1drv.ms/Bulk4,Bulk6'                                    
        list ipset '/1drv.com/Bulk4,Bulk6'                                   
        list ipset '/graph.microsoft.com/BE4,BE6'                            
        list ipset '/web.whatsapp.com/BE4,BE6'   
5 Likes

Thanks. I am totally bemused then why I have to lose so much bandwidth to get this to work.

Somebody else said earlier it's because the issue is outside my network, I still don't really know what this means.

Could it be a router issue? Is my Archer C7 too old? Would a new router be better?

@ldir what are you using?

I use an APU2 - since you're not using the built-in wifi I think the c7 should be good enough...just. Check cpu usage.

One quick and dirty way of doing that is to log into the router via SSH and then run top -d 1 and then observe the value in the % idle column at the top, if that value get close to say 10% percent (on a single core router, for traffic shaping it is typically the sirq% that go up when idle% goes down) or lower that would indicate your router running out of CPU cycles to spend on traffic shaping, which would result in undesired latency under load increases. CPU time accounting is a bit peculiar so looking at the time the CPU does nothing (idle) is the simplest was to gauge the CPU load. With multicore CPUs you either need a top variant that can show values per CPU or you need to adjust the reference value; on a dual core CPU, idle 50% could mean either one CPU is maxed out (0%) and the other is fully idle (100%) or a mixture. Keep in mind that can ATM really only runs on a single CPU and hence a single maxed CPU can already be s a sign of critical overload.

Whilst downloading a single torrent @moeller0

Well, that looks a lot like CPU overload. The issue is that traffic shaping is not necessarily that CPU bandwidth dependent, but to maintain low latencies SQM needs get access to the CPU quickly enough that there are no "gaps" in the transmission (at the desired shaper-rate). For a number of modern multicore SoCs SQM runs intro issues, as its sustained load is not high enough to keep the CPU in the high-frequency/high-power regime, but then the scaled down CPU is not powerful enough to process SQMs bursty loads in a timely enough fashion and the throughput suffers.

What services are you running on your router? Sometimes ostensibly sane and cheap services can require way more CPU cycles than one expects (case in point SQM itself, traffic shaping takes way more CPU cycles than it should).

1 Like

How would I get a list?

I am quite willing to upgrade the router if it's necessary but I would like to future proof myself. What would handle 1Gb with SQM?

Mmmh, maybe just run top -d 1for a while, and look at the %CPU column that should show which process consume considerable CPU time. (or use ps aux to just get an instantaneous list of processes)

That requires some beefy CPU, so either x86 or ARM cores. There are reports here in the forum, that the raspberry Pi4B paired with an TP-Link UE300 USB§ Gigabit ethernet dongle will allow traffic shaping up to a gigabit (if configured correctly), but to replace your current router you will also need a gigabit ethernet switch. So cost wise you are looking at ballpark 100EUR (depending on the Pi4B model you buy, just make sure that you have sufficient cooling).

Cheaper devices like the Brume or turris MOX will probably allow shaping up to around 500-600 Mbps (combined up and downlink) and might be an option at least for your current link.

Please note that gl-inet and turris will ship customised OSes based on OpenWrt, as neither of the three seems to be supported by current stable OpenWrt builds. That said, I expect the RPi4B to be added hopefully with the coming 20.X stable version of OpenWrt (albeit it seems more like a 21.X version by now).

In terms of x86, have you got any suggestions? What kind of hardware would I need on a minimum for 1Gb with cake?

Sorry, no real first hand experience I can offer. Personally I would probably go the raspberry pi4B route first, since it should be considerably cheaper.
But then I limited my 100/40 link down to 49/31 as that was the maximum my old router could achieve, and I value low latency under load more than pure throughput :wink:

1 Like

@moeller0 I've swapped to a Zyxel NBG6617 in the meantime. I hope this will buy me a bit of breathing room.

Mmmh, for your current internet access speed that might do. Could you repeat the top test while torrenting with SQM active, please?
So we see whether that helped?

Had a ping monitor running via Think Broadband for a couple of days.

This to me isn't looking too good but the pings from the router, e.g. to BBC or Google are consistently low even when downloading. I have set the download and upload speeds to be low.

Why does Think Broadband show such a different picture?