CAKE w/ Adaptive Bandwidth

Thank you so much for the quick reply! I configured cake very shoddily with SQM in Luci. It's quite possible I don't have that part configured properly. And of course, my hotspot/modem is locking onto Band 14 right now which is a little better on latency and worse on speed than the usual Band 2 which I get higher speeds and wilder latencies/bufferbloat. Sorry for the linked images for cloudflare tests, as a new user it only would let me embed one :confused:

tc -s qdisc (cake-autorate off)

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 359614 bytes 1425 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 13432 bytes 118 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 116831 bytes 356 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 66918 bytes 199 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 88856 bytes 258 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 73577 bytes 494 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 15315820570 bytes 12612487 pkt (dropped 0, overlimits 0 requeues 152)
 backlog 0b 0p requeues 152
  maxpacket 16272 drop_overlimit 0 new_flow_count 4639 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 14874328 bytes 68443 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1512 drop_overlimit 0 new_flow_count 1124 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
root@OpenWrt:~/cake-autorate# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
 Sent 461634 bytes 2037 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 29703 bytes 270 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 172923 bytes 588 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 82442 bytes 294 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 96671 bytes 323 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1356 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 79895 bytes 562 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 15316156816 bytes 12613109 pkt (dropped 0, overlimits 0 requeues 152)
 backlog 0b 0p requeues 152
  maxpacket 16272 drop_overlimit 0 new_flow_count 4639 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 14895754 bytes 68512 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1512 drop_overlimit 0 new_flow_count 1124 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

tc -s qdisc (with cake-autorate on)

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 806f: dev eth0 root refcnt 6 bandwidth 2463Kbit besteffort dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 44 mpu 96
 Sent 26809725 bytes 116051 pkt (dropped 3002, overlimits 178115 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 242348b of 4Mb
 capacity estimate: 7Mbit
 min/max network layer size:           28 /    1342
 min/max overhead-adjusted size:       96 /    1386
 average network hdr offset:           14

                  Tin 0
  thresh       2463Kbit
  target         6.61ms
  interval        102ms
  pk_delay        450us
  av_delay         42us
  sp_delay          3us
  backlog            0b
  pkts           119053
  bytes        29221326
  way_inds            3
  way_miss          250
  way_cols            0
  drops            3002
  marks               0
  ack_drop            0
  sp_flows            4
  bk_flows            1
  un_flows            0
  max_len         13560
  quantum           300

qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
 Sent 123373419 bytes 113567 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 16186715285 bytes 13378412 pkt (dropped 0, overlimits 0 requeues 810)
 backlog 0b 0p requeues 810
  maxpacket 58308 drop_overlimit 0 new_flow_count 9756 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wlan0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 4Mb ecn drop_batch 64
 Sent 15302259 bytes 69812 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 1512 drop_overlimit 0 new_flow_count 1130 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8070: dev ifb4eth0 root refcnt 2 bandwidth 25555Kbit besteffort dual-dsthost nat wash no-ack-filter split-gso rtt 100ms noatm overhead 44 mpu 96
 Sent 124853077 bytes 113403 pkt (dropped 164, overlimits 180912 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 505730b of 4Mb
 capacity estimate: 40Mbit
 min/max network layer size:           46 /    1342
 min/max overhead-adjusted size:       96 /    1386
 average network hdr offset:           14

                  Tin 0
  thresh      25555Kbit
  target            5ms
  interval        100ms
  pk_delay        935us
  av_delay        115us
  sp_delay          4us
  backlog            0b
  pkts           113567
  bytes       125074381
  way_inds            7
  way_miss          243
  way_cols            0
  drops             164
  marks               0
  ack_drop            0
  sp_flows            3
  bk_flows            1
  un_flows            0
  max_len         13560
  quantum           779

ifstatus WAN | grep device

"l3_device": "eth0",
        "device": "eth0",

Waveform test, no cake: https://www.waveform.com/tools/bufferbloat?test-id=da5334c9-2278-4b7e-a47c-dd2f0f04ea01

Cloudflare, no cake: https://imgur.com/a/PH7hXvO

Waveform test, with cake-autorate: https://www.waveform.com/tools/bufferbloat?test-id=4d57c4cf-b869-47d2-ab47-2fb4697615b0

Cloudflare, with cake-autorate: https://imgur.com/a/iW5e1Kx

My autorate config (no overrides, tried playing with them a few times but not sure exactly what I'm tweaking with it):

dl_if=ifb4eth0 # download interface
ul_if=eth0     # upload interface

# Set either of the below to 0 to adjust one direction only
# or alternatively set both to 0 to simply use cake-autorate to monitor a connection
adjust_dl_shaper_rate=1 # enable (1) or disable (0) actually changing the dl shaper rate
adjust_ul_shaper_rate=1 # enable (1) or disable (0) actually changing the ul shaper rate

min_dl_shaper_rate_kbps=5000  # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=15000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=40000  # maximum bandwidth for download (Kbit/s)

min_ul_shaper_rate_kbps=500  # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=1000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=5000  # maximum bandwidth for upload (Kbit/s)

My SQM configuration, if it's any help (saw 44 mentioned as a safe overhead, also FirstNet has a strange MTU of 1328, so I changed max packet size from default of 2047 to 1875 as I had assumed that that number itself assumes an MTU of 1500, so I figured out the difference and changed it just screwing around trying things lol)
https://imgur.com/a/SoJTmVK

That really is not correct, or rather not aligned how these numbers are interpreted by tc's stab option, however cake does its own atm accounting and ignores these numbers and your link does not use atm in the first place...

If your gaming spikes came without cross traffic from the rest of your network and also occur without cake running, these might not be fixable by a local traffic shaper....
EDIT: I mixed up two threads and thought your challenge was gaming.... will have a closer look later today and hopefully come up with a more meaninful reply....

For cake-autorate tests, the best you can do is to export a log file directly after your throughput tests so we see how throughput and latency develops over time... as well as what shaper settings the script selected.

These are a bit tricky... at 500 Kbps a single full MTU sized packet (depending on the used encapsulation around 1540 bytes in size, but let's just run the numbers for 1500) will take:

[milliseconds/second] * ([bytes] * [bits/byte]) / ([Kilobit/second] * [bit/Kilobit]) = [milliseconds]
1000 * (1500 * 8) / (500 * 1000^1) = 24 ms

That means on average your upload delay under load will be around 12ms and the jitter quite large. At those speeds, all we can do is to manage the pain.... and at 1Mbps the per packet transmission time is still 12 milliseconds... autorate will adjust the delay threshold so the shaper will take this into account, but your experiences delay will be higher by necessity. If you are lucky the minimal upload rate rarely triggers (but we need to look at your autorate logs to confirm that) but the base_ul rate will be where the controller starts after some idle time, and 1Mbps is still in the "manage the pain region".

What are the delay thresholds you configured for autorate?

1 Like

If I'm understanding the question correctly, I have the thresholds at the default 30ms on both upload and download

Maybe you could post your full config file again? As well as a log (@Lynx could you remind us on the exact invocation to extract the log with the current code? I have not been running any recent autorate, so do not know how this is done ATM...)

Hi,

I already use cake with piece of cake script and the luci app.
Installing your script will allow me to just select autorate under the same luci app?
Or should I remove anything cake related before installing your script?

Thanks.

No, cake-autorate is a different beast, however it is compatible with sqm/piece_of_cake in that the autorate script expects you to somehow instantiate your initial cake qdiscs, which is what sqm can deliver easily.

You will need something or need to edit the script to start the required cake instances.

Hi @moeller0
Thank you for your reply.

I went through the github page and had a quick read.
It seems that maybe for me it won't do much. I have FTTH connection so I always have a stable bandwidth from my ISP.

Also already with piece of cake I get A+
So I don't believe it will dustify the extra cpu load.
Especially since I already run a schedule to automatically do speedtest and re-apply the SQM values everyday...

That been said, this is a good solution and I wish I had it at the before FTTH years!

2 Likes

I am in a similar boat, my dsl connection is refreshingly boring and always delivers maximal contracted rates (fullsync), all autorate can do on such a link is 'not getting in the way'...
Now on a variable link rate from a less customer friendly ISP things can look quite differently...

1 Like

** cries in FTTH with highly variable several unstable isp-**

That's very fortunate, I'd imagine that cake-autorate Could~ at the very least improve the micro seconds uS instead of ms.

if you live near the servers you're doing the speed tests, everything will essentially be A+ regardless, at that point you wanna look and see if the jitter varies, and just assume if theres high jitter on the network there might be an underlying issue that the isp should've resolved.~~ autorate might be able to improve~

I've seen cake-autorate being able to lower my jitter and overall bloat significantly, and it noticeably impacts the response and fluidity of online games, now the only people that teleport are the ones with high ping and not everyone
(I don't remember this being an issue in like 90% of games 10-15 years ago but here we are)

( we shouldn't have to worry about this normally unless the whole countries networks get overloaded during certain times.... where I currently reside.. it seems that there is alooooot of catching up to do when it comes to stability and quality... speed isn't everything.... especially if you get long duration 3000ms+ spikes while on a 2.5gbps line :expressionless: )

The autorate works very efficiently now and I struggled setting it up initially So I'd like to recommended it to almost everyone but not everyone has the resources to try it out sadly.

I'm looking forward to the projects This community puts together its super interesting everything they're doing and improving

Hmmm, so autorate can not do this, its control loop operates in the single to low double digit millisecond period range... And even cake's per-flow control needs to see queuing delay >= 5 for at least 100ms before it engages... My gut feeling is what you see is caused by cake's flow queueing scheduler and not the AQM, that scheduler does a much better job in interleaving packets of different flows on the egress, so that no flow is delayed for too long, just because another flow just dumped a bunch of packets in one burst into a shared queue. The scheduler (if combined with a traffic shaper that keeps the bottleneck queue under control of that scheduler) does a terrific job of sharing a link between different traffic aggregates (and autorate does a great job of keeping the scheduler in control of the queue on variable rate links).

I admittedly mostly see reports of games who are unhappy with what they see on their links, but I am not 100% sure modern on-line games network stacks are as robust and reliable as one would expect, given the multi decade history of quick reaction-time gated on-line games...

Yes, @Lynx did a great job here! If you have time, maybe consider writing a quick how-to for to setting it up, the sort of how-to you would like ot have at hand when you started?

Fortunately the Installation is very simple to understand but limited exposure to using SSH&CMDs was the most difficult part towards understanding how to set it up, I wouldn't know how to make it easier to understand

The Auto-rate seems to do great during my.. abnormal experience, its capable of keeping the rates as best as it could at least... given the.. most abnormal results I've gotten in the last 2 weeks.

Would you happen to know what would cause the.. Latency/network to have such a weird State, asfaik Its not exactly ~peak hours~ or whatever excuse the isp's here will try to get away with. it's essentially double the latency, 15-30x the jitter and high packet loss


What differed between these twi tests? I would guess these used different remote end points (both seem to use cloudflare infrastructure, but waveform uses their CDN while their own test apparently uses some dedicated locations, you cut of the server location map (likely for privacy reasons) that might allow to make some estimates). Aren't you behind MAP-T, so with no direct IPv4? Is there a chance of using IPv6 directly?

The difference as far as I can tell is 10 mins, both routed towards the same server im forced to bounce off that route due to..the. routing usually it will be osaka->tokyo (with NTT west even though theres like 2 other locations closer than both of those)

Though on a normal day, running those tests one after the other would normally have 2ms-5ms difference (and sometimes the ipv4 works and sometimes the ipv6.. idk the logic behind it, but if I run the tes.....


it would appear the v6 address likes to route through tokyo and the v4 through osaka~

(I think my ipv6 addr is "fake/internally routed" through the gateway as i cant reach any sites that use ipv6 without the map-e protocol~ (which is just ipv4 encapsulated by ipv6 asfaik i can ping with both v4 and v6 but with neither without the map-e protocol to make things worse..... the v4 and v6 are permanently assigned to my... I wanna cry T_T)

--- using openwrt over the isp recommended routers.... I tend to have less problems with openwrt if you can believe that :confused: I clearly have no idea what I'm doing still....

I am thoroughly confused now, I would have expected IPv6 to be more consistent given that no translation appears necessary...

1 Like


So my network under stable hours (demon hour)
The script is doing well managing the bloat regardless~ even during the peek hell hours It was able to stay within 60~70ms (which is... double my normal ping)

That looks considerably nicer, I wonder, can try to get the IP addresses of the remote endpoint of the cloudflare speedtest and run a mtr -ezb6w -c 100 ${THE_SPEEDTEST_SERVERS.IPv6.ADDRESS} both during demon hour as well as when things are bad. I wonder whether there might be differential routing in play as well?

1 Like

image
Just curious, would this~ command work, this is the ip i'm getting from cloudflare through the STUN protocol during the speed test (as thats the only one that showed up).

Though I'm not getting anything following after inputting that command while doing the speed test/before

Sorry, leave the ${} part out.... that is how in shell you decorate a variable, so ${THE_SPEEDTEST_SERVERS.IPv6.ADDRESS} was shorthand for 2606:4700:90:0:2a8f:ccfa:caad:78b6 and not for a literal ${2606:4700:90:0:2a8f:ccfa:caad:78b6}. I should have given a real example instead of trying to be "cute".

Caught that Just ran a test with the mtr prompt, does it save the results some where as I only get
image

image