CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

moeller0 · December 3, 2023, 10:20am

Hmmm, so autorate can not do this, its control loop operates in the single to low double digit millisecond period range... And even cake's per-flow control needs to see queuing delay >= 5 for at least 100ms before it engages... My gut feeling is what you see is caused by cake's flow queueing scheduler and not the AQM, that scheduler does a much better job in interleaving packets of different flows on the egress, so that no flow is delayed for too long, just because another flow just dumped a bunch of packets in one burst into a shared queue. The scheduler (if combined with a traffic shaper that keeps the bottleneck queue under control of that scheduler) does a terrific job of sharing a link between different traffic aggregates (and autorate does a great job of keeping the scheduler in control of the queue on variable rate links).

I admittedly mostly see reports of games who are unhappy with what they see on their links, but I am not 100% sure modern on-line games network stacks are as robust and reliable as one would expect, given the multi decade history of quick reaction-time gated on-line games...

Yes, @Lynx did a great job here! If you have time, maybe consider writing a quick how-to for to setting it up, the sort of how-to you would like ot have at hand when you started?

Clap · December 3, 2023, 10:42am

Fortunately the Installation is very simple to understand but limited exposure to using SSH&CMDs was the most difficult part towards understanding how to set it up, I wouldn't know how to make it easier to understand

The Auto-rate seems to do great during my.. abnormal experience, its capable of keeping the rates as best as it could at least... given the.. most abnormal results I've gotten in the last 2 weeks.

Would you happen to know what would cause the.. Latency/network to have such a weird State, asfaik Its not exactly ~peak hours~ or whatever excuse the isp's here will try to get away with. it's essentially double the latency, 15-30x the jitter and high packet loss

moeller0 · December 3, 2023, 12:20pm

What differed between these twi tests? I would guess these used different remote end points (both seem to use cloudflare infrastructure, but waveform uses their CDN while their own test apparently uses some dedicated locations, you cut of the server location map (likely for privacy reasons) that might allow to make some estimates). Aren't you behind MAP-T, so with no direct IPv4? Is there a chance of using IPv6 directly?

Clap · December 3, 2023, 1:36pm

The difference as far as I can tell is 10 mins, both routed towards the same server im forced to bounce off that route due to..the. routing usually it will be osaka->tokyo (with NTT west even though theres like 2 other locations closer than both of those)

Though on a normal day, running those tests one after the other would normally have 2ms-5ms difference (and sometimes the ipv4 works and sometimes the ipv6.. idk the logic behind it, but if I run the tes.....

it would appear the v6 address likes to route through tokyo and the v4 through osaka~

(I think my ipv6 addr is "fake/internally routed" through the gateway as i cant reach any sites that use ipv6 without the map-e protocol~ (which is just ipv4 encapsulated by ipv6 asfaik i can ping with both v4 and v6 but with neither without the map-e protocol to make things worse..... the v4 and v6 are permanently assigned to my... I wanna cry T_T)

--- using openwrt over the isp recommended routers.... I tend to have less problems with openwrt if you can believe that I clearly have no idea what I'm doing still....

moeller0 · December 3, 2023, 2:20pm

I am thoroughly confused now, I would have expected IPv6 to be more consistent given that no translation appears necessary...

Clap · December 3, 2023, 5:16pm

So my network under stable hours (demon hour)
The script is doing well managing the bloat regardless~ even during the peek hell hours It was able to stay within 60~70ms (which is... double my normal ping)

moeller0 · December 4, 2023, 7:13am

That looks considerably nicer, I wonder, can try to get the IP addresses of the remote endpoint of the cloudflare speedtest and run a mtr -ezb6w -c 100 ${THE_SPEEDTEST_SERVERS.IPv6.ADDRESS} both during demon hour as well as when things are bad. I wonder whether there might be differential routing in play as well?

Clap · December 4, 2023, 2:28pm

Just curious, would this~ command work, this is the ip i'm getting from cloudflare through the STUN protocol during the speed test (as thats the only one that showed up).

Though I'm not getting anything following after inputting that command while doing the speed test/before

moeller0 · December 4, 2023, 2:30pm

Sorry, leave the ${} part out.... that is how in shell you decorate a variable, so ${THE_SPEEDTEST_SERVERS.IPv6.ADDRESS} was shorthand for 2606:4700:90:0:2a8f:ccfa:caad:78b6 and not for a literal ${2606:4700:90:0:2a8f:ccfa:caad:78b6}. I should have given a real example instead of trying to be "cute".

Clap · December 4, 2023, 2:35pm

Caught that Just ran a test with the mtr prompt, does it save the results some where as I only get

moeller0 · December 4, 2023, 2:36pm

With -c 100 it will take north of 100 seconds to return something (if you leave out the -c 100 it will run interactively showing intermediate results almost immediately).

Clap · December 4, 2023, 2:44pm

This is during a stable~ connection, the unstable connections are completely variable, or whenever the ISP decides to give you the middle finger

HOST: Fangler                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev 
1. ASN<- cloud   reduced caboose)                                            0.0%    10    2.5   2.3   1.9   3.1   0.3  
2. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0  
3. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0  
4. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0  
5. AS4713   2001:380:c510:b::1 (2001:380:c510:b::1)                         10.0%    10   21.3  21.1  18.8  29.8   3.5  
6. AS4713   2001:380:c500:25::1 (2001:380:c500:25::1)                        0.0%    10   19.3  19.7  18.8  22.5   1.2  
7. AS4713   2001:380:c200:7::1 (2001:380:c200:7::1)                          0.0%    10   27.6  27.4  27.1  27.6   0.2       
[MPLS: Lbl 3954 TC 0 S u TTL 1]
[MPLS: Lbl 2 TC 0 S u TTL 1]
8. AS4713   2001:380:c200:8::2 (2001:380:c200:8::2)                          0.0%    10   28.3  31.8  27.3  52.0   7.5  
9. AS2914   ae-12.r02.osakjp02.jp.bb.gin.ntt.net (2001:218:2000:5000::929)   0.0%    10   27.4  28.0  27.1  32.3   1.5 
10. AS2914   2001:218:2000:5000::162 (2001:218:2000:5000::162)                0.0%    10   27.6  31.5  27.6  43.3   6.0 
11. AS13335  2400:cb00:370:3:: (2400:cb00:370:3::)                            0.0%    10   54.5  31.1  27.5  54.5   8.3 
12. AS13335  2606:4700::6810:4426 (2606:4700::6810:4426)                      0.0%    10   27.3  29.1  27.3  33.5   1.9

I uhh... I'll try to fix the above later.
This is what it spat out~

moeller0 · December 4, 2023, 2:55pm

Great, this is with reasonably high throughput and decent latency, will be interesting to compare this with data from peak-hour when the latency doubles...

Clap · December 6, 2023, 4:10pm

mmmm It may take a few days for the similar situation, I honestly don't know what causes it but eventually the network will just suffer for no apparent/reasonable reason... for a couple of days..

coricycle · December 11, 2023, 12:30am

Hi, I have difficulties to set up cake-autorate/SQM adequately. I deactivated cake-autorate and SQM, went to waveform.com for a RTT test and also used gping, which gave me roughly the same data:

I set the thresholds for cake-autorate as following;

min_dl_shaper_rate_kbps=5000
base_dl_shaper_rate_kbps=75000
max_dl_shaper_rate_kbps=88000

After launching cake-autorate and activate SQM (with the following settings)....

...It seems it does not really much to reduce bufferbloat

moeller0 · December 11, 2023, 5:52am

Please post your full config file and a link to the autorate log preferably containing a speedtest and a link/screenshot of that speedtest's results...

Clap · December 15, 2023, 7:19am

Ensure the openwrt image is EXT4 as the squashFS doesnt seem compatible (for me at least~)

Also make sure the cake-autorate service is enabled, and you type start~

All these below will be involved in the SSH method through either WindowsPowerShell, or in a linux equivalent Shell~ (ex. ssh root@192.168.1.1 <- default openwrt ssh)

service <- this alone will show all services, try to see if cake-autorate is listed and enabled

service cake-autorate enable
(there is enabled and enable, enable will enable it to load on start up)

service cake-autorate start
(will start the actual service on startup)

service cake-autorate info
(just to see if the service has properly loaded, asfaik it will start on squash but unable to actually work~)

@moeller0 i'm confident I'll be able to get the mtr results by tomorrow/today, I'm 100% positive that its more apparent on the weekends.... Though I am curious why my latency literally doubled~ I don't Think my ISP is incompetent.. just trying too many new protocols with new methods that... isn't really efficient but... I want to blame the cellular network lacking ipv6 as the primary connection method for a majority of some weird wonky bonanza issues

Clap · December 15, 2023, 12:28pm

                                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS-   2400:-(2400:-)                                                   0.0%    10    2.0   2.7   2.0   3.6   0.5
  2. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  3. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  4. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  5. AS4713   2001:380:c510:16::1 (2001:380:c510:16::1)                        0.0%    10   19.0  21.7  18.3  25.1   2.6
  6. AS4713   2001:380:c500:26::1 (2001:380:c500:26::1)                        0.0%    10   30.8  24.3  19.8  30.8   4.4
  7. AS4713   2001:380:c300:6::1 (2001:380:c300:6::1)                          0.0%    10   33.7  37.8  30.6  64.1  10.0
       [MPLS: Lbl 4847 TC 0 S u TTL 1]
       [MPLS: Lbl 2 TC 0 S u TTL 1]
  8. AS4713   2001:380:c300:7::2 (2001:380:c300:7::2)                          0.0%    10   33.8  47.2  33.7 126.0  28.8
  9. AS2914   ae-11.r02.osakjp02.jp.bb.gin.ntt.net (2001:218:2000:5000::931)   0.0%    10   42.5  36.9  33.4  48.2   5.0
 10. AS2914   2001:218:2000:5000::162 (2001:218:2000:5000::162)                0.0%    10   49.2  51.3  34.3  96.7  22.5
 11. AS13335  2400:cb00:51:3:: (2400:cb00:51:3::)                              0.0%    10   45.3  47.6  33.8  73.4  14.0
 12. AS13335  2606:4700::6810:4326 (2606:4700::6810:4326)                      0.0%    10   39.2  36.5  33.3  47.6   4.4

along with the speed test, idk whats doubling my ping, hopefully this makes it a bit more clear

Along with a Waveform test~

HOST: Fangler                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS-   2400:-                                                 )           80.0%    10    2.8   2.7   2.7   2.8   0.1
  2. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  3. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  4. AS???    ???                                                             100.0    10    0.0   0.0   0.0   0.0   0.0
  5. AS4713   2001:380:c510:16::1 (2001:380:c510:16::1)                        0.0%    10   21.3  24.3  19.4  32.4   4.4
  6. AS4713   2001:380:c500:26::1 (2001:380:c500:26::1)                        0.0%    10   23.2  25.4  19.6  38.7   5.8
  7. AS4713   2001:380:c300:6::1 (2001:380:c300:6::1)                          0.0%    10   31.7  38.4  31.4  57.7   9.2
       [MPLS: Lbl 4847 TC 0 S u TTL 1]
       [MPLS: Lbl 2 TC 0 S u TTL 1]
  8. AS4713   2001:380:c300:7::2 (2001:380:c300:7::2)                          0.0%    10   43.2  40.6  33.4  49.0   5.1
  9. AS2914   ae-11.r02.osakjp02.jp.bb.gin.ntt.net (2001:218:2000:5000::931)   0.0%    10   39.4  39.0  33.8  44.6   3.8
 10. AS2914   2001:218:2000:5000::162 (2001:218:2000:5000::162)                0.0%    10   49.8  42.4  34.3  49.8   5.1
 11. AS13335  2400:cb00:51:3:: (2400:cb00:51:3::)                             10.0%    10   45.9  50.9  33.7  73.2  10.5
 12. AS13335  2606:4700::6810:4326 (2606:4700::6810:4326)                      0.0%    10   48.2  39.9  33.3  49.3   5.6

moeller0 · December 15, 2023, 2:44pm

AS13335: NTT Communications Corporation (OCN)
AS2914: NTT Global IP Network
AS13335: Cloudflare

~20ms to the first reliably responding hop, and the addition al ~10ms, likely for the MPLS tunnel, all within your ISPs own network (you are NTT customer, no?) More or less similar to your earlier MTR results.

I am probably a bit daft today, but where do you see the doubling of the ping? (I believe your assessment, I just can not see where in the data, and having stared long enough, I am sure today is a day for asking, not for staring even longer )

Clap · December 15, 2023, 5:48pm

Latency will on a normal day should be from 28ms-30ms ingames, however when my internet does similar things to the above post my ping will be doubled well either way.. 45ms-55ms isn't normal (latency is doubled on ingress, as the mean is 88ms and thats the ping i'd get in games)

(60ms~80ms not jittering up to 60ms-80ms and going back down, but staying at 60ms-80ms)

I'm not sure if this is due to the loss% tbh but its not... a pleasant experience especially since it lasts 3-4 hours and this is for everyone in the country outside of tokyo/osaka asfaik.

These "issues" are entirely new for me as.... when I was in the states the latency was stable all the time and only improved with cake-autorate. (i had 80ms~bit far from the.. mainland then with cake it was down to 60ms on a 300mbps using a docsis 3.1 isp provided modem on the line )

However using cake-autorate in japan can improve the jitter ~
All i've noticed is that.... sadly the internet feels completely unstable with and without cake... and I can't figure out why.. as my isp doesn't really want to communicate/unable to?