Won’t help.
The other things you mention might help. I’m no DOCSIS expert.
To definitely confirm this jitter is because of your internet connection, could you switch your modem to the router mode and connect to it directly from your gaming device? Also could you try with another device to ensure this isn’t some hardware problem with your main gaming rig? Lower your graphic settings to the minimum to exclude any bottlenecks.
1 Like
Hi guys,
I am pretty new to OpenWRT but have found the default CAKE SQM to be really successful at reducing bufferbloat on my 4g connection. My download speed typically fluctuates between 20-40mb.
I currently have a batch file (thanks chatGPT) on my desktop that sets the CAKE download limit at 20mb - I run this when I want to minimise my latency/bufferbloat for online gaming. When finished I have another script that sets it to 40mb for the rest of the time when higher latency is acceptable.
However given the fluctuating download speed, I think what would be even better would be to have a button that sets a target latency under load rather than setting the maximum download speed, and the router automatically reduces the download limit until this target latency under load is hit. Eg I would have a button that sets a "+5ms" latency under load for my online gaming, and a "+70ms" or whatever for the rest of the time. This would better achieve my goals and especially when the connection moves outside its typical range.
Does this make sense and would this be achievable using autorate? From the docs it sounds like autorate can potentially set a target latency although not a target "latency under load" which I think is what causes jitter/spikes on online gaming if someone else is using the connection.
I did try to install autorate but halfway through the installation instructions ran out of space on my basic router. I would consider upgrading my router if something like this could be possible. Many thanks for any help!
1 Like
From the documentation of cake-autorate:
Setting the download bandwidth to 20 MBit/s when gaming and 40 Mbit/s otherwise is a rather crude and I would think using cake-autorate would be more appropriate because the usable bandwidth always fluctuates with 4G connections. I should know as I use 4G myself.
cake-autorate will always punish cake bandwidth (regardless of load) if the latencies across the reflectors exceed the threshold (albeit this is a simplification), and this will only bite when there is an active load since otherwise there will be no throttling effect.
At the moment cake-autorate would simply have to be restarted between the threshold changes, i.e. you'd alter the config file(s) then restart cake-autorate to load them in.
Facilitating dynamic adjustment of the cake-autorate thresholds has not yet been implemented, but perhaps this would be useful for your use case. We should think about whether any other values might warrant such dynamic adjustment. One option might be to offer an option to reload the entire config on a running instance. Regardless, care would need to be taken to ensure no conflict arises when dynamically changing parameters in this way and there might be required some extra code to account for that.
Dynamic adjustment of parameters based on active connections has been discussed before in this or one of the historical threads. We could, for example, monitor for the presence of certain DSCPs in the active connections and then alter the parameters to reflect something like a detected 'gaming mode'.
2 Likes
Hi and many thanks for your reply. Personally I am happy to continue manually trigger the "gaming mode" (latency priority) myself via my desktop shortcut - this is probably "safer" in that there would be no chance of delays in detection/scaling which in a worst case could boot me out of a server due to lag spikes / packet loss .
If I move to autorate which threshold would I need to reduce in my gaming mode, is it xl_owd_delta_delay_thr_ms?
I am going to try installing openwrt and autorate on the other router I have which is https://openwrt.org/toh/tp-link/archer_c6_v3 , it apparently has twice as much flash memory as the one I tried on (C6 V2) so hopefully will have more luck!
Thanks again
1 Like
I also wondered if you or anyone else may be able to share what link layer adaptation settings you use with 4G? The docs suggested googling for ISP but I haven't been able to find anything. I am with talkmobile (vodafone) on band 8 in the UK. Many thanks!
But doesn’t your usable bufferbloat-free bandwidth sometimes dip below 20 Mbit/s? If that happens then the bufferbloat associated with rates exceeding that level will go unchecked.
You could set 20 Mbit/s as the maximum cake-autorate bandwidth, 5 Mbit/s as the minimum and 10 Mbit/s as the base.
The response in cake-autorate is rapid (20 Hz by default I think, and can be made even more rapid but it quickly gets to the point where there is no gain and a lot of unnecessary load on CPU).
These:
# average owd delta threshold in ms up to which the maximum adjust_up_load_high is applied to the shaper rate adjustment
# for average owd deltas between avg_owd_delta_max_adjust_up_thr_ms and owd_delta_thr_ms, the adjustment is scaled linearly
# from max_adjust_up_load_high (at avg_owd_delta_max_adjust_up_thr_ms) to min_adjust_up_load_high (at owd_delta_thr_ms)
dl_avg_owd_delta_max_adjust_up_thr_ms=10.0 # (milliseconds)
ul_avg_owd_delta_max_adjust_up_thr_ms=10.0 # (milliseconds)
# owd delta threshold in ms is the extent of OWD increase to classify as a delay
# these are automatically adjusted based on maximum on the wire packet size
# (adjustment significant at sub 12Mbit/s rates, else negligible)
dl_owd_delta_delay_thr_ms=30.0 # (milliseconds)
ul_owd_delta_delay_thr_ms=30.0 # (milliseconds)
# average owd delta threshold in ms beyond which the maximum adjust_down_bufferbloat is applied to the shaper rate adjustment
# for average owd deltas between owd_delta_thr_ms and avg_owd_delta_max_adjust_up_thr_ms, the adjustment is scaled linearly
# from min_adjust_down_bufferbloat (at owd_delta_thr_ms) to min_adjust_up_load_high (at avg_owd_delta_max_adjust_down_thr_ms)
dl_avg_owd_delta_max_adjust_down_thr_ms=60.0 # (milliseconds)
ul_avg_owd_delta_max_adjust_down_thr_ms=60.0 # (milliseconds)
2 Likes
Hi! Thanks for this amazing piece of tech. It's been great improvement for my connection!
I've been using settings I found on this thread:
min_dl_shaper_rate_kbps=10000 # minimum bandwidth for download (Kbit/s)
base_dl_shaper_rate_kbps=100000 # steady state bandwidth for download (Kbit/s)
max_dl_shaper_rate_kbps=1000000 # maximum bandwidth for download (Kbit/s)
min_ul_shaper_rate_kbps=15000 # minimum bandwidth for upload (Kbit/s)
base_ul_shaper_rate_kbps=20000 # steady state bandwidth for upload (KBit/s)
max_ul_shaper_rate_kbps=45000 # maximum bandwidth for upload (Kbit/s)
connection_active_thr_kbps=2000 # threshold in Kbit/s below which dl/ul is considered idle
# Logging toggles for various stats
output_processing_stats=0 # enable (1) or disable (0) output monitoring lines showing processing stats
output_load_stats=0 # enable (1) or disable (0) output monitoring lines showing achieved loads
output_reflector_stats=0 # enable (1) or disable (0) output monitoring lines showing reflector stats
output_summary_stats=0 # enable (1) or disable (0) output monitoring lines showing summary stats
output_cpu_stats=0 # enable (1) or disable (0) output monitoring lines showing CPU usage percentages
output_cpu_raw_stats=0 # enable (1) or disable (0) output monitoring lines showing raw CPU usage lines
# *** OVERRIDES ***
### See defaults.sh for additional configuration options
### that can be set in this configuration file to override the defaults.
### Place any such overrides below this line.
dl_avg_owd_delta_max_adjust_up_thr_ms=10.0
ul_avg_owd_delta_max_adjust_up_thr_ms=10.0
dl_owd_delta_delay_thr_ms=20.0
ul_owd_delta_delay_thr_ms=20.0
dl_avg_owd_delta_max_adjust_down_thr_ms=40.0
ul_avg_owd_delta_max_adjust_down_thr_ms=40.0
My connection is 5G NSA 1000mbits theoretical max download and 50mbits upload. Router running cake-autorate is NanoPi R6S. I recorded small set of speedtests with output_processing_stats, output_load_stats enabled as instructed in ANALYSIS page. The span is quite short, only few minutes and few speedtests.
https://drive.google.com/file/d/1HM6VnFw5IU6YSB1TaZK3jsAP2BY5C8Zv/view?usp=sharing
Would be really appreciated to have some tips to optimize this. Let me know if its too small sample size.
Thanks in advance!
1 Like
Hey there. Thanks for the feedback!
I think we need more data relating to a sustained download - perhaps an .ISO from a well-connected host?
Zooming in on the small portion where there is some loading:
there are some pretty hefty latency excursions, but each are reacted to with rate punishments. So generally speaking it seems that cake-autorate is doing the right thing within the parameters set.
Looks like you could tighten up the delay thresholds if you wanted to, but at the expense of bandwidth:
The purple line there represents the 20ms delay threshold, but it looks to me like this could be dropped to say 15ms:
You could try:
dl_avg_owd_delta_max_adjust_up_thr_ms=7.5
ul_avg_owd_delta_max_adjust_up_thr_ms=7.5
dl_owd_delta_delay_thr_ms=15.0
ul_owd_delta_delay_thr_ms=15.0
dl_avg_owd_delta_max_adjust_down_thr_ms=30.0
ul_avg_owd_delta_max_adjust_down_thr_ms=30.0
for comparison.
But ultimately what works best for you will hinge on your preferences in terms of bandwidth and latency. Probably in your position I'd try to reduce latency a bit further since you have so much bandwidth to spare.
What do:
https://www.waveform.com/tools/bufferbloat
report about your connection with and without cake-autorate?
Thanks for very indepth reply!
I applied new proposed settings and did tests requested 
-
I tried to do more sustained download now with 1000GB test file on my homelab server, let it download few gigabytes before cancelling. Here are new logs:
https://drive.google.com/file/d/1B8qPfYrDX5qRhGsPgzTYrj4lm6s3iSZC/view?usp=sharing
Not sure if any adjustments are needed as results are really really nice!
Without:
https://www.waveform.com/tools/bufferbloat?test-id=3adf1fc0-84c1-40b0-962e-86c24499ef87
With:
https://www.waveform.com/tools/bufferbloat?test-id=6879fa6e-915f-4dad-96d5-e9c0bd2a893c
Small note on these tests that download speed should be ignored as I started download on my server during loaded download test phase. This was just to create even more load.
Comparing old settings to new proposed settings, speed seems to be about the same 400-700 mbits depending on time of the day etc.
I got couple of "related" questions that popped out:
-When enabling the regular SQM does link layer adaptation has any effect? I chose settings from this thread and been sticking to those.
Ethernet with overhead
Per Packet Overhead 44
Maximum packet size 2047
Rate table size 128
Minimum packet size (MPU) 88
-As it seems to be that usable "stable" bandwidth is 400-700 depending on time of the day, I think it would make sense for me to switch my plan to 600mbits max download. I suppose this wouldn't have effect on bufferbloat as speed will be just capped on ISP side if that limit is reached?
Big thanks again for all the efforts you've done to this project, both on the actual software and here helping out people. You should really consider accepting some donations for your efforts!
1 Like
Static SQM configurations are mildly sensitive to the overhead settings (that is, if the settings are too small, bufferbloat will tend to crop up under specific usage scenarios), cake-autorate however is more tolerant, as configuring the overhead too small will result in higher latency, which in turn will make cake-autprate react... in an ideal world we would configure both static sqm and cake autorate with the veridical overhead, but the world being as it is all I can say is cake-autprate is more forgiving...
Hard to predict, as that depends on how the ISP actually configures this on their end, but my gut feeling is, bufferbloat is not magically getting better by booking a lower speed tier from the ISP...
Thanks for quick reply!
I'll let settings be as they are now then. Especially now when I'm testing new settings better not to introduce new variables...
I was thinking to lower the speed tier just because its a bit cheaper and even without SQM Im not getting often above 700mbits 
I suppose I can just give it a shot as changing speed tiers is quite simple thru their online service.
1 Like
Ah, downgrading the access rate fr economic reasons sounds reasonable to me (not that my opinion matters in how you spend your money 
). We user a 100/40 Mbps DSL link as a family of five with (static SQM) and I so far have not regretted it (I can get 250/40 DSL or 1000/50 cable, but 100/40 is plenty and cheaper than the alternatives). That dsl link is really really boring in its reliability, so I never really got around to run cake-autorate, as a simple static sqm configuration already works well (so all cake-autorate could do is not get into the way *).
*) Not fully true, the data logging cake-autorate does would still be nice to run continuously, sort of like a flight recorder, if something odd happens to check post hoc whether bufferbloat might have been involved.
You really got some nice connections available, slightly jealous
My options are either 5G or VDSL2. VDSL2 line is 100/10 and it has interleaving(Not sure if this was the correct term), so base latency is quite bad. On top of that its as expensive as "uncapped" 1000mbps 5G.
Luckily the 5G basestation is really close, I havent even bothered with outdoor antenna yet... Only issue I had early on was that ISP had powersaving during the night and they shut off the cell, gladly they removed the powersaving just by me calling to them
Data logging is in deed always fun, I was planning to export cake-autorate logs to Prometheus as per guide. Not that I understand much from them, but they do look nice!
Sorry for a bit off topic talk!
Where are you based? I’m a bit jealous of your amazing 5G connection.
We haven't seen cake-autorate operate at this level before. Exciting stuff.
@moeller0 any thoughts on the oscillation above? I think this looks pretty good.
@moeller0 do you suppose with @Duppo's lower-latency 5G connection and powerful NanoPi R6S that it would be worth trying a higher refresh interval than the default 20 Hz?
@Duppo do you see better or worse results with doubling the reflector frequency?
no_pingers=6 # number of pingers to maintain
reflector_ping_interval_s=0.15 # (seconds, e.g. 0.2s or 2s)
Please also enable the CPU log lines so we can verify none of your cores are getting saturated.
Thanks, glad to hear that my connection is really good I'm located in Finland. A bit outside of major city. 2 largest providers have expanded their 5G networks really rapidly so coverage is really good. I'm lucky to have basestation on top of my apartment building so I suppose that helps too.
Only thing I wish I could test out 5G SA mode, but basestation only has N78 band, while neighbours stations do have both N78 and N28.
I'll do few tests with faster frequency and report back. Talking about that I do think I added few reflectors, but kept the default ones as well. (Ones added were my ISP DNS servers, but ill double check).
Timecourse:
Raw CDFs:
Delta CDFs:
Samples per reflector:
ReflectorID: 156.154.70.3; N: 29
ReflectorID: 156.154.71.1; N: 29
ReflectorID: 208.67.220.123; N: 26
ReflectorID: 208.67.220.2; N: 31
ReflectorID: 208.67.222.123; N: 28
ReflectorID: 208.67.222.2; N: 31
ReflectorID: 94.140.15.15; N: 2
DL: maximum 95.000%-ile delta delay over all 7 reflectors: 9.545 ms.
DL: maximum 99.000%-ile delta delay over all 7 reflectors: 15.485 ms.
DL: maximum 99.500%-ile delta delay over all 7 reflectors: 15.485 ms.
DL: maximum 99.900%-ile delta delay over all 7 reflectors: 15.485 ms.
DL: maximum 99.950%-ile delta delay over all 7 reflectors: 15.485 ms.
DL: maximum 99.990%-ile delta delay over all 7 reflectors: 15.485 ms.
DL: maximum 99.999%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 95.000%-ile delta delay over all 7 reflectors: 9.545 ms.
UL: maximum 99.000%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 99.500%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 99.900%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 99.950%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 99.990%-ile delta delay over all 7 reflectors: 15.485 ms.
UL: maximum 99.999%-ile delta delay over all 7 reflectors: 15.485 ms.
Files: https://easyupload.io/m/376vnj
Anything stand out to you @moeller0?
1 Like
Heres log with all logging on and proposed reflector frequency changes:
https://easyupload.io/qu48m6
(Changed to easyupload too as forums now doesnt like my gdrive link :D)
I tried to monitor ping during load, but couldn't tell difference thats outside of margin of error. Both tests gave between +9-12ms during download load. I'll do some more tests tomorrow manually pinging with higher frequency
During last test I ran htop on R6S and all cores seemed fine, but logs will tell hows its doing.
Thanks!
Hey, I did some better(?) testing now with and without reflector frequency changes.
With default settings:
https://i.imgur.com/wjr5ruE.png
With reflector frequency changes:
https://i.imgur.com/xE5O1CO.png
I loaded connection with download on my server and used gping with 0.2s interval and 180s buffer. If you know better way / tools to test it, let me know
Ignoring the few spikes happening here and there it seems that with reflector changes there is slight advantage. I did run test few times and results were similar.
1 Like
Agreed.
Sure, why not... would be interesting to learn whether that helps or not.
Nothing I notice...
I love gping, but I note that these two plots have quite different scaling, making it hard to see at a glance which is better...
2 Likes