CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

moeller0 · July 21, 2022, 10:44am

Pet peeve of mine, but that speedtest in its default configuration will report inflated values...
See: https://github.com/librespeed/speedtest/blob/master/speedtest_worker.js:
overheadCompensationFactor: 1.06, //can be changed to compensatie for transport overhead. (see doc.md for some other values)
I think this is the consequence of the author empirically trying to make sure his speed test reports the same figure as some windows system tool. However that means the test results are neither here nor there, they are neither a veridical measurement of the actual goodput (the payload rate) not of the gross rate including all overhead (as different link technologies carry different overhead, case in point ATM/AAL5 will have a link-layer overhead of a whopping 9% in addition to all other protocol overhead).

I would probably opt for one of the other speedtests that try to be less clever and boringly report just the achieved payload-rate (which in itself is already harder than one would expect, if multiple parallel data flows are used).

MangoMan · July 21, 2022, 10:46am

The first sentence I understand in your post (the technical explanations are way over my head )

Any particular suggestions? Should the waveform bufferbloat test be sufficient?

moeller0 · July 21, 2022, 10:48am

autorate will relieve the pressure of getting the overhead compensation right somewhat, if the overhead is wrong (too small) the consequence is increased likelihood of encountering increased delays, but autorate is directly controlling via measureing the delay, so is more tolerant against getting the overhead wrong than a static sqm configuration. However, I still would aim at getting this right if just out of principle ;).
The wiki does not say anything about overhead on LTE links, because we have not found reliable information that allows us to figure that out precisely, but it seems that the applicable overhead is generally much smaller than 44 bytes so your configuration should be fine.

moeller0 · July 21, 2022, 10:54am

Ah sorry, that test by default will report values that are 6%-age points higher than the actually measured values, which I consider sub-optimal (which I also informed the tests developer about).

The waveform test is not bad, but even Ookla's speedtest.net got much better recently (the on-line test now will show three latency values, one for idle, one for download and one for upload, not 100% clear whether that is the maximum RTT encountered during these phases or the median or mean, but it is better than the old idle only value; however these values are not yet reported/aggregated in the "results" overview page which collects multiple test results).

MangoMan · July 21, 2022, 11:29am

Just did it but the script actually stopped logging for most of the test and showed this error:

./CAKE-autorate.sh: line 584: ((: (1658402778622798 - 091404: value too great for base (error token is "091404")

and now I can't start it again:

Error: /tmp/CAKE-autorate already exists. Is another instance running? Exiting script.

while I'm also not able anymore to "stop" the service (which worked before):

# service cake-autorate stop
Command failed: Not found

Looks like I messed something up

Lynx · July 21, 2022, 11:43am

Weird timestamp from ping response! Otherwise this is my bad - it's that leading 0 and so I need to specify base 10 to deal with this weird eventuality. I will look to see if it possible to set this globally because we've seen this issue come up elsewhere. I'll fix this in any case.

For now easy solution is either reboot router or kill all instances of bash/ping and rm /tmp/CAKE-autorate.

MangoMan · July 21, 2022, 1:43pm

I did this

So newest test:

and the new paste for it

Also wondering about the upper limits now. Just when I tested I spotted the download (without sqm/cake) to reach 2XMbps for a second and that reminded me actually that I do automatic speed tests on a regular interval

I think it's clearly visible when I activated the

And yes, with longer history it's clear that our connections actually "performs" better (in terms of the max dl rate) than I remembered/expected:

Lynx · July 21, 2022, 2:24pm

@MangoMan this looks pretty good to me.

Do you have visibility of your LTE stats? Curious that download capacity seems rather poorer than upload capacity. Is your cell tower heavily congested?

MangoMan · July 21, 2022, 3:18pm

Just watched the stats for some minutes and the range at the moment (very dry so best signal of the year) is something like that:

Signal Strength:	-71, -72, -74, -75, -76 dBm
RSRP:	-97 dBm
SINR:	0, 1, 2, 3, 5, 6, 7, 9 dB
RSRQ:	-7, -8, -9, -10, -11 dB

Yea, that's interesting but was always the case. My theory is that they put more "effort" in shaping the downstream (because it is what like 99% of the users mostly use) and not to much carrying about the upstream (maybe don't shape it at all)

Lynx · July 23, 2022, 7:43am

Hey have you kept the autorate script running? If so could you provide the new graphs from your automatic speed tests / latency measurement?

How has latency / reactiveness been in general? Internet feel snappy?

Perhaps there is still room for improvement in the parameters.

Have you considered using an outdoor antenna? If not have you really tried optimizing position of the LTE modem? Mine works best face down on its side underneath a window between wall and floor. My neighbour's works best in an unused fireplace against side wall. There is typically a lot of room for improvement by adjusting the position of the modem. Tenacity pays off.

MangoMan · July 23, 2022, 9:20am

yes, indeed. With the last settings mentioned

Here we go:

It is actually a outdoor antenna (with integrated modem).

It is already on the highest spot available installed (pver the roof)

Well, I don't need start inside the house as there is hardly any reception

MangoMan · July 23, 2022, 6:18pm

Like one can imagen with this tight budget available it's never a pleasure but I have the feeling the responsive of web surfing improved a bit (less waiting "ages" for a site to appear in the browser after sending a URL). On the other hand I'm not quite sure how about the situation with video streaming and if it tends to pauses more for buffering. Guess some more watching necessary here

richb-hanover-priv · July 25, 2022, 6:47pm

New Idea - I'm not sure where to put it: I wrote up a challenge on Github to see if we could "misuse" CAKE-autorate to learn more about the underlying links. Here are my thoughts...

github.com/lynxthecat/CAKE-autorate

Instrumenting CAKE-autorate to learn more about underlying links

opened 08:07PM - 24 Jul 22 UTC

richb-hanover

Here's an ill-formed thought to throw out for the collective wisdom to consider:… At this point, CAKE-autorate appears to be doing a great job of maximizing transfer rates while minimizing latency. It does this by adjusting the CAKE parameters to provide good service to its human masters :-) This is good. Speculation: What would it mean to use CAKE-autorate "backwards" - to characterize the speed of the underlying links. I've heard a bunch of assertions about various link technologies: - I believe a fiber optic connection runs very stably at/near the ISP's rated speed - I suspect that DSL/VDSL also similar stable data rates - It is widely believed that Cable Modem speeds exhibit 'sag' in the high-traffic evening hours - Cellphone modems offer wildly differing data rates over time. Is there any pattern to them? - Starlink - there's a reason to think it might have a 15-minute periodicity in its speeds. But I don't know of an existing consistent/reliable way to measure/confirm any of the above assertions. My Challenge: - CAKE-autorate is continually adjusting the CAKE parameters. Would it be useful to examine this information over a long term? - If so, how could we sample the readings in a useful, sufficiently granular way to display "what's going on" with the underlying link? - How could we make sense of the data without being drowned in multiple data samples per second? Possible Use Case: Add a "CAKE Link-Speed" tab to the Realtime Graphs page that showed [ indication of the link speeds | variation in the link speeds ] over time. Thanks!

Enjoy!

MangoMan · July 26, 2022, 8:42am

I have:

Today my router went very laggy (wifi clients don't connect properly) and internal pings are huge by times and resolving host name is troublesome.

The luci interface isn't available but I get this "site" instead:

Bad Gateway
The process did not produce any response

Logging into ssh worked but took very long. Getting top to output the load average:

CPU:  61% usr  36% sys   0% nic   0% idle   0% io   0% irq   1% sirq
Load average: 25.63 18.79 15.93 8/313 28884

I wonder if cake-autorate is to blame? Should it run that often?

11537     1 root     R     2168   2%  17% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
16429     1 root     R     2140   2%  17% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11527     1 root     R     2168   2%   7% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11519     1 root     R     2168   2%   6% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
16462 16429 root     S     2096   2%   4% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11525     1 root     R     2168   2%   2% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11521     1 root     S     2168   2%   2% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11517     1 root     S     2168   2%   2% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11509     1 root     S     2168   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10583     1 root     S     2176   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11523     1 root     R     2168   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
28827 28813 root     R     1288   1%   1% top
10951     1 root     S     2204   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10831     1 root     S     2196   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10513     1 root     S     2172   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10547     1 root     S     2172   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
 1414     1 root     S     4376   4%   1% /usr/sbin/hostapd -s -g /var/run/hostapd/global
10963     1 root     S     2208   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10839     1 root     S     2196   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11184     1 root     S     2168   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10933     1 root     S     2204   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10881     1 root     S     2200   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10877     1 root     S     2200   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10791     1 root     S     2192   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10695     1 root     S     2184   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11223     1 root     S     2168   2%   1% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
    7     2 root     RW       0   0%   1% [ksoftirqd/0]
10925     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10809     1 root     S     2196   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10737     1 root     S     2188   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10733     1 root     S     2188   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10681     1 root     S     2184   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10641     1 root     S     2180   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10561     1 root     S     2176   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
10563     1 root     R     2176   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11530     1 root     R     2168   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11246     1 root     S     2168   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
11231     1 root     S     2168   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh

I did run `services cake-autorate stop" now (I checked an the "services" was running).
It took like 3 minutes or more to finish and the web interface somewhat came available again. Still ssh and luci are the opposite of responsive.

I guess I need a hard restart now (probably then all logs will be lost to debug this )

Lynx · July 26, 2022, 8:59am

Hmm. Is there anything in the log file in /tmp? From top looks like you had many instances running simultaneously unless you have increased the number of pingers to well beyond the default (4).

The cpu stats are misleading if you have schedutil enabled since it clocks down the processor to the minimum needed to keep everything running.

I spent quite a lot of time tweaking the bash implementation to get the CPU down as far as possible, but bash is still bash. For my sub 100Mbit/s connection the CPU use on my RT3200 is sufficiently minor not to affect anything with default settings. the settings could be tweaked to reduce CPU usage of desired.

MangoMan · July 26, 2022, 9:09am

I didn't restart yet but it's impossible to reach it ...so I power cycled the device

I kept the defaults for everything (only set the dl/ul min/max..)

MangoMan · August 7, 2022, 8:24am

Does CAKE eat my RAM? Linux ate it!

My luci web interface is everything from responsive and while it now looks good again memory wise:

one minute ago I only had like 1 MiB available

Establishing a ssh connection takes a good a amount of seconds before I even asked for the password and even more seconds afterwards. Then I get only half of the greeting:



BusyBox v1.33.2 (2022-02-16 20:29:10 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.2, r16495-bf0c965af0
 -----------------------------------------------------

and like a good minute later just the command prompt which is everything apart from responisve

I just enabled fast transition on wifi - maybe this hogs on memory together with ?

Total available now:

Who ate my ram?

My cake-autorate options regarding output/debug are all

# *** OUTPUT OPTIONS ***

output_processing_stats=0 # enable (1) or disable (0) output monitoring lines showing processing stats
output_cake_changes=0     # enable (1) or disable (0) output monitoring lines showing cake bandwidth changes
debug=0			  # enable (1) or disable (0) out of debug lines

EDIT: just checking top looks like upnp is going crazy (CPU wise) actually:

15131  1086 root     R     1176   1%  79% lua /usr/libexec/rpcd/luci.upnp call get_status

but for the cake-autorate script - is it normal it runs many times in parallel?

15970     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15954     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15952     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15948     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15950     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15958     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15968     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15960     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15963     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15965     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15956     1 root     S     2208   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15920     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15930     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15918     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15922     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15924     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15926     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh
15911     1 root     S     2204   2%   0% {CAKE-autorate.s} /bin/bash /root/CAKE-autorate/CAKE-autorate.sh

EDIT:

Restarting the router did it bring back onto a normal level:

I also have less CAKE-autorate scripts running now (like 7 I see in top).

moeller0 · August 7, 2022, 9:01am

Each cake instance by default (in OpenWrt) will at most take 4 MB for its queues per instance, so a normal install with one upload and one download instance should not take much more than 2 * 4 = 8 MB.

1* script instances sounds like a lot, but how many latency reflectors do you have configured?

Mmmh, do you have cases where the script has to replace reflectors? @Lynx in case of a stuck reflector do we kill the old process that pinged that reflector for good?

Lynx · August 7, 2022, 9:27am

Yes we kill off bad ping process here:

github.com

lynxthecat/CAKE-autorate/blob/72a74345a24fa2056a72fdb2323f8765f88a82fb/CAKE-autorate.sh#L285


      
          				if(($no_reflectors>$no_pingers)); then
          
          
					# pingers always use reflectors[0]..[$no_pingers-1] as the initial set
          					# and the additional reflectors are spare reflectors should any from initial set go stale
          					# a bad reflector in the initial set is replaced with $reflectors[$no_pingers]
          					# $reflectors[$no_pingers] is then unset
          					# and the the bad reflector moved to the back of the queue (last element in $reflectors[])
          					# and finally the indices for $reflectors are updated to reflect the new order
          	
          					(($debug)) && echo "DEBUG: Replacing reflector: "${reflectors[$pinger]}" with "${reflectors[$no_pingers]}"."
          					kill ${pinger_pids[$pinger]} 2> /dev/null
          					bad_reflector=${reflectors[$pinger]}
          					# overwrite the bad reflector with the reflector that is next in the queue (the one after 0..$no_pingers-1)
          					reflectors[$pinger]=${reflectors[$no_pingers]}
          					# remove the new reflector from the list of additional reflectors beginning from $reflectors[$no_pingers]
          					unset reflectors[$no_pingers]
          					# bad reflector goes to the back of the queue
          					reflectors+=($bad_reflector)
          					# reset array indices
          					reflectors=(${reflectors[*]})
          					# set up the new pinger with the new reflector and retain pid

And with sleep will kill off all ping processes here:

github.com

lynxthecat/CAKE-autorate/blob/72a74345a24fa2056a72fdb2323f8765f88a82fb/CAKE-autorate.sh#L634


      
          	done</tmp/CAKE-autorate/ping_fifo
          
          
	(($debug)) && {(( ${PIPESTATUS[0]} == 142 )) && echo "DEBUG Warning: global ping response timeout. Enforcing minimum shaper rates." || echo "DEBUG Connection idle. Enforcing minimum shaper rates.";}
          	
          	# in any case, we broke out of processing loop, so conservatively set hard minimums and wait until there is a load increase again
          	dl_shaper_rate_kbps=$min_dl_shaper_rate_kbps
          	ul_shaper_rate_kbps=$min_ul_shaper_rate_kbps
          	set_shaper_rates
          
          
	# Kill off ping processes
          	kill $maintain_pingers_pid 2> /dev/null
          
          
	# reset idle timer
          	t_sustained_connection_idle_us=0
          
          
	# verify interfaces are up (e.g. following ping response timeout from interfaces going down)
          	verify_ifs_up
          
          
	# wait until load increases again
          	while true
          	do

@MangoMan are you using a service file? Although the script shouldn't allow multiple instances to run by exiting where its tmp folder already exists.

Can you also try setting sleep duration really short and threshold high to make it sleep often and verify that the processes are not increasing?

Or try setting bad reflectors so we get continual rotation and also monitor processes?

We should work out why you are seeing duplicate processes. Really hope you can help me work out what's going wrong.

But it's not affecting me at all.

@moeller0 can you see anything wrong with the kill processes?

On my system when the pipe is broken because the fifo file is removed the corresponding monitor background process is also killed. Could it be on some systems that doesn't happen for some reason?

github.com

lynxthecat/CAKE-autorate/blob/72a74345a24fa2056a72fdb2323f8765f88a82fb/CAKE-autorate.sh#L181


      
          		do
          			((timestamp_usecs_past_minute=${EPOCHREALTIME/./}%60000000))
          			if (( ($timestamp_usecs_past_minute > ($sss_time_us-$sss_compensation_pre_duration_us)) && ($timestamp_usecs_past_minute < ($sss_time_us+$sss_compensation_post_duration_us)) )); then
          				load_condition=$load_condition"_sss"
          				break
          			fi
          		done			
          	fi
          }
          
          
monitor_reflector_responses() 
          {
          	# ping reflector, maintain baseline and output deltas to a common fifo
          
          
	local pinger=$1
          	local rtt_baseline_us=$2
          
          
	while read -r  timestamp _ _ _ reflector seq_rtt
          	do
          		# If no match then skip onto the next one
          		[[ $seq_rtt =~ icmp_[s|r]eq=([0-9]+).*time=([0-9]+)\.?([0-9]+)?[[:space:]]ms ]] || continue

Pipe is here:

github.com

lynxthecat/CAKE-autorate/blob/72a74345a24fa2056a72fdb2323f8765f88a82fb/CAKE-autorate.sh#L210


      
          		rtt_delta_us=$(( $rtt_us-$rtt_baseline_us ))
          
          
		alpha=$(( (( $rtt_delta_us >=0 )) ? $alpha_baseline_increase : $alpha_baseline_decrease ))
          
          
		rtt_baseline_us=$(( ( (1000-$alpha)*$rtt_baseline_us+$alpha*$rtt_us )/1000 ))
          
          
		printf '%s %s %s %s %s %s\n' "$timestamp" "$reflector" "$seq" "$rtt_baseline_us" "$rtt_us" "$rtt_delta_us" > /tmp/CAKE-autorate/ping_fifo
          	
          		printf '%s' "${timestamp//[[\[\].]}" > /tmp/CAKE-autorate/reflector_${pinger}_last_timestamp_us
          
          
	done</tmp/CAKE-autorate/pinger_${pinger}_fifo
          }
          
          
kill_pingers()
          {
          	for (( pinger=0; pinger<$no_pingers; pinger++))
          	do
          		kill ${pinger_pids[$pinger]} 2> /dev/null
          		[[ -p /tmp/CAKE-autorate/pinger_${pinger}_fifo ]] && rm /tmp/CAKE-autorate/pinger_${pinger}_fifo
          	done
          	exit

MangoMan · August 7, 2022, 10:02am

Yes, went exactly like described in the githubs.

Any hint were to find that config? Probably in the conf file I expect?

It's weird - but it might be the second time now that it happens (router got already unresponsive in the past ones since I deployed )... but I guess it takes many days that it hogs on all the ram available..