I think the suggestion is to increase them separately but decrease together. That's A fairly reasonable approach. It may be problematic if there is a lot of asymmetry on the line. If we had a reasonably powerful language I might suggest to do something like a logistic regression on bufferbloat as a function of up and down bandwidth over the last 5 mins of data and set bandwidth to the value that results in say 1% chance.
But definitely not in shell script!!
On an RPi you could do all this in Julia quite nicely, you could run pings in a separate thread, read the bandwidth file every say 500msec and get a nice time series of bandwidth vs ping time. You could fit a logistic regression every minute and store that info permanently, from that you could Estimate a function of time of day, if there's A reasonable pattern. The time of day function could set a prior for a Bayesian estimate based on last minute of current actuals, then you could just immediately jump to the "optimal" BW...
Doing something like that in shell would require some serious pain meds
But aren't lines like the above just so much fun ! It's like brute force wins sometimes. With enough determination and grit.
Thank you for this suggestion. Makes sense to me. Only increase if there is bandwidth demand, and upon detection of bufferbloat we scale both down.
Seems intriguing, but I have always favoured simple hacks. I used to work in an organisation with two groups: 'Advanced Signal and Information Processing' and 'Pattern and Information Processing'. The former just hacked stuff together and got things to work. The latter spent ages working on these funky Bayesian models and stuff, but the processing was so difficult and hard to get to work in real time.
+1; but for normal consumer routers, I believe only stuff like sh, awk, lua are realistic options, for raspberries however I agree, all of python, julia, R or even octave would be options... but even then I would probably first see how far one gets with a limited set of heuristics (like changing thresholds based on the current tie of day, day-of week or similar)
I fully agree. A much simpler design however seems possible, and maybe that could be good enough?
Yes if a simple hack works go for it. However an Rpi4 has a ton of processor power. With 4 cores at 1400 MHz. Fitting a logistic regression to a hundred data points or so should take maybe 10 milliseconds.
I think for the shaper adjuster our real-time demands are in the "within a few seconds" range which might allow bayesian trickery (and I like about a Bayesian approach how we can simply feed the last posterior into the new posterior)
I did suppose cron job that issues 'tc change' calls at different times of the day might work well enough. But feels like a cop out / just intellectually unsatisfactory. And in fairness congestion is variable. I haven't measured. Only observe that around 4pm onwards the available bandwidth tends to reduce until perhaps around 10pm or something. Presumably with LTE connection hinges upon a lot of different factors. Like how many individuals are within the cell tower and streaming Lady Gaga on Amazon Music during their commute.
My RT3200 (only £80 on Amazon) seems to have lots of spare CPU cycles and RAM, at least since I only have max 80Mbit/s. I read in another thread with irqbalance it can even handle 1Gbit sqm.
And thank you @moeller0 and @dlakelan for your continued insights. Have been enjoying this tremendously.
Its possible with a detailed dataset, to develop an algorithm using something like Julia, and then code it into a compiled binary using maybe a C program. A shell script could collect data and every 20 seconds ask the C oracle what speed to set.
Am I the only one immature enough to find the idea of giving a girl's name to a programming language rather curious. She has been designed for high performance and interactive use. Are some of the comments here:
written with even a degree of humour?
The CAKE was only half baked, but then we put Julia into action!
The Julia developers claim its not named after anything. I always think of the fractals... The Julia con has an explicit code of conduct to not anthropomorphize it due to inclusive considerations...
Anyway, I think collecting a days worth of data and take a look at it might be a worthwhile step. Your existing shell script if it logged time and BW to a file and ping time to a file... Could be enough
IMHO most of these codes of conduct boil down to: behave like a sensible human being and treat others with courtesy and respect. Personally I can get behind that message quite easily (after all that is what parents, school and society taught me to be desirable behavior in the first place)
I wouldn't write new code in lua, luci is slowly migrating away from lua to client side javascript, once that covers the base luci package set, there's no reason to install lua on OpenWrt anymore. That won't happen over night, but it's progressing piece by piece, steadily.
Except of all the "real" programming languages I could see in the package archive, Lua was the smallest and lightest. For example Node.js, Python, and Erlang are like 10MB whereas lua and its libraries are under 1MB. Its still a quite viable language for small systems even if its not included.
I guess you could just source files in /tmp/autorate-ingress.sh and just assign variables in there. But that might run into issues if you try to change these files while reading. Another idea would be to store these in /etc/config/sqm and use the uci function to get/set them (I assume that these apply the appropriate locking, but have not confirmed that)....
Sure, you can call your functions from inside the main loop, just as you already do for get_avg_RTT...
I tried infinite while loop to call 'update_rates', but it seems the variable change from function call inside while loop do not persist? So maybe something else is needed?
root@OpenWrt:~# cat ./sqm-autorate
#!/bin/sh
max_ul_rate=35000
min_ul_rate=25000
max_dl_rate=60000
min_dl_rate=20000
read -d '' reflectors << EOF
1.1.1.1
8.8.8.8
EOF
alpha=0.1
ratio_adjust_down=0.25
ratio_adjust_up=0.125
max_delta_RTT=20
rx_bytes_path="/sys/class/net/wan/statistics/rx_bytes"
tx_bytes_path="/sys/class/net/wan/statistics/tx_bytes"
no_reflectors=$(echo "$reflectors" | wc -l)
rx_load_thresh=$(echo "scale=4; $min_dl_rate/2"|bc)
tx_load_thresh=$(echo "scale=4; $min_ul_rate/2"|bc)
RTTs=$(mktemp)
function get_avg_RTT {
for reflector in $reflectors;
do
echo $(ping -i 0.2 -c 10 $reflector | tail -1 | awk '{print $4}' | cut -d '/' -f 2) >> $RTTs&
done
wait
avg_RTT=$(echo $(cat $RTTs) | awk '{ total += $2; count++ } END { print total/count }')
> $RTTs
}
function update_rates {
t_start=$(date +%s)
get_avg_RTT
delta_RTT=$(echo "scale=4; $avg_RTT - $prev_avg_RTT" | bc)
prev_avg_RTT=$(echo "scale=4; (1-$alpha)*$prev_avg_RTT+$alpha*$avg_RTT" | bc)
echo "delta_RTT=$delta_RTT"
if [ $(echo "$delta_RTT > $max_delta_RTT" | bc) ]; then
cur_dl_rate=$(echo "scale=4; $cur_dl_rate-$ratio_adjust_down*($max_dl_rate-$min_dl_rate)" | bc)
if [ $(echo "$cur_dl_rate<$min_dl_rate" | bc) ]; then
cur_dl_rate=$min_dl_rate;
fi
cur_ul_rate=$(echo "scale=4; $cur_ul_rate-($max_ul_rate-$min_ul_rate)/4" | bc)
if [ $(echo "$cur_ul_rate<$min_ul_rate" | bc) ]; then
cur_ul_rate=$min_ul_rate;
fi
fi
cur_rx_bytes=$(cat $rx_bytes_path)
cur_tx_bytes=$(cat $tx_bytes_path)
t_cur_bytes=$(date +%s)
rx_load=$(echo "scale=4; (1/1000)*(($cur_rx_bytes-$prev_rx_bytes)/($t_cur_bytes-$t_prev_bytes))"|bc)
tx_load=$(echo "scale=4; (1/1000)*(($cur_tx_bytes-$prev_tx_bytes)/($t_cur_bytes-$t_prev_bytes))"|bc)
t_prev_bytes=$t_cur_bytes
prev_rx_bytes=$cur_rx_bytes
prev_tx_bytes=$cur_tx_bytes
echo "$delta_RTT < $max_delta_RTT && $rx_load > $rx_load_thresh"
if [ $(echo "$delta_RTT < $max_delta_RTT && $rx_load > $rx_load_thresh" |bc) ]; then
cur_dl_rate=$(echo "scale=4; $cur_dl_rate + $ratio_adjust_up*($max_dl_rate-$min_dl_rate)"|bc)
fi
if [ $(echo "$delta_RTT < $max_delta_RTT && $tx_load > $tx_load_thresh" |bc) ]; then
cur_ul_rate=$(echo "scale=4; $cur_ul_rate + $ratio_adjust_up*($max_ul_rate-$min_ul_rate)"|bc)
fi
if [ $(echo "$delta_RTT < $max_delta_RTT && $rx_load < $rx_load_thresh" |bc) ]; then
echo "scale=4; $cur_dl_rate - $ratio_adjust_down*($max_dl_rate-$min_dl_rate)"
cur_dl_rate=$(echo "scale=4; $cur_dl_rate - $ratio_adjust_down*($max_dl_rate-$min_dl_rate)"|bc)
fi
if [ $(echo "$delta_RTT < $max_delta_RTT && $tx_load < $tx_load_thresh" |bc) ]; then
cur_ul_rate=$(echo "scale=4; $cur_ul_rate - $ratio_adjust_down*($max_ul_rate-$min_ul_rate)"|bc)
fi
t_end=$(date +%s)
sleep $(echo "2-($t_start-$t_end)"|bc)
echo $cur_dl_rate
echo $cur_ul_rate
}
get_avg_RTT
prev_avg_RTT=$avg_RTT;
cur_dl_rate=min_dl_rate
cur_ul_rate=min_ul_rate
t_prev_bytes=$(date +%s)
prev_rx_bytes=$(cat $rx_bytes_path)
prev_tx_bytes=$(cat $tx_bytes_path)
while true
do
update_rates
done
Here 'cur_dl_rate' should keep decrementing to 0 and into negative I think (as I have not yet implementing the lower min floors), but for some reason it stays the same. So I think for some reason variable change is not persistent. I tried replacing while loop with multiple 'update_rates' calls but that didn't work either. Hmm.
Well you guard this by f [ $(echo "$delta_RTT > $max_delta_RTT" | bc) ]; then and set max_delta_RTT=20
delta_RTT is calculated before: delta_RTT=$(echo "scale=4; $avg_RTT - $prev_avg_RTT" | bc)
but prev_avg_RTT does not seem to be assigned anywhere.... so how does delta_RTT actually look?
Maybe piping this though bc hides the expected error?
By the way, I prefer a more verbose style for shell functions where I use echo to create a "return" value:
function get_avg_RTT {
for reflector in $reflectors;
do
echo $(ping -i 0.2 -c 10 $reflector | tail -1 | awk '{print $4}' | cut -d '/' -f 2) >> $RTTs&
done
wait
avg_RTT=$(echo $(cat $RTTs) | awk '{ total += $2; count++ } END { print total/count }')
> $RTTs
echo "$avg_RTT"
}
I would enable the individual clauses one by one. Updating variables is not really an issue as a quick test showed, so it probably is about the individual conditionals and whether they and/or their sequence makes sense...
! It's like brute force wins sometimes. With enough determination and grit.