SQM - bufferbloat -> dyamical adjustment of bandwidth needed

[camelcamro]

Hello,
must of users are having an INTERNET conenction via cable and mostly having good bandwidth speed.
but there are also a lot of users whic are having an internet connection via LTE, 3G, or adsl which is very different - depends on day or/and time, (eg:on weekend or on public holidays on rushhours the internet speed is muchmore lower as providers are selling/charging you.
(almost all contracts are having this "maximum speed", which mostly can't be reached ...

in my case ... paying fora 50/20mbit LTE (austria), but the real speed is totally different
see my picture ...

a bigger challenge is in costa rica, where you have totally worst situation:
I''m using in costa rica 3 routers with l3 different providers and loadbalancing + routing "wasted" traffic to 1 of the unlimited providers sim cards.

kolbi: 3G + 4G = speed differs from 14mbit / 8mbit to 1,0mbit/0,3mbit !!
movistar 3G = speed differs from 3,5mbit / 1mbit to 0,5mbit/0,2mbit !!
claro: 3G = speed differs from 2,5mbit / 0,5mbit to 0,4mbit/0,2mbit !!

so, the challenge is, i can't setup SQM fix to the slowest speed which i have per day usage ...

what would be needed is like " SQM with dyamical bandwidth adjustment"

the idea would be to do every X-hour via a cron job to run a script:
like:
1.) stop/disable SQM if activated
2.) run PING test to measure latency
3.) run PING test + the speed test (if wanted i can upload a speedtest.py (python script, which i found on internet) to measure the maximum speed , and adjust via a luci script the SQM settings *
3.) start SQM with new settings
4.) run a simple DL test of 60 seconds and check with a PING the latency time *, if bufferbloat is better

is anyone already doing that ?

PLS: * .....for 2 + 4 -> the next problem is, that if you run a speedtest all few hours that is generating a lot of traffic, and most providers are not having an unlimited traffic tarif (specially in latina america such kind of 3g/4G traiffs are not existing)

did anyone already do such kind of thinkings or implementation/scripting ?

cu camel

[mbo2o]

That might be a bit unstable. You should use a negative feedback after doing your speed test.
So you new limit should be w1 * current_limit + w2 * new_measurement

Where w1+w2 = 1

For example
w1 = 0.8
w2 = 0.2

[slh]

Your suggestion is pretty difficult if the router itself (which has to employ sqm) doesn't really know the physical connection parameters itself, because it has to use an external modem. Leaving the router in the dark about the physical connection speed and the practical connection speed (e.g. if the congestion isn't on the last mile, but at the ISP's side/ overbooking). While your suggestion of doing regular speed tests sounds nice at first and might fight these particular issues, it introduces two new ones instead...

1. this could quickly become very expensive for metered connections, as any type of speedtest (in particular automated ones) requires quite a bit of data transfer.

2. doing a speed test only gains useful data if nothing else uses (significant) amounts of throughput, you can't get any kind of meaningful information if one of your clients is a p2p supernode. the other problem is simply 'overloading' your WAN connection at regular intervalls, negatively affecting the services you actually want to use (think doing a speedtest while having a SIP call, while playing an online 1st person shooter with ping time going through the roof, watching a livestream of some kind, etc.).

[camelcamro]

i guess you're right ... a speedtest is senseless and produce to much load and would maybe interfear with other services which are used (eg: voice call, etc ...

another idea is to create a kind of logic:
eg: DL 2mbit / UL 1mbit as base ...
and checking the throughput all the time + let running a ping all the time ...
if the throughput reach over 80% of the current max speed settings in SQM ... verify latency times on the running PING ...
if ping duration time still OK, then increase SQM max speed setting with eg: 10%
the other case, if ping latency increase on high traffic (eg: ober 80%) => then reduce max speed in SQM settings and restart SQM service

better ideas ? are welcome

in any case a SQM with an different bandwidth speed is almost senseless as long as it is not taking care about what is possible (i mean the effective speed) as this needs to be adapted to this.
and to wait that sprout (alfalfa.mit.edu) will be a new standard internet protocol ... can't wait for that.

[moeller0]

Well you can use

tc qdisc change dev eth0 root cake bandwidth 900Mbit

to change the bandwidth of the running cake instance (in this example on eth0) at run time without any packet loss. So all you need to do is find a way to divine the available bandwidth and you are off to the races
In all earnest, I just showed you how to implement the easy part, the tracking of the bandwidth is the tricky part. Especially since in your case neither up nor download are constant enough to allow for example to use the idea behind gargoyle's ACC design, where you keep pining to an external target and interpret all increases in delay as signs of downstream congestion to which you respond to shaping the download speed down (with occasional tries at higher bandwidths to also track increasing bandwidth), but that relies on either Down- or Upstream being the single point on congestion, if both can congest (as in your case if the available link has noticeably less bandwidth that cake is configured to) it will get hard to disambiguate up and downstream delay increases. In theory one way delay measurement would solve that part of the issue, but as far as I know there are no open on-way measureent systems in the internet (and they would require anyway that both the reflector's clock and your router's clock are reasonable closely phase-locked).

Best Regards

P.S. cake also has the autorate-ingress keyword, which tries to track a variable download speed (with the configured speed being the upper limit IIRC), but I blieve this will require backpressure from the actual "GSM"-modems to be useful, so most likely this will not solve your problem...

[camelcamro]

well, as i need to test any solution I'm trying now to set every hour to max DL speed:
UL st mostly always the same as 1:50 and is not that overloaded and is throttled by provider ....
additional, i have no data limit on that provider, but it differs a lot (from 600 to 5000kbit in DL)
what i do:
1.) stopping SQM
2.) doing high load on DL (DL speedtest)
3.) calculate max bandwidth + average +95%
4.) set uci valuesfor my 3g-4G
5.) start SQM

and execute this script every hour ...

if anyone is having better idea .. pls support.
maybe better as nothing.

#!/bin/sh

/etc/init.d/sqm stop 2> /dev/null

/usr/bin/speedtest.sh Germany 25 > /dev/null &..
# &>/dev/null
sleep 2

i=0
bandw_sum=0

while [ $i -lt 20 ]
do
    i=$(( $i + 1 ))
    #echo "-- i=${i} --"
    bandw=`bwm-ng -I 3g-4G -u bits -c 1 |grep '3g-4G' | sed -e's/  */ /g' | cut -d " " -f 3`
    #echo "bandw=${bandw}"
    bandw_sum=`echo $bandw_sum + $bandw | bc`
    #echo "bandw_sum=${bandw_sum}"
done

bandw_avg=`echo $bandw_sum / 20 | bc`
bandw_avg_sqm=$(($bandw_avg*95/100))

#minimum, if something goes wrong with query for speed
if [ $bandw_avg_sqm -lt 650 ]; then
    bandw_avg_sqm=650
fi

echo "`date '+%Y-%m-%d %H:%M:%S'` - SQM - Speed Reset to: bandw_avg=${bandw_avg} / bandw_avg=${bandw_avg_sqm}"
logger -t SQM "SQM - Speed Reset to: bandw_avg=${bandw_avg} / bandw_avg=${bandw_avg_sqm}"

#bandw_avg_rounded=`echo (($bandw_avg + 0.5 ) / 1) | bc`
#echo "bandw_avg_rounded=${bandw_avg_rounded}"

uci set sqm.cfg02b68a.download=$bandw_avg_sqm
uci commit

/etc/init.d/sqm start 2> /dev/null

[mbo2o]

It might make more sense to do a hourly weekday test and same for weekend without sqm. Plot the results and maybe repeat for a few days to get good confidence in the results.

Then use cron to set sqm parameters based on your observations.

It is not necessary to have dynamic sqm based on periodic test.

[camelcamro]

well, i do it every hour for a while (eg: 1-2 weeks)
then I'M sure, there is a trend ... but then you have also week, weekend, public holidays,rain, etc.. on wich it differs)
if bad weather the users are more using internet
.
that is my goal (to reduce the wasted dl) and to avoid to overload the internet every hour.
and after a while, i can grep from logs where are the peaks.

[mueslimak3r]

@camelcamro it's been a few years. How's your script holding up to the test of time?