Help to make sh script adjust SQM automaticaly

[moeller0]

Interesting idea, as far as I can tell evenroute offers a commercial product, the iqrouter, based on this approach (not identical in all details). You might want to have a look at https://forum.openwrt.org/t/sqm-autorate-ingress-can-i-set-thresholds-for-this/55771 where @dlakelan developed something functionally similar using erlang...

@bagus91

So my logic is like this

1. disable client internet

This is pretty intrusive, as you will have repeated outages due to your schedules measurement runs...

@bagus91

• do speed test use netperf / iperf3 to specific server to get download & upload average speed

This is pretty sane, you need your own measurement endpoints to get halfway reliable speedtests, but keep in mind that you will measure the whole network path, so if the transit/peering between the network housing your automatically adjusted SQM instance and the AS/datacenter housing the measurement endpoint is congested, you will throttle all outgoing traffic, even if destined to non congested networks.

@bagus91

I want this solution because my ISP have not fix speed, it guarantee me up to 15mbps, but sometimes it can be more than it, so better i take advantage by adjust SQM regulary. So it will be loss for me if my internet can reach 30mbps but I put 14mbps on SQM

The downside is, that when your ISP just throttled you from 30 to 14 you will need to endure quite substantial bufferbloat until your adjustment script is run again.

[bagus91]

You might want to have a look at SQM autorate-ingress: Can I set thresholds for this? where @dlakelan developed something functionally similar using erlang...

Thanks! will take a look on it

This is pretty intrusive, as you will have repeated outages due to your schedules measurement runs...

I did this so the speedtest result from script will more accurate to avoid some client downloading in same time with speedtest process

This is pretty sane, you need your own measurement endpoints to get halfway reliable speedtests, but keep in mind that you will measure the whole network path, so if the transit/peering between the network housing your automatically adjusted SQM instance and the AS/datacenter housing the measurement endpoint is congested, you will throttle all outgoing traffic, even if destined to non congested networks.

my plan is keep it automatic because if i do it manually it will cancel my idea/plan hehe
but i should can set minimum speed for my internet, to prevent sqm config speed lower than my ISP speed promise. so if i set minimum bandwidth 14mbps for download & upload, then speedtest result lower than it, it will set to 14mbps.

The downside is, that when your ISP just throttled you from 30 to 14 you will need to endure quite substantial bufferbloat until your adjustment script is run again.

yah i plan to make it every 8 hour adjustment
or when i play game and feel lagging, i should do manual adjustment

[Bndwdthseekr]

@dlakelan's script works pretty well for me so far. It may be just the thing you are looking for! I have been using this script on and off when needed, and it's great when I am doing latency sensitive things and definitely keeps the buffer bloat under control.

I have a shell script to check latency while running a speed test from a network server if 80% of the bandwidth isn't being used to prevent the margin from being increased without a load on the connection, but its really just something I'm playing around with in my spare time. Its going to have to become a whole lot more seamless than it currently is now before I'm comfortable releasing it. I haven't really felt the need to improve/use it either, so not much progress has been made. If @dlakelan has some time to add a feature that would check how much bandwidth is being used so the target isn't increased if the connection is idle it could work as well as my idea I think.

I think ultimate solution may be a combination of several things, but I am more focused on putting my ISP in hot water with the FCC until they provide more back-haul to support the meager bandwidth I pay to get. You are extremely lucky to have an ISP who will give you a minimum guarantee and not try dodgy underhanded tactics to dodge the issue.

[bagus91]

finally i succedd create script to auto adjust SQM from betterspeedtest.sh result
i made custom code to get average speedtest result and set it into sqm. i use cron every 6 hour to do speedtest. I'm so happy coz it met my own requirement and i learnt new things about bash script

[Sparks]

I’d be interested to try it too. I’ve got crappy adsl and speeds fluctuate depending on a when it rains would be great to be able to tweak sqm periodically

[bagus91]

great!
ill share it some other time. my script is so messed up in many files. ill make it in 2 script file so it will be easy to share
in your condition i think my script not really match because i use average. for your condition maybe better use result from speedtest right away. if I have, ill assist or help you after o share my script

[bagus91]

Hi @Sparks
this is my custom script https://github.com/baguswahyu/autoSQM-damasus.bagus
dont forget to download betterspeedtest.sh in here https://github.com/richb-hanover/OpenWrtScripts/blob/master/betterspeedtest.sh
put all of the files in /root/
sh autoSQM.sh to run the script for auto adjust sqm, and make sure check uci sqm script. mine is eth1, adjust it with your local configuration
make sure you have netperf package in your openwrt, this require netperf to do speedtest
put on cron to make script auto run
cc @moeller0 @Bndwdthseekr | if you guys wanna try and test my script, feedback is welcome

note : if you want, you can add script code to disable internet from client when script running, it will give more accurate results

[Bndwdthseekr]

@Sparks If it only fluctuates when it rains, you might want to look into adjusting based on modem sync. This would get you much more accurate results if it isn't a congestion issue on the ISP's side of things like it is in my case... I'm not sure how you would get that realtime line rate if your ISP uses proprietary modems though, but I'm sure it could be done.

@bagus91 I'm glad to see more people thinking about this issue. Your approach is interesting. Do you find it to be disruptive to clients on the network since you are stopping and restarting SQM? What do you think about setting the limits to the maximum expected bandwidth, then running a speed test just long enough to find your usable bandwidth with something like bwm-ng?

For example, if your maximum bandwidth is 30 megabit/s I would use the below command, and run a speedtest while using bwm-ng to get the average bandwidth on the wan iface during the test, then use the same command with your average plugged in after the test is complete. The speed test will just be to fully saturate the connection in case your clients aren't using much. This way SQM will still be somewhat effective during the speed test. (of course, you will have to input your WAN iface and other settings... I just copied the below from my system. you can use 'tc -s qdisc' command to find yours if you are unsure)

tc qdisc change root dev ifb4pppoe-wan cake bandwidth 30000Kbit dual-dsthost nat overhead 44 ingress

Just some thoughts... you have given me a few ideas too, hopefully I will have some time to work on it! Oh, and I HATE my ISP

[dtaht]

if you do live speedtests on a live network, it helps to capture the ifconfig statistics rather than rely entirely on the speedtest result. That gives you the real total data transfered over a sampling interval, from all sources.

[Bndwdthseekr]

Yeah, that's pretty much what I was getting at. If bwm-ng is used (see below) while running a speed test to fully saturate the connection, the samples of the connection under full load can be averaged and get you pretty close to the max bandwidth you can expect at the time.

bwm-ng -I pppoe-wan -u bits -c 5 |grep 'pppoe-wan' | sed -e's/  */ /g' | cut -d " " -f 3

OUTPUT:
root@OpenWrt:~# bwm-ng -I pppoe-wan -u bits -c 5 |grep 'pppoe-wan' | sed -e's/  */ /g' | cut -d " " -f 3
0.00
6271.40
6057.51
6065.72
5986.98

I'm not sure if this is the best solution, just a thought I had to get around having to disable the internet for the clients to get an accurate reading like @bagus91 had mentioned.

Of course getting that from ifconfig would probably be the best way to do it now that I think about it... and it wouldn't depend on bwm-ng to work.

[bagus91]

wow that's good method @Bndwdthseekr
i was thinking measure wan speed but i don't have idea how to measure it that time
ill test and try mix my script with ur command to get more realistic speed from wan
right now im doing speedtest without disconnect clients. but if that time some client use full bandwidth. of course it will disturb soeedtrst result. but until now. i still get stable speed from speedtest result by do adjustment 4 times a day

[dtaht]

Never heard of this tool, thx. @richb-hanover-priv ? @moeller0 does this help?

[richb-hanover-priv]

I think the approach is intriguing. On a period basis:

• Run a process to guarantee the link is fully-loaded
• Make some measurement of max up and down transfer rates
• Adjust the SQM params based on those measured speeds.

I don't know if these random thoughts might help/be feasible:

• Using ifconfig values would seem to capture the actual number of bytes transferred in each direction. Thus it's not affected by other client traffic. You also avoid needing to install another package.

• To get data rates, measure now with ifconfig, and measure again 30 seconds later to get two data samples. Subtract those values and divide the result by 30 seconds to get byte rate for that interval, and multiply by 8 to get bit rate.

• I don't think it is necessary to shut off "normal client" traffic during these SQM-Adjustment measurement intervals. The process to load the up and down links will "fill in the gaps" of the client traffic to ensure the link is fully-loaded. This will be a brief time on a periodic basis, and will not significantly harm client traffic.

• It is probably necessary to shut off SQM during the SQM-Adjustment measurement interval. (If the SQM parameters happen to be set too low, they would prevent the link from achieving its full capacity.)

[bagus91]

my script still not use interface traffic.for now my speedtest result still okay. but it will be more accurate of when i do speedtest, data i capture come from interface
will be so happy of you could contribute in my github code hehe
until now i still cant get improve my script to read interface traffic

[bagus91]

aside from bwm-ng
do u have other method to get interface traffic while doing speedtest?
just trying to get data from default package hehe

its kinda complicated for me to extract data with bwm-ng
because the speed test i used is do download & upload separate. if i start bwm-ng befotr and after speedtest. the data not accurate coz bwm-ng will give low value when i measure upload but upload test is not start yet

[Bndwdthseekr]

It's probably best to do as @richb-hanover-priv suggested and capture the total bytes transferred from your WAN interface in 'ifconfig' at the beginning of that speedtest, and again at the end.

If you want to use bwm-ng, I think adding this: -T avg to the command would give you an average? Then you can take one sample while you're running the speedtest. I'm not 100% sure about this though... I can check later this afternoon.

[Bndwdthseekr]

Try this:

cat /sys/class/net/pppoe-wan/statistics/rx_bytes
cat /sys/class/net/pppoe-wan/statistics/tx_bytes

Replace 'pppoe-wan' with your iface of course, and that's your data point, which is updated in realtime

Edit:
Maybe this will help. Its lazy as... but it works over here. I didn't bother adding upload as mine is always constant.

 #set to max bandwidth connection can support
 tc qdisc change root dev ifb4pppoe-wan cake bandwidth 6944Kbit dual-dsthost nat overhead 44 ingress
 #run speedtest
 sh /root/betterspeedtest.sh -t 15 -n 15 -p www.google.com -H netperf-west.bufferbloat.net &
 sleep 4
 sample1=`cat /sys/class/net/pppoe-wan/statistics/rx_bytes`
 sleep 10
 sample2=`cat /sys/class/net/pppoe-wan/statistics/rx_bytes`
 echo Sample1: $sample1
 echo Sample2:	$sample2
 #get average speed for download & upload
 download=$(($sample2-$sample1))
 echo DIFF: $download
 totalkb=$(($download/10))
 downloadKbps=$(($totalkb/125))
 echo CURRENT Kbps: $downloadKbps

 #set to average bandwidth during the test
 tc qdisc change root dev ifb4pppoe-wan cake bandwidth "$downloadKbps"Kbit dual-dsthost nat overhead 44 ingress
 
 echo "auto sqm DONE! Download set to: "$downloadKbps""

[Knomax]

I want to thank you @bagus91 now it is working under 1Mbps speed but i will let @bagus91 to tell the details.
Again thank you and if i can test something else to improve it i will be very happy to help.

[bagus91]

i think for this one, i need user netperfrunner.sh to test dowload & upload speed in same time
next time when i have time to research and test, ill make my script have option to choose which method will be used
thanks for your help @Bndwdthseekr