OK - I'll just code up facility to independently change download and upload shaper rate during our compensation period and @gba can test that.
Just to the Netherlands to visit family (my wife is Dutch). They seem to have rather better internet infrastructure (and infrastructure in general) as compared to what we are stuck with in Scotland (the SNP, just like our NHS, leave a lot to be desired!).
OK @gba and others, I have merged the changes from the 'starlink-testing' branch to the main branch. Clearly more work is to be done in terms of figuring out how to best optimize for Starlink, but I like these changes so far to the overall flow anyway.
I now prepend 'dl' or 'ul' to the 'load_condition' identifiers, as follows:
1657044342.191073 25874 1296 97 5 [1657044342.173162] 8.8.8.8 53 36441 46800 10369 25990 1 dl_high_sss ul_low_sss 26695 25000
1657044342.249335 23124 1444 86 5 [1657044342.230145] 8.8.4.4 52 38338 49200 10872 25986 0 dl_high ul_low 26961 25250
1657044342.293023 23124 1444 85 5 [1657044342.269984] 1.1.1.1 56 39484 46900 7423 25976 0 dl_high ul_low 27230 25250
Thus we have the following form now for the load identifiers:
[dl|ul]_[low|med|high]_[bb|bb_sss]
And this allows differentiation in the case of Starlink satellite switch compensation based on whether the load is download or upload.
For now I have amended the get_next_shaper_rate() function such that we only drop down to the minimum shaper rate in respect of upload:
case $load_condition in
# Starlink satelite switching compensation, so drop down to minimum rate through switching period
ul*sss)
shaper_rate_kbps=$min_shaper_rate_kbps
;;
# bufferbloat detected, so decrease the rate providing not inside bufferbloat refractory period
*bb*)
if (( $t_next_rate_us > ($t_last_bufferbloat_us+$bufferbloat_refractory_period_us) )); then
adjusted_achieved_rate_kbps=$(( ($achieved_rate_kbps*$achieved_rate_adjust_down_bufferbloat)/1000 ))
adjusted_shaper_rate_kbps=$(( ($shaper_rate_kbps*$shaper_rate_adjust_down_bufferbloat)/1000 ))
shaper_rate_kbps=$(( $adjusted_achieved_rate_kbps < $adjusted_shaper_rate_kbps ? $adjusted_achieved_rate_kbps : $adjusted_shaper_rate_kbps ))
t_last_bufferbloat_us=${EPOCHREALTIME/./}
fi
;;
# high load, so increase rate providing not inside bufferbloat refractory period
*high*)
if (( $t_next_rate_us > ($t_last_bufferbloat_us+$bufferbloat_refractory_period_us) )); then
shaper_rate_kbps=$(( ($shaper_rate_kbps*$shaper_rate_adjust_up_load_high)/1000 ))
fi
;;
# medium load, so just maintain rate as is, i.e. do nothing
*med*)
:
;;
# low or idle load, so determine whether to decay down towards base rate, decay up towards base rate, or set as base rate
*low*|*idle*)
if (($t_next_rate_us > ($t_last_decay_us+$decay_refractory_period_us) )); then
if (($shaper_rate_kbps > $base_shaper_rate_kbps)); then
decayed_shaper_rate_kbps=$(( ($shaper_rate_kbps*$shaper_rate_adjust_down_load_low)/1000 ))
shaper_rate_kbps=$(( $decayed_shaper_rate_kbps > $base_shaper_rate_kbps ? $decayed_shaper_rate_kbps : $base_shaper_rate_kbps))
elif (($shaper_rate_kbps < $base_shaper_rate_kbps)); then
decayed_shaper_rate_kbps=$(( ($shaper_rate_kbps*$shaper_rate_adjust_up_load_low)/1000 ))
shaper_rate_kbps=$(( $decayed_shaper_rate_kbps < $base_shaper_rate_kbps ? $decayed_shaper_rate_kbps : $base_shaper_rate_kbps))
fi
t_last_decay_us=${EPOCHREALTIME/./}
fi
;;
esac
@gba you can tweak this behaviour as desired for testing. If you want to drop down to the minimum shaper rate for both upload and download, then just replace 'ul * sss' with '*sss'. Or alternatively we can split things out and provide separate patterns 'ul * sss' and 'dl * sss' to match against upload and download separately, and then we can, for example, drop down to the minimum shaper rate for upload and min(base shaper rate, previous shaper_rate) for download.
Latest code for testing is in the main branch here:
Thanks for making the updates. I'm running it right now and will try to do some tests when I get a chance. Enjoy your vacation!
Testing as well. The ul*sss setting appears to improve download bandwidth without affecting my latency, but I'll need to test more to validate that.
After this thunderstorm passes though, so maybe not until tomorrow.
Excellent - that was my hope. I'm eager to see the findings from the further testing.
The script collection on GitHub seems a bit too expansive for my needs. I'm looking for a simple time-based switch of the settings: from 00:00 to 12:00 (AM period) I have 100mbit/s and from 12:00 to 00:00 (PM) I have 70mbit/s. So essentially I just want to run a script that changes the download/upload rate in the SQM config.
I'm willing to eat the memory cost of running crond 27/4, but I dont know how to change and then apply config rate for SQM. I guess I could just make 2 copeis and copy them over to current config with basic cp.
Looking at the script I see
tc qdisc change root dev $interface cake bandwidth ${shaper_rate_kbps}Kbit 2> /dev/null
command however, it does not make much sense to me. What "bandwidth" stands for? Upload? Download? Combined? Does the command only change the value but wont apply new settings or does it apply them instantly?
So shaper rate change is instantaneous. You just set the bandwidth for the appropriate interfaces (check output of 'tc qdisc ls') using that command you correctly identified from the code - one call for each interface.
Does your ISP really provide exactly those rates at those times?
So my understanding I dont need to restart any service to apply said changes?
tc qdisc ls gives me my current setup:
tc qdisc ls
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
qdisc noqueue 0: dev br-guest root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 8015: dev br-wan root refcnt 2 bandwidth 48Mbit besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 44 mpu 84
qdisc ingress ffff: dev br-wan parent ffff:fff1 ----------------
qdisc noqueue 0: dev eth0.2 root refcnt 2
qdisc cake 8016: dev ifb4br-wan root refcnt 2 bandwidth 48Mbit besteffort triple-isolate nonat wash no-ack-filter split-gso rtt 100.0ms noatm overhead 44 mpu 84
ifb4br-wan an "upload interface"?rate if I want to change rate only or do I need to copy the rest of the settings: besteffort triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms noatm overhead 44 mpu 84 refcnt 2?tc qdisc change root dev br-wan cake bandwidth 68000Kbit 2 > /dev/null for download, what's 2 is for?I'm not exactly sure about actual meaning behind the question. This is the plan's conditions and I assume the limits will be set at specific or close to specific time. Few minutes wont kill me.
If you are asking "is your ISP provides these speeds with 100% window" then I think the answer is obvious for any ISP in any country. However, I doubt it is gonna be as unstable as mobile network so some occasional dips are okay/expected (hence I dont think autorate script is really relevant to my case and it probably will hurt more than do good).
Hard to tell from the outside. What does ifstatus wan | grep -e device return?
Because this is how Linux interfaces operate; each interface will only allow to instantiate qdisc (like cake) on its egress side. So we need to play some tricks to get a qdisc instantiated in incoming/ingress traffic. The trick sqm uses ist to use an intermediary functional block device short IFB for that purpose, the IFB is essentially an abstraction that allows up to copy all incoming packets from the true ingress interface to, and this IFB will then send these same packets to the kernel, but sending happens in the IFB's egress side where Linux will allow us to instantiate a qdisc like cake.
You can call this an anti-pattern if you will as well as counter-intuitive (I agree to the latter), but that is a discussion you need to have with the upstream linux kernel network maintainers, sqm is no real position to change that.
IIRC just specifying the rate is enough, after all you instruct a change and you specify what to change....
That means reference count 2 which probably means two other object hold references to the qdisc so these probably should not be deleted without first caring for those dependencies.
The important part is 2 > /dev/null which is described here in short it will silence any error output from that command.
ifstatus wan | grep -e device
"l3_device": "br-wan",
"device": "br-wan",
I knew what dev/null redirection means, I just thought 2 was part of the previous command argument.
Thanks that does not help much but it seems to imply that ifb4-br-wan is your ingress/download interface while br-wan is your egress interface.
No. A combination of other interests taking priority, wanting to see how the AQL/ATF debugging turns out, and my initial inspection indicated sch_arl will likely need changes for my intended use will make this slow going. As I mentioned above, measuring RTT may not be what I'm looking for - tweaking how a wifi AP adjusts rates may be where I end up. I can already demonstrate that by slowing clients down a bit, I seem to fix some issues.
If I was to go forward with passive measurement of RTT's, I think I would use pping to make an assessment and then look at how to implement something closer to pping in sch_arl. i.e. pping apparently can measure RTT at arbitrary "capture" points while sch_arl seems to be built for an end point.
If your interested in moving sch_arl forward, don't wait for me - I could take years (and never finish).
HTH
I haven't had much time to do much testing with the latest version of the Starlink autorate code (though I am running it), but I did spend a little time looking with more detail at my previous flent runs and found a couple interesting things. I feel like this information could better inform how the autorate script is set up specifically for Starlink, although I'm not exactly sure what to change to improve it.
My observations are:
Starlink's download bandwidth fluctuates wildly, going from < 20 Mbps to > 120 Mbps in a matter of seconds. We already knew this. But looking at these flent graphs it is very obvious that the download bandwidth moves inversely to the ping latency. I'll have some examples where they are practically a mirror image of each other. This means there should be a lot of improvement from the autorate script, if the correct parameters are dialed in.
I had already found that sudden changes in latency occurred at those 15 second Starlink transitions, but flent makes it clear that sudden changes in download bandwidth also occurs at those same increments. There are some 15 second windows where the average download bandwidth is over 100 Mbps but the latency is fairly low. There are also 15 second windows where the average download bandwidth is < 10 Mbps and the latency is spiking to more than 300 ms.
It makes me wonder if, for Starlink, basically treating each 15 second block as a separate autorate period would be the way to go. Reset bandwidth back to the base shaper rate at the beginning of each one and be prepared to throttle back hard if the latency starts to spike? Not sure.
Of course then there is the question, as @dtaht brought up, how does Starlink decide how much bandwidth to give you in the next 15 second cycle (I'm making an assumption that it decides that every 15 seconds but that's just an assumption)? There likely is a fairness algorithm; will the autorate interfere with that?
Anyway, here are some graphs so you can see what I'm talking about. This was done with no autorate so that doesn't interfere, it is just using fixed CAKE with 200/30 down/up rate set so there shouldn't be many packets CAKE is dropping.
Thanks a lot @gba. Hopefully others can give their thoughts on the extra and helpful looking data you have provided.
So how about in addition to setting the upload shaper rate to minimum, we set the download shaper rate to the base rate? The base rate is intended to be the safe harbour bandwidth. It's not the minimum and is what we always decay back to absent load. So I think it makes sense to go back to base given your indication that bandwidth seems to kind of reset.
@moeller0 given what @gba states above this seems like the best we can get, or?
That could be achieved by just adding in the appropriate case statement:
case $load_condition in
# Starlink satelite switching compensation
dl*sss)
shaper_rate_kbps=$base_shaper_rate_kbps
;;
ul*sss)
shaper_rate_kbps=$min_shaper_rate_kbps
;;
I've got a question. If you enable sqm on upload only, does hardware offloading work on download?
As far as I can tell sqm does not interfere with "hardware offloading" at all, rather the opposite, packets being handled by hardware offloading will not be visible to and hence not be handled by sqm.
@gba any updates? Would you like me to implement the change above regarding setting download rate to base rate in addition to setting upload rate to minimum on satellite switch times?
Unfortunately I haven't had much time to do testing this week, although I've been running with it and I haven't noticed any issues. If you want to make that change I think it would be a good one to test, I think I would probably be able to run a few tests this week comparing the two versions.
OK, ready with this commit: