CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

Installing and Using OpenWrt
61

Ok @Lynx convinced me to work on this for a bit. I've got what looks like a working bandwidth measuring timer task... every second or so it reads the rx and tx amounts and calculates the Kbps through a given interface and reports it to the adjuster thread.

Now I think the idea is if the current bw in either direction is less than 1/2 the current set point, we cut the set point by some factor... so that it converges down towards being double the current bandwidth, or the minimum... so I'll try that out. @Lynx can test tomorrow since it's night time where he is.

What I'm relying on here is that if there's no delays, it'll try to bump up the bandwidth automatically (existing functionality)... if it goes above 2x the current usage, the bw monitor will bump it back down... so it'll oscillate a little... if our bandwidth usage goes up then we'll stop bumping it back down and it'll drift up.... if the delays occur, it'll drift back down... if our usage drops... bandwidth monitor will bump down.

I pushed it to the devel branch in my repo. Test it out!

2 Likes
62

Thank you very much indeed for your work on this @dlakelan. I hope in the long run this work will benefit others too - it certainly seems I am not the only one facing this issue.

It still runs. But presumably needs a tweak or two to work. Bandwidth seems to stay at the minimum set values even when saturating the connection.

I have set:

root@OpenWrt:~/sqmfeedback# tail sqmfeedback.erl
    monitor_ifaces([{"tc qdisc change root dev wan cake bandwidth ~BKbit",25000,30000,35000},
                    {"tc qdisc change root dev veth-lan cake bandwidth ~BKbit",20000,40000,60000}],
    ["google.com","one.one.one.one","quad9.net","facebook.com",
     "gstatic.com","cloudflare.com","fbcdn.com","akamai.com","amazon.com"],
                  "wan"),
    receive
        _ -> true
    end.

Does this look OK? Is 'wan' the correct Iface in my wan / veth-lan setup?

This gives:

root@OpenWrt:~/sqmfeedback# erl -pa /root/sqmfeedback -eval 'sqmfeedback:main().'
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]
 
Eshell V11.0  (abort with ^G)
1> Received bandwidth report: Rx: -233.727273, Tx: -247.000000 at 1634283819035
ping google.com with results: [50.8,52.3,53.1,53.8,57.2]
ping akamai.com with results: [46.0,48.8,49.8,50.0,50.3]
ping one.one.one.one with results: [45.8,46.1,47.3,51.6,52.8]
ping gstatic.com with results: [50.8,52.2,52.8,53.3,57.3]
ping amazon.com with results: [120.0,122.0,123.0,124.0,124.0]
ping cloudflare.com with results: [45.4,50.0,51.8,51.8,53.1]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 32000Kbit
ping facebook.com with results: [260.0,263.0,263.0,263.0,264.0]
ping fbcdn.com with results: [264.0,269.0,271.0,271.0,272.0]
ping quad9.net with results: [172.0,173.0,173.0,183.0,184.0]
Received bandwidth report: Rx: 0.373062, Tx: 0.493217 at 1634283820067
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 25600Kbit
Received bandwidth report: Rx: 0.357642, Tx: 0.163836 at 1634283821068
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20480Kbit
Received bandwidth report: Rx: 0.185814, Tx: 1.337662 at 1634283822069
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 0.366633, Tx: 0.180819 at 1634283823070
Received bandwidth report: Rx: 0.586414, Tx: 1.390609 at 1634283824071
Received bandwidth report: Rx: 0.231768, Tx: 0.121878 at 1634283825072
Received bandwidth report: Rx: 0.045954, Tx: 0.105894 at 1634283826073
Received bandwidth report: Rx: 0.139860, Tx: 1.304695 at 1634283827074
Received bandwidth report: Rx: 0.045954, Tx: 0.000000 at 1634283828075
Received bandwidth report: Rx: 0.000000, Tx: 0.000000 at 1634283829076
Received bandwidth report: Rx: 0.185814, Tx: 1.288711 at 1634283830077
Received bandwidth report: Rx: 7.204795, Tx: 1.836164 at 1634283831078
ping gstatic.com with results: [56.3,57.3,57.4,59.4,59.9]
Received bandwidth report: Rx: 1.482517, Tx: 5.648352 at 1634283832079
Received bandwidth report: Rx: 0.000000, Tx: 0.000000 at 1634283833080
Received bandwidth report: Rx: 0.357642, Tx: 1.412587 at 1634283834081
ping facebook.com with results: [260.0,261.0,268.0,269.0,273.0]
Received bandwidth report: Rx: 0.217782, Tx: 0.121878 at 1634283835082
Received bandwidth report: Rx: 0.169830, Tx: 0.000000 at 1634283836083
Received bandwidth report: Rx: 0.713287, Tx: 2.024975 at 1634283837084
ping quad9.net with results: [174.0,180.0,180.0,194.0,215.0]
ping amazon.com with results: [110.0,120.0,121.0,121.0,122.0]
Received bandwidth report: Rx: 1.720280, Tx: 1.832168 at 1634283838085
ping google.com with results: [51.8,52.0,52.0,52.1,56.0]
ping akamai.com with results: [39.8,47.8,50.8,51.7,52.3]
Received bandwidth report: Rx: 5.974026, Tx: 13.636364 at 1634283839086
Received bandwidth report: Rx: 0.357642, Tx: 1.378621 at 1634283840087
Received bandwidth report: Rx: 0.000000, Tx: 0.000000 at 1634283841088
Received bandwidth report: Rx: 0.547453, Tx: 0.756244 at 1634283842089
ping one.one.one.one with results: [46.1,46.7,53.7,54.6,55.3]
Received bandwidth report: Rx: 0.217782, Tx: 1.256743 at 1634283843090
Received bandwidth report: Rx: 0.185814, Tx: 0.713287 at 1634283844091
Received bandwidth report: Rx: 0.000000, Tx: 0.000000 at 1634283845092
Received bandwidth report: Rx: 6.125874, Tx: 5.946054 at 1634283846093
ping fbcdn.com with results: [266.0,272.0,273.0,274.0,274.0]
Received bandwidth report: Rx: 10.461538, Tx: 5.675325 at 1634283847094
Received bandwidth report: Rx: 5.168831, Tx: 3.501499 at 1634283848095
Received bandwidth report: Rx: 539.390609, Tx: 24.990010 at 1634283849096
ping cloudflare.com with results: [42.1,44.2,44.8,51.7,59.8]
Received bandwidth report: Rx: 243.877123, Tx: 22.543457 at 1634283850097
Received bandwidth report: Rx: 3.771229, Tx: 7.016983 at 1634283851098
Received bandwidth report: Rx: 508.993007, Tx: 102.466533 at 1634283852099
ping akamai.com with results: [42.9,44.9,47.7,49.9,52.8]
Received bandwidth report: Rx: 707.213786, Tx: 43.338661 at 1634283853100
Received bandwidth report: Rx: 1783.322677, Tx: 69.046953 at 1634283854101
Received bandwidth report: Rx: 1181.783217, Tx: 36.843157 at 1634283855102
Received bandwidth report: Rx: 1666.090909, Tx: 88.775225 at 1634283856103
ping one.one.one.one with results: [39.7,43.3,44.0,50.0,57.3]
Received bandwidth report: Rx: 942.753247, Tx: 39.353646 at 1634283857104
ping google.com with results: [44.0,46.4,51.1,53.7,54.0]
Checking up on things: 1634283859
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 27596Kbit
tc qdisc change root dev veth-lan cake bandwidth 22077Kbit
Received bandwidth report: Rx: 1344.208791, Tx: 19.273726 at 1634283858105
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283860
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 27408Kbit
tc qdisc change root dev veth-lan cake bandwidth 21926Kbit
Received bandwidth report: Rx: 1437.792208, Tx: 22.337662 at 1634283859106
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping gstatic.com with results: [46.8,49.1,50.2,56.7,58.1]
Checking up on things: 1634283861
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 25270Kbit
tc qdisc change root dev veth-lan cake bandwidth 20216Kbit
Received bandwidth report: Rx: 1321.988012, Tx: 25.590410 at 1634283860107
tc qdisc change root dev wan cake bandwidth 25000Kbit
ping quad9.net with results: [176.0,176.0,179.0,179.0,182.0]
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping facebook.com with results: [253.0,260.0,261.0,267.0,270.0]
Checking up on things: 1634283862
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 26733Kbit
tc qdisc change root dev veth-lan cake bandwidth 21387Kbit
Received bandwidth report: Rx: 1067.035964, Tx: 14.969031 at 1634283861108
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping fbcdn.com with results: [265.0,266.0,266.0,267.0,267.0]
Checking up on things: 1634283863
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 27247Kbit
tc qdisc change root dev veth-lan cake bandwidth 21798Kbit
Received bandwidth report: Rx: 818.894106, Tx: 10.707293 at 1634283862109
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283864
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 26381Kbit
tc qdisc change root dev veth-lan cake bandwidth 21105Kbit
Received bandwidth report: Rx: 1196.752248, Tx: 12.388611 at 1634283863110
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283865
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 27808Kbit
tc qdisc change root dev veth-lan cake bandwidth 22246Kbit
Received bandwidth report: Rx: 1612.997003, Tx: 15.794206 at 1634283864111
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283866
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 27145Kbit
tc qdisc change root dev veth-lan cake bandwidth 21716Kbit
Received bandwidth report: Rx: 1134.584416, Tx: 17.666334 at 1634283865112
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283867
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
tc qdisc change root dev wan cake bandwidth 25351Kbit
tc qdisc change root dev veth-lan cake bandwidth 20281Kbit
Received bandwidth report: Rx: 1500.595405, Tx: 36.003996 at 1634283866113
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283868
Full Delayed Site List: [{[113,117,97,100,57,46,110,101,116],42.0,1634283838}]
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 27787Kbit
tc qdisc change root dev veth-lan cake bandwidth 22229Kbit
Received bandwidth report: Rx: 1337.538462, Tx: 12.563437 at 1634283867114
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping amazon.com with results: [112.0,113.0,115.0,117.0,122.0]
Checking up on things: 1634283869
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 27748Kbit
tc qdisc change root dev veth-lan cake bandwidth 22198Kbit
Received bandwidth report: Rx: 1649.670330, Tx: 27.512488 at 1634283868115
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283870
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26301Kbit
tc qdisc change root dev veth-lan cake bandwidth 21040Kbit
Received bandwidth report: Rx: 1431.821179, Tx: 25.378621 at 1634283869116
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283871
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26368Kbit
tc qdisc change root dev veth-lan cake bandwidth 21095Kbit
Received bandwidth report: Rx: 666.027972, Tx: 7.206793 at 1634283870117
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283872
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26527Kbit
tc qdisc change root dev veth-lan cake bandwidth 21222Kbit
Received bandwidth report: Rx: 802.356643, Tx: 28.064935 at 1634283871118
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping cloudflare.com with results: [37.2,39.3,45.3,48.7,48.7]
Checking up on things: 1634283873
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26808Kbit
tc qdisc change root dev veth-lan cake bandwidth 21446Kbit
ping google.com with results: [49.1,50.1,51.1,56.1,57.4]
Received bandwidth report: Rx: 1310.789211, Tx: 13.230769 at 1634283872119
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping one.one.one.one with results: [41.6,48.1,50.4,50.6,55.0]
Checking up on things: 1634283874
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 25224Kbit
tc qdisc change root dev veth-lan cake bandwidth 20179Kbit
Received bandwidth report: Rx: 1395.594406, Tx: 12.823177 at 1634283873120
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283875
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26771Kbit
tc qdisc change root dev veth-lan cake bandwidth 21417Kbit
Received bandwidth report: Rx: 1854.289710, Tx: 36.299700 at 1634283874121
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
ping gstatic.com with results: [46.2,48.1,48.8,50.6,63.9]
ping quad9.net with results: [178.0,179.0,181.0,182.0,196.0]
Checking up on things: 1634283876
Full Delayed Site List: []
Recent Delayed Site List: []
tc qdisc change root dev wan cake bandwidth 26241Kbit
tc qdisc change root dev veth-lan cake bandwidth 20993Kbit
Received bandwidth report: Rx: 1495.528472, Tx: 25.836164 at 1634283875122
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Checking up on things: 1634283877
Full Delayed Site List: []
Recent Delayed Site List: []

More here:

Here is another run during an active download:

Does this look as expected?

Incidentally, it seems my DNS server: CleanBrowsing Family, which for the curious reader keen to safeguard against obscene material, I would highly recommend, does not like 'dns.google.com', presumably because it has the term 'dns' in it. This gives:

image
image1250×440 28.8 KB

I am working on another patent opposition today, but will keep a close eye on this thread.

63

Will take a look at the logic. It checks the BW usage every second but the delays are checked less often, I think this means the down diffusion is much faster than the up diffusion and it never gets far above the minimum. I'll work out some of the math and see how to balance it. The base functionality is there and working though!

If you want to see what happens try changing the constant to 0.99 in the code that sends a factor message after receiving the bandwidth report. That should slow the downward decline a lot.

64

Great that the base functionality is there!

Here is what happens with 0.99 with an active download saturating the download (still seems to drop to min bandwidth pretty rapidly):

root@OpenWrt:/etc/init.d# erl -pa /root/sqmfeedback -eval 'sqmfeedback:main().'
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]

Eshell V11.0  (abort with ^G)
1> Received bandwidth report: Rx: -22051.727273, Tx: -2022.181818 at 1634309190904
tc qdisc change root dev wan cake bandwidth 29700Kbit
ping cloudflare.com with results: [46.2,62.3,87.4,113.0,119.0]
ping gstatic.com with results: [63.2,68.4,111.0,124.0,141.0]
ping one.one.one.one with results: [46.4,51.1,90.2,103.0,122.0]
ping google.com with results: [73.0,75.1,100.0,130.0,155.0]
ping amazon.com with results: [119.0,126.0,140.0,149.0,265.0]
ping akamai.com with results: [44.7,62.7,64.2,77.2,158.0]
tc qdisc change root dev veth-lan cake bandwidth 39600Kbit
ping quad9.net with results: [191.0,197.0,206.0,213.0,284.0]
ping fbcdn.com with results: [261.0,261.0,274.0,307.0,334.0]
ping facebook.com with results: [262.0,267.0,268.0,302.0,324.0]
Received bandwidth report: Rx: 1845.914000, Tx: 88.232000 at 1634309191904
tc qdisc change root dev wan cake bandwidth 29403Kbit
tc qdisc change root dev veth-lan cake bandwidth 39204Kbit
Received bandwidth report: Rx: 2008.251748, Tx: 100.411588 at 1634309192905
tc qdisc change root dev wan cake bandwidth 29109Kbit
tc qdisc change root dev veth-lan cake bandwidth 38812Kbit
Received bandwidth report: Rx: 2450.651349, Tx: 110.525475 at 1634309193906
tc qdisc change root dev wan cake bandwidth 28818Kbit
tc qdisc change root dev veth-lan cake bandwidth 38424Kbit
Received bandwidth report: Rx: 2799.244755, Tx: 119.228771 at 1634309194907
tc qdisc change root dev wan cake bandwidth 28530Kbit
tc qdisc change root dev veth-lan cake bandwidth 38040Kbit
Received bandwidth report: Rx: 3747.994012, Tx: 189.798403 at 1634309195909
tc qdisc change root dev wan cake bandwidth 28245Kbit
tc qdisc change root dev veth-lan cake bandwidth 37660Kbit
Received bandwidth report: Rx: 3904.488000, Tx: 173.432000 at 1634309196909
tc qdisc change root dev wan cake bandwidth 27963Kbit
tc qdisc change root dev veth-lan cake bandwidth 37283Kbit
Received bandwidth report: Rx: 4323.506494, Tx: 386.677323 at 1634309197910
tc qdisc change root dev wan cake bandwidth 27683Kbit
tc qdisc change root dev veth-lan cake bandwidth 36910Kbit
Received bandwidth report: Rx: 4085.478521, Tx: 388.819181 at 1634309198911
tc qdisc change root dev wan cake bandwidth 27406Kbit
tc qdisc change root dev veth-lan cake bandwidth 36541Kbit
Received bandwidth report: Rx: 4008.425574, Tx: 383.564436 at 1634309199912
tc qdisc change root dev wan cake bandwidth 27132Kbit
tc qdisc change root dev veth-lan cake bandwidth 36176Kbit
Received bandwidth report: Rx: 4178.517483, Tx: 399.692308 at 1634309200913
tc qdisc change root dev wan cake bandwidth 26861Kbit
tc qdisc change root dev veth-lan cake bandwidth 35814Kbit
Received bandwidth report: Rx: 4216.523477, Tx: 378.354645 at 1634309201914
tc qdisc change root dev wan cake bandwidth 26592Kbit
tc qdisc change root dev veth-lan cake bandwidth 35456Kbit
ping amazon.com with results: [123.0,130.0,131.0,132.0,144.0]
Received bandwidth report: Rx: 3405.412587, Tx: 331.392607 at 1634309202915
tc qdisc change root dev wan cake bandwidth 26326Kbit
tc qdisc change root dev veth-lan cake bandwidth 35101Kbit
Received bandwidth report: Rx: 3555.822178, Tx: 278.603397 at 1634309203916
tc qdisc change root dev wan cake bandwidth 26063Kbit
tc qdisc change root dev veth-lan cake bandwidth 34750Kbit
ping akamai.com with results: [46.8,54.5,67.3,95.4,135.0]
Received bandwidth report: Rx: 2185.218781, Tx: 195.174825 at 1634309204917
tc qdisc change root dev wan cake bandwidth 25802Kbit
tc qdisc change root dev veth-lan cake bandwidth 34403Kbit
ping cloudflare.com with results: [39.8,43.0,48.2,51.1,55.4]
Received bandwidth report: Rx: 2738.347652, Tx: 243.678322 at 1634309205918
tc qdisc change root dev wan cake bandwidth 25544Kbit
tc qdisc change root dev veth-lan cake bandwidth 34059Kbit
ping quad9.net with results: [224.0,232.0,248.0,256.0,286.0]
Received bandwidth report: Rx: 3188.889111, Tx: 273.388611 at 1634309206919
tc qdisc change root dev wan cake bandwidth 25289Kbit
tc qdisc change root dev veth-lan cake bandwidth 33718Kbit
ping one.one.one.one with results: [61.2,86.0,104.0,136.0,150.0]
Received bandwidth report: Rx: 3190.881119, Tx: 282.335664 at 1634309207920
tc qdisc change root dev wan cake bandwidth 25036Kbit
tc qdisc change root dev veth-lan cake bandwidth 33381Kbit
Received bandwidth report: Rx: 1706.281718, Tx: 153.748252 at 1634309208921
ping google.com with results: [115.0,151.0,228.0,284.0,356.0]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 33047Kbit
Received bandwidth report: Rx: 3104.195804, Tx: 182.273726 at 1634309209922
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 32717Kbit
ping fbcdn.com with results: [328.0,363.0,378.0,406.0,423.0]
Received bandwidth report: Rx: 2943.246753, Tx: 159.610390 at 1634309210923
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 32390Kbit
Received bandwidth report: Rx: 2940.309690, Tx: 190.537463 at 1634309211924
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 32066Kbit
Received bandwidth report: Rx: 3040.463536, Tx: 193.482517 at 1634309212925
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 31745Kbit
Received bandwidth report: Rx: 3246.587413, Tx: 286.167832 at 1634309213926
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 31428Kbit
Received bandwidth report: Rx: 3172.079920, Tx: 257.098901 at 1634309214927
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 31114Kbit
Received bandwidth report: Rx: 3263.811189, Tx: 292.634366 at 1634309215928
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 30803Kbit
ping gstatic.com with results: [66.1,69.2,73.4,80.0,129.0]
Received bandwidth report: Rx: 2933.728272, Tx: 283.190809 at 1634309216929
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 30495Kbit
Received bandwidth report: Rx: 2800.861139, Tx: 259.220779 at 1634309217930
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 30190Kbit
Received bandwidth report: Rx: 3765.487026, Tx: 279.227545 at 1634309218932
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 29888Kbit
ping facebook.com with results: [262.0,268.0,269.0,295.0,341.0]
Received bandwidth report: Rx: 2399.318000, Tx: 191.680000 at 1634309219932
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 29589Kbit
Received bandwidth report: Rx: 2816.977023, Tx: 244.141858 at 1634309220933
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 29293Kbit
Received bandwidth report: Rx: 3246.547453, Tx: 271.924076 at 1634309221934
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 29000Kbit
Received bandwidth report: Rx: 3158.257742, Tx: 253.070929 at 1634309222935
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 28710Kbit
Received bandwidth report: Rx: 2744.837163, Tx: 213.090909 at 1634309223936
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 28423Kbit
Received bandwidth report: Rx: 2060.199800, Tx: 190.297702 at 1634309224937
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 28139Kbit
ping akamai.com with results: [44.4,44.6,44.9,47.2,49.8]
Received bandwidth report: Rx: 1959.834166, Tx: 163.060939 at 1634309225938
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 27858Kbit
ping quad9.net with results: [174.0,179.0,185.0,187.0,188.0]
Received bandwidth report: Rx: 2094.232767, Tx: 169.694306 at 1634309226939
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 27579Kbit
ping one.one.one.one with results: [46.9,49.3,52.9,55.8,72.6]
Received bandwidth report: Rx: 2647.769231, Tx: 232.650350 at 1634309227940
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 27303Kbit
Received bandwidth report: Rx: 3433.914086, Tx: 320.545455 at 1634309228941
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 27030Kbit
Checking up on things: 1634309230
Full Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
Recent Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 24702Kbit
Received bandwidth report: Rx: 3197.958042, Tx: 286.417582 at 1634309229942
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 24455Kbit
ping cloudflare.com with results: [47.1,49.3,49.3,52.1,53.3]
Checking up on things: 1634309231
Full Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
Recent Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 21412Kbit
Received bandwidth report: Rx: 2593.716284, Tx: 228.051948 at 1634309230943
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 21198Kbit
ping facebook.com with results: [260.0,260.0,262.0,262.0,272.0]
Checking up on things: 1634309232
Full Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
Recent Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 2511.702595, Tx: 187.730539 at 1634309231945
ping amazon.com with results: [122.0,122.0,125.0,127.0,127.0]
Checking up on things: 1634309233
Full Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205},{[97,109,97,122,111,110,46,99,111,109],139.0,1634309203}]
Recent Delayed Site List: [{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 2109.320000, Tx: 180.154000 at 1634309232945
ping fbcdn.com with results: [260.0,263.0,264.0,266.0,267.0]
Checking up on things: 1634309234
Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205}]
Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 2147.076923, Tx: 171.262737 at 1634309233946
Checking up on things: 1634309235
Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206},{[97,107,97,109,97,105,46,99,111,109],103.5,1634309205}]
Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 1601.502498, Tx: 124.893107 at 1634309234947
Checking up on things: 1634309236
Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207},{[99,108,111,117,100,102,108,97,114,101,46,99,111,109],72.8,1634309206}]
Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 1580.815185, Tx: 128.863137 at 1634309235948
ping google.com with results: [62.5,65.3,66.2,66.7,67.4]
Checking up on things: 1634309237
Full Delayed Site List: [{[103,111,111,103,108,101,46,99,111,109],84.8,1634309237},{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208},{[113,117,97,100,57,46,110,101,116],80.0,1634309207}]
Recent Delayed Site List: [{[103,111,111,103,108,101,46,99,111,109],84.8,1634309237},{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 1604.095904, Tx: 144.039960 at 1634309236949
Checking up on things: 1634309238
Full Delayed Site List: [{[103,111,111,103,108,101,46,99,111,109],84.8,1634309237},{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],88.8,1634309208}]
Recent Delayed Site List: [{[103,111,111,103,108,101,46,99,111,109],84.8,1634309237},{[102,98,99,100,110,46,99,111,109],71.0,1634309234},{[111,110,101,46,111,110,101,46,111,110,101,46,111,110,101],51.9,1634309228},{[113,117,97,100,57,46,110,101,116],47.0,1634309227},{[97,107,97,109,97,105,46,99,111,109],22.4,1634309226},{[102,97,99,101,98,111,111,107,46,99,111,109],74.0,1634309220},{[103,115,116,97,116,105,99,46,99,111,109],72.6,1634309217},{[102,98,99,100,110,46,99,111,109],149.0,1634309211},{[103,111,111,103,108,101,46,99,111,109],256.0,1634309209}]
tc qdisc change root dev wan cake bandwidth 25000Kbit
tc qdisc change root dev veth-lan cake bandwidth 20000Kbit
Received bandwidth report: Rx: 1722.679321, Tx: 131.768232 at 1634309237950

BREAK: (a)bort (A)bort with dump (c)ontinue (p)roc info (i)nfo
       (l)oaded (v)ersion (k)ill (D)b-tables (d)istribution

What do the values in the lines:

Received bandwidth report: Rx: 2943.246753, Tx: 159.610390 at 1634309210923

indicate?

65

This says you're using 2.9 Mbps down and 159kbps up at that time.

66

Ah, but at that time I had a download going at full pelt - so 20Mbit/s or so. So is it that the bandwidth that is getting determined is out for some reason?

67

aha, yes it's bytes vs bits! thanks I just realized... I'll fix that.

Ok I pushed a version that multiplies by 8 appropriately it's in devel... still haven't adjusted the time constant so try this first, then we can adjust the time constant.

1 Like
68

I started download a little after running and this is what I see now:

So bandwidth looks more sensible now. But still just converges down to min and stays there (with active download saturating download at the min 20MBit/s) and then calls tc qdisc change every second at min bandwidth. This is with the factor at 0.8.

69

So, it's actually behaving as designed... It hits the low end, and then as the bandwidth gets above 1/2 the current value it stops lowering... Every time it looks for a delay (which is every 40 seconds) it will see no delay and raise the level... but if the streams don't rise fast enough... then after a while it will be below the halfway mark and will start lowering again. So, it's now about balancing the math...

Right now it checks the bandwidth usage every 1 second, but it checks the delays every 40 seconds or so. Checking for delays more often is a problem because we don't want to ping-flood things, but I could slow down how fast it reacts to bandwidth ... like I could only send the message to down-scale 1/10 or 1/20 of the time (randomly using an RNG).

The big issue is it takes time for downloads to notice they can get more bandwidth (TCP "slow start") so upping the bandwidth on cake doesn't immediately make the download go faster... it has to climb slowly... as it's climbing, the bandwidth checker can clamp it back down.

As I suspected there is some interaction between TCP bandwidth finding and this "throttle control". So we're going to have to come to a balance probably mostly by testing. I'm going to add a random number generator to send the bandwidth scaling only some of the time... you can then tweak the numbers...

70

Ok, I pushed a version that uses a random number to determine how often to actually scale down the bandwidth. I'm starting with 0.05 which is once every 20 seconds on average. you can tweak the frequency and the degree of scaling (the 0.99 factor) and try to get something that performs well for you. It will depend on the behavior of TCP scaling and such so it's not very easy to just derive a good balance by first principles without testing.

71

Great thanks. And thank you for the helpful explanations. If anyone has any thoughts on guesses about these constants please chip in, and otherwise I will experiment with different values.

72

I'd suggest to have the bandwidth decay be slow-ish... so like on average it reduces by 50% every 3 minutes or something... If we stick with 0.05 as the random constant, then we want x^(360.05) = 1/2 so x = 0.926 or so. Try that to start with... if you find too much jitter because you're only checking every 20 seconds... you could do something like the random constant 0.1 and then

x^(3600.1) = 1/2, so x = 0.962

I think the point is that after a "long time" (minutes and minutes) of not utilizing much bandwidth, the whole thing decays to the minimum, but it should be not super aggressive.

We might need to tweak it a few other ways, but start with this.

1 Like
73

Here is a run based on the devel branch with only tweaks at the end:

Are you able to discern what is happening from this? Something doesn't seem right. The reported rx and tx are low but it stays on max bandwidth. Is it supposed to call tc qdisc change every second? Ultimately it crashes and stops calling tc qdisc change.

=CRASH REPORT==== 15-Oct-2021::18:36:56.658126 ===
  crasher:
    initial call: sqmfeedback:adjuster/1
    pid: <0.80.0>
    registered_name: []
    exception error: emfile
      in function  open_port/2
         called as open_port({spawn,"/bin/sh -s unix:cmd"},
                             [binary,stderr_to_stdout,stream,in,hide,out])
      in call from os:cmd/2 (os.erl, line 275)
      in call from sqmfeedback:new_bandwidth/2 (sqmfeedback.erl, line 39)
      in call from sqmfeedback:adjuster/1 (sqmfeedback.erl, line 180)
    ancestors: [<0.9.0>]
    message_queue_len: 0
    messages: []
    links: [<0.9.0>]
    dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.47293030>,
                                next => #Fun<rand.0.47293030>,type => exsss,
                                uniform => #Fun<rand.1.47293030>,
                                uniform_n => #Fun<rand.2.47293030>},
                              [143092564726364658|84188395106486865]}}]
    trap_exit: false
    status: running
    heap_size: 987
    stack_size: 28
    reductions: 609645
  neighbours:

Checking up on things: 1634323017
Full Delayed Site List: [{[102,97,99,101,98,111,111,107,46,99,111,109],57.0,1634322994}]
Recent Delayed Site List: [{[102,97,99,101,98,111,111,107,46,99,111,109],57.0,1634322994}]
respawning bw usage monitor;
=CRASH REPORT==== 15-Oct-2021::18:36:57.697625 ===
  crasher:
    initial call: sqmfeedback:monitor_bw/2
    pid: <0.82.0>
    registered_name: []
    exception error: no match of right hand side value {error,emfile}
      in function  sqmfeedback:read_bw/1 (sqmfeedback.erl, line 94)
      in call from sqmfeedback:monitor_bw/5 (sqmfeedback.erl, line 102)
    ancestors: [<0.9.0>]
    message_queue_len: 0
    messages: []
    links: [<0.9.0>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 141670
  neighbours:
74

Yeah, I can see what's up. The two different timers are an issue (checking bandwidth vs checking delays). I'll work on a fix.

Ok, as of now it sends the timer ticks to everyone every second... but it only reacts randomly 1/20 of the time on average in both the delay detection, and bandwidth detection. Also the bandwidth detection downgrades bandwidth at 0.85 which is faster than the upgrader can maximally upgrade (1.12*0.85 = .952)

So as long as the used bandwidth is less than 1/2 the current set point, the decay will dominate... once you get above half the set-point, then only upward diffusion will be allowed, and only if there are no detected delays... I think this should work! pushed to devel

As for the crash... I don't know what that's about. It seems to have crashed when it tried to run tc to set cake's settings... but I don't know why. Erlang should just restart that process... If you encounter that again, I'll look at it.

75

Many thanks. I think this is beginning to look healthier now:

right? I think it has kept the bandwidth at a minimum here despite call for more (by me downloading file whilst Netflix is live) because this is evening and during a time of congestion. Earlier today it was uncongested. But at this time I wouldn't be surprised if 25 or so is the limit without bufferbloat. Is that what the the log shows?

.. but .. oh no it crashed again! Will try to recreate with log.

Working on getting the log. It seems that it works well for a good while but after a while of running if I start a download it goes back to tc qdisc change every 1s. Is that what you would expect?

So it's like if low bandwidth I see tc qdisc change infrequently. But then if I start my download I start to see tc qdisc change every 1s.

In any case - bingo! Got it all here from start to crash and showing the rapid tc qdisc change again (when I start download):

76

It looks like it's behaving ok, in particular as you say it's not raising the bandwidth because there are delays to the sites. The big issue is the crash. I'm wondering if that crash really matters, or if it just restarts things that crash. In Erlang one of the principles is "let it fail", so that's a valid thing to do. On the other hand ... why is it crashing? It's like maybe it's out of file descriptors. Do you see a bunch of zombie processes if you do a full ps aux at the command line?

77

Once it crashes it stops issuing any 'tc qdisc change' calls and so the bandwidth just stays constant until it is rerun.

root@OpenWrt:~# ps aux
ps: unrecognized option: a
BusyBox v1.34.1 (2021-10-11 19:54:16 UTC) multi-call binary.

Usage: ps

Show list of processes

        w       Wide output
root@OpenWrt:~# ps
  PID USER       VSZ STAT COMMAND
    1 root      1672 S    /sbin/procd
    2 root         0 SW   [kthreadd]
    3 root         0 IW<  [rcu_gp]
    4 root         0 IW<  [rcu_par_gp]
    6 root         0 IW<  [kworker/0:0H-ev]
    8 root         0 IW<  [mm_percpu_wq]
    9 root         0 SW   [rcu_tasks_trace]
   10 root         0 SW   [ksoftirqd/0]
   11 root         0 IW   [rcu_sched]
   12 root         0 SW   [migration/0]
   13 root         0 SW   [cpuhp/0]
   14 root         0 SW   [cpuhp/1]
   15 root         0 SW   [migration/1]
   16 root         0 SW   [ksoftirqd/1]
   18 root         0 IW<  [kworker/1:0H-ev]
   19 root         0 IW<  [netns]
  318 root         0 SW   [oom_reaper]
  319 root         0 IW<  [writeback]
  320 root         0 SW   [kcompactd0]
  331 root         0 IW<  [pencrypt_serial]
  333 root         0 IW<  [pdecrypt_serial]
  358 root         0 IW<  [kblockd]
  359 root         0 IW<  [blkcg_punt_bio]
  373 root         0 IW<  [ata_sff]
  400 root         0 SW   [watchdogd]
  413 root         0 IW<  [kworker/0:1H-kb]
  421 root         0 SW   [kswapd0]
  424 root         0 IW<  [kthrotld]
  561 root         0 SW   [hwrng]
  590 root         0 IW<  [iscsi_eh]
  594 root         0 IW<  [iscsi_destroy]
  631 root         0 SW   [spi0]
  637 root         0 SW   [spi1]
  714 root         0 IW<  [ipv6_addrconf]
  715 root         0 IW<  [kworker/1:1H-kb]
  716 root         0 IW<  [dsa_ordered]
  804 root         0 SW   [ubi_bgt0d]
  809 root         0 SW   [irq/38-gpio-key]
  810 root         0 SW   [irq/140-gpio-ke]
  826 root         0 IW<  [uas]
  892 root         0 SW   [ubifs_bgt0_5]
  987 ubus      1264 S    /sbin/ubusd
  989 root       836 S    /sbin/askfirst /usr/libexec/login.sh
 1028 root       972 S    /sbin/urngd
 1170 root         0 IW<  [cfg80211]
 1184 root         0 SW   [napi/phy0-5]
 1185 root         0 SW   [napi/phy0-6]
 1189 root         0 SW   [mt76-tx phy0]
 1201 root         0 SW   [napi/phy1-7]
 1202 root         0 SW   [napi/phy1-8]
 1204 root         0 SW   [napi/phy1-9]
 1213 root         0 SW   [mt76-tx phy1]
 1398 logd      1208 S    /sbin/logd -S 64
 1456 root      2312 S    /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1720 root      1032 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 1818 root      2448 S    {hostapd} /sbin/ujail -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c --
 1819 root      2448 S    {wpa_supplicant} /sbin/ujail -n wpa_supplicant -U network -G network -C /etc/capabilities/wp
 1823 network   5848 S    /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1826 network   5988 S    /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1881 root      1876 S    /sbin/netifd
 1943 root      1452 S    /usr/sbin/odhcpd
 2417 root       992 S    odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 wan
 2419 root      1236 S    udhcpc -p /var/run/udhcpc-wan.pid -s /lib/netifd/dhcp.script -f -t 0 -i wan -x hostname:Open
 2774 root         0 IW<  [wg-crypt-vpn]
 3204 root      4924 S    /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 1
 3291 root      1624 S    /usr/bin/dbus-daemon --system
 3318 nobody    2328 S    avahi-daemon: running [OpenWrt.local]
 3377 root      1448 S    /sbin/blockd
 3867 root      1152 S    /usr/sbin/irqbalance -f -t 10
 4697 root      2448 S    {dnsmasq} /sbin/ujail -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/
 4698 dnsmasq   1496 S    /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411
 4820 root      729m S    rclone mount OneDrive:/Scanned Documents/ /tmp/OneDrive/ --use-mmap --buffer-size 0 --cache-
 4948 root     35008 S    /usr/sbin/smbd -F
 4949 root     22120 S    /usr/sbin/nmbd -F
 5008 root      2448 S    {ntpd} /sbin/ujail -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -
 5033 ntp       1236 S    /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.n
 5501 root     33488 S    {smbd-notifyd} /usr/sbin/smbd -F
 5502 root     33496 S    {cleanupd} /usr/sbin/smbd -F
 7587 root         0 IW   [kworker/u4:0-ev]
 8322 root      1052 S    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 8323 root      1256 S    -ash
 8838 root      1052 R    /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 8839 root      1248 S    -ash
10179 root         0 IW   [kworker/1:1-wg-]
10194 root         0 IW   [kworker/1:2-eve]
10212 root         0 IW   [kworker/u4:1-ph]
10213 root         0 IW   [kworker/0:0-wg-]
10221 root         0 IW   [kworker/u4:5-ph]
10230 root         0 IW   [kworker/0:1-wg-]
10241 root         0 IW   [kworker/0:2-wg-]
10242 root         0 IW   [kworker/0:3-wg-]
10243 root         0 IW   [kworker/1:0-wg-]
10244 root         0 IW   [kworker/u4:3-ev]
10261 root      1244 R    ps
root@OpenWrt:~# ps aux
ps: unrecognized option: a
BusyBox v1.34.1 (2021-10-11 19:54:16 UTC) multi-call binary.

Usage: ps

Show list of processes

        w       Wide output
root@OpenWrt:~# free -m
              total        used        free      shared  buff/cache   available
Mem:         493936       65928      347056        3184       80952      375124
Swap:             0           0           0
root@OpenWrt:~#
78

yeah, busybox ps is limited... Ok. let me look at the crash situation... I don't know why it can't run tc, but I think we can restart that thread and move on.

79

Is it expected that it is issuing 'tc qdisc change' calls every 1 second (around line 1200) when I start a download - see the way the seconds counter increments:

1. tc qdisc change root dev wan cake bandwidth 25000Kbit

2. tc qdisc change root dev veth-lan cake bandwidth 20000Kbit

3. Received bandwidth report: Rx: 18005.378621, Tx: 780.091908 at 1634328536942

4. Checking up on things: 1634328538

5. Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

6. Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

7. tc qdisc change root dev wan cake bandwidth 25000Kbit

8. tc qdisc change root dev veth-lan cake bandwidth 20000Kbit

9. Received bandwidth report: Rx: 18804.019980, Tx: 1076.115884 at 1634328537943

10. Checking up on things: 1634328539

11. Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

12. Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

13. tc qdisc change root dev wan cake bandwidth 25000Kbit

14. tc qdisc change root dev veth-lan cake bandwidth 20000Kbit

15. Received bandwidth report: Rx: 16923.508492, Tx: 914.741259 at 1634328538944

16. ping gstatic.com with results: [58.4,58.7,67.9,70.6,71.4]

17. Checking up on things: 1634328540

18. Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

19. Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

20. tc qdisc change root dev wan cake bandwidth 25000Kbit

21. tc qdisc change root dev veth-lan cake bandwidth 20000Kbit

22. Received bandwidth report: Rx: 10560.447552, Tx: 339.164835 at 1634328539945

23. Checking up on things: 1634328541

24. Full Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512},{[103,111,111,103,108,101,46,99,111,109],26.5,1634328511}]

25. Recent Delayed Site List: [{[102,98,99,100,110,46,99,111,109],27.0,1634328536},{[102,97,99,101,98,111,111,107,46,99,111,109],33.0,1634328524},{[97,109,97,122,111,110,46,99,111,109],37.0,1634328521},{[97,107,97,109,97,105,46,99,111,109],48.1,1634328514},{[113,117,97,100,57,46,110,101,116],22.0,1634328512}]

26. tc qdisc change root dev wan cake bandwidth 25000Kbit

27. tc qdisc change root dev veth-lan cake bandwidth 20000Kbit

Could that be related?

80

when there are delays it's currently issuing them every second... but I added the random downscaling factor to that as well, maybe that will help.. pushing...