CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

[Lynx]

@_FailSafe

Does the --clock-skew argument help at all in your case?

Some fascinating stuff on the internet about that.

ccs06hotornot.pdf

Nmap Development: Re: Clock Skew

[moeller0]

@dlakelan

Ok, so I think what I'm suggesting, to be clear, is that at start-up, we use mtr to find the path, and then we probe outward from the router to find the closest reflector that gives fairly reliable regular ping responses.

That is a rationale approach, but that suffers from the fact that intermediate nodes are known to be best effort ICMP reflectors (actually end hosts are best effort, so intermediate hops are "better effort" only ).

Sure, we can add some of those to the mix, but I certainly would include end-hosts as these have a better track record in reflecting (we still need to keep the rate low enough to not be classified as offender (which is also true for intermediate hops, especially if they are shared by a large number of users)). The assumption that a "close" by hop is shared by fewer users is intuitive, but IMHO does not really reflect how ISPs aggregate/terminate customers access links. Some keep it small, some hoard many users (often seen at DOCSIS head-ends).

@dlakelan

if such a package were rolled out at scale to millions of OpenWrt based routers.

That is not going to happen, realistically speaking. This will be something for a few enthusiasts, maybe a few thousand but millions? ATM this script still requires manual intervention/configuration to be useful so I do not see this automatically starting at millions of devices.....

BTW, we need to get enough OWD samples that our baseline is reasonably correct, but for that even every 10 seconds would help.

@dlakelan

Not once tp-link realizes that if they install it by default they can get a competitive edge in responsiveness and hence marketing to tech-know-nothings

Then tp-link better deploy a fleet of reliable enough reflectors for their users ;). As far as I know they do not distribute SQM at all currently.

@dlakelan

No, that's what I'm suggesting. we idle on regular pings to the 2nd or 3rd hop along our path sending them about say every 200ms,

That is already a factor 5 faster than I envision... once per tick seems okay, if we sample enough reflectors concurrently.

@dlakelan

This lowers the load on 9.9.9.9 etc by a factor of 100x

except when load pushes the script to increase rates at which point OWD will be the norm not the exception....

[BlueKingMuch]

@dlakelan

Can someone try a traceroute to 9.9.9.9 and then try hping to each step along the way, see what happens?

unfortunately only quad9 seems to answer fully on my line

root@turris:~# traceroute 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 38 byte packets
 1  100.126.0.244 (100.126.0.244)  9.402 ms  8.798 ms  9.124 ms
 2  100.126.0.46 (100.126.0.46)  8.707 ms  9.031 ms  9.151 ms
 3  as42-vie.pch.net (193.203.0.33)  9.712 ms  9.783 ms  9.552 ms
 4  dns9.quad9.net (9.9.9.9)  9.746 ms !C  8.829 ms !C  9.579 ms !C
root@turris:~#
root@turris:~# hping3 100.126.0.244 --icmp --icmp-ts -i u1000 -c 10
HPING 100.126.0.244 (pppoe-wan 100.126.0.244): icmp mode set, 28 headers + 0 dat                    a bytes

--- 100.126.0.244 hping statistic ---
10 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@turris:~# hping3 100.126.0.46 --icmp --icmp-ts -i u1000 -c 10
HPING 100.126.0.46 (pppoe-wan 100.126.0.46): icmp mode set, 28 headers + 0 data                     bytes

--- 100.126.0.46 hping statistic ---
10 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@turris:~# hping3 193.203.0.33 --icmp --icmp-ts -i u1000 -c 10
HPING 193.203.0.33 (pppoe-wan 193.203.0.33): icmp mode set, 28 headers + 0 data                     bytes

--- 193.203.0.33 hping statistic ---
10 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@turris:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 10
HPING 9.9.9.9 (pppoe-wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=40 ip=9.9.9.9 ttl=61 id=59201 icmp_seq=0 rtt=29.5 ms
ICMP timestamp: Originate=60139973 Receive=60139978 Transmit=60139978
ICMP timestamp RTT tsrtt=29

len=40 ip=9.9.9.9 ttl=61 id=59202 icmp_seq=1 rtt=28.9 ms
ICMP timestamp: Originate=60139974 Receive=60139979 Transmit=60139979
ICMP timestamp RTT tsrtt=29

len=40 ip=9.9.9.9 ttl=61 id=59203 icmp_seq=2 rtt=28.2 ms
ICMP timestamp: Originate=60139975 Receive=60139980 Transmit=60139980
ICMP timestamp RTT tsrtt=28

len=40 ip=9.9.9.9 ttl=61 id=59204 icmp_seq=3 rtt=27.5 ms
ICMP timestamp: Originate=60139976 Receive=60139981 Transmit=60139981
ICMP timestamp RTT tsrtt=28

len=40 ip=9.9.9.9 ttl=61 id=59205 icmp_seq=4 rtt=26.9 ms
ICMP timestamp: Originate=60139977 Receive=60139982 Transmit=60139982
ICMP timestamp RTT tsrtt=27

len=40 ip=9.9.9.9 ttl=61 id=59206 icmp_seq=5 rtt=26.2 ms
ICMP timestamp: Originate=60139978 Receive=60139983 Transmit=60139983
ICMP timestamp RTT tsrtt=26

len=40 ip=9.9.9.9 ttl=61 id=59207 icmp_seq=6 rtt=25.5 ms
ICMP timestamp: Originate=60139979 Receive=60139984 Transmit=60139984
ICMP timestamp RTT tsrtt=26

len=40 ip=9.9.9.9 ttl=61 id=59208 icmp_seq=7 rtt=24.9 ms
ICMP timestamp: Originate=60139980 Receive=60139985 Transmit=60139985
ICMP timestamp RTT tsrtt=25

len=40 ip=9.9.9.9 ttl=61 id=59209 icmp_seq=8 rtt=24.2 ms
ICMP timestamp: Originate=60139981 Receive=60139986 Transmit=60139986
ICMP timestamp RTT tsrtt=24

len=40 ip=9.9.9.9 ttl=61 id=59210 icmp_seq=9 rtt=23.6 ms
ICMP timestamp: Originate=60139982 Receive=60139987 Transmit=60139987
ICMP timestamp RTT tsrtt=24


--- 9.9.9.9 hping statistic ---
10 packets tramitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 23.6/26.5/29.5 ms

[BlueKingMuch]

@Lynx

it doesn't seem to work in the WSL2 environment in Windows

I used a Hyper-V instance of a Server - 64bit Ubuntu 14.04LTS which is recommended in turris' forums for compiling/building stuff for their customized openWRT, but it should be good for plain openWRT also

I just had to upgrade the "tar" program manually to 1.28 because the git integration in build toolchain needs it

[Lynx]

Managed to compile using the same Ubuntu in WSL2 using the SDK for my particular snapshot.

[hnyman]

@dlakelan

@Lynx

I think to use this script you have to be somewhere pretty high on the spectrum.

Not once tp-link realizes that if they install it by default they can get a competitive edge in responsiveness and hence marketing to tech-know-nothings

We still do not know who used OpenWrt download servers for something like that...
The DDoS aspect is real.

Note the below thread from Q1/2019

TL-WR1043ND snapshot images - High download numbers - Spanish users needed

The issue Quite unusual issue where we need the help of spanish residents: Since 30.09.2018 we see abnormally high download numbers for the TL-WR1043ND snapshot factory image in the download stats, which create a tremendous amount of traffic. [grafik] See also https://downloads.openwrt.org/stats/#countries [grafik] Analysis Analysis of the download servers logfiles revealed that 99,7% of the 1043nd snapshot downloads originate from Spain 75% Wget; 25% uclient-fetch both without version …

[moeller0]

Mmmh, I managed to disable the NTP client on my router completely... after enabing and forcing an NTP time update, the offset towards 9.9.9.9 becomes believably small again.

@tievolu thank you for demonstrating how well ICMP timestamps can work! Without your proof of principle, I for one would have still thought these timestamps to be completely useless (based on some cursory testing I performed a few years ago). I am also amazed how many end-nodes are willing to supply their time!

[Lynx]

Haha that was a funny read! Makes me feel slightly guilty for targeting certain Linux .ISO downloads to help saturate my connection. Next time I will target something I don't like. Maybe Scottish nationalist propaganda videos.

We should be circumspect with pings.

[BlueKingMuch]

I just found out that my ISP's official DNS Servers do reply with icmp timestamps, maybe the local ISP's DNS Servers are also good targets for measurement for others?

[dlakelan]

@BlueKingMuch

maybe the local ISP's DNS Servers are also good targets for measurement for others?

That's also a good candidate, at least you have a contract with your ISP to provide network services, you have no such contract with quad 9 etc. Also presumably using the ISP equipment means you aren't going over any peering links etc.

Another thought I had was to somehow use a DDNS type solution. Let each router register their IP address with a dynamic dns type service, and then have the pings go to a selection of these routers. so the systems using the pings also provide the reflectors.

We should also NOT hard-code IP addresses. Some ISPs could be IPv6 only for example. If we're going to use quad9 let's use: dns9.quad9.net

similarly for other services: dns.google and one.one.one.one

[moeller0]

@dlakelan

We should also NOT hard-code IP addresses. Some ISPs could be IPv6 only for example. If we're going to use quad9 let's use: dns9.quad9.net

similarly for other services: dns.google and one.one.one.one

But then we rely on our local DNS server caching these, otherwise the name lookup will make a single call quite costly....

[dlakelan]

True, I'm thinking we just run pings at the right frequency though. once the ping is started we just leave it running?

[Lochnair]

Looks like my ISP (Telenor) filters ICMP timestamp requests, so no with their DNS servers for me.

[moeller0]

That is not what we do right now... the challenge is getting the most recent value out of a running ping instance in shell... so ATM ping is run for a number of samples and its output massages and stored to disk (for multiple ping instances) and then that disk file is massages to get the minimum over all instances. The script does a bit more and then sleeps until the next tick time, rinse and repeat...
Doing a remote name lookup every tick is not going to be helpful.... so if we want symbolic names the script should convert these into IP addresses early on and then only handle those addresses, no? For the time being, IP addresses sound fine, but for bigger deployment I believe your idea is relevant and worth thinking over.

[dlakelan]

I guess resolving names early is probably good enough.

I do think ultimately lua is the better choice here. shell is just too limited for doing "smart" things.

[moeller0]

Something slightly off-topic, but the great @dtaht alerted me to the existence of gping which is a nice tool to show ping RTTs in realtime in a terminal window, pretty nifty to run parallel while testing the autoratescript to get some visible insight in what is happening second by second.

[dtaht]

Over here, in part based on all this enthusiastic research on this thread, I started drafting a design for a new tool that would combine a load test with icmp unreachable messages. https://github.com/dtaht/wtbb . I thought "I will damned" if I'll write it in C (using libcurl as a base), but it turned out I'd be equally damned if I tried to write it in rust, and in part, because qping was written in rust it seemed attractive as a starting point or influencer as to how that exercise went.

So I'm still floating around that portion of the solution space, I have a tendency to write the spec and man page first, while trying to find the least number of libs to leverage - notably looking over the sources to tcptraceroute.

libcurl is really excellent as to how you can instrument it, nothing in rust comes close, or perhaps I don't understand rust well enough - or both! Doing threads is not really necessary (I think), but rust does those well. It's really remarkable how short the basic code could be, with tokio, compared to a C version...

[Lynx]

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes

len=46 ip=9.9.9.9 ttl=54 id=47019 icmp_seq=0 rtt=89.5 ms
ICMP timestamp: Originate=82696653 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=90

len=46 ip=9.9.9.9 ttl=54 id=47021 icmp_seq=2 rtt=87.5 ms
ICMP timestamp: Originate=82696656 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=87

len=46 ip=9.9.9.9 ttl=54 id=47020 icmp_seq=1 rtt=88.8 ms
ICMP timestamp: Originate=82696655 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=88

Any major awk experts here able to determine for each result:

uplink time = Receive - Originate
downlink time = Originate + rtt - Transmit

And then determine the minimum across the matches?

Something like multi-line match on rtt=X Originate=Y Receive=Z, do the calculations, and keep track of the minimum for each match. And then just output the minimums.

It's possible but seems to be a nightmare in awk

[Lochnair]

@moeller0

Something slightly off-topic, but the great @dtaht alerted me to the existence of gping which is a nice tool to show ping RTTs in realtime in a terminal window, pretty nifty to run parallel while testing the autoratescript to get some visible insight in what is happening second by second.

gping is pretty sweet. I'll usually always have it running in a workspace

@dtaht

libcurl is really excellent as to how you can instrument it, nothing in rust comes close, or perhaps I don't understand rust well enough - or both! Doing threads is not really necessary (I think), but rust does those well. It's really remarkable how short the basic code could be, with tokio, compared to a C version...

Rust is nice, but no support in master for it yet it would seem

Rust-lang (rustc/cargo) for OpenWrt - testing needed

It is in Draft PR status because it needs more wide-range testing. It works for the devices I have to actually test it, and overall should probably be redone to slim it down and see what other ways I can do it. It's still a WIP but you're welcome to test it.

@_FailSafe

I'm very willing to help test and confirm the patch(es) once ready. Anything I can do to help, let me know!

Thanks! Probably gonna be tomorrow or Friday depending on what needs doing IRL

[Lochnair]

@Lynx

This seems to be a nightmare in awk

I'm very tempted make a Lua port of your script and continue this work on that just avoid these headaches