CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

moeller0 · 2021-12-08T20:38:17.172Z

That is not what we do right now... the challenge is getting the most recent value out of a running ping instance in shell... so ATM ping is run for a number of samples and its output massages and stored to disk (for multiple ping instances) and then that disk file is massages to get the minimum over all instances. The script does a bit more and then sleeps until the next tick time, rinse and repeat...
Doing a remote name lookup every tick is not going to be helpful.... so if we want symbolic names the script should convert these into IP addresses early on and then only handle those addresses, no? For the time being, IP addresses sound fine, but for bigger deployment I believe your idea is relevant and worth thinking over.

dlakelan · 2021-12-08T20:45:01.798Z

I guess resolving names early is probably good enough.

I do think ultimately lua is the better choice here. shell is just too limited for doing "smart" things.

moeller0 · 2021-12-08T21:22:26.098Z

Something slightly off-topic, but the great @dtaht alerted me to the existence of gping which is a nice tool to show ping RTTs in realtime in a terminal window, pretty nifty to run parallel while testing the autoratescript to get some visible insight in what is happening second by second.

dtaht · 2021-12-08T21:41:03.470Z

Over here, in part based on all this enthusiastic research on this thread, I started drafting a design for a new tool that would combine a load test with icmp unreachable messages. https://github.com/dtaht/wtbb . I thought "I will damned" if I'll write it in C (using libcurl as a base), but it turned out I'd be equally damned if I tried to write it in rust, and in part, because qping was written in rust it seemed attractive as a starting point or influencer as to how that exercise went.

So I'm still floating around that portion of the solution space, I have a tendency to write the spec and man page first, while trying to find the least number of libs to leverage - notably looking over the sources to tcptraceroute.

libcurl is really excellent as to how you can instrument it, nothing in rust comes close, or perhaps I don't understand rust well enough - or both! Doing threads is not really necessary (I think), but rust does those well. It's really remarkable how short the basic code could be, with tokio, compared to a C version...

Lynx · 2021-12-08T23:04:58.642Z

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes

len=46 ip=9.9.9.9 ttl=54 id=47019 icmp_seq=0 rtt=89.5 ms
ICMP timestamp: Originate=82696653 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=90

len=46 ip=9.9.9.9 ttl=54 id=47021 icmp_seq=2 rtt=87.5 ms
ICMP timestamp: Originate=82696656 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=87

len=46 ip=9.9.9.9 ttl=54 id=47020 icmp_seq=1 rtt=88.8 ms
ICMP timestamp: Originate=82696655 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=88

Any major awk experts here able to determine for each result:

uplink time = Receive - Originate
downlink time = Originate + rtt - Transmit

And then determine the minimum across the matches?

Something like multi-line match on rtt=X Originate=Y Receive=Z, do the calculations, and keep track of the minimum for each match. And then just output the minimums.

It's possible but seems to be a nightmare in awk

Lochnair · 2021-12-08T23:06:21.340Z

moeller0:

Something slightly off-topic, but the great @dtaht alerted me to the existence of gping which is a nice tool to show ping RTTs in realtime in a terminal window, pretty nifty to run parallel while testing the autoratescript to get some visible insight in what is happening second by second.

gping is pretty sweet. I'll usually always have it running in a workspace

dtaht:

libcurl is really excellent as to how you can instrument it, nothing in rust comes close, or perhaps I don't understand rust well enough - or both! Doing threads is not really necessary (I think), but rust does those well. It's really remarkable how short the basic code could be, with tokio, compared to a C version...

Rust is nice, but no support in master for it yet it would seem

Rust-lang (rustc/cargo) for OpenWrt - testing needed Community Builds, Projects & Packages

It is in Draft PR status because it needs more wide-range testing. It works for the devices I have to actually test it, and overall should probably be redone to slim it down and see what other ways I can do it. It's still a WIP but you're welcome to test it.

_FailSafe:

I'm very willing to help test and confirm the patch(es) once ready. Anything I can do to help, let me know!

Thanks! Probably gonna be tomorrow or Friday depending on what needs doing IRL

Lochnair · 2021-12-08T23:08:23.671Z

Lynx:

This seems to be a nightmare in awk

I'm very tempted make a Lua port of your script and continue this work on that just avoid these headaches

Lynx · 2021-12-08T23:09:29.621Z

Honestly I think this is the only really hard bit...

@vgaetera I think this might be one for you? Seems like a really tough awk mult-line match with regexp, calculations and keeping track of minimums across matches.

@moeller0 is this beyond you as well? I can probably crack it if I spend an entire day.

dlakelan · 2021-12-08T23:16:39.574Z

Lochnair:

I'm very tempted make a Lua port of your script and continue this work on that just avoid these headaches

DO IT! This needs to be in Lua. I am happy to help but don't have the bandwidth for a full port.

dlakelan · 2021-12-08T23:37:24.492Z

Lynx:

icmp_seq=2

Anyone else notice that these example pings came back out of order?

dlakelan · 2021-12-08T23:41:30.064Z

Lochnair:

Rust is nice, but no support in master for it yet it would seem

I think that's for the compiler, but can't rust output standalone binaries?

Lynx:

Something like multi-line match on rtt=X

I think you match on rtt= and then in the body you read two more lines and pull them apart. Using getline

O’Reilly Online Learning

Effective awk Programming, 4th Edition

Chapter 4. Reading Input Files In the typical awk program, awk reads all input either from the standard input (by default, this is the keyboard, but often it is a … - Selection from Effective awk Programming, 4th Edition [Book]

interestingly another option for a small but nontrivial programming language is javascript via https://duktape.org/

but not currently in OpenWrt. what would it take to get it included?

Nevermind, reading up on it, it seems like it's for embedding in a C/C++ program, so we could build a "hello world" that included it and then write the logic in javascript.

_FailSafe · 2021-12-09T00:20:47.668Z

I started a lua port of this script the other day when I was asking about alternate languages (after others mentioned python was too heavy for smaller resource hardware). I ditched the effort and actually tossed the work when it sounded like shell was the preferred path forward. Bummer.

I’m no lua expert, but willing to get my hands dirty with it again. You’ll probably develop faster, but I’ll be willing to contribute where possible.

Lochnair · 2021-12-09T01:05:41.340Z

Lynx:

Honestly I think this is the only really hard bit...

Fair enough, I just imagine I'd probably spend more time on figuring out how to make awk parse the output, than just doing a port

dlakelan:

DO IT! This needs to be in Lua. I am happy to help but don't have the bandwidth for a full port.

_FailSafe:

I started a lua port of this script the other day when I was asking about alternate languages (after others mentioned python was too heavy for smaller resource hardware). I ditched the effort and actually tossed the work when it sounded like shell was the preferred path forward. Bummer.

Ah damn, well I'll let you know when I have anything worth looking at.

dlakelan:

I think that's for the compiler, but can't rust output standalone binaries?

Found a stackoverflow answer about from a few years back, at the time at least Rust statically linked everything except libc. And I'm sure it can be done easily now. I was mostly asking with getting an offical OpenWrt package made in mind. I assume just downloading the binaries into a package would be frowned upon.

dlakelan:

interestingly another option for a small but nontrivial programming language is javascript via https://duktape.org/

but not currently in OpenWrt. what would it take to get it included?

Nevermind, reading up on it, it seems like it's for embedding in a C/C++ program, so we could build a "hello world" that included it and then write the logic in javascript.

Looks pretty neat, and yeah we'd probably need something written in C to set up the engine and get things running, but that should be easy enough I imagine.

Had a quick look at what Alpine does to compile it, should be simple enough
https://git.alpinelinux.org/aports/tree/community/duktape/APKBUILD
And if it works for Alpine without patches, we shouldn't have trouble on OpenWrt with targetting musl libc either.

_FailSafe · 2021-12-09T05:45:20.360Z

Lynx:

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes

len=46 ip=9.9.9.9 ttl=54 id=47019 icmp_seq=0 rtt=89.5 ms
ICMP timestamp: Originate=82696653 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=90

len=46 ip=9.9.9.9 ttl=54 id=47021 icmp_seq=2 rtt=87.5 ms
ICMP timestamp: Originate=82696656 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=87

len=46 ip=9.9.9.9 ttl=54 id=47020 icmp_seq=1 rtt=88.8 ms
ICMP timestamp: Originate=82696655 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=88
Any major awk experts here able to determine for each result:

uplink time = Receive - Originate
downlink time = Originate + rtt - Transmit

And then determine the minimum across the matches?

Something like multi-line match on rtt=X Originate=Y Receive=Z, do the calculations, and keep track of the minimum for each match. And then just output the minimums.

It's possible but seems to be a nightmare in awk

I think I have it working for you:

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
    awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
    { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
    rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
    uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
    min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
    min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
    END { print min_uplink_time, min_downlink_time }'
6 12.9

If you want to confirm the results are true, you can run this version for a test:

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
    awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
    { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
    rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
    uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
    min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
    min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
    { print uplink_time, downlink_time, min_uplink_time, min_downlink_time }'
7 12.9 7 12.9
6 12.9 6 12.9
14 23.8 6 12.9

The first command should give you just the final min_uplink_time and min_downlink_time across all three executions. If you need the final values delimited with something other than a space, that's an easy mod to the awk statement.

In the second command:

First column is uplink_time for the current record set
Second column is downlink_time for the current record set
Third column is overall min_uplink_time
Fourth column is overall min_downlink_time

Update...
Ran another test with more iterations to further confirm the overall min logic holds. Let me know if you see any issues I need to fix:

root@OpenWrt:~# time hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 20 2> /dev/null | tail -n+2  | \
    awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
    { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); rx=$11; sub(/Receive=/, "", rx); \
    tx=$12; sub(/Transmit=/, "", tx); uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
    min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
    min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
    { print uplink_time, downlink_time, min_uplink_time, min_downlink_time }'
13 26.9 13 26.9
17 21.9 13 21.9
16 21.8 13 21.8
15 21.8 13 21.8
14 21.7 13 21.7
13 21.6 13 21.6
12 21.5 12 21.5
11 21.5 11 21.5
10 21.4 10 21.4
9 21.3 9 21.3
8 21.3 8 21.3
7 21.2 7 21.2
6 21.2 6 21.2
10 16.1 6 16.1
8 17 6 16.1
7 17 6 16.1
6 16.9 6 16.1
5 16.8 5 16.1
4 16.8 4 16.1
13 26.6 4 16.1

real	0m0.244s
user	0m0.005s
sys	0m0.000s

moeller0 · 2021-12-09T08:14:37.695Z

Pretty cool! I think we should be better off reducing the number of samples an increasing the delay though.

Here is a slight modification that also prints out the last valid originate timestamp (this strictly only makes sense for -c 1, but for sanity checking, I believe having any real timestamp from the run there is helpful), and allows one default out of bounds value:

root@turris:~# hping3 1.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
>     awk 'BEGIN { RS = ""; FS = " "; OOB_val= 3600000 ; min_uplink_time=OOB_val; min_downlink_time=OOB_val; orig=OOB_val ; min_rtt=OOB_val}; \
>     { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
>     rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
>     uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
>     min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
>     min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time;  \
>     min_rtt = rtt<min_rtt ? rtt : min_rtt; } \
>     END { print orig, min_uplink_time, min_downlink_time, min_rtt }'

3600000 3600000 3600000 3600000
root@turris:~# 
root@turris:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
>     awk 'BEGIN { RS = ""; FS = " "; OOB_val= 3600000 ; min_uplink_time=OOB_val; min_downlink_time=OOB_val; orig=OOB_val ; min_rtt=OOB_val}; \
>     { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
>     rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
>     uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
>     min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
>     min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time;  \
>     min_rtt = rtt<min_rtt ? rtt : min_rtt; } \
>     END { print orig, min_uplink_time, min_downlink_time, min_rtt }'
29548827 5 12.7 18.7

Then we can store this into /tmp/run/autorate-sqm/${reflector}_last_sample and read this in after all reflectors have been processed to calculate the baselines and then store these out to /tmp/run/autorate-sqm/${reflector}_baseline` or similar....

Lynx · 2021-12-09T08:40:15.560Z

@_FailSafe Crikey Charlie! Amazing.

@moeller0 and @_FailSafe the burning question now is does it show increase in one way delay upon bufferbloat on upload and/or download? So if run in an infinite while loop with sleep 0.5s does it show increase in the affected direction? I can't test now but would be superb if we can show this actually works.

@moeller0 I believe in shell it is possible to write to variables like:

min_downlink_owd$n=

Where n is the nth reflector. Would square brackets deal with the 10 or greater issue we encountered before? Wouldn't that be nicer than writing out to files even if in tmp? I only did that so far because I wanted to fork pings to background and concurrently write to file. And then retrieve those following a wait call to wait on all the forked ping processes completing.

moeller0 · 2021-12-09T08:52:14.529Z

Lynx:

I only did that so far because I wanted to fork pings to background and concurrently write to file.

I think we still would like to do that, no? instead of hammering 1 reflector for X samples we should hammer X reflectors for 1 sample each.... with somewhat persistent state, we could think about randomly drawing these five from a larger pool, IFF it turns out that the deltaOWDs are relative stable over reflectors.
I wonder whether @_FailSafe's code should move into its own shel script, so that we can truly run it in the background?

Lynx:

And then retrieve those following a wait call to wait on all the forked ping processes completing.

I thought this was a clever twist worth maintaining... the beauty of keeping state on files (on tmpfs) would be that we could keep state over crashes of the script....

Lynx · 2021-12-09T08:57:55.198Z

The burning question for me is does this actually work. If not all this timestamp stuff is in vain!

I imagine it will but I am keen to see some plots in Excel on this thread. Will test as soon as I can.

BlueKingMuch · 2021-12-09T09:23:12.709Z

I have an Idea for using local ISP's reflectors. why not scan it with nmap and find out usable reflectors?

Proof of concept by myself:

root@turris:/tmp# ###so, we want reliably find out reflectors from our local ISP ####
root@turris:/tmp# ###let's check the traceroute###
root@turris:/tmp# traceroute 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 38 byte packets
 1  100.126.0.244 (100.126.0.244)  9.701 ms  9.291 ms  8.933 ms
 2  100.126.0.46 (100.126.0.46)  8.660 ms  8.440 ms  8.582 ms
 3  as42-vie.pch.net (193.203.0.33)  9.243 ms  9.546 ms  9.308 ms
 4  dns9.quad9.net (9.9.9.9)  9.232 ms !C  8.546 ms !C  8.669 ms !C
root@turris:/tmp# ### interesting, let's try out if 1st hop responds ###
root@turris:/tmp# hping3 100.126.0.244 --icmp --icmp-ts -i u1000 -c 1
HPING 100.126.0.244 (pppoe-wan 100.126.0.244): icmp mode set, 28 headers + 0 data bytes

--- 100.126.0.244 hping statistic ---
1 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@turris:/tmp# ### sadly not :( ###
root@turris:/tmp# ### but, what if we scan the network where the first hop ends?###
root@turris:/tmp# ### let's try nmap, with nmap -sP -PP, it check's if host is responding with times
tamps! ###
root@turris:/tmp# ### for example nmap with 9.9.9.9 ###
root@turris:/tmp# nmap -sP -PP 9.9.9.9
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:12 CET
Nmap scan report for dns9.quad9.net (9.9.9.9)
Host is up (0.0094s latency).
Nmap done: 1 IP address (1 host up) scanned in 0.38 seconds
root@turris:/tmp# ### sanity checking with hping3 ###
root@turris:/tmp# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 1
HPING 9.9.9.9 (pppoe-wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=40 ip=9.9.9.9 ttl=61 id=56849 icmp_seq=0 rtt=19.8 ms
ICMP timestamp: Originate=33220732 Receive=33220737 Transmit=33220737
ICMP timestamp RTT tsrtt=20


--- 9.9.9.9 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.8/19.8/19.8 ms
root@turris:/tmp# ### that looks good so far, but does nmap -sP -PP reliably differ hosts which sen
d icmp timestamps from those who do not? let's check on the first hop! ###
root@turris:/tmp# nmap -sP -PP 100.126.0.244
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:14 CET
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 2.20 seconds
root@turris:/tmp# ### it does! yay! ###
root@turris:/tmp# ### now, what if we scan the first hop's network, if any host there is responding
(so we can use this as reflectors) ? ###
root@turris:/tmp# ### let's scan the network! ###
root@turris:/tmp# nmap -sP -PP 100.126.0.1-254
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:17 CET
Nmap scan report for 100.126.0.1
Host is up (0.026s latency).
Nmap scan report for 100.126.0.2
Host is up (0.0090s latency).
Nmap scan report for 100.126.0.13
Host is up (0.014s latency).
Nmap scan report for 100.126.0.14
Host is up (0.011s latency).
Nmap scan report for 100.126.0.17
Host is up (0.015s latency).
Nmap scan report for 100.126.0.18
Host is up (0.011s latency).
Nmap scan report for 100.126.0.21
Host is up (0.025s latency).
Nmap scan report for 100.126.0.22
Host is up (0.014s latency).
Nmap scan report for 100.126.0.26
Host is up (0.0098s latency).
Nmap scan report for 100.126.0.49
Host is up (0.016s latency).
Nmap scan report for 100.126.0.54
Host is up (0.016s latency).
Nmap scan report for 100.126.0.58
Host is up (0.016s latency).
Nmap scan report for 100.126.0.62
Host is up (0.016s latency).
Nmap scan report for 100.126.0.66
Host is up (0.017s latency).
Nmap scan report for 100.126.0.70
Host is up (0.018s latency).
Nmap scan report for 100.126.0.82
Host is up (0.021s latency).
Nmap scan report for 100.126.0.86
Host is up (0.022s latency).
Nmap scan report for 100.126.0.90
Host is up (0.019s latency).
Nmap scan report for 100.126.0.94
Host is up (0.021s latency).
Nmap scan report for 100.126.0.98
Host is up (0.012s latency).
Nmap scan report for 100.126.0.100
Host is up (0.037s latency).
Nmap scan report for 100.126.0.102
Host is up (0.037s latency).
Nmap scan report for 100.126.0.106
Host is up (0.015s latency).
Nmap scan report for 100.126.0.107
Host is up (0.024s latency).
Nmap scan report for 100.126.0.108
Host is up (0.018s latency).
Nmap scan report for 100.126.0.109
Host is up (0.027s latency).
Nmap scan report for 100.126.0.110
Host is up (0.029s latency).
Nmap scan report for 100.126.0.111
Host is up (0.018s latency).
Nmap scan report for 100.126.0.112
Host is up (0.025s latency).
Nmap scan report for 100.126.0.113
Host is up (0.035s latency).
Nmap scan report for 100.126.0.114
Host is up (0.016s latency).
Nmap scan report for 100.126.0.115
Host is up (0.026s latency).
Nmap scan report for 100.126.0.201
Host is up (0.012s latency).
Nmap scan report for 100.126.0.202
Host is up (0.015s latency).
Nmap scan report for 100.126.0.203
Host is up (0.015s latency).
Nmap scan report for 100.126.0.204
Host is up (0.015s latency).
Nmap scan report for 100.126.0.206
Host is up (0.015s latency).
Nmap scan report for 100.126.0.213
Host is up (0.010s latency).
Nmap scan report for 100.126.0.249
Host is up (0.015s latency).
Nmap scan report for 100.126.0.250
Host is up (0.015s latency).
Nmap scan report for 100.126.0.252
Host is up (0.031s latency).
Nmap scan report for 100.126.0.253
Host is up (0.032s latency).
Nmap scan report for 100.126.0.254
Host is up (0.026s latency).
Nmap done: 254 IP addresses (43 hosts up) scanned in 1.79 seconds
root@turris:/tmp# ### perfect! there's plenty of reflectors which we can use! let's pick out one of
these to check if our assumption is correct ###
root@turris:/tmp# hping3  100.126.0.203 --icmp --icmp-ts -i u1000 -c 1
HPING 100.126.0.203 (pppoe-wan 100.126.0.203): icmp mode set, 28 headers + 0 data bytes
len=40 ip=100.126.0.203 ttl=60 id=62083 icmp_seq=0 rtt=9.8 ms
ICMP timestamp: Originate=33494932 Receive=33494935 Transmit=33494935
ICMP timestamp RTT tsrtt=10


--- 100.126.0.203 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 9.8/9.8/9.8 ms

as you can see, i found plenty of usable reflectors inside the network of the first hop. could anyone test with their ISP's if this could be a feasible solution?

I am 99.99% sure this could by automated in a startup script

Lynx · 2021-12-09T09:27:54.816Z

Timestamp capabilities?