That is not what we do right now... the challenge is getting the most recent value out of a running ping instance in shell... so ATM ping is run for a number of samples and its output massages and stored to disk (for multiple ping instances) and then that disk file is massages to get the minimum over all instances. The script does a bit more and then sleeps until the next tick time, rinse and repeat...
Doing a remote name lookup every tick is not going to be helpful.... so if we want symbolic names the script should convert these into IP addresses early on and then only handle those addresses, no? For the time being, IP addresses sound fine, but for bigger deployment I believe your idea is relevant and worth thinking over.
I guess resolving names early is probably good enough.
I do think ultimately lua is the better choice here. shell is just too limited for doing "smart" things.
1 Like
Something slightly off-topic, but the great @dtaht alerted me to the existence of gping which is a nice tool to show ping RTTs in realtime in a terminal window, pretty nifty to run parallel while testing the autoratescript to get some visible insight in what is happening second by second.
2 Likes
Over here, in part based on all this enthusiastic research on this thread, I started drafting a design for a new tool that would combine a load test with icmp unreachable messages. https://github.com/dtaht/wtbb . I thought "I will damned" if I'll write it in C (using libcurl as a base), but it turned out I'd be equally damned if I tried to write it in rust, and in part, because qping was written in rust it seemed attractive as a starting point or influencer as to how that exercise went.
So I'm still floating around that portion of the solution space, I have a tendency to write the spec and man page first, while trying to find the least number of libs to leverage - notably looking over the sources to tcptraceroute.
libcurl is really excellent as to how you can instrument it, nothing in rust comes close, or perhaps I don't understand rust well enough - or both! Doing threads is not really necessary (I think), but rust does those well. It's really remarkable how short the basic code could be, with tokio, compared to a C version...
3 Likes
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=54 id=47019 icmp_seq=0 rtt=89.5 ms
ICMP timestamp: Originate=82696653 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=90
len=46 ip=9.9.9.9 ttl=54 id=47021 icmp_seq=2 rtt=87.5 ms
ICMP timestamp: Originate=82696656 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=87
len=46 ip=9.9.9.9 ttl=54 id=47020 icmp_seq=1 rtt=88.8 ms
ICMP timestamp: Originate=82696655 Receive=82696703 Transmit=82696703
ICMP timestamp RTT tsrtt=88
Any major awk experts here able to determine for each result:
uplink time = Receive - Originate
downlink time = Originate + rtt - Transmit
And then determine the minimum across the matches?
Something like multi-line match on rtt=X Originate=Y Receive=Z, do the calculations, and keep track of the minimum for each match. And then just output the minimums.
It's possible but seems to be a nightmare in awk 
gping is pretty sweet. I'll usually always have it running in a workspace
Rust is nice, but no support in master for it yet it would seem
Thanks! Probably gonna be tomorrow or Friday depending on what needs doing IRL
I'm very tempted make a Lua port of your script and continue this work on that just avoid these headaches 
Honestly I think this is the only really hard bit...
@vgaetera I think this might be one for you? Seems like a really tough awk mult-line match with regexp, calculations and keeping track of minimums across matches.
@moeller0 is this beyond you as well? I can probably crack it if I spend an entire day.
DO IT! This needs to be in Lua. I am happy to help but don't have the bandwidth for a full port.
1 Like
Anyone else notice that these example pings came back out of order?
I think that's for the compiler, but can't rust output standalone binaries?
I think you match on rtt= and then in the body you read two more lines and pull them apart. Using getline
interestingly another option for a small but nontrivial programming language is javascript via https://duktape.org/
but not currently in OpenWrt. what would it take to get it included?
Nevermind, reading up on it, it seems like it's for embedding in a C/C++ program, so we could build a "hello world" that included it and then write the logic in javascript.
I started a lua port of this script the other day when I was asking about alternate languages (after others mentioned python was too heavy for smaller resource hardware). I ditched the effort and actually tossed the work when it sounded like shell was the preferred path forward. Bummer.
I’m no lua expert, but willing to get my hands dirty with it again. You’ll probably develop faster, but I’ll be willing to contribute where possible.
Fair enough, I just imagine I'd probably spend more time on figuring out how to make awk parse the output, than just doing a port 
Ah damn, well I'll let you know when I have anything worth looking at.
Found a stackoverflow answer about from a few years back, at the time at least Rust statically linked everything except libc. And I'm sure it can be done easily now. I was mostly asking with getting an offical OpenWrt package made in mind. I assume just downloading the binaries into a package would be frowned upon.
Looks pretty neat, and yeah we'd probably need something written in C to set up the engine and get things running, but that should be easy enough I imagine.
Had a quick look at what Alpine does to compile it, should be simple enough
https://git.alpinelinux.org/aports/tree/community/duktape/APKBUILD
And if it works for Alpine without patches, we shouldn't have trouble on OpenWrt with targetting musl libc either.
I think I have it working for you:
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
{ rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
END { print min_uplink_time, min_downlink_time }'
6 12.9
If you want to confirm the results are true, you can run this version for a test:
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
{ rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
{ print uplink_time, downlink_time, min_uplink_time, min_downlink_time }'
7 12.9 7 12.9
6 12.9 6 12.9
14 23.8 6 12.9
The first command should give you just the final min_uplink_time and min_downlink_time across all three executions. If you need the final values delimited with something other than a space, that's an easy mod to the awk statement.
In the second command:
- First column is
uplink_timefor the current record set - Second column is
downlink_timefor the current record set - Third column is overall
min_uplink_time - Fourth column is overall
min_downlink_time
Update...
Ran another test with more iterations to further confirm the overall min logic holds. Let me know if you see any issues I need to fix:
root@OpenWrt:~# time hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 20 2> /dev/null | tail -n+2 | \
awk 'BEGIN { RS = ""; FS = " "; min_uplink_time=100000; min_downlink_time=100000 }; \
{ rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); rx=$11; sub(/Receive=/, "", rx); \
tx=$12; sub(/Transmit=/, "", tx); uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; } \
{ print uplink_time, downlink_time, min_uplink_time, min_downlink_time }'
13 26.9 13 26.9
17 21.9 13 21.9
16 21.8 13 21.8
15 21.8 13 21.8
14 21.7 13 21.7
13 21.6 13 21.6
12 21.5 12 21.5
11 21.5 11 21.5
10 21.4 10 21.4
9 21.3 9 21.3
8 21.3 8 21.3
7 21.2 7 21.2
6 21.2 6 21.2
10 16.1 6 16.1
8 17 6 16.1
7 17 6 16.1
6 16.9 6 16.1
5 16.8 5 16.1
4 16.8 4 16.1
13 26.6 4 16.1
real 0m0.244s
user 0m0.005s
sys 0m0.000s
3 Likes
Pretty cool! I think we should be better off reducing the number of samples an increasing the delay though.
Here is a slight modification that also prints out the last valid originate timestamp (this strictly only makes sense for -c 1, but for sanity checking, I believe having any real timestamp from the run there is helpful), and allows one default out of bounds value:
root@turris:~# hping3 1.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
> awk 'BEGIN { RS = ""; FS = " "; OOB_val= 3600000 ; min_uplink_time=OOB_val; min_downlink_time=OOB_val; orig=OOB_val ; min_rtt=OOB_val}; \
> { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
> rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
> uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
> min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
> min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; \
> min_rtt = rtt<min_rtt ? rtt : min_rtt; } \
> END { print orig, min_uplink_time, min_downlink_time, min_rtt }'
3600000 3600000 3600000 3600000
root@turris:~#
root@turris:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 3 2> /dev/null | tail -n+2 | \
> awk 'BEGIN { RS = ""; FS = " "; OOB_val= 3600000 ; min_uplink_time=OOB_val; min_downlink_time=OOB_val; orig=OOB_val ; min_rtt=OOB_val}; \
> { rtt=$6; sub(/rtt=/, "", rtt); orig=$10; sub(/Originate=/, "", orig); \
> rx=$11; sub(/Receive=/, "", rx); tx=$12; sub(/Transmit=/, "", tx); \
> uplink_time=rx-orig; downlink_time=orig+rtt-tx; \
> min_uplink_time = uplink_time<min_uplink_time ? uplink_time : min_uplink_time; \
> min_downlink_time = downlink_time<min_downlink_time ? downlink_time : min_downlink_time; \
> min_rtt = rtt<min_rtt ? rtt : min_rtt; } \
> END { print orig, min_uplink_time, min_downlink_time, min_rtt }'
29548827 5 12.7 18.7
Then we can store this into /tmp/run/autorate-sqm/${reflector}_last_sample and read this in after all reflectors have been processed to calculate the baselines and then store these out to /tmp/run/autorate-sqm/${reflector}_baseline` or similar....
@_FailSafe Crikey Charlie! Amazing.
@moeller0 and @_FailSafe the burning question now is does it show increase in one way delay upon bufferbloat on upload and/or download? So if run in an infinite while loop with sleep 0.5s does it show increase in the affected direction? I can't test now but would be superb if we can show this actually works.
@moeller0 I believe in shell it is possible to write to variables like:
min_downlink_owd$n=
Where n is the nth reflector. Would square brackets deal with the 10 or greater issue we encountered before? Wouldn't that be nicer than writing out to files even if in tmp? I only did that so far because I wanted to fork pings to background and concurrently write to file. And then retrieve those following a wait call to wait on all the forked ping processes completing.
I think we still would like to do that, no? instead of hammering 1 reflector for X samples we should hammer X reflectors for 1 sample each.... with somewhat persistent state, we could think about randomly drawing these five from a larger pool, IFF it turns out that the deltaOWDs are relative stable over reflectors.
I wonder whether @_FailSafe's code should move into its own shel script, so that we can truly run it in the background?
I thought this was a clever twist worth maintaining... the beauty of keeping state on files (on tmpfs) would be that we could keep state over crashes of the script....
1 Like
The burning question for me is does this actually work. If not all this timestamp stuff is in vain!
I imagine it will but I am keen to see some plots in Excel on this thread. Will test as soon as I can.
I have an Idea for using local ISP's reflectors. why not scan it with nmap and find out usable reflectors?
Proof of concept by myself:
root@turris:/tmp# ###so, we want reliably find out reflectors from our local ISP ####
root@turris:/tmp# ###let's check the traceroute###
root@turris:/tmp# traceroute 9.9.9.9
traceroute to 9.9.9.9 (9.9.9.9), 30 hops max, 38 byte packets
1 100.126.0.244 (100.126.0.244) 9.701 ms 9.291 ms 8.933 ms
2 100.126.0.46 (100.126.0.46) 8.660 ms 8.440 ms 8.582 ms
3 as42-vie.pch.net (193.203.0.33) 9.243 ms 9.546 ms 9.308 ms
4 dns9.quad9.net (9.9.9.9) 9.232 ms !C 8.546 ms !C 8.669 ms !C
root@turris:/tmp# ### interesting, let's try out if 1st hop responds ###
root@turris:/tmp# hping3 100.126.0.244 --icmp --icmp-ts -i u1000 -c 1
HPING 100.126.0.244 (pppoe-wan 100.126.0.244): icmp mode set, 28 headers + 0 data bytes
--- 100.126.0.244 hping statistic ---
1 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@turris:/tmp# ### sadly not :( ###
root@turris:/tmp# ### but, what if we scan the network where the first hop ends?###
root@turris:/tmp# ### let's try nmap, with nmap -sP -PP, it check's if host is responding with times
tamps! ###
root@turris:/tmp# ### for example nmap with 9.9.9.9 ###
root@turris:/tmp# nmap -sP -PP 9.9.9.9
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:12 CET
Nmap scan report for dns9.quad9.net (9.9.9.9)
Host is up (0.0094s latency).
Nmap done: 1 IP address (1 host up) scanned in 0.38 seconds
root@turris:/tmp# ### sanity checking with hping3 ###
root@turris:/tmp# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 1
HPING 9.9.9.9 (pppoe-wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=40 ip=9.9.9.9 ttl=61 id=56849 icmp_seq=0 rtt=19.8 ms
ICMP timestamp: Originate=33220732 Receive=33220737 Transmit=33220737
ICMP timestamp RTT tsrtt=20
--- 9.9.9.9 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.8/19.8/19.8 ms
root@turris:/tmp# ### that looks good so far, but does nmap -sP -PP reliably differ hosts which sen
d icmp timestamps from those who do not? let's check on the first hop! ###
root@turris:/tmp# nmap -sP -PP 100.126.0.244
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:14 CET
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 2.20 seconds
root@turris:/tmp# ### it does! yay! ###
root@turris:/tmp# ### now, what if we scan the first hop's network, if any host there is responding
(so we can use this as reflectors) ? ###
root@turris:/tmp# ### let's scan the network! ###
root@turris:/tmp# nmap -sP -PP 100.126.0.1-254
Starting Nmap 7.70 ( https://nmap.org ) at 2021-12-09 10:17 CET
Nmap scan report for 100.126.0.1
Host is up (0.026s latency).
Nmap scan report for 100.126.0.2
Host is up (0.0090s latency).
Nmap scan report for 100.126.0.13
Host is up (0.014s latency).
Nmap scan report for 100.126.0.14
Host is up (0.011s latency).
Nmap scan report for 100.126.0.17
Host is up (0.015s latency).
Nmap scan report for 100.126.0.18
Host is up (0.011s latency).
Nmap scan report for 100.126.0.21
Host is up (0.025s latency).
Nmap scan report for 100.126.0.22
Host is up (0.014s latency).
Nmap scan report for 100.126.0.26
Host is up (0.0098s latency).
Nmap scan report for 100.126.0.49
Host is up (0.016s latency).
Nmap scan report for 100.126.0.54
Host is up (0.016s latency).
Nmap scan report for 100.126.0.58
Host is up (0.016s latency).
Nmap scan report for 100.126.0.62
Host is up (0.016s latency).
Nmap scan report for 100.126.0.66
Host is up (0.017s latency).
Nmap scan report for 100.126.0.70
Host is up (0.018s latency).
Nmap scan report for 100.126.0.82
Host is up (0.021s latency).
Nmap scan report for 100.126.0.86
Host is up (0.022s latency).
Nmap scan report for 100.126.0.90
Host is up (0.019s latency).
Nmap scan report for 100.126.0.94
Host is up (0.021s latency).
Nmap scan report for 100.126.0.98
Host is up (0.012s latency).
Nmap scan report for 100.126.0.100
Host is up (0.037s latency).
Nmap scan report for 100.126.0.102
Host is up (0.037s latency).
Nmap scan report for 100.126.0.106
Host is up (0.015s latency).
Nmap scan report for 100.126.0.107
Host is up (0.024s latency).
Nmap scan report for 100.126.0.108
Host is up (0.018s latency).
Nmap scan report for 100.126.0.109
Host is up (0.027s latency).
Nmap scan report for 100.126.0.110
Host is up (0.029s latency).
Nmap scan report for 100.126.0.111
Host is up (0.018s latency).
Nmap scan report for 100.126.0.112
Host is up (0.025s latency).
Nmap scan report for 100.126.0.113
Host is up (0.035s latency).
Nmap scan report for 100.126.0.114
Host is up (0.016s latency).
Nmap scan report for 100.126.0.115
Host is up (0.026s latency).
Nmap scan report for 100.126.0.201
Host is up (0.012s latency).
Nmap scan report for 100.126.0.202
Host is up (0.015s latency).
Nmap scan report for 100.126.0.203
Host is up (0.015s latency).
Nmap scan report for 100.126.0.204
Host is up (0.015s latency).
Nmap scan report for 100.126.0.206
Host is up (0.015s latency).
Nmap scan report for 100.126.0.213
Host is up (0.010s latency).
Nmap scan report for 100.126.0.249
Host is up (0.015s latency).
Nmap scan report for 100.126.0.250
Host is up (0.015s latency).
Nmap scan report for 100.126.0.252
Host is up (0.031s latency).
Nmap scan report for 100.126.0.253
Host is up (0.032s latency).
Nmap scan report for 100.126.0.254
Host is up (0.026s latency).
Nmap done: 254 IP addresses (43 hosts up) scanned in 1.79 seconds
root@turris:/tmp# ### perfect! there's plenty of reflectors which we can use! let's pick out one of
these to check if our assumption is correct ###
root@turris:/tmp# hping3 100.126.0.203 --icmp --icmp-ts -i u1000 -c 1
HPING 100.126.0.203 (pppoe-wan 100.126.0.203): icmp mode set, 28 headers + 0 data bytes
len=40 ip=100.126.0.203 ttl=60 id=62083 icmp_seq=0 rtt=9.8 ms
ICMP timestamp: Originate=33494932 Receive=33494935 Transmit=33494935
ICMP timestamp RTT tsrtt=10
--- 100.126.0.203 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 9.8/9.8/9.8 ms
as you can see, i found plenty of usable reflectors inside the network of the first hop. could anyone test with their ISP's if this could be a feasible solution?
I am 99.99% sure this could by automated in a startup script
Timestamp capabilities?