I wonder whether https://subversion.xray.aps.anl.gov/trac/dbw/browser/trunk/Xalert/stevens/tcpipi/icmptime/icmptime.c might not be helpful, instead of going for the full hping3 we might just get a small binary that does get us ICMP timestamps, not sure about the license/usability of that code for our purpose though. Piggy-backing on an existing maintained project is probably better
icmptime.c:20:10: fatal error: netinet/ip_var.h: No such file or directory
20 | #include <netinet/ip_var.h>
| ^~~~~~~~~~~~~~~~~~
compilation terminated.
This is on WSL2. Not sure how it would fare for OpenWrt.
Yeah the code is ancient and does not actually assume Linux to be a thing (effortlessly explained by the code's age), I mostly linked that because the whole thing seems rather short and some of us wanted parts of this project to move to C-code 
.
1 Like
I'm working on this now, but slow going. Will update if/when I make some headway...
2 Likes
Ace - thanks for trying. Don't forget to post issues on this thread because help is pretty fast. It's like placing a match next to a tinderbox. So much eagerness from many different parties. It's quite amazing really.
1 Like
For fun, here is @tievolu's complete list of type13 reflectors from a well connected place:
Pings done = 114 RTT/2 min = 3 | Request | Reply | Round Trip |
-------------------------------------------------------------------------------------------------------------------------------------
| Ping Target | Offset | Current | Average ? jitter | Current | Average ? jitter | Current | Average ? jitter |
-------------------------------------------------------------------------------------------------------------------------------------
| 27.112.79.80 | 0 | ERROR | N/A ? N/A | ERROR | N/A ? N/A | ERROR | N/A ? N/A |
| 79.110.170.43 | 0 | ERROR | N/A ? N/A | ERROR | N/A ? N/A | ERROR | N/A ? N/A |
| 146.112.61.106 | -60 | 3 | 3.95 ? 0.75 | 4 | 3.67 ? 0.55 | 7 | 7.62 ? 0.70 |
| 149.112.112.112 | -40 | 3 | 3.71 ? 0.80 | 4 | 4.05 ? 0.95 | 7 | 7.76 ? 0.35 |
| 208.67.220.220 | -59 | 4 | 3.76 ? 0.70 | 3 | 4.00 ? 0.70 | 7 | 7.76 ? 0.40 |
| 208.67.222.222 | -63 | 4 | 3.62 ? 0.95 | 3 | 4.14 ? 1.00 | 7 | 7.76 ? 0.75 |
| 9.9.9.10 | -60 | 3 | 3.76 ? 0.95 | 5 | 4.05 ? 1.05 | 8 | 7.81 ? 0.20 |
| 149.112.112.11 | -60 | 4 | 3.67 ? 1.25 | 4 | 4.14 ? 1.20 | 8 | 7.81 ? 0.25 |
| 146.112.41.3 | -62 | 5 | 3.71 ? 0.95 | 3 | 4.10 ? 1.15 | 8 | 7.81 ? 0.40 |
| 9.9.9.9 | -62 | 3 | 3.67 ? 1.20 | 5 | 4.14 ? 1.30 | 8 | 7.81 ? 0.50 |
| 94.140.15.15 | -63 | 5 | 3.81 ? 0.90 | 3 | 4.05 ? 0.90 | 8 | 7.86 ? 0.20 |
| 94.140.15.16 | -62 | 3 | 3.62 ? 0.95 | 5 | 4.29 ? 0.95 | 8 | 7.90 ? 0.20 |
| 149.112.112.10 | -59 | 3 | 3.52 ? 0.90 | 5 | 4.38 ? 1.10 | 8 | 7.90 ? 0.50 |
| 94.140.14.14 | -63 | 3 | 3.86 ? 1.05 | 5 | 4.14 ? 1.05 | 8 | 8.00 ? 0.00 |
| 94.140.14.15 | -63 | 3 | 3.67 ? 1.10 | 5 | 4.43 ? 1.10 | 8 | 8.10 ? 0.20 |
| 146.112.41.2 | -59 | 3 | 3.57 ? 0.80 | 5 | 4.95 ? 2.50 | 8 | 8.52 ? 2.10 |
| 9.9.9.11 | -61 | 3 | 5.05 ? 2.95 | 4 | 4.00 ? 1.20 | 7 | 9.05 ? 2.75 |
| 185.228.168.10 | 321 | 4 | 5.71 ? 4.95 | 4 | 4.48 ? 0.95 | 8 | 10.19 ? 4.20 |
| 185.233.106.232 | 26088 | 3 | 4.57 ? 1.40 | 7 | 6.48 ? 1.35 | 10 | 11.05 ? 0.25 |
| 185.233.107.4 | 12251 | 4 | 4.19 ? 1.70 | 7 | 7.00 ? 1.80 | 11 | 11.19 ? 0.40 |
| 185.228.168.168 | 321 | 3 | 6.81 ? 6.00 | 5 | 4.62 ? 1.75 | 8 | 11.43 ? 6.85 |
| 185.235.81.1 | -59 | 3 | 3.90 ? 1.50 | 8 | 8.05 ? 1.60 | 11 | 11.95 ? 0.40 |
| 176.9.1.117 | -61 | 7 | 5.10 ? 2.55 | 6 | 7.29 ? 2.50 | 13 | 12.38 ? 0.55 |
| 74.82.42.42 | -60 | 9 | 5.71 ? 3.80 | 6 | 6.71 ? 2.15 | 15 | 12.43 ? 2.25 |
| 176.9.93.198 | -59 | 6 | 4.71 ? 2.60 | 6 | 8.43 ? 3.80 | 12 | 13.14 ? 2.40 |
| 185.95.218.42 | -62 | 5 | 5.38 ? 2.40 | 9 | 8.76 ? 2.40 | 14 | 14.14 ? 0.20 |
| 185.95.218.43 | -63 | 3 | 4.57 ? 2.30 | 11 | 9.71 ? 1.90 | 14 | 14.29 ? 0.60 |
| 193.17.47.1 | -62 | 7 | 4.81 ? 2.60 | 8 | 9.62 ? 2.65 | 15 | 14.43 ? 0.45 |
| 185.43.135.1 | -63 | 4 | 4.62 ? 2.05 | 10 | 10.05 ? 2.50 | 14 | 14.67 ? 1.05 |
| 93.104.213.190 | -63 | 3 | 5.05 ? 2.20 | 11 | 10.24 ? 2.05 | 14 | 15.29 ? 2.05 |
| 130.59.31.248 | -63 | 8 | 5.52 ? 3.45 | 9 | 11.48 ? 3.45 | 17 | 17.00 ? 0.00 |
| 130.59.31.251 | -64 | 4 | 4.57 ? 1.65 | 13 | 12.52 ? 1.75 | 17 | 17.10 ? 0.20 |
| 185.150.99.255 | -65 | 3 | 6.38 ? 3.55 | 12 | 11.10 ? 4.55 | 15 | 17.48 ? 3.60 |
| 213.196.191.96 | -63 | ERROR | 6.33 ? 4.90 | ERROR | 12.38 ? 5.20 | ERROR | 18.71 ? 7.30 |
| 193.19.108.3 | -228670 | 5 | 5.62 ? 3.40 | 15 | 13.81 ? 3.50 | 20 | 19.43 ? 0.40 |
| 5.1.66.255 | -64 | 4 | 4.86 ? 1.85 | 15 | 14.86 ? 1.95 | 19 | 19.71 ? 0.40 |
| 194.242.2.3 | -228670 | 3 | 5.38 ? 4.00 | 16 | 14.43 ? 2.70 | 19 | 19.81 ? 1.60 |
| 146.255.56.98 | 223 | 3 | 6.57 ? 3.90 | 17 | 13.57 ? 4.10 | 20 | 20.14 ? 0.20 |
| 212.78.94.40 | -66 | 4 | 4.52 ? 2.25 | 16 | 15.76 ? 2.35 | 20 | 20.29 ? 0.50 |
| 193.19.108.2 | -228671 | 4 | 4.24 ? 1.65 | 18 | 18.05 ? 2.05 | 22 | 22.29 ? 0.60 |
| 185.216.27.142 | -63 | 10 | 7.48 ? 5.95 | 7 | 14.81 ? 5.75 | 17 | 22.29 ? 8.00 |
| 194.242.2.2 | -228670 | 5 | 4.95 ? 2.20 | 18 | 17.76 ? 2.25 | 23 | 22.71 ? 0.45 |
| 185.64.79.5 | -66 | 3 | 5.38 ? 3.70 | 24 | 17.81 ? 4.00 | 27 | 23.19 ? 0.40 |
| 37.252.230.153 | -67 | 3 | 5.67 ? 4.45 | 21 | 18.05 ? 4.25 | 24 | 23.71 ? 0.40 |
| 209.250.226.191 | -70 | 6 | 4.43 ? 2.40 | 18 | 19.76 ? 2.30 | 24 | 24.19 ? 0.30 |
| 46.227.200.55 | -67 | 5 | 6.05 ? 3.65 | 20 | 19.10 ? 3.65 | 25 | 25.14 ? 0.20 |
| 46.227.200.54 | -67 | 5 | 4.95 ? 2.70 | 20 | 20.52 ? 1.70 | 25 | 25.48 ? 1.00 |
| 37.252.232.95 | -67 | 9 | 5.10 ? 2.95 | 17 | 20.43 ? 2.80 | 26 | 25.52 ? 0.65 |
| 95.216.229.153 | -75 | 4 | 5.10 ? 1.75 | 28 | 26.24 ? 2.25 | 32 | 31.33 ? 1.10 |
| 185.134.196.55 | -63 | 3 | 4.62 ? 2.40 | 12 | 30.52 ? 12.50 | 15 | 35.14 ? 10.60 |
| 185.134.196.54 | -63 | 8 | 4.86 ? 2.35 | 7 | 35.05 ? 12.65 | 15 | 39.90 ? 10.50 |
| 46.239.223.80 | -74 | 4 | 5.00 ? 1.90 | 51 | 49.62 ? 2.20 | 55 | 54.62 ? 0.40 |
| 35.245.234.132 | -95 | 9 | 11.57 ? 1.85 | 82 | 79.62 ? 1.95 | 91 | 91.19 ? 0.30 |
| 35.230.160.38 | -119 | 26 | 16.95 ? 2.15 | 66 | 74.81 ? 2.05 | 92 | 91.76 ? 0.40 |
| 149.112.122.10 | -99 | 7 | 8.67 ? 2.85 | 87 | 85.19 ? 2.75 | 94 | 93.86 ? 0.30 |
| 149.112.121.20 | -101 | 5 | 6.81 ? 2.50 | 89 | 87.14 ? 2.50 | 94 | 93.95 ? 0.10 |
| 149.112.122.20 | -99 | 5 | 8.38 ? 2.20 | 89 | 85.62 ? 2.40 | 94 | 94.00 ? 0.20 |
| 149.112.121.10 | -98 | 6 | 9.33 ? 2.25 | 88 | 84.71 ? 2.15 | 94 | 94.05 ? 0.30 |
| 149.112.122.30 | -101 | 4 | 6.14 ? 4.70 | 90 | 88.10 ? 4.80 | 94 | 94.24 ? 0.50 |
| 149.112.121.30 | -101 | 3 | 4.19 ? 1.70 | 92 | 90.10 ? 1.95 | 95 | 94.29 ? 0.45 |
| 35.237.220.84 | -98 | 21 | 21.76 ? 2.80 | 81 | 80.14 ? 2.90 | 102 | 101.90 ? 0.50 |
| 35.231.247.227 | -95 | 9 | 13.52 ? 3.05 | 92 | 88.43 ? 3.15 | 101 | 101.95 ? 0.90 |
| 174.138.29.175 | -61 | 85 | 87.10 ? 2.75 | 88 | 85.33 ? 2.90 | 173 | 172.43 ? 0.35 |
| 139.162.112.47 | -14 | ERROR | 168.90 ? 3.00 | ERROR | 84.05 ? 3.05 | ERROR | 252.95 ? 0.45 |
| 172.104.93.80 | -15 | 178 | 179.19 ? 1.65 | 88 | 86.38 ? 1.45 | 266 | 265.57 ? 0.60 |
| 101.102.103.104 | -1 | 195 | 200.62 ? 5.15 | 88 | 82.38 ? 5.15 | 283 | 283.00 ? 0.00 |
| 101.101.101.101 | -8 | 200 | 203.05 ? 2.65 | 87 | 84.86 ? 3.00 | 287 | 287.90 ? 0.65 |
-------------------------------------------------------------------------------------------------------------------------------------
note how small the offsets are to most of these....
Do the offsets seem to be staying the same roughly between runs?
I wonder if systemd is responsible for this. By default it comes with an ntp client whereas before that you had to actually install ntpd
Not working yet, but posting this here in case anyone wants to work in parallel toward getting this to compile: https://github.com/Fail-Safe/hping3/blob/main/Makefile
(I'm not a Makefile expert, so please don't take anything already in the file as correct or even necessary.)
1 Like
On first glance they look relatively stable, but the script prints out one of these tables (always ordered by Round Trip I believe, so the exact sequence is not fixed) per iteration making it tedious to track the offset per IP-address.
Since I was running this on my macos laptop outside of my home network, systemd would need to be active at the reflector sites, and I do have my reservations about systemd's prevalence on servers...
it's the standard for Debian, which is I think a very common server distribution.
In my experience with my own scripts, the offsets are generally very small (mostly <5ms) and stable once ntpd has settled on an accurate time for your system clock (i.e. about 24 hours after a reboot).
In terms of offset size there are some outliers with much larger offsets, as shown in @moeller0's output above (presumably these hosts aren't sending back timestamps based on milliseconds-since-midnight-UTC), but even these offsets are pretty stable.
Just comment out this line to remove the sorting by RTT:
my @results_to_print = sort {&get_sort_string($a) cmp &get_sort_string($b)} @results_to_print;
2 Likes
just for update on compiling i give up. (at least for today).
I tried first with the arm compiler, then with the toolkit (seems to be the makefile which does problems, but i don't understand it completely), in both ways I am differently stuck.
Hope you or another person has more luck or knowledge. or both.
The bunch of hosts with offset around -228670 all resolve to the same owner in Sweden via whois, well possible that they have a 3-4 minute offset of their main clock....
Then, thank you very much for sharing your code and the timestamp reflector list.
Thanks, I guess I need to export the data in a way that is easier to parse for post processing....
It works well in my monster perl script. For example, here's the script detecting bufferbloat in the upload direction:
Wed Dec 1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec 1 01:06:50 2021 [2034-00056166]: | Bandwidth usage | 8.26.56.26 | 129.250.35.251 | 208.67.222.222 |
Wed Dec 1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec 1 01:06:50 2021 [2034-00056166]: | TX Mbps | RX Mbps | Rqst | Rply | Rqst | Rply | Rqst | Rply |
Wed Dec 1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
...
Wed Dec 1 01:07:40 2021 [2034-00056189]: | 34.578 | 0.774 | 13 | 14 | 16 | 12 | 15 | 14 |
Wed Dec 1 01:07:42 2021 [2034-00056190]: | 32.304 | 0.695 | 15 | 13 | 11 | 13 | 11 | 14 |
Wed Dec 1 01:07:44 2021 [2034-00056191]: | 29.493 | 0.638 | 13 | 13 | 14 | 13 | 12 | 14 |
Wed Dec 1 01:07:46 2021 [2034-00056192]: | 34.485 | 1.725 | 11 | 14 | 14 | 13 | 10 | 14 |
Wed Dec 1 01:07:48 2021 [2034-00056193]: | 33.479 | 0.721 | 14 | 13 | 13 | 13 | 13 | 13 |
Wed Dec 1 01:07:50 2021 [2034-00056194]: | 26.218 | 0.545 | 50 X | 14 | 55 X | 13 | 52 X | 14 |
Wed Dec 1 01:07:52 2021 [2034-00056195]: | 26.952 | 0.566 | 35 X | 13 | 32 X | 14 | 36 X | 14 |
Wed Dec 1 01:07:54 2021 [2034-00056196]: | 29.268 | 0.687 | 68 X | 13 | 66 X | 14 | 69 X | 14 |
Timestamped pings are used to separate out request (upload) and reply (download) latency in exactly the same way as the example script I posted a few weeks back.
You are right. I forgot one of the main sources of a working technical solution to the problem of ascertaining direction of bufferbloat. Your contribution is so much appreciated in this respect, @tievolu! You cracked one of the major problems.
From memory you use nping, right? So did you face the issues identified above with nping? That it is a little slow? I have found results are improved by getting the tick duration really small, which seems impossible with nping.
Yeah, I use nping. I ping three reflectors once every two seconds, with the ping requests set to time out after one second. Now that you mention it I did find that things would break if I tried to do it more frequently than that, but for my own purposes two seconds is sufficient.
My script still uses the tactic of resting at max bandwidth and reducing it temporarily if latency gets bad, with "bad latency" defined as at least 3 out of the last 10 requests/replies exceeding 30ms for all reflectors.
Usually bad latency looks like the example above, with all three reflectors returning bad results at the same time, all back to back, and only one bandwidth reduction is needed to address it, so the script typically mitigates bufferbloat after about six seconds.
My script is very much tailored to my own connection and I haven't made any attempt to implement a moving ping baseline or any of the other clever stuff you guys are working on. Another thing to note is that I'm doing all this on a Qotom x86 box, so results might be different on more typical router hardware.
Had a feeling this question would be waiting for me when I got home 
But no not yet. I wanted to test on the router, and for that I need to break out a cross-compilation toolchain.
Safest way to get the dependencies right is to make a package for it I'd assume, so I'd gotta remember how that works again as well. Or I could always compile a static binary so there's no dependency issues.
And I'm not entirely sure how to use the thing, the notes in the fork helps, but I still don't know what the MAC addresses are for.
1 Like
Here's where I sit with compiling it. Not great to be honest...
Summary
user@4fbfcf8066d8:~/openwrt$ make package/network/utils/hping3/compile V=sc | tee build.log
make[1]: Entering directory '/home/user/openwrt'
make[2]: Entering directory '/home/user/openwrt/package/libs/toolchain'
echo "libc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libgcc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libatomic" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libstdcpp" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libpthread" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "librt" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
make[2]: Leaving directory '/home/user/openwrt/package/libs/toolchain'
time: package/libs/toolchain/compile#0.14#0.18#0.31
make[2]: Entering directory '/home/user/openwrt/package/network/utils/hping3'
rm -f /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built
touch /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built_check
ccache_cc -O2 -ftree-vectorize -march=ivybridge -mfpmath=sse -mabi=sysv -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769=hping3-2021-12-07-3547c769 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/user/openwrt/staging_dir/target-x86_64_musl/usr/include -ffunction-sections -fdata-sections -o /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/hping3
/home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/lib/crt1.o: in function `_start_c':
/home/user/openwrt/build_dir/toolchain-x86_64_gcc-11.2.0_musl/musl-1.2.2/crt/crt1.c:18: undefined reference to `main'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:60: /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built] Error 1
make[2]: Leaving directory '/home/user/openwrt/package/network/utils/hping3'
time: package/network/utils/hping3/compile#1.12#0.19#1.29
ERROR: package/network/utils/hping3 failed to build.
make[1]: *** [package/Makefile:116: package/network/utils/hping3/compile] Error 1
make[1]: Leaving directory '/home/user/openwrt'
make: *** [/home/user/openwrt/include/toplevel.mk:230: package/network/utils/hping3/compile] Error 2
I think you need to run a make make package/network/utils/hping3/clean first to get all the errors.
Tried here myself and it's complaining about some byteorder file. The ./configure script does some weird stuff. It doesn't seem to handle cross compilation well.
This part is the one causing me grief:
CC=${CC:=cc}
echo build byteorder.c...
$CC byteorder.c -o byteorder || exit 1
INSTALL_MANPATH=`echo $MANPATH|cut -f1 -d:`
if [ "$INSTALL_MANPATH" = "" ]; then
INSTALL_MANPATH="/usr/local/man"
fi
BYTEORDER=`./byteorder -m`
echo create byteorder.h...
cat > byteorder.h <<EOF
#ifndef __BYTEORDER_H
#define __BYTEORDER_H
EOF
echo \#ifndef $BYTEORDER >> byteorder.h
echo \#define $BYTEORDER >> byteorder.h
echo \#endif /\* $BYTEORDER \*/ >> byteorder.h
cat >> byteorder.h <<EOF
#endif /* __BYTEORDER_H */
EOF
It compiles byteorder with the cross compiler, and then tries to run it after. If you don't QEMU setup with binfmt, that's gonna fail hard. Gonna look at what the byteorder program does and see if we can be rid of it
2 Likes