You are right. I forgot one of the main sources of a working technical solution to the problem of ascertaining direction of bufferbloat. Your contribution is so much appreciated in this respect, @tievolu! You cracked one of the major problems.
From memory you use nping, right? So did you face the issues identified above with nping? That it is a little slow? I have found results are improved by getting the tick duration really small, which seems impossible with nping.
Yeah, I use nping. I ping three reflectors once every two seconds, with the ping requests set to time out after one second. Now that you mention it I did find that things would break if I tried to do it more frequently than that, but for my own purposes two seconds is sufficient.
My script still uses the tactic of resting at max bandwidth and reducing it temporarily if latency gets bad, with "bad latency" defined as at least 3 out of the last 10 requests/replies exceeding 30ms for all reflectors.
Usually bad latency looks like the example above, with all three reflectors returning bad results at the same time, all back to back, and only one bandwidth reduction is needed to address it, so the script typically mitigates bufferbloat after about six seconds.
My script is very much tailored to my own connection and I haven't made any attempt to implement a moving ping baseline or any of the other clever stuff you guys are working on. Another thing to note is that I'm doing all this on a Qotom x86 box, so results might be different on more typical router hardware.
Had a feeling this question would be waiting for me when I got home 
But no not yet. I wanted to test on the router, and for that I need to break out a cross-compilation toolchain.
Safest way to get the dependencies right is to make a package for it I'd assume, so I'd gotta remember how that works again as well. Or I could always compile a static binary so there's no dependency issues.
And I'm not entirely sure how to use the thing, the notes in the fork helps, but I still don't know what the MAC addresses are for.
Here's where I sit with compiling it. Not great to be honest...
user@4fbfcf8066d8:~/openwrt$ make package/network/utils/hping3/compile V=sc | tee build.log
make[1]: Entering directory '/home/user/openwrt'
make[2]: Entering directory '/home/user/openwrt/package/libs/toolchain'
echo "libc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libgcc" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libatomic" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libstdcpp" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "libpthread" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
echo "librt" >> /home/user/openwrt/staging_dir/target-x86_64_musl/pkginfo/toolchain.default.install
make[2]: Leaving directory '/home/user/openwrt/package/libs/toolchain'
time: package/libs/toolchain/compile#0.14#0.18#0.31
make[2]: Entering directory '/home/user/openwrt/package/network/utils/hping3'
rm -f /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built
touch /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built_check
ccache_cc -O2 -ftree-vectorize -march=ivybridge -mfpmath=sse -mabi=sysv -pipe -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769=hping3-2021-12-07-3547c769 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/user/openwrt/staging_dir/target-x86_64_musl/usr/include -ffunction-sections -fdata-sections -o /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/hping3
/home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/bin/ld: /home/user/openwrt/staging_dir/toolchain-x86_64_gcc-11.2.0_musl/lib/gcc/x86_64-openwrt-linux-musl/11.2.0/../../../../x86_64-openwrt-linux-musl/lib/crt1.o: in function `_start_c':
/home/user/openwrt/build_dir/toolchain-x86_64_gcc-11.2.0_musl/musl-1.2.2/crt/crt1.c:18: undefined reference to `main'
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:60: /home/user/openwrt/build_dir/target-x86_64_musl/hping3-2021-12-07-3547c769/.built] Error 1
make[2]: Leaving directory '/home/user/openwrt/package/network/utils/hping3'
time: package/network/utils/hping3/compile#1.12#0.19#1.29
ERROR: package/network/utils/hping3 failed to build.
make[1]: *** [package/Makefile:116: package/network/utils/hping3/compile] Error 1
make[1]: Leaving directory '/home/user/openwrt'
make: *** [/home/user/openwrt/include/toplevel.mk:230: package/network/utils/hping3/compile] Error 2
I think you need to run a make make package/network/utils/hping3/clean first to get all the errors.
Tried here myself and it's complaining about some byteorder file. The ./configure script does some weird stuff. It doesn't seem to handle cross compilation well.
This part is the one causing me grief:
CC=${CC:=cc}
echo build byteorder.c...
$CC byteorder.c -o byteorder || exit 1
INSTALL_MANPATH=`echo $MANPATH|cut -f1 -d:`
if [ "$INSTALL_MANPATH" = "" ]; then
INSTALL_MANPATH="/usr/local/man"
fi
BYTEORDER=`./byteorder -m`
echo create byteorder.h...
cat > byteorder.h <<EOF
#ifndef __BYTEORDER_H
#define __BYTEORDER_H
EOF
echo \#ifndef $BYTEORDER >> byteorder.h
echo \#define $BYTEORDER >> byteorder.h
echo \#endif /\* $BYTEORDER \*/ >> byteorder.h
cat >> byteorder.h <<EOF
#endif /* __BYTEORDER_H */
EOF
It compiles byteorder with the cross compiler, and then tries to run it after. If you don't QEMU setup with binfmt, that's gonna fail hard. Gonna look at what the byteorder program does and see if we can be rid of it
@_FailSafe
Alright I think I got it to compile, had to modify sources to get it working
Modified OpenWrt Makefile: https://github.com/Lochnair/hping3/blob/main/Makefile
My changes to source in this repo: https://github.com/Lochnair/hping
YMMV, quick and dirty changes to make it compile. Gonna go test if it actually runs as well
Edit: Makefile is based on the original from when OpenWrt apparently had it in the repo
It's definitely getting further for me with your updated Makefile and source. But I'm still getting a build error (I'm x86_64_musl): https://pastebin.com/Ghq0y8Be
@Lochnair you are one of my present heroes / Jedi Masters of this thread. Thank you for your efforts!
Are you building against master? I built hping successfully with the 21.02.1 SDK for x86_64.
I am but a young Padawan, but I try my best 
I am, yes. I'll try against the 21.02.1 branch a little later this evening. Thanks for the tip!
Okay I dunno why it worked on the older toolchain, but the delaytable variable was defined in both hping2.h and main.c which made GCC / linker mad. Made the one in hping2.h extern and it looks to compile with latest snapshot SDK as well.
That did the trick! I got a clean compile now--thank you! I just had to remember to grab your latest commit hash to update my Makefile. Funny commit messages you have there 
Okay, so the binary technically doesn't error, but also doesn't seem to be functioning? @Lochnair You're not facing the same behavior are you?
root@OpenWrt:~# hping3 1.1.1.1 --icmp --icmp-ts -c 2000 -i u500 -I eth0
HPING 1.1.1.1 (eth0 1.1.1.1): icmp mode set, 28 headers + 0 data bytes
--- 1.1.1.1 hping statistic ---
2000 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
If I don't specify an interface (-I) then it uses lo which obviously won't work:
root@OpenWrt:~# hping3 1.1.1.1 --icmp --icmp-ts -c 2000 -i u500
HPING 1.1.1.1 (lo 1.1.1.1): icmp mode set, 28 headers + 0 data bytes
--- 1.1.1.1 hping statistic ---
2000 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
Yeah lets not talk about those 
Nope, works as expected for me. Although I get no response from 1.1.1.1, I think between the bigger DNS providers only 9.9.9.9 answered timestamp requests @Lynx?
Weird, I'm not seeing that behavior. Shows the WAN interface as it should here.
Yes I think 9.9.9.9 answers timestamp ICMP whereas 1.1.1.1 and 8.8.8.8 have been configured not to.
I'll keep poking around at it this evening. I'm wondering if it's something unique to my configuration. I just am not getting any responses with hping3:
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -c 2000 -i u500 -I eth0
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
--- 9.9.9.9 hping statistic ---
2000 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms
root@OpenWrt:~# ping 9.9.9.9
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=57 time=24.6 ms
64 bytes from 9.9.9.9: icmp_seq=2 ttl=57 time=25.1 ms
64 bytes from 9.9.9.9: icmp_seq=3 ttl=57 time=20.9 ms
64 bytes from 9.9.9.9: icmp_seq=4 ttl=57 time=23.1 ms
64 bytes from 9.9.9.9: icmp_seq=5 ttl=57 time=21.7 ms
^C
--- 9.9.9.9 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 20.885/23.072/25.149/1.639 ms
Update... I think I see what's going on. My eth0 (WAN) iface has an additional static IP on my cable modem's subnet so I can communicate with my modem's status page. That is the IP that hping3 is assuming when I specify eth0. I'll pull that IP off the iface in a bit and test again.
Update 2... Yep--it was having the modem subnet (192.168.100.0/24) IP statically set on my eth0 iface that was messing with hping3. Took that off the iface and I'm seeing success here!
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -c 50 -i u500
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=57 id=35394 icmp_seq=0 rtt=19.9 ms
ICMP timestamp: Originate=3499462 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=20
len=46 ip=9.9.9.9 ttl=57 id=35395 icmp_seq=1 rtt=19.4 ms
ICMP timestamp: Originate=3499463 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=19
len=46 ip=9.9.9.9 ttl=57 id=35396 icmp_seq=2 rtt=18.9 ms
ICMP timestamp: Originate=3499463 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=19
len=46 ip=9.9.9.9 ttl=57 id=35397 icmp_seq=3 rtt=18.4 ms
ICMP timestamp: Originate=3499464 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=18
len=46 ip=9.9.9.9 ttl=57 id=35398 icmp_seq=4 rtt=17.9 ms
ICMP timestamp: Originate=3499464 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=18
len=46 ip=9.9.9.9 ttl=57 id=35399 icmp_seq=5 rtt=17.4 ms
ICMP timestamp: Originate=3499465 Receive=3499470 Transmit=3499470
ICMP timestamp RTT tsrtt=17
len=46 ip=9.9.9.9 ttl=57 id=35404 icmp_seq=6 rtt=46.8 ms
ICMP timestamp: Originate=3499466 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=46
len=46 ip=9.9.9.9 ttl=57 id=35405 icmp_seq=7 rtt=46.3 ms
ICMP timestamp: Originate=3499466 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=46
len=46 ip=9.9.9.9 ttl=57 id=35406 icmp_seq=8 rtt=45.8 ms
ICMP timestamp: Originate=3499467 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=45
len=46 ip=9.9.9.9 ttl=57 id=35407 icmp_seq=9 rtt=45.3 ms
ICMP timestamp: Originate=3499467 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=45
len=46 ip=9.9.9.9 ttl=57 id=35408 icmp_seq=10 rtt=44.8 ms
ICMP timestamp: Originate=3499468 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=44
len=46 ip=9.9.9.9 ttl=57 id=35409 icmp_seq=11 rtt=44.3 ms
ICMP timestamp: Originate=3499468 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=44
len=46 ip=9.9.9.9 ttl=57 id=35410 icmp_seq=12 rtt=43.8 ms
ICMP timestamp: Originate=3499469 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=43
len=46 ip=9.9.9.9 ttl=57 id=35411 icmp_seq=13 rtt=43.3 ms
ICMP timestamp: Originate=3499469 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=43
len=46 ip=9.9.9.9 ttl=57 id=35412 icmp_seq=14 rtt=42.8 ms
ICMP timestamp: Originate=3499470 Receive=3499480 Transmit=3499480
ICMP timestamp RTT tsrtt=42
len=46 ip=9.9.9.9 ttl=57 id=35420 icmp_seq=15 rtt=42.3 ms
ICMP timestamp: Originate=3499470 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=43
len=46 ip=9.9.9.9 ttl=57 id=35421 icmp_seq=16 rtt=41.8 ms
ICMP timestamp: Originate=3499471 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=42
len=46 ip=9.9.9.9 ttl=57 id=35422 icmp_seq=17 rtt=41.3 ms
ICMP timestamp: Originate=3499471 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=42
len=46 ip=9.9.9.9 ttl=57 id=35423 icmp_seq=18 rtt=40.8 ms
ICMP timestamp: Originate=3499472 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=41
len=46 ip=9.9.9.9 ttl=57 id=35424 icmp_seq=19 rtt=40.3 ms
ICMP timestamp: Originate=3499472 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=41
len=46 ip=9.9.9.9 ttl=57 id=35425 icmp_seq=20 rtt=39.7 ms
ICMP timestamp: Originate=3499473 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=40
len=46 ip=9.9.9.9 ttl=57 id=35426 icmp_seq=21 rtt=39.2 ms
ICMP timestamp: Originate=3499473 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=40
len=46 ip=9.9.9.9 ttl=57 id=35427 icmp_seq=22 rtt=38.7 ms
ICMP timestamp: Originate=3499474 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=39
len=46 ip=9.9.9.9 ttl=57 id=35428 icmp_seq=23 rtt=38.2 ms
ICMP timestamp: Originate=3499474 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=39
len=46 ip=9.9.9.9 ttl=57 id=35429 icmp_seq=24 rtt=37.7 ms
ICMP timestamp: Originate=3499475 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=38
len=46 ip=9.9.9.9 ttl=57 id=35430 icmp_seq=25 rtt=37.1 ms
ICMP timestamp: Originate=3499475 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=38
len=46 ip=9.9.9.9 ttl=57 id=35431 icmp_seq=26 rtt=36.6 ms
ICMP timestamp: Originate=3499476 Receive=3499490 Transmit=3499490
ICMP timestamp RTT tsrtt=37
len=46 ip=9.9.9.9 ttl=57 id=35436 icmp_seq=27 rtt=36.1 ms
ICMP timestamp: Originate=3499477 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=36
len=46 ip=9.9.9.9 ttl=57 id=35437 icmp_seq=28 rtt=35.6 ms
ICMP timestamp: Originate=3499477 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=36
len=46 ip=9.9.9.9 ttl=57 id=35438 icmp_seq=29 rtt=35.0 ms
ICMP timestamp: Originate=3499478 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=35
len=46 ip=9.9.9.9 ttl=57 id=35439 icmp_seq=30 rtt=34.5 ms
ICMP timestamp: Originate=3499478 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=35
len=46 ip=9.9.9.9 ttl=57 id=35440 icmp_seq=31 rtt=34.0 ms
ICMP timestamp: Originate=3499479 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=34
len=46 ip=9.9.9.9 ttl=57 id=35441 icmp_seq=32 rtt=33.4 ms
ICMP timestamp: Originate=3499479 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=34
len=46 ip=9.9.9.9 ttl=57 id=35442 icmp_seq=33 rtt=32.9 ms
ICMP timestamp: Originate=3499480 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=33
len=46 ip=9.9.9.9 ttl=57 id=35443 icmp_seq=34 rtt=32.4 ms
ICMP timestamp: Originate=3499480 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=33
len=46 ip=9.9.9.9 ttl=57 id=35444 icmp_seq=35 rtt=31.9 ms
ICMP timestamp: Originate=3499481 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=32
len=46 ip=9.9.9.9 ttl=57 id=35445 icmp_seq=36 rtt=31.3 ms
ICMP timestamp: Originate=3499481 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=32
len=46 ip=9.9.9.9 ttl=57 id=35446 icmp_seq=37 rtt=30.8 ms
ICMP timestamp: Originate=3499482 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=31
len=46 ip=9.9.9.9 ttl=57 id=35447 icmp_seq=38 rtt=30.3 ms
ICMP timestamp: Originate=3499482 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=31
len=46 ip=9.9.9.9 ttl=57 id=35448 icmp_seq=39 rtt=29.8 ms
ICMP timestamp: Originate=3499483 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=30
len=46 ip=9.9.9.9 ttl=57 id=35449 icmp_seq=40 rtt=29.3 ms
ICMP timestamp: Originate=3499483 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=30
len=46 ip=9.9.9.9 ttl=57 id=35450 icmp_seq=41 rtt=28.8 ms
ICMP timestamp: Originate=3499484 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=29
len=46 ip=9.9.9.9 ttl=57 id=35451 icmp_seq=42 rtt=28.3 ms
ICMP timestamp: Originate=3499484 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=29
len=46 ip=9.9.9.9 ttl=57 id=35452 icmp_seq=43 rtt=27.8 ms
ICMP timestamp: Originate=3499485 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=28
len=46 ip=9.9.9.9 ttl=57 id=35453 icmp_seq=44 rtt=27.3 ms
ICMP timestamp: Originate=3499486 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=27
len=46 ip=9.9.9.9 ttl=57 id=35454 icmp_seq=45 rtt=26.8 ms
ICMP timestamp: Originate=3499486 Receive=3499498 Transmit=3499498
ICMP timestamp RTT tsrtt=27
len=46 ip=9.9.9.9 ttl=57 id=35456 icmp_seq=46 rtt=26.2 ms
ICMP timestamp: Originate=3499487 Receive=3499500 Transmit=3499500
ICMP timestamp RTT tsrtt=26
len=46 ip=9.9.9.9 ttl=57 id=35457 icmp_seq=47 rtt=25.7 ms
ICMP timestamp: Originate=3499487 Receive=3499500 Transmit=3499500
ICMP timestamp RTT tsrtt=26
len=46 ip=9.9.9.9 ttl=57 id=35458 icmp_seq=48 rtt=25.2 ms
ICMP timestamp: Originate=3499488 Receive=3499500 Transmit=3499500
ICMP timestamp RTT tsrtt=25
len=46 ip=9.9.9.9 ttl=57 id=35459 icmp_seq=49 rtt=24.7 ms
ICMP timestamp: Originate=3499488 Receive=3499500 Transmit=3499500
ICMP timestamp RTT tsrtt=25
--- 9.9.9.9 hping statistic ---
50 packets tramitted, 50 packets received, 0% packet loss
round-trip min/avg/max = 17.4/33.7/46.8 ms
FINALLY
root@turris:/tmp# hping3 9.9.9.9 --icmp --icmp-ts -c 2000 -i u500 > ping.txt
--- 9.9.9.9 hping statistic ---
2000 packets tramitted, 2000 packets received, 0% packet loss
round-trip min/avg/max = 8.9/18.8/28.9 ms
i saved the (quite big) output to a file, output can be viewed here: https://www.toptal.com/developers/hastebin/ogovewoyoh.apache
Since both of us seem to be using turris routers, would you share the hping3.ipk? I could participate in testing then...
OpenDNS and Comodo DNS also respond to timestamped ping requests. A sizeable list of potential reflectors can be found here. (I compiled this list by simply trying out all the hosts listed as DoH servers here.)
Sure!
Edit for clarification: this is hping3.ipk for Turris Omnia, compiled with latest build at this Time (Turris OS 5.3.1)
https://easyupload.io/d5zc69