There is no fix except patching hping3 (which I won't do, because we have users on systems other than OpenWRT) or installing coreutils-stdbuf.
BTW @patrakov will hping3 also suffer from the drift issue? I am not sure how it controls sending out requests - whether those wait for responses or not. The OWD handling seems nice and it can go to very very small intervals for custom VPS solutions.
You're talking about ucode?
From a cursory reading it doesn't look like (currently) supports sockets, nor does the system function let you get stdout/err from an external command.
I guess it would be possible to call a ucode script from C, and handle functions ucode doesn't support in the C part.
Sure, I can try to whip something up.
I haven't looked at your script in a long time now, so I'd need some pointers on what exactly you'd need from it 
Oh sweet! Well we've been experimenting with different ping binaries. And I think two desirable characteristics would be:
- icmp-timestamp support like hping3:
root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=56 id=32110 icmp_seq=0 rtt=67.9 ms
ICMP timestamp: Originate=33554447 Receive=33554476 Transmit=33554476
ICMP timestamp RTT tsrtt=68
- round robin support like fping:
root@OpenWrt:~# fping 1.1.1.1 9.9.9.9 -c 10
1.1.1.1 : [0], 64 bytes, 38.0 ms (38.0 avg, 0% loss)
9.9.9.9 : [0], 64 bytes, 42.6 ms (42.6 avg, 0% loss)
You see I found that setting up multiple instances of pingers results in drift in the sends so that whilst I can start them off with nice granularity, the sends can drift resulting in undesirable spacing between the sends.
As far as I know there is no ping utility that simultaneously supports both options as above. @moeller0 asked the fping guys if they might add ICMP type 13 support on GitHub, but who knows what will happen with that.
- one line per reflector response:
Output on just one line per response would also be desirable to simplify line-based reads/processing. At the moment I add a wrapper per ping binary, have the binary output to a fifo, and then ultimately read in lines from the fifo (link to code) like this:
while read -t $stall_detection_timeout_s timestamp reflector seq dl_owd_baseline_us dl_owd_us dl_owd_delta_us ul_owd_baseline_us ul_owd_us ul_owd_delta_us
...
done</tmp/cake-autorate/ping_fifo
Overall fping works well but it doesn't support OWDs.
Summary
So ultimately from my perspective the C program would desirably output one line per ICMP type 13 response from multiple reflectors, this line providing the extractable owd info for processing in a bash while/read loop.
For me having a custom C-originating binary for this would be super exciting. Bash facilitates rapid customisation and control, and I think the existing code works well for several users now, but being limited to the existing ping binaries available in OpenWrt has been a challenge, and so a custom OpenWrt-originating solution would really help unlock more possibilities.
@patrakov, @moeller0 any other desirable properties of a custom C-based pinger for these autorate utilities?
Ability to set the source IP, interface, or firewall mark. Not critical, because one can wrap it into mwan3 use lte custom-pinger --args.
1 Like
-
Is obviously already covered, but I guess it wouldn't hurt to have it handle ICMP echo as well?
I'd imagine it would simplify your code a bit if you don't have to interpret different outputs -
Aah, okay I think I understand better why you were talking about drift previously now
What it currently does is loop through the reflectors, so that should be covered. I think the easiest improvement that can be made to the C code to avoid drift is switching from nanosleep (only relative time) to clock_nanosleep (allows absolute time). That should - based on my man page reading be more reliable. -
And yes definitely.
As far as I can tell a firewall mark only needs the SO_MARK sockopt set, so that should be a simply addition.
1 Like
Actually I would like to see something like:
timeout=B; timestamp=XXXXX.XXXX; ip=N.N.N.N; id=NN; icmp_seq=N; originate= YYYYYY; receive=YYYYYY; transmit= YYYYYY; selfreceive= YYYYYY; RTT=XXXXX.XXXX
so something which condenses the desired information into a single line AND also includes the selfreceive timestamp. Hping3 does not seem to report that so it needs to be calculated from origonate+rtt. The idea about timeout is to explicitly signal a record where no response was gotten in time...
I still think that a more robust approach would be to fix these drifts up periodically instead od requiring a drift free delay source, but the latter sure makes the bash script simpler 
Yes, see above it should contain all relevant information conveniently packaged into a single line for easier parsing.
+1.
Yes that would be nice for A/B testing.
I do not think that is required, The problem is that with iputils-ping with one process per reflector over some time all pings are sent pretty much at the same time instead of nicely interleaved. For a delay collector that samples multiple reflectors it would be enough if the interval between sending requests ist more or less constant. So drift against the wall clock is acceptable, drift and lumping of probe transmissions is not.
1 Like
The next-generation branch looks intriguing. (I haven't had a chance to try it.)
Would you also update the Changelog to talk about the features of 1.1 and adjust the cake-autorate-setup.sh script to pull in the proper packages, etc? it might be worth a short note at the head of the README noting the 1.1 version, with a link to the Changelog for details. Thanks!
1 Like
Thank you @moeller0 for doing this calculation. It confirmed my intuition that pinging "all the time" would burn up a lot of data. (A couple US datapoints: my "standard" phone has 2GBytes of data per month; my cellphone modem has 10Gbytes/month. Each weighs in around $40/month.)
Fortunately, CAKE-autorate doesn't ping all the time - only when there is traffic.
Furthermore, we can develop guidance to say that if you're worried about your cell data plan, then you could decrease reflector_ping_interval_s significantly. (Cell modem latency is dreadful all on its own - varying between 10's of milliseconds up to a couple hundred msec moment to moment - even on an unloaded link. So attempting to set CAKE parameters from sub-second measurements is not likely to be useful.)
1 Like
Could pinging this way raise any red flags?
How many pings would it take for a reflector or a service provider to suspect malicious activity?
Presumably at some degree. I think most here would agree that testing big hitters like Google or Cloudflare at 5 Hz per reflector (interval 0.2 seconds) is not worthy of a public flogging.
1 Like
It looks like bufferbloat is happening but it's not being detected because the maximum permitted latencies are too high.
Try being a bit stricter:
dl_max_idle_latency = 200
dl_max_loaded_latency = 250
ul_max_idle_latency = 200
ul_max_loaded_latency = 250
We do not really know. One more reason why the bash code allows to interleave reflectors to synthesize a high effective delay measurement rate without over taxing individual reflectors and why all approaches have some reflector sanity checking and recycling.Also this is why the sampling rates are configurable
@patrakov has a very finicky ISP that allows only relative small ICMP rates, requiring him to reduce the sampling rate considerably.
The thing is, it really does not matter on what we agree here ;). Observation however seems to indicate that rates up to 10 Hz seem to not cause problems and some reflectors might allow even more. I would expect that reflectors might have a global ICMP limit and not a limit per end point, so the acceptable rate per reflector might change deoending on the current total ICMP load on a server.
1 Like
Having been on the receiving end of high frequency pinging in the past, I'd beg to differ (and that were just 5000-20000 rogue clients trying to download a small static file and checking for changes from a beefy root server). Obviously google and cloudflare have other means and icmp is a little lighter than http get, but once it crosses a certain threshold, it will be noticed and acted upon.
1 Like
So you think 5 Hz pinging is excessive? Or that there is some threshold beyond 5 Hz that is excessive? Nobody doubts the latter, but can we really fault sending ICMP requests to a big-name reflector at 0.2 seconds per request, i.e. 5 Hz? Really?
Based on my experiences as server admin, I'd call anything doing automated pings more often than once in 30s 24/7 excessive and wouldn't accept any code pinging more often than once every 5 minutes (and you'd have to have very good reasons to go that low, rather than 15-20 minutes). A few hundreds of these clients will already generate significant load on the server.
It's all fine with just a handful of clients, but a few hundreds and rising will be an issue that would need heavy handed attention.
2 Likes
We don't have to only use "big name" reflectors. There are literally thousands of ICMP type 13 reflectors out there, and probably 1000x as many that will respond to regular pings.
A while back I wrote some code to go looking for ICMP type 13 reflectors and found over 100,000 of them:
In my perl autorate implementation I do round robin pings at a rate of 10Hz to a pool of 30 reflectors (randomly chosen from a list of ~5000 for my region), so each one is never pinged more than once every 1.5s. I could easily use an even bigger pool to reduce the frequency per reflector even further.
If a reflector misbehaves I just throw it away and grab a new random reflector from the list.
That is only a subjective opinion though isn't it? If you look into the publicly documented infrastructure of Cloudflare and the like it is quite staggering.