CAKE w/ Adaptive Bandwidth

@moeller0 finally - OWDs in cake-autorate using @Lochnair's tsping binary.

@patrakov would you like to test?

Samples per reflector:
ReflectorID: 185.228.168.10; N: 91
ReflectorID: 185.228.168.9; N: 91
ReflectorID: 9.9.9.11; N: 90
ReflectorID: 9.9.9.9; N: 88
DL: maximum 95.000%-ile delta delay over all 4 reflectors: 18.670 ms.
DL: maximum 99.000%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.500%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.900%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.950%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.990%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.999%-ile delta delay over all 4 reflectors: 38.850 ms.
UL: maximum 95.000%-ile delta delay over all 4 reflectors: 4.990 ms.
UL: maximum 99.000%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.500%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.900%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.950%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.990%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.999%-ile delta delay over all 4 reflectors: 11.985 ms.
INFO: Writing plot as: ./output.timecourse.pdf
INFO: fn_parse_autorate_log took: 16.0685 seconds.
1 Like

Not necessarily today, as the LTE link is good at night (20+ Mbps) and doesn't really need cake-autorate. And I won't have anything demanding (like Discord) until Wednesday.

Regarding mimicking fping options - yes I agree that it makes sense. That's assuming a sane interpretation of the timeout, in particular the case when it is greater than the period.

1 Like

Why? It is not that fping is a widely used binary with lots of documentation...

also does fping allow period smaller than interval?

Yes it does:

hms-beagle2:~ smoeller$ fping --timestamp --loop --period 100 --interval 200 1.1.1.1 9.9.9.9
[1678125449.51688] 1.1.1.1 : [0], 64 bytes, 33.5 ms (33.5 avg, 0% loss)
[1678125449.71822] 9.9.9.9 : [0], 64 bytes, 32.9 ms (32.9 avg, 0% loss)
[1678125449.92214] 1.1.1.1 : [1], 64 bytes, 33.0 ms (33.2 avg, 0% loss)
[1678125450.12517] 9.9.9.9 : [1], 64 bytes, 33.3 ms (33.1 avg, 0% loss)
[1678125450.33064] 1.1.1.1 : [2], 64 bytes, 33.6 ms (33.4 avg, 0% loss)
[1678125450.53283] 9.9.9.9 : [2], 64 bytes, 33.6 ms (33.3 avg, 0% loss)
[1678125450.73605] 1.1.1.1 : [3], 64 bytes, 33.7 ms (33.4 avg, 0% loss)
[1678125450.93910] 9.9.9.9 : [3], 64 bytes, 33.4 ms (33.3 avg, 0% loss)
^C

however the results are "interesting" at best... tspings variant is IMHO better in that you can only specify things the code will deliver...

Normally I am all for re-using existing command line switches and behaviour, but here I am not convinced that fping is the example to follow.

1 Like

I like how apparently both autorate itself as well as the plotting code defaulted to do the right thing once fed with OWDs.

This will need testing and likely there will be one or two issues to iron out:

But it at least allowed me to produce the plot above using @Lochnair's tsping binary, which is available here:

and which includes support for ICMP type 13 (timestamp) requests and responses, and hence working with one way delays (OWDs) rather than round trip times (RTTs).

tsping is not yet an official OpenWrt package, but for anyone wanting to test there are simple instructions for building on OpenWrt here:

For any readers not sure about the significance of this, this facilitates determining the direction of bufferbloat (download or upload) and hence better control over the download and upload shaper rates since they can be controlled more independently in dependence upon the respective download and upload OWDs. Hitherto cake-autorate has fudged the issue by working with RTTs and erred on the side of caution.

Here is an example set of config overrides that work for my connection:

pinger_binary=tsping

reflectors=(
"9.9.9.9" "9.9.9.10" "9.9.9.11" # Quad9
"185.228.168.9" "185.228.168.10" # CleanBrowsing
)

no_pingers=5

ping_prefix_string="stdbuf -oL"

But does this make sense to you:

# tsping --help
Usage: tsping [OPTION...] IP1 IP2 IP3 ...
tsping -- a simple application to send ICMP echo/timestamp requests

  -e, --icmp-echo            Use ICMP echo requests
  -t, --icmp-ts              Use ICMP timestamp requests (default)
  -r, --target-spacing=TIME  Time to wait between pinging each target in ms
                             (default 0)
  -s, --sleep-time=TIME      Time to wait between each round of pinging in ms
                             (default 100)
  -D, --print-timestamps     Print UNIX timestamps for responses
  -m, --machine-readable[=DELIMITER]
                             Output results in a machine readable format
  -f, --fw-mark=MARK         Firewall mark to set on outgoing packets
  -i, --interface=INTERFACE  Interface to bind to
  -?, --help                 Give this help list
      --usage                Give a short usage message

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

With the above, my understanding is that sleep-time 1000ms and target-spacing 500ms means: every 1000ms start round robin and send out first ICMP to first target, and then wait 500ms to send out interval to next target in that round robin.

So effective interval between responses of 500ms.

But I see:

root@OpenWrt-1:~# tsping --print-timestamps --machine-readable=' ' --sleep-time 1000 --target-spacing 500 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678128598.015933 9.9.9.10 0 67797969 67797993 67797993 67798015 46 22 24
1678128599.535897 9.9.9.9 1 67799470 67799504 67799504 67799535 65 31 34
1678128600.016869 9.9.9.10 1 67799970 67799994 67799994 67800016 46 22 24
1678128601.515681 9.9.9.9 2 67801471 67801494 67801494 67801515 44 21 23
1678128602.017769 9.9.9.10 2 67801971 67801993 67801993 67802017 46 24 22
1678128603.516854 9.9.9.9 3 67803471 67803493 67803493 67803516 45 23 22
1678128604.015840 9.9.9.10 3 67803972 67803993 67803993 67804015 43 22 21

That is, 1.5s and 500ms spacing between responses.

This is much clearer to me:

−p, −−period= MSEC

In looping or counting modes (−l, −c, or −C), this parameter sets the time in milliseconds that fping waits between successive packets to an individual target. Default is 1000 and minimum is 10.

−i, −−interval= MSEC

The minimum amount of time (in milliseconds) between sending a ping packet to any target (default is 10, minimum is 1).

Nope, with e.g. tow IPs, you will do:
probe1, 500ms, probe2, 1000ms
so your total period is 1500ms...

And that is what you see, 500ms between the members of the set (9.9.9.9's first response is missing)

  1. 9.9.9.9: 0
  2. 9.9.9.10: 500ms later
    next round
  3. 9.9.9.9: 1000ms after 2), so exactly the requested sleep time.

Except you can specify stuff that fping does not complain about and does not deliver, period < interval...

My take is that you got used to fpings way of specifying this and hence confuse sleep with period, even though these are clearly different concepts. What like about @Lochnair's version is that you can not request impossible timings... it does become a tiny bit more involved to calculate the effective per reflector rate.

1 Like

Ah, I think I get it now. So they are both spacings between sends. Target is the spacing between targets, and sleep is the spacing between rounds (i.e. the spacing between the last send and the next first send)?

Ah, not quite:

root@OpenWrt-1:~# tsping --print-timestamps --machine-readable=' ' --sleep-time 1000 --target-spacing 1000 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678131109.591224 9.9.9.9 0 70309547 70309563 70309563 70309591 44 28 16
1678131110.598166 9.9.9.10 0 70310547 70310574 70310574 70310598 51 24 27
1678131112.597135 9.9.9.9 1 70312547 70312573 70312573 70312597 50 24 26
1678131113.598123 9.9.9.10 1 70313548 70313573 70313573 70313598 50 25 25
1678131115.609165 9.9.9.9 2 70315548 70315584 70315584 70315609 61 25 36
1678131116.598139 9.9.9.10 2 70316548 70316573 70316573 70316598 50 25 25

So there is target + sleep between rounds? But it should just be sleep right? Or?

Ah, now I understand your question @Lochnair:

Yes, I think it should be skipped. @moeller0 do you agree?

Good question, I guessthe question is convenience, do you want to be able to specify a "normal" continous sampling with just one number, or do you essentially always want having to configure sleep >= target. I honestly see only two usecases:
a) equitemporal sampling like autorate does now
b) batched sampling where you queuery with target 0 and then use sleep to define the effective period

since we mostly use a) I am fine with not having to configure sleep, but the behaviour and help text should be adjusted to:

  -s, --sleep-time=TIME      Time to wait between each round of pinging in ms
                             (default to target for equidistant sampling)

I have already filed a bug with a different (more consistent) proposal: https://github.com/Lochnair/tsping/issues/7

@patrakov any clue why this happens:

root@OpenWrt-1:~# tsping --print-timestamps --machine-readable=' ' --sleep-time 100 --target-spacing 1000 9.9.9.10 9.9.9.9
Starting tsping 0.2.2 - pinging 2 targets
1678132116.263362 9.9.9.10 0 71316226 71316239 71316239 71316263 37 24 13
1678132118.388443 9.9.9.10 1 71318327 71318364 71318364 71318388 61 24 37
1678132119.377351 9.9.9.9 1 71319327 71319353 71319353 71319377 50 24 26
1678132120.490509 9.9.9.10 2 71320428 71320464 71320464 71320490 62 26 36

root@OpenWrt-1:~# stdbuf -oL tsping --print-timestamps --machine-readable=' ' --sleep-time 100 --target-spacing 1000 9.9.9.10 9.9.9.9
Starting tsping 0.2.2 - pinging 2 targets
1678132123.268491 9.9.9.10 0 71323218 71323244 71323244 71323268 50 24 26
Something went wrong while sending: -1
1678132125.367498 9.9.9.10 1 71325318 71325343 71325343 71325367 49 24 25
Something went wrong while sending: -1
1678132127.468529 9.9.9.10 2 71327419 71327443 71327443 71327468 49 25 24
Something went wrong while sending: -1

For some reason the address family is corrupted:

[pid 26979] sendto(3, "\r\0\346\253ia\0\0\4B\236\260\0\0\0\0\0\0\0\0", 20, 0, {sa_family=0x7c60 /* AF_??? */, sa_data="\36\205\t\t\t\t\270\352'\205\177\0\0\0"}, 16) = -1 EAFNOSUPPORT (Address family not supported by protocol)

I have compiled it on the desktop, and valgrind complains:

==25044== Thread 3:
==25044== Syscall param socketcall.sendto(to.sa_family) points to uninitialised byte(s)
==25044==    at 0x498DEAC: sendto (sendto.c:27)
==25044==    by 0x10AB28: send_icmp_timestamp_request (main.c:94)
==25044==    by 0x10B60C: sender_loop (main.c:304)
==25044==    by 0x4909BB4: start_thread (pthread_create.c:444)
==25044==    by 0x498BCB3: clone (clone.S:100)
==25044==  Address 0x4a6e3d0 is 0 bytes inside a block of size 32 alloc'd
==25044==    at 0x4846CC3: realloc (vg_replace_malloc.c:1437)
==25044==    by 0x10A76B: parse_opt (args.h:103)
==25044==    by 0x499846C: group_parse (argp-parse.c:257)
==25044==    by 0x499846C: parser_parse_arg (argp-parse.c:693)
==25044==    by 0x499846C: parser_parse_next (argp-parse.c:865)
==25044==    by 0x499846C: argp_parse (argp-parse.c:921)
==25044==    by 0x10B747: main (main.c:341)
1 Like

Simply do not do this, white space is arguable the worst possible delimiter use something explicit here like ; that will allow you to detect empty fields. All you need to do is something
like:

orig_IFs=${IFS}
IFS=";" read -t ...
IFS=${orig_IFS}

to have read use proper delimiters...

root@OpenWrt-1:~# stdbuf -oL tsping --print-timestamps --machine-readable='=' --sleep-time 100 --target-spacing 1000 9.9.9.10 9.9.9.9
Starting tsping 0.2.2 - pinging 2 targets
1678133127.068839=9.9.9.10=0=72327019=72327044=72327044=72327068=49=24=25
Something went wrong while sending: -1
1678133129.167795=9.9.9.10=1=72329120=72329143=72329143=72329167=47=24=23

This was a general remark about not using ' ' or '/tab' as delimiters... delimiters need to be human readable and allow for unambiguous encoding of empty fields.

diff --git a/args.h b/args.h
index 3296020..6262cbd 100644
--- a/args.h
+++ b/args.h
@@ -101,6 +101,8 @@ parse_opt (int key, char *arg, struct argp_state *state)
                
                case ARGP_KEY_ARG:
                        arguments->targets = realloc(arguments->targets, (arguments->targets_len + 1) * sizeof(struct sockaddr_in));
+                       arguments->targets[arguments->targets_len].sin_family = AF_INET;
+                       arguments->targets[arguments->targets_len].sin_port = 0;
 
                        if (inet_pton(AF_INET, arg, &arguments->targets[arguments->targets_len].sin_addr) != 1) {
                                printf("Invalid IP address: %s\n", arg);

Tested on desktop and on the router, this gets rid of the error.

@Lochnair could you please release a fixed version? I don't care for #7 (sleep duration syntax) right now, but uninitialized variables are important. And this is a good time to add the setbuf() call.

2 Likes

I had a go at one of your irtt data sets (with octave*) and I am probably not understanding the data all that well, but could it be that the clocks of both sides are badly synchronized and both drifting? That, or I am looking at the wrong fields of irtt's json output :wink:

*) I use matlab at work so this is the only thing I am somewhat fluent in, but boy compared to commercial matlab, octave is a bumpy ride...

P.S.: The json parsing is still dog-slow, but you only need to do this once per input file, I just needed to find a loading function that actually works, and then massage irtt's output into a simple 2D table to make plotting simple, but since that is now done, refining the plot should not be witchcraft.

P.P.S.: Here is the current state of the plotting:


Since the clocks are out of sync this only shows the RTT, but adds the inter-quartile mean (the number we assume Ookla's speedtests.net reports) and the number of lost probes separated by direction. What is missing are for each of rtt/send/receive CDF and distribution plots. Unlike with the autorate data we have no achieved throughput or CPU load numbers to show concurrently... However irtt timestamps appear to be unix epoch in nanoseconds, so this could be co-plotted with autorate data containing throughput data....

2 Likes

Good catch. Will get this fixed in a moment.

I did try to use Valgrind to catch issues like this, but had some issues with missing debug symbols in glibc that I didn't get fixed before tonight.

1 Like

Really neat that you've written this. I'm not aware of any decent ping binary that offers round robin ICMP type 13 requests. So a nice addition to the OpenWrt landscape.