CAKE w/ Adaptive Bandwidth [August 2022 to March 2024]

Sleep between rounds happens after the last send.

I'll just quote the whole code for the sending loop here:

while (1)
	{
		if (sentICMP > 0 && sentICMP % 100 == 0 && receivedICMP == 0) {
			fprintf(stderr, "Warning: We've sent %lu packets, but received none.\n", sentICMP);
		}

		for (int i = 0; i < targets_len; i++)
		{
			switch (args->icmp_type) {
				case 8:
					send_icmp_echo_request(sock_fd, &targets[i], thread_data->id, htons(seq));
					break;
				case 13:
					send_icmp_timestamp_request(sock_fd, &targets[i], thread_data->id, htons(seq));
					break;
				default:
					fprintf(stderr, "sender: Invalid ICMP type: %u, exiting..\n", args->icmp_type);
					pthread_exit(NULL);
			}

			if (args->target_spacing > 0 && args->targets_len > 1)
				nanosleep(&spacing_time, NULL);
		}

		seq++;
		if (args->sleep_time > 0)
			nanosleep(&sleep_time, NULL);
	}

I didn't consider this when I first wrote it, but this nanosleep(&spacing_time, NULL); should probably be skipped after the last target has been pinged, or?

Not sure. But why does this come through in batches:

tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9 9.9.9.10 9.
9.9.11 185.228.168.9 185.228.168.10 | while read -r line; do echo ${line}; done

Whereas this is smooth:

root@OpenWrt-1:~/cake-autorate#  fping 9.9.9.9 --loop | while read line; do echo ${line}; done
9.9.9.9 : [0], 64 bytes, 49.5 ms (49.5 avg, 0% loss)
9.9.9.9 : [1], 64 bytes, 49.8 ms (49.6 avg, 0% loss)
9.9.9.9 : [2], 64 bytes, 48.9 ms (49.4 avg, 0% loss)

Ah, stackoverflow has the answer for that one I believe:

Also, just wanted to mention that apparently in UNIX a newline will typically only flush the buffer if stdout is a terminal. If the output is being redirected to a file, a newline won't flush.

Didn't catch this since it works without a pipe, I guess buffering for stdout needs to be disabled entirely

Can that be done from the tsping binary?

Yes, the SO answer says you can disable buffering for stdout as simply as setbuf(stdout, NULL);, so it's an easy fix

1 Like

Sweet. Any chance you could push those fixes? I've finished the tsping wrapper for cake-autorate and dying to test this out!

Please use stdbuf -oL as a wrapper as a temporary workaround. To solve the problem properly, please add a call to setvbuf() to the C code.

1 Like

Why does stdbuf -oL change the operation of tsping in this way? See:

rroot@OpenWrt-1:~/cake-autorate# stdbuf -oL tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9
9.9.9.10 9.9.9.11 185.228.168.9 185.228.168.10
Starting tsping 0.2.2 - pinging 5 targets
1678121889.858740 9.9.9.9 0 61089809 61089834 61089834 61089858 49 24 25
Something went wrong while sending: -1
1678121890.457717 9.9.9.11 0 61090409 61090433 61090433 61090457 48 24 24
1678121890.763506 185.228.168.9 0 61090710 61090737 61090737 61090763 53 26 27
1678121891.059733 185.228.168.10 0 61091010 61091035 61091035 61091059 49 24 25
1678121891.358621 9.9.9.9 1 61091310 61091333 61091333 61091358 48 25 23
Something went wrong while sending: -1
1678121891.957635 9.9.9.11 1 61091911 61091933 61091933 61091957 46 24 22
1678121892.258752 185.228.168.9 1 61092212 61092236 61092236 61092258 46 22 24
1678121892.558668 185.228.168.10 1 61092512 61092534 61092534 61092558 46 24 22
1678121892.857691 9.9.9.9 2 61092812 61092833 61092833 61092857 45 24 21
Something went wrong while sending: -1
1678121893.457572 9.9.9.11 2 61093413 61093433 61093433 61093457 44 24 20
1678121893.758712 185.228.168.9 2 61093713 61093737 61093737 61093758 45 21 24
1678121894.059684 185.228.168.10 2 61094013 61094035 61094035 61094059 46 24 22
1678121894.357632 9.9.9.9 3 61094313 61094333 61094333 61094357 44 24 20
Something went wrong while sending: -1
^C
root@OpenWrt-1:~/cake-autorate# tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9 9.9.9.10 9.
9.9.11 185.228.168.9 185.228.168.10
Starting tsping 0.2.2 - pinging 5 targets
1678122015.593081 9.9.9.9 0 61215543 61215564 61215564 61215593 50 29 21
1678122015.899172 9.9.9.10 0 61215843 61215871 61215871 61215899 56 28 28
1678122016.201457 9.9.9.11 0 61216144 61216175 61216175 61216201 57 26 31
1678122016.507425 185.228.168.9 0 61216444 61216481 61216481 61216507 63 26 37
1678122016.792759 185.228.168.10 0 61216744 61216768 61216768 61216792 48 24 24
1678122017.100950 9.9.9.9 1 61217045 61217076 61217076 61217100 55 24 31
1678122017.403393 9.9.9.10 1 61217345 61217374 61217374 61217403 58 29 29
1678122017.739296 9.9.9.11 1 61217645 61217711 61217711 61217739 94 28 66
1678122017.989428 185.228.168.9 1 61217945 61217967 61217967 61217989 44 22 22
1678122018.295854 185.228.168.10 1 61218245 61218268 61218268 61218295 50 27 23
1678122018.592690 9.9.9.9 2 61218546 61218566 61218566 61218592 46 26 20
1678122018.919731 9.9.9.10 2 61218846 61218895 61218895 61218919 73 24 49
1678122019.205083 9.9.9.11 2 61219146 61219171 61219171 61219205 59 34 25

Yes I will get those fixes up asap

Getting the columns right, would be simplest to fix by changing them to have the same ordering, so let me know if there's any preferences on that

That is really weird, and seems to only affect 9.9.9.10. I've neglected to use errno if something goes wrong when sending, so will add that as well so we can better see what it's complaining about

Regarding time interval, fping uses:

−p, −−period= MSEC

In looping or counting modes (−l, −c, or −C), this parameter sets the time in milliseconds that fping waits between successive packets to an individual target. Default is 1000 and minimum is 10.

−i, −−interval= MSEC

The minimum amount of time (in milliseconds) between sending a ping packet to any target (default is 10, minimum is 1).

So with:

fping ${ping_extra_args} --timestamp --loop --period "${reflector_ping_interval_ms}" --interval "${ping_response_interval_ms}" --timeout 10000 "${reflectors[@]:0:${no_pingers}}"

As I understand it, this means that ${reflector_ping_interval_ms} separates ping sends to a particular target, and ${ping_response_interval_ms} separates ping sends to any target.

I think this system in fping probably is optimal. @patrakov do you agree?

@moeller0 finally - OWDs in cake-autorate using @Lochnair's tsping binary.

@patrakov would you like to test?

Samples per reflector:
ReflectorID: 185.228.168.10; N: 91
ReflectorID: 185.228.168.9; N: 91
ReflectorID: 9.9.9.11; N: 90
ReflectorID: 9.9.9.9; N: 88
DL: maximum 95.000%-ile delta delay over all 4 reflectors: 18.670 ms.
DL: maximum 99.000%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.500%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.900%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.950%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.990%-ile delta delay over all 4 reflectors: 38.850 ms.
DL: maximum 99.999%-ile delta delay over all 4 reflectors: 38.850 ms.
UL: maximum 95.000%-ile delta delay over all 4 reflectors: 4.990 ms.
UL: maximum 99.000%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.500%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.900%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.950%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.990%-ile delta delay over all 4 reflectors: 11.985 ms.
UL: maximum 99.999%-ile delta delay over all 4 reflectors: 11.985 ms.
INFO: Writing plot as: ./output.timecourse.pdf
INFO: fn_parse_autorate_log took: 16.0685 seconds.
1 Like

Not necessarily today, as the LTE link is good at night (20+ Mbps) and doesn't really need cake-autorate. And I won't have anything demanding (like Discord) until Wednesday.

Regarding mimicking fping options - yes I agree that it makes sense. That's assuming a sane interpretation of the timeout, in particular the case when it is greater than the period.

1 Like

Why? It is not that fping is a widely used binary with lots of documentation...

also does fping allow period smaller than interval?

Yes it does:

hms-beagle2:~ smoeller$ fping --timestamp --loop --period 100 --interval 200 1.1.1.1 9.9.9.9
[1678125449.51688] 1.1.1.1 : [0], 64 bytes, 33.5 ms (33.5 avg, 0% loss)
[1678125449.71822] 9.9.9.9 : [0], 64 bytes, 32.9 ms (32.9 avg, 0% loss)
[1678125449.92214] 1.1.1.1 : [1], 64 bytes, 33.0 ms (33.2 avg, 0% loss)
[1678125450.12517] 9.9.9.9 : [1], 64 bytes, 33.3 ms (33.1 avg, 0% loss)
[1678125450.33064] 1.1.1.1 : [2], 64 bytes, 33.6 ms (33.4 avg, 0% loss)
[1678125450.53283] 9.9.9.9 : [2], 64 bytes, 33.6 ms (33.3 avg, 0% loss)
[1678125450.73605] 1.1.1.1 : [3], 64 bytes, 33.7 ms (33.4 avg, 0% loss)
[1678125450.93910] 9.9.9.9 : [3], 64 bytes, 33.4 ms (33.3 avg, 0% loss)
^C

however the results are "interesting" at best... tspings variant is IMHO better in that you can only specify things the code will deliver...

Normally I am all for re-using existing command line switches and behaviour, but here I am not convinced that fping is the example to follow.

1 Like

I like how apparently both autorate itself as well as the plotting code defaulted to do the right thing once fed with OWDs.

This will need testing and likely there will be one or two issues to iron out:

But it at least allowed me to produce the plot above using @Lochnair's tsping binary, which is available here:

and which includes support for ICMP type 13 (timestamp) requests and responses, and hence working with one way delays (OWDs) rather than round trip times (RTTs).

tsping is not yet an official OpenWrt package, but for anyone wanting to test there are simple instructions for building on OpenWrt here:

For any readers not sure about the significance of this, this facilitates determining the direction of bufferbloat (download or upload) and hence better control over the download and upload shaper rates since they can be controlled more independently in dependence upon the respective download and upload OWDs. Hitherto cake-autorate has fudged the issue by working with RTTs and erred on the side of caution.

Here is an example set of config overrides that work for my connection:

pinger_binary=tsping

reflectors=(
"9.9.9.9" "9.9.9.10" "9.9.9.11" # Quad9
"185.228.168.9" "185.228.168.10" # CleanBrowsing
)

no_pingers=5

ping_prefix_string="stdbuf -oL"

But does this make sense to you:

# tsping --help
Usage: tsping [OPTION...] IP1 IP2 IP3 ...
tsping -- a simple application to send ICMP echo/timestamp requests

  -e, --icmp-echo            Use ICMP echo requests
  -t, --icmp-ts              Use ICMP timestamp requests (default)
  -r, --target-spacing=TIME  Time to wait between pinging each target in ms
                             (default 0)
  -s, --sleep-time=TIME      Time to wait between each round of pinging in ms
                             (default 100)
  -D, --print-timestamps     Print UNIX timestamps for responses
  -m, --machine-readable[=DELIMITER]
                             Output results in a machine readable format
  -f, --fw-mark=MARK         Firewall mark to set on outgoing packets
  -i, --interface=INTERFACE  Interface to bind to
  -?, --help                 Give this help list
      --usage                Give a short usage message

Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.

With the above, my understanding is that sleep-time 1000ms and target-spacing 500ms means: every 1000ms start round robin and send out first ICMP to first target, and then wait 500ms to send out interval to next target in that round robin.

So effective interval between responses of 500ms.

But I see:

root@OpenWrt-1:~# tsping --print-timestamps --machine-readable=' ' --sleep-time 1000 --target-spacing 500 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678128598.015933 9.9.9.10 0 67797969 67797993 67797993 67798015 46 22 24
1678128599.535897 9.9.9.9 1 67799470 67799504 67799504 67799535 65 31 34
1678128600.016869 9.9.9.10 1 67799970 67799994 67799994 67800016 46 22 24
1678128601.515681 9.9.9.9 2 67801471 67801494 67801494 67801515 44 21 23
1678128602.017769 9.9.9.10 2 67801971 67801993 67801993 67802017 46 24 22
1678128603.516854 9.9.9.9 3 67803471 67803493 67803493 67803516 45 23 22
1678128604.015840 9.9.9.10 3 67803972 67803993 67803993 67804015 43 22 21

That is, 1.5s and 500ms spacing between responses.

This is much clearer to me:

−p, −−period= MSEC

In looping or counting modes (−l, −c, or −C), this parameter sets the time in milliseconds that fping waits between successive packets to an individual target. Default is 1000 and minimum is 10.

−i, −−interval= MSEC

The minimum amount of time (in milliseconds) between sending a ping packet to any target (default is 10, minimum is 1).

Nope, with e.g. tow IPs, you will do:
probe1, 500ms, probe2, 1000ms
so your total period is 1500ms...

And that is what you see, 500ms between the members of the set (9.9.9.9's first response is missing)

  1. 9.9.9.9: 0
  2. 9.9.9.10: 500ms later
    next round
  3. 9.9.9.9: 1000ms after 2), so exactly the requested sleep time.

Except you can specify stuff that fping does not complain about and does not deliver, period < interval...

My take is that you got used to fpings way of specifying this and hence confuse sleep with period, even though these are clearly different concepts. What like about @Lochnair's version is that you can not request impossible timings... it does become a tiny bit more involved to calculate the effective per reflector rate.

1 Like

Ah, I think I get it now. So they are both spacings between sends. Target is the spacing between targets, and sleep is the spacing between rounds (i.e. the spacing between the last send and the next first send)?

Ah, not quite:

root@OpenWrt-1:~# tsping --print-timestamps --machine-readable=' ' --sleep-time 1000 --target-spacing 1000 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678131109.591224 9.9.9.9 0 70309547 70309563 70309563 70309591 44 28 16
1678131110.598166 9.9.9.10 0 70310547 70310574 70310574 70310598 51 24 27
1678131112.597135 9.9.9.9 1 70312547 70312573 70312573 70312597 50 24 26
1678131113.598123 9.9.9.10 1 70313548 70313573 70313573 70313598 50 25 25
1678131115.609165 9.9.9.9 2 70315548 70315584 70315584 70315609 61 25 36
1678131116.598139 9.9.9.10 2 70316548 70316573 70316573 70316598 50 25 25

So there is target + sleep between rounds? But it should just be sleep right? Or?

Ah, now I understand your question @Lochnair:

Yes, I think it should be skipped. @moeller0 do you agree?