CAKE w/ Adaptive Bandwidth

Well you could throttle the actual speedtest and only run it if you want to use the link...

But that would not help.

In the past IIRC Telekom Mobile in the US used a "trick" in which the unlocked links during a speedtest... but that obviously was not above the board and they stopped doing this once caught... (Their argument was that they wanted to show customers the capacity they could have, except they did not bother to inform customers about this fact.)

Any idea what 'finished' relates to:

root@OpenWrt-1:~# tsping 9.9.9.9
Starting tsping 0.2.2 - pinging 1 targets
9.9.9.9         : [0] Down: 50373319, Up: 50373344, RTT: 50373344, Originate: 50373378, Received: 59, Transmit: 34, Finished: 25
9.9.9.9         : [1] Down: 50373419, Up: 50373434, RTT: 50373434, Originate: 50373468, Received: 49, Transmit: 34, Finished: 15
9.9.9.9         : [2] Down: 50373520, Up: 50373534, RTT: 50373534, Originate: 50373568, Received: 48, Transmit: 34, Finished: 14
9.9.9.9         : [3] Down: 50373620, Up: 50373633, RTT: 50373633, Originate: 50373666, Received: 46, Transmit: 33, Finished: 13
9.9.9.9         : [4] Down: 50373720, Up: 50373723, RTT: 50373723, Originate: 50373756, Received: 36, Transmit: 33, Finished: 3
9.9.9.9         : [5] Down: 50373821, Up: 50373833, RTT: 50373833, Originate: 50373866, Received: 45, Transmit: 33, Finished: 12
9.9.9.9         : [6] Down: 50373921, Up: 50373933, RTT: 50373933, Originate: 50373966, Received: 45, Transmit: 33, Finished: 12
9.9.9.9         : [7] Down: 50374021, Up: 50374033, RTT: 50374033, Originate: 50374067, Received: 46, Transmit: 34, Finished: 12
9.9.9.9         : [8] Down: 50374121, Up: 50374132, RTT: 50374132, Originate: 50374166, Received: 45, Transmit: 34, Finished: 11
9.9.9.9         : [9] Down: 50374222, Up: 50374233, RTT: 50374233, Originate: 50374266, Received: 44, Transmit: 33, Finished: 11
^C
root@OpenWrt-1:~# tsping -m 9.9.9.9
Starting tsping 0.2.2 - pinging 1 targets
9.9.9.9,0,50395549,50395563,50395563,50395596,47,33,14
9.9.9.9,1,50395650,50395663,50395663,50395696,46,33,13
9.9.9.9,2,50395750,50395763,50395763,50395796,46,33,13
9.9.9.9,3,50395850,50395863,50395863,50395896,46,33,13
9.9.9.9,4,50395951,50395963,50395963,50395996,45,33,12
9.9.9.9,5,50396051,50396063,50396063,50396096,45,33,12
9.9.9.9,6,50396151,50396163,50396163,50396196,45,33,12
9.9.9.9,7,50396252,50396271,50396271,50396303,51,32,19
9.9.9.9,8,50396352,50396363,50396363,50396396,44,33,11
9.9.9.9,9,50396452,50396463,50396463,50396496,44,33,11

@Lochnair?

All I will say is: :person_facepalming:

Didn't catch that the ordering of the columns is different when I changed how the printing works...

	char FMT_ICMP_TIMESTAMP_HUMAN[] = "%-15s : [%u] Down: %d, Up: %d, RTT: %d, Originate: %u, Received: %u, Transmit: %u, Finished: %u\n";

and

				printf(FMT_OUTPUT, ip, result.sequence, result.originateTime, result.receiveTime, result.transmitTime, result.finishedTime, rtt, down_time, up_time);

1 Like

Yes, somehow I didn't catch that when testing my changes :see_no_evil:
So need to fix that up

Any chance you could also add option to print out in us?

ICMP timestamps only have millisecond resolution... so that is the natural way of reporting them. However bash-autorate uses µs due to bash's EPOCHREALTIME reporting in µs.... so µs would be convenient...

I have updated the post with more data:

EDIT: useless, I did not record OWDs properly. Please wait for the next Discord meeting, which is on Wednesday.

Something doesn't feel right with the time intervals.

root@OpenWrt-1:~/cake-autorate# tsping  --print-timestamps --machine-readable=' ' --sleep-time 200 --target-spacing 100 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678118756.728162 9.9.9.9 0 57956682 57956703 57956703 57956728 46 25 21
1678118756.829150 9.9.9.10 0 57956782 57956803 57956803 57956829 47 26 21
1678118757.129260 9.9.9.9 1 57957082 57957103 57957103 57957129 47 26 21
1678118757.229163 9.9.9.10 1 57957183 57957204 57957204 57957229 46 25 21
1678118757.528159 9.9.9.9 2 57957483 57957503 57957503 57957528 45 25 20
1678118757.630785 9.9.9.10 2 57957583 57957603 57957603 57957630 47 27 20
1678118757.928160 9.9.9.9 3 57957883 57957903 57957903 57957928 45 25 20
1678118758.028105 9.9.9.10 3 57957984 57958003 57958003 57958028 44 25 19
1678118758.328203 9.9.9.9 4 57958284 57958303 57958303 57958328 44 25 19
1678118758.428184 9.9.9.10 4 57958384 57958403 57958403 57958428 44 25 19
1678118758.728151 9.9.9.9 5 57958685 57958703 57958703 57958728 43 25 18
1678118759.130258 9.9.9.9 6 57959085 57959105 57959105 57959130 45 25 20
1678118759.229255 9.9.9.10 6 57959186 57959203 57959203 57959229 43 26 17
1678118759.528163 9.9.9.9 7 57959486 57959502 57959502 57959528 42 26 16
1678118759.630210 9.9.9.10 7 57959586 57959603 57959603 57959630 44 27 17
1678118759.928276 9.9.9.9 8 57959886 57959903 57959903 57959928 42 25 17
1678118760.028298 9.9.9.10 8 57959987 57960003 57960003 57960028 41 25 16

I expected this to give a round robin spacing of 1s, with 500ms spacing between reflectors, to give one response every 500ms, but we don't see that:

root@OpenWrt-1:~/cake-autorate# tsping  --print-timestamps --machine-readable=' ' --sleep-time 1000 --target-spacing 500 9.9.9.9 9.9.9.10
Starting tsping 0.2.2 - pinging 2 targets
1678119096.278095 9.9.9.9 0 58296228 58296253 58296253 58296278 50 25 25
1678119096.768772 9.9.9.10 0 58296728 58296743 58296743 58296768 40 25 15
1678119098.277984 9.9.9.9 1 58298229 58298253 58298253 58298277 48 24 24
1678119098.777821 9.9.9.10 1 58298729 58298753 58298753 58298777 48 24 24
1678119100.278943 9.9.9.9 2 58300229 58300253 58300253 58300278 49 25 24
1678119100.769150 9.9.9.10 2 58300730 58300743 58300743 58300769 39 26 13
1678119102.277942 9.9.9.9 3 58302230 58302253 58302253 58302277 47 24 23
1678119102.778920 9.9.9.10 3 58302730 58302753 58302753 58302778 48 25 23

@Lochnair is the sleep between rounds from the last response rather than from the first send?

I'm trying to figure out how these correspond with fpings.

Sleep between rounds happens after the last send.

I'll just quote the whole code for the sending loop here:

while (1)
	{
		if (sentICMP > 0 && sentICMP % 100 == 0 && receivedICMP == 0) {
			fprintf(stderr, "Warning: We've sent %lu packets, but received none.\n", sentICMP);
		}

		for (int i = 0; i < targets_len; i++)
		{
			switch (args->icmp_type) {
				case 8:
					send_icmp_echo_request(sock_fd, &targets[i], thread_data->id, htons(seq));
					break;
				case 13:
					send_icmp_timestamp_request(sock_fd, &targets[i], thread_data->id, htons(seq));
					break;
				default:
					fprintf(stderr, "sender: Invalid ICMP type: %u, exiting..\n", args->icmp_type);
					pthread_exit(NULL);
			}

			if (args->target_spacing > 0 && args->targets_len > 1)
				nanosleep(&spacing_time, NULL);
		}

		seq++;
		if (args->sleep_time > 0)
			nanosleep(&sleep_time, NULL);
	}

I didn't consider this when I first wrote it, but this nanosleep(&spacing_time, NULL); should probably be skipped after the last target has been pinged, or?

Not sure. But why does this come through in batches:

tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9 9.9.9.10 9.
9.9.11 185.228.168.9 185.228.168.10 | while read -r line; do echo ${line}; done

Whereas this is smooth:

root@OpenWrt-1:~/cake-autorate#  fping 9.9.9.9 --loop | while read line; do echo ${line}; done
9.9.9.9 : [0], 64 bytes, 49.5 ms (49.5 avg, 0% loss)
9.9.9.9 : [1], 64 bytes, 49.8 ms (49.6 avg, 0% loss)
9.9.9.9 : [2], 64 bytes, 48.9 ms (49.4 avg, 0% loss)

Ah, stackoverflow has the answer for that one I believe:

Also, just wanted to mention that apparently in UNIX a newline will typically only flush the buffer if stdout is a terminal. If the output is being redirected to a file, a newline won't flush.

Didn't catch this since it works without a pipe, I guess buffering for stdout needs to be disabled entirely

Can that be done from the tsping binary?

Yes, the SO answer says you can disable buffering for stdout as simply as setbuf(stdout, NULL);, so it's an easy fix

1 Like

Sweet. Any chance you could push those fixes? I've finished the tsping wrapper for cake-autorate and dying to test this out!

Please use stdbuf -oL as a wrapper as a temporary workaround. To solve the problem properly, please add a call to setvbuf() to the C code.

1 Like

Why does stdbuf -oL change the operation of tsping in this way? See:

rroot@OpenWrt-1:~/cake-autorate# stdbuf -oL tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9
9.9.9.10 9.9.9.11 185.228.168.9 185.228.168.10
Starting tsping 0.2.2 - pinging 5 targets
1678121889.858740 9.9.9.9 0 61089809 61089834 61089834 61089858 49 24 25
Something went wrong while sending: -1
1678121890.457717 9.9.9.11 0 61090409 61090433 61090433 61090457 48 24 24
1678121890.763506 185.228.168.9 0 61090710 61090737 61090737 61090763 53 26 27
1678121891.059733 185.228.168.10 0 61091010 61091035 61091035 61091059 49 24 25
1678121891.358621 9.9.9.9 1 61091310 61091333 61091333 61091358 48 25 23
Something went wrong while sending: -1
1678121891.957635 9.9.9.11 1 61091911 61091933 61091933 61091957 46 24 22
1678121892.258752 185.228.168.9 1 61092212 61092236 61092236 61092258 46 22 24
1678121892.558668 185.228.168.10 1 61092512 61092534 61092534 61092558 46 24 22
1678121892.857691 9.9.9.9 2 61092812 61092833 61092833 61092857 45 24 21
Something went wrong while sending: -1
1678121893.457572 9.9.9.11 2 61093413 61093433 61093433 61093457 44 24 20
1678121893.758712 185.228.168.9 2 61093713 61093737 61093737 61093758 45 21 24
1678121894.059684 185.228.168.10 2 61094013 61094035 61094035 61094059 46 24 22
1678121894.357632 9.9.9.9 3 61094313 61094333 61094333 61094357 44 24 20
Something went wrong while sending: -1
^C
root@OpenWrt-1:~/cake-autorate# tsping  --print-timestamps --machine-readable=' ' --sleep-time 0 --target-spacing 300 9.9.9.9 9.9.9.10 9.
9.9.11 185.228.168.9 185.228.168.10
Starting tsping 0.2.2 - pinging 5 targets
1678122015.593081 9.9.9.9 0 61215543 61215564 61215564 61215593 50 29 21
1678122015.899172 9.9.9.10 0 61215843 61215871 61215871 61215899 56 28 28
1678122016.201457 9.9.9.11 0 61216144 61216175 61216175 61216201 57 26 31
1678122016.507425 185.228.168.9 0 61216444 61216481 61216481 61216507 63 26 37
1678122016.792759 185.228.168.10 0 61216744 61216768 61216768 61216792 48 24 24
1678122017.100950 9.9.9.9 1 61217045 61217076 61217076 61217100 55 24 31
1678122017.403393 9.9.9.10 1 61217345 61217374 61217374 61217403 58 29 29
1678122017.739296 9.9.9.11 1 61217645 61217711 61217711 61217739 94 28 66
1678122017.989428 185.228.168.9 1 61217945 61217967 61217967 61217989 44 22 22
1678122018.295854 185.228.168.10 1 61218245 61218268 61218268 61218295 50 27 23
1678122018.592690 9.9.9.9 2 61218546 61218566 61218566 61218592 46 26 20
1678122018.919731 9.9.9.10 2 61218846 61218895 61218895 61218919 73 24 49
1678122019.205083 9.9.9.11 2 61219146 61219171 61219171 61219205 59 34 25

Yes I will get those fixes up asap

Getting the columns right, would be simplest to fix by changing them to have the same ordering, so let me know if there's any preferences on that

That is really weird, and seems to only affect 9.9.9.10. I've neglected to use errno if something goes wrong when sending, so will add that as well so we can better see what it's complaining about

Regarding time interval, fping uses:

−p, −−period= MSEC

In looping or counting modes (−l, −c, or −C), this parameter sets the time in milliseconds that fping waits between successive packets to an individual target. Default is 1000 and minimum is 10.

−i, −−interval= MSEC

The minimum amount of time (in milliseconds) between sending a ping packet to any target (default is 10, minimum is 1).

So with:

fping ${ping_extra_args} --timestamp --loop --period "${reflector_ping_interval_ms}" --interval "${ping_response_interval_ms}" --timeout 10000 "${reflectors[@]:0:${no_pingers}}"

As I understand it, this means that ${reflector_ping_interval_ms} separates ping sends to a particular target, and ${ping_response_interval_ms} separates ping sends to any target.

I think this system in fping probably is optimal. @patrakov do you agree?