CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

And if I understand correctly we can deal with this by creating baseline based on the difference (can be positive or negative) and then looking at the increase between ticks. Only problem comes in with the clock synchronization events, which would somehow need to be managed? Is this an example where the two pole butterworth would deal better with step change.

And do we use this:

uplink time = Receive - Originate
downlink time = rtt - uplink time

And create separate baselines for each and look for increases associated with bufferbloat in uplink or downlink?

Oh, and these two baselines is per reflector. Or something funky like offset kept for each reflector and somehow amalgamated into overall baseline for each direction. But former approach maybe easier?

Yes.

My first assumption is that these should be rare enough to ignore, which works better if we can rely on a hard minimum rate setting helping us over periods of artificial bufferbloat caused by clocks changing out of phase and then the EWMA adjusting to the new value over time (it should converge fast on the minimum which should hep with bufferbloat)... alternatively we could look for ridiculously large jumps and immediately reset the EWMA in such cases.... but let's only go there if we have to, no? Side-note: the current code has a similar problem in the case when the nearest reflector goes away for whatever reason there will be a jump in the current_RTT values that will take a while to be reflected in baselineRTT, but that is a conscious and acceptable trade-off IMHO (says the guy who can not even tests the script properly :wink: )

Not sure.

technically it seems cleaner to do (Originate + rtt) - Transmit here allowing for the upstream system taking a break between receive and transit, but that will in all likelihood have zero practical relevance... given that the 1ms time stamp resolution is a looong time for a recent CPUs.

+1; that is the idea, but now we need to keep track of two EWMAs per reflector instead of a single EWMA for the whole system.

1 Like

Sweet, then if we can get hping3 made an Official OpenWrt package - who can help with this - any thoughts @richb-hanover-priv(?) then I at least know now how to implement this in script. Unless someone beats me to it.

This offers the potential to get true directional rate control, which is rather exciting.

image

grafik

I'm trying, i have no experience with compiling software, but there are several tutorials which i read. Maybe I am succesful and can provide a hperf3.ipk, but it can only be for Turris Omnia then i suppose

sadly there is no libpcap-dev for openwrt, with it (and much space on the router), somebody could build it "relatively" easy direct on the router and then share

If anyone from the experienced coders out here reads this: PLEASE HALP US

1 Like

Building on the router is generally only for very beefy routers, like SBCs (RPi, Pine64 etc).

I wonder whether https://subversion.xray.aps.anl.gov/trac/dbw/browser/trunk/Xalert/stevens/tcpipi/icmptime/icmptime.c might not be helpful, instead of going for the full hping3 we might just get a small binary that does get us ICMP timestamps, not sure about the license/usability of that code for our purpose though. Piggy-backing on an existing maintained project is probably better

icmptime.c:20:10: fatal error: netinet/ip_var.h: No such file or directory
   20 | #include <netinet/ip_var.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.

This is on WSL2. Not sure how it would fare for OpenWrt.

Yeah the code is ancient and does not actually assume Linux to be a thing (effortlessly explained by the code's age), I mostly linked that because the whole thing seems rather short and some of us wanted parts of this project to move to C-code :wink: .

1 Like

I'm working on this now, but slow going. Will update if/when I make some headway...

2 Likes

Ace - thanks for trying. Don't forget to post issues on this thread because help is pretty fast. It's like placing a match next to a tinderbox. So much eagerness from many different parties. It's quite amazing really.

1 Like

For fun, here is @tievolu's complete list of type13 reflectors from a well connected place:

  Pings done = 114    RTT/2 min =   3  |                     Request  |                       Reply  |                  Round Trip  |  
-------------------------------------------------------------------------------------------------------------------------------------
| Ping Target      |           Offset  |  Current | Average ? jitter  |  Current | Average ? jitter  |  Current | Average ? jitter  |  
-------------------------------------------------------------------------------------------------------------------------------------
| 27.112.79.80     |                0  |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |  
| 79.110.170.43    |                0  |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |    ERROR |     N/A ? N/A     |  
| 146.112.61.106   |              -60  |        3 |    3.95 ? 0.75    |        4 |    3.67 ? 0.55    |        7 |    7.62 ? 0.70    |  
| 149.112.112.112  |              -40  |        3 |    3.71 ? 0.80    |        4 |    4.05 ? 0.95    |        7 |    7.76 ? 0.35    |  
| 208.67.220.220   |              -59  |        4 |    3.76 ? 0.70    |        3 |    4.00 ? 0.70    |        7 |    7.76 ? 0.40    |  
| 208.67.222.222   |              -63  |        4 |    3.62 ? 0.95    |        3 |    4.14 ? 1.00    |        7 |    7.76 ? 0.75    |  
| 9.9.9.10         |              -60  |        3 |    3.76 ? 0.95    |        5 |    4.05 ? 1.05    |        8 |    7.81 ? 0.20    |  
| 149.112.112.11   |              -60  |        4 |    3.67 ? 1.25    |        4 |    4.14 ? 1.20    |        8 |    7.81 ? 0.25    |  
| 146.112.41.3     |              -62  |        5 |    3.71 ? 0.95    |        3 |    4.10 ? 1.15    |        8 |    7.81 ? 0.40    |  
| 9.9.9.9          |              -62  |        3 |    3.67 ? 1.20    |        5 |    4.14 ? 1.30    |        8 |    7.81 ? 0.50    |  
| 94.140.15.15     |              -63  |        5 |    3.81 ? 0.90    |        3 |    4.05 ? 0.90    |        8 |    7.86 ? 0.20    |  
| 94.140.15.16     |              -62  |        3 |    3.62 ? 0.95    |        5 |    4.29 ? 0.95    |        8 |    7.90 ? 0.20    |  
| 149.112.112.10   |              -59  |        3 |    3.52 ? 0.90    |        5 |    4.38 ? 1.10    |        8 |    7.90 ? 0.50    |  
| 94.140.14.14     |              -63  |        3 |    3.86 ? 1.05    |        5 |    4.14 ? 1.05    |        8 |    8.00 ? 0.00    |  
| 94.140.14.15     |              -63  |        3 |    3.67 ? 1.10    |        5 |    4.43 ? 1.10    |        8 |    8.10 ? 0.20    |  
| 146.112.41.2     |              -59  |        3 |    3.57 ? 0.80    |        5 |    4.95 ? 2.50    |        8 |    8.52 ? 2.10    |  
| 9.9.9.11         |              -61  |        3 |    5.05 ? 2.95    |        4 |    4.00 ? 1.20    |        7 |    9.05 ? 2.75    |  
| 185.228.168.10   |              321  |        4 |    5.71 ? 4.95    |        4 |    4.48 ? 0.95    |        8 |   10.19 ? 4.20    |  
| 185.233.106.232  |            26088  |        3 |    4.57 ? 1.40    |        7 |    6.48 ? 1.35    |       10 |   11.05 ? 0.25    |  
| 185.233.107.4    |            12251  |        4 |    4.19 ? 1.70    |        7 |    7.00 ? 1.80    |       11 |   11.19 ? 0.40    |  
| 185.228.168.168  |              321  |        3 |    6.81 ? 6.00    |        5 |    4.62 ? 1.75    |        8 |   11.43 ? 6.85    |  
| 185.235.81.1     |              -59  |        3 |    3.90 ? 1.50    |        8 |    8.05 ? 1.60    |       11 |   11.95 ? 0.40    |  
| 176.9.1.117      |              -61  |        7 |    5.10 ? 2.55    |        6 |    7.29 ? 2.50    |       13 |   12.38 ? 0.55    |  
| 74.82.42.42      |              -60  |        9 |    5.71 ? 3.80    |        6 |    6.71 ? 2.15    |       15 |   12.43 ? 2.25    |  
| 176.9.93.198     |              -59  |        6 |    4.71 ? 2.60    |        6 |    8.43 ? 3.80    |       12 |   13.14 ? 2.40    |  
| 185.95.218.42    |              -62  |        5 |    5.38 ? 2.40    |        9 |    8.76 ? 2.40    |       14 |   14.14 ? 0.20    |  
| 185.95.218.43    |              -63  |        3 |    4.57 ? 2.30    |       11 |    9.71 ? 1.90    |       14 |   14.29 ? 0.60    |  
| 193.17.47.1      |              -62  |        7 |    4.81 ? 2.60    |        8 |    9.62 ? 2.65    |       15 |   14.43 ? 0.45    |  
| 185.43.135.1     |              -63  |        4 |    4.62 ? 2.05    |       10 |   10.05 ? 2.50    |       14 |   14.67 ? 1.05    |  
| 93.104.213.190   |              -63  |        3 |    5.05 ? 2.20    |       11 |   10.24 ? 2.05    |       14 |   15.29 ? 2.05    |  
| 130.59.31.248    |              -63  |        8 |    5.52 ? 3.45    |        9 |   11.48 ? 3.45    |       17 |   17.00 ? 0.00    |  
| 130.59.31.251    |              -64  |        4 |    4.57 ? 1.65    |       13 |   12.52 ? 1.75    |       17 |   17.10 ? 0.20    |  
| 185.150.99.255   |              -65  |        3 |    6.38 ? 3.55    |       12 |   11.10 ? 4.55    |       15 |   17.48 ? 3.60    |  
| 213.196.191.96   |              -63  |    ERROR |    6.33 ? 4.90    |    ERROR |   12.38 ? 5.20    |    ERROR |   18.71 ? 7.30    |  
| 193.19.108.3     |          -228670  |        5 |    5.62 ? 3.40    |       15 |   13.81 ? 3.50    |       20 |   19.43 ? 0.40    |  
| 5.1.66.255       |              -64  |        4 |    4.86 ? 1.85    |       15 |   14.86 ? 1.95    |       19 |   19.71 ? 0.40    |  
| 194.242.2.3      |          -228670  |        3 |    5.38 ? 4.00    |       16 |   14.43 ? 2.70    |       19 |   19.81 ? 1.60    |  
| 146.255.56.98    |              223  |        3 |    6.57 ? 3.90    |       17 |   13.57 ? 4.10    |       20 |   20.14 ? 0.20    |  
| 212.78.94.40     |              -66  |        4 |    4.52 ? 2.25    |       16 |   15.76 ? 2.35    |       20 |   20.29 ? 0.50    |  
| 193.19.108.2     |          -228671  |        4 |    4.24 ? 1.65    |       18 |   18.05 ? 2.05    |       22 |   22.29 ? 0.60    |  
| 185.216.27.142   |              -63  |       10 |    7.48 ? 5.95    |        7 |   14.81 ? 5.75    |       17 |   22.29 ? 8.00    |  
| 194.242.2.2      |          -228670  |        5 |    4.95 ? 2.20    |       18 |   17.76 ? 2.25    |       23 |   22.71 ? 0.45    |  
| 185.64.79.5      |              -66  |        3 |    5.38 ? 3.70    |       24 |   17.81 ? 4.00    |       27 |   23.19 ? 0.40    |  
| 37.252.230.153   |              -67  |        3 |    5.67 ? 4.45    |       21 |   18.05 ? 4.25    |       24 |   23.71 ? 0.40    |  
| 209.250.226.191  |              -70  |        6 |    4.43 ? 2.40    |       18 |   19.76 ? 2.30    |       24 |   24.19 ? 0.30    |  
| 46.227.200.55    |              -67  |        5 |    6.05 ? 3.65    |       20 |   19.10 ? 3.65    |       25 |   25.14 ? 0.20    |  
| 46.227.200.54    |              -67  |        5 |    4.95 ? 2.70    |       20 |   20.52 ? 1.70    |       25 |   25.48 ? 1.00    |  
| 37.252.232.95    |              -67  |        9 |    5.10 ? 2.95    |       17 |   20.43 ? 2.80    |       26 |   25.52 ? 0.65    |  
| 95.216.229.153   |              -75  |        4 |    5.10 ? 1.75    |       28 |   26.24 ? 2.25    |       32 |   31.33 ? 1.10    |  
| 185.134.196.55   |              -63  |        3 |    4.62 ? 2.40    |       12 |   30.52 ? 12.50   |       15 |   35.14 ? 10.60   |  
| 185.134.196.54   |              -63  |        8 |    4.86 ? 2.35    |        7 |   35.05 ? 12.65   |       15 |   39.90 ? 10.50   |  
| 46.239.223.80    |              -74  |        4 |    5.00 ? 1.90    |       51 |   49.62 ? 2.20    |       55 |   54.62 ? 0.40    |  
| 35.245.234.132   |              -95  |        9 |   11.57 ? 1.85    |       82 |   79.62 ? 1.95    |       91 |   91.19 ? 0.30    |  
| 35.230.160.38    |             -119  |       26 |   16.95 ? 2.15    |       66 |   74.81 ? 2.05    |       92 |   91.76 ? 0.40    |  
| 149.112.122.10   |              -99  |        7 |    8.67 ? 2.85    |       87 |   85.19 ? 2.75    |       94 |   93.86 ? 0.30    |  
| 149.112.121.20   |             -101  |        5 |    6.81 ? 2.50    |       89 |   87.14 ? 2.50    |       94 |   93.95 ? 0.10    |  
| 149.112.122.20   |              -99  |        5 |    8.38 ? 2.20    |       89 |   85.62 ? 2.40    |       94 |   94.00 ? 0.20    |  
| 149.112.121.10   |              -98  |        6 |    9.33 ? 2.25    |       88 |   84.71 ? 2.15    |       94 |   94.05 ? 0.30    |  
| 149.112.122.30   |             -101  |        4 |    6.14 ? 4.70    |       90 |   88.10 ? 4.80    |       94 |   94.24 ? 0.50    |  
| 149.112.121.30   |             -101  |        3 |    4.19 ? 1.70    |       92 |   90.10 ? 1.95    |       95 |   94.29 ? 0.45    |  
| 35.237.220.84    |              -98  |       21 |   21.76 ? 2.80    |       81 |   80.14 ? 2.90    |      102 |  101.90 ? 0.50    |  
| 35.231.247.227   |              -95  |        9 |   13.52 ? 3.05    |       92 |   88.43 ? 3.15    |      101 |  101.95 ? 0.90    |  
| 174.138.29.175   |              -61  |       85 |   87.10 ? 2.75    |       88 |   85.33 ? 2.90    |      173 |  172.43 ? 0.35    |  
| 139.162.112.47   |              -14  |    ERROR |  168.90 ? 3.00    |    ERROR |   84.05 ? 3.05    |    ERROR |  252.95 ? 0.45    |  
| 172.104.93.80    |              -15  |      178 |  179.19 ? 1.65    |       88 |   86.38 ? 1.45    |      266 |  265.57 ? 0.60    |  
| 101.102.103.104  |               -1  |      195 |  200.62 ? 5.15    |       88 |   82.38 ? 5.15    |      283 |  283.00 ? 0.00    |  
| 101.101.101.101  |               -8  |      200 |  203.05 ? 2.65    |       87 |   84.86 ? 3.00    |      287 |  287.90 ? 0.65    |  
-------------------------------------------------------------------------------------------------------------------------------------

note how small the offsets are to most of these....

Do the offsets seem to be staying the same roughly between runs?

I wonder if systemd is responsible for this. By default it comes with an ntp client whereas before that you had to actually install ntpd

Not working yet, but posting this here in case anyone wants to work in parallel toward getting this to compile: https://github.com/Fail-Safe/hping3/blob/main/Makefile

(I'm not a Makefile expert, so please don't take anything already in the file as correct or even necessary.)

1 Like

On first glance they look relatively stable, but the script prints out one of these tables (always ordered by Round Trip I believe, so the exact sequence is not fixed) per iteration making it tedious to track the offset per IP-address.

Since I was running this on my macos laptop outside of my home network, systemd would need to be active at the reflector sites, and I do have my reservations about systemd's prevalence on servers...

it's the standard for Debian, which is I think a very common server distribution.

In my experience with my own scripts, the offsets are generally very small (mostly <5ms) and stable once ntpd has settled on an accurate time for your system clock (i.e. about 24 hours after a reboot).

In terms of offset size there are some outliers with much larger offsets, as shown in @moeller0's output above (presumably these hosts aren't sending back timestamps based on milliseconds-since-midnight-UTC), but even these offsets are pretty stable.

Just comment out this line to remove the sorting by RTT:

my @results_to_print = sort {&get_sort_string($a) cmp &get_sort_string($b)} @results_to_print;

2 Likes

just for update on compiling i give up. (at least for today).

I tried first with the arm compiler, then with the toolkit (seems to be the makefile which does problems, but i don't understand it completely), in both ways I am differently stuck.

Hope you or another person has more luck or knowledge. or both.

The bunch of hosts with offset around -228670 all resolve to the same owner in Sweden via whois, well possible that they have a 3-4 minute offset of their main clock....

Then, thank you very much for sharing your code and the timestamp reflector list.

Thanks, I guess I need to export the data in a way that is easier to parse for post processing....

It works well in my monster perl script. For example, here's the script detecting bufferbloat in the upload direction:

Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec  1 01:06:50 2021 [2034-00056166]: |  Bandwidth usage  |      8.26.56.26 |  129.250.35.251 |  208.67.222.222 |
Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
Wed Dec  1 01:06:50 2021 [2034-00056166]: | TX Mbps | RX Mbps |  Rqst  |  Rply  |  Rqst  |  Rply  |  Rqst  |  Rply  |
Wed Dec  1 01:06:50 2021 [2034-00056166]: ---------------------------------------------------------------------------
...
Wed Dec  1 01:07:40 2021 [2034-00056189]: |  34.578 |   0.774 |   13   |   14   |   16   |   12   |   15   |   14   |
Wed Dec  1 01:07:42 2021 [2034-00056190]: |  32.304 |   0.695 |   15   |   13   |   11   |   13   |   11   |   14   |
Wed Dec  1 01:07:44 2021 [2034-00056191]: |  29.493 |   0.638 |   13   |   13   |   14   |   13   |   12   |   14   |
Wed Dec  1 01:07:46 2021 [2034-00056192]: |  34.485 |   1.725 |   11   |   14   |   14   |   13   |   10   |   14   |
Wed Dec  1 01:07:48 2021 [2034-00056193]: |  33.479 |   0.721 |   14   |   13   |   13   |   13   |   13   |   13   |
Wed Dec  1 01:07:50 2021 [2034-00056194]: |  26.218 |   0.545 |   50 X |   14   |   55 X |   13   |   52 X |   14   |
Wed Dec  1 01:07:52 2021 [2034-00056195]: |  26.952 |   0.566 |   35 X |   13   |   32 X |   14   |   36 X |   14   |
Wed Dec  1 01:07:54 2021 [2034-00056196]: |  29.268 |   0.687 |   68 X |   13   |   66 X |   14   |   69 X |   14   |

Timestamped pings are used to separate out request (upload) and reply (download) latency in exactly the same way as the example script I posted a few weeks back.