CAKE w/ Adaptive Bandwidth [October 2021 to September 2022]

moeller0 · December 8, 2021, 10:07am

Thanks a lot! Installation succeeded and at ~110 KB this is lighter than I feared:

root@turris:~$ ls -all /usr/sbin/hping3 
-rwxr-xr-x    1 root     root        102543 Dec  8 01:01 /usr/sbin/hping3

root@turris:~$ ls -all /usr/bin/ping
-rwsr-xr-x    1 root     root         33067 Nov  1 10:16 /usr/bin/ping

root@turris:~$ ls -all /usr/bin/ping6
-rwsr-xr-x    1 root     root         28895 Nov  1 10:16 /usr/bin/ping6

root@turris:~$ ls -all /usr/bin/nping 
-rwxr-xr-x    1 root     root        362555 Nov  1 10:16 /usr/bin/nping

larger than both pings (I believe for newer iputils there is only a single ping binary for both IP4 and IP6), but considerably smaller than nping and considerably faster at execution:

root@turris:~$ time hping3 9.9.9.9 --icmp --icmp-ts -c 1
HPING 9.9.9.9 (pppoe-wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=40 ip=9.9.9.9 ttl=60 id=3284 icmp_seq=0 rtt=19.7 ms
ICMP timestamp: Originate=36988061 Receive=36990206 Transmit=36990206
ICMP timestamp RTT tsrtt=19


--- 9.9.9.9 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.7/19.7/19.7 ms
real    0m 0.24s
user    0m 0.00s
sys     0m 0.00s
root@turris:~$ time nping --icmp-type 13 9.9.9.9 -c 1 --icmp-orig-time now

Starting Nping 0.7.70 ( https://nmap.org/nping ) at 2021-12-08 11:16 CET
SENT (0.1489s) ICMP [77.6.135.184 > 9.9.9.9 Timestamp request (type=13/code=0) id=59080 seq=1 orig=37001000 recv=0 trans=0] IP [ttl=64 id=55460 iplen=40 ]
RCVD (0.3070s) ICMP [9.9.9.9 > 77.6.135.184 Timestamp reply (type=14/code=0) id=59080 seq=1 orig=37001000 recv=37003318 trans=37003318] IP [ttl=60 id=9913 iplen=40 ]
 
Max rtt: 157.947ms | Min rtt: 157.947ms | Avg rtt: 157.947ms
Raw packets sent: 1 (40B) | Rcvd: 1 (40B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.23 seconds
real    0m 1.24s
user    0m 0.01s
sys     0m 0.00s

And as I just noticed nping truncates the originate timestamp to full seconds.... so many thanks to you , @Lochnair, and @_FailSafe for charting a path how to get hping3 into OpenWrt (again).

Lynx · December 8, 2021, 10:30am

These lines:

seem so easy to work with. Just noticed that's a big offset between originate and receive, no?

I cannot so easily test since I am using RT3200 + snapshot and it doesn't seem to work in the WSL2 environment in Windows:

lynx@DESKTOP:~$ hping3 9.9.9.9 --icmp --icmp-ts -c 1
[open_sockraw] socket(): Operation not permitted
[main] can't open raw socket
lynx@DESKTOP:~$ sudo hping3 9.9.9.9 --icmp --icmp-ts -c 1
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes

--- 9.9.9.9 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

@Lochnair are we in a position for pull request to get adopted as official OpenWrt package? I confess I am not familiar with this process so 100% reliant on your Jedi skills here.

moeller0 · December 8, 2021, 10:55am

Yes, this is because my router's clock is obviously not well synchronized (anymore), from my two internal Linux hosts I see less divergence:

user@happy-horse:~> cat /etc/os-release | grep PRETTY_NAME ; sudo time hping3 9.9.9.9 --icmp --icmp-ts -c 1
PRETTY_NAME="openSUSE Leap 15.2"
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=59 id=27328 icmp_seq=0 rtt=23.6 ms
ICMP timestamp: Originate=39150476 Receive=39150486 Transmit=39150486
ICMP timestamp RTT tsrtt=24


--- 9.9.9.9 hping statistic ---
1 packets tramitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 23.6/23.6/23.6 ms
0.00user 0.00system 0:00.09elapsed 6%CPU (0avgtext+0avgdata 4136maxresident)k
0inputs+0outputs (0major+224minor)pagefaults 0swaps

and

user@work-horse:~/SCRATCH$ cat /etc/os-release | grep PRETTY_NAME ; sudo time hping3 9.9.9.9 --icmp --icmp-ts -c 1
PRETTY_NAME="Ubuntu 20.04.3 LTS"
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=59 id=39988 icmp_seq=0 rtt=19.9 ms
ICMP timestamp: Originate=39179219 Receive=39179228 Transmit=39179228
ICMP timestamp RTT tsrtt=20


--- 9.9.9.9 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.9/19.9/19.9 ms
0.00user 0.00system 0:00.10elapsed 1%CPU (0avgtext+0avgdata 3204maxresident)k
0inputs+0outputs (0major+165minor)pagefaults 0swaps

I guess time to make sure that my network time is properly synced again.... But that is also the reason why we should not work with the raw differences...

Lynx · December 8, 2021, 10:59am

Ah yes. What is default OpenWrt ntp behaviour I wonder? I think my router loses time every reboot, so presumably the time shifting will give effects as discussed above. Would it help to force an ntpupdate on script initialization?

moeller0 · December 8, 2021, 11:05am

No idea. My issue is that I added a GPS-disciplined NTP server to my internal network but only did a half-assed job at making sure all devices get their high-quality time from there. Now, I was under the impression that I had fixed the router, but apparently I did not (the router has a battery backed real time clock, so does not suffer from your specific concern).

Not sure, I would opt for just letting the normal auto-baseline adjustment code deal with that....

_FailSafe · December 8, 2021, 12:16pm

You build your image or use a pre-built? If you build your own, it’s very straightforward to add the hping3 package @Lochnair has in his repo from our tweaking and testing yesterday.

I would be willing to also cross-compile the binary for you if that would help your immediate testing.

Lynx · December 8, 2021, 12:23pm

Presently just pre-built using attended sysupgrade. But can switch over.

Do you know if I can build using SDK from git feed? So far it looks like this utility is a clear winner. Next week I am off work for Christmas and with the benefit of way more free time then I plan to modify the script to work with the timestamps from this utility, and once the script logic is settled spend time optimising the script as much as possible for performance and robustness.

Assuming everyone is positive about this utility it seems an important goal is to try to get it made an official OpenWrt package.

_FailSafe · December 8, 2021, 12:34pm

I'm not sure I follow, but if you already have an OpenWrt build environment you can just create a new sub-directory for the hping3 package. I created mine as package/network/utils/hping3 and put @Lochnair's Makefile in that directory.

Next, run a make menuconfig and add the hping3 utility from the Network category. Save and build!

My offer stands to build the hping3 for your platform to get it in your hands quicker, @Lynx, if you'd like. Just reply with your ARCH from cat /etc/openwrt_release if you'd like me to do so!

Lynx · December 8, 2021, 1:01pm

I was concerned about the integrity of my manliness so I built using SDK. Terrific work @Lochnair!

So far so good:

root@OpenWrt:~# hping3 9.9.9.9 --icmp --icmp-ts -i u1000 -c 10
HPING 9.9.9.9 (wan 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=54 id=31927 icmp_seq=0 rtt=79.3 ms
ICMP timestamp: Originate=47167204 Receive=47167241 Transmit=47167241
ICMP timestamp RTT tsrtt=79

len=46 ip=9.9.9.9 ttl=54 id=31926 icmp_seq=2 rtt=77.5 ms
ICMP timestamp: Originate=47167206 Receive=47167241 Transmit=47167241
ICMP timestamp RTT tsrtt=77

len=46 ip=9.9.9.9 ttl=54 id=31928 icmp_seq=3 rtt=76.7 ms
ICMP timestamp: Originate=47167207 Receive=47167241 Transmit=47167241
ICMP timestamp RTT tsrtt=77

len=46 ip=9.9.9.9 ttl=54 id=31929 icmp_seq=1 rtt=79.1 ms
ICMP timestamp: Originate=47167205 Receive=47167241 Transmit=47167241
ICMP timestamp RTT tsrtt=79

len=46 ip=9.9.9.9 ttl=54 id=31931 icmp_seq=6 rtt=92.9 ms
ICMP timestamp: Originate=47167210 Receive=47167249 Transmit=47167249
ICMP timestamp RTT tsrtt=93

len=46 ip=9.9.9.9 ttl=54 id=31933 icmp_seq=4 rtt=95.4 ms
ICMP timestamp: Originate=47167208 Receive=47167249 Transmit=47167249
ICMP timestamp RTT tsrtt=95

len=46 ip=9.9.9.9 ttl=54 id=31932 icmp_seq=7 rtt=92.5 ms
ICMP timestamp: Originate=47167211 Receive=47167249 Transmit=47167249
ICMP timestamp RTT tsrtt=93

len=46 ip=9.9.9.9 ttl=54 id=31934 icmp_seq=8 rtt=91.8 ms
ICMP timestamp: Originate=47167212 Receive=47167249 Transmit=47167249
ICMP timestamp RTT tsrtt=92

len=46 ip=9.9.9.9 ttl=54 id=31935 icmp_seq=5 rtt=95.2 ms
ICMP timestamp: Originate=47167209 Receive=47167249 Transmit=47167249
ICMP timestamp RTT tsrtt=95

len=46 ip=9.9.9.9 ttl=54 id=31936 icmp_seq=9 rtt=91.3 ms
ICMP timestamp: Originate=47167213 Receive=47167250 Transmit=47167250
ICMP timestamp RTT tsrtt=92


--- 9.9.9.9 hping statistic ---
10 packets tramitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 76.7/87.2/95.4 ms

Lochnair · December 8, 2021, 1:12pm

Need to clean up my changes first at the very least. I don't really know this process either, but I think it's better to add my changes as patches the build systems applies at build time, so we can stay on the upstream repository.

_FailSafe · December 8, 2021, 1:18pm

I'm very willing to help test and confirm the patch(es) once ready. Anything I can do to help, let me know!

Lynx · December 8, 2021, 1:19pm

@moeller0 at the moment we have:

# get minimum RTT across entire set of reflectors
get_RTT() {

for reflector in $reflectors;
do
        echo $(/usr/bin/ping -i 0.00 -c 10 $reflector | tail -1 | awk '{print $4}' | cut -d '/' -f 2) >> $RTTs&
done
wait
RTT=$(echo $(cat $RTTs) | awk 'min=="" || $1 < min {min=$1} END {print min}')
> $RTTs
}

So actually this takes the minimum average across 10 pings. That is not quite the minimum of minimums. But it has worked well so far. Any thoughts?

_FailSafe · December 8, 2021, 1:20pm

LOL I understand! Glad the SDK route worked out for getting it built.

moeller0 · December 8, 2021, 2:09pm

My local version changed that to 3 and I think that we are fine with a single probe per reflector as long as we have say 5 reflectors. But I read this as extracting the reported min field of the ping summary line so that should be minimum of minimums, no?
We then should calculate the DL_dOWD (download delta one-way-delay) and the UL_dOWD per reflector and take the minimum of each over the 5 reflectors.

Since OWD reflectors might be picky, maybe we should still also still run the RTT code and in case we have no valid OWDs in a tick evaluate the RTTs instead?

Lynx · December 8, 2021, 2:35pm

root@OpenWrt:~# ping -i 0.00 -c 10 8.8.8.8 | tail -1
rtt min/avg/max/mdev = 63.362/71.922/83.376/4.892 ms, pipe 5, ipg/ewma 19.637/72.387 ms

So I think the -f 2 in:

ping -i 0.00 -c 10 8.8.8.8 | tail -1 | awk '{print $4}' | cut -d '/' -f 2

will actually work with the average I think (sorry I may have been unclear earlier in this thread), but no matter for now anyway. Will try firstly working the minimums as you suggest. Shouldn't take long.

moeller0 · December 8, 2021, 2:49pm

Ah, I guess I need to step through this procedure, but this is not really critical anyways, except that -i 0.00 is not a good idea this basically floods 10 ICMP packets into the link which for sufficiently slow links might already start to noticeably influence each others' delay. Sure pings are small packets, but that still adds up.

dlakelan · December 8, 2021, 3:35pm

Agreed. Let's consider a little the psychophysics of the situation. How much delay is unbearable? We can certainly notice on VOIP if a buffer fails, typical jitter buffers are say 3 packets and that's 60ms. But this is the minimum noticeable delay. So if we can respond in 3 times that amount it should be sufficient. So 180ms. We certainly don't need 10 pings, so I suggest 3 pings spaced .06 seconds apart.

I'm still concerned about rolling this out at scale. If I were 9.9.9.9 and I started to see pings flood in I'd certainly just start dropping them above 20/s or so.

Can someone try a traceroute to 9.9.9.9 and then try hping to each step along the way, see what happens?

I'll do it from my network as well

So, results from my network, using hping3 from Debian.

9.9.9.9 responds with timestamps (yay).
Nothing else along the path responds at all when I request timestamps (boo, ??)
Everything else along the path responds fine with regular icmp pings without timestamps (yay).

Let's see what others report.

Also, my Debian based router responds with timestamps, but the response time is WAY longer than regular ping.

it responds to a regular ping in 0.23 ms, it responds to a timestamp request in about 4-12 ms

Also also, even though my desktop machine is synchronized to my router and my router is synchronized to several Stratum 1 time servers, there's a ~ 30-40 ms time difference between my desktop (which just woke from sleep a half hour ago) and the router.

moeller0 · December 8, 2021, 3:58pm

In fact, real bufferbloat that we can mitigate will be persistent as long as:
bottleneck_rate < load_rate <= shaper_rate
And hence earlier latency measurements will be probably less helpful as more recent ones. So maybe just go for a single sample per reflector? Which will also take the least time, potentially allowing faster tick rates (which in turn might still be limited by how fast we can get ICMP timestamps).

Sure:

user@work-horse:~$ sudo mtr -ezb4w -c 30 9.9.9.9
Start: 2021-12-08T16:42:35+0100
HOST: work-horse                                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    _gateway (192.168.42.1)                                              0.0%    30    0.3   0.3   0.3   0.4   0.0
  2. AS6805   loopback1.0001.acln.01.ham.de.net.telefonica.de (62.52.200.147)      0.0%    30    9.4  10.3   9.0  22.8   3.1
  3. AS6805   bundle-ether27.0006.dbrx.01.ham.de.net.telefonica.de (62.53.12.14)   0.0%    30    9.5   9.7   9.1  10.6   0.3
  4. AS6805   ae1-0.0002.prrx.01.ham.de.net.telefonica.de (62.53.6.207)            0.0%    30    9.6   9.7   8.9  17.1   1.6
  5. AS???    as42.hamburg.megaport.com (193.42.155.3)                             0.0%    30   18.5  19.2  18.1  33.3   2.7
  6. AS19281  dns9.quad9.net (9.9.9.9)                                             0.0%    30   17.8  18.0  17.4  20.3   0.5

user@work-horse:~$ sudo hping3 62.52.200.147 --icmp --icmp-ts -c 1
HPING 62.52.200.147 (eth0 62.52.200.147): icmp mode set, 28 headers + 0 data bytes
len=46 ip=62.52.200.147 ttl=254 id=18582 icmp_seq=0 rtt=27.9 ms
ICMP timestamp: Originate=56640783 Receive=3716931435 Transmit=3716931435
ICMP timestamp RTT tsrtt=28

--- 62.52.200.147 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 27.9/27.9/27.9 ms


moeller@work-horse:~$ sudo hping3 62.53.12.14 --icmp --icmp-ts -c 1
HPING 62.53.12.14 (eth0 62.53.12.14): icmp mode set, 28 headers + 0 data bytes

--- 62.53.12.14 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms


moeller@work-horse:~$ sudo hping3 62.53.6.207 --icmp --icmp-ts -c 1
HPING 62.53.6.207 (eth0 62.53.6.207): icmp mode set, 28 headers + 0 data bytes
len=46 ip=62.53.6.207 ttl=61 id=12445 icmp_seq=0 rtt=15.8 ms
ICMP timestamp: Originate=56673715 Receive=56673719 Transmit=56673719
ICMP timestamp RTT tsrtt=16

--- 62.53.6.207 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 15.8/15.8/15.8 ms


user@work-horse:~$ sudo hping3 193.42.155.3 --icmp --icmp-ts -c 1
HPING 193.42.155.3 (eth0 193.42.155.3): icmp mode set, 28 headers + 0 data bytes

--- 193.42.155.3 hping statistic ---
1 packets transmitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms


user@work-horse:~$ sudo hping3 9.9.9.9 --icmp --icmp-ts -c 1
HPING 9.9.9.9 (eth0 9.9.9.9): icmp mode set, 28 headers + 0 data bytes
len=46 ip=9.9.9.9 ttl=59 id=3368 icmp_seq=0 rtt=19.9 ms
ICMP timestamp: Originate=56724315 Receive=56724324 Transmit=56724324
ICMP timestamp RTT tsrtt=20

--- 9.9.9.9 hping statistic ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 19.9/19.9/19.9 ms

Repeated measurement against 62.52.200.147 result in somewhat fluctuating rtt values, while against 9.9.9.9 are relatively flat, also I note 62.52.200.147 is having "funky time".

dlakelan · December 8, 2021, 4:01pm

How about just regular pings against say 62.52.200.147 ?

how's the fluctuation in rtt?

For example, here's my first hop after the ISP router in my closet:

64 bytes from 162.207.92.1: icmp_seq=1 ttl=62 time=2.16 ms
64 bytes from 162.207.92.1: icmp_seq=2 ttl=62 time=1.98 ms
64 bytes from 162.207.92.1: icmp_seq=3 ttl=62 time=2.15 ms
64 bytes from 162.207.92.1: icmp_seq=4 ttl=62 time=2.13 ms
64 bytes from 162.207.92.1: icmp_seq=5 ttl=62 time=2.95 ms
64 bytes from 162.207.92.1: icmp_seq=6 ttl=62 time=2.10 ms
64 bytes from 162.207.92.1: icmp_seq=7 ttl=62 time=1.96 ms
64 bytes from 162.207.92.1: icmp_seq=8 ttl=62 time=3.03 ms
64 bytes from 162.207.92.1: icmp_seq=9 ttl=62 time=1.74 ms
64 bytes from 162.207.92.1: icmp_seq=10 ttl=62 time=2.06 ms

which seems to be a perfectly suitable reflector. To the extent that more people could use a close hop for basic RTT and then maybe when an issue is detected send a probe to 9.9.9.9 for each-way delay inspection, that could maybe lower the traffic across the internet and concentrated to 9.9.9.9 and others by several orders of magnitude

For fun, here's pings at 0.2 s intervals during a speed test:

dlakelan@tintin:~$ ping -i 0.2 162.207.92.1
PING 162.207.92.1 (162.207.92.1) 56(84) bytes of data.
64 bytes from 162.207.92.1: icmp_seq=1 ttl=62 time=2.06 ms
64 bytes from 162.207.92.1: icmp_seq=2 ttl=62 time=2.18 ms
64 bytes from 162.207.92.1: icmp_seq=3 ttl=62 time=2.50 ms
64 bytes from 162.207.92.1: icmp_seq=4 ttl=62 time=2.57 ms
64 bytes from 162.207.92.1: icmp_seq=5 ttl=62 time=1.81 ms
64 bytes from 162.207.92.1: icmp_seq=6 ttl=62 time=1.93 ms
64 bytes from 162.207.92.1: icmp_seq=7 ttl=62 time=1.83 ms
64 bytes from 162.207.92.1: icmp_seq=8 ttl=62 time=2.27 ms
64 bytes from 162.207.92.1: icmp_seq=9 ttl=62 time=1.76 ms
64 bytes from 162.207.92.1: icmp_seq=10 ttl=62 time=1.62 ms
64 bytes from 162.207.92.1: icmp_seq=11 ttl=62 time=1.59 ms
64 bytes from 162.207.92.1: icmp_seq=12 ttl=62 time=1.66 ms
64 bytes from 162.207.92.1: icmp_seq=13 ttl=62 time=1.47 ms
64 bytes from 162.207.92.1: icmp_seq=14 ttl=62 time=2.04 ms
64 bytes from 162.207.92.1: icmp_seq=15 ttl=62 time=1.62 ms
64 bytes from 162.207.92.1: icmp_seq=16 ttl=62 time=1.51 ms
64 bytes from 162.207.92.1: icmp_seq=17 ttl=62 time=1.52 ms
64 bytes from 162.207.92.1: icmp_seq=18 ttl=62 time=1.48 ms
64 bytes from 162.207.92.1: icmp_seq=19 ttl=62 time=1.70 ms
64 bytes from 162.207.92.1: icmp_seq=20 ttl=62 time=1.69 ms
64 bytes from 162.207.92.1: icmp_seq=21 ttl=62 time=2.22 ms
64 bytes from 162.207.92.1: icmp_seq=22 ttl=62 time=12.3 ms
64 bytes from 162.207.92.1: icmp_seq=23 ttl=62 time=1.66 ms
64 bytes from 162.207.92.1: icmp_seq=24 ttl=62 time=1.46 ms
64 bytes from 162.207.92.1: icmp_seq=25 ttl=62 time=2.13 ms
64 bytes from 162.207.92.1: icmp_seq=26 ttl=62 time=1.72 ms
64 bytes from 162.207.92.1: icmp_seq=27 ttl=62 time=1.68 ms
64 bytes from 162.207.92.1: icmp_seq=28 ttl=62 time=1.43 ms
64 bytes from 162.207.92.1: icmp_seq=29 ttl=62 time=1.66 ms
64 bytes from 162.207.92.1: icmp_seq=30 ttl=62 time=1.80 ms
64 bytes from 162.207.92.1: icmp_seq=31 ttl=62 time=1.50 ms
64 bytes from 162.207.92.1: icmp_seq=32 ttl=62 time=1.45 ms
64 bytes from 162.207.92.1: icmp_seq=33 ttl=62 time=3.85 ms
64 bytes from 162.207.92.1: icmp_seq=34 ttl=62 time=2.01 ms
64 bytes from 162.207.92.1: icmp_seq=35 ttl=62 time=1.79 ms
64 bytes from 162.207.92.1: icmp_seq=36 ttl=62 time=1.55 ms
64 bytes from 162.207.92.1: icmp_seq=37 ttl=62 time=1.33 ms
64 bytes from 162.207.92.1: icmp_seq=38 ttl=62 time=2.02 ms
64 bytes from 162.207.92.1: icmp_seq=39 ttl=62 time=1.63 ms
64 bytes from 162.207.92.1: icmp_seq=40 ttl=62 time=6.91 ms
64 bytes from 162.207.92.1: icmp_seq=41 ttl=62 time=2.16 ms
64 bytes from 162.207.92.1: icmp_seq=42 ttl=62 time=6.90 ms
64 bytes from 162.207.92.1: icmp_seq=43 ttl=62 time=1.60 ms
64 bytes from 162.207.92.1: icmp_seq=44 ttl=62 time=4.47 ms
64 bytes from 162.207.92.1: icmp_seq=46 ttl=62 time=7.67 ms
64 bytes from 162.207.92.1: icmp_seq=47 ttl=62 time=4.87 ms
64 bytes from 162.207.92.1: icmp_seq=48 ttl=62 time=14.4 ms
64 bytes from 162.207.92.1: icmp_seq=49 ttl=62 time=4.65 ms
64 bytes from 162.207.92.1: icmp_seq=50 ttl=62 time=6.36 ms
64 bytes from 162.207.92.1: icmp_seq=51 ttl=62 time=6.47 ms
64 bytes from 162.207.92.1: icmp_seq=52 ttl=62 time=9.54 ms
64 bytes from 162.207.92.1: icmp_seq=53 ttl=62 time=5.70 ms
64 bytes from 162.207.92.1: icmp_seq=54 ttl=62 time=4.62 ms
64 bytes from 162.207.92.1: icmp_seq=55 ttl=62 time=4.37 ms
64 bytes from 162.207.92.1: icmp_seq=56 ttl=62 time=7.26 ms
64 bytes from 162.207.92.1: icmp_seq=57 ttl=62 time=5.99 ms
64 bytes from 162.207.92.1: icmp_seq=58 ttl=62 time=4.67 ms
64 bytes from 162.207.92.1: icmp_seq=59 ttl=62 time=5.38 ms
64 bytes from 162.207.92.1: icmp_seq=60 ttl=62 time=4.94 ms
64 bytes from 162.207.92.1: icmp_seq=61 ttl=62 time=10.5 ms
64 bytes from 162.207.92.1: icmp_seq=62 ttl=62 time=3.34 ms
64 bytes from 162.207.92.1: icmp_seq=63 ttl=62 time=4.22 ms
64 bytes from 162.207.92.1: icmp_seq=64 ttl=62 time=5.56 ms
64 bytes from 162.207.92.1: icmp_seq=65 ttl=62 time=6.66 ms
64 bytes from 162.207.92.1: icmp_seq=66 ttl=62 time=6.96 ms
64 bytes from 162.207.92.1: icmp_seq=67 ttl=62 time=4.23 ms
64 bytes from 162.207.92.1: icmp_seq=68 ttl=62 time=4.07 ms
64 bytes from 162.207.92.1: icmp_seq=69 ttl=62 time=7.58 ms
64 bytes from 162.207.92.1: icmp_seq=70 ttl=62 time=5.71 ms
64 bytes from 162.207.92.1: icmp_seq=71 ttl=62 time=12.4 ms
64 bytes from 162.207.92.1: icmp_seq=72 ttl=62 time=8.01 ms
64 bytes from 162.207.92.1: icmp_seq=73 ttl=62 time=10.2 ms
64 bytes from 162.207.92.1: icmp_seq=74 ttl=62 time=3.70 ms
64 bytes from 162.207.92.1: icmp_seq=75 ttl=62 time=10.7 ms
64 bytes from 162.207.92.1: icmp_seq=76 ttl=62 time=2.09 ms
64 bytes from 162.207.92.1: icmp_seq=77 ttl=62 time=11.7 ms
64 bytes from 162.207.92.1: icmp_seq=78 ttl=62 time=2.18 ms
64 bytes from 162.207.92.1: icmp_seq=79 ttl=62 time=1.70 ms
64 bytes from 162.207.92.1: icmp_seq=80 ttl=62 time=1.55 ms
64 bytes from 162.207.92.1: icmp_seq=81 ttl=62 time=1.38 ms
64 bytes from 162.207.92.1: icmp_seq=82 ttl=62 time=1.55 ms
64 bytes from 162.207.92.1: icmp_seq=83 ttl=62 time=1.67 ms
64 bytes from 162.207.92.1: icmp_seq=84 ttl=62 time=1.62 ms
64 bytes from 162.207.92.1: icmp_seq=85 ttl=62 time=1.62 ms
64 bytes from 162.207.92.1: icmp_seq=86 ttl=62 time=1.47 ms
64 bytes from 162.207.92.1: icmp_seq=87 ttl=62 time=1.69 ms
64 bytes from 162.207.92.1: icmp_seq=88 ttl=62 time=1.57 ms
64 bytes from 162.207.92.1: icmp_seq=89 ttl=62 time=1.95 ms
64 bytes from 162.207.92.1: icmp_seq=90 ttl=62 time=2.04 ms
64 bytes from 162.207.92.1: icmp_seq=91 ttl=62 time=1.60 ms
64 bytes from 162.207.92.1: icmp_seq=92 ttl=62 time=1.45 ms
64 bytes from 162.207.92.1: icmp_seq=93 ttl=62 time=1.69 ms
64 bytes from 162.207.92.1: icmp_seq=94 ttl=62 time=1.69 ms
64 bytes from 162.207.92.1: icmp_seq=95 ttl=62 time=1.65 ms
64 bytes from 162.207.92.1: icmp_seq=96 ttl=62 time=1.46 ms
64 bytes from 162.207.92.1: icmp_seq=97 ttl=62 time=1.57 ms
64 bytes from 162.207.92.1: icmp_seq=98 ttl=62 time=1.95 ms
64 bytes from 162.207.92.1: icmp_seq=99 ttl=62 time=1.52 ms
64 bytes from 162.207.92.1: icmp_seq=100 ttl=62 time=1.55 ms

Lynx · December 8, 2021, 4:06pm

But the whole point of timestamps is to offer directionality information about bufferbloat to know whether to reduce upload rate, download rate, or both. And 'funky time' described above had a good reason relating to the way the nodes handle ping requests in terms of hardcore fast processors or weak processors, or something along these lines?