Personally I think these "power-boost" like very short rate bursts allowed by some ISPs are mostly attempts at making their customers short speed-tests look great. I would simply ignore these at least as these techniques appear pretty rarely used. But then I might be wrong here based on my link not offering that feature ;).
Well as before if we reduce the rate drastically we need to give flows some time to react and during that time transient queueing will increase.
Any plans to auto-fine the adaptive settings?
I have tried a few settings with my LTE connection but no luck.
As far as I know, for bash-autorate there are no real plans to make anything auto-tuning; we do play with the idea of making some off-line tool to at least deduce recommendations for the delay-thresholds based on a log file recorded on a quiescent network, but that is mostly vapor ware right now.
Maybe you post your config file and a log file recorded over say 15-30 minutes with mostly no traffic interspersed by a few speedtests (note that in fast.com you can configure to also test the upload, and also extend the test duration to 30 seconds or more) so we have mostly low/no load samples but also a few high load samples as well.
I agree, it skews speed tests since their test time window is short. However, I do see noticeable (not just in theory) increased responsiveness when browsing. With my current settings cloudflare speed test reports for upload:
26.56Mbps for 3s
32.88MBps for 2.5s
22.36Mbps for the reminder of the test
So, for uploads cake-autorate already accommodates the burst a little. On the other hand, downloads do not burst as they could.
I 'll try it.
OK, after running it for a couple days it blocked again. I had it logging to syslog but unfortunately it must have happened overnight and the log was rotated away on my router so I don't have anything in the logs from when it first started blocking.
But I'm leaving it in this blocking state for now and you can let me know if there's anything you want me to try to figure out where it is blocking.
I did try strace but I don't know if it's helpful.
First, I had to figure out which process was what:
root@OpenWrt:~# ps | grep auto
6468 root 2020 S {cake-autorate_l} /bin/bash /root/cake-autorate/cake-autorate_launcher.sh
6470 root 2568 S {cake-autorate.s} /bin/bash /root/cake-autorate/cake-autorate.sh /root/cake-autorate/cake-autorate_config.primary.sh
6489 root 2432 S {cake-autorate.s} /bin/bash /root/cake-autorate/cake-autorate.sh /root/cake-autorate/cake-autorate_config.primary.sh
6500 root 2484 S {cake-autorate.s} /bin/bash /root/cake-autorate/cake-autorate.sh /root/cake-autorate/cake-autorate_config.primary.sh
6523 root 2524 S {cake-autorate.s} /bin/bash /root/cake-autorate/cake-autorate.sh /root/cake-autorate/cake-autorate_config.primary.sh
20730 root 1316 S tail -F cake-autorate.primary.log
20749 root 1312 S grep auto
25275 root 2652 S {cake-autorate.s} /bin/bash /root/cake-autorate/cake-autorate.sh /root/cake-autorate/cake-autorate_config.primary.sh
root@OpenWrt:~# pgrep -P 6468
6470
6471
root@OpenWrt:~# ps | grep 6471
6471 root 1312 S sleep inf
20754 root 1312 S grep 6471
So it looks like the launcher spawned PID 6470 as the autorate and 6471 as this infinite sleep.
But when I do:
root@OpenWrt:~# strace -y -p 6470
strace: Process 6470 attached
wait4(-1,
^Cstrace: Process 6470 detached
It doesn't finish printing that wait4 line...does anyone know why?
Other processes:
root@OpenWrt:~# strace -y -p 6489
strace: Process 6489 attached
read(0<pipe:[6055]>,
root@OpenWrt:~# strace -y -p 6500
strace: Process 6500 attached
(streams of output for this one)
....
read(0</tmp/run/cake-autorate/primary/log_fifo>, "4", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, ";", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, " ", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "1", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "\n", 1) = 1
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 16) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
open("/var/log/cake-autorate.primary.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3</tmp/log/cake-autorate.primary.log>
fcntl64(1</dev/null>, F_GETFD) = 0
fcntl64(1</dev/null>, F_DUPFD, 10) = 13</dev/null>
fcntl64(1</dev/null>, F_GETFD) = 0
fcntl64(13</dev/null>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/log/cake-autorate.primary.log>, 1</dev/null>) = 1</tmp/log/cake-autorate.primary.log>
close(3</tmp/log/cake-autorate.primary.log>) = 0
writev(1</tmp/log/cake-autorate.primary.log>, [{iov_base="LOAD; 2023-01-26-11:19:58; 16747"..., iov_len=84}, {iov_base="\n", iov_len=1}], 2) = 85
dup2(13</dev/null>, 1</tmp/log/cake-autorate.primary.log>) = 1</dev/null>
fcntl64(13</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(13</dev/null>) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/log_fifo>, TIOCGWINSZ, 0x7fac2330) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/log_fifo>, 0, 0x7fac2330, SEEK_CUR) = -1 ESPIPE (Invalid seek)
read(0</tmp/run/cake-autorate/primary/log_fifo>, ^Cstrace: Process 6500 detached
<detached ...>
root@OpenWrt:~# strace -y -p 6523
strace: Process 6523 attached
More streams of output
.....
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
open("/var/run/cake-autorate/primary/ul_shaper_rate_kbps", O_RDONLY|O_LARGEFILE) = 3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(0</dev/null>, F_DUPFD, 10) = 12</dev/null>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(12</dev/null>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, 0</dev/null>) = 0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>
close(3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, TIOCGWINSZ, 0x7fac1a28) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, 0, [0], SEEK_CUR) = 0
read(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, "10000", 4096) = 5
read(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, "", 4096) = 0
dup2(12</dev/null>, 0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>) = 0</dev/null>
fcntl64(12</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(12</dev/null>) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e14000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e12000
munmap(0x77e14000, 4096) = 0
munmap(0x77e12000, 4096) = 0
statx(AT_FDCWD</>, "/var/run/cake-autorate/primary/log_fifo", AT_STATX_SYNC_AS_STAT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFIFO|0644, stx_size=0, ...}) = 0
open("/var/run/cake-autorate/primary/log_fifo", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3</tmp/run/cake-autorate/primary/log_fifo>
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_GETFD) = 0
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_DUPFD, 10) = 12</tmp/run/cake-autorate/primary/log_fifo>
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_GETFD) = 0
fcntl64(12</tmp/run/cake-autorate/primary/log_fifo>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/run/cake-autorate/primary/log_fifo>, 1</tmp/run/cake-autorate/primary/log_fifo>) = 1</tmp/run/cake-autorate/primary/log_fifo>
close(3</tmp/run/cake-autorate/primary/log_fifo>) = 0
munmap(0x77e15000, 23) = 0
open("/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3</tmp/TZ>
fstat64(3</tmp/TZ>, {st_mode=S_IFREG|0644^Cst_size=23, ...}) = 0
strace: Process 6523 detached
root@OpenWrt:~# strace -y -p 25275
strace: Process 25275 attached
wait4(-1,
.... it just sits here
Anything else to try?
Regarding process 6470 and process 25275: wait4(-1, ...) means that it waits for "any child process" to exit. It doesn't finish printing, because the remaining "arguments" are just pointers to places in memory where the results of waiting should be placed, and nothing has been written there yet.
Regarding process 6489: it is trying to read a pipe, which is presumably connected to process 6055 (not sure). The pipe is on its stdin, but it doesn't mean anything, it's just how bash works internally.
Regarding process 6500: it's a logger. Since it still works, please download /tmp/log/cake-autorate.primary.log before it is autorotated.
Regarding process 6523: it's the one which monitors the achieved rates. What it does looks like normal operation.
Next time, when running strace, please add the -s 1000 arguments, so that strings are not truncated as early as they are now.
Yes please keep it like this for a while if you can since this is invaluable to prove what's gone wrong.
The sleep inf is a red herring - it's just used inside the cake-autorate_launcher.sh and gets killed upon service stop.
I am guessing that process 25275 corresponds with maintain_pingers() because it has a much larger number than the others resulting from sucessive kills and recreations associated with sleep and wake from sleep:
@patrakov is there anything else we can do with strace here to work out what's happening? I'd really like to know what line of the cake-autorate.sh this corresponds with.
Ah, so I am guessing code it stuck here:
As called by:
OR
If the latter, and if it's blocking on the FIFO read, then @patrakov the read timeout didn't work here:
And @moeller0 this would again point to either your suggestion of checking every read for specific value, or perhaps better just unjamming FIFO in the trapped function by writing something - anything - to it here:
So if I'm right, the fix may be as simple as adding to line 401:
printf "${RANDOM}\n" > ${run_path}/pinger_${pinger}_fifo
@gba if you re-run the strace does it always show the same?
Also can you post the full .log file here (even though it is the present log file some time after the event that led to this)?
Finally, perhaps you could try running:
printf "${RANDOM}\n" > ${run_path}/pinger_0_fifo
And see if that gets cake-autorate.sh unstuck?
I thought we only read FIFOs in monitor_reflector_responses_() and that likely should be dead, as I bet the wait4 is in kill_and_wait_by_pid_name(). This looks like:
a) while read -t 1 timestamp reflector _ seq_rtt 2>/dev/null does NOT seem to actually exit by virtue of the timeout.
b) mutual exclusion of places that kill and start monitor_reflector_responses_ seems pretty much required.
c) maybe an explicit "please terminate yourself" message/logic in monitor_reflector_responses_() is in order.
To test my hypothesis @gba should simply try:
echo "anything" > /tmp/run/cake-autorate/primary/pinger_0_fifo
this should get the sleep unstuck.
OK before I try unsticking, here's the followup to strace with the longer lines:
PID 6500:
read(0</tmp/run/cake-autorate/primary/log_fifo>, "9", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "5", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "4", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, ";", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, " ", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "1", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "0", 1) = 1
read(0</tmp/run/cake-autorate/primary/log_fifo>, "\n", 1) = 1
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 16) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
open("/var/log/cake-autorate.primary.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0666) = 3</tmp/log/cake-autorate.primary.log>
fcntl64(1</dev/null>, F_GETFD) = 0
fcntl64(1</dev/null>, F_DUPFD, 10) = 13</dev/null>
fcntl64(1</dev/null>, F_GETFD) = 0
fcntl64(13</dev/null>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/log/cake-autorate.primary.log>, 1</dev/null>) = 1</tmp/log/cake-autorate.primary.log>
close(3</tmp/log/cake-autorate.primary.log>) = 0
writev(1</tmp/log/cake-autorate.primary.log>, [{iov_base="LOAD; 2023-01-26-12:41:06; 1674758466.597987; 1674758466.597209; 57; 106; 89054; 10000", iov_len=86}, {iov_base="\n", iov_len=1}], 2) = 87
dup2(13</dev/null>, 1</tmp/log/cake-autorate.primary.log>) = 1</dev/null>
fcntl64(13</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(13</dev/null>) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/log_fifo>, TIOCGWINSZ, 0x7fac2330) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/log_fifo>, 0, 0x7fac2330, SEEK_CUR) = -1 ESPIPE (Invalid seek)
read(0</tmp/run/cake-autorate/primary/log_fifo>, ^Cstrace: Process 6500 detached
<detached ...>
PID 6523:
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/dl_shaper_rate_kbps>, TIOCGWINSZ, 0x7fac19f8) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/dl_shaper_rate_kbps>, 0, [0], SEEK_CUR) = 0
read(0</tmp/run/cake-autorate/primary/dl_shaper_rate_kbps>, "89054", 4096) = 5
read(0</tmp/run/cake-autorate/primary/dl_shaper_rate_kbps>, "", 4096) = 0
dup2(12</dev/null>, 0</tmp/run/cake-autorate/primary/dl_shaper_rate_kbps>) = 0</dev/null>
fcntl64(12</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(12</dev/null>) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e14000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e12000
munmap(0x77e14000, 4096) = 0
munmap(0x77e12000, 4096) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 16) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
open("/var/run/cake-autorate/primary/ul_shaper_rate_kbps", O_RDONLY|O_LARGEFILE) = 3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(0</dev/null>, F_DUPFD, 10) = 12</dev/null>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(12</dev/null>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, 0</dev/null>) = 0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>
close(3</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, TIOCGWINSZ, 0x7fac1a28) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, 0, [0], SEEK_CUR) = 0
read(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, "10000", 4096) = 5
read(0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>, "", 4096) = 0
dup2(12</dev/null>, 0</tmp/run/cake-autorate/primary/ul_shaper_rate_kbps>) = 0</dev/null>
fcntl64(12</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(12</dev/null>) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e14000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e12000
munmap(0x77e14000, 4096) = 0
munmap(0x77e12000, 4096) = 0
statx(AT_FDCWD</>, "/var/run/cake-autorate/primary/log_fifo", AT_STATX_SYNC_AS_STAT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFIFO|0644, stx_size=0, ...}) = 0
open("/var/run/cake-autorate/primary/log_fifo", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3</tmp/run/cake-autorate/primary/log_fifo>
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_GETFD) = 0
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_DUPFD, 10) = 12</tmp/run/cake-autorate/primary/log_fifo>
fcntl64(1</tmp/run/cake-autorate/primary/log_fifo>, F_GETFD) = 0
fcntl64(12</tmp/run/cake-autorate/primary/log_fifo>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/run/cake-autorate/primary/log_fifo>, 1</tmp/run/cake-autorate/primary/log_fifo>) = 1</tmp/run/cake-autorate/primary/log_fifo>
close(3</tmp/run/cake-autorate/primary/log_fifo>) = 0
munmap(0x77e15000, 23) = 0
open("/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3</tmp/TZ>
fstat64(3</tmp/TZ>, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
mmap2(NULL, 23, PROT_READ, MAP_SHARED, 3</tmp/TZ>, 0) = 0x77e15000
close(3</tmp/TZ>) = 0
munmap(0x77e15000, 23) = 0
open("/etc/TZ", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC) = 3</tmp/TZ>
fstat64(3</tmp/TZ>, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
mmap2(NULL, 23, PROT_READ, MAP_SHARED, 3</tmp/TZ>, 0) = 0x77e15000
close(3</tmp/TZ>) = 0
writev(1</tmp/run/cake-autorate/primary/log_fifo>, [{iov_base="LOAD; 2023-01-26-12:41:39; 1674758499.964866; 1674758499.964051; 26; 10; 89054; 10000", iov_len=85}, {iov_base="\n", iov_len=1}], 2) = 86
dup2(12</tmp/run/cake-autorate/primary/log_fifo>, 1</tmp/run/cake-autorate/primary/log_fifo>) = 1</tmp/run/cake-autorate/primary/log_fifo>
fcntl64(12</tmp/run/cake-autorate/primary/log_fifo>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(12</tmp/run/cake-autorate/primary/log_fifo>) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 16) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
open("/var/run/cake-autorate/primary/max_wire_packet_rtt_us", O_RDONLY|O_LARGEFILE) = 3</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(0</dev/null>, F_DUPFD, 10) = 12</dev/null>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(12</dev/null>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>, 0</dev/null>) = 0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>
close(3</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 16) = 0
ioctl(0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>, TIOCGWINSZ, 0x7fac2160) = -1 ENOTTY (Not a tty)
_llseek(0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>, 0, [0], SEEK_CUR) = 0
read(0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>, "1334", 4096) = 4
read(0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>, "", 4096) = 0
dup2(12</dev/null>, 0</tmp/run/cake-autorate/primary/max_wire_packet_rtt_us>) = 0</dev/null>
fcntl64(12</dev/null>, F_GETFD) = 0x1 (flags FD_CLOEXEC)
close(12</dev/null>) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e14000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x77e12000
munmap(0x77e14000, 4096) = 0
munmap(0x77e12000, 4096) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 16) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 16) = 0
statx(AT_FDCWD</>, "/sys/class/net/ifb4eth0/statistics/tx_bytes", AT_STATX_SYNC_AS_STAT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0444, stx_size=4096, ...}) = 0
open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3</dev/null>
fcntl64(2<pipe:[6055]>, F_GETFD) = 0
fcntl64(2<pipe:[6055]>, F_DUPFD, 10) = 12<pipe:[6055]>
fcntl64(2<pipe:[6055]>, F_GETFD) = 0
fcntl64(12<pipe:[6055]>, F_SETFD, FD_CLOEXEC) = 0
dup2(3</dev/null>, 2<pipe:[6055]>) = 2</dev/null>
close(3</dev/null>) = 0
open("/sys/class/net/ifb4eth0/statistics/tx_bytes", O_RDONLY|O_LARGEFILE) = 3</sys/devices/virtual/net/ifb4eth0/statistics/tx_bytes>
fcntl64(0</dev/null>, F_GETFD) = 0
fcntl64(0</dev/null>, F_DUPFD, 10) = 13</dev/null>
fcntl64(0</dev/null>, F_GETFD) = 0^Cstrace: Process 6523 detached
And here's the log file:
DEBUG; 2023-01-26-12:41:07; 1674758467.271144; Starting: rotate_log_file with PID: 6500
DEBUG; 2023-01-26-12:41:07; 1674758467.284839; Starting: print_headers with PID: 6500
DATA_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; DL_ACHIEVED_RATE_KBPS; UL_ACHIEVED_RATE_KBPS; DL_LOAD_PERCENT; UL_LOAD_PERCENT; RTT_TIMESTAMP; REFLECTOR; SEQUENCE; DL_OWD_BASELINE; DL_OWD_US; DL_OWD_DELTA_EWMA_US; DL_OWD_DELTA_US; DL_ADJ_DELAY_THR; UL_OWD_BASELINE; UL_OWD_US; UL_OWD_DELTA_EWMA_US; UL_OWD_DELTA_US; UL_ADJ_DELAY_THR; SUM_DL_DELAYS; SUM_UL_DELAYS; DL_LOAD_CONDITION; UL_LOAD_CONDITION; CAKE_DL_RATE_KBPS; CAKE_UL_RATE_KBPS
LOAD_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; DL_ACHIEVED_RATE_KBPS; UL_ACHIEVED_RATE_KBPS; CAKE_DL_RATE_KBPS; CAKE_UL_RATE_KBPS
REFLECTOR_HEADER; LOG_DATETIME; LOG_TIMESTAMP; PROC_TIME_US; REFLECTOR; DL_MIN_BASELINE_US; DL_BASELINE_US; DL_BASELINE_DELTA_US; DL_BASELINE_DELTA_THR_US; DL_MIN_DELTA_EWMA_US; DL_DELTA_EWMA_US; DL_DELTA_EWMA_DELTA_US; DL_DELTA_EWMA_DELTA_THR; UL_MIN_BASELINE_US; UL_BASELINE_US; UL_BASELINE_DELTA_US; UL_BASELINE_DELTA_THR_US; UL_MIN_DELTA_EWMA_US; UL_DELTA_EWMA_US; UL_DELTA_EWMA_DELTA_US; UL_DELTA_EWMA_DELTA_THR
LOAD; 2023-01-26-12:41:07; 1674758467.438336; 1674758467.437551; 38; 20; 89054; 10000
LOAD; 2023-01-26-12:41:07; 1674758467.648407; 1674758467.647621; 9; 4; 89054; 10000
LOAD; 2023-01-26-12:41:07; 1674758467.857979; 1674758467.857198; 18; 15; 89054; 10000
LOAD; 2023-01-26-12:41:08; 1674758468.068210; 1674758468.067431; 112; 166; 89054; 10000
LOAD; 2023-01-26-12:41:08; 1674758468.277926; 1674758468.277147; 62; 15; 89054; 10000
LOAD; 2023-01-26-12:41:08; 1674758468.487915; 1674758468.487137; 25; 12; 89054; 10000
LOAD; 2023-01-26-12:41:08; 1674758468.697920; 1674758468.697145; 31; 18; 89054; 10000
LOAD; 2023-01-26-12:41:08; 1674758468.908270; 1674758468.907491; 20; 34; 89054; 10000
LOAD; 2023-01-26-12:41:09; 1674758469.118269; 1674758469.117483; 58; 47; 89054; 10000
LOAD; 2023-01-26-12:41:09; 1674758469.328529; 1674758469.327726; 52; 97; 89054; 10000
LOAD; 2023-01-26-12:41:09; 1674758469.537930; 1674758469.537152; 117; 23; 89054; 10000
LOAD; 2023-01-26-12:41:09; 1674758469.748245; 1674758469.747464; 40; 35; 89054; 10000
LOAD; 2023-01-26-12:41:09; 1674758469.957916; 1674758469.957141; 81; 19; 89054; 10000
LOAD; 2023-01-26-12:41:10; 1674758470.167962; 1674758470.167177; 26; 11; 89054; 10000
LOAD; 2023-01-26-12:41:10; 1674758470.377898; 1674758470.377115; 50; 28; 89054; 10000
LOAD; 2023-01-26-12:41:10; 1674758470.587924; 1674758470.587137; 24; 15; 89054; 10000
LOAD; 2023-01-26-12:41:10; 1674758470.798010; 1674758470.797227; 32; 20; 89054; 10000
LOAD; 2023-01-26-12:41:11; 1674758471.007934; 1674758471.007153; 69; 30; 89054; 10000
LOAD; 2023-01-26-12:41:11; 1674758471.217907; 1674758471.217127; 27; 9; 89054; 10000
LOAD; 2023-01-26-12:41:11; 1674758471.428278; 1674758471.427495; 33; 15; 89054; 10000
LOAD; 2023-01-26-12:41:11; 1674758471.637929; 1674758471.637150; 37; 19; 89054; 10000
LOAD; 2023-01-26-12:41:11; 1674758471.847961; 1674758471.847176; 15; 6; 89054; 10000
LOAD; 2023-01-26-12:41:12; 1674758472.058128; 1674758472.057355; 29; 14; 89054; 10000
LOAD; 2023-01-26-12:41:12; 1674758472.267818; 1674758472.267041; 36; 50; 89054; 10000
LOAD; 2023-01-26-12:41:12; 1674758472.477932; 1674758472.477147; 54; 55; 89054; 10000
LOAD; 2023-01-26-12:41:12; 1674758472.688436; 1674758472.687644; 35; 7; 89054; 10000
LOAD; 2023-01-26-12:41:12; 1674758472.898002; 1674758472.897217; 23; 11; 89054; 10000
LOAD; 2023-01-26-12:41:13; 1674758473.108006; 1674758473.107223; 189; 207; 89054; 10000
LOAD; 2023-01-26-12:41:13; 1674758473.317986; 1674758473.317204; 58; 19; 89054; 10000
LOAD; 2023-01-26-12:41:13; 1674758473.528052; 1674758473.527277; 52; 33; 89054; 10000
LOAD; 2023-01-26-12:41:13; 1674758473.737961; 1674758473.737181; 52; 57; 89054; 10000
LOAD; 2023-01-26-12:41:13; 1674758473.948419; 1674758473.947631; 323; 61; 89054; 10000
LOAD; 2023-01-26-12:41:14; 1674758474.157911; 1674758474.157130; 27; 25; 89054; 10000
LOAD; 2023-01-26-12:41:14; 1674758474.367937; 1674758474.367156; 34; 40; 89054; 10000
LOAD; 2023-01-26-12:41:14; 1674758474.577987; 1674758474.577200; 35; 17; 89054; 10000
LOAD; 2023-01-26-12:41:14; 1674758474.787941; 1674758474.787154; 19; 9; 89054; 10000
LOAD; 2023-01-26-12:41:14; 1674758474.997981; 1674758474.997202; 22; 11; 89054; 10000
LOAD; 2023-01-26-12:41:15; 1674758475.207951; 1674758475.207173; 51; 50; 89054; 10000
LOAD; 2023-01-26-12:41:15; 1674758475.417923; 1674758475.417147; 37; 21; 89054; 10000
LOAD; 2023-01-26-12:41:15; 1674758475.628416; 1674758475.627623; 22; 9; 89054; 10000
LOAD; 2023-01-26-12:41:15; 1674758475.837839; 1674758475.837066; 29; 18; 89054; 10000
LOAD; 2023-01-26-12:41:16; 1674758476.047945; 1674758476.047164; 27; 12; 89054; 10000
LOAD; 2023-01-26-12:41:16; 1674758476.257913; 1674758476.257135; 24; 9; 89054; 10000
LOAD; 2023-01-26-12:41:16; 1674758476.467970; 1674758476.467194; 27; 15; 89054; 10000
LOAD; 2023-01-26-12:41:16; 1674758476.677913; 1674758476.677133; 22; 11; 89054; 10000
LOAD; 2023-01-26-12:41:16; 1674758476.888301; 1674758476.887521; 25; 19; 89054; 10000
LOAD; 2023-01-26-12:41:17; 1674758477.098014; 1674758477.097232; 80; 35; 89054; 10000
LOAD; 2023-01-26-12:41:17; 1674758477.307923; 1674758477.307146; 26; 9; 89054; 10000
LOAD; 2023-01-26-12:41:17; 1674758477.517904; 1674758477.517138; 35; 22; 89054; 10000
LOAD; 2023-01-26-12:41:17; 1674758477.727949; 1674758477.727173; 24; 13; 89054; 10000
LOAD; 2023-01-26-12:41:17; 1674758477.937871; 1674758477.937096; 31; 150; 89054; 10000
LOAD; 2023-01-26-12:41:18; 1674758478.147959; 1674758478.147181; 107; 56; 89054; 10000
LOAD; 2023-01-26-12:41:18; 1674758478.357932; 1674758478.357154; 93; 4; 89054; 10000
LOAD; 2023-01-26-12:41:18; 1674758478.567924; 1674758478.567147; 30; 20; 89054; 10000
LOAD; 2023-01-26-12:41:18; 1674758478.777955; 1674758478.777176; 32; 16; 89054; 10000
LOAD; 2023-01-26-12:41:18; 1674758478.987947; 1674758478.987170; 20; 7; 89054; 10000
LOAD; 2023-01-26-12:41:19; 1674758479.197883; 1674758479.197110; 22; 7; 89054; 10000
LOAD; 2023-01-26-12:41:19; 1674758479.407919; 1674758479.407142; 101; 102; 89054; 10000
LOAD; 2023-01-26-12:41:19; 1674758479.618083; 1674758479.617300; 72; 32; 89054; 10000
LOAD; 2023-01-26-12:41:19; 1674758479.827953; 1674758479.827177; 52; 38; 89054; 10000
LOAD; 2023-01-26-12:41:20; 1674758480.038000; 1674758480.037211; 38; 18; 89054; 10000
LOAD; 2023-01-26-12:41:20; 1674758480.247939; 1674758480.247156; 207; 101; 89054; 10000
LOAD; 2023-01-26-12:41:20; 1674758480.457913; 1674758480.457136; 25; 11; 89054; 10000
LOAD; 2023-01-26-12:41:20; 1674758480.667902; 1674758480.667131; 48; 38; 89054; 10000
LOAD; 2023-01-26-12:41:20; 1674758480.877854; 1674758480.877083; 43; 53; 89054; 10000
LOAD; 2023-01-26-12:41:21; 1674758481.087924; 1674758481.087139; 83; 67; 89054; 10000
LOAD; 2023-01-26-12:41:21; 1674758481.298122; 1674758481.297344; 36; 12; 89054; 10000
LOAD; 2023-01-26-12:41:21; 1674758481.508278; 1674758481.507496; 1416; 69; 89054; 10000
LOAD; 2023-01-26-12:41:21; 1674758481.717804; 1674758481.717033; 51; 29; 89054; 10000
LOAD; 2023-01-26-12:41:21; 1674758481.928155; 1674758481.927370; 84; 51; 89054; 10000
LOAD; 2023-01-26-12:41:22; 1674758482.137950; 1674758482.137167; 88; 77; 89054; 10000
LOAD; 2023-01-26-12:41:22; 1674758482.348296; 1674758482.347513; 27; 11; 89054; 10000
LOAD; 2023-01-26-12:41:22; 1674758482.557926; 1674758482.557143; 32; 19; 89054; 10000
LOAD; 2023-01-26-12:41:22; 1674758482.767847; 1674758482.767076; 32; 15; 89054; 10000
LOAD; 2023-01-26-12:41:22; 1674758482.978318; 1674758482.977541; 129; 184; 89054; 10000
LOAD; 2023-01-26-12:41:23; 1674758483.188151; 1674758483.187359; 190; 12; 89054; 10000
LOAD; 2023-01-26-12:41:23; 1674758483.397921; 1674758483.397135; 1383; 207; 89054; 10000
LOAD; 2023-01-26-12:41:23; 1674758483.608536; 1674758483.607741; 13393; 182; 89054; 10000
LOAD; 2023-01-26-12:41:23; 1674758483.817961; 1674758483.817174; 9678; 79; 89054; 10000
LOAD; 2023-01-26-12:41:24; 1674758484.027993; 1674758484.027211; 19; 9; 89054; 10000
LOAD; 2023-01-26-12:41:24; 1674758484.238508; 1674758484.237714; 26; 153; 89054; 10000
LOAD; 2023-01-26-12:41:24; 1674758484.447976; 1674758484.447186; 178; 53; 89054; 10000
LOAD; 2023-01-26-12:41:24; 1674758484.657888; 1674758484.657108; 44; 7; 89054; 10000
LOAD; 2023-01-26-12:41:24; 1674758484.867903; 1674758484.867125; 24; 18; 89054; 10000
LOAD; 2023-01-26-12:41:25; 1674758485.077992; 1674758485.077210; 34; 12; 89054; 10000
LOAD; 2023-01-26-12:41:25; 1674758485.287927; 1674758485.287153; 51; 30; 89054; 10000
LOAD; 2023-01-26-12:41:25; 1674758485.497888; 1674758485.497113; 16; 7; 89054; 10000
LOAD; 2023-01-26-12:41:25; 1674758485.707919; 1674758485.707139; 91; 38; 89054; 10000
LOAD; 2023-01-26-12:41:25; 1674758485.918022; 1674758485.917245; 32; 12; 89054; 10000
LOAD; 2023-01-26-12:41:26; 1674758486.127904; 1674758486.127125; 90; 54; 89054; 10000
LOAD; 2023-01-26-12:41:26; 1674758486.337977; 1674758486.337193; 35; 31; 89054; 10000
LOAD; 2023-01-26-12:41:26; 1674758486.547950; 1674758486.547169; 66; 20; 89054; 10000
LOAD; 2023-01-26-12:41:26; 1674758486.757938; 1674758486.757150; 17; 7; 89054; 10000
LOAD; 2023-01-26-12:41:26; 1674758486.969404; 1674758486.968628; 111; 57; 89054; 10000
LOAD; 2023-01-26-12:41:27; 1674758487.178399; 1674758487.177609; 52; 56; 89054; 10000
LOAD; 2023-01-26-12:41:27; 1674758487.387944; 1674758487.387164; 66; 36; 89054; 10000
LOAD; 2023-01-26-12:41:27; 1674758487.597937; 1674758487.597156; 36; 52; 89054; 10000
LOAD; 2023-01-26-12:41:27; 1674758487.808180; 1674758487.807409; 29; 15; 89054; 10000
LOAD; 2023-01-26-12:41:28; 1674758488.017999; 1674758488.017217; 87; 283; 89054; 10000
LOAD; 2023-01-26-12:41:28; 1674758488.227964; 1674758488.227182; 178; 13; 89054; 10000
LOAD; 2023-01-26-12:41:28; 1674758488.437932; 1674758488.437148; 45; 9; 89054; 10000
LOAD; 2023-01-26-12:41:28; 1674758488.648368; 1674758488.647565; 73; 56; 89054; 10000
LOAD; 2023-01-26-12:41:28; 1674758488.857935; 1674758488.857151; 20; 15; 89054; 10000
LOAD; 2023-01-26-12:41:29; 1674758489.067971; 1674758489.067193; 64; 16; 89054; 10000
LOAD; 2023-01-26-12:41:29; 1674758489.277924; 1674758489.277145; 33; 73; 89054; 10000
LOAD; 2023-01-26-12:41:29; 1674758489.488295; 1674758489.487504; 129; 29; 89054; 10000
LOAD; 2023-01-26-12:41:29; 1674758489.698638; 1674758489.697671; 17; 40; 89054; 10000
LOAD; 2023-01-26-12:41:29; 1674758489.907904; 1674758489.907121; 27; 11; 89054; 10000
LOAD; 2023-01-26-12:41:30; 1674758490.117965; 1674758490.117192; 14; 5; 89054; 10000
LOAD; 2023-01-26-12:41:30; 1674758490.327880; 1674758490.327103; 38; 31; 89054; 10000
LOAD; 2023-01-26-12:41:30; 1674758490.537947; 1674758490.537159; 26; 127; 89054; 10000
LOAD; 2023-01-26-12:41:30; 1674758490.747949; 1674758490.747164; 67; 45; 89054; 10000
LOAD; 2023-01-26-12:41:30; 1674758490.958287; 1674758490.957499; 154; 65; 89054; 10000
LOAD; 2023-01-26-12:41:31; 1674758491.167906; 1674758491.167126; 20; 12; 89054; 10000
LOAD; 2023-01-26-12:41:31; 1674758491.377910; 1674758491.377132; 17; 6; 89054; 10000
LOAD; 2023-01-26-12:41:31; 1674758491.588589; 1674758491.587807; 26; 15; 89054; 10000
LOAD; 2023-01-26-12:41:31; 1674758491.797945; 1674758491.797170; 34; 17; 89054; 10000
LOAD; 2023-01-26-12:41:32; 1674758492.008711; 1674758492.007928; 281; 504; 89054; 10000
LOAD; 2023-01-26-12:41:32; 1674758492.217938; 1674758492.217162; 227; 60; 89054; 10000
LOAD; 2023-01-26-12:41:32; 1674758492.427957; 1674758492.427177; 123; 40; 89054; 10000
LOAD; 2023-01-26-12:41:32; 1674758492.637888; 1674758492.637112; 9; 9; 89054; 10000
LOAD; 2023-01-26-12:41:32; 1674758492.847926; 1674758492.847146; 63; 17; 89054; 10000
LOAD; 2023-01-26-12:41:33; 1674758493.058378; 1674758493.057583; 85; 229; 89054; 10000
LOAD; 2023-01-26-12:41:33; 1674758493.267975; 1674758493.267186; 212; 69; 89054; 10000
LOAD; 2023-01-26-12:41:33; 1674758493.477921; 1674758493.477148; 33; 13; 89054; 10000
LOAD; 2023-01-26-12:41:33; 1674758493.688218; 1674758493.687407; 66; 16; 89054; 10000
LOAD; 2023-01-26-12:41:33; 1674758493.898112; 1674758493.897325; 63; 14; 89054; 10000
LOAD; 2023-01-26-12:41:34; 1674758494.107972; 1674758494.107194; 57; 27; 89054; 10000
LOAD; 2023-01-26-12:41:34; 1674758494.317927; 1674758494.317151; 70; 111; 89054; 10000
LOAD; 2023-01-26-12:41:34; 1674758494.527950; 1674758494.527168; 104; 14; 89054; 10000
LOAD; 2023-01-26-12:41:34; 1674758494.737928; 1674758494.737151; 26; 16; 89054; 10000
LOAD; 2023-01-26-12:41:34; 1674758494.947916; 1674758494.947134; 21; 13; 89054; 10000
LOAD; 2023-01-26-12:41:35; 1674758495.157987; 1674758495.157208; 59; 19; 89054; 10000
LOAD; 2023-01-26-12:41:35; 1674758495.367934; 1674758495.367158; 39; 26; 89054; 10000
LOAD; 2023-01-26-12:41:35; 1674758495.577932; 1674758495.577145; 39; 38; 89054; 10000
LOAD; 2023-01-26-12:41:35; 1674758495.787973; 1674758495.787189; 50; 16; 89054; 10000
LOAD; 2023-01-26-12:41:35; 1674758495.997936; 1674758495.997157; 29; 11; 89054; 10000
LOAD; 2023-01-26-12:41:36; 1674758496.208417; 1674758496.207615; 32; 18; 89054; 10000
LOAD; 2023-01-26-12:41:36; 1674758496.417984; 1674758496.417203; 38; 18; 89054; 10000
LOAD; 2023-01-26-12:41:36; 1674758496.627916; 1674758496.627135; 10; 7; 89054; 10000
LOAD; 2023-01-26-12:41:36; 1674758496.837965; 1674758496.837187; 24; 17; 89054; 10000
LOAD; 2023-01-26-12:41:37; 1674758497.047961; 1674758497.047190; 40; 18; 89054; 10000
LOAD; 2023-01-26-12:41:37; 1674758497.257974; 1674758497.257205; 35; 20; 89054; 10000
LOAD; 2023-01-26-12:41:37; 1674758497.468967; 1674758497.468155; 29; 13; 89054; 10000
LOAD; 2023-01-26-12:41:37; 1674758497.677985; 1674758497.677210; 15; 7; 89054; 10000
LOAD; 2023-01-26-12:41:37; 1674758497.888484; 1674758497.887705; 32; 18; 89054; 10000
LOAD; 2023-01-26-12:41:38; 1674758498.099114; 1674758498.098321; 157; 176; 89054; 10000
LOAD; 2023-01-26-12:41:38; 1674758498.308447; 1674758498.307664; 348; 78; 89054; 10000
LOAD; 2023-01-26-12:41:38; 1674758498.518057; 1674758498.517275; 33; 12; 89054; 10000
LOAD; 2023-01-26-12:41:38; 1674758498.831606; 1674758498.830784; 23; 15; 89054; 10000
LOAD; 2023-01-26-12:41:39; 1674758499.058771; 1674758499.057935; 27; 18; 89054; 10000
LOAD; 2023-01-26-12:41:39; 1674758499.275024; 1674758499.274204; 44; 46; 89054; 10000
LOAD; 2023-01-26-12:41:39; 1674758499.519325; 1674758499.518203; 115; 106; 89054; 10000
LOAD; 2023-01-26-12:41:39; 1674758499.760507; 1674758499.759549; 45; 6; 89054; 10000
LOAD; 2023-01-26-12:41:39; 1674758499.964866; 1674758499.964051; 26; 10; 89054; 10000
LOAD; 2023-01-26-12:41:40; 1674758500.048905; 1674758500.048124; 21; 10; 89054; 10000
LOAD; 2023-01-26-12:41:40; 1674758500.238077; 1674758500.237303; 39; 20; 89054; 10000
LOAD; 2023-01-26-12:41:40; 1674758500.448526; 1674758500.447718; 22; 9; 89054; 10000
LOAD; 2023-01-26-12:41:40; 1674758500.657965; 1674758500.657183; 20; 14; 89054; 10000
LOAD; 2023-01-26-12:41:40; 1674758500.868152; 1674758500.867370; 32; 23; 89054; 10000
LOAD; 2023-01-26-12:41:41; 1674758501.078016; 1674758501.077231; 76; 25; 89054; 10000
LOAD; 2023-01-26-12:41:41; 1674758501.288278; 1674758501.287467; 72; 58; 89054; 10000
LOAD; 2023-01-26-12:41:41; 1674758501.498212; 1674758501.497428; 56; 19; 89054; 10000
LOAD; 2023-01-26-12:41:41; 1674758501.708528; 1674758501.707742; 24; 10; 89054; 10000
LOAD; 2023-01-26-12:41:41; 1674758501.918118; 1674758501.917345; 58; 35; 89054; 10000
LOAD; 2023-01-26-12:41:42; 1674758502.127997; 1674758502.127212; 12; 5; 89054; 10000
LOAD; 2023-01-26-12:41:42; 1674758502.337957; 1674758502.337175; 40; 15; 89054; 10000
LOAD; 2023-01-26-12:41:42; 1674758502.547963; 1674758502.547175; 38; 22; 89054; 10000
LOAD; 2023-01-26-12:41:42; 1674758502.757916; 1674758502.757134; 23; 9; 89054; 10000
LOAD; 2023-01-26-12:41:42; 1674758502.967990; 1674758502.967210; 31; 123; 89054; 10000
LOAD; 2023-01-26-12:41:43; 1674758503.177944; 1674758503.177157; 198; 133; 89054; 10000
LOAD; 2023-01-26-12:41:43; 1674758503.387942; 1674758503.387161; 74; 21; 89054; 10000
LOAD; 2023-01-26-12:41:43; 1674758503.597945; 1674758503.597165; 28; 15; 89054; 10000
LOAD; 2023-01-26-12:41:43; 1674758503.808089; 1674758503.807306; 27; 11; 89054; 10000
LOAD; 2023-01-26-12:41:44; 1674758504.018192; 1674758504.017405; 29; 23; 89054; 10000
LOAD; 2023-01-26-12:41:44; 1674758504.228379; 1674758504.227584; 39; 23; 89054; 10000
LOAD; 2023-01-26-12:41:44; 1674758504.437897; 1674758504.437123; 51; 50; 89054; 10000
LOAD; 2023-01-26-12:41:44; 1674758504.647940; 1674758504.647161; 37; 23; 89054; 10000
LOAD; 2023-01-26-12:41:44; 1674758504.857920; 1674758504.857140; 24; 16; 89054; 10000
LOAD; 2023-01-26-12:41:45; 1674758505.067933; 1674758505.067153; 20; 33; 89054; 10000
LOAD; 2023-01-26-12:41:45; 1674758505.277977; 1674758505.277200; 46; 32; 89054; 10000
LOAD; 2023-01-26-12:41:45; 1674758505.487948; 1674758505.487166; 135; 94; 89054; 10000
LOAD; 2023-01-26-12:41:45; 1674758505.697970; 1674758505.697191; 57; 18; 89054; 10000
LOAD; 2023-01-26-12:41:45; 1674758505.907922; 1674758505.907140; 20; 9; 89054; 10000
LOAD; 2023-01-26-12:41:46; 1674758506.118551; 1674758506.117754; 24; 14; 89054; 10000
LOAD; 2023-01-26-12:41:46; 1674758506.328443; 1674758506.327660; 63; 51; 89054; 10000
LOAD; 2023-01-26-12:41:46; 1674758506.537931; 1674758506.537150; 28; 15; 89054; 10000
LOAD; 2023-01-26-12:41:46; 1674758506.747925; 1674758506.747150; 29; 9; 89054; 10000
LOAD; 2023-01-26-12:41:46; 1674758506.957907; 1674758506.957126; 71; 25; 89054; 10000
LOAD; 2023-01-26-12:41:47; 1674758507.167984; 1674758507.167199; 23; 15; 89054; 10000
LOAD; 2023-01-26-12:41:47; 1674758507.377829; 1674758507.377044; 44; 17; 89054; 10000
LOAD; 2023-01-26-12:41:47; 1674758507.587950; 1674758507.587172; 27; 9; 89054; 10000
LOAD; 2023-01-26-12:41:47; 1674758507.797905; 1674758507.797124; 29; 13; 89054; 10000
LOAD; 2023-01-26-12:41:48; 1674758508.007911; 1674758508.007132; 24; 213; 89054; 10000
LOAD; 2023-01-26-12:41:48; 1674758508.218006; 1674758508.217219; 132; 9; 89054; 10000
LOAD; 2023-01-26-12:41:48; 1674758508.428172; 1674758508.427395; 70; 17; 89054; 10000
LOAD; 2023-01-26-12:41:48; 1674758508.637952; 1674758508.637172; 28; 15; 89054; 10000
LOAD; 2023-01-26-12:41:48; 1674758508.847946; 1674758508.847164; 10; 4; 89054; 10000
LOAD; 2023-01-26-12:41:49; 1674758509.057910; 1674758509.057128; 14; 6; 89054; 10000
LOAD; 2023-01-26-12:41:49; 1674758509.267886; 1674758509.267107; 19; 102; 89054; 10000
LOAD; 2023-01-26-12:41:49; 1674758509.477972; 1674758509.477181; 137; 18; 89054; 10000
LOAD; 2023-01-26-12:41:49; 1674758509.687916; 1674758509.687139; 49; 9; 89054; 10000
LOAD; 2023-01-26-12:41:49; 1674758509.897904; 1674758509.897125; 22; 16; 89054; 10000
LOAD; 2023-01-26-12:41:50; 1674758510.107948; 1674758510.107158; 25; 19; 89054; 10000
LOAD; 2023-01-26-12:41:50; 1674758510.317989; 1674758510.317209; 20; 9; 89054; 10000
LOAD; 2023-01-26-12:41:50; 1674758510.527946; 1674758510.527162; 38; 19; 89054; 10000
LOAD; 2023-01-26-12:41:50; 1674758510.737956; 1674758510.737173; 31; 17; 89054; 10000
LOAD; 2023-01-26-12:41:50; 1674758510.947909; 1674758510.947125; 41; 20; 89054; 10000
LOAD; 2023-01-26-12:41:51; 1674758511.157937; 1674758511.157156; 34; 48; 89054; 10000
LOAD; 2023-01-26-12:41:51; 1674758511.367914; 1674758511.367138; 29; 11; 89054; 10000
LOAD; 2023-01-26-12:41:51; 1674758511.577954; 1674758511.577172; 35; 20; 89054; 10000
LOAD; 2023-01-26-12:41:51; 1674758511.787960; 1674758511.787172; 26; 11; 89054; 10000
LOAD; 2023-01-26-12:41:51; 1674758511.997974; 1674758511.997190; 30; 49; 89054; 10000
LOAD; 2023-01-26-12:41:52; 1674758512.207964; 1674758512.207194; 17; 6; 89054; 10000
LOAD; 2023-01-26-12:41:52; 1674758512.417936; 1674758512.417159; 21; 10; 89054; 10000
LOAD; 2023-01-26-12:41:52; 1674758512.627968; 1674758512.627191; 23; 9; 89054; 10000
LOAD; 2023-01-26-12:41:52; 1674758512.838240; 1674758512.837385; 28; 31; 89054; 10000
LOAD; 2023-01-26-12:41:53; 1674758513.047900; 1674758513.047124; 115; 197; 89054; 10000
LOAD; 2023-01-26-12:41:53; 1674758513.257988; 1674758513.257208; 141; 15; 89054; 10000
LOAD; 2023-01-26-12:41:53; 1674758513.467939; 1674758513.467162; 35; 15; 89054; 10000
LOAD; 2023-01-26-12:41:53; 1674758513.678408; 1674758513.677622; 33; 19; 89054; 10000
LOAD; 2023-01-26-12:41:53; 1674758513.887952; 1674758513.887173; 22; 11; 89054; 10000
LOAD; 2023-01-26-12:41:54; 1674758514.098259; 1674758514.097469; 18; 9; 89054; 10000
LOAD; 2023-01-26-12:41:54; 1674758514.308004; 1674758514.307223; 27; 97; 89054; 10000
LOAD; 2023-01-26-12:41:54; 1674758514.517994; 1674758514.517214; 116; 23; 89054; 10000
LOAD; 2023-01-26-12:41:54; 1674758514.727971; 1674758514.727191; 111; 111; 89054; 10000
LOAD; 2023-01-26-12:41:54; 1674758514.938174; 1674758514.937406; 37; 16; 89054; 10000
LOAD; 2023-01-26-12:41:55; 1674758515.148050; 1674758515.147274; 37; 22; 89054; 10000
LOAD; 2023-01-26-12:41:55; 1674758515.357951; 1674758515.357169; 37; 20; 89054; 10000
LOAD; 2023-01-26-12:41:55; 1674758515.567947; 1674758515.567167; 37; 15; 89054; 10000
LOAD; 2023-01-26-12:41:55; 1674758515.777911; 1674758515.777140; 20; 20; 89054; 10000
LOAD; 2023-01-26-12:41:55; 1674758515.988354; 1674758515.987565; 35; 16; 89054; 10000
LOAD; 2023-01-26-12:41:56; 1674758516.197897; 1674758516.197117; 26; 23; 89054; 10000
LOAD; 2023-01-26-12:41:56; 1674758516.408002; 1674758516.407231; 28; 9; 89054; 10000
LOAD; 2023-01-26-12:41:56; 1674758516.617959; 1674758516.617175; 29; 19; 89054; 10000
LOAD; 2023-01-26-12:41:56; 1674758516.827954; 1674758516.827174; 31; 10; 89054; 10000
LOAD; 2023-01-26-12:41:57; 1674758517.037943; 1674758517.037159; 40; 38; 89054; 10000
LOAD; 2023-01-26-12:41:57; 1674758517.248008; 1674758517.247225; 66; 38; 89054; 10000
LOAD; 2023-01-26-12:41:57; 1674758517.457968; 1674758517.457187; 45; 19; 89054; 10000
LOAD; 2023-01-26-12:41:57; 1674758517.667949; 1674758517.667166; 32; 18; 89054; 10000
LOAD; 2023-01-26-12:41:57; 1674758517.877973; 1674758517.877183; 100; 22; 89054; 10000
LOAD; 2023-01-26-12:41:58; 1674758518.087956; 1674758518.087177; 47; 102; 89054; 10000
LOAD; 2023-01-26-12:41:58; 1674758518.297989; 1674758518.297211; 82; 9; 89054; 10000
LOAD; 2023-01-26-12:41:58; 1674758518.507952; 1674758518.507178; 30; 9; 89054; 10000
LOAD; 2023-01-26-12:41:58; 1674758518.717960; 1674758518.717181; 20; 10; 89054; 10000
LOAD; 2023-01-26-12:41:58; 1674758518.928306; 1674758518.927501; 58; 24; 89054; 10000
LOAD; 2023-01-26-12:41:59; 1674758519.137933; 1674758519.137153; 41; 16; 89054; 10000
LOAD; 2023-01-26-12:41:59; 1674758519.348733; 1674758519.347911; 76; 154; 89054; 10000
LOAD; 2023-01-26-12:41:59; 1674758519.557950; 1674758519.557172; 114; 22; 89054; 10000
LOAD; 2023-01-26-12:41:59; 1674758519.767966; 1674758519.767188; 25; 34; 89054; 10000
LOAD; 2023-01-26-12:41:59; 1674758519.977955; 1674758519.977175; 234; 32; 89054; 10000
LOAD; 2023-01-26-12:42:00; 1674758520.188381; 1674758520.187591; 35; 30; 89054; 10000
LOAD; 2023-01-26-12:42:00; 1674758520.397996; 1674758520.397214; 52; 43; 89054; 10000
LOAD; 2023-01-26-12:42:00; 1674758520.607947; 1674758520.607172; 62; 37; 89054; 10000
LOAD; 2023-01-26-12:42:00; 1674758520.817960; 1674758520.817181; 34; 19; 89054; 10000
LOAD; 2023-01-26-12:42:01; 1674758521.028170; 1674758521.027395; 33; 26; 89054; 10000
LOAD; 2023-01-26-12:42:01; 1674758521.238294; 1674758521.237502; 31; 17; 89054; 10000
LOAD; 2023-01-26-12:42:01; 1674758521.448033; 1674758521.447245; 44; 19; 89054; 10000
LOAD; 2023-01-26-12:42:01; 1674758521.657937; 1674758521.657159; 25; 15; 89054; 10000
LOAD; 2023-01-26-12:42:01; 1674758521.867982; 1674758521.867206; 52; 28; 89054; 10000
LOAD; 2023-01-26-12:42:02; 1674758522.077975; 1674758522.077192; 32; 26; 89054; 10000
LOAD; 2023-01-26-12:42:02; 1674758522.287983; 1674758522.287201; 32; 16; 89054; 10000
LOAD; 2023-01-26-12:42:02; 1674758522.497941; 1674758522.497155; 34; 22; 89054; 10000
LOAD; 2023-01-26-12:42:02; 1674758522.707968; 1674758522.707177; 25; 17; 89054; 10000
LOAD; 2023-01-26-12:42:02; 1674758522.917975; 1674758522.917202; 21; 15; 89054; 10000
LOAD; 2023-01-26-12:42:03; 1674758523.127957; 1674758523.127181; 111; 280; 89054; 10000
LOAD; 2023-01-26-12:42:03; 1674758523.338209; 1674758523.337294; 120; 31; 89054; 10000
LOAD; 2023-01-26-12:42:03; 1674758523.547937; 1674758523.547161; 39; 20; 89054; 10000
LOAD; 2023-01-26-12:42:03; 1674758523.757923; 1674758523.757146; 15; 7; 89054; 10000
LOAD; 2023-01-26-12:42:03; 1674758523.967970; 1674758523.967187; 41; 35; 89054; 10000
LOAD; 2023-01-26-12:42:04; 1674758524.177965; 1674758524.177185; 22; 9; 89054; 10000
LOAD; 2023-01-26-12:42:04; 1674758524.388006; 1674758524.387222; 56; 104; 89054; 10000
LOAD; 2023-01-26-12:42:04; 1674758524.597958; 1674758524.597180; 96; 6; 89054; 10000
LOAD; 2023-01-26-12:42:04; 1674758524.807945; 1674758524.807173; 31; 12; 89054; 10000
LOAD; 2023-01-26-12:42:05; 1674758525.018128; 1674758525.017346; 87; 42; 89054; 10000
LOAD; 2023-01-26-12:42:05; 1674758525.227967; 1674758525.227182; 56; 31; 89054; 10000
LOAD; 2023-01-26-12:42:05; 1674758525.438320; 1674758525.437533; 31; 19; 89054; 10000
LOAD; 2023-01-26-12:42:05; 1674758525.648293; 1674758525.647487; 27; 9; 89054; 10000
LOAD; 2023-01-26-12:42:05; 1674758525.858263; 1674758525.857482; 97; 62; 89054; 10000
LOAD; 2023-01-26-12:42:06; 1674758526.067958; 1674758526.067179; 61; 18; 89054; 10000
LOAD; 2023-01-26-12:42:06; 1674758526.277937; 1674758526.277157; 24; 9; 89054; 10000
LOAD; 2023-01-26-12:42:06; 1674758526.487849; 1674758526.487074; 29; 9; 89054; 10000
LOAD; 2023-01-26-12:42:06; 1674758526.697911; 1674758526.697133; 14; 7; 89054; 10000
LOAD; 2023-01-26-12:42:06; 1674758526.907925; 1674758526.907142; 24; 9; 89054; 10000
LOAD; 2023-01-26-12:42:07; 1674758527.117969; 1674758527.117189; 31; 25; 89054; 10000
LOAD; 2023-01-26-12:42:07; 1674758527.327966; 1674758527.327183; 163; 35; 89054; 10000
LOAD; 2023-01-26-12:42:07; 1674758527.537963; 1674758527.537184; 22; 11; 89054; 10000
LOAD; 2023-01-26-12:42:07; 1674758527.748338; 1674758527.747554; 14; 4; 89054; 10000
LOAD; 2023-01-26-12:42:07; 1674758527.957864; 1674758527.957085; 35; 25; 89054; 10000
LOAD; 2023-01-26-12:42:08; 1674758528.167947; 1674758528.167168; 144; 113; 89054; 10000
LOAD; 2023-01-26-12:42:08; 1674758528.377982; 1674758528.377200; 37; 14; 89054; 10000
LOAD; 2023-01-26-12:42:08; 1674758528.587922; 1674758528.587145; 33; 28; 89054; 10000
LOAD; 2023-01-26-12:42:08; 1674758528.797958; 1674758528.797178; 18; 4; 89054; 10000
LOAD; 2023-01-26-12:42:09; 1674758529.008796; 1674758529.008001; 18; 15; 89054; 10000
LOAD; 2023-01-26-12:42:09; 1674758529.217969; 1674758529.217185; 47; 40; 89054; 10000
LOAD; 2023-01-26-12:42:09; 1674758529.428053; 1674758529.427269; 63; 91; 89054; 10000
LOAD; 2023-01-26-12:42:09; 1674758529.637932; 1674758529.637152; 70; 34; 89054; 10000
LOAD; 2023-01-26-12:42:09; 1674758529.848276; 1674758529.847496; 27; 25; 89054; 10000
LOAD; 2023-01-26-12:42:10; 1674758530.057954; 1674758530.057176; 91; 39; 89054; 10000
LOAD; 2023-01-26-12:42:10; 1674758530.268322; 1674758530.267526; 19; 21; 89054; 10000
LOAD; 2023-01-26-12:42:10; 1674758530.477830; 1674758530.477058; 154; 34; 89054; 10000
LOAD; 2023-01-26-12:42:10; 1674758530.687974; 1674758530.687181; 51; 27; 89054; 10000
LOAD; 2023-01-26-12:42:10; 1674758530.898189; 1674758530.897400; 69; 25; 89054; 10000
LOAD; 2023-01-26-12:42:11; 1674758531.108123; 1674758531.107338; 69; 30; 89054; 10000
LOAD; 2023-01-26-12:42:11; 1674758531.318135; 1674758531.317349; 247; 584; 89054; 10000
LOAD; 2023-01-26-12:42:11; 1674758531.528454; 1674758531.527660; 350; 341; 89054; 10000
LOAD; 2023-01-26-12:42:11; 1674758531.738026; 1674758531.737249; 220; 228; 89054; 10000
LOAD; 2023-01-26-12:42:11; 1674758531.947948; 1674758531.947172; 78; 43; 89054; 10000
LOAD; 2023-01-26-12:42:12; 1674758532.157939; 1674758532.157155; 34; 14; 89054; 10000
LOAD; 2023-01-26-12:42:12; 1674758532.368130; 1674758532.367341; 28; 47; 89054; 10000
LOAD; 2023-01-26-12:42:12; 1674758532.578347; 1674758532.577557; 48; 47; 89054; 10000
LOAD; 2023-01-26-12:42:12; 1674758532.787958; 1674758532.787180; 47; 14; 89054; 10000
LOAD; 2023-01-26-12:42:12; 1674758532.997982; 1674758532.997197; 34; 48; 89054; 10000
LOAD; 2023-01-26-12:42:13; 1674758533.207974; 1674758533.207196; 63; 58; 89054; 10000
OK I just ran:
echo "anything" > /tmp/run/cake-autorate/primary/pinger_0_fifo
and nothing happened. I did a speed test but still just LOAD; lines in the log, also confirmed with "tc -s qdisc" that nothing is changing.
No, it would need gdb and bash compiled with debug symbols. And a much easier solution exists: set -x. Except that it will fill the logs quickly.
Can you repeat that for the 25275 process?
Also is fping running and can you post output of:
ls /var/run/cake-autorate/primary/
Assuming 'primary' is your identifier?
root@OpenWrt:~/cake-autorate# strace -y -s 1000 -p 25275
strace: Process 25275 attached
strace: Exit of unknown pid 21396 ignored
wait4(-1, ^Cstrace: Process 25275 detached
<detached ...>
root@OpenWrt:~/cake-autorate# ps | grep fping
17509 root 2060 S fping --timestamp --loop --period 300 --interval 50 --timeout 10000 9.9.9.10 8.8.8.8 9.9.9.9 208.67.220.2 208.67.220.220 94.140.14.15
root@OpenWrt:~/cake-autorate# ls /var/run/cake-autorate/primary/
compensated_dl_delay_thr_us reflector_156-154-70-3_delta_ewma_us reflector_156-154-71-5_dl_owd_delta_ewma_us reflector_208-67-220-220_ul_owd_delta_ewma_us reflector_9-9-9-11_delta_ewma_us
compensated_ul_delay_thr_us reflector_156-154-70-3_dl_owd_baseline_us reflector_156-154-71-5_last_timestamp_us reflector_208-67-220-2_baseline_us reflector_9-9-9-11_last_timestamp_us
dl_achieved_rate_kbps reflector_156-154-70-3_dl_owd_delta_ewma_us reflector_156-154-71-5_ul_owd_baseline_us reflector_208-67-220-2_delta_ewma_us reflector_9-9-9-9_baseline_us
dl_load_percent reflector_156-154-70-3_last_timestamp_us reflector_156-154-71-5_ul_owd_delta_ewma_us reflector_208-67-220-2_last_timestamp_us reflector_9-9-9-9_delta_ewma_us
dl_shaper_rate_kbps reflector_156-154-70-3_ul_owd_baseline_us reflector_185-228-168-10_baseline_us reflector_208-67-222-123_baseline_us reflector_9-9-9-9_last_timestamp_us
log_fifo reflector_156-154-70-3_ul_owd_delta_ewma_us reflector_185-228-168-10_delta_ewma_us reflector_208-67-222-123_delta_ewma_us reflector_94-140-14-140_baseline_us
maintain_log_file_pid reflector_156-154-70-4_baseline_us reflector_185-228-168-10_dl_owd_baseline_us reflector_208-67-222-123_dl_owd_baseline_us reflector_94-140-14-140_delta_ewma_us
max_wire_packet_rtt_us reflector_156-154-70-4_delta_ewma_us reflector_185-228-168-10_dl_owd_delta_ewma_us reflector_208-67-222-123_dl_owd_delta_ewma_us reflector_94-140-14-140_last_timestamp_us
ping_fifo reflector_156-154-70-4_last_timestamp_us reflector_185-228-168-10_last_timestamp_us reflector_208-67-222-123_last_timestamp_us reflector_94-140-14-141_baseline_us
pinger_0_fifo reflector_156-154-70-5_baseline_us reflector_185-228-168-10_ul_owd_baseline_us reflector_208-67-222-123_ul_owd_baseline_us reflector_94-140-14-141_delta_ewma_us
reflector_1-0-0-1_baseline_us reflector_156-154-70-5_delta_ewma_us reflector_185-228-168-10_ul_owd_delta_ewma_us reflector_208-67-222-123_ul_owd_delta_ewma_us reflector_94-140-14-141_dl_owd_baseline_us
reflector_1-0-0-1_delta_ewma_us reflector_156-154-70-5_last_timestamp_us reflector_185-228-168-9_baseline_us reflector_208-67-222-2_baseline_us reflector_94-140-14-141_dl_owd_delta_ewma_us
reflector_1-0-0-1_last_timestamp_us reflector_156-154-71-1_baseline_us reflector_185-228-168-9_delta_ewma_us reflector_208-67-222-2_delta_ewma_us reflector_94-140-14-141_last_timestamp_us
reflector_1-1-1-1_baseline_us reflector_156-154-71-1_delta_ewma_us reflector_185-228-168-9_dl_owd_baseline_us reflector_208-67-222-2_last_timestamp_us reflector_94-140-14-141_ul_owd_baseline_us
reflector_1-1-1-1_delta_ewma_us reflector_156-154-71-1_last_timestamp_us reflector_185-228-168-9_dl_owd_delta_ewma_us reflector_64-6-65-6_baseline_us reflector_94-140-14-141_ul_owd_delta_ewma_us
reflector_1-1-1-1_dl_owd_baseline_us reflector_156-154-71-2_baseline_us reflector_185-228-168-9_last_timestamp_us reflector_64-6-65-6_delta_ewma_us reflector_94-140-14-15_baseline_us
reflector_1-1-1-1_dl_owd_delta_ewma_us reflector_156-154-71-2_delta_ewma_us reflector_185-228-168-9_ul_owd_baseline_us reflector_64-6-65-6_last_timestamp_us reflector_94-140-14-15_delta_ewma_us
reflector_1-1-1-1_last_timestamp_us reflector_156-154-71-2_last_timestamp_us reflector_185-228-168-9_ul_owd_delta_ewma_us reflector_8-8-4-4_baseline_us reflector_94-140-14-15_last_timestamp_us
reflector_1-1-1-1_ul_owd_baseline_us reflector_156-154-71-3_baseline_us reflector_208-67-220-123_baseline_us reflector_8-8-4-4_delta_ewma_us reflector_94-140-15-15_baseline_us
reflector_1-1-1-1_ul_owd_delta_ewma_us reflector_156-154-71-3_delta_ewma_us reflector_208-67-220-123_delta_ewma_us reflector_8-8-4-4_dl_owd_baseline_us reflector_94-140-15-15_delta_ewma_us
reflector_156-154-70-1_baseline_us reflector_156-154-71-3_last_timestamp_us reflector_208-67-220-123_dl_owd_baseline_us reflector_8-8-4-4_dl_owd_delta_ewma_us reflector_94-140-15-15_last_timestamp_us
reflector_156-154-70-1_delta_ewma_us reflector_156-154-71-4_baseline_us reflector_208-67-220-123_dl_owd_delta_ewma_us reflector_8-8-4-4_last_timestamp_us reflector_94-140-15-16_baseline_us
reflector_156-154-70-1_last_timestamp_us reflector_156-154-71-4_delta_ewma_us reflector_208-67-220-123_last_timestamp_us reflector_8-8-4-4_ul_owd_baseline_us reflector_94-140-15-16_delta_ewma_us
reflector_156-154-70-2_baseline_us reflector_156-154-71-4_dl_owd_baseline_us reflector_208-67-220-123_ul_owd_baseline_us reflector_8-8-4-4_ul_owd_delta_ewma_us reflector_94-140-15-16_last_timestamp_us
reflector_156-154-70-2_delta_ewma_us reflector_156-154-71-4_dl_owd_delta_ewma_us reflector_208-67-220-123_ul_owd_delta_ewma_us reflector_8-8-8-8_baseline_us reflectors_last_timestamp_us
reflector_156-154-70-2_dl_owd_baseline_us reflector_156-154-71-4_last_timestamp_us reflector_208-67-220-220_baseline_us reflector_8-8-8-8_delta_ewma_us sleep_fifo
reflector_156-154-70-2_dl_owd_delta_ewma_us reflector_156-154-71-4_ul_owd_baseline_us reflector_208-67-220-220_delta_ewma_us reflector_8-8-8-8_last_timestamp_us ul_achieved_rate_kbps
reflector_156-154-70-2_last_timestamp_us reflector_156-154-71-4_ul_owd_delta_ewma_us reflector_208-67-220-220_dl_owd_baseline_us reflector_9-9-9-10_baseline_us ul_load_percent
reflector_156-154-70-2_ul_owd_baseline_us reflector_156-154-71-5_baseline_us reflector_208-67-220-220_dl_owd_delta_ewma_us reflector_9-9-9-10_delta_ewma_us ul_shaper_rate_kbps
reflector_156-154-70-2_ul_owd_delta_ewma_us reflector_156-154-71-5_delta_ewma_us reflector_208-67-220-220_last_timestamp_us reflector_9-9-9-10_last_timestamp_us
reflector_156-154-70-3_baseline_us reflector_156-154-71-5_dl_owd_baseline_us reflector_208-67-220-220_ul_owd_baseline_us reflector_9-9-9-11_baseline_us
Here's with the times in that directory, I can see some haven't been updated in over 13 hours, I'm guessing that's when it stopped working right:
root@OpenWrt:~/cake-autorate# ls -l /var/run/cake-autorate/primary/
-rw-r--r-- 1 root root 5 Jan 25 23:27 compensated_dl_delay_thr_us
-rw-r--r-- 1 root root 5 Jan 25 23:27 compensated_ul_delay_thr_us
-rw-r--r-- 1 root root 3 Jan 26 13:25 dl_achieved_rate_kbps
-rw-r--r-- 1 root root 1 Jan 25 23:27 dl_load_percent
-rw-r--r-- 1 root root 5 Jan 25 23:27 dl_shaper_rate_kbps
prw-r--r-- 1 root root 0 Jan 26 13:25 log_fifo
-rw-r--r-- 1 root root 5 Jan 25 10:05 maintain_log_file_pid
-rw-r--r-- 1 root root 4 Jan 25 23:27 max_wire_packet_rtt_us
prw-r--r-- 1 root root 0 Jan 25 23:28 ping_fifo
prw-r--r-- 1 root root 0 Jan 26 12:45 pinger_0_fifo
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_1-0-0-1_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_1-0-0-1_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_1-0-0-1_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_1-1-1-1_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_1-1-1-1_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_1-1-1-1_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_1-1-1-1_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_1-1-1-1_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_1-1-1-1_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_1-1-1-1_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-70-1_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-70-1_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-70-1_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-70-2_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-70-2_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-2_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_156-154-70-2_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_156-154-70-2_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-2_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_156-154-70-2_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-70-3_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-70-3_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-3_dl_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-3_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_156-154-70-3_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-3_ul_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_156-154-70-3_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-70-4_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-70-4_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-70-4_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-70-5_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-70-5_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-70-5_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-71-1_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-71-1_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-71-1_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-71-2_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-71-2_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-71-2_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_156-154-71-3_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_156-154-71-3_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-71-3_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-71-4_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:37 reflector_156-154-71-4_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 21:42 reflector_156-154-71-4_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:42 reflector_156-154-71-4_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_156-154-71-4_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 21:42 reflector_156-154-71-4_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:42 reflector_156-154-71-4_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-71-5_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_156-154-71-5_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:27 reflector_156-154-71-5_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:27 reflector_156-154-71-5_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:27 reflector_156-154-71-5_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:27 reflector_156-154-71-5_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:27 reflector_156-154-71-5_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_185-228-168-10_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_185-228-168-10_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_185-228-168-10_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_185-228-168-10_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_185-228-168-10_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_185-228-168-10_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_185-228-168-10_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_185-228-168-9_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_185-228-168-9_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_185-228-168-9_dl_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_185-228-168-9_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_185-228-168-9_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_185-228-168-9_ul_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_185-228-168-9_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-220-123_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-220-123_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_208-67-220-123_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_208-67-220-123_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_208-67-220-123_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_208-67-220-123_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_208-67-220-123_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-220-220_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-220-220_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_208-67-220-220_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_208-67-220-220_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_208-67-220-220_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 21:43 reflector_208-67-220-220_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 21:43 reflector_208-67-220-220_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_208-67-220-2_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_208-67-220-2_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_208-67-220-2_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-222-123_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_208-67-222-123_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_208-67-222-123_dl_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_208-67-222-123_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_208-67-222-123_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_208-67-222-123_ul_owd_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:28 reflector_208-67-222-123_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_208-67-222-2_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_208-67-222-2_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_208-67-222-2_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_64-6-65-6_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_64-6-65-6_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_64-6-65-6_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_8-8-4-4_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_8-8-4-4_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_8-8-4-4_dl_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_8-8-4-4_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflector_8-8-4-4_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_8-8-4-4_ul_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:28 reflector_8-8-4-4_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_8-8-8-8_baseline_us
-rw-r--r-- 1 root root 4 Jan 25 23:37 reflector_8-8-8-8_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_8-8-8-8_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_9-9-9-10_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_9-9-9-10_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_9-9-9-10_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_9-9-9-11_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_9-9-9-11_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_9-9-9-11_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_9-9-9-9_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_9-9-9-9_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_9-9-9-9_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-14-140_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-14-140_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_94-140-14-140_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-14-141_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-14-141_delta_ewma_us
-rw-r--r-- 1 root root 5 Jan 25 21:41 reflector_94-140-14-141_dl_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 21:41 reflector_94-140-14-141_dl_owd_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_94-140-14-141_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 21:41 reflector_94-140-14-141_ul_owd_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 21:41 reflector_94-140-14-141_ul_owd_delta_ewma_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_94-140-14-15_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_94-140-14-15_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_94-140-14-15_last_timestamp_us
-rw-r--r-- 1 root root 6 Jan 25 23:37 reflector_94-140-15-15_baseline_us
-rw-r--r-- 1 root root 1 Jan 25 23:37 reflector_94-140-15-15_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_94-140-15-15_last_timestamp_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-15-16_baseline_us
-rw-r--r-- 1 root root 5 Jan 25 23:37 reflector_94-140-15-16_delta_ewma_us
-rw-r--r-- 1 root root 16 Jan 25 23:26 reflector_94-140-15-16_last_timestamp_us
-rw-r--r-- 1 root root 16 Jan 25 23:28 reflectors_last_timestamp_us
prw-r--r-- 1 root root 0 Jan 25 10:05 sleep_fifo
-rw-r--r-- 1 root root 2 Jan 26 13:25 ul_achieved_rate_kbps
-rw-r--r-- 1 root root 1 Jan 25 23:27 ul_load_percent
-rw-r--r-- 1 root root 5 Jan 25 23:26 ul_shaper_rate_kbps
root@OpenWrt:~/cake-autorate# date
Thu Jan 26 13:25:49 CST 2023
set -x unfortunately can only be set from the script itself.
Ah fping still running @moeller0 and @patrakov so I think this means it's blocking on the wait for kill of fping process and not the monitor process? So the kill for fping presumably got blocked and wait runs indefinitely.
Not sure what other debugging steps to try other than killing the fping process to see if that gets everything unstuck. Failing which maybe killing the maintain pingers process too. But either of these steps should only be run if we've exhausted any other angles.
Could you retry while at the same time running
In my opinion, if this is the problematic code, then using an explicit termination request is leaps and bounds superior to just papering over the race with another write to the FIFO as that does not really solve the problem, but just modifies the time window the problem might trigger. But again, this interpretation requires this to be the problematic spot.
I am thinking now the problem is with killing fping.