For bash experts (pipe buffering)

For Developers
21

Am I right that in principle the fifo blocking could be relied upon such that ping is blocked whilst fifo is not being read?

A named pipe (also called a named FIFO, or just FIFO) is a pipe whose access point is a file kept on the file system. By opening this file for reading, a process gets access to the reading end of the pipe. By opening the file for writing, the process gets access to the writing end of the pipe. If a process opens the file for reading, it is blocked until another process opens the file for writing. The same goes the other way around.

http://www.cs.kent.edu/~ruttan/sysprog/lectures/shmem/pipes.html

That seems to me promising. Like I set up ping -> fifo, then the ping write is blocked until the fifo read/processing is finished. Something along these lines? Or I am fundamentally misunderstanding things?

Someone also suggested using 'read -t 0.X' to determine that previous read was the latest.

22

Size is a major issue, but so is system performance - bash takes a long time to start up. A while ago I was using Debian on a SUN SPARCstation 20 with a single 50 MHz SuperSPARC CPU, bash took more than 20-30s to start up (login, getty, xterm, etc.), while busybox ash was available pretty much instantly.

23

I'm as much a "non expert" as you. All I can contribute from here is that I tired several ways (excluding the ftee example which i did not try) and could only achieve the objective by "emptying" the pipe.

Note, it helped me to think of the named pipe as a file descriptor similar to stdout and stderr.

If I was in your situation, I'd both investigate @moeller0's suggestions about a single shot ping and/or adapt my example to your case and check the cpu cost. If it's too much, likely you will have to go to a "compiled" solution regardless.

24

This seems close:

#!/bin/bash

ping -D -i 0.1 1.1.1.1 | while read ping_line
do
        if [ -z $(read -t 0.01) ]; then
                echo $last_line
                sleep 1
        fi
        last_line=$ping_line
done

So the idea there is to keep reading until read times out, and then the last read should be the latest sample. This isn't quite behaving as expected, but I wonder if this might work?

But I still favour the fifo based approach if that can be made to work.

If anyone has any ideas please chime in!

In the meantime, here is for hoping:

25

This may move the needle somewhat:

#!/bin/bash

pipe=/tmp/test
mkfifo $pipe

ping -i 0.1 1.1.1.1 > $pipe&

cat $pipe > /dev/null&

while true
do
        head -1 $pipe
        sleep 1
done

root@OpenWrt:~/CAKE-autorate# ./test.sh
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=12 ttl=54 time=48.7 ms
64 bytes from 1.1.1.1: icmp_seq=23 ttl=54 time=48.7 ms
64 bytes from 1.1.1.1: icmp_seq=34 ttl=54 time=48.7 ms
64 bytes from 1.1.1.1: icmp_seq=45 ttl=54 time=45.7 ms

So then I'd just sleep in the while loop for the ping interval.

I don't completely understand this. Can it be relied upon to always work or is this dangerous?

Alternative formulation:

#!/bin/bash

pipe=/tmp/test
mkfifo $pipe

ping -i 0.1 1.1.1.1 > $pipe&

cat $pipe > /dev/null&

cat $pipe | while read line
do
        echo $line
done

This seems like it may offer a candidate because I can put sleeps inside to simulate processing delays and it works.

26

The question is, does the head "consume" the data in the pipe (reducing the memory load of the pipe) or does it just read it?

But I really want to understand how/where the simple write single-shot to file, parse from file approach goes "mental" hogging too much CPU cycles, because it solves almost all concurrency issues between producer (ping) and consumer (shell) and should not be worse than having the ping binary constantly resident in memory...

27

It's just when I tried that I struggled to work with sample frequency of 10Hz on my RT3200. It ended up being more like 3-4Hz with the overhead of calling ping or shell interaction. And on my LTE I have found a reliable 10Hz is important. With ping just running in the background I get really nice and smooth lines to work with, albeit the challenge is how to handle the lines.

28

Assuming, I correctly understand the functionality, you want to achieve, I would use 2 processes:

  • p1) do single pings in endless loop, flock, write to /tmp/ping.out, ulock, sleep 0.1
  • p2) in endless loop, flock, read /tmp/ping.out, ulock, process data read

Using named pipes (FIFO, queue) has the disadvantage, you need to get rid of 'redundant' lines. Only possible by sequential read, as there is no way to read specific entry from queue. You can not even determine number of entries in queue.

Just my few cents.

29

I think the consumption is achieved with the cat to /dev/null. That is why head in 1st example or echo in 2nd example keeps showing new values.

@moeller0 what do you make of this:

With:

#!/bin/bash

while true
do
        ping -D -c 1 1.1.1.1
        sleep 0.1
done
~

root@OpenWrt:~/CAKE-autorate# ./test.sh
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760047.759557] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=56.3 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 56.296/56.296/56.296/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760047.914019] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=46.3 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 46.310/46.310/46.310/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.073018] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=50.5 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 50.492/50.492/50.492/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.234065] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=53.0 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 52.970/52.970/52.970/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.397008] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=55.2 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 55.199/55.199/55.199/0.000 ms
^C

Or here is with 0.05 sleep:

root@OpenWrt:~/CAKE-autorate# ./test.sh
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760047.759557] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=56.3 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 56.296/56.296/56.296/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760047.914019] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=46.3 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 46.310/46.310/46.310/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.073018] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=50.5 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 50.492/50.492/50.492/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.234065] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=53.0 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 52.970/52.970/52.970/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760048.397008] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=55.2 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 55.199/55.199/55.199/0.000 ms
^C
root@OpenWrt:~/CAKE-autorate# ^C
root@OpenWrt:~/CAKE-autorate# vi test.sh
root@OpenWrt:~/CAKE-autorate# ./test.sh
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760182.527140] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=62.3 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.291/62.291/62.291/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760182.649151] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=63.6 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 63.590/63.590/63.590/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760182.769452] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=62.7 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 62.749/62.749/62.749/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760182.889985] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=63.1 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 63.124/63.124/63.124/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760183.009447] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=61.2 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 61.191/61.191/61.191/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760183.126431] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=58.0 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 58.030/58.030/58.030/0.000 ms
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
[1646760183.258458] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=72.1 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 72.063/72.063/72.063/0.000 ms
^C

How important is it to have even spacing? I guess we really want the most up to date samples we can achieve. Won't doing this at 10Hz for 4 reflectors result in massive CPU overhead given 40Hz calls of ping and passing back and forth to bash.

I can try again if you think worthwhile.

For comparison purposes, here is the output from a fifo approach:

#!/bin/bash

pipe=/tmp/test
mkfifo $pipe

ping -D -i 0.1 1.1.1.1 > $pipe&

cat $pipe > /dev/null&

while true
do
        head -1 $pipe
done

root@OpenWrt:~/CAKE-autorate# ./test.sh
[1646761665.736721] 64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=44.6 ms
[1646761665.950656] 64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=56.1 ms
[1646761666.139707] 64 bytes from 1.1.1.1: icmp_seq=5 ttl=54 time=45.2 ms
[1646761666.354629] 64 bytes from 1.1.1.1: icmp_seq=7 ttl=54 time=58.3 ms
[1646761666.556416] 64 bytes from 1.1.1.1: icmp_seq=9 ttl=54 time=58.1 ms
[1646761666.750159] 64 bytes from 1.1.1.1: icmp_seq=11 ttl=54 time=50.2 ms
[1646761666.964655] 64 bytes from 1.1.1.1: icmp_seq=13 ttl=54 time=63.3 ms
[1646761667.152936] 64 bytes from 1.1.1.1: icmp_seq=15 ttl=54 time=52.8 ms
[1646761667.354276] 64 bytes from 1.1.1.1: icmp_seq=17 ttl=54 time=53.1 ms
[1646761667.565643] 64 bytes from 1.1.1.1: icmp_seq=19 ttl=54 time=63.3 ms
[1646761667.768973] 64 bytes from 1.1.1.1: icmp_seq=21 ttl=54 time=65.3 ms
[1646761667.951002] 64 bytes from 1.1.1.1: icmp_seq=23 ttl=54 time=45.7 ms
[1646761668.180158] 64 bytes from 1.1.1.1: icmp_seq=25 ttl=54 time=73.9 ms
[1646761668.373990] 64 bytes from 1.1.1.1: icmp_seq=27 ttl=54 time=66.7 ms
[1646761668.555103] 64 bytes from 1.1.1.1: icmp_seq=29 ttl=54 time=46.1 ms
[1646761668.757098] 64 bytes from 1.1.1.1: icmp_seq=31 ttl=54 time=46.8 ms
30

Well, if you do this for four reflectors you only need to run each at 10/4 2.5 Hz to get an effective 10 Hz delay measurement....

31

Looking at the timestamps both are similarly bad... in the first version however that can be corrected for by not sleeping for a fixed duration but calculating the sleep duration as a function of the last RTT...

I guess the bigger question is how does this look under load, is calling and exiting ping too costly or not?

1 Like
32 
#!/bin/bash

pipe=/tmp/test
mkfifo $pipe

ping -D -i 0.1 1.1.1.1 > $pipe&

cat $pipe > /dev/null&

while true
do
        head -1 $pipe
done

If my understanding is correct, with the above fifo approach what happens is that the background ping writes to fifo buffer and the background cat writes the fifo buffer to /dev/null.

So I think a stream of fish is set up flowing from ping to /dev/null.

As a fisherman, I can pluck individual fish out from this stream with 'head -1'.

These are up to date because the stream only has one fish at a time.

It seems to work.

In a simple test bash script if I sleep enough to get roughly 0.1s interval with single shot the cpu usage jumps to 3%. And that's just for one reflector.

But with my background ping and fifo stream approach the cpu usage stays at 0%.

33

Yeah, the interesting test is when both approaches are doing the full thing, including the main control loop ;)...

34

I'm interested in your spacing idea about changing spacing based on RTT.

What is the optimum spacing between results I wonder?

35

Situations like this are where my new found "love hate" relationship with awk start. You can likely do all the ping output streaming, picking the result you need based on a RTT (i.e. non integer math and built in functions to work with date/times), no need for a named pipe for ping, etc if you wrote the code in awk. Not to mention you get access to hash like data structures in awk, better array handling, and likely very fast execution time with minimal cpu load.

You might still need a named pipe to collect the output from awk in your bash script. But working through coding an awk program is where i usually decide, I don't really want it that much.

Good luck - it looks like you close.

1 Like
36

Thanks. My original approach was ash plus awk but it was messy to read and I struggled to work with it. That's still in my main branch.

Do you think my understanding about the fifo between ping and null offering a place to lift off individual lines is correct?

Here is the latest code:

The relevant part is in the monitor_reflector_path.sh script. Here:

And here:

On my RT3200 cpu usage is about 2% with four reflectors and 10Hz pings. The code certainly seems to work well compared to anything else I've tested at handling autorate for CAKE. I'm hopeful this new fifo stream approach will fix the cpu usage spike others were seeing with the previous 'tail -f' plus '> file' truncation approach.

37

Someone posted a perl based approach to the buffering problem here:

But in the comments he seemed to like my background cat to null idea for consuming data and n bash implementation.

38

I think it's close enough concept wise (I might express it slightly differently tho but that's not important).

You should test under load as output might become unpredictable when the system struggles for cpu cycles (or if the ping rate is too fast I think). *nix os's are not good at real time applications and likely will not behave in predicable fashion (under load) like a PLC might.

1 Like
39

You can't control that unfortunately, what you can do is space the ICMP transmissions to be roughly equidistant in time, and if the RTT is roughly stable the result spacing will also be. The idea would be to sleep for ${desired_interval} - ${current_RTT} or similar....

I have no idea about the optimum, but would guess somewhere between 0.5 to 0.1seconds should do....

40

OK great. Glad I'm not a million miles away since I'm new to bash and pipes. There's not so much out there on the subject. It seems a bit niche. Presumably because most people have moved on from bash to pastures greener by then. But I've been enjoying my excursions with bash.

Yes the stackoverflow gentleman indicates there is a race condition. Not sure whether I need to be concerned with that. So long as the 'head -1' picks up a fish without staring at the stream too long then it should be OK. But if the bear is transfixed we have a problem because we want to be as up to date as possible. Any thoughts on what governs race between cat and the head pickup?