Unix Shell - EOF on reading a pipe when there is a writer

Author

EOF on reading a pipe when there is a writer

Enrique Perez-Terron

2005-09-17, 2:48 am

Hello,

I need to capture a copy of the output of a script into a file,
and I try to use 'tee' for that:

bash my_script | tee -a logfile

Now it turns out that my_script (which is not really "my" script)
starts a process that does not terminate (and should not terminate)

Actually, I should rather start by the symptoms: The script that
contains the above pipeline never goes on afterwards. That is, I have
a script containing

bash my_script | tee -a logfile
echo "Hello World"

and it never outputs "Hello World". When I attach the process with
strace -p $the_pid, I get

waitpid(-1,... (unfinished)

and investigating I find that the 'tee' process is still around,
even my_script has terminated.

Then I discover that there is a process with the process group id of
the my_script process, but with PPID set to one.
This is really a daemon process of some kind, and probably should have
closed its standard output/error. As it is, it is keeping the writer
side of the pipe open, and so 'tee' does not terminate, and the shell
remains in waitpid().

To mitigate this, I try

bash my_script | (tee -a logfile &)
echo "Hello World"

What I want to achieve is that the shell waits until my_script has
finished, but allows the tee process go on and capture any output
coming later from the daemons started by my_script.

I thought it ought to work this way: The shell creates a pipe
and two subshells, a writer and a reader side. The shell then waits
for both subshells to finish. The writer side should work the normal
way, send its output into the pipe and finish. The reader side
forks yet another process, which execs "tee". At this point there
are two processes on the reader end of the pipe, the reader subshell
and the "tee" sub-sub-process. The reader subshell then terminates
without waiting for "tee", leaving "tee" as the sole reader of the
pipe. As long as there is still a reader process, the pipe should
remain open for the writer end to write into.

But this does not seem to work. To investigate this a little further,
I wrote a new myscript, with the following contents:

#!/bin/bash
echo Important

and a wrapper "q" with the contents

#!/bin/bash
myscript | {
tee -a logfile &
}
echo PIPESTATUS: ${PIPESTATUS[*]}
echo "Hello World"

and running the latter I get the output

PIPESTATUS: 141 0
Hello World

It shows that myscript receives a SIGPIPE meaning that there
is no reader on the pipe. (141 = 128 + 13, and 13 is the signal
number of SIGPIPE). That in turn, should mean that the "tee"
process terminates prematurely.

Then I mended the tee command to read:

strace -o tee.strace tee -a logfile &

Then the final part of tee.strace is

open("logfile", O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE, 0666) = 3
fstat64(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7d98000
fstat64(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
_llseek(3, 0, [0], SEEK_SET) = 0
munmap(0xb7d98000, 4096) = 0
read(0, "", 8192) = 0
close(3) = 0
close(0) = 0
exit_group(0) = ?

We here see that tee opens the logfile, allocates a buffer,
checks the size of the output file (or what else is it doing
fstat for?), discards the buffer, tries to read some data from
the standard input, and gets an end-of-file indication.

Why?

-Enrique

Enrique Perez-Terron

2005-09-17, 9:04 pm

On Sat, 17 Sep 2005 00:41:40 +0200, Enrique Perez-Terron <enrio@online.no>
wrote:

> bash my_script | (tee -a logfile &)
> echo "Hello World"
>
> What I want to achieve is that the shell waits until my_script has
> finished, but allows the tee process go on and capture any output
> coming later from the daemons started by my_script.
[snip]
> But this does not seem to work. To investigate this a little further,
> I wrote a new myscript, with the following contents:
>
> #!/bin/bash
> echo Important
>
> and a wrapper "q" with the contents
>
> #!/bin/bash
> myscript | {
> tee -a logfile &
> }
> echo PIPESTATUS: ${PIPESTATUS[*]}
> echo "Hello World"
>
> and running the latter I get the output
>
> PIPESTATUS: 141 0
> Hello World

Problem solved.

strace revealed that bash redirects the standard input of a process
that is run in the background using &, from /dev/null.

Solution:

#!/bin/bash
myscript | {
exec 3<&0
(tee -a logfile <&3) &
}

runs tee connected to the pipe, yet in the background. Execution
continues when myscript finishes even if tee is still running and
reading data from myscript's chhild processes.

-Enrique