|
Home > Archive > Unix Programming > February 2006 > Program hanging when read()ing from pipe
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Program hanging when read()ing from pipe
|
|
| Julien 2006-02-04, 5:52 pm |
| Hello,
I'm writing a program in C which uses a pipe()/fork()/exec() mechanism
to execute a program as a child and make it communicate with the parent
process. At some point, the child process is supposed to stop but
read()
doesn't seem to get an EOF and the parent hangs.
'ps' reports that the child process is defunct.
If I execute the child from a terminal and do exactly the same things
as
the master process does, it exits normally.
What can I do to get things to work properly ?
Thanks in advance for your help.
| |
| Nils O. Selåsdal 2006-02-04, 5:52 pm |
| Julien wrote:
> Hello,
>
> I'm writing a program in C which uses a pipe()/fork()/exec() mechanism
> to execute a program as a child and make it communicate with the parent
> process. At some point, the child process is supposed to stop but
> read()
> doesn't seem to get an EOF and the parent hangs.
> 'ps' reports that the child process is defunct.
Oh - means it exited. You need call wait/waitpid/or similar
on the child (do it in a signal handler for SIGCHLD). Processes
stick around untill you call wait on them
> If I execute the child from a terminal and do exactly the same things
> as
> the master process does, it exits normally.
> What can I do to get things to work properly ?
>
> Thanks in advance for your help.
Did you remember to close the writing end (or atleast the one you don't
use) in the child, and close the reading end in the parent ?
| |
| Ian Collins 2006-02-04, 5:52 pm |
| Julien wrote:
> Hello,
>
> I'm writing a program in C which uses a pipe()/fork()/exec() mechanism
> to execute a program as a child and make it communicate with the parent
> process. At some point, the child process is supposed to stop but
> read()
> doesn't seem to get an EOF and the parent hangs.
> 'ps' reports that the child process is defunct.
Which order are things supposed to die? If the child is defunct, the
parent must have terminated.
--
Ian Collins.
| |
| Barry Margolin 2006-02-04, 5:52 pm |
| In article <1139090292.579903@drone2-svc-skyt.qsi.net.nz>,
Ian Collins <ian-news@hotmail.com> wrote:
> Julien wrote:
>
> Which order are things supposed to die? If the child is defunct, the
> parent must have terminated.
No, it's the other way around. Defunct means the child has died, but
the parent hasn't yet called one of the wait() functions to get its
termination status.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
| Ian Collins 2006-02-04, 5:52 pm |
| Barry Margolin wrote:
>
>
> No, it's the other way around. Defunct means the child has died, but
> the parent hasn't yet called one of the wait() functions to get its
> termination status.
>
Doesn't this also happen if the parent has terminated before the child
and can't call one of the wait() functions, or does init clean up?
--
Ian Collins.
| |
| Barry Margolin 2006-02-04, 8:47 pm |
| In article <1139094474.89209@drone2-svc-skyt.qsi.net.nz>,
Ian Collins <ian-news@hotmail.com> wrote:
> Barry Margolin wrote:
> Doesn't this also happen if the parent has terminated before the child
> and can't call one of the wait() functions, or does init clean up?
When the parent terminates first, init becomes the parent. Then the
definition becomes the same -- when the child terminates, it will become
a zombie for the brief period until init calls wait().
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
|
|
| Julien 2006-02-05, 5:06 pm |
| Thanks for the answers.
[vbcol=seagreen]
Yes, and I can read and write through the pipes. I actually get what I
expect from the external program that is launched as child.
I did some rewriting and the child process now ends normally (not
defunct anymore) ; waitpid returns the child process' pid.
The while(r=read(...)){...} loop still hangs however because there is
nothing to read in the pipe's buffer. Why doesn't it get EOF ?
I don't know if that has something to do with the problem, but before
terminating, the child spawns another process, which runs in the
background and has init as parent.
| |
| Mr. Uh Clem 2006-02-05, 5:06 pm |
| Julien wrote:
> Thanks for the answers.
>
>
>
>
> Yes, and I can read and write through the pipes. I actually get what I
> expect from the external program that is launched as child.
>
> I did some rewriting and the child process now ends normally (not
> defunct anymore) ; waitpid returns the child process' pid.
> The while(r=read(...)){...} loop still hangs however because there is
> nothing to read in the pipe's buffer. Why doesn't it get EOF ?
>
> I don't know if that has something to do with the problem, but before
> terminating, the child spawns another process, which runs in the
> background and has init as parent.
>
If the grand child inherits a copy of the pipe to the parent,
the pipe will still have an end open when the child exits, so
the parent will not see EOF. (You can have this same sort of
problem if a parent is spawning multiple children. A subsequent
child can inherit the parent's end of a pipe to a previous
child, making it impossible for the parent to close() the
pipe to the previous child and have it get EOF.)
--
Clem
"If you push something hard enough, it will fall over."
- Fudd's first law of opposition
| |
| Julien 2006-02-06, 8:04 am |
| > If the grand child inherits a copy of the pipe to the parent,
> the pipe will still have an end open when the child exits, so
> the parent will not see EOF. (You can have this same sort of
> problem if a parent is spawning multiple children. A subsequent
> child can inherit the parent's end of a pipe to a previous
> child, making it impossible for the parent to close() the
> pipe to the previous child and have it get EOF.)
Is there anyway to check that ? It would be strange since the grand
child is supposed to be in the background, so I guess it doesn't need
any stdio.
| |
| Thomas Maier-Komor 2006-02-06, 8:04 am |
| Julien wrote:
>
> Is there anyway to check that ? It would be strange since the grand
> child is supposed to be in the background, so I guess it doesn't need
> any stdio.
On Solaris you can achieve this with pfiles and taking a look at the ino
field. In the general case, you will probably have to go through
/proc/<pid>/fd/* and look what you find there. Maybe you could generate
a core file with gcore and take a look at that...
Tom
| |
| Alex Fraser 2006-02-06, 8:04 am |
| "Julien" <jdemoor@gmail.com> wrote in message
news:1139176251.838363.61190@g43g2000cwa.googlegroups.com...
[snip]
> The while(r=read(...)){...} loop still hangs however because there is
> nothing to read in the pipe's buffer. Why doesn't it get EOF ?
read() on the read end of a pipe descriptor (in the default blocking mode)
blocks if and only if (1) there is no data available and (2) the write end
of the pipe is open in one or more processes. It returns 0 (indicating EOF)
if 1 is true but 2 is not, and >0 if 1 is false (irrespective of 2).
If you are looping on read() but not getting EOF, one or more processes must
have the write end of the pipe open. I suggest using lsof or equivalent to
identify which.
Alex
| |
| Julien 2006-02-06, 5:56 pm |
| Alex Fraser wrote:
> read() on the read end of a pipe descriptor (in the default blocking mode)
> blocks if and only if (1) there is no data available and (2) the write end
> of the pipe is open in one or more processes. It returns 0 (indicating EOF)
> if 1 is true but 2 is not, and >0 if 1 is false (irrespective of 2).
>
> If you are looping on read() but not getting EOF, one or more processes must
> have the write end of the pipe open. I suggest using lsof or equivalent to
> identify which.
I did that and found that the grand child inherits the pipe, as was
suggested. Since the child sends the SIGCHLD signal, I think I'll be
able to close the pipe at the right time and get things to work. Thanks
for the replies.
| |
| Alex Fraser 2006-02-06, 5:56 pm |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e77d10$0$11094$636a55ce@news.free.fr...
> Alex Fraser wrote:
[snip]
>
> I did that and found that the grand child inherits the pipe, as was
> suggested. Since the child sends the SIGCHLD signal, I think I'll be
> able to close the pipe at the right time and get things to work.
That is not guaranteed to work in general because SIGCHLD could be delivered
before you have drained the pipe. Depending on the circumstances (which you
haven't fully explained), either the child (perhaps between fork and exec,
in the grandchild process) or the grandchild should close the write end.
Alex
| |
| Julien 2006-02-06, 5:56 pm |
| > That is not guaranteed to work in general because SIGCHLD could be delivered
> before you have drained the pipe. Depending on the circumstances (which you
> haven't fully explained), either the child (perhaps between fork and exec,
> in the grandchild process) or the grandchild should close the write end.
I can close the writing end neither from the child because the
exec()uted process will be writing to it nor from the grandchild because
it's an external program (and I'd prefer not to modify the sources).
Do you see any other problem if SIGCHLD is treated differently if
catched while in the reading loop (for processing before next read()) ?
| |
| Alex Fraser 2006-02-06, 8:48 pm |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e7aab2$0$625$636a55ce@news.free.fr...
>
> I can close the writing end neither from the child because the
> exec()uted process will be writing to it nor from the grandchild because
> it's an external program (and I'd prefer not to modify the sources).
What is the number of the inherited descriptor for the write end of the pipe
in the external program?
If it is not 1 or 2, then you should almost certainly close the write end of
the pipe in the child program, after fork, in the grandchild of the first
process, before exec'ing the external program.
If it is, it might make sense for the child to arrange that the external
program has the descriptor pointing at /dev/null instead of the pipe.
I can't say much for sure, because you have not provided sufficient
information.
> Do you see any other problem if SIGCHLD is treated differently if
> catched while in the reading loop (for processing before next read()) ?
Sorry, I'm not sure what you mean.
Alex
| |
| Julien 2006-02-07, 7:51 am |
| Alex Fraser wrote:
> "Julien" <j_____@nospamhotmail.fr> wrote in message
> news:43e7aab2$0$625$636a55ce@news.free.fr...
>
>
>
> What is the number of the inherited descriptor for the write end of the pipe
> in the external program?
1.
> If it is, it might make sense for the child to arrange that the external
> program has the descriptor pointing at /dev/null instead of the pipe.
But I have no control over its behaviour.
>
>
> Sorry, I'm not sure what you mean.
That's why I tried to cancel my previous message. I actually solved the
problem by catching SIGCHLD and using select() to check if there is
anything more to read in the buffer prior to closing the read end of the
pipe in the parent process. If the buffer is not empty, then read() is
called one more time.
| |
| Alex Fraser 2006-02-07, 7:51 am |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e877f2$0$1239$636a55ce@news.free.fr...
> Alex Fraser wrote:
[snip]
>
> 1.
>
>
> But I have no control over its behaviour.
Do you have control over the child program (the one spawned by the process
which doesn't get EOF)?
Does the external program need the write end of the pipe?
If the answers to the above are yes and no respectively, then in the child
program do something like:
if (fork() == 0) {
int fd = open("/dev/null", O_WRONLY);
dup2(fd, 1);
close(fd);
/* exec external program */
}
> I actually solved the problem by catching SIGCHLD and using select() to
> check if there is anything more to read in the buffer prior to closing
> the read end of the pipe in the parent process. If the buffer is not
> empty, then read() is called one more time.
This approach still has some issues. A slight improvement would be to set
the descriptor to non-blocking and loop read() until you get EAGAIN.
As I said before, I can't say much for sure, because you have not provided
sufficient information. But it seems very likely is that the parent process
is *not* the right place to solve the problem; the fact you observe the
problem there is merely a symptom that some part of the whole system (which
includes the child and external programs) is broken.
Alex
| |
| Julien 2006-02-07, 7:51 am |
| > Do you have control over the child program (the one spawned by the process
> which doesn't get EOF)?
No.
> Does the external program need the write end of the pipe?
Yes.
> This approach still has some issues. A slight improvement would be to set
> the descriptor to non-blocking and loop read() until you get EAGAIN.
What issues do think of ? If the read is in non-blocking mode and I get
EAGAIN doesn't mean the program is terminated.
> As I said before, I can't say much for sure, because you have not provided
> sufficient information. But it seems very likely is that the parent process
> is *not* the right place to solve the problem; the fact you observe the
> problem there is merely a symptom that some part of the whole system (which
> includes the child and external programs) is broken.
I think that a program that goes into the background shouldn't inherit
its parent's stdout, but that's not under my control.
What happens for to program at the write end of the pipe when the other
end is closed ? Is the output redirected to /dev/null ?
| |
| Alex Fraser 2006-02-07, 6:03 pm |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e8a53e$0$12588$636a15ce@news.free.fr...
> No.
>
> Yes.
Does the external program write something to the pipe that you need to read
in its grandparent? If so, how can you be sure it is read if the grandparent
stops reading after the child program ends? If not, why does the external
program need the write end of the pipe?
I have repeatedly hinted that more information is needed. Unless you take
the hint, don't expect a reply.
Alex
| |
| Julien 2006-02-07, 6:03 pm |
| > I have repeatedly hinted that more information is needed. Unless you take
> the hint, don't expect a reply.
That's because I didn't see what more I could say. Here's a summary of
the situation :
My process forks. The child exec()s an external program. The parent
communicates with it through pipes connected to the child's
std{in,out,err}. At some point, the child (over which I have no control)
spawns another external process which goes in the background but
unfortunately inherits its parent's stdio. But my process does not need
to communicate with it and it is not supposed to output anything (appart
from some error messages that we be safely ignored). It should actually
stop the while(read()) loop (on the pipe connected to stdout, stderr is
checked later). Since the pipe is inherited between the external
programs, read() is blocking. That's what I plan to solve with a SIGCHLD
handler.
> Does the external program write something to the pipe that you need to
read in its grandparent?
No.
> If not, why does the external program need the write end of the pipe ?
The child needs it to communicate but the grand child doesn't - but
inherits it after the child's fork().
Do you still see any issue with the parent closing the pipes upon
receiving SIGCHLD (after calling read() if there is something in the pipe) ?
| |
| Alex Fraser 2006-02-07, 6:03 pm |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e8b734$0$592$636a15ce@news.free.fr...
> My process forks. The child exec()s an external program. The parent
> communicates with it through pipes connected to the child's
> std{in,out,err}. At some point, the child (over which I have no control)
> spawns another external process which goes in the background but
> unfortunately inherits its parent's stdio. But my process does not need
> to communicate with it and it is not supposed to output anything (appart
> from some error messages that we be safely ignored). It should actually
> stop the while(read()) loop (on the pipe connected to stdout, stderr is
> checked later).
This could cause a deadlock where the child is blocked writing to stderr
(because it fills the pipe buffer) while your program is blocked reading the
child's stdout (after you've read everything written before the child
blocked). You need to use non-blocking I/O (in conjunction with
select()/poll()) to avoid this possibility.
Similarly, there is a potential deadlock where you block writing to the
child's stdin and the child blocks writing to stdout.
> Since the pipe is inherited between the external programs, read() is
> blocking. That's what I plan to solve with a SIGCHLD handler.
>
>
> No.
>
>
> The child needs it to communicate but the grand child doesn't - but
> inherits it after the child's fork().
>
> Do you still see any issue with the parent closing the pipes upon
> receiving SIGCHLD (after calling read() if there is something in the
> pipe) ?
You previously wrote:
> I actually solved the problem by catching SIGCHLD and using select() to
> check if there is anything more to read in the buffer prior to closing
> the read end of the pipe in the parent process. If the buffer is not
> empty, then read() is called one more time.
There are two solvable issues I can see with this. The first is fairly
obvious: what if there are more bytes in the pipe than you read in the extra
call to read()? The second is rather theoretical: even if select() says the
pipe is readable, that doesn't mean a subsequent read() won't block.
Both of these issues are solved by the alternative I gave: after you get
SIGCHLD, make the pipe non-blocking and call read() in a loop until it fails
with EAGAIN.
In practice, I think this will read everything previously written to the
pipe (which, in your case, being done after getting SIGCHLD, must include
everything written by the child - if the child has exited it can't write any
more!). However, I am not convinced that this is guaranteed.
I have thought of another solution, although I don't know if it makes sense
or even is practical or possible in your case. If it does make sense, and is
both practical and possible, it is IMO preferable to the above (in part
because of the niggling uncertainty).
The idea is very simple: somehow (I can't suggest how) arrange for the child
to spawn a "stub" program instead of the external program. The stub closes
the pipe(s), open()s descriptors 0, 1, and 2 so they point at /dev/null,
then execs the real program (without forking).
With this in place, you should get EOF in your program when you were
originally expecting it. You won't need to handle SIGCHLD - just call wait()
after you get EOF.
From what you've said, it seems likely that I would consider one (or both)
of the external programs broken. Either the first should spawn the second
with stdin, out and err redirected to /dev/null, or the second should close
them when it starts.
Alex
| |
| Julien 2006-02-07, 6:03 pm |
| > This could cause a deadlock where the child is blocked writing to stderr
> (because it fills the pipe buffer) while your program is blocked reading the
> child's stdout (after you've read everything written before the child
> blocked). You need to use non-blocking I/O (in conjunction with
> select()/poll()) to avoid this possibility.
OK, so I'll have to do it at least for stderr.
> Similarly, there is a potential deadlock where you block writing to the
> child's stdin and the child blocks writing to stdout.
It won't happen because each write to the child is done at a time when
the child expects it.
> There are two solvable issues I can see with this. The first is fairly
> obvious: what if there are more bytes in the pipe than you read in the extra
> call to read()?
while(function() || read()>0){}
where function() closes the pipe if there is nothing to read and SIGCHLD
has been received; returns 1 if pipe still open, 0 if closed.
> The second is rather theoretical: even if select() says the
> pipe is readable, that doesn't mean a subsequent read() won't block.
Can you explain this further ? I read in the manual that I can use
select precisely to check if a read() will not block.
> I have thought of another solution, although I don't know if it makes sense
> or even is practical or possible in your case. If it does make sense, and is
> both practical and possible, it is IMO preferable to the above (in part
> because of the niggling uncertainty).
>
> The idea is very simple: somehow (I can't suggest how) arrange for the child
> to spawn a "stub" program instead of the external program. The stub closes
> the pipe(s), open()s descriptors 0, 1, and 2 so they point at /dev/null,
> then execs the real program (without forking).
>
> With this in place, you should get EOF in your program when you were
> originally expecting it. You won't need to handle SIGCHLD - just call wait()
> after you get EOF.
I don't think it's possible to do this in my case.
| |
| Alex Fraser 2006-02-07, 6:03 pm |
| "Julien" <j_____@nospamhotmail.fr> wrote in message
news:43e8e2f1$0$1720$636a55ce@news.free.fr...
>
> OK, so I'll have to do it at least for stderr.
>
>
> It won't happen because each write to the child is done at a time when
> the child expects it.
Fair enough.
You've reminded me of something I meant to mention before. If it is possible
to determine from the data you read from the child that you have read
everything, then do so. This amounts to detecting an in-band EOF indication.
>
> while(function() || read()>0){}
> where function() closes the pipe if there is nothing to read and SIGCHLD
> has been received; returns 1 if pipe still open, 0 if closed.
It didn't sound from your description like you had a loop.
>
> Can you explain this further ? I read in the manual that I can use
> select precisely to check if a read() will not block.
There is a (small) period of time between when select() determines that the
descriptor is readable and when you call read(). select() can't tell you
that a future read() *will not* block; what it can (and does) tell you is
that, at the time it checked, a read() *would not have* blocked.
This is not a big deal. Just make the descriptor non-blocking with fcntl(),
and handle the possible EAGAIN error by acting as though select() hadn't
said the descriptor was readable.
Alex
| |
| Julien 2006-02-14, 5:54 pm |
| >
> It didn't sound from your description like you had a loop.
>
Sorry for that, I should have mentionned it.
Thanks for your help, I have the program running very fine now.
|
|
|
|
|