Hi,

I fork and exec a child and use a pipe to comunicate the child's output
to the parent. The parent waits until the child is complete before
reading the output.

It works fine where the child has output of equal or less than 10240
characters, however above that the child blocks. I guess the pipe is
full and that's why it's blocking.

I'd like to know exactly how this limit comes about and if it can be
changed.

I have printed PIPE_MAX from limits.h but that is 5120 bytes - so I
think this is a red herring.

Let me know your thoughts.

[ Post a follow-up to this message ]

smiffy wrote:
> I fork and exec a child and use a pipe to comunicate the child's output
> to the parent. The parent waits until the child is complete before
> reading the output.

change your construction ... dont wait until the child has finished ...
read from the pipe until the read returns an error.

> It works fine where the child has output of equal or less than 10240
> characters, however above that the child blocks. I guess the pipe is
> full and that's why it's blocking.

Yes, that's why the idea to wait until the child has finished is silly.
Imagine your child produces 1000GB of data ... who shall buffer that?

> I'd like to know exactly how this limit comes about and if it can be
> changed.

It usually cannot be changed without modification in the kernel. It is
not reasonable to change it.

> I have printed PIPE_MAX from limits.h but that is 5120 bytes - so I
> think this is a red herring.

No it isn't PIP_MAX is the amount of data that doens't get fragmented if
there are concurrent writes to the pipe.

Rainer

[ Post a follow-up to this message ]

smiffy wrote:
> Hi,
>
> I fork and exec a child and use a pipe to comunicate the child's output
> to the parent. The parent waits until the child is complete before
> reading the output.
This is flawd.
The parent should start reading the pipe after it forks the child,
and then keep reading until EOF.

[ Post a follow-up to this message ]

In article <e9l63g$cfh$2@daniel-new.mch.sbs.de>,
Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:

> smiffy wrote: 
>
> change your construction ... dont wait until the child has finished ...
> read from the pipe until the read returns an error.

Or EOF, which is more likely.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

[ Post a follow-up to this message ]

Barry Margolin wrote:

 
[vbcol=seagreen] 
[vbcol=seagreen]
> Or EOF, which is more likely.

Well, to be exact ... EOF is a FILE-thing.
But you're right. read() will return 0.

I mainly use all IO togehter with poll()/select() and
treat a POLLIN and no data on read() as an error and close.
(That prooved to be a quite robust construction). But therefore
I don't care much if read returned 0 or -1.

Rainer

[ Post a follow-up to this message ]

Rainer Temme wrote:

> I mainly use all IO togehter with poll()/select() and
> treat a POLLIN and no data on read() as an error and close.
> (That prooved to be a quite robust construction). But therefore
> I don't care much if read returned 0 or -1.

What if 'read()' errors out with EWOULDBLOCK? (Which can definitely
happen with connected UDP.)

DS

[ Post a follow-up to this message ]

davids@webmaster.com wrote: 
[vbcol=seagreen]
> What if 'read()' errors out with EWOULDBLOCK? (Which can definitely
> happen with connected UDP.)

Actually, if poll() select() have indicated POLLIN,
a read()/recv()/recvfrom() shall be called without blocking.
Therefore, if it returns with EWOULDBLOCK that's certainly an error.

The only situation where I know that the receive-calls will
return -1 and errno==EWOULDBLOCK, is when the socket is
set to non-blocking, and you call the receive-function
when there is no data available.

But that is not what I'm doing. I call poll/select before any
receive-call.

Rainer

[ Post a follow-up to this message ]

Rainer Temme wrote:

> Actually, if poll() select() have indicated POLLIN,
> a read()/recv()/recvfrom() shall be called without blocking.

See the other thread.

> Therefore, if it returns with EWOULDBLOCK that's certainly an error.

Consider if the socket is UDP and the UDP packet is discarded by the
kernel between the POLLIN hit and the call to 'recvfrom'. In that case,
it *will* block and this really does happen.

> The only situation where I know that the receive-calls will
> return -1 and errno==EWOULDBLOCK, is when the socket is
> set to non-blocking, and you call the receive-function
> when there is no data available.

> But that is not what I'm doing. I call poll/select before any
> receive-call.

That doesn't guarantee the data will *still* be available when you
later come around to calling a receive function. See the other thread.

DS

[ Post a follow-up to this message ]

davids@webmaster.com schrieb:
> Consider if the socket is UDP and the UDP packet is discarded by the
> kernel between the POLLIN hit and the call to 'recvfrom'. In that case,
> it *will* block and this really does happen.

No, the ip-stack will not push a packet to the udp-stack before it's
reassembled. After it's reassembled and accepted by udp there is no
reason to discard it (that fact that it has been accepted makes clear,
that there was enough bufferspace).

So, either a packet is discarded (in this case POLLIN is not signalled)
or POLLIN is signalled and there is something to read, or an
error-condition.

Rainer

[ Post a follow-up to this message ]

Rainer Temme wrote:

> davids@webmaster.com schrieb:
 
[vbcol=seagreen]
> No, the ip-stack will not push a packet to the udp-stack before it's
> reassembled. After it's reassembled and accepted by udp there is no
> reason to discard it (that fact that it has been accepted makes clear,
> that there was enough bufferspace).

> So, either a packet is discarded (in this case POLLIN is not signalled)
> or POLLIN is signalled and there is something to read, or an
> error-condition.

You are incorrect. Linux in fact does that and it in fact broke code.

You can argue all you want that this is the way it should be, but no
standard guarantees or requires this behavior and at least one popular
UNIX (Linux) does not in fact provide this behavior.

In fact, this use to be a denial of service attack against inetd which
didn't set its sockets non-blocking.

DS

[ Post a follow-up to this message ]