|
Home > Archive > Unix Programming > July 2006 > pipe write buffer
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
| smiffy 2006-07-19, 8:04 am |
| Hi,
I fork and exec a child and use a pipe to comunicate the child's output
to the parent. The parent waits until the child is complete before
reading the output.
It works fine where the child has output of equal or less than 10240
characters, however above that the child blocks. I guess the pipe is
full and that's why it's blocking.
I'd like to know exactly how this limit comes about and if it can be
changed.
I have printed PIPE_MAX from limits.h but that is 5120 bytes - so I
think this is a red herring.
Let me know your thoughts.
| |
| Rainer Temme 2006-07-19, 8:04 am |
| smiffy wrote:
> I fork and exec a child and use a pipe to comunicate the child's output
> to the parent. The parent waits until the child is complete before
> reading the output.
change your construction ... dont wait until the child has finished ...
read from the pipe until the read returns an error.
> It works fine where the child has output of equal or less than 10240
> characters, however above that the child blocks. I guess the pipe is
> full and that's why it's blocking.
Yes, that's why the idea to wait until the child has finished is silly.
Imagine your child produces 1000GB of data ... who shall buffer that?
> I'd like to know exactly how this limit comes about and if it can be
> changed.
It usually cannot be changed without modification in the kernel. It is
not reasonable to change it.
> I have printed PIPE_MAX from limits.h but that is 5120 bytes - so I
> think this is a red herring.
No it isn't PIP_MAX is the amount of data that doens't get fragmented if
there are concurrent writes to the pipe.
Rainer
| |
| Nils O. Selåsdal 2006-07-19, 1:40 pm |
| smiffy wrote:
> Hi,
>
> I fork and exec a child and use a pipe to comunicate the child's output
> to the parent. The parent waits until the child is complete before
> reading the output.
This is flawd.
The parent should start reading the pipe after it forks the child,
and then keep reading until EOF.
| |
| Barry Margolin 2006-07-20, 1:47 am |
| In article <e9l63g$cfh$2@daniel-new.mch.sbs.de>,
Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:
> smiffy wrote:
>
> change your construction ... dont wait until the child has finished ...
> read from the pipe until the read returns an error.
Or EOF, which is more likely.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
| Rainer Temme 2006-07-20, 7:51 am |
| Barry Margolin wrote:
[vbcol=seagreen]
[vbcol=seagreen]
> Or EOF, which is more likely.
Well, to be exact ... EOF is a FILE-thing.
But you're right. read() will return 0.
I mainly use all IO togehter with poll()/select() and
treat a POLLIN and no data on read() as an error and close.
(That prooved to be a quite robust construction). But therefore
I don't care much if read returned 0 or -1.
Rainer
| |
| davids@webmaster.com 2006-07-20, 7:51 am |
|
Rainer Temme wrote:
> I mainly use all IO togehter with poll()/select() and
> treat a POLLIN and no data on read() as an error and close.
> (That prooved to be a quite robust construction). But therefore
> I don't care much if read returned 0 or -1.
What if 'read()' errors out with EWOULDBLOCK? (Which can definitely
happen with connected UDP.)
DS
| |
| Rainer Temme 2006-07-20, 7:51 am |
| davids@webmaster.com wrote:
[vbcol=seagreen]
> What if 'read()' errors out with EWOULDBLOCK? (Which can definitely
> happen with connected UDP.)
Actually, if poll() select() have indicated POLLIN,
a read()/recv()/recvfrom() shall be called without blocking.
Therefore, if it returns with EWOULDBLOCK that's certainly an error.
The only situation where I know that the receive-calls will
return -1 and errno==EWOULDBLOCK, is when the socket is
set to non-blocking, and you call the receive-function
when there is no data available.
But that is not what I'm doing. I call poll/select before any
receive-call.
Rainer
| |
| davids@webmaster.com 2006-07-20, 7:24 pm |
|
Rainer Temme wrote:
> Actually, if poll() select() have indicated POLLIN,
> a read()/recv()/recvfrom() shall be called without blocking.
See the other thread.
> Therefore, if it returns with EWOULDBLOCK that's certainly an error.
Consider if the socket is UDP and the UDP packet is discarded by the
kernel between the POLLIN hit and the call to 'recvfrom'. In that case,
it *will* block and this really does happen.
> The only situation where I know that the receive-calls will
> return -1 and errno==EWOULDBLOCK, is when the socket is
> set to non-blocking, and you call the receive-function
> when there is no data available.
> But that is not what I'm doing. I call poll/select before any
> receive-call.
That doesn't guarantee the data will *still* be available when you
later come around to calling a receive function. See the other thread.
DS
| |
| Rainer Temme 2006-07-20, 7:24 pm |
| davids@webmaster.com schrieb:
> Consider if the socket is UDP and the UDP packet is discarded by the
> kernel between the POLLIN hit and the call to 'recvfrom'. In that case,
> it *will* block and this really does happen.
No, the ip-stack will not push a packet to the udp-stack before it's
reassembled. After it's reassembled and accepted by udp there is no
reason to discard it (that fact that it has been accepted makes clear,
that there was enough bufferspace).
So, either a packet is discarded (in this case POLLIN is not signalled)
or POLLIN is signalled and there is something to read, or an
error-condition.
Rainer
| |
| davids@webmaster.com 2006-07-21, 1:29 am |
|
Rainer Temme wrote:
> davids@webmaster.com schrieb:
[vbcol=seagreen]
> No, the ip-stack will not push a packet to the udp-stack before it's
> reassembled. After it's reassembled and accepted by udp there is no
> reason to discard it (that fact that it has been accepted makes clear,
> that there was enough bufferspace).
> So, either a packet is discarded (in this case POLLIN is not signalled)
> or POLLIN is signalled and there is something to read, or an
> error-condition.
You are incorrect. Linux in fact does that and it in fact broke code.
You can argue all you want that this is the way it should be, but no
standard guarantees or requires this behavior and at least one popular
UNIX (Linux) does not in fact provide this behavior.
In fact, this use to be a denial of service attack against inetd which
didn't set its sockets non-blocking.
DS
| |
| Rainer Temme 2006-07-21, 1:29 am |
| davids@webmaster.com wrote:
>
> You are incorrect. Linux in fact does that and it in fact broke code.
David,
please show us/me sume code that shows this behaviour.
Rainer
| |
| William Ahern 2006-07-21, 7:24 pm |
| On Fri, 21 Jul 2006 08:02:25 +0200, Rainer Temme wrote:
> davids@webmaster.com wrote:
>
> David,
>
> please show us/me sume code that shows this behaviour.
I recently patched C-Ares because I hit this exact problem
http://daniel.haxx.se/projects/c-ares/
Do you want to see the kernel code, or just any code. If just any code,
see the CVS web repository above, specifically ares_process.c. Though,
it's nothing special.
I will attest that Linux (at least 2.4 kernel) can and will return read
readiness but have nothing available in the queue to read, at least for
UDP sockets. I wrote a "high-performance" resolver library and was
perplexed why a single thread using getaddrinfo(3) was destroying the
multiplexed I/O version. Turns out under high-load Linux was dropping UDP
packets like mad. Increasing the socket buffer size through setsockopt()
had negligible effect. I saw plain as day the strace and gdb logs show
read readiness on a descriptor followed immediately by a read error with
EAGAIN. Correlated w/ network traces I also saw UDP DNS replies just
disappear into the abyss, sometimes triggering readiness on their own,
sometimes not (if they came too close to each other).
| |
| davids@webmaster.com 2006-07-22, 1:22 am |
|
Rainer Temme wrote:
> davids@webmaster.com wrote:
>
> David,
>
> please show us/me sume code that shows this behaviour.
>
> Rainer
See the Linux inetd denial of service attack thread. Unfortunately,
inetd assumed that a read hit for 'select' guaranteed that a
subseqeuent 'read' would not block. However, the Linux kernel decided
to drop the packet in-between the 'select' hit and the 'read' hit
(which it has the absolute right to do at any time), and 'inetd'
blocked in 'read'.
DS
| |
| Rainer Temme 2006-07-22, 7:38 am |
| davids@webmaster.com wrote:
[vbcol=seagreen]
[vbcol=seagreen]
> See the Linux inetd denial of service attack thread. Unfortunately,
> inetd assumed that a read hit for 'select' guaranteed that a
> subseqeuent 'read' would not block. However, the Linux kernel decided
> to drop the packet in-between the 'select' hit and the 'read' hit
> (which it has the absolute right to do at any time), and 'inetd'
> blocked in 'read'.
No David, you're on the wrong trail here...
Following your argumentation, a system without a UDP stack could
claim to have one, but unfortunately it's implementation
NEVER delivers any packets ... but according to your argument ...
that's covered by standards.
The question is NOT about UDP at all.
select() / poll() have indicated that a read() recv() ... call can be
made WITHOUT blocking (even if the fd in used is in blocking mode).
This is a guarantee to the process not only for a few milliseconds.
Its the kernels responsibility to maintain an appropriate state on
this fd until the application finds time to care for this, even
if it takes five years (so to say).
An implementation failing to do so simply has an error!
What you do is to pretend, that this errorneous behaviour is
coverd by the UDP standard. (Well see above, this is argueable)
Nevertheless the kernel-implementation has a flaw, since it MUST NOT
UNDER NO CIRCUMSTANCES NOT EVEN UNDER MEMORY PRESSURE block a call
when it promised not to block it before that.
Rainer
| |
| Barry Margolin 2006-07-22, 1:18 pm |
| In article <e9snde$pn5$1@online.de>,
Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:
> select() / poll() have indicated that a read() recv() ... call can be
> made WITHOUT blocking (even if the fd in used is in blocking mode).
No, it's an indication that a read()/recv() *could have been* made
without blocking at the time that select() performed its check.
> This is a guarantee to the process not only for a few milliseconds.
> Its the kernels responsibility to maintain an appropriate state on
> this fd until the application finds time to care for this, even
> if it takes five years (so to say).
This is simply impossible to guarantee in general. Consider two
processes both calling select() on the same stream. Both will receive
an indication that the socket is readable, but only the first one to
call read() will get data, the other will block or get an EWOULDBLOCK
error.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
| Rainer Temme 2006-07-22, 7:20 pm |
| Barry Margolin wrote:
>
> This is simply impossible to guarantee in general. Consider two
> processes both calling select() on the same stream. Both will receive
> an indication that the socket is readable, but only the first one to
> call read() will get data, the other will block or get an EWOULDBLOCK
> error.
That's not the same situation ... when you use a construction, where
more that multiple readers are trying to read something from
the stream, at least one of them got something. I admit, that the
circumstances come close to what we were discussing, but in this case
the kernel cannot be blamed ... in the other case it's a simple
implementation-error.
Rainer
| |
| Alex Fraser 2006-07-22, 7:20 pm |
| "Rainer Temme" <Rainer.Temme@NoSpam.Siemens.Com> wrote in message
news:e9tqji$1or$1@online.de...
> Barry Margolin wrote:
>
> That's not the same situation ... when you use a construction, where
> more that multiple readers are trying to read something from
> the stream, at least one of them got something. I admit, that the
> circumstances come close to what we were discussing, but in this case
> the kernel cannot be blamed ... in the other case it's a simple
> implementation-error.
Something can only reasonably be called an "implementation error" if it does
not comply with required behaviour. As far as I can see, you have not even
attempted to prove that the behaviour implied by the guarantee above is
required.
I (for one) don't really care whether it is or is not required, because in
my experience it is trivial to write code that will work correctly in either
case. To me, in the absence of a clear requirement, it seems prudent to do
this - in other words, to code defensively.
Alex
| |
| davids@webmaster.com 2006-07-22, 7:20 pm |
|
Rainer Temme wrote:
[vbcol=seagreen]
> No David, you're on the wrong trail here...
Please explain why.
> Following your argumentation, a system without a UDP stack could
> claim to have one, but unfortunately it's implementation
> NEVER delivers any packets ... but according to your argument ...
> that's covered by standards.
I agree that that's allowed by the standard for 'select'. That's
because 'select' is protocol-independent, so it can only guarantee
what's possible for all the protocol it's expected to support.
> The question is NOT about UDP at all.
I agree. UDP is just an example. If it can legally happen for 'select'
with UDP, then it can legally happen for 'select'. If the question is
what 'select' guarantees, if it doesn't happen with select/UDP, then
'select' doesn't guarantee it.
> select() / poll() have indicated that a read() recv() ... call can be
> made WITHOUT blocking (even if the fd in used is in blocking mode).
No, not "can be made", "could have been made". The indiciation is
correct at the time it is generated, but becomes stale as soon as your
code gets it. This is the same for every other indication provided by
every other function, 'stat', 'access'. It is a bug to assume that it
must be true later.
> This is a guarantee to the process not only for a few milliseconds.
> Its the kernels responsibility to maintain an appropriate state on
> this fd until the application finds time to care for this, even
> if it takes five years (so to say).
No, it is not. POSIX does not say this and it defies common sense. You
might as well say that if I 'stat' a file, the kernel shouldn't let
another process delete or modify that file just in case I plan to
follow the 'stat' with some other system call that relies on the
response from 'stat'. It's nonsensical.
> An implementation failing to do so simply has an error!
Nope.
> What you do is to pretend, that this errorneous behaviour is
> coverd by the UDP standard. (Well see above, this is argueable)
> Nevertheless the kernel-implementation has a flaw, since it MUST NOT
> UNDER NO CIRCUMSTANCES NOT EVEN UNDER MEMORY PRESSURE block a call
> when it promised not to block it before that.
Where does it say that there is a promise of future behavior? This is
not the case in any other status function.
Bluntly, you are just making this up and asserting it as fact. It is
not a POSIX requirement.
DS
| |
| Rainer Temme 2006-07-23, 7:23 am |
| davids@webmaster.com wrote:
> Please explain why.
See below...
[vbcol=seagreen]
> No, not "can be made", "could have been made". The indiciation is
> correct at the time it is generated, but becomes stale as soon as your
> code gets it. This is the same for every other indication provided by
> every other function, 'stat', 'access'. It is a bug to assume that it
> must be true later.
[vbcol=seagreen]
> No, it is not. POSIX does not say this and it defies common sense. You
> might as well say that if I 'stat' a file, the kernel shouldn't let
> another process delete or modify that file just in case I plan to
> follow the 'stat' with some other system call that relies on the
> response from 'stat'. It's nonsensical.
>
>
> Nope.
Ok let's have another look at it...
The problem boils down to wether poll() select() deliver "static"
information or wether they deliver a snapshot from state at call-time
(which is subject to change without notice).
Additionally, you claim, that no standard tells that poll() select()
musts deliver static results.
To simplify things ... let's exclude the whole problematic with
"multiple readers callin read after poll/select" because the
problem here is build up in userspace (and can be solved there)
whereas the other is in kernelland. This also excludes your
argumentation with the stat() call. (Because if I'm the only one
to modify a file I can assume the file to have the properties stat()
was telling me in a call.)
So, what would happen to us, if poll() select() where allowed to
deliver snapshot-information (that can change state in the time between
select() and read/write)?
1) Usage of fd's with blocking IO is a complete NONONO.
Why? Because your process can be blocked in read() write at any time
even if a previous poll select where telling (in this case only
pretending) the call wouldn't block.
Multiplexing of fd's with fd's left in blocking mode would
actually be impossible. The poll() select() calls would be rendered
completely unusable, if you could only expect the information to be
a snapshot.
2) Usage of fd'd with non-blocking IO is not much better.
Why? Because we cannot prevent our process from falling into
a busy loop! Noting is preventing a "poll POLLIN read EAGAIN" loop.
If poll only deliveres a snapshot, a system that was idle in
one second can become completely overloaded within a second with
tousands of runable processes that do not do any usefull work
anymore. Even trating EAGAIN/EWOULDBLOCK as an error after POLLIN
(as I usually do) will not help much, because this will only change
the loop-content a bit ...
"poll POLLIN read EAGAIN close create_newsocket"
So the definition that poll() select() will give us only a snapshot-like
result is not helpfull at all. With this definition it becomes
impossible to write stable code.
Being part of a standard or not, the only mode that makes sense
if you think about it, is the static mode, where a state has to
be maintained until the process reacts on the state with a call.
Finally, I think this thread is not on-topic for too long and should
be terminated here ... I think the arguments are exchanged, and
I'm not going to contribute much more.
And, David, wether you think I'm "making this all up" as you
wrote in your last message or not, is something I really
don't care much about. To write it to usenet might be a bit off-road
of common accepted netiquette.
Rainer
| |
| Nils O. Selåsdal 2006-07-23, 7:24 am |
| Barry Margolin wrote:
> In article <e9snde$pn5$1@online.de>,
> Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:
>
>
> No, it's an indication that a read()/recv() *could have been* made
> without blocking at the time that select() performed its check.
This is not what the opengroup documentation says.
It sure leaves things to be desired, but it merely says
"there is data available". This can be interpreted
many ways. If there is data available, then a following read
should not block. It there was data available at the time of
select returning, but it magically vanishes - well... The former
is more reliable and useful - reliable and useful interfacs
are better than unreliable ones :-)
You can also find a bit discussion on the opengroup mailinglists,
though I've not seen any confirmation one way or another what is
"right".
| |
| Alex Fraser 2006-07-23, 7:24 am |
| "Rainer Temme" <Rainer.Temme@NoSpam.Siemens.Com> wrote in message
news:e9vgj9$j68$1@online.de...
[snip]
> So, what would happen to us, if poll() select() where allowed to
> deliver snapshot-information (that can change state in the time between
> select() and read/write)?
>
> 1) Usage of fd's with blocking IO is a complete NONONO.
Suppose poll()/select() results reflect a "static" state. When it says a
descriptor is writeable, you could only be certain to write one byte without
blocking. You usually want to write more than one byte at a time, therefore
when poll()/select() says a descriptor is writeable, a subsequent
write()/send() call might block unless the descriptor is non-blocking. So in
practice, blocking writes are "a complete NONONO" anyway.
> 2) Usage of fd'd with non-blocking IO is not much better.
>
> Why? Because we cannot prevent our process from falling into
> a busy loop! Noting is preventing a "poll POLLIN read EAGAIN" loop.
Nothing prevents POSIX-specified functions from suspending the calling
process for a minute before performing whatever POSIX requires either.
Except one thing: quality of implementation.
Nobody has suggested that a read() call returning EAGAIN after
select()/poll() said it was readable should happen often, just that it
*might* happen.
[snip]
> So the definition that poll() select() will give us only a snapshot-like
> result is not helpfull at all.
I agree, it does not offer any benefit to the user, but it does not impose
any hardship either.
> With this definition it becomes impossible to write stable code.
No more so than usual, as far as I can see.
Alex
| |
| joe@invalid.address 2006-07-23, 1:21 pm |
| Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> writes:
> davids@webmaster.com wrote:
>
> Ok let's have another look at it...
>
> The problem boils down to wether poll() select() deliver "static"
> information or wether they deliver a snapshot from state at call-time
> (which is subject to change without notice).
>
> Additionally, you claim, that no standard tells that poll() select()
> musts deliver static results.
This is an argument I've had many times over the years. I have to say
I agree with David and Barry now about this particular part of it. The
standard doesn't say that static results are required. Reality is that
sometimes they should not even be expected, for example with UDP.
I'd say the standard should be more explicit about what it's not
guaranteeing, but it isn't. And even if it was I don't think it would
be that helpful, because you wind up having to program to the platform
more than the standard anyway.
I still have a disagreement with David about what "blocking" means,
but that's probably not worth worrying much about.
Joe
| |
| Barry Margolin 2006-07-23, 7:20 pm |
| In article <e9tqji$1or$1@online.de>,
Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:
> Barry Margolin wrote:
>
> That's not the same situation ... when you use a construction, where
> more that multiple readers are trying to read something from
> the stream, at least one of them got something. I admit, that the
> circumstances come close to what we were discussing, but in this case
> the kernel cannot be blamed ... in the other case it's a simple
> implementation-error.
Unless the spec specifically mentions the "multiple callers" exception,
it's not possible for the guarantee to be what was intended. So either
it's always guaranteed or it's never guaranteed, and I believe the only
reasonable interpretation of the standard is the latter.
Perhaps it's a bug in the standard, and they meant to say that it's
always guaranteed unless there are concurrent callers of select() on the
same stream. But until they publish an errata saying so, AND most
implementations are updated to conform, applications should be written
defensively to handle potential blocking after select() returns.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
| |
| Brian Raiter 2006-07-24, 7:22 pm |
| > 1) Usage of fd's with blocking IO is a complete NONONO.
Not at all. Quite often this is exactly what you want programs to do:
block indefinitely until data is available or the source is determined
to be empty.
> 2) Usage of fd'd with non-blocking IO is not much better.
>
> Why? Because we cannot prevent our process from falling into a busy
> loop!
Standards frequently fail to require useful behavior that is difficult
to realistically codify. Instead they rely on "market pressure" to
ensure that people make good implementations available. Nothing in any
standard prevents an implementation from always returning -1 from
open(), but who would use such a system?
The problem with the argument is that it boils down to: "It's not fair
to expect us to write code without this guarantee." This is a weak
response to assertion: "This is what the standard says and what real
implementations do."
b
| |
| Rainer Temme 2006-07-24, 7:22 pm |
| Brian Raiter wrote:
[vbcol=seagreen]
> Not at all. Quite often this is exactly what you want programs to do:
> block indefinitely until data is available or the source is determined
> to be empty.
No, it's exactly what you DONT want when multiplexing fds! You simply
cannot afford to be blocked in a call, since if you would be, you dont
serve your other fds any more.
> The problem with the argument is that it boils down to: "It's not fair
> to expect us to write code without this guarantee." This is a weak
> response to assertion: "This is what the standard says and what real
> implementations do."
Fair or not, my point is, that it is simply not possible to write
really reliable software on what the standard seems to say.
BTW: the behaviour in discussion is noted as 'BUG' in the linux
man-pages. Besides the standard saying something or not saying
something conventional wisdom is still alive.
Rainer
| |
| Brian Raiter 2006-07-24, 7:22 pm |
| >> Not at all. Quite often this is exactly what you want programs to do:
>
> No, it's exactly what you DONT want when multiplexing fds!
Obviously in that particular situation you don't want to block. And
there are many other situations where blocking is good and the right
thing to do.
>
> Fair or not, my point is, that it is simply not possible to write
> really reliable software on what the standard seems to say.
I defer here to the other responses.
b
| |
| Barry Margolin 2006-07-24, 7:22 pm |
| In article <ea38ri$asq$1@online.de>,
Rainer Temme <Rainer.Temme@NoSpam.Siemens.Com> wrote:
> BTW: the behaviour in discussion is noted as 'BUG' in the linux
> man-pages. Besides the standard saying something or not saying
> something conventional wisdom is still alive.
It's long-standing tradition in Unix/Linux to describe undesirable
features as "BUG" in the man pages. For instance, I remember seeing man
pages for "find" that said that its non-traditional option style is a
bug.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
|
|
|
|
|