|
Home > Archive > Unix Programming > September 2004 > Do pipes buffer?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
|
|
|
| I'm a little confused on how the buffering of data sent to descriptors
created by pipe() and socketpair() work. They are buffered, right? So
if I write just a few bytes to such a pipe, the write won't block,
even in blocking mode, right? I'm a little confused over the issue of
flushing, though. I've read that if I did write too few bytes to a
pipe, those bytes will not be committed until the buffer fills, or is
flushed. Is this true? If so, that sucks, because I'd be writing 4
bytes to a pipe, expecting them to be read very soon, and can't wait
for the buffer to flush. And the select() in the receiving process
won't react because it doesn't see anything to be read yet.
| |
| Jens.Toerring@physik.fu-berlin.de 2004-09-22, 9:22 pm |
| In comp.unix.programmer Jo <JoJoTwilligo@hotmail.com> wrote:
> I'm a little confused on how the buffering of data sent to descriptors
> created by pipe() and socketpair() work. They are buffered, right? So
> if I write just a few bytes to such a pipe, the write won't block,
> even in blocking mode, right?
Yes, typically a pipe has a few kB of space you can write to with-
out the other side reading anything before it writes start to block.
> I'm a little confused over the issue of
> flushing, though. I've read that if I did write too few bytes to a
> pipe, those bytes will not be committed until the buffer fills, or is
> flushed. Is this true?
Write how? What you're writing looks like the behavior of standard
C functions like printf(3) etc. that do internal buffering instead
of writing the data out immediately. If you're using one of these
functions you either have to call fflush(3) in order to get this
buffer (the internal buffer of the f/printf() functions) written
out or to switch off internal buffering using setvbuf(3).
But that's not an issue with the write(2) function - it doesn't
buffer anything. And it's also not an issue with the pipe - what
you have written to it is immediately available to the other side
for reading, the buffer is only there to allow a certain amount
of data to be written to the pipe without writes blocking.
> If so, that sucks, because I'd be writing 4
> bytes to a pipe, expecting them to be read very soon, and can't wait
> for the buffer to flush. And the select() in the receiving process
> won't react because it doesn't see anything to be read yet.
If you use write(2) to put data into the pipe that won't be a
problem and select() will get triggered even if there's only
a single byte available to be read.
Regards, Jens
--
\ Jens Thoms Toerring ___ Jens.Toerring@physik.fu-berlin.de
\__________________________ http://www.toerring.de
| |
|
| Jo wrote:
> I'm a little confused on how the buffering of data sent to
descriptors
> created by pipe() and socketpair() work. They are buffered, right? So
Right.
> if I write just a few bytes to such a pipe, the write won't block,
> even in blocking mode, right? I'm a little confused over the issue of
> flushing, though. I've read that if I did write too few bytes to a
> pipe, those bytes will not be committed until the buffer fills, or is
> flushed. Is this true? If so, that sucks, because I'd be writing 4
> bytes to a pipe, expecting them to be read very soon, and can't wait
That just depends on the kind of I/O that you are using. If you are
using std i/o then it will buffered, otherwise not.
Cheers,
-Manu
------------
Manu Garg
http://manugarg.freezope.org
| |
| William Ahern 2004-09-22, 9:22 pm |
| In comp.os.linux.development.apps Jo <JoJoTwilligo@hotmail.com> wrote:
> I'm a little confused on how the buffering of data sent to descriptors
> created by pipe() and socketpair() work. They are buffered, right? So
> if I write just a few bytes to such a pipe, the write won't block,
> even in blocking mode, right? I'm a little confused over the issue of
> flushing, though. I've read that if I did write too few bytes to a
> pipe, those bytes will not be committed until the buffer fills, or is
> flushed. Is this true? If so, that sucks, because I'd be writing 4
> bytes to a pipe, expecting them to be read very soon, and can't wait
> for the buffer to flush. And the select() in the receiving process
> won't react because it doesn't see anything to be read yet.
There are two I/O interfaces: stdio and Unix. The stdio interface is the
portable ISO/ANSI C interface: fopen(3), fclose(3), fread(3), fwrite(3),
printf(3), fprintf(3), getchar(3), putchar(3), etc. These should be
available on any hosted C implementation, regardless of operating system.
The Unix interface includes open(2), close(2), read(2), write(2), pipe(2),
socketpair(2), etc. Something like fdopen(3) spans both (though isn't part
of the ISO/ANSI C interface itself, but is rather a Unix API).
The stdio interface can be buffered, and typically on Unix will be buffered
by default. You can disable stdio buffering--or change the buffer type or
size--by using setvbuf(3).
In short, the Unix interfaces do not buffer, the stdio interfaces can
buffer--line buffering when attached to a terminal, fixed-size buffering
otherwise. On Unix the stdio interfaces use the Unix interface internally,
and maximize the efficiency of calling the heavier weight Unix system calls
by queueing data into larger chunks. This is why it's not horribly slowly to
read in a large file or stream using the very useful getchar(3).
It is generally not recommended to mix the use of the two interfaces. Stick
to one or the other. Using both on the same underlying descriptor--abusing
fileno(3) or fdopen(3)--can result in very unexpected behavior.
- Bill
| |
| William Ahern 2004-09-22, 9:22 pm |
| In comp.os.linux.development.apps William Ahern <william@wilbur.25thandclement.com> wrote:
> In short, the Unix interfaces do not buffer, the stdio interfaces can
> buffer--line buffering when attached to a terminal, fixed-size buffering
> otherwise. On Unix the stdio interfaces use the Unix interface internally,
> and maximize the efficiency of calling the heavier weight Unix system calls
> by queueing data into larger chunks. This is why it's not horribly slowly to
> read in a large file or stream using the very useful getchar(3).
Oops. As others have pointed out, the kernel does have an internal buffer.
From an application perspective though, and given a single reader and
writer, this buffering is more-or-less inconsequential regarding how you
construct your program. It becomes an issue when you enter the realm of
multiple readers and writers.
| |
|
| Great, thanks guys. I was pretty sure, but now I'm absolutely sure.
|
|
|
|
|