|
Home > Archive > Unix Programming > January 2004 > writing to unbuffered stream
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
writing to unbuffered stream
|
|
| john k. 2004-01-23, 5:15 pm |
| I have an Solaris 2.8 application (AppX) that flattens an incoming
stream of structured data, and writes the result to stdout using
printf(). I observe a periodic loss of data when AppX stdout is
directed to a FIFO file (named pipe), and the FIFO file is then read
into a regular file. For example:
AppX(1) > file.out // Both AppX instances receive
AppX(2) > FIFO_FILE // the same stream of input.
cat FIFO_FIFO > fifo.out
90% of the time, file.out == fifo.out.
printf() man page indicates possible failure if writing to an
unbuffered stream; at time of failure, printf() starts returning "-1"
but errno == 0
Q: Is there a prescribed method for writing to an unbuffered stream?
thank you,
-john
| |
| Barry Margolin 2004-01-23, 5:15 pm |
| In article <cd4a1fe2.0311051237.6d67eea9@posting.google.com>,
john k. <john_keeling@agilent.com> wrote:quote:
>I have an Solaris 2.8 application (AppX) that flattens an incoming
>stream of structured data, and writes the result to stdout using
>printf(). I observe a periodic loss of data when AppX stdout is
>directed to a FIFO file (named pipe), and the FIFO file is then read
>into a regular file. For example:
>
> AppX(1) > file.out // Both AppX instances receive
> AppX(2) > FIFO_FILE // the same stream of input.
> cat FIFO_FIFO > fifo.out
>
>
>90% of the time, file.out == fifo.out.
>
>printf() man page indicates possible failure if writing to an
>unbuffered stream; at time of failure, printf() starts returning "-1"
>but errno == 0
My man page says the failure will only happen if one of these errors is
encountered:
EFBIG The file is a regular file and an attempt was
made to write at or beyond the offset max-
imum.
EILSEQ An invalid character has been detected.
quote:
>Q: Is there a prescribed method for writing to an unbuffered stream?
What makes you think the streams are unbuffered? By default, all stdio
streams are buffered; they're line-buffered if writing to a terminal,
fully-buffered otherwise. Since none of your processes are writing to a
terminal, they're all fully-buffered unless the application uses setbuf()
to disable buffering.
Anyway, using unbuffered streams is usually the *solution* to loss of data.
With a buffered stream, if the process is killed, anything in the buffer is
lost, but if the stream is unbuffered then it is always written
immediately. If you're losing data, my guess is that the streams are
buffered, and something killed AppX(2) before it had a chance to flush its
buffers.
--
Barry Margolin, barry.margolin@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
| |
| Chuck Dillon 2004-01-23, 5:16 pm |
| john k. wrote:
quote:
> I have an Solaris 2.8 application (AppX) that flattens an incoming
> stream of structured data, and writes the result to stdout using
> printf(). I observe a periodic loss of data when AppX stdout is
> directed to a FIFO file (named pipe), and the FIFO file is then read
> into a regular file. For example:
What does periodic mean? Are there periodic problems in what is
generated by a single process or do you mean occasionally not all of
the data is there?
At any rate, I suggest you use truss to capture the system calls of
AppX(2) and see what it tells you.
-- ced
quote:
>
> AppX(1) > file.out // Both AppX instances receive
> AppX(2) > FIFO_FILE // the same stream of input.
> cat FIFO_FIFO > fifo.out
>
>
> 90% of the time, file.out == fifo.out.
>
> printf() man page indicates possible failure if writing to an
> unbuffered stream; at time of failure, printf() starts returning "-1"
> but errno == 0
>
> Q: Is there a prescribed method for writing to an unbuffered stream?
>
> thank you,
>
> -john
--
Chuck Dillon
Senior Software Engineer
NimbleGen Systems Inc.
| |
| john k. 2004-01-23, 5:16 pm |
| Some addition information on the problem:
* AppX(2) isn't terminating unexpectedly; it actually completes
execution. The missing data has thusfar always occurred ~middle of the
data stream. For example, in one test run involving 100,000 records,
records 17762 thru 17807 were missing.
* At onset of problem, printf() starts returning "-1"; errno = 0. Even
though data is eventually written to stdout, printf() continues
returning "-1" for remainder of program execution.
| |
| Geoff Clare 2004-01-23, 5:17 pm |
| Barry Margolin <barry.margolin@level3.com> writes:
quote:
[QUOTE][color=darkred]
>My man page says the failure will only happen if one of these errors is
>encountered:
quote:
> EFBIG The file is a regular file and an attempt was
> made to write at or beyond the offset max-
> imum.
quote:
> EILSEQ An invalid character has been detected.
Are you sure you haven't missed a note in the man page that tells
you to refer to another man page (e.g. fputc) for most of the errors
that printf can encounter? There are lots more errors for printf
than just the above two.
--
Geoff Clare <nospam@gclare.org.uk>
|
|
|
|
|