Unix Programming - fdopen, sockets, stdio

Author

fdopen, sockets, stdio

Daniel C. Bastos

2005-10-24, 3:48 pm

Do you guys use the stdio to write to and read from sockets? I've been
learning the techniques of DJB, for example, but I don't think that's
a good idea for myself. I was wondering if fdopen() could be enough
for me if I use FILE *out and FILE *in in order to avoid problems with
buffering (like reading after writing without flushing.)

I wonder how you guys do it... Do you guys write your own library or
you use the stdio making sure you avoid the buffering problems or
what? Any advice is interesting.

I am not very fond of the stdio, but I don't think I have reason for
it. I remember reading Stevens and he was warning of the problems, but
the problems are solvable, right? He seems to use the stdio himself by
working around the problems. Also, DJB seems to criticize the stdio; I
have not seen him using any libraries, except the libc and his own
stuff. I like how easy life gets when I use the stdio.

Michael Wojcik

2005-10-25, 5:54 pm

In article <slrndliac4.l7l.dbastos@my.free.dom.to>, "Daniel C. Bastos" <dbastos@my.free.dom.to> writes:
> Do you guys use the stdio to write to and read from sockets?

I don't, and I've been writing sockets-based software professionally
for over fifteen years. From past threads on comp.protocols.tcp-ip,
it's my impression that using stdio for socket I/O is neither
widespread nor widely recommended.

> I was wondering if fdopen() could be enough
> for me if I use FILE *out and FILE *in in order to avoid problems with
> buffering (like reading after writing without flushing.)

Using two separate FILE* objects won't prevent buffering issues.

For example: if you try to read from a FILE* that's associated with a
socket and has full buffering enabled, and the peer sends less data
than the buffer size but does not close the connection (eg because
it's waiting for a response), then the read operation could hang
trying to fill the buffer.

Similarly, if you're reading from a FILE* which has line buffering
enabled, and the peer tries to send a complete line but the network
connection fails after only part of the line is transmitted, you'll
hang waiting for a newline that will never arrive. Unless you have
unacked sends or keepalives are enabled, that read will hang forever.

AFAICT it's not safe to use stdio with a TCP stream unless buffering
is disabled for those FILE* objects; and if buffering is disabled,
why use stdio?

> I wonder how you guys do it... Do you guys write your own library or
> you use the stdio making sure you avoid the buffering problems or
> what? Any advice is interesting.

I write basic functions that call the socket API directly and
manipulate buffers in a manner suitable for my application(s). Some
of those are in libraries; some are just a module linked into an
application. It depends on the application.

> I like how easy life gets when I use the stdio.

I don't believe stdio makes socket I/O significantly easier, if you
want reliable and safe code. send and recv are not tricky functions.
And if you need anything more sophisticated - such as multiplexing
with select or poll, or scatter/gather I/O, etc - then stdio just
gets in the way.

--
Michael Wojcik michael.wojcik@microfocus.com

Do we have boyfriends? We are interested in delicious food and sweets.
And tiny animals like the cat. -- Naoko Yamano

Barry Margolin

2005-10-26, 2:49 am

In article <djmbeh0hfe@news2.newsguy.com>,
mwojcik@newsguy.com (Michael Wojcik) wrote:

> In article <slrndliac4.l7l.dbastos@my.free.dom.to>, "Daniel C. Bastos"
> <dbastos@my.free.dom.to> writes:
>
> I don't, and I've been writing sockets-based software professionally
> for over fifteen years. From past threads on comp.protocols.tcp-ip,
> it's my impression that using stdio for socket I/O is neither
> widespread nor widely recommended.
>
>
> Using two separate FILE* objects won't prevent buffering issues.
>
> For example: if you try to read from a FILE* that's associated with a
> socket and has full buffering enabled, and the peer sends less data
> than the buffer size but does not close the connection (eg because
> it's waiting for a response), then the read operation could hang
> trying to fill the buffer.

stdio functions don't usually wait for the buffer to fill. If they did,
programs that use stdio to read from stdin would hang when it's
connected to the terminal.

>
> Similarly, if you're reading from a FILE* which has line buffering

Line buffering is an *output* feature, not an input feature.

>
> I don't believe stdio makes socket I/O significantly easier, if you
> want reliable and safe code.

It's easier to use stdio than to write your own buffering functions.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Michael Wojcik

2005-10-28, 4:53 pm

In article <barmar-3791EA.22162825102005@comcast.dca.giganews.com>, Barry Margolin <barmar@alum.mit.edu> writes:
> In article <djmbeh0hfe@news2.newsguy.com>,
> mwojcik@newsguy.com (Michael Wojcik) wrote:
>
> stdio functions don't usually wait for the buffer to fill. If they did,
> programs that use stdio to read from stdin would hang when it's
> connected to the terminal.

stdin, when connected to an interactive device, will not be fully-
buffered:

As initially opened, the standard error stream is not fully
buffered; the standard input and standard output steams are
fully buffered if and only if the stream can be determined
not to refer to an interactive device.
(ISO 9899:1999 7.19.3 #7)

So the example of stdin connected to a terminal is irrelevant.

stdin is typically line buffered when connected to a terminal
device:

Line buffering is typically used on a stream when it refers
to a terminal (e.g., standard input and standard output).
(Stevens, APUE, 123)

and that does indeed cause read functions to wait for a complete
line, or a full buffer. This is trivially demonstrated by a
program that calls getchar() and prints the result; run it
interactively, type a non-EOL character, and it will wait until
you provide a newline.

>
> Line buffering is an *output* feature, not an input feature.

What a bizarre claim. It's most certainly available for both
input and output, and is the default for stdin when connected to
a terminal.

>
> It's easier to use stdio than to write your own buffering functions.

My point is that what you save in avoiding writing your own
buffering functions is less than what you lose in correct
behavior and control.

--
Michael Wojcik michael.wojcik@microfocus.com

Thanatos, thanatos! The labourer, dropping his lever,
Hides a black letter close to his heart and goes,
Thanatos, thanatos, home for the day and for ever. -- George Barker

Michael Wojcik

2005-10-28, 4:53 pm

In article <djqsmb01i59@news4.newsguy.com>, mwojcik@newsguy.com (Michael Wojcik) writes:
> In article <barmar-3791EA.22162825102005@comcast.dca.giganews.com>, Barry Margolin <barmar@alum.mit.edu> writes:
>
> stdin, when connected to an interactive device, will not be fully-
> buffered:
>
> As initially opened, the standard error stream is not fully
> buffered; the standard input and standard output steams are
> fully buffered if and only if the stream can be determined
> not to refer to an interactive device.
> (ISO 9899:1999 7.19.3 #7)
>
> So the example of stdin connected to a terminal is irrelevant.
>
> stdin is typically line buffered when connected to a terminal
> device:
>
> Line buffering is typically used on a stream when it refers
> to a terminal (e.g., standard input and standard output).
> (Stevens, APUE, 123)
>
> and that does indeed cause read functions to wait for a complete
> line, or a full buffer. This is trivially demonstrated by a
> program that calls getchar() and prints the result; run it
> interactively, type a non-EOL character, and it will wait until
> you provide a newline.

Argh. Obviously that "trivial demonstration" is trivially wrong;
that's tty buffering, not stdio buffering. While stdin is line-
buffered by default, that doesn't prevent getchar() from returning
a single character before a newline has been encountered; and I
believe you're correct insofar as line- and full-buffering have
the same effect on input.

My apologies.

What's odd is that I could *swear* that we had this discussion
years ago in comp.protocols.tcp-ip, that at that time *I*
recommended using stdio, and one of the regulars - and I thought
it was Barry - raised concerns about it. But Google shows
nothing of the sort.

--
Michael Wojcik michael.wojcik@microfocus.com

Advertising Copy in a Second Language Dept.:
The precious ovum itself is proof of the oath sworn to those who set
eyes upon Mokona: Your wishes will be granted if you are able to invest
it with eternal radiance... -- Noriyuki Zinguzi

Barry Margolin

2005-10-28, 4:53 pm

In article <djr4im01p2l@news4.newsguy.com>,
mwojcik@newsguy.com (Michael Wojcik) wrote:

> What's odd is that I could *swear* that we had this discussion
> years ago in comp.protocols.tcp-ip, that at that time *I*
> recommended using stdio, and one of the regulars - and I thought
> it was Barry - raised concerns about it. But Google shows
> nothing of the sort.

The usual concern over using fdopen() is with mixing stdio and system
calls. But if you stick to stdio functions once you call fdopen(), I
think you should generally be safe.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Daniel C. Bastos

2005-10-29, 7:47 am

In article <barmar-BBB9DE.22083427102005@comcast.dca.giganews.com>,
Barry Margolin wrote:

> In article <djr4im01p2l@news4.newsguy.com>,
> mwojcik@newsguy.com (Michael Wojcik) wrote:
>
>
> The usual concern over using fdopen() is with mixing stdio and system
> calls. But if you stick to stdio functions once you call fdopen(), I
> think you should generally be safe.

I don't clearly know what you mean by mixing system calls with stdio.

To read, I'm calling read() and extracting line by line and calling
read() again on demand. To write, I fdopen() using FILE* out, then I
fprintf() and fflush() when I need. So the reading buffer is separate
from the writing buffer and I only use the stdio for writing.

Is this to be mixing system calls and stdio?

Barry Margolin

2005-10-31, 2:49 am

In article <slrndm68qu.160l.dbast0s@my.free.dom.to>,
"Daniel C. Bastos" <dbast0s@yahoo.com.br> wrote:

> In article <barmar-BBB9DE.22083427102005@comcast.dca.giganews.com>,
> Barry Margolin wrote:
>
>
> I don't clearly know what you mean by mixing system calls with stdio.

read(), write(), select(), send(), and recv() are Unix system calls.

fprintf(), fputc(), fgets(), and fgetc() are stdio functions.

Trying to use functions from both sets on the same descriptor is tricky.

>
> To read, I'm calling read() and extracting line by line and calling
> read() again on demand. To write, I fdopen() using FILE* out, then I
> fprintf() and fflush() when I need. So the reading buffer is separate
> from the writing buffer and I only use the stdio for writing.
>
> Is this to be mixing system calls and stdio?

That should be pretty safe, since the two directions are independent.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Brian Raiter

2005-10-31, 8:50 pm

>> To read, I'm calling read() and extracting line by line and calling
>
> That should be pretty safe, since the two directions are independent.

Though potentially rather confusing. At the least I would comment such
code carefully.

b