|
Home > Archive > Unix Programming > July 2004 > non blocking sockets
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
non blocking sockets
|
|
| j0mbolar 2004-07-23, 2:50 am |
| Is it more appropriate for an irc client to use non-blocking sockets?
if so, why? Currently my bot uses blocking sockets but I have been
told that non-blocking sockets are preferred except they never
provided any reason as to why they are preferred.
| |
| Lev Walkin 2004-07-23, 2:50 am |
| j0mbolar wrote:
> Is it more appropriate for an irc client to use non-blocking sockets?
yes.
> if so, why?
to avoid blocking (not THAT blocking, see below).
> Currently my bot uses blocking sockets but I have been
> told that non-blocking sockets are preferred except they never
> provided any reason as to why they are preferred.
Suppose:
- You want to write 8192 bytes of data into the socket.
- The socket buffer size is 4096 (kind of low, but illustrative)
- The IRC server is malicious.
If you're doing write(sockfd, buf, 8192), the client will block
until part of data is drained. If the IRC server is malicious, it
can prohibit receiving data by making its receive buffer zero.
Your TCP stack will not be able to push data to a remote system,
and your bot will freeze. This situation may persist for indefinite
amount of time.
In some variation of this scenario it is possible to harm your
client even if IRC server is not malicious.
--
Lev Walkin
vlm@lionet.info
| |
| Erik Max Francis 2004-07-23, 2:50 am |
| Lev Walkin wrote:
> If you're doing write(sockfd, buf, 8192), the client will block
> until part of data is drained. If the IRC server is malicious, it
> can prohibit receiving data by making its receive buffer zero.
> Your TCP stack will not be able to push data to a remote system,
> and your bot will freeze. This situation may persist for indefinite
> amount of time.
So only make the write call when you know the socket is ready for write,
which select/poll indicates for you.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ Love is not love which alters when it alteration finds
-- William Shakespeare, _Sonnets_, 116
| |
| Lev Walkin 2004-07-23, 2:50 am |
| Erik Max Francis wrote:
> Lev Walkin wrote:
>
>
>
>
> So only make the write call when you know the socket is ready for write,
> which select/poll indicates for you.
Select/poll indicates only _writeability_. It does not tell you anything
about ability to write the whole amount without blocking.
--
Lev Walkin
vlm@lionet.info
| |
| Erik Max Francis 2004-07-23, 2:50 am |
| Lev Walkin wrote:
> Select/poll indicates only _writeability_. It does not tell you
> anything
> about ability to write the whole amount without blocking.
That's why you only try to empty an application-side client buffer in
chunks that are smaller than the system-side socket buffer.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ Love is not love which alters when it alteration finds
-- William Shakespeare, _Sonnets_, 116
| |
| Frank Cusack 2004-07-23, 2:50 am |
| On Fri, 23 Jul 2004 01:37:12 -0700 Erik Max Francis <max@alcyone.com> wrote:
> Lev Walkin wrote:
>
>
> So only make the write call when you know the socket is ready for write,
> which select/poll indicates for you.
No it doesn't, select/poll indicates writability of at least 1 byte, only,
and only at the time it is called. By the time you actually do the write,
your call may block. When using select/poll, you must always use non-
blocking sockets.
/fc
| |
| Lev Walkin 2004-07-23, 7:50 am |
| Erik Max Francis wrote:
> Lev Walkin wrote:
>
>
>
>
> That's why you only try to empty an application-side client buffer in
> chunks that are smaller than the system-side socket buffer.
>
Okay. What about reads? You can easily block on read, when the remote
side is awaiting some data to be send from your process.
--
Lev Walkin
vlm@lionet.info
| |
| Erik Max Francis 2004-07-23, 7:50 am |
| Lev Walkin wrote:
> Okay. What about reads? You can easily block on read, when the remote
> side is awaiting some data to be send from your process.
Select/poll also indicates readability. If you don't try to read from
the socket unless select/poll says it's ready for reading, you never
block.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ Love is not love which alters when it alteration finds
-- William Shakespeare, _Sonnets_, 116
| |
| Lev Walkin 2004-07-23, 7:50 am |
| Erik Max Francis wrote:
> Lev Walkin wrote:
>
>
>
>
> Select/poll also indicates readability. If you don't try to read from
> the socket unless select/poll says it's ready for reading, you never
> block.
Never? Even if another thread or a process sharing this descriptor reads this
data before you get a chance to read it yourself?
Too many "but's".
--
Lev Walkin
vlm@lionet.info
| |
| Erik Max Francis 2004-07-23, 7:50 am |
| Lev Walkin wrote:
> Never? Even if another thread or a process sharing this descriptor
> reads this
> data before you get a chance to read it yourself?
Only one thread and one process should be processing sockets. There are
race conditions all over the place if that's violated, and to fix those
you need mutexes for each socket which gets incredibly expensive.
One process, one thread handles socket transactions. That process has
application-side read and write buffers, and uses select/poll to read
and write from ready sockets when select/poll indicates that they're
ready for read or write. That's all.
> Too many "but's".
You're the one inventing the _buts_. They're not relevant.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ Adultery is the application of democracy to love.
-- H.L. Mencken
| |
| Lev Walkin 2004-07-23, 7:50 am |
| Erik Max Francis wrote:
> Lev Walkin wrote:
>
>
>
> You're the one inventing the _buts_. They're not relevant.
Wasn't it you who suggested the following?
=== cut ===
That's why you only try to empty an application-side client buffer in
chunks that are smaller than the system-side socket buffer.
=== cut ===
Chunks that are smaller than "the system-side socket buffer", is a pretty
big "but". Considering that it is not enough: you must ensure that your
chunk is smaller than amount of _free space_ in this buffer, not the
buffer size itself.
Unless you're doing a micro-management of your buffers by doing setsockopts(),
setting watermark, and other stuff, you cannot expect this to hold true in
every environment where your program operates. And if you do this, you're
lowering portability with each additional setsockopt(). The cost of _making
sure_ that you're doing it completely right is low, yet is non-zero, and
easily exceeds the cost of writing your application using the non-blocking
socket in the first place.
I didn't say it was impossible to use blocking sockets. Just merely
"more appropriate" in case of an irc client. Will you continue fighting
this point of view?
--
Lev Walkin
vlm@lionet.info
| |
| Erik Max Francis 2004-07-23, 7:50 am |
| Lev Walkin wrote:
> Chunks that are smaller than "the system-side socket buffer", is a
> pretty
> big "but". Considering that it is not enough: you must ensure that
> your
> chunk is smaller than amount of _free space_ in this buffer, not the
> buffer size itself.
No, that's why write(2) returns the number of bytes successfully
written. If there isn't enough space in the socket-side buffer, it only
writes what it can. Select/poll says you can write, you write, it fills
the socket-side buffer, tells you where it left off. write(2) returns a
value for a reason; you're suggesting ignoring it.
> I didn't say it was impossible to use blocking sockets. Just merely
> "more appropriate" in case of an irc client. Will you continue
> fighting
> this point of view?
Yes, since it's wrong.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ Adultery is the application of democracy to love.
-- H.L. Mencken
| |
| Alex Fraser 2004-07-23, 5:53 pm |
| "j0mbolar" <j0mbolar@engineer.com> wrote in message
news:2d31a9f9.0407222355.51d650ad@posting.google.com...
> Is it more appropriate for an irc client to use non-blocking sockets?
> if so, why?
In the specific case of an IRC client, non-blocking sockets are strongly
preferred because they allow you to implement event-driven I/O which is
natural for the protocol.
> Currently my bot uses blocking sockets but I have been told that
> non-blocking sockets are preferred except they never provided any reason
> as to why they are preferred.
Can you see the difference between a synchronous protocol such as SMTP
(without extensions), where blocking sockets can be used in a
straight-forward manner, and an asynchronous protocol such as IRC?
Alex
| |
| James Antill 2004-07-23, 5:53 pm |
| On Fri, 23 Jul 2004 02:44:22 -0700, Erik Max Francis wrote:
> Lev Walkin wrote:
>
>
> No, that's why write(2) returns the number of bytes successfully
> written.
True.
> If there isn't enough space in the socket-side buffer, it only
> writes what it can.
Not true. SuS says:
If the O_NONBLOCK flag is clear, a write request may cause the thread to
block, but on normal completion it shall return nbyte.
....Ie. the thread/process can block until all submitted data is sent.
Maybe you were confused by the wording for writing to fds backed by files
where the physical media (Ie. disk) is full?
>
> Yes, since it's wrong.
No, it isn't. Even ignoring the inherent race condition of
calling poll/select and then calling read/write at some future point,
read/write with a blocking fd is allowed to block.
For a client you'd also want async. connect(), which also requires the fd
be set to non-blocking.
I'm confused though about why you'd go to all this trouble to implement
an event mechanism, local buffer storage etc. but not want to change your
fds to non-blocking?
--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/
| |
| Alex Fraser 2004-07-23, 5:53 pm |
| "James Antill" <james-netnews@and.org> wrote in message
news:pan.2004.07.23.14.52.05.527797@and.org...
> No, it isn't. Even ignoring the inherent race condition of
> calling poll/select and then calling read/write at some future point,
> read/write with a blocking fd is allowed to block.
write(), certainly, but is read() allowed to block? (It's at least mostly an
academic point, I know.)
Alex
| |
| Barry Margolin 2004-07-23, 5:53 pm |
| In article <2mcubkFlmd8rU1@uni-berlin.de>,
"Alex Fraser" <me@privacy.net> wrote:
> "James Antill" <james-netnews@and.org> wrote in message
> news:pan.2004.07.23.14.52.05.527797@and.org...
>
> write(), certainly, but is read() allowed to block? (It's at least mostly an
> academic point, I know.)
There *have* been reports in this newsgroup of cases where select()
indicated that a socket was ready to read, but a subsequent read()
blocked anyway. Some of these may have been programmer error (the
obvious case being multiple threads/processes reading from the same
socket), but I think there have been cases where we were stumped as to
the reason, and chalked it up to flukes in the kernel implementation.
So the common wisdom is to always use non-blocking I/O along with
select/poll, just to be safe. I can't really think of any good reason
you would *want* to use blocking I/O in these situations anyway.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| David Schwartz 2004-07-23, 5:53 pm |
|
"Barry Margolin" <barmar@alum.mit.edu> wrote in message
news:barmar-3A8CD3.13504123072004@comcast.dca.giganews.com...
> There *have* been reports in this newsgroup of cases where select()
> indicated that a socket was ready to read, but a subsequent read()
> blocked anyway. Some of these may have been programmer error (the
> obvious case being multiple threads/processes reading from the same
> socket), but I think there have been cases where we were stumped as to
> the reason, and chalked it up to flukes in the kernel implementation.
One possibility is that something else gobbled up the data. However, the
argument is simply that select is protocol-neutral and nothing stops someone
from designing a protocol where the other side can 'retract' data after it
has been sent but before it has been accepted by the other side. So, in
principle, the data could be 'retracted' before it has been received.
DS
| |
| j0mbolar 2004-07-24, 2:48 am |
| "Alex Fraser" <me@privacy.net> wrote in message news:<2mctcoFlmi6aU1@uni-berlin.de>...
> "j0mbolar" <j0mbolar@engineer.com> wrote in message
> news:2d31a9f9.0407222355.51d650ad@posting.google.com...
>
> In the specific case of an IRC client, non-blocking sockets are strongly
> preferred because they allow you to implement event-driven I/O which is
> natural for the protocol.
>
>
> Can you see the difference between a synchronous protocol such as SMTP
> (without extensions), where blocking sockets can be used in a
> straight-forward manner, and an asynchronous protocol such as IRC?
>
what constitutes a synchronus protocol and asynchronus protocol?
> Alex
| |
| Michael B Allen 2004-07-28, 6:19 pm |
| On Fri, 23 Jul 2004 13:50:41 -0400, Barry Margolin wrote:
> There *have* been reports in this newsgroup of cases where select()
> indicated that a socket was ready to read, but a subsequent read()
> blocked anyway. Some of these may have been programmer error (the
I don't think I'm going to beleive that until I see it myself.
> So the common wisdom is to always use non-blocking I/O along with
> select/poll, just to be safe. I can't really think of any good reason
> you would *want* to use blocking I/O in these situations anyway.
Both select/poll and non-blocking I/O are clumsy but select/poll is a
little less clumsy. With non-blocking I/O the code is awkward compared
to the traditional programming style. With select/poll you know you
can read/write a certain amount of data. And I would imagine it is more
efficient as the read/write-ability tests can be performed with 1 system
call for all descriptors as opposed to 1 for each descriptor. That might
be significant when you have 985 out of 1000 IRC clients doing nothing.
Mike
| |
| Michael B Allen 2004-07-28, 6:19 pm |
| On Fri, 23 Jul 2004 04:49:05 -0400, Frank Cusack wrote:
>
> No it doesn't, select/poll indicates writability of at least 1 byte,
> only, and only at the time it is called. By the time you actually do
> the write, your call may block. When using select/poll, you must always
> use non- blocking sockets.
What are you talking about? If the descriptor is writeable and there are
no other processes/threads involved it will still be writable. Other
than closing the descriptor or some kind of error occuring what is
going to happen that will make it non-writeable? You don't have to use
non-blocking sockets with select/poll. That would be somewhat redundant
as conceptually they do the same thing (facilitate I/O multiplexing).
Mike
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Michael B Allen wrote:
> On Fri, 23 Jul 2004 13:50:41 -0400, Barry Margolin wrote:
>
>
>
>
> I don't think I'm going to beleive that until I see it myself.
You don't have to experience an accident to start driving defensively.
Non-blocking I/O with select/poll is easy enough to consider it even
if you don't believe that using blocking I/O will hurt you.
>
>
> Both select/poll and non-blocking I/O are clumsy but select/poll is a
> little less clumsy. With non-blocking I/O the code is awkward compared
> to the traditional programming style.
First, non-blocking I/O does not make sense without select/poll or other
means to wait for an event. Asserting that select/poll is less clumsy
that non-blocking I/O is either too obvious, or borders with nonsense.
> With select/poll you know you can read/write a certain amount of data.
No. With select/poll you know that during the time the process has spent
doing select() or poll() system call, some data became available for
doing read() or some buffer space has been created so write was possible().
This guarantee is lost the moment you're out of select()/poll(), or even
earlier. Moreover, select/poll never guaranteed that you can read() or
write() even a single byte. "Ready for reading" != "can read certain amount".
The thing that saves everybody, is that the window between select() and
read()/write() is small enough.
> And I would imagine it is more
> efficient as the read/write-ability tests can be performed with 1 system
> call for all descriptors as opposed to 1 for each descriptor. That might
> be significant when you have 985 out of 1000 IRC clients doing nothing.
Yes, but you seem to ignore the fact that select/poll are used for
non-blocking I/O most natively, so this paragraph also holds true for
non-blocking approach.
P.S. For 985 out of 1000 clients doing nothing, it is better to consider
kqueue, /dev/poll or rt signals instead of select/poll.
--
Lev Walkin
vlm@lionet.info
| |
| Michael B Allen 2004-07-28, 6:19 pm |
| On Fri, 23 Jul 2004 12:40:54 -0400, Alex Fraser wrote:
>
> In the specific case of an IRC client, non-blocking sockets are strongly
> preferred because they allow you to implement event-driven I/O which is
> natural for the protocol.
Actually select is conceptually more "event-driven" than non-blocking
I/O. With non-blocking I/O you have to try to read/write the descriptor
only to fail and waste a system call. Select and poll are specifically
designed ot communiate I/O events from the kernel.
In practice you could write a very good IRC server using either
method. Everything else being equal I would not dare predict which
method would actually work better. I suspect the real argument for
using one method over the other would be code quality, readability,
maintainability, portability, etc. Meaning if everyone else on my team
claimed non-blocking I/O was definitely the way to go then I'd probably
keep my mouth shut and play along.
Mike
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Michael B Allen wrote:
> On Fri, 23 Jul 2004 04:49:05 -0400, Frank Cusack wrote:
>
>
>
> What are you talking about? If the descriptor is writeable and there are
> no other processes/threads involved it will still be writable.
Not true.
> Other than closing the descriptor or some kind of error occuring
That's why, at least.
> what is going to happen that will make it non-writeable?
You've eliminated closing the descriptor and some kind of error occuring
on the connection. I can add at least one more: somebody had decreased
the amount of buffers the kernel allows for socket to tie up. Consequently,
if this happens between a select()/poll() and a write(), write() will not
be able to write anything, because there is no space at this moment.
Note: this is not an error, and the socket is not closed. This says that
you've missed at least one case where the expected select() semantics
does not guarantee that you can write without blocking. There are more.
This argument alone is sufficient to make conscious programmers to always
use non-blocking programming with select/poll.
> You don't have to use
> non-blocking sockets with select/poll. That would be somewhat redundant
> as conceptually they do the same thing (facilitate I/O multiplexing).
Please note that select/poll with non-blocking sockets is virtually
indistinguishable from select/poll with blocking sockets. That's from
a programming perspective. Given that there _might_ be the case that
a program may block unexpectedly using s/p+blocking approach, the
cost of converting program to use s/p+non-blocking may be as low
as adding a single fcntl() when the socket is being created + reacting
on EAGAIN/EWOULDBLOCK in two places.
By using that simple fcntl(), you're basically granting your program
an eternal safety from blocking issues.
What's the point of arguing about virtues of s/p+blocking, then?
--
Lev Walkin
vlm@lionet.info
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Michael B Allen wrote:
> On Fri, 23 Jul 2004 12:40:54 -0400, Alex Fraser wrote:
>
>
>
> Actually select is conceptually more "event-driven" than non-blocking
> I/O. With non-blocking I/O you have to try to read/write the descriptor
> only to fail and waste a system call. Select and poll are specifically
> designed ot communiate I/O events from the kernel.
Mike, nobody tries to say that non-blocking is better then select!
non-blocking approach USES select/poll. You're missing the point.
--
Lev Walkin
vlm@lionet.info
| |
| Michael B Allen 2004-07-28, 6:19 pm |
| On Sat, 24 Jul 2004 05:12:14 -0400, Lev Walkin wrote:
>
> No. With select/poll you know that during the time the process has spent
> doing select() or poll() system call, some data became available for
> doing read() or some buffer space has been created so write was
> possible(). This guarantee is lost the moment you're out of
> select()/poll(), or even earlier. Moreover, select/poll never guaranteed
> that you can read() or write() even a single byte. "Ready for reading"
> != "can read certain amount".
>
> The thing that saves everybody, is that the window between select() and
> read()/write() is small enough.
If select indicates that there is data to read then provided the
descriptor is not closed or in error the data is guaranteed to be
there. Where's it going to go? If by the time the caller actually gets
around to reading the data the kernel is busy doing something else then
yes, the caller will wait. But the caller will not *block*. Blocking is a
condition that can be indefiniate if no more data is available. Waiting
because the kernel is doing some strange DMA operation is a different
thing entirely. And non-blocking I/O is subject to that behavior too
albeit to a lesser extent.
Mike
| |
| Michael B Allen 2004-07-28, 6:19 pm |
| On Sat, 24 Jul 2004 05:20:32 -0400, Lev Walkin wrote:
>
> You've eliminated closing the descriptor and some kind of error occuring
> on the connection. I can add at least one more: somebody had decreased
> the amount of buffers the kernel allows for socket to tie up.
> Consequently, if this happens between a select()/poll() and a write(),
> write() will not be able to write anything, because there is no space at
> this moment.
Assuming that the kernel could dynamically change the size of used
socket buffers (which I seriously doubt) I don't see how such a thing
could take place so frequently that it would impact the porformace of
your application. Regardless, this is not blocking. It might cause a
delay but it would not block unless buffers became so scarce that the
machine would effectively be dead anyway.
Mike
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Michael B Allen wrote:
> On Sat, 24 Jul 2004 05:20:32 -0400, Lev Walkin wrote:
>
>
>
> Assuming that the kernel could dynamically change the size of used
> socket buffers (which I seriously doubt) I don't see how such a thing
> could take place so frequently that it would impact the porformace of
> your application. Regardless, this is not blocking. It might cause a
> delay but it would not block unless buffers became so scarce that the
> machine would effectively be dead anyway.
man setsockopt
--
Lev Walkin
vlm@lionet.info
| |
| Bjorn Reese 2004-07-28, 6:19 pm |
| On Fri, 23 Jul 2004 00:55:26 -0700, j0mbolar wrote:
> Is it more appropriate for an irc client to use non-blocking sockets?
> if so, why? Currently my bot uses blocking sockets but I have been
> told that non-blocking sockets are preferred except they never
> provided any reason as to why they are preferred.
If your client supports DCC (Direct Client to Client) then you
should use non-blocking connects, because the other client may
be too slow in establishing the connection, which may cause you
to react too slowly to a ping request from the server.
--
mail1dotstofanetdotdk
| |
| Alex Fraser 2004-07-28, 6:19 pm |
| "j0mbolar" <j0mbolar@engineer.com> wrote in message
news:2d31a9f9.0407232334.4132ae98@posting.google.com...
> "Alex Fraser" <me@privacy.net> wrote in message
> news:<2mctcoFlmi6aU1@uni-berlin.de>...
>
> what constitutes a synchronus protocol and asynchronus protocol?
In a synchronous protocol, the client sends a request and waits for the
response. Then (perhaps) sends another request and waits for the response to
that, and so on. For example, in SMTP, the client sends a MAIL command,
waits for the server to accept or reject the sender, then continues with
RCPT command(s), and so on.
In an asynchronous protocol, this strict sequence is not prescribed. For
example, an IRC client might send a JOIN command, but the next thing
received from the server could be a PRIVMSG. In short, either side can send
anything at any time.
Alex
| |
| Alex Fraser 2004-07-28, 6:19 pm |
| "Michael B Allen" <mba2000@ioplex.com> wrote in message
news:pan.2004.07.24.05.14.13.899463.23023@ioplex.com...
> On Fri, 23 Jul 2004 12:40:54 -0400, Alex Fraser wrote:
>
> Actually select is conceptually more "event-driven" than non-blocking
> I/O.
Yes; just to be clear, I was suggesting using select() (or poll(), or...) in
conjunction with non-blocking sockets.
Alex
| |
| Alex Fraser 2004-07-28, 6:19 pm |
| "Michael B Allen" <mba2000@ioplex.com> wrote in message
news:pan.2004.07.24.05.33.02.877172.23023@ioplex.com...
> If select indicates that there is data to read then provided the
> descriptor is not closed or in error the data is guaranteed to be
> there.
Where is this guarantee (with the implied guarantee that a subsequent
blocking read() would not block) documented? (This is rephrasing the
question I asked before.)
If it's not documented, it's not guaranteed as far as I'm concerned. Even
if - as in this case - I cannot see any reason for it to be other than as
you describe.
Alex
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "Alex Fraser" <me@privacy.net> writes:
> "Michael B Allen" <mba2000@ioplex.com> wrote in message
> news:pan.2004.07.24.05.33.02.877172.23023@ioplex.com...
>
> Where is this guarantee (with the implied guarantee that a
> subsequent blocking read() would not block) documented? (This is
> rephrasing the question I asked before.)
POSIX/SUS:
A descriptor shall be considered ready for reading when a call to an
input function with O_NONBLOCK clear would not block, whether or not
the function would transfer data successfully. (The function might
return data, an end-of-file indication, or an error other than one
indicating that it is blocked, and in each of these cases the
descriptor shall be considered ready for reading.)
A descriptor shall be considered ready for writing when a call to an
output function with O_NONBLOCK clear would not block, whether or
not the function would transfer data successfully.
http://www.opengroup.org/onlinepubs...ons/select.html
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| David Schwartz 2004-07-28, 6:19 pm |
|
<joe@invalid.address> wrote in message
news:m3smbfoa22.fsf@invalid.address...
> "Alex Fraser" <me@privacy.net> writes:
There is no guarantee a subsequent blocking read will not block. The
guarantee is only that it would not have blocked had it been issued at the
point when you were told the socket was ready.
[vbcol=seagreen]
> POSIX/SUS:
>
> A descriptor shall be considered ready for reading when a call to an
> input function with O_NONBLOCK clear would not block, whether or not
> the function would transfer data successfully. (The function might
> return data, an end-of-file indication, or an error other than one
> indicating that it is blocked, and in each of these cases the
> descriptor shall be considered ready for reading.)
This is an instantaneous thing. By "would not block", they mean it would
not block if it had been issued at the instant the determination was made.
The OS cannot guarantee what will happen in the future.
> A descriptor shall be considered ready for writing when a call to an
> output function with O_NONBLOCK clear would not block, whether or
> not the function would transfer data successfully.
Would not block at that instant.
DS
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "David Schwartz" <davids@webmaster.com> writes:
> <joe@invalid.address> wrote in message
> news:m3smbfoa22.fsf@invalid.address...
>
>
> There is no guarantee a subsequent blocking read will not
> block. The guarantee is only that it would not have blocked had it
> been issued at the point when you were told the socket was ready.
That's not what the standard says.
>
> This is an instantaneous thing. By "would not block", they mean
> it would not block if it had been issued at the instant the
> determination was made. The OS cannot guarantee what will happen in
> the future.
The standard doesn't use the term instant, it says it will not block.
What would "instant" mean here? Define how long an instant is, and
where that definition is documented.
You're talking about waiting for a page fault or something of the
kind, not blocking. What the standard says is that it will not
block. It's right there, read it. It says "would not block". If the
standard considered waiting for a page fault to be blocking, why would
it say "would not block"?
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| David Schwartz 2004-07-28, 6:19 pm |
|
<joe@invalid.address> wrote in message
news:m3n01no3h5.fsf@invalid.address...
> "David Schwartz" <davids@webmaster.com> writes:
[vbcol=seagreen]
> That's not what the standard says.
The standard doesn't have to say it doesn't provide a guarantee, it just
has to fail to provide that guarantee.
[vbcol=seagreen]
> The standard doesn't use the term instant, it says it will not block.
When?
> What would "instant" mean here? Define how long an instant is, and
> where that definition is documented.
The indications 'select' gives you are just like any other indication
you might get. You see your 'low oil' light come on. What does that mean? It
means that at some point your oil was low. If the indication is latched for
you (as select's is), then there's no guarantee your oil is still low. It
could have sloshed around or someone might have added some. At the instant
the system decided to give you a hit from select, a read would not have
blocked.
> You're talking about waiting for a page fault or something of the
> kind, not blocking. What the standard says is that it will not
> block. It's right there, read it. It says "would not block". If the
> standard considered waiting for a page fault to be blocking, why would
> it say "would not block"?
The indication is that an operation, had it been attempted at the
instant the indication was generated, would not have blocked. You cannot
localize the instant that the indication was generated any better than it
was after you called 'select' and before it returned. Somewhere in there,
the indication was generated, and at that time the operation would not have
blocked.
POSIX does not say that a future operation will not block, nor could it,
because there are plenty of things we know of that would cause a future
operation to block.
DS
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "David Schwartz" <davids@webmaster.com> writes:
> <joe@invalid.address> wrote in message
> news:m3n01no3h5.fsf@invalid.address...
>
[vbcol=seagreen]
>
>
> When?
Exactly.
>
> The indications 'select' gives you are just like any other
> indication you might get. You see your 'low oil' light come on. What
> does that mean? It means that at some point your oil was low. If the
> indication is latched for you (as select's is), then there's no
> guarantee your oil is still low. It could have sloshed around or
> someone might have added some. At the instant the system decided to
> give you a hit from select, a read would not have blocked.
Ignoring for the moment that I don't agree with the analogy, I'm still
waiting to hear what "instant" means, and where that definition is
documented.
>
> The indication is that an operation, had it been attempted at
> the instant the indication was generated, would not have
> blocked. You cannot localize the instant that the indication was
> generated any better than it was after you called 'select' and
> before it returned. Somewhere in there, the indication was
> generated, and at that time the operation would not have blocked.
There's that pesky word "instant" again. At the time that select()
determines a read would not block, it hasn't returned a value to the
caller yet. So how could the state at that "instant" be of any value
to the caller?
How could the caller attempt the read at that instant? Clearly it
can't, so it seems odd to me that the standard would make a statement
like "would not block" with reference to the return from select(), if
that return value should be considered to be out of date as soon as
you get it.
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "David Schwartz" <davids@webmaster.com> writes:
> <joe@invalid.address> wrote in message
> news:m3n01no3h5.fsf@invalid.address...
>
>
>
>
> The standard doesn't have to say it doesn't provide a guarantee,
> it just has to fail to provide that guarantee.
It looks to me like it does provide such a guarantee. Consider:
3.76 Blocked Process (or Thread)
A process (or thread) that is waiting for some condition (other than
the availability of a processor) to be satisfied before it can
continue execution.
3.77 Blocking
A property of an open file description that causes function calls
associated with it to wait for the requested action to be performed
before returning.
http://www.opengroup.org/onlinepubs....html#tag_03_76
Now, if this definition of blocking agrees with you, then the
following statement is completely worthless:
[vbcol=seagreen]
I don't see how it can possibly mean that the call to an input
function would have to happen at the same "instant" that select()
decided a call to an input function would not block, because it's
impossible to issue a call to an input function at that point.
I don't see how it can mean both things at once. Either the standard
doesn't include waiting for page faults and such to be blocking, or it
guarantees that these kinds of things will not happen.
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| Erik Max Francis 2004-07-28, 6:19 pm |
| joe@invalid.address wrote:
> I don't see how it can possibly mean that the call to an input
> function would have to happen at the same "instant" that select()
> decided a call to an input function would not block, because it's
> impossible to issue a call to an input function at that point.
>
> I don't see how it can mean both things at once. Either the standard
> doesn't include waiting for page faults and such to be blocking, or it
> guarantees that these kinds of things will not happen.
Indeed. You are running into a situation where people are trying to
stretch standardese to fit their preconceptions about how to use
blocking sockets with select/poll. If what they were saying were true,
then the "would not block" guarantees that select/poll make would be
totally useless, defeating a large part of the purpose for them (which
is using them with blocking sockets).
Thus, either the standards are pointless or they're reading them the
wrong way. In the grand scheme of things, it's far more likely that
it's the latter rather than the former.
--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
\__/ They make it a desert and call it peace.
-- Tacitus
| |
| David Schwartz 2004-07-28, 6:19 pm |
|
"Erik Max Francis" <max@alcyone.com> wrote in message
news:4104A546.BF5E8287@alcyone.com...
> joe@invalid.address wrote:
> Indeed. You are running into a situation where people are trying to
> stretch standardese to fit their preconceptions about how to use
> blocking sockets with select/poll. If what they were saying were true,
> then the "would not block" guarantees that select/poll make would be
> totally useless, defeating a large part of the purpose for them (which
> is using them with blocking sockets).
The only purpose of the 'select'/'poll' functions is to cue you that
something has changed with respect to the socket.
> Thus, either the standards are pointless or they're reading them the
> wrong way. In the grand scheme of things, it's far more likely that
> it's the latter rather than the former.
They're not pointless, they're just subtle. The standard does not
require the platform to guarantee that the condition will continue to remain
true, in fact, it's impossible for the platform to do this.
DS
| |
| David Schwartz 2004-07-28, 6:19 pm |
|
<joe@invalid.address> wrote in message
news:m3bri3nv4x.fsf@invalid.address...
> "David Schwartz" <davids@webmaster.com> writes:
[vbcol=seagreen]
> It looks to me like it does provide such a guarantee. Consider:
>
> 3.76 Blocked Process (or Thread)
>
> A process (or thread) that is waiting for some condition (other than
> the availability of a processor) to be satisfied before it can
> continue execution.
This is not intended to be a precise definition. The implementation can,
if it's able to, consider a socket to be blocked if accessing it would
require a page fault. However, it's not required to. The implementation, to
be useful, must be able to consider anything that requires waiting for a
totally external event (that could take arbitrarily long) to be blocking.
> 3.77 Blocking
>
> A property of an open file description that causes function calls
> associated with it to wait for the requested action to be performed
> before returning.
>
> http://www.opengroup.org/onlinepubs....html#tag_03_76
>
> Now, if this definition of blocking agrees with you, then the
> following statement is completely worthless:
Note that it's not very clear on what consitutes waiting. Something fast
and internal might or might not be considered waiting, but I think we can
all agree that waiting for the other side to source or sink data *must* be
considered waiting. Other things (like waiting for local memory to be
available) might or might not be considered blocking.
[vbcol=seagreen]
> I don't see how it can possibly mean that the call to an input
> function would have to happen at the same "instant" that select()
> decided a call to an input function would not block, because it's
> impossible to issue a call to an input function at that point.
I can't see how it could mean anything else. Every indication means the
condition it indicates was true at the time the indication was provided.
> I don't see how it can mean both things at once. Either the standard
> doesn't include waiting for page faults and such to be blocking, or it
> guarantees that these kinds of things will not happen.
The standard doesn't require an implementation to consider waiting for a
page fault or local memory to become available to be blocking. It allows an
implementation to be smart if it can be. What it most certainly does *not*
do is require a 'select' indication to reserve the memory needed to ensure a
future operation won't cause a block. It's perfectly legal to get a write
indication for select and have a future write return 'ENOMEM' because the
memory that allowed the write to complete is no longer available.
DS
| |
| James Antill 2004-07-28, 6:19 pm |
| On Sun, 25 Jul 2004 23:31:34 -0700, Erik Max Francis wrote:
> Indeed. You are running into a situation where people are trying to
> stretch standardese to fit their preconceptions about how to use
> blocking sockets with select/poll.
And, of course, you aren't ... and the many trying to inform you must
have no operational experience. *sigh*
> If what they were saying were true,
> then the "would not block" guarantees that select/poll make would be
> totally useless, defeating a large part of the purpose for them (which
> is using them with blocking sockets).
No, this is where you misunderstand. The design of select/poll are to
give hints, for you can call read/write on non-blocking fds
(otherwise you'd have to constantly loop through all fds calling both
variants). They were never meant as _guarantees_. The descriptions
specifically say "may", but hell just read what I'd written
previously in that if you poll for POLLOUT on an socket fd and then try
to write more than can fit _it'll block_.
You can test this, it's very easy, go do it now before replying that
we're all clueless. I'll even help you out, go get...
http://www.and.org/vstr/basic_cat.c.html
http://www.and.org/vstr/examples/ex_utils.h.html
....and paste the io_block() function into the cat example and call it
before the write() calls in full_write() and see whether it blocks in
write() or poll().
> Thus, either the standards are pointless or they're reading them the
> wrong way. In the grand scheme of things, it's far more likely that
> it's the latter rather than the former.
Or, just maybe, when using poll() it's assumed you are using non-blocking
sockets because:
1. It's no more painful than not doing, and has much better error paths if
(failing all else) you screw up the event propagation and read() or
write() twice etc.
2. It's assumed with blocking IO that the caller generally prefers
to wait until the entire IO completes (read() from sockets being the
exception here).
But, anyway, feel free to refuse me with actual facts ... like starting
with an explanation of how you do write() calls with more than 1 byte
lengths without just hoping and praying you don't block.
--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "David Schwartz" <davids@webmaster.com> writes:
> <joe@invalid.address> wrote in message
> news:m3bri3nv4x.fsf@invalid.address...
>
>
>
> This is not intended to be a precise definition. The
> implementation can, if it's able to, consider a socket to be blocked
> if accessing it would require a page fault. However, it's not
> required to. The implementation, to be useful, must be able to
> consider anything that requires waiting for a totally external event
> (that could take arbitrarily long) to be blocking.
I'm having a hard time because you keep making assertions like this
without giving any backing for them. I've asked a couple times where
your ideas are documented in the standard, but I don't see a response
to that. If these are your opinions about what the standard should
mean, that's ok with me, and we'll just agree to disagree.
However, what the standard *says* is that it won't block. Whatever you
want to say blocking is or isn't, the standard says it won't.
>
> Note that it's not very clear on what consitutes
> waiting.
No, but it says that it won't happen if select returns an indication
that the descriptor is ready to read. Note that select has to return
in order to do that. The calling code can't do anything until it gets
the return value from select. It specifically does not say (and cannot
mean) that the application can only rely on the return from select
before select returns the value.
> Something fast and internal might or might not be considered
> waiting, but I think we can all agree that waiting for the other
> side to source or sink data *must* be considered waiting. Other
> things (like waiting for local memory to be available) might or
> might not be considered blocking.
Sure, we can agree on that. That's what blocking normally means in my
experience.
>
>
> I can't see how it could mean anything else. Every indication
> means the condition it indicates was true at the time the indication
> was provided.
How could it mean that? How could it possibly mean that the return
from select is only valid before it's been returned?
>
> The standard doesn't require an implementation to consider
> waiting for a page fault or local memory to become available to be
> blocking. It allows an implementation to be smart if it can be. What
> it most certainly does *not* do is require a 'select' indication to
> reserve the memory needed to ensure a future operation won't cause a
> block. It's perfectly legal to get a write indication for select and
> have a future write return 'ENOMEM' because the memory that allowed
> the write to complete is no longer available.
Where is this documented?
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| Barry Margolin 2004-07-28, 6:19 pm |
| In article <m3zn5mmp6n.fsf@invalid.address>, joe@invalid.address wrote:
> However, what the standard *says* is that it won't block. Whatever you
> want to say blocking is or isn't, the standard says it won't.
The standard doesn't say "won't block", it says "wouldn't block". I.e.
it wouldn't have blocked at the time that select/poll determined that
the socket was ready.
The standard is meant to be read by people who are knowledgeable in the
design of operating systems. We know that blocking is based on dynamic
state, so it doesn't make sense to interpret the specification as
anything other than a claim about a potentially transient property of
the socket.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| Barry Margolin <barmar@alum.mit.edu> writes:
> In article <m3zn5mmp6n.fsf@invalid.address>, joe@invalid.address wrote:
>
>
> The standard doesn't say "won't block", it says "wouldn't block".
> I.e. it wouldn't have blocked at the time that select/poll
> determined that the socket was ready.
Well, maybe I'm missing something, but this is what I see:
Upon successful completion, the pselect() or select() function shall
modify the objects pointed to by the readfds, writefds, and errorfds
arguments to indicate which file descriptors are ready for reading,
ready for writing, or have an error condition pending, respectively,
The "shall" in that sentence indicates to me a guarantee that the
contents of the fd_set objects indicate whether or not the descriptors
are ready for reading, writing or have an exceptional condition
pending. Those contents are not available until after select returns.
Being ready for reading is further defined as:
A descriptor shall be considered ready for reading when a call to an
input function with O_NONBLOCK clear would not block, whether or not
the function would transfer data successfully. (The function might
return data, an end-of-file indication, or an error other than one
indicating that it is blocked, and in each of these cases the
descriptor shall be considered ready for reading.)
I understand this to say that whether or not a descriptor is ready for
reading isn't determined by some unknowable state before select
returns, but by the values in the fd_set object *after* it returns.
If the values in the fd_set objects are to be considered out of date
when they return, why would the standard define the values of those
objects to be the indicators of whether or not a read operation will
block?
> The standard is meant to be read by people who are knowledgeable in
> the design of operating systems. We know that blocking is based on
> dynamic state, so it doesn't make sense to interpret the
> specification as anything other than a claim about a potentially
> transient property of the socket.
I always thought the purpose of standards was to define the behavior
of what's being standardized, so that people could design and program
to a predictable way of doing things.
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| Barry Margolin 2004-07-28, 6:19 pm |
| In article <m3r7qymhsi.fsf@invalid.address>, joe@invalid.address wrote:
> Barry Margolin <barmar@alum.mit.edu> writes:
>
>
> Well, maybe I'm missing something, but this is what I see:
>
> Upon successful completion, the pselect() or select() function shall
> modify the objects pointed to by the readfds, writefds, and errorfds
> arguments to indicate which file descriptors are ready for reading,
> ready for writing, or have an error condition pending, respectively,
>
> The "shall" in that sentence indicates to me a guarantee that the
> contents of the fd_set objects indicate whether or not the descriptors
> are ready for reading, writing or have an exceptional condition
> pending. Those contents are not available until after select returns.
>
> Being ready for reading is further defined as:
>
> A descriptor shall be considered ready for reading when a call to an
> input function with O_NONBLOCK clear would not block, whether or not
> the function would transfer data successfully. (The function might
> return data, an end-of-file indication, or an error other than one
> indicating that it is blocked, and in each of these cases the
> descriptor shall be considered ready for reading.)
>
> I understand this to say that whether or not a descriptor is ready for
> reading isn't determined by some unknowable state before select
> returns, but by the values in the fd_set object *after* it returns.
But the determination is made while select() is running. So all it's
telling you is that the call would not have blocked at the time that
select() made the determination to add the FD to the result set. If
something about the descriptor changes after that, how could select()
possibly be expected to deal with it?
> If the values in the fd_set objects are to be considered out of date
> when they return, why would the standard define the values of those
> objects to be the indicators of whether or not a read operation will
> block?
The purpose of select() is to avoid trying to access all the *other*
descriptors. If select() didn't exist, you would have to iterate
through all the possible descriptors, checking whether they can be read
or written. Select() allows you to limit your scanning to the ones that
are believed to be ready. But it shouldn't be viewed as a guarantee
that you can read something.
I.e. it's just a way to optimize your programs.
>
> I always thought the purpose of standards was to define the behavior
> of what's being standardized, so that people could design and program
> to a predictable way of doing things.
Of course. But they have to be read in context. Just as you have to
understand English, you also have to understand how computers work. And
part of that is realizing that computers can't predict the future.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| Barry Margolin <barmar@alum.mit.edu> writes:
> In article <m3r7qymhsi.fsf@invalid.address>, joe@invalid.address
> wrote:
>
[vbcol=seagreen]
>
> But the determination is made while select() is running. So all
> it's telling you is that the call would not have blocked at the time
> that select() made the determination to add the FD to the result
> set. If something about the descriptor changes after that, how
> could select() possibly be expected to deal with it?
Well then I don't understand why the standard says what it does. Why
does the standard tell programmers that select shall return an fd_set
object which says whether or not a read on a descriptor will block?
The standard doesn't say that the values in the fd_set objects are
only hints, it says that they "shall" convey something. When the word
"shall" is used in a standard it conveys a guarantee, does it not?
>
> The purpose of select() is to avoid trying to access all the *other*
> descriptors. If select() didn't exist, you would have to iterate
> through all the possible descriptors, checking whether they can be
> read or written. Select() allows you to limit your scanning to the
> ones that are believed to be ready. But it shouldn't be viewed as a
> guarantee that you can read something.
>
> I.e. it's just a way to optimize your programs.
If the standard said that I'd be fine with it. However, as I've said
several times, that's not what the standard says. The standard says
that a bit in an fd_set will not be set unless the associated
descriptor is ready for reading/writing. The definition of ready for
reading/writing is
A descriptor shall be considered ready for reading when a call to an
input function with O_NONBLOCK clear would not block, whether or not
the function would transfer data successfully. (The function might
return data, an end-of-file indication, or an error other than one
indicating that it is blocked, and in each of these cases the
descriptor shall be considered ready for reading.)
The point I'm trying to make is that this is tied to what select
returns. It doesn't say that the bit means the descriptor was ready to
read at some time in the past, it says that if you get an fd_set
object with the bit set, it's ready to read.
If the standard really means something else, I think it needs to be
changed.
>
> Of course. But they have to be read in context. Just as you have
> to understand English, you also have to understand how computers
> work. And part of that is realizing that computers can't predict
> the future.
I don't expect computers to predict the future. I just expect that if
the standard says "shall" I can depend on it. It's then up to the
implementors of the OS to either be compliant or not.
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| David Schwartz 2004-07-28, 6:19 pm |
| <joe@invalid.address> wrote in message
news:m3r7qymhsi.fsf@invalid.address...
> Barry Margolin <barmar@alum.mit.edu> writes:
>
> Well, maybe I'm missing something, but this is what I see:
>
> Upon successful completion, the pselect() or select() function shall
> modify the objects pointed to by the readfds, writefds, and errorfds
> arguments to indicate which file descriptors are ready for reading,
> ready for writing, or have an error condition pending, respectively,
>
> The "shall" in that sentence indicates to me a guarantee that the
> contents of the fd_set objects indicate whether or not the descriptors
> are ready for reading, writing or have an exceptional condition
> pending. Those contents are not available until after select returns.
I'm sorry, that's utterly absurd. You might as well argue that 'stat'
guarantees that the size of the file can't change since you can't access the
size until after 'stat' returns and standard says that 'stat' shall return
the size of the file.
> Being ready for reading is further defined as:
>
> A descriptor shall be considered ready for reading when a call to an
> input function with O_NONBLOCK clear would not block, whether or not
> the function would transfer data successfully. (The function might
> return data, an end-of-file indication, or an error other than one
> indicating that it is blocked, and in each of these cases the
> descriptor shall be considered ready for reading.)
>
> I understand this to say that whether or not a descriptor is ready for
> reading isn't determined by some unknowable state before select
> returns, but by the values in the fd_set object *after* it returns.
I don't understand your understanding. Are you saying that the
implementation is prohibited from changing the state of the socket after
'select' returns? Until when?
> If the values in the fd_set objects are to be considered out of date
> when they return, why would the standard define the values of those
> objects to be the indicators of whether or not a read operation will
> block?
Because that's how every other function works. Why does 'stat' return
the size of a file if that might not be the size by the time you get around
to checking it?
[vbcol=seagreen]
> I always thought the purpose of standards was to define the behavior
> of what's being standardized, so that people could design and program
> to a predictable way of doing things.
You two are not disagreeing with each other. You are just assuming that
a guarantee exists when it does not. It's well known that the return value
of functions can be out of date by the time you get around to noticing those
values.
DS
| |
| Frank Cusack 2004-07-28, 6:19 pm |
| On Tue, 27 Jul 2004 03:04:51 GMT joe@invalid.address wrote:
> If the standard said that I'd be fine with it. However, as I've said
> several times, that's not what the standard says.
Yes it is, you are not thinking clearly. No non-atomic operations can
be guaranteed to succeed as an entire set. The future cannot be known.
> The standard says
> that a bit in an fd_set will not be set unless the associated
> descriptor is ready for reading/writing. The definition of ready for
> reading/writing is
>
> A descriptor shall be considered ready for reading when a call to an
> input function with O_NONBLOCK clear would not block, whether or not
> the function would transfer data successfully. (The function might
> return data, an end-of-file indication, or an error other than one
> indicating that it is blocked, and in each of these cases the
> descriptor shall be considered ready for reading.)
>
> The point I'm trying to make is that this is tied to what select
> returns. It doesn't say that the bit means the descriptor was ready to
> read at some time in the past, it says that if you get an fd_set
> object with the bit set, it's ready to read.
Right, not at some time in the past, at "now", ie, the time that
select returns. *Not* at some time in the future either, namely your
next call to read()/write().
> If the standard really means something else, I think it needs to be
> changed.
....
> I don't expect computers to predict the future. I just expect that if
> the standard says "shall" I can depend on it. It's then up to the
> implementors of the OS to either be compliant or not.
It's obvious that the standard is talking about "now", not "sometime
later when you get around to calling write()". The standard does not
need to be changed, it is as correct as possible as-is. It takes a
certain standard-fu to read it correctly, but there's no room for
interpretation here -- the future cannot be predicted and therefore
the standard cannot be talking about some future writability event.
Consider this:
client server
<- receive window 4000
write(3000) -> send
<- ack, receive window 1000
select() ... 1000 bytes avail,
return writability
*context switch*,
or app does stuff
<- ack, receive window 0
write(1) -> blocks
/fc
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Frank Cusack wrote:
>
> Consider this:
>
> client server
> <- receive window 4000
> write(3000) -> send
> <- ack, receive window 1000
> select() ... 1000 bytes avail,
> return writability
> *context switch*,
> or app does stuff
> <- ack, receive window 0
> write(1) -> blocks
write(1) won't block in this case: as there's room inside the kernel
buffer. you would have to shrink the send buffer before write()
using setsockopt() to hit this.
--
Lev Walkin
vlm@lionet.info
| |
| Frank Cusack 2004-07-28, 6:19 pm |
| On Mon, 26 Jul 2004 23:44:24 -0700 Lev Walkin <vlm@lionet.info> wrote:
> write(1) won't block in this case: as there's room inside the kernel
> buffer. you would have to shrink the send buffer before write()
> using setsockopt() to hit this.
Yup, I guess you're right, the sender would just buffer, but why would
write() block if you decreased the send buffer? It would just return
n-bufsize.
/fc
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| Frank Cusack wrote:
> On Mon, 26 Jul 2004 23:44:24 -0700 Lev Walkin <vlm@lionet.info> wrote:
>
>
>
> Yup, I guess you're right, the sender would just buffer, but why would
> write() block if you decreased the send buffer? It would just return
> n-bufsize.
because if you don't block if there's no space in the buffer, this
would be a non-blocking behavior, which contradicts with the problem
definition: "to make a _blocking_ socket block after select() has
indicated writeability". Note that we're working on a blocking socket,
so allowing write() to return with 0/-1 in this case is not an option:
this would be a non-blocking behavior.
--
Lev Walkin
vlm@lionet.info
| |
| James Antill 2004-07-28, 6:19 pm |
| On Mon, 26 Jul 2004 23:44:24 -0700, Lev Walkin wrote:
> write(1) won't block in this case: as there's room inside the kernel
> buffer. you would have to shrink the send buffer before write()
> using setsockopt() to hit this.
It's a wonderful delusion you're having, and I'm sure not testing your
theory helps a lot (or listening to anyone who has tested it). However
back in the real world, if the length of the write() is bigger than the
kernel buffer it WILL block if you are not in a non-blocking state.
--
James Antill -- james@and.org
Need an efficient and powerful string library for C?
http://www.and.org/vstr/
| |
| Lev Walkin 2004-07-28, 6:19 pm |
| James Antill wrote:
> On Mon, 26 Jul 2004 23:44:24 -0700, Lev Walkin wrote:
>
>
>
>
> It's a wonderful delusion you're having, and I'm sure not testing your
> theory helps a lot (or listening to anyone who has tested it). However
> back in the real world, if the length of the write() is bigger than the
> kernel buffer it WILL block if you are not in a non-blocking state.
It's a wonderful thing not to read the thread before answering to a paragraph
taken out of context.
I was very precise to point out that the length of write was LESSER than
the available buffer size, so no blocking would occur in this case. You seem
to have missed this bold print. Please review the thread before answering.
--
Lev Walkin
vlm@lionet.info
| |
| Barry Margolin 2004-07-28, 6:19 pm |
| In article <m3k6wqm9po.fsf@invalid.address>, joe@invalid.address wrote:
> Well then I don't understand why the standard says what it does. Why
> does the standard tell programmers that select shall return an fd_set
> object which says whether or not a read on a descriptor will block?
You keep saying "will not block", but the standard says "would not
block". They aren't the same tense, and the distinction is important.
--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
| |
| joe@invalid.address 2004-07-28, 6:19 pm |
| "David Schwartz" <davids@webmaster.com> writes:
> <joe@invalid.address> wrote in message
> news:m3r7qymhsi.fsf@invalid.address...
>
>
>
> I'm sorry, that's utterly absurd. You might as well argue that
> 'stat' guarantees that the size of the file can't change since you
> can't access the size until after 'stat' returns and standard says
> that 'stat' shall return the size of the file.
No, I wouldn't argue that, but it seems to me that the case of a
socket with data ready to read isn't analogous to the stat case.
Certainly something could change the file size between the time I call
stat() and the time I use the file size returned by stat(). However,
are you saying that data which has already been received on a tcp
connection can be discarded by the system and no longer be available
when read() is called?
I don't think that's what you're saying, I think what you're saying is
that the data which had been read may have been paged out to disk
before the call to read(). I don't disagree with that. It seems to me
that our disagreement is in what constitutes blocking. In my
experience, the word "blocking" is used when the function could be
delayed indefinitely. I don't see that waiting for a page fault is
like that.
The reason I think the distinction is important is that many
applications don't care about waiting for page faults, but they would
care about being delayed indefinitely. I've seen programmers write
code that exclusively uses non-blocking sockets because this
distinction isn't understood. The code winds up being more complicated
than it needs to be if the application isn't concerned with page
faults or other short duration waits, such as waiting to be scheduled
to run again.
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers
| |
| Frank Cusack 2004-07-28, 6:19 pm |
| On Wed, 28 Jul 2004 08:33:04 GMT joe@invalid.address wrote:
> The reason I think the distinction is important is that many
> applications don't care about waiting for page faults, but they would
> care about being delayed indefinitely. I've seen programmers write
> code that exclusively uses non-blocking sockets because this
> distinction isn't understood. The code winds up being more complicated
> than it needs to be if the application isn't concerned with page
> faults or other short duration waits, such as waiting to be scheduled
> to run again.
huh? non-blocking is not complicated at all. The select loop is the
complicated part, once you have that non-blocking is just testing for
EAGAIN. geez.
write() on a socket is just like send().
send() will block if there's not enough buffer space.
This has nothing to do with select().
You'd rather write the code to manage buffer space than handle EAGAIN???
How you're even going to know what buffer space is available, I have
no idea.
/fc
| |
| David Schwartz 2004-07-28, 6:19 pm |
|
<joe@invalid.address> wrote in message
news:m38yd44jlr.fsf@invalid.address...
> "David Schwartz" <davids@webmaster.com> writes:
>
[vbcol=seagreen]
> Certainly something could change the file size between the time I call
> stat() and the time I use the file size returned by stat(). However,
> are you saying that data which has already been received on a tcp
> connection can be discarded by the system and no longer be available
> when read() is called?
I'm sorry, who said anything about TCP? This is a protocol neutral
issue. There could exist a network protocol that allowed the sender to
'rescind' data before it was accepted by the receiving application. Would
you argue that 'select' could not be used with such a protocol? Or would you
argue that the implementation has to remember that there's been a read hit
on 'select' and act as if the data was accepted and lie to the other end?
(Even though one could 'select' on a socket for read just for, say,
statistical information about readiness and have no intention of ever
reading from it. After all, 'select' is level-based, not edge based.)
> I don't think that's what you're saying, I think what you're saying is
> that the data which had been read may have been paged out to disk
> before the call to read(). I don't disagree with that. It seems to me
> that our disagreement is in what constitutes blocking. In my
> experience, the word "blocking" is used when the function could be
> delayed indefinitely. I don't see that waiting for a page fault is
> like that.
If a system has the capability to treat a page in as blocking, POSIX
doesn't prohibit it from doing so. POSIX only requires an implementation to
consider a socket to be blocking if the delay could be indefinite (for
example, waiting for the other side to send data is clearly potentially
indefinite) but it doesn't prohibit the implementation from being smart
about things like paging.
But in any event, this isn't about paging. This isn't about TCP. This is
about whether POSIX requires an implementation to predict the future for
every possible situation with every possible protocool.
> The reason I think the distinction is important is that many
> applications don't care about waiting for page faults, but they would
> care about being delayed indefinitely. I've seen programmers write
> code that exclusively uses non-blocking sockets because this
> distinction isn't understood. The code winds up being more complicated
> than it needs to be if the application isn't concerned with page
> faults or other short duration waits, such as waiting to be scheduled
> to run again.
The distinction is important. But I've seen the reverse just as often --
servers that perform burstily because the programmer didn't take page faults
into account. However, this isn't about page faults. This is about whether
or not a specific guarantee exists in the standard, and my position is that
it doesn't.
DS
| |
| David Schwartz 2004-07-29, 8:47 pm |
|
"Erik Max Francis" <max@alcyone.com> wrote in message
news:4100D7DE.5FB8DE89@alcyone.com...
> Lev Walkin wrote:
[vbcol=seagreen]
> Only one thread and one process should be processing sockets. There are
> race conditions all over the place if that's violated, and to fix those
> you need mutexes for each socket which gets incredibly expensive.
That's crap, utter crap. For UDP, it's perfectly natural to have any
thread that wants to write a datagram to the socket. For TCP, it's perfectly
natural to have sending code and receiving code that can run concurrently in
diffierent threads.
> One process, one thread handles socket transactions. That process has
> application-side read and write buffers, and uses select/poll to read
> and write from ready sockets when select/poll indicates that they're
> ready for read or write. That's all.
That's a very bad architecture. A single page fault can cause your
performance to get bursty, and that can create a backup from which you
cannot recover as the working set size increases to exceed the processor
cache sizes.
DS
| |
| David Schwartz 2004-07-29, 8:47 pm |
|
<joe@invalid.address> wrote in message
news:m3zn5mmp6n.fsf@invalid.address...
> "David Schwartz" <davids@webmaster.com> writes:
>
[vbcol=seagreen]
[vbcol=seagreen]
[vbcol=seagreen]
> I'm having a hard time because you keep making assertions like this
> without giving any backing for them. I've asked a couple times where
> your ideas are documented in the standard, but I don't see a response
> to that.
It's in the thread you keep quoting. It clearly says that a blocked
process or thread is one that is "waiting for some condition (other than the
availability fo a processor)". This can certainly include a page fault.
> If these are your opinions about what the standard should
> mean, that's ok with me, and we'll just agree to disagree.
The standard doesn't say that a page fault must be considered a block
nor does it say it cannot be.
> However, what the standard *says* is that it won't block. Whatever you
> want to say blocking is or isn't, the standard says it won't.
Huh? Where does the standard say that if 'select' returns a hit for
write, a future write won't block? We *know* that's not true (because no
particular amount of bytes are guaranteed to be writable). So you think it's
true for read even though it's not true for write. Why? Where does the
standard make this distinction?
[vbcol=seagreen]
> No, but it says that it won't happen if select returns an indication
> that the descriptor is ready to read.
Won't *when*? Are you seriously arguing that the platform must somehow
ensure that a subsequent read or write does not block no matter how long the
program waits or what happens in the interim? That's absolutely ludicrous.
*No* other function works at all like this.
> Note that select has to return
> in order to do that. The calling code can't do anything until it gets
> the return value from select. It specifically does not say (and cannot
> mean) that the application can only rely on the return from select
> before select returns the value.
This is true for *every* other function. The return value can be out of
date by the time the function returns.
[vbcol=seagreen]
> How could it mean that? How could it possibly mean that the return
> from select is only valid before it's been returned?
When is the size of a file returned from a 'stat' call valid? *Every*
function that reports information reports the information as of the time the
indication was generated. No function works in the way you suggest 'select'
is supposed to, remebering that you were given information such that the
information remains valid in the future. It's absurd.
[vbcol=seagreen]
> Where is this documented?
Where is it documented that POSIX does *not* contain the requirement?!?!
Are you seriously arguing that standards must document what guarantees they
do not provide? Where does it say that the size returned by 'stat' may not
be valid by the time you look at it or the file you got in 'readdir' may not
exist? *No* indication function guarantees that the indication is valid
except at the time it was internally generated.
DS
| |
| joe@invalid.address 2004-07-29, 8:47 pm |
| "David Schwartz" <davids@webmaster.com> writes:
Sheesh, you smell blood or something? I was just letting it go and you
keep coming back at it.
I think I understand your position now. I'll let it go at that.
Joe
--
An Italian is COMBING his hair in suburban DES MOINES!
- yow
|
|
|
|
|