|
Home > Archive > Unix Programming > February 2007 > close() Interrupted By Signal (Resumption Fails)
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
close() Interrupted By Signal (Resumption Fails)
|
|
|
| I have read everything I could find about this. If close() is
interrupted by a signal (because SA_RESTART isn't set), the file
descriptor passed to the function is in an unspecified state. What
exactly does this mean? GNU documentation even suggests its own macro
to loop on close() until it's not interrupted.
Does "unspecified" mean it might have finished closing, might be in
the middle of closing...? Likely the system call detected and
interrupted on the signal in the middle of disk I/O on close(). What
if by the time the close() is restarted by the user-space code, the
file finished closing? The descriptor is now free...the kernel happens
to switch to a different thread which opens a new file and that file
re-uses that descriptor...now back in the first thread, close() is
called again because EINTR was received...and now back in the second
thread, the file that was just successfully opened is closed!
This should not be so difficult. We cannot be expected to do a process-
wide block on creating new files until a loop has finished for closing
one. And with "unspecified" state, how can we even know if doing so
would work?
I'm trying to handle signals in an intelligent way. I can use
sigaction with SA_RESTART for all signals I handle, and just remember
that "select" will still fail. I can let signals I don't expect
terminate the program. But this doesn't even solve the problem. I have
seen discussion and documentation of a problem where down inside glibc
some signal handlers were getting reset for the duration of library
calls and somehow EINTR was bubbling back up to user-space.
Is robust error-handling in Unix really possible? I can't disable
signals completely (not considering STOP and KILL) because a library
can turn them back on. I can't get the kernel to restart all my system
functions saying "I understand that functionality can be valuable, but
it isn't for me" thereby forcing me to alter my paradigm of
programming because I can't disable a feature that serves -me- no
purpose and no good.
Am I forced to write my programs as well as I can, and then say,
"Anything else that happens is so rare it's like winning the lottery,
so just crash and let inittab bring me back up"? The best thing about
computers is they are 100% deterministic, and yet things are getting
flung around so I can't even grab them. Please tell me some way I can
handle everything I am supposed to be able to handle, by logic.
| |
| David Schwartz 2007-02-27, 1:24 am |
| On Feb 26, 8:10 pm, "JAKJ" <J...@j-a-k-j.com> wrote:
I agree with everything you say but this:
> I can't disable
> signals completely (not considering STOP and KILL) because a library
> can turn them back on.
You and your libraries have to cooperate on signals. Basically, you
cannot allow a signal to interrupt a 'close' unless that signal is
fatal to the process. Any library that doesn't cooperate with you in
this policy is broken. Any library that tampers with signals in an
undocumented way is broken.
DS
| |
| Rainer Weikusat 2007-02-27, 7:21 am |
| "JAKJ" <JAKJ@j-a-k-j.com> writes:
> I have read everything I could find about this. If close() is
> interrupted by a signal (because SA_RESTART isn't set), the file
> descriptor passed to the function is in an unspecified state. What
> exactly does this mean?
The exact meaning of 'unspecified' is 'unspecified'. Anything more
detailed would be a property of a particular implementation.
> GNU documentation even suggests its own macro
> to loop on close() until it's not interrupted.
>
> Does "unspecified" mean it might have finished closing, might be in
> the middle of closing...? Likely the system call detected and
> interrupted on the signal in the middle of disk I/O on close().
A UNIX(*) (or similarly structured) system does disk I/O
asynchronously (and has always done so since the mid-1970s), which
means that _no_ system call from userspace (except the *sync-calls)
has a 1:1 correspondence with any actual I/O to a disk.
> What if by the time the close() is restarted by the user-space code,
> the file finished closing? The descriptor is now free...the kernel happens
> to switch to a different thread which opens a new file and that file
> re-uses that descriptor...now back in the first thread, close() is
> called again because EINTR was received...and now back in the second
> thread, the file that was just successfully opened is closed!
Congratulations. You have just discovered race conditions in
multi-threade programs and just need to discover locking as well to be
able to deal with them.
[...]
> Is robust error-handling in Unix really possible?
Define 'robust error handling'.
[...]
> Am I forced to write my programs as well as I can, and then say,
> "Anything else that happens is so rare it's like winning the lottery,
> so just crash and let inittab bring me back up"?
How are you going to prevent someone with sufficient privileges
killing your program at any inconvenient time, some cleaning lady
unplugging the power cord or unexpected arbian visitors suddenly
flying through a window?
| |
|
| On Feb 27, 4:58 am, Rainer Weikusat <rainer.weiku...@sncag.com> wrote:
> The exact meaning of 'unspecified' is 'unspecified'. Anything more
> detailed would be a property of a particular implementation.
I can accept that in the abstract. But where is the documentation for
each particular implementation on what this means? Once you limit
yourself to a particular kernel, you should be able to find out such
things without reading source.
> A UNIX(*) (or similarly structured) system does disk I/O
> asynchronously (and has always done so since the mid-1970s), which
> means that _no_ system call from userspace (except the *sync-calls)
> has a 1:1 correspondence with any actual I/O to a disk.
This just implies that when close() returns uninterrupted, the
operation is pending and possibly not complete. This is fine. But my
problem is if close() does not finish doing what it needed to do to
queue the asynchronous I/O in the kernel. If it were documented that
it can just be restarted, then fine, but the documentation says "we
have no idea what happened" when it says "unspecified". A half-closed
file is possible data loss and resource leaking. So much for robust
error-checking if a process gets a SIGCHLD and its I/O bunks up.
> Congratulations. You have just discovered race conditions in
> multi-threade programs and just need to discover locking as well to be
> able to deal with them.
I'm well-aware of race conditions. So you really are saying I have to
create a mutex just to close a file? Which leaves out the issue of
"unspecified" meaning I may not even be able to finish closing it,
even if access to it is locked before the close() began.
> Define 'robust error handling'.
Being able to handle all expected situations deterministically. Assume
a program with no signals: no interrupions. Now add signals: system
calls can be interrupted and return EINTR. Give me a list of all
system calls that can generate EINTR: I put all of them in a loop that
restarts them if EINTR is received. Tell me all system calls can
generate EINTR: I put every system call in such a loop. My program now
has signals and interrupted system calls with no failure.
Except...not all system calls can be restarted. connect() is another
good example, on blocking sockets. (And don't tell me "just use
unblocking sockets" because blocking sockets exist, work, and serve a
purpose.)
I'm not trying to program for the unexpected. I'm trying to program
for the expected. It's a perfectly good programming paradigm to me:
"Account for everything you can think of, and let everything else
crash you. Find out what crashed you, and now you can account for
that." As I said, computers are deterministic. Cosmic rays aren't
going to interrupt my system calls and give weird error codes.
Anything that happens in my program, I did, a library did, the kernel
did, the hardware did...and on some level, everything is predictable.
I can accept "I just got an error code I don't have in my list. Log
and die." I can't accept "A normal, intended kernel function just
happened and left part of my program in an unknown state with no way
to proceed."
> How are you going to prevent someone with sufficient privileges
> killing your program at any inconvenient time, some cleaning lady
> unplugging the power cord or unexpected arbian visitors suddenly
> flying through a window?
That falls under the "unexpected" category. Like the OOM killer. So is
a user deciding to play around with kill(). That's perfectly
acceptable, and in that case "robust error handling" is the bring-it-
back-up procedure.
Let's say my program doesn't expect to be in a resizeable terminal.
(Let's also pretend SIGWINCH defaulted to "Term".) The window resized
and my program crashed. Now I realize I have to account for this, so I
install a signal handler that updates internal variables as to the
current window size and marks the display as stale. Now everything's
working as expected. Except...SIGWINCH comes in while I happen to be
in a close() call, which is vanishingly unlikely BUT POSSIBLE. "Bad
luck, pal. You must've broken a mirror this week." That's just not
acceptable.
If I issue a system call, I expect to get either success or one of a
set of expected error conditions. How (and if) I handle these errors
is up to me. But the kernel is doing the work, the kernel knows
anything that can go wrong, and the kernel issues the error codes. I
find it unlikely that the kernel people just said "This is so unlikely
that it doesn't matter. Let the user deal with it. Or let nobody deal
with it. One messed-up descriptor in a year's time isn't going to
bring down a system." So this has to be documented -somewhere-, and
please don't tell me "read source" for something that's in the public
standard.
I've got to be able to recover from errors and conditions I expect to
receive. "We interrupted you. Ordinarily you would just resume...but
nobody knows what will happen if you do." If you're going to pull a
stunt like that, just crash me and spare me the headache.
| |
| Rainer Weikusat 2007-02-27, 1:19 pm |
| "JAKJ" <JAKJ@j-a-k-j.com> writes:
> On Feb 27, 4:58 am, Rainer Weikusat <rainer.weiku...@sncag.com> wrote:
>
> I can accept that in the abstract. But where is the documentation for
> each particular implementation on what this means?
Supposedly, in the 'system calls' section of the corresponding reference
manual.
[...]
>
> This just implies that when close() returns uninterrupted, the
> operation is pending and possibly not complete.
No, this does not mean anything different from what I wrote. 'close'
is a system call that closes a descriptor.
> This is fine. But my problem is if close() does not finish doing
> what it needed to do to queue the asynchronous I/O in the kernel.
'close' is still a system call that closes a descriptor. In
particular, it doesn't submit data to the kernel, that would be
write.
> If it were documented that it can just be restarted, then fine, but
> the documentation says "we have no idea what happened" when it says
> "unspecified".
The UNIX(*) standard says the 'the state of the descriptor is
unspecified' in this respect. Presumably, the 'state of the
descriptor' is wether the kernel considers it to be open or closed.
> A half-closed file is possible data loss and resource leaking.
Since close closes a descriptor, the call necessary to have at least
some guarantees wrt actual I/O is (yesterday, today and tomorrow)
still fsync (or fdatasync).
> So much for robust error-checking if a process gets a SIGCHLD and
> its I/O bunks up.
Surprisingly, close still closes a file descriptor. Can you give an
example what exactly 'a half-closed file' is supposed to be, and on
which UNIX(*) you have managed to create one by which sequence of
actions and what happened then?
>
> I'm well-aware of race conditions. So you really are saying I have to
> create a mutex just to close a file?
If you want to blindly redo close calls with some number that may
correspond to an open or a closed file descriptor and other threads
call open concurrently, you need some form of locking to preven your
close loop from closing a descriptor returned by open to a different
thread.
>
> Being able to handle all expected situations deterministically. Assume
> a program with no signals: no interrupions. Now add signals: system
> calls can be interrupted and return EINTR.
Theoretically, system calls can only return EINTR if the thread of
execution that is currently blocked in the kernel catches a signal,
the signal handler returns and 'automatic restarts' were not requested
when installing it.
[...]
> Except...not all system calls can be restarted. connect() is another
> good example, on blocking sockets.
In case of an interrupted connect, the connection is supposed to be
completed asynchronuously, according to SUS.
| |
| James Carlson 2007-02-27, 1:19 pm |
| "JAKJ" <JAKJ@j-a-k-j.com> writes:
> On Feb 27, 4:58 am, Rainer Weikusat <rainer.weiku...@sncag.com> wrote:
>
>
> I can accept that in the abstract. But where is the documentation for
> each particular implementation on what this means?
See your system man pages -- both for the close(2) system call and for
the particular I/O descriptors (the drivers) you're trying to close.
If they don't tell you, then you *might* try complaining to the
vendor.
The big problem here is that vendors _may_ still choose to leave the
behavior unspecified, which means that your application can't actually
rely on it. Portable applications must not rely on it, which is why
at least one other poster has pointed you to the UNIX specification.
(For the systems I know something about, the undocumented behavior is
that close(2) always closes the descriptor from the point of view of
the application, and the only thing that can be interrupted by a
signal in that process is the draining of previously written data.
This means that after EINTR, you should not reissue your close() call
[it's already closed], and in some contexts [particularly with
connection-oriented network sockets and serial ports], you may have
lost data that didn't finish draining. Use some other mechanism to
drain first if you care about the data.)
> I'm not trying to program for the unexpected. I'm trying to program
> for the expected. It's a perfectly good programming paradigm to me:
> "Account for everything you can think of, and let everything else
> crash you. Find out what crashed you, and now you can account for
> that." As I said, computers are deterministic.
Computers are, but not all specifications are. Some have "undefined"
semantics that you must code around.
> Cosmic rays aren't
> going to interrupt my system calls and give weird error codes.
> Anything that happens in my program, I did, a library did, the kernel
> did, the hardware did...and on some level, everything is predictable.
In the face of unspecified interfaces, the above prediction
necessarily assumes that your application relies on undocumented and
unsupported system behavior -- by definition.
That's generally not a good trait for a program to have.
> I can accept "I just got an error code I don't have in my list. Log
> and die." I can't accept "A normal, intended kernel function just
> happened and left part of my program in an unknown state with no way
> to proceed."
"Sorry."
> I've got to be able to recover from errors and conditions I expect to
> receive. "We interrupted you. Ordinarily you would just resume...but
> nobody knows what will happen if you do." If you're going to pull a
> stunt like that, just crash me and spare me the headache.
You could just call assert() on the result from close() to get that
behavior, if you want. It's your call.
--
James Carlson, Solaris Networking <james.d.carlson@sun.com>
Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
| |
|
| On Feb 27, 9:36 am, Rainer Weikusat <rainer.weiku...@sncag.com> wrote:
>
> Surprisingly, close still closes a file descriptor. Can you give an
> example what exactly 'a half-closed file' is supposed to be, and on
> which UNIX(*) you have managed to create one by which sequence of
> actions and what happened then?
Well, I found one mention that Linux does not return EINTR for close()
because close() works on the file descriptor and all the rest is
asynchronous. Paraphrasing Linus, the file descriptor is released long
before any "slow" operations which are interruptible are initiated, so
even without a signal interrupting the call, another thread can
receive that file descriptor before the close() even returns to
userspace. So it would seem you're right. But then again, it doesn't
make sense that it could ever return EIO. Unless "normally" it returns
either 0 or EIO, but if interrupted it returns EINTR instead and
whatever might have generated EIO runs detached from the thread of
execution so the IO error is silently ignored. Fsync not withstanding.
But this makes no sense -at-all- to me. Why would there exist a
function that is interruptible that isn't meant to be restarted? If
close() always succeeds in closing the file descriptor, whether or not
the async I/O succeeds or fails, why is it interruptible at all? Why
isn't it implemented so that it simply returns success when the file
descriptor is closed, and require you to use fsync to detect lingering
I/O errors?
> If you want to blindly redo close calls with some number that may
> correspond to an open or a closed file descriptor and other threads
> call open concurrently, you need some form of locking to preven your
> close loop from closing a descriptor returned by open to a different
> thread.
I don't want to "blindly" do anything. I want my kernel to tell me
what it wants me to do. Does it want me to retry on EINTR? In that
case, the file descriptor should still be valid and I shall retry.
Does it want me to carry on and simply know that I need to use other
means to detect I/O errors if close() wasn't able to inform me? In
that case, the file descriptor should be invalid, being closed, and I
will carry on. If I have to put a mutex on close() so the descriptor
can't be reassigned before I'm sure it's invalid, I will be annoyed,
but I will do it, because I know that's what needs to be done.
So there's no portable way to do this unless I plan for armageddon at
every turn? Just write different code for each environment based on
its kernel or libc?
>
>
> Theoretically, system calls can only return EINTR if the thread of
> execution that is currently blocked in the kernel catches a signal,
> the signal handler returns and 'automatic restarts' were not requested
> when installing it.
Some system calls (like select() and connect()) are not restarted
automatically by SA_RESTART. SIGSTOP/SIGCONT is the prime example. I
can manually set every signal handler to SIG_IGN, or set every one to
a custom handler with SA_RESTART, or even block all signals, and
SIGSTOP/SIGCONT will still break connect() with EINTR.
The portable solution would seem to be to use non-blocking sockets for
every operation, since select() can be safely restarted every time.
Linux (supposedly) allows you to restart connect() without having to
use select()/poll() on EINTR, which seems to be exclusive to Linux.
Why do blocking connect()s exist in a signalling environment if they
don't work simply?
>
> In case of an interrupted connect, the connection is supposed to be
> completed asynchronuously, according to SUS.
Or to use non-blocking connect in the first place, yes. People do not
use blocking I/O for performance, because select() blocks too. They do
it for ease of programming and simplicity of code. When you're
programming for Unix, there is no way to say "My program doesn't use
signals. I don't want to deal with them. I accept that if I get a
signal like SIGSEGV or SIGTERM, I will be simply kill()ed off." Even a
daemon can be issued SIGSTOP which will break code that doesn't
account for EINTR.
What's the point of writing code that uses a blocking connect() when
your error-handling code has to then invoke select() as on a non-
blocking connect()? That's actually MORE complex, and longer, than
just using non-blocking connect() in the first place. What is this?
"Feel free to use non-blocking connect(), but better hope nothing
unusual happens, or you're on your own."
Which raises even another point. Linux, on its lonesome, supposedly
allows you to restart connect() with connect() just as you could
restart any other restartable call if you didn't have SA_RESTART set.
So why wouldn't connect() automatically restart with SA_RESTART on
Linux? It would still require different code for Linux and non-Linux,
but at least it would be sensible.
I understand why people want select() to be interruptible. I do not
understand why I cannot specify that in my application I do not want
select() to be interrupted. Why do I not have this choice? "It's too
hard to write a kernel that way" or "We haven't done it that way yet"
I could understand. But "You shouldn't want that"...? My signal
handler uses the "self-pipe trick" because pselect isn't available
except on the newest Linux kernel. I'm not going to act on any signals
until my processing is done and I'm back at select() anyway, which
returns when the pipe has a signal in it. There is no place in my
entire program whatsoever that benefits from any system call ever
being interrupted by anything. Why do I have to deal with this?
| |
| Rainer Weikusat 2007-02-27, 1:19 pm |
| "JAKJ" <JAKJ@j-a-k-j.com> writes:
> On Feb 27, 9:36 am, Rainer Weikusat <rainer.weiku...@sncag.com> wrote:
>
> Well, I found one mention that Linux does not return EINTR for close()
> because close() works on the file descriptor and all the rest is
> asynchronous. Paraphrasing Linus, the file descriptor is released long
> before any "slow" operations which are interruptible are initiated, so
> even without a signal interrupting the call, another thread can
> receive that file descriptor before the close() even returns to
> userspace. So it would seem you're right. But then again, it doesn't
> make sense that it could ever return EIO. Unless "normally" it returns
> either 0 or EIO, but if interrupted it returns EINTR instead and
> whatever might have generated EIO runs detached from the thread of
> execution so the IO error is silently ignored. Fsync not withstanding.
>
> But this makes no sense -at-all- to me. Why would there exist a
> function that is interruptible that isn't meant to be restarted?
I think you are misinterpreting the 'is unspecified'. This just means
that if close returns EINTR, the file descriptor is either closed or
open, and which of both depends on the implementation you are
using. For instance, the HP-UX 11.22 documenation says:
[EINTR] An attempt to close a slow device or connection
or file with pending aio requests was interrupted
by a signal. The file descriptor still points to
an open device or connection or file.
Contrasting to this, you'll find the following text for AIX 5.3:
The close subroutine attempts to cancel outstanding
asynchronous I/O requests on this file descriptor. If the
asynchronous I/O requests cannot be canceled, the application
is blocked until the requests have completed.
[...]
The close subroutine is blocked until all subroutines which
use the file descriptor return to usr space. For example,
when a thread is calling close and another thread is calling
select with the same file descriptor, the close subroutine
does not return until the select call returns.
[...]
If the close subroutine is interrupted by a signal that is
caught, it returns a value of -1, the errno global variable
is set to EINTR and the state of the FileDescriptor parameter
is closed.
[...]
>
> I don't want to "blindly" do anything. I want my kernel to tell me
> what it wants me to do. Does it want me to retry on EINTR? In that
> case, the file descriptor should still be valid and I shall retry.
> Does it want me to carry on and simply know that I need to use other
> means to detect I/O errors if close() wasn't able to inform me? In
> that case, the file descriptor should be invalid, being closed, and I
> will carry on. If I have to put a mutex on close() so the descriptor
> can't be reassigned before I'm sure it's invalid, I will be annoyed,
> but I will do it, because I know that's what needs to be done.
>
> So there's no portable way to do this unless I plan for armageddon at
> every turn? Just write different code for each environment based on
> its kernel or libc?
Since existing implementations behave differently in this respect
(example above) and likely already behaved differently before anything
more standardized than SVID, there is no really portable way to do
this (for instance, the Solaris man page just repeats the 'is
unspecified' text from SUS). This implies that your code can only support
some finite set of 'supported' target platforms and somebody trying to
use it on an unsupported platform may need to port a tiny piece of it.
>
> Some system calls (like select() and connect()) are not restarted
> automatically by SA_RESTART. SIGSTOP/SIGCONT is the prime example.
With select, it is (according to SUS) implementation defined. At least
on my Linux machine here, an unhandled SIGCONT does not interrupt a
connect. Neither does an ignored SIGHUP.
[...]
>
> Or to use non-blocking connect in the first place, yes.
Or don't set up signal handlers before connect or block handled
signals during connect.
[...]
> I understand why people want select() to be interruptible. I do not
> understand why I cannot specify that in my application I do not want
> select() to be interrupted. Why do I not have this choice?
Because the person or group of persons who originally implemented this
considered it to be the Right Thing or at least the Less Evil
Thing. As with all opinions, valid arguments for the opposite can
usually be found and eventually, the reasoning that led to the opinion
may have been nonsense from the start.
| |
|
| Well, I appreciate you helping me with this and remaining civil even
through my frustration. I think the correct solution is to support one
kernel/libset at a time and delve into it to garner all the little
details, which will allow me to be as robust as humanly possible. And
I'll just have slightly modified sources for each one. (I don't like
autoconf and its derivatives. Tools like that are insanely valuable
for "getting it working", but make people less concerned about making
things portable-able in the first place.)
I've got to say, actually enjoying programming is a valuable trait to
counteract frustration with it! Keeps you determined.
|
|
|
|
|