Unix Programming - shmctl(IPC

Author

shmctl(IPC_RMID) oddity

phil_gg04@treefic.com

2005-04-22, 6:00 pm

Dear Experts,

[Linux, 2.6.3 kernel, Debian]

I have a program [*] that uses shared memory to communicate between its
many processes. I create a key_t when the program starts, and then the
first time that each process wants to use the shared memory it calls
shmget() and shmat().

One thing that I had previously been ignoring is that shared memory
segments will hang around using up RAM or swap space after the program
has finished unless you do something to remove them. Today I have been
trying to fix this.

As I understand it:

- Each segment keeps a count of how many processes are attached to it.
- Processes detach when they terminate.
- When the count gets down to zero the segment will be destroyed but
only if shmctl(IPC_RMID) had been called at some point.

So my plan was to call shmctl(IPC_RMID) immediately after attaching to
the segment. The segment does now seem to be destroyed when the
program terminates, so in that respect it has worked.

BUT, adding the call to shmctl(IPC_RMID) has broken everything else,
because it seems to have turned the segment into an anonymous one (like
IPC_PRIVATE).

Specifically, when I use ipcs or look in /proc/sysvipc/shm, I see
entries where the key is zero. (IPC_PRIVATE==0).

The result is that the first process creates the segment, but
subsequent processes also create new segments rather than getting the
existing one.

Removing the shmctl(IPC_RMID) call makes it all work again, with the
expected keys in the /proc/sysvipc/shm contents.

The correct key value is being passed according to strace.

I am rather baffled by this. Is what I am doing sensible? Is anyone
else doing this successfully?

Here is the relevant bit of code. It is made slightly more complicated
by the need to detect when the segment is first created so that it can
be initialised.

// earlier anyterm_shm_key is set to time().

bool exists = true;
// This will fail if it has not been created yet
anyterm_shm_id = shmget(anyterm_shm_key, 0, 0600);
if (anyterm_shm_id==-1) {
if (errno==ENOENT) {
// OK, it hadn't been created: we'll create it
exists = false;
int sz = sizeof(...);
anyterm_shm_id = shmget(anyterm_shm_key, sz,
IPC_CREAT|IPC_EXCL|0600);
}
}
if (anyterm_shm_id==-1) {
report error...
}

ad = (anyterm_data*) shmat(anyterm_shm_id, NULL, 0);
if ((int)ad==-1) {
report error...
}

// Here is the bit that breaks it:
// int rc = shmctl(anyterm_shm_id, IPC_RMID, NULL);
// if (rc==-1) {
// report error...
// }

if (!exists) {
initialise shared memory structures
}

[*] This is for Anyterm: http://chezphil.org/anyterm/

Thanks for any help!

--Phil.

phil_gg04@treefic.com

2005-04-22, 8:47 pm

Answering my own question....

> shmctl(IPC_RMID)
> seems to have turned the segment into an anonymous one (like
IPC_PRIVATE).

I decided to have a look at the kernel source, and yes, in the code for
IPC_RMID there's a bit that says:

shp->shm_perm.key = IPC_PRIVATE;

So once you've called shmctl(IPC_RMID) for a segment you can't attach
to it.

OK, so how am I going to make sure that the segment eventually gets
destroyed? New processes, which may want to connect to the shared
segment, can be created at any time. I suppose that I have to add a
signal handler to trap termination.

--Phil.

Kenny McCormack

2005-04-23, 5:51 pm

In article <1114213301.332271.58400@z14g2000cwz.googlegroups.com>,
<phil_gg04@treefic.com> wrote:
....
>OK, so how am I going to make sure that the segment eventually gets
>destroyed? New processes, which may want to connect to the shared
>segment, can be created at any time. I suppose that I have to add a
>signal handler to trap termination.

This question comes up every so often - and it does seem like one ought, in
the pursuit of being thought of as a good system citizen, try to do this
(that is, delete the shared memory when it is no longer used). As you can
tell from my phrasing here, I now doubt the truth of this.

Obviously, once you delete the shared memory, nobody else can connect to
it, so I am a little confused by the part of your post(s) that seems to
imply that you might somehow expect otherwise. Anyway, the general method
is for all the processes that need to attach to get attached, and then you
you delete it. The expected (hoped for) behavior is analogous to that of
a deleted file - that is, the file doesn't actually go away (in the sense
of the data it contains becoming toast) until the last close on the file.

Now, having said that, the consensus seems to be that:
1) Linux does the right thing (that is, as in the deleted file
case, the structure is left intact until the last close)
2) The standard does not require this.
3) Solaris may or may not do the right thing. My testing indicates
that it does, but others may have posted otherwise.

Finally, my own view is that ultimately it doesn't matter. As long as you
make sure that any process that wants to use it, will use the existing one
(i.e., not create a new one), then there's really no problem with leaving
it around.

Casper H.S. Dik

2005-04-24, 2:46 pm

gazelle@yin.interaccess.com (Kenny McCormack) writes:

>Now, having said that, the consensus seems to be that:
> 1) Linux does the right thing (that is, as in the deleted file
>case, the structure is left intact until the last close)
> 2) The standard does not require this.
> 3) Solaris may or may not do the right thing. My testing indicates
>that it does, but others may have posted otherwise.

The Solaris manual says:

IPC_RMID Remove the shared memory identifier speci-
fied by shmid from the system. The segment
referenced by the identifier will be des-
troyed when all processes with the segment
attached have either detached the segment or
exited. If the segment is not attached to
any process when IPC_RMID is invoked, it
will be destroyed immediately. This command
can be executed only by a process that has
appropriate privileges or an effective user
ID equal to the value of shm_perm.cuid or
shm_perm.uid in the data structure associ-
ated with shmid.

But the standard says:

IPC_RMID
Remove the shared memory identifier specified by shmid from the
system and destroy the shared memory segment and shmid_ds data
structure associated with it. IPC_RMID can only be executed by a
process that has an effective user ID equal to either that of a
process with appropriate privileges or to the value of
shm_perm.cuid or shm_perm.uid in the shmid_ds data structure
associated with shmid.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

phil_gg04@treefic.com

2005-04-25, 2:53 am

> Obviously, once you delete the shared memory, nobody else can connect
to it

This wasn't obvious to me; the linux man page says:

IPC_RMID is used to mark the segment as destroyed. It will
actually
be destroyed after the last detach. (I.e.,
when the
shm_nattch member of the associated structure
shmid_ds is
zero.) The user must be the owner, creator, or the
super-
user.

This doesn't say anything about new attachments being prevented after
the call.

> Anyway, the general method is for all the processes that need to
> attach to get attached, and then you you delete it.

My problem is that new processes are being created frequently; I don't
know when the "last" process has attached. So I believe that I have to
trap termination signals and delete the segment if this is the last
attached process.

--Phil.

phil_gg04@treefic.com

2005-04-25, 2:53 am

> But the standard says:
>
> IPC_RMID
> Remove the shared memory identifier specified by shmid from the
> system and destroy the shared memory segment and shmid_ds data
> structure associated with it. IPC_RMID can only be executed by a
> process that has an effective user ID equal to either that of a
> process with appropriate privileges or to the value of
> shm_perm.cuid or shm_perm.uid in the shmid_ds data structure
> associated with shmid.

Thanks Casper. So for portable code I musn't call shmctl(IPC_RMID)
until I really want it to go away.

--Phil.

Michael Kerrisk

2005-04-25, 2:53 am

On 22 Apr 2005 16:41:41 -0700, phil_gg04@treefic.com wrote:

>Answering my own question....
>
>IPC_PRIVATE).
>
>I decided to have a look at the kernel source, and yes, in the code for
>IPC_RMID there's a bit that says:
>
> shp->shm_perm.key = IPC_PRIVATE;
>
>So once you've called shmctl(IPC_RMID) for a segment you can't attach
>to it.

I think you're misreading the source here. The above line has the
ffect that we can no longer apply shmget() on the key for this segment
-- but we can still do shmat() using the segment identfier.

Cheers,

Michael

Michael Kerrisk

2005-04-25, 7:55 am

On Sat, 23 Apr 2005 22:45:22 GMT, gazelle@yin.interaccess.com (Kenny
McCormack) wrote:

>In article <1114213301.332271.58400@z14g2000cwz.googlegroups.com>,
> <phil_gg04@treefic.com> wrote:
>...
>
>This question comes up every so often - and it does seem like one ought, in
>the pursuit of being thought of as a good system citizen, try to do this
>(that is, delete the shared memory when it is no longer used). As you can
>tell from my phrasing here, I now doubt the truth of this.
>
>Obviously, once you delete the shared memory, nobody else can connect to
>it, so I am a little confused by the part of your post(s) that seems to
>imply that you might somehow expect otherwise.

Am I misunderstanding the OP or are you? AFAICS the OP is saying that
he has *observed* this bahaviour on Linux. He (incorrectly) expects
it to apply on other Unix systems.

>Anyway, the general method
>is for all the processes that need to attach to get attached, and then you
>you delete it.

Yes.

> The expected (hoped for) behavior is analogous to that of
>a deleted file - that is, the file doesn't actually go away (in the sense
>of the data it contains becoming toast) until the last close on the file.
>
>Now, having said that, the consensus seems to be that:
> 1) Linux does the right thing (that is, as in the deleted file
>case, the structure is left intact until the last close)

Linux does the right thing in that it doesn't remove the segment until
all processes have detached.

However, Linux is aberrant in allowing new attaches to a segment that
is already marked for deletion. This behaviour is not required by
SUSv3 (though I can't find any text in the standard that prohibits
it). I don't know of any other implementation that does this (I've
tested a few). Reportedly this feature crept into Linux by accident,
and some applications now depend on it.

> 2) The standard does not require this.

IIRC correctly the standard is weakly written in this area. All
implementations that I know of will only actually remove the segment
after the last detach (this differs from the effect of IPC_RMID for
other System IPC objects (i.e., message queues, semaphores) since they
don't have the notion of "attachment" to the object).

Michael Kerrisk

2005-04-25, 7:55 am

On 24 Apr 2005 13:26:06 GMT, Casper H.S. Dik <Casper.Dik@Sun.COM>
wrote:

>gazelle@yin.interaccess.com (Kenny McCormack) writes:
>
>
>The Solaris manual says:
>
> IPC_RMID Remove the shared memory identifier speci-
> fied by shmid from the system. The segment
> referenced by the identifier will be des-
> troyed when all processes with the segment
> attached have either detached the segment or
> exited. If the segment is not attached to
> any process when IPC_RMID is invoked, it
> will be destroyed immediately. This command
> can be executed only by a process that has
> appropriate privileges or an effective user
> ID equal to the value of shm_perm.cuid or
> shm_perm.uid in the data structure associ-
> ated with shmid.
>
>But the standard says:
>
>IPC_RMID
> Remove the shared memory identifier specified by shmid from the
> system and destroy the shared memory segment and shmid_ds data
> structure associated with it. IPC_RMID can only be executed by a
> process that has an effective user ID equal to either that of a
> process with appropriate privileges or to the value of
> shm_perm.cuid or shm_perm.uid in the shmid_ds data structure
> associated with shmid.

Yes -- the standard doesn't accurately describe reality on existing
implementations. The Solaris man page does.

Casper H.S. Dik

2005-04-25, 7:55 am

Michael Kerrisk <michael.kerrisk.at.gmx.net@nospam.com> writes:

>However, Linux is aberrant in allowing new attaches to a segment that
>is already marked for deletion. This behaviour is not required by
>SUSv3 (though I can't find any text in the standard that prohibits
>it). I don't know of any other implementation that does this (I've
>tested a few). Reportedly this feature crept into Linux by accident,
>and some applications now depend on it.

The SUSv2 manual page claims it should be destroyed; that would preclude
further attaches but even continued use.

[vbcol=seagreen]
>IIRC correctly the standard is weakly written in this area. All
>implementations that I know of will only actually remove the segment
>after the last detach (this differs from the effect of IPC_RMID for
>other System IPC objects (i.e., message queues, semaphores) since they
>don't have the notion of "attachment" to the object).

Indeed; I would expect the identifier to be gone (the segment becomes
anonymous) and then on last close it is removed.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

phil_gg04@treefic.com

2005-04-25, 5:53 pm

> I think you're misreading the source here. The above line has the
> effect that we can no longer apply shmget() on the key for this
segment
> -- but we can still do shmat() using the segment identfier.

Michael, that's interesting. Currently I choose a key when the
application starts and each process calls shmget to get an identifier
and then shmat to attach. Are you suggesting that I only need to do
shmget once, pass the identifier to the new processes, and have them
just call shmat? I suppose that I was considering the identifiers to
be local to processes, but now I'm unsure.

--Phil.

Michael Kerrisk

2005-04-26, 2:49 am

On 25 Apr 2005 14:10:32 -0700, phil_gg04@treefic.com wrote:

>segment
>
>Michael, that's interesting. Currently I choose a key when the
>application starts and each process calls shmget to get an identifier
>and then shmat to attach. Are you suggesting that I only need to do
>shmget once, pass the identifier to the new processes, and have them
>just call shmat? I suppose that I was considering the identifiers to
>be local to processes, but now I'm unsure.

Phil,

Yes -- you're getting it. Identifiers are not local to processes;
they are system-wide. (They're not like file descriptors.)

It is indeed possible to do a shmget() in just one process and then
pass the identifier along to other processes that do a shmat() (the
complication is how to pass the ID -- perhaps simplest to write it to
a file).

One technique for guaranteeing a new unique segment is the following:

id = shmget(IPC_PRIVATE, IPC_CREAT | mode);

and then write the ID to a file for the other processes to read.

IPC_PRIVATE guarantees a new segment whose ID does not match that of
any existing segment.

Cheers,

Michael