I have a question about shared memory.  As I understand it, there
are two main types of shared memory available on most systems.
There's System V shared memory (shmget(), shmat(), etc.), and there
is mmap().  They both have advantages and disadvantages.

Now, let's imagine you have a daemon running, and some independent
client process wants to communicate with the daemon using some form
of shared memory.

One approach is for both the daemon and the client to agree on
arguments to pass to ftok(); then, they can both attach to the
same shared memory segment.  But, most versions of Unix have
limits on the size and number of System V shared memory segments,
which can easily make this impractical.  (It's very undesirable
for your software's install instructions to require the user to
change kernel tunables!)

mmap() style (where you mmap() /dev/zero or similar) doesn't impose
these kinds of size limits, but is there any way that a client and a
server can share memory via mmap() if neither is a child of the other?
Obviously, you could mmap a plain file in the filesystem, but it seems
like this would waste disk space and create unneeded physical I/O to
the disk.

One idea I had is to open /dev/zero in the server, then pass file
descriptor passing to pass the descriptor to the client, then have
them both mmap() the same region, but I'm uncertain that even works[1].

Maybe I need a good book to read on this and related subjects...

- Logan

[1]  Partly I'm uncertain because I don't fully understand the
semantics of mmap()ing /dev/zero; if any two processes mmap()
a plain file, they will share data, but separate processes
mmap()ing /dev/zero seem to end up with separate spaces mapped
into their address space.

[ Post a follow-up to this message ]

Logan Shaw wrote:
> I have a question about shared memory.  As I understand it, there
> are two main types of shared memory available on most systems.
> There's System V shared memory (shmget(), shmat(), etc.), and there
> is mmap().  They both have advantages and disadvantages.
>
Can you use mmap in this way?  I thought it mapped the file into the
process's address space, which isn't visible to other processes.

> Now, let's imagine you have a daemon running, and some independent
> client process wants to communicate with the daemon using some form
> of shared memory.
>
> One approach is for both the daemon and the client to agree on
> arguments to pass to ftok(); then, they can both attach to the
> same shared memory segment.  But, most versions of Unix have
> limits on the size and number of System V shared memory segments,
> which can easily make this impractical.  (It's very undesirable
> for your software's install instructions to require the user to
> change kernel tunables!)
>
Tell that to Oracle!

How many and how big do you have to go?

--
Ian Collins.

[ Post a follow-up to this message ]

Ian Collins wrote:
> Logan Shaw wrote: 
> Can you use mmap in this way?  I thought it mapped the file into the
> process's address space, which isn't visible to other processes.

Yeah, you can definitely use mmap() to change a file in a way that
the changes are visible to other programs either doing read()s or
reading through mmap().  In fact, the Solaris version of cp now does
this instead of regular read() and write() if one or the other of
the arguments is a plain file because it gets better performance
that way.

So I guess the point is that while the process's address space isn't
visible to other processes, files are, and if you make a change in
a file that's mapped into two processes, the data changes in both
places.  (The VM system is using the file as backing store for the
pages, after all.)

But, it does seem to require using the MAP_SHARED flag to mmap().
I tried using MAP_PRIVATE instead, which seems nonsensical to me
when mmap()ing a plain file BTW, and all my writes didn't seem
to affect the file's contents.
 
[vbcol=seagreen]
> Tell that to Oracle!

Yeah, well...  In their case, I guess usually you dedicate a machine
to their product anyway, so maybe it's OK if they require you to make
radical configuration changes.  But I still think it's bad form.  :-)

> How many and how big do you have to go?

Well, basically the specific reason is that I'm toying with writing a
program that will stream large amounts of data from one process to
another.  In principle, I could just use a pipe, but for performance
reasons, shared memory seems better.  I've already written a test
program that uses shared memory and semaphores to make a pseudo-pipe
between a parent and its child, but I'd like to be able to do it
between arbitrary processes, or at least to know whether it's possible.

Anyway, the thing is that I would like the user to be able to say how
large a buffer they need: some of this data might come from disk and
stream over the network (or vice versa), so larger buffers are better,
and for performance reasons, I'd rather avoid copying the data more
than necessary, which means a large shared buffer is even better.
And the number of buffers needed would just depend on what the user
is trying to accomplish.

Also, in general, I hate putting arbitrary limits on things if there
is no good reason to do so.  It just makes your software suck.  :-)

- Logan

[ Post a follow-up to this message ]

Logan Shaw <lshaw-usenet@austin.rr.com> writes:
 
>
> Well, basically the specific reason is that I'm toying with writing a
> program that will stream large amounts of data from one process to
> another.  In principle, I could just use a pipe, but for performance
> reasons, shared memory seems better.  I've already written a test
> program that uses shared memory and semaphores to make a pseudo-pipe
> between a parent and its child, but I'd like to be able to do it
> between arbitrary processes, or at least to know whether it's possible.

If you STREAM the data from one process to the another, you don't
necessarily need a big buffer.  The size of the shared memory circular
buffer you'll use will impose a synchronization between the two
processes, the smaller the more synchrones they'll be, but  it is
entirely independant of the size of the data to be STREAMED.


> Anyway, the thing is that I would like the user to be able to say how
> large a buffer they need: some of this data might come from disk and
> stream over the network (or vice versa), so larger buffers are better,

Even large buffers can be filled.


> and for performance reasons, I'd rather avoid copying the data more
> than necessary, which means a large shared buffer is even better.
> And the number of buffers needed would just depend on what the user
> is trying to accomplish.
>
> Also, in general, I hate putting arbitrary limits on things if there
> is no good reason to do so.  It just makes your software suck.  :-)
>
>    - Logan

--
__Pascal Bourguignon__                     http://www.informatimago.com/

"Specifications are for the weak and timid!"

[ Post a follow-up to this message ]

Logan Shaw wrote:
> I have a question about shared memory.  As I understand it, there
> are two main types of shared memory available on most systems.
> There's System V shared memory (shmget(), shmat(), etc.), and there
> is mmap().  They both have advantages and disadvantages.

Not really. The two types are System V which you have mentioned, and
POSIX shm_open().

> Now, let's imagine you have a daemon running, and some independent
> client process wants to communicate with the daemon using some form
> of shared memory.
>
> One approach is for both the daemon and the client to agree on
> arguments to pass to ftok(); then, they can both attach to the
> same shared memory segment.

The same is possible with POSIX shared memory, with the difference that
ftok() is not needed anymore.

[]

> mmap() style (where you mmap() /dev/zero or similar) doesn't impose
> these kinds of size limits, but is there any way that a client and a
> server can share memory via mmap() if neither is a child of the other?
> Obviously, you could mmap a plain file in the filesystem, but it seems
> like this would waste disk space and create unneeded physical I/O to
> the disk.

Pass mmap() the descriptor obtained via shm_open(), this is how POSIX
shared memory works.

http://www.opengroup.org/onlinepubs...s/shm_open.html

[ Post a follow-up to this message ]

Pascal Bourguignon wrote:
> Logan Shaw <lshaw-usenet@austin.rr.com> writes:
 
[vbcol=seagreen]
> If you STREAM the data from one process to the another, you don't
> necessarily need a big buffer.  The size of the shared memory circular
> buffer you'll use will impose a synchronization between the two
> processes, the smaller the more synchrones they'll be, but  it is
> entirely independant of the size of the data to be STREAMED.

The specific idea is to copy a bunch of files to an archive (or
several archives) on the local machine or a remote machine.
The rate the source makes the data available will tend to vary
since smaller files have extra overhead and cause the disks to
seek, where as large files have higher I/O rates.  The rate that
the consumer is able to consume the data will tend to vary when
the remote end is across a network, since available network
bandwidth can vary.  Therefore, the larger the buffer, the better.
Yes, there is a law of diminishing returns, but in this particular
case it may not kick in until relatively late.

> Even large buffers can be filled.

I'm much more concerned about the buffer becoming empty.  :-)

- Logan

[ Post a follow-up to this message ]

In article <2JkBf.32391$SD1.29432@tornado.texas.rr.com>, Logan Shaw <lshaw-usenet@austin.rr.
com> writes:
> Ian Collins wrote: 

The process's address space is private to the process[1], but that
says nothing about the pages mapped into that space.  Data pages are
usually private, but they can be shared; text pages are often shared.
[vbcol=seagreen]
> Yeah, you can definitely use mmap() to change a file in a way that
> the changes are visible to other programs either doing read()s or
> reading through mmap().  In fact, the Solaris version of cp now does
> this instead of regular read() and write() if one or the other of
> the arguments is a plain file because it gets better performance
> that way.

That's different from shared memory.  cp (I assume) maps its source
file with MAP_PRIVATE - it has no need to share it with any other
process.  It maps the target file with MAP_SHARED only because
MAP_PRIVATE maps the pages as copy-on-write (COW), so cp has to use
MAP_SHARED in order to actually change the target file.  It's not
sharing any pages with any other process, which is what shared memory
is for.

mmap'ing a file just means that the backing store for those pages is
that file rather than space on a paging device.

Ordinary data pages are backed up by pagesize (typically 4KB) blocks
on a paging device.  If you run short on free physical memory, the
kernel may write some data pages out to a paging device and free up
those physical pages in RAM; then, when the process that owns that
data goes to read or write to that part of its address space, the
kernel pages them back in again.

mmap a file with MAP_SHARED and all you're saying is, "hey, if I read
from these pages, and they haven't been paged in yet, fill them from
this file; and if I write to them, and then you need to flush them,
write them to this file rather than to a paging device".  (And the
only difference between that and MAP_PRIVATE is that MAP_PRIVATE
changes the write behavior to be "if I write to them, make a copy and
update that instead of the file".  The copy will be backed up with
normal paging space.)

Note that this means that even with MAP_SHARED changes aren't
necessarily "seen" right away by other processes that are looking at
the file.  See the man pages for mmap, munmap, and msync; the kernel
decides when to flush dirty pages to disk.  (Some Unix implementations
provide extensions to force flushing under other circumstances, but
this is all POSIX provides.)

> So I guess the point is that while the process's address space isn't
> visible to other processes, files are, and if you make a change in
> a file that's mapped into two processes, the data changes in both
> places.

Not immediately, and that's not quite right anyway.  MAP_SHARED makes
the page shared, ie not COW.  The sharing has nothing to do with it
being backed by a file, which is why MAP_SHARED pages remain shared
between parent and child after a fork.

Using mmap for shared memory between unrelated processes requires a
file because that's how the processes can identify the pages that
they're all going to map.

> But, it does seem to require using the MAP_SHARED flag to mmap().
> I tried using MAP_PRIVATE instead, which seems nonsensical to me
> when mmap()ing a plain file BTW, and all my writes didn't seem
> to affect the file's contents.

MAP_PRIVATE with a file is perfectly sensible; it means "I want to
map this file as COW", which is useful in many circumstances: if
you're only going to read it, for example, or if you don't want
your changes to its data to be reflected in it.

Writes to pages mapped from files with MAP_PRIVATE don't affect the
file's contents because MAP_PRIVATE markes the pages as COW.  It's
right in the man pages...


Getting back to your original problem: why not let the user decide
how big the buffer is going to be?  It'll only actually consume
space in the filesystem if pages have to be flushed to disk, and
in that case you'd be taking up paging space anyway.

I don't offhand know of a way to prevent the OS from flushing the
pages to disk when the last reference to the mapping is removed,
but you can always ftruncate the file to 0 afterward.  Actually,
that might work before you unmap, though it's not entirely clear
to me whether that's well-defined behavior under SUSv3.

Hmm... Another possibility: one process always creates the file
(and sets its size and maps it), and the other then opens it,
maps it, and removes it.  The file will persist only as long as
at least one process has an open reference to it; when they exit,
its space will be returned to the filesystem.



1. In vast majority of existing Unix implementations, anyway.
There's at least one UNIX-branded OS which doesn't use process-
private addressing: OS/400, where all processes (except the ones
running under PASE) share a single 128-bit address space, with object
protection attributes to impose the necessary permissions. But it's
definitely an oddity, and despite the branding most Unix users
wouldn't recognize it as a Unix implementation.

2. Note that non-XSI Unixes aren't required to enlarge a file if
ftruncate specifies a larger size; if you have to support such a
platform, you can lseek to the desired size and write a single byte
instead.

--
Michael Wojcik                  michael.wojcik@microfocus.com

The way things were, were the way things were, and they stayed that way
because they had always been that way.  -- Jon Osborne

[ Post a follow-up to this message ]

Michael Wojcik wrote:
> In article <2JkBf.32391$SD1.29432@tornado.texas.rr.com>, Logan Shaw <lshaw
-usenet@austin.rr.com> writes:
> 
>
>
> The process's address space is private to the process[1], but that
> says nothing about the pages mapped into that space.  Data pages are
> usually private, but they can be shared; text pages are often shared.
>
>
Thanks for the detail Michael.

I've never attempted to share mmap'd pages, I've always used shared
memory or doors.

[good stuff snipped]

Thanks.

--
Ian Collins.

[ Post a follow-up to this message ]

Michael Wojcik wrote:
> mmap a file with MAP_SHARED and all you're saying is, "hey, if I read
> from these pages, and they haven't been paged in yet, fill them from
> this file; and if I write to them, and then you need to flush them,
> write them to this file rather than to a paging device".  (And the
> only difference between that and MAP_PRIVATE is that MAP_PRIVATE
> changes the write behavior to be "if I write to them, make a copy and
> update that instead of the file".  The copy will be backed up with
> normal paging space.)

Thanks.  That's a good explanation, which provides a good model of
the semantics, although if I understand correctly, older Unix kernels
(like SunOS 4.x and older versions of BSDs) didn't do copy-on-write
at all, so it seems like whether the copy is lazy (copy-on-write)
or eager seems unspecified.  But all that changes is the performance.

> Note that this means that even with MAP_SHARED changes aren't
> necessarily "seen" right away by other processes that are looking at
> the file.  See the man pages for mmap, munmap, and msync; the kernel
> decides when to flush dirty pages to disk.

I agree that the kernel doesn't have to instantaneously flush pages
to disk; however, I'm surprised to learn that it might also not have
to keep them in sync.  Flushing them to disk isn't necessary in order
to keep them in sync -- you just map each process's page in its virtual
address space to the same physical memory, and they stay in sync
regardless of whether it's all instantly flushed to disk or is never
flushed to disk.

I'd be interested to know if there is in fact any guarantee in a
standard somewhere about whether two things mmap()ed (with MAP_SHARED)
to the same region of the same file are or are not guaranteed to be
kept in sync.  I could see arguments for doing it either way, because
guaranteeing they stay in sync could be helpful to the caller of
mmap(), but guaranteeing would also be more work for the system.
(For example, I can imagine a NUMA machine running Unix where several
nodes have their own local memory area; requiring them to keep everything
in sync would be quite a burden.)

I guess we're really getting into two different issues here:  whether
writes are being committed to stable storage, and whether shared parts
of files are guaranteed to always appear to be in sync.

> Getting back to your original problem: why not let the user decide
> how big the buffer is going to be?

That's exactly what I'm trying to do, which is why I'm looking for
a shared memory mechanism without arbitrary limits.

> It'll only actually consume
> space in the filesystem if pages have to be flushed to disk, and
> in that case you'd be taking up paging space anyway.
>
> I don't offhand know of a way to prevent the OS from flushing the
> pages to disk when the last reference to the mapping is removed,
> but you can always ftruncate the file to 0 afterward.

From where I stand, it seems like there is no way to prevent the
pages from being flushed to disk.  They could be flushed to disk
due to memory pressure, or some arbitrary limit that some kernel
chooses to put on certain kinds of pages that can be resident for
each process, or really anything.  At least I don't know of any
restrictions that prevent the kernel from writing pages to disk
at any time.  It seems like even if you do mlock() or similar,
the kernel is still free to keep the pages locked in memory but
also copy them to their backing store as frequently as it wants.

On the other hand, I guess the kernel is free to do that with
just about any form of memory, shared or otherwise, if it really
feels like it.  It might write it back to the file it's mapped
from, or it might write it to swap space...

> Hmm... Another possibility: one process always creates the file
> (and sets its size and maps it), and the other then opens it,
> maps it, and removes it.  The file will persist only as long as
> at least one process has an open reference to it; when they exit,
> its space will be returned to the filesystem.

Which is pretty good, although I'd really prefer not to require
that the user have a usable filesystem with enough space available
or to deal with the difficulties of letting the user figure out
what the pathname to that filesystem is and pass it to the program
(or having the program just try to guess).  It's not an insurmountable
problem, but it's not pretty either...

- Logan

[ Post a follow-up to this message ]

On 2006-01-24, Ian Collins <ian-news@hotmail.com> wrote:
> Logan Shaw wrote: 
> Tell that to Oracle!

No, tell it to the operating system vendors! IMHO, there really isn't
anything bad with a software vendor using the very useful IPC facility
that nearly all of their target OSes provide.

The problem here is rather the fact that almost all of today's Unixish
OSes have default resource limits on SysV IPC so low that it is hardly
usable for any of today's applications in the default configuration.
Also, most Unixes are unable to dynamically change these resource
limits without requiring a reboot. For this really the OS vendors are
to be blamed.

[Just my 2p.]

Regards
--
Jan Andres <jandres@gmx.net>

[ Post a follow-up to this message ]