Unix Programming - Call a Function in Shared Memory?

Author

Call a Function in Shared Memory?

Michael B Allen

2004-09-26, 5:55 pm

Can a function be placed in shared memory such that two otherwise
unrelated processes can call it?

I have an /almost/ pointer-less hashmap that can be manipulated by
separate processes in shared memory. It uses offsets from the beginning
of memory from which all objects assocated with the map are derived. This
works great if all processes are forked from the process that itialized
the hashmap but I would like to access the map from processes that are
NOT forked from a common parent concurrently. The problem is that the
hash_fn and cmp_fn function pointers maintained by the map are only
valid to the process that set them:

struct hashmap {
int table_size_index;
hash_fn hash;
cmp_fn cmp;
void *context;
unsigned int size;
unsigned int load_factor_high;
unsigned int load_factor_low;
struct allocator *al;
ref_t table;
};

So I guess "pointer-less" isn't quite true.

I could temporarily set the function pointers before calling each map
function but that's somewhat crude.

Provided I know that the functions do not call other functions can I
copy them into shared memory as well effectively making the map truely
pointer-less? Is it possible to determine how big a function is? Is
there alignment to worry about?

Thanks,
Mike

Frank Cusack

2004-09-26, 5:55 pm

On Sun, 26 Sep 2004 04:30:38 -0400 Michael B Allen <mba2000@ioplex.com> wrote:
> Can a function be placed in shared memory such that two otherwise
> unrelated processes can call it?

Sure. The dynamic linker in all modern unices does this.

....
> Provided I know that the functions do not call other functions can I
> copy them into shared memory as well effectively making the map truely
> pointer-less?

See dlopen(3).

> Is it possible to determine how big a function is?

Sure, see libelf(3).

> Is there alignment to worry about?

Yes.

Maybe you can dlopen() a shared object (or yourself), get a handle
to your function, find the size of it, and copy it into shared mem.
Global symbols will complicate matters (as if it's not complex enough).

Maybe you could simply link your app against a shared lib containing
the shared function. Every app that needs to access this shared
function merely needs to link against the lib and the dynamic linker
will see to it that the function is available in shared mem, but
perhaps not the same address. Maybe there's a way you can influence
that.

/fc

Paul Pluzhnikov

2004-09-26, 5:55 pm

Michael B Allen <mba2000@ioplex.com> writes:

> Can a function be placed in shared memory such that two otherwise
> unrelated processes can call it?

Yes, but there are many complications ...

> I have an /almost/ pointer-less hashmap that can be manipulated by [...]

> I could temporarily set the function pointers before calling each map
> function but that's somewhat crude.

That would also require that you synchronize access to this map
between processes with a semaphore or process-shared mutex.

> Provided I know that the functions do not call other functions can I
> copy them into shared memory as well effectively making the map truely
> pointer-less?

Yes, if these functions are not accessing any global data and are
compiled position-independent.

> Is it possible to determine how big a function is?

Yes, but this is architecture-dependent.

> Is there alignment to worry about?

Yes.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Paul Pluzhnikov

2004-09-26, 5:55 pm

Frank Cusack <fcusack@fcusack.com> writes:

> On Sun, 26 Sep 2004 04:30:38 -0400 Michael B Allen <mba2000@ioplex.com> wrote:
>
> Sure. The dynamic linker in all modern unices does this.

No dynamic linker I know of places anything in (SysV or POSIX)
shared memory.

> Maybe you could simply link your app against a shared lib containing
> the shared function. [...]

That would generally *not* work. Consider:

struct map { int a; void (*print)(struct map *); }

// code in shared library "libshared.so"
void print_simple(struct map *m) { printf("a = %d\n", m->a); }
void print_fancy(struct map *m) { printf("map(%p).a = %d\n", m, m->a); }

// code in process1
int main()
{
... create and initialize map1 and map2 in shared memory
map1->print = print_simple;
map2->print = print_fancy;
}

// code in process2
int main()
{
... attach to map1 and map2
map1->print(map1);
map2->print(map2);
}

Even if both processes link against libshared.so, there is no reason
to expect that the address of print_simple will be the same in both
(unless the processes are fork()ed from one another, or are running
the same executable).

And if it isn't, process2 will likely crash on the call to map1->print().

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Barry Margolin

2004-09-26, 5:55 pm

In article <m3ekkpuky7.fsf@salmon.parasoft.com>,
Paul Pluzhnikov <ppluzhnikov-nsp@charter.net> wrote:

> Frank Cusack <fcusack@fcusack.com> writes:
>
[vbcol=seagreen]
> No dynamic linker I know of places anything in (SysV or POSIX)
> shared memory.

It mmaps the shared library file, and all processes that link the object
share this memory.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Andrew Gabriel

2004-09-26, 5:55 pm

In article <pan.2004.09.26.04.30.37.550970.12601@ioplex.com>,
Michael B Allen <mba2000@ioplex.com> writes:
> Can a function be placed in shared memory such that two otherwise
> unrelated processes can call it?

That's not the right way to solve this problem.

> I have an /almost/ pointer-less hashmap that can be manipulated by
> separate processes in shared memory. It uses offsets from the beginning
> of memory from which all objects assocated with the map are derived. This
> works great if all processes are forked from the process that itialized
> the hashmap but I would like to access the map from processes that are
> NOT forked from a common parent concurrently. The problem is that the
> hash_fn and cmp_fn function pointers maintained by the map are only
> valid to the process that set them:

Use something other than the function addresses to identify the function.
The most obvious way would be to have an array of function pointers in
the processes, and have the array index stored in the struct.

> struct hashmap {
> int table_size_index;
> hash_fn hash;
> cmp_fn cmp;
> void *context;
> unsigned int size;
> unsigned int load_factor_high;
> unsigned int load_factor_low;
> struct allocator *al;
> ref_t table;
> };
>
> So I guess "pointer-less" isn't quite true.

You seem to have 'context' and 'al' pointers in that struct too.

--
Andrew Gabriel
Consultant Software Engineer

Paul Pluzhnikov

2004-09-26, 5:55 pm

Barry Margolin <barmar@alum.mit.edu> writes:

>
> It mmaps the shared library file [...]

Indeed, but it mmap()s it with MAP_PRIVATE, *not* MAP_SHARED,
as can be seen in strace/truss output:

open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF [...]
old_mmap(NULL, 1201988, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40034000
old_mmap(0x4014f000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11a000) = 0x4014f000
old_mmap(0x40156000, 14148, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40156000
close(3) = 0

It (almost) might as well malloc() that memory.

The physical RAM pages are shared, but this is not at all what
people call "shared memory", and not what OP is asking about.

In particular, modifications to these pages (as in
e.g. self-modifying code), will not be seen by other processes.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Michael B Allen

2004-09-26, 5:55 pm

On Sun, 26 Sep 2004 10:48:27 -0400, Paul Pluzhnikov wrote:

>
> That would also require that you synchronize access to this map between
> processes with a semaphore or process-shared mutex.

Well yeah, I have to do that regardless.

>
> Yes, if these functions are not accessing any global data and are
> compiled position-independent.

They do not and they are.

>
> Yes, but this is architecture-dependent.

Well this isn't something that would need to be done frequently in a
generic way. So can I just extract this information from tools like nm
or objdump?

$ objdump -t libmba.so.0.8.11 | egrep 'hash_str|cmp_str'
000050e8 g F .text 00000059 cmp_str
0000506c g F .text 00000031 hash_str

>
> Yes.

What is the minimum alignment that would be considered
conservative? Obviously page alignment be sufficient yes??

Thanks,
Mike

Frank Cusack

2004-09-26, 5:55 pm

On 26 Sep 2004 08:08:32 -0700 Paul Pluzhnikov <ppluzhnikov-nsp@charter.net> wrote:
> Frank Cusack <fcusack@fcusack.com> writes:
>
>
> No dynamic linker I know of places anything in (SysV or POSIX)
> shared memory.

The OP didn't say SYSV shared memory. It appears he simply wants all
these processes to not duplicate the RAM usage. Clearly he did mean
SYSV shared mem, but given the problem (eg he didn't say anything
about self-modifying code) I'm simply answering the question
literally. (Because SYSV shared mem isn't required for this problem
as stated. And also I did mention copying the function into SYSV
shared mem.)

>
> That would generally *not* work. Consider:
....
> Even if both processes link against libshared.so, there is no reason
> to expect that the address of print_simple will be the same in both
> (unless the processes are fork()ed from one another, or are running
> the same executable).

Right, that's why I explicitly said:

On Sun, 26 Sep 2004 03:09:32 -0700 Frank Cusack <fcusack@fcusack.com> wrote:
> perhaps not the same address. Maybe there's a way you can influence
> that.

I wouldn't have mentioned this if it wasn't clear that this was a problem.
And it is really a hint ... you can influence this (perhaps not 100%
reliably but also perhaps reliably enough for a given environment).

> And if it isn't, process2 will likely crash on the call to map1->print().

Easily handled. The fn pointers could be pointers-to-pointers in
process local address space. Then you need only ensure that the local
pointed-to address space is the same for every process; mmap() can
guarantee this. You load the shared lib, which is at a different
address for each process but because of the extra indirection it
works. (I hope you can decipher that! And that it makes sense.
You'd call *map1->print().)

This basically accomplishes what the OP wants, given that he didn't
mention updating the shared function, etc. I'm assuming the same
function for every element in the data structure as well, otherwise
it's a lot of work.

On 26 Sep 2004 10:37:48 -0700 Paul Pluzhnikov <ppluzhnikov-nsp@charter.net> wrote:
> Barry Margolin <barmar@alum.mit.edu> writes:
>
>
> Indeed, but it mmap()s it with MAP_PRIVATE, *not* MAP_SHARED,
> as can be seen in strace/truss output:
>
> open("/lib/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF [...]
> old_mmap(NULL, 1201988, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40034000
> old_mmap(0x4014f000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11a000) = 0x4014f000
> old_mmap(0x40156000, 14148, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40156000
> close(3) = 0
>
> It (almost) might as well malloc() that memory.

??? mmap as above is simply an address space mapping.

> The physical RAM pages are shared, but this is not at all what
> people call "shared memory", and not what OP is asking about.
>
> In particular, modifications to these pages (as in
> e.g. self-modifying code), will not be seen by other processes.

For read-only it's the same. Global vars, even read-only ones, mess
this up also (both for SYSV shared mem and my shared lib suggestion).

/fc

Frank Cusack

2004-09-26, 5:55 pm

Frank Cusack

2004-09-26, 5:55 pm

On Sun, 26 Sep 2004 14:48:41 -0400 Michael B Allen <mba2000@ioplex.com> wrote:
> What is the minimum alignment that would be considered
> conservative? Obviously page alignment be sufficient yes??

Double alignment should be enough.

/fc

Paul Pluzhnikov

2004-09-26, 8:56 pm

Michael B Allen <mba2000@ioplex.com> writes:

> What is the minimum alignment that would be considered
> conservative?

The strictest code alignment I know of is 16 bytes (for IA64).

> Obviously page alignment be sufficient yes??

Sure, that (I think) would guarantee no further alignment problems
till the end of time :-)

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Michael B Allen

2004-09-27, 2:48 am

On Sun, 26 Sep 2004 13:04:56 -0400, Andrew Gabriel wrote:

>
> You seem to have 'context' and 'al' pointers in that struct too.

Well the context member is user defined and can just be NULL. But I
just realized that the al member is indeed a problem. A big problem. I
was thinking it wasn't bacause it's already in shared memory. Actually
that pointer points to the beginning of shared memory (the allocator
stucture is stored at the beginning). The problem is that is the base
address that is used to convert all pointers to offsets.

:-/

I suppose I could make the base address global data but that could get
ugly. Any ideas?

Mike

Michael B Allen

2004-09-27, 2:48 am

On Mon, 27 Sep 2004 01:24:36 -0400, Michael B Allen wrote:

> On Sun, 26 Sep 2004 13:04:56 -0400, Andrew Gabriel wrote:
>
>
> Well the context member is user defined and can just be NULL. But I just
> realized that the al member is indeed a problem. A big problem. I was
> thinking it wasn't bacause it's already in shared memory. Actually that
> pointer points to the beginning of shared memory (the allocator stucture
> is stored at the beginning). The problem is that is the base address
> that is used to convert all pointers to offsets.

Actually there's a very simple solution to this problem. Hint: a hashmap
in shared memory resides at a fixed offset relative to the beginning of
that shared memory segment.

Mike

Nils O. Selåsdal

2004-09-27, 5:55 pm

On Sun, 26 Sep 2004 10:37:48 -0700, Paul Pluzhnikov wrote:

> Barry Margolin <barmar@alum.mit.edu> writes:
>
>
> Indeed, but it mmap()s it with MAP_PRIVATE, *not* MAP_SHARED,
> as can be seen in strace/truss output:
>
> open("/lib/libc.so.6", O_RDONLY) = 3
> read(3, "\177ELF [...]
> old_mmap(NULL, 1201988, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x40034000
> old_mmap(0x4014f000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x11a000) = 0x4014f000
> old_mmap(0x40156000, 14148, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40156000
> close(3) = 0
>
> It (almost) might as well malloc() that memory.
No. These are likely copy on write protected. And you don't write to code
sections. It could also be that the code could not be loaded at it's
compile time address, in which case the code must be modified though.

Paul Pluzhnikov

2004-09-27, 5:55 pm

Nils O. Selåsdal <NOS@Utel.no> writes:

> No.

No what?

Are you saying the loader couldn't have simply malloc()ed a block
of memory, and read PIC-compiled code into it?

Why not?

This will make the shared library code not reside in shared RAM pages
anymore, which is why I said "almost"; but it sure *can* be done.

> These are likely copy on write protected.

Which is exactly my point -- they aren't "shared memory" in a sense
that modifications by one process are visible to any other process.

Nor are they "likely COW" -- they *are* COW: from "man mmap":

MAP_PRIVATE
Create a private copy-on-write mapping.

> And you don't write to code sections.

Speak for yourself. I often *do* write to them (dynamic code
patching).

> It could also be that the code could not be loaded at it's
> compile time address, in which case the code must be modified though.

For PIC code this should not matter.

For non-PIC code the loader does have to apply relocations,
destroying the RAM shared-ness, with the end-result being almost
exactly the same as if loader malloc()ed large blocks of memory,
read code/data into them, applied relocations and mprotect()ed them
appropriately.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.