Unix Programming - Calling a Function in Shared Memory

Author

Calling a Function in Shared Memory

Michael B Allen

2005-05-15, 8:35 am

I have a hashmap in a file that is mmap'd into shared memory when
needed. I use size_t offsets from the beginning of the mapping to track
everything. This has worked great with Linux 2.4. But I just tried 2.6
and I'm getting a segfault.

Two issues have conspired against me to cause this problem. One is
that multiple processes need to access the map concurrently. Two, the
hashmap API hides the hash and compare functions as pointers within the
hashmap structure.

Because a pointer in one process (a function pointer in this case) is not
valid in another I had to copy the function into shared memory. Let me
reiterate that a little. The function of interest is called 'hash_str'. I
can locate this function within it's library with objdump:

$ objdump -t libmba.so.0.8.32 | grep hash_str
0000594c g F .text 0000004d hash_str

Now I can dissassemble this like:

$ objdump --start-address=0x594c --stop-address=0x5999 \
-d libmba.so.0.8.32

libmba.so.0.8.32: file format elf32-i386

Disassembly of section .init:
Disassembly of section .plt:
Disassembly of section .text:

0000594c <hash_str>:
594c: 55 push %ebp
594d: 89 e5 mov %esp,%ebp
594f: 83 ec 0c sub $0xc,%esp
5952: c7 45 fc 05 15 00 00 movl $0x1505,0xfffffffc(%ebp)
5959: 8b 45 08 mov 0x8(%ebp),%eax
595c: 89 45 f8 mov %eax,0xfffffff8(%ebp)
595f: 83 7d 0c 00 cmpl $0x0,0xc(%ebp)
5963: 74 09 je 596e <hash_str+0x22>
5965: 8b 45 08 mov 0x8(%ebp),%eax
5968: 03 45 0c add 0xc(%ebp),%eax
596b: 89 45 f8 mov %eax,0xfffffff8(%ebp)
596e: 8b 45 f8 mov 0xfffffff8(%ebp),%eax
5971: 0f b6 00 movzbl (%eax),%eax
5974: 89 45 f4 mov %eax,0xfffffff4(%ebp)
5977: 8b 55 f4 mov 0xfffffff4(%ebp),%edx
597a: 8d 45 f8 lea 0xfffffff8(%ebp),%eax
597d: ff 00 incl (%eax)
597f: 85 d2 test %edx,%edx
5981: 74 11 je 5994 <hash_str+0x48>
5983: 8b 45 fc mov 0xfffffffc(%ebp),%eax
5986: c1 e0 05 shl $0x5,%eax
5989: 03 45 fc add 0xfffffffc(%ebp),%eax
598c: 03 45 f4 add 0xfffffff4(%ebp),%eax
598f: 89 45 fc mov %eax,0xfffffffc(%ebp)
5992: eb da jmp 596e <hash_str+0x22>
5994: 8b 45 fc mov 0xfffffffc(%ebp),%eax
5997: c9 leave
5998: c3 ret
Disassembly of section .fini:

When the program starts I actually allocate 15 + 0x004d bytes in the
shared memory and memcpy the above function .text section there aligned
on a 16 byte boundry. Yeah, I know this is highly non-portable but I
don't really have a lot of choices.

So when one of the processes calls the hashmap_get function, which needs
to call the hash function, it gets a pointer to the shared memory and
casts it to hash_fn. This much works. I can print the memory at the
pointer just before the segfault and see the above code:

00000: 55 89 e5 83 ec 0c c7 45 fc 05 15 00 00 8b 45 08 |U......E......E.|
00010: 89 45 f8 83 7d 0c 00 74 09 8b 45 08 03 45 0c 89 |.E..}..t..E..E..|
00020: 45 f8 8b 45 f8 0f b6 00 89 45 f4 8b 55 f4 8d 45 |E..E.....E..U..E|
00030: f8 ff 00 85 d2 74 11 8b 45 fc c1 e0 05 03 45 fc |.....t..E.....E.|
00040: 03 45 f4 89 45 fc eb da 8b 45 fc c9 c3 |.E..E....E... |

But when I call the function it segfaults:

Program received signal SIGSEGV, Segmentation fault.
0x007d9b66 in hashmap_get (h=0xb7508188, key=0x804f8a7) at src/hashmap.c:398
398 hs = hf(key, ALADR(al, h->context));
(gdb) bt
#0 0x007d9b66 in hashmap_get (h=0xb7508188, key=0x804f8a7) at src/hashmap.c:398

So I can only conclude that 2.6 does not like executing arbirary code
in a memory mapping like this.

What can I do? Is there a workaround?

Thanks,
Mike

moi

2005-05-15, 8:35 am

Michael B Allen wrote:
> I have a hashmap in a file that is mmap'd into shared memory when
> needed. I use size_t offsets from the beginning of the mapping to track
> everything. This has worked great with Linux 2.4. But I just tried 2.6
> and I'm getting a segfault.
>
> Two issues have conspired against me to cause this problem. One is
> that multiple processes need to access the map concurrently. Two, the
> hashmap API hides the hash and compare functions as pointers within the
> hashmap structure.
>
> Because a pointer in one process (a function pointer in this case) is not
> valid in another I had to copy the function into shared memory. Let me
> reiterate that a little. The function of interest is called 'hash_str'. I
> can locate this function within it's library with objdump:

Why would you copy a library function ? All processes will have mapped
the library function into their address space.
[ the problem may be how to locate them. ;-]

> So I can only conclude that 2.6 does not like executing arbirary code
> in a memory mapping like this.
>
> What can I do? Is there a workaround?
>

You seem to have found a way to avoid pointers in the shared structure,
by replacing them with offsets. Could you do the same with
functionpointers ?
If not (functions may be located at different offsets in different
"instances" of the same code), you would need to add an extra layer
of indirection.

IIRC there have been changes in the linux kernel/loader, to "randomise"
the loading, in order to avoid trivial stackoverrun/return into libc
exploits. That would affect your program as well.

HTH,
AvK

Paul Pluzhnikov

2005-05-15, 5:49 pm

Michael B Allen <mba2000@ioplex.com> writes:

> But when I call the function it segfaults:

Let me guess: you are on an AMD Athlon and you didn't mprotect()
the area into which you copied hash_str() with PROT_EXEC?

> Program received signal SIGSEGV, Segmentation fault.
> 0x007d9b66 in hashmap_get (h=0xb7508188, key=0x804f8a7) at src/hashmap.c:398
> 398 hs = hf(key, ALADR(al, h->context));

If my guess above is incorrect, what is the output from

x/15i $pc-15
info reg

> So I can only conclude that 2.6 does not like executing arbirary code
> in a memory mapping like this.

That conclusion is certainly wrong.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Michael B Allen

2005-05-15, 5:49 pm

On Sun, 15 May 2005 12:41:27 +0200, moi wrote:

>
> Why would you copy a library function ? All processes will have mapped
> the library function into their address space.
> [ the problem may be how to locate them. ;-]

Exactly. My struct hashmap has size_t offsets to .text of functions in
shared memory. If I used pointers instead I would have to set them in
struct hashmap before using it each time. Or I would have to change
each hashmap API function to accept the hash and compare function
pointers. Either way is too ugly.

> You seem to have found a way to avoid pointers in the shared structure,
> by replacing them with offsets. Could you do the same with
> functionpointers ?

I *do*. It's easy to put struct or a string in shared memory and refer
to it by offset but a function is code.

> If not (functions may be located at different offsets in different
> "instances" of the same code), you would need to add an extra layer
> of indirection.

Exactly. By puting the function in shared memory it is always at the same
offset within that shared memory. So each process can get a pointer to
the function by simply doing (void *)(baseaddr + offset).

> IIRC there have been changes in the linux kernel/loader, to "randomise"
> the loading, in order to avoid trivial stackoverrun/return into libc
> exploits. That would affect your program as well.

I have noticed the addresses are changing a lot. But this should have
no effect on my code as long as everything the processes access are
translated to offsets and converted to pointers based on the shread
memory base address on demand.

Indeed the problem was that I simply forgot PROT_EXEC (I guess 2.4 didn't
need it) and the code now works fine.

Thanks,
Mike

Paul Pluzhnikov

2005-05-15, 5:49 pm

Michael B Allen <mba2000@ioplex.com> writes:

> Indeed the problem was that I simply forgot PROT_EXEC (I guess 2.4 didn't
> need it) and the code now works fine.

The 2.4 kernels did not enable NX/XD bit in the Intel/AMD processors
because these bits did not exist when 2.4 was developed.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.