I am writing a custom malloc debugger for my application, and while
gathering the stack trace for a given call to malloc(), I find I need
a fast way of testing whether a given address is readable.

I'd much prefer to avoid messing with SEGV handlers (we already have
one, and I don't want to interfere with it), and /proc/PID/maps is too
slow to parse through (I can't cache it, since it changes frequently).

So I've been trying to use write(), since it is supposed to return
EFAULT if an address is not readable. However, my first attempt failed
completely by always returning true:

bool is_readable(void* address) {
static int dummy_fd = open("/dev/null", O_WRONLY);
int status = write(dummy_fd, address, sizeof(void*));
return ! (status == -1 && errno == EFAULT);
}

I'm guessing the problem is that dummy_fd refers to a file structure
within the kernel whose write() function pointer does nothing, but
always returns success. If I set dummy_fd to 1 (stdout), the check
succeeds.

Currently, I am doing:

static int dummy_fd = -1;
if (dummy_fd < 0) {
dummy_fd = open("/tmp/addrcheck", O_WRONLY | O_CREAT);
unlink("/tmp/addrcheck");
}

It seems to be working fine, but I'm worried about the speed, and
about creating an always-growing file that will eat my disk space. I
could periodically seek back to the beginning to fix the second, but
it feels like there ought to be a simpler way to solve the original
problem: how can I tell whether a given 4-byte address range is mapped
and readable? (Which is itself, I suppose, a proxy for "is this a
valid stack address?", so perhaps that's the ultimate question here.)

[ Post a follow-up to this message ]

sphink@gmail.com <sphink@gmail.com> wrote:
> I am writing a custom malloc debugger for my application, and while
> gathering the stack trace for a given call to malloc(), I find I need
> a fast way of testing whether a given address is readable.

Finding out if an address can be read from isn't of much help
when you want to write a "malloc() debugger". There are lots
and lots of addresses you potentially could read from (basi-
cally all address that's within your processes address space)
but which don't "belong" to you in the sense that they are
addresses that belongs to a memory chunk you got from malloc()
etc. So testing if an address can be read from won't help you
a bit in a "malloc() debugger" - all you potentially could
catch is reads from outside your address space (or that may
be marked as non-readable) but those you already rather easily
find because you get a segmentation fault and every debugger
will give you a backtrace for those. The "interesting" bugs
are those where you access memory you don't "own" (i.e. are
outside of memory chunks you got from malloc() etc. or aren't
variables of your program) but which don't result in a segmen-
tation fault, so the program continues to run as if nothing
bad had happened.
Regards, Jens
--
\   Jens Thoms Toerring  ___      jt@toerring.de
\__________________________      http://toerring.de

[ Post a follow-up to this message ]

sphink@gmail.com wrote On 02/15/07 17:07,:
> I am writing a custom malloc debugger for my application, and while
> gathering the stack trace for a given call to malloc(), I find I need
> a fast way of testing whether a given address is readable.
> [...]
> how can I tell whether a given 4-byte address range is mapped
> and readable? (Which is itself, I suppose, a proxy for "is this a
> valid stack address?", so perhaps that's the ultimate question here.)

You might be able to use mincore() for the first question.

The second question seems simpler: Find the stack base (you
only need to do this once), find the address of an `auto' variable
on your own stack frame, and check that the test address lies
between them.  Things get harder for multi-threaded programs
(where an  "on stack" address might be on some other thread's
stack).  It would also be hard on a system where stack frames
weren't necessarily contiguous -- but if you're already walking
up the stack frame by frame, maybe you could perform the test
as part of the stack walk.

--
Eric.Sosman@sun.com

[ Post a follow-up to this message ]

"sphink@gmail.com" <sphink@gmail.com> writes:

> Currently, I am doing:
>
>   static int dummy_fd = -1;
>   if (dummy_fd < 0) {
>     dummy_fd = open("/tmp/addrcheck", O_WRONLY | O_CREAT);
>     unlink("/tmp/addrcheck");
>   }
>
> It seems to be working fine, but I'm worried about the speed, and
> about creating an always-growing file that will eat my disk space. I
> could periodically seek back to the beginning to fix the second, but
> it feels like there ought to be a simpler way to solve the original
> problem: how can I tell whether a given 4-byte address range is mapped
> and readable? (Which is itself, I suppose, a proxy for "is this a
> valid stack address?", so perhaps that's the ultimate question here.)

Unfortunately there is no portable cleaner way to do it.

On Solaris, you have /proc/self/map which provides a binary
interface to the "process address map"; but no such luck on Linux.

You could do it faster by arranging to interpose all calls to
mmap/munmap/mprotect, and keeping track of what's readable yourself.
However this is difficult to do -- Linux glibc and Solaris libc
perform "direct syscall" to do mmap under some conditions.

If you only need to address single-threaded code, and only worry
about reading stack addresses, then the task is almost trivial:
it's easy to know where top of the stack is, and everything between
top and current stack pointer is readable.

Solaris also provides access to thread stack segment via
thr_stksegment(), but no such luck on Linux either.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

[ Post a follow-up to this message ]

jt@toerring.de (Jens Thoms Toerring) writes:

> sphink@gmail.com <sphink@gmail.com> wrote:
>
> Finding out if an address can be read from isn't of much help
> when you want to write a "malloc() debugger".

It is if all you want to do is record allocation/deallocation
stack traces.

> There are lots
> and lots of addresses you potentially could read from (basi-
> cally all address that's within your processes address space)
> but which don't "belong" to you in the sense that they are
> addresses that belongs to a memory chunk you got from malloc()
> etc.

He isn't trying to determine whether "random" addresses are readable.
He is trying to say -- here is my saved ebp. Previous ebp is
supposed to be stored at *ebp. Can I dereference current ebp
(to get the previous one), or will that cause a SIGSEGV ?

> So testing if an address can be read from won't help you
> a bit in a "malloc() debugger" - all you potentially could
> catch is reads from outside your address space (or that may

You are thinking that he does Valgrind-style malloc debugger.
But what he appears to be doing is a "debug-malloc" implementation,
similar to e.g. mpatrol.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

[ Post a follow-up to this message ]

On Feb 15, 3:36 pm, Paul Pluzhnikov <ppluzhnikov-...@charter.net>
wrote:
> j...@toerring.de (Jens Thoms Toerring) writes:
> 
> 
>
> It is if all you want to do is record allocation/deallocation
> stack traces.

Yes, exactly.
 
>
> You are thinking that he does Valgrind-style malloc debugger.
> But what he appears to be doing is a "debug-malloc" implementation,
> similar to e.g. mpatrol.

I'm trying to track down memory leaks, and I noticed that libc
helpfully exports malloc(), free(), and calloc() as weak symbols. So I
define my own and do stack walks to figure out what's what. (When
printing out the leaked memory, I do the full scan of /proc/self/maps
-- yes, it exists in Linux, it's just in text form -- and nm/readelf
to map addresses to symbols, but that's later on and can be slower
because it only runs on what was actually leaked.)

I tried at least half a dozen existing tools, but none of them quite
worked with my application. Multithreading killed several, even though
I only care about the main thread. I forget now what I ran into with
the others. But writing my own only took about a day (plus at least 5
more to get it right, but that doesn't count, does it? Stupid
prelinked libraries...) And writing my own had the advantage of being
able to add in some application-specific context information.

I'm using the stack address range trick. strace was showing far too
many 4-byte writes to my bogus file, and I remembered seeing the stack
address range thing before, so it seemed like a nice speedup. Although
it makes me a little nervous since I've run into noncontiguous stacks
before. But I don't trust it, so I fall back on the write() trick.
Which is the wrong way around if I'm really worried about
noncontiguous stack space, but I just need to find this damn leak
right now so I'm more worried about the libraries that are compiled
without frame pointers, and the one (I may have been imagining this)
that seems to have its own memory regions in the stack trace.

[ Post a follow-up to this message ]

On Thu, 15 Feb 2007 23:20:34 -0800, sphink@gmail.com wrote:
<snip>
> noncontiguous stack space, but I just need to find this damn leak
> right now so I'm more worried about the libraries that are compiled
> without frame pointers, and the one (I may have been imagining this)
> that seems to have its own memory regions in the stack trace.

Valgrind will find your leak right quick, assuming your allocating using
malloc().

Valgrind gets an A+ from me, despite the fact that it doesn't work with
custom allocators (unless they're pull from a fixed area, like a large
static region).

If you have some esoteric reason for not using Valgrind, you can
always read the code and take some hints regarding how they store
addresses (hint: several hash tables, nested in a very intentional manner).
I never delved into the bytecode interpreter, but the memcheck module at
least was amazingly straightforward.

[ Post a follow-up to this message ]

On Feb 16, 12:09 am, William Ahern <will...@25thandClement.com> wrote:
> If you have some esoteric reason for not using Valgrind, you can
> always read the code and take some hints regarding how they store
> addresses (hint: several hash tables, nested in a very intentional manner)
.
> I never delved into the bytecode interpreter, but the memcheck module at
> least was amazingly straightforward.

I'd love to use valgrind. That's my first tool of choice. It causes
some issues because of the massive slowdown it induces (this is a
realtime interactive animation system, so the slowdown tends to change
the behavior fairly drastically unless we replay canned input), but
otherwise it's great.

Except that it causes an abort() deep within the OMP thread library,
which is used by IPP (Intel's Performance Primitives library). It's
possible that something I'm doing is triggering the abort() by
mangling memory, but I haven't been able to figure it out. For once,
it's not because of MMX or SSE instructions that IPP uses and valgrind
doesn't support; valgrind is now excellent in that area.

It worked fine with older versions of IPP, but we've started relying
on the newer stuff.

[ Post a follow-up to this message ]

"sphink@gmail.com" <sphink@gmail.com> writes:

> I'm using the stack address range trick. strace was showing far too
> many 4-byte writes to my bogus file, and I remembered seeing the stack
> address range thing before, so it seemed like a nice speedup. Although
> it makes me a little nervous since I've run into noncontiguous stacks
> before.

Non-contiguous stacks are extremely rare in multithreaded programs --
they are somewhat more common in "co-routine style" task switching,
but that's very rarely used when "real" threads are available.

Since this is your code, why are you worried about non-contiguous
stacks? Unless *your* code does it, nobody else is likely to.
Almost certainly not Intel libraries.

> I just need to find this damn leak

If you just want to find a memory leak, writing full-blown malloc
replacement and stack unwinder may be an overkill. Of course since
you've already spent 5 days on it, you are not likely to abandon
that approach now, but you may want to consider simply logging all
allocations and deallocatins (and file/line) into a large static
buffer, and periodically "compact" that buffer by removing matched
alloc/dealloc pairs. If your leak comes from the application code
(as opposed to being allocated by e.g. libc on your behalf), you
are likely to find that leak in no time.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

[ Post a follow-up to this message ]

On Feb 16, 10:52 pm, Paul Pluzhnikov <ppluzhnikov-...@charter.net>
wrote:
> "sph...@gmail.com" <sph...@gmail.com> writes: 
>
> If you just want to find a memory leak, writing full-blown malloc
> replacement and stack unwinder may be an overkill. Of course since
> you've already spent 5 days on it, you are not likely to abandon
> that approach now, but you may want to consider simply logging all
> allocations and deallocatins (and file/line) into a large static
> buffer, and periodically "compact" that buffer by removing matched
> alloc/dealloc pairs. If your leak comes from the application code
> (as opposed to being allocated by e.g. libc on your behalf), you
> are likely to find that leak in no time.

I did that long ago, by overriding the global operators new and
delete. (My application code almost never calls malloc directly, and
the places where it does are easy to verify.)

Overriding malloc() caught the leak. The memory was being allocated by
libperl.so. It appears to be a bug in version 5.8.5 of the PERL shared
lib, though not in 5.8.0, when playing certain games with eval(). I
haven't tried anything newer.

It looks like there may be a secondary, much more occasional leak in
libxml2. I've had one leak there in the past.

[ Post a follow-up to this message ]