Unix Programming - finding memory leak in a long running process

Author

finding memory leak in a long running process - need tips

Shea Martin

2004-03-11, 12:35 pm

Sun Workshop 5.0, though I also have Sun Studio 9 demo compilers. With
Studio 9 I don't have a gui, so I have to use ddd as front end for
debugger.

I have a daemon which seems to have a memory leak, as its size has
tripled over the last 4 days.

Does anyone have any experience in detecting memory leaks in
daemons/servers/long-running-process?

Any tips on hunting down a leak? I don't have time to step through a
debugger for 2 days. Getting a core, and analyzing it doesn't really
help, as I can't see lost memory (that I know of). I am not sure if a
memcheck in the the debugger would work or not, as the process never
really ends. I tried print statements in all Constructor/Destructors,
and it appears to be 1:1, so I must be missing something elsewhere.

Any ideas would be great.

~Shea M.

Shea Martin

2004-03-11, 12:35 pm

Shea Martin wrote:
> Sun Workshop 5.0, though I also have Sun Studio 9 demo compilers. With
> Studio 9 I don't have a gui, so I have to use ddd as front end for
> debugger.
>
> I have a daemon which seems to have a memory leak, as its size has
> tripled over the last 4 days.
>
> Does anyone have any experience in detecting memory leaks in
> daemons/servers/long-running-process?
>
> Any tips on hunting down a leak? I don't have time to step through a
> debugger for 2 days. Getting a core, and analyzing it doesn't really
> help, as I can't see lost memory (that I know of). I am not sure if a
> memcheck in the the debugger would work or not, as the process never
> really ends. I tried print statements in all Constructor/Destructors,
> and it appears to be 1:1, so I must be missing something elsewhere.
>
> Any ideas would be great.
>
> ~Shea M.
>

I should mention that the daemon opens 3 files for writing. My process
has grown from 2.5 MB to 6.2 MB, and the total size of the 3 files is
only 9k.

~S

Sean Burke

2004-03-11, 1:34 pm

Shea Martin <smartin@arcis.com> writes:

> Shea Martin wrote:
>
> I should mention that the daemon opens 3 files for writing. My process
> has grown from 2.5 MB to 6.2 MB, and the total size of the 3 files is
> only 9k.

I have found tools like Purify to be useful for finding memory
leaks on Solaris, especially the subtler ones. More info on
Purify is at www.rational.com (which redirects to IBM now).

-SEan

carl mcguire

2004-03-11, 2:34 pm

Sean Burke wrote:
>
> Purify is at www.rational.com (which redirects to IBM now).
>
> -SEan
>

When did that happen? Does that increase the chances of seeing Pufify
for AIX? Pardon me if that's a stupid question.

Carl

Fletcher Glenn

2004-03-11, 2:34 pm

Shea Martin wrote:
> Shea Martin wrote:
>
>
> I should mention that the daemon opens 3 files for writing. My process
> has grown from 2.5 MB to 6.2 MB, and the total size of the 3 files is
> only 9k.
>
> ~S
>

If your daemon does a lot of memory allocation and deallocation, you may
be seeing a normal result of an allocator that does not do "garbage
collection". Whenever the allocator cannot find an appropriate block to
allocate, rather than looking for adjacent unallocated blocks to fill
the request, it will request more memory from the system. However, at
some point you tend to reach an equilibrium, but not necessarily.
Sometimes it helps a great deal to allocate large chunks, and re-use
those chunks yourself, only going to the allocator if you need more
memory. An example would be: if you are maintaining a dynamically sized
table, get 64 entries at a time instead of the single entry you need.
You have to keep count of the usage yourself.

--

Fletcher Glenn

Paul Pluzhnikov

2004-03-11, 11:34 pm

carl mcguire <abuse@spamcop.net> writes:

>
> When did that happen?

PureAtria was aquired by Rational in April of 1997.
Rational was aquired by IBM in February of 2003.

> Does that increase the chances of seeing Pufify for AIX?

I've heard that it is "in the works".

Purify competitors on AIX are Insure++ from ParaSoft and ZeroFault
from ZeroFault Group. Rather than waiting for Purify, you may want
to give competition a try.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

carl mcguire

2004-03-12, 6:34 am

Paul Pluzhnikov wrote:

> Purify competitors on AIX are Insure++ from ParaSoft and ZeroFault
> from ZeroFault Group. Rather than waiting for Purify, you may want
> to give competition a try.
>
> Cheers,

Already have done (both), but it's a political problem when developing
across AIX, Solaris, HP, Linux etc. It is difficult enough to get budget
approval for one of these tools, let alone two. Plus, we are partnered
with IBM ;-)

Thanks.

Paul Pluzhnikov

2004-03-12, 10:38 am

carl mcguire <abuse@spamcop.net> writes:

> but it's a political problem when developing across AIX, Solaris,
> HP, Linux etc.

Insure supports all of the above (on all processors). Purify doesn't.

> Plus, we are partnered with IBM ;-)

If your memory leak can wait; sure, go for it.

> It is difficult enough to get budget approval for one of these tools

This looks like a red herring to me: you tell your manager:

We have a problem [describe the problem and its effects] (if you
only have *one* problem, good for you).

I can look for it "by hand", and it will only take me a couple
of days to find (if I am lucky). And I would not know if I
introduced any new ones by fixing the ones I find.

Or I can use automation, find problems like this in an hour,
and *know* the problem is fixed. That will cost $X,000

Any manager in his right mind will make a quick calculation,
multiply savings by the number of developers, and make the right
decision :-)

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Nick Landsberg

2004-03-12, 2:36 pm

Paul Pluzhnikov wrote:
> carl mcguire <abuse@spamcop.net> writes:

[SNIP]

>=20
>=20
>=20
>=20
> This looks like a red herring to me: you tell your manager:
>=20
> We have a problem [describe the problem and its effects] (if you
> only have *one* problem, good for you).
>=20
> I can look for it "by hand", and it will only take me a couple
> of days to find (if I am lucky). And I would not know if I
> introduced any new ones by fixing the ones I find.
>=20
> Or I can use automation, find problems like this in an hour,
> and *know* the problem is fixed. That will cost $X,000
>=20
> Any manager in his right mind will make a quick calculation,
> multiply savings by the number of developers, and make the right
> decision :-)
>=20
> Cheers,

Paul,

1. First, you assume that a manager has a mind

2. There are situations in certain business climates
where the bean counters have edicted "Thou shalt not purchase
ANYTHING without a 'business case' and executive level approval."

Often, by the time you get that (sometimes weeks), the
current problem has been solved and there is no longer
a perceived need for the product ... until the next time
it (in this case a memory leak) happens.

As the old saying goes, "penny wise and pound foolish."

--=20
=D1
"It is impossible to make anything foolproof because fools are so=20
ingenious" - A. Bloch

Shea Martin

2004-03-12, 2:36 pm

Paul Pluzhnikov wrote:
> carl mcguire <abuse@spamcop.net> writes:
>
>
>
>
> Insure supports all of the above (on all processors). Purify doesn't.
>
>
>
>
> If your memory leak can wait; sure, go for it.
>
>
>
>
> This looks like a red herring to me: you tell your manager:
>
> We have a problem [describe the problem and its effects] (if you
> only have *one* problem, good for you).
>
> I can look for it "by hand", and it will only take me a couple
> of days to find (if I am lucky). And I would not know if I
> introduced any new ones by fixing the ones I find.
>
> Or I can use automation, find problems like this in an hour,
> and *know* the problem is fixed. That will cost $X,000
>
> Any manager in his right mind will make a quick calculation,
> multiply savings by the number of developers, and make the right
> decision :-)
>
> Cheers,
As I junior developer, I am worried that my manager (not a programmer) would
feel that I am not good enough to find it by hand, and a poor programmer to find
the leak in the first place.
~S

Shea Martin

2004-03-12, 3:35 pm

Fletcher Glenn wrote:
>
>
> Shea Martin wrote:
>
>
> If your daemon does a lot of memory allocation and deallocation, you may
> be seeing a normal result of an allocator that does not do "garbage
> collection". Whenever the allocator cannot find an appropriate block to
> allocate, rather than looking for adjacent unallocated blocks to fill
> the request, it will request more memory from the system. However, at
> some point you tend to reach an equilibrium, but not necessarily.
> Sometimes it helps a great deal to allocate large chunks, and re-use
> those chunks yourself, only going to the allocator if you need more
> memory. An example would be: if you are maintaining a dynamically sized
> table, get 64 entries at a time instead of the single entry you need.
> You have to keep count of the usage yourself.
>
> --
>
> Fletcher Glenn
>

Basically the allocation is for items in a Queue. Every connection gets a "new"
request handler (thread) which is deleted after it's run methods. I have
verified that all threads are being properly deleted. I am not allocating
anything on the heap in the constructor of my request thread.

The daemon stores items in a queue. Seeing as I never know how large the queues
is going to be, I used a std::map as the underlying storage facility. The map
stores pointers to a Job object. I suspect the leak has something to to with this.

Sun's mem checker keeps giving me a RUI in this situation:
char *buf = new char[512];
somefunc(buf);

where somefunc(char *givenbuf) does this:

strlen(givenbuf); //this gives a RUI access violation.

I have got hung up trying to fix this RUI, and got off track from tracking a
possible leak. :-) Any know how to fix this RUI problem? Note it does not
*appear* to be a real problem at runtime.

~S

Joe Halpin

2004-03-12, 3:35 pm

Shea Martin <shea@snowsquirrel.ca> writes:

> As I junior developer, I am worried that my manager (not a
> programmer) would feel that I am not good enough to find it by hand,
> and a poor programmer to find the leak in the first place.

There are other options besides buying purify or whatever. Take a look
at

http://dmalloc.com/

for a free debugging malloc replacement you can link in while trying
to debug. More options (some free some not) are at

http://www.cs.colorado.edu/homes/zo...allocDebug.html

Joe

Paul Pluzhnikov

2004-03-13, 3:33 am

Joe Halpin <jhalpin@nortelnetworks.com_.nospam> writes:

> Shea Martin <shea@snowsquirrel.ca> writes:
>

If he is not a programmer, explain to him that finding memory leaks
and other problems like this "by hand" is akin to digging a trench
1 meter deep with a showel. It can be done, but if your trench
is longer than about 2 meters, you'll be *much* better of with a
specialized tool.
[color=darkred]
> There are other options besides buying purify or whatever. Take a look

True. And much can be learned while trying to apply them, especially
on AIX, where intercepting all malloc()s is more challenging than
on most other UNIXes.

Shea Martin <shea@snowsquirrel.ca> writes:

> Sun's mem checker keeps giving me a RUI in this situation:
> char *buf = new char[512];
> somefunc(buf);
> where somefunc(char *givenbuf) does this:
> strlen(givenbuf); //this gives a RUI access violation.

Sun's mem checker happens to be correct.

> I have got hung up trying to fix this RUI, and got off track from
> tracking a possible leak. :-) Any know how to fix this RUI problem?

I can see that you *really* need more than just malloc debugger.
It is good to see that you are at least trying to use tools available
to you.

To fix the RUI, do this:

char *buf = new char[512];
buf[0] = '\0';
somefunc(buf);

> Note it does not *appear* to be a real problem at runtime.

Oh, it *is* a real problem, and can cause your program to crash at
the most innopportune moment. If you haven't seen the crash yet,
it is because the *most* inopportune moment has not happened yet :-)

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Shea Martin

2004-03-13, 11:34 am

Paul Pluzhnikov wrote:
> Joe Halpin <jhalpin@nortelnetworks.com_.nospam> writes:
>
>
>
>
> If he is not a programmer, explain to him that finding memory leaks
> and other problems like this "by hand" is akin to digging a trench
> 1 meter deep with a showel. It can be done, but if your trench
> is longer than about 2 meters, you'll be *much* better of with a
> specialized tool.
>
>
>
>
> True. And much can be learned while trying to apply them, especially
> on AIX, where intercepting all malloc()s is more challenging than
> on most other UNIXes.
>
> Shea Martin <shea@snowsquirrel.ca> writes:
>
>
>
>
> Sun's mem checker happens to be correct.
>
>
>
>
> I can see that you *really* need more than just malloc debugger.
> It is good to see that you are at least trying to use tools available
> to you.
>
> To fix the RUI, do this:
>
> char *buf = new char[512];
> buf[0] = '\0';
> somefunc(buf);
>
>
>
>
> Oh, it *is* a real problem, and can cause your program to crash at
> the most innopportune moment. If you haven't seen the crash yet,
> it is because the *most* inopportune moment has not happened yet :-)
>
> Cheers,

Ok I am not that dumb. My example should have been:

char *buf = new char[512];
int read = read(socket_descriptor, buf, 512);
buf[read] = 0;
somefunc(buf);

where somefunc(char *givenbuf) does this:
strlen(givenbuf); //this gives a RUI access violation.

Now why does this give me a RUI?

Nick Landsberg

2004-03-13, 12:34 pm

Shea Martin wrote:

> Paul Pluzhnikov wrote:
>=20
[color=darkred]
k[color=darkred]
>=20
>=20
> Ok I am not that dumb. My example should have been:
>=20
> char *buf =3D new char[512];
> int read =3D read(socket_descriptor, buf, 512);
> buf[read] =3D 0;
> somefunc(buf);
>=20
> where somefunc(char *givenbuf) does this:
> strlen(givenbuf); //this gives a RUI access violation.
>=20
> Now why does this give me a RUI?

Conjecture:

You never checked that the "read" variable was >=3D0.
It could have returned -1. In which case you are
setting some unknown location to 0;

As it happens, some implementations of
malloc() use the memory location just BEFORE
the pointer they return to store the
size of the memory segment (in order to
be able to manage the memory pool).

If you actually change that value, free() may
not be able to do the right bookkeeping.

Assume that at buf[-1] there was a some
portion of the representation of the size
(512) in this case. It was probably stored
as some kind of int, either 16 bit or 32 bit.
You may have just zeroed out some of
bits of that int value. (which ones get
zeroed depends on the endianness of your machine).

The strlen() may be giving you the access
violation because memory allocated by your
"new" may not be zeroing out the allocated
memory, and strlen() goes off into never-never
land looking for the '\0' character
which it may never find because you wrote
the '\0' to the wrong place.

--=20
=D1
"It is impossible to make anything foolproof because fools are so=20
ingenious" - A. Bloch

Paul Pluzhnikov

2004-03-13, 1:35 pm

Shea Martin <shea@snowsquirrel.ca> writes:

[You need to snip a bit more when quoting.]

> Ok I am not that dumb. My example should have been:

Good to know :-)

There are multiple problems with your code:

> char *buf = new char[512];
> int read = read(socket_descriptor, buf, 512);
> buf[read] = 0;

If read() fails and returns -1, you'll write into buf[-1] and corrupt
the heap. You should always check return values from system calls.

If read succeeds in reading 512 bytes, you'll write into buf[512]
(which is one byte past the end of buf) and corrupt heap again.

Also, assignment should be "buf[read] = '\0';" -- there is a
difference between 0, NULL and '\0' (aka NUL), don't mix them up.

> somefunc(buf);
>
> where somefunc(char *givenbuf) does this:
> strlen(givenbuf); //this gives a RUI access violation.
>
> Now why does this give me a RUI?

Don't know. It should have given you an WUA -- write to unallocated
memory. What is the exact RUI message?

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Bjorn Reese

2004-03-14, 4:34 am

On Sat, 13 Mar 2004 09:45:10 -0800, Paul Pluzhnikov wrote:

> Also, assignment should be "buf[read] = '\0';" -- there is a
> difference between 0, NULL and '\0' (aka NUL), don't mix them up.

My preference is either 0 or (char)0, if I want to be explicit.
The simple reason is that I have seen too many people spent too
much time looking at '0' thinking that they are seeing '\0' during
code reviews and debugging sessions.

--
mail1dotstofanetdotdk

Frank Cusack

2004-03-14, 6:33 am

On 13 Mar 2004 09:45:10 -0800 Paul Pluzhnikov <ppluzhnikov-nsp@charter.net> wrote:
> Also, assignment should be "buf[read] = '\0';" -- there is a
> difference between 0, NULL and '\0' (aka NUL), don't mix them up.

What is the difference between 0 and NULL?

I thought NULL was *defined to be* 0. Checking on a glibc-2.3 RH9 system
and a Solaris 9 system, both define it similarly. glibc goes through
the effort of defining it as void *0 for C (but not C++), although I
don't understand the point there either, I thought unadorned 0 was
*defined to be* void *0.

'\0' aka NUL is of course, a different thing.

/fc

Erik Max Francis

2004-03-14, 7:35 am

Frank Cusack wrote:

> What is the difference between 0 and NULL?

0 is an integral constant. In a pointer context, it's the null pointer
constant. NULL is a preprocessor macro which expands to the null
pointer constant (but it's implementation defined exactly what it
expands to). In C, (void *) 0 is also a valid null pointer constant and
so NULL can be defined as that as well, but in C++ it cannot (implicit
casts from void * are disallowed).

> I thought NULL was *defined to be* 0.

That's certainly a valid definition of NULL. In a null pointer context,
there's no difference between 0 and NULL.

The tricky part gets to where you're not in a null pointer context
(varargs argument list, no prototype in scope) and you need a null
pointer constant. In those cases the proper full cast (not just void *)
is required, whether you're using 0 or NULL.

--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
\__/ Oh, what lies there are in kisses.
-- Heinrich Heine

Erik Max Francis

2004-03-14, 7:35 am

Paul Pluzhnikov wrote:

> Also, assignment should be "buf[read] = '\0';" -- there is a
> difference between 0, NULL and '\0' (aka NUL), don't mix them up.

Then you may be surprised to realize there's no difference between 0 and
'\0' in C. They're both constants of type int with the same value.
(Not true in C++, where char literals are of type char.)

--
__ Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
/ \ San Jose, CA, USA && 37 20 N 121 53 W && &tSftDotIotE
\__/ Oh, what lies there are in kisses.
-- Heinrich Heine

Frank Cusack

2004-03-14, 3:35 pm

On Sun, 14 Mar 2004 03:42:06 -0800 Erik Max Francis <max@alcyone.com> wrote:
> Frank Cusack wrote:
>
....[color=darkred]
>
> The tricky part gets to where you're not in a null pointer context
> (varargs argument list, no prototype in scope) and you need a null
> pointer constant. In those cases the proper full cast (not just void *)
> is required, whether you're using 0 or NULL.

ahh ... very enlightening!

Kedar

2004-03-15, 4:34 am

Hi,
I am many sytem() function calls many times in my C program.But
after 1-2 successful executions the daemon crashes with the error:
Hi all,
I have written a multithreaded program in C/UNIX using POSIX
threads (pthreads).
In the code,I am calling a child process using exec and also a system
call.
These calls are made n number of times.
After 3-4 (average) executions the daemon crashes with error as
follws:

********** Internal heap ERROR KGHALO2 addr=0x0 *********

****************************************
**************
HEAP DUMP heap name="Alloc environm" desc=0x848a90c
extent sz=0x1024 alt=32 het=32767 rec=0 flg=2 opc=2
parent=0 owner=0 nex=0 xsz=0x1024
EXTENT 0
Chunk 892c778 sz= 2832 free " "
Chunk 892d288 sz= 1292 freeable assoc with mark prv=0
EXTENT 1
Chunk 84b75d0 sz= 252 free " "
Chunk 84b76cc sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b76f4 sz= 208 free " "
Chunk 84b77c4 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b77ec sz= 164 free " "
Chunk 84b7890 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b78b8 sz= 124 freeable assoc with mark prv=0 nxt=0
Chunk 84b7934 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b79a0 sz= 532 free " "
Chunk 84b7bb4 sz= 428 freeable assoc with mark prv=0 nxt=0
Chunk 84b7d60 sz= 284 free " "
Chunk 84b7e7c sz= 20 freeable assoc with mark prv=0 nxt=0
Chunk 84b7e90 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7eb8 sz= 28 free " "
Chunk 84b7ed4 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7efc sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7f24 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7f4c sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7f74 sz= 28 free " "
Chunk 84b7f90 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7fb8 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b7fe0 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b8008 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b8030 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b8058 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b8080 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b80a8 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b80d0 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84b80f8 sz= 40 freeable assoc with mark prv=0
nxt=084bb054 sz= 84 free " "
Chunk 84bb0a8 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84bb0d0 sz= 1212 free " "
Chunk 84bb58c sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84bb5b4 sz= 40 freeable assoc with mark prv=0 nxt=0
Chunk 84bb5dc sz= 40 freeable assoc with mark prv=0 nxt=0
EXTENT 3
Chunk 8498748 sz= 14552 freeable "Alloc server h "
ds=84ba974
EXTENT 4
Chunk 8497528 sz= 24 perm "perm " alo=24
Chunk 8497540 sz= 2772 recreate "Alloc server h " latch=0
ds 84ba974 sz= 17324 ct= 2
8498748 sz= 14552
Chunk 8498014 sz= 1312 freeable assoc with mark prv=0 nxt=0
Bucket 0 size=272
Chunk 84b7d60 sz= 284 free " "
Chunk 84b76f4 sz= 208 free " "
Chunk 84b75d0 sz= 252 free " "
Chunk 84bb054 sz= 84 free " "
Chunk 84b7eb8 sz= 28 free " "
Chunk 84baaa4 sz= 44 free " "
Chunk 84baaf8 sz= 128 free " "
Chunk 84b77ec sz= 164 free " "
Chunk 84ba5e8 sz= 236 free " "
Chunk 84b7f74 sz= 28 free " "
Chunk 84b8580 sz= 28 free " "
Chunk 84b795c sz= 28 free " "
Bucket 1 size=528
Chunk 84b79a0 sz= 532 free " "
Bucket 2 size=1040
Chunk 892c778 sz= 2832 free " "
Chunk 84baba0 sz= 1084 free " "
Chunk 84bb0d0 sz= 1212 free " "
Total free space = 7172
UNPINNED RECREATABLE CHUNKS (lru first):
PERMANENT CHUNKS:
Chunk 8497528 sz= 24 perm "perm " alo=24
Permanent space = 24
****************************************
**************
Hla: 0
kghalo bad size 0x720025e0
********** Internal heap ERROR KGHALO2 addr=0x0 *********

****************************************
**************
HEAP DUMP heap name="Alloc environm" desc=0x848a90c
extent sz=0x1024 alt=32 het=32767 rec=0 flg=2 opc=2
parent=0 owner=0 nex=0 xsz=0x1024

Pls could you guide me on what could be reason for this? and how this
could be avoided?

Waiting for your reply.

Warm Regards
Kedar

Fletcher Glenn <fletcher@removethisfoglight.com> wrote in message news:<4050B340.4030408@removethisfoglight.com>...
> Shea Martin wrote:
>
> If your daemon does a lot of memory allocation and deallocation, you may
> be seeing a normal result of an allocator that does not do "garbage
> collection". Whenever the allocator cannot find an appropriate block to
> allocate, rather than looking for adjacent unallocated blocks to fill
> the request, it will request more memory from the system. However, at
> some point you tend to reach an equilibrium, but not necessarily.
> Sometimes it helps a great deal to allocate large chunks, and re-use
> those chunks yourself, only going to the allocator if you need more
> memory. An example would be: if you are maintaining a dynamically sized
> table, get 64 entries at a time instead of the single entry you need.
> You have to keep count of the usage yourself.

Nils Petter Vaskinn

2004-03-15, 5:35 am

On Fri, 12 Mar 2004 19:41:17 +0000, Shea Martin wrote:

> Sun's mem checker keeps giving me a RUI in this situation:
> char *buf = new char[512];

Allocate memory for a series of 512 chars containing random garbage.

> somefunc(buf);

Call function.

> where somefunc(char *givenbuf) does this:
>
> strlen(givenbuf); //this gives a RUI access violation.

Treat random garbage as if it was a string. Meaning strlen will keep going
until it hits a char that has the value '\0'

Add the line:

buf[0] = '\0';

after alloocating the memory to make it a null terminated c string. But
maybe you should consider std::string.

--
NPV

"the large print giveth, and the small print taketh away"
Tom Waits - Step right up

carl mcguire

2004-03-15, 6:34 am

Paul Pluzhnikov wrote:
> carl mcguire <abuse@spamcop.net> writes:
>
>
>
>
> Insure supports all of the above (on all processors). Purify doesn't.
>
True, but last time I got quotes Purify was cheaper. Times are hard...

>
>
> If your memory leak can wait; sure, go for it.
>
The trial version of ZeroFault on AIX tells me there are no leaks there
at the moment, as does dbx on Solaris with check -leaks option.

>
>
> This looks like a red herring to me: you tell your manager:
>
> We have a problem [describe the problem and its effects] (if you
> only have *one* problem, good for you).
>
> I can look for it "by hand", and it will only take me a couple
> of days to find (if I am lucky). And I would not know if I
> introduced any new ones by fixing the ones I find.
>
> Or I can use automation, find problems like this in an hour,
> and *know* the problem is fixed. That will cost $X,000
>
> Any manager in his right mind will make a quick calculation,
> multiply savings by the number of developers, and make the right
> decision :-)
>
> Cheers,

When we have such a problem, such an argument is pretty much cut and
dried. I prefer to not get into the situation in the first place but
when was the last time you met a proactive manager? ;-)

Thanks,
Carl

Casper H.S. Dik

2004-03-15, 10:36 am

carl mcguire <abuse@spamcop.net> writes:

>The trial version of ZeroFault on AIX tells me there are no leaks there
>at the moment, as does dbx on Solaris with check -leaks option.

Since some update of Solaris 9, there's "libumem" which allows
you to debug memory leaks and other problems without intervention
of a debugger.

Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

Shea Martin

2004-03-15, 1:34 pm

Shea Martin wrote:
> Fletcher Glenn wrote:
>
>
> Basically the allocation is for items in a Queue. Every connection gets
> a "new" request handler (thread) which is deleted after it's run
> methods. I have verified that all threads are being properly deleted.
> I am not allocating anything on the heap in the constructor of my
> request thread.
>
> The daemon stores items in a queue. Seeing as I never know how large
> the queues is going to be, I used a std::map as the underlying storage
> facility. The map stores pointers to a Job object. I suspect the leak
> has something to to with this.
>
> Sun's mem checker keeps giving me a RUI in this situation:
> char *buf = new char[512];
> somefunc(buf);
>
> where somefunc(char *givenbuf) does this:
>
> strlen(givenbuf); //this gives a RUI access violation.
>
> I have got hung up trying to fix this RUI, and got off track from
> tracking a possible leak. :-) Any know how to fix this RUI problem?
> Note it does not *appear* to be a real problem at runtime.
>
> ~S
I am checking for read to return a >0. Before passing the buffer. In
fact, I have stepped through the debugger to verify that there is a
string in the buffer. The string starts at the correct memory address,
etc. Yet when strlen() gets the string, I get the RUI.

~S

David Schwartz

2004-03-15, 2:36 pm

"Shea Martin" <smartin@arcis.com> wrote in message
news:OIm5c.2948$G3.25919@localhost...

Temporarily, add this line before the call to 'somefunc':

memset(buf, 32, 512);

See if the RUI goes away or if you get a crash.
[color=darkred]
> I am checking for read to return a >0. Before passing the buffer. In
> fact, I have stepped through the debugger to verify that there is a string
> in the buffer. The string starts at the correct memory address, etc. Yet
> when strlen() gets the string, I get the RUI.

Did you make sure the string is terminated with a zero?

DS

Shea Martin

2004-03-15, 2:36 pm

David Schwartz wrote:
> "Shea Martin" <smartin@arcis.com> wrote in message
> news:OIm5c.2948$G3.25919@localhost...
>
>
>
>
>
>
> Temporarily, add this line before the call to 'somefunc':
>
> memset(buf, 32, 512);
A memset fixed the problem. Though I am not sure why. I had a null
terminater on the string in 'buf'. There was definitely valid contents
in 'buf', so strlen should not have accessed anything beyond the NULL.

Anyway, the memset puts my mind at ease. Thanks.

>
> See if the RUI goes away or if you get a crash.
>
>
>
>
>
> Did you make sure the string is terminated with a zero?
definitely.
>
> DS
>

~S

Sean Burke

2004-03-15, 8:34 pm

Shea Martin <smartin@arcis.com> writes:

> David Schwartz wrote:
> A memset fixed the problem. Though I am not sure why. I had a null
> terminater on the string in 'buf'. There was definitely valid contents
> in 'buf', so strlen should not have accessed anything beyond the NULL.
>
> Anyway, the memset puts my mind at ease. Thanks.

I've seen this sort of thing in the past using Purify.

I've always suspected that Sun's strlen() was using the
algorithm that reads four bytes at a time, resulting in
reads of one or more uninitialized bytes beyond the
null character.

Can anyone comment whether this is the explanation?

-SEan

Paul Pluzhnikov

2004-03-16, 5:36 am

Sean Burke <foobar@mystery.org> writes:

> I've always suspected that Sun's strlen() was using the
> algorithm that reads four bytes at a time, resulting in
> reads of one or more uninitialized bytes beyond the
> null character.

It does. The main loop looks like this:

# on entry %o1 is 4-byte aligned pointer into the string,
# %o3 == 0x7efefeff; %o4 == 0x81010100

0xff2b6f2c: strlen+0x0080: ld [%o1], %o2 # load 4 bytes
0xff2b6f30: strlen+0x0084: add %o1, 0x4, %o1 # point to next 4 bytes
0xff2b6f34: strlen+0x0088: add %o2, %o3, %o5
0xff2b6f38: strlen+0x008c: xor %o5, %o2, %o5
0xff2b6f3c: strlen+0x0090: and %o5, %o4, %o5
0xff2b6f40: strlen+0x0094: cmp %o5, %o4
0xff2b6f44: strlen+0x0098: be,a strlen+0x80 # next 4 bytes, unless NUL was found,
# or one of the char had high bit set

> Can anyone comment whether this is the explanation?

My trivial test cases did not produce a RUM under purify, nor RUI
under dbx, so I don't know ...

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Shea Martin

2004-03-16, 6:35 am

Paul Pluzhnikov wrote:
> Sean Burke <foobar@mystery.org> writes:
>
>
>
>
> It does. The main loop looks like this:
>
> # on entry %o1 is 4-byte aligned pointer into the string,
> # %o3 == 0x7efefeff; %o4 == 0x81010100
>
> 0xff2b6f2c: strlen+0x0080: ld [%o1], %o2 # load 4 bytes
> 0xff2b6f30: strlen+0x0084: add %o1, 0x4, %o1 # point to next 4 bytes
> 0xff2b6f34: strlen+0x0088: add %o2, %o3, %o5
> 0xff2b6f38: strlen+0x008c: xor %o5, %o2, %o5
> 0xff2b6f3c: strlen+0x0090: and %o5, %o4, %o5
> 0xff2b6f40: strlen+0x0094: cmp %o5, %o4
> 0xff2b6f44: strlen+0x0098: be,a strlen+0x80 # next 4 bytes, unless NUL was found,
> # or one of the char had high bit set
>
>
>
>
>
> My trivial test cases did not produce a RUM under purify, nor RUI
> under dbx, so I don't know ...
>
> Cheers,
Thanks, that is good to know.

Maybe someone should let the dbx team know. ;-)
~S

Bjorn Reese

2004-03-16, 2:36 pm

On Tue, 16 Mar 2004 00:55:17 +0000, Sean Burke wrote:

> I've seen this sort of thing in the past using Purify.

I've seen similar behavior (Uninitialized Memory Read)
for write(). Allocating and initializing aligned memory
made the problem go away. I drew the same conclusion as
you about reading words, and ended up suppressing all
UMR from write() in my .purify.

However, I am more inclined to believe that the original
poster has an off-by-one error. If 512 bytes are read in
the code below, then the 'read' variable (kind of confusing
to use the same name for a variable and a function) is 512,
and buf[read] will write one byte past the end of the buffer.

- - - - - - - - - - - - - - - - - - - - - - - -
char *buf = new char[512];
int read = read(socket_descriptor, buf, 512);
buf[read] = 0;
- - - - - - - - - - - - - - - - - - - - - - - -

--
mail1dotstofanetdotdk

David Schwartz

2004-03-16, 6:40 pm

"Bjorn Reese" <breese@see.signature> wrote in message
news:pan.2004.03.16.19.17.06.195042@see.signature...

> However, I am more inclined to believe that the original
> poster has an off-by-one error. If 512 bytes are read in
> the code below, then the 'read' variable (kind of confusing
> to use the same name for a variable and a function) is 512,
> and buf[read] will write one byte past the end of the buffer.
>
> - - - - - - - - - - - - - - - - - - - - - - - -
> char *buf = new char[512];
> int read = read(socket_descriptor, buf, 512);
> buf[read] = 0;
> - - - - - - - - - - - - - - - - - - - - - - - -

Not likely, because he said a 'memset' fixed it.

DS

Bjorn Reese

2004-03-18, 3:35 pm

On Tue, 16 Mar 2004 14:06:52 -0800, David Schwartz wrote:

> Not likely, because he said a 'memset' fixed it.

Sorry, I missed that.

He still has an off-by-one problem through.

--
mail1dotstofanetdotdk