Hi all,

running the following small test program on different unices to test
malloc's out-of-memory error handling with this result:

FreeBSD 3.4-RELEASE   -  512 MB malloc: Cannot allocate memory
SunOS 5.7 sparc       - 1124 MB malloc: Resource temporarily unavailable
AIX 1 5               -  128 MB malloc: Not enough space
HP-UX B.11.00 9000/785- 1015 MB malloc: Not enough space

Linux 2.2.17              i686 -  588 MB malloc: Cannot allocate memory
Linux 2.4.19            x86_64 - 1958 MB Killed
Linux 2.4.17 	          i686 - 1802 MB Killed
Linux 2.4.17-mckinley-smp ia64 - 2015 MB Killed

-- snip ----
#include <stdlib.h>
#include <stdio.h>

#define MEG (1024*1024)

int main(int argc, const char* argv[]) {
int   chunksize = 128*1024;	/* 128 kB chunks */
unsigned long  sum = 0;	/* allocated memory */

printf("Allocating memory in chunks a %dkB\n", chunksize/1024);
for(;;) {
char* p;
int   i;

sum += chunksize;
if (sum % MEG == 0) {
fprintf(stdout, "\r%9lu MB ", sum/MEG);
fflush(stdout);
}
p = malloc(chunksize);
if(!p) {
perror("malloc");
return 1;
}
/* touch the allocated memory space */
for (i=0; i<chunksize; i+= 1024) p[i] = ' ';
};
fprintf(stdout, "\nDone.\n");
return 0;
}
-- snap ----


On newer Linux systems a call to malloc() only returns NULL if there are
no more addresses available. Since older Linux systems and other unices
returns NULL if there is no more physical memory availabel.
Does anyone know why this behavior has changed in Linux?

If my test program tries to access the memory address returned from
malloc it receives a kill signal (which can't be handled).
Any (simple) ideas how to realize that there is no more physical memory
availabel, so my program can terminate with an out-of-memory error
instead of being killed?


Thanks,
Pascal

[ Post a follow-up to this message ]

In comp.os.linux.development.system Pascal Bolzhauser <pascal@concept.de> wrote:
> On newer Linux systems a call to malloc() only returns NULL if there are
> no more addresses available. Since older Linux systems and other unices
> returns NULL if there is no more physical memory availabel.
> Does anyone know why this behavior has changed in Linux?

It hasn't. It's been configurable for ages. See "memory_overcommit" in
proc.

Up till 2.1 I think it was the default not to allocate memory on
demand, but only on write. Some people got annoyed at malloc always
saying "yes, fine" (calloc will be "better", since it zeros the memory)
and wrote their own gnu_malloc wrappers. So Linus made it configurable
and changed the default way back then to be no-overcommit. I've always
kept it as overcommit.

> If my test program tries to access the memory address returned from
> malloc it receives a kill signal (which can't be handled).

> Any (simple) ideas how to realize that there is no more physical memory
> availabel,

Don't use memory overcommit, or trample memory as you get it.

Peter

[ Post a follow-up to this message ]

Pascal Bolzhauser <pascal@concept.de> writes:

>FreeBSD 3.4-RELEASE   -  512 MB malloc: Cannot allocate memory
>SunOS 5.7 sparc       - 1124 MB malloc: Resource temporarily unavailable
>AIX 1 5               -  128 MB malloc: Not enough space
>HP-UX B.11.00 9000/785- 1015 MB malloc: Not enough space
>Linux 2.2.17              i686 -  588 MB malloc: Cannot allocate memory

Standards compliant.


>Linux 2.4.19            x86_64 - 1958 MB Killed
>Linux 2.4.17 	          i686 - 1802 MB Killed
>Linux 2.4.17-mckinley-smp ia64 - 2015 MB Killed

Non compliant.

I think you need to disable "lazy swap allocation" or whatever they
call it (enable memory overcommit?)

>If my test program tries to access the memory address returned from
>malloc it receives a kill signal (which can't be handled).
>Any (simple) ideas how to realize that there is no more physical memory
>availabel, so my program can terminate with an out-of-memory error
>instead of being killed?

The C standard says that such behaviour is not allowed.

Casper
--
Expressed in this posting are my opinions.  They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.

[ Post a follow-up to this message ]

>> If my test program tries to access the memory address returned from 
[vbcol=seagreen]
> The C standard says that such behaviour is not allowed.

Interesting.  I never noticed that it disallows it.  Do you remember which
part of the standard disallows it?


Stefan

[ Post a follow-up to this message ]

"P.T. Breuer" schrieb 
>
> It hasn't. It's been configurable for ages.
> See "memory_overcommit" in proc.
>
What's the idea behind overcommit?

Regards
Martin

[ Post a follow-up to this message ]

In comp.unix.programmer Martin Blume <mblume@socha.net> wrote:
> "P.T. Breuer" schrieb 
> What's the idea behind overcommit?

Many programs are rather bad behaved and malloc() lots and lots of
memory they never use (or only at a much later time), but that won't
work when malloc() fails. That led to the idea of having malloc()
return whatever they ask for but only to try to make good on that
promise when the program actually tries to access to that memory
and to kill the program (or some other program that asked for even
more memory) in case there's not enough memory left.

Regards, Jens
--
\   Jens Thoms Toerring  ___  Jens.Toerring@physik.fu-berlin.de
\__________________________  http://www.toerring.de

[ Post a follow-up to this message ]

Stefan Monnier wrote: 
>
> 
>
>
> Interesting.  I never noticed that it disallows it.  Do you remember which
> part of the standard disallows it?

This has been a topic of recurring debate in the C groups,
because the C Standard has no explicit language prohibiting
lazy allocation.  However, the Standard *does* say that if
malloc(non_zero_size) returns a non-NULL pointer, then all
the `non_zero_size' bytes are available for storing data.
An implementation that allows

int *p = malloc(sizeof *p);
if (p != NULL) *p = 42;

to fail is not a conforming implementation.

Of course, the same thing can be said about the ^C key,
CPU time limits, and pulling the electric plug: All these
things and more can cause a C program to stop when the Standard
says it should still be running, so all of them make the
implementation non-conforming.  "Usefully non-conforming," one
might say, which leads one to wonder why lazy allocation should
be singled out as Bad when control-C is recognized as Good ...

Personally, I am in the "lazy allocation is Bad" camp.
But I recognize that the infidels on the other side are not
completely devoid of reason.

--
Eric.Sosman@sun.com

[ Post a follow-up to this message ]

"Martin Blume" <mblume@socha.net> writes:

> "P.T. Breuer" schrieb 
> What's the idea behind overcommit?

Consider a 1GB process doing system("/bin/ls"); It would do if (fork()
== 0) exec ...  Now while the child is active the process temporarily
needs 2GB memory because in theory the child could touch and copy all
memory before calling exec. This used to be a real problem on Sun
based name servers. named would grow quite big, and it would
occasionally fork some small helpers. People had to add quite big swap
partitions that were never needed just to work around the true commit.
Another big issue is with older fortran programs. They don't allow
malloc() easily, so people just declared very big arrays, but only
used small parts of them. With a non overcommit system these fortran
programs don't load at all or only when you waste a lot of disk space
for never used swap space.

In practice having a lot of swap is also not a great advantage
anyways, even with true overcommit. When some process allocates much
more virtual memory than you have real memory it will thrash the whole
system with a swap storm instead of getting killed relatively quickly
when it gets out of control. The bigger the swap the worse the swap
storm.

So even turning off overcommit and adding swap doesn't help very
much. The only thing that helps is to never use significantly more
memory than you have real memory, but a lot of programs don't like
that.

-Andi

[ Post a follow-up to this message ]

"Andi Kleen" schrieb 
> 
>
>
> Consider a 1GB process doing system("/bin/ls");
> ...
> Another big issue is with older fortran programs.
> ...

Thanks, Andi and Jens for the explanations. Now it makes sense.

Regards
Martin

[ Post a follow-up to this message ]

In comp.os.linux.development.system Martin Blume <mblume@socha.net> wrote:
> "P.T. Breuer" schrieb 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^[vbcol=seagreen] 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[vb
col=seagreen] 
> What's the idea behind overcommit?

What he said (see underline above). And what I said.

Peter

[ Post a follow-up to this message ]