I am using Apache 2.2.2 and when using mod_python in a certain way, I am see
ing
significant ongoing increases in memory use by Apache child processes.
Initially I thought I had stuffed up recent changes in mod_python 3.3 out of
subversion trunk that I had been making, but I went back to mod_python 3.2.1
0
and am seeing the same problem.

Can some one please run the following test and use "top" or some other means
 of
monitoring memory use to see if you can duplicate the problem. The test is r
eally
quite simple actually.

# .htaccess

PythonFixupHandler handlers
AddHandler mod_python .py
PythonHandler handlers

# handlers.py

from mod_python import apache

def handler(req):
req.content_type = 'text/plain'
req.write('handler')
return apache.OK

def fixuphandler(req):
return apache.OK

The command I have been using to test is:

ab -c 1 -n 5000 http://localhost:8082/~grahamd/MODPYTHON-155/hello.py

What is strange is that if I have only the fixup handler or response handler
invoked and not both, then memory leak isn't present. It is only when both a
re
being invoked that it leaks memory.

This occurs for me on Mac OS X 10.4, Python 2.3.5 and Apache 2.2.2, with
either mod_python 3.2.10 or mod_python 3.3. I haven't yet tested with older
Apache 2.0.58 as yet as don't have it on my box at present, but will downloa
d
it and see if that makes a difference.

I guess what I am trying to working out is if this is an issue with Apache 2
.2.2
or whether it is mod_python and problem has been there for a while. Thus if
people with both Apache 2.0.X and Apache 2.2.X could test it it would be gre
at.

Any help appreciated.

Thanks.

Graham

[ Post a follow-up to this message ]

I get it on Apache 2.0.59 as well. :-(

I will thus be interested to see what others get, as appears to be an existi
ng
mod_python issue.

BTW, this is with worker MPM.

Graham

Graham Dumpleton wrote ..
> I am using Apache 2.2.2 and when using mod_python in a certain way, I am
> seeing
> significant ongoing increases in memory use by Apache child processes.
> Initially I thought I had stuffed up recent changes in mod_python 3.3 out
> of
> subversion trunk that I had been making, but I went back to mod_python
> 3.2.10
> and am seeing the same problem.
>
> Can some one please run the following test and use "top" or some other
> means of
> monitoring memory use to see if you can duplicate the problem. The test
> is really
> quite simple actually.
>
> # .htaccess
>
> PythonFixupHandler handlers
> AddHandler mod_python .py
> PythonHandler handlers
>
> # handlers.py
>
> from mod_python import apache
>
> def handler(req):
>   req.content_type = 'text/plain'
>   req.write('handler')
>   return apache.OK
>
> def fixuphandler(req):
>   return apache.OK
>
> The command I have been using to test is:
>
>   ab -c 1 -n 5000 http://localhost:8082/~grahamd/MODPYTHON-155/hello.py
>
> What is strange is that if I have only the fixup handler or response handl
er
> invoked and not both, then memory leak isn't present. It is only when both
> are
> being invoked that it leaks memory.
>
> This occurs for me on Mac OS X 10.4, Python 2.3.5 and Apache 2.2.2, with
> either mod_python 3.2.10 or mod_python 3.3. I haven't yet tested with olde
r
> Apache 2.0.58 as yet as don't have it on my box at present, but will downl
oad
> it and see if that makes a difference.
>
> I guess what I am trying to working out is if this is an issue with Apache
> 2.2.2
> or whether it is mod_python and problem has been there for a while. Thus
> if
> people with both Apache 2.0.X and Apache 2.2.X could test it it would be
> great.
>
> Any help appreciated.
>
> Thanks.
>
> Graham

[ Post a follow-up to this message ]

The good news is that this isn't a problem introduced with mod_python
3.2.10
so doesn't necessarily have to stop that being released.

The bad news though is that is is also in mod_python 3.2.8, so it has
been
there for some time.

The question now is whether anyone else sees it on other platforms or
whether
it is something specific to my machine.

Graham

On 31/07/2006, at 4:53 PM, Graham Dumpleton wrote:
[vbcol=seagreen]
> I get it on Apache 2.0.59 as well. :-(
>
> I will thus be interested to see what others get, as appears to be
> an existing
> mod_python issue.
>
> BTW, this is with worker MPM.
>
> Graham
>
> Graham Dumpleton wrote .. 

[ Post a follow-up to this message ]

Okay, found the source of the memory leak. The problem goes right back
to 3.1.4 which also has the problem when tested.

The problem code is in python_handler() in 'src/mod_python.c'.
Specifically
the code does:

if (!hle) {
/* create a handler list object from dynamically registered
handlers */
request_obj->hlo = (hlistobject *)MpHList_FromHLEntry
(dynhle, 1);
}
else {
/* create a handler list object */
request_obj->hlo = (hlistobject *)MpHList_FromHLEntry(hle, 1);

/* add dynamically registered handlers, if any */
if (dynhle) {
MpHList_Append(request_obj->hlo, dynhle);
}
}

Problem is that request_obj->hlo can already be set by a prior phase's
handler and by simply assigning to request_obj->hlo you get a memory
leak as it is a Python object and it isn't being decref'd.

Thus, before this 'if' statement, it would appear that the following
should
be inserted.

if (request_obj->hlo)
Py_DECREF(request_obj->hlo);

or:

Py_XDECREF(request_obj->hlo);

Still need to do some more checking, but adding that seems to get rid of
the problem.

Now what do we do about 3.2.10? Given that this thing leaks really badly
when triggered shows that no one must be using multiple handler phases
at the same time, so may be safe to still release 3.2.10 and we fix
it in next
backport release and 3.3.

Comments.

Graham

On 31/07/2006, at 7:24 PM, Graham Dumpleton wrote:
[vbcol=seagreen]
> The good news is that this isn't a problem introduced with
> mod_python 3.2.10
> so doesn't necessarily have to stop that being released.
>
> The bad news though is that is is also in mod_python 3.2.8, so it
> has been
> there for some time.
>
> The question now is whether anyone else sees it on other platforms
> or whether
> it is something specific to my machine.
>
> Graham
>
> On 31/07/2006, at 4:53 PM, Graham Dumpleton wrote:
> 

[ Post a follow-up to this message ]

Graham Dumpleton wrote:
> Okay, found the source of the memory leak. The problem goes right back
> to 3.1.4 which also has the problem when tested.
..
> Now what do we do about 3.2.10? Given that this thing leaks really badly
> when triggered shows that no one must be using multiple handler phases
> at the same time, so may be safe to still release 3.2.10 and we fix it
> in next backport release and 3.3.

Since this is a confirmed non-regression, especially one which has
existed for a long time, I don't think it makes sense to call off 3.2.10
- especially not at this late stage when all that is still needed is
official rubber-stamping of the release.

There are two very important new features coming in this release:
* Apache 2.2 support
* don't die in configure when using a recent bash

Max.

Attachment: signature.asc
This has been downloaded 0 time(s).

[ Post a follow-up to this message ]

Here is further confirmation that it leaks like crazy for:

mod_python 3.2.10, Linux Ubuntu 6.06, Apache 2.0.55 (mpm-worker), Python
2.4.3

Jim

Graham Dumpleton wrote:
> I get it on Apache 2.0.59 as well. :-(
>
> I will thus be interested to see what others get, as appears to be an exis
ting
> mod_python issue.
>
> BTW, this is with worker MPM.
>
> Graham
>
> Graham Dumpleton wrote .. 
>

[ Post a follow-up to this message ]

Max Bowsher wrote:
> Graham Dumpleton wrote: 
> ... 
>
> Since this is a confirmed non-regression, especially one which has
> existed for a long time, I don't think it makes sense to call off 3.2.10
> - especially not at this late stage when all that is still needed is
> official rubber-stamping of the release.
>
> There are two very important new features coming in this release:
>   * Apache 2.2 support
>   * don't die in configure when using a recent bash
>

Agreed. Hopefully Grisha will have a chance to sign and upload the
3.2.10 tarball in the near future.

Jim

[ Post a follow-up to this message ]