Incremental memory use when performing streaming over long period.
------------------------------------------------------------------

Key: MODPYTHON-241
URL: https://issues.apache.org/jira/browse/MODPYTHON-241
Project: mod_python
Issue Type: Bug
Components: core
Affects Versions: 3.3.1, 3.2.10
Reporter: Graham Dumpleton
Priority: Minor


As first raised in:

http://www.modpython.org/pipermail/...uly/023964.html

when streaming data from mod_python handler over a long period of time, memo
ry use will keep increasing.

Note that this isn't actually a memory leak though, as the memory gets recla
imed at the end of the request.

The underlying problem arises from mod_python using the ap_rwrite() function
 to write output data. When this function is used it will create a bucket ch
ain for each block of data written, with the bucket chain coming from the re
quest memory pool. There is
no attempt to reuse the bucket chain for successive writes, so memory will i
ncrease on every call.

The normal solution to this problem is to retain the bucket chain and use it
 from one call to the next. In mod_python though this isn't simple to achiev
e because calls to response data may or may not be flushed at the discretion
 of the user code. The reta
ining of the bucket chain and using it from one call to the next generally c
an only be done when data is always being flushed. A more complicated system
 would be needed which can cope with a need to buffer data at times and yet 
flush at other times.

One approach may be to not actually use Apache itself to buffer data between
 flushes, but to use Python data structures instead to accumulate references
 to Python string objects in a Python list so objects are still valid. When 
flushed only then construct
a chain of buckets which use direct reference to raw data of Python string o
bjects.

[ Post a follow-up to this message ]