| Graham Dumpleton (JIRA) 2007-08-07, 7:13 am |
| Incremental memory use when performing streaming over long period.
------------------------------------------------------------------
Key: MODPYTHON-241
URL: https://issues.apache.org/jira/browse/MODPYTHON-241
Project: mod_python
Issue Type: Bug
Components: core
Affects Versions: 3.3.1, 3.2.10
Reporter: Graham Dumpleton
Priority: Minor
As first raised in:
http://www.modpython.org/pipermail/...uly/023964.html
when streaming data from mod_python handler over a long period of time, memory use will keep increasing.
Note that this isn't actually a memory leak though, as the memory gets reclaimed at the end of the request.
The underlying problem arises from mod_python using the ap_rwrite() function to write output data. When this function is used it will create a bucket chain for each block of data written, with the bucket chain coming from the request memory pool. There is
no attempt to reuse the bucket chain for successive writes, so memory will increase on every call.
The normal solution to this problem is to retain the bucket chain and use it from one call to the next. In mod_python though this isn't simple to achieve because calls to response data may or may not be flushed at the discretion of the user code. The reta
ining of the bucket chain and using it from one call to the next generally can only be done when data is always being flushed. A more complicated system would be needed which can cope with a need to buffer data at times and yet flush at other times.
One approach may be to not actually use Apache itself to buffer data between flushes, but to use Python data structures instead to accumulate references to Python string objects in a Python list so objects are still valid. When flushed only then construct
a chain of buckets which use direct reference to raw data of Python string objects.
|