Using compression on request content and FieldStorage can cause problems.
-------------------------------------------------------------------------

Key: MODPYTHON-215
URL: https://issues.apache.org/jira/browse/MODPYTHON-215
Project: mod_python
Issue Type: Bug
Components: core
Affects Versions: 3.2.10, 3.3
Reporter: Graham Dumpleton


The problems occur when a client uses 'Content-Encoding' of 'gzip', ie., it 
compresses the content of a POST request and Apache is configured to decompr
ess it.

The first issue is that the original content length header will be the lengt
h of data in the body before it is decompressed. If the content type is "app
lication/x-www-form-urlencoded" this will result in not all the decompressed
 data (which would normally
be greater in size) being read in and used as it uses the original value of 
the content length header explicitly when doing the read.

if ctype.startswith("application/x-www-form-urlencoded"):
pairs = parse_qsl(req.read(clen), keep_blank_values)

Note that the change in content size is not a problem with "multipart/*" POS
T requests as it reads data in blocks until req.read() returns an empty stri
ng to indicate end of data and doesn't consult content length at all.

The second issue is if Apache 2.2 is being used and the mod_filter FilterPro
tocol directive is used to correctly indicate that input filters are being u
sed which will change the size of the content, then it will actually zap the
 content length header, rem
oving it from the set of input headers. If this is done it will cause FieldS
torage to wrongly flag the request as not having a content length specified 
in the first place.

try:
clen = int(req.headers_in["content-length"])
except (KeyError, ValueError):
# absent content-length is not acceptable
raise apache.SERVER_RETURN, apache.HTTP_LENGTH_REQUIRED

What should be done to fix this later issue is for FieldStorage to rely on t
he fact that req.read() internally performs a check on first read to ensure 
that the content length header was set. It would just need to be checked tha
t this check is before mod_
filter zaps the content length header on determining that a input filter whi
ch actually change the length.

As to the attempt to use the content length header when it is "application/x
-www-form-urlencoded", it perhaps can do blocked reads as well until it accu
mulates all the content, join it back together and then use it. Note that it
 can't just use req.read()
with no arguments because of MODPYTHON-212.

[ Post a follow-up to this message ]