Apache Mod-Python - mod_python 3.2.9-rc2 available for testing

Author

mod_python 3.2.9-rc2 available for testing

Jim Gallacher

2006-06-23, 1:12 am

OK, this time for real.

The mod_python 3.2.9-rc2 tarball is available for testing. This release
adds support for apache 2.2 as well as some other useful backports from
the development branch. For information on the changes from 3.2.8 take a
look at doc-html/node98.html in the tarball.

Here are the rules:

In order for a file to be officially announced, it has to be tested by
developers on the dev list. Anyone subscribed to this list can (and
should feel obligated to :-) ) test it, and provide feedback *to _this_
list*! (Not the mod_python@modpython.org list, and preferably not me
personally).

The files are (temporarily) available here:

http://people.apache.org/~jgallacher/mod_python/dist/
http://people.apache.org/~jgallache...n-3.2.9-rc1.tgz
http://people.apache.org/~jgallache...2.9-rc1.tgz.md5

Please download it, then do the usual

$ ./configure --with-apxs=/wherever/it/is
$ make
$ (su)
# make install

Then (as non-root user!)

$ cd test
$ Python test.py

And see if any tests fail. If they pass, send a +1 to the list, if they
fail, send the details (the versions of OS, Python and Apache, the test
output, and suggestions, if any).

If this tarball looks good, I'll tag it svn and create a 3.2.9 final
tarball.

Thank you for your assistance,
Jim Gallacher

Nicolas Lehuen

2006-06-23, 7:14 am

+1 Windows XP SP2, ActivePython 2.4.3, Apache 2.0.58

Regards,
Nicolas

2006/6/23, Jim Gallacher <jgallacher@apache.org>:
> OK, this time for real.

>
> The mod_python 3.2.9-rc2 tarball is available for testing. This release
> adds support for apache 2.2 as well as some other useful backports from
> the development branch. For information on the changes from 3.2.8 take a
> look at doc-html/node98.html in the tarball.
>
> Here are the rules:
>
> In order for a file to be officially announced, it has to be tested by
> developers on the dev list. Anyone subscribed to this list can (and
> should feel obligated to :-) ) test it, and provide feedback *to _this_
> list*! (Not the mod_python@modpython.org list, and preferably not me
> personally).
>
> The files are (temporarily) available here:
>
> http://people.apache.org/~jgallacher/mod_python/dist/
> http://people.apache.org/~jgallache...n-3.2.9-rc1.tgz
> http://people.apache.org/~jgallache...2.9-rc1.tgz.md5
>
> Please download it, then do the usual
>
> $ ./configure --with-apxs=/wherever/it/is
> $ make
> $ (su)
> # make install
>
> Then (as non-root user!)
>
> $ cd test
> $ Python test.py
>
> And see if any tests fail. If they pass, send a +1 to the list, if they
> fail, send the details (the versions of OS, Python and Apache, the test
> output, and suggestions, if any).
>
> If this tarball looks good, I'll tag it svn and create a 3.2.9 final
> tarball.
>
> Thank you for your assistance,
> Jim Gallacher
>
>
>
>

Max Bowsher

2006-06-23, 7:14 am

Jim Gallacher wrote:
> The mod_python 3.2.9-rc2 tarball is available for testing.

Something about the mod_python.util changes has either exposed a bug in
Trac, or introduced a bug into mod_python - I'm not sure which yet.

3.2.x r416547 with r393781 reverted works fine for me
3.2.x r416548 seems to be behaving as if any path info in the URL beyond
the object designated as a mod_python handler is being truncated after
the first slash - i.e.:

If the configuration is:

<Location /trac>
SetHandler mod_python
...
</Location>

then an URL like:

http://host/trac/foo/bar/baz

gets treated as:

http://host/trac/foo/

I'll try to narrow down the source of the problem.

Max.

Jeff Hinrichs - DM&T

2006-06-23, 7:14 am

+1 FreeBSD 6.1p2 / Python 2.4.3 / Apache 2.2.2

On 6/23/06, Nicolas Lehuen <nicolas@lehuen.com> wrote:
> +1 Windows XP SP2, ActivePython 2.4.3, Apache 2.0.58
>
> Regards,
> Nicolas
>
> 2006/6/23, Jim Gallacher <jgallacher@apache.org>:
>

Jim Gallacher

2006-06-23, 7:14 am

+1 Linux Debian Sid, apache 2.0.55, Python 2.3.5

David Fraser

2006-06-23, 1:12 pm

I've recently been bitten by the Apache 2.0.47 requirement as mentioned
in the following mails:
http://www.modpython.org/pipermail/...ary/020280.html
http://www.modpython.org/pipermail/...May/021135.html
http://www.modpython.org/pipermail/...May/021133.html

I've attached a patch to update the requirement in the docs - would be
good to have this in for 3.2.9
Have also put this in JIRA at
http://issues.apache.org/jira/browser/MODPYTHON-174

Jim Gallacher wrote:
> OK, this time for real.

>
> The mod_python 3.2.9-rc2 tarball is available for testing. This release
> adds support for apache 2.2 as well as some other useful backports from
> the development branch. For information on the changes from 3.2.8 take a
> look at doc-html/node98.html in the tarball.
>
> Here are the rules:
>
> In order for a file to be officially announced, it has to be tested by
> developers on the dev list. Anyone subscribed to this list can (and
> should feel obligated to :-) ) test it, and provide feedback *to _this_
> list*! (Not the mod_python@modpython.org list, and preferably not me
> personally).
>
> The files are (temporarily) available here:
>
> http://people.apache.org/~jgallacher/mod_python/dist/
> http://people.apache.org/~jgallache...n-3.2.9-rc1.tgz
>
> http://people.apache.org/~jgallache...2.9-rc1.tgz.md5
>
>
> Please download it, then do the usual
>
> $ ./configure --with-apxs=/wherever/it/is
> $ make
> $ (su)
> # make install
>
> Then (as non-root user!)
>
> $ cd test
> $ Python test.py
>
> And see if any tests fail. If they pass, send a +1 to the list, if they
> fail, send the details (the versions of OS, Python and Apache, the test
> output, and suggestions, if any).
>
> If this tarball looks good, I'll tag it svn and create a 3.2.9 final
> tarball.
>
> Thank you for your assistance,
> Jim Gallacher
>
>

Jim Gallacher

2006-06-23, 1:12 pm

Thanks for the patch David. I'll include it for 3.2.9.

Jim

David Fraser wrote:
> I've recently been bitten by the Apache 2.0.47 requirement as mentioned
> in the following mails:
> http://www.modpython.org/pipermail/...ary/020280.html
> http://www.modpython.org/pipermail/...May/021135.html
> http://www.modpython.org/pipermail/...May/021133.html
>
> I've attached a patch to update the requirement in the docs - would be
> good to have this in for 3.2.9
> Have also put this in JIRA at
> http://issues.apache.org/jira/browser/MODPYTHON-174
>
> Jim Gallacher wrote:
>
>
> ------------------------------------------------------------------------
>
> Index: Doc/modpython2.tex
> ========================================
===========================
> --- Doc/modpython2.tex (revision 416697)
> +++ Doc/modpython2.tex (working copy)
> @@ -16,7 +16,7 @@
> \item
> Python 2.2.1 or later. Earlier versions of Python will not work.
> \item
> - Apache 2.0.40 or later (For Apache 1.3.x, use mod_python version 2.7.x).
> + Apache 2.0.47 or later (For Apache 1.3.x, use mod_python version 2.7.x).
> \end{itemize}
>
> In order to compile mod_python you will need to have the include files

Jim Gallacher

2006-06-23, 1:12 pm

Max Bowsher wrote:
> Jim Gallacher wrote:
>
> Something about the mod_python.util changes has either exposed a bug in
> Trac, or introduced a bug into mod_python - I'm not sure which yet.
>
> 3.2.x r416547 with r393781 reverted works fine for me
> 3.2.x r416548 seems to be behaving as if any path info in the URL beyond
> the object designated as a mod_python handler is being truncated after
> the first slash - i.e.:
>
> If the configuration is:
>
> <Location /trac>
> SetHandler mod_python
> ...
> </Location>
>
> then an URL like:
>
> http://host/trac/foo/bar/baz
>
> gets treated as:
>
> http://host/trac/foo/
>
>
> I'll try to narrow down the source of the problem.

I installed trac and can reproduce the behaviour. The latest changes to
Field *should* work, so I'm thinking something is a little wonky in
FieldStorage. No time right now but I'll be able to dig into it later
today - if you haven't found the solution by then Max. ;)

Jim

Max Bowsher

2006-06-23, 1:12 pm

Jim Gallacher wrote:
> Max Bowsher wrote:
>
> I installed trac and can reproduce the behaviour. The latest changes to
> Field *should* work, so I'm thinking something is a little wonky in
> FieldStorage. No time right now but I'll be able to dig into it later
> today - if you haven't found the solution by then Max. ;)

OK, I've found the solution.

The root of the problem is that Trac wants to be able to add extra
fields to a FieldStorage itself, and has been jumping through all sorts
of crazy hoops in the internals of FieldStorage to make this happen.
The recent changes have moved all the hoops, and Trac is now failing
utterly.

First problem, is that the new FieldStorage memoizes self.dictionary the
first time it creates it. However, Trac expects to be able to append to
FieldStorage.list to add new fields. Since the list member is documented
in the public API docs, this isn't entirely unreasonable. In any case,
list ought to have a leading underscore if it is supposed to be private.

OK, so suppose you comment out the line "self.dictionary = result", so
that the dictionary gets rebuilt every time it is needed. Does it work? No!

The fun doesn't stop there. The second problem is related to the way
that pre-3.2.9 mod_python likes to stuff everything into unnecessary
StringIO objects. Trac tries to replicate this, and this causes problems
since 3.2.9 no longer expects StringIO objects everywhere. Trac is
actually relying on pre-3.2.9 behaviour of transforming
string-in-StringIO Fields into StringFields during
FieldStorage.__getitem__, but this no longer happens in 3.2.9.

Here is the patch that seems to make things work again:
--- util-bad.py 2006-06-23 08:12:00.000000000 -0500
+++ util.py 2006-06-23 09:51:32.000000000 -0500
@@ -323,10 +323,17 @@
def __getitem__(self, key):
"""Dictionary style indexing."""
found = self.dictionary[key]
- if len(found) == 1:
- return found[0]
+ newfound = []
+ for item in found:
+ if isinstance(item.file, FileType) or \
+ isinstance(getattr(item.file, 'file', None), FileType):
+ newfound.append(item)
+ else:
+ newfound.append(StringField(item.value))
+ if len(newfound) == 1:
+ return newfound[0]
else:
- return found
+ return newfound

def get(self, key, default):
try:
@@ -374,7 +381,6 @@
if list is None:
raise TypeError, "not indexable"
result = {}
- self.dictionary = result
for item in list:
if item.name in result:
result[item.name].append(item)

Yes, ugh.

So, I guess the two questions to answer now are:

* How much non-compatibility is acceptable in a patch release?

* How are applications supposed to perform write operations on a
FieldStorage, in 3.3 and the future?

Max.

Mike Looijmans

2006-06-27, 1:12 pm

Having written most of the issue "93" code, here's my opinion:

> * How much non-compatibility is acceptable in a patch release?

None.
Though it hurts my personal feelings that my patch did manage to break
something (who imagined anyone trying to hack data into the FS object?),
we cannot break other people's software with a mere patch.

I don't have a clue what the "trac" software is, but I imagine that even
for this particular piece of software, I may be able to come up with
some form of emulation (e.g. using __setattr__ hooks) that still keeps
it working with the patch. The question is of course, where does it stop?

Note that the "old" code still contains a bug that the patch solves. The
problem is that the old code does not work correctly if the objects
created in a make_file callback are not real files created with file().

> * How are applications supposed to perform write operations on a
> FieldStorage, in 3.3 and the future?

Since we claim that FieldStorage behaves like a dictionary, the obvious
syntax would be:

form['mykey'] = 'value'

This would require a __setitem__ method which should look something like
this (with some extra code to handle lists):

def __setitem__(self, key, value):
if self.dictionary is None:
# Create self.dictionary as in __getitem__
self.dictionary[key] = StringField(value)

Jim Gallacher

2006-06-27, 1:12 pm

Mike Looijmans wrote:
> Having written most of the issue "93" code, here's my opinion:
>
>
> None.
> Though it hurts my personal feelings that my patch did manage to break
> something (who imagined anyone trying to hack data into the FS object?),
> we cannot break other people's software with a mere patch.

I think this surprised many of us, as no one on the list seems to have
thought of that use case. Trac subclasses FieldStorage to get behaviour
more in line with cgi.py. We don't have any prohibitions on subclassing,
so although we didn't foresee this usage I think it is still legitimate.

> I don't have a clue what the "trac" software is, but I imagine that even
> for this particular piece of software, I may be able to come up with
> some form of emulation (e.g. using __setattr__ hooks) that still keeps
> it working with the patch. The question is of course, where does it stop?

Trac is bugtracker / wiki that seems to be getting some traction. ;)

> Note that the "old" code still contains a bug that the patch solves. The
> problem is that the old code does not work correctly if the objects
> created in a make_file callback are not real files created with file().

So do you think we can release 3.2.9 with the old 3.2.8 code, or should
this block the release until we have a correct fix? I'm hoping we can do
a 3.3 release in October or November, FYI.

>
> Since we claim that FieldStorage behaves like a dictionary, the obvious
> syntax would be:
>
> form['mykey'] = 'value'
>
> This would require a __setitem__ method which should look something like
> this (with some extra code to handle lists):
>
> def __setitem__(self, key, value):
> if self.dictionary is None:
> # Create self.dictionary as in __getitem__
> self.dictionary[key] = StringField(value)

Trac also appends to the FieldStorage list attribute, which complicates
things a little further.

The '93' code also adds the add_field() method. Although this is not
documented, there is nothing to indicate that it is a private method
either. Calling add_field on a FieldStorage instance will not likely
give the results a user expects, so we need to give some attention to
that as well.

Jim

Jorey Bump

2006-06-27, 1:12 pm

Jim Gallacher wrote:
> Mike Looijmans wrote:
>
> I think this surprised many of us, as no one on the list seems to have
> thought of that use case. Trac subclasses FieldStorage to get behaviour
> more in line with cgi.py. We don't have any prohibitions on subclassing,
> so although we didn't foresee this usage I think it is still legitimate.
>
>
> Trac is bugtracker / wiki that seems to be getting some traction. ;)

It's also a frontend to subversion (and other version control systems,
using plugins), which I think is responsible for its explosive
popularity. It may well become the premier application using (but not
exclusively dependent on) mod_python. I haven't used it (I'm waiting for
a stable darcs plugin), but you can get an idea of its features here:

http://projects.edgewall.com/trac/

This definitely a project we don't want to break. If I get a chance I'll
try to set up a test bed to make sure future release candidates are
compatible.

Do we have any trac developers on this list?

Graham Dumpleton

2006-06-27, 7:12 pm

Jim Gallacher wrote ..
> So do you think we can release 3.2.9 with the old 3.2.8 code, or should
> this block the release until we have a correct fix? I'm hoping we can do
> a 3.3 release in October or November, FYI.

I don't think it is worth trying to work on a fix that makes new
FieldStorage code work with Trac. Just go with 3.2.8 FieldStorage code
in 3.2.9. In 3.3 then use new code with understanding that it will break
Trac.

I see this as acceptable as version of Trac in subversion trunk does
away with all the FieldStorage fiddles and instead they base it on a
WSGI wrapper which I presume therefore avoids use of FieldStorage.

We just need to ensure that this newer version of Trac is released
before we release mod_python 3.3 and that we notify them that older
versions of Trac will not work with mod_python 3.3 and people will need
to use their most latest version of Trac instead.

Graham

Mike Looijmans

2006-06-28, 7:12 am

>>So do you think we can release 3.2.9 with the old 3.2.8 code, or should
>
> I don't think it is worth trying to work on a fix that makes new
> FieldStorage code work with Trac. Just go with 3.2.8 FieldStorage code
> in 3.2.9. In 3.3 then use new code with understanding that it will break
> Trac.
>
> I see this as acceptable as version of Trac in subversion trunk does
> away with all the FieldStorage fiddles and instead they base it on a
> WSGI wrapper which I presume therefore avoids use of FieldStorage.
>
> We just need to ensure that this newer version of Trac is released
> before we release mod_python 3.3 and that we notify them that older
> versions of Trac will not work with mod_python 3.3 and people will need
> to use their most latest version of Trac instead.

You have my +1 on that. Let's keep the 3.2.8 implementation for now, and do the rework in 3.3 in
cooperation with the Trac people.

Mike.