[ http://issues.apache.org/jira/brows...HON-77?page=all ]

Graham Dumpleton updated MODPYTHON-77:
--------------------------------------

Attachment: grahamd_20051105.tar.gz

Here is my first go at an alternate patch for this problem. Patch was made a
gainst SVN head, believed to be 3.2.4b.

All the change in effect does is save the first interpreter as main_interpre
ter, but most importantly it does this in Apache child process and not in th
e parent process before the fork occurs as was the case before. This avoids 
problems on Mac OS X where
Apache would crash on "restart" and on Linux where Apache would crash after 
the request had been handled.

Note that this change doesn't use any of the PEP GIL specific calls nor does
 it do anything specific to make anything work on Python 2.3.5. Except for m
oving one thread state swap call from the parent process context to the chil
d process context, all thre
ad management code is the same.

The changes work fine on:

Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5

Test example was gilstate.tar.gz attached to MODPYTHON-77.

Also passed on mod_python/test suite on Mac OS X. There were failures of tes
t suite on Linux, but those failures occurred before patches were applied as
 well.

The changes need to be tested on Win32 Apache as well as system where no thr
ead support compiled into Python.

For those of you following this issue, if you can test this change, indicate
 if it works or fails and if it fails indicate specifically how it is failin
g. From any failures can then start to understand the other changes Boyan ha
s made and what is required
and what isn't.


> The multiple interpreter concept of mod_python is broken for Python extens
ion modules since Python 2.3
> --------------------------------------------------------------------------
----------------------------
>
>          Key: MODPYTHON-77
>          URL: http://issues.apache.org/jira/browse/MODPYTHON-77
>      Project: mod_python
>         Type: Bug
>   Components: core
>     Versions: 3.1.4
>  Environment: Python >= 2.3
>     Reporter: Boyan Boyadjiev
>  Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz,
 grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff
, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extens
ion modules since Python 2.3 because of the PEP 311 (Simplified Global Inter
preter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
>     This proposal identifies a solution for extension authors with
>     complex multi-threaded requirements, but that only require a
>     single "PyInterpreterState".  There is no attempt to cater for
>     extensions that require multiple interpreter states.  At the time
>     of writing, no extension has been identified that requires
>     multiple PyInterpreterStates, and indeed it is not clear if that
>     facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with
 Python >= 2.3, because they are supposed to work only with the first interpreter st
ate initialized for the current process (a problem we experienced). The first interp
ret
er state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter st
ate into the "interpreters" dictionary in the function python_init (MAIN_INT
ERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
>                        apr_pool_t *plog, server_rec *s)
> {
>     ...
>     /* initialize global Python interpreter if necessary */
>     if (! Py_IsInitialized())
>     {
>         /* initialze the interpreter */
>         Py_Initialize();
> #ifdef WITH_THREAD
>         /* create and acquire the interpreter lock */
>         PyEval_InitThreads();
> #endif
>         /* create the obCallBack dictionary */
>         interpreters = PyDict_New();
>         if (! interpreters) {
>             ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
>                          "python_init: PyDict_New() failed! No more memory
?");
>             exit(1);
>         }
>         {
>             /*
>             Workaround PEP 311 - Simplified Global Interpreter Lock Acquis
ition for Extensions
>             BEGIN
>             */
>             PyObject *p = 0;
>             interpreterdata * idata = (interpreterdata *)malloc(sizeof(int
erpreterdata));
>             PyThreadState* currentThreadState = PyThreadState_Get();
>             PyInterpreterState *istate = currentThreadState->interp;
>             idata->istate = istate;
>             /* obcallback will be created on first use */
>             idata->obcallback = NULL;
>             p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout =
 1*/
>             PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->
refcout = 2*/
>             Py_DECREF(p); /*p->refcout = 1*/
>             /*
>             END
>             Workaround PEP 311 - Simplified Global Interpreter Lock Acquis
ition for Extensions
>             */
>         }
>         /* Release the thread state because we will never use
>          * the main interpreter, only sub interpreters created later. */
>         PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
>         /* release the lock; now other threads can run */
>         PyEval_ReleaseLock();
> #endif
>     }
>     return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interprete
r, which will remove leaky reference to the PyCObject with the interpreter data. Thi
s was not a real problem, but now I see fewer leaks in BoundsChecker :-).

[ Post a follow-up to this message ]