[ https://issues.apache.org/jira/brow...action_12523668 ]

Deron Meranda commented on MODPYTHON-217:
-----------------------------------------

Another real-world case where this comes up is with the Xapian
search engine (http://xapian.org/).  It also is SWIG generated.
See their bug #185,
http://www.xapian.org/cgi-bin/bugzi..._bug.cgi?id=185


> Python 2.3 and simplified GIL state API still causes problems.
> --------------------------------------------------------------
>
>                 Key: MODPYTHON-217
>                 URL: https://issues.apache.org/jira/browse/MODPYTHON-217
>             Project: mod_python
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 3.3.x, 3.2.10
>            Reporter: Graham Dumpleton
>
> There are still problems in mod_python when third party extension modules are used
 which make use of the Python simplified GIL state API. Specifically, if C code make
s calls to PyGILState_Ensure() when it already holds the Python GIL and has an activ
e t
hread state, but the active thread was one which was originally created outs
ide of Python, such as by Apache with mod_python, then the call to PyGILStat
e_Ensure() results in a dead lock. This is occurring because the PyGILState_
Ensure() function finds it
does not know anything about the current thread ID and therefore assumes it 
needs to create a new thread state for it and make that the active thread st
ate. In doing that though it tries to acquire the GIL when the thread alread
y holds it, resulting in th
e deadlock.
> At this stage it is believed this only occurs with Python 2.3 and shouldn't be a p
roblem with later versions of Python. This is the case as later versions of Python w
ere modified so that PyThreadState_New() will register the thread with the underpinn
ing
s of the PyGILState_???() mechanism and because it already knows about it, it will not attem
pt to create a new thread state object and so the deadlock will not occur.
> When C code is hand crafted it is unlikely that anyone would call PyGILSta
te_Ensure() when they already know they hold the GIL, ie., when calling out 
of Python code, but SWIG when used with the -threads option does exactly thi
s. For example:
> SWIGINTERN PyObject *_wrap_ap_get_server_version(PyObject *SWIGUNUSEDPARM(
self), PyObject *args) {  PyObject *resultobj = 0;
>   char *result = 0 ;
>
>   SWIG_PYTHON_THREAD_BEGIN_BLOCK;
>   if (!PyArg_ParseTuple(args,(char *)":ap_get_server_version")) SWIG_fail;
>   {
>     SWIG_PYTHON_THREAD_BEGIN_ALLOW;
>     result = (char *)ap_get_server_version();
>     SWIG_PYTHON_THREAD_END_ALLOW;
>   }
>   resultobj = SWIG_FromCharPtr((const char *)result);
>   SWIG_PYTHON_THREAD_END_BLOCK;
>   return resultobj;
> fail:
>   SWIG_PYTHON_THREAD_END_BLOCK;
>   return NULL;
> }
> where SWIG_PYTHON_THREAD_BEGIN_BLOCK eventually expands to a call to PyGIL
State_Ensure().
> The only solution to this problem would be for mod_python to behave differently wh
en it is acquiring a thread state against the main Python interpreter, ie. 'main_int
erpreter'. Specifically, rather than use PyThreadState_New() etc, it should call PyG
ILS
tate_Ensure() instead. When needing to release the main Python interpreter i
t should call PyGILState_Release(). It should continue to work as before for
 all other interpreters and anyone with GILState code would need to ensure t
hey are using 'main_interpr
eter'.
> Unfortunately at the moment making this change is not completely straight forward 
as when an interpreter is being released it isn't known that it is the main Python i
nterpreter. Thus some knowledge that it is the main interpreter would need to be car
rie
d into the call. The only other option is that the list of all active interpreters is traver
sed and the last one in the list is assumed to be the main interpreter. Then if the interpre
ter being released matches that then act differently.
> Note that at present the acquire/release methods are exported as optional function
s so that other Apache modules use them. It is unlikely that anyone is making use of
 them, so changing the in/out parameters of the functions would possibly be acceptab
le.

> Also note that maintaining support for threaded and non threaded Python and now th
is GIL state variation is going to make the code even more messy. As such it might b
e worthwhile considering dropping support for non threaded Python. These days Python
 de
faults to being threaded anyway, and highly unlikely that someone would want
 to use non thread Python with mod_python anyway. When ever any development 
of mod_python is done, no testing is ever done with a non threaded Python an
yway and so it is possibly
questionable if that support still works.
> FWIW, this issue came up as a result of queries expressed in:
>   http://www.modpython.org/pipermail/...ril/023445.html
> The GIL state issue has attempted to be addressed before in MODPYTHON-77 but this 
particular issue not captured in that.

[ Post a follow-up to this message ]