Python 2.3 and simplified GIL state API still causes problems.
--------------------------------------------------------------

Key: MODPYTHON-217
URL: https://issues.apache.org/jira/browse/MODPYTHON-217
Project: mod_python
Issue Type: Bug
Components: core
Affects Versions: 3.2.10, 3.3
Reporter: Graham Dumpleton


There are still problems in mod_python when third party extension modules ar
e used which make use of the Python simplified GIL state API. Specifically, 
if C code makes calls to PyGILState_Ensure() when it already holds the Pytho
n GIL and has an active thr
ead state, but the active thread was one which was originally created outsid
e of Python, such as by Apache with mod_python, then the call to PyGILState_
Ensure() results in a dead lock. This is occurring because the PyGILState_En
sure() function finds it do
es not know anything about the current thread ID and therefore assumes it ne
eds to create a new thread state for it and make that the active thread stat
e. In doing that though it tries to acquire the GIL when the thread already 
holds it, resulting in the
deadlock.

At this stage it is believed this only occurs with Python 2.3 and shouldn't 
be a problem with later versions of Python. This is the case as later versio
ns of Python were modified so that PyThreadState_New() will register the thr
ead with the underpinnings
of the PyGILState_???() mechanism and because it already knows about it, it 
will not attempt to create a new thread state object and so the deadlock wil
l not occur.

When C code is hand crafted it is unlikely that anyone would call PyGILState
_Ensure() when they already know they hold the GIL, ie., when calling out of
 Python code, but SWIG when used with the -threads option does exactly this.
 For example:

SWIGINTERN PyObject *_wrap_ap_get_server_version(PyObject *SWIGUNUSEDPARM(se
lf), PyObject *args) {  PyObject *resultobj = 0;
char *result = 0 ;

SWIG_PYTHON_THREAD_BEGIN_BLOCK;
if (!PyArg_ParseTuple(args,(char *)":ap_get_server_version")) SWIG_fail;
{
SWIG_PYTHON_THREAD_BEGIN_ALLOW;
result = (char *)ap_get_server_version();
SWIG_PYTHON_THREAD_END_ALLOW;
}
resultobj = SWIG_FromCharPtr((const char *)result);
SWIG_PYTHON_THREAD_END_BLOCK;
return resultobj;
fail:
SWIG_PYTHON_THREAD_END_BLOCK;
return NULL;
}

where SWIG_PYTHON_THREAD_BEGIN_BLOCK eventually expands to a call to PyGILSt
ate_Ensure().

The only solution to this problem would be for mod_python to behave differen
tly when it is acquiring a thread state against the main Python interpreter,
 ie. 'main_interpreter'. Specifically, rather than use PyThreadState_New() e
tc, it should call PyGILSta
te_Ensure() instead. When needing to release the main Python interpreter it 
should call PyGILState_Release(). It should continue to work as before for a
ll other interpreters and anyone with GILState code would need to ensure the
y are using 'main_interpret
er'.

Unfortunately at the moment making this change is not completely straight fo
rward as when an interpreter is being released it isn't known that it is the
 main Python interpreter. Thus some knowledge that it is the main interprete
r would need to be carried
into the call. The only other option is that the list of all active interpre
ters is traversed and the last one in the list is assumed to be the main int
erpreter. Then if the interpreter being released matches that then act diffe
rently.

Note that at present the acquire/release methods are exported as optional fu
nctions so that other Apache modules use them. It is unlikely that anyone is
 making use of them, so changing the in/out parameters of the functions woul
d possibly be acceptable.

Also note that maintaining support for threaded and non threaded Python and 
now this GIL state variation is going to make the code even more messy. As s
uch it might be worthwhile considering dropping support for non threaded Pyt
hon. These days Python defa
ults to being threaded anyway, and highly unlikely that someone would want t
o use non thread Python with mod_python anyway. When ever any development of
 mod_python is done, no testing is ever done with a non threaded Python anyw
ay and so it is possibly qu
estionable if that support still works.

FWIW, this issue came up as a result of queries expressed in:

http://www.modpython.org/pipermail/...ril/023445.html

The GIL state issue has attempted to be addressed before in MODPYTHON-77 but
 this particular issue not captured in that.

[ Post a follow-up to this message ]