| Boyan Boyadjiev (JIRA) 2005-09-03, 7:46 am |
| [ http://issues.apache.org/jira/brows...HON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: gil_test.c
I hope that, the following can help clarifying where the problem comes from.
The root of the evil was the introduction of the new Python APIs PyGILState_Ensure and PyGILState_Release. They are designed to make the process of embedding Python and the extension development much easier, but they also break the idea of having multiple
interpreters. If you look at the implementation of the PyGILState_Ensure you will see that the Python code executed in a PyGILState_Ensure/PyGILState_Release scope is using the autoInterpreterState, which is the one initialized by Py_Initialize. So you
can easily mix multiple interpreters in the same call stack... Another fact is that more and more essential extensions are using those new Python APIs. That's why I thought, that the most acceptable way of 'un-breaking' the concept of having multiple inte
rpreters and using complex Python extensions is:
- To specify, that the use of Python extensions can work only for interpreters[MAIN_INTERPRETER]
- The interpreters[MAIN_INTERPRETER] must be the same as the pystate.c->autoInterpreterState, so the one created by Py_Initiualize.
I've written a very simplified pace of code doing both embedding and extending Python in order to exemplify what I'm talking about (see the attached gil_test.c).
Case 1: Python - it works:
Python 2.3.5 (#62, Sep 2 2005, 19:32:54) [MSC v.1200 32 bit (Intel)] on win32
sys.testAttr = testAttr[vbcol=seagreen]
attribute_found = 1[vbcol=seagreen]
Case 2: mod_python (not patched) - does not work:
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
Case 3: mod_python (patched) - works
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
[vbcol=seagreen]
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpret
er state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
|