Note that I'm constrained to use Python 2.6. I have a Python 2.6 application that uses a C++ multi-threaded API library built with boost-python. My use-case was simply to execute a Python function callback from a C++ boost thread but despite the many different attempts and researching all the available online resources I haven't found any way that works. All the proposed solutions revolve around a different combination of the functions: Py_Initialize*, PyEval_InitThreads, PyGILState_Ensure, PyGILState_Release but after trying all possible combinations nothing works in practice e.g.
Embedding Python in multi-threaded C++ applications with code here
PyEval_InitThreads in Python 3: How/when to call it? (the saga continues ad nauseum)
Therefore, this question: how can I start and run a Python thread from C++? I basically want to: create it, run it with a Python target function object and forget about it.
Is that possible?
Based on the text below from your question:
Run a Python thread from C++? I basically want to: create it, run it with a Python target function object and forget about it.
You may find useful to simply spawn a process using sytem:
system("python myscript.py")
And if you need to include arguments:
string args = "arg1 arg2 arg3 ... argn"
system("python myscript.py " + args)
You should call PyEval_InitThreads from your init routine (and leave the GIL held). Then, you can spawn a pthread (using boost), and in that thread, call PyGILState_Ensure, then call your python callback (PyObject_CallFunction?), release any returned value or deal with any error (PyErr_Print?), release the GIL with PyGILState_Release, and let the thread die.
void *my_thread(void *arg)
{
PyGILState_STATE gstate;
PyObject *result;
gstate = PyGILState_Ensure();
if ((result = PyObject_CallFunction(func, NULL))) {
Py_DECREF(result);
}
else {
PyErr_Print();
}
PyGILState_Release(gstate);
return NULL;
}
The answer to the OP How to call Python from a boost thread? also answers this OP or DP (Derived:)). It clearly demonstrates how to start a thread from C++ that callbacks to Python. I have tested it and works perfectly though needs adaptation for pre-C++11. It uses Boost Python, is indeed an all inclusive 5* answer and the example source code is here.
Related
I have a C++ project which was written by others, which calls python code at the end of the execution.
In the C++ initializer, it defines a pHandle:
pHandle_ = PyObject_CallObject(pLoad_, NULL);
PyGILState_Release(gstate);
Then it calls python code this way:
PyObject *pyValue = PyObject_CallObject(pProcess_, pArgs);
pProcess_ and pArgs are created earlier. The python code's file name is 'runLogic.py' and the function executed in runLogic.py is 'process()'.
Is there a way to break into the python's process() function while I debug in c++ through GDB? In C++ code, I can step through until the line above, copied again below:
PyObject *pyValue = PyObject_CallObject(pProcess_, pArgs);
Then I don't know how to jump into the Python's function.
Is there a way to do that? I want to trace the full logic of the code, in addition to C++'s code.
Then I don't know how to jump into the Python's function.
Presumably you want to now step through Python code.
GDB doesn't know how to do that, but pdb can.
Assuming you can modify runLogic.py, add import pdb to the top of the file, and pdb.set_trace() in the process() function.
This will let you step through Python code. If you have breakpoints in C++, and Python code calls back into C++, these breakpoints will still be active, you should be able to debug both C++ and Python.
I have an Python 3 interpreter embedded into an C++ MPI application. This application loads a script and passes it to the interpreter.
When I execute the program on 1 process without the MPI launcher (simply calling ./myprogram), the script is executed properly and its "print" statements output to the terminal. When the script has an error, I print it on the C++ side using PyErr_Print().
However when I lauch the program through mpirun (even on a single process), I don't get any output from the "print" in the python code. I also don't get anything from PyErr_Print() when my script has errors.
I guess there is something in the way Python deals with standard output that do not match the way MPI (actuall Mpich here) deals with redirecting the processes' output to the launcher and finally to the terminal.
Any idea on how to solve this?
[edit, following the advice from this issue]
You need to flush_io() after each call to PyErr_Print, where flush_io could be this function:
void flush_io(void)
{
PyObject *type, *value, *traceback;
PyErr_Fetch(&type, &value, &traceback); // in Python/pythonrun.c, they save the traceback, let's do the the same
for (auto& s: {"stdout", "stderr"}) {
PyObject *f = PySys_GetObject(s);
if (f) PyObject_CallMethod(f, "flush", NULL);
else PyErr_Clear();
}
PyErr_Restore(type, value, traceback);
}
[below my old analysis, it still has some interesting info]
I ended up with the same issue (PyErr_Print not working from an mpirun). Tracing back (some gdb of python3 involved) and comparing the working thing (./myprogram) and non-working thing (mpirun -np 1 ./myprogram), I ended up in _io_TextIOWrapper_write_impl at ./Modules/_io/textio.c:1277 (python-3.6.0 by the way).
The only difference between the 2 runs is that self->line_buffering is 1 vs. 0 (at this point self represents sys.stderr).
Then, in pylifecycle.c:1128, we can see who decided this value:
if (isatty || Py_UnbufferedStdioFlag)
line_buffering = Py_True;
So it seems that MPI does something to stderr before launching the program, which makes it not a tty. I haven't investigated if there's an option in mpirun to keep the tty flag on stderr ... if someone knows, it'd be interesting (though on second thought mpi probably has good reasons to put his file descriptors in place of stdout&stderr, for its --output-filename for example).
With this info, I can come out with 3 solutions (the first 2 are quick-fixes, the 3rd is better):
1/ in the C code that starts the python interpreter, set the buffering flag before creating sys.stderr. The code becomes :
Py_UnbufferedStdioFlag = 1; // force line_buffering for _all_ I/O
Py_Initialize();
This brings Python's traceback back to screen in all situations; but will probably give catastrophic I/O ... so only an acceptable solution in debug mode.
2/ in the python (embedded) script, at the very beginning add this :
import sys
#sys.stderr.line_buffering = True # would be nice, but readonly attribute !
sys.stderr = open("error.log", 'w', buffering=1 )
The script then dumps the traceback to this file error.log.
I also tried adding a call to fflush(stderr) or fflush(NULL) right after the PyErr_Print() ... but this didn't work (because sys.stderr has its own internal buffering). That'd be a nice solution though.
3/ After a little more digging, I found the perfect function in
Python/pythonrun.c:57:static void flush_io(void);
It is in fact called after each PyErr_Print in this file.
Unfortunately it's static (only exists in that file, no reference to it in Python.h, at least in 3.6.0). I copied the function from this file to myprogram and it turns out to do exactly the job.
I have a shared library (DLL) in C++ with some C-style API which I use from Python. There is a function which takes a C callback as an argument. As far as I understood, Python extension module (a separate DLL) is required to do that. This module shall pass a native "proxy" callback to API, then call Python object from that callback. I try do use it as follows (it fails):
from ctypes import CDLL
from test_extension import set_python_callback
def callback():
pass
if __name__ == '__main__':
library = CDLL('test_dll.shared-library')
set_python_callback(library, callback)
library.trigger_callback()
I'm using Windows 8 x64 environment with Python 2.7.6 (32 bit) and MinGW 32-bit compiler (mingw32). Python extension is built by distutils. API library can be changed if necessary, however, Python-specific code has to be kept in a separate library.
Here is my code, reduced. All error-checking was removed, however, when performed it showed no errors either from Python API or from Windows API.
API library:
/* typedef void (*callback_t)(); */
static callback_t s_callback = nullptr;
void DLL_PUBLIC set_callback(callback_t callback) {
s_callback = callback;
}
void DLL_PUBLIC trigger_callback() {
s_callback(); // <--------------------------(1)
}
Python extension module function set_python_callback(library, callback) takes a ctypes.CDLL object and Python callable. It then extracts native DLL handle from the former:
PyObject* handle_object = PyObject_GetAttrString(library, "_handle");
const HMODULE handle = static_cast<const HMODULE>(
PyLong_AsVoidPtr(handle_object));
const native_set_callback_t native_api_routine =
reinterpret_cast<const native_set_callback_t>(
GetProcAddress(handle, "set_callback"));
native_api_routine(common_callback);
C function common_callback() is implemented as follows:
extern "C" void DLL_PUBLIC common_callback() {
// <--------------------------------------- (2)
PyObject *arguments = Py_BuildValue("()");
PyObject *callback_result = PyEval_CallObject(s_callback, arguments);
Py_DECREF(arguments);
Py_XDECREF(callback_result);
}
The error message says:
Invoking callback from DLL... Traceback (most recent call last):
File "script.py", line 14, in <module>
library.trigger_callback()
WindowsError: exception: access violation reading 0x00000028
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute. The strange thing is, changing the way common_callback() call Python code (e. g. passing Py_None instead of empty tuple) changes the address in the error message.
Maybe this is somehow related to the calling convention, but I have no idea where it's wrong exactly or how to fix it.
Solved the issue by registering the thread for GIL as described in documentation.
Please note that original API DLL, Python extension DLL, and the script were executed in single-threaded mode and no additional thread had been created, either Python-managed, or not. However, there had been a synchronization issue for sure, because using Py_AddPendingCall() also worked. (Using is is discouraged; I found it analyzing signals module sources, it lead to the solution above.)
This statement of mine was incorrect:
Using debug print, I traced the error down to (1). Any code at (2) point doesn't execute.
I was mislead by the fact that some of Python API functions still worked (e. g. Py_BuildValue or Py_Initialize), but most didn't (e. g. PySys_WriteStdout, PyObject_Call, etc).
I'am writing a multi-threaded program in C. Before creating the threads, a global python environment is initialized by calling Py_Initialize(). Then, in every created thread, the global python environment is shared, and each thread calls a python method with the parameters converted in C. Everything works well until here.
When I use time.sleep() in loaded python modules, the C program raises a Segmentation Fault. Furthermore, the loaded python module is supposed to load another C lib to continue the work. I've written the following stupid counter lib to test it:
# python part, call the counter function
lib = ctypes.cdll.LoadLibrary(libpycount.so)
for i in xrange(10):
lib.count()
// C part, dummy countings
#include <stdio.h>
int counter = 1;
void
count() {
printf("counter:%d \n", counter);
counter++;
}
I guess that it might be because I didn't manage the complex thread creation in the right way. And I've found Non-Python created threads in the python doc.
Any ideas or suggestions?
My problem has been solved. You may have your problems more particular, so I'm trying to write my solution in a more generic way here. Hope it helps.
- In main C thread
initialize the Python environment at the very begining:
/*define a global variable to store the main python thread state*/
PyThreadState * mainThreadState = NULL;
if(!Py_IsInitialized())
Py_Initialize();
mainThreadState = = PyThreadState_Get();
Then start the C threads:
pthread_create(pthread_id, NULL, thread_entrance, NULL);
- In every thread, or we can say in the body of thread_entrance function
prepare the environment:
/*get the lock and create new python thread state*/
PyEval_AcquireLock();
PyInterpreterState * mainInterpreterState = mainThreadState->interp;
PyThreadState * myThreadState = PyThreadState_New(mainInterpreterState);
PyEval_ReleaseLock(); /*don't forget to release the lock*/
/*
* some C manipulations here
*/
put embeded Python code here:
/*get the lock and put your C-Python code here*/
PyEval_AcquireLock();
PyThreadState_Swap(myThreadState); /*swap your python thread state*/
PyEval_CallObject(py_function, py_arguments);
/*or just something like PyRun_SimpleString("print \"hello world\""); for test*/
PyThreadState_Swap(NULL); /*clean the thread state before leaving*/
PyEval_ReleaseLock();
- back to main C thread
when every thread finishes their works, finalize the python environment
pthread_join(pthread_id, NULL);
PyEval_RestoreThread(mainThreadState);
Py_Finalize();
Question is whether the Python interpreter is thread safe -- This is what the documentation says about running multiple interpreters in the same process space;
Bugs and caveats: Because sub-interpreters (and the main interpreter)
are part of the same process, the insulation between them isn't
perfect -- for example, using low-level file operations like
os.close() they can (accidentally or maliciously) affect each other's
open files. Because of the way extensions are shared between
(sub-)interpreters, some extensions may not work properly; this is
especially likely when the extension makes use of (static) global
variables, or when the extension manipulates its module's dictionary
after its initialization. It is possible to insert objects created in
one sub-interpreter into a namespace of another sub-interpreter; this
should be done with great care to avoid sharing user-defined
functions, methods, instances or classes between sub-interpreters,
since import operations executed by such objects may affect the wrong
(sub-)interpreter's dictionary of loaded modules. (XXX This is a
hard-to-fix bug that will be addressed in a future release.)
...and I don't think that Python threads are the same thing as native threads such as found in C/C++
I have a python program that calls into a c++ library, which wishes to release all the python locks so that other python threads can run.
Using PyEval_SaveThread and PyEval_ReleaseThread I get errors that the thread state is NULL:
Fatal Python error: PyEval_SaveThread: NULL tstate
However, the lower-level function seem to accept the NULL state happily:
PyThreadState *s;
s = PyThreadState_Swap(NULL);
// (now s = 0)
PyEval_ReleaseLock();
// ....
PyEval_AcquireLock();
PyThreadState_Swap(s);
// everything seems to be just fine :)
Answer: no, it is never meant to be NULL (if it is, it's a fatal error). Turned out this was because I was linking against two different versions of python, one via boost_python, and the other directly.
Top Tip:
use ldd or otool -L to check your library dependencies when funny stuff happens ;)