I am new to python and C-extensions. I am writing a python code where I have created two threads and have defined a callback function py_cb().
In one thread I am appending to a global list after certain time intervals, whereas from the other thread, I am calling a C-extension library api. The C-extension api spawns a thread that calls the python callback function (py_cb) defined in my original file. In the callback function, I am trying to display the global list, but it seems it has a different id. I checked it using id(listName). Is there a way I can use the right global variable in the C-extension python callback function? I was assuming that global values will be shared across all the threads, is that not the case? Also, please suggest better solutions.
Following is the C code:
char *MODULE_NAME = "simple";
---
PyMODINIT_FUNC init_cmodule(void)
{
PyObject *m = Py_InitModule3("_cmodule",module_methods,module_docstring);
if (m == NULL)
return;
}
static PyObject *cmodule_cmodule(PyObject *self, PyObject *args)
{
int value = register_cb();
PyObject *ret = Py_BuildValue("i",value);
return ret;
}
void notifyFooHandle()
{
printf("inside the notify_foo function\n");
pModule = PyImport_ImportModule(MODULE_NAME);
if (!pModule) {
printf ("Failed to load the module\n");
return;
}
PyObject *pFunc;
pFunc = PyObject_GetAttrString(pModule, "py_cb");
if (pFunc && PyCallable_Check(pFunc)) {
PyObject_CallObject(pFunc,NULL);
}
else {
Py_DECREF(pFunc);
PyErr_Print();
printf("Failed to send notification\n");
return;
}
return;
}
void notify_foo(void)
{
int t = 5;
while (t < 10) {
notifyFooHandle();
sleep(5);
t++;
}
return;
}
int register_cb(void)
{
pthread_t notify_t;
int rc = pthread_create(¬ify_t, NULL, (void *) ¬ify_foo,NULL);
---
}
Following is the python callback API in simple.py file:
def py_cb():
print "Received a call back from cmodule"
global myList
print "Global myList is ", id(myList)
print myList
return
Can you show us the code in your C library? If you're initializing a python VM there and then calling py_cb, it's easy to understand why the lists are different: you have two different python VM instances.
EDIT:
I think your problem is you're using two different python instances. First, you have your main program in python. In that instance you have a global "myList" so every function invoked from there will be accessing that particular instance of "myList". Then, you load a C module. When that C module opens you original python module in order to load py_cb, you are using a different python instance, you'll have a second "myList". In short, you have TWO different python instances running, the one you create when you run your main python script and one you create inside your C library to call py_cb.
If you want to share a common python instance, you'll have to create you instance in C, make it globally accessible in your C module, insert there you C functions and then run your main python function. When python calls to a C function, you'll be inside your original address space and not a new one, and when you make a call back to python you'll be using always the same python instance.
Related
I'm learning C++, and in particular C interface to Python. Right now, my focus is on calling or importing python objects from C++ main program.
I've been studying the following link but couldn't understand some concepts. (https://www.codeproject.com/Articles/820116/Embedding-Python-program-in-a-C-Cplusplus-code)
Following is the sections of the tutorial that I can't understand fully.
My questions are:
calling of module:
Is it correct for me to assume "CPyObject pModule = PyImport_Import(pName)" is doing this job?
importing of objects:
i. Is it correct for me to assume "CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger")" is doing this job?
ii.If I want to import a dataframe from python to C++ as a CPyObject, how can I manipulate this object in C++. I'm asking because there is no equivalent object to dataframe in C++.
3) Is there anything else I need to do to make sure my Python module file is visible and callable to C++ ? Such as saving them in the same folder?
Consider the following Python program, stored in pyemb3.py:
def getInteger():
print('Python function getInteger() called')
c = 100*50/30
return c
Now we want to call the function getInteger() from the following C++ code and print the value returned this function. This is the client C++ code:
#include <stdio.h>
#include <Python.h>
#include <pyhelper.hpp>
int main()
{
CPyInstance hInstance;
CPyObject pName = PyUnicode_FromString("pyemb3");
CPyObject pModule = PyImport_Import(pName);
if(pModule)
{
CPyObject pFunc = PyObject_GetAttrString(pModule, "getInteger");
if(pFunc && PyCallable_Check(pFunc))
{
CPyObject pValue = PyObject_CallObject(pFunc, NULL);
printf_s("C: getInteger() = %ld\n", PyLong_AsLong(pValue));
}
else
{
printf("ERROR: function getInteger()\n");
}
}
else
{
printf_s("ERROR: Module not imported\n");
}
return 0;
}
The problem is that 100*50/30 is not an integer, it is a float.
to get an integer use integer division: 100*50//30
If you are not sure about the returned type, you can use the Py_TYPE macro on pValue or just simply check for the type you are looking for with: PyLong_Check or PyLong_CheckExact
1: if PyImport_Import does not return null then the import was successful and the module was already executed by the time the function returned.
2: The PyObject_GetAttrString or the PyObject_GetAttr is the right way to get the imported module's objects.
3: Use these flags to ensure Python is embedded. Use Py_SetPath before Py_Initialize to add your module's path to sys.path.
In the following MCVE two different scripts get executed. Both don't do anything special, the first one has an empty function, the second script does literally nothing. But depending on the script PyEval_EvalCode increfs the passed global/local dictionary or not. The documentation doesn't say anything. But depending on the script I have a dangling dictionary after it got executed. When do I have to decref them afterwards or when not? Or what am I missing here? The following C snippet outputs the refcount of the passed dictionaries.
I tried this on Windows with the standard Python-2.7 interpreter.
#pragma comment(lib, "C:\\Python27\\libs\\python27.lib")
#include "C:\\Python27\\include\\Python.h"
static int execute(const char* script)
{
PyObject* globals = PyDict_New();
PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins());
PyObject* code = Py_CompileString(script, "test", Py_file_input);
if (!code)
{
PyErr_Print();
return 0;
}
PyObject* result = PyEval_EvalCode((PyCodeObject*)code, globals, nullptr);
Py_DECREF(code);
if (!result)
{
PyErr_Print();
return 0;
}
Py_DECREF(result);
printf("Refcount of globals: %zd\n", globals->ob_refcnt);
Py_DECREF(globals); // missing decref spotted by user2357112
return 0;
}
int main()
{
const char* script = nullptr;
Py_Initialize();
// First script contains a function
script =
"def main():\n" \
" pass\n" \
"main()\n";
execute(script);
// second script does nothing
script = "12345";
execute(script);
Py_Finalize();
return 0;
}
The extra reference is the function's own reference to its global variable dict. This reference is not your concern, and you do not need to decref that reference.
You only need to perform decrefs to account for clearing references you own.
After tracing everything down I realized there is a circular reference. The function gets a reference to the globals dictionary, and the dictionary keeps the function alive. Calling the garbage collector right after the script got executed fixes the problem and the circular reference gets resolved.
execute(script);
PyGC_Collect();
P.S. Credit goes also to user2357112 who spotted a missing decref in my MCVE.
As a learning process of Python C API I am trying to call a Python function inside of functor passed to tbb parallel_for.
Operation of calling the function crashes instance of Python process.
I am not doing anything not thread safe. I get the item from a list and I then call a Python function with the item passed as an argument to the function. In the end I set the item back to the list. Any hints what have I done wrong ?
It is most likely that you forgot to grab Global Interpreter Lock (GIL) when you call Python function back from C++. For example, TBB module for Python implements this using swig:
class PyCaller : public swig::SwigPtr_PyObject {
public:
using swig::SwigPtr_PyObject::SwigPtr_PyObject; // gets constructors
void operator()() const {
SWIG_PYTHON_THREAD_BEGIN_BLOCK;
PyObject* r = PyObject_CallFunctionObjArgs((PyObject*)*this, NULL);
if(r) Py_DECREF(r);
SWIG_PYTHON_THREAD_END_BLOCK;
}
};
// Usage:
tbb::task_group tg;
void enqueue( PyObject *c ) {
tg.run( PyCaller(c) );
}
And you can see how SWIG implements it - here.
Other options to consider include using Numba's #cfunc(nopython=True) decorator and Cython's nogil attribute which both make function faster and enable Python's function to run in parallel as they remove GIL from the compiled function.
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
I am currently using pybind11 to embed python into my program.
I call the update() function in which I want to run python code on all objects (hundreds of objects) created from a certain class. The problem is, that I need to hand over a pointer to the object(this) that is calling the python code to python.
So basically I have a class like that (minimal pseudo C++ Code):
class A
{
public:
A(Data* d)
{
dat = d;
createPyObject(passToPython);
}
void update()
{
launchPythonCode(passToPython);
}
Data* getData(){return dat;}
static void createPythonClassDef()
{
pybind11::class_<A>(module, "A")
.def("getData",&A::getData);
}
private:
void createPyObject(PyObject* p)
{
//HowTo?
}
Data* dat;
PyObject* passToPython;
};
What I tried:
Adding a function returning the pointer to the class and creating a Python localdictionary containing a function getA() that would point to that function. So each Object would contain their own localdictionary with the getA() function returning its pointer.
That should give Python the object but it didn't work.
I hope someone can tell me how to correctly embed python for that purpose.