I am currently using pybind11 to embed python into my program.
I call the update() function in which I want to run python code on all objects (hundreds of objects) created from a certain class. The problem is, that I need to hand over a pointer to the object(this) that is calling the python code to python.
So basically I have a class like that (minimal pseudo C++ Code):
class A
{
public:
A(Data* d)
{
dat = d;
createPyObject(passToPython);
}
void update()
{
launchPythonCode(passToPython);
}
Data* getData(){return dat;}
static void createPythonClassDef()
{
pybind11::class_<A>(module, "A")
.def("getData",&A::getData);
}
private:
void createPyObject(PyObject* p)
{
//HowTo?
}
Data* dat;
PyObject* passToPython;
};
What I tried:
Adding a function returning the pointer to the class and creating a Python localdictionary containing a function getA() that would point to that function. So each Object would contain their own localdictionary with the getA() function returning its pointer.
That should give Python the object but it didn't work.
I hope someone can tell me how to correctly embed python for that purpose.
Related
I have a legacy code in C++ (which would be a huge pain to edit) and I need to use it in Python 2 for speed reasons.
I have two classes. One is responsible for loading huge amount of data from memory, in a form of std::string and converting it to internal representation MiddleClass. Second one is converting it from internal representation MiddleClass back to std::string.
class Load {
Load(const std::string & data) { ... };
MiddleClass load() { ... };
};
class Save {
Save(std::string & data) { .... };
void save(const MiddleClass & middleclass) { ... };
};
My goal is, to use this setup in Python 2 like this:
import datahandler # my lib
import requests
request = request.get("url-to-data")
loader = datahandler.Load(request.content) # my C++ class Load
internal_representation = loader.load()
.
.
.
result_variable = str() # or None or something not important
saver = datahandler.Save(result_variable) # my C++ class Save
saver.save(internal_representation)
How can I achieve this?
I've run into trouble, right from the start.
Simple variant:
BOOST_PYTHON_MODULE(datahandler)
{
class_<MiddleClass>("MiddleClass");\
// some .defs - not important
class <Load>("Load", init<const std::string &>())
.def("load". &Load::load);
class <Save>("Save", init<std::string &>())
.def("save". &Save::save);
}
Will compile, no worries, but data which are loaded are somehow mangled, which leads me to thinking, that I am doing it terribly wrongly.
Also I found this bit offtopic SO question, which told me, that I can't have std::string &, because Python strings are immutable.
So conclusion: I have no idea what to do now :( Can anyone here help me? Thanks.
Take as reference this working example.
Define your C++ classes. For instance:
class MiddleClass {
public:
explicit MiddleClass(const std::string& data) : parent_data_(data) {}
void print() {
std::cout << parent_data_ << std::endl;
}
private:
std::string parent_data_;
};
class Loader {
public:
explicit Loader(const std::string& data) :
data_(data){
};
MiddleClass load() {
return MiddleClass(data_);
};
private:
std::string data_;
};
Create the boost bindings
boost::python::class_<MiddleClass>("MiddleClass",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("print_data", &MiddleClass::print);
boost::python::class_<Loader>("Loader",
boost::python::init<const std::string&>(boost::python::arg("data"), ""))
.def("load", &Loader::load);
Install your library in the right python site-package.
Enjoy it in python:
from my_cool_package import MiddleClass, Loader
example_string = "whatever"
loader = Loader(data=example_string)
# Get the middle class
middle_class = loader.load()
# Print the data in the middle class
middle_class.print_data()
The expected output:
whatever
So, I have found a solution. Prove me wrong, but I think, that what am I trying to achieve is impossible.
Python has immutable strings, so passing a "reference" of string to function and expecting ability to change it from inside a function is simply not valid.
Take this code as an example:
variable = "Hello"
def changer(var):
var = "Bye"
changer(variable)
print(variable)
Prints "Hello". In Python, you can't make it work differently. (although to be exact, it is still being passed as a reference, but when you modify Python string, you just create a new one and a new reference).
So, how to get arround this?
Simple! Create a C++ wrapper, that will handle passing reference on std::string and return copy of resulting string. Not very effective, but you probably can't make it better.
Sample code of SaveWrapper class:
class SaveWrapper {
public:
// some constructor
std::string save(MiddleClass & value) {
std::string result;
Save saver(result);
saver.save(value);
return result;
}
};
Which can be easily "ported" to Python!
I have a python extension module written in C++, which contains multiple functions. One of these generates an instance of a custom structure, which I then want to use with other functions of my module in Python as follows
import MyModule
var = MyModule.genFunc()
MyModule.readFunc(var)
To do this, I've tried using PyCapsule objects to pass a pointer to these objects between Python and C, but this produces errors when attempting to read them in the second C function ("PyCapsule_GetPointer called with invalid PyCapsule object"). Python, however, if asked to print the PyCapsule object (var) correctly identifies it as a "'capsule object "testcapsule"'. My C code appears as follows:
struct MyStruct {
int value;
};
static PyObject* genFunc(PyObject* self, PyObject *args) {
MyStruct var;
PyObject *capsuleTest;
var.value = 1;
capsuleTest = PyCapsule_New(&var, "testcapsule", NULL);
return capsuleTest;
}
static PyObject* readFunc(PyObject* self, PyObject *args) {
PyCapsule_GetPointer(args, "testcapsule");
return 0;
}
Thank you for your help.
Like stated in a comment to your question, you'll run into an issue when reading data from the local variable MyStruct var. For this you can use the third destructor to PyCapsule_New.
But that's not the reason for your problem just now. You're using PyCapsule_GetPointer(args, "testcapsule") on the args parameter. And since it's not a capsule, even though var is one, you might have defined the signature of the function as METH_VARARGS. Instead you need to unpack the tuple or use METH_O.
I try to expose two different classes to python, but I don't get it to compile. I tried to follow the boost::python example, which works quite well. But if I try to write the wrapper classes for my classes it doesn't work. I have provided two minimal examples below:
struct Base
{
virtual ~Base() {}
virtual std::unique_ptr<std::string> f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
std::unique_ptr<std::string> f()
{
return this->get_override("f")();
}
};
and
struct Base
{
virtual ~Base() {}
virtual void f() = 0;
};
struct BaseWrap : Base, python::wrapper<Base>
{
void f()
{
return this->get_override("f")();
}
};
The first one does not compile because of the unique pointer(I think boost::python does not use unique pointers?) and the second example complains about the return statement inside the void function. Can someone help me how to solve this problems?
The examples are failing to compile because:
The first example attempts to convert an unspecified type (the return type of override::operator()) to an incompatible type. In particular, Boost.Python does not currently support std::unique_ptr, and hence will not convert to it.
The second example attempts to return the unspecified type mentioned above when the calling function declares that it returns void.
From a Python perspective, strings are immutable, and attempting to transferring ownership of a string from Python to C++ violates semantics. However, one could create a copy of a string within C++, and pass ownership of the copied string to C++. For example:
std::unique_ptr<std::string> BaseWrap::f()
{
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("f")();
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
The object returned from this->get_override("f")() has an unspecified type, but can be used to convert to C++ types. The invocation of the override will throw if Python throws, and the conversion to the C++ type will throw if the object returned from Python is not convertible to the C++ type.
Here is a complete example demonstrating two ways to adapt the returned Python object to a C++ object. As mentioned above, the override conversion can be used. Alternatively, one can use boost::python::extract<>, allowing one to check if the conversion will fail before performing the conversion:
#include <memory> // std::unique_ptr
#include <boost/algorithm/string.hpp> // boost::to_upper_copy
#include <boost/python.hpp>
struct base
{
virtual ~base() {}
virtual std::unique_ptr<std::string> perform() = 0;
};
struct base_wrap : base, boost::python::wrapper<base>
{
std::unique_ptr<std::string> perform()
{
namespace python = boost::python;
// This could throw if the Python method throws or the Python
// method returns a value that is not convertible to std::string.
std::string result = this->get_override("perform")();
// Alternatively, an extract could be used to defer extracting the
// result.
python::object method(this->get_override("perform"));
python::extract<std::string> extractor(method());
// Check that extractor contains a std::string without throwing.
assert(extractor.check());
// extractor() would throw if it did not contain a std::string.
assert(result == extractor());
// Adapt the result to the return type.
return std::unique_ptr<std::string>(new std::string(result));
}
};
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<base_wrap, boost::noncopyable>("Base", python::init<>())
.def("perform", python::pure_virtual(&base::perform))
;
python::def("make_upper", +[](base* object) {
auto result = object->perform(); // Force dispatch through base_wrap.
assert(result);
return boost::to_upper_copy(*result);
});
}
Interactive usage:
>>> import example
>>> class Derived(example.Base):
... def perform(self):
... return "abc"
...
>>> derived = Derived()
>>> assert("ABC" == example.make_upper(derived))
I am new to python and C-extensions. I am writing a python code where I have created two threads and have defined a callback function py_cb().
In one thread I am appending to a global list after certain time intervals, whereas from the other thread, I am calling a C-extension library api. The C-extension api spawns a thread that calls the python callback function (py_cb) defined in my original file. In the callback function, I am trying to display the global list, but it seems it has a different id. I checked it using id(listName). Is there a way I can use the right global variable in the C-extension python callback function? I was assuming that global values will be shared across all the threads, is that not the case? Also, please suggest better solutions.
Following is the C code:
char *MODULE_NAME = "simple";
---
PyMODINIT_FUNC init_cmodule(void)
{
PyObject *m = Py_InitModule3("_cmodule",module_methods,module_docstring);
if (m == NULL)
return;
}
static PyObject *cmodule_cmodule(PyObject *self, PyObject *args)
{
int value = register_cb();
PyObject *ret = Py_BuildValue("i",value);
return ret;
}
void notifyFooHandle()
{
printf("inside the notify_foo function\n");
pModule = PyImport_ImportModule(MODULE_NAME);
if (!pModule) {
printf ("Failed to load the module\n");
return;
}
PyObject *pFunc;
pFunc = PyObject_GetAttrString(pModule, "py_cb");
if (pFunc && PyCallable_Check(pFunc)) {
PyObject_CallObject(pFunc,NULL);
}
else {
Py_DECREF(pFunc);
PyErr_Print();
printf("Failed to send notification\n");
return;
}
return;
}
void notify_foo(void)
{
int t = 5;
while (t < 10) {
notifyFooHandle();
sleep(5);
t++;
}
return;
}
int register_cb(void)
{
pthread_t notify_t;
int rc = pthread_create(¬ify_t, NULL, (void *) ¬ify_foo,NULL);
---
}
Following is the python callback API in simple.py file:
def py_cb():
print "Received a call back from cmodule"
global myList
print "Global myList is ", id(myList)
print myList
return
Can you show us the code in your C library? If you're initializing a python VM there and then calling py_cb, it's easy to understand why the lists are different: you have two different python VM instances.
EDIT:
I think your problem is you're using two different python instances. First, you have your main program in python. In that instance you have a global "myList" so every function invoked from there will be accessing that particular instance of "myList". Then, you load a C module. When that C module opens you original python module in order to load py_cb, you are using a different python instance, you'll have a second "myList". In short, you have TWO different python instances running, the one you create when you run your main python script and one you create inside your C library to call py_cb.
If you want to share a common python instance, you'll have to create you instance in C, make it globally accessible in your C module, insert there you C functions and then run your main python function. When python calls to a C function, you'll be inside your original address space and not a new one, and when you make a call back to python you'll be using always the same python instance.
I am trying to embed python within an exiting C++ application. However I am stuck at trying to call an instance member function from the python script.
I have an existing class in which I have a private member function that wraps the native function and meets the PyCFunction interface.
class MY_CLASS {
public:
void MY_CLASS();
void submit(TypeA a, int b);
private:
PyObject* PythonMethod_submit(PyObject*, PyObject*);
}
When I create the PyMethodDef I use the method name to identify the function I want however I get a compliation error indicating I have to the wrong type signature.
MY_CLASS::MY_CLASS() {
// Python C/API Intilization stuff
PyMethodDef ModuleMethods[] = {
{ "submit",
PythonMethod_submit,
METH_VARARGS,
"docstring" }
};
Py_InitModule("myclass", ModuleMethods);
// Further interaction with the embeded interpreter
}
If I use a static function as described in the Python C API this process works, when I use the class member I recieve an error asserting PyObject* (MY_CLASS::)(PyObject*, PyObject*) does not match PyObject* (*)(PyObject*, PyObject*).
Is it possible to pass a member function as function pointer as stated in the error?
No. A non-static member function has a MY_CLASS * implicitly passed in (as this), whereas a static member function, which behaves like a C function, does not. See this page for the different type signatures: http://www.parashift.com/c++-faq/fnptr-vs-memfnptr-types.html
What you can do as an alternative is have a member in your Python object that holds a pointer (MY_CLASS *) to your C++ object.