So, I have a simple event library, written in C++ and using the Boost libraries. I wanted to expose said library to Python, so naturally I turned to Boost::Python. I got the code to compile, eventually, but now I'm faced with quite the problem: my library uses higher-order programming techniques. For example, the library is made up of three main classes: an event class, an event manager class, and an event listener class. The event listener class poses a problem. Code:
class listener{
public:
listener(){}
void alert(cham::event::event e){
if (responses[e.getName()])
responses[e.getName()](e.getData());
}
void setResponse(std::string n, boost::function<void (std::string d)> c){responses.insert(make_pair(n, c));}
void setManager(_manager<listener> *m){manager = m;}
private:
std::map<std::string, boost::function<void (std::string d)> > responses;
_manager<listener> *manager;
As you can see, the function setResponse is the problem. It requires a function to be passed to it, and, unfortunately, Boost::Python does not apply it's converter magic in this situation. When called like the following:
>>> import chameleon
>>> man = chameleon.manager()
>>> lis = chameleon.listener()
>>> def oup(s):
... print s
...
>>> lis.setResponse("event", oup)
it gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
listener.setResponse(listener, str, function)
did not match C++ signature:
setResponse(cham::event::listener {lvalue}, std::string, boost::function<void ()(std::string)>)
So, my question is, how could I fix this? It would have to either use overloading or a wrapper, as I would like the library to remain callable by C++.
You will need a wrapper around setResponse, which takes a boost::python::object instead of a function. It should store this bp::object in a known location (probably a member variable of a listener subclass).
Then pass a different c++ function to the base setResponse, that will know how to lookup and call the function in the bp::object. If events are to be called on a different thread, you will also need to ensure proper handling of python's Global Interpreter Lock, as discussed here: boost.python not supporting parallelism?.
Related
I'm trying to access a class inside a .dll file using pythonnet and I am unable to create and instance of a class without the following error. I'm using the latest version of pythonnet (2.5.2) and Python 3.10.5.
Error
Traceback (most recent call last):
x = IDispenser()
TypeError: interface takes exactly one argument
Python Code
import clr
import sys
dll_path = "C:\\Program Files\\Seyonic\\Dispenser Control Monitor 3.8"
if dll_path not in sys.path:
sys.path.append("C:\\Program Files\\Seyonic\\Dispenser Control Monitor 3.8")
assert dll_path in sys.path
clr.AddReference("Seyonic.Dispenser")
from Seyonic.Dispenser import IDispenser
x = IDispenser()
DLL
namespace Seyonic.Dispenser
{
public interface IDispenser
{
// ----------------------------------------------------------------------------
// Properties
// ----------------------------------------------------------------------------
int Data { get;}
...etc
}
}
After digging through the dll file (doesn't seem to take any arguments for using these classes) I think this is a pythonnet / python3 problem. Here is an excerpt from https://github.com/DynamoDS/Dynamo/wiki/Work-in-progress-to-improve-Python-3-support:
Python classes cannot implement .NET interfaces (Not planned)
This is probably an advanced usage scenario of Python and interoperability with .NET. Here a Python class is defined in such a way that it implements a .NET interface. The idea behind this is that instances of the class can be created and passed as arguments to methods accepting them. Here is an example of how this could look like:
import clr
clr.AddReference('SomeLibrary')
from SomeLibrary import SomeInterface, SomeClass
class MyClass(SomeInterface):
def __init__(self):
pass
inst = MyClass()
result = SomeClass.SomeMethodTakingSomeInterface(inst)
OUT = result
Given a valid library was provided, the previous code sample would work without issues in the IronPython2 engine. When using CPython3, however, you can get TypeError : interface takes exactly one argument or TypeError : object does not implement SomeInterface, depending on the interface and method definitions. The required changes in Python .NET to make this work seem big when compared to the relevance we have detected for this use case, so we have decided to not plan this work yet.
I'm using PyBind11 to run a Python interpreter, and I need to call a Python function in c++ with some pointer arguments.
According to the docs of pybind11, it looks like that a argument being passed to Python side should be freed normally by the Python interpreter, instead of the c++ main program.
But this time the argument is a pointer to a static object, it should NOT be freed by anyone. How to code such a binding/calling?
I know that pybind11::return_value_policy::reference can be used to prevent a returning result from being freed, but it is for a returning object, not for arguments.
Any hint will be appreciated!
You can define a python wrapper inside an embedded module and access it from inside the python code. Then you can specify an appropriate return value policy when defining the getter, like this:
extern MyObject global_object;
PYBIND11_EMBEDDED_MODULE(statics, m) {
m.def(
"global_object",
[]() -> MyObject& {
return global_object;
},
pybind11::return_value_policy::reference
);
}
Then either call the statics.global_object() directly from the python code, or, if you still want to use pass it as an argument from C++, call the statics.global_object() in the interpreter and store the result in C++ as a py::object which you can then pass as an argument.
I have a program that is written in C++ and can be extended with extensions that are written in python using python's C api, basically there is python interpreter in the program which can load python modules that contains some hooks to various events.
I can pass simple data types from C to python, such as string or integer. I can do the same the other way, I can call internal C++ api's I created from the python modules. What I can't do, however is to marshall more complex data types, such as classes. For example I have this thing:
class X
{
int a;
int b;
};
I could call some python hook that would look like this:
def some_hook(a, b):
print(str(a + b))
but I would rather do
def some_hook(x):
print(str(x.a + x.b))
I know how to call a python function from C using PyObject_CallObject and how to give it some simple python vars as parameters (like integer or string). But I have no idea how to construct python classes from within C/C++, fill them up with data, and then pass them as parameters to PyObject_CallObject.
My idea was to create some proxy classes like in SWIG (swig.org) that I would fill up with data from C++ class and then back. I created some .py file that contains the declaration of these classes, but still, I have no idea how would I instantiate them from within C/C++ and fill up? Neither how I would do it the other way - turn PyObject into C/C++ class?
In a nutshell: I want to be able to turn C++ class into a python class, call a python function with that class as a parameter, and then call some other function from python with this class that would be just a C++ API that would turn the python class into C++.
Some background reading on how to call python from C: http://docs.python.org/3/extending/embedding.html
Source code of my app C++ that loads and uses python extensions: https://github.com/huggle/huggle3-qt-lx/blob/master/huggle/pythonengine.cpp
I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.
Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c
The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.
The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.
Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!
I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.
I have a C++ class that requires a function pointer in it's constructor (float(*myfunction)(vector<float>*))
I've already exposed some function pointers to Python.
The ideal way to use this class is something like this:
import mymodule
mymodule.some_class(mymodule.some_function)
So I tell Boost about this class like so:
class_<SomeClass>("some_class", init<float(*)(vector<float>*)>);
But I get:
error: no matching function for call to 'register_shared_ptr1(Sample (*)(std::vector<double, std::allocator<double> >*))'
when I try to compile it.
So, does anyone have any ideas on how I can fix the error without losing the flexibility gained from function pointers (ie no falling back to strings that indicate which function to call)?
Also, the main point of writing this code in C++ is for speed. So it would be nice if I was still able to keep that benefit (the function pointer gets assigned to a member variable during initialization and will get called over a million times later on).
OK, so this is a fairly difficult question to answer in general. The root cause of your problem is that there really is no python type which is exactly equivalent to a C function pointer. Python functions are sort-of close, but their interface doesn't match for a few reasons.
Firstly, I want to mention the technique for wrapping a constructor from here:
http://wiki.python.org/moin/boost.python/HowTo#namedconstructors.2BAC8factories.28asPythoninitializers.29. This lets you write an __init__ function for your object that doesn't directly correspond to an actual C++ constructor. Note also, that you might have to specify boost::python::no_init in the boost::python::class_ construction, and then def a real __init__ function later, if your object isn't default-constructible.
Back to the question:
Is there only a small set of functions that you'll usually want to pass in? In that case, you could just declare a special enum (or specialized class), make an overload of your constructor that accepts the enum, and use that to look up the real function pointer. You can't directly call the functions yourself from python using this approach, but it's not that bad, and the performance will be the same as using real function pointers.
If you want to provide a general approach that will work for any python callable, things get more complex. You'll have to add a constructor to your C++ object that accepts a general functor, e.g. using boost::function or std::tr1::function. You could replace the existing constructor if you wanted, because function pointers will convert to this type correctly.
So, assuming you've added a boost::function constructor to SomeClass, you should add these functions to your python wrapping code:
struct WrapPythonCallable
{
typedef float * result_type;
explicit WrapPythonCallable(const boost::python::object & wrapped)
: wrapped_(wrapped)
{ }
float * operator()(vector<float>* arg) const
{
//Do whatever you need to do to convert into a
//boost::python::object here
boost::python::object arg_as_python_object = /* ... */;
//Call out to python with the object - note that wrapped_
//is callable using an operator() overload, and returns
//a boost::python::object.
//Also, the call can throw boost::python::error_already_set -
//you might want to handle that here.
boost::python::object result_object = wrapped_(arg_as_python_object);
//Do whatever you need to do to extract a float * from result_object,
//maybe using boost::python::extract
float * result = /* ... */;
return result;
}
boost::python::object wrapped_;
};
//This function is the "constructor wrapper" that you'll add to SomeClass.
//Change the return type to match the holder type for SomeClass, like if it's
//held using a shared_ptr.
std::auto_ptr<SomeClass> CreateSomeClassFromPython(
const boost::python::object & callable)
{
return std::auto_ptr<SomeClass>(
new SomeClass(WrapPythonCallable(callable)));
}
//Later, when telling Boost.Python about SomeClass:
class_<SomeClass>("some_class", no_init)
.def("__init__", make_constructor(&CreateSomeClassFromPython));
I've left out details on how to convert pointers to and from python - that's obviously something that you'll have to work out, because there are object lifetime issues there.
If you need to call the function pointers that you'll pass in to this function from Python, then you'll need to def these functions using Boost.Python at some point. This second approach will work fine with these def'd functions, but calling them will be slow, because objects will be unnecessarily converted to and from Python every time they're called.
To fix this, you can modify CreateSomeClassFromPython to recognize known or common function objects, and replace them with their real function pointers. You can compare python objects' identity in C++ using object1.ptr() == object2.ptr(), equivalent to id(object1) == id(object2) in python.
Finally, you can of course combine the general approach with the enum approach. Be aware when doing this, that boost::python's overloading rules are different from C++'s, and this can bite you when dealing with functions like CreateSomeClassFromPython. Boost.Python tests functions in the order that they are def'd to see if the runtime arguments can be converted to the C++ argument types. So, CreateSomeClassFromPython will prevent single-argument constructors def'd later than it from being used, because its argument matches any python object. Be sure to put it after other single-argument __init__ functions.
If you find yourself doing this sort of thing a lot, then you might want to look at the general boost::function wrapping technique (mentioned on the same page with the named constructor technique): http://wiki.python.org/moin/boost.python/HowTo?action=AttachFile&do=view&target=py_boost_function.hpp.