Develop Reference

Pybind11: external type as return value

With pybind11 I need to bind a function that returns an object type whose binding is already available from another extension module (also created with pybind11). I know this is generally possible since the docs say so and I've also managed to build a complete toy example where things work fine. However, for some reason I don't understand, I cannot make it work If the external type is the Tensor type from https://github.com/deephealthproject/pyeddl. I know that -fvisibility=hidden is not an issue because:
it's also present in the toy example;
I've also tried rebuilding pyeddl without it.
src/dummy_bindings.cpp:
#include <pybind11/pybind11.h>
#include <eddl/tensor/tensor.h>
Tensor* mkTensor() {
return new Tensor();
}
PYBIND11_MODULE(_foo, m) {
m.def("mkTensor", &mkTensor);
}
setup.py:
from distutils.core import setup, Extension
import pybind11
ext = Extension(
"_foo",
sources=["src/dummy_bindings.cpp"],
include_dirs=[
"src",
pybind11.get_include(),
pybind11.get_include(user=True)
],
libraries=["eddl"],
)
setup(
name="foo",
ext_modules=[ext]
)
Test python code:
import pyeddl._core
import _foo
print(pyeddl._core.Tensor)
t = _foo.mkTensor()
This leads to:
<class 'pyeddl._core.Tensor'>
Traceback (most recent call last):
File "foo.py", line 5, in <module>
t = _foo.mkTensor()
TypeError: Unable to convert function return value to a Python type! The signature was
() -> Tensor
My understanding is that the import pyeddl._core should make the Tensor type visible at runtime and make the call to mkTensor work. At least, it does work in the toy example. However, I get the above error. I'm also the author of the pyeddl bindings, so I'm wondering if I'm doing something wrong there that prevents the type from being findable by other extensions. Or perhaps there is something more about the way pybind11 decides if a value can be converted to a Python type that I don't understand.

The reason it didn't work is that the two extensions were compiled with different versions of gcc.

Python 'static core class'

Currently I'm making a game server, I actually want to make a base project for all my server products in Python. I used to use C# but I wanted to do something different so I started on Python. Although I don't know how to do something.
In C# I used to make one static 'core' class holding all data, containing the entry point and there I boot the server. Here's a small example:
using System;
namespace SERVER
{
public static class Engine
{
public static DatabaseHelper DatabaseHelper { get; private set; }
static void Main(string[] args)
{
DatabaseHelper = new DatabaseHelper();
}
}
}
Then I just could use in every class:
Engine.DatabaseHelper.SomeMethod();
Now, I want the same concept in my Python project, but I don't know how to do it.
I got this first:
DynamicEmu.py
import Engine
engine = Engine.Engine()
Engine.py
from Network.Game.GameConnectionListener import GameConnectionListener
class Engine:
gameConnection = None
def __init__(self):
gameConnection = GameConnectionListener()
def tuple_to_addr(self, tpl):
return '{0}:{1}'.format(tpl[0], str(tpl[1]))
And then in my GameConnectionListener.py I would use:
import DynamicEmu
In order to do
DynamicEmu.engine.tuple_to_addr(TUPLE HERE)
But I get this error
Traceback (most recent call last):
File "C:/Users/Josh/PycharmProjects/DynamicEmu/DynamicEmu.py", line 1, in <module>
import Engine
File "C:\Users\Josh\PycharmProjects\DynamicEmu\Engine.py", line 3, in <module>
from Network.Game.GameConnectionListener import GameConnectionListener
File "C:\Users\Josh\PycharmProjects\DynamicEmu\Network\Game\GameConnectionListener.py", line 4, in <module>
import DynamicEmu
File "C:\Users\Josh\PycharmProjects\DynamicEmu\DynamicEmu.py", line 3, in <module>
engine = Engine.Engine()
AttributeError: module 'Engine' has no attribute 'Engine'
I suck in Python so if I'm doing something REALLY wrong I understand, I want to learn so I'd appreciate every piece of help I can get :)

Typical Python code doesn't use static members very much.1 So I would advise making most of your variables instance variables. For instance, in DynamicEmu.py, don't have a module-level Engine instance. Instead, have a class which is initialized with an Engine instance
class DynamicEmu:
def __init__(self, engine):
self.engine = engine
...
or a function that accepts an Engine instance as an argument
def dynamic_emu(engine):
...
It's hard to be more specific without knowing what the code in DynamicEmu.py is supposed to do.
Similarly, in GameConnectionListener.py you would define your listener like
class GameConnectionListener:
def __init__(self, engine):
self.engine = engine
...
and so on with other code. (Bear in mind that Python functions are objects, and can be passed as arguments and used as callbacks. Consider whether you could get away with a function, not a class.)
Obviously, that Engine instance has to be created somewhere. A logical place might be in your main() function, or in some setup function that gets called early in the program. You can then pass it around to other functions and methods as needed.
In my experience, writing a large project tends to work out best when you are clear about the dependencies between different parts of the code. For example, be clear about whether the game engine uses the listener or the listener uses the game engine. If the engine uses the listener, then you should be using GameConnectionListener in Engine, but not vice-versa. The GameConnectionListener should be designed in a way that it doesn't know about the Engine. This isn't always possible, of course, but to whatever extent you can do it, I think it helps. It reduces or eliminates circular dependencies as you have in your example code, and it also helps you write each component of the program separately without having to worry about others at the same time.
1Or, what C# calls static members; in Python they're called class members, but nearly the same functionality is often provided by module-level variables, which are simpler.

how to include shared object in python [duplicate]

I'm just getting started with ctypes and would like to use a C++ class that I have exported in a dll file from within python using ctypes.
So lets say my C++ code looks something like this:
class MyClass {
public:
int test();
...
I would know create a .dll file that contains this class and then load the .dll file in python using ctypes.
Now how would I create an Object of type MyClass and call its test function? Is that even possible with ctypes? Alternatively I would consider using SWIG or Boost.Python but ctypes seems like the easiest option for small projects.

Besides Boost.Python(which is probably a more friendly solution for larger projects that require one-to-one mapping of C++ classes to python classes), you could provide on the C++ side a C interface. It's one solution of many so it has its own trade offs, but I will present it for the benefit of those who aren't familiar with the technique. For full disclosure, with this approach one wouldn't be interfacing C++ to python, but C++ to C to Python. Below I included an example that meets your requirements to show you the general idea of the extern "c" facility of C++ compilers.
//YourFile.cpp (compiled into a .dll or .so file)
#include <new> //For std::nothrow
//Either include a header defining your class, or define it here.
extern "C" //Tells the compile to use C-linkage for the next scope.
{
//Note: The interface this linkage region needs to use C only.
void * CreateInstanceOfClass( void )
{
// Note: Inside the function body, I can use C++.
return new(std::nothrow) MyClass;
}
//Thanks Chris.
void DeleteInstanceOfClass (void *ptr)
{
delete(std::nothrow) ptr;
}
int CallMemberTest(void *ptr)
{
// Note: A downside here is the lack of type safety.
// You could always internally(in the C++ library) save a reference to all
// pointers created of type MyClass and verify it is an element in that
//structure.
//
// Per comments with Andre, we should avoid throwing exceptions.
try
{
MyClass * ref = reinterpret_cast<MyClass *>(ptr);
return ref->Test();
}
catch(...)
{
return -1; //assuming -1 is an error condition.
}
}
} //End C linkage scope.
You can compile this code with
gcc -shared -o test.so test.cpp
#creates test.so in your current working directory.
In your python code you could do something like this (interactive prompt from 2.7 shown):
>>> from ctypes import cdll
>>> stdc=cdll.LoadLibrary("libc.so.6") # or similar to load c library
>>> stdcpp=cdll.LoadLibrary("libstdc++.so.6") # or similar to load c++ library
>>> myLib=cdll.LoadLibrary("/path/to/test.so")
>>> spam = myLib.CreateInstanceOfClass()
>>> spam
[outputs the pointer address of the element]
>>> value=CallMemberTest(spam)
[does whatever Test does to the spam reference of the object]
I'm sure Boost.Python does something similar under the hood, but perhaps understanding the lower levels concepts is helpful. I would be more excited about this method if you were attempting to access functionality of a C++ library and a one-to-one mapping was not required.
For more information on C/C++ interaction check out this page from Sun: http://dsc.sun.com/solaris/articles/mixing.html#cpp_from_c

The short story is that there is no standard binary interface for C++ in the way that there is for C. Different compilers output different binaries for the same C++ dynamic libraries, due to name mangling and different ways to handle the stack between library function calls.
So, unfortunately, there really isn't a portable way to access C++ libraries in general. But, for one compiler at a time, it's no problem.
This blog post also has a short overview of why this currently won't work. Maybe after C++0x comes out, we'll have a standard ABI for C++? Until then, you're probably not going to have any way to access C++ classes through Python's ctypes.

The answer by AudaAero is very good but not complete (at least for me).
On my system (Debian Stretch x64 with GCC and G++ 6.3.0, Python 3.5.3) I have segfaults as soon has I call a member function that access a member value of the class.
I diagnosticated by printing pointer values to stdout that the void* pointer coded on 64 bits in wrappers is being represented on 32 bits in Python. Thus big problems occurs when it is passed back to a member function wrapper.
The solution I found is to change:
spam = myLib.CreateInstanceOfClass()
Into
Class_ctor_wrapper = myLib.CreateInstanceOfClass
Class_ctor_wrapper.restype = c_void_p
spam = c_void_p(Class_ctor_wrapper())
So two things were missing: setting the return type to c_void_p (the default is int) and then creating a c_void_p object (not just an integer).
I wish I could have written a comment but I still lack 27 rep points.

Extending AudaAero's and Gabriel Devillers answer I would complete the class object instance creation by:
stdc=c_void_p(cdll.LoadLibrary("libc.so.6"))
using ctypes c_void_p data type ensures the proper representation of the class object pointer within python.
Also make sure that the dll's memory management be handled by the dll (allocated memory in the dll should be deallocated also in the dll, and not in python)!

I ran into the same problem. From trial and error and some internet research (not necessarily from knowing the g++ compiler or C++ very well), I came across this particular solution that seems to be working quite well for me.
//model.hpp
class Model{
public:
static Model* CreateModel(char* model_name) asm("CreateModel"); // static method, creates an instance of the class
double GetValue(uint32_t index) asm("GetValue"); // object method
}
#model.py
from ctypes import ...
if __name__ == '__main__':
# load dll as model_dll
# Static Method Signature
fCreateModel = getattr(model_dll, 'CreateModel') # or model_dll.CreateModel
fCreateModel.argtypes = [c_char_p]
fCreateModel.restype = c_void_p
# Object Method Signature
fGetValue = getattr(model_dll, 'GetValue') # or model_dll.GetValue
fGetValue.argtypes = [c_void_p, c_uint32] # Notice two Params
fGetValue.restype = c_double
# Calling the Methods
obj_ptr = fCreateModel(c_char_p(b"new_model"))
val = fGetValue(obj_ptr, c_int32(0)) # pass in obj_ptr as first param of obj method
>>> nm -Dg libmodel.so
U cbrt#GLIBC_2.2.5
U close#GLIBC_2.2.5
00000000000033a0 T CreateModel # <----- Static Method
U __cxa_atexit#GLIBC_2.2.5
w __cxa_finalize#GLIBC_2.2.5
U fprintf#GLIBC_2.2.5
0000000000002b40 T GetValue # <----- Object Method
w __gmon_start__
...
...
... # Mangled Symbol Names Below
0000000000002430 T _ZN12SHMEMWrapper4HashEPKc
0000000000006120 B _ZN12SHMEMWrapper8info_mapE
00000000000033f0 T _ZN5Model12DestroyModelEPKc
0000000000002b20 T _ZN5Model14GetLinearIndexElll
First, I was able to avoid the extern "C" directive completely by instead using the asm keyword which, to my knowledge, asks the compiler to use a given name instead of the generated one when exporting the function to the shared object lib's symbol table. This allowed me to avoid the weird symbol names that the C++ compiler generates automatically. They look something like the _ZN1... pattern you see above. Then in a program using Python ctypes, I was able to access the class functions directly using the custom name I gave them. The program looks like fhandle = mydll.myfunc or fhandler = getattr(mydll, 'myfunc') instead of fhandle = getattr(mydll, '_ZN12...myfunc...'). Of course, you could just use the long name; it would make no difference, but I figure the shorter name is a little cleaner and doesn't require using nm to read the symbol table and extract the names in the first place.
Second, in the spirit of Python's style of object oriented programming, I decided to try passing in my class' object pointer as the first argument of the class object method, just like when we pass self in as the first method in Python object methods. To my surprise, it worked! See the Python section above. Apparently, if you set the first argument in the fhandle.argtypes argument to c_void_ptr and pass in the ptr you get from your class' static factory method, the program should execute cleanly. Class static methods seem to work as one would expect like in Python; just use the original function signature.
I'm using g++ 12.1.1, python 3.10.5 on Arch Linux. I hope this helps someone.

Python/Plone - Is it possible to call a global utility at module level of a py file?

I have a product for a plone site with a module containing a utility class, and in the module that will/should use this utility, I am trying to have it setup at the module level.
In the module containing the utility (my.product.testutility), I have this:
from five import grok
from zope.interface import Interface
class ITestUtil(Interface):
"""Interface of utility
"""
def returnTest(self):
"""return a string for now
"""
class TestUtil(object):
"""Utility class test
"""
grok.implements(ITestUtil)
def returnTest(self):
return "testing"
grok.global_utility(TestUtil, name="TestUtility")
In the module that will use this utility (my.product.stringtesting):
from five import grok
from my.package.testutility import ITestUtil
from zope import component, schema
from Products.CMFCore.interfaces import ISiteRoot
utilForTesting = component.getUtility(ITestUtil, name="TestUtility")
class IStringTest(Interface):
......
class View(grok.View):
def returnStringForTest(self):
return utilForTesting.returnTest()
I also had the template file that would call the returnStringForTest to display the string on the rendered page.
I end up getting this error unfortunately:
ComponentLookupError: (< InterfaceClass my.product.testutility.ITestUtil >, "TestUtility")
I did try several different things like using grok.GlobalUtility as a base as opposed to making it an object registering it through grok.global_utility. I did remove the name parameter in the class using this while testing this.
The documentation I was trying to follow was the References on the grok site, looking at the directives page where it has the global utility information.
Also, I am using grok 0.9.
Edit:
The version of Plone I am using is Plone 4 and the version of python I am using is 2.7.
Is it possible to have the utility set up at the module level like I was trying?

You can do what you want without relying on Zope at all.
You can change the line in my.product.stringtesting:
from
utilForTesting = component.getUtility(ITestUtil, name="TestUtility")
to
utilForTesting = TestUtil()

Higher-Order Programming Using Boost::Python

So, I have a simple event library, written in C++ and using the Boost libraries. I wanted to expose said library to Python, so naturally I turned to Boost::Python. I got the code to compile, eventually, but now I'm faced with quite the problem: my library uses higher-order programming techniques. For example, the library is made up of three main classes: an event class, an event manager class, and an event listener class. The event listener class poses a problem. Code:
class listener{
public:
listener(){}
void alert(cham::event::event e){
if (responses[e.getName()])
responses[e.getName()](e.getData());
}
void setResponse(std::string n, boost::function<void (std::string d)> c){responses.insert(make_pair(n, c));}
void setManager(_manager<listener> *m){manager = m;}
private:
std::map<std::string, boost::function<void (std::string d)> > responses;
_manager<listener> *manager;
As you can see, the function setResponse is the problem. It requires a function to be passed to it, and, unfortunately, Boost::Python does not apply it's converter magic in this situation. When called like the following:
>>> import chameleon
>>> man = chameleon.manager()
>>> lis = chameleon.listener()
>>> def oup(s):
... print s
...
>>> lis.setResponse("event", oup)
it gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
listener.setResponse(listener, str, function)
did not match C++ signature:
setResponse(cham::event::listener {lvalue}, std::string, boost::function<void ()(std::string)>)
So, my question is, how could I fix this? It would have to either use overloading or a wrapper, as I would like the library to remain callable by C++.

You will need a wrapper around setResponse, which takes a boost::python::object instead of a function. It should store this bp::object in a known location (probably a member variable of a listener subclass).
Then pass a different c++ function to the base setResponse, that will know how to lookup and call the function in the bp::object. If events are to be called on a different thread, you will also need to ensure proper handling of python's Global Interpreter Lock, as discussed here: boost.python not supporting parallelism?.