Develop Reference

what is the python C api function that returns a PyObject using the object name

I'm embedding Python into my C++ and creating PyObjects to represent my data/objects (ints, doubles, strings, etcetera).
I've put in several hours trying to find the answer to the above question, I expected there'd be a "name" property or "name()" method to set, and reference, the canonical object name used in Python script (as global/local objects), that there'd be a function:
PyObject *PyObject_ByName(PyObject *PyObjs, char* name)
Return value: New reference. Part of the Stable ABI.
Return a new PyObject reference from an array of PyObjs that match the 'name', or NULL on failure.
What am I missing? I see all the other pieces in place.
MSEXPORT PyObject* PyVariant(VarObj* data, tLvVarErr* error) {
PyObject* module_name, * module, * dict, * python_class, * object;
PyObject *pValue = NULL; // <- what I want to set name
switch (data->data.index())
{
case VarIdx::I32:
pValue = PyLong_FromLong((long) get<int32_t>(data->data));
break;
case VarIdx::DBL:
pValue = PyFloat_FromDouble ((double) get<double>(data->data));
break;
case VarIdx::Str :
pValue = PyUnicode_FromString((char*) get<string*>(data->data)->c_str());
break;
default:
return NULL;
}
return pValue;
}

What you need to know is that Python namespaces are implemented based on dictionaries (the exception is the local namespace inside the function). Global name lookup is just to obtain the global namespace dictionary and search through the string key. Therefore, what you need to do is:
Use PyEval_GetGlobals to get the global namespace.
Use PyUnicode_FromString to construct a string object through a C string.
Use PyDict_GetItem to get object from the global namespace.
As for the _PyDict_LoadGlobal I mentioned in the comment area, it is an API used internally by Python to quickly load objects (starting with an underscore of function names). You should avoid using it when writing C extensions.

Transfer a c++ object between cython and pybind11 using python capsules

I have two c++ libraries that expose python API but using two different frameworks(pybind11 and cython). I need to transfer an object between them (both ways) using python capsules. Since cython and pybind11 use the python capsules different ways, is it even possible to make it work?
I have library A that defines a class Foo and exposes it with pybind11 to python. Library B exposes its API using cython. LibB owns a shared_ptr<Foo> which is a member of one of LibB's classes, say - Bar.
Bar returns the shared_ptr<Foo> member as a PyCapsule which I capture in pybind11 of the Foo class. I'm unpacking the shared_ptr<Foo> from the capsule, returning it to python and the user can operate on this object in python using pybind11 bindings for Foo.
Then I need to put it back in a capsule in pybind11 and return back to Bar.
Bar's python API operates on PyObject and PyCapsule because cython allows that. pybind11 and thus Foo's API does not accept those types and I'm forced to use pybind11::object and pybind11::capsule.
Everything works fine until the moment when I'm trying to use the pybind11::capsule created in pybind11, inside a cython method of class Bar which expects a PyCapsule*.
The shared_ptr<Foo> inside the pybind11::capsule is corrupted and my app crashes.
Has anyone tried to make those 2 libs talk to each other?
libA -> class Foo
namespace foo{
class Foo {
public:
void foo() {...}
}
}
libB -> class Bar
namespace bar {
class Bar {
public:
PyObject* get_foo() {
const char * capsule_name = "foo_in_capsule";
return PyCapsule_New(&m_foo, capsule_name, nullptr);
}
static Bar fooToBar(PyObject * capsule) {
void * foo_ptr = PyCapsule_GetPointer(capsule, "foo_in_capsule");
auto foo = static_cast<std::shared_ptr<foo::Foo>*>(foo_ptr);
// here the shared_ptr is corrupted (garbage numbers returned for use_count() and get() )
std::cout << "checking the capsule: " << foo->use_count() << " " << foo->get() << std::endl
Bar b;
b.m_foo = *foo; //this is what I would like to get
return b;
}
std::shared_ptr<Foo> m_foo;
};
}
pybind11 for Foo
void regclass_foo_Foo(py::module m)
{
py::class_<foo::Foo, std::shared_ptr<foo::Foo>> foo(m, "Foo");
foo.def("foo", &foo::Foo::foo);
foo.def_static("from_capsule", [](py::object* capsule) {
auto* pycapsule_ptr = capsule->ptr();
auto* foo_ptr = reinterpret_cast<std::shared_ptr<foo::Foo>*>(PyCapsule_GetPointer(pycapsule_ptr, "foo_in_capsule"));
return *foo_ptr;
});
foo.def_static("to_capsule", [](std::shared_ptr<foo::Foo>& foo_from_python) {
auto pybind_capsule = py::capsule(&foo_from_python, "foo_in_capsule", nullptr);
return pybind_capsule;
});
}
cython for Bar
cdef extern from "bar.hpp" namespace "bar":
cdef cppclass Bar:
object get_foo() except +
def foo_to_bar(capsule):
b = C.fooToBar(capsule)
return b
putting it all together in python
from bar import Bar, foo_to_bar
from foo import Foo
bar = Bar(... some arguments ...)
capsule1 = bar.get_foo()
foo_from_capsule = Foo.from_capsule(capsule1)
// this is the important part - need to operate on foo using its python api
print("checking if foo works", foo_from_capsule.foo())
// and use it to create another bar object with a (possibly) modified foo object
capsule2 = Foo.to_capsule(foo_from_capsule)
bar2 = foo_to_bar(capsule2)

There's too many unfinished details in your code for me to even test your PyCapsule version. My view is that the issue is with the lifetime of the shared pointers - your capsule points to a shared pointer that's lifetime is tied to the Bar it's in. However, the capsule may outlive that. You should probably be creating a new shared_ptr<Foo>* (with new), pointing to that in your capsule, and defining a destructor (for the capsule) to delete it.
An outline of an alternative approach that I think should work better is as follows:
Write your classes purely in terms of C++ types, so get_foo and foo_to_bar just take/return shared_ptr<Foo>.
Define PyBar as a proper Cython class, rather than using capsules:
cdef public class PyBar [object PyBarStruct, type PyBarType]:
cdef shared_ptr[Bar] ptr
cdef public PyBar PyBar_from_shared_ptr(shared_ptr[Bar] b):
cdef PyBar x = PyBar()
x.ptr = b
return x
This generates a header file containing definitions of PyBarStruct and PyBarType (you probably don't need the latter). I also define a basic module-level function to create a PyBar from a shared pointer (and make that public as well, so it appears in the header too).
Then use PyBind11 to define a custom type-caster to/from shared_ptr<Bar>. load would be something like:
bool load(handle src, bool) {
auto bar_mod = py::import("bar");
auto bar_type = py::getattr(bar_mod,"Bar");
if (!py::isinstance(src,bar_type)) {
return false;
}
// now cast to my PyBarStruct
auto ptr = reinterpret_cast<PyBarStruct*>(src.ptr());
value = ptr->ptr; // access the shared_ptr of the struct
}
while the C++ to Python caster would be something like
static handle cast(std::shared_ptr<Bar> src, return_value_policy /* policy */, handle /* parent */) {
auto bar_mod = py::import("bar"); // See note...
return PyBar_from_shared_ptr(src);
}
I've ensured to include py::import("bar") in both functions because I don't think it's safe to use the Cython-defined functions until the module has been imported somewhere and importing it in the casters does ensure that.
This code is untested so almost certainly has errors, but should give a cleaner approach than PyCapsule.

boost-python when C++ method returns std::map<string,X*>

I'm exposing an API to Python, written in C++ that I have no access to change, using Boost Python.
I have successfully exposed methods returning references to a std:map where the key,value pairs are value types - eg:
class_< std::map<std::string, std::string> >("StringMap")
.def(map_indexing_suite< std::map<std::string, std::string>, true >());
This works seamlessly. But when trying to achieve a similar result where the map values are pointers to classes I've exposed within the API doesn't work:
struct X_wrap : X, wrapper<X>
{
X_wrap(int i): X(i) {}
// virtual methods here, omitted for brevity - as unlikely to be the issue
}
BOOST_PYTHON_MODULE(my_py_extension)
{
class_< std::map<std::string, X*> >("XPtrMap")
.def(map_indexing_suite< std::map<std::string, X*> >());
class_<X_wrap, boost::noncopyable, bases<XBase> >("X", init<int>())
// other definitions omitted
}
Error seen in g++ 7.3.0:
/usr/include/boost/python/detail/caller.hpp:100:98: error: ‘struct boost::python::detail::specify_a_return_value_policy_to_wrap_functions_returning<X*>’ has no member named ‘get_pytype’
I understand why the compiler is complaining - the X* in the map needs to be wrapped in a call policy so that it can be returned to Python, just like with a basic method that returns a raw pointer.
My question is what is the best way to do this?
From Googling it strikes that I can perhaps specify a DerivedPolicies child class of map_indexing_suite that will overload the necessary parts to wrap the X* in an appropriate return_value_policy. However so far I've be unsuccessful in putting anything together that the compiler doesn't bawk at!
I also suspect I can literally copy-and-paste the whole map_indexing_suite and rename it, and make the changes therein to produce a new indexing_suite with the right return_value_policy, but this seems ugly compared to the solution using DerviedPolicies - assuming I'm right that DeriviedPolicies can be used at all!
Any help, pointers, or examples gratefully received!
EDIT
I have proved that the cut-and-paste option works with a single trivial change of is_class to is_pointer. It's curious that is_pointer is not allowed in the original as the target policy can handle pointers. I'm yet to convince myself that it's an object lifetime restriction that means pointers are not allowed in the original?
The whole class is public so I suspect it's possible to avoid the full cut-and-paste by simply inheriting from map_indexing_suite or perhaps by using the mysterious DerivedPolicies parameter?
extension_def(Class& cl)
{
// Wrap the map's element (value_type)
std::string elem_name = "mapptr_indexing_suite_";
object class_name(cl.attr("__name__"));
extract<std::string> class_name_extractor(class_name);
elem_name += class_name_extractor();
elem_name += "_entry";
typedef typename mpl::if_<
mpl::and_<is_pointer<data_type>, mpl::bool_<!NoProxy> >
, return_internal_reference<>
, default_call_policies
>::type get_data_return_policy;
class_<value_type>(elem_name.c_str())
.def("__repr__", &DerivedPolicies::print_elem)
.def("data", &DerivedPolicies::get_data, get_data_return_policy())
.def("key", &DerivedPolicies::get_key)
;
}
EDIT 2
Now see answer

Slightly cleaner implementation from the cut-and-paste is to inherit map_indexing_suite - a few tweaks are needed to make this work.
This seems reasonably sensible - if someone chimes in with a neater solution or can better explain DerivedPolicies then great, otherwise I'll accept the below as the answer in a few days or so...
using namespace boost;
using namespace boost::python;
//Forward declaration
template <class Container, bool NoProxy, class DerivedPolicies>
class mapptr_indexing_suite;
template <class Container, bool NoProxy>
class final_mapptr_derived_policies
: public mapptr_indexing_suite<Container,
NoProxy, final_mapptr_derived_policies<Container, NoProxy> > {};
template <
class Container,
bool NoProxy = false,
class DerivedPolicies
= final_mapptr_derived_policies<Container, NoProxy> >
class mapptr_indexing_suite
: public map_indexing_suite<
Container,
NoProxy,
DerivedPolicies
>
{
public:
// Must be explicit if the compiler is
// going to take from the base class
using typename map_indexing_suite<
Container,NoProxy,DerivedPolicies>::data_type;
using typename map_indexing_suite<
Container,NoProxy,DerivedPolicies>::value_type;
// Only one class needs to be overridden from the base
template <class Class>
static void
extension_def(Class& cl)
{
// Wrap the map's element (value_type)
std::string elem_name = "mapptr_indexing_suite_";
object class_name(cl.attr("__name__"));
extract<std::string> class_name_extractor(class_name);
elem_name += class_name_extractor();
elem_name += "_entry";
// use of is_pointer here is the only
// difference to the base map_indexing_suite
typedef typename mpl::if_<
mpl::and_<std::is_pointer<data_type>, mpl::bool_<!NoProxy> >
, return_internal_reference<>
, default_call_policies
>::type get_data_return_policy;
class_<value_type>(elem_name.c_str())
.def("__repr__", &DerivedPolicies::print_elem)
.def("data", &DerivedPolicies::get_data, get_data_return_policy())
.def("key", &DerivedPolicies::get_key)
;
}
};

Breaking up SWIG Python interface -- containers create namespace conflict

Our code base currently supports a single SWIG interface file (for Python) that has grown over the years to include roughly 300 C++ classes (technically interfaces), all of which inherit from a single base class, and all of which exist in a single global namespace. This allows us, with a minimal amount of SWIG code, to implement dynamic casting among the C++ classes that the SWIG classes represent while at the same time simplifying by keeping the C++ inheritance structure out of SWIG.
As long as we compiled our SWIG interface in a single module, this mechanism worked well -- but as the SWIG interface file has grown it has become difficult to manage, and compile/link times have grown. To address this I split the interface file up into separate modules by the names of the derived classes -- one module for class names beginning with "A" to "G", one for names beginning with "H" to "N", etc., resulting in four derived-class modules and a base class module. I was able to get these modules to compile and link, and exhibit expected behavior for the dynamic casting, following the method outlined here: (http://www.swig.org/Doc3.0/SWIGDocumentation.html#Modules_nn1)
However, breaking the single module into four parts (five parts counting the base class) causes problems with the namespace when containers come into play. Consider the following function, from a class in my v-to-z interface file:
void RemoveIsolated(const std::vector<global::IFoo*> spRemoveIsolated) {
…
}
That takes a vector of one of the derived classes that exist in the global namespace. This worked without issue when I had only one module but now class IFoo lives in the a-to-g module -- so if I cast something to an IFoo*, it's an a-to-g.IFoo*. However, the function demands a global::IFoo*.
This seems to be a situation that could be addressed by the SWIG template mechanism. I've seen discussions in which people have had success by means of at one point (possibly in the interface file for the base class??) declaring
%template(FooVector) std::vector<global::Foo*>;
And at another point (possibly in the interface file for the derived class??):
%template () std::vector<global::Foo*>;
But my attempts to implement this have not been successful. The discussions are somewhat ambiguous, it's possible that I'm doing something wrong. Can anyone provide clarification, ideally with an example?

The piece of information it looks like you're missing is the %import directive, which lets modules cooperate with the definition of types, without repeating them and still ending up with a single wrapped type. The documentation suggests using this to reduce module size even.
Probably all you need to do is have your v-to-z module %import the a-to-g module to get this working for you. (Personally I'd have tried to divide them up by functionality rather than alphabetically though, so the dependency between then wouldn't be an issue)

Thanks for your suggestion Flexo. Importing the a-to-g module did not work; the C++ compiler complained that all of the classes (interfaces) declared there were not part of the global namespace when it tried to compile to v-to-z wrapper file. However, going through the exercise led me to question why we had been having success previously when we were compiling a single module. It turned out that we were using a typemapping macro in the interface file for the single module that would take a
const std::vector<global::IFoo*>
and map it thusly:
TYPEMAPMACRO(global::IFoo, SWIGTYPE_p_global__IFoo)
for vector containers. The macro itself, for anyone who's interested, is:
%define TYPEMAPMACRO(type, name)
%typemap(in) const std::vector {
/*Check if is a list */
std::vector vec;
void *pobj = 0;
if(PyTuple_Check($input))
{
size_t size = PyTuple_Size($input);
for (size_t j = 0; j < size; j++) {
PyObject *o = PyTuple_GetItem($input, j);
void *argp1 = 0 ;
int res1 = SWIG_ConvertPtr(o, &argp1, name, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "Typemap of std::vector" "', argument " "1"" of type '" """'");
}
vec.push_back(reinterpret_cast< type * >(argp1));
}
$1 = vec;
}
else if (SWIG_IsOK(SWIG_ConvertPtr($input, &pobj, name, 0 | 0 ))) {
PyObject *o = $input;
void *argp1 = 0 ;
int res1 = SWIG_ConvertPtr(o, &argp1, name, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "Typemap of std::vector" "', argument " "1"" of type '" """'");
}
vec.push_back(reinterpret_cast< type * >(argp1));
$1 = vec;
}
else {
PyErr_SetString(PyExc_TypeError, "not a list");
return NULL;
}
}
%typecheck(SWIG_TYPECHECK_POINTER) std::vector {
void *pobj = 0;
if(!PyTuple_Check($input) && !SWIG_IsOK(SWIG_ConvertPtr($input, &pobj, name, 0 | 0 ))) {
$1 = 0;
PyErr_Clear();
} else {
$1 = 1;
}
}
%enddef
My sense is that this is standard boilerplate stuff, I don't claim to understand it well as it's someone else's code, but what I do understand now that I did not before is that I needed to place the macro for the typemap before the function that uses the typemap (e.g the "RemoveIsolated" example above). That ordering had been broken when I divided my big module up into smaller ones.

Reusing SWIG mappings in custom typemap

I'm currently working on a Python wrapper for a C++ library for which I want to use SWIG. In my C++ library I have a method with the following signature:
std::vector<SomeClass> getMembers();
Now I know that SWIG has built-in std::vector support but I want to explicitly convert std::vectors to Python Lists (I just think it is cleaner). For that I have the following typemap:
template<typename T>
PyObject* toList(vector<T> vec){
size_t size = vec.size();
PyObject *o = PyList_New(size);
for(size_t i = 0; i < size; i++){
PyList_SetItem(o, i, toPythonInstance<T>(vec[i]));
}
return o;
}
%define OUTPUT_VEC_TO_LIST(type)
%typemap (out) std::vector<type> {
$result = toList<type>($1);
}
%enddef
Now the template method:
template<T>
PyObject* toPythonInstance(T& val){}
Can be specialized to add support for the necessary datatypes. The problem I'm facing now is the following:
SomeClass is wrapped automatically by SWIG. So what I like to do is to reuse this wrapper in my vector type map, i.e. have the following:
template<>
PyObject* toPythonInstance<SomeClass>(SomeClass& val){
//call some SWIG macro to automatically wrap the given instance to
//a Python object
}
Inspecting the code generated by SWIG, I already found the following functions
SWIG_NewPointerObj(...);
SWIG_ConvertPtr(...);
which seem to be responsible for doing exactly what I want. However, I do not want to interfere with any internals of SWIG. So if somebody knows how to achieve what I want with the "public" SWIG interface, I would be very glad!

SWIG actually makes a whole bunch of runtime information part of an external interface, see http://www.swig.org/Doc3.0/Modules.html#Modules_external_run_time for details. This includes the functions you're likely to want for your efforts.
I disagree with your assessment that mapping std::vector to list is cleaner in Python - you always end up copying and visiting every member of that vector to do this. In effect you make a copy of the original container and end up with two containers, so changes to the Python list won't be reflected back on the underlying C++ container. The Python supplied std::vector wrapping should implement the protocols you care about to enable pythonic syntax by default, and can support the ABCs correctly too.. (And if they don't I'm up for writing patches!)