I have a couple of header files that are already defined in C (C++ if we're being technical, but the header is C compatible) that I use to define a bunch of data types (structs). I would like to make use of these in a python script that I am going use to test the corresponding C++ application. This is mostly to avoid having to redefine them in python as some of the structs are unwieldy, but also it would be nice to have them defined in one place so if changes happen down the road it will be easier to adapt.
When I started looking into this I thought that this was certainly doable but none of the examples I have come across get me quite there. The closest I got was using cffi. I got a simple example working how I want it to:
Data types header:
// mylib.h
struct Point2D
{
float x;
float y;
};
struct Point3D
{
float x;
float y;
float z;
};
Python code:
from cffi import FFI
with open("./mylib.h", "r") as fo:
header_text = fo.read()
ffi = FFI()
ffi.cdef(header_text)
point = ffi.new("struct Point2D*")
But this fails if I have #includes or #ifdefs in the header file, per the cffi documentation:
The declarations can contain types, functions, constants and global
variables. What you pass to the cdef() must not contain more than
that; in particular, #ifdef or #include directives are not supported.
Are there any tricks I can do to make this work?
You cannot directly access C structs in Python. You will need to 'bind' C functions to Python functions. This only allows you to access C functions from Python - not a C struct.
Testing C++ is generally done using Google Test. If you require using Python to test C++ functionality then you will need to create bindings in Python to access the C++ functions (as C functions using extern "C").
You can only bind to a C/C++ library. Google "Call C functions in Python" for more.
I'm currently working on a Python wrapper for a C++ library for which I want to use SWIG. In my C++ library I have a method with the following signature:
std::vector<SomeClass> getMembers();
Now I know that SWIG has built-in std::vector support but I want to explicitly convert std::vectors to Python Lists (I just think it is cleaner). For that I have the following typemap:
template<typename T>
PyObject* toList(vector<T> vec){
size_t size = vec.size();
PyObject *o = PyList_New(size);
for(size_t i = 0; i < size; i++){
PyList_SetItem(o, i, toPythonInstance<T>(vec[i]));
}
return o;
}
%define OUTPUT_VEC_TO_LIST(type)
%typemap (out) std::vector<type> {
$result = toList<type>($1);
}
%enddef
Now the template method:
template<T>
PyObject* toPythonInstance(T& val){}
Can be specialized to add support for the necessary datatypes. The problem I'm facing now is the following:
SomeClass is wrapped automatically by SWIG. So what I like to do is to reuse this wrapper in my vector type map, i.e. have the following:
template<>
PyObject* toPythonInstance<SomeClass>(SomeClass& val){
//call some SWIG macro to automatically wrap the given instance to
//a Python object
}
Inspecting the code generated by SWIG, I already found the following functions
SWIG_NewPointerObj(...);
SWIG_ConvertPtr(...);
which seem to be responsible for doing exactly what I want. However, I do not want to interfere with any internals of SWIG. So if somebody knows how to achieve what I want with the "public" SWIG interface, I would be very glad!
SWIG actually makes a whole bunch of runtime information part of an external interface, see http://www.swig.org/Doc3.0/Modules.html#Modules_external_run_time for details. This includes the functions you're likely to want for your efforts.
I disagree with your assessment that mapping std::vector to list is cleaner in Python - you always end up copying and visiting every member of that vector to do this. In effect you make a copy of the original container and end up with two containers, so changes to the Python list won't be reflected back on the underlying C++ container. The Python supplied std::vector wrapping should implement the protocols you care about to enable pythonic syntax by default, and can support the ABCs correctly too.. (And if they don't I'm up for writing patches!)
I have this function is C++:
#include <windows.h>
#include <iphlpapi.h>
#include <stdio.h>
#include <iostream>
#pragma comment(lib, "iphlpapi.lib")
void printInterfaces(){
ULONG buflen = sizeof(IP_ADAPTER_INFO);
IP_ADAPTER_INFO *pAdapterInfo = (IP_ADAPTER_INFO *)malloc(buflen);
if (GetAdaptersInfo(pAdapterInfo, &buflen) == ERROR_BUFFER_OVERFLOW) {
free(pAdapterInfo);
pAdapterInfo = (IP_ADAPTER_INFO *)malloc(buflen);
}
if (GetAdaptersInfo(pAdapterInfo, &buflen) == NO_ERROR) {
for (IP_ADAPTER_INFO *pAdapter = pAdapterInfo; pAdapter; pAdapter = pAdapter->Next) {
if (pAdapter -> IpAddressList.IpAddress.String != "0.0.0.0"){
std::cout << "IP: " <<
pAdapter->IpAddressList.IpAddress.String <<
" Description: " <<
pAdapter-> Description << std::endl;
}
}
}
if (pAdapterInfo) free(pAdapterInfo);
}
I am writing a sniffer with python and I want to get the names of interfaces on windows, so this c++ code prints the IP address and the Description, Is there a way to call this function from python and to make it return the interfaces as a list with tuples? and also I have a problem there when doing != "0.0.0.0" it runs but doesn't filter the interfaces with ip "0.0.0.0". what is the correct way to do it? Also I am more familiar with C#, Is importing C# is easier then C++?
typically the python wrappers, like swig: http://www.swig.org/papers/PyTutorial98/PyTutorial98.pdf or Python.Boost actually wraps the functions.
so a c++ function with type void, will never return a list.
it prints to standardout, so you need to capture that and parse it to generate a python list..
You may want to return something in that c++ function that can actually be interpreted by a python wrapper to a list.
try returning a std::vector and read this: https://wiki.python.org/moin/boost.python/StlContainers
Quoted from Python.Boost:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
You can see another good and great solution depicted with example here.
I have a c++ library containing only .h and .lib files (no cpp files) for communicating with a piece of hardware and I need to use this library from Python. I don't have much experience with c/++, so it's all a little alien to me.
The .h files look something like this:
#define MSG_DLL_VERSION 10
typedef struct {
ULONG ulDLLVersion;
// vipmsg variables
PMSGACCOUNTS pMsgAccounts;
PMSGSEGMENT pMsgSegment;
USHORT usMsgSegmentNum;
} MSGSTATICDATA, *PMSGSTATICDATA;
VOID msgGetDLLRedirections ( PMSGSTATICDATA *pData );
VOID msgSetDLLRedirections ( PMSGSTATICDATA pData );
Looking around, here's what I've found:
Cython is primarily for C. It can do c++, but requires the .cpp files
Boost.Python also requires .cpp files
cffi is C only
ctypes is C only
So, what would be the best approach?
Boost.Python also requires .cpp files
No it does not. You can wrap your library with Boost.Python. The way you expose C++ code using Boost.Python is to write a shared library using the various macros provided by Boost.Python.
The documentation for Boost.Python demonstrates wrapping this C++ type:
struct World
{
void set(std::string msg) { this->msg = msg; }
std::string greet() { return msg; }
std::string msg;
};
The wrapper looks like this:
#include <boost/python.hpp>
using namespace boost::python;
BOOST_PYTHON_MODULE(hello)
{
class_<World>("World")
.def("greet", &World::greet)
.def("set", &World::set)
;
}
You can do that with your library. You do need to write the wrapper. But the fact that the class and methods being wrapped are defined in a .lib library file rather than a .cpp source file is irrelevant.
Update
Looking at the sample code from the header file this seems much more like a C style library than C++. You could certainly use Boost.Python for that. SWIG would also be an option. Or ctypes.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
The community reviewed whether to reopen this question last year and left it closed:
Original close reason(s) were not resolved
Improve this question
What would be the quickest way to construct a Python binding to a C or C++ library?
(I am using Windows if this matters.)
ctypes module is part of the standard library, and therefore is more stable and widely available than swig, which always tended to give me problems.
With ctypes, you need to satisfy any compile time dependency on python, and your binding will work on any python that has ctypes, not just the one it was compiled against.
Suppose you have a simple C++ example class you want to talk to in a file called foo.cpp:
#include <iostream>
class Foo{
public:
void bar(){
std::cout << "Hello" << std::endl;
}
};
Since ctypes can only talk to C functions, you need to provide those declaring them as extern "C"
extern "C" {
Foo* Foo_new(){ return new Foo(); }
void Foo_bar(Foo* foo){ foo->bar(); }
}
Next you have to compile this to a shared library
g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
And finally you have to write your python wrapper (e.g. in fooWrapper.py)
from ctypes import cdll
lib = cdll.LoadLibrary('./libfoo.so')
class Foo(object):
def __init__(self):
self.obj = lib.Foo_new()
def bar(self):
lib.Foo_bar(self.obj)
Once you have that you can call it like
f = Foo()
f.bar() #and you will see "Hello" on the screen
You should have a look at Boost.Python. Here is the short introduction taken from their website:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
There is also pybind11, which is like a lightweight version of Boost.Python and compatible with all modern C++ compilers:
https://pybind11.readthedocs.io/en/latest/
The quickest way to do this is using SWIG.
Example from SWIG tutorial:
/* File : example.c */
int fact(int n) {
if (n <= 1) return 1;
else return n*fact(n-1);
}
Interface file:
/* example.i */
%module example
%{
/* Put header files here or function declarations like below */
extern int fact(int n);
%}
extern int fact(int n);
Building a Python module on Unix:
swig -python example.i
gcc -fPIC -c example.c example_wrap.c -I/usr/local/include/python2.7
gcc -shared example.o example_wrap.o -o _example.so
Usage:
>>> import example
>>> example.fact(5)
120
Note that you have to have python-dev. Also in some systems python header files will be in /usr/include/python2.7 based on the way you have installed it.
From the tutorial:
SWIG is a fairly complete C++ compiler with support for nearly every language feature. This includes preprocessing, pointers, classes, inheritance, and even C++ templates. SWIG can also be used to package structures and classes into proxy classes in the target language — exposing the underlying functionality in a very natural manner.
I started my journey in the Python <-> C++ binding from this page, with the objective of linking high level data types (multidimensional STL vectors with Python lists) :-)
Having tried the solutions based on both ctypes and boost.python (and not being a software engineer) I have found them complex when high level datatypes binding is required, while I have found SWIG much more simple for such cases.
This example uses therefore SWIG, and it has been tested in Linux (but SWIG is available and is widely used in Windows too).
The objective is to make a C++ function available to Python that takes a matrix in form of a 2D STL vector and returns an average of each row (as a 1D STL vector).
The code in C++ ("code.cpp") is as follow:
#include <vector>
#include "code.h"
using namespace std;
vector<double> average (vector< vector<double> > i_matrix) {
// Compute average of each row..
vector <double> averages;
for (int r = 0; r < i_matrix.size(); r++){
double rsum = 0.0;
double ncols= i_matrix[r].size();
for (int c = 0; c< i_matrix[r].size(); c++){
rsum += i_matrix[r][c];
}
averages.push_back(rsum/ncols);
}
return averages;
}
The equivalent header ("code.h") is:
#ifndef _code
#define _code
#include <vector>
std::vector<double> average (std::vector< std::vector<double> > i_matrix);
#endif
We first compile the C++ code to create an object file:
g++ -c -fPIC code.cpp
We then define a SWIG interface definition file ("code.i") for our C++ functions.
%module code
%{
#include "code.h"
%}
%include "std_vector.i"
namespace std {
/* On a side note, the names VecDouble and VecVecdouble can be changed, but the order of first the inner vector matters! */
%template(VecDouble) vector<double>;
%template(VecVecdouble) vector< vector<double> >;
}
%include "code.h"
Using SWIG, we generate a C++ interface source code from the SWIG interface definition file..
swig -c++ -python code.i
We finally compile the generated C++ interface source file and link everything together to generate a shared library that is directly importable by Python (the "_" matters):
g++ -c -fPIC code_wrap.cxx -I/usr/include/python2.7 -I/usr/lib/python2.7
g++ -shared -Wl,-soname,_code.so -o _code.so code.o code_wrap.o
We can now use the function in Python scripts:
#!/usr/bin/env python
import code
a= [[3,5,7],[8,10,12]]
print a
b = code.average(a)
print "Assignment done"
print a
print b
For modern C++, use cppyy:
http://cppyy.readthedocs.io/en/latest/
It's based on Cling, the C++ interpreter for Clang/LLVM. Bindings are at run-time and no additional intermediate language is necessary. Thanks to Clang, it supports C++17.
Install it using pip:
$ pip install cppyy
For small projects, simply load the relevant library and the headers that you are interested in. E.g. take the code from the ctypes example is this thread, but split in header and code sections:
$ cat foo.h
class Foo {
public:
void bar();
};
$ cat foo.cpp
#include "foo.h"
#include <iostream>
void Foo::bar() { std::cout << "Hello" << std::endl; }
Compile it:
$ g++ -c -fPIC foo.cpp -o foo.o
$ g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
and use it:
$ python
>>> import cppyy
>>> cppyy.include("foo.h")
>>> cppyy.load_library("foo")
>>> from cppyy.gbl import Foo
>>> f = Foo()
>>> f.bar()
Hello
>>>
Large projects are supported with auto-loading of prepared reflection information and the cmake fragments to create them, so that users of installed packages can simply run:
$ python
>>> import cppyy
>>> f = cppyy.gbl.Foo()
>>> f.bar()
Hello
>>>
Thanks to LLVM, advanced features are possible, such as automatic template instantiation. To continue the example:
>>> v = cppyy.gbl.std.vector[cppyy.gbl.Foo]()
>>> v.push_back(f)
>>> len(v)
1
>>> v[0].bar()
Hello
>>>
Note: I'm the author of cppyy.
pybind11 minimal runnable example
pybind11 was previously mentioned at https://stackoverflow.com/a/38542539/895245 but I would like to give here a concrete usage example and some further discussion about implementation.
All and all, I highly recommend pybind11 because it is really easy to use: you just include a header and then pybind11 uses template magic to inspect the C++ class you want to expose to Python and does that transparently.
The downside of this template magic is that it slows down compilation immediately adding a few seconds to any file that uses pybind11, see for example the investigation done on this issue. PyTorch agrees. A proposal to remediate this problem has been made at: https://github.com/pybind/pybind11/pull/2445
Here is a minimal runnable example to give you a feel of how awesome pybind11 is:
class_test.cpp
#include <string>
#include <pybind11/pybind11.h>
struct ClassTest {
ClassTest(const std::string &name, int i) : name(name), i(i) { }
void setName(const std::string &name_) { name = name_; }
const std::string getName() const { return name + "z"; }
void setI(const int i) { this->i = i; }
const int getI() const { return i + 1; }
std::string name;
int i;
};
namespace py = pybind11;
PYBIND11_PLUGIN(class_test) {
py::module m("my_module", "pybind11 example plugin");
py::class_<ClassTest>(m, "ClassTest")
.def(py::init<const std::string &, int>())
.def("setName", &ClassTest::setName)
.def("getName", &ClassTest::getName)
.def_readwrite("name", &ClassTest::name)
.def("setI", &ClassTest::setI)
.def("getI", &ClassTest::getI)
.def_readwrite("i", &ClassTest::i);
return m.ptr();
}
class_test_main.py
#!/usr/bin/env python3
import class_test
my_class_test = class_test.ClassTest("abc", 1);
print(my_class_test.getName())
print(my_class_test.getI())
my_class_test.setName("012")
my_class_test.setI(2)
print(my_class_test.getName())
print(my_class_test.getI())
assert(my_class_test.getName() == "012z")
assert(my_class_test.getI() == 3)
Compile and run:
#!/usr/bin/env bash
set -eux
sudo apt install pybind11-dev
g++ `python3-config --cflags` -shared -std=c++11 -fPIC class_test.cpp \
-o class_test`python3-config --extension-suffix` `python3-config --libs`
./class_test_main.py
Stdout output:
abcz
2
012z
3
If we tried to use a wrong type as in:
my_class_test.setI("abc")
it blows up as expected:
Traceback (most recent call last):
File "/home/ciro/test/./class_test_main.py", line 9, in <module>
my_class_test.setI("abc")
TypeError: setI(): incompatible function arguments. The following argument types are supported:
1. (self: my_module.ClassTest, arg0: int) -> None
Invoked with: <my_module.ClassTest object at 0x7f2980254fb0>, 'abc'
This example shows how pybind11 allows you to effortlessly expose the ClassTest C++ class to Python!
Notably, Pybind11 automatically understands from the C++ code that name is an std::string, and therefore should be mapped to a Python str object.
Compilation produces a file named class_test.cpython-36m-x86_64-linux-gnu.so which class_test_main.py automatically picks up as the definition point for the class_test natively defined module.
Perhaps the realization of how awesome this is only sinks in if you try to do the same thing by hand with the native Python API, see for example this example of doing that, which has about 10x more code: https://github.com/cirosantilli/python-cheat/blob/4f676f62e87810582ad53b2fb426b74eae52aad5/py_from_c/pure.c On that example you can see how the C code has to painfully and explicitly define the Python class bit by bit with all the information it contains (members, methods, further metadata...). See also:
Can python-C++ extension get a C++ object and call its member function?
Exposing a C++ class instance to a python embedded interpreter
A full and minimal example for a class (not method) with Python C Extension?
Embedding Python in C++ and calling methods from the C++ code with Boost.Python
Inheritance in Python C++ extension
pybind11 claims to be similar to Boost.Python which was mentioned at https://stackoverflow.com/a/145436/895245 but more minimal because it is freed from the bloat of being inside the Boost project:
pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent Boost.Python library by David Abrahams: to minimize boilerplate code in traditional extension modules by inferring type information using compile-time introspection.
The main issue with Boost.Python—and the reason for creating such a similar project—is Boost. Boost is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available, this heavy machinery has become an excessively large and unnecessary dependency.
Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn't relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depend on Python (2.7 or 3.x, or PyPy2.7 >= 5.7) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (specifically: tuples, lambda functions and variadic templates). Since its creation, this library has grown beyond Boost.Python in many ways, leading to dramatically simpler binding code in many common situations.
pybind11 is also the only non-native alternative hightlighted by the current Microsoft Python C binding documentation at: https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2019 (archive).
Tested on Ubuntu 18.04, pybind11 2.0.1, Python 3.6.8, GCC 7.4.0.
I think cffi for python can be an option.
The goal is to call C code from Python. You should be able to do so
without learning a 3rd language: every alternative requires you to
learn their own language (Cython, SWIG) or API (ctypes). So we tried
to assume that you know Python and C and minimize the extra bits of
API that you need to learn.
http://cffi.readthedocs.org/en/release-0.7/
I love cppyy, it makes it very easy to extend Python with C++ code, dramatically increasing performance when needed.
It is powerful and frankly very simple to use,
here it is an example of how you can create a numpy array and pass it to a class member function in C++.
cppyy_test.py
import cppyy
import numpy as np
cppyy.include('Buffer.h')
s = cppyy.gbl.Buffer()
numpy_array = np.empty(32000, np.float64)
s.get_numpy_array(numpy_array.data, numpy_array.size)
print(numpy_array[:20])
Buffer.h
struct Buffer {
void get_numpy_array(double *ad, int size) {
for( long i=0; i < size; i++)
ad[i]=i;
}
};
You can also create a Python module very easily (with CMake), this way you will avoid recompile the C++ code all the times.
The question is how to call a C function from Python, if I understood correctly. Then the best bet are Ctypes (BTW portable across all variants of Python).
>>> from ctypes import *
>>> libc = cdll.msvcrt
>>> print libc.time(None)
1438069008
>>> printf = libc.printf
>>> printf("Hello, %s\n", "World!")
Hello, World!
14
>>> printf("%d bottles of beer\n", 42)
42 bottles of beer
19
For a detailed guide you may want to refer to my blog article.
Cython is definitely the way to go, unless you anticipate writing Java wrappers, in which case SWIG may be preferable.
I recommend using the runcython command line utility, it makes the process of using Cython extremely easy. If you need to pass structured data to C++, take a look at Google's protobuf library, it's very convenient.
Here is a minimal examples I made that uses both tools:
https://github.com/nicodjimenez/python2cpp
Hope it can be a useful starting point.
First you should decide what is your particular purpose. The official Python documentation on extending and embedding the Python interpreter was mentioned above, I can add a good overview of binary extensions. The use cases can be divided into 3 categories:
accelerator modules: to run faster than the equivalent pure Python code runs in CPython.
wrapper modules: to expose existing C interfaces to Python code.
low level system access: to access lower level features of the CPython runtime, the operating system, or the underlying hardware.
In order to give some broader perspective for other interested and since your initial question is a bit vague ("to a C or C++ library") I think this information might be interesting to you. On the link above you can read on disadvantages of using binary extensions and its alternatives.
Apart from the other answers suggested, if you want an accelerator module, you can try Numba. It works "by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool)".