Develop Reference

Return a python function from Python C-API

I am currently in the process of writing a small python module using the Python API, that will speed up some of the slower python code, that is repeatedly run in a in a simulation of sorts. My issue is that currently this code is takes a bunch of arguments, that in many use cases won't change. For example the function signature will be like: func(x,a,b,c,d,e), but after an initialisation only x will change. I therefore will have the python code littered with lambda x : func(x,a,b,c,d,e) where I wrap these before use. I have observed that this actually introduces quite a bit of calling overhead.
My idea to fix this was to create a PyObject* that is essentially C++ lambda instead of the python one. The main issue with this is that I have not found a way to create PyObjects from C++ lambdas, or even lower level functions. Since functions/lambdas in python can be passed as arguments I assume it is possible, but is there a clean way I'm missing.

I would seriously consider using swig instead of pybind11 for example. It's just peace of mind. If you don't want to use swig directly, you can at least see what swig does to wrap up features like proxy objects.
http://www.swig.org/Doc2.0/SWIGPlus.html#SWIGPlus_nn38

python embedding in C++ with functions returning shared_ptr(pybind11/boost_python)

I am currently working on a usecase where I have implemented APIs in python and they need to return struct, vector and map to C++ caller functions. Could you please help me with an example how to do that. All the usecases I see in pybind/boost python is just simply python being embedded in C++ without any return value.
I came across your blog as well that explained the two libraries pretty well. But I could not proceed because of this usecase.
The scenario which I need to implement is a python -> C++ interface where the C++ caller will start a session and call various python objects when it wants and then close the session. I am ok with solution both in boost::python or pybind11

"All the usecases I see in pybind/boost python is just simply python being embedded in C++ without any return value."
PyBind11 Example with return value
The example the called python function returns an int which they assign first to a py::object and then cast to an int. For STL containers like std::map, std::vector there is automatic conversion if you #include <pybind11/stl.h>. I am unsure about conversion to struct, but I would use one of PyBind's built in STL conversions between Python and C++ and then construct the struct on the C++ side.

Python support for Qt dll

I have my own C++ library project(with source) written in Qt and it uses QTcpsocket, QUdpSocket, QSerialPort signals and slots.
I would like to support this library in Python as well.
What is the preferred way to do this?
Writing a wrapper in Python, if so does it have obstacles?
Dont know if PyQt is just for this purpose?
Or do you thnink is it better to rewrite the lib in Python by just implementing the logic used in C++ library project.
As this is library is part of a SDK, same applies for supporting QT dll with .NET as well in fact, as a second step after supporting Python.
Example API of Qt.
quint16 SendCommandAsync(CmdBaseSpv1* pcommand,
ConnectionArg connectionArg,
emitLogOptions::emitLogOption logOption,
LogInfo &txLogInfo,
LogInfo &rxLogInfo);
I want to call this function from Python.
Function parameters CmdBaseSpv1, ConnectionArg, emitLogOption, LogInfo are all Qt classes.
Some of these arguments are using the QOBJECT base class.
As you see from the function name; it is an Asynchronous function call. Result will emit a signal so I need to get async result as well.

I'll write down what I know about wrapping C++ libraries and will try to source it, but as a huge disclaimer, I have only used this for something very, very simple myself.
You ask about rewriting the library in Python. I would say this depends. If the code is trivial, then I don't see why not. If it is larger and has to be kept up-to-date with other code (as you imply with .Net), I wouldn't. It makes sense to reuse the same code for both.
 My suggestion
From what I see of your code I would try to wrap it using boost::python or SWIG.
How to wrap
The main trouble is going to be to create CmdBaseSpv1, ConnectionArg, etc. in Python.
If you don't need any Qt-classes to instantiate your classes, this should be straightforward. However, in case you need the Qt types inside of Python (e.g. because the constructor of CmdBaseSpv1 requires a QString), your task is a lot more complicated because you need a way to convert a Python-string into a QString. If you can, you should only use stl-types.
Everything in Python
The simplest way to wrap a small C library is to use the cffi module (or ctypes). You can write the full binding in Python. However, this is a lot of manual work if your API is large and can get difficult.
There is another problem: ctypes is only compatible with C, not C++. So you'd need to change your interface to be compatible with C, internally you could still use C++ and Qt.
Wrap by hand
An alternative is to wrap the library calls yourself. You can either do this by using the Python API. There are also a few libraries that help you create the bindings. Boost::python seems especially promising and works with C++.
Binding generators
If your API is very large, you should use a binding generator which parses the C++ code and generates the bindings itself. For example sip is one of them. It is used to create the bindings for the whole Qt library. There are a few binding generators out there, one mentioned in the Python docs is SWIG. PySide uses Shiboken and also has a nice description of it on their website.
SWIG has the additional advantage, that you can create bindings for multiple languages, including C#.
PyQt
PyQt is a binding generated from Qt using sip. You'll probably not need it, unless you need to access the full power of Qt from inside Python. If this is the case, consider using sipfor generating the bindings, so things like the signal-slot mechanism are compatible between your library and PyQt.
Challenges with bindings
Bindings come with a few challenges because Python and C++ are different in some key areas.
Memory-management
Memory management in Python is almost automatic, in C++ you're required to do it manually. For example
def myfunc():
mywidget = QWidget()
at the end of myfunc() mywidget gets garbage collected. In C++ however
void myfunc() {
auto mywidget = new QWidget();
}
mywidget is still around. This means that even when inside Python, you need to take care of the C++ memory management. The problems I've seen are memory leaks and dangling pointers. Watch out for this when using callbacks, you don't want Python to garbage collect the callback while C++ thinks it's still alive.
Exceptions
Not all programming languages have exceptions or deal with them the same way. For example, it would be nice if an exception inside C++ can be caught inside Python.
Links to related question
How to wrap a c++ library for python? (example of boost::python)
Exposing a C++ API to Python (discussion about boost::python, SWIG and more)
https://stackoverflow.com/a/5686873 (discusses Cython, another choice)

Is there a way to generate a c++ class from a python class and bind it a compile time?

Is there a way to generate a relatively clean c++ class from a python class and bind it at compile-time?
For instance, if I have this python class:
class CarDef:
acceleration = 1000.0
brake = 1500.0
inertia = acceleration * 0.1 * brake
def __init__(self):
pass
I'd like to have the corresponding c++ class:
class CarDef
{
public:
double acceleration;
double brake;
double inertia;
CarDef()
: acceleration( 1000.0 )
, brake( 1500.0 )
, inertia ( 150000.0 )
{};
};
The resulting c++ class could be different, as well as the original python class: I could use a "getter methods" paradigm instead of class attributes.
What I'm trying to achieve is to create resource files in python that I'll be able to use in my c++ application. The goal is to reduce as much as possible the amount of code the end-user will have to write to add and use parameters; and it must avoid string comparison during the "running phase" (it's allowed during the "initialization phase").
I'd like the user to have to enter the resource name only twice: once in the python class, and once in the place where the resource will be used in the c++, assuming that the "magic" is going to bind the two items (either at run-time (which I doubt could be done without string comparison) or at compile time (an in-between step generates c++ class before the project is compiled)). This is why I'm going from python to c++; I believe that going from c++ to python would require at least 2 python files: one that is generated and one that inherits from the latter (to avoid overwriting already specified resources).
The end-user use would look like this:
// A singleton definition manager
class DefManager
{
CarDef mCarDef;
public:
static DefManager& GetReference()
{
static DefManager instance;
return instance;
}
CarDef& getCarDef() { return mCarDef; }
};
// One would use the generated CarDef class like this:
mCar.setSpeed( mCar.getSpeed() + DefManager.GetReference().getCarDef().acceleration );
With this in mind, the python code is strictly outside of the c++ code.
One obvious problem I see is how to know what type a python attribute or method returns. I've seen a bit of examples of Cython, and it's seems to be able to use types (which is great!), but I haven't seen any examples where it could do what I need. Also, c generated code seems to still need Python.h and thus the cpython api libraries when compiling.
Is there any ways I could achieve this? Are there better way to do it?
I'm using python 3.2+.
I'm using MS Visual Studio 2010 (we plan to upgrade to 2013 soon).
I'm on a 32 bit system (but we plan to upgrade to 64 bit soon, OS and developed software).

There is a way to go from C++ to Python but I do not know of any way of going from Python to C++. If you don't mind writing your code in C++ first, you can use the tool SWIG to auto generated for you Python classes.
Do note there are a few limitations around exception handling. You can set up to have your Python code throw C++ exceptions but the type of exception can be lost in translation. You also need to pay attention to handling of reference counted objects. SWIG will generate reference counting for Python which can sometimes delete objects unexpectedly.
If you don't like using a tool such as SWIG, there is also Boost.Python for C++. Again, this is C++ for Python bindings and does not auto generate C++ from Python.

You could embed python in your C++ code or vice versa. There are tons of helper functions, though a little ugly, can be very powerful and might be able to accomplish what you want, though I'm not sure I'm entirely understanding your question. This doesn't require the cython api, but does still require Python.h.

There is a logical problem with doing what you ask.
Python is weakly typed.
In python one can even change the type of a certain data member during run time.
So say you have two objects of type
CarDef
Lets call them obj1 and obj2.
Lets say you have a setter:
setIntX(self):
self.x = 5
and lets say you also have a setter:
setStringX(self):
self.x = "5"
Then what type will member x have in your C++ class?
This can only be decided during run time, and more than one C++ class might be necessary to model one python class.
However a template class from python might be possible, and quite interesting actually.
Also maybe a general solution is not possible, but if you assume no member have ambiguous type it is possible.

How should I embed Python in a C++ Builder / Delphi 2010 application?

I'm interested in experimenting with embedding Python in my application, to let the user run Python scripts within the application environment, accessing internal (C++-implemented) objects, etc. I'm quite new to this so don't know exactly what I'm doing.
I have read Embedding Python in Another Application, though this seems to talk only about a C API and flat C functions, not classes or objects (unless I've missed something) and its "Embedding Python in C++" section is only two sentences long. However, I also came across how to use boost::python and this looks excellent.
There's one problem: boost::python is not supported by C++ Builder 2010.
So, given this, what is the best approach for embedding Python in a C++ application compiled with C++ Builder 2010, and, using whichever technique is best, how do you expose / integrate classes and objects to give the Python coder access to the object-oriented internals of a program? Have I missed a standard approach? Is exposing internal classes or instantiated objects to Python as objects easy, or is the API truly C-style or flat / non-OO, and if so what's the best approach to mimic an underlying OO layer through such an API?
Note: I actually use RAD Studio, which includes both C++ Builder and Delphi. It may be possible to make use of some sort of Delphi-specific binding, but the ones I've encountered are six or seven years old, or are new-ish (Python 2.6) but don't seem to have any documentation and have comments in the issue list like "Anyone reads thiese [sic] comments anyway? Anyone working on this project?" which is not encouraging. But please feel free to include Delphi-specific answers especially if you think it's likely they'll work in a combined D+CB app. I appreciate all answers even if they aren't quite perfect - I can research, I just need pointers on where to go. A native C++ solution would probably be ideal, though, since using VCL-derived objects has its own limitations.
Thanks for your input!

You should not be afraid of the P4D project at google groups. It seems inactive because, in part, it is very stable and full-featured already. Those components are used in the much more active PyScripter application which is one of the best python development editors currently available. PyScripter is written in Delphi and uses the P4D components. As such, it also presents a very comprehensive example of how to use the P4D components, although the examples provided with the P4D source checkout are already good enough to get started.

Is exposing internal classes or
instantiated objects to Python as
objects easy, or is the API truly
C-style or flat / non-OO, and if so
what's the best approach to mimic an
underlying OO layer through such an
API?
You have already answered yourself. The latter part of the sentence is correct.
Objects and classes do not exist in C++ as soon as you compile, only a few structures (vtables), and also another ones explaining some OO data, provided that RTTI is activated. That's why it is not possible to bridge the gap between Python and C++ using classes and objects.
You can build that surely by yourself, creating a set of C functions along with some data structures, and then an OO-layer. But you cannot do that out of the box.
For instance, class Car:
class Car {
public:
int getDoors()
{ return this->doors; }
protected:
int doors;
};
Is translated to:
struct Car {
int doors;
};
int Car_getDoors(Car * this)
{
return this->doors;
}
And a call to getDoors:
c->getDoors()
Is translated as:
Car_getDoors( c )

You can generate C++ to $SCRIPTLANG wrappers with swig.