How should I embed Python in a C++ Builder / Delphi 2010 application? - python

I'm interested in experimenting with embedding Python in my application, to let the user run Python scripts within the application environment, accessing internal (C++-implemented) objects, etc. I'm quite new to this so don't know exactly what I'm doing.
I have read Embedding Python in Another Application, though this seems to talk only about a C API and flat C functions, not classes or objects (unless I've missed something) and its "Embedding Python in C++" section is only two sentences long. However, I also came across how to use boost::python and this looks excellent.
There's one problem: boost::python is not supported by C++ Builder 2010.
So, given this, what is the best approach for embedding Python in a C++ application compiled with C++ Builder 2010, and, using whichever technique is best, how do you expose / integrate classes and objects to give the Python coder access to the object-oriented internals of a program? Have I missed a standard approach? Is exposing internal classes or instantiated objects to Python as objects easy, or is the API truly C-style or flat / non-OO, and if so what's the best approach to mimic an underlying OO layer through such an API?
Note: I actually use RAD Studio, which includes both C++ Builder and Delphi. It may be possible to make use of some sort of Delphi-specific binding, but the ones I've encountered are six or seven years old, or are new-ish (Python 2.6) but don't seem to have any documentation and have comments in the issue list like "Anyone reads thiese [sic] comments anyway? Anyone working on this project?" which is not encouraging. But please feel free to include Delphi-specific answers especially if you think it's likely they'll work in a combined D+CB app. I appreciate all answers even if they aren't quite perfect - I can research, I just need pointers on where to go. A native C++ solution would probably be ideal, though, since using VCL-derived objects has its own limitations.
Thanks for your input!

You should not be afraid of the P4D project at google groups. It seems inactive because, in part, it is very stable and full-featured already. Those components are used in the much more active PyScripter application which is one of the best python development editors currently available. PyScripter is written in Delphi and uses the P4D components. As such, it also presents a very comprehensive example of how to use the P4D components, although the examples provided with the P4D source checkout are already good enough to get started.

Is exposing internal classes or
instantiated objects to Python as
objects easy, or is the API truly
C-style or flat / non-OO, and if so
what's the best approach to mimic an
underlying OO layer through such an
API?
You have already answered yourself. The latter part of the sentence is correct.
Objects and classes do not exist in C++ as soon as you compile, only a few structures (vtables), and also another ones explaining some OO data, provided that RTTI is activated. That's why it is not possible to bridge the gap between Python and C++ using classes and objects.
You can build that surely by yourself, creating a set of C functions along with some data structures, and then an OO-layer. But you cannot do that out of the box.
For instance, class Car:
class Car {
public:
int getDoors()
{ return this->doors; }
protected:
int doors;
};
Is translated to:
struct Car {
int doors;
};
int Car_getDoors(Car * this)
{
return this->doors;
}
And a call to getDoors:
c->getDoors()
Is translated as:
Car_getDoors( c )

You can generate C++ to $SCRIPTLANG wrappers with swig.

Related

How to convert a Python deque to a C++ container?

I have to pass a Python deque to a C++ function but I can't find any documentation on how to convert a deque to a C++ container (for this case, either a deque or a vector would do). Searched everywhere but I couldn't find a solution. There's gotta be a simple way to do this right?
Thanks
There's no "simple way". You'll need to do that by hand (or maybe with some kind of interface generator), in a data-specific way.
Take a look at C API Introduction — Python 3.8.3 documentation for an overview of Python data model at C level. It has absolutely nothing to do with any C++ types.
You'll need to somehow extract the data you need from Python objects, via Python C API, and construct the C++ structure you need from that.
SWIG may help you here. Take a look at SWIG-4.0 Documentation -- it has a section about how to do conversion between C++ types and those from a scripting language.

Wrapping C++ class to use in Python

I have a device which can be controlled through a C++ class (https://github.com/stanleyseow/RF24/tree/master/RPi/RF24).
I'd like to be able to use this class in Python, and thought I could wrap it.
I found many ways to do it, but not much detailed documentation with examples. In particular, I found Boost, Cython, SWIG and the native C Python API.
Which one is the best method in which case ? And do you have some links to detailed documentations / examples about this ?
Thanks !
There is no "best"; it depends entirely on your circumstances.
For a single class, the native C Python API isn't too difficult,
but you do have to create an entire module, then the class. It
would be simpler if you exposed a procedural interface, rather
than a class. If you only have one instance of the device, this
would be an appropriate solution.
SWIG is very good for taking C++ class definitions and
generating a Python module which contains them. The resulting
code is relatively complex, since SWIG tries to cover all
possible versions of Python; for anything 2.7 or later (and
perhaps a little earlier), you can do everything directly in
C++, without any intermediate Python.
Boost makes extensive use of templates. This isn't really an
appropriate solution for the problem; it adds a lot of
complexity for something that is relatively simple if done with
external tools, rather than metaprogramming. Still, if the
underlying complexity doesn't scare you, it might not be that
hard to use.
I'm not familiar with Cython.
Globally, if all you have is one instance of one simple class,
using the native C API is probably no more difficult than the
other solutions, and introduces a minimum of added internal
complexity.

Is there a way to generate a c++ class from a python class and bind it a compile time?

Is there a way to generate a relatively clean c++ class from a python class and bind it at compile-time?
For instance, if I have this python class:
class CarDef:
acceleration = 1000.0
brake = 1500.0
inertia = acceleration * 0.1 * brake
def __init__(self):
pass
I'd like to have the corresponding c++ class:
class CarDef
{
public:
double acceleration;
double brake;
double inertia;
CarDef()
: acceleration( 1000.0 )
, brake( 1500.0 )
, inertia ( 150000.0 )
{};
};
The resulting c++ class could be different, as well as the original python class: I could use a "getter methods" paradigm instead of class attributes.
What I'm trying to achieve is to create resource files in python that I'll be able to use in my c++ application. The goal is to reduce as much as possible the amount of code the end-user will have to write to add and use parameters; and it must avoid string comparison during the "running phase" (it's allowed during the "initialization phase").
I'd like the user to have to enter the resource name only twice: once in the python class, and once in the place where the resource will be used in the c++, assuming that the "magic" is going to bind the two items (either at run-time (which I doubt could be done without string comparison) or at compile time (an in-between step generates c++ class before the project is compiled)). This is why I'm going from python to c++; I believe that going from c++ to python would require at least 2 python files: one that is generated and one that inherits from the latter (to avoid overwriting already specified resources).
The end-user use would look like this:
// A singleton definition manager
class DefManager
{
CarDef mCarDef;
public:
static DefManager& GetReference()
{
static DefManager instance;
return instance;
}
CarDef& getCarDef() { return mCarDef; }
};
// One would use the generated CarDef class like this:
mCar.setSpeed( mCar.getSpeed() + DefManager.GetReference().getCarDef().acceleration );
With this in mind, the python code is strictly outside of the c++ code.
One obvious problem I see is how to know what type a python attribute or method returns. I've seen a bit of examples of Cython, and it's seems to be able to use types (which is great!), but I haven't seen any examples where it could do what I need. Also, c generated code seems to still need Python.h and thus the cpython api libraries when compiling.
Is there any ways I could achieve this? Are there better way to do it?
I'm using python 3.2+.
I'm using MS Visual Studio 2010 (we plan to upgrade to 2013 soon).
I'm on a 32 bit system (but we plan to upgrade to 64 bit soon, OS and developed software).
There is a way to go from C++ to Python but I do not know of any way of going from Python to C++. If you don't mind writing your code in C++ first, you can use the tool SWIG to auto generated for you Python classes.
Do note there are a few limitations around exception handling. You can set up to have your Python code throw C++ exceptions but the type of exception can be lost in translation. You also need to pay attention to handling of reference counted objects. SWIG will generate reference counting for Python which can sometimes delete objects unexpectedly.
If you don't like using a tool such as SWIG, there is also Boost.Python for C++. Again, this is C++ for Python bindings and does not auto generate C++ from Python.
You could embed python in your C++ code or vice versa. There are tons of helper functions, though a little ugly, can be very powerful and might be able to accomplish what you want, though I'm not sure I'm entirely understanding your question. This doesn't require the cython api, but does still require Python.h.
There is a logical problem with doing what you ask.
Python is weakly typed.
In python one can even change the type of a certain data member during run time.
So say you have two objects of type
CarDef
Lets call them obj1 and obj2.
Lets say you have a setter:
setIntX(self):
self.x = 5
and lets say you also have a setter:
setStringX(self):
self.x = "5"
Then what type will member x have in your C++ class?
This can only be decided during run time, and more than one C++ class might be necessary to model one python class.
However a template class from python might be possible, and quite interesting actually.
Also maybe a general solution is not possible, but if you assume no member have ambiguous type it is possible.

Interfacing C/C++ libraries with Python

I have a C++ library that I need to be able to interface with python. I read this question to understand the choice I need to adapt.
I saw SWIG and Cython and wanted to go with SWIG, mainly because my python programming experience is very minimal. However, I realise that with Swig I have to write an interface (.i extensions) for every class. Now, my C++ project is huge and I feel it will take me a lot of time to get the wrappers in place (or maybe I am wrong).
So right now since my application is large I need to make a choice. In the quoted thread I came across Boost Python. Now I can no longer decide and want input from people who can tell me the pros and cons of one over the other. Note my preference is on easy of use and how quickly can it be done. I am willing to compromise system performance for this. I would appreciate immensely if someone could provide me a SWIG implemented project or Boost Python implemented project link (a complete module instead of a sample tutorial would be much better !)
Boost::python provides a nearly wrapper-less interface between C++ and Python. It also allows you to write custom converters and other neat things that make the Python interfaces much nicer. The interfaces are pure C++, but they rely on templates and clever design patterns to make it look all nice and declarative. You also get the benefit of your connector code being checked by the compiler directly.
With Swig, you write interface declarations in Swig's own DSL, which takes a few days to get a hang of. In addition, it always inserts a wrapper layer, so it could be a bit slower. However, it does have the nice feature of automatically converting many things for ya without having to declare anything extra. The wrappers it generates are quite hard to debug though.
IMHO boost::python is the better choice, because you're working pretty directly with CPython's native C interfaces. I use Swig for Java and C++ interaction, because JNI is a bear, Python's C interface is actually quite usable all by itself.
If you already have a bunch of Swig wrappers, I would keep those because you'd have to redo all that work. However, starting a new project, or if you require maximum performance, boost::python all the way!

Python: SWIG vs ctypes

In python, under what circumstances is SWIG a better choice than ctypes for calling entry points in shared libraries? Let's assume you don't already have the SWIG interface file(s). What are the performance metrics of the two?
I have a rich experience of using swig. SWIG claims that it is a rapid solution for wrapping things. But in real life...
Cons:
SWIG is developed to be general, for everyone and for 20+ languages. Generally, it leads to drawbacks:
- needs configuration (SWIG .i templates), sometimes it is tricky,
- lack of treatment of some special cases (see python properties further),
- lack of performance for some languages.
Python cons:
1) Code style inconsistency. C++ and python have very different code styles (that is obvious, certainly), the possibilities of a swig of making target code more Pythonish is very limited. As an example, it is butt-heart to create properties from getters and setters. See this q&a
2) Lack of broad community. SWIG has some good documentation. But if one caught something that is not in the documentation, there is no information at all. No blogs nor googling helps. So one has to heavily dig SWIG generated code in such cases... That is terrible, I could say...
Pros:
In simple cases, it is really rapid, easy and straight forward
If you produced swig interface files once, you can wrap this C++ code to ANY of other 20+ languages (!!!).
One big concern about SWIG is a performance. Since version 2.04 SWIG includes '-builtin' flag which makes SWIG even faster than other automated ways of wrapping. At least some benchmarks shows this.
When to USE SWIG?
So I concluded for myself two cases when the swig is good to use:
2) If one needs to wrap C++ code for several languages. Or if potentially there could be a time when one needs to distribute the code for several languages. Using SWIG is reliable in this case.
1) If one needs to rapidly wrap just several functions from some C++ library for end use.
Live experience
Update :
It is a year and a half passed as we did a conversion of our library by using SWIG.
First, we made a python version. There were several moments when we experienced troubles with SWIG - it is true. But right now we expanded our library to Java and .NET. So we have 3 languages with 1 SWIG. And I could say that SWIG rocks in terms of saving a LOT of time.
Update 2:
It is two years as we use SWIG for this library. SWIG is integrated into our build system. Recently we had major API change of C++ library. SWIG worked perfectly. The only thing we needed to do is to add several %rename to .i files so our CppCamelStyleFunctions() now looks_more_pythonish in python. First I was concerned about some problems that could arise, but nothing went wrong. It was amazing. Just several edits and everything distributed in 3 languages. Now I am confident that it was a good solution to use SWIG in our case.
Update 3:
It is 3+ years we use SWIG for our library. Major change: python part was totally rewritten in pure python. The reason is that Python is used for the majority of applications of our library now. Even if the pure python version works slower than C++ wrapping, it is more convenient for users to work with pure python, not struggling with native libraries.
SWIG is still used for .NET and Java versions.
The Main question here "Would we use SWIG for python if we started the project from the beginning?". We would! SWIG allowed us to rapidly distribute our product to many languages. It worked for a period of time which gave us the opportunity for better understanding our users requirements.
SWIG generates (rather ugly) C or C++ code. It is straightforward to use for simple functions (things that can be translated directly) and reasonably easy to use for more complex functions (such as functions with output parameters that need an extra translation step to represent in Python.) For more powerful interfacing you often need to write bits of C as part of the interface file. For anything but simple use you will need to know about CPython and how it represents objects -- not hard, but something to keep in mind.
ctypes allows you to directly access C functions, structures and other data, and load arbitrary shared libraries. You do not need to write any C for this, but you do need to understand how C works. It is, you could argue, the flip side of SWIG: it doesn't generate code and it doesn't require a compiler at runtime, but for anything but simple use it does require that you understand how things like C datatypes, casting, memory management and alignment work. You also need to manually or automatically translate C structs, unions and arrays into the equivalent ctypes datastructure, including the right memory layout.
It is likely that in pure execution, SWIG is faster than ctypes -- because the management around the actual work is done in C at compiletime rather than in Python at runtime. However, unless you interface a lot of different C functions but each only a few times, it's unlikely the overhead will be really noticeable.
In development time, ctypes has a much lower startup cost: you don't have to learn about interface files, you don't have to generate .c files and compile them, you don't have to check out and silence warnings. You can just jump in and start using a single C function with minimal effort, then expand it to more. And you get to test and try things out directly in the Python interpreter. Wrapping lots of code is somewhat tedious, although there are attempts to make that simpler (like ctypes-configure.)
SWIG, on the other hand, can be used to generate wrappers for multiple languages (barring language-specific details that need filling in, like the custom C code I mentioned above.) When wrapping lots and lots of code that SWIG can handle with little help, the code generation can also be a lot simpler to set up than the ctypes equivalents.
CTypes is very cool and much easier than SWIG, but it has the drawback that poorly or malevolently-written python code can actually crash the python process. You should also consider boost python. IMHO it's actually easier than swig while giving you more control over the final python interface. If you are using C++ anyway, you also don't add any other languages to your mix.
In my experience, ctypes does have a big disadvantage: when something goes wrong (and it invariably will for any complex interfaces), it's a hell to debug.
The problem is that a big part of your stack is obscured by ctypes/ffi magic and there is no easy way to determine how did you get to a particular point and why parameter values are what they are..
You can also use Pyrex, which can act as glue between high-level Python code and low-level C code. lxml is written in Pyrex, for instance.
ctypes is great, but does not handle C++ classes. I've also found ctypes is about 10% slower than a direct C binding, but that will highly depend on what you are calling.
If you are going to go with ctypes, definitely check out the Pyglet and Pyopengl projects, that have massive examples of ctype bindings.
I'm going to be contrarian and suggest that, if you can, you should write your extension library using the standard Python API. It's really well-integrated from both a C and Python perspective... if you have any experience with the Perl API, you will find it a very pleasant surprise.
Ctypes is nice too, but as others have said, it doesn't do C++.
How big is the library you're trying to wrap? How quickly does the codebase change? Any other maintenance issues? These will all probably affect the choice of the best way to write the Python bindings.
Just wanted to add a few more considerations that I didn't see mentioned yet.
[EDIT: Ooops, didn't see Mike Steder's answer]
If you want to try using a non Cpython implementation (like PyPy, IronPython or Jython), then ctypes is about the only way to go. PyPy doesn't allow writing C-extensions, so that rules out pyrex/cython and Boost.python. For the same reason, ctypes is the only mechanism that will work for IronPython and (eventually, once they get it all working) jython.
As someone else mentioned, no compilation is required. This means that if a new version of the .dll or .so comes out, you can just drop it in, and load that new version. As long as the none of the interfaces changed, it's a drop in replacement.
Something to keep in mind is that SWIG targets only the CPython implementation. Since ctypes is also supported by the PyPy and IronPython implementations it may be worth writing your modules with ctypes for compatibility with the wider Python ecosystem.
I have found SWIG to be be a little bloated in its approach (in general, not just Python) and difficult to implement without having to cross the sore point of writing Python code with an explicit mindset to be SWIG friendly, rather than writing clean well-written Python code. It is, IMHO, a much more straightforward process to write C bindings to C++ (if using C++) and then use ctypes to interface to any C layer.
If the library you are interfacing to has a C interface as part of the library, another advantage of ctypes is that you don't have to compile a separate python-binding library to access third-party libraries. This is particularly nice in formulating a pure-python solution that avoids cross-platform compilation issues (for those third-party libs offered on disparate platforms). Having to embed compiled code into a package you wish to deploy on something like PyPi in a cross-platform friendly way is a pain; one of my most irritating points about Python packages using SWIG or underlying explicit C code is their general inavailability cross-platform. So consider this if you are working with cross-platform available third party libraries and developing a python solution around them.
As a real-world example, consider PyGTK. This (I believe) uses SWIG to generate C code to interface to the GTK C calls. I used this for the briefest time only to find it a real pain to set up and use, with quirky odd errors if you didn't do things in the correct order on setup and just in general. It was such a frustrating experience, and when I looked at the interace definitions provided by GTK on the web I realized what a simple excercise it would be to write a translator of those interface to python ctypes interface. A project called PyGGI was born, and in ONE day I was able to rewrite PyGTK to be a much more functiona and useful product that matches cleanly to the GTK C-object-oriented interfaces. And it required no compilation of C-code making it cross-platform friendly. (I was actually after interfacing to webkitgtk, which isn't so cross-platform). I can also easily deploy PyGGI to any platform supporting GTK.

Categories

Resources