Develop Reference

Python Extension Dll Installation

I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.

Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.

Get unmanaged DLL path to load Python independent of the version

I looked for a way to retrieve the path to a dll that is installed in the user's computer, but where the path could change depending on where they decided to install it. Couldn't find anything, so I wrote this with my own findings (feel free to add your own).
Some background:
I'm writing a module that loads Python into C++ (my users' machines have installed Python in their path, but the python version and path may vary between users)
However, I've found 2 issues:
The linker creates dll version dependencies even if I'm using functions available to any python 3 release (e.g. python 3.6 requires python36.dll).
The PYTHONPATH must be set in order to find the installed modules.
For the first issue, I used LoadLibrary to load at run time the appropriate dll, but that still left the burden of configuration on the user (he had to configure which dll was in his system, and where it was installed). Works fine if your user knows about his configuration, which is not the case for many of my users.
So that brings me to the guessing:
I was able to load python3.dll (which is located right next to the python36.dll or python38.dll) and I needed the path to the dll to calculate PTYHONPATH (and potentially, use python3 version to get the right dll to use, like python37.dll, python38.dll, etc.)

I don't think there is a nice way to allow for different Python minor versions (eg 3.6 or 3.8) unless you define a stable ABI for Python. However from my experience is that this makes for a poor subset of Python that doesn't support things like PyMemoryView. I would also suggest using delay load linker flags if you can isolate the Python part of your code to its own DLL. That way you can have a config file that reads where the Python library is located at runtime and load it from the appropriate path.

First, load the library using LoadLibraryA (or LoadLibraryW) and then use GetModuleFileNameA (or GetModuleFileNameW) to get the fullpath
//#include <stdio.h>
//#include <iostream>
HMODULE pythonLib = nullptr;
pythonLib = LoadLibraryA("python3.dll");
if (pythonLib != nullptr) {
char path[MAX_PATH];
GetModuleFileNameA(pythonLib, path, MAX_PATH);
std::cout << path << std::endl;
}

Make executable file from multiple pyx files using cython

I am trying to make one unix executable file from my python source files.
I have two file, p1.py and p2.py
p1.py :-
from p2 import test_func
print (test_func())
p2.py :-
def test_func():
return ('Test')
Now, as we can see p1.py is dependent on p2.py . I want to make an executable file by combining two files together. I am using cython.
I changed the file names to p1.pyx and p2.pyx respectively.
Now, I can make file executable by using cython,
cython p1.pyx --embed
It will generate a C source file called p1.c . Next we can use gcc to make it executable,
gcc -Os -I /usr/include/python3.5m -o test p1.c -lpython3.5m -lpthread -lm -lutil -ldl
But how to combine two files into one executable ?

People are tempted to do this because it's fairly easy to do for the simplest case (one module, no dependencies). #ead's answer is good but honestly pretty fiddly and it is handling the next simplest case (two modules that you have complete control of, no dependencies).
In general a Python program will depend on a range of external modules. Python comes with a large standard library which most programs use to an extent. There's a wide range of third party libraries for maths, GUIs, web frameworks. Even tracing those dependencies through the libraries and working out what you need to build is complicated, and tools such as PyInstaller attempt it but aren't 100% reliable.
When you're compiling all these Python modules you're likely to come across a few Cython incompatibilities/bugs. It's generally pretty good, but struggles with features like introspection, so it's unlikely a large project will compile cleanly and entirely.
On top of that many of those modules are compiled modules written either in C, or using tools such as SWIG, F2Py, Cython, boost-python, etc.. These compiled modules may have their own unique idiosyncrasies that make them difficult to link together into one large blob.
In summary, it may be possible, but for non-trivial programs it is not a good idea however appealing it seems. Tools like PyInstaller and Py2Exe that use a much simpler approach (bundle everything into a giant zip file) are much more suitable for this task (and even then they struggle to be really robust).
Note this answer is posted with the intention of making this question a canonical duplicate for this problem. While an answer showing how it might be done is useful, "don't do this" is probably the best solution for the vast majority of people.

There are some loops you have to jump through to make it work.
First, you must be aware that the resulting executable is a very slim layer which just delegates the whole work to (i.e. calls functions from) pythonX.Ym.so. You can see this dependency when calling
ldd test
...
libpythonX.Ym.so.1.0 => not found
...
So, to run the program you either need to have the LD_LIBRARY_PATH showing to the location of the libpythonX.Ym.so or build the exe with --rpath option, otherwise at the start-up of test dynamic loader will throw an error similar to
/test: error while loading shared libraries: libpythonX.Ym.so.1.0: cannot open shared object file: No such file or directory
The generic build command would look like following:
gcc -fPIC <other flags> -o test p1.c -I<path_python_include> -L<path_python_lib> -Wl,-rpath=<path_python_lib> -lpython3.6m <other_needed_libs>
It is also possible to build against static version of the python-library, thus eliminating run time dependency on the libpythonX.Ym, see for example this SO-post.
The resulting executable test behaves exactly the same as if it were a python-interpreter. This means that now, test will fail because it will not find the module p2.
One simple solution were to cythonize the p2-module inplace (cythonize p2.pyx -i): you would get the desired behavior - however, you would have to distribute the resulting shared-object p2.so along with test.
It is easy to bundle both extension into one executable - just pass both cythonized c-files to gcc:
# creates p1.c:
cython --empbed p1.pyx
# creates p2.c:
cython p2.pyx
gcc ... -o test p1.c p2.c ...
But now a new (or old) problem arises: the resulting test-executable cannot once again find the module p2, because there is no p2.py and no p2.so on the python-path.
There are two similar SO questions about this problem, here and here. In your case the proposed solutions are kind of overkill, here it is enough to initialize the p2 module before it gets imported in the p1.pyx-file to make it work:
# making init-function from other modules accessible:
cdef extern object PyInit_p2();
#init/load p2-module manually
PyInit_p2() #Cython handles error, i.e. if NULL returned
# actually using already cached imported module
# no search in python path needed
from p2 import test_func
print(test_func())
Calling the init-function of a module prior to importing it (actually the module will not be really imported a second time, only looked up in the cache) works also if there are cyclic dependencies between modules. For example if module p2 imports module p3, which imports p2in its turn.
Warning: Since Cython 0.29, Cython uses multi-phase initialization per default for Python>=3.5, thus calling PyInit_p2 is not enough (see e.g. this SO-post). To switch off this multi-phase initialization -DCYTHON_PEP489_MULTI_PHASE_INIT=0should be passed to gcc or similar to other compilers.
Note: However, even after all of the above, the embedded interpreter will need its standard libraries (see for example this SO-post) - there is much more work to do to make it truly standalone! So maybe one should heed #DavidW's advice:
"don't do this" is probably the best solution for the vast majority of
people.
A word of warning: if we declare PyInit_p2() as
from cpython cimport PyObject
cdef extern PyObject *PyInit_p2();
PyInit_p2(); # TODO: error handling if NULL is returned
Cython will no longer handle the errors and its our responsibility. Instead of
PyObject *__pyx_t_1 = NULL;
__pyx_t_1 = PyInit_p2(); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 4, __pyx_L1_error)
__Pyx_GOTREF(__pyx_t_1);
__Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
produced for object-version, the generated code becomes just:
(void)(PyInit_p2());
i.e. no error checking!
On the other hand using
cdef extern from *:
"""
PyObject *PyInit_p2(void);
"""
object PyInit_p2()
will not work with g++ - one has to add extern C to declaration.

using a C extension in python, without installing it as a module

I am writing C extensions for python. I am just experimenting for the time being and I have written a hello world extension that looks like this :
#include <Python2.7/Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] = "helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs},
{NULL,NULL,0,NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,"Extension module example!");
}
the code works perfectly fine, after installing it from a setup.py file I wrote, and installing it from command line
python setup.py install
What I want is the following :
I want to use the C file as a python extension module, without installing it, that is I want to use it as just another python file in my project, and not a file that I need to install before my python modules get to use its functionality. Is there some way of doing this ?

You can simply compile the extension without installing (usually something like python setup.py build). Then you have to make sure the interpreter can find the compiled module (for example by copying it next to a script that imports it, or setting PYTHONPATH).

You can create your "own interpreter" by not extending python, but embedding it into your application. In that way, your objects will be always available for the users who are running your program. This is a pretty common thing to do in certain cases, for example look at the Blender project where all the bpy, bmesh and bge modules are already included.
The downside is, your users can't use the python command directly, they have to use your hello_world_python instead. (But of course you can provide your extension as a module as well.) And that also means, you have to compile and distribute your application for all platforms you want to support -- in case you want to distribute it as a binary, to make your users lives a bit easier.
For further information on embedding python into your program, read the propriate sections of the documentation:
Embedding Python in Another Application
Personal suggestion: Use Python 3.5 whenever you can, and stop supporting the old 2.x versions. For more information, read this article: Should I use Python 2 or Python 3 for my development activity?

C++ with Python embedding: crash if Python not installed

I'm developing on Windows, and I've searched everywhere without finding anyone talking about this kind of thing.
I made a C++ app on my desktop that embedded Python 3.1 using MSVC. I linked python31.lib and included python31.dll in the app's run folder alongside the executable. It works great. My extension and embedding code definitely works and there are no crashes.
I sent the run folder to my friend who doesn't have Python installed, and the app crashes for him during the scripting setup phase.
A few hours ago, I tried the app on my laptop that has Python 2.6 installed. I got the same crash behavior as my friend, and through debugging found that it was the Py_Initialize() call that fails.
I installed Python 3.1 on my laptop without changing the app code. I ran it and it runs perfectly. I uninstalled Python 3.1 and the app crashes again. I put in code in my app to dynamically link from the local python31.dll, to ensure that it was using it, but I still get the crash.
I don't know if the interpreter needs more than the DLL to start up or what. I haven't been able to find any resources on this. The Python documentation and other guides do not seem to ever address how to distribute your C/C++ applications that use Python embedding without having the users install Python locally. I know it's more of an issue on Windows than on Unix, but I've seen a number of Windows C/C++ applications that embed Python locally and I'm not sure how they do it.
What else do I need other than the DLL? Why does it work when I install Python and then stop working when I uninstall it? It sounds like it should be so trivial; maybe that's why nobody really talks about it. Nevertheless, I can't really explain how to deal with this crash issue.
Thank you very much in advance.

In addition to pythonxy.dll, you also need the entire Python library, i.e. the contents of the lib folder, plus the extension modules, i.e. the contents of the DLLs folder. Without the standard library, Python won't even start, since it tries to find os.py (in 3.x; string.py in 2.x). On startup, it imports a number of modules, in particular site.py.
There are various locations where it searches for the standard library; in your cases, it eventually finds it in the registry. Before, uses the executable name (as set through Py_SetProgramName) trying to find the landmark; it also checks for a file python31.zip which should be a zipped copy of the standard library. It also checks for a environment variable PYTHONHOME.
You are free to strip the library from stuff that you don't need; there are various tools that compute dependencies statically (modulefinder in particular).
If you want to minimize the number of files, you can
link all extension modules statically into your pythonxy.dll, or even link pythonxy.dll statically into your application
use the freeze tool; this will allow linking the byte code of the standard library into your pythonxy.dll.
(alternatively to 2.) use pythonxy.zip for the standard library.

Nice. And if you do not want to zip, copy Python26\DLLs & Python26\lib to your exe directory as:
.\myexe.exe
.\python26.dll
.\Python26\DLLs
.\Python26\lib
And then set PYTHONHOME with Py_SetPythonHome() API. Apparently, this API is not in the list of "allowed" calls before Py_Initialize();
Below worked for me on Windows (Python not installed):
#include "stdafx.h"
#include <iostream>
#include "Python.h"
using namespace std;
int _tmain(int argc, _TCHAR* argv[])
{
char pySearchPath[] = "Python26";
Py_SetPythonHome(pySearchPath);
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print 'Today is',ctime(time())\n");
//cerr << Py_GetPath() << endl;
Py_Finalize();
return 0;
}
Good that the search path is relative w.r.t the exe. Py_GetPath can show you where all it is looking for the modules.

A zip of the Python standard library worked for me with Python27.
I zipped the contents of Lib and dll, and made sure there was no additional python27-subfolder or Lib or dll subfolder. i.e. just a zip named python27.zip containing all the files.
I copied that zip and the python27.dll alongside the executable.

I wanted to add some additional info for others who might still be having troubles with this, as I was. I was eventually able to get my application working using the method proposed by user sambha, that is:
Program Files (x86)\
MyApplicationFolder\
MyApplication.exe
python27.dll
Python27\
DLLs\ (contents of DLLs folder)
Lib\ (contents of Lib folder)
...with one important addition: I also needed to install the MSVCR90.DLL. I'm using Python 2.7 and apparently python27.dll requires the MSVCR90.DLL (and maybe other MSVC*90.DLLs).
I solved this by downloading, installing, and running the 'vcredist_x86.exe' package from http://www.microsoft.com/en-us/download/details.aspx?id=29 . I think, though I am not certain, that you need to do it this way at least on Win7, as opposed to simply placing the MSVC*90.DLLs alongside your .exe as you may have done in the past. The Microsoft installer places the files and registers them in a special way under Win7.
I also tried the .zip file method but that did not work, even with the MSVCR90.DLL installed.