Develop Reference

Python Extension Dll Installation

I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.

Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.

Get unmanaged DLL path to load Python independent of the version

I looked for a way to retrieve the path to a dll that is installed in the user's computer, but where the path could change depending on where they decided to install it. Couldn't find anything, so I wrote this with my own findings (feel free to add your own).
Some background:
I'm writing a module that loads Python into C++ (my users' machines have installed Python in their path, but the python version and path may vary between users)
However, I've found 2 issues:
The linker creates dll version dependencies even if I'm using functions available to any python 3 release (e.g. python 3.6 requires python36.dll).
The PYTHONPATH must be set in order to find the installed modules.
For the first issue, I used LoadLibrary to load at run time the appropriate dll, but that still left the burden of configuration on the user (he had to configure which dll was in his system, and where it was installed). Works fine if your user knows about his configuration, which is not the case for many of my users.
So that brings me to the guessing:
I was able to load python3.dll (which is located right next to the python36.dll or python38.dll) and I needed the path to the dll to calculate PTYHONPATH (and potentially, use python3 version to get the right dll to use, like python37.dll, python38.dll, etc.)

I don't think there is a nice way to allow for different Python minor versions (eg 3.6 or 3.8) unless you define a stable ABI for Python. However from my experience is that this makes for a poor subset of Python that doesn't support things like PyMemoryView. I would also suggest using delay load linker flags if you can isolate the Python part of your code to its own DLL. That way you can have a config file that reads where the Python library is located at runtime and load it from the appropriate path.

First, load the library using LoadLibraryA (or LoadLibraryW) and then use GetModuleFileNameA (or GetModuleFileNameW) to get the fullpath
//#include <stdio.h>
//#include <iostream>
HMODULE pythonLib = nullptr;
pythonLib = LoadLibraryA("python3.dll");
if (pythonLib != nullptr) {
char path[MAX_PATH];
GetModuleFileNameA(pythonLib, path, MAX_PATH);
std::cout << path << std::endl;
}

Deploying C++ program with python embedded

I have a Qt C++ project in which, for some functionality,I have added Python. The C++ function calls python script and returns the values.Like the below Example.
PyObject *pName,*pModule,*pFunc;
PyObject *pArgs,*pValue;
const char *module="getBeamData";
pName=PyUnicode_FromString(module);
pModule=PyImport_Import(pName);
if(pModule!=NULL){
pFunc=PyObject_GetAttrString(pModule,"getBeamDose");
if(pFunc&&PyCallable_Check(pFunc)){
pArgs=PyTuple_New(2);
PyTuple_SetItem(pArgs,0,PyBytes_FromString(path.toStdString().c_str()));
PyTuple_SetItem(pArgs,1,PyLong_FromLong(fraction));
pValue=PyObject_CallObject(pFunc,pArgs);
It calls the script getBeamdata.py. Functionality works fine. Do I have to install python and libraries(like numpy) for all customer and ship script ? Or is there any other way without sending script.

Without statically compiling your Python script you will need to distribute:
requirements.txt (for all libraries/modules required by your script).
details on the correct version of Python required by the script (2.7/3 etc).
Alternatively you could look into creating an executable using py2exe and distribute that instead, although you'll need to modify the way your program interacts with the Python script.
Some further details on distributing Python can be found here.

If I were you, I'd do it with some sort of installer.
Now in that installer, you can of course embed the wheel version of your script, which is in addition going to be installed using setuptools.
In that setuptools script you can mention dependencies, but that means that your client should have either Python installed on his/her computer, or you should ship your own version of Python embedded with the software.
Such functionality can be also be achieved using Boost.Python

using a C extension in python, without installing it as a module

I am writing C extensions for python. I am just experimenting for the time being and I have written a hello world extension that looks like this :
#include <Python2.7/Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] = "helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs},
{NULL,NULL,0,NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,"Extension module example!");
}
the code works perfectly fine, after installing it from a setup.py file I wrote, and installing it from command line
python setup.py install
What I want is the following :
I want to use the C file as a python extension module, without installing it, that is I want to use it as just another python file in my project, and not a file that I need to install before my python modules get to use its functionality. Is there some way of doing this ?

You can simply compile the extension without installing (usually something like python setup.py build). Then you have to make sure the interpreter can find the compiled module (for example by copying it next to a script that imports it, or setting PYTHONPATH).

You can create your "own interpreter" by not extending python, but embedding it into your application. In that way, your objects will be always available for the users who are running your program. This is a pretty common thing to do in certain cases, for example look at the Blender project where all the bpy, bmesh and bge modules are already included.
The downside is, your users can't use the python command directly, they have to use your hello_world_python instead. (But of course you can provide your extension as a module as well.) And that also means, you have to compile and distribute your application for all platforms you want to support -- in case you want to distribute it as a binary, to make your users lives a bit easier.
For further information on embedding python into your program, read the propriate sections of the documentation:
Embedding Python in Another Application
Personal suggestion: Use Python 3.5 whenever you can, and stop supporting the old 2.x versions. For more information, read this article: Should I use Python 2 or Python 3 for my development activity?

Installing python library with C dependancies

I'd like to install the pypoker-eval package. The author provided a nice readme for how to install the package in windows:
pypoker-eval on Windows
install python
open visual studio project and change directories to include python directories and fix the path to the poker-eval project
build the project, there may be an undefined ssize you can typedef as int
locate the newly built dll pypokereval.dll then look at the exports and you will find something similar to init_pokereval_2_4
rename the dll to _pokereval_2_4.pyd (your version numbers) and move it into the test.py directory
edit the .py files so the imported module is _pokereval_2_4 (your version numbers) not a dynamically generated one
To be quite honest though, these instructions really lost me. I've tried doing some research on my own, and I understand that because the library has some C code I need to compile it first. I wasn't able to figure out though what the purpose of changing the directories to include python directories, does this part mean change the working directory? Also is the compiler I use important? I've dabbled some in C but that was with the Code::Blocks IDE.
I think my main problem is I don't really understand what happens when I install a package. Pip has spoiled me and kept this whole process as a block box and now with something like this I feel completely lost. Any recommended reading on installing packages with C dependencies or tips on how I can get this package installed would be much appreciated!
For reference I'm using Python 2.7 on Windows 7.