I am wrapping C++ code for use in Python using SWIG. The C++ module I am wrapping has C++ dependencies of other modules located within a different package. However, rather than directly importing/including these files, I would like to import a previously created Python library/dynamic shared library to deal with the dependencies. I want to use this method because I do not want to hard-code files in this package for them to work. I will simply have access to the shared library.
Currently, without importing the library, compiling the new module with the wrapper file results in the error:
"fatal error: string: No such file or directory compilation terminated."
as a header file the new module depends on is not available within this package. I do not want to copy the required headers the new module has a dependency on into this package.
I would like to know if this is possible within either the SWIG interface file or CMake.
Thanks for your help.
Related
I have a C++ library where I have a Python binding with PyBind11. My C++ library gets compiled against another C++ library which also has a Python wrapper. To ensure consistency, my library built into a directory which has the version of the upstream library in its path. This way we can have multiple different virtual environments with the upstream library. During runtime my Python code will determine which compiled version to load.
My Python code then uses my function get_libwidget_library_dir() to determine the matching path based on the upstream library version:
soabi = sysconfig.get_config_var("SOABI")
file_path = os.path.join(
get_libwidget_library_dir(), "libwidget", "binding", f"libwidget_pybind.{soabi}.so"
)
spec = importlib.util.spec_from_file_location("libwidget_pybind", file_path)
module = importlib.util.module_from_spec(spec)
These hacks are only needed because we build the library in-tree and then import it in the checkout. If I was to compile that for all different versions of the upstream library, make Python wheels and put them into the virtual enviroments, it would likely be much more standard. But then it becomes more painful to develop the library as I cannot just build and launch.
All this works, except that mypy doesn't understand what this library is. I can generate the stub file libwidget.pyi from the compiled library, but I am not sure where to put it. The module path is only determined during runtime, it seems that mypy as a static checker by construction cannot check that.
Is the only way to make mypy aware of the stubs to get rid of the runtime dependency on the path? Or is there some way to add the stubs there as well?
I can make as much out as its build system generates a shared object, and also a Python module as a proxy to that shared object. But how does the Python runtime end up doing a dlopen of the generated shared object and bind Python method calls on to corresponding functions in the shared library?
I also found a reference in Python's import system about shared libraries, but nothing beyond that. Does CPython treat .so files in the module path as Python modules?
When doing import module Python will look for files with various extensions that could be python modules. That can be module.py, but also module.so (on Linux) or module.pyd (on Windows).
When loading a shared object, Python will load it like any dynamic library and then it will call the module's init method: It must be named PyInit_{module_name_here} and exported in the shared library.
You can read more about it here.
I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.
Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.
I am using CMake to build some python bindings for my code using Pybind11. It is working well, but they get compiled in the main build directory. I would like them to be built on build\python directory. I am trying the following:
pybind11_add_module(myModule src/main.cpp)
set_target_properties(myModule PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/python")
But it is not working as intended, myModule is still built on the build directory as if the set_target_properties was not being called.
In the official pybind11_add_module documentation it is written:
This function behaves very much like CMake’s builtin add_library (in fact, it’s a wrapper function around that command). It will add a library target called to be built from the listed source files. In addition, it will take care of all the Python-specific compiler and linker flags as well as the OS- and Python-version-specific file extension. The produced target can be further manipulated with regular CMake commands.
So I assumed that set_target_properties could be used to indicate a different output directory after it, is this not the case? If not, how can this be done?
Thank you in advance!
pybind11 module is a library of either a SHARED or MODULE type.
Build directory for SHARED libraries is specified via LIBRARY_OUTPUT_DIRECTORY on all platforms except for Windows (and its dll's).
Build directory for MODULE libraries is specified via LIBRARY_OUTPUT_DIRECTORY on all platforms without exception.
Detailed description of types for output artifacts in CMake and corresponded OUTPUT variables can be found in the documentation.
I have a C++ library (we'll call it Example in the following) for which I wrote Python bindings using the boost.python library. This Python-wrapped library will be called pyExample. The entire project is built using CMake and the resulting Python-wrapped library is a file named libpyExample.so.
When I use the Python bindings from a Python script located in the same directory as libpyExample.so, I simply have to write:
import libpyExample
libpyExample.hello_world()
and this executes a hello_world() function exposed by the wrapping process.
What I want to do
For convenience, I would like my pyExample library to be available from anywhere simply using the command
import pyExample
I also want pyExample to be easily installable in any virtualenv in just one command. So I thought a convenient process would be to use setuptools to make that happen. That would therefore imply:
Making libpyExample.so visible for any Python script
Changing the name under which the module is accessed
I did find many things about compiling C++ extensions with setuptools, but nothing about packaging a pre-compiled C++ extension. Is what I want to do even possible?
What I do not want to do
I don't want to build the pyExample library with setuptools, I would like to avoid modifying the existing project too much. The CMake build is just fine, I can retrieve the libpyExample.so file very easily.
If I understand your question correctly, you have the following situation:
you have an existing CMake-based build of a C++ library with Python bindings
you want to package this library with setuptools
The latter then allows you to call python setup.py install --user, which installs the lib in the site-packages directory and makes it available from every path in your system.
What you want is possible, if you overload the classes that setuptools uses to build extensions, so that those classes actually call your CMake build system. This is not trivial, but you can find a working example here, provided by the pybind11 project:
https://github.com/pybind/cmake_example
Have a look into setup.py, you will see how the classes build_ext and Extension are inherited from and modified to execute the CMake build.
This should work out of the box for you or with little modification - if your build requires special -D flags to be set.
I hope this helps!