I have a project, cloudgen, that I would like to add bindings for Python so I can access some of the underlying functions. I have stubbed out the initial work on a branch. Because the main executable is built with cmake, I decided to use scikit-build to manage the build and use pybind11 to deal with the binding (following this example repo).
When I run pip install . in a virtual environment, everything appears to work as expected. I find the executable is installed to <prefix>/bin, the library goes into <prefix>/lib, and the module goes into <prefix>/lib/pythonX.Y/site-packages/cloudgen. In fact, if I run pip uninstall cloudgen, all of the correct files are uninstalled. However, my problems arise when I start to test the Python bindings. I find two separate but related problems.
If I installed into an Anaconda environment, the module is able to resolve the path to the shared library and pass the tests, but the executable does not resolve the path to the library.
On the other hand, if I installed into a virtual environment using python -m venv, both the module and the executable are unable to resolve the path to the shared library.
Searching around, I came across this question which notes I could manipulate LD_LIBRARY_PATH (or equivalently DYLD_LIBRARY_PATH on macOS or PATH on Windows), but that is normally frowned upon. That question references an open issue that refers to including additional build products (which as I said appears to not be my problem) but doesn't address the library path resolution. I also came across this question asking about distributing the build products using scikit-build and this question using setuptools directly. Neither of the questions or answers address the library path resolution.
My question is: What is the correct way to distribute a package that contains an executable, shared library, and Python binding module and have the path resolution Just Work™?
A minimal working example is a bit much, but I created a gist to demonstrate the behavior.
After a bit more digging (and carefully reading the CMake documentation on RPATH), the correct answer appears to be explicitly setting RPATH on installation. The relevant change to the linked gist is to add the following to the CMakeLists.txt after creating the targets (adapted from the linked Wiki):
if (SKBUILD)
find_package(PythonExtensions REQUIRED)
set(lib_path "${PYTHON_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
else()
set(lib_path "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_LIBDIR}")
endif()
list(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${lib_path}" is_system)
if ("${is_system}" STREQUAL "-1")
set_target_properties(mwe.exe PROPERTIES
INSTALL_RPATH_USE_LINK_PATH TRUE
INSTALL_RPATH "${lib_path}")
# The following is necessary for installation in a virtual
# environment `python -m pip venv env`
set_target_properties(_mwe PROPERTIES
INSTALL_RPATH_USE_LINK_PATH TRUE
INSTALL_RPATH "${lib_path}")
endif()
This does require following the rest of the details about setting the RPATH such as including (literally from the linked CMake Wiki):
# use, i.e. don't skip the full RPATH for the build tree
set(CMAKE_SKIP_BUILD_RPATH FALSE)
# when building, don't use the install RPATH already
# (but later on when installing)
set(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
earlier in the CMakeLists.txt. The net result is this should disable the dynamic searching of the library path (LD_LIBRARY_PATH or DYLD_LIBRARY_PATH as appropriate) for the installed executable and Python module.
Related
I was trying to understand the purpose of use_develop and from the docs, I found this:
Install the current package in development mode with develop mode. For pip this uses -e option, so should be avoided if you’ve specified a custom install_command that does not support -e.
I don't understand what "development mode" means. Is this a python concept or it's specific to tox? Either way what does it mean?
development mode or editable installs is a Python concept, or even more specific a Python packaging concept.
Usually, when you package a Python application or library, the source files are packaged into a "container", either a wheel or a source distribution.
This is a good thing to distribute a package, but not for developing, as then the source files are no longer accessible.
editable installs is a concept, that instead of "moving/copying" the files into the package container, the files are just symlinked (at least this is one way).
So when you edit the source files, also the package is updated immediately.
For tox this also means that the files in the root of the source tree are importable by Python, not only the one in the package.
This might be comfortable, but there is one huge caveat. If you misconfigure the packaging setup, maybe the tests ran by tox are green, but it is entirely possible that you forget to include the source files in the package you deliver to your users.
As distutils is being removed from Python in the > 3.10 versions, and setuptools will not be added to the stdlib, I want to replace an existing setup.py recipe for building/installing a C++ library Cython extension (i.e. not primarily a Python package, not run in a venv, etc.) with some custom code.
The Cython part is working fine, and I just about managed to construct an equivalent call to the C++ compiler from that previously executed by distutils, by using config-var info from sysconfig... though the latter was very trial and error, with no documentation or particular consistency to the config-var collection as far as I could tell.
But I am now stuck on identifying what directory to install my build extension .so into, within the target prefix of my build. Depending on the platform and path scheme in use, and the prefix itself, the subdirs could be in lib or lib64, a pythonX.Y subdir of some sort, and a final site-packages, dist-packages or other directory. This decision was previously made by distutils but I can't find any equivalent code to return such path decisions in other stdlib packages.
Any suggestions of answers or best-practice approaches? (Other than "use setuptools", please!)
I maintain a Python utility that allows bpy to be installable as a Python module. Due to the hugeness of the spurce code, and the length of time it takes to download the libraries, I have chosen to provide this module as a wheel.
Unfortunately, platform differences and Blender runtime expectations makes support for this tricky at times.
Currently, one of my big goals is to get the Blender addon scripts directory to install into the correct location. The directory (simply named after the version of Blender API) has to exist in the same directory as the Python executable.
Unfortunately the way that setuptools works (or at least the way that I have it configured) the 2.79 directory is not always placed as a sibling to the Python executable. It fails on Windows platforms outside of virtual environments.
However, I noticed in setuptools documentation that you can specify eager_resources that supposedly guarantees the location of extracted files.
https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-resource-extraction
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resource-extraction
There was a lot of hand waving and jargon in the documentation, and 0 examples. I'm really confused as to how to structure my setup.py file in order to guarantee the resource extraction. Currently, I just label the whole 2.79 directory as "scripts" in my setuptools Extension and ship it.
Is there a way to write my setup.py and package my module so as to guarantee the 2.79 directory's location is the same as the currently running python executable when someone runs
py -3.6.8-32 -m pip install bpy
Besides simply "hacking it in"? I was considering writing a install_requires module that would simply move it if possible but that is mangling with the user's file system and kind of hacky. However it's the route I am going to go if this proves impossible.
Here is the original issue for anyone interested.
https://github.com/TylerGubala/blenderpy/issues/13
My build process is identical to the process descsribed in my answer here
https://stackoverflow.com/a/51575996/6767685
Maybe try the data_files option of distutils/setuptools.
You could start by adding data_files=[('mydata', ['setup.py'],)], to your setuptools.setup function call. Build a wheel, then install it and see if you can find mydata/setup.py somewhere in your sys.prefix.
In your case the difficult part will be to compute the actual target directory (mydata in this example). It will depend on the platform (Linux, Windows, etc.), if it's in a virtual environment or not, if it's a global or local install (not actually feasible with wheels currently, see update below) and so on.
Finally of course, check that everything gets removed cleanly on uninstall. It's a bit unnecessary when working with virtual environments, but very important in case of a global installation.
Update
Looks like your use case requires a custom step at install time of your package (since the location of the binary for the Python interpreter relative to sys.prefix can not be known in advance). This can not be done currently with wheels. You have seen it yourself in this discussion.
Knowing this, my recommendation would be to follow the advice from Jan Vlcinsky in his comment for his answer to this question:
Post install script after installing a wheel.
Add an extra setuptools console entry point to your package (let's call it bpyconfigure).
Instruct the users of your package to run it immediately after installing your package (pip install bpy && bpyconfigure).
The purpose of bpyconfigure should be clearly stated (in the documentation and maybe also as a notice shown in the console right after starting bpyconfigure) since it would write into locations of the file system where pip install does not usually write.
bpyconfigure should figure out where is the Python interpreter, and where to write the extra data.
The extra data to write should be packaged as package_data, so that it can be found with pkg_resources.
Of course bpyconfigure --uninstall should be available as well!
I am using pydev and a virtualenv (which has already been set up successfully). How do you add quantlib (and for that matter any python wrapper plus its C++ native library) to a virtualenv?
I successfully built quantlib and the quantlib-SWIG from source as described here. I notice that after the boost build, //usr/local/lib contains libQuantLib.* files which are probably the native libs.
I then tried copying libQuantLib.* to my virtualenv/lib/python2.7/site-packages, as described here but eclipse still complains about unresolved imports (at this point I am also externally referencing //usr/local/lib/QuantLib-SWIG-1.4/Python/build/lib.linux-x86_64-2.7/QuantLib folder). I am not sure if I had this correctly working.
I have seen this solution, but I really want everything contained in the virtualenv - both the python wrapper and C++ libraries, so everything is resolved when I set the project's pydev interpreter as my virtualenv.
I am unsure what best practices are here.
I'm not familiar with the way the virtualenv is set up. However: from the fact that your Python modules are in virtualenv/lib/python2.7/site-packages, I'd guess that the native libraries should go in virtualenv/lib. The correct way to have everything set up there, though, would be to tell the build machinery where you want the library; in your case (and assuming my guess above is correct) you'd do it by building QuantLib with:
./configure --prefix=/path/to/virtualenv
make
make install
where /path/to/virtualenv is the path to your virtualenv, including the virtualenv folder (but not lib). This will put header files and native libraries in the correct place in the virtualenv. After this, build QuantLib-SWIG using the QuantLib libraries you just installed: I think the easiest way is to do it from within the virtualenv (that is, using the Python interpreter inside it). Activate the env, enter the QuantLib-SWIG/Python directory, and run:
export PATH=/path/to/virtualenv/bin:$PATH
python setup.py build
python setup.py install
where setting PATH as above might be needed to find the correct quantlib-config script. (By the way, you should end up with just a QuantLib Python module in site-packages, not the whole build/lib.linux-x86_64-2.7 thing you have now.)
First of all let me state that I am a proponent of generic software (in general ;-). I am no expert on Python, but it seems that the 'virtualenv' utility solves pretty much the same problem 'chroot' can help to solve - bootstrapping a directory tree that can be passed as root, thus effectively protecting the real directory tree, if needed.
Since I am no expert in Python as already mentioned, I wonder - what problem can virtualenv solve that chroot cannot? I mean, can't I just set up a nice fake root tree (possibly using union mounting), chroot into it, and do pip install a package I want in my new environment, and then play around within the bounds of my new environment, running python scripts and what not?
Am I missing something here?
Update:
Can't one install packages/modules locally in whatever application directory, I mean, without root privileges and subsequently without overwriting or adding files to /usr/lib or /usr/local/lib? It appears that this is what virtualenv does, however I think it has to symlink or otherwise provide a python interpreter for each environment one creates, does it not?
bootstrapping a directory tree that can be passed as root
That's not what virtualenv does, except (to some degree) for Python packages. It provides a place where these can be installed without replacing the rest of the filesystem. It also works without root privileges and it's portable as it needs no kernel support, unlike chroot, which (I presume) won't work on Windows.
Can't one install packages/modules locally in whatever application directory
Yes, but virtualenv does one more thing, which is that it disables (by default at least) the system's Python package directories. That means you can test whether your package correctly installs all of its dependencies (you might have forgotten to list one because it's already installed on your system) and it allows installing different versions in case you need either newer or older versions. The ability to install older versions should not be overlooked because sometimes new versions of packages introduce bugs.