My goal is to embed the python interpreter using PyBind11, but to use the interpreter from a virtual env, such that installing dependencies using pip does not clutter the system paths.
There is not much information online around this topic. Embedding python with pybind11. Virtual environment doesn't work hardcodes the venv at compile time. This is insufficient, since the venv does not exist at that time, but will be created when the scripting engine is enabled at runtime.
The plan for now is to pip install --target into a cache dir and add it to the sys.path. Using the system interpreter. This is "okayish" but not using the system interpreter would be peferable.
You do not need a virtual environment to ship isolated dependencies with your software, or to use them locally, you simply need what you already have: an isolated lib folder.
pybind11 does not have its own interpreter (AFAIK), it embeds the one from the Python build you're using when compiling, so you should already have a matching version of the Python interpreter you're embedding (this is the executable you pass to PYTHON_EXECUTABLE if you're using CMake).
The only things that you should change is to not use sys.path but rather site.addsitedir, since sys.path will not handle everything (e.g., .pth files), while site.addsitedir will, so simply:
py::module_::import("site").attr("addsitedir")(/* whatever */);
Related
When installing packages with sudo apt-get install or building libraries from source inside a python virtual environment (I am not talking about pip install), does doing it inside a python virtual environment isolate the applications being installed? I mean do they exist only inside the python virtual environment?
Things that a virtual environment gives you an isolated version of:
You get a separate PATH entry, so unqualified command-line references to python, pip, etc., will refer to the selected Python distribution. This can be convenient if you have many copies of Python installed on the system (common on developer workstations). This means that a shebang line like #!/usr/bin/env python will "do the right thing" inside of a virtualenv (on a Unix or Unix-like system, at least).
You get a separate site-packages directory, so Python packages (installed using pip or built locally inside this environment using e.g. setup.py build) are installed locally to the virtualenv and not in a system-wide location. This is especially useful on systems where the core Python interpreter is installed in a place where unprivileged users are not allowed to write files, as it allows each user to have their own private virtualenvs with third-party packages installed, without needing to use sudo or equivalent to install those third-party packages system-wide.
... and that's about it.
A virtual environment will not isolate you from:
Your operating system (Linux, Windows) or machine architecture (x86).
Scripts that reference a particular Python interpreter directly (e.g. #!/usr/bin/python).
Non-Python things on your system PATH (e.g. third party programs or utilities installed via your operating system's package manager).
Non-Python libraries or headers that are installed into a operating system specific location (e.g. /usr/lib, /usr/include, /usr/local/lib, /usr/local/include).
Python packages that are installed using the operating system's package manager (e.g. apt) rather than a Python package manager (pip) might not be visible from the the virtualenv's site-packages folder, but the "native" parts of such packages (in e.g. /usr/lib) will (probably) still be visible.
As per the comment by #deceze, virtual environments have no influence over apt operations.
When building from source, any compiled binaries will be linked to the python binaries of that environment. So if your virtualenv python version varies from the system version, and you use the system python (path problems usually), you can encounter runtime linking errors.
As for isolation, this same property (binary compatibility) isolates you from system upgrades which might change your system python binaries. Generally we're stable in the 2.x and 3.x, so it isn't likely to happen. But has, and can.
And of course, when building from source inside a virtualenv, installed packages are stashed in that virtualenv; no other python binary will have access to those packages, unless you are manipulating your path or PYTHONPATH in strange ways.
I use a Python virtual environment. Basically, it works fine, but I run into problems when compiling some Python bindings, namely with libIGL and pybind11.
CMake has the following Python-related variables:
PYTHON_EXECUTABLE /users/me/libs/pyvenv/bin/python
PYTHON_INCLUDE_DIR /usr/include/python2.7
PYTHON_LIBRARY /usr/lib64/libpython2.7.so
It seems that it can detect the executable which is a python3.5 of the previously-activated virtual environment properly, but it finds some wrong 2.7 paths for include and library.
So I'd like to just set those paths manually to my virtual environment. I browsed around in the directory structure of the virtual environment, and I think I found the includes in /users/me/libs/pyvenv/include/python3.5m. But I can't find the libpython*, there's no *.so file at all in my virtual environment. So which library should I use in that case?
Restrict python libs to match version of found interpreter in cmake:
find_package(PythonInterp REQUIRED)
find_package(PythonLibs "${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}" REQUIRED )
Or use FindPython if cmake>=3.12 is availble
Similarly to this question, you could run python from within your virtualenv and run
import pybind11
print(pybind11.__file__)
# '/home/me/.pyenv/versions/py36/lib/python3.6/site-packages/pybind11/__init__.py'
I have a python package that will depend on a significant number of external, 3rdparty shared libraries. It seems as if the convention is to copy the shared libraries to Library/bin (at least on windows using anaconda). I would prefer to keep the extra shared libraries needed for my package in a separate, package specific folder. Is there an (easy) way to do this with pip or conda or something else?
My first cut at this will be on windows, but linux will be next, so I would like an approach that will work similarly on both platforms.
Also, I am aware of and use virtual environments. But I am looking for a way to isolate the shared libraries needed specifically for one module/package from the other libraries within a virtual environment if possible and not use a separate virtual environment.
Create a virtual environment using the venv command:
Official Python Docs for venv
This will allow you to create a new python environment which you can configure using the conventional methods you've already found.
But it will also leave your main python environment unpolluted by any changes you make in that virtual environment.
I recently learned how to use virtualenv and virtualenvwrapper in my workflow but I've seen pyenv mentioned in a few guides but I can't seem to get an understanding of what pyenv is and how it is different/similar to virtualenv. Is pyenv a better/newer replacement for virtualenv or a complimentary tool? If the latter what does it do differently and how do the two (and virtualenvwrapper if applicable) work together?
Pyenv and virtualenv are very different tools that work in different ways to do different things:
Pyenv is a bash extension - will not work on Windows - that intercepts your calls to python, pip, etc., to direct them to one of several of the system python tool-chains. So you always have all the libraries that you have installed in the selected python version available - as such it is good for users who have to switch between different versions of python.
VirtualEnv, is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact your module.
venv python > 3.3
Note that from Python 3.3 onward there is a built in implementation of VirtualEnv called venv (with, on some installations a wrapper called pyvenv - this wrapper is deprecated in Python 3.6), which should probably be used in preference. To avoid possible issues with the wrapper it is often a good idea to use it directly by using /path/to/python3 -m venv desired/env/path or you can use the excellent py python selector on windows with py -3 -m venv desired/env/path. It will create the directory specified with desired/env/path configure and populate it appropriately. In general it is very much like using VirtualEnv.
Additional Tools
There are a number of tools that it is worth mentioning, and considering, as they can help with the use of one or more of the above:
VirtualEnvWrapper Manage and simplify the use and management of VirtualEnv - Cross Platform.
pyenv-virtualenv, installed by pyenv-installer, which gives PyEnv tools for managing and interfacing to VirtualEnv - with this you can have a base installation that includes more than one version of python and create isolated environments within each of them - Linux/OS-X. Suggested by Johann Visagie
PyInstaller can take your python code, possibly developed & tested under VirtualEnv, and bundle it up so that it can run one platforms that do not have your version of python installed - Note that it is not a cross compiler you will need a Windows (virtual-)machine to build Windows installs, etc., but it can be handy even where you can be sure that python will be installed but cannot be sure that the version of python and all the libraries will be compatible with your code.
Short version:
virtualenv allows you to create local (per-directory), independent python installations by cloning from existing ones
pyenv allows you to install (build from source) different versions of Python alongside each other; you can then clone them with virtualenv or use pyenv to select which one to run at any given time
Longer version:
Virtualenv allows you to create a custom Python installation e.g. in a subdirectory of your project. This is done by cloning from an existing Python installation somewhere on your system (some files are copied, some are reused/shared to save space). Each of your projects can thus have their own python (or even several) under their respective virtualenv. It is perfectly fine for some/all virtualenvs to even have the same version of python (e.g. 3.8.5) without conflict - they live separately and don't know about each other. If you want to use any of those pythons from shell, you have to activate it (by running a script which will temporarily modify your PATH to ensure that that virtualenv's bin/ directory comes first). From that point, calling python (or pip etc.) will invoke that virtualenv's version until you deactivate it (which restores the PATH). It is also possible to call into a virtualenv Python using its absolute path - this can be useful e.g. when invoking Python from a script.
Pyenv operates on a wider scale than virtualenv. It is used to install (build from source) arbitrary versions of Python (it holds a register of available versions). By default, they're all installed alongside each other under ~/.pyenv, so they're "more global" than virtualenv. Then, it allows you to configure which version of Python to run when you use the python command (without virtualenv). This can be done at a global level or, separately, per directory (by placing a .python-version file in a directory). It's done by prepending pyenv's shim python script to your PATH (permanently, unlike in virtualenv) which then decides which "real" python to invoke. You can even configure pyenv to call into one of your virtualenv pythons (by using the pyenv-virtualenv plugin). You can also duplicate Python versions (by giving them different names) and let them diverge.
Using pyenv can be a convenient way of installing Python for subsequent virtualenv use.
Background
Windows does not include a compiler by default, and installing a compiler (and perhaps configuring Python to use it) is a complicated enough task that many developers avoid doing so. To this end, many packages that have binary dependencies are available as a precompiled Windows executable that contains binary files. As an example, there is psycopg.
The executable file is an installer. When executed, it provides a graphical interface that locates an installed version of Python via the registry, and it installs the Python library and the included binary dependencies in the global Python installation.
However, this is not always desirable. Particularly when using virtualenv, developers don't want to install the library globally. They want the library installed in the virtual environment. Since this environment is not represented in the registry, the graphical installer cannot locate it. Luckily, a command similar to the following can be used to install the library to the virtual environment:
C:\> C:\virtualenv\Scripts\activate.bat
(virtualenv) C:\> easy_install psycopg2-2.5.win32-py2.7-pg9.2.4-release.exe
Note that this works regardless of whether easy_install comes from setuptools or distribute.
The actual question
Why does this command work? What is it about the exe that allows easy_install to process it?
I have noticed that the exe seems to be some kind of zip file. 7-Zip is able to open it for browsing, and these exe files that easy_install can process seem to have a common file structure. They have a top directory named PLATLIB which contains an egg-info file or folder and another (possibly more than 1?) folder. Are these exes just Python eggs with some kind of executable wrapped around them? How might I produce one myself? (Or to word it differently, is there some standard way of producing exes like this?)
Edit
Bonus question: Why doesn't pip work with these files?