I have a python package that will depend on a significant number of external, 3rdparty shared libraries. It seems as if the convention is to copy the shared libraries to Library/bin (at least on windows using anaconda). I would prefer to keep the extra shared libraries needed for my package in a separate, package specific folder. Is there an (easy) way to do this with pip or conda or something else?
My first cut at this will be on windows, but linux will be next, so I would like an approach that will work similarly on both platforms.
Also, I am aware of and use virtual environments. But I am looking for a way to isolate the shared libraries needed specifically for one module/package from the other libraries within a virtual environment if possible and not use a separate virtual environment.
Create a virtual environment using the venv command:
Official Python Docs for venv
This will allow you to create a new python environment which you can configure using the conventional methods you've already found.
But it will also leave your main python environment unpolluted by any changes you make in that virtual environment.
Related
My goal is to embed the python interpreter using PyBind11, but to use the interpreter from a virtual env, such that installing dependencies using pip does not clutter the system paths.
There is not much information online around this topic. Embedding python with pybind11. Virtual environment doesn't work hardcodes the venv at compile time. This is insufficient, since the venv does not exist at that time, but will be created when the scripting engine is enabled at runtime.
The plan for now is to pip install --target into a cache dir and add it to the sys.path. Using the system interpreter. This is "okayish" but not using the system interpreter would be peferable.
You do not need a virtual environment to ship isolated dependencies with your software, or to use them locally, you simply need what you already have: an isolated lib folder.
pybind11 does not have its own interpreter (AFAIK), it embeds the one from the Python build you're using when compiling, so you should already have a matching version of the Python interpreter you're embedding (this is the executable you pass to PYTHON_EXECUTABLE if you're using CMake).
The only things that you should change is to not use sys.path but rather site.addsitedir, since sys.path will not handle everything (e.g., .pth files), while site.addsitedir will, so simply:
py::module_::import("site").attr("addsitedir")(/* whatever */);
I'm working on a script in python that relies on several different packages and libraries. When this script is transferred to another machine, the packages it needs in order to run are sometimes not present or are older versions that do not have the same functionality and cause the script to fail.
I was considering using a virtual environment, but I can't find a way to have the script use the specific environment I design as it's default, and in order to use the environment a user must manually activate it from the command line.
I've also looked into trying to check the versions of the packages installed on the machine, and if they are not sufficient then updating them from the script as described here:
Installing python module within code
Is there any easier/surefire way to make sure that the needed packages will always be available regardless of where it's run?
The normal approach is to create an installation script and have that manage your dependencies. Then when you move your project to a new environment your installer will check that all dependencies are present.
I recommend you check out setuptools: https://setuptools.readthedocs.io/en/latest/
If you don't want to install dependencies whenever you need to use your script somewhere new, then you could package your script into a Docker container.
If the problem is ensuring the required packages are available in a new environment or virtual environment, you could use pip and generate a requirements.txt and check it in version control or use a tool to do that for you, like pipenv.
If you would prefer to generate a requirements.txt by hand, you should:
Install your depencencies using pip
Type pip freeze > requirements.txt to generate a requirements.txt file
Check requirements.txt in you source management software
When you need to setup a new environment, use pip install -m requirements.txt
The solution that I've been using has been to include a custom library (folder with all of my desired packages) in the folder with my script, and I simply import them from there:
from Customlib import pkg1, pkg2,...
As long as the custom library and script stay together in the same folder, it will always have access to the right packages and the correct versions of those packages.
I'm not sure how robust this solution actually is or what possible bugs may arise from this if it is passed from machine to machine, but for now this seems to work.
When installing packages with sudo apt-get install or building libraries from source inside a python virtual environment (I am not talking about pip install), does doing it inside a python virtual environment isolate the applications being installed? I mean do they exist only inside the python virtual environment?
Things that a virtual environment gives you an isolated version of:
You get a separate PATH entry, so unqualified command-line references to python, pip, etc., will refer to the selected Python distribution. This can be convenient if you have many copies of Python installed on the system (common on developer workstations). This means that a shebang line like #!/usr/bin/env python will "do the right thing" inside of a virtualenv (on a Unix or Unix-like system, at least).
You get a separate site-packages directory, so Python packages (installed using pip or built locally inside this environment using e.g. setup.py build) are installed locally to the virtualenv and not in a system-wide location. This is especially useful on systems where the core Python interpreter is installed in a place where unprivileged users are not allowed to write files, as it allows each user to have their own private virtualenvs with third-party packages installed, without needing to use sudo or equivalent to install those third-party packages system-wide.
... and that's about it.
A virtual environment will not isolate you from:
Your operating system (Linux, Windows) or machine architecture (x86).
Scripts that reference a particular Python interpreter directly (e.g. #!/usr/bin/python).
Non-Python things on your system PATH (e.g. third party programs or utilities installed via your operating system's package manager).
Non-Python libraries or headers that are installed into a operating system specific location (e.g. /usr/lib, /usr/include, /usr/local/lib, /usr/local/include).
Python packages that are installed using the operating system's package manager (e.g. apt) rather than a Python package manager (pip) might not be visible from the the virtualenv's site-packages folder, but the "native" parts of such packages (in e.g. /usr/lib) will (probably) still be visible.
As per the comment by #deceze, virtual environments have no influence over apt operations.
When building from source, any compiled binaries will be linked to the python binaries of that environment. So if your virtualenv python version varies from the system version, and you use the system python (path problems usually), you can encounter runtime linking errors.
As for isolation, this same property (binary compatibility) isolates you from system upgrades which might change your system python binaries. Generally we're stable in the 2.x and 3.x, so it isn't likely to happen. But has, and can.
And of course, when building from source inside a virtualenv, installed packages are stashed in that virtualenv; no other python binary will have access to those packages, unless you are manipulating your path or PYTHONPATH in strange ways.
I use python virtual environments regularly and generally execute the standard method of installing any necessary site packages after running installing the virtual environment. However, when I don't want to use all of the original environment's site-packages then in order to install a number of additional packages to the new environment I find the process tedious, especially when non pypi pacakges are required for my work. I haven't managed to find a means of moving a global site package into an existing virtual environment. The work flow for this is as follows
I'd like to be able to first create a virtual environment from a full python installation, without using the --system-site-packagesoption. Then, I'd like to be able to selectively move site-pacakges over to the new virtual environment.
This is, of course, something that I can get by without, but it would be a convenience if it is possible.
I recently learned how to use virtualenv and virtualenvwrapper in my workflow but I've seen pyenv mentioned in a few guides but I can't seem to get an understanding of what pyenv is and how it is different/similar to virtualenv. Is pyenv a better/newer replacement for virtualenv or a complimentary tool? If the latter what does it do differently and how do the two (and virtualenvwrapper if applicable) work together?
Pyenv and virtualenv are very different tools that work in different ways to do different things:
Pyenv is a bash extension - will not work on Windows - that intercepts your calls to python, pip, etc., to direct them to one of several of the system python tool-chains. So you always have all the libraries that you have installed in the selected python version available - as such it is good for users who have to switch between different versions of python.
VirtualEnv, is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact your module.
venv python > 3.3
Note that from Python 3.3 onward there is a built in implementation of VirtualEnv called venv (with, on some installations a wrapper called pyvenv - this wrapper is deprecated in Python 3.6), which should probably be used in preference. To avoid possible issues with the wrapper it is often a good idea to use it directly by using /path/to/python3 -m venv desired/env/path or you can use the excellent py python selector on windows with py -3 -m venv desired/env/path. It will create the directory specified with desired/env/path configure and populate it appropriately. In general it is very much like using VirtualEnv.
Additional Tools
There are a number of tools that it is worth mentioning, and considering, as they can help with the use of one or more of the above:
VirtualEnvWrapper Manage and simplify the use and management of VirtualEnv - Cross Platform.
pyenv-virtualenv, installed by pyenv-installer, which gives PyEnv tools for managing and interfacing to VirtualEnv - with this you can have a base installation that includes more than one version of python and create isolated environments within each of them - Linux/OS-X. Suggested by Johann Visagie
PyInstaller can take your python code, possibly developed & tested under VirtualEnv, and bundle it up so that it can run one platforms that do not have your version of python installed - Note that it is not a cross compiler you will need a Windows (virtual-)machine to build Windows installs, etc., but it can be handy even where you can be sure that python will be installed but cannot be sure that the version of python and all the libraries will be compatible with your code.
Short version:
virtualenv allows you to create local (per-directory), independent python installations by cloning from existing ones
pyenv allows you to install (build from source) different versions of Python alongside each other; you can then clone them with virtualenv or use pyenv to select which one to run at any given time
Longer version:
Virtualenv allows you to create a custom Python installation e.g. in a subdirectory of your project. This is done by cloning from an existing Python installation somewhere on your system (some files are copied, some are reused/shared to save space). Each of your projects can thus have their own python (or even several) under their respective virtualenv. It is perfectly fine for some/all virtualenvs to even have the same version of python (e.g. 3.8.5) without conflict - they live separately and don't know about each other. If you want to use any of those pythons from shell, you have to activate it (by running a script which will temporarily modify your PATH to ensure that that virtualenv's bin/ directory comes first). From that point, calling python (or pip etc.) will invoke that virtualenv's version until you deactivate it (which restores the PATH). It is also possible to call into a virtualenv Python using its absolute path - this can be useful e.g. when invoking Python from a script.
Pyenv operates on a wider scale than virtualenv. It is used to install (build from source) arbitrary versions of Python (it holds a register of available versions). By default, they're all installed alongside each other under ~/.pyenv, so they're "more global" than virtualenv. Then, it allows you to configure which version of Python to run when you use the python command (without virtualenv). This can be done at a global level or, separately, per directory (by placing a .python-version file in a directory). It's done by prepending pyenv's shim python script to your PATH (permanently, unlike in virtualenv) which then decides which "real" python to invoke. You can even configure pyenv to call into one of your virtualenv pythons (by using the pyenv-virtualenv plugin). You can also duplicate Python versions (by giving them different names) and let them diverge.
Using pyenv can be a convenient way of installing Python for subsequent virtualenv use.