what 'operations' does a python virtual enviornement really isolate? - python

When installing packages with sudo apt-get install or building libraries from source inside a python virtual environment (I am not talking about pip install), does doing it inside a python virtual environment isolate the applications being installed? I mean do they exist only inside the python virtual environment?

Things that a virtual environment gives you an isolated version of:
You get a separate PATH entry, so unqualified command-line references to python, pip, etc., will refer to the selected Python distribution. This can be convenient if you have many copies of Python installed on the system (common on developer workstations). This means that a shebang line like #!/usr/bin/env python will "do the right thing" inside of a virtualenv (on a Unix or Unix-like system, at least).
You get a separate site-packages directory, so Python packages (installed using pip or built locally inside this environment using e.g. setup.py build) are installed locally to the virtualenv and not in a system-wide location. This is especially useful on systems where the core Python interpreter is installed in a place where unprivileged users are not allowed to write files, as it allows each user to have their own private virtualenvs with third-party packages installed, without needing to use sudo or equivalent to install those third-party packages system-wide.
... and that's about it.
A virtual environment will not isolate you from:
Your operating system (Linux, Windows) or machine architecture (x86).
Scripts that reference a particular Python interpreter directly (e.g. #!/usr/bin/python).
Non-Python things on your system PATH (e.g. third party programs or utilities installed via your operating system's package manager).
Non-Python libraries or headers that are installed into a operating system specific location (e.g. /usr/lib, /usr/include, /usr/local/lib, /usr/local/include).
Python packages that are installed using the operating system's package manager (e.g. apt) rather than a Python package manager (pip) might not be visible from the the virtualenv's site-packages folder, but the "native" parts of such packages (in e.g. /usr/lib) will (probably) still be visible.

As per the comment by #deceze, virtual environments have no influence over apt operations.
When building from source, any compiled binaries will be linked to the python binaries of that environment. So if your virtualenv python version varies from the system version, and you use the system python (path problems usually), you can encounter runtime linking errors.
As for isolation, this same property (binary compatibility) isolates you from system upgrades which might change your system python binaries. Generally we're stable in the 2.x and 3.x, so it isn't likely to happen. But has, and can.
And of course, when building from source inside a virtualenv, installed packages are stashed in that virtualenv; no other python binary will have access to those packages, unless you are manipulating your path or PYTHONPATH in strange ways.

Related

Using PyBind11 with a virtual environment created at runtime

My goal is to embed the python interpreter using PyBind11, but to use the interpreter from a virtual env, such that installing dependencies using pip does not clutter the system paths.
There is not much information online around this topic. Embedding python with pybind11. Virtual environment doesn't work hardcodes the venv at compile time. This is insufficient, since the venv does not exist at that time, but will be created when the scripting engine is enabled at runtime.
The plan for now is to pip install --target into a cache dir and add it to the sys.path. Using the system interpreter. This is "okayish" but not using the system interpreter would be peferable.
You do not need a virtual environment to ship isolated dependencies with your software, or to use them locally, you simply need what you already have: an isolated lib folder.
pybind11 does not have its own interpreter (AFAIK), it embeds the one from the Python build you're using when compiling, so you should already have a matching version of the Python interpreter you're embedding (this is the executable you pass to PYTHON_EXECUTABLE if you're using CMake).
The only things that you should change is to not use sys.path but rather site.addsitedir, since sys.path will not handle everything (e.g., .pth files), while site.addsitedir will, so simply:
py::module_::import("site").attr("addsitedir")(/* whatever */);

Does miniconda installation affect standard python installation?

I had first installed python using the standard python distribution available on their official website and I would be using pip to install all necessary packages.
However, now I wish to use miniconda, since it is a better choice for data science.
But, it installs python along with It and I don't want to disturb my earlier setup of pip+Python.
Will installing miniconda affect my python installation.
Is there a way of installing it without disturbing the python installation?
I am on a Windows operating system.
You can safely install Anaconada (or Miniconda) on top of other Python installations. It goes into a completely different folder on your local disk. But leave the default installation options on default, especially don't add Python to your path.
The important thing is that you activate your environment before you use it via
conda activate
and then start Python from there (or let your IDE do that for you).
(base)> python
Without activatation conda doesn't work and calling python from the command prompt will start your 'standard installation' again.
The advantage of Anaconda is that it guarantees maximumum consistency for the 'scientific stack' and in case you are still missing some 3rd party packages you can always install them aditionally via `pip install' into an activated conda environment.

install flask for python2 when python 3 is default

I am trying to install flask and a few other modules for Python2.
When I try to install them using command pip install flask, it installs these for Python3.
This has created major issues because things like django are not compatible with Python3.
When I want to run a program using Python2, I cant use any of these modules.
Question
How do I use pip to install modules into a specified version of Python?
Try:
python2.7 -m pip install flask
Python “Virtual Environments” allow Python packages to be installed in an isolated location for a particular application, rather than being installed globally.
Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtual environments can help you. They have their own installation directories and they don’t share libraries with other virtual environments.
Currently, there are two common tools for creating Python virtual environments:
venv is available by default in Python 3.3 and later, and installs pip and setuptools into created virtual environments in Python 3.4 and later.
virtualenv needs to be installed separately, but supports Python 2.6+ and Python 3.3+, and pip, setuptools and wheel are always installed into created virtual environments by default (regardless of Python version).
The basic usage is like so:
Using virtualenv:
virtualenv <DIR>
source <DIR>/bin/activate
Using venv:
python3 -m venv <DIR>
source <DIR>/bin/activate
For more information, see the virtualenv docs or the venv docs.
Managing multiple virtual environments directly can become tedious, so the dependency management tutorial introduces a higher level tool, Pipenv, that automatically manages a separate virtual environment for each project and application that you work on.
https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments

Do anaconda packages interfere with system python

I have a system with certain python version and packages installed suing the distribution repositories. For some project (calculation) I need newer version the the packages. I am thinking of installing anaconda and use conda virtual environments. Will this broke programs that must use the system packages?
(note: I tried virtual enviroment, but I couldn't install a newver version of matplotlib, because of problems with pygtk)
No this will not break your system's python. As long as you don't tick the option "register miniconda as the default system python" (or whatever that option is called depending on your OS).
One of the key benefits of conda is that you can create isolated python environments, fully independent of each other.

What is the relationship between virtualenv and pyenv?

I recently learned how to use virtualenv and virtualenvwrapper in my workflow but I've seen pyenv mentioned in a few guides but I can't seem to get an understanding of what pyenv is and how it is different/similar to virtualenv. Is pyenv a better/newer replacement for virtualenv or a complimentary tool? If the latter what does it do differently and how do the two (and virtualenvwrapper if applicable) work together?
Pyenv and virtualenv are very different tools that work in different ways to do different things:
Pyenv is a bash extension - will not work on Windows - that intercepts your calls to python, pip, etc., to direct them to one of several of the system python tool-chains. So you always have all the libraries that you have installed in the selected python version available - as such it is good for users who have to switch between different versions of python.
VirtualEnv, is pure python so works everywhere, it makes a copy of, optionally a specific version of, python and pip local to the activate environment which may or may not include links to the current system tool-chain, if it does not you can install just a known subset of libraries into that environment. As such it is almost certainly much better for testing and deployment as you know exactly which libraries, at which versions, are used and a global change will not impact your module.
venv python > 3.3
Note that from Python 3.3 onward there is a built in implementation of VirtualEnv called venv (with, on some installations a wrapper called pyvenv - this wrapper is deprecated in Python 3.6), which should probably be used in preference. To avoid possible issues with the wrapper it is often a good idea to use it directly by using /path/to/python3 -m venv desired/env/path or you can use the excellent py python selector on windows with py -3 -m venv desired/env/path. It will create the directory specified with desired/env/path configure and populate it appropriately. In general it is very much like using VirtualEnv.
Additional Tools
There are a number of tools that it is worth mentioning, and considering, as they can help with the use of one or more of the above:
VirtualEnvWrapper Manage and simplify the use and management of VirtualEnv - Cross Platform.
pyenv-virtualenv, installed by pyenv-installer, which gives PyEnv tools for managing and interfacing to VirtualEnv - with this you can have a base installation that includes more than one version of python and create isolated environments within each of them - Linux/OS-X. Suggested by Johann Visagie
PyInstaller can take your python code, possibly developed & tested under VirtualEnv, and bundle it up so that it can run one platforms that do not have your version of python installed - Note that it is not a cross compiler you will need a Windows (virtual-)machine to build Windows installs, etc., but it can be handy even where you can be sure that python will be installed but cannot be sure that the version of python and all the libraries will be compatible with your code.
Short version:
virtualenv allows you to create local (per-directory), independent python installations by cloning from existing ones
pyenv allows you to install (build from source) different versions of Python alongside each other; you can then clone them with virtualenv or use pyenv to select which one to run at any given time
Longer version:
Virtualenv allows you to create a custom Python installation e.g. in a subdirectory of your project. This is done by cloning from an existing Python installation somewhere on your system (some files are copied, some are reused/shared to save space). Each of your projects can thus have their own python (or even several) under their respective virtualenv. It is perfectly fine for some/all virtualenvs to even have the same version of python (e.g. 3.8.5) without conflict - they live separately and don't know about each other. If you want to use any of those pythons from shell, you have to activate it (by running a script which will temporarily modify your PATH to ensure that that virtualenv's bin/ directory comes first). From that point, calling python (or pip etc.) will invoke that virtualenv's version until you deactivate it (which restores the PATH). It is also possible to call into a virtualenv Python using its absolute path - this can be useful e.g. when invoking Python from a script.
Pyenv operates on a wider scale than virtualenv. It is used to install (build from source) arbitrary versions of Python (it holds a register of available versions). By default, they're all installed alongside each other under ~/.pyenv, so they're "more global" than virtualenv. Then, it allows you to configure which version of Python to run when you use the python command (without virtualenv). This can be done at a global level or, separately, per directory (by placing a .python-version file in a directory). It's done by prepending pyenv's shim python script to your PATH (permanently, unlike in virtualenv) which then decides which "real" python to invoke. You can even configure pyenv to call into one of your virtualenv pythons (by using the pyenv-virtualenv plugin). You can also duplicate Python versions (by giving them different names) and let them diverge.
Using pyenv can be a convenient way of installing Python for subsequent virtualenv use.

Categories

Resources