Related
I'm working with a distributed package system (happens to be CVMFS but I don't think the particulars are relevant). The setup script for that environment adds some locations to PYTHONPATH.
Now, working in that environment, I want to install newer versions of some packages that are already found within that path. Installing the packages is easy, I can either do
python -m pip install --user --upgrade <packages>
or, my preferred approach, use a virtual environment
python -m venv myenv
source myenv/bin/activate
python -m pip install --upgrade <packages>
Installation works fine. But then when I try to run python, the directories in PYTHONPATH are search before either my user-site directory or the virtual environment, and I get the old versions of packages.
Is there any way to force packages in my virtual environment or user-site path to have priority, without having to manually edit sys.path or PYTHONPATH?
I've created a Python virtual environment, and activate it by doing:
joe#joe-mint $ source ./venvs/deep-learning/bin/activate
Which turns the prompt into:
(deep-learning) joe#joe-mint $
Now whenever I run a python package or try to install one, the system seems to ignore the fact that it's in a virtual environment and does things system-wide:
(deep-learning) joe#joe-mint $ which pip
/usr/local/bin/pip
The same happens when I try to install new packages that aren't on my system; it installs them to the system files (i.e. /usr/bin) instead of the virtual environment.
What's wrong with my virtual environment? How do I get it to ignore system files and do everything inside the environment?
I've looked at this question which says to use an explicit flag when creating the virtual environment to make it use the local environment packages, but I used python-3.5 -m venv to create the virtual environment, and this flag is removed in this version as it's now a default option.
I've also looked at this question and can confirm that the VIRTUAL_ENV variable is set correctly in the activate file of the virtual environment.
Here was the problem:
It seems that if you run pip on a venv without a local pip installation, then it will default to the system's pip outside the venv. Even if you've activated a virtual environment, this seems to want to install packages on the system rather than in the venv.
Here was the solution:
First, I had to install the virtual environment without pip due to a bug that has long remained unresolved.
Second, I installed pip in the virtual environment as per the instruction here. However, doing so required using some temporary folders that for some reason my user didn't have access to. So this failed, and the only way I could get it to work was to become root.
sudo su
activate ..../venvs/deep-learning/bin/activate to activate the virtual environment.
curl --silent --show-error --retry 5 https://bootstrap.pypa.io/get-pip.py | python as per the answer linked above.
Although which pip now indicated the correct pip (inside the venv) was being used, running pip would use the system one! Deactivating (deactivate) and reactivating the venv solved this.
Now it took me a while to realise that having installed this as root caused me permission errors when trying to install more packages using pip inside the virtual environment.
chown <user>:<group> -R ..../venvs/deep-learning/*
And that was it. After these steps, I could activate the venv and run pip correctly. It would use the pip inside the venv, and install packages inside the venv.
As we all know, Mac OS ships with its own python pre installed.
The recommendation seems to be to leave that alone and use homebrew to install a fresh python into the system.
My issue is that after installing python (and pip) using homebrew, pip is installing packages into the Mac OS site-packages instead of my own. I have confirmed I am running the "homebrew" pip:
$ which pip
/usr/local/bin/pip
But then when I pip install something I can se it is installed at:
/lib/python2.7/site-packages
Pip should be installing at /usr/local/lib/python2.7/site-packages unless i'm miss understanding something.
The surprising thing is that checking with -V yields a surprising result:
pip -V
pip 7.1.0 from /usr/local/lib/python2.7/site-packages (python 2.7)
Running pip list just after running pip install does not show the packages that were supposedly just installed by it but went to the wrong site-packages.
Adding to this, the packages installed on the /lib/python2.7/site-packages are not recognized by my $PYTHONPATH and as such I cannot use them.
To add even more confusion, I decided to use a virtualenv, but I was amazed as even using pip with the virtualenv active kept installing to the /lib/python2.7/site-packages instead of the virtualenv site-packages.
So, somehow I ended up with a homebrew pip, that installs packages outside of the homebrew site-packages and a python interpreter that cant use the packages installed by pip.
How do you recommend I go about finding the root cause and having a smooth python experience? :)
You can easily find you site-packages directory by invoking this command
python -c 'import site; print(site.getsitepackages())'
I think after you activate a virtualenv your python path should point to that environments site-package location--if not it's probably not activated. Only once you activate it will you run pip so it installs in that virtual env's site-packages. if it's not activated, it will go in whatever other site-packages it already knows about:
Step 1: create a virtual env
a la... virtualenv venv
Do this only once!
Step 2: Activate the vitual env
something like source /venv/bin/activate
Needs doing every time you want to use this virtual environment
Step 3: run pip commands, watch them get installed in the virtual env site-packages!
If you do step 3 before step 2 your not actually using the virtual environment you created, so all bets are off--That's probably the reason pip is still installing to the old location.
Now, my overall recommendation is to go further and use pyenv to install specific version of python into your /Users/username/.pyenv folder and abandon both the default OSX and homebrew packages. It's simple and you can control easily the exact version of python to use by simple issuing of command to change versions.
THEN use virtualenv in python2 or pyvenv if in python3 (not to be confused with pyenv) to build vitual environments with their own local site-packages to store pip modules. When you activate a virtualenv, your $PYTHONPATH will switch to the specific location.
The flow would then be:
Use pyenv to pull down and switch to a specific version of python you want to use--overriding homebrew and the OSX version.
Create your vitrualenv. This will create a bin that will link to the pyenv python stack you just specified in the previous step.
Activate the virtual env, and proceed.
Totally control your environment!
For one, you could try updating pip with pip install --upgrade pip command, which might or might not redirect your pip path.
Two, and I should have really started with this one is to set the preferred pip executable path in the .bash_profile or .zshrc if you are using one. The way you do it (on Mac) is by holding Shift+Command+Period to reveal hidden files, going to the User folder and opening the .bash_profile/.zshrc with a text editor. Afterward, add the path/to/bin where the pip that you require is. Like export PATH="User/Username/anaconda3/bin:$PATH" or /usr/local/bin or path/to/venv/bin. Whatever code you write in the end will overwrite the previous one.
Three, if you don't wanna change your default pip, but rather wanna use a different version for that specific case just include the full path of the pip executable like /usr/local/bin/pip list or Users/Username/Desktop/venv/bin/pip install module.
Using virtualenv, I run my projects with the default version of Python (2.7). On one project, I need to use Python 3.4.
I used brew install python3 to install it on my Mac. Now, how do I create a virtualenv that uses the new version?
e.g. sudo virtualenv envPython3
If I try:
virtualenv -p python3 test
I get:
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.4.0_1/Frameworks/Python.framework/Versions/3.4'
New python executable in test/bin/python3.4
Also creating executable in test/bin/python
Failed to import the site module
Traceback (most recent call last):
File "/Users/user/Documents/workspace/test/test/bin/../lib/python3.4/site.py", line 67, in <module>
import os
File "/Users/user/Documents/workspace/test/test/bin/../lib/python3.4/os.py", line 634, in <module>
from _collections_abc import MutableMapping
ImportError: No module named '_collections_abc'
ERROR: The executable test/bin/python3.4 is not functioning
ERROR: It thinks sys.prefix is '/Users/user/Documents/workspace/test' (should be '/Users/user/Documents/workspace/test/test')
ERROR: virtualenv is not compatible with this system or executable
simply run
virtualenv -p python3 envname
Update after OP's edit:
There was a bug in the OP's version of virtualenv, as described here. The problem was fixed by running:
pip install --upgrade virtualenv
Python 3 has a built-in support for virtual environments - venv. It might be better to use that instead. Referring to the docs:
Creation of virtual environments is done by executing the pyvenv
script:
pyvenv /path/to/new/virtual/environment
Update for Python 3.6 and newer:
As pawciobiel correctly comments, pyvenv is deprecated as of Python 3.6 and the new way is:
python3 -m venv /path/to/new/virtual/environment
I'v tried pyenv and it's very handy for switching python versions (global, local in folder or in the virtualenv):
brew install pyenv
then install Python version you want:
pyenv install 3.5.0
and simply create virtualenv with path to needed interpreter version:
virtualenv -p /Users/johnny/.pyenv/versions/3.5.0/bin/python3.5 myenv
That's it, check the version:
. ./myenv/bin/activate && python -V
There are also plugin for pyenv pyenv-virtualenv but it didn't work for me somehow.
Install prerequisites.
sudo apt-get install python3 python3-pip virtualenvwrapper
Create a Python3 based virtual environment. Optionally enable --system-site-packages flag.
mkvirtualenv -p /usr/bin/python3 <venv-name>
Set into the virtual environment.
workon <venv-name>
Install other requirements using pip package manager.
pip install -r requirements.txt
pip install <package_name>
When working on multiple python projects simultaneously it is usually recommended to install common packages like pdbpp globally and then reuse them in virtualenvs.
Using this technique saves a lot of time spent on fetching packages and installing them, apart from consuming minimal disk space and network bandwidth.
sudo -H pip3 -v install pdbpp
mkvirtualenv -p $(which python3) --system-site-packages <venv-name>
Django specific instructions
If there are a lot of system wide python packages then it is recommended to not use --system-site-packages flag especially during development since I have noticed that it slows down Django startup a lot. I presume Django environment initialisation is manually scanning and appending all site packages from the system path which might be the reason. Even python manage.py shell becomes very slow.
Having said that experiment which option works better. Might be safe to just skip --system-site-packages flag for Django projects.
virtualenv --python=/usr/bin/python3 <name of env>
worked for me.
This is all you need, in order to run a virtual environment in python / python3
First if virtualenv not installed, run
pip3 install virtualenv
Now Run:
virtualenv -p python3 <env name> # you can specify full path instead <env_name> to install the files in a different location other than the current location
Sometime the cmd virtualenv fails, if so use this:
python3 -m virtualenv <env_name> # you can specify full path instead <env_name> to install the files in a different location other than the current location
Now activate the virtual env:
source <env_name>/bin/activate
Or:
source `pwd`/<env_name>/bin/activate
Now run
which python
You should see the full path to your dir and <env_name>/bin/python suffix
To exit the virtualenv, run:
deactivate
To troubleshoot Python location got to here
You can specify specific Version of Python while creating environment.
It's mentioned in virtualenv.py
virtualenv --python=python3.5 envname
In some cases this has to be the full path to the executable:
virtualenv --python=/Users/username/.pyenv/versions/3.6.0/bin/python3.6 envname
How -p works
parser.add_option(
'-p', '--python',
dest='python',
metavar='PYTHON_EXE',
help='The Python interpreter to use, e.g., --python=python3.5 will use the python3.5 '
'interpreter to create the new environment. The default is the interpreter that '
'virtualenv was installed with (%s)' % sys.executable)
I had the same ERROR message. tbrisker's solution did not work in my case. Instead this solved the issue:
$ python3 -m venv .env
In addition to the other answers, I recommend checking what instance of virtualenv you are executing:
which virtualenv
If this turns up something in /usr/local/bin, then it is possible - even likely - that you installed virtualenv (possibly using an instance of easy_tools or pip) without using your system's package manager (brew in OP's case). This was my problem.
Years ago - when I was even more ignorant - I had installed virtualenv and it was masking my system's package-provided virtualenv.
After removing this old, broken virtualenv, my problems went away.
The below simple commands can create a virtual env with version 3.5
apt-get install python3-venv
python3.5 -m venv <your env name>
if you want virtual env version as 3.6
python3.6 -m venv <your env name>
Python now comes with its own implementation of virtual environment, by the name of "venv". I would suggest using that, instead of virtualenv.
Quoting from venv - docs,
Deprecated since version 3.6: pyvenv was the recommended tool for
creating virtual environments for Python 3.3 and 3.4, and is
deprecated in Python 3.6.
Changed in version 3.5: The use of venv is now recommended for
creating virtual environments.
For windows, to initiate venv on some project, open cmd:
python -m venv "c:\path\to\myenv"
(Would suggest using double quote around directory path if it contains any spaces. Ex: "C:/My Dox/Spaced Directory/Something")
Once venv is set up, you will see some new folders inside your project directory. One of them would be "Scripts".
To activate or invoke venv you need:
C:\> <venv>\Scripts\activate.bat
You can deactivate a virtual environment by typing “deactivate” in your shell. With this, you are now ready to install your project specific libraries, which will reside under the folder "Lib".
================================ Edit 1 ====================================
The scenario which will be discussed below is not what originally asked, just adding this in case someone use vscode with python extension
In case, you use vs code with its python extension, you might face an issue with its pylint which points to the global installation. In this case, pylint won't be able to see the modules that are installed in your virtual environment and hence will show errors while importing.
Here is a simple method to get past this.
cd Workspace\Scripts
.\Activate.ps1
code .
We are basically activating the environment first and then invoking vs-code so that pylint starts within the environment and can see all local packages.
In python3.6 I tried
python3 -m venv myenv,
as per the documentation, but it was taking so long. So the very simple and quick command is
python -m venv yourenv
It worked for me on python3.6.
On Mac I had to do the following to get it to work.
mkvirtualenv --python=/usr/bin/python3 YourEnvNameHere
If you install python3 (brew install python3) along with virtualenv burrito, you can then do mkvirtualenv -p $(which python3) env_name
Of course, I know virtualenv burrito is just a wrapper, but it has served me well over the years, reducing some learning curves.
virtualenv --python=/usr/local/bin/python3 <VIRTUAL ENV NAME>
this will add python3
path for your virtual enviroment.
It worked for me
virtualenv --no-site-packages --distribute -p /usr/bin/python3 ~/.virtualenvs/py3
For those having troubles while working with Anaconda3 (Python 3).
You could use
conda create -n name_of_your_virtualenv python=python_version
To activate the environment ( Linux, MacOS)
source activate name_of_your_virtualenv
For Windows
activate name_of_your_virtualenv
I tried all the above stuff, it still didn't work. So as a brute force, I just re-installed the anaconda, re-installed the virtualenv... and it worked.
Amans-MacBook-Pro:~ amanmadan$ pip install virtualenv
You are using pip version 6.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting virtualenv
Downloading virtualenv-15.0.3-py2.py3-none-any.whl (3.5MB)
100% |████████████████████████████████| 3.5MB 114kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-15.0.3
Amans-MacBook-Pro:python amanmadan$ virtualenv my_env
New python executable in /Users/amanmadan/Documents/HadoopStuff/python/my_env/bin/python
Installing setuptools, pip, wheel...done.
Amans-MacBook-Pro:python amanmadan$
I wanted to keep python 2.7.5 as default version on Centos 7 but have python 3.6.1 in a virtual environment running alongside other virtual environments in python 2.x
I found the below link the best solution for the newest python version ( python 3.6.1)
https://www.digitalocean.com/community/tutorial_series/how-to-install-and-set-up-a-local-programming-environment-for-python-3.
It shows the steps for different platforms but the basic steps are
Install python3.x (if not present) for your platform
Install python3.x-devel for your platform
Create virtual environment in python 3.x
(for example $ python3.6 -m venv virenv_test_p3/ )
Activate the testenvironment for python 3.x
(for example source virenv_test_p3/bin/activate)
Install the packages which you want to use in your new python 3 virtual environment and which are supported ( for example pip install Django==1.11.2)
On Windows command line, the following worked for me. First find out where your python executables are located:
where python
This will output the paths to the different python.exe on your system. Here were mine:
C:\Users\carandangc\Anaconda3\python.exe
C:\Python27\python.exe
So for Python3, this was located in the first path for me, so I cd to the root folder of the application where I want to create a virtual environment folder. Then I run the following which includes the path to my Python3 executable, naming my virtual environment 'venv':
virtualenv --python=/Users/carandangc/Anaconda3/python.exe venv
Next, activate the virtual environment:
call venv\Scripts\activate.bat
Finally, install the dependencies for this virtual environment:
pip install -r requirements.txt
This requirements.txt could be populated manually if you know the libraries/modules needed for your application in the virtual environment. If you had the application running in another environment, then you can automatically produce the dependencies by running the following (cd to the application folder in the environment where it is working):
pip freeze > requirements.txt
Then once you have the requirements.txt that you have 'frozen', then you can install the requirements on another machine or clean environment with the following (after cd to the application folder):
pip install -r requirements.txt
To see your python version in the virtual environment, run:
python --version
Then voila...you have your Python3 running in your virtual environment. Output for me:
Python 3.7.2
For those of you who are using pipenv and want to install specific version:
pipenv install --python 3.6
I got the same error due to it being a conflict with miniconda3 install so when you type "which virtualenv" and if you've installed miniconda and it's pointing to that install you can either remove it (if your like me and haven't moved to it yet) or change your environment variable to point to the install you want.
I created a virtualenv with the following command.
mkvirtualenv --distribute --system-site-packages "$1"
After starting the virtualenv with workon, I type ipython. It prompts me
WARNING: Attempting to work in a virtualenv. If you encounter problems, please install IPython inside the virtualenv.
When I try to install ipython with the virtualenv, I got the following error message:
pip install ipython
Requirement already satisfied (use --upgrade to upgrade): ipython in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Cleaning up...
Does anyone know how to install inside the virtualenv?
Avoiding Headaches and Best Practices:
Virtual Environments are not part of your git project (they don't need to be versioned) !
They can reside on the project folder (locally), but, ignored on your .gitignore.
After activating the virtual environment of your project, never "sudo pip install package".
After finishing your work, always "deactivate" your environment.
Avoid renaming your project folder.
For a better representation, here's a simulation:
creating a folder for your projects/environments
$ mkdir envs
creating environment
$ cd envs/
$ virtualenv google_drive
New python executable in google_drive/bin/python
Installing setuptools, pip...done.
activating environment
$ source google_drive/bin/activate
installing packages
(google_drive) $ pip install PyDrive
Downloading/unpacking PyDrive
Downloading PyDrive-1.3.1-py2-none-any.whl
...
...
...
Successfully installed PyDrive PyYAML google-api-python-client oauth2client six uritemplate httplib2 pyasn1 rsa pyasn1-modules
Cleaning up...
package available inside the environment
(google_drive) $ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pydrive.auth
>>>
>>> gdrive = pydrive.auth.GoogleAuth()
>>>
deactivate environment
(google_drive) $ deactivate
$
package NOT AVAILABLE outside the environment
$ python
Python 2.7.6 (default, Oct 26 2016, 20:32:10)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import pydrive.auth
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pydrive.auth
>>>
Notes:
Why not sudo?
Virtualenv creates a whole new environment for you, defining $PATH and some other variables and settings. When you use sudo pip install package, you are running Virtualenv as root, escaping the whole environment which was created, and then, installing the package on global site-packages, and not inside the project folder where you have a Virtual Environment, although you have activated the environment.
If you rename the folder of your project...
...you'll have to adjust some variables from some files inside the bin directory of your project.
For example:
bin/pip, line 1 (She Bang)
bin/activate, line 42 (VIRTUAL_ENV)
Create your virtualenv with --no-site-packages if you don't want it to be able to use external libraries:
virtualenv --no-site-packages my-virtualenv
. my-virtualenv/bin/activate
pip install ipython
Otherwise, as in your example, it can see a library installed in your system Python environment as satisfying your requested dependency.
For Python 3 :
### install library `virtualenv`
$ pip3 install virtualenv
### call module `venv` with the name for your environment
$ python3 -m venv venv_name
### activate the created environment
$ source venv_name/bin/activate #key step
### install the packages
(venv_name) user#host: pip3 install "package-name"
Well i don't have an appropriate reason regarding why this behavior occurs but then i just found a small work around
Inside the VirtualEnvironment
pip install -Iv package_name==version_number
now this will install the version in your virtual environment
Additionally you can check inside the virtual environment with this
pip install yolk
yolk -l
This shall give you the details of all the installed packages in both the locations(system and virtualenv)
While some might say its not appropriate to use --system-site-packages (it may be true), but what if you have already done a lot of stuffs inside your virtualenv? Now you dont want to redo everything from the scratch.
You may use this as a hack and be careful from the next time :)
To use the environment virtualenv has created, you first need to source env/bin/activate. After that, just install packages using pip install package-name.
You can go to the folder where your venv exists and right click -> git bash here.
Then you just right python -m pip install ipython and it will install inside the folder.
I find it even more convenient with the virtualenv package that creates the venv inside the project's folder.
To further clarify the other answer here:
Under the current version of virtualenv, the --no-site-packages flag is the default behavior, so you don't need to specify it. However, you are overriding the default by explicitly using the --system-site-packages flag, and that's probably not what you want. The default behavior (without specifying either flag) is to create the virtual environment such that when you are using it, any Python packages installed outside the environment are not accessible. That's typically the right choice because it best isolates the virtual environment from your local computer environment. Python packages installed within the environment will not affect your local computer and vice versa.
Secondly, to use a virtual environment after it's been created, you need to navigate into the virtual environment directory and then run:
bin/activate
What this does is to configure environment variables so that Python packages and any executables in the virtual environment's bin folders will be used before those in the standard locations on your local computer. So, for example, when you type "pip", the version of pip that is inside your virtual environment will run instead of the version of pip on your local machine. This is desirable because pip inside the virtual environment will install packages inside the virtual environment.
The problem you are having is because you are running programs (like ipython) from your local machine, when you instead want to install and run copies of those programs isolated inside your virtual environment. You set this up by creating the environment (without specifying any site-packages flags if you are using the current version), running the activate script mentioned above, then running pip to install any packages you need (which will go inside the environment).
From documentation https://docs.python.org/3/library/venv.html:
The pyvenv script has been deprecated as of Python 3.6 in favor of using python3 -m venv to help prevent any potential confusion as to which Python interpreter a virtual environment will be based on.
In order to create a virtual environment for particular project, create a file /home/user/path/to/create_venv.sh:
#!/usr/bin/env bash
# define path to your project's directory
PROJECT_DIR=/home/user/path/to/Project1
# a directory with virtual environment
# will be created in your Project1 directory
# it recommended to add this path into your .gitignore
VENV_DIR="${PROJECT_DIR}"/venv
# https://docs.python.org/3/library/venv.html
python3 -m venv "${VENV_DIR}"
# activates the newly created virtual environment
. "${VENV_DIR}"/bin/activate
# prints activated version of Python
python3 -V
pip3 install --upgrade pip
# Write here all Python libraries which you want to install over pip
# An example or requirements.txt see here:
# https://docs.python.org/3/tutorial/venv.html#managing-packages-with-pip
pip3 install -r "${PROJECT_DIR}"/requirements.txt
echo "Virtual environment ${VENV_DIR} has been created"
deactivate
Then run this script in the console:
$ bash /home/user/path/to/create_venv.sh
I had the same issue and the --no-site-packages did not work for me. I discovered on this older mailing list archive that you are able to force an installation in the virtualenv using the -U flag for pip, eg pip -U ipython. You may verify this works using the bash command which ipython while in the virtualenv.
source: https://mail.python.org/pipermail/python-list/2010-March/571663.html
Sharing what has worked for me in both Ubuntu and Windows. This is for python3. To do for python2, replace "3" with "2":
Ubuntu
pip install virtualenv --user
virtualenv -p python3 /tmp/VIRTUAL
source /tmp/VIRTUAL/bin/activate
which python3
To install any package: pip install package
To get out of the virtual environment: deactivate
To activate again: source /tmp/VIRTUAL/bin/activate
Full explanation here.
Windows
(Assuming you have MiniConda installed and are in the Start Menu > Anaconda > Anaconda Terminal)
conda create -n VIRTUAL python=3
activate VIRTUAL
To install any package: pip install package or conda install package
To get out of the virtual environment: deactivate
To activate again: activate VIRTUAL
Full explanation here.
Sharing a personal case if it helps. It is that a virtual environment was previously arranged. Its path can be displayed by
echo $VIRTUAL_ENV
Make sure that the it is writable to the current user. If not, using
sudo ipython
would certainly clear off the warning message.
In anaconda, if $VIRTUAL_ENV is independently arranged, one can simply delete this folder or rename it, and then restart the shell. Anaconda will recover to its default setup.