I need to find the python packages within my virtual environment.
When I run pip freeze I can see all the packages in the virtual environment.
However, when I cd ./venv/ and run tree Pillow (or manually search) I cannot find any installed package.
Any idea where they are stored?
Ask pip to tell you where it installed the project with the following command:
$ path/to/venv/bin/python -m pip show --files Pillow
from virtualenv do pip -V
It will be in : <virtual env root dir> + lib
example from my terminal
(vams_test) C:\VAMS\jiras\vams>pip -V
pip 19.2.3 from c:\vams\code\copy_v0.1\vams_test\lib\site-packages\pip (python 3.7)
I'm to deploy Python into a production system and the python script I have has a number of modules associated with it.
Is there a way to install python with only a specific list of modules? Abit like with generating a jar, you can have a folder with all the other dependency jar's in a folder, which is nice and clean. I don't want to compile the python code so I want something similar.
(Note: I also don't want to create a virtual environment - I want the default environment like this)
You can either use virtualenv, which basically is what the name suggests, or you can use Docker, which personally I prefer
If you don't want to do like what Amir is suggesting above, then 2 other options are available:
Copy those modules and place them in the same folder where your script is installed
Create a requirements.txt file with the name & version of those modules and then run "pip install -r requirements.txt" to install these modules in your site-packages folder
To manage your python packages you can use great virtualenv tool, it looks really simple and works well on linux/macOS/Windows. Any package which will be installed in activated virtualenv will be available only in this virtualenv, so you can have for example 3 different versions of "Django" package on your machine and work with them using different virtual environments:
Install virtualenv:
$ pip3 install virtualenv
Create your virtualenv:
$ virtualenv -p python3 my_virtualenv_name
Activate your virtualenv:
$ . my_virtualenv_name/bin/activate
Check what packages have been installed:
$ pip freeze
Install any package for example "Django":
$ pip install Django
Confirm installation:
$ pip freeze | grep Django
Uninstall any package from your virtual environment:
$ pip uninstall Django -y
Uninstall all packages from your virtual environment:
$ pip freeze | xargs pip uninstall -y
Deactivate virtualenv
$ deactivate
More info in the official documentation: https://virtualenv.pypa.io/en/latest/
I do not have root previlege on a linux server so I want to creat a virtual python according to creating a "virtual" python.
After I run virtual-python.py, I do have python in ~/bin/python:
Then, according to setuptools PyPI page, I download ez_setup.py and run ~/bin/python ez_setup.py. Error occurs:
What should I do?
Looking at the linked website, it looks outdated. You use pip, not easy_install.
For installing development packages, I always take the following rules in account:
The system package manager is responsible for system-wide packages, so never use sudo pip. This doesn't just match the question, but this is always a good idea.
The package manager packages are probably outdated. You'll want an up-to-date version for development tools.
I recommend the following way to install local development tools.
$ # Install pip and setuptools on a user level
$ curl https://bootstrap.pypa.io/get-pip.py | python - --user
$ # Add the executables to your path. Add this to your `.bashrc` or `.profile` as well
$ export PATH=$PATH/$HOME/.local/bin
At this point pip should be accessible from the command line and usable without sudo. Use this to install virtualenv, the most widely used tools to set up virtual environments.
$ pip install virtualenv --user
Now simply use virtualenv to set up an environment to run your application in:
$ virtualenv myapp
Now activate the virtual environment and do whatever you would like to do with it. Note that after activating the virtual environment, pip refers to pip installed inside of the virtualenv, not the one installed on a user level.
$ source myapp/bin/activate
(myapp)$ pip install -r requirements.txt # This is just an example
You'll want to create a new virtual environment for each application you run on the server, so the dependencies can't conflict.
On Windows 8, i have the following structure for a Python 3 project:
../Project/
../Project/app/app.py
../Project/app/setup.py
From the app folder, i invoke the following commands to create and enter into a virtual environment:
pyvenv.py venv
cd venv\Scripts
activate.bat
cd ../..
Now i would like to install the Nose unit testing framework into my virtual environment:
pip install nose
... and Nose gets installed into the global folder (In my case, C:\Python33).
When i invoke python setup.py install, my custom module gets installed to the virtual environment. Why doesn't PIP do the same?
It works well for me after following docs:
Common installation tools such as Distribute and pip work as expected
with venvs - i.e. when a venv is active, they install Python packages
into the venv without needing to be told to do so explicitly. Of
course, you need to install them into the venv first: this could be
done by running distribute_setup.py with the venv activated, followed
by running easy_install pip. Alternatively, you could download the
source tarballs and run python setup.py install after unpacking, with
the venv activated.
Is there a way to upgrade the version of python used in a virtualenv (e.g. if a bugfix release comes out)?
I could pip freeze --local > requirements.txt, then remove the directory and pip install -r requirements.txt, but this requires a lot of reinstallation of large libraries, for instance, numpy, which I use a lot.
I can see this is an advantage when upgrading from, e.g., 2.6 -> 2.7, but what about 2.7.x -> 2.7.y?
If you happen to be using the venv module that comes with Python 3.3+, it supports an --upgrade option.
Per the docs:
Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place
python3 -m venv --upgrade ENV_DIR
Did you see this? If I haven't misunderstand that answer, you may try to create a new virtualenv on top of the old one. You just need to know which python is going to use your virtualenv (you will need to see your virtualenv version).
If your virtualenv is installed with the same python version of the old one and upgrading your virtualenv package is not an option, you may want to read this in order to install a virtualenv with the python version you want.
EDIT
I've tested this approach (the one that create a new virtualenv on top of the old one) and it worked fine for me. I think you may have some problems if you change from python 2.6 to 2.7 or 2.7 to 3.x but if you just upgrade inside the same version (staying at 2.7 as you want) you shouldn't have any problem, as all the packages are held in the same folders for both python versions (2.7.x and 2.7.y packages are inside your_env/lib/python2.7/).
If you change your virtualenv python version, you will need to install all your packages again for that version (or just link the packages you need into the new version packages folder, i.e: your_env/lib/python_newversion/site-packages)
Updated again:
The following method might not work in newer versions of virtualenv. Before you try to make modifications to the old virtualenv, you should save the dependencies in a requirement file (pip freeze > requirements.txt) and make a backup of it somewhere else. If anything goes wrong, you can still create a new virtualenv and install the old dependencies in it (pip install -r requirements.txt).
Updated: I changed the answer 5 months after I originally answered. The following method is more convenient and robust.
Side effect: it also fixes the Symbol not found: _SSLv2_method exception when you do import ssl in a virtual environment after upgrading Python to v2.7.8.
Notice: Currently, this is for Python 2.7.x only.
If you're using Homebrew Python on OS X, first deactivate all virtualenv, then upgrade Python:
brew update && brew upgrade python
Run the following commands (<EXISTING_ENV_PATH> is path of your virtual environment):
cd <EXISTING_ENV_PATH>
rm .Python
rm bin/pip{,2,2.7}
rm bin/python{,2,2.7}
rm -r include/python2.7
rm lib/python2.7/*
rm -r lib/python2.7/distutils
rm lib/python2.7/site-packages/easy_install.*
rm -r lib/python2.7/site-packages/pip
rm -r lib/python2.7/site-packages/pip-*.dist-info
rm -r lib/python2.7/site-packages/setuptools
rm -r lib/python2.7/site-packages/setuptools-*.dist-info
Finally, re-create your virtual environment:
virtualenv <EXISTING_ENV_PATH>
By doing so, old Python core files and standard libraries (plus setuptools and pip) are removed, while the custom libraries installed in site-packages are preserved and working, as soon as they are in pure Python. Binary libraries may or may not need to be reinstalled to function properly.
This worked for me on 5 virtual environments with Django installed.
BTW, if ./manage.py compilemessages is not working afterwards, try this:
brew install gettext && brew link gettext --force
Step 1: Freeze requirement & take a back-up of existing env
pip freeze > requirements.txt
deactivate
mv env env_old
Step 2: Install Python 3.7 & activate virutal environment
sudo apt-get install python3.7-venv
python3.7 -m venv env
source env/bin/activate
python --version
Step 3: Install requirements
sudo apt-get install python3.7-dev
pip3 install -r requirements.txt
How to upgrade the Python version for an existing virtualenvwrapper project and keep the same name
I'm adding an answer for anyone using Doug Hellmann's excellent virtualenvwrapper specifically since the existing answers didn't do it for me.
Some context:
I work on some projects that are Python 2 and some that are Python 3; while I'd love to use python3 -m venv, it doesn't support Python 2 environments
When I start a new project, I use mkproject which creates the virtual environment, creates an empty project directory, and cds into it
I want to continue using virtualenvwrapper's workon command to activate any project irrespective of Python version
Directions:
Let's say your existing project is named foo and is currently running Python 2 (mkproject -p python2 foo), though the commands are the same whether upgrading from 2.x to 3.x, 3.6.0 to 3.6.1, etc. I'm also assuming you're currently inside the activated virtual environment.
1. Deactivate and remove the old virtual environment:
$ deactivate
$ rmvirtualenv foo
Note that if you've added any custom commands to the hooks (e.g., bin/postactivate) you'd need to save those before removing the environment.
2. Stash the real project in a temp directory:
$ cd ..
$ mv foo foo-tmp
3. Create the new virtual environment (and project dir) and activate:
$ mkproject -p python3 foo
4. Replace the empty generated project dir with real project, change back into project dir:
$ cd ..
$ mv -f foo-tmp foo
$ cdproject
5. Re-install dependencies, confirm new Python version, etc:
$ pip install -r requirements.txt
$ python --version
If this is a common use case, I'll consider opening a PR to add something like $ upgradevirtualenv / $ upgradeproject to virtualenvwrapper.
Let's consider that the environment that one wants to update has the name venv.
1. Backup venv requirements (optional)
First of all, backup the requirements of the virtual environment:
pip freeze > requirements.txt
deactivate
#Move the folder to a new one
mv venv venv_old
2. Install Python
Assuming that one doesn't have sudo access, pyenv is a reliable and fast way to install Python. For that, one should run
curl https://pyenv.run | bash
and then
exec $SHELL
as suggested here.
If, when one tries to update pyenv
pyenv update
and one gets the error
bash: pyenv: command not found
that is because pyenv path wasn't exported to .bashrc. It can be solved by executing the following commands:
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
Then restart the shell
exec "$SHELL"
Now one should install the Python version that one wants. Let's say version 3.8.3
pyenv install 3.8.3
One can confirm if it was properly installed by running
pyenv versions
The output should be the location and the versions (in this case 3.8.3)
3. Create the new virtual environment
Finally, with the new Python version installed, create a new virtual environment (let's call it venv)
python3.8 -m venv venv
Activate it
source venv/bin/activate
and install the requirements
pip install -r requirements.txt
Now one should be up and running with a new environment.
I wasn't able to create a new virtualenv on top of the old one. But there are tools in pip which make it much faster to re-install requirements into a brand new venv. Pip can build each of the items in your requirements.txt into a wheel package, and store that in a local cache. When you create a new venv and run pip install in it, pip will automatically use the prebuilt wheels if it finds them. Wheels install much faster than running setup.py for each module.
My ~/.pip/pip.conf looks like this:
[global]
download-cache = /Users/me/.pip/download-cache
find-links =
/Users/me/.pip/wheels/
[wheel]
wheel-dir = /Users/me/.pip/wheels
I install wheel (pip install wheel), then run pip wheel -r requirements.txt. This stores the built wheels in the wheel-dir in my pip.conf.
From then on, any time I pip install any of these requirements, it installs them from the wheels, which is pretty quick.
I just want to clarify, because some of the answers refer to venv and others refer to virtualenv.
Use of the -p or --python flag is supported on virtualenv, but not on venv. If you have more than one Python version and you want to specify which one to create the venv with, do it on the command line, like this:
malikarumi#Tetuoan2:~/Projects$ python3.6 -m venv {path to pre-existing dir you want venv in}
You can of course upgrade with venv as others have pointed out, but that assumes you have already upgraded the Python that was used to create that venv in the first place. You can't upgrade to a Python version you don't already have on your system somewhere, so make sure to get the version you want, first, then make all the venvs you want from it.
This approach always works for me:
# First of all, delete all broken links. Replace my_project_name` to your virtual env name
find ~/.virtualenvs/my_project_name/ -type l -delete
# Then create new links to the current Python version
virtualenv ~/.virtualenvs/my_project_name/
# It's it. Just repeat for each virtualenv located in ~/.virtualenvs
Taken from:
https://github.com/1st/python-on-osx#python-virtualenv
https://gist.github.com/1st/ced02a1c64ac7b82bb27e432eea6b068
If you're using pipenv, I don't know if it's possible to upgrade an environment in place, but at least for minor version upgrades it seems to be smart enough not to rebuild packages from scratch when it creates a new environment. E.g., from 3.6.4 to 3.6.5:
$ pipenv --python 3.6.5 install
Virtualenv already exists!
Removing existing virtualenv…
Creating a v$ pipenv --python 3.6.5 install
Virtualenv already exists!
Removing existing virtualenv…
Creating a virtualenv for this project…
Using /usr/local/bin/python3.6m (3.6.5) to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3.6m
Using base prefix '/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/python3.6
Also creating executable in /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD
Installing dependencies from Pipfile.lock (84dd0e)…
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 47/47 — 00:00:24
To activate this project's virtualenv, run the following:
$ pipenv shell
$ pipenv shell
Spawning environment shell (/bin/bash). Use 'exit' to leave.
. /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/activate
bash-3.2$ . /Users/dmoles/.local/share/virtualenvs/autoscale-aBUhewiD/bin/activate
(autoscale-aBUhewiD) bash-3.2$ python
Python 3.6.5 (default, Mar 30 2018, 06:41:53)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>>
I moved my home directory from one mac to another (Mountain Lion to Yosemite) and didn't realize about the broken virtualenv until I lost hold of the old laptop. I had the virtualenv point to Python 2.7 installed by brew and since Yosemite came with Python 2.7, I wanted to update my virtualenv to the system python. When I ran virtualenv on top of the existing directory, I was getting OSError: [Errno 17] File exists: '/Users/hdara/bin/python2.7/lib/python2.7/config' error. By trial and error, I worked around this issue by removing a few links and fixing up a few more manually. This is what I finally did (similar to what #Rockalite did, but simpler):
cd <virtualenv-root>
rm lib/python2.7/config
rm lib/python2.7/lib-dynload
rm include/python2.7
rm .Python
cd lib/python2.7
gfind . -type l -xtype l | while read f; do ln -s -f /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/${f#./} $f; done
After this, I was able to just run virtualenv on top of the existing directory.
On OS X or macOS using Homebrew to install and upgrade Python3 I had to delete symbolic links before python -m venv --upgrade ENV_DIR would work.
I saved the following in upgrade_python3.sh so I would remember how months from now when I need to do it again:
brew upgrade python3
find ~/.virtualenvs/ -type l -delete
find ~/.virtualenvs/ -type d -mindepth 1 -maxdepth 1 -exec python3 -m venv --upgrade "{}" \;
UPDATE: while this seemed to work well at first, when I ran py.test it gave an error. In the end I just re-created the environment from a requirements file.
For everyone with the problem
Error: Command '['/Users/me/Sites/site/venv3/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
You have to install python3.6-venv
sudo apt-get install python3.6-venv