Develop Reference

pip version is old in virtual environment [duplicate]

As part of the compilation step for a new python version, I fetch and run get-pip.py, to have the latest pip installed next to the python executable:
$ /opt/python/3.7.0/bin/python --version
Python 3.7.0
$ /opt/python/3.7.0/bin/pip --version
pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)
I have 25 such versions under /opt/python, although I mostly use the five latest versions of each major.minor version that is not EOL. To setup an invironment I used to run virtualenv or my virtualenvutils with the -p /opt/python/X.Y.Z/bin/python option to get a virtual environment with a specific version.
With Python 3.7 this gives the imp module deprecation warning:
$ virtualenv -p /opt/python/3.7.0/bin/python /tmp/py37virtualenv
Running virtualenv with interpreter /opt/python/3.7.0/bin/python
Using base prefix '/opt/python/3.7.0'
/opt/util/virtualenvutils/lib/python3.6/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
New python executable in /tmp/py37virtualenv/bin/python
Installing setuptools, pip, wheel...done.
I have little hope this will be solved in virtualenv, as this has had a PendingDeprecationWarning at least since 2014 (as can be seen from the output in this question)
While investigating replacing virtualenv with python -m venv in virtualenvutils, I first created a new venv based virtual environment by hand:
$ /opt/python/3.7.0/bin/python -m venv /tmp/py37venv
$ /tmp/py37venv/bin/pip --version
pip 10.0.1 from /tmp/py37venv/lib/python3.7/site-packages/pip (python 3.7)
That has an old pip version! If you use it, you'll get:
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command
In the virtual environment created with virtualenv you immediately get the latest version:
$ /tmp/py37virtualenv/bin/pip --version
pip 18.0 from /tmp/py37virtualenv/lib/python3.7/site-packages/pip (python 3.7)
I can run a post-creation step:
/tmp/py37venv/bin/pip install -U --disable-pip-version-check pip
which will take extra time. And if there was a some security update for pip, this would imply running the non-secure version to get a secure version, an ideal point of attack.
From virtualenvutils it is trivial to do the multiple steps to create a pip-less virtualenv and then add pip using get-pip.py. From the command-line this is not so simple:
$ /opt/python/3.7.0/bin/python -m venv --without-pip /tmp/py37venvnopip
$ /tmp/py37venvnopip/bin/python -c "from urllib.request import urlopen; response = urlopen('https://bootstrap.pypa.io/get-pip'); open('/tmp/tmp_get_pip.py', 'w').write(response.read())"
$ /opt/python/3.7.0/bin/python /tmp/tmp_get_pip.py
......
$ /opt/python/3.7.0/bin/pip --version
pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)
What is causing /opt/python/3.7.0/bin/python -m venv to take that old pip version? Is that the version available when 3.7.0 was released?
How can I update my install under /opt/python/3.7.0 in some way so that using /opt/python/3.7.0/bin/python -m venv creates a virtualenv with the latest pip version without reverting to scripts, aliases or using multiple commands? Having the latest pip installed under /opt/python/3.7.0 obviously is not enough.
There are two bundled wheels:
/opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/setuptools-39.0.1-py2.py3-none-any.whl
/opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/pip-10.0.1-py2.py3-none-any.whl
I suspect I need to update those. Is there a better way than updating those by hand? Some option for /some/python -m venv would be nice.
(And running /some/python -m ensurepip --upgrade doesn't do the trick)
Running the deprecated /opt/python/3.7.0/bin/pyvenv has the same old pip version problem.

The trick is not to install the bundled version of pip (which will almost always be out of date), but to use it to install the most current version from the internet.
Standard library venv offers a --without-pip flag that can help here. After creating the virtual environment without pip, you can then you can "execute" ensurepip's wheel directly thanks to Python's zip importer. This is both faster and less hacky than installing pip and then immediately using that same pip installation to uninstall itself and upgrade.
Code speaks louder than words, so here's an example bash function for the process I've described:
# in ~/.bashrc or wherever
function ve() {
local py="python3"
if [ ! -d ./.venv ]; then
echo "creating venv..."
if ! $py -m venv .venv --prompt=$(basename $PWD) --without-pip; then
echo "ERROR: Problem creating venv" >&2
return 1
else
local whl=$($py -c "import pathlib, ensurepip; whl = list(pathlib.Path(ensurepip.__path__[0]).glob('_bundled/pip*.whl'))[0]; print(whl)")
echo "boostrapping pip using $whl"
.venv/bin/python $whl/pip install --upgrade pip setuptools wheel
source .venv/bin/activate
fi
else
source .venv/bin/activate
fi
}
If you prefer the older project virtualenv, it also offers --no-pip, --no-setuptools, and --no-wheel flags to achieve the same on Python 2.7.
Note: Python 3.9+ venv has an --upgrade-deps option to immediately upgrade the pip/setuptools versions after creating an environment, see https://bugs.python.org/issue34556 for more info about that. I don't use this option because it still goes through an unnecessary install/uninstall of the vendored versions, which is inferior to the method of creating an environment with the latest versions directly as shown above.

I use upgrade-ensurepip to update those pip and setuptools wheel files that are part of the ensurepip package. It's not as elegant as being able to upgrade ensurepip via pip, but it's still preferable to doing it manually.
https://pypi.org/project/upgrade-ensurepip/

It is an expected behavior. python -m venv calls python -m ensurepip to install pip and This answer shows that ensurepip would only install the bundled version even with --upgrade option. There isn't any official option to update the bundled pip and setuptools.
Well I have also no good idea to fix this problem as it just is the designed behavior. I would like to give two suggestions:
Use pipenv. It is really good! And it will be the next-generation official package manager in the future(Although there is a big problem related to current Pypi's structure. In short, a package manager can only decide the dependencies with downloading the whole package. This gives a huge difficulty to building dependencies graph.).
Implement your custom EnvBuilder, actually there is an official example about this. And in the example, it also use get-pip.py to install the latest pip.

Virtualenv with pyenv gives wrong python version

I have pyenv installed on an up to date debian testing distribution, and have python 2.7.1 in it.
I create a virtualenv for a project with this version, but when activated it gives me the python system version (3.7).
Here is what I did:
$ pyenv virtualenv 2.7.16 my_project-2.7
Requirement already satisfied: virtualenv in /home/user/.local/lib/python2.7/site-packages (15.1.0)
You are using pip version 18.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Using base prefix '/usr'
New python executable in /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/bin/python
Installing setuptools, pip, wheel...
done.
Installing pip from https://bootstrap.pypa.io/get-pip.py...
Collecting pip
Using cached https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 19.1.1
Uninstalling pip-19.1.1:
Successfully uninstalled pip-19.1.1
Successfully installed pip-19.1.1
$ pyenv activate my_project-2.7
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
$ python --version
Python 3.7.3
$ pip --version
pip 19.1.1 from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/site-packages/pip (python 3.7)
It is strange. The virtualenv is correctly set into ~/.pyenv/version/2.7.16, but I can see that pip come from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/
I think I missed something, or I might have some mess in my python installation, but I can't find out what this python3.7 lib is doing here.
How can I have this virtualenv created for python2.7 ?
Edit
I tried to completely remove pyenv (removing the $(pyenv root) directory), then reinstall it with pyenv-installer but it did not change anything.
I created two virtualenvs for two projects I have (the above 2.7 and one in 3.5).
$ pyenv virtualenvs
2.7.16/envs/my_project-2.7 (created from /usr)
3.6.8/envs/other_project-3.6 (created from /home/user/.pyenv/versions/3.6.8)
my_project-2.7 (created from /usr)
other_project-3.6 (created from /home/user/.pyenv/versions/3.6.8)
I think the problem could be found from this created from /usr but I don't understand what it means. Why was this virtualenv not created from the pyenv python version ? Is it at all relevant ?
It seems that when I create a 3.+ virtual env, venv is used. But on 2.7, virtualenv is used and fails.

WFM as written if I install pyenv and pyenv-virtualenv from scratch according to the linked instructions.
I can see that pip come from /home/user/.pyenv/versions/2.7.16/envs/my_project-2.7/lib/python3.7/
This shows that most probably, that virtual environment already exists and has a wrong Python version.
I can only conjecture how it ended up like this. Most probably, you were messing with that virtualenv or with pyenv envvars by hand; or ran scripts/have something in your bash stratup scripts that do that. E.g. if you are using pyenv-virtualenv commands to manage your environments, you are not supposed to mix them with regular virtualenv commands.
Deleting this virtualenv with pyenv virtualenv-delete my_project-2.7 then recreating it should help.
If not, delete ~/.pyenv and everything relevant from your startup scripts, restart the shell and reinstall pyenv and pyenv-virtualenv.
For a less destructive fixing, you need to carefully inspect your shell environment (envvars, aliases, defined shell functions) and startup scripts for anything that might conflict with pyenv's work; if that doesn't help, debug pyenv machinery (export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }' helps here very much) to find out what exactly is happening.

Using pip on Windows installed with both python 2.7 and 3.5

I am using Windows 10. Currently, I have Python 2.7 installed. I would like to install Python 3.5 as well. However, if I have both 2.7 and 3.5 installed, when I run pip, how do I get the direct the package to be installed to the desired Python version?

You will have to use the absolute path of pip.
E.g: if I installed python 3 to C:\python35, I would use:
C:\> python35\Scripts\pip.exe install packagename
Or if you're on linux, use pip3 install packagename
If you don't specify a full path, it will use whichever pip is in your path.

Because usually i change my intepreter to run something(i got 2 diff projects with both 2 and 3), i use these solution:
Add path to the environment as usual (of course)
Rename ur python.exe , in my case i want to run python 3 using command python3 on my cmd. So i renamed my python.exe in python3.x directory with python3. Itll works with python 2 ofc.
Then to use pip in both python, i use this command.
python3 -m pip install 'somepackage'
and to run pip on python2
python -m pip install 'somepackage'
This is may not the best solution out there, but i like this one
** WINDOWS **
ref : https://datascience.com.co/how-to-install-python-2-7-and-3-6-in-windows-10-add-python-path-281e7eae62a

In my case, I have Python 2.7 and Python 3.4, with the Python Launcher for Windows.
This is the output when running this commands:
PS C:\> pip -V
pip 9.0.1 from c:\python27\lib\site-packages (python 2.7)
PS C:\> pip3 -V
pip 9.0.1 from C:\Python34\lib\site-packages (python 3.4)
I'll note that in my Python27\Scripts\ directory, I have pip.exe, pip2.exe and pip2.7.exe.
And in my Python34\Scripts\ directory, I have pip.exe, pip3.exe and pip3.4.exe.
So all of these .exe files help you when you have different versions of Python installed at the same time.
Of course, for this to work, you have to have the respective Scriptsdirectries in your Path system enviroment variable.

The answer from Farhan.K will work. However, I think a more convenient way would be to rename python35\Scripts\pip.exe to python35\Scripts\pip3.exe assuming python 3 is installed in C:\python35.
After renaming, you can use pip3 when installing packages to python v3 and pip when installing packages to python v2. Without the renaming, your computer will use whichever pip is in your path.

I would advise against ever calling any pip script directly (nor pip3, pip2.7.exe, anything like that).
Instead, a surefire way is to always prefer the explicit variant of calling pip's executable module for a specific Python interpreter:
path/to/pythonX.Y -m pip somecommand
path/to/venv/bin/python -m pip somecommand
C:\path\to\venv\Scripts\python.exe -m pip somecommand
There are many advantages to this, for example:
It is explicit for which Python interpreter the projects will be pip-installed (Python 2 or 3, inside the virtual environment or not, etc.)
For a virtual environment, one can pip-install (or do other things) without activating it: path/to/venv/bin/python -m pip install SomeProject
Under Windows this is the only way to safely upgrade pip itself path\to\venv\Scripts\python.exe -m pip install --upgrade pip
But yes, if all is perfectly setup, then python3 -m pip install SomeProject and pip3 install SomeProject should do the exact same thing, but there are way too many cases where there is an issue with the setup and things don't work as expected and users get confused (as shown by the many questions about this topic on this platform).
References
Brett Cannon's article "Why you should use python -m pip"
pip's documentation section on "Upgrading pip"
venv's documentation section on "Creating virtual environments": "You don’t specifically need to activate an environment [...]"

I ran across an issue with running pip with absolute path. This might be related to WinPython's installation routine and the order of installing Python 3.6 first, 2.7 second, or Python 3.6 being in the path.
No matter which pip was called, it was activating the 3.6 one:
λ C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\Scripts\pip2.exe --version
pip 9.0.1 from C:\prog\WinPython-64bit-3.6.1.0Zero\python-3.6.1.amd64\lib\site-packages (python 3.6)
What finally did the trick was calling pip as a module of the respective python binary:
λ C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\python.exe -m pip --version
pip 9.0.1 from C:\prog\WinPython-64bit-2.7.13.1Zero\python-2.7.13.amd64\lib\site-packages (python 2.7)
Hope that might help someone with similar issues.

I tried many things , then finally
pip3 install --upgrade pip worked for me as i was facing this issue since i had both python3 and python2.7 installed on my system.
mind the pip3 in the beginning and pip in the end.
And yes you do have to run in admin mode the command prompt and make sure if the path is set properly.

1-open command prompt and change direction using the command cd C:\Python35\Scripts
2- write the command pip3 install --upgrade pip
3- close the command prompt and reopen it again to return to the default direction and use the command pip3.exe install package_name to install any package you want

Multiple python installations and pip, dude, where is my site-packages?

As we all know, Mac OS ships with its own python pre installed.
The recommendation seems to be to leave that alone and use homebrew to install a fresh python into the system.
My issue is that after installing python (and pip) using homebrew, pip is installing packages into the Mac OS site-packages instead of my own. I have confirmed I am running the "homebrew" pip:
$ which pip
/usr/local/bin/pip
But then when I pip install something I can se it is installed at:
/lib/python2.7/site-packages
Pip should be installing at /usr/local/lib/python2.7/site-packages unless i'm miss understanding something.
The surprising thing is that checking with -V yields a surprising result:
pip -V
pip 7.1.0 from /usr/local/lib/python2.7/site-packages (python 2.7)
Running pip list just after running pip install does not show the packages that were supposedly just installed by it but went to the wrong site-packages.
Adding to this, the packages installed on the /lib/python2.7/site-packages are not recognized by my $PYTHONPATH and as such I cannot use them.
To add even more confusion, I decided to use a virtualenv, but I was amazed as even using pip with the virtualenv active kept installing to the /lib/python2.7/site-packages instead of the virtualenv site-packages.
So, somehow I ended up with a homebrew pip, that installs packages outside of the homebrew site-packages and a python interpreter that cant use the packages installed by pip.
How do you recommend I go about finding the root cause and having a smooth python experience? :)

You can easily find you site-packages directory by invoking this command
python -c 'import site; print(site.getsitepackages())'

I think after you activate a virtualenv your python path should point to that environments site-package location--if not it's probably not activated. Only once you activate it will you run pip so it installs in that virtual env's site-packages. if it's not activated, it will go in whatever other site-packages it already knows about:
Step 1: create a virtual env
a la... virtualenv venv
Do this only once!
Step 2: Activate the vitual env
something like source /venv/bin/activate
Needs doing every time you want to use this virtual environment
Step 3: run pip commands, watch them get installed in the virtual env site-packages!
If you do step 3 before step 2 your not actually using the virtual environment you created, so all bets are off--That's probably the reason pip is still installing to the old location.
Now, my overall recommendation is to go further and use pyenv to install specific version of python into your /Users/username/.pyenv folder and abandon both the default OSX and homebrew packages. It's simple and you can control easily the exact version of python to use by simple issuing of command to change versions.
THEN use virtualenv in python2 or pyvenv if in python3 (not to be confused with pyenv) to build vitual environments with their own local site-packages to store pip modules. When you activate a virtualenv, your $PYTHONPATH will switch to the specific location.
The flow would then be:
Use pyenv to pull down and switch to a specific version of python you want to use--overriding homebrew and the OSX version.
Create your vitrualenv. This will create a bin that will link to the pyenv python stack you just specified in the previous step.
Activate the virtual env, and proceed.
Totally control your environment!

For one, you could try updating pip with pip install --upgrade pip command, which might or might not redirect your pip path.
Two, and I should have really started with this one is to set the preferred pip executable path in the .bash_profile or .zshrc if you are using one. The way you do it (on Mac) is by holding Shift+Command+Period to reveal hidden files, going to the User folder and opening the .bash_profile/.zshrc with a text editor. Afterward, add the path/to/bin where the pip that you require is. Like export PATH="User/Username/anaconda3/bin:$PATH" or /usr/local/bin or path/to/venv/bin. Whatever code you write in the end will overwrite the previous one.
Three, if you don't wanna change your default pip, but rather wanna use a different version for that specific case just include the full path of the pip executable like /usr/local/bin/pip list or Users/Username/Desktop/venv/bin/pip install module.

Python - No pip when creating a virtual environment

So I heard about the proper way to install packages into python by creating a new virtual environment for every project. Being on a mac (10.8) I have installed python3 using Homebrew, then I installed pip and virtualenv on this copy.
Now here comes the problem:
I create a new virtualenv, and activate it using:
virtualenv testing
source testing/bin/activate
When I type
which python
/Users/mik/Desktop/testing/bin/python
But typing
which pip
/usr/local/bin/pip
(learned of this when trying to install a package in the virtual environment, and it installed in the system wide installation in /usr/local/)
Inside the folder testing there is no file referring to pip
Extra Question: How does pip know which python to install the files to, for example pip list (which I believe refers to python 2.7) outputs the names of packages installed on python 3.3

I'll start with the last question as it explains what is happening.
The commands pip and easy_install are python scripts which are made executable on the filesystem. The python they use is the python that the first line tells to run the script. e.g. in /usr/bin/easy_install it is #!/usr/bin/python This will be Apple's python. So easy_install will install the 2.7 version of pip and virtualenv and will ignore your python3.3 setup.
The way to instal into python 3 is to install the 3.3 version of pip and virtualenv, the easiest way would be to install the Homebrew package for them. I think it is easier and less confusing to use just one package manager (Homebrew here) and not two (i.e. Homebrew and python).
You can also install easy_install directly. The way to do this is install the distribute package using python3.3 explicitly.
Python 3.4 will make this much easier as pip will always be available