Develop Reference

Why is pip asking me to upgrade when it's already up-to-date?

I have a brand new installation of python 3.7.1 64-bit and I'm using the latest pyCharm as my IDE. I have no other installation of python on this machine.
I go to install numpy and I get this message:
(venv) C:\Users\John\PycharmProjects\project>pip install numpy
Requirement already satisfied: numpy in c:\users\john\pycharmprojects\pysims\venv\lib\site-packages (1.15.4)
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
So I run the suggested command but it's already up-to-date
(venv) C:\Users\John\PycharmProjects\project>python -m pip install --upgrade pip
Requirement already up-to-date: pip in c:\users\john\pycharmprojects\pysims\venv\lib\site-packages (18.1)
So I check the version but it's still the old version
(venv) C:\Users\John\PycharmProjects\pySIMS>pip -V
pip 10.0.1 from c:\users\john\pycharmprojects\pysims\venv\lib\site-packages\pip-10.0.1-py3.7.egg\pip (python 3.7)
I thought I'd try py -3 to upgrade and it works.
(venv) C:\Users\John\PycharmProjects\pySIMS>py -3 -m pip install --upgrade pip
Collecting pip
Using cached https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 10.0.1
Uninstalling pip-10.0.1:
Successfully uninstalled pip-10.0.1
Successfully installed pip-18.1
But the version is still old
(venv) C:\Users\John\PycharmProjects\pySIMS>pip -V
pip 10.0.1 from c:\users\john\pycharmprojects\pysims\venv\lib\site-packages\pip-10.0.1-py3.7.egg\pip (python 3.7)
WHAT IS GOING ON? Am I missing something totally obvious? I've never had an issue like this working in Python 2 but since I've moved to Python 3 it's been nothing but errors.

This is looking like you have multiple installation of pip, one that comes first in the PATH (pip) and another that is recognized by python (python -m pip).
Try running the command:
pip show pip
and
python -m pip show pip
And check if the path are the same.
If not i would suggest uninstalling the undesired one, or change your your PATH environment variable to have the folder containing the correct pip come before the folder with the wrong one.

When I got this error I had 2 pip version folders in site-packages. One pip-19.2.3.dist-info and another something like pip-10.0.1. I deleted the first one, leaving only the default version. Then running python -m pip install --upgrade pip fixed the problem

I had a similar issue where I was upgrading pip; however, my issue what that I had a pip.conf file that pointed to a repo that didn't have the version to upgrade to. I removed the pip.conf to allow it to go to the default repo and it was able to download the correct version and upgrade.

1.Open you project setting(File>Settings)
2.Project>Project Interpreter
3.find pip and lick the triangle on the right(in the red circle I draw)

Package updates on the system affecting virtual environment?

I am in a virtual environment. I verified that in my active terminal using which python and can confirm that I am in fact in a virtual environment. Printing pip list, according to the official documentation, should list the packages in this virtual environment. Here's an output:
Package Version
---------------------------------- -------------------
alabaster 0.7.10
anaconda-client 1.6.5
... (truncated)
pip 10.0.1
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
When I fire up another terminal, this time making sure I'm outside of that virtual environment and proceed to upgrade pip:
pip install --upgrade pip
Upon doing that, I verified that my pip package on the system has been updated to version 18.0. Here is the confusing part: I switched back into the virtual environment and use pip list and the pip version in my virtual environment is now pip 18.0.
Why is it that upgrading the pip version outside of that environment subsequently update the pip in my virtual environment from 10.0.1 at all? Have I misunderstood how virtual environments work? I'm not new to python but have not used virtual environment so forgive me if it's something very fundamental. In my understanding the primary value in using virtual envs is that I can be upgrading my system-wide packages (such as pip, flask etc) without any of that changes being affected in my virtual environments. Virtual environments should be isolated environment at all?
If it matters, I'm using the default venv and not virtualenvwrapper or any other wrapper tool.

The Problem
tl;dr: The problem is with conda. I use conda and apparently that causes some issues with managing and installing packages in virtual environment created via venv because a local instance of pip was not present.
Solution A: conda install -n myenv pip where myenv refers to the name of your virtual environment
Solution B: Use conda list, conda create to work with environments in a way that is 100% compatible with conda
/end tl;dr
Here's a breakdown of the problem. When I'm using an anaconda's version of python and I decide to look at the list of packages I have in my environment; Supposed I were to call pip freeze or pip list, whether I'm in a virtual environment or not wouldn't matter. It returns exactly the same list of packages from conda's corresponding site-packages folder.
When I'm inside a virtual environment, running which python does seem to point to an isolated version of the python instance (acco is the name of that instance):
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ which python
/Users/Samuel/Dropbox/Projects/Python/Acco/acco/bin/python
However, while in this virtual environment, running pip list or pip list --local would still refer to the same set of packages - and that is because it is still point to the conda's version of package directory (Yes, even in a virtual environment).
Specifically, with or without, inside or outside of a virtual environment, the pip list points to packages installed in the /anaconda3/lib/../site-packages directory:
import sys
sys.prefix
'/Users/Samuel/anaconda3'
import site
site.getsitepackages()
['/Users/Samuel/anaconda3/lib/python3.6/site-packages']
The really problematic part of this is that while in the virtual environment, you have essentially no libraries installed. Installing from requirements.txt' usingpip install -r requirements.txtor just installing packages at all using plain oldpip install` wouldn't work if any of the packages you're attempting to install already exist at the conda's directory. Instead, you'll get a message that doesn't look like an error, and it stops just right there. The package you're attempting to install is not installed into the local directory.
# same as pip install Flask==0.12.2
pip install -r requirements.txt
Requirement already satisfied: Flask==0.12.2 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Werkzeug>=0.7 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Jinja2>=2.4 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (2.9.6)
Requirement already satisfied: itsdangerous>=0.21 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.24)
Requirement already satisfied: click>=2.0 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (6.7)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.12.2->-r requirements.txt (line 1)) (1.0)
As a reminder, we're still executing your python commands using the python in our virtual environment (../Dropbox/Projects/Python/Acco/acco/bin/python) and not the conda distribution. This virtual environment has no packages in its isolated lib folder, and you can't install any libraries into it as pip will be stopped with the "requirement already satisfied" message and exit (or terminate its attempt).
This means that while in that virtual environment, trying to run a python script or app that has dependencies will certainly fail. Building from the above example, your Flask app app.py won't run because it couldn't find flask. Goes without saying, since your virtual environment has no packages and it won't let you install any.
The Solution
The solution is that if you're using a conda distribution of python, check the packages you have installed on your system using conda list instead of pip list for maximal consistency.
(acco) Samuels-MacBook-Pro:Acco Samuel$ conda list
# packages in environment at /Users/Samuel/anaconda3:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py36h2fc01ae_0
_r-mutex 1.0.0 mro_2
alabaster 0.7.10 py36h174008c_0
anaconda custom py36ha4fed55_0
Notice that conda tells you the environment from which the libraries are listed from. At this stage:
Deactivate the environment using deactive
Make virtual environment using conda create --name acco python=3.6.3 flask sqlite, here we're using acco as the virtual environment's name, a specific version of python, and optionally some other dependencies
Activate the environment using source activate acco
Now, when you do conda list again, being in your virtual environment:
# packages in environment at /Users/Samuel/anaconda3/envs/acco:
#
# Name Version Build Channel
ca-certificates 2018.03.07 0
certifi 2018.8.13 py36_0
click 6.7 py36hec950be_0
flask 1.0.2 py36_1
If you want to manage the dependencies of your app using an old-school requirement.txt file, then simplest way is to conda list --export > requirements.txt. From this point on, use conda list in place of pip list and use conda create as well as source activate in place of its venv counterpart.
The anaconda's main documentation suggest:
To use pip in your environment, in your Terminal window or an Anaconda
Prompt, run:
conda install -n acco pip
source activate acco
pip <pip_subcommand>
This solution worked too. Following the steps above, I ended up with:
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ pip list
Package Version
------------ ---------
certifi 2018.8.13
click 6.7
Flask 1.0.2
itsdangerous 0.24
Jinja2 2.10
MarkupSafe 1.0
pip 10.0.1
setuptools 40.0.0
Werkzeug 0.14.1
wheel 0.31.1
As a bonus, I double-checked to see that a conda environment created this way would be able to handle pip list or pip install: same error. However, when using conda update (for example: conda update sqlite), conda install or any of the conda-counterpart code, they all work as fully intended. Updating sqlite in your local environment update exactly that copy of sqlite and not the one in your anaconda system. Whew!

Why can't conda find pip installed module in prefix (within conda evironment)

I used pip to install the Resource module to the default conda environment on my laptop: (C:\Users\my_username\Anaconda2). I think it is called root. I installed pip to the conda environment and so I'm 90% sure the resource was installed within the environment. And indeed running conda list shows that the package is listed within the environment. Here is a section of the output:
# packages in environment at C:\Users\conna\Anaconda2:
#
qtpy 1.2.1 py27_0
requests 2.14.2 py27_0
Resource 0.2.0 <pip>
rope 0.9.4 py27_1
ruamel_yaml 0.11.14 py27_1
scandir 1.5 py27_0
scikit-image 0.13.0 np112py27_0
However when I run
conda update Resource
I get the following error:
PackageNotInstalledError: Package is not installed in prefix.
prefix: C:\Users\conna\Anaconda2
package name: Resource
How is it possible that conda list shows the module is present but conda update can't see them? I also noticed that conda update doesn't recognize any packages with <pip>. What is happening?

conda only manages the packages that are installed using a conda command. If you installed a package with pip (or using python setup.py install or develop) it will show up with conda list (because that shows all packages no matter how they were installed) but conda won't manage that package. Simply because it doesn't know how!
So if you installed a package with pip you also need to upgrade/update it with pip:
pip install [package_name] --upgrade

Try this;
pip install Resource --upgrade

Does conda update packages from pypi installed using pip install?

I use Anaconda (because it is awesome), and the packages available through conda install are quite extensive. However now and then I do need to install a package that isn't available in the conda repositories, and so get it from pypi instead.
My question: when I run the command conda update --all, will conda also update these pypi packages? Or do I have to update them separately? The conda docs don't seem to contain an answer to this. This question and answer seems to indicate that no, conda does not manage pypi packages, but I'm still uncertain.

No, conda update and conda install don't update packages installed with pip (or install them using pip).
These conda commands only check your "default" anaconda-channels or the ones specified with -c, they ignore everything else. One exception is conda list which shows also the packages installed with pip, these are marked with <pip> and won't be updated.
One example using pip and six:
$ conda create -n testenv python=3.5
Fetching package metadata .................
Solving package specifications: .
Package plan for installation in environment testenv:
The following NEW packages will be INSTALLED:
pip: 9.0.1-py35_1
python: 3.5.3-3
setuptools: 27.2.0-py35_1
vs2015_runtime: 14.0.25123-0
wheel: 0.29.0-py35_0
Proceed ([y]/n)? y
$ activate testenv
Installing six with pip (old version):
(testenv) $ pip install six==1.6
Collecting six==1.6
Downloading six-1.6.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.6.0
conda update doesn't update it (note that six isn't listed in the "all requested packages" but it's listed in conda list):
(testenv) $ conda update --all
Fetching package metadata .................
Solving package specifications: .
# All requested packages already installed.
# packages in environment at testenv:
#
pip 9.0.1 py35_1
python 3.5.3 3
setuptools 27.2.0 py35_1
vs2015_runtime 14.0.25123 0
wheel 0.29.0 py35_0
(testenv) $ conda list
# packages in environment at testenv:
#
pip 9.0.1 py35_1
python 3.5.3 3
setuptools 27.2.0 py35_1
six 1.6.0 <pip>
vs2015_runtime 14.0.25123 0
wheel 0.29.0 py35_0
But it can be upgraded with pip:
(testenv) $ pip install six --upgrade
Collecting six
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six
Found existing installation: six 1.6.0
Uninstalling six-1.6.0:
Successfully uninstalled six-1.6.0
Successfully installed six-1.10.0
Just to show that there is a newer version of six in the anaconda channel (which was ignored when I did conda update):
(testenv) $ conda install six
Fetching package metadata .................
Solving package specifications: .
Package plan for installation in environment testenv:
The following NEW packages will be INSTALLED:
six: 1.10.0-py35_0
Proceed ([y]/n)?

Conda 4.6 has an experimental feature to enable interoperability with pip-installed packages. Use conda config --set pip_interop_enabled true. Non-conda-installed python packages that can be "managed" by conda (i.e. removed) may be updated/changed to satisfy the current solve. Manageable packages were typically installed from wheels. Sdists installed with newer versions of pip are also typically manageable. However conda won't switch out the non-conda-installed package for a conda package if the versions are equivalent.
Non-conda-installed python packages that can't be managed will anchor the environment in place until they are removed by other means. An example of unmanageable packages are "editable" installs that used pip install -e.
All of this applies to conda update --all.

This question is old, but here's a batch script that might help with automating this process on Windows. It involves going through conda list and finding packages marked with the pypi tag, which are then subsequently upgraded with pip --upgrade en masse (assuming they are out-of-date; otherwise the standard Requirement already up-to-date message will be returned).
Place the following in a batch file (e.g., condapip.bat) and try it out:
#echo off
set packages=pip install --upgrade
for /f "tokens=1" %%i in ('conda list ^| findstr /R /C:"pypi"') do (call :join %%i)
#echo on
%packages%
#echo off
goto :eof
:join
set packages=%packages% %1
goto :eof

pyvenv installs wrong pip version

I'm creating a virtualenv with pyvenv env but the installed pip version is outdated. I can manually update pip in the virtual env but I would like to have the correct version automatically.
Without venv activated:
~> pip -V
pip 7.1.2 from /usr/lib/python3.4/site-packages (python 3.4)
Installing the venv:
~> pyvenv env
~> source env/bin/activate.fish
With venv activated:
~> pip -V
pip 6.0.8 from /home/syntonym/test/env/lib/python3.4/site-packages (python 3.4)
I'm using arch and have pip managed by both, pip and pacman (the arch packet manager), which is probably not a good idea. I still have no idea from where pyvenv gets a 6.0.8 version of pip or how to fix it. Reinstalling with pacman did not help.
EDIT:
ensurepip claims it's already up to date:
~> python -m ensurepip --upgrade
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: setuptools in /usr/lib/python3.4/site-packages
Requirement already up-to-date: pip in /usr/lib/python3.4/site-packages

You can upgrade pip in a virtual environment manually by executing
pip install -U pip
You are facing this problem, because venv uses ensurepip to add pip into new environments:
Unless the --without-pip option is given, ensurepip will be invoked to
bootstrap pip into the virtual environment.
Ensurepip package won't download from the internet or grab files from anywhere else, because all required components are already included into the package. Doing so would add security flaws and is thus unsupported.
Ensurepip is not designed to give you the newest pip, but just "a" pip. To get the newest one use the manual way at the beginning of this post.