I am in a virtual environment. I verified that in my active terminal using which python and can confirm that I am in fact in a virtual environment. Printing pip list, according to the official documentation, should list the packages in this virtual environment. Here's an output:
Package Version
---------------------------------- -------------------
alabaster 0.7.10
anaconda-client 1.6.5
... (truncated)
pip 10.0.1
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
When I fire up another terminal, this time making sure I'm outside of that virtual environment and proceed to upgrade pip:
pip install --upgrade pip
Upon doing that, I verified that my pip package on the system has been updated to version 18.0. Here is the confusing part: I switched back into the virtual environment and use pip list and the pip version in my virtual environment is now pip 18.0.
Why is it that upgrading the pip version outside of that environment subsequently update the pip in my virtual environment from 10.0.1 at all? Have I misunderstood how virtual environments work? I'm not new to python but have not used virtual environment so forgive me if it's something very fundamental. In my understanding the primary value in using virtual envs is that I can be upgrading my system-wide packages (such as pip, flask etc) without any of that changes being affected in my virtual environments. Virtual environments should be isolated environment at all?
If it matters, I'm using the default venv and not virtualenvwrapper or any other wrapper tool.
The Problem
tl;dr: The problem is with conda. I use conda and apparently that causes some issues with managing and installing packages in virtual environment created via venv because a local instance of pip was not present.
Solution A: conda install -n myenv pip where myenv refers to the name of your virtual environment
Solution B: Use conda list, conda create to work with environments in a way that is 100% compatible with conda
/end tl;dr
Here's a breakdown of the problem. When I'm using an anaconda's version of python and I decide to look at the list of packages I have in my environment; Supposed I were to call pip freeze or pip list, whether I'm in a virtual environment or not wouldn't matter. It returns exactly the same list of packages from conda's corresponding site-packages folder.
When I'm inside a virtual environment, running which python does seem to point to an isolated version of the python instance (acco is the name of that instance):
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ which python
/Users/Samuel/Dropbox/Projects/Python/Acco/acco/bin/python
However, while in this virtual environment, running pip list or pip list --local would still refer to the same set of packages - and that is because it is still point to the conda's version of package directory (Yes, even in a virtual environment).
Specifically, with or without, inside or outside of a virtual environment, the pip list points to packages installed in the /anaconda3/lib/../site-packages directory:
import sys
sys.prefix
'/Users/Samuel/anaconda3'
import site
site.getsitepackages()
['/Users/Samuel/anaconda3/lib/python3.6/site-packages']
The really problematic part of this is that while in the virtual environment, you have essentially no libraries installed. Installing from requirements.txt' usingpip install -r requirements.txtor just installing packages at all using plain oldpip install` wouldn't work if any of the packages you're attempting to install already exist at the conda's directory. Instead, you'll get a message that doesn't look like an error, and it stops just right there. The package you're attempting to install is not installed into the local directory.
# same as pip install Flask==0.12.2
pip install -r requirements.txt
Requirement already satisfied: Flask==0.12.2 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Werkzeug>=0.7 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Jinja2>=2.4 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (2.9.6)
Requirement already satisfied: itsdangerous>=0.21 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.24)
Requirement already satisfied: click>=2.0 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (6.7)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.12.2->-r requirements.txt (line 1)) (1.0)
As a reminder, we're still executing your python commands using the python in our virtual environment (../Dropbox/Projects/Python/Acco/acco/bin/python) and not the conda distribution. This virtual environment has no packages in its isolated lib folder, and you can't install any libraries into it as pip will be stopped with the "requirement already satisfied" message and exit (or terminate its attempt).
This means that while in that virtual environment, trying to run a python script or app that has dependencies will certainly fail. Building from the above example, your Flask app app.py won't run because it couldn't find flask. Goes without saying, since your virtual environment has no packages and it won't let you install any.
The Solution
The solution is that if you're using a conda distribution of python, check the packages you have installed on your system using conda list instead of pip list for maximal consistency.
(acco) Samuels-MacBook-Pro:Acco Samuel$ conda list
# packages in environment at /Users/Samuel/anaconda3:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py36h2fc01ae_0
_r-mutex 1.0.0 mro_2
alabaster 0.7.10 py36h174008c_0
anaconda custom py36ha4fed55_0
Notice that conda tells you the environment from which the libraries are listed from. At this stage:
Deactivate the environment using deactive
Make virtual environment using conda create --name acco python=3.6.3 flask sqlite, here we're using acco as the virtual environment's name, a specific version of python, and optionally some other dependencies
Activate the environment using source activate acco
Now, when you do conda list again, being in your virtual environment:
# packages in environment at /Users/Samuel/anaconda3/envs/acco:
#
# Name Version Build Channel
ca-certificates 2018.03.07 0
certifi 2018.8.13 py36_0
click 6.7 py36hec950be_0
flask 1.0.2 py36_1
If you want to manage the dependencies of your app using an old-school requirement.txt file, then simplest way is to conda list --export > requirements.txt. From this point on, use conda list in place of pip list and use conda create as well as source activate in place of its venv counterpart.
The anaconda's main documentation suggest:
To use pip in your environment, in your Terminal window or an Anaconda
Prompt, run:
conda install -n acco pip
source activate acco
pip <pip_subcommand>
This solution worked too. Following the steps above, I ended up with:
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ pip list
Package Version
------------ ---------
certifi 2018.8.13
click 6.7
Flask 1.0.2
itsdangerous 0.24
Jinja2 2.10
MarkupSafe 1.0
pip 10.0.1
setuptools 40.0.0
Werkzeug 0.14.1
wheel 0.31.1
As a bonus, I double-checked to see that a conda environment created this way would be able to handle pip list or pip install: same error. However, when using conda update (for example: conda update sqlite), conda install or any of the conda-counterpart code, they all work as fully intended. Updating sqlite in your local environment update exactly that copy of sqlite and not the one in your anaconda system. Whew!
Related
I am using Anaconda to build environments. I've been using pip install my_module the entire time, and I've recently realized this is incorrect. The environment I made is called dp_offfsets_environment and I have been able to run my flask app I wrote in IntelliJ using an SDK linked to this environment.
Why is this? Seen below, when calling conda list no packages are installed under my environment so I would expect my flask app to fail.
What is pip list displaying below? Is it showing packages installed in my base environment?
How do you recommend uninstalling these packages and moving them into the actual environment?
(dp_offsets_environment) C:\WINDOWS\system32>pip list
Package Version
---------------------- -------
click 8.0.3
colorama 0.4.4
cycler 0.10.0
Flask 2.0.2
fonttools 4.29.1
importlib-metadata 4.10.1
itsdangerous 2.0.1
Jinja2 3.0.3
kiwisolver 1.3.1
MarkupSafe 2.0.1
matlab 0.1
mysql-connector-python 8.0.26
packaging 21.3
pandas 1.3.0
Pillow 8.3.1
pip 22.0.2
pyparsing 2.4.7
python-dateutil 2.8.1
pytz 2021.1
setuptools 47.1.0
six 1.16.0
typing_extensions 4.0.1
Werkzeug 2.0.2
wheel 0.37.1
zipp 3.7.0
(dp_offsets_environment) C:\WINDOWS\system32>conda list
# packages in environment at C:\Users\ckurtz\.conda\envs\dp_offsets_environment:
#
# Name Version Build Channel
When you run conda create -n dp_offsets_environment, then conda creates an empty environment, i.e. with no packages. conda does not install python, pip or anything else in the environment. So when you run pip with that env activated, you will still get whatever pip is first located in your PATH (most likely from your base env).
So whenever you ran pip install nothing was installed to your currently active conda env and your flask app was probably not using that conda env, but also used the python that corresponds to your pip.
You need to install python/pip to your env and then install to your env.
See this as an example (I added annotation with ###):
(base) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### We are in base, so pip from base is found
(base) C:\Users\FlyingTeller>conda create -n someEnv
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
### Note: No packages are being installed
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate someEnv
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) C:\Users\FlyingTeller>conda activate someEnv
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### <----- pip still from base env
(someEnv) C:\Users\FlyingTeller>conda install python pip
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
added / updated specs:
- pip
- python
### Download and installation pruned
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\envs\someEnv\Scripts\pip.exe ### Only now pip is in this env
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe
(someEnv) C:\Users\FlyingTeller>
Side Note
I've been using pip install my_module the entire time, and I've recently realized this is incorrect
This is a misconception of a commonly given advice "Do not use pip in a conda env". This advice is always a bit strong. There is no reason to refrain from using pip in a conda env all together. There is a good reason to be careful and aware of certain things that can easily break your env. There is also a good post on anaconda.org about this.
For anyone looking back on this, I discovered I had another instance of Anaconda on my machine. My company gives out laptops with Spyder installed which by default, has it's own version of Anaconda. I did not know Spyder came with Anaconda so I accidently installed two versions of Anaconda during my initial setup. To fix this I had to go through and uninstall both instances of Anaconda on my machine including their packages. Once completely removed, I reinstalled Anaconda, and my problems were alleviated.
I'm not sure I understand what's going on here, since I create a virtual environment and it seems to be using the "all users" installation, but whenever I update pip, it does so in a user-specific folder?
I've just installed the latest Python version (using the Windows installer from python.org). I carefully selected the custom installation option and selected to add Python to environment variables and "install for all users", so I saw how the default user folder changed to C:\Program Files\Python39, as seen here:
Then I tried to update pip (from version 21.1.1 to 21.1.2) so I ran python -m pip install --upgrade pip and got this message:
Defaulting to user installation because normal site-packages is not
writeable Requirement already satisfied: pip in c:\program
files\python39\lib\site-packages (21.1.1) Collecting pip Using
cached pip-21.1.2-py3-none-any.whl (1.5 MB) Installing collected
packages: pip WARNING: The scripts pip.exe, pip3.9.exe and pip3.exe
are installed in
'C:\Users\Currentuser\AppData\Roaming\Python\Python39\Scripts' which is
not on PATH. Consider adding this directory to PATH or, if you
prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.1.2 WARNING: You are using pip version
21.1.1; however, version 21.1.2 is available. You should consider upgrading via the 'C:\Program Files\Python39\python.exe -m pip install
--upgrade pip' command.
So what's with this C:\Users\Currentuser\AppData\Roaming\Python\Python39 folder?
If I run pip --version` I get:
pip 21.1.2 from
C:\Users\Currentuser\AppData\Roaming\Python\Python39\site-packages\pip
(python 3.9)
So it looks like it has created a user-only installation in C:\Users\Currentuser\AppData\Roaming\Python\Python39?
The problem is that now I have 2 versions, and venv uses the old one. I created a C:\Users\Currentuser\Desktop\test folder and then created a virtual environment within it. In this environment, pip list shows me it uses the old pip version:
Package Version
---------- -------
pip 21.1.1
setuptools 56.0.0
WARNING: You are using pip version 21.1.1; however, version 21.1.2 is available.
You should consider upgrading via the 'c:\users\currentuser\desktop\test\my-venv\scripts\python.exe -m pip install --upgrade pip' command.
So how do I prevent Python from using this C:\Users\Currentuser\AppData\Roaming\Python\Python39 folder and force it to use the C:\Program Files\Python39\ I installed to?
create virtual environment conda create test python=3.7
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/mac/miniconda3/envs/test
added / updated specs:
- python=3.7
The following NEW packages will be INSTALLED:
ca-certificates pkgs/main/osx-64::ca-certificates-2019.10.16-0
certifi pkgs/main/osx-64::certifi-2019.9.11-py37_0
libcxx pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
libcxxabi pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
libedit pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
libffi pkgs/main/osx-64::libffi-3.2.1-h475c297_4
ncurses pkgs/main/osx-64::ncurses-6.1-h0a44026_1
openssl pkgs/main/osx-64::openssl-1.1.1d-h1de35cc_3
pip pkgs/main/osx-64::pip-19.3.1-py37_0
python pkgs/main/osx-64::python-3.7.5-h359304d_0
readline pkgs/main/osx-64::readline-7.0-h1de35cc_5
setuptools pkgs/main/osx-64::setuptools-42.0.1-py37_0
sqlite pkgs/main/osx-64::sqlite-3.30.1-ha441bb4_0
tk pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
wheel pkgs/main/osx-64::wheel-0.33.6-py37_0
xz pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3
check the reference of pip, which is correct.
(test) mac#mac-MBP ~ % which pip
/Users/mac/miniconda3/envs/test/bin/pip
(test) mac#mac-MBP ~ % pip --version
pip 19.3.1 from /Users/mac/miniconda3/envs/test/lib/python3.7/site-packages/pip (python 3.7)
install pip install jupyter
shows
Requirement already satisfied: jupyter in ./.local/lib/python3.7/site-packages (1.0.0)
I can use conda to uninstall and reinstall pip, which will solve the Requirement already satisfied issue, but it also upgrades python3.7 to 3.8 for some reason. I have to use 3.7 because tensorflow currently only supports 3.7.
Not sure what's wrong here. I have used the same commands to create environment and also worked smoothly. The only change I can think of is that I changed from bash to zsh because of Catalina. Any help would be appreciated.
The issue was I installed Jupyter with --user flag before, so the package was installed in the user directory outside of the conda env directory. pip list --user shows these packages.
The solution is to uninstall these packages first.
pip freeze --user > packages.txt
pip uninstall -r packages.txt
You can call the pip with the python version you want to install the package for. For more detail please go through the link am pasting below:
Install a module using pip for specific python version
and do not forget to restart your anaconda after installing.
Hope it'll solve your issue
While installing Turicreate using pip on my macOS it gave me an error,
Could not find a version that satisfies the requirement turicreate (from versions: )
No matching distribution found for turicreate
I did the following steps:
conda create -n venv python=2.7 anaconda
source activate venv
pip install -U turicreate
pip install --upgrade pip
Requirement already up-to-date: pip in ./anaconda/lib/python3.5/site-packages
Although you create your venv virtual environment with python=2.7, the message you get when trying to upgrade pip indicates that you are running it in Python 3.5; the path
./anaconda/lib/python3.5/site-packages
clearly refers to the base Anaconda installation, and not to your venv virtual environment.
Given that, the error message you get is not unexpected, since Turicreate is not yet available for Python 3.5:
System Requirements
Python 2.7 (Python 3.5+ support coming soon)
x86_64 architecture
I use Anaconda (because it is awesome), and the packages available through conda install are quite extensive. However now and then I do need to install a package that isn't available in the conda repositories, and so get it from pypi instead.
My question: when I run the command conda update --all, will conda also update these pypi packages? Or do I have to update them separately? The conda docs don't seem to contain an answer to this. This question and answer seems to indicate that no, conda does not manage pypi packages, but I'm still uncertain.
No, conda update and conda install don't update packages installed with pip (or install them using pip).
These conda commands only check your "default" anaconda-channels or the ones specified with -c, they ignore everything else. One exception is conda list which shows also the packages installed with pip, these are marked with <pip> and won't be updated.
One example using pip and six:
$ conda create -n testenv python=3.5
Fetching package metadata .................
Solving package specifications: .
Package plan for installation in environment testenv:
The following NEW packages will be INSTALLED:
pip: 9.0.1-py35_1
python: 3.5.3-3
setuptools: 27.2.0-py35_1
vs2015_runtime: 14.0.25123-0
wheel: 0.29.0-py35_0
Proceed ([y]/n)? y
$ activate testenv
Installing six with pip (old version):
(testenv) $ pip install six==1.6
Collecting six==1.6
Downloading six-1.6.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.6.0
conda update doesn't update it (note that six isn't listed in the "all requested packages" but it's listed in conda list):
(testenv) $ conda update --all
Fetching package metadata .................
Solving package specifications: .
# All requested packages already installed.
# packages in environment at testenv:
#
pip 9.0.1 py35_1
python 3.5.3 3
setuptools 27.2.0 py35_1
vs2015_runtime 14.0.25123 0
wheel 0.29.0 py35_0
(testenv) $ conda list
# packages in environment at testenv:
#
pip 9.0.1 py35_1
python 3.5.3 3
setuptools 27.2.0 py35_1
six 1.6.0 <pip>
vs2015_runtime 14.0.25123 0
wheel 0.29.0 py35_0
But it can be upgraded with pip:
(testenv) $ pip install six --upgrade
Collecting six
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: six
Found existing installation: six 1.6.0
Uninstalling six-1.6.0:
Successfully uninstalled six-1.6.0
Successfully installed six-1.10.0
Just to show that there is a newer version of six in the anaconda channel (which was ignored when I did conda update):
(testenv) $ conda install six
Fetching package metadata .................
Solving package specifications: .
Package plan for installation in environment testenv:
The following NEW packages will be INSTALLED:
six: 1.10.0-py35_0
Proceed ([y]/n)?
Conda 4.6 has an experimental feature to enable interoperability with pip-installed packages. Use conda config --set pip_interop_enabled true. Non-conda-installed python packages that can be "managed" by conda (i.e. removed) may be updated/changed to satisfy the current solve. Manageable packages were typically installed from wheels. Sdists installed with newer versions of pip are also typically manageable. However conda won't switch out the non-conda-installed package for a conda package if the versions are equivalent.
Non-conda-installed python packages that can't be managed will anchor the environment in place until they are removed by other means. An example of unmanageable packages are "editable" installs that used pip install -e.
All of this applies to conda update --all.
This question is old, but here's a batch script that might help with automating this process on Windows. It involves going through conda list and finding packages marked with the pypi tag, which are then subsequently upgraded with pip --upgrade en masse (assuming they are out-of-date; otherwise the standard Requirement already up-to-date message will be returned).
Place the following in a batch file (e.g., condapip.bat) and try it out:
#echo off
set packages=pip install --upgrade
for /f "tokens=1" %%i in ('conda list ^| findstr /R /C:"pypi"') do (call :join %%i)
#echo on
%packages%
#echo off
goto :eof
:join
set packages=%packages% %1
goto :eof