Develop Reference

Why is my flask app running with no modules installed in my environment?

I am using Anaconda to build environments. I've been using pip install my_module the entire time, and I've recently realized this is incorrect. The environment I made is called dp_offfsets_environment and I have been able to run my flask app I wrote in IntelliJ using an SDK linked to this environment.
Why is this? Seen below, when calling conda list no packages are installed under my environment so I would expect my flask app to fail.
What is pip list displaying below? Is it showing packages installed in my base environment?
How do you recommend uninstalling these packages and moving them into the actual environment?
(dp_offsets_environment) C:\WINDOWS\system32>pip list
Package Version
---------------------- -------
click 8.0.3
colorama 0.4.4
cycler 0.10.0
Flask 2.0.2
fonttools 4.29.1
importlib-metadata 4.10.1
itsdangerous 2.0.1
Jinja2 3.0.3
kiwisolver 1.3.1
MarkupSafe 2.0.1
matlab 0.1
mysql-connector-python 8.0.26
packaging 21.3
pandas 1.3.0
Pillow 8.3.1
pip 22.0.2
pyparsing 2.4.7
python-dateutil 2.8.1
pytz 2021.1
setuptools 47.1.0
six 1.16.0
typing_extensions 4.0.1
Werkzeug 2.0.2
wheel 0.37.1
zipp 3.7.0
(dp_offsets_environment) C:\WINDOWS\system32>conda list
# packages in environment at C:\Users\ckurtz\.conda\envs\dp_offsets_environment:
#
# Name Version Build Channel

When you run conda create -n dp_offsets_environment, then conda creates an empty environment, i.e. with no packages. conda does not install python, pip or anything else in the environment. So when you run pip with that env activated, you will still get whatever pip is first located in your PATH (most likely from your base env).
So whenever you ran pip install nothing was installed to your currently active conda env and your flask app was probably not using that conda env, but also used the python that corresponds to your pip.
You need to install python/pip to your env and then install to your env.
See this as an example (I added annotation with ###):
(base) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### We are in base, so pip from base is found
(base) C:\Users\FlyingTeller>conda create -n someEnv
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
### Note: No packages are being installed
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate someEnv
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) C:\Users\FlyingTeller>conda activate someEnv
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### <----- pip still from base env
(someEnv) C:\Users\FlyingTeller>conda install python pip
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
added / updated specs:
- pip
- python
### Download and installation pruned
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\envs\someEnv\Scripts\pip.exe ### Only now pip is in this env
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe
(someEnv) C:\Users\FlyingTeller>
Side Note
I've been using pip install my_module the entire time, and I've recently realized this is incorrect
This is a misconception of a commonly given advice "Do not use pip in a conda env". This advice is always a bit strong. There is no reason to refrain from using pip in a conda env all together. There is a good reason to be careful and aware of certain things that can easily break your env. There is also a good post on anaconda.org about this.

For anyone looking back on this, I discovered I had another instance of Anaconda on my machine. My company gives out laptops with Spyder installed which by default, has it's own version of Anaconda. I did not know Spyder came with Anaconda so I accidently installed two versions of Anaconda during my initial setup. To fix this I had to go through and uninstall both instances of Anaconda on my machine including their packages. Once completely removed, I reinstalled Anaconda, and my problems were alleviated.

Package updates on the system affecting virtual environment?

I am in a virtual environment. I verified that in my active terminal using which python and can confirm that I am in fact in a virtual environment. Printing pip list, according to the official documentation, should list the packages in this virtual environment. Here's an output:
Package Version
---------------------------------- -------------------
alabaster 0.7.10
anaconda-client 1.6.5
... (truncated)
pip 10.0.1
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
When I fire up another terminal, this time making sure I'm outside of that virtual environment and proceed to upgrade pip:
pip install --upgrade pip
Upon doing that, I verified that my pip package on the system has been updated to version 18.0. Here is the confusing part: I switched back into the virtual environment and use pip list and the pip version in my virtual environment is now pip 18.0.
Why is it that upgrading the pip version outside of that environment subsequently update the pip in my virtual environment from 10.0.1 at all? Have I misunderstood how virtual environments work? I'm not new to python but have not used virtual environment so forgive me if it's something very fundamental. In my understanding the primary value in using virtual envs is that I can be upgrading my system-wide packages (such as pip, flask etc) without any of that changes being affected in my virtual environments. Virtual environments should be isolated environment at all?
If it matters, I'm using the default venv and not virtualenvwrapper or any other wrapper tool.

The Problem
tl;dr: The problem is with conda. I use conda and apparently that causes some issues with managing and installing packages in virtual environment created via venv because a local instance of pip was not present.
Solution A: conda install -n myenv pip where myenv refers to the name of your virtual environment
Solution B: Use conda list, conda create to work with environments in a way that is 100% compatible with conda
/end tl;dr
Here's a breakdown of the problem. When I'm using an anaconda's version of python and I decide to look at the list of packages I have in my environment; Supposed I were to call pip freeze or pip list, whether I'm in a virtual environment or not wouldn't matter. It returns exactly the same list of packages from conda's corresponding site-packages folder.
When I'm inside a virtual environment, running which python does seem to point to an isolated version of the python instance (acco is the name of that instance):
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ which python
/Users/Samuel/Dropbox/Projects/Python/Acco/acco/bin/python
However, while in this virtual environment, running pip list or pip list --local would still refer to the same set of packages - and that is because it is still point to the conda's version of package directory (Yes, even in a virtual environment).
Specifically, with or without, inside or outside of a virtual environment, the pip list points to packages installed in the /anaconda3/lib/../site-packages directory:
import sys
sys.prefix
'/Users/Samuel/anaconda3'
import site
site.getsitepackages()
['/Users/Samuel/anaconda3/lib/python3.6/site-packages']
The really problematic part of this is that while in the virtual environment, you have essentially no libraries installed. Installing from requirements.txt' usingpip install -r requirements.txtor just installing packages at all using plain oldpip install` wouldn't work if any of the packages you're attempting to install already exist at the conda's directory. Instead, you'll get a message that doesn't look like an error, and it stops just right there. The package you're attempting to install is not installed into the local directory.
# same as pip install Flask==0.12.2
pip install -r requirements.txt
Requirement already satisfied: Flask==0.12.2 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Werkzeug>=0.7 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Jinja2>=2.4 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (2.9.6)
Requirement already satisfied: itsdangerous>=0.21 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.24)
Requirement already satisfied: click>=2.0 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (6.7)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.12.2->-r requirements.txt (line 1)) (1.0)
As a reminder, we're still executing your python commands using the python in our virtual environment (../Dropbox/Projects/Python/Acco/acco/bin/python) and not the conda distribution. This virtual environment has no packages in its isolated lib folder, and you can't install any libraries into it as pip will be stopped with the "requirement already satisfied" message and exit (or terminate its attempt).
This means that while in that virtual environment, trying to run a python script or app that has dependencies will certainly fail. Building from the above example, your Flask app app.py won't run because it couldn't find flask. Goes without saying, since your virtual environment has no packages and it won't let you install any.
The Solution
The solution is that if you're using a conda distribution of python, check the packages you have installed on your system using conda list instead of pip list for maximal consistency.
(acco) Samuels-MacBook-Pro:Acco Samuel$ conda list
# packages in environment at /Users/Samuel/anaconda3:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py36h2fc01ae_0
_r-mutex 1.0.0 mro_2
alabaster 0.7.10 py36h174008c_0
anaconda custom py36ha4fed55_0
Notice that conda tells you the environment from which the libraries are listed from. At this stage:
Deactivate the environment using deactive
Make virtual environment using conda create --name acco python=3.6.3 flask sqlite, here we're using acco as the virtual environment's name, a specific version of python, and optionally some other dependencies
Activate the environment using source activate acco
Now, when you do conda list again, being in your virtual environment:
# packages in environment at /Users/Samuel/anaconda3/envs/acco:
#
# Name Version Build Channel
ca-certificates 2018.03.07 0
certifi 2018.8.13 py36_0
click 6.7 py36hec950be_0
flask 1.0.2 py36_1
If you want to manage the dependencies of your app using an old-school requirement.txt file, then simplest way is to conda list --export > requirements.txt. From this point on, use conda list in place of pip list and use conda create as well as source activate in place of its venv counterpart.
The anaconda's main documentation suggest:
To use pip in your environment, in your Terminal window or an Anaconda
Prompt, run:
conda install -n acco pip
source activate acco
pip <pip_subcommand>
This solution worked too. Following the steps above, I ended up with:
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ pip list
Package Version
------------ ---------
certifi 2018.8.13
click 6.7
Flask 1.0.2
itsdangerous 0.24
Jinja2 2.10
MarkupSafe 1.0
pip 10.0.1
setuptools 40.0.0
Werkzeug 0.14.1
wheel 0.31.1
As a bonus, I double-checked to see that a conda environment created this way would be able to handle pip list or pip install: same error. However, when using conda update (for example: conda update sqlite), conda install or any of the conda-counterpart code, they all work as fully intended. Updating sqlite in your local environment update exactly that copy of sqlite and not the one in your anaconda system. Whew!

Why can't conda find pip installed module in prefix (within conda evironment)

I used pip to install the Resource module to the default conda environment on my laptop: (C:\Users\my_username\Anaconda2). I think it is called root. I installed pip to the conda environment and so I'm 90% sure the resource was installed within the environment. And indeed running conda list shows that the package is listed within the environment. Here is a section of the output:
# packages in environment at C:\Users\conna\Anaconda2:
#
qtpy 1.2.1 py27_0
requests 2.14.2 py27_0
Resource 0.2.0 <pip>
rope 0.9.4 py27_1
ruamel_yaml 0.11.14 py27_1
scandir 1.5 py27_0
scikit-image 0.13.0 np112py27_0
However when I run
conda update Resource
I get the following error:
PackageNotInstalledError: Package is not installed in prefix.
prefix: C:\Users\conna\Anaconda2
package name: Resource
How is it possible that conda list shows the module is present but conda update can't see them? I also noticed that conda update doesn't recognize any packages with <pip>. What is happening?

conda only manages the packages that are installed using a conda command. If you installed a package with pip (or using python setup.py install or develop) it will show up with conda list (because that shows all packages no matter how they were installed) but conda won't manage that package. Simply because it doesn't know how!
So if you installed a package with pip you also need to upgrade/update it with pip:
pip install [package_name] --upgrade

Try this;
pip install Resource --upgrade

How to upgrade scikit-learn package in anaconda

I am trying to upgrade package of scikit-learn from 0.16 to 0.17. For that I am trying to use binaries from this website: http://www.lfd.uci.edu/~gohlke/pythonlibs/#scikit-learn. I have Windows 7 x64 bit. I downloaded the relevant package locally and gave following commands and got Requirement already up-to-date:
C:\Users\skumar>pip install --upgrade --use-wheel --no-index --find-links=../../
SOURCE/APPS scikit-learn
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: scikit-learn in c:\anaconda3\lib\site-packages
Then I tried to upgrade it from remote site and got similar result:
C:\Users\skumar>pip install --upgrade --use-wheel --no-index --trusted-host www.
lfd.uci.edu --find-links=http://www.lfd.uci.edu/~gohlke/pythonlibs/ scikit-learn
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: scikit-learn in c:\anaconda3\lib\site-packages
On Remote site there are two versions i.e., 0.16 & 0.17. Is there a way to specify version in command? Or how do you install/upgrade wheel file?

Anaconda comes with the conda package manager which is designed to handle these kinds of upgrades. Start by updating conda itself to get the most recent package lists:
conda update conda
And then install the version of scikit-learn you want
conda install scikit-learn=0.17
All necessary dependencies will be upgraded as well. If you have trouble with conda on Windows, there are some relevant FAQ here: http://docs.continuum.io/anaconda/faq

Following Worked for me for scikit-learn on Anaconda-Jupyter Notebook.
Upgrading my scikit-learn from 0.19.1 to 0.19.2 in anaconda installed on Ubuntu on Google VM instance:
Run the following commands in the terminal:
First, check existing available packages with versions by using:
conda list
It will show different packages and their installed versions in the output. Here check for scikit-learn. e.g. for me, the output was:
scikit-learn 0.19.1 py36hedc7406_0
Now I want to Upgrade to 0.19.2 July 2018 release i.e. latest available version.
conda config --append channels conda-forge
conda install scikit-learn=0.19.2
As you are trying to upgrade to 0.17 version try the following command:
conda install scikit-learn=0.17
Now check the required version of the scikit-learn is installed correctly or not by using:
conda list
For me the Output was:
scikit-learn 0.19.2 py36_blas_openblasha84fab4_201 [blas_openblas] conda-forge
Note: Don't use pip command if you are using Anaconda or Miniconda
I tried following commands:
!conda update conda
!pip install -U scikit-learn
It will install the required packages also will show in the conda list but if you try to import that package it will not work.
On the website http://scikit-learn.org/stable/install.html it is mentioned as:
Warning To upgrade or uninstall scikit-learn installed with Anaconda or conda you should not use the pip.

So to upgrade scikit-learn package, you have to follow below process
Step-1: Open your terminal(Ctrl+Alt+t)
Step-2: Now for checking currently installed packages along with the
versions installed on your
conda environment by typing conda list
Step-3: Now for upgrade type below command
conda update scikit-learn
Hope it helps!!

I would suggest using conda. Conda is an anconda specific package manager. If you want to know more about conda, read the conda docs.
Using conda in the command line, the command below would install scipy 0.17.
conda install scipy=0.17.0

Updating a Specific Library - scikit-learn:
Anaconda (conda):
conda install scikit-learn
Pip Installs Packages (pip):
pip install --upgrade scikit-learn
Verify Update:
conda list scikit-learn
It should now display the current (and desired) version of the scikit-learn library.
For me personally, I tried using the conda command to update the scikit-learn library and it acted as if it were installing the latest version to then later discover (with an execution of the conda list scikit-learn command) that it was the same version as previously and never updated (or recognized the update?). When I used the pip command, it worked like a charm and correctly updated the scikit-learn library to the latest version!
Hope this helps!
More in-depth details of latest version can be found here (be mindful this applies to the scikit-learn library version of 0.22):
Release Highlights for scikit-learn 0.22

I made it work to update to 0.24.1, on Windows 10 64bits, so I share the way I did it with the GUI:
launch Anaconda3 gui
on the left menu, click "environments"
next to "base (root)", click on the green arrow/triangle
select "Open Terminal"
type the command line:
conda install scikit-learn==0.24.1
It worked without error.

If you are using Jupyter in anaconda, after conda update scikit-learn in terminal, close anaconda and restart, otherwise the error will occur again.

Conda and Miniconda doesn't include celery in repository

I am trying to install celery using conda in my Miniconda python distribution. I am using PyCharm to manage my project and packages. I have set up the project to use python installed with miniconda as distribution. Trying to install celery from pycharm package manager gives me no results. As well as trying to install conda from command line. Is there a way to install celerey using conda? Is there a way to use both pip and conda in parallerl? Can this be done through PyCharm?I also have python 2.7.10 installed in my windows pc. So I have two python installments in my system one in
C:\Python27
and one in
C:\Miniconda2
conda install celery return the following
Fetching package metadata: ....
Error: No packages found in current win-64 channels matching: celery
You can search for this package on anaconda.org with
anaconda search -t conda celery
You may need to install the anaconda-client command line client with
conda install anaconda-client

You can check the URL.
https://anaconda.org/conda-forge/celery
conda install -c conda-forge celery=3.1.25

I don't think celery is available as a conda package. You can install with pip into a conda environment, yes. But if at all possible you want to try to stick to conda packages.