Develop Reference

Why is my flask app running with no modules installed in my environment?

I am using Anaconda to build environments. I've been using pip install my_module the entire time, and I've recently realized this is incorrect. The environment I made is called dp_offfsets_environment and I have been able to run my flask app I wrote in IntelliJ using an SDK linked to this environment.
Why is this? Seen below, when calling conda list no packages are installed under my environment so I would expect my flask app to fail.
What is pip list displaying below? Is it showing packages installed in my base environment?
How do you recommend uninstalling these packages and moving them into the actual environment?
(dp_offsets_environment) C:\WINDOWS\system32>pip list
Package Version
---------------------- -------
click 8.0.3
colorama 0.4.4
cycler 0.10.0
Flask 2.0.2
fonttools 4.29.1
importlib-metadata 4.10.1
itsdangerous 2.0.1
Jinja2 3.0.3
kiwisolver 1.3.1
MarkupSafe 2.0.1
matlab 0.1
mysql-connector-python 8.0.26
packaging 21.3
pandas 1.3.0
Pillow 8.3.1
pip 22.0.2
pyparsing 2.4.7
python-dateutil 2.8.1
pytz 2021.1
setuptools 47.1.0
six 1.16.0
typing_extensions 4.0.1
Werkzeug 2.0.2
wheel 0.37.1
zipp 3.7.0
(dp_offsets_environment) C:\WINDOWS\system32>conda list
# packages in environment at C:\Users\ckurtz\.conda\envs\dp_offsets_environment:
#
# Name Version Build Channel

When you run conda create -n dp_offsets_environment, then conda creates an empty environment, i.e. with no packages. conda does not install python, pip or anything else in the environment. So when you run pip with that env activated, you will still get whatever pip is first located in your PATH (most likely from your base env).
So whenever you ran pip install nothing was installed to your currently active conda env and your flask app was probably not using that conda env, but also used the python that corresponds to your pip.
You need to install python/pip to your env and then install to your env.
See this as an example (I added annotation with ###):
(base) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### We are in base, so pip from base is found
(base) C:\Users\FlyingTeller>conda create -n someEnv
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
### Note: No packages are being installed
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate someEnv
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) C:\Users\FlyingTeller>conda activate someEnv
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe ### <----- pip still from base env
(someEnv) C:\Users\FlyingTeller>conda install python pip
Collecting package metadata (current_repodata.json): done
Solving environment: done
Please update conda by running
$ conda update -n base conda
## Package Plan ##
environment location: C:\Users\FlyingTeller\miniconda3\envs\someEnv
added / updated specs:
- pip
- python
### Download and installation pruned
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(someEnv) C:\Users\FlyingTeller>where pip
C:\Users\FlyingTeller\miniconda3\envs\someEnv\Scripts\pip.exe ### Only now pip is in this env
C:\Users\FlyingTeller\miniconda3\Scripts\pip.exe
(someEnv) C:\Users\FlyingTeller>
Side Note
I've been using pip install my_module the entire time, and I've recently realized this is incorrect
This is a misconception of a commonly given advice "Do not use pip in a conda env". This advice is always a bit strong. There is no reason to refrain from using pip in a conda env all together. There is a good reason to be careful and aware of certain things that can easily break your env. There is also a good post on anaconda.org about this.

For anyone looking back on this, I discovered I had another instance of Anaconda on my machine. My company gives out laptops with Spyder installed which by default, has it's own version of Anaconda. I did not know Spyder came with Anaconda so I accidently installed two versions of Anaconda during my initial setup. To fix this I had to go through and uninstall both instances of Anaconda on my machine including their packages. Once completely removed, I reinstalled Anaconda, and my problems were alleviated.

local pip in conda environment checks globally and says Requirement already satisfied

create virtual environment conda create test python=3.7
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/mac/miniconda3/envs/test
added / updated specs:
- python=3.7
The following NEW packages will be INSTALLED:
ca-certificates pkgs/main/osx-64::ca-certificates-2019.10.16-0
certifi pkgs/main/osx-64::certifi-2019.9.11-py37_0
libcxx pkgs/main/osx-64::libcxx-4.0.1-hcfea43d_1
libcxxabi pkgs/main/osx-64::libcxxabi-4.0.1-hcfea43d_1
libedit pkgs/main/osx-64::libedit-3.1.20181209-hb402a30_0
libffi pkgs/main/osx-64::libffi-3.2.1-h475c297_4
ncurses pkgs/main/osx-64::ncurses-6.1-h0a44026_1
openssl pkgs/main/osx-64::openssl-1.1.1d-h1de35cc_3
pip pkgs/main/osx-64::pip-19.3.1-py37_0
python pkgs/main/osx-64::python-3.7.5-h359304d_0
readline pkgs/main/osx-64::readline-7.0-h1de35cc_5
setuptools pkgs/main/osx-64::setuptools-42.0.1-py37_0
sqlite pkgs/main/osx-64::sqlite-3.30.1-ha441bb4_0
tk pkgs/main/osx-64::tk-8.6.8-ha441bb4_0
wheel pkgs/main/osx-64::wheel-0.33.6-py37_0
xz pkgs/main/osx-64::xz-5.2.4-h1de35cc_4
zlib pkgs/main/osx-64::zlib-1.2.11-h1de35cc_3
check the reference of pip, which is correct.
(test) mac#mac-MBP ~ % which pip
/Users/mac/miniconda3/envs/test/bin/pip
(test) mac#mac-MBP ~ % pip --version
pip 19.3.1 from /Users/mac/miniconda3/envs/test/lib/python3.7/site-packages/pip (python 3.7)
install pip install jupyter
shows
Requirement already satisfied: jupyter in ./.local/lib/python3.7/site-packages (1.0.0)
I can use conda to uninstall and reinstall pip, which will solve the Requirement already satisfied issue, but it also upgrades python3.7 to 3.8 for some reason. I have to use 3.7 because tensorflow currently only supports 3.7.
Not sure what's wrong here. I have used the same commands to create environment and also worked smoothly. The only change I can think of is that I changed from bash to zsh because of Catalina. Any help would be appreciated.

The issue was I installed Jupyter with --user flag before, so the package was installed in the user directory outside of the conda env directory. pip list --user shows these packages.
The solution is to uninstall these packages first.
pip freeze --user > packages.txt
pip uninstall -r packages.txt

You can call the pip with the python version you want to install the package for. For more detail please go through the link am pasting below:
Install a module using pip for specific python version
and do not forget to restart your anaconda after installing.
Hope it'll solve your issue

Why does pip install from github fail when pip install -e succeeds?

There's a git repo at https://github.com/Ivancic91/LammpsIO that I can install in "editable" mode by cloning and running pip install -e /path/to/LammpsIO. When I do this, I'm able to run import LammpsIO successfully in python.
However, when I pip install directly from github with pip install git+https://github.com/Ivancic91/LammpsIO.git I run into problems. The actual pip install command looks successful with the usual success messages:
Successfully built LammpsIO
Installing collected packages: LammpsIO
Successfully installed LammpsIO-0.0.1
But when I try to import LammpsIO, I get ModuleNotFoundError: No module named 'LammpsIO'.
I thought this might be some sort of path error, but which pip and which python both point to the same conda environment folder.
Here are a couple more outputs in case anyone needs them.
After pip install git+...:
conda list returns:
krb5 1.16.3 h24a3359_1000 conda-forge
lammpsio 0.0.1 pypi_0 pypi
libcurl 7.64.0 h76de61e_0 conda-forge
pip list returns:
kiwisolver 1.0.1
LammpsIO 0.0.1
MarkupSafe 1.1.0
After pip install -e ...:
conda list returns:
krb5 1.16.3 h24a3359_1000 conda-forge
lammpsio 0.0.1 dev_0 libcurl 7.64.0 h76de61e_0 conda-forge
pip list returns:
kiwisolver 1.0.1
LammpsIO 0.0.1 /path/to/LammpsIO/src
MarkupSafe 1.1.0
I'm guessing this has to do with the unorthodox structure of LammpsIO, whose setup.py file contains:
package_dir = {'': 'src'},
packages = find_packages(where='src'),
but I'm surprised that pip install from github fails where pip install -e /path works.

Your reasoning seems sound here. I'd think that what you're trying to would work. I created a new virtualenv and tried the github install myself. I then looked in site-packages. What I see is strange. There's a 'LammpsIO-0.0.1.dist-info' directory containing basic metadata, but no Lammps package itself. I did the find 'find . -iname "*mmps*"' in site-packages, and only got that info directory. So it seems like the package didn't actually install. - LammpsIO does show up when I do "pip list".
If you "install" the package by download a zip and doing "pip install [zip]", you get the same output from pip and end up with the same problem. So it has nothing to do with installing directly from github. It has to do with actually installing the package vs just linking its source into your enviroment .
My guess is that the Lammps installer is just broken - Since '-e' doesn't install anything, but just creates a link, that would explain why it works but a regular install does not.

Package updates on the system affecting virtual environment?

I am in a virtual environment. I verified that in my active terminal using which python and can confirm that I am in fact in a virtual environment. Printing pip list, according to the official documentation, should list the packages in this virtual environment. Here's an output:
Package Version
---------------------------------- -------------------
alabaster 0.7.10
anaconda-client 1.6.5
... (truncated)
pip 10.0.1
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
When I fire up another terminal, this time making sure I'm outside of that virtual environment and proceed to upgrade pip:
pip install --upgrade pip
Upon doing that, I verified that my pip package on the system has been updated to version 18.0. Here is the confusing part: I switched back into the virtual environment and use pip list and the pip version in my virtual environment is now pip 18.0.
Why is it that upgrading the pip version outside of that environment subsequently update the pip in my virtual environment from 10.0.1 at all? Have I misunderstood how virtual environments work? I'm not new to python but have not used virtual environment so forgive me if it's something very fundamental. In my understanding the primary value in using virtual envs is that I can be upgrading my system-wide packages (such as pip, flask etc) without any of that changes being affected in my virtual environments. Virtual environments should be isolated environment at all?
If it matters, I'm using the default venv and not virtualenvwrapper or any other wrapper tool.

The Problem
tl;dr: The problem is with conda. I use conda and apparently that causes some issues with managing and installing packages in virtual environment created via venv because a local instance of pip was not present.
Solution A: conda install -n myenv pip where myenv refers to the name of your virtual environment
Solution B: Use conda list, conda create to work with environments in a way that is 100% compatible with conda
/end tl;dr
Here's a breakdown of the problem. When I'm using an anaconda's version of python and I decide to look at the list of packages I have in my environment; Supposed I were to call pip freeze or pip list, whether I'm in a virtual environment or not wouldn't matter. It returns exactly the same list of packages from conda's corresponding site-packages folder.
When I'm inside a virtual environment, running which python does seem to point to an isolated version of the python instance (acco is the name of that instance):
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ which python
/Users/Samuel/Dropbox/Projects/Python/Acco/acco/bin/python
However, while in this virtual environment, running pip list or pip list --local would still refer to the same set of packages - and that is because it is still point to the conda's version of package directory (Yes, even in a virtual environment).
Specifically, with or without, inside or outside of a virtual environment, the pip list points to packages installed in the /anaconda3/lib/../site-packages directory:
import sys
sys.prefix
'/Users/Samuel/anaconda3'
import site
site.getsitepackages()
['/Users/Samuel/anaconda3/lib/python3.6/site-packages']
The really problematic part of this is that while in the virtual environment, you have essentially no libraries installed. Installing from requirements.txt' usingpip install -r requirements.txtor just installing packages at all using plain oldpip install` wouldn't work if any of the packages you're attempting to install already exist at the conda's directory. Instead, you'll get a message that doesn't look like an error, and it stops just right there. The package you're attempting to install is not installed into the local directory.
# same as pip install Flask==0.12.2
pip install -r requirements.txt
Requirement already satisfied: Flask==0.12.2 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Werkzeug>=0.7 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.12.2)
Requirement already satisfied: Jinja2>=2.4 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (2.9.6)
Requirement already satisfied: itsdangerous>=0.21 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (0.24)
Requirement already satisfied: click>=2.0 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Flask==0.12.2->-r requirements.txt (line 1)) (6.7)
Requirement already satisfied: MarkupSafe>=0.23 in /Users/Samuel/anaconda3/lib/python3.6/site-packages (from Jinja2>=2.4->Flask==0.12.2->-r requirements.txt (line 1)) (1.0)
As a reminder, we're still executing your python commands using the python in our virtual environment (../Dropbox/Projects/Python/Acco/acco/bin/python) and not the conda distribution. This virtual environment has no packages in its isolated lib folder, and you can't install any libraries into it as pip will be stopped with the "requirement already satisfied" message and exit (or terminate its attempt).
This means that while in that virtual environment, trying to run a python script or app that has dependencies will certainly fail. Building from the above example, your Flask app app.py won't run because it couldn't find flask. Goes without saying, since your virtual environment has no packages and it won't let you install any.
The Solution
The solution is that if you're using a conda distribution of python, check the packages you have installed on your system using conda list instead of pip list for maximal consistency.
(acco) Samuels-MacBook-Pro:Acco Samuel$ conda list
# packages in environment at /Users/Samuel/anaconda3:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py36h2fc01ae_0
_r-mutex 1.0.0 mro_2
alabaster 0.7.10 py36h174008c_0
anaconda custom py36ha4fed55_0
Notice that conda tells you the environment from which the libraries are listed from. At this stage:
Deactivate the environment using deactive
Make virtual environment using conda create --name acco python=3.6.3 flask sqlite, here we're using acco as the virtual environment's name, a specific version of python, and optionally some other dependencies
Activate the environment using source activate acco
Now, when you do conda list again, being in your virtual environment:
# packages in environment at /Users/Samuel/anaconda3/envs/acco:
#
# Name Version Build Channel
ca-certificates 2018.03.07 0
certifi 2018.8.13 py36_0
click 6.7 py36hec950be_0
flask 1.0.2 py36_1
If you want to manage the dependencies of your app using an old-school requirement.txt file, then simplest way is to conda list --export > requirements.txt. From this point on, use conda list in place of pip list and use conda create as well as source activate in place of its venv counterpart.
The anaconda's main documentation suggest:
To use pip in your environment, in your Terminal window or an Anaconda
Prompt, run:
conda install -n acco pip
source activate acco
pip <pip_subcommand>
This solution worked too. Following the steps above, I ended up with:
(acco) Samuels-MacBook-Pro:Accomplish Samuel$ pip list
Package Version
------------ ---------
certifi 2018.8.13
click 6.7
Flask 1.0.2
itsdangerous 0.24
Jinja2 2.10
MarkupSafe 1.0
pip 10.0.1
setuptools 40.0.0
Werkzeug 0.14.1
wheel 0.31.1
As a bonus, I double-checked to see that a conda environment created this way would be able to handle pip list or pip install: same error. However, when using conda update (for example: conda update sqlite), conda install or any of the conda-counterpart code, they all work as fully intended. Updating sqlite in your local environment update exactly that copy of sqlite and not the one in your anaconda system. Whew!

Why can't conda find pip installed module in prefix (within conda evironment)

I used pip to install the Resource module to the default conda environment on my laptop: (C:\Users\my_username\Anaconda2). I think it is called root. I installed pip to the conda environment and so I'm 90% sure the resource was installed within the environment. And indeed running conda list shows that the package is listed within the environment. Here is a section of the output:
# packages in environment at C:\Users\conna\Anaconda2:
#
qtpy 1.2.1 py27_0
requests 2.14.2 py27_0
Resource 0.2.0 <pip>
rope 0.9.4 py27_1
ruamel_yaml 0.11.14 py27_1
scandir 1.5 py27_0
scikit-image 0.13.0 np112py27_0
However when I run
conda update Resource
I get the following error:
PackageNotInstalledError: Package is not installed in prefix.
prefix: C:\Users\conna\Anaconda2
package name: Resource
How is it possible that conda list shows the module is present but conda update can't see them? I also noticed that conda update doesn't recognize any packages with <pip>. What is happening?

conda only manages the packages that are installed using a conda command. If you installed a package with pip (or using python setup.py install or develop) it will show up with conda list (because that shows all packages no matter how they were installed) but conda won't manage that package. Simply because it doesn't know how!
So if you installed a package with pip you also need to upgrade/update it with pip:
pip install [package_name] --upgrade

Try this;
pip install Resource --upgrade