Develop Reference

Python setup.py: Could not find suitable distribution for Requirement.parse('tensorflow')

I have tensorflow listed as a requirement in the install_requires section of the setup.py of my project.
When I attempt to install my project into a new Anaconda environment I get the following error:
$ python setup.py install
...
Searching for tensorflow
Reading https://pypi.org/simple/tensorflow/
No local packages or working download links found for tensorflow
error: Could not find suitable distribution for Requirement.parse('tensorflow')
I can get past this by installing tensorflow "manually" via conda:
$ conda install tensorflow
Once I do this the install via setup.py works without a hitch.
Am I mistaken in assuming that something is amiss with my environment? If not then what is going on and how can I avoid this issue? (My concern is that users of my package will not be able to install from source using setup.py)
Another oddity that I assume is related or may provide a clue is that the version of TensorFlow listed in my Anaconda environment is 2.0 but if I import it when running Python it appears to instead be using version 1.15. For example:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvdata_test:
#
# Name Version Build Channel
tensorflow 2.0.0 mkl_py37h66b46cc_0
tensorflow-base 2.0.0 mkl_py37h9204916_0
tensorflow-estimator 2.0.0 pyh2649769_0
$ python
Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.15.0-rc2'
This is on a Dell laptop running Ubuntu 18.04 without a GPU, so perhaps the version shown in the interpreter is akin to tensorflow-cpu? If I run pip freeze I see tensorflow==2.0.0 and tensorflow-cpu==1.15.0rc2, which is a bit confusing...

This appears to have been caused by using Python version 3.8, which is currently an unsupported version of Python. Once I created a new Anaconda environment with Python version 3.7 this issue went away.
The only remaining issue is this error that I see when I run pip install -e . for my project which includes tensorflow:
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorboard<1.16.0,>=1.15.0, but you'll have tensorboard 2.1.0 which is incompatible.
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorflow-estimator==1.15.1, but you'll have tensorflow-estimator 2.1.0 which is incompatible.
So the issue of tensorflow-cpu version 1.15.0rc2 actually being installed when version 2.1.0 shows as being the installed version is still a mystery. To wit:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvd:
#
# Name Version Build Channel
tensorflow 2.1.0 pypi_0 pypi
tensorflow-estimator 2.1.0 pypi_0 pypi
$ python -c "import tensorflow as tf; print(tf.__version__)"
1.15.0-rc2

Anaconda selecting correct enrivonment but wrong sys.path?

I have different anaconda environments. When starting up one of them it seems as if sys.path is still set to some totally wrong directory. Am I not understanding the concepts of environments correctly or is this an error concerning my anaconda setup?
My environments:
fabianwerner ~$ conda env list
# conda environments:
#
base * /Users/fabianwerner/anaconda3/anaconda3
my-rdkit-env /Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env
openpose /Users/fabianwerner/anaconda3/anaconda3/envs/openpose
fabianwerner ~$
I activate the environment 'openpose':
fabianwerner ~$ conda activate openpose
(openpose) fabianwerner ~$
I will try to import 'keras' so let us make sure that it is installed:
(openpose) fabianwerner ~$ pip list | grep -i keras
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
Keras (2.2.4)
Keras-Applications (1.0.7)
Keras-Preprocessing (1.0.9)
You are using pip version 9.0.1, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(openpose) fabianwerner ~$
I start python and try to import keras:
(openpose) fabianwerner ~$ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 26 2018, 08:42:37)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import keras
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'keras'
>>>
Let us check where python looks for modules:
>>> import sys
>>> print(sys.path)
['', '/Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env/lib/python36.zip', '/Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env/lib/python3.6', '/Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env/lib/python3.6/lib-dynload', '/Users/fabianwerner/.local/lib/python3.6/site-packages', '/Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env/lib/python3.6/site-packages']
>>>
so I activated the environment 'openpose' but it still looks for modules in the other (wrong!) environment 'my-rdkit-env'!
How to fix this?
EDIT:
I cannot see any obvious problem with the anaconda setup as conda info -a gives the right path:
(openpose) fabianwerner ~$ conda info -a
active environment : openpose
active env location : /Users/fabianwerner/anaconda3/anaconda3/envs/openpose
shell level : 2
user config file : /Users/fabianwerner/.condarc
populated config files :
conda version : 4.5.11
conda-build version : 3.15.1
python version : 3.7.0.final.0
base environment : /Users/fabianwerner/anaconda3/anaconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/osx-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/osx-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /Users/fabianwerner/anaconda3/anaconda3/pkgs
/Users/fabianwerner/.conda/pkgs
envs directories : /Users/fabianwerner/anaconda3/anaconda3/envs
/Users/fabianwerner/.conda/envs
platform : osx-64
user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Darwin/17.7.0 OSX/10.13.6
UID:GID : 502:20
netrc file : None
offline mode : False
# conda environments:
#
base /Users/fabianwerner/anaconda3/anaconda3
my-rdkit-env /Users/fabianwerner/anaconda3/anaconda3/envs/my-rdkit-env
openpose * /Users/fabianwerner/anaconda3/anaconda3/envs/openpose
strokinat0r /Users/fabianwerner/anaconda3/anaconda3/envs/strokinat0r
sys.version: 3.7.0 (default, Jun 28 2018, 07:39:16)
...
sys.prefix: /Users/fabianwerner/anaconda3/anaconda3
sys.executable: /Users/fabianwerner/anaconda3/anaconda3/bin/python
conda location: /Users/fabianwerner/anaconda3/anaconda3/lib/python3.7/site-packages/conda
conda-build: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-build
conda-convert: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-convert
conda-develop: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-develop
conda-env: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-env
conda-index: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-index
conda-inspect: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-inspect
conda-metapackage: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-metapackage
conda-render: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-render
conda-server: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-server
conda-skeleton: /Users/fabianwerner/anaconda3/anaconda3/bin/conda-skeleton
user site dirs: ~/.local/lib/python3.6
CIO_TEST: <not set>
CONDA_DEFAULT_ENV: openpose
CONDA_EXE: /Users/fabianwerner/anaconda3/anaconda3/bin/conda
CONDA_PREFIX: /Users/fabianwerner/anaconda3/anaconda3/envs/openpose
CONDA_PREFIX_1: /Users/fabianwerner/anaconda3/anaconda3
CONDA_PROMPT_MODIFIER: (openpose)
CONDA_PYTHON_EXE: /Users/fabianwerner/anaconda3/anaconda3/bin/python
CONDA_ROOT: /Users/fabianwerner/anaconda3/anaconda3
CONDA_SHLVL: 2
PATH: /Users/fabianwerner/anaconda3/anaconda3/envs/openpose/bin:/Users/fabianwerner/anaconda3/anaconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/GIMP-2.10.app/Contents/MacOS
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license
(openpose) fabianwerner ~$
Also, I do not seem to have an environment variable set that relates to python:
(openpose) fabianwerner ~$ printenv | grep -i python
CONDA_PYTHON_EXE=/Users/fabianwerner/anaconda3/anaconda3/bin/python
(openpose) fabianwerner ~$
My setup:
Mac OS-X 10.13.6 (High Sierra)
anaconda version 4.5.11
Thanks,
FW

The Python interpreter you started in your example is not the one in the environment.
conda info -a says python version : 3.7.0.final.0 and yet your interpreter says Python 3.6.5
The problem should become apparent when you activate your environment and run which python which should be pointing to the activated env but probably doesn't.
How did you create those environments? Make sure to set the python=XX option or the new environment uses the interpreter from the base/root environment rather than installing a new one. I.e. conda create -n my_environment python=3.7
Edit:
Sorry, I just looked up and tested conda info -a. python version : XX seems to be referring to the base env not the currently active one.
I'm leaving this answer here, since even though my reasoning seems to be wrong, it may still be helpful.

Compiling scikit-image on Windows to run tests

I'm stumped. I'm developing some enhancements to scikit-image which are failing the automated build tests, probably due to rounding errors. I therefore need to get the automated tests running on my Windows system so that I can debug and work out what's wrong. I've so far tried two approaches, neither of which are working:
In my Anaconda Python 3.6 environment, when I try to run the automated tests, I am getting the following error:
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
...which I have found reference to in other contexts, but have not been able to eliminate.
Since the automated test do run (but fail) on a Python 3.5-based system, I thought things might work if I tried a local Python 3.5 environment. Here, I am running into the issue that, despite being installed, the environment cannot find the MS C++ compiler cl.exe. It is installed in C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\ and is found and executed by my Python 3.6 environment, but my Python 3.5 environment doesn't find it despite me adding that directory to my PATH. I should add that my Python 3.6 environment finds it without the directory being added to the PATH. I understand that both Python 3.5 and 3.6 use MSVC 14.0.
I would prefer to fix the problem in my Python 3.6 environment if possible. Any assistance much appreciated.
Update
I have made a box-fresh Python 3.6 conda environment as follows:
conda create --name sk36 python=3.6
conda activate sk36
conda install scikit-image --only-deps
conda install cython
git clone https://github.com/scikit-image/scikit-image.git
cd scikit-image
pip install -e .
pytest skimage/feature
The specific error I am getting is as follows:
..\Anaconda3\lib\site-packages\py\_path\local.py:662: in pyimport
__import__(modname)
skimage\__init__.py:135: in <module>
from .data import data_dir
skimage\data\__init__.py:13: in <module>
from ..io import imread, use_plugin
skimage\io\__init__.py:7: in <module>
from .manage_plugins import *
skimage\io\manage_plugins.py:24: in <module>
from .collection import imread_collection_wrapper
skimage\io\collection.py:12: in <module>
from ..external.tifffile import TiffFile
skimage\external\tifffile\__init__.py:1: in <module>
from .tifffile import imsave, imread, imshow, TiffFile, TiffWriter, TiffSequence
skimage\external\tifffile\tifffile.py:292: in <module>
from . import _tifffile
E RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
...which appears to have something to do with tifffile. Since this package wasn't originally explicitly installed in my new environment, I tried installing various versions of it, including some which downgraded numpy and scipy. Still the same error as above.
Having done some more research it would appear that something is seeing numpy 1.13.x when in fact version 1.15.4 is installed. Here is the full output from conda list:
# Name Version Build Channel
blas 1.0 mkl anaconda
ca-certificates 2018.03.07 0 anaconda
certifi 2018.10.15 py36_0 anaconda
cloudpickle 0.6.1 py36_0 anaconda
cycler 0.10.0 py36h009560c_0 anaconda
cython 0.29 py36ha925a31_0 anaconda
dask-core 0.20.0 py36_0 anaconda
decorator 4.3.0 py36_0 anaconda
freetype 2.9.1 ha9979f8_1 anaconda
icc_rt 2017.0.4 h97af966_0 anaconda
icu 58.2 ha66f8fd_1 anaconda
imageio 2.4.1 py36_0 anaconda
intel-openmp 2019.0 118 anaconda
jpeg 9b hb83a4c4_2 anaconda
kiwisolver 1.0.1 py36h6538335_0 anaconda
libpng 1.6.35 h2a8f88b_0 anaconda
libtiff 4.0.9 h36446d0_2 anaconda
matplotlib 3.0.1 py36hc8f65d3_0 anaconda
mkl 2019.0 118 anaconda
mkl_fft 1.0.6 py36hdbbee80_0 anaconda
mkl_random 1.0.1 py36h77b88f5_1 anaconda
networkx 2.2 py36_1 anaconda
numpy 1.15.4 py36ha559c80_0 anaconda
numpy-base 1.15.4 py36h8128ebf_0 anaconda
olefile 0.46 py36_0 anaconda
openssl 1.0.2p hfa6e2cd_0 anaconda
package_has_been_revoked 1.0 0 enable_revoked
pillow 5.3.0 py36hdc69c19_0 anaconda
pip 18.1 py36_0 anaconda
pyparsing 2.3.0 py36_0 anaconda
pyqt 5.9.2 py36h6538335_2 anaconda
python 3.6.7 h33f27b4_1 anaconda
python-dateutil 2.7.5 py36_0 anaconda
pytz 2018.7 py36_0 anaconda
pywavelets 1.0.1 py36h8c2d366_0 anaconda
qt 5.9.6 vc14h1e9a669_2 anaconda
scikit-image 0.15.dev0 <pip>
scipy 1.1.0 py36h4f6bf74_1 anaconda
setuptools 40.5.0 py36_0 anaconda
sip 4.19.8 py36h6538335_0 anaconda
six 1.11.0 py36_1 anaconda
sqlite 3.25.2 hfa6e2cd_0 anaconda
tifffile 0.15.1 py36h452e1ab_1001 conda-forge
tk 8.6.8 hfa6e2cd_0 anaconda
toolz 0.9.0 py36_0 anaconda
tornado 5.1.1 py36hfa6e2cd_0 anaconda
vc 14.1 h21ff451_3 anaconda
vs2015_runtime 15.5.2 3 anaconda
wheel 0.32.2 py36_0 anaconda
wincertstore 0.2 py36h7fe50ca_0 anaconda
zlib 1.2.11 h8395fce_2 anaconda
Update 2
I've solved the problem for Python 3.6, and I think there's enough information above for the astute to be able to work out what was wrong. I'll put the solution in an answer below.
A cleanly built Python 3.5 environment can't find the compiler, so that issue still remains.

One approach you could try is to upgrade your numpy with
pip install numpy --upgrade
as described here: RuntimeError: module compiled against API version a but this version of numpy is 9
Otherwise (if for some reason you cannot upgrade numpy) I would suggest going with a virtual environment for scikit-image project. I just tried it on Windows 10 and was able to successfully execute tests. My steps (from cmd, inside the project folder):
conda uninstall scikit-image to remove any previously built/installed versions
conda -n scikit-image python=3.6 to create a virtual environment for this project (I used python 3.6, but you can change it to 3.5)
activate scikit-image activated the new virtual env
pip install -r requirements.txt -- installed dependencies (without this step I wasn't getting the dependencies for tests installed)
pip install -e .
pytest

It turns out that pytest wasn't actually installed in the correct environment, it was being invoked from base which did indeed have numpy 1.13.3 installed. Installing it in the cleanly built Python 3.6 environment solved the problem for Python 3.6 at least.

Anaconda + Spark - changing python version for ipython notebooks

I installed Anaconda, and try to work with spark on top.
When I launch spark with Ipython_OPTS="notebook", the python version is the one associated with anaconda's version of python for the notebook.
$ conda search python
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
ipython 0.13 py26_0 defaults
* 4.1.2 py35_1 defaults
ipython-notebook 0.13.2 py27_0 defaults
4.0.4 py27_0 defaults
4.0.4 py34_0 defaults
4.0.4 py35_0 defaults
python 1.0.1 0 defaults
. 2.7.11 0 defaults
* 3.5.1 0 defaults
And if start spark-shell I can precise the environment version on which I am interested (I want 2.7.11) :
$ PYSPARK_PYTHON=/Applications/anaconda/anaconda/envs/vingt-sept/bin/python pyspark
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:57:58)
but if I start spark with the ipython notebook then it defaults back to the python 3.5 version :-(
How can I link the default ipython version to the same version as the one of my env "vingt-sept" ?

Similar to how you are setting the PYSPARK_PYTHON environment variable for the pyspark shell, you can also set this environment variable in your IPython/Jupyter notebook using:
import os
os.environ["PYSPARK_PYTHON"] = "/Applications/anaconda/anaconda/envs/vingt-sept/bin/python"
Refer to this blog post for more information about setting PYSPARK_PYTHON and other Spark-related environment variables from your notebook.

Packages from Conda env not found in Jupyer Notebook

I created an environment called imagescraper and installed pip with it.
I then proceed to use pip to install a package called ImageScraper;
>>activate imagescraper
[imagescraper]>>pip install ImageScraper
Just to ensure that I have the package successfully installed:
>>conda list
[imagescraper] C:\Users\John>conda list
# packages in environment at C:\Anaconda2\envs\imagescrap
#
future 0.15.2 <pip>
imagescraper 2.0.7 <pip>
lxml 3.6.0 <pip>
numpy 1.11.0 <pip>
pandas 0.18.0 <pip>
pip 8.1.1 py27_1
python 2.7.11 4
python-dateutil 2.5.2 <pip>
pytz 2016.3 <pip>
requests 2.9.1 <pip>
setproctitle 1.1.9 <pip>
setuptools 20.3 py27_0
simplepool 0.1 <pip>
six 1.10.0 <pip>
vs2008_runtime 9.00.30729.1 0
wheel 0.29.0 py27_0
Before I launch Jupyter notebook, just to check where we are getting the path from:
[imagescraper] C:\Users\John>python
Python 2.7.11 |Continuum Analytics, Inc.| (default, Feb 16 2016, 09:58:36) [MSC
v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import sys
>>> sys.executable
'C:\\Anaconda2\\envs\\imagescraper\\python.exe'
>>> import image_scraper
Seems ok, so I proceed to launch Jupyter notebook using
[imagescraper]>>jupyter notebook
Within the notebook I created a new book and when i tried the same;
import image_scraper
I am returned with:
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-6c2b65c9cdeb> in <module>()
----> 1 import image_scraper
ImportError: No module named image_scraper
Doing the same to check the paths within Jupyter notebook, I get this;
import sys
sys.executable
'C:\\Anaconda2\\python.exe'
Which tells me that it is not referring to the environment where I installed the modules in.
Is there a way I can ensure that my notebooks all refer to its own env packages?

Here are two possible solutions:
You can register a new kernel based on your imagescraper environment. The kernel will start from the imagescraper environment and thus sees all its packages.
source activate imagescraper
conda install ipykernel
ipython kernel install --name imagescraper
This will add a new kernel named imagescraper to your jupyter dashboard.
Another solution is to install jupyter notebook into the imagescraper environment and start jupyter from the enviroment. This requires activating imagescraper whenever you start jupyter notebook.
source activate imagescraper
conda install notebook
jupyter notebook