I'm stumped. I'm developing some enhancements to scikit-image which are failing the automated build tests, probably due to rounding errors. I therefore need to get the automated tests running on my Windows system so that I can debug and work out what's wrong. I've so far tried two approaches, neither of which are working:
In my Anaconda Python 3.6 environment, when I try to run the automated tests, I am getting the following error:
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
...which I have found reference to in other contexts, but have not been able to eliminate.
Since the automated test do run (but fail) on a Python 3.5-based system, I thought things might work if I tried a local Python 3.5 environment. Here, I am running into the issue that, despite being installed, the environment cannot find the MS C++ compiler cl.exe. It is installed in C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.15.26726\bin\HostX86\x64\ and is found and executed by my Python 3.6 environment, but my Python 3.5 environment doesn't find it despite me adding that directory to my PATH. I should add that my Python 3.6 environment finds it without the directory being added to the PATH. I understand that both Python 3.5 and 3.6 use MSVC 14.0.
I would prefer to fix the problem in my Python 3.6 environment if possible. Any assistance much appreciated.
Update
I have made a box-fresh Python 3.6 conda environment as follows:
conda create --name sk36 python=3.6
conda activate sk36
conda install scikit-image --only-deps
conda install cython
git clone https://github.com/scikit-image/scikit-image.git
cd scikit-image
pip install -e .
pytest skimage/feature
The specific error I am getting is as follows:
..\Anaconda3\lib\site-packages\py\_path\local.py:662: in pyimport
__import__(modname)
skimage\__init__.py:135: in <module>
from .data import data_dir
skimage\data\__init__.py:13: in <module>
from ..io import imread, use_plugin
skimage\io\__init__.py:7: in <module>
from .manage_plugins import *
skimage\io\manage_plugins.py:24: in <module>
from .collection import imread_collection_wrapper
skimage\io\collection.py:12: in <module>
from ..external.tifffile import TiffFile
skimage\external\tifffile\__init__.py:1: in <module>
from .tifffile import imsave, imread, imshow, TiffFile, TiffWriter, TiffSequence
skimage\external\tifffile\tifffile.py:292: in <module>
from . import _tifffile
E RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
...which appears to have something to do with tifffile. Since this package wasn't originally explicitly installed in my new environment, I tried installing various versions of it, including some which downgraded numpy and scipy. Still the same error as above.
Having done some more research it would appear that something is seeing numpy 1.13.x when in fact version 1.15.4 is installed. Here is the full output from conda list:
# Name Version Build Channel
blas 1.0 mkl anaconda
ca-certificates 2018.03.07 0 anaconda
certifi 2018.10.15 py36_0 anaconda
cloudpickle 0.6.1 py36_0 anaconda
cycler 0.10.0 py36h009560c_0 anaconda
cython 0.29 py36ha925a31_0 anaconda
dask-core 0.20.0 py36_0 anaconda
decorator 4.3.0 py36_0 anaconda
freetype 2.9.1 ha9979f8_1 anaconda
icc_rt 2017.0.4 h97af966_0 anaconda
icu 58.2 ha66f8fd_1 anaconda
imageio 2.4.1 py36_0 anaconda
intel-openmp 2019.0 118 anaconda
jpeg 9b hb83a4c4_2 anaconda
kiwisolver 1.0.1 py36h6538335_0 anaconda
libpng 1.6.35 h2a8f88b_0 anaconda
libtiff 4.0.9 h36446d0_2 anaconda
matplotlib 3.0.1 py36hc8f65d3_0 anaconda
mkl 2019.0 118 anaconda
mkl_fft 1.0.6 py36hdbbee80_0 anaconda
mkl_random 1.0.1 py36h77b88f5_1 anaconda
networkx 2.2 py36_1 anaconda
numpy 1.15.4 py36ha559c80_0 anaconda
numpy-base 1.15.4 py36h8128ebf_0 anaconda
olefile 0.46 py36_0 anaconda
openssl 1.0.2p hfa6e2cd_0 anaconda
package_has_been_revoked 1.0 0 enable_revoked
pillow 5.3.0 py36hdc69c19_0 anaconda
pip 18.1 py36_0 anaconda
pyparsing 2.3.0 py36_0 anaconda
pyqt 5.9.2 py36h6538335_2 anaconda
python 3.6.7 h33f27b4_1 anaconda
python-dateutil 2.7.5 py36_0 anaconda
pytz 2018.7 py36_0 anaconda
pywavelets 1.0.1 py36h8c2d366_0 anaconda
qt 5.9.6 vc14h1e9a669_2 anaconda
scikit-image 0.15.dev0 <pip>
scipy 1.1.0 py36h4f6bf74_1 anaconda
setuptools 40.5.0 py36_0 anaconda
sip 4.19.8 py36h6538335_0 anaconda
six 1.11.0 py36_1 anaconda
sqlite 3.25.2 hfa6e2cd_0 anaconda
tifffile 0.15.1 py36h452e1ab_1001 conda-forge
tk 8.6.8 hfa6e2cd_0 anaconda
toolz 0.9.0 py36_0 anaconda
tornado 5.1.1 py36hfa6e2cd_0 anaconda
vc 14.1 h21ff451_3 anaconda
vs2015_runtime 15.5.2 3 anaconda
wheel 0.32.2 py36_0 anaconda
wincertstore 0.2 py36h7fe50ca_0 anaconda
zlib 1.2.11 h8395fce_2 anaconda
Update 2
I've solved the problem for Python 3.6, and I think there's enough information above for the astute to be able to work out what was wrong. I'll put the solution in an answer below.
A cleanly built Python 3.5 environment can't find the compiler, so that issue still remains.
One approach you could try is to upgrade your numpy with
pip install numpy --upgrade
as described here: RuntimeError: module compiled against API version a but this version of numpy is 9
Otherwise (if for some reason you cannot upgrade numpy) I would suggest going with a virtual environment for scikit-image project. I just tried it on Windows 10 and was able to successfully execute tests. My steps (from cmd, inside the project folder):
conda uninstall scikit-image to remove any previously built/installed versions
conda -n scikit-image python=3.6 to create a virtual environment for this project (I used python 3.6, but you can change it to 3.5)
activate scikit-image activated the new virtual env
pip install -r requirements.txt -- installed dependencies (without this step I wasn't getting the dependencies for tests installed)
pip install -e .
pytest
It turns out that pytest wasn't actually installed in the correct environment, it was being invoked from base which did indeed have numpy 1.13.3 installed. Installing it in the cleanly built Python 3.6 environment solved the problem for Python 3.6 at least.
Related
There are many posts on this site which reference, typically in passing, the idea of setting pip_interop_enabled=True within some environment. This makes conda and pip3 somehow interact better, I am told. To be precise, people say conda will search PyPI for packages that don't exist in the main channels if this is true. They also say it's "experimental."
Here is conda's documentation about this. It notes that much of conda's behavior in recent versions has also improved even with pip_interop_enabled=False, leading to questions about what this setting even does.
Here is my question: in real terms, what does all of this mean?
Is the only difference that conda will search PyPI if this is True and not if it's False?
Are there other things that it does? For instance, if I need to install some package from pip, will conda know better not to clobber it if this setting is True?
What, to be precise, goes wrong if I set this to True? Are there known edge cases that somehow break things if this "experimental" setting is set to True?
Why would I ever not want to set this?
Not a PyPI Searching Feature
First, let's clarify: Conda will not "search PyPI" - that is not what the pip_interop_enabled configuration option adds. Rather, it enables the solver to allow a package already installed with pip to satisfy a dependency requirement of a Conda package. Note that the option is about Pip interoperability (as distinct from PyPI) and it doesn't matter whether the package was sourced from PyPI, GitHub, local, etc..
Example: scipy -> numpy
Let's consider a simple example to illustrate the behavior. Start with the following environment that has Python 3.10 and numpy installed from PyPI.
pip_interop.yaml
name: pip_interop
channels:
- conda-forge
dependencies:
- python=3.10
- pip
## PyPI packages
- pip:
- numpy
which we can create with
conda env create -n pip_interop -f pip_interop.yaml
and verify that the numpy is from PyPI:
$ conda list -n pip_interop numpy
# packages in environment at /Users/user/mambaforge/envs/pip_interop:
#
# Name Version Build Channel
numpy 1.24.2 pypi_0
Let's see what would happen installing scipy and in particular, how it satisfies its numpy dependency.
Installing without Pip interoperability
In default mode, we see the following behavior
$ conda install -n pip_interop scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/user/mambaforge/envs/pip_interop
added / updated specs:
- scipy
The following packages will be downloaded:
package | build
---------------------------|-----------------
cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB
numpy-1.24.2 | py310h788a5b3_0 6.1 MB
scipy-1.10.0 | py310h240c617_2 20.2 MB
------------------------------------------------------------
Total: 27.4 MB
The following NEW packages will be INSTALLED:
appdirs conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0
brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005
certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0
cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3
charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0
cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0
idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0
libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas
libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas
libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0
libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27
libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27
liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas
libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3
llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0
numpy conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0
packaging conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0
pooch conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0
pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0
pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0
pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6
python_abi conda-forge/osx-64::python_abi-3.10-3_cp310
requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0
scipy conda-forge/osx-64::scipy-1.10.0-py310h240c617_2
urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0
Proceed ([y]/n)?
Observe that despite numpy already being installed in the environment, Conda is proposing to replace it with a Conda version. That is, only considers the information in conda-meta/ to determine whether a package is installed and won't check the environment's lib/python3.10/site-packages/.
Installing with Pip interoperability
Now we try it with the pip_interop_enabled turned on:
$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n foo scipy
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/user/mambaforge/envs/pip_interop
added / updated specs:
- scipy
The following packages will be downloaded:
package | build
---------------------------|-----------------
cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB
scipy-1.10.0 | py310h240c617_2 20.2 MB
------------------------------------------------------------
Total: 21.3 MB
The following NEW packages will be INSTALLED:
appdirs conda-forge/noarch::appdirs-1.4.4-pyh9f0ad1d_0
brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005
certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0
cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3
charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0
cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0
idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0
libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas
libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas
libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0
libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27
libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27
liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas
libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3
llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0
packaging conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0
pooch conda-forge/noarch::pooch-1.6.0-pyhd8ed1ab_0
pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0
pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0
pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6
python_abi conda-forge/osx-64::python_abi-3.10-3_cp310
requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0
scipy conda-forge/osx-64::scipy-1.10.0-py310h240c617_2
urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0
Proceed ([y]/n)?
Note that now the numpy is not proposed to be replaced and this is because the existing pip-installed version is consider able to satisfy the dependency.
Why is this experimental?
There may be multiple reasons why this remains experimental after several years. One important reason is that Conda only tests its package builds against Conda builds of the dependencies. So, it cannot guarantee that the packages are functionally exchangeable.
Furthermore, Conda packages often bring in non-Python dependencies. There has been a rise in wheel deployments, which is the PyPI approach to this, but isn't ubiquitous. There are still many "wrapper" packages out there where the PyPI version assumes some binary is on PATH, whereas the installation of the Conda package guarantees the binary is also installed.
Another important issue is that the PyPI-Conda name mapping is not well-defined. That is, the name of a package in PyPI may not correspond to its Conda package name. This can directly lead to cryptic issues when the names diverge. Specifically, Conda will not correctly recognize that a pip-installed package satisfies the requirement when the names don't match. Hence, the is some unexpected heterogeneity in how the interoperability applies.
Example: torch vs pytorch
In the Python ecosystem, the torch module is provided by the PyPI package torch. However, the package torch in PyPI goes by pytorch on Conda channels.
Here's how this can lead to inconsistent behavior. Let's begin with torch installed from PyPI:
pip_interop.yaml
name: pip_interop
channels:
- conda-forge
dependencies:
- python=3.10
- pip
## PyPI packages
- pip:
- torch
Creating with:
conda env create -n pip_interop -f pip_interop.yaml
Now if we install torchvision from Conda, even with the pip_interop_enabled on, we get:
$ CONDA_PIP_INTEROP_ENABLED=1 conda install -n pip_interop torchvision
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/user/mambaforge/envs/pip_interop
added / updated specs:
- torchvision
The following packages will be downloaded:
package | build
---------------------------|-----------------
cryptography-39.0.1 | py310hdd0c95c_0 1.1 MB
jpeg-9e | hb7f2c08_3 226 KB
libprotobuf-3.21.12 | hbc0c0cd_0 1.8 MB
mkl-2022.2.1 | h44ed08c_16952 113.1 MB
numpy-1.24.2 | py310h788a5b3_0 6.1 MB
pillow-9.4.0 | py310h306a057_1 44.1 MB
pytorch-1.13.1 |cpu_py310h2bbf33f_1 56.9 MB
sleef-3.5.1 | h6db0672_2 1.0 MB
torchvision-0.14.1 |cpu_py310hd5ee960_0 5.9 MB
------------------------------------------------------------
Total: 230.1 MB
The following NEW packages will be INSTALLED:
brotlipy conda-forge/osx-64::brotlipy-0.7.0-py310h90acd4f_1005
certifi conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0
cffi conda-forge/osx-64::cffi-1.15.1-py310ha78151a_3
charset-normalizer conda-forge/noarch::charset-normalizer-2.1.1-pyhd8ed1ab_0
cryptography conda-forge/osx-64::cryptography-39.0.1-py310hdd0c95c_0
freetype conda-forge/osx-64::freetype-2.12.1-h3f81eb7_1
idna conda-forge/noarch::idna-3.4-pyhd8ed1ab_0
jpeg conda-forge/osx-64::jpeg-9e-hb7f2c08_3
lcms2 conda-forge/osx-64::lcms2-2.14-h29502cd_1
lerc conda-forge/osx-64::lerc-4.0.0-hb486fe8_0
libblas conda-forge/osx-64::libblas-3.9.0-16_osx64_openblas
libcblas conda-forge/osx-64::libcblas-3.9.0-16_osx64_openblas
libcxx conda-forge/osx-64::libcxx-14.0.6-hccf4f1f_0
libdeflate conda-forge/osx-64::libdeflate-1.17-hac1461d_0
libgfortran conda-forge/osx-64::libgfortran-5.0.0-11_3_0_h97931a8_27
libgfortran5 conda-forge/osx-64::libgfortran5-11.3.0-h082f757_27
liblapack conda-forge/osx-64::liblapack-3.9.0-16_osx64_openblas
libopenblas conda-forge/osx-64::libopenblas-0.3.21-openmp_h429af6e_3
libpng conda-forge/osx-64::libpng-1.6.39-ha978bb4_0
libprotobuf conda-forge/osx-64::libprotobuf-3.21.12-hbc0c0cd_0
libtiff conda-forge/osx-64::libtiff-4.5.0-hee9004a_2
libwebp-base conda-forge/osx-64::libwebp-base-1.2.4-h775f41a_0
libxcb conda-forge/osx-64::libxcb-1.13-h0d85af4_1004
llvm-openmp conda-forge/osx-64::llvm-openmp-15.0.7-h61d9ccf_0
mkl conda-forge/osx-64::mkl-2022.2.1-h44ed08c_16952
numpy conda-forge/osx-64::numpy-1.24.2-py310h788a5b3_0
openjpeg conda-forge/osx-64::openjpeg-2.5.0-h13ac156_2
pillow conda-forge/osx-64::pillow-9.4.0-py310h306a057_1
pthread-stubs conda-forge/osx-64::pthread-stubs-0.4-hc929b4f_1001
pycparser conda-forge/noarch::pycparser-2.21-pyhd8ed1ab_0
pyopenssl conda-forge/noarch::pyopenssl-23.0.0-pyhd8ed1ab_0
pysocks conda-forge/noarch::pysocks-1.7.1-pyha2e5f31_6
python_abi conda-forge/osx-64::python_abi-3.10-3_cp310
pytorch conda-forge/osx-64::pytorch-1.13.1-cpu_py310h2bbf33f_1
requests conda-forge/noarch::requests-2.28.2-pyhd8ed1ab_0
sleef conda-forge/osx-64::sleef-3.5.1-h6db0672_2
tbb conda-forge/osx-64::tbb-2021.7.0-hb8565cd_1
torchvision conda-forge/osx-64::torchvision-0.14.1-cpu_py310hd5ee960_0
typing_extensions conda-forge/noarch::typing_extensions-4.4.0-pyha770c72_0
urllib3 conda-forge/noarch::urllib3-1.26.14-pyhd8ed1ab_0
xorg-libxau conda-forge/osx-64::xorg-libxau-1.0.9-h35c211d_0
xorg-libxdmcp conda-forge/osx-64::xorg-libxdmcp-1.1.3-h35c211d_0
zstd conda-forge/osx-64::zstd-1.5.2-hbc0c0cd_6
Proceed ([y]/n)?
That is, Conda still tries to install pytorch and this means that it will lead to clobbering of the existing torch package installed from PyPI. This has the potential to having residual files from the clobbered version of the package intermixed with the clobbering version.
Basically, this is undefined behavior and the Conda software may not give you any warning about potential problems.
I have tensorflow listed as a requirement in the install_requires section of the setup.py of my project.
When I attempt to install my project into a new Anaconda environment I get the following error:
$ python setup.py install
...
Searching for tensorflow
Reading https://pypi.org/simple/tensorflow/
No local packages or working download links found for tensorflow
error: Could not find suitable distribution for Requirement.parse('tensorflow')
I can get past this by installing tensorflow "manually" via conda:
$ conda install tensorflow
Once I do this the install via setup.py works without a hitch.
Am I mistaken in assuming that something is amiss with my environment? If not then what is going on and how can I avoid this issue? (My concern is that users of my package will not be able to install from source using setup.py)
Another oddity that I assume is related or may provide a clue is that the version of TensorFlow listed in my Anaconda environment is 2.0 but if I import it when running Python it appears to instead be using version 1.15. For example:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvdata_test:
#
# Name Version Build Channel
tensorflow 2.0.0 mkl_py37h66b46cc_0
tensorflow-base 2.0.0 mkl_py37h9204916_0
tensorflow-estimator 2.0.0 pyh2649769_0
$ python
Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.15.0-rc2'
This is on a Dell laptop running Ubuntu 18.04 without a GPU, so perhaps the version shown in the interpreter is akin to tensorflow-cpu? If I run pip freeze I see tensorflow==2.0.0 and tensorflow-cpu==1.15.0rc2, which is a bit confusing...
This appears to have been caused by using Python version 3.8, which is currently an unsupported version of Python. Once I created a new Anaconda environment with Python version 3.7 this issue went away.
The only remaining issue is this error that I see when I run pip install -e . for my project which includes tensorflow:
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorboard<1.16.0,>=1.15.0, but you'll have tensorboard 2.1.0 which is incompatible.
ERROR: tensorflow-cpu 1.15.0rc2 has requirement tensorflow-estimator==1.15.1, but you'll have tensorflow-estimator 2.1.0 which is incompatible.
So the issue of tensorflow-cpu version 1.15.0rc2 actually being installed when version 2.1.0 shows as being the installed version is still a mystery. To wit:
$ conda list tensorflow
# packages in environment at /home/james/miniconda3/envs/cvd:
#
# Name Version Build Channel
tensorflow 2.1.0 pypi_0 pypi
tensorflow-estimator 2.1.0 pypi_0 pypi
$ python -c "import tensorflow as tf; print(tf.__version__)"
1.15.0-rc2
I am working on a windows amd64 machine with all the latest anaconda packages; In an attempt to solve this problem I went and updated them all with conda update --all to be sure, but the issue is still present.
As I understand it, PyQt5 is included in the latest anaconda packages, but I haven't found many topics on similar issues, and the solutions I tried didnt work.
I tried running pyuic5 too, which comes included with anaconda and depends on PyQt, and it crashes python instantly.
Here's the output of conda list qt:
pyqt 5.6.0 py35_2
PyQt5 5.8.2 <pip>
qt 5.6.2 vc14_3 [vc14]
qtawesome 0.4.4 py35_0
qtconsole 4.3.0 py35_0
qtpy 1.2.1 py35_0
Reinstalling Anaconda or in my case removing it solved the issue.
Confirmed # win 10 x 64, python 3.6.1 , MiniAnaconda
I'm new to Python and recently installed PyCharm 2016.3 on Windows 10. I'm also using Anaconda 3.
I don't know much about package management and would like to understand it better. Normally I just use conda update --all but I noticed (by checking the package list of my local PyCharm Interpreter) that this doesn't upgrade all packages to the latest version.
One such package is Pillow of which there's a version 4.0.0 but conda (4.3.11) won't update it past 3.4.2. I tried conda install pillow: 4.0.0 and got:
UnsatisfiableError: The following specifications were found to be in conflict:
- pillow 4.0.0*
- python 3.5*
- spyder-app
Use "conda info <package>" to see the dependencies for each package.
Later I found out that Pillow is also available on conda-forge so I tried conda install -c conda-forge pillow=4.0.0 and got:
The following NEW packages will be INSTALLED:
libiconv: 1.14-vc14_4 conda-forge [vc14]
libxml2: 2.9.3-vc14_9 conda-forge [vc14]
olefile: 0.44-py35_0 conda-forge
vc: 14-0 conda-forge
The following packages will be UPDATED:
freetype: 2.5.5-vc14_2 [vc14] --> 2.7-vc14_0 conda-forge [vc14]
jpeg: 8d-vc14_2 [vc14] --> 9b-vc14_0 conda-forge [vc14]
libtiff: 4.0.6-vc14_2 [vc14] --> 4.0.6-vc14_7 conda-forge [vc14]
pillow: 3.4.2-py35_0 --> 4.0.0-py35_2 conda-forge
The following packages will be SUPERCEDED by a higher-priority channel:
conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge
conda-env: 2.6.0-0 --> 2.6.0-0 conda-forge
qt: 4.8.7-vc14_9 [vc14] --> 4.8.7-vc14_6 conda-forge [vc14]
I decided not to proceed and instead tried pip install pillow. Since this command doesn't ask for confirmation the package was simply installed. Now when I type conda list I get:
Pillow 4.0.0 <pip>
pillow 3.4.2 py35_0
The package list of the PyCharm Interpreter now shows Pillow as being version 4.0.0 but conda update pillow still returns:
# All requested packages already installed.
pillow 3.4.2 py35_0
My questions are:
1) What should I rely on to keep all my packages up to date, without compatibility issues?
2) Why did conda install pillow: 4.0.0 return an error but conda install -c conda-forge pillow=4.0.0 didn't?
3) What do the * next to pillow 4.0.0 and python 3.5 in the list of dependencies mean?
4) Since now I have both Pillow 3.4.2 (in /anaconda3/pkgs) and Pillow 4.0.0 (in /anaconda3/lib/site-packages) which one would be used if I imported Pillow?
5) Does the superseding conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge mean conda is getting downgraded?
6) What is the difference between the tags pip, py35_0, py35_4, np111py35_2, etc?
7) PyCharm tells me there's a version 2.9.5 of package Jinja2 but both normal conda and conda-forge only find 2.9.4. From which channel is PyCharm getting this information?
Ok, I can't answer all of your questions but here goes:
1) Conda defers to the "pain up front" approach for handling dependency/conflict resolution. You'll have to get all of your packages to play nicely together in the repo's/channels that you have available to even make a package or keep them in an environment together. You can try running it with --force or --no-deps to try getting it in but ..... that can cause issues for you in the future (I don't know if that would even work with the later versions of conda, it changes a lot). Simply keeping packages up to date, and up to latest, I would just use pip. Its come a long way in the last few years (https://glyph.twistedmatrix.com/2016/08/python-packaging.html)
2) I am not completely sure, I believe it would have something to do with providing an explicit non-url channel for conda to look at. Typically you pass it the URL to the conda-forge repo (I think, again we don't use conda-forge internally).
3) The * means you are ignoring the patch/build 4.0.0 == Major.Minor.Build. Likewise, 3.5* == any version of 3.5
4) I would import pillow in a terminal, and then print out the module to see where its getting pulled from, why guess?
5) pass (although I think so)
6)
pip : means you installed that package via pip. It will not be picked up if you do conda list --explicit
py35_0 : has a requirement / only available to envs / packages that use python 3.5
py35_4 : not sure (always forget that one)
np111py35_2 : requires python3.5 and also numpy 1.11 (I think *)
7) I tend to steer clear of pycharm, I believe that you can inspect the python interpreter that pycharm is pointing at to see what environment its in. Based on the root environment, you can do a conda info and get a list of all of the channels you are pointing to.
Note: if you are going to use conda, you may just want to add conda-forge to your channels list instead of passing the -c (but seeing how the other channels are organized should help you see how you should pass the -c flag)
I want to install a package I made into one of my conda envorinments (For Python 3.5). Yet, when I use conda list, I can find it (called spbusiness):
...
sockjs-tornado 1.0.3 py35_0
spbusiness 0.1 <pip>
sphinx 1.4.6 py35_0
spyder 3.0.0 py35_0
...
Now when I want to use it with iPython it raises:
ImportError: No module named 'spbusiness'
When I import pip and asking for all installed packages:
import pip
pip.get_installed_distributions()
It is found in the \anaconda3\lib\site-packages folder:
...
SQLAlchemy 1.0.13 (c:\users\martin\anaconda3\lib\site-packages),
spyder 3.0.0 (c:\users\martin\anaconda3\lib\site-packages),
spbusiness 0.1 (c:\users\martin\anaconda3\lib\site-packages),
sockjs-tornado 1.0.3 (c:\users\martin\anaconda3\lib\site-packages),
snowballstemmer 1.2.1 (c:\users\martin\anaconda3\lib\site-packages),
...
I used the Python interpreter from Anaconda directory ~\Anaconda3\python.exe for setup.py install. It returned:
running install
running build
running build_py
running install_lib
running install_egg_info
Removing [...]\Anaconda3\Lib\site-packages\spbusiness-0.1-py3.5.egg-info
Writing [...]\Anaconda3\Lib\site-packages\spbusiness-0.1-py3.5.egg-info
Still I can't use it in iPython nor in Jupyter Notebooks. I probably made something wrong but I can't get it the right way.
Thanks for help in forward.
Martin