Develop Reference

conda install and conda build result in different dependency versions

I'm trying to build a package which includes h5py. When using conda build, it seems to install the wrong version of the dependency. It installs 3.2.1-py37h6c542dc_0, which includes hdf5: 1.10.6-nompi_h6a2412b_1114.
The problem is that this hdf5 lib, seems to have these setting:
(Read-Only) S3 VFD: yes
This causes an error for me. When just running conda install h5py==3.2.1, it does install the right version (hdf5-1.10.6-nompi_h3c11f04_101).
Why is there a difference?

"Why is there a difference?
Using conda install h5py=3.2.1 additionally includes all the previous constraints in the current environment, whereas during a conda build run, a new environment is created only with requirements that the package specifies. That is, it is more like running conda create -n foo h5py=3.2.1.
So, that covers the mechanism, but we can also look at the particular package dependencies to see why the current environment constrains to the older hdf5-1.10.6-nompi_h3c11f04_101, which OP states is preferred. Here is the package info for the two:
hdf5-1.10.6-nompi_h6a2412b_1114
$ mamba search --info conda-forge/linux-64::hdf5[version='1.10.6',build='nompi_h6a2412b_1114']
hdf5 1.10.6 nompi_h6a2412b_1114
-------------------------------
file name : hdf5-1.10.6-nompi_h6a2412b_1114.tar.bz2
name : hdf5
version : 1.10.6
build : nompi_h6a2412b_1114
build number: 1114
size : 3.1 MB
license : LicenseRef-HDF5
subdir : linux-64
url : https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.10.6-nompi_h6a2412b_1114.tar.bz2
md5 : 0a2984b78f51148d7ff6219abe73509e
timestamp : 2021-01-08 23:10:11 UTC
dependencies:
- libcurl >=7.71.1,<8.0a0
- libgcc-ng >=9.3.0
- libgfortran-ng
- libgfortran5 >=9.3.0
- libstdcxx-ng >=9.3.0
- openssl >=1.1.1i,<1.1.2a
- zlib >=1.2.11,<1.3.0a0
hdf5-1.10.6-nompi_h3c11f04_101
$ mamba search --info conda-forge/linux-64::hdf5[version='1.10.6',build='nompi_h3c11f04_101']
hdf5 1.10.6 nompi_h3c11f04_101
------------------------------
file name : hdf5-1.10.6-nompi_h3c11f04_101.tar.bz2
name : hdf5
version : 1.10.6
build : nompi_h3c11f04_101
build number: 101
size : 3.0 MB
license : HDF5
subdir : linux-64
url : https://conda.anaconda.org/conda-forge/linux-64/hdf5-1.10.6-nompi_h3c11f04_101.tar.bz2
md5 : 9f1ccc4d36edf8ea15ce19f52cf6d601
timestamp : 2020-07-31 12:26:29 UTC
dependencies:
- libgcc-ng >=7.5.0
- libgfortran-ng >=7,<8.0a0
- libstdcxx-ng >=7.5.0
- zlib >=1.2.11,<1.3.0a0
The difference here is that the latter works with older versions of libgcc-ng, libstdcxx-ng, and libgfortran-ng (below 9.3.0), as well as has no constraint on openssl or libcurl. So, we can guess that the current environment where the conda install h5py=3.2.1 was invoked has one of these restrictions.

How to install fiona 1.6?

I am trying to install fiona=1.6 but I get the following error
conda install fiona=1.6
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Collecting package metadata: done
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/noarch::flask-cors==3.0.7=py_0
- conda-forge/osx-64::blaze==0.11.3=py36_0
- conda-forge/noarch::flask==1.0.2=py_2
failed
PackagesNotFoundError: The following packages are not available from current channels:
- fiona=1.6 -> gdal==1.11.4
Current channels:
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
If I try to install gdal==1.11.4, I get the following
conda install -c conda-forge gdal=1.11.4
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Collecting package metadata: done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/noarch::flask-cors==3.0.7=py_0
- conda-forge/osx-64::blaze==0.11.3=py36_0
- conda-forge/noarch::flask==1.0.2=py_2
failed
PackagesNotFoundError: The following packages are not available from current channels:
- gdal=1.11.4
Current channels:
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/osx-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
This is the result of conda info
conda info
active environment : base
active env location : /anaconda3
shell level : 1
user config file : /Users/massaro/.condarc
populated config files : /Users/massaro/.condarc
conda version : 4.6.11
conda-build version : 3.17.8
python version : 3.6.8.final.0
base environment : /anaconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/osx-64
https://conda.anaconda.org/conda-forge/noarch
package cache : /anaconda3/pkgs
/Users/massaro/.conda/pkgs
envs directories : /anaconda3/envs
/Users/massaro/.conda/envs
platform : osx-64
user-agent : conda/4.6.11 requests/2.21.0 CPython/3.6.8 Darwin/17.5.0 OSX/10.13.4
UID:GID : 502:20
netrc file : None

Python Versions
The Conda Forge channel only has gdal v1.11.4 for Python 2.7, 3.4, and 3.5. You either need to use a newer version of Fiona (current is 1.8) or make a new env that includes one of those older Python versions.
For example,
conda create -n fiona_1_6 fiona=1.6 python=3.5
Channel defaults is Required
Another issue you face is that you have removed the defaults channel from your configuration (as per your conda info). It is impossible to install fiona=1.6 with only the conda-forge channel. My recommendation would be to have both conda-forge and defaults in your configuration, but just set conda-forge to have higher priority (if that's what you want). You can do this like so,
conda config --append channels defaults
If you really don't want to include defaults, but just want a temporary workaround, then you can simply run the first command with a --channels | -c flag
conda create -n fiona_1_6 -c conda-forge -c defaults fiona=1.6 python=3.5
This will still give conda-forge precedence, but allow missing dependencies to be sourced from defaults.
Environment File
If you have more than just Fiona that you require, it may be cleaner to put together a requirements file, like so
fiona_1_6.yaml
name: fiona_1_6
channels:
- conda-forge
- defaults
dependencies:
- python=3.5
- fiona=1.6
- osmnx
Then create the new environment with this:
conda env create -f fiona_1_6.yaml

Doing what the error message told me to,
To search for alternate channels that may provide the conda package you're
looking for, navigate to https://anaconda.org
and typing in gdal in the search box led me to https://anaconda.org/conda-forge/gdal which has this installation instruction:
conda install -c conda-forge gdal=1.11.4
Try that to install the gdal dependency, maybe?

Conda can't find packages at pypi.org/simple

I have a problem with installing packages via conda. It can not find any package in channel https://pypi.org simple.
conda install logbook
returns:
PS C:\WINDOWS\system32> conda config --add channels
https://pypi.org/simple
Warning: 'https://pypi.org/simple' already in 'channels' list, moving to the top
PS C:\WINDOWS\system32> conda install Logbook
Fetching package metadata ....
WARNING: The remote server could not find the noarch directory for the
requested channel with url: https://pypi.org/simple
It is possible you have given conda an invalid channel. Please double-check
your conda configuration using `conda config --show`.
If the requested url is in fact a valid conda channel, please request that the
channel administrator create `noarch/repodata.json` and associated
`noarch/repodata.json.bz2` files, even if `noarch/repodata.json` is empty.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json
...........
PackageNotFoundError: Packages missing in current channels:
- logbook
We have searched for the packages in the following channels:
- https://pypi.org/simple/win-64
- https://pypi.org/simple/noarch
- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch
I have checked manually in browser, that the logbook module is in the list on page https://pypi.org/simple.
hoever, it seems that conda seeks packages in https://pypi.org/simple/win-64, but the win-64 directory does not exist there.
conda config --show says, that configured channel is https://pypi.org/simple
add_anaconda_token: True
add_pip_as_python_dependency: True
allow_non_channel_urls: True
allow_softlinks: False
always_copy: False
always_softlink: False
always_yes: False
anaconda_upload: None
auto_update_conda: True
changeps1: True
channel_alias: https://conda.anaconda.org
channel_priority: True
channels:
- https://pypi.org/simple
- https://pypi.org/simple/
- defaults
client_ssl_cert: None
client_ssl_cert_key: None
clobber: False
create_default_packages: []
custom_channels:
pkgs/main: https://repo.continuum.io/
pkgs/free: https://repo.continuum.io/
pkgs/r: https://repo.continuum.io/
pkgs/pro: https://repo.continuum.io/
pkgs/msys2: https://repo.continuum.io/
C:/Program%20Files/Anaconda3/conda-bld: file:///
Any advice?

A conda channel has to have a specific layout (win-64, win-32, ...) and the package has to be built in a conda-specific way (see Building conda packages with conda skeleton). The packages are generally build against specific Python versions (although there should also be noarch packages) and/or numpy versions and for different platforms (windows, linux, mac 32bit or 64bit).
You cannot directly install packages from PyPI using conda because PyPI doesn't qualify as conda-channel and even if it were the packages there aren't build like conda-packages. But you can install them using pip (within conda).
But: You could check if the packages you want are distributed in a conda-conforming channel (a very popular channel currently is conda-forge). At a first glance several channels contain a package named logbook in the anaconda cloud (search results for logbook).
If you find a channel that distributes the desired version of your package (and against the desired Python version and platform) then just use:
conda install -c channel_name logbook

Many Python products are available for Windows on the Gohlke page.
Assuming you have 64-bit Py3.6 installed in its own environment, called Py36, you can do this.
Download Logbook-1.1.0-cp36-cp36m-win_amd64.whl from Gohlke (or whatever version it is that you need).
Then within a command prompt:
activate the conda environment where you want to install Logbook.
Use pip to install the whl that you have downloaded within that environment.
Verify success, if you wish.
C:\scratch>activate Py36
(Py36) C:\scratch>pip install Logbook-1.1.0-cp36-cp36m-win_amd64.whl
Processing c:\scratch\logbook-1.1.0-cp36-cp36m-win_amd64.whl
Installing collected packages: Logbook
Successfully installed Logbook-1.1.0
(Py36) C:\scratch>conda list
# packages in environment at C:\ProgramData\Miniconda3\envs\Py36:
#
beautifulsoup4 4.6.0 <pip>
certifi 2017.11.5 py36hb8ac631_0
chardet 3.0.4 <pip>
idna 2.6 <pip>
Logbook 1.1.0 <pip>
opencv-python 3.4.0+contrib <pip>
pip 9.0.1 py36h226ae91_4
python 3.6.3 h3b118a2_4
requests 2.18.4 <pip>
setuptools 36.5.0 py36h65f9e6e_0
urllib3 1.22 <pip>
vc 14 h2379b0c_2
vs2015_runtime 14.0.25123 hd4c4e62_2
wheel 0.30.0 py36h6c3ec14_1
wikipedia 1.4.0 <pip>
wincertstore 0.2 py36h7fe50ca_0
Edit: Answer to query in comment.
I entered this:
conda env export -n Py36 -f Py36.yml
Content of yaml file.
name: Py36
channels:
- defaults
dependencies:
- certifi=2017.11.5=py36hb8ac631_0
- pip=9.0.1=py36h226ae91_4
- python=3.6.3=h3b118a2_4
- setuptools=36.5.0=py36h65f9e6e_0
- vc=14=h2379b0c_2
- vs2015_runtime=14.0.25123=hd4c4e62_2
- wheel=0.30.0=py36h6c3ec14_1
- wincertstore=0.2=py36h7fe50ca_0
- pip:
- beautifulsoup4==4.6.0
- chardet==3.0.4
- idna==2.6
- logbook==1.1.0
- opencv-python==3.4.0+contrib
- requests==2.18.4
- urllib3==1.22
- wikipedia==1.4.0
prefix: C:\ProgramData\Miniconda3\envs\Py36

Conda cannot install pydot-ng with python3 (Unsatisfiable error)

I have Anaconda 4.3.23 on Ubuntu 16.04 LTS. When I try to install pydot-ng package using
conda install pydot-ng
I get the following error:
UnsatisfiableError: The following specifications were found to be in conflict:
- pydot-ng -> python 2.7* -> openssl 1.0.1*
- python 3.6*
Use "conda info <package>" to see the dependencies for each package.
Upon running conda info pydot-ng i see
pydot-ng 1.0.0.15 py27_0
------------------------
file name : pydot-ng-1.0.0.15-py27_0.tar.bz2
name : pydot-ng
version : 1.0.0.15
build string: py27_0
build number: 0
channel : defaults
size : 45 KB
arch : x86_64
date : 2015-09-09
license : MIT
md5 : 8b81a344723e64ec3545b5f030caca47
noarch : None
platform : linux
url : https://repo.continuum.io/pkgs/free/linux-64/pydot-ng-
1.0.0.15-py27_0.tar.bz2
dependencies:
pyparsing
python 2.7*
pydot-ng 1.0.0.15 py34_0
------------------------
file name : pydot-ng-1.0.0.15-py34_0.tar.bz2
name : pydot-ng
version : 1.0.0.15
build string: py34_0
build number: 0
channel : defaults
size : 46 KB
arch : x86_64
date : 2015-09-09
license : MIT
md5 : 13e3a10b45edfb38d91a51d6b3ccabc7
noarch : None
platform : linux
url : https://repo.continuum.io/pkgs/free/linux-64/pydot-ng-
1.0.0.15-py34_0.tar.bz2
dependencies:
pyparsing
python 3.4*
What is the problem? Shouldn't conda be able to install the pydot-ng 1.0.0.15 py34_0 version which would be compatible with python 3.6 (I presume)?

That's not how conda works - the package is built for a specific version of Python, unless the package builder specifically says that its a universal package and can be installed for many versions. The simplest solution to this is to create a new environment to install this package:
conda create -n pydotng python=3.4 pydot-ng

Managing packages: PyCharm vs conda vs pip

I'm new to Python and recently installed PyCharm 2016.3 on Windows 10. I'm also using Anaconda 3.
I don't know much about package management and would like to understand it better. Normally I just use conda update --all but I noticed (by checking the package list of my local PyCharm Interpreter) that this doesn't upgrade all packages to the latest version.
One such package is Pillow of which there's a version 4.0.0 but conda (4.3.11) won't update it past 3.4.2. I tried conda install pillow: 4.0.0 and got:
UnsatisfiableError: The following specifications were found to be in conflict:
- pillow 4.0.0*
- python 3.5*
- spyder-app
Use "conda info <package>" to see the dependencies for each package.
Later I found out that Pillow is also available on conda-forge so I tried conda install -c conda-forge pillow=4.0.0 and got:
The following NEW packages will be INSTALLED:
libiconv: 1.14-vc14_4 conda-forge [vc14]
libxml2: 2.9.3-vc14_9 conda-forge [vc14]
olefile: 0.44-py35_0 conda-forge
vc: 14-0 conda-forge
The following packages will be UPDATED:
freetype: 2.5.5-vc14_2 [vc14] --> 2.7-vc14_0 conda-forge [vc14]
jpeg: 8d-vc14_2 [vc14] --> 9b-vc14_0 conda-forge [vc14]
libtiff: 4.0.6-vc14_2 [vc14] --> 4.0.6-vc14_7 conda-forge [vc14]
pillow: 3.4.2-py35_0 --> 4.0.0-py35_2 conda-forge
The following packages will be SUPERCEDED by a higher-priority channel:
conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge
conda-env: 2.6.0-0 --> 2.6.0-0 conda-forge
qt: 4.8.7-vc14_9 [vc14] --> 4.8.7-vc14_6 conda-forge [vc14]
I decided not to proceed and instead tried pip install pillow. Since this command doesn't ask for confirmation the package was simply installed. Now when I type conda list I get:
Pillow 4.0.0 <pip>
pillow 3.4.2 py35_0
The package list of the PyCharm Interpreter now shows Pillow as being version 4.0.0 but conda update pillow still returns:
# All requested packages already installed.
pillow 3.4.2 py35_0
My questions are:
1) What should I rely on to keep all my packages up to date, without compatibility issues?
2) Why did conda install pillow: 4.0.0 return an error but conda install -c conda-forge pillow=4.0.0 didn't?
3) What do the * next to pillow 4.0.0 and python 3.5 in the list of dependencies mean?
4) Since now I have both Pillow 3.4.2 (in /anaconda3/pkgs) and Pillow 4.0.0 (in /anaconda3/lib/site-packages) which one would be used if I imported Pillow?
5) Does the superseding conda: 4.3.11-py35_0 --> 4.2.13-py35_0 conda-forge mean conda is getting downgraded?
6) What is the difference between the tags pip, py35_0, py35_4, np111py35_2, etc?
7) PyCharm tells me there's a version 2.9.5 of package Jinja2 but both normal conda and conda-forge only find 2.9.4. From which channel is PyCharm getting this information?

Ok, I can't answer all of your questions but here goes:
1) Conda defers to the "pain up front" approach for handling dependency/conflict resolution. You'll have to get all of your packages to play nicely together in the repo's/channels that you have available to even make a package or keep them in an environment together. You can try running it with --force or --no-deps to try getting it in but ..... that can cause issues for you in the future (I don't know if that would even work with the later versions of conda, it changes a lot). Simply keeping packages up to date, and up to latest, I would just use pip. Its come a long way in the last few years (https://glyph.twistedmatrix.com/2016/08/python-packaging.html)
2) I am not completely sure, I believe it would have something to do with providing an explicit non-url channel for conda to look at. Typically you pass it the URL to the conda-forge repo (I think, again we don't use conda-forge internally).
3) The * means you are ignoring the patch/build 4.0.0 == Major.Minor.Build. Likewise, 3.5* == any version of 3.5
4) I would import pillow in a terminal, and then print out the module to see where its getting pulled from, why guess?
5) pass (although I think so)
6)
pip : means you installed that package via pip. It will not be picked up if you do conda list --explicit
py35_0 : has a requirement / only available to envs / packages that use python 3.5
py35_4 : not sure (always forget that one)
np111py35_2 : requires python3.5 and also numpy 1.11 (I think *)
7) I tend to steer clear of pycharm, I believe that you can inspect the python interpreter that pycharm is pointing at to see what environment its in. Based on the root environment, you can do a conda info and get a list of all of the channels you are pointing to.
Note: if you are going to use conda, you may just want to add conda-forge to your channels list instead of passing the -c (but seeing how the other channels are organized should help you see how you should pass the -c flag)