Is it possible to set the channel_priority to strict when creating an environment using the yaml file? For example:
name: my_environment
channels:
- conda-forge
dependencies:
- python
- geopandas
- rasterio
One additional note is that a specified channel for a given package does not need to be listed in the channels section. I find this safer as it does not risk to (re-)install some other package from unexpected channel.
So, for example:
channels:
- defaults
dependencies:
- python =3.8
- ...
# specifically from conda-forge (but only those):
- conda-forge::nbsphinx
Instead of:
# NO!
channels:
- defaults
- conda-forge
dependencies:
- python =3.8
- ...
- conda-forge::nbsphinx
Importantly, this seems to install only the specified packages from conda-forge, and it doesn't try to (re-)install conda-forge versions of packages that are in the dependency graph of those, but are already available (perhaps with a slightly less cutting-edge version) from pkgs/main.
Thanks to merv.
A workaround is to specify the channel for each package:
name: my_environment
channels:
- conda-forge
dependencies:
- conda-forge::python
- conda-forge::geopandas
- conda-forge::rasterio
A straightforward way is to first create the empty environment and set the channel priority to strict, then install the packages from the spec file:
conda create new_env
conda activate new_env
conda config --env --add channels conda-forge
conda config --env --set channel_priority strict
conda env update --name new_env --file env.yml
Note: if using a .txt spec file instead of .yml then replace the last line with
conda install --name new_env --file env.txt
reference doc:
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#id13
To directly answer your question, no I don't think it is possible to add channel_priority to the environment file. Maybe raise this as a Conda issue (if it isn't there already ;).
Something else worth trying would be adding defaults to the channel listing explicitly with lower priority like so...
name: my_environment
channels:
- conda-forge
- defaults
dependencies:
- python
- geopandas
- rasterio
Related
I'm trying out OpenCV with Python bindings for which I'm using the following YML file:
name: opencv-python-sandbox
channels:
- menpo
- conda-forge
- defaults
dependencies:
- jupyter=1.0.0
- jupyterlab=0.34.9
- keras=2.9.0
- matplotlib=3.5.2
- numpy=1.23.1
- opencv-python==4.6.0.66
- pandas=1.4.3
- python=3.8.0
- scikit-learn=1.1.1
- scipy=1.8.1
- tensorboard=2.9.1
- tensorflow=2.9.1
When I rain it threw some errors and says that it is not able to resolve OpenCV and Tensorflow:
(ml-sandbox) joesan#joesan-InfinityBook-S-14-v5:~/Projects/Private/ml-projects/ml-sandbox/opencv-python-sandbox$ conda env create -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
- tensorflow=2.9.1
- opencv-python==4.6.0.66
How to get this fixed? Do I need to add pip to my environment.yml and then manually install opencv via pip after activating the conda environment?
Not sure why this was not answered by anyone else as this seems to be a very common problem. Nevertheless, I was able to solve this by adding pip as a dependency in my environment.yml and use pip to install OpenCV and any other libraries that won't resolve with Conda.
My environment.yml looks like this:
name: ml-sandbox
channels:
- menpo
- conda-forge
- defaults
dependencies:
- jupyter=1.0.0
- jupyterlab=0.34.9
- keras=2.9.0
- matplotlib=3.5.2
- pandas=1.4.3
- python=3.8.0
- pip=22.1.2
- scikit-learn=1.1.1
- scipy=1.8.1
- tensorboard=2.9.1
- pip:
- numpy==1.23.1
- opencv-contrib-python==4.6.0.66
You have fixed it yourself by moving the requirements to the pip section, which results in an installation from Pypi. I just wanted to add explanation why your original attempt did not work and suggestions in case you want to strictly stick to using conda. Note that for both tensorflow and opencv, the packages provided on conda-forge are not maintained by the respective developers, often resulting in them lacking behind in versions.
The python bindings for openCV are called py-opencv on conda forge and have different version strings, so you would need to put py-opencv==4.6.0 in your yml
tensorflow on conda-forge goes only up to 2.8.1. So when strictly sticking to conda, you would need to downgrade the version
You can always check available versions for packages by using conda search -c <channel> <package-name> from your terminal
I added conda-forge to the conda channels:
$ conda config --show channels
channels:
- conda-forge
- defaults
my requirements.txt contains, among others, these lines:
ipython-genutils==0.2.0
jupyter-client==6.1.12
jupyterlab-pygments==0.1.2
appnope==0.1.2
jupyterlab-widgets==1.0.0
data==0.4
prometheus-client==0.11.0
latex==0.7.0
scipy==1.5.4
jupyter-core==4.7.1
jupyter-console==6.4.0
async-generator==1.10
vg==1.10.0
sklearn==0.0
postgis==1.0.4
When I try to create a new environment from this requirements.txt using conda with
conda create --name myenv --file requirements.txt
I get the following errors:
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- ipython-genutils==0.2.0
- jupyter-client==6.1.12
- jupyterlab-pygments==0.1.2
- appnope==0.1.2
- jupyterlab-widgets==1.0.0
- data==0.4
- prometheus-client==0.11.0
- latex==0.7.0
- scipy==1.5.4
- jupyter-core==4.7.1
- jupyter-console==6.4.0
- async-generator==1.10
- vg==1.10.0
- sklearn==0.0
- postgis==1.0.4
Current channels:
- https://conda.anaconda.org/conda-forge/linux-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
As you can see, conda-forge is listed under "current channels" and ipython-genutils==0.2.0 is available in conda-forge. However, the package is not found. How can I fix this problem?
I tried both conda config --set channel_priority flexible and ... stable
I run Ubuntu 20.04 LTS, Python 3.10 and Conda 4.12.0
It looks to me like this should have been a requirements.txt to be used by pip. Note that conda packages can have slightly different names than what is available on pypi.
ipython-genutils is not the correct name, looking at the link you have provided, the name of the package is ipython_genutils with an underscore. The same is true for the other packages that you have written with a hyphen. They should all be spelled with an underscore.
That leaves
- sklearn==0.0
- latex==0.7.0
- vg==1.10.0
- scipy==1.5.4
- postgis==1.0.4
- data==0.4
- appnope==0.1.2
sklearn==0.0 seems to be a corrupt line in your file. The package's name is scikit-learn. latex, vg and data are not available on conda channels as far as I can tell. The same goes for scipy==1.5.4, only 1.5.3 and 1.6 are available. postgis only goes back to 2.4.3 on conda-forge , see here, but also seems to be different from what is available on pypi. appnope is a package only available for macOS, see it's description:
Simple package for disabling App Nap on macOS >= 10.9, which can be problematic.
So with that in mind, we can create a yml file that installs from both conda channels and from pip (Changes to your file: replaced - with _, removed appnope, added pip dependency, renamed sklearn to scikit-learn and moved it together with latex, scipy, vg, data, postgis to pip requirements. If you are flexible with scipy==1.5.4, I would advise to change it to scipy==1.5.3 or scipy==1.6.0 and move scipy and sklearn out of the pip installed packages):
name: myenv
dependencies:
- ipython_genutils==0.2.0
- jupyter_client==6.1.12
- jupyterlab_pygments==0.1.2
- jupyterlab_widgets==1.0.0
- prometheus_client==0.11.0
- jupyter_core==4.7.1
- jupyter_console==6.4.0
- async_generator==1.10
- pip
- pip:
- scikit-learn
- latex==0.7.0
- scipy==1.5.4
- vg==1.10.0
- data==0.4.0
- postgis==1.0.4
Save this as environment.yml and then do
conda env create -f environment.yml
According to the documentation, if I use
conda env export > file.yml
I am able to share the environment with others. For better cross-platform compatibility, a better way would be:
conda env export --from-history > file.yml
listing only the packages explicitly requested (and not their associated dependencies).
That is what I did, I created a requirement yml file with the second command. Here it is:
name: torch
channels:
- defaults
dependencies:
- python=3.8
- humanize
- nltk
- pandas
- lxml
- numpy
- bs4
- fire
- neptune-client
- tqdm
- pyyaml
- torchaudio
- pytorch
- cudatoolkit=11.3
- torchvision
Among those packages, some were installed from the channel conda-forge: channels seem to be lost in the yaml file.
Indeed, if I try and use that file for cloning the environment (same machine):
conda env create -n torch2 --file=file.yml
I get an error for the packages installed from conda-forge (I explicitly installed from conda-forge only neptune-client and fire):
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
- torchaudio
- neptune-client
- fire
However, it seems that channels should be included in the yml. For example, on this github issue page I read:
Currently, conda env export does include channels information.
that closes the issue.
What am I missing?
NOTE: pytorch was installed with
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
from the official web page.
What am I missing?
The docs are wrong - or misleading (at best) when it is communicated that conda env export --from-history exports the channels that packages are installed from, or even any channels at all. This is not the behavior you get, nor is it what I get myself:
$ conda env export | head -n 8
name: smithy
channels:
- bioconda
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
$ conda env export --from-history | head -n 8
name: smithy
channels:
- defaults
dependencies:
- mamba
- constructor
- cookiecutter
- conda-build
Note that conda env export does include channel information, but in a highly-pinned way that's almost guaranteed to not work across platforms. So that's not going to work for your use case. I'm not sure if this is a bug or an oversight, but it's clearly not producing the desired result for the user.
Now to offer a (opinionated) recommendation on how to proceed: your best bet is to semi-manually curate an environment YAML file yourself and use that as a single source of truth. It looks like you can use your name: torch ... file a s a starting point, adding in the channels and maybe some other details as you go. Don't forget you can tie an individual package to a channel with the channel::package syntax a la
name: torch
channels:
- defaults
- conda-forge
- torch
dependencies:
- python=3.8
<SNIP>
- pytorch::torchaudio
- pytorch::pytorch
- pytorch::cudatoolkit=11.3
- pytorch::torchvision
I am trying to install fiona=1.6 but I get the following error
conda install fiona=1.6
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Collecting package metadata: done
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/noarch::flask-cors==3.0.7=py_0
- conda-forge/osx-64::blaze==0.11.3=py36_0
- conda-forge/noarch::flask==1.0.2=py_2
failed
PackagesNotFoundError: The following packages are not available from current channels:
- fiona=1.6 -> gdal==1.11.4
Current channels:
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
If I try to install gdal==1.11.4, I get the following
conda install -c conda-forge gdal=1.11.4
WARNING: The conda.compat module is deprecated and will be removed in a future release.
Collecting package metadata: done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/noarch::flask-cors==3.0.7=py_0
- conda-forge/osx-64::blaze==0.11.3=py36_0
- conda-forge/noarch::flask==1.0.2=py_2
failed
PackagesNotFoundError: The following packages are not available from current channels:
- gdal=1.11.4
Current channels:
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/osx-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
This is the result of conda info
conda info
active environment : base
active env location : /anaconda3
shell level : 1
user config file : /Users/massaro/.condarc
populated config files : /Users/massaro/.condarc
conda version : 4.6.11
conda-build version : 3.17.8
python version : 3.6.8.final.0
base environment : /anaconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/osx-64
https://conda.anaconda.org/conda-forge/noarch
package cache : /anaconda3/pkgs
/Users/massaro/.conda/pkgs
envs directories : /anaconda3/envs
/Users/massaro/.conda/envs
platform : osx-64
user-agent : conda/4.6.11 requests/2.21.0 CPython/3.6.8 Darwin/17.5.0 OSX/10.13.4
UID:GID : 502:20
netrc file : None
Python Versions
The Conda Forge channel only has gdal v1.11.4 for Python 2.7, 3.4, and 3.5. You either need to use a newer version of Fiona (current is 1.8) or make a new env that includes one of those older Python versions.
For example,
conda create -n fiona_1_6 fiona=1.6 python=3.5
Channel defaults is Required
Another issue you face is that you have removed the defaults channel from your configuration (as per your conda info). It is impossible to install fiona=1.6 with only the conda-forge channel. My recommendation would be to have both conda-forge and defaults in your configuration, but just set conda-forge to have higher priority (if that's what you want). You can do this like so,
conda config --append channels defaults
If you really don't want to include defaults, but just want a temporary workaround, then you can simply run the first command with a --channels | -c flag
conda create -n fiona_1_6 -c conda-forge -c defaults fiona=1.6 python=3.5
This will still give conda-forge precedence, but allow missing dependencies to be sourced from defaults.
Environment File
If you have more than just Fiona that you require, it may be cleaner to put together a requirements file, like so
fiona_1_6.yaml
name: fiona_1_6
channels:
- conda-forge
- defaults
dependencies:
- python=3.5
- fiona=1.6
- osmnx
Then create the new environment with this:
conda env create -f fiona_1_6.yaml
Doing what the error message told me to,
To search for alternate channels that may provide the conda package you're
looking for, navigate to https://anaconda.org
and typing in gdal in the search box led me to https://anaconda.org/conda-forge/gdal which has this installation instruction:
conda install -c conda-forge gdal=1.11.4
Try that to install the gdal dependency, maybe?
I have a problem with installing packages via conda. It can not find any package in channel https://pypi.org simple.
conda install logbook
returns:
PS C:\WINDOWS\system32> conda config --add channels
https://pypi.org/simple
Warning: 'https://pypi.org/simple' already in 'channels' list, moving to the top
PS C:\WINDOWS\system32> conda install Logbook
Fetching package metadata ....
WARNING: The remote server could not find the noarch directory for the
requested channel with url: https://pypi.org/simple
It is possible you have given conda an invalid channel. Please double-check
your conda configuration using `conda config --show`.
If the requested url is in fact a valid conda channel, please request that the
channel administrator create `noarch/repodata.json` and associated
`noarch/repodata.json.bz2` files, even if `noarch/repodata.json` is empty.
$ mkdir noarch
$ echo '{}' > noarch/repodata.json
$ bzip2 -k noarch/repodata.json
...........
PackageNotFoundError: Packages missing in current channels:
- logbook
We have searched for the packages in the following channels:
- https://pypi.org/simple/win-64
- https://pypi.org/simple/noarch
- https://repo.continuum.io/pkgs/main/win-64
- https://repo.continuum.io/pkgs/main/noarch
- https://repo.continuum.io/pkgs/free/win-64
- https://repo.continuum.io/pkgs/free/noarch
- https://repo.continuum.io/pkgs/r/win-64
- https://repo.continuum.io/pkgs/r/noarch
- https://repo.continuum.io/pkgs/pro/win-64
- https://repo.continuum.io/pkgs/pro/noarch
- https://repo.continuum.io/pkgs/msys2/win-64
- https://repo.continuum.io/pkgs/msys2/noarch
I have checked manually in browser, that the logbook module is in the list on page https://pypi.org/simple.
hoever, it seems that conda seeks packages in https://pypi.org/simple/win-64, but the win-64 directory does not exist there.
conda config --show says, that configured channel is https://pypi.org/simple
add_anaconda_token: True
add_pip_as_python_dependency: True
allow_non_channel_urls: True
allow_softlinks: False
always_copy: False
always_softlink: False
always_yes: False
anaconda_upload: None
auto_update_conda: True
changeps1: True
channel_alias: https://conda.anaconda.org
channel_priority: True
channels:
- https://pypi.org/simple
- https://pypi.org/simple/
- defaults
client_ssl_cert: None
client_ssl_cert_key: None
clobber: False
create_default_packages: []
custom_channels:
pkgs/main: https://repo.continuum.io/
pkgs/free: https://repo.continuum.io/
pkgs/r: https://repo.continuum.io/
pkgs/pro: https://repo.continuum.io/
pkgs/msys2: https://repo.continuum.io/
C:/Program%20Files/Anaconda3/conda-bld: file:///
Any advice?
A conda channel has to have a specific layout (win-64, win-32, ...) and the package has to be built in a conda-specific way (see Building conda packages with conda skeleton). The packages are generally build against specific Python versions (although there should also be noarch packages) and/or numpy versions and for different platforms (windows, linux, mac 32bit or 64bit).
You cannot directly install packages from PyPI using conda because PyPI doesn't qualify as conda-channel and even if it were the packages there aren't build like conda-packages. But you can install them using pip (within conda).
But: You could check if the packages you want are distributed in a conda-conforming channel (a very popular channel currently is conda-forge). At a first glance several channels contain a package named logbook in the anaconda cloud (search results for logbook).
If you find a channel that distributes the desired version of your package (and against the desired Python version and platform) then just use:
conda install -c channel_name logbook
Many Python products are available for Windows on the Gohlke page.
Assuming you have 64-bit Py3.6 installed in its own environment, called Py36, you can do this.
Download Logbook-1.1.0-cp36-cp36m-win_amd64.whl from Gohlke (or whatever version it is that you need).
Then within a command prompt:
activate the conda environment where you want to install Logbook.
Use pip to install the whl that you have downloaded within that environment.
Verify success, if you wish.
C:\scratch>activate Py36
(Py36) C:\scratch>pip install Logbook-1.1.0-cp36-cp36m-win_amd64.whl
Processing c:\scratch\logbook-1.1.0-cp36-cp36m-win_amd64.whl
Installing collected packages: Logbook
Successfully installed Logbook-1.1.0
(Py36) C:\scratch>conda list
# packages in environment at C:\ProgramData\Miniconda3\envs\Py36:
#
beautifulsoup4 4.6.0 <pip>
certifi 2017.11.5 py36hb8ac631_0
chardet 3.0.4 <pip>
idna 2.6 <pip>
Logbook 1.1.0 <pip>
opencv-python 3.4.0+contrib <pip>
pip 9.0.1 py36h226ae91_4
python 3.6.3 h3b118a2_4
requests 2.18.4 <pip>
setuptools 36.5.0 py36h65f9e6e_0
urllib3 1.22 <pip>
vc 14 h2379b0c_2
vs2015_runtime 14.0.25123 hd4c4e62_2
wheel 0.30.0 py36h6c3ec14_1
wikipedia 1.4.0 <pip>
wincertstore 0.2 py36h7fe50ca_0
Edit: Answer to query in comment.
I entered this:
conda env export -n Py36 -f Py36.yml
Content of yaml file.
name: Py36
channels:
- defaults
dependencies:
- certifi=2017.11.5=py36hb8ac631_0
- pip=9.0.1=py36h226ae91_4
- python=3.6.3=h3b118a2_4
- setuptools=36.5.0=py36h65f9e6e_0
- vc=14=h2379b0c_2
- vs2015_runtime=14.0.25123=hd4c4e62_2
- wheel=0.30.0=py36h6c3ec14_1
- wincertstore=0.2=py36h7fe50ca_0
- pip:
- beautifulsoup4==4.6.0
- chardet==3.0.4
- idna==2.6
- logbook==1.1.0
- opencv-python==3.4.0+contrib
- requests==2.18.4
- urllib3==1.22
- wikipedia==1.4.0
prefix: C:\ProgramData\Miniconda3\envs\Py36