In a python script that I run within a Snakemake rule, I’m trying to use the python package pysam, which is installed in one of my conda environments. It is also listed in the yaml file I generated from the conda env.
I’m specifying the yaml of that conda env in a Snakemake rule and then running a python script that uses pysam, but I’m getting a ModuleNotFoundError: No module named ‘pysam’. Without using Snakemake, when I run the script within the conda environment, it works just fine.
Inside my Snakefile:
rule match:
input: "input.bam"
output: "output.tsv"
conda: "/path/to/condaenv.yml"
script: "scripts/match.py"
I am running it with:
snakemake --profile lsf --use-conda.
This is the error:
Activating conda environment: /path/to/.snakemake/conda/3824034a93030
Traceback (most recent call last):
File "/path/to/scripts/tmppa0ynjrn.match.py", line 10, in <module>
import pysam
ModuleNotFoundError: No module named 'pysam'
Edit: here is my yaml --
name: chisel
channels:
- bioconda
- r
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
- backports=1.0=py_2
- backports.functools_lru_cache=1.6.1=py_0
- backports_abc=0.5=py_1
- bcftools=1.12=h45bccc9_1
- bwa=0.7.17=h5bf99c6_8
- bzip2=1.0.8=h7f98852_4
- c-ares=1.17.1=h7f98852_1
- ca-certificates=2021.5.30=ha878542_0
- certifi=2019.11.28=py27h8c360ce_1
- chisel=1.0.0=pyhdfd78af_0
- cycler=0.10.0=py_2
- freetype=2.10.4=h0708190_1
- functools32=3.2.3.2=py_3
- futures=3.3.0=py27h8c360ce_1
- gawk=5.1.0=h7f98852_0
- gsl=2.6=he838d99_2
- htslib=1.12=h9093b5e_1
- icu=64.2=he1b5a44_1
- kiwisolver=1.1.0=py27h9e3301b_1
- krb5=1.19.1=hcc1bbae_0
- ld_impl_linux-64=2.35.1=hea4e1c9_2
- libblas=3.9.0=8_openblas
- libcblas=3.9.0=8_openblas
- libcurl=7.77.0=h2574ce0_0
- libdeflate=1.7=h7f98852_5
- libedit=3.1.20191231=he28a2e2_2
- libev=4.33=h516909a_1
- libffi=3.2.1=he1b5a44_1007
- libgcc-ng=9.3.0=h2828fa1_19
- libgfortran-ng=7.5.0=h14aa051_19
- libgfortran4=7.5.0=h14aa051_19
- libgomp=9.3.0=h2828fa1_19
- liblapack=3.9.0=8_openblas
- libnghttp2=1.43.0=h812cca2_0
- libopenblas=0.3.12=pthreads_hb3c22a3_1
- libpng=1.6.37=h21135ba_2
- libssh2=1.9.0=ha56f1ee_6
- libstdcxx-ng=9.3.0=h6de172a_19
- matplotlib-base=2.2.5=py27h250f245_1
- ncurses=6.2=h58526e2_4
- numpy=1.16.5=py27h95a1406_0
- openssl=1.1.1k=h7f98852_0
- pandas=0.24.2=py27hb3f55d8_0
- patsy=0.5.1=py_0
- perl=5.32.1=0_h7f98852_perl5
- pip=20.1.1=pyh9f0ad1d_0
- pyparsing=2.4.7=pyh9f0ad1d_0
- pysam=0.16.0.1=py27hc729bab_3
- python=2.7.15=h5a48372_1011_cpython
- python-dateutil=2.8.1=py_0
- python_abi=2.7=1_cp27mu
- pytz=2020.1=pyh9f0ad1d_0
- readline=8.1=h46c0cb4_0
- samtools=1.12=h9aed4be_1
- scipy=1.2.1=py27h921218d_2
- seaborn=0.9.0=py_2
- setuptools=44.0.0=py27_0
- singledispatch=3.6.1=pyh44b312d_0
- six=1.16.0=pyh6c4a22f_0
- sqlite=3.35.5=h74cdb3f_0
- statsmodels=0.10.2=py27hc1659b7_0
- subprocess32=3.5.4=py27h516909a_0
- tk=8.6.10=h21135ba_1
- tornado=5.1.1=py27h14c3975_1000
- wheel=0.36.2=pyhd3deb0d_0
- xz=5.2.5=h516909a_1
- zlib=1.2.11=h516909a_1010
prefix: /home/leventhe/miniconda3/envs/chisel
Is there something else I need to do in Snakemake in order to be able to use the module from the conda env in the python script?
Related
I'm trying to create an environment with conda after having installed Miniconda3, on macOS. I am using the conda env create -f SEM1-CB.yml command through terminal but it gives me the following error:
Collecting package metadata (repodata.json): done
Solving environment: failed
ResolvePackageNotFound:
- xz==5.2.5=h62dcd97_1
- mkl-service==2.4.0=py310h2bbff1b_0
- numpy-base==1.21.2=py310h0829f74_0
- libzlib==1.2.11=h8ffe710_1013
- vc==14.2=hb210afc_5
- lz4-c==1.9.3=h8ffe710_1
- numpy==1.21.2=py310hfca59bb_0
- jpeg==9d=h8ffe710_0
- mkl_random==1.2.2=py310h4ed8f06_0
- zlib==1.2.11=h8ffe710_1013
- libffi==3.4.2=h8ffe710_5
- mkl==2021.4.0=haa95532_640
- setuptools==58.5.3=py310h5588dad_0
- mkl_fft==1.3.1=py310ha0764ea_0
- ca-certificates==2021.10.26=haa95532_4
- libtiff==4.3.0=hd413186_2
- openssl==3.0.0=h8ffe710_2
- intel-openmp==2021.4.0=haa95532_3556
- libdeflate==1.8=h8ffe710_0
- zstd==1.5.0=h6255e5f_0
- python==3.10.0=hcf16a7b_2_cpython
- vs2015_runtime==14.29.30037=h902a5da_5
- lcms2==2.12=h2a16943_0
- lerc==3.0=h0e60522_0
- openjpeg==2.4.0=hb211442_1
- pillow==8.4.0=py310h22f3323_0
- tk==8.6.11=h8ffe710_1
- ucrt==10.0.20348.0=h57928b3_0
- freetype==2.10.4=h546665d_1
- jbig==2.1=h8d14728_2003
- libpng==1.6.37=h1d00b33_2
- sqlite==3.36.0=h8ffe710_2
- bzip2==1.0.8=h8ffe710_4
The SEM1-CB.yml file is as follows:
name: SEM1-CB
channels:
- conda-forge
- defaults
dependencies:
- blas=1.0=mkl
- bzip2=1.0.8=h8ffe710_4
- ca-certificates=2021.10.26=haa95532_4
- freetype=2.10.4=h546665d_1
- intel-openmp=2021.4.0=haa95532_3556
- jbig=2.1=h8d14728_2003
- jpeg=9d=h8ffe710_0
- lcms2=2.12=h2a16943_0
- lerc=3.0=h0e60522_0
- libdeflate=1.8=h8ffe710_0
- libffi=3.4.2=h8ffe710_5
- libpng=1.6.37=h1d00b33_2
- libtiff=4.3.0=hd413186_2
- libzlib=1.2.11=h8ffe710_1013
- lz4-c=1.9.3=h8ffe710_1
- mkl=2021.4.0=haa95532_640
- mkl-service=2.4.0=py310h2bbff1b_0
- mkl_fft=1.3.1=py310ha0764ea_0
- mkl_random=1.2.2=py310h4ed8f06_0
- numpy=1.21.2=py310hfca59bb_0
- numpy-base=1.21.2=py310h0829f74_0
- olefile=0.46=pyh9f0ad1d_1
- openjpeg=2.4.0=hb211442_1
- openssl=3.0.0=h8ffe710_2
- pillow=8.4.0=py310h22f3323_0
- pip=21.3.1=pyhd8ed1ab_0
- pysimplegui=4.55.1=pyhd8ed1ab_0
- python=3.10.0=hcf16a7b_2_cpython
- python_abi=3.10=2_cp310
- setuptools=58.5.3=py310h5588dad_0
- six=1.16.0=pyhd3eb1b0_0
- sqlite=3.36.0=h8ffe710_2
- tk=8.6.11=h8ffe710_1
- tzdata=2021e=he74cb21_0
- ucrt=10.0.20348.0=h57928b3_0
- vc=14.2=hb210afc_5
- vs2015_runtime=14.29.30037=h902a5da_5
- wheel=0.37.0=pyhd8ed1ab_1
- xz=5.2.5=h62dcd97_1
- zlib=1.2.11=h8ffe710_1013
- zstd=1.5.0=h6255e5f_0
- pip:
- click==8.0.3
- colorama==0.4.4
- flask==2.0.2
- itsdangerous==2.0.1
- jinja2==3.0.3
- markupsafe==2.0.1
- pymata4==1.15
- pyserial==3.5
- werkzeug==2.0.2
prefix:
I need to create and activate this environment in order to add it as interpreter on visual studio code. I have tried the procedure on microsoft windows and it works, but I need it to work even on macOS.
Thank you in advance for any help!
In SEM1-CB.yml file you have Windows-specific conda package builds specified, which aren't available in OSX index, hence ends up showing you ResolvePackageNotFound error. Check the screenshot below I took from: https://anaconda.org/conda-forge/xz/files
To fix this, when you export your conda environment file in Windows run this command: conda env export --from-history --name env_name > file_name.yml or manually remove build numbers from the file.
I have conda installed locally on my Windows PC and also installed remotely on a Linux server. I already have conda packages installed locally on my Windows PC, and I want to install the same packages on the Linux server. I have already tried the following steps:
Create a requirements.txt file containing the currently installed packages and their versions using the Anaconda Prompt on my Windows PC using the command conda list -e > requirements.txt.
Transfer this requirements.txt file to my Linux server.
Install these packages in my conda base environment using the command conda install --yes --file requirements.txt.
However, I get the following error message on my Linux server when I try to complete step 3:
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- ca-certificates==2020.12.8=haa95532_0
- m2w64-gcc-libs-core==5.3.0=7
- audioread==2.1.8=pypi_0
- pywin32-ctypes==0.2.0=py38_1000
- notebook==6.1.5=py38haa95532_0
- librosa==0.8.0=pypi_0
- numpy-base==1.19.2=py38ha3acd2a_0
- psutil==5.7.2=py38he774522_0
- numpy==1.19.2=py38hadc3359_0
- regex==2020.11.13=py38h2bbff1b_0
- spyder-kernels==1.10.0=py38haa95532_0
- appdirs==1.4.4=pypi_0
- ujson==4.0.1=py38ha925a31_0
- setuptools==51.0.0=py38haa95532_2
- sklearn-crfsuite==0.3.6=pypi_0
- pywinpty==0.5.7=py38_0
- m2w64-gmp==6.1.0=2
- pyyaml==5.3.1=py38he774522_1
- bzip2==1.0.8=he774522_0
- sounddevice==0.4.1=pypi_0
- certifi==2020.12.5=py38haa95532_0
- gpytorch==1.3.0=pypi_0
- winpty==0.4.3=4
- pyzmq==20.0.0=py38hd77b12b_1
- pytorch==1.7.1=py3.8_cpu_0
- lazy-object-proxy==1.4.3=py38h2bbff1b_2
- zeromq==4.3.3=ha925a31_3
- ipython==7.19.0=py38hd4e2768_0
- mkl_fft==1.2.0=py38h45dec08_0
- conda-package-handling==1.7.2=py38h76e460a_0
- vc==14.2=h21ff451_1
- cpuonly==1.0=0
- pip==20.3.1=py38haa95532_0
- tornado==6.1=py38h2bbff1b_0
- libarchive==3.4.2=h5e25573_0
- msys2-conda-epoch==20160418=1
- pandocfilters==1.4.3=py38haa95532_1
- scikit-learn==0.23.2=pypi_0
- torchaudio==0.7.2=py38
- soundfile==0.10.3.post1=pypi_0
- gsl==2.4=hfa6e2cd_4
- kiwisolver==1.3.0=py38hd77b12b_0
- argon2-cffi==20.1.0=py38he774522_1
- dataclasses==0.6=pypi_0
- libtiff==4.1.0=h56a325e_1
- torchvision==0.8.2=py38_cpu
- m2w64-libwinpthread-git==5.0.0.4634.697f757=2
- numba==0.51.2=pypi_0
- pooch==1.2.0=pypi_0
- cvxopt==1.2.0=py38hdc3235a_0
- tabulate==0.8.7=pypi_0
- pillow==8.0.1=py38h4fa10fc_0
- libpng==1.6.37=h2a8f88b_0
- libiconv==1.15=h1df5818_7
- rtree==0.9.4=py38h21ff451_1
- qt==5.9.7=vc14h73c81de_0
- ruamel_yaml==0.15.87=py38he774522_1
- libsodium==1.0.18=h62dcd97_0
- yaml==0.2.5=he774522_0
- m2w64-gcc-libs==5.3.0=7
- libspatialindex==1.9.3=h33f27b4_0
- jedi==0.17.2=py38haa95532_1
- tk==8.6.10=he774522_0
- six==1.15.0=py38haa95532_0
- python-crfsuite==0.9.7=pypi_0
- spyder==4.2.0=py38haa95532_0
- cffi==1.14.4=py38hcd4344a_0
- xz==5.2.5=h62dcd97_0
- console_shortcut==0.1.1=4
- sqlite==3.33.0=h2a8f88b_0
- pycosat==0.6.3=py38h2bbff1b_0
- pyrsistent==0.17.3=py38he774522_0
- markupsafe==1.1.1=py38he774522_0
- bcrypt==3.2.0=py38he774522_0
- libuv==1.40.0=he774522_0
- brotlipy==0.7.0=py38h2bbff1b_1003
- mistune==0.8.4=py38he774522_1000
- wrapt==1.11.2=py38he774522_0
- powershell_shortcut==0.0.1=3
- mkl-service==2.3.0=py38h196d8e1_0
- pysocks==1.7.1=py38haa95532_0
- typeguard==2.10.0=pypi_0
- jpeg==9b=hb83a4c4_2
- libxml2==2.9.10=hb89e7f3_3
- freetype==2.10.4=hd328e21_0
- python==3.8.5=h5fd99cc_1
- liblief==0.10.1=ha925a31_0
- sip==4.19.13=py38ha925a31_0
- scipy==1.5.4=pypi_0
- pywin32==227=py38he774522_1
- nltk==3.5=pypi_0
- py-lief==0.10.1=py38ha925a31_0
- threadpoolctl==2.1.0=pypi_0
- zlib==1.2.11=h62dcd97_4
- cudatoolkit==10.2.89=h74a9793_1
- zstd==1.4.5=h04227a9_0
- mkl_random==1.1.1=py38h47e9c7a_0
- glpk==4.65=hdc00fd2_2
- ninja==1.10.2=py38h6d14046_0
- joblib==0.17.0=pypi_0
- typed-ast==1.4.1=py38he774522_0
- pandas==1.1.3=py38ha925a31_0
- llvmlite==0.34.0=pypi_0
- resampy==0.2.2=pypi_0
- pynacl==1.4.0=py38h62dcd97_1
- vs2015_runtime==14.27.29016=h5e58377_2
- icu==58.2=ha925a31_3
- matplotlib-base==3.3.2=py38hba9282a_0
- menuinst==1.4.16=py38he774522_1
- pyqt==5.9.2=py38ha925a31_4
- cryptography==3.3.1=py38hcd4344a_0
- jupyter_core==4.7.0=py38haa95532_0
- ax-platform==0.1.19=pypi_0
- botorch==0.3.3=pypi_0
- win_inet_pton==1.1.0=py38haa95532_0
- pkginfo==1.6.1=py38haa95532_0
- openssl==1.1.1i=h2bbff1b_0
- wincertstore==0.2=py38_0
- matplotlib==3.3.2=pypi_0
- lz4-c==1.9.2=hf4a77e7_3
- pandoc==2.11=h9490d1a_0
- conda==4.9.2=py38haa95532_0
- ad3==2.3.dev0=pypi_0
- retrying==1.3.3=pypi_0
- plotly==4.14.1=pypi_0
- m2w64-gcc-libgfortran==5.3.0=6
- watchdog==0.10.4=py38haa95532_0
- chardet==3.0.4=py38haa95532_1003
Current channels:
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/linux-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://repo.anaconda.com/pkgs/pro/linux-64
- https://repo.anaconda.com/pkgs/pro/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
I am aware that the problem is that I am not looking in the correct conda channels, as the error message suggests, but I am not sure how to solve this problem.
Thanks for the help.
The official documentation has some additional steps to have environment working cross-platform. Here is the link to that.
However, if you are not using packages that are only available in anaconda channels, you can do the following.
Have pip on both your conda environment (windows and linux server)
Make requirements.txt file using pip freeze instead of conda's one. From the windows machine
$ pip freeze > requirements.txt
Install the packages in linux server normally with pip.
$ pip install -r requirements.txt
I am not saying that it is the best option. But this way usually is easier can support other environment management packages like pyenv.
I work with conda environments and need some pip packages as well, e.g. pre-compiled wheels from ~gohlke.
At the moment I have two files: environment.yml for conda with:
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
and requirements.txt for pip which can be used after activating above conda environment:
# run: pip install -i requirements.txt
docx
gooey
http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl
Is there a possibility to combine them in one file (for conda)?
Pip dependencies can be included in the environment.yml file like this (docs):
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- numpy=1.13.3 # pin version for conda
- pip:
# works for regular pip packages
- docx
- gooey
- matplotlib==2.0.0 # pin version for pip
# and for wheels
- http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl
It also works for .whl files in the same directory (see Dengar's answer) as well as with common pip packages.
One can also use the requirements.txt directly in the YAML. For example,
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
- -r requirements.txt
Basically, any option you can run with pip install you can run in a YAML. See the Advanced Pip Example for a showcase of other capabilities.
Important Note
A previous version of this answer (and Conda's Advanced Pip Example) used a substandard file URI syntax:
- -r file:requirements.txt
Pip v21.2.1 introduced stricter behavior for URI parsing and no longer supports this. See this answer for details.
Just want to add that adding a wheel in the directory also works. I was getting this error when using the entire URL:
HTTP error 404 while getting http://www.lfd.uci.edu/~gohlke/pythonlibs/f9r7rmd8/opencv_python-3.1.0-cp35-none-win_amd64.whl
Ended up downloading the wheel and saving it into the same directory as the yml file.
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
- opencv_python-3.1.0-cp35-none-win_amd64.whl
If you want to do it automatically it seems that if you do:
conda env export > environment.yml
already has the pip things you need. No need to run pip freeze > requirements4pip.txt separately for me or include it as a
- pip:
- -r file:requirements.txt
as another answer has mentioned.
See my yml file:
$ cat environment.yml
name: myenv
channels:
- pytorch
- dglteam
- defaults
- conda-forge
dependencies:
- _libgcc_mutex=0.1=main
- absl-py=0.12.0=py38h06a4308_0
- aiohttp=3.7.4=py38h27cfd23_1
- async-timeout=3.0.1=py38h06a4308_0
- attrs=20.3.0=pyhd3eb1b0_0
- beautifulsoup4=4.9.3=pyha847dfd_0
- blas=1.0=mkl
- blinker=1.4=py38h06a4308_0
- brotlipy=0.7.0=py38h27cfd23_1003
- bzip2=1.0.8=h7b6447c_0
- c-ares=1.17.1=h27cfd23_0
- ca-certificates=2021.4.13=h06a4308_1
- cachetools=4.2.1=pyhd3eb1b0_0
- cairo=1.14.12=h8948797_3
- certifi=2020.12.5=py38h06a4308_0
- cffi=1.14.0=py38h2e261b9_0
- chardet=3.0.4=py38h06a4308_1003
- click=7.1.2=pyhd3eb1b0_0
- conda=4.10.1=py38h06a4308_1
- conda-build=3.21.4=py38h06a4308_0
- conda-package-handling=1.7.3=py38h27cfd23_1
- coverage=5.5=py38h27cfd23_2
- cryptography=3.4.7=py38hd23ed53_0
- cudatoolkit=11.0.221=h6bb024c_0
- cycler=0.10.0=py38_0
- cython=0.29.23=py38h2531618_0
- dbus=1.13.18=hb2f20db_0
- decorator=4.4.2=pyhd3eb1b0_0
- dgl-cuda11.0=0.6.1=py38_0
- dill=0.3.3=pyhd3eb1b0_0
- expat=2.3.0=h2531618_2
- filelock=3.0.12=pyhd3eb1b0_1
- fontconfig=2.13.1=h6c09931_0
- freetype=2.10.4=h7ca028e_0
- fribidi=1.0.10=h7b6447c_0
- gettext=0.21.0=hf68c758_0
- glib=2.66.3=h58526e2_0
- glob2=0.7=pyhd3eb1b0_0
- google-auth=1.29.0=pyhd3eb1b0_0
- google-auth-oauthlib=0.4.4=pyhd3eb1b0_0
- graphite2=1.3.14=h23475e2_0
- graphviz=2.40.1=h21bd128_2
- grpcio=1.36.1=py38h2157cd5_1
- gst-plugins-base=1.14.0=h8213a91_2
- gstreamer=1.14.0=h28cd5cc_2
- harfbuzz=1.8.8=hffaf4a1_0
- icu=58.2=he6710b0_3
- idna=2.10=pyhd3eb1b0_0
- importlib-metadata=3.10.0=py38h06a4308_0
- intel-openmp=2021.2.0=h06a4308_610
- jinja2=2.11.3=pyhd3eb1b0_0
- joblib=1.0.1=pyhd3eb1b0_0
- jpeg=9b=h024ee3a_2
- kiwisolver=1.3.1=py38h2531618_0
- lcms2=2.12=h3be6417_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libarchive=3.4.2=h62408e4_0
- libffi=3.2.1=hf484d3e_1007
- libgcc-ng=9.1.0=hdf63c60_0
- libgfortran-ng=7.3.0=hdf63c60_0
- libglib=2.66.3=hbe7bbb4_0
- libiconv=1.16=h516909a_0
- liblief=0.10.1=he6710b0_0
- libpng=1.6.37=h21135ba_2
- libprotobuf=3.14.0=h8c45485_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- libtiff=4.1.0=h2733197_1
- libuuid=1.0.3=h1bed415_2
- libuv=1.40.0=h7b6447c_0
- libxcb=1.14=h7b6447c_0
- libxml2=2.9.10=hb55368b_3
- lz4-c=1.9.2=he1b5a44_3
- markdown=3.3.4=py38h06a4308_0
- markupsafe=1.1.1=py38h7b6447c_0
- matplotlib=3.3.4=py38h06a4308_0
- matplotlib-base=3.3.4=py38h62a2d02_0
- mkl=2020.2=256
- mkl-service=2.3.0=py38h1e0a361_2
- mkl_fft=1.3.0=py38h54f3939_0
- mkl_random=1.2.0=py38hc5bc63f_1
- multidict=5.1.0=py38h27cfd23_2
- ncurses=6.2=he6710b0_1
- networkx=2.5.1=pyhd3eb1b0_0
- ninja=1.10.2=hff7bd54_1
- numpy=1.19.2=py38h54aff64_0
- numpy-base=1.19.2=py38hfa32c7d_0
- oauthlib=3.1.0=py_0
- olefile=0.46=pyh9f0ad1d_1
- openssl=1.1.1k=h27cfd23_0
- pandas=1.2.4=py38h2531618_0
- pango=1.42.4=h049681c_0
- patchelf=0.12=h2531618_1
- pcre=8.44=he6710b0_0
- pillow=8.2.0=py38he98fc37_0
- pip=21.0.1=py38h06a4308_0
- pixman=0.40.0=h7b6447c_0
- pkginfo=1.7.0=py38h06a4308_0
- protobuf=3.14.0=py38h2531618_1
- psutil=5.8.0=py38h27cfd23_1
- py-lief=0.10.1=py38h403a769_0
- pyasn1=0.4.8=py_0
- pyasn1-modules=0.2.8=py_0
- pycosat=0.6.3=py38h7b6447c_1
- pycparser=2.20=py_2
- pyjwt=2.0.1=pyhd8ed1ab_1
- pyopenssl=20.0.1=pyhd3eb1b0_1
- pyparsing=2.4.7=pyhd3eb1b0_0
- pyqt=5.9.2=py38h05f1152_4
- pysocks=1.7.1=py38h06a4308_0
- python=3.8.2=hcf32534_0
- python-dateutil=2.8.1=pyhd3eb1b0_0
- python-libarchive-c=2.9=pyhd3eb1b0_1
- python_abi=3.8=1_cp38
- pytorch=1.7.1=py3.8_cuda11.0.221_cudnn8.0.5_0
- pytz=2021.1=pyhd3eb1b0_0
- pyyaml=5.4.1=py38h27cfd23_1
- qt=5.9.7=h5867ecd_1
- readline=8.1=h27cfd23_0
- requests=2.25.1=pyhd3eb1b0_0
- requests-oauthlib=1.3.0=py_0
- ripgrep=12.1.1=0
- rsa=4.7.2=pyhd3eb1b0_1
- ruamel_yaml=0.15.100=py38h27cfd23_0
- scikit-learn=0.24.1=py38ha9443f7_0
- scipy=1.6.2=py38h91f5cce_0
- setuptools=52.0.0=py38h06a4308_0
- sip=4.19.13=py38he6710b0_0
- six=1.15.0=pyh9f0ad1d_0
- soupsieve=2.2.1=pyhd3eb1b0_0
- sqlite=3.35.4=hdfb4753_0
- tensorboard=2.4.0=pyhc547734_0
- tensorboard-plugin-wit=1.6.0=py_0
- threadpoolctl=2.1.0=pyh5ca1d4c_0
- tk=8.6.10=hbc83047_0
- torchaudio=0.7.2=py38
- torchtext=0.8.1=py38
- torchvision=0.8.2=py38_cu110
- tornado=6.1=py38h27cfd23_0
- typing-extensions=3.7.4.3=0
- typing_extensions=3.7.4.3=py_0
- urllib3=1.26.4=pyhd3eb1b0_0
- werkzeug=1.0.1=pyhd3eb1b0_0
- wheel=0.36.2=pyhd3eb1b0_0
- xz=5.2.5=h7b6447c_0
- yaml=0.2.5=h7b6447c_0
- yarl=1.6.3=py38h27cfd23_0
- zipp=3.4.1=pyhd3eb1b0_0
- zlib=1.2.11=h7b6447c_3
- zstd=1.4.5=h9ceee32_0
- pip:
- aioconsole==0.3.1
- lark-parser==0.6.5
- lmdb==0.94
- pexpect==4.6.0
- progressbar2==3.39.3
- ptyprocess==0.7.0
- pycapnp==1.0.0
- python-utils==2.5.6
- sexpdata==0.0.3
- tqdm==4.56.0
prefix: /home/miranda9/miniconda3/envs/myenv
Note that at the time of this writing doing conda env create --file environment.yml to create the yml env results in an error:
$ conda env create --file environment.yml
CondaValueError: prefix already exists: /home/miranda9/miniconda3/envs/myenv
I would like to know how to install python libraries using yml file without making a new environment. I already have tensorflow environment in conda. I want to install list of libraries into this tensorflow environment. It is the only way I know manually add each of these libraries but it is very hard to do this list. Please give me solution for that
This is yml file:
name: virtual_platform
channels:
- menpo
- conda-forge
- peterjc123
- defaults
dependencies:
- ffmpeg=3.2.4=1
- freetype=2.7=vc14_1
- imageio=2.2.0=py35_0
- libtiff=4.0.6=vc14_7
- olefile=0.44=py35_0
- pillow=4.2.1=py35_0
- vc=14=0
- alabaster=0.7.10=py35_0
- astroid=1.5.3=py35_0
- babel=2.5.0=py35_0
- bleach=1.5.0=py35_0
- certifi=2016.2.28=py35_0
- cffi=1.10.0=py35_0
- chardet=3.0.4=py35_0
- colorama=0.3.9=py35_0
- decorator=4.1.2=py35_0
- docutils=0.14=py35_0
- entrypoints=0.2.3=py35_0
- html5lib=0.9999999=py35_0
- icu=57.1=vc14_0
- imagesize=0.7.1=py35_0
- ipykernel=4.6.1=py35_0
- ipython=6.1.0=py35_0
- ipython_genutils=0.2.0=py35_0
- isort=4.2.15=py35_0
- jedi=0.10.2=py35_2
- jinja2=2.9.6=py35_0
- jpeg=9b=vc14_0
- jsonschema=2.6.0=py35_0
- jupyter_client=5.1.0=py35_0
- jupyter_core=4.3.0=py35_0
- lazy-object-proxy=1.3.1=py35_0
- libpng=1.6.30=vc14_1
- markupsafe=1.0=py35_0
- mistune=0.7.4=py35_0
- mkl=2017.0.3=0
- nbconvert=5.2.1=py35_0
- nbformat=4.4.0=py35_0
- numpy=1.13.1=py35_0
- numpydoc=0.7.0=py35_0
- openssl=1.0.2l=vc14_0
- pandocfilters=1.4.2=py35_0
- path.py=10.3.1=py35_0
- pickleshare=0.7.4=py35_0
- pip=9.0.1=py35_1
- prompt_toolkit=1.0.15=py35_0
- psutil=5.2.2=py35_0
- pycodestyle=2.3.1=py35_0
- pycparser=2.18=py35_0
- pyflakes=1.6.0=py35_0
- pygments=2.2.0=py35_0
- pylint=1.7.2=py35_0
- pyqt=5.6.0=py35_2
- python=3.5.4=0
- python-dateutil=2.6.1=py35_0
- pytz=2017.2=py35_0
- pyzmq=16.0.2=py35_0
- qt=5.6.2=vc14_6
- qtawesome=0.4.4=py35_0
- qtconsole=4.3.1=py35_0
- qtpy=1.3.1=py35_0
- requests=2.14.2=py35_0
- rope=0.9.4=py35_1
- setuptools=36.4.0=py35_1
- simplegeneric=0.8.1=py35_1
- singledispatch=3.4.0.3=py35_0
- sip=4.18=py35_0
- six=1.10.0=py35_1
- snowballstemmer=1.2.1=py35_0
- sphinx=1.6.3=py35_0
- sphinxcontrib=1.0=py35_0
- sphinxcontrib-websupport=1.0.1=py35_0
- spyder=3.2.3=py35_0
- testpath=0.3.1=py35_0
- tornado=4.5.2=py35_0
- traitlets=4.3.2=py35_0
- vs2015_runtime=14.0.25420=0
- wcwidth=0.1.7=py35_0
- wheel=0.29.0=py35_0
- win_unicode_console=0.5=py35_0
- wincertstore=0.2=py35_0
- wrapt=1.10.11=py35_0
- zlib=1.2.11=vc14_0
- opencv3=3.1.0=py35_0
- pytorch=0.1.12=py35_0.1.12cu80
- torch==0.1.12
- torchvision==0.1.9
- pip:
- ipython-genutils==0.2.0
- jupyter-client==5.1.0
- jupyter-core==4.3.0
- prompt-toolkit==1.0.15
- pyyaml==3.12
- rope-py3k==0.9.4.post1
- torch==0.1.12
- torchvision==0.1.9
- win-unicode-console==0.5
You can use the conda env update command:
conda env update --name <your env name> -f <your file>.yml
or, if the environment you want to update is already activated, then
conda env update -f <your file>.yml
If you want to create the environment from your yml file:
conda env create -f environment.yml
The name of your environment is virtual_platform. If you want another name, just edit your yml name to desired name.
It is not recommended to install packages to your base environment but if that is what you want, and I believe you should not, you need to create a requirement.txt from dependencies listed on your yml.
Copy and paste all the dependencies
packages and there version to requirements.txt as:
python ==3.5
ffmpeg=3.2.4
freetype=2.7
imageio=2.2.0
...
Then do:
conda install --yes --file requirements.txt
The problem is that this will fail if any dependence fail to install. So I will recommend installing using yml which means having an environment separate from the rest.
I work with conda environments and need some pip packages as well, e.g. pre-compiled wheels from ~gohlke.
At the moment I have two files: environment.yml for conda with:
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
and requirements.txt for pip which can be used after activating above conda environment:
# run: pip install -i requirements.txt
docx
gooey
http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl
Is there a possibility to combine them in one file (for conda)?
Pip dependencies can be included in the environment.yml file like this (docs):
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- numpy=1.13.3 # pin version for conda
- pip:
# works for regular pip packages
- docx
- gooey
- matplotlib==2.0.0 # pin version for pip
# and for wheels
- http://www.lfd.uci.edu/~gohlke/pythonlibs/bofhrmxk/opencv_python-3.1.0-cp35-none-win_amd64.whl
It also works for .whl files in the same directory (see Dengar's answer) as well as with common pip packages.
One can also use the requirements.txt directly in the YAML. For example,
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
- -r requirements.txt
Basically, any option you can run with pip install you can run in a YAML. See the Advanced Pip Example for a showcase of other capabilities.
Important Note
A previous version of this answer (and Conda's Advanced Pip Example) used a substandard file URI syntax:
- -r file:requirements.txt
Pip v21.2.1 introduced stricter behavior for URI parsing and no longer supports this. See this answer for details.
Just want to add that adding a wheel in the directory also works. I was getting this error when using the entire URL:
HTTP error 404 while getting http://www.lfd.uci.edu/~gohlke/pythonlibs/f9r7rmd8/opencv_python-3.1.0-cp35-none-win_amd64.whl
Ended up downloading the wheel and saving it into the same directory as the yml file.
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
- opencv_python-3.1.0-cp35-none-win_amd64.whl
If you want to do it automatically it seems that if you do:
conda env export > environment.yml
already has the pip things you need. No need to run pip freeze > requirements4pip.txt separately for me or include it as a
- pip:
- -r file:requirements.txt
as another answer has mentioned.
See my yml file:
$ cat environment.yml
name: myenv
channels:
- pytorch
- dglteam
- defaults
- conda-forge
dependencies:
- _libgcc_mutex=0.1=main
- absl-py=0.12.0=py38h06a4308_0
- aiohttp=3.7.4=py38h27cfd23_1
- async-timeout=3.0.1=py38h06a4308_0
- attrs=20.3.0=pyhd3eb1b0_0
- beautifulsoup4=4.9.3=pyha847dfd_0
- blas=1.0=mkl
- blinker=1.4=py38h06a4308_0
- brotlipy=0.7.0=py38h27cfd23_1003
- bzip2=1.0.8=h7b6447c_0
- c-ares=1.17.1=h27cfd23_0
- ca-certificates=2021.4.13=h06a4308_1
- cachetools=4.2.1=pyhd3eb1b0_0
- cairo=1.14.12=h8948797_3
- certifi=2020.12.5=py38h06a4308_0
- cffi=1.14.0=py38h2e261b9_0
- chardet=3.0.4=py38h06a4308_1003
- click=7.1.2=pyhd3eb1b0_0
- conda=4.10.1=py38h06a4308_1
- conda-build=3.21.4=py38h06a4308_0
- conda-package-handling=1.7.3=py38h27cfd23_1
- coverage=5.5=py38h27cfd23_2
- cryptography=3.4.7=py38hd23ed53_0
- cudatoolkit=11.0.221=h6bb024c_0
- cycler=0.10.0=py38_0
- cython=0.29.23=py38h2531618_0
- dbus=1.13.18=hb2f20db_0
- decorator=4.4.2=pyhd3eb1b0_0
- dgl-cuda11.0=0.6.1=py38_0
- dill=0.3.3=pyhd3eb1b0_0
- expat=2.3.0=h2531618_2
- filelock=3.0.12=pyhd3eb1b0_1
- fontconfig=2.13.1=h6c09931_0
- freetype=2.10.4=h7ca028e_0
- fribidi=1.0.10=h7b6447c_0
- gettext=0.21.0=hf68c758_0
- glib=2.66.3=h58526e2_0
- glob2=0.7=pyhd3eb1b0_0
- google-auth=1.29.0=pyhd3eb1b0_0
- google-auth-oauthlib=0.4.4=pyhd3eb1b0_0
- graphite2=1.3.14=h23475e2_0
- graphviz=2.40.1=h21bd128_2
- grpcio=1.36.1=py38h2157cd5_1
- gst-plugins-base=1.14.0=h8213a91_2
- gstreamer=1.14.0=h28cd5cc_2
- harfbuzz=1.8.8=hffaf4a1_0
- icu=58.2=he6710b0_3
- idna=2.10=pyhd3eb1b0_0
- importlib-metadata=3.10.0=py38h06a4308_0
- intel-openmp=2021.2.0=h06a4308_610
- jinja2=2.11.3=pyhd3eb1b0_0
- joblib=1.0.1=pyhd3eb1b0_0
- jpeg=9b=h024ee3a_2
- kiwisolver=1.3.1=py38h2531618_0
- lcms2=2.12=h3be6417_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libarchive=3.4.2=h62408e4_0
- libffi=3.2.1=hf484d3e_1007
- libgcc-ng=9.1.0=hdf63c60_0
- libgfortran-ng=7.3.0=hdf63c60_0
- libglib=2.66.3=hbe7bbb4_0
- libiconv=1.16=h516909a_0
- liblief=0.10.1=he6710b0_0
- libpng=1.6.37=h21135ba_2
- libprotobuf=3.14.0=h8c45485_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- libtiff=4.1.0=h2733197_1
- libuuid=1.0.3=h1bed415_2
- libuv=1.40.0=h7b6447c_0
- libxcb=1.14=h7b6447c_0
- libxml2=2.9.10=hb55368b_3
- lz4-c=1.9.2=he1b5a44_3
- markdown=3.3.4=py38h06a4308_0
- markupsafe=1.1.1=py38h7b6447c_0
- matplotlib=3.3.4=py38h06a4308_0
- matplotlib-base=3.3.4=py38h62a2d02_0
- mkl=2020.2=256
- mkl-service=2.3.0=py38h1e0a361_2
- mkl_fft=1.3.0=py38h54f3939_0
- mkl_random=1.2.0=py38hc5bc63f_1
- multidict=5.1.0=py38h27cfd23_2
- ncurses=6.2=he6710b0_1
- networkx=2.5.1=pyhd3eb1b0_0
- ninja=1.10.2=hff7bd54_1
- numpy=1.19.2=py38h54aff64_0
- numpy-base=1.19.2=py38hfa32c7d_0
- oauthlib=3.1.0=py_0
- olefile=0.46=pyh9f0ad1d_1
- openssl=1.1.1k=h27cfd23_0
- pandas=1.2.4=py38h2531618_0
- pango=1.42.4=h049681c_0
- patchelf=0.12=h2531618_1
- pcre=8.44=he6710b0_0
- pillow=8.2.0=py38he98fc37_0
- pip=21.0.1=py38h06a4308_0
- pixman=0.40.0=h7b6447c_0
- pkginfo=1.7.0=py38h06a4308_0
- protobuf=3.14.0=py38h2531618_1
- psutil=5.8.0=py38h27cfd23_1
- py-lief=0.10.1=py38h403a769_0
- pyasn1=0.4.8=py_0
- pyasn1-modules=0.2.8=py_0
- pycosat=0.6.3=py38h7b6447c_1
- pycparser=2.20=py_2
- pyjwt=2.0.1=pyhd8ed1ab_1
- pyopenssl=20.0.1=pyhd3eb1b0_1
- pyparsing=2.4.7=pyhd3eb1b0_0
- pyqt=5.9.2=py38h05f1152_4
- pysocks=1.7.1=py38h06a4308_0
- python=3.8.2=hcf32534_0
- python-dateutil=2.8.1=pyhd3eb1b0_0
- python-libarchive-c=2.9=pyhd3eb1b0_1
- python_abi=3.8=1_cp38
- pytorch=1.7.1=py3.8_cuda11.0.221_cudnn8.0.5_0
- pytz=2021.1=pyhd3eb1b0_0
- pyyaml=5.4.1=py38h27cfd23_1
- qt=5.9.7=h5867ecd_1
- readline=8.1=h27cfd23_0
- requests=2.25.1=pyhd3eb1b0_0
- requests-oauthlib=1.3.0=py_0
- ripgrep=12.1.1=0
- rsa=4.7.2=pyhd3eb1b0_1
- ruamel_yaml=0.15.100=py38h27cfd23_0
- scikit-learn=0.24.1=py38ha9443f7_0
- scipy=1.6.2=py38h91f5cce_0
- setuptools=52.0.0=py38h06a4308_0
- sip=4.19.13=py38he6710b0_0
- six=1.15.0=pyh9f0ad1d_0
- soupsieve=2.2.1=pyhd3eb1b0_0
- sqlite=3.35.4=hdfb4753_0
- tensorboard=2.4.0=pyhc547734_0
- tensorboard-plugin-wit=1.6.0=py_0
- threadpoolctl=2.1.0=pyh5ca1d4c_0
- tk=8.6.10=hbc83047_0
- torchaudio=0.7.2=py38
- torchtext=0.8.1=py38
- torchvision=0.8.2=py38_cu110
- tornado=6.1=py38h27cfd23_0
- typing-extensions=3.7.4.3=0
- typing_extensions=3.7.4.3=py_0
- urllib3=1.26.4=pyhd3eb1b0_0
- werkzeug=1.0.1=pyhd3eb1b0_0
- wheel=0.36.2=pyhd3eb1b0_0
- xz=5.2.5=h7b6447c_0
- yaml=0.2.5=h7b6447c_0
- yarl=1.6.3=py38h27cfd23_0
- zipp=3.4.1=pyhd3eb1b0_0
- zlib=1.2.11=h7b6447c_3
- zstd=1.4.5=h9ceee32_0
- pip:
- aioconsole==0.3.1
- lark-parser==0.6.5
- lmdb==0.94
- pexpect==4.6.0
- progressbar2==3.39.3
- ptyprocess==0.7.0
- pycapnp==1.0.0
- python-utils==2.5.6
- sexpdata==0.0.3
- tqdm==4.56.0
prefix: /home/miranda9/miniconda3/envs/myenv
Note that at the time of this writing doing conda env create --file environment.yml to create the yml env results in an error:
$ conda env create --file environment.yml
CondaValueError: prefix already exists: /home/miranda9/miniconda3/envs/myenv