I am trying to setup the environment in my Unix terminal. When I run my yaml file with make, I am getting the error
make: *** [Makefile:105: environment-dev] Error 247
But when I remove conda-forge from yaml file, all my packages are getting installed but in the end I am getting the error
Adding activation of '/home/xxx/yyy/.env' to conda 'env-abc' environment...
/bin/sh: 1: .: Can't open /home/xxx/yyy/.env
/bin/sh: 1: cannot create : Directory nonexistent
Please find the below yaml file
channels:
- defaults
- conda-forge
dependencies:
- python==3.7.1
- pip:
- -r src/requirements.txt
- jupyterlab==0.35.*
- flake8==3.7.*
- -e . # The project package.
Command I used:
make filename
Please advice
My Bad. I just create .env file and it worked well.
Related
I have a project on ReadTheDocs that I'm trying to build. I'm using a very basic .readthedocs.yaml file that reads:
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the version of Python and other tools you might need
build:
os: ubuntu-20.04
tools:
python: "3.9"
# You can also specify other tool versions:
# nodejs: "16"
# rust: "1.55"
# golang: "1.17"
# Build documentation in the docs/ directory with Sphinx
sphinx:
builder: html
configuration: docs/source/conf.py
fail_on_warning: true
# If using Sphinx, optionally build your docs in additional formats such as PDF
# formats:
# - pdf
# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
conda:
environment: environment.yml
Unfortunately, the RTD build logs seem to tell me that after cloning and writing out the environment.yml file, the build process runs python env create --quiet --name develop --file environment.yml. This obviously fails with "no such file or directory" (Error 2) since, well, no such file or directory as env exists in the directory structure. Shouldn't RTD be running conda create here? How do I make it do the right thing?
Thanks,
Eli
This problem is described in https://github.com/readthedocs/readthedocs.org/issues/8595
In summary, python: "3.9" now means that CPython 3.9 + venv are used, regardless of conda.environment.
If one wants to use a conda environment, python: "miniconda3-4.7" or python: "mambaforge-4.10" need to be specified.
In the future, a better error message should be shown at least. Feel free to upvote the issue.
I'm trying to Create New conda environment by 'Anaconda Prompt' usnig yml File in Windows 10.
So here is the steps i made through:
1. using cd command i changed the directory to dir which my yml file located. (suppose my yml file is in c:/Users/<USER NAME>/.jupyter )
2. Then i used conda env create -f Python 310.yml command to create new conda env.
and what i got is:
SpecNotFound: Invalid name, try the format: user/package
Now I don't know how can I solve this problem and exactly what is the meaning of this error.
Appendix
my Python 310.yml file contains these stuff:
drop the "env" in "conda env create" it's just "conda create ..."
The other answer says to:
drop the "env" in "conda env create" it's just "conda create ..."
This did not fix it for me by itself, but it did work after trying that and then after that trying the whole
conda env create -n myenv-dev --file my_env.yml
command again.
issue solved by changing contents of Python 310.yml and renaming yml file to Python310.yml. Here is the final .yml file content:
name: Python3.9
channels:
- defaults
dependencies:
- numpy
- pandas
- matplotlib
- pip
- python=3.9.*
- python-dateutil
- pytz
- scikit-learn
- scipy
- statsmodels
- xlrd
- openpyxl
- lxml
- html5lib
- beautifulsoup4
- jupyter
- pip:
- pmdarima
- tensorflow
- keras
prefix: C:\Users\Shayan\Anaconda3\envs\Python3.9
You can use conda create -f Python 310.yml.
In my case the problem was that the file did not exist. So make sure that the file exists before running the update command.
Make sure your directory is in "quotes", bc I got this error having whitespace inside it...
My solution was because I was running the following command on windows
conda env create -n spa --file .\environment.yml
and in order to get it to run on my Ubuntu WSL2 just needed a minor update:
conda env create -n spa --file environment.yml
I put the name of the yml file in double quotes and used the -f= notation and powershell no longer gave me an error. For you, that might mean:
conda env create -f="Python 310.yml"
I would like to be able to install R packages from GitHub in a R conda environment created by Snakemake, as well as python libraries via pip in a python environment. I'll use these environments in a whole set of rules thereafter.
My initial thought was to create a rule running a script to install the specified packages.
For instance, my initial run was: snakemake -j1 --use-conda -R create_r_environment.
My Snakefile:
rule create_r_environment:
conda:
"envs/r.yaml"
script:
"scripts/r-dependencies.R"
rule create_python_environment:
conda:
"envs/python.yaml"
script:
"scripts/python-dependencies.py"
My envs/r.yaml file:
channels:
- conda-forge
dependencies:
- r=4.0
My r-dependencies.R file:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
My envs/pyton.yaml file:
channels:
- conda-forge
dependencies:
- python=3.8.2
My python-dependencies.py file:
!pip install gseapy
The log output:
Building DAG of jobs...
Creating conda environment envs/r.yaml...
Downloading and installing remote packages.
Environment for envs/r.yaml created (location: .snakemake/conda/388,repos = "http://cran.us.r-project.org")f7df8)
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 create_r_environment
1
[Fri Oct 30 22:38:56 2020]
rule create_r_environment:
jobid: 0
Activating conda environment: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8
[Fri Oct 30 22:38:57 2020]
Error in rule create_r_environment:
jobid: 0
conda-env: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8
RuleException:
CalledProcessError in line 5 of /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/Snakefile:
Command 'source /home/cmcouto-silva/miniconda3/bin/activate '/home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/conda/388f7df8'; set -euo pipefail; Rscript --vanilla /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/scripts/tmpa6jdxovx.r-dependencies.R' returned non-zero exit status 1.
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2168, in run_wrapper
File "/home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/Snakefile", line 5, in __rule_create_r_environment
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 529, in _callback
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/concurrent/futures/thread.py", line 57, in run
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
File "/home/cmcouto-silva/miniconda3/envs/snakemake/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2199, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/cmcouto-silva/cmcouto.silva#usp.br/lab_files/phd_data/SO/.snakemake/log/2020-10-30T223743.852983.snakemake.log
My folder structure:
.
├── envs
│ ├── python.yaml
│ └── r.yaml
├── scripts
│ ├── python-dependencies.py
│ └── r-dependencies.R
└── Snakefile
It successfully creates the environment but fails when running the script, and I don't know why. I've changed the envs/r.yaml file content to install.packages("data.table") to see if there was an issue with the github package, but it's not. It fails anyway. The same occurs when I run the rule create_python_environment (output not showed here).
Any help?
Edit after the accepted answer
As #dariober pointed out, I forgot to install the remotes package before calling it in the script. I did it in the .yaml file, and it worked well. Also, I installed the pip libraries using shell instead of a python file.
I would like to highlight some points though, just in case anyone's facing the same or similar problem:
First, I could successfully install further packages I needed to, but some of them require specific libraries (e.g. libcurl), which is installed in my system, but it's not recognized inside the Snakemake conda environment, forcing me to either install it in the Snakemake conda environment (which is good for reproducibility, although I don't know how to do that yet) or specify the path library. Maybe a better option would be using a container just like #merv commented out.
Second, I figured out that Snakemake already provides a way to install pip libraries using the .yaml file. From the documentation, it looks like this:
name: stats2
channels:
- javascript
dependencies:
- python=3.6 # or 2.7
- bokeh=0.9.2
- numpy=1.9.*
- nodejs=0.10.*
- flask
- pip:
- Flask-Testing
I think there are quite a few wrong things:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never"): In your r.yaml you should include the remotes package.
!pip install gseapy is not valid python code. If anything, it is code to be executed by shell but I'm not sure that leading ! is correct. Also, gseapy is available from bioconda I don;t see why you should install it with pip.
Before OP edited the question
My envs/r.yaml file:
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
It's odd that you get the conda environment correctly created since that r.yaml is not a valid environment file.
This is what I tried to recreate your issue:
r.yaml
cat r.yaml
remotes::install_github("ramiromagno/gwasrapidd", upgrade = "never")
Snakefile:
cat Snakefile
rule create_r_environment:
conda:
"r.yaml"
script:
"r-dependencies.R"
Execute:
snakemake -j1 --use-conda -R create_r_environment
Building DAG of jobs...
Creating conda environment r.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /home/dario/Downloads/r.yaml:
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 80, in execute
directory=os.getcwd())
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/specs/__init__.py", line 40, in detect
if spec.can_handle():
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/specs/yaml_file.py", line 18, in can_handle
self._environment = env.from_file(self.filename)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 151, in from_file
return from_yaml(yamlstr, filename=filename)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 137, in from_yaml
data = validate_keys(data, kwargs)
File "/home/dario/miniconda3/lib/python3.7/site-packages/conda_env/env.py", line 35, in validate_keys
new_data = data.copy() if data else {}
AttributeError: 'str' object has no attribute 'copy'
`$ /home/dario/miniconda3/bin/conda-env create --file /home/dario/Downloads/.snakemake/conda/095b0ca2.yaml --prefix /home/dario/Downloads/.snakemake/conda/095b0ca2`
environment variables:
CIO_TEST=<not set>
CMAKE_PREFIX_PATH=/home/dario/miniconda3/envs/tritume:/home/dario/miniconda3/envs/tritum
e/x86_64-conda-linux-gnu/sysroot/usr
CONDA_AUTO_UPDATE_CONDA=false
CONDA_BUILD_SYSROOT=/home/dario/miniconda3/envs/tritume/x86_64-conda-linux-gnu/sysroot
CONDA_DEFAULT_ENV=tritume
CONDA_EXE=/home/dario/miniconda3/bin/conda
CONDA_PREFIX=/home/dario/miniconda3/envs/tritume
CONDA_PROMPT_MODIFIER=(tritume)
CONDA_PYTHON_EXE=/home/dario/miniconda3/bin/python
CONDA_ROOT=/home/dario/miniconda3
CONDA_SHLVL=1
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
PATH=/home/dario/miniconda3/envs/tritume/bin:/home/dario/miniconda3/condabi
n:/opt/gradle/gradle-5.2/bin:/home/dario/.local/share/umake/bin:/home/
dario/.local/bin:/home/dario/bin:/opt/gradle/gradle-5.2/bin:/usr/local
/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/loc
al/games:/snap/bin:/usr/lib/jvm/java-10-oracle/bin:/usr/lib/jvm/java-1
0-oracle/db/bin
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
WINDOWPATH=2
active environment : tritume
active env location : /home/dario/miniconda3/envs/tritume
shell level : 1
user config file : /home/dario/.condarc
populated config files : /home/dario/.condarc
conda version : 4.8.3
conda-build version : not installed
python version : 3.7.6.final.0
virtual packages : __glibc=2.27
base environment : /home/dario/miniconda3 (writable)
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/dario/miniconda3/pkgs
/home/dario/.conda/pkgs
envs directories : /home/dario/miniconda3/envs
/home/dario/.conda/envs
platform : linux-64
user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.6 Linux/4.15.0-91-generic ubuntu/18.04.4 glibc/2.27
UID:GID : 1001:1001
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?
[y/N]:
Timeout reached. No report sent.
File "/home/dario/miniconda3/envs/tritume/lib/python3.6/site-packages/snakemake/deployment/conda.py", line 320, in create
Anyway, your error says:
... r-dependencies.R' returned non-zero exit status 1
What do you have in r-dependencies.R?
I took these from yml file of packages that my current environment is missing. How do I just install these within my current environment?
channels:
- defaults
dependencies:
- appdirs=1.4.3=py36h28b3542_0
- asn1crypto=0.24.0=py36_0
- attrs=18.2.0=py36h28b3542_0
- blas=1.0=mkl
- cffi=1.11.5=py36h6174b99_1
- constantly=15.1.0=py36h28b3542_0
- cryptography=2.3.1=py36hdbc3d79_0
- freetype=2.9.1=hb4e5f40_0
- html5lib=1.0.1=py36_0
- idna=2.7=py36_0
- incremental=17.5.0=py36_0
- intel-openmp=2019.0=118
- libgfortran=3.0.1=h93005f0_2
- libxml2=2.9.4=0
- libxslt=1.1.29=hc208041_6
- lxml=4.1.1=py36h6c891f4_0
- mkl=2019.0=118
- mkl_fft=1.0.6=py36hb8a8100_0
- mkl_random=1.0.1=py36h5d10147_1
- numpy=1.15.3=py36h6a91979_0
- numpy-base=1.15.3=py36h8a80b8c_0
- pyasn1=0.4.4=py36h28b3542_0
- pyasn1-modules=0.2.2=py36_0
- pycparser=2.19=py36_0
- pyopenssl=18.0.0=py36_0
- service_identity=17.0.0=py36h28b3542_0
- twisted=17.5.0=py36_0
- zope=1.0=py36_1
- zope.interface=4.5.0=py36h1de35cc_0
- pip:
- absl-py==0.2.2
- ete3==3.1.1
- grpcio==1.12.1
Conda Env Update
If you have a YAML file, then the most efficacious way to apply it to a given env is with conda env update:
conda env update --file environment.yml
⚠️ Warning: The conda env commands don't prompt you to review and approve the transactions - it simply executes the changes. Be sure to carefully review the YAML file to ensure all of the changes are desired.
Conda Install
The format that Conda accepts for conda install --file is that which matches the output of conda list --export. It's not a YAML, but a simple text file with one package per line, similar to the one produced by pip freeze, except for the single equality sign ('=' rather than ==).
conda list --export
appdirs=1.4.3=py36h28b3542_0
asn1crypto=0.24.0=py36_0
...
zope=1.0=py36_1
zope.interface=4.5.0=py36h1de35cc_0
Note that the builds are not required, e.g., the following would also work and may actually be slightly more portable across architectures
appdirs=1.4.3
asn1crypto=0.24.0
...
zope=1.0
zope.interface=4.5.0
Unfortunately, conda install doesn't support PyPI packages; you'd have to install those separately via pip install in your activated env.
I am trying to install a Python Environment using a yml file. However I run into the following error :
NoPackagesFoundError: Package missing in current Win64 channels:
- freetype 2.5.5 1
I am using the following code to install the environment using Anaconda 2 (after putting the .yml file in the current directory for Python.)
conda env create -f environment.yml
The yml file is as follows taken from the url ( https://github.com/zooniverse/Data-digging/blob/master/example_scripts/basic_project_stats.yml)
name: python279_volcrowe
channels:
- defaults
dependencies:
- cycler=0.10.0=py27_0
- freetype=2.5.5=1
- jpeg=8d=1
- lcms=1.19=0
- libpng=1.6.22=0
- matplotlib=1.5.1=np110py27_0
- numpy=1.10.1=py27_0
- openssl=1.0.2g=0
- pandas=0.17.1=np110py27_0
- pil=1.1.7=py27_2
- pip=8.1.1=py27_0
- pyparsing=2.1.4=py27_0
- pyqt=4.11.4=py27_3
- python=2.7.11=0
- python-dateutil=2.4.2=py27_0
- pytz=2015.7=py27_0
- qt=4.8.7=3
- readline=6.2=2
- requests=2.9.1=py27_0
- setuptools=20.3=py27_0
- sip=4.16.9=py27_0
- six=1.10.0=py27_0
- sqlite=3.9.2=0
- tk=8.5.18=0
- wheel=0.29.0=py27_0
- zlib=1.2.8=0
prefix: /Users/Public/Anaconda2/envs/python279_volcrowe
I was not sure what the last line in the yml file which is 'prefix' is doing, but I changed it to the directory where the environments in my computer reside.
As mentioned in a similar post I tried installing a win 64 channel using:
conda config --add channels bioninja
I am able to add the bioninja thing but after it when I run the following command:
conda env create -f environment.yml
It gives me the same Win64 channel error as earlier.
Can anyone please help me solve this issue ?
It happened to me when I downloaded the wrong .yml file.
I was working on windows and i run:
conda env create -f aind-environment-unix.yml
I have downloaded the correct file and run:
conda env create -f aind-environment-windows.yml
and i solved the problem