First, a bit of context : I'm curently trying to work on MineRL, and to put this in a docker. I'm using WSL 2 on my Windows machine to make docker work correctly. There is the github where i found the dockerfile I used : GitHub.
When running the dockerfile with
docker build - < Dockerfile
Everything works perfectly until i reach step [9/18], Which is about conda.
Here is the error i had :
=> ERROR [ 9/18] RUN /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 0.8s
------
> [ 9/18] RUN /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 && /home/user/miniconda/bin/conda clean -ya:
#12 0.739 /bin/sh: 1: /home/user/miniconda/bin/conda: not found
------
executor failed running [/bin/sh -c /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 && /home/user/miniconda/bin/conda clean -ya]: exit code: 127
I found a solution on this stack overflow post but it couldn't resolve my problems.
Then i am asking anyone who is used to docker, conda and maybe MineRL some help on this please!
EDIT : Thanks to #lote and #DobbyTheElf, I mannaged to modify some part of the dockerfile to correspond to my configuration, but it still cannot find the conda file. Here is my new Dockerfile.
To be sure about what #DobbyTheElf told me, I ran this command to find the path to the folder "miniconda3" and the result is underneath the request :
bloster#DESKTOP-UATBDIM:~/miniconda3/bin$ find / -name miniconda3
/home/bloster/miniconda3
So i edited again my code to what it is now (by the time my last git commit is called "first push").
The error remains the same for now...
Related
I have a docker file like :
FROM conda/miniconda3-centos7
WORKDIR /tmp
COPY app/ /tmp
RUN conda install gcc_linux-64
RUN conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
The code gets stuck after the first RUN conda command. The error i get is :
WARNING: The conda.compat module is deprecated and will be removed in a future release.
==> WARNING: A newer version of conda exists. <==
current version: 4.6.11
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
Removing intermediate container 277edb28a107
---> e6b51d71eac0
Step 7/8 : RUN conda install gxx_linux-64
---> Running in 94166fbfff2a
Traceback (most recent call last):
File "/usr/local/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
The command '/bin/sh -c conda install gxx_linux-64' returned a non-zero code: 1
Can you please suggest?
Adding conda update -n base -c defaults conda in your Dockerfile solves the mentioned problem.
You could also consider using && for optimizing the creation of docker images. Read more about it here.
An optimized Dockerfile would be:
FROM conda/miniconda3-centos7
WORKDIR /arnav
COPY app/ /arnav
RUN conda update -n base -c defaults conda \
&& conda install gcc_linux-64 && conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
I am a fairly inexperienced programmer and am struggling with installing Conda into Deepnote. Pip install doesn't work for certain packages. For example, I'm trying to install rdkit, a cheminformatics package, which has rather complex installation instructions or a simple 1 line of code managed through the Anaconda/mini-conda distribution. I really like the Deepnote notebooks and I would really appreciate any help here, please.
So far, I have found this useful code for Conda installation on Google Colab: https://github.com/dataprofessor/code/blob/master/python/google_colab_install_conda.ipynb
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
Whilst this successfully works on Google Colab, I'm not sure why it fails as shown below on Deepnote:
--2020-12-04 22:58:34-- https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.130.3, 104.16.131.3, 2606:4700::6810:8203, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.130.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85055499 (81M) [application/x-sh]
Saving to: ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’
Miniconda3-py37_4.8 100%[===================>] 81.12M 133MB/s in 0.6s
2020-12-04 22:58:35 (133 MB/s) - ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’ saved [85055499/85055499]
PREFIX=/usr/local
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 392: /usr/local/conda.exe: Permission denied
chmod: cannot access '/usr/local/conda.exe': No such file or directory
Unpacking payload ...
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 404: /usr/local/conda.exe: No such file or directory
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 406: /usr/local/conda.exe: No such file or directory
I also want to do conda install -c bioconda gromacs, which I cannot find a work around for, so I am hoping someone can help me resolve this query.
Many thanks in advance!
P.S. I am on a Mac OS
You can check this notebook from Daniel Zvara
Using Conda in Deepnote in 3 simple steps
In sum
# 1. Install Conda and make Conda packages available in current environment
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!sudo bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
# 2. Package installation
# !sudo conda install -y [EXTRA_OPTIONS] package_name
# So for example Keras from conda-forge channel
!sudo conda install -y -c conda-forge keras
# 3. Package usage
import keras
Also found an alternative solution on the Deepnote community: https://community.deepnote.com/c/custom-environments/custom-environment-for-installing-conda-packages
Can set up a custom environment with the following Dockerfile (just fill in the placeholder for the packages you require).
FROM gcr.io/deepnote-200602/templates/deepnote
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
RUN bash ~/miniconda.sh -b -p $HOME/miniconda
ENV PATH $HOME/miniconda/bin:$PATH
RUN conda install python=3.7 ipykernel -y
RUN conda install <insert packages here> -y
RUN python -m ipykernel install --user --name=conda
ENV DEFAULT_KERNEL_NAME "conda"
And the custom environment in Deepnote is set up through the Environment tab in the sidebar, giving you an option for the Dockerfile. Copy & paste the above, then click build and restart machine - the environment will be set up allowing you to use conda.
You can now choose Anaconda in the built-in environments in Deepnote.
I am trying to create a docker image with miniconda3 intalled. Instead of using directly the base image offered in docker hub, I want to start from scratch by creating my own Dockerfile and putting there the commands of the Dockerfile of the continuumio/miniconda3 image, which are:
FROM debian:latest
# $ docker build . -t continuumio/miniconda3:latest -t continuumio/miniconda3:4.5.11
# $ docker run --rm -it continuumio/miniconda3:latest /bin/bash
# $ docker push continuumio/miniconda3:latest
# $ docker push continuumio/miniconda3:4.5.11
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH /opt/conda/bin:$PATH
RUN apt-get update --fix-missing && \
apt-get install -y wget bzip2 ca-certificates curl git && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh && \
/opt/conda/bin/conda clean -tipsy && \
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "conda activate base" >> ~/.bashrc
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT [ "/usr/bin/tini", "--" ]
CMD [ "/bin/bash" ]
Building and running the container works just fine. For reference, here is the output of conda info in the container:
(base) root#def48bd1ed5d:/# conda info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /root/.condarc
populated config files :
conda version : 4.5.11
conda-build version : not installed
python version : 3.7.0.final.0
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/4.4.0-187-generic debian/10 glibc/2.28
UID:GID : 0:0
netrc file : None
offline mode : False
The problem appears whenever I try to use conda to install a module, for example conda install jupyter -y. The process starts and at some point during the installation I get this error:
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(base) root#def48bd1ed5d:/#
After this, it seems the installation is corrupeted. If I try to use the conda command to call for exmaple conda info again, I get the same error:
(base) root#def48bd1ed5d:/# conda info
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(base) root#def48bd1ed5d:/#
miniconda3 version specified in installation line inside the Dockerfile is the not the latest one
That Dockerfile that you used to build local image will install miniconda3-4.5.11 not the latest version. You can find it here:
...
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda &&
...
And also in this way with docker:
$ docker build --tag miniconda3:test .
$ docker docker run -i -t miniconda3:test /bin/bash
$ docker history --no-trunc miniconda3:test | grep Miniconda3
/bin/sh -c wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && ...
Ok, now let's look at official continuumio/miniconda3:
$ docker run -i -t continuumio/miniconda3 /bin/bash
and then:
$ docker history --no-trunc continuumio/miniconda3 | grep Miniconda3
/bin/sh -c wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && ...
As you can see continuumio/miniconda3 image from DockerHub installs latest miniconda3 4.8.2 and not the 4.5.11 version. Thus, your local image built from that Dockerfile will produce container with miniconda3:4.5.11.
Change in python version breaks conda
Now, let's figure out why conda fails. First build and run:
$ docker build --tag miniconda3:test .
$ docker docker run -i -t miniconda3:test /bin/bash
Get some info:
(base) root#61cafd17d954:/# conda info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /root/.condarc
populated config files :
conda version : 4.5.11
conda-build version : not installed
python version : 3.7.0.final.0
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/5.4.0-48-generic debian/10 glibc/2.28
UID:GID : 0:0
netrc file : None
offline mode : False
Well, we have conda:4.5.11 with python:3.7.0.
Now, we are going to install jupyter, for example:
(base) root#61cafd17d954:/# conda install jupyter
You may notice, that this installation will update python:
The following packages will be UPDATED:
...
python: 3.7.0-hc3d631a_0 --> 3.8.5-h7579374_1
...
If you proceed, this will update python and will break conda:
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
This is quite known issue and you can find more info on this issue in How does using conda to install a package change my python version and remove conda? answer, and in official conda repo: No module named conda.cli.main #2463.
Update conda or use Dockerfile for the miniconda3:latest
There are 3 possible solutions to this issue:
Edit your Dockerfile by replacing this line:
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
with
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
Use the latest official Dockerfile from ContinuumIO, you can find it here.
Update conda inside container before usage:
(base) root#61cafd17d954:/# conda update conda
Problem
I am trying to install python packages that have C dependencies on AWS Elastic Beanstalk (namely : fbprophet and xgboost)
Elastic Beanstalk python installs packages from requirements.txt by default with pip or pipenv on Amazon Linux 2
However, fbprophet and xgboost have dependencies in C that need to be compiled before installing them with pip. conda comes with these libraries precompiled so they are a lot easier to install with conda.
What I have tried
Here is my attempt at installing them with conda using a .config file in .ebextensions folder :
commands:
00_download_conda:
command: 'wget http://repo.continuum.io/archive/Anaconda3-2020.02-Linux-x86_64.sh'
test: test ! -d /anaconda
01_install_conda:
command: 'bash Anaconda3-2020.02-Linux-x86_64.sh -b -f -p /anaconda'
test: test ! -d /anaconda
02_reload_bash:
command: 'source ~/.bashrc'
03_create_home:
command: 'mkdir -p /home/wsgi'
04_conda_env:
command: 'conda env create -f environment.yml'
05_activate_env:
command: 'conda activate demo_forecast'
However this does not work and throws this error :
[2020-04-21T18:18:22.285Z] INFO [3699] - [Application update app-8acc-200421_201612#4/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_0_test_empty_dash/Command 03_conda_env] : Activity execution failed, because: /bin/sh: conda: command not found
(ElasticBeanstalk::ExternalInvocationError)
[2020-04-21T18:18:22.285Z] INFO [3699] - [Application update app-8acc-200421_201612#4/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_0_test_empty_dash/Command 03_conda_env] : Activity failed.
So it seems that sourcing the .bashrc does not create the conda alias
I am aware of this question and its answer , however it is a little bit old and does not provide enough details for my case, because it does not go through with installing packages using conda.
Another way would be to try and install and compile the C dependencies before pip installing the requirement, but I had no success there for now.
Thank you for the help !
I'm using to Docker to build a Python container with the intention of having a reproducible environment on several machines, which are a bunch of development Macbooks and several AWS EC2 servers.
The container is based on continuumio/miniconda3, i.e. Dockerfile starts with
FROM continuumio/miniconda3
A few days ago on Ubuntu the conda install and conda upgrade commands in the Docker file complained about a new Conda version (4.11) being available:
==> WARNING: A newer version of conda exists. <==
current version: 4.4.10
latest version: 4.4.11
Please update conda by running
$ conda update -n base conda
If I ignore this, package installations quit with an error:
Downloading and Extracting Packages
The command '/bin/sh -c conda install -y pandas=0.22.0 matplotlib
scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2
sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras
xgboost' returned a non-zero code: 1
When I add this conda update... to the Docker file, things work again.
What's really annoying, however, is that the update that makes things run in Ubuntu does not work on Mac Docker. I get the following error:
CondaEnvironmentNotFoundError: Could not find environment: base .
You can list all discoverable environments with `conda info --envs`.
Note that I get this error when I docker build the same Docker file that works on the Ubuntu machine, which kind of ruins the whole point about using Docker in the first place. On the Mac, the old version of the file (without conda update -n base conda) still runs fine and installs all packages.
Docker / Conda experts, any ideas?
Edit: Here's the full Dockerfile (the one that works in Ubuntu):
# Use an official Python runtime as a parent image
FROM continuumio/miniconda3
WORKDIR /app/dev/predictive.analytics
RUN apt-get update; \
apt-get install -y gcc tmux htop
RUN conda update -y -n base conda
RUN conda config --add channels babbel; \
conda config --add channels conda-forge;
RUN conda install -y pandas=0.22.0 matplotlib scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2 sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras xgboost
RUN pip install recordclass sultan
RUN conda upgrade -y python
ENV DATA_DIR /host/data
ENV PYTHONPATH /host/predictive.analytics/python
ENV PATH="/host/predictive.analytics:${PATH}"
Perhaps you're using an outdated miniconda on one of the build machine, try doing docker build --pull --no-cache.
Docker doesn't necessarily pull the latest image from the repository, so unless you do a --pull, it is possible that some of your machines may be starting the build with outdated base image.