Develop Reference

Docker build issue with conda (miniconda)

First, a bit of context : I'm curently trying to work on MineRL, and to put this in a docker. I'm using WSL 2 on my Windows machine to make docker work correctly. There is the github where i found the dockerfile I used : GitHub.
When running the dockerfile with
docker build - < Dockerfile
Everything works perfectly until i reach step [9/18], Which is about conda.
Here is the error i had :
=> ERROR [ 9/18] RUN /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 0.8s
------
> [ 9/18] RUN /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 && /home/user/miniconda/bin/conda clean -ya:
#12 0.739 /bin/sh: 1: /home/user/miniconda/bin/conda: not found
------
executor failed running [/bin/sh -c /home/user/miniconda/bin/conda install conda-build && /home/user/miniconda/bin/conda create -y --name py37 python=3.7.3 && /home/user/miniconda/bin/conda clean -ya]: exit code: 127
I found a solution on this stack overflow post but it couldn't resolve my problems.
Then i am asking anyone who is used to docker, conda and maybe MineRL some help on this please!
EDIT : Thanks to #lote and #DobbyTheElf, I mannaged to modify some part of the dockerfile to correspond to my configuration, but it still cannot find the conda file. Here is my new Dockerfile.
To be sure about what #DobbyTheElf told me, I ran this command to find the path to the folder "miniconda3" and the result is underneath the request :
bloster#DESKTOP-UATBDIM:~/miniconda3/bin$ find / -name miniconda3
/home/bloster/miniconda3
So i edited again my code to what it is now (by the time my last git commit is called "first push").
The error remains the same for now...

CKAN docker compose with custom extensions

I have been trying to customize the default CKAN docker image to include my prefered extensions. The problem is that the precedence of installing pip packages, configuring the ini files and, building and running the main CKAN image is not clear for me.
I have tried adding a layer to the Dockerfile (see below) but extensions are not available after running docker-compose up. The add-extensions.sh file contains pip install commands for each extension.
RUN sh $CKAN_VENV/src/ckan/contrib/docker/add-extensions.sh
I have also tried to include my commands in the docker-compose file itself, inside the ckan service, as follows.
command: >
sh -c "
pip install ckanext-geoview &&
pip install ckanext-datarequests &&
ckan config-tool "/etc/ckan/production.ini" -f "/etc/ckan/custom-config.ini" &&
ckan config-tool "/etc/ckan/production.ini" -s app:main -e ckan.plugins='stats text_view image_view recline_view datastore datapusher resource_proxy geo_view datarequests' &&
ckan -c /etc/ckan/production.ini run --host 0.0.0.0"
But this is not working either. So what is the recommended way of including custom plugins inside the default CKAN image?

Installing Conda on Deepnote

I am a fairly inexperienced programmer and am struggling with installing Conda into Deepnote. Pip install doesn't work for certain packages. For example, I'm trying to install rdkit, a cheminformatics package, which has rather complex installation instructions or a simple 1 line of code managed through the Anaconda/mini-conda distribution. I really like the Deepnote notebooks and I would really appreciate any help here, please.
So far, I have found this useful code for Conda installation on Google Colab: https://github.com/dataprofessor/code/blob/master/python/google_colab_install_conda.ipynb
! wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
! chmod +x Miniconda3-py37_4.8.2-Linux-x86_64.sh
! bash ./Miniconda3-py37_4.8.2-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
Whilst this successfully works on Google Colab, I'm not sure why it fails as shown below on Deepnote:
--2020-12-04 22:58:34-- https://repo.anaconda.com/miniconda/Miniconda3-py37_4.8.2-Linux-x86_64.sh
Resolving repo.anaconda.com (repo.anaconda.com)... 104.16.130.3, 104.16.131.3, 2606:4700::6810:8203, ...
Connecting to repo.anaconda.com (repo.anaconda.com)|104.16.130.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 85055499 (81M) [application/x-sh]
Saving to: ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’
Miniconda3-py37_4.8 100%[===================>] 81.12M 133MB/s in 0.6s
2020-12-04 22:58:35 (133 MB/s) - ‘Miniconda3-py37_4.8.2-Linux-x86_64.sh’ saved [85055499/85055499]
PREFIX=/usr/local
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 392: /usr/local/conda.exe: Permission denied
chmod: cannot access '/usr/local/conda.exe': No such file or directory
Unpacking payload ...
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 404: /usr/local/conda.exe: No such file or directory
./Miniconda3-py37_4.8.2-Linux-x86_64.sh: line 406: /usr/local/conda.exe: No such file or directory
I also want to do conda install -c bioconda gromacs, which I cannot find a work around for, so I am hoping someone can help me resolve this query.
Many thanks in advance!
P.S. I am on a Mac OS

You can check this notebook from Daniel Zvara
Using Conda in Deepnote in 3 simple steps
In sum
# 1. Install Conda and make Conda packages available in current environment
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!sudo bash ./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
import sys
sys.path.append('/usr/local/lib/python3.7/site-packages/')
# 2. Package installation
# !sudo conda install -y [EXTRA_OPTIONS] package_name
# So for example Keras from conda-forge channel
!sudo conda install -y -c conda-forge keras
# 3. Package usage
import keras

Also found an alternative solution on the Deepnote community: https://community.deepnote.com/c/custom-environments/custom-environment-for-installing-conda-packages
Can set up a custom environment with the following Dockerfile (just fill in the placeholder for the packages you require).
FROM gcr.io/deepnote-200602/templates/deepnote
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
RUN bash ~/miniconda.sh -b -p $HOME/miniconda
ENV PATH $HOME/miniconda/bin:$PATH
RUN conda install python=3.7 ipykernel -y
RUN conda install <insert packages here> -y
RUN python -m ipykernel install --user --name=conda
ENV DEFAULT_KERNEL_NAME "conda"
And the custom environment in Deepnote is set up through the Environment tab in the sidebar, giving you an option for the Dockerfile. Copy & paste the above, then click build and restart machine - the environment will be set up allowing you to use conda.

You can now choose Anaconda in the built-in environments in Deepnote.

How to run pytest in jenkins

I am trying to run pytest in jenkins.
When i try to install pytest in build option in jenkins it says pip command not found. Even tried setting a virtual env but no success.
I AM RUNNING JENKINS IN DOCKER CONTAINER
#!/bin/bash
cd /usr/bin
pip install pytest
py.test test_11.py
#!/bin/bash
source env1/bin/activate
pip install pytest
py.test test_11.py
Dockerfile
FROM Jenkins
USER root
Errors:
Started by user admin
Running as SYSTEM
Building on master in workspace /var/jenkins_home/workspace/pyproject
[pyproject] $ /bin/bash /tmp/jenkins5312265766264018610.sh
/tmp/jenkins5312265766264018610.sh: line 4: pip: command not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
Started by user admin
Running as SYSTEM
Building on master in workspace /var/jenkins_home/workspace/pyproject
[pyproject] $ /bin/bash /tmp/jenkins6002566555689593419.sh
/tmp/jenkins6002566555689593419.sh: line 4: pip: command not found
/tmp/jenkins6002566555689593419.sh: line 5: py.test: command not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE

well, the error is daylight clear, pip is not installed in the running environment.
I did some digging myself and found out that jenkins image has only python2.7 installed, and pip is not installed.
I would start by installing pip first and continue from there, so modify Dockerfile to:
FROM jenkins
USER root
RUN apt-get update && apt-get install -y python-pip && rm -rf /var/lib/apt/lists/*
hope this helps you find your way.
more helpful information could be:
your jenkins pipeline script (at least until step 'Execute shell')
python version you intend to use.
how and where you run the virtual-env creation command.

Docker and Conda: Differences when building the same container on Mac and on Ubuntu

I'm using to Docker to build a Python container with the intention of having a reproducible environment on several machines, which are a bunch of development Macbooks and several AWS EC2 servers.
The container is based on continuumio/miniconda3, i.e. Dockerfile starts with
FROM continuumio/miniconda3
A few days ago on Ubuntu the conda install and conda upgrade commands in the Docker file complained about a new Conda version (4.11) being available:
==> WARNING: A newer version of conda exists. <==
current version: 4.4.10
latest version: 4.4.11
Please update conda by running
$ conda update -n base conda
If I ignore this, package installations quit with an error:
Downloading and Extracting Packages
The command '/bin/sh -c conda install -y pandas=0.22.0 matplotlib
scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2
sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras
xgboost' returned a non-zero code: 1
When I add this conda update... to the Docker file, things work again.
What's really annoying, however, is that the update that makes things run in Ubuntu does not work on Mac Docker. I get the following error:
CondaEnvironmentNotFoundError: Could not find environment: base .
You can list all discoverable environments with `conda info --envs`.
Note that I get this error when I docker build the same Docker file that works on the Ubuntu machine, which kind of ruins the whole point about using Docker in the first place. On the Mac, the old version of the file (without conda update -n base conda) still runs fine and installs all packages.
Docker / Conda experts, any ideas?
Edit: Here's the full Dockerfile (the one that works in Ubuntu):
# Use an official Python runtime as a parent image
FROM continuumio/miniconda3
WORKDIR /app/dev/predictive.analytics
RUN apt-get update; \
apt-get install -y gcc tmux htop
RUN conda update -y -n base conda
RUN conda config --add channels babbel; \
conda config --add channels conda-forge;
RUN conda install -y pandas=0.22.0 matplotlib scikit-learn=0.19.1 pathos lazy openpyxl pytables dill pydro psycopg2 sqlalchemy pyarrow arrow-cpp parquet-cpp scipy tensorflow keras xgboost
RUN pip install recordclass sultan
RUN conda upgrade -y python
ENV DATA_DIR /host/data
ENV PYTHONPATH /host/predictive.analytics/python
ENV PATH="/host/predictive.analytics:${PATH}"

Perhaps you're using an outdated miniconda on one of the build machine, try doing docker build --pull --no-cache.
Docker doesn't necessarily pull the latest image from the repository, so unless you do a --pull, it is possible that some of your machines may be starting the build with outdated base image.