I have a docker file like :
FROM conda/miniconda3-centos7
WORKDIR /tmp
COPY app/ /tmp
RUN conda install gcc_linux-64
RUN conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
The code gets stuck after the first RUN conda command. The error i get is :
WARNING: The conda.compat module is deprecated and will be removed in a future release.
==> WARNING: A newer version of conda exists. <==
current version: 4.6.11
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
Removing intermediate container 277edb28a107
---> e6b51d71eac0
Step 7/8 : RUN conda install gxx_linux-64
---> Running in 94166fbfff2a
Traceback (most recent call last):
File "/usr/local/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
The command '/bin/sh -c conda install gxx_linux-64' returned a non-zero code: 1
Can you please suggest?
Adding conda update -n base -c defaults conda in your Dockerfile solves the mentioned problem.
You could also consider using && for optimizing the creation of docker images. Read more about it here.
An optimized Dockerfile would be:
FROM conda/miniconda3-centos7
WORKDIR /arnav
COPY app/ /arnav
RUN conda update -n base -c defaults conda \
&& conda install gcc_linux-64 && conda install gxx_linux-64
CMD ["python", "Hello_World.py"]
Related
I'm trying to install torchreid which is a library for person re-identification in PyTorch. I've followed the steps mentioned on the git repository but is getting this error.
#conda install
!wget -c https://repo.anaconda.com/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh
!chmod +x Miniconda3-4.5.4-Linux-x86_64.sh
!bash ./Miniconda3-4.5.4-Linux-x86_64.sh -b -f -p /usr/local
!conda install -q -y --prefix /usr/local python=3.6 ujson
import sys
sys.path.append('/usr/local/lib/python3.6/site-packages')
!git clone https://github.com/KaiyangZhou/deep-person-reid.git
!cd deep-person-reid/
!conda create --name torchreid python=3.7
!conda activate torchreid
!pip install -r /content/deep-person-reid/requirements.txt
!conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
import torchreid
error
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-4faf46a39b5d> in <module>()
----> 1 import torchreid
ModuleNotFoundError: No module named 'torchreid'
Run below command,
# install torchreid (don't need to re-build it if you modify the source code)
python setup.py develop
First deactivate you created conda environment 'torchreid'. Activate it again and try to import.
deactivate torchreid
conda activate torchreid
Note: you are installing torchreid in a specifica conda environment, which in not accessible from outside of that environment.
So I'n new-ish to Docker.
What I'm trying to do:
create a docker image (that puts you into an anaconda enviroment and downloads some libraries via conda), within that enviroment run a setup.py install so that the library CPLEX is also in the image
Build the image. Next I run the image, and set a scratch directory that should have all the python programs in it that I want to run.
This is my docker file:
## syntax=docker/dockerfile:1
from continuumio/miniconda3
RUN conda create --name opt python=3.7
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate opt"
RUN conda install -c conda-forge fenics
RUN conda install -c conda-forge scipy
RUN conda install -c conda-forge numpy
RUN conda install -c conda-forge matplotlib
RUN conda install -c conda-forge ipopt
RUN conda install -c conda-forge glpk
RUN conda install -c conda-forge pyomo
RUN conda install -c conda-forge pandas
RUN conda install -c conda-forge scikit-learn
ENTRYPOINT [ "python" ]
COPY CPLEX_Studio129/ /CPLEX_Studio129/
COPY . /codes/
RUN cd /CPLEX_Studio129/ && python cplex/python/3.7/x86-64_osx/setup.py install
WORKDIR /codes/
RUN cd /codes/
Next I build the image, and then run the image :
sudo docker build -t ex:latest .
docker run --rm -it --entrypoint /bin/bash ex
(base) root#bc9beca93d69:/codes# python test_cplex.py
Traceback (most recent call last):
File "/codes/test_cplex.py", line 1, in <module>
import cplex
ModuleNotFoundError: No module named 'cplex'
(base) root#bc9beca93d69:/codes# ls
For what its worth, if I make /CPLEX_Studio129/ the WORKDIR, and install the library, then start a python in that directory, and try to import cplex it works. But my hope was that it would work when I run a program that imports CPLEX in /codes/, which it cannot find.
Any tips, and a bried explation of where I went wrong would be greatly appreciated.
I am trying to create a docker image with miniconda3 intalled. Instead of using directly the base image offered in docker hub, I want to start from scratch by creating my own Dockerfile and putting there the commands of the Dockerfile of the continuumio/miniconda3 image, which are:
FROM debian:latest
# $ docker build . -t continuumio/miniconda3:latest -t continuumio/miniconda3:4.5.11
# $ docker run --rm -it continuumio/miniconda3:latest /bin/bash
# $ docker push continuumio/miniconda3:latest
# $ docker push continuumio/miniconda3:4.5.11
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH /opt/conda/bin:$PATH
RUN apt-get update --fix-missing && \
apt-get install -y wget bzip2 ca-certificates curl git && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda && \
rm ~/miniconda.sh && \
/opt/conda/bin/conda clean -tipsy && \
ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
echo "conda activate base" >> ~/.bashrc
ENV TINI_VERSION v0.16.1
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /usr/bin/tini
RUN chmod +x /usr/bin/tini
ENTRYPOINT [ "/usr/bin/tini", "--" ]
CMD [ "/bin/bash" ]
Building and running the container works just fine. For reference, here is the output of conda info in the container:
(base) root#def48bd1ed5d:/# conda info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /root/.condarc
populated config files :
conda version : 4.5.11
conda-build version : not installed
python version : 3.7.0.final.0
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/4.4.0-187-generic debian/10 glibc/2.28
UID:GID : 0:0
netrc file : None
offline mode : False
The problem appears whenever I try to use conda to install a module, for example conda install jupyter -y. The process starts and at some point during the installation I get this error:
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(base) root#def48bd1ed5d:/#
After this, it seems the installation is corrupeted. If I try to use the conda command to call for exmaple conda info again, I get the same error:
(base) root#def48bd1ed5d:/# conda info
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(base) root#def48bd1ed5d:/#
miniconda3 version specified in installation line inside the Dockerfile is the not the latest one
That Dockerfile that you used to build local image will install miniconda3-4.5.11 not the latest version. You can find it here:
...
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p /opt/conda &&
...
And also in this way with docker:
$ docker build --tag miniconda3:test .
$ docker docker run -i -t miniconda3:test /bin/bash
$ docker history --no-trunc miniconda3:test | grep Miniconda3
/bin/sh -c wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && ...
Ok, now let's look at official continuumio/miniconda3:
$ docker run -i -t continuumio/miniconda3 /bin/bash
and then:
$ docker history --no-trunc continuumio/miniconda3 | grep Miniconda3
/bin/sh -c wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && ...
As you can see continuumio/miniconda3 image from DockerHub installs latest miniconda3 4.8.2 and not the 4.5.11 version. Thus, your local image built from that Dockerfile will produce container with miniconda3:4.5.11.
Change in python version breaks conda
Now, let's figure out why conda fails. First build and run:
$ docker build --tag miniconda3:test .
$ docker docker run -i -t miniconda3:test /bin/bash
Get some info:
(base) root#61cafd17d954:/# conda info
active environment : base
active env location : /opt/conda
shell level : 1
user config file : /root/.condarc
populated config files :
conda version : 4.5.11
conda-build version : not installed
python version : 3.7.0.final.0
base environment : /opt/conda (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /opt/conda/pkgs
/root/.conda/pkgs
envs directories : /opt/conda/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.19.1 CPython/3.7.0 Linux/5.4.0-48-generic debian/10 glibc/2.28
UID:GID : 0:0
netrc file : None
offline mode : False
Well, we have conda:4.5.11 with python:3.7.0.
Now, we are going to install jupyter, for example:
(base) root#61cafd17d954:/# conda install jupyter
You may notice, that this installation will update python:
The following packages will be UPDATED:
...
python: 3.7.0-hc3d631a_0 --> 3.8.5-h7579374_1
...
If you proceed, this will update python and will break conda:
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
File "/opt/conda/bin/conda", line 7, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
This is quite known issue and you can find more info on this issue in How does using conda to install a package change my python version and remove conda? answer, and in official conda repo: No module named conda.cli.main #2463.
Update conda or use Dockerfile for the miniconda3:latest
There are 3 possible solutions to this issue:
Edit your Dockerfile by replacing this line:
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.5.11-Linux-x86_64.sh -O ~/miniconda.sh && \
with
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
Use the latest official Dockerfile from ContinuumIO, you can find it here.
Update conda inside container before usage:
(base) root#61cafd17d954:/# conda update conda
I used conda-pack to move environment to different server but it failed with this error
(jupyterhub_new2) xxxxx#yyyyyy: bin $ ./jupyterhub --version
Traceback (most recent call last):
File "./jupyterhub", line 8, in <module>
from jupyterhub.app import main
ModuleNotFoundError: No module named 'jupyterhub'
I've created conda env on remote server (with internet access). I've installed python3.6 jupterhub and conda-pack.
conda install python=3.6.8
conda install -c conda-forge conda-pack
conda install jupyterhub=1.0.0=py36_0
conda install anaconda-clean=1.1.0=py36_1
conda install -c conda-forge dockerspawner=0.11.1
conda install -c conda-forge jupyterhub-ldapauthenticator=1.2.2=py36_0
Jupyterhub is running so I pack the environment and moved it into test-server (without internet)
conda pack -n jupyterhub_new
I extracted environment and activated it.
sudo mkdir /mnt/data/miniconda3/envs/jupyterhub_new
sudo tar -xzf jupyterhub_new.tar.gz -C /mnt/data/miniconda3/envs/jupyterhub_new
eval "$(conda shell.bash hook)"
conda activate jupyterhub_new
But in the end Jupyterhub in environment can't be even started
(jupyterhub_new2) xxxxx#yyyyyy: bin $ which conda
/mnt/data/miniconda3/bin/conda
(jupyterhub_new2) xxxxx#yyyyyy: bin $ which python
/mnt/data/miniconda3/envs/jupyterhub_new2/bin/python
(jupyterhub_new2) xxxxx#yyyyyy: bin $ conda -V
conda 4.7.12
(jupyterhub_new2) xxxxx#yyyyyy: bin $ jupyterhub -V
Traceback (most recent call last):
File "/mnt/data/miniconda3/envs/jupyterhub_new2/bin/jupyterhub", line 8, in <module>
from jupyterhub.app import main
ModuleNotFoundError: No module named 'jupyterhub'
Conda on the is installed from the same file on the both servers.
test-server is Red Hat Enterprise Linux Server release 7.6 (Maipo), remote server is CentOS Linux 7 (Core)
Any idea how to fixed this ? I've seen many errors in conda with "No module named XXXX" but I didn't find any solution not requiring internet access.
edit 1: yes, jupyter is installed
(jupyterhub_new2) xxxx#yyyy: ~ $ conda list -n jupyterhub_new2 jupyter
# packages in environment at /mnt/data/miniconda3/envs/jupyterhub_new2:
#
# Name Version Build Channel
jupyterhub 1.0.0 py36_0
jupyterhub-ldapauthenticator 1.2.2 py36_0 conda-forge
(jupyterhub_new2) xxxx#yyyyy: ~ $
I am building a docker image that uses conda (conda 4.5.10 & miniconda 4.5.4 are installed on the base image) to create an environment and install some packages. I am having a problem whereby after the conda create this subsequent docker command fails:
RUN source /opt/conda/bin/activate /opt/conda/envs/python2 && \
pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir --upgrade -r /requirements_py2_pip.txt && \
rm /requirements_py2_pip.txt
Error is:
Step 12/17 : RUN source /opt/conda/bin/activate /opt/conda/envs/python2 && pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir --upgrade -r /requirements_py2_pip.txt && rm /requirements_py2_pip.txt
---> Running in e828112b4ae0 .
Could not find platform independent libraries
Could not find platform dependent libraries
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007fe6d4851700 (most recent call first):
The command '/bin/bash -c source /opt/conda/bin/activate /opt/conda/envs/python2 && pip install --no-cache-dir --upgrade pip && pip install --no-cache-dir --upgrade -r /requirements_py2_pip.txt && rm /requirements_py2_pip.txt' returned a non-zero code: 134
The fact that it fails intermittently really bugged me, if it failed every time I wouldn't mind so much but it fails approximately 50% of the time. I googled around and found this SO thread: ImportError: No module named 'encodings' that suggested it may be a problem with the conda environments. Hence, I added this docker command prior to the one that was failing intermittently:
RUN conda env list
Annoyingly that command fails intermittently too. When it succeeds it returns:
Step 12/18 : RUN conda env list
---> Running in 05eaa6f726a9
# conda environments:
#
base * /opt/conda
python3 /opt/conda/envs/python3
python3.6 /opt/conda/envs/python3.6
When it fails it returns:
Step 12/18 : RUN conda env list
---> Running in 20247acc3824
Could not find platform independent libraries
Could not find platform dependent libraries
Consider setting $PYTHONHOME to [:]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007f7a5f5ef700 (most recent call first):
The command '/bin/bash -c conda env list' returned a non-zero code: 139
Which as you will see is very similar to the previous error. This suggested to me that there was a problem with the conda environment which is created using this command:
# Create a Python 2.x environment using conda including at least the ipython kernel
# and the kernda utility. Add any additional packages you want available for use
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 --file requirements_py2_conda.txt && \
source /opt/conda/bin/activate /opt/conda/envs/python2 && \
conda remove --quiet --yes --force qt pyqt && \
conda clean -tipsy && \
rm -rf $CONDA_DIR/envs/python2/share/jupyter/lab/staging && \
rm -rf /usr/local/share/.cache /tmp/* /opt/conda/pkgs/* /opt/conda/envs/python2/pkgs/* \
/requirements_py2_conda.txt /opt/conda/envs/python3/pkgs/*
I compared the output of that command in both a succeeded and a failed build. Both install the same packages, they just happen to install them in a different order.
By the way, requirements_py2_conda.txt contains:
beautifulsoup4=4.6.*
bokeh=0.13.*
bz2file=0.98
cloudpickle=0.5.*
colour=0.1.*
configparser=3.5.*
cython=0.28.*
dill=0.2.*
fastparquet=0.1.*
future=0.16.*
gensim=3.4.*
graphviz=2.40.*
h5py=2.8.*
hdf5=1.10.*
imageio=2.3.*
ipykernel=4.8.*
ipython=5.8.*
ipywidgets=7.4.*
keras=2.2.*
lxml=4.2.*
matplotlib=2.2.*
mysqlclient=1.3.*
mpld3=0.3
nltk=3.3.*
nose=1.3.*
numba=0.39.*
numexpr=2.6.*
numpy=1.15.*
pandas=0.23.*
pathlib2=2.3.*
patsy=0.5.*
pexpect=4.6.*
pivottablejs=0.9.*
protobuf=3.*
pyemd=0.5.*
pymc3=3.5
pyparsing=2.2.*
pystan=2.17.*
pytest=3.7.*
python=2.7.*
pylint
py-xgboost=0.72.*
pyyaml=3.13
requests=2.19.*
scandir=1.*
scikit-image=0.14.*
scikit-learn=0.19.*
scipy=1.1.*
seaborn=0.9.*
sh=1.12.*
simplegeneric=0.8.*
singledispatch=3.4.0.*
six=1.11.*
sortedcontainers=2.0.*
sqlalchemy=1.2.*
SQLAlchemy=1.2.*
statsmodels=0.9.*
subprocess32=3.5.*
sympy=1.2.*
tabulate=0.8.*
tensorflow=1.10.*
texlive-core=20180414
theano=1.0.*
widgetsnbextension=3.4.*
xlrd=1.1.*
The error message suggests
Consider setting $PYTHONHOME to [:]
In order to see if that was the issue I added:
echo $PYTHONHOME
This resulted in the same output (i.e. empty PYTHONHOME) both when the build succeeds and when it fails:
Step 12/20 : RUN echo $PYTHONHOME
---> Running in 59ce6f2776c7
Removing intermediate container 59ce6f2776c7
---> cee1ad9f695e
I'm now stumped. I don't know what to do next in order to investigate and I'm not familiar with conda (I didn't write the Dockerfile, but these intermittent failures are blocking me hence I'm investigating). Any suggestions of things I could do to try and uncover the problem would be welcomed.