I build dockerimages for a jupyterlab with some apt and pip packages.
docker image ls:
jupyterlab 3.7.12 9d4cc5c15853 27 minutes ago 2.3GB
jupyterlab 3.9.9 1962a1eb6bff 25 hours ago 1.12GB
The filesize of the images differs quite much, I only took a another python version.
Here are the dockerfiles for comparison:
This dockerfile is build with python 3.7.12 as baseimage:
FROM python:3.7.12-slim-buster
WORKDIR /
COPY wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl ./wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir numpy && \
pip install --no-cache-dir /wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl && \
pip install --no-cache-dir jupyterlab
# pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
This is the one with python 3.9.9 as baseimage:
FROM python:3.9.9-slim-buster
WORKDIR /
COPY wheels ./wheels
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
The wheels folder and some other files are deleted in both cases. Why is there such a big difference in filesize???
I solved the issue by removing "apt-get python3-dev", it was version 3.7.2. My baseimage is python 3.7.12 so maybe there was a conflict, I can not explain it.
The size of the image:
#docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterlab 3.7.12_nodev 6112bcaff54d 17 hours ago 1.18GB
jupyterlab 3.9.9 1962a1eb6bff 2 days ago 1.12GB
Related
I am trying to install python3 and pip.
So, I have a dockerfile
(I also tried with ubuntu:20.04)-
(Note- I have made the Dockerfile short, for full dockerfile you can see here)
FROM ubuntu:latest
WORKDIR /app
COPY . /app
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository universe
ENV PYTHONUNBUFFERED=1
RUN apt install -y python3
RUN apt-get -y install python3-pip
RUN pip install --no-cache --upgrade pip setuptools
But it always give me error-
---> Running in ba9f299b852b
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package python3-pip
The command '/bin/sh -c apt-get -y install python3-pip' returned a non-zero code: 100
While finding more out in the error, I found out that while the universal repositories were added, python3-pip didnt got installed, while other things like python3-sip got installed.
How do I fix this error and why did it happened?
The software-properties-common meta-packages installs a bunch of softwares that are not required in a docker image and make its the size quite huge.
This simpler Dockerfile does the job for me.
FROM ubuntu:latest
WORKDIR /app
#COPY . /app
RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
python3 \
python3-pip \
&& apt-get clean all
RUN pip install --no-cache --upgrade pip setuptools
RUN pip --version # just for test
Output:
STEP 1/5: FROM ubuntu:latest
STEP 2/5: WORKDIR /app
--> Using cache 9f1d34a70ba9e13074601d16456fda97b2de87a0be3b18ddc6963fb65aaee096
--> 9f1d34a70ba
STEP 3/5: RUN apt-get update && apt-get install --assume-yes --no-install-recommends --quiet python3 python3-pip && apt-get clean all
[...]
STEP 4/5: RUN pip install --no-cache --upgrade pip setuptools
Collecting pip
Downloading pip-21.3-py3-none-any.whl (1.7 MB)
Collecting setuptools
Downloading setuptools-58.2.0-py3-none-any.whl (946 kB)
Installing collected packages: pip, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Attempting uninstall: setuptools
Found existing installation: setuptools 45.2.0
Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed pip-21.3 setuptools-58.2.0
STEP 5/5: RUN pip --version
pip 21.3 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
COMMIT staskoverflow-ubuntu
--> 2498ddc1767
Successfully tagged localhost/staskoverflow-ubuntu:latest
2498ddc17672584d2dd38bf23ae63521d9adb43465cf8c7b415c33dc2aa0aa01
The tool pip is updated to the version 21.3 and the OS-provided one is not uninstalled (this is a pip feature).
Anyway this dockefile, more similar to yours, works too:
RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
software-properties-common \
&& add-apt-repository universe \
&& apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
python3 \
python3-pip \
&& apt-get clean all
RUN pip install --no-cache --upgrade pip setuptools
After adding the universe repository, you should run again apt-get update to retrieve the new lists of packages with the new apt configuration. This is the root cause of the problem I think.
I am building an image using the following Dockerfile:
FROM ubuntu:18.04
RUN apt-get update \
&& apt-get install -y python3-pip python3-dev \
&& cd /usr/local/bin \
&& ln -s /usr/bin/python3 python \
&& pip3 install --upgrade pip
# Setup the Python's configs
RUN pip install --upgrade pip && \
pip install --no-cache-dir matplotlib==3.0.2 pandas==0.23.4 numpy==1.16.3 && \
pip install --no-cache-dir pybase64 && \
pip install --no-cache-dir scipy && \
pip install --no-cache-dir dask[complete] && \
pip install --no-cache-dir dash==1.6.1 dash-core-components==1.5.1 dash-bootstrap-components==0.7.1 dash-html-components==1.0.2 dash-table==4.5.1 dash-daq==0.2.2 && \
pip install --no-cache-dir plotly && \
pip install --no-cache-dir adjustText && \
pip install --no-cache-dir networkx && \
pip install --no-cache-dir scikit-learn && \
pip install --no-cache-dir tzlocal
# Setup the R configs
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
RUN add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/'
RUN apt update
ENV DEBIAN_FRONTEND=noninteractive
RUN apt install -y r-base
RUN pip install rpy2==2.9.4
RUN apt-get -y install libxml2 libxml2-dev libcurl4-gnutls-dev libssl-dev
RUN echo "r <- getOption('repos'); r['CRAN'] <- 'https://cran.r-project.org'; options(repos = r);" > ~/.Rprofile
RUN Rscript -e "install.packages('BiocManager')"
RUN Rscript -e "BiocManager::install('ggplot2')"
RUN Rscript -e "BiocManager::install('DESeq2')"
RUN Rscript -e "BiocManager::install('RColorBrewer')"
RUN Rscript -e "BiocManager::install('ggrepel')"
RUN Rscript -e "BiocManager::install('factoextra')"
RUN Rscript -e "BiocManager::install('FactoMineR')"
RUN Rscript -e "BiocManager::install('apeglm')"
When I build this on Linux I launched the web-app from the container and it's run fine.
But, when I build this on windows using Docker Toolbox although the installation of factoextra and FactoMineR work successfully, when I launch the web-app it's raise an error:
Error in library("factoextra") : there is no package called ‘factoextra’
Do you have any idea what's might cause this problem? It's very strange because when I build the image the installation of these 2 packages seems to work successfully.
I'm new to Docker and currently trying to create a Dockerfile with installing the python packages and its libraries as shown here:
FROM balenalib/fincm3-debian-python:latest
# RUN install_packages git
RUN apt-get update && apt-get install python \
&& apt-get install pip3 \
apt-get install libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev \
pip3 install pyaudio \
pip3 install numpy \
pip3 install matplotlib \
pip3 install scipy \
pip3 install librosa \
# Set our working directory
WORKDIR /usr/src/app
COPY Recorder.py /usr/src/app
# Recorder.py will run when container starts up on the device
CMD ["python","/usr/src/app/Recorder.py"]
However, while I am trying to push this Dockerfile, the error is generated with
Error: The command '/bin/sh -c apt-get update && apt-get install python && apt-get install pip3 apt-get install libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev pip3 install pyaudio
pip3 install numpy pip3 install matplotlib pip3 install scipy pip3 install librosa WORKDIR /usr/src/app' returned a non-zero code: 100
Moving python packages in requirements.txt and installing python3-pip worked with python:3 base image.
# RUN install_packages git
RUN apt-get update \
&& apt-get install -y python \
&& apt-get install -y python3-pip
RUN pip install -r requirements.txt
The package you are looking for is called python3-pip.
Next, you need both && (to separate commands) and \ (to continue the command line). So, in summary, that should be:
FROM balenalib/fincm3-debian-python:latest
RUN apt-get update && apt-get install python && \
apt-get install -y \
python3-pip libportaudio0 libportaudio2 libportaudiocpp0 \
portaudio19-dev && \
pip3 install pyaudio numpy matplotlib \
scipy librosa
# Set our working directory
WORKDIR /usr/src/app
COPY Recorder.py /usr/src/app
# Recorder.py will run when container starts up on the device
CMD ["python","/usr/src/app/Recorder.py"]
I believe you have more than one problem in this Dockerfile, and when you put all commands together with && and \, you don't know which one is triggering the error. I suggest splitting them for debugging purposes, when they all work then you can put then together. Once you understand each individual error is easier to check and solve them. this question has valuable info: how to install pip in docker
Try this:
1- packages are triggers Y/n questions, give -y to guarantee it passes
2- using the backslashes to refer to a new command, you should use &&, backslashes refer to breaking line, you can use \ and then &&
3- pip3 and libportaudio0 packages doesn't exist.
E: Unable to locate package libportaudio0
I found out about the errors dividing the Dockerfile like this and removing the problems mentioned:
RUN apt-get update
RUN apt-get install python -y\
&& apt-get install python3-pip -y
RUN apt-get install libportaudio2 libportaudiocpp0 portaudio19-dev -y
RUN pip3 install pyaudio numpy matplotlib \
scipy librosa
If you want to put the commands together:
RUN apt-get update \
&& apt-get install python -y \
&& apt-get install python3-pip -y \
&& apt-get install libportaudio2 libportaudiocpp0 portaudio19-dev -y \
&& pip3 install pyaudio numpy matplotlib \
scipy librosa
I also suggest adding a pip requirements file, would make things cleaner.
I'm trying to build an image using DockerFile in AmazonSageMaker but I'm getting the following error.
Traceback (most recent call last):
File "/usr/bin/pip3", line 9, in
from pip import main ImportError: cannot import name 'main' The command '/bin/sh -c pip3 install --upgrade pip setuptools wheel &&
pip3 install mxnet-cu90 --upgrade --pre && pip3 install
keras-mxnet --upgrade --pre' returned a non-zero code: 1
My DockerFile is as below
FROM nvidia/cuda:9.0-runtime
RUN apt-get update && \
apt-get -y install build-essential libopencv-dev libopenblas-dev libjemalloc-dev libgfortran3 \
python-dev python3-dev python3-pip wget curl
COPY train_siamese_network.py /opt/program/train RUN chmod +x /opt/program/train
RUN mkdir /root/.keras COPY keras.json /root/.keras/
RUN pip3 install --upgrade pip setuptools wheel && \
pip3 install mxnet-cu90 --upgrade --pre && \
pip3 install keras-mxnet --upgrade --pre
RUN rm -rf /var/lib/apt/lists/* RUN rm -rf /root/.cache
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"
ENV PATH="/opt/program:${PATH}"
WORKDIR /opt/program
you can run a container use image nvidia/cuda:9.0-runtime, try pip install command in the container and see what the command return
I am trying to install scipy from a Dockerfile and I cannot for the life of me figure out how.
Here is the Dockerfile:
FROM python:3.5
ENV HOME /root
# Install dependencies
RUN apt-get update
RUN apt-get install -y gcc
RUN apt-get install -y build-essential
RUN apt-get install -y zlib1g-dev
RUN apt-get install -y wget
RUN apt-get install -y unzip
RUN apt-get install -y cmake
RUN apt-get install -y python3-dev
RUN apt-get install -y gfortran
RUN apt-get install -y python-numpy
RUN apt-get install -y python-matplotlib
RUN apt-get install -y ipython
RUN apt-get install -y ipython-notebook
RUN apt-get install -y python-pandas
RUN apt-get install -y python-sympy
RUN apt-get install -y python-nose
# Install Python packages
RUN pip install --upgrade pip
RUN pip install cython
# Install scipy
RUN apt-get install -y python-scipy
This builds an image, but when I run the container and try to import scipy it says:
Python 3.5.1 (default, Mar 9 2016, 03:30:07)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'scipy'
I have tried using RUN pip install scipy and RUN pip install git+https://github.com/scipy/scipy.git but those throw an error before completing the build.
You're using Python 3 but installing the Python 2 packages. Change your Dockerfile to the following:
FROM python:3.5
ENV HOME /root
ENV PYTHONPATH "/usr/lib/python3/dist-packages:/usr/local/lib/python3.5/site-packages"
# Install dependencies
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get autoremove -y \
&& apt-get install -y \
gcc \
build-essential \
zlib1g-dev \
wget \
unzip \
cmake \
python3-dev \
gfortran \
libblas-dev \
liblapack-dev \
libatlas-base-dev \
&& apt-get clean
# Install Python packages
RUN pip install --upgrade pip \
&& pip install \
ipython[all] \
numpy \
nose \
matplotlib \
pandas \
scipy \
sympy \
cython \
&& rm -fr /root/.cache