Develop Reference

Filesize of dockerimage doubles with different python version as baseimage

I build dockerimages for a jupyterlab with some apt and pip packages.
docker image ls:
jupyterlab 3.7.12 9d4cc5c15853 27 minutes ago 2.3GB
jupyterlab 3.9.9 1962a1eb6bff 25 hours ago 1.12GB
The filesize of the images differs quite much, I only took a another python version.
Here are the dockerfiles for comparison:
This dockerfile is build with python 3.7.12 as baseimage:
FROM python:3.7.12-slim-buster
WORKDIR /
COPY wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl ./wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir numpy && \
pip install --no-cache-dir /wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl && \
pip install --no-cache-dir jupyterlab
# pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
This is the one with python 3.9.9 as baseimage:
FROM python:3.9.9-slim-buster
WORKDIR /
COPY wheels ./wheels
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
The wheels folder and some other files are deleted in both cases. Why is there such a big difference in filesize???

I solved the issue by removing "apt-get python3-dev", it was version 3.7.2. My baseimage is python 3.7.12 so maybe there was a conflict, I can not explain it.
The size of the image:
#docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterlab 3.7.12_nodev 6112bcaff54d 17 hours ago 1.18GB
jupyterlab 3.9.9 1962a1eb6bff 2 days ago 1.12GB

Unable to install PIP & PYMSSQL in docker

I am trying to build a docker image with
Python
PIP
pymssql
below is my Dockerfile
FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get update && apt-get install -y telnet && apt-get install -y ksh && apt-get install -y python2.7.x && apt-get update && apt-get install -y python-pip python-dev build-essential && apt-get -y clean && rm -rf /var/lib/apt/lists/*
RUN pip install pymssql==2.1.3
ENTRYPOINT ['python']
it is throwing the following error:

If you don't mind to use python3 (otherwise use virtual env) I rate you to do :
RUN apt-get install python3-pip
Then you will be able to install whatever pip package you need : by doing :
RUN pip3 install <your_pip_pkg>

Python-pip is only available on Ubuntu Bionic. see python-pip.
You need to switch from focal to bionic. Universe repository should be enabled.

ModuleNotFoundError: No module named 'turbodbc' on running the docker image

I'm using Cloudera Hive ODBC driver in my code and I'm trying to containerize the app.
Below is my Dockerfile,
FROM ubuntu:18.04
FROM continuumio/anaconda3
FROM node:10
RUN conda update -n base -c defaults conda
RUN conda create -n env python=3.7
RUN echo "conda activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y \
python-pip python-dev python-setuptools \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
RUN pip install pyyaml pandas numpy pymysql sqlalchemy schedule tornado
RUN apt-get update && apt-get install -y --no-install-recommends git unzip unixodbc unixodbc-dev
RUN conda install -c conda-forge turbodbc=3.1.1
RUN apt-get update && apt-get install -y gettext nano vim -y
RUN yarn install --modules-folder ./static
WORKDIR /app
COPY entry.sh /usr/local/bin/
COPY . /app/
ENV SSH_PASSWD "root:Docker!"
RUN apt-get update \
&& apt-get install -y --no-install-recommends dialog \
&& apt-get update \
&& apt-get install -y --no-install-recommends openssh-server \
&& echo "$SSH_PASSWD" | chpasswd
COPY sshd_config /etc/ssh/
COPY entry.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/entry.sh
EXPOSE 5000 2222 22 80 8000
CMD ["entry.sh"]
Building Image is getting successful, but I see when I run the docker image, I see below error
Traceback (most recent call last):
File "app.py", line 14, in <module>
from abc_scheduler import scheduler_main
File "/app/abc_scheduler.py", line 5, in <module>
from methods import Methods
File "/app/methods.py", line 10, in <module>
from utils import *
File "/app/utils.py", line 2, in <module>
from turbodbc import connect, make_options
ModuleNotFoundError: No module named 'turbodbc'
I have tried many other ODBC's inside my Dockerfile, but no luck. Any help would be great.

As suggested by #DavidMaze, I managed create a successful Dockerfile & is shown below
FROM ubuntu:latest
FROM continuumio/anaconda3
FROM node:10
RUN conda update -n base -c defaults conda
RUN conda create -n env python=3.7
RUN echo 'conda init bash' >/.bashrc
RUN echo "conda activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
RUN apt-get update && apt-get install -y \
curl apt-utils apt-transport-https debconf-utils gcc build-essential \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y \
python-pip python-dev python-setuptools \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
# ==================TURBODBC========================
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get dist-upgrade -y
RUN apt-get install -y alien # optional
COPY ClouderaHiveODBC-2.6.1.1001-1.x86_64.rpm /opt/cloudera/
RUN alien /opt/cloudera/ClouderaHiveODBC-2.6.1.1001-1.x86_64.rpm
RUN dpkg -i clouderahiveodbc_2.6.1.1001-2_amd64.deb
# ==================END=============================
RUN conda install --name env -c conda-forge turbodbc=4.1.1 tornado=6.0.4 pyyaml pymysql schedule sqlalchemy pyarrow numpy=1.19.3\
pandas=1.1.4 pybind11 pyarrow
COPY odbc.ini /etc/
RUN apt-get update && apt-get install -y gettext nano vim -y
RUN yarn install --modules-folder ./static
WORKDIR /app
COPY . /app/
ENV SSH_PASSWD "root:Docker!"
RUN apt-get update \
&& apt-get install -y --no-install-recommends dialog \
&& apt-get update \
&& apt-get install -y --no-install-recommends openssh-server \
&& echo "$SSH_PASSWD" | chpasswd
COPY sshd_config /etc/ssh/
COPY entry.sh /usr/local/bin/
RUN chmod u+x /usr/local/bin/entry.sh
EXPOSE 9988 2222 22 80 8000
CMD ["entry.sh"]
Keep a copy of ClouderaHiveODBC-2.6.1.1001-1.x86_64.rpm in the current directory
Keep the below files as well :
odbc.ini - which has the DB info
entry.sh - which is shell script and has a command - python app.py
ssh_config - file without any extension has the information as shown below
Port 2222
ListenAddress 0.0.0.0
LoginGraceTime 180
X11Forwarding yes
Ciphers aes128-cbc,3des-cbc,aes256-cbc
MACs hmac-sha1,hmac-sha1-96
StrictModes yes
SyslogFacility DAEMON
PrintMotd no
IgnoreRhosts no
#deprecated option
#RhostsAuthentication no
RhostsRSAAuthentication yes
RSAAuthentication no
PasswordAuthentication yes
PermitEmptyPasswords no
PermitRootLogin yes

I want to expand the answer by showing an approach that works without conda being necessary. In other words, a full-pip minimum viable docker setup for installing turbodbc. I've fully documented the solution in this Github comment in the official turbodbc repo.

Run install python packages on Dockerfile

I'm new to Docker and currently trying to create a Dockerfile with installing the python packages and its libraries as shown here:
FROM balenalib/fincm3-debian-python:latest
# RUN install_packages git
RUN apt-get update && apt-get install python \
&& apt-get install pip3 \
apt-get install libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev \
pip3 install pyaudio \
pip3 install numpy \
pip3 install matplotlib \
pip3 install scipy \
pip3 install librosa \
# Set our working directory
WORKDIR /usr/src/app
COPY Recorder.py /usr/src/app
# Recorder.py will run when container starts up on the device
CMD ["python","/usr/src/app/Recorder.py"]
However, while I am trying to push this Dockerfile, the error is generated with
Error: The command '/bin/sh -c apt-get update && apt-get install python && apt-get install pip3 apt-get install libportaudio0 libportaudio2 libportaudiocpp0 portaudio19-dev pip3 install pyaudio
pip3 install numpy pip3 install matplotlib pip3 install scipy pip3 install librosa WORKDIR /usr/src/app' returned a non-zero code: 100

Moving python packages in requirements.txt and installing python3-pip worked with python:3 base image.
# RUN install_packages git
RUN apt-get update \
&& apt-get install -y python \
&& apt-get install -y python3-pip
RUN pip install -r requirements.txt

The package you are looking for is called python3-pip.
Next, you need both && (to separate commands) and \ (to continue the command line). So, in summary, that should be:
FROM balenalib/fincm3-debian-python:latest
RUN apt-get update && apt-get install python && \
apt-get install -y \
python3-pip libportaudio0 libportaudio2 libportaudiocpp0 \
portaudio19-dev && \
pip3 install pyaudio numpy matplotlib \
scipy librosa
# Set our working directory
WORKDIR /usr/src/app
COPY Recorder.py /usr/src/app
# Recorder.py will run when container starts up on the device
CMD ["python","/usr/src/app/Recorder.py"]

I believe you have more than one problem in this Dockerfile, and when you put all commands together with && and \, you don't know which one is triggering the error. I suggest splitting them for debugging purposes, when they all work then you can put then together. Once you understand each individual error is easier to check and solve them. this question has valuable info: how to install pip in docker
Try this:
1- packages are triggers Y/n questions, give -y to guarantee it passes
2- using the backslashes to refer to a new command, you should use &&, backslashes refer to breaking line, you can use \ and then &&
3- pip3 and libportaudio0 packages doesn't exist.
E: Unable to locate package libportaudio0
I found out about the errors dividing the Dockerfile like this and removing the problems mentioned:
RUN apt-get update
RUN apt-get install python -y\
&& apt-get install python3-pip -y
RUN apt-get install libportaudio2 libportaudiocpp0 portaudio19-dev -y
RUN pip3 install pyaudio numpy matplotlib \
scipy librosa
If you want to put the commands together:
RUN apt-get update \
&& apt-get install python -y \
&& apt-get install python3-pip -y \
&& apt-get install libportaudio2 libportaudiocpp0 portaudio19-dev -y \
&& pip3 install pyaudio numpy matplotlib \
scipy librosa
I also suggest adding a pip requirements file, would make things cleaner.

Dockerfile returning a non-zero code:1

I'm trying to build an image using DockerFile in AmazonSageMaker but I'm getting the following error.
Traceback (most recent call last):
File "/usr/bin/pip3", line 9, in
from pip import main ImportError: cannot import name 'main' The command '/bin/sh -c pip3 install --upgrade pip setuptools wheel &&
pip3 install mxnet-cu90 --upgrade --pre && pip3 install
keras-mxnet --upgrade --pre' returned a non-zero code: 1
My DockerFile is as below
FROM nvidia/cuda:9.0-runtime
RUN apt-get update && \
apt-get -y install build-essential libopencv-dev libopenblas-dev libjemalloc-dev libgfortran3 \
python-dev python3-dev python3-pip wget curl
COPY train_siamese_network.py /opt/program/train RUN chmod +x /opt/program/train
RUN mkdir /root/.keras COPY keras.json /root/.keras/
RUN pip3 install --upgrade pip setuptools wheel && \
pip3 install mxnet-cu90 --upgrade --pre && \
pip3 install keras-mxnet --upgrade --pre
RUN rm -rf /var/lib/apt/lists/* RUN rm -rf /root/.cache
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/usr/local/lib"
ENV PATH="/opt/program:${PATH}"
WORKDIR /opt/program

you can run a container use image nvidia/cuda:9.0-runtime, try pip install command in the container and see what the command return