I have something like the following in my Dockerfile:
ADD api/requirements.txt /app/
# Install apt and pip packages
RUN set -ex \
&& BUILD_DEPS=" \
build-essential \
gcc \
python3-dev \
libsasl2-dev libldap2-dev libssl-dev \
libpq-dev libffi-dev \
python3 python3-pip python3-wheel python3-setuptools \
zlib1g-dev libjpeg-dev libpng-dev \
" \
&& RUN_DEPS=" \
postgresql-client \
nginx \
libsasl2-dev libldap2-dev libssl-dev python3-dev \
python3 python3-pip python3-setuptools python3-wheel \
wireguard \
gunicorn3 \
curl \
zlib1g-dev libjpeg-dev libpng-dev \
" \
&& apt-get update && apt-get install -y $BUILD_DEPS \
&& pip3 install --no-cache-dir --default-timeout=100000 -r /app/requirements.txt \
&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false $BUILD_DEPS \
&& apt-get update && apt-get install -y --no-install-recommends $RUN_DEPS \
&& rm -rf /var/lib/apt/lists/*
One pypackage I have in my requirements.txt is six. However, the debian package python3-six is a dependency of one of those packages in BUILD_DEPS. This means that when it runs pip install -r requirements.txt, it skips installing six as it's already supplied by the distro package.
Is there a way to force that pypackage to install anyway?
Related
For a Docker container I am making a build of OpenCV which should work together with CUDA.
Here is an excerpt from the Dockerfile:
ARG OPENCV_VERSION=4.7.0
RUN cd /opt/ &&\
# Download and unzip OpenCV and opencv_contrib and delte zip files
wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
unzip $OPENCV_VERSION.zip &&\
rm $OPENCV_VERSION.zip &&\
wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
unzip ${OPENCV_VERSION}.zip &&\
rm ${OPENCV_VERSION}.zip &&\
# Create build folder and switch to it
mkdir /opt/opencv-${OPENCV_VERSION}/build && cd /opt/opencv-${OPENCV_VERSION}/build &&\
# Cmake configure
cmake \
-DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
-DWITH_CUDA=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
# Install path will be /usr/local/lib (lib is implicit)
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. &&\
# Make
make -j"$(nproc)" && \
# Install to /usr/local/lib
make install && \
ldconfig &&\
# Remove OpenCV sources and build folder
rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}
So far this works, but I have to create a very small container. If I now want to use OpenCV with CUDA, I end up with 2.7GB with all the data, etc.
I would now like to get this smaller. My idea would be to omit the extensions of OpenCV, but then I can no longer create CUDA with OpenCV. Since there are probably libraries in there which are needed.
Is there a way to do this without the libraries or can I delete them and "tell" OpenCV not to look for them? If I just leave them out there are a lot of error messages in OpenCV.
Here is the whole Dockerfile, it is not perfect:
FROM ghcr.io/ifm/ifm3d:latest-l4t-arm64 AS buildstage
COPY requirements.txt /tmp/
COPY python /tmp/python
USER root
ARG JETPACK_VERSION_BASE="r32.4"
ARG JETPACK_VERSION="${JETPACK_VERSION_BASE}.3"
ARG BASE_IMAGE="nvcr.io/nvidia/l4t-base:${JETPACK_VERSION}"
ARG SOC="t186"
ADD --chown=root:root https://repo.download.nvidia.com/jetson/jetson-ota-public.asc /etc/apt/trusted.gpg.d/jetson-ota-public.asc
RUN chmod 644 /etc/apt/trusted.gpg.d/jetson-ota-public.asc \
&& apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
&& echo "deb https://repo.download.nvidia.com/jetson/common ${JETPACK_VERSION_BASE} main" > /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& echo "deb https://repo.download.nvidia.com/jetson/${SOC} ${JETPACK_VERSION_BASE} main" >> /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& cat /etc/apt/sources.list.d/nvidia-l4t-apt-source.list \
&& apt-get update \
&& rm -rf /var/lib/apt/lists/*
ARG CUDA=10.2
ENV CUDA=${CUDA}
ENV PATH /usr/local/cuda-$CUDA/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/cuda-$CUDA/targets/aarch64-linux/lib:${LD_LIBRARY_PATH}
RUN ldconfig
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES all
ARG OPENCV_VERSION=4.7.0
RUN apt-get update && \
apt-get install -y \
bc \
bzip2 \
language-pack-en-base \
python3-distutils \
python3-pip \
build-essential \
cuda-libraries-dev-${CUDA} \
cuda-cudart-dev-${CUDA} \
cuda-compiler-${CUDA} \
libnvinfer-samples \
ca-certificates \
python-dev \
git \
cmake \
wget \
unzip \
yasm \
pkg-config \
libswscale-dev \
libtbb2 \
libtbb-dev \
libjpeg-dev \
libpng-dev \
libtiff-dev \
libavformat-dev \
libpq-dev \
libxine2-dev \
libglew-dev \
libtiff5-dev \
zlib1g-dev \
libjpeg-dev \
libavcodec-dev \
libavformat-dev \
libavutil-dev \
libpostproc-dev \
libswscale-dev \
libeigen3-dev \
libtbb-dev \
libgtk2.0-dev \
pkg-config \
## Python
python3-numpy \
libopencv-dev \
g++ \
libcudnn8 \
nvidia-cudnn8 \
libsm6 \
libxrender-dev \
&& rm -rf /var/lib/apt/lists/*
USER ifm
RUN python3 -m pip install --upgrade pip
RUN pip3 install --requirement /tmp/requirements.txt
USER root
ARG OPENCV_VERSION=4.7.0
RUN cd /opt/ &&\
# Download and unzip OpenCV and opencv_contrib and delte zip files
wget https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip &&\
unzip $OPENCV_VERSION.zip &&\
rm $OPENCV_VERSION.zip &&\
wget https://github.com/opencv/opencv_contrib/archive/$OPENCV_VERSION.zip &&\
unzip ${OPENCV_VERSION}.zip &&\
rm ${OPENCV_VERSION}.zip &&\
# Create build folder and switch to it
mkdir /opt/opencv-${OPENCV_VERSION}/build && cd /opt/opencv-${OPENCV_VERSION}/build &&\
# Cmake configure
cmake \
-DOPENCV_EXTRA_MODULES_PATH=/opt/opencv_contrib-${OPENCV_VERSION}/modules \
-DWITH_CUDA=ON \
-DCMAKE_BUILD_TYPE=RELEASE \
# Install path will be /usr/local/lib (lib is implicit)
-DCMAKE_INSTALL_PREFIX=/usr/local \
.. &&\
# Make
make -j"$(nproc)" && \
# Install to /usr/local/lib
make install && \
ldconfig &&\
# Remove OpenCV sources and build folder
rm -rf /opt/opencv-${OPENCV_VERSION} && rm -rf /opt/opencv_contrib-${OPENCV_VERSION}
#RUN ls /usr/local/lib/libopencv*
FROM ghcr.io/ifm/ifm3d:latest-l4t-arm64
USER root
#COPY --from=buildstage /usr/local/include/opencv4/ /usr/local/include/opencv4/
COPY --from=buildstage /home/ifm/venv/lib/python3.9/site-packages /home/ifm/venv/lib/python3.9/site-packages
COPY --from=buildstage /usr/local/lib/python3.9/site-packages/cv2/ /home/ifm/venv/lib/python3.9/site-packages/cv2/
COPY --from=buildstage /usr/local/lib/libopencv* /usr/local/lib/
COPY --from=buildstage /usr/lib/x86_64-linux-gnu/ /usr/lib/x86_64-linux-gnu/
#RUN ldconfig
COPY python /tmp/python
The basis container it's a l4t for TX2
Several try to build the container, but currently only OpenCV works without CUDA
I must run some code inside a container, but the code requires Python 3.9.10, and I am struggling to install version 3.9.10.
I can modify the Dockerfile, but I can't change the base image (which is based on Ubuntu) to python/whatever.
I tried apt install, but the installed version is 3.9.5.
I also tried using conda to install version 3.9.10 but is incredibly painful to activate its environments automatically.
Suggestions?
I ended up compiling the version I needed and changing the system defaults, heres my Makefile.
FROM nvidia/cuda:11.4.2-cudnn8-runtime-ubuntu20.04
ENV TZ=America/Sao_Paulo
ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
RUN apt-get update \
&& apt-get install --no-install-recommends --no-install-suggests -y \
build-essential \
curl \
git \
libbz2-dev \
libffi-dev \
liblzma-dev \
libncursesw5-dev \
libreadline-dev \
libsqlite3-dev \
libssl-dev \
libxml2-dev \
libxmlsec1-dev \
llvm \
make \
tk-dev \
wget \
xz-utils \
zlib1g-dev \
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*
# compile python 3.9.10
RUN git clone --depth 1 https://github.com/python/cpython.git --branch v3.9.10 \
&& cd /cpython \
&& ./configure --enable-optimizations \
&& make \
&& make install \
&& update-alternatives --install /usr/bin/python python /usr/local/bin/python3 999 \
&& rm -rf /cpython
# ...
# commands that require both Python 3.9.10 and "nvidia stuff"
# ...
I build dockerimages for a jupyterlab with some apt and pip packages.
docker image ls:
jupyterlab 3.7.12 9d4cc5c15853 27 minutes ago 2.3GB
jupyterlab 3.9.9 1962a1eb6bff 25 hours ago 1.12GB
The filesize of the images differs quite much, I only took a another python version.
Here are the dockerfiles for comparison:
This dockerfile is build with python 3.7.12 as baseimage:
FROM python:3.7.12-slim-buster
WORKDIR /
COPY wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl ./wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir numpy && \
pip install --no-cache-dir /wheels/scipy-1.7.3-cp37-cp37m-linux_armv7l.whl && \
pip install --no-cache-dir jupyterlab
# pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
# pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
This is the one with python 3.9.9 as baseimage:
FROM python:3.9.9-slim-buster
WORKDIR /
COPY wheels ./wheels
## INSTALL WITH APK
RUN apt-get update && apt-get install -y \
g++ \
gcc \
python3-dev \
libjpeg-dev \
zlib1g-dev \
make \
wget \
libatlas-base-dev \
libffi-dev
## INSTALL WITH PIP
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir pillow && \
pip install --no-cache-dir matplotlib && \
pip install --no-cache-dir pandas && \
pip install --no-cache-dir setuptools && \
pip install --no-cache-dir cffi && \
pip install --no-cache-dir GLIBC && \
pip install --no-cache-dir /wheels/numpy-1.21.4-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/scipy-1.7.2-cp39-cp39-linux_armv7l.whl && \
pip install --no-cache-dir /wheels/jupyterlab-4.0.0a15-py3-none-any.whl
...
The wheels folder and some other files are deleted in both cases. Why is there such a big difference in filesize???
I solved the issue by removing "apt-get python3-dev", it was version 3.7.2. My baseimage is python 3.7.12 so maybe there was a conflict, I can not explain it.
The size of the image:
#docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
jupyterlab 3.7.12_nodev 6112bcaff54d 17 hours ago 1.18GB
jupyterlab 3.9.9 1962a1eb6bff 2 days ago 1.12GB
I have a python script where various ping check has been performed . The same py file runs perfectly when triggered locally , but when i run the job from jenkins after building docker image the below error comes up and ping check always return false.
sh: 1: ping: not found
Can someone help me why this error is coming, do I need to update dockerfile with some update or some jenkins configuration is required.
Updating docker file
FROM ubuntu:18.04
ADD requirements.txt /requirements.txt
ADD jenkins_bot.py /jenkins_bot.py
ADD src /src
ADD config /config
ADD run_unit_tests.sh /run_unit_tests.sh
ADD utests /utests
WORKDIR /
RUN DEBIAN_FRONTEND=noninteractive && apt-get -qq update && \
apt-get -y install apt-utils && \
apt-get -qqy install ssh && \
apt-get -qqy install build-essential \
python3-dev \
python3-setuptools \
libfreetype6-dev \
libxft-dev && \
apt-get -qqy install python3-pip && \
pip3 install -r /requirements.txt
#RUN sh run_unit_tests.sh
Thanks in advance.
Ping is missing from in the ubuntu base image. You can update your dockerfile as follows to install it:
FROM ubuntu:18.04
ADD requirements.txt /requirements.txt
ADD jenkins_bot.py /jenkins_bot.py
ADD src /src
ADD config /config
ADD run_unit_tests.sh /run_unit_tests.sh
ADD utests /utests
WORKDIR /
RUN DEBIAN_FRONTEND=noninteractive && apt-get -qq update && \
apt-get -y install apt-utils && \
apt-get install -y iputils-ping && \
apt-get -qqy install ssh && \
apt-get -qqy install build-essential \
python3-dev \
python3-setuptools \
libfreetype6-dev \
libxft-dev && \
apt-get -qqy install python3-pip && \
pip3 install -r /requirements.txt
#RUN sh run_unit_tests.sh
I am trying to build Python and Pip from source. Following is the Dockerfile for it:
FROM buildpack-deps:stretch
ARG PYTHON_VERSION="3.6.5"
ARG PIP_VERSION="20.2"
ARG INSTALLATION_PREFIX="/opt/python/${PYTHON_VERSION}"
WORKDIR /usr/src/python
RUN wget -q https://www.python.org/ftp/python/$PYTHON_VERSION/Python-${PYTHON_VERSION}.tgz \
&& apt-get update \
&& apt-get install -y \
make \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
wget \
curl \
llvm \
libncurses5-dev \
libncursesw5-dev \
xz-utils \
tk-dev
RUN tar xvf Python-${PYTHON_VERSION}.tgz \
&& rm -f Python-${PYTHON_VERSION}.tgz
RUN cd Python-$PYTHON_VERSION \
&& ./configure \
--prefix=${INSTALLATION_PREFIX} \
--build=$(dpkg-architecture --query DEB_BUILD_GNU_TYPE) \
--enable-loadable-sqlite-extensions \
--enable-shared \
--with-system-expat \
--with-system-ffi \
--without-ensurepip \
&& make -j $(nproc) \
&& make altinstall
ENV LD_LIBRARY_PATH="${INSTALLATION_PREFIX}/lib:$LD_LIBRARY_PATH"
RUN wget -q https://bootstrap.pypa.io/get-pip.py -O /get-pip.py \
&& /usr/src/python/Python-3.6.5/python /get-pip.py \
--prefix ${INSTALLATION_PREFIX} \
--disable-pip-version-check \
--no-cache-dir \
--no-warn-script-location \
pip==${PIP_VERSION}
WORKDIR /
RUN mv ${INSTALLATION_PREFIX} /tmp/test
ENV LD_LIBRARY_PATH="/tmp/test/lib:$LD_LIBRARY_PATH"
ENV PATH="/tmp/test/bin:$PATH"
When I build the above Dockerfile, it builds fine and I am able to do python3.6 --version and pip --version and I get the expected results.
Now because of a requirement that I have(Imagine building different Python versions from source in a docker image and store these built runtimes somewhere and install them on demand to a location before using it), I need to move this built Python and Pip to a different location, for example /tmp/test...I was hoping that Python and Pip executables would work just as before but when I do a pip --version I get the following error:
root#12c243458190:/# pip --version
bash: /tmp/test/bin/pip: /opt/python/3.6.5/bin/python3.6: bad interpreter: No such file or directory
pip seems to have hard-coded the location of Python to the installation prefix it was given when it was built.
Any ideas on how to fix this issue?