Develop Reference

How can we use opencv in a multistage docker image?

I recently learned about the concept of building docker images based on a multi-staged Dockerfile.
I have been trying simple examples of multi-staged Dockerfiles, and they were working fine. However, when I tried implementing the concept for my own application, I was facing some issues.
My application is about object detection in videos, so I use python and Tensorflow.
Here is my Dockerfile:
FROM python:3-slim AS base
WORKDIR /objectDetector
COPY detect_objects.py .
COPY detector.py .
COPY requirements.txt .
ADD data /objectDetector/data/
ADD models /objectDetector/models/
RUN apt-get update && \
apt-get install protobuf-compiler -y && \
apt-get install ffmpeg libsm6 libxext6 -y && \
apt-get install gcc -y
RUN pip3 install update && python3 -m pip install --upgrade pip
RUN pip3 install tensorflow-cpu==2.9.1
RUN pip3 install opencv-python==4.6.0.66
RUN pip3 install opencv-contrib-python
WORKDIR /objectDetector/models/research
RUN protoc object_detection/protos/*.proto --python_out=.
RUN cp object_detection/packages/tf2/setup.py .
RUN python -m pip install .
RUN python object_detection/builders/model_builder_tf2_test.py
WORKDIR /objectDetector/models/research
RUN pip3 install wheel && pip3 wheel . --wheel-dir=./wheels
FROM python:3-slim
RUN pip3 install update && python3 -m pip install --upgrade pip
COPY --from=base /objectDetector /objectDetector
WORKDIR /objectDetector
RUN pip3 install --no-index --find-links=/objectDetector/models/research/wheels -r requirements.txt
When I try to run my application in the final stage of the container, I receive the following error:
root#3f062f9a5d64:/objectDetector# python detect_objects.py
Traceback (most recent call last):
File "/objectDetector/detect_objects.py", line 3, in <module>
import cv2
ModuleNotFoundError: No module named 'cv2'
So per my understanding, it seems that opencv-python is not successfully moved from the 1st stage to the 2nd.
I have been searching around, and I found some good blogs and questions tackling the issue of multi-staging Dockerfiles, specifically for python libraries. However, it seems I missing something here.
Here are some references that I have been following to solve the issue:
How do I reduce a python (docker) image size using a multi-stage build?
Multi-stage build usage for cuda,cudnn,opencv and ffmpeg #806
So my question is: How can we use opencv in a multistage docker image?

How do I install dateinfer inside my Docker image

Some background : I'm new to understanding docker images and containers and how to write DOCKERFILE. I currently have a Dockerfile which installs all the dependencies that I want through PIP install command and so, it was very simple to build and deploy images.
But I currently have a new requirement to use the Dateinfer module and that cannot be installed through the pip install command.
The repo has to be first cloned and then has to be installed and I'm having difficulty achieving this through a DOCKERFILE. The current work around I've been following for now is to run the container and install it manually in the directory with all the other dependencies and Committing the changes with dateinfer installed.But this is a very tedious and time consuming process and I want to achieve the same by just mentioning it in the DOCKERFILE along with all my other dependencies.
This is what my Dockerfile looks like:
FROM ubuntu:20.04
RUN apt update
RUN apt upgrade -y
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y libenchant1c2a
RUN apt install git -y
RUN pip3 install argparse
RUN pip3 install boto3
RUN pip3 install numpy==1.19.1
RUN pip3 install scipy
RUN pip3 install pandas
RUN pip3 install scikit-learn
RUN pip3 install matplotlib
RUN pip3 install plotly
RUN pip3 install kaleido
RUN pip3 install fpdf
RUN pip3 install regex
RUN pip3 install pyenchant
RUN pip3 install openpyxl
ADD core.py /
ENTRYPOINT [ "/usr/bin/python3.8", "/core.py”]
So when I try to install Dateinfer like this:
RUN git clone https://github.com/nedap/dateinfer.git
RUN cd dateinfer
RUN pip3 install .
It throws the following error :
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
The command '/bin/sh -c pip3 install .' returned a non-zero code: 1
How do I solve this?

Each RUN directive in a Dockerfile runs in its own subshell. If you write something like this:
RUN cd dateinfer
That is a no-op: it starts a new shell, changes directory, and then the shell exits. When the next RUN command executes, you're back in the / directory.
The easiest way of resolving this is to include your commands in a single RUN statement:
RUN git clone https://github.com/nedap/dateinfer.git && \
cd dateinfer && \
pip3 install .
In fact, you would benefit from doing this with your other pip install commands as well; rather than a bunch of individual RUN
commands, consider instead:
RUN pip3 install \
argparse \
boto3 \
numpy==1.19.1 \
scipy \
pandas \
scikit-learn \
matplotlib \
plotly \
kaleido \
fpdf \
regex \
pyenchant \
openpyxl
That will generally be faster because pip only needs to resolve
dependencies once.
Rather than specifying all the packages individually on the command
line, you could also put them into a requirements.txt file, and then
use pip install -r requirements.txt.

Unable to locate package python3-pip in Dockerfile while having Universal Repos enabled

I am trying to install python3 and pip.
So, I have a dockerfile
(I also tried with ubuntu:20.04)-
(Note- I have made the Dockerfile short, for full dockerfile you can see here)
FROM ubuntu:latest
WORKDIR /app
COPY . /app
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository universe
ENV PYTHONUNBUFFERED=1
RUN apt install -y python3
RUN apt-get -y install python3-pip
RUN pip install --no-cache --upgrade pip setuptools
But it always give me error-
---> Running in ba9f299b852b
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package python3-pip
The command '/bin/sh -c apt-get -y install python3-pip' returned a non-zero code: 100
While finding more out in the error, I found out that while the universal repositories were added, python3-pip didnt got installed, while other things like python3-sip got installed.
How do I fix this error and why did it happened?

The software-properties-common meta-packages installs a bunch of softwares that are not required in a docker image and make its the size quite huge.
This simpler Dockerfile does the job for me.
FROM ubuntu:latest
WORKDIR /app
#COPY . /app
RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
python3 \
python3-pip \
&& apt-get clean all
RUN pip install --no-cache --upgrade pip setuptools
RUN pip --version # just for test
Output:
STEP 1/5: FROM ubuntu:latest
STEP 2/5: WORKDIR /app
--> Using cache 9f1d34a70ba9e13074601d16456fda97b2de87a0be3b18ddc6963fb65aaee096
--> 9f1d34a70ba
STEP 3/5: RUN apt-get update && apt-get install --assume-yes --no-install-recommends --quiet python3 python3-pip && apt-get clean all
[...]
STEP 4/5: RUN pip install --no-cache --upgrade pip setuptools
Collecting pip
Downloading pip-21.3-py3-none-any.whl (1.7 MB)
Collecting setuptools
Downloading setuptools-58.2.0-py3-none-any.whl (946 kB)
Installing collected packages: pip, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Attempting uninstall: setuptools
Found existing installation: setuptools 45.2.0
Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed pip-21.3 setuptools-58.2.0
STEP 5/5: RUN pip --version
pip 21.3 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
COMMIT staskoverflow-ubuntu
--> 2498ddc1767
Successfully tagged localhost/staskoverflow-ubuntu:latest
2498ddc17672584d2dd38bf23ae63521d9adb43465cf8c7b415c33dc2aa0aa01
The tool pip is updated to the version 21.3 and the OS-provided one is not uninstalled (this is a pip feature).
Anyway this dockefile, more similar to yours, works too:
RUN apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
software-properties-common \
&& add-apt-repository universe \
&& apt-get update \
&& apt-get install --assume-yes --no-install-recommends --quiet \
python3 \
python3-pip \
&& apt-get clean all
RUN pip install --no-cache --upgrade pip setuptools
After adding the universe repository, you should run again apt-get update to retrieve the new lists of packages with the new apt configuration. This is the root cause of the problem I think.

docker image build fails at RUN pip install -r requirements.txt in a Flask app

I have a Flask application with the following requirements.txt
chalice
matplotlib
sklearn
numpy
scipy
pandas
flask
flask_restful
and the following Dockerfile:
FROM python:3.6.1-alpine
WORKDIR /project
ADD . /project
RUN pip install -r requirements.txt
CMD ["python","app.py"]
Running the command docker image build -t clf_test .
generates the following error:
You are using pip version 9.0.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
It seems that the matplotlib can't get installed for some reason.
Running the pip install -r requirements.txt locally doesn't produce any errors

matplotlib must be built from source, and compiling it requires a number of supporting libraries as well as a functioning C compiler. You can figure out what these are and install them so that it builds properly...
...or you can just base your Dockerfile on the non-alpine python:3.6.1 image and then apt-get install python3-matplotlib before installing your other requirements. E.g., this builds without errors:
FROM python:3.6.1
WORKDIR /project
ADD . /project
RUN apt update; apt-get -y install python3-matplotlib
RUN pip install -r requirements.txt
CMD ["python","app.py"]

cant install pip in ubuntu 18.04 docker /bin/sh: 1: pip: not found

I am getting the error using pip in my docker image.
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository universe
RUN apt-get install -y \
python3.6 \
python3-pip
ENV PYTHONUNBUFFERED 1
RUN mkdir /api
WORKDIR /api
COPY . /api/
RUN pip install pipenv
RUN ls
RUN pipenv sync
I installed python 3.6 and pip3 but getting
Step 9/11 : RUN pip install pipenv
---> Running in b184de4eb28e
/bin/sh: 1: pip: not found

To run pip for python3 use pip3, not pip.

Another solution.
You can add this line (after apt-get install). It will upgrade pip to the version you need, for instance:
RUN pip3 install --upgrade pip==20.0.1
and you can then use pip install from requirements file (for instance):
RUN pip install -r requirements.txt