Some background : I'm new to understanding docker images and containers and how to write DOCKERFILE. I currently have a Dockerfile which installs all the dependencies that I want through PIP install command and so, it was very simple to build and deploy images.
But I currently have a new requirement to use the Dateinfer module and that cannot be installed through the pip install command.
The repo has to be first cloned and then has to be installed and I'm having difficulty achieving this through a DOCKERFILE. The current work around I've been following for now is to run the container and install it manually in the directory with all the other dependencies and Committing the changes with dateinfer installed.But this is a very tedious and time consuming process and I want to achieve the same by just mentioning it in the DOCKERFILE along with all my other dependencies.
This is what my Dockerfile looks like:
FROM ubuntu:20.04
RUN apt update
RUN apt upgrade -y
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt-get -y install tzdata
RUN apt-get install -y libenchant1c2a
RUN apt install git -y
RUN pip3 install argparse
RUN pip3 install boto3
RUN pip3 install numpy==1.19.1
RUN pip3 install scipy
RUN pip3 install pandas
RUN pip3 install scikit-learn
RUN pip3 install matplotlib
RUN pip3 install plotly
RUN pip3 install kaleido
RUN pip3 install fpdf
RUN pip3 install regex
RUN pip3 install pyenchant
RUN pip3 install openpyxl
ADD core.py /
ENTRYPOINT [ "/usr/bin/python3.8", "/core.py”]
So when I try to install Dateinfer like this:
RUN git clone https://github.com/nedap/dateinfer.git
RUN cd dateinfer
RUN pip3 install .
It throws the following error :
ERROR: Directory '.' is not installable. Neither 'setup.py' nor 'pyproject.toml' found.
The command '/bin/sh -c pip3 install .' returned a non-zero code: 1
How do I solve this?
Each RUN directive in a Dockerfile runs in its own subshell. If you write something like this:
RUN cd dateinfer
That is a no-op: it starts a new shell, changes directory, and then the shell exits. When the next RUN command executes, you're back in the / directory.
The easiest way of resolving this is to include your commands in a single RUN statement:
RUN git clone https://github.com/nedap/dateinfer.git && \
cd dateinfer && \
pip3 install .
In fact, you would benefit from doing this with your other pip install commands as well; rather than a bunch of individual RUN
commands, consider instead:
RUN pip3 install \
argparse \
boto3 \
numpy==1.19.1 \
scipy \
pandas \
scikit-learn \
matplotlib \
plotly \
kaleido \
fpdf \
regex \
pyenchant \
openpyxl
That will generally be faster because pip only needs to resolve
dependencies once.
Rather than specifying all the packages individually on the command
line, you could also put them into a requirements.txt file, and then
use pip install -r requirements.txt.
Related
I recently learned about the concept of building docker images based on a multi-staged Dockerfile.
I have been trying simple examples of multi-staged Dockerfiles, and they were working fine. However, when I tried implementing the concept for my own application, I was facing some issues.
My application is about object detection in videos, so I use python and Tensorflow.
Here is my Dockerfile:
FROM python:3-slim AS base
WORKDIR /objectDetector
COPY detect_objects.py .
COPY detector.py .
COPY requirements.txt .
ADD data /objectDetector/data/
ADD models /objectDetector/models/
RUN apt-get update && \
apt-get install protobuf-compiler -y && \
apt-get install ffmpeg libsm6 libxext6 -y && \
apt-get install gcc -y
RUN pip3 install update && python3 -m pip install --upgrade pip
RUN pip3 install tensorflow-cpu==2.9.1
RUN pip3 install opencv-python==4.6.0.66
RUN pip3 install opencv-contrib-python
WORKDIR /objectDetector/models/research
RUN protoc object_detection/protos/*.proto --python_out=.
RUN cp object_detection/packages/tf2/setup.py .
RUN python -m pip install .
RUN python object_detection/builders/model_builder_tf2_test.py
WORKDIR /objectDetector/models/research
RUN pip3 install wheel && pip3 wheel . --wheel-dir=./wheels
FROM python:3-slim
RUN pip3 install update && python3 -m pip install --upgrade pip
COPY --from=base /objectDetector /objectDetector
WORKDIR /objectDetector
RUN pip3 install --no-index --find-links=/objectDetector/models/research/wheels -r requirements.txt
When I try to run my application in the final stage of the container, I receive the following error:
root#3f062f9a5d64:/objectDetector# python detect_objects.py
Traceback (most recent call last):
File "/objectDetector/detect_objects.py", line 3, in <module>
import cv2
ModuleNotFoundError: No module named 'cv2'
So per my understanding, it seems that opencv-python is not successfully moved from the 1st stage to the 2nd.
I have been searching around, and I found some good blogs and questions tackling the issue of multi-staging Dockerfiles, specifically for python libraries. However, it seems I missing something here.
Here are some references that I have been following to solve the issue:
How do I reduce a python (docker) image size using a multi-stage build?
Multi-stage build usage for cuda,cudnn,opencv and ffmpeg #806
So my question is: How can we use opencv in a multistage docker image?
I added pyenchant to my requirements files then
I tried adding this in my dockerfile (python version 3.10) (this worked on version 3.7 but not now):
RUN apt install --yes libenchant-2-2
RUN apt-get install -y gdebi
RUN curl -sLO http://ftp.de.debian.org/debian/pool/main/e/enchant/libenchant1c2a_1.6.0-11.1+b1_amd64.deb && gdebi -n libenchant1c2a_1.6.0-11.1+b1_amd64.deb
but i tried to import enchant got this error
No module named 'enchant'
And when i tried this
RUN apt install --yes libenchant1c2a
the error was files not found
Any idea how to install it ?
You should include your entire dockerfile including the specific base image. I was able to import enchant with this dockerfile.
You could replace the CMD line with something like CMD ["python", "-u", "main.py"] to run your file.
FROM python:3.10.5-slim-buster
RUN apt update && apt install -y libenchant-dev
RUN pip install pyenchant
CMD ["python", "-c", "'import enchant'"]
I have a python code that contains a dockerfile like this:
FROM python:3.8.3 AS base
RUN apt-get update
RUN apt-get -y install software-properties-common cron vim
RUN apt-get update
RUN apt-get -y install python3-pip
RUN pip3 install pandas
RUN pip3 install sklearn
RUN pip3 install SQLAlchemy
RUN pip3 install ConfigParser
RUN pip3 install psycopg2
RUN pip3 install numpy
RUN pip3 install xgboost
RUN pip3 install xlrd
RUN pip3 install matplotlib
FROM base AS publish
WORKDIR /app
COPY . /app
RUN touch /var/log/cron.log
RUN (crontab -l ; echo "* * * * * python /app/main.py" >> /var/log/cron.log") | crontab
CMD cron && tail -f /var/log/cron.log
I want to execute my main.py python script in every minute, but my main.py is not running when i build and run the docker container. I have found some solutions on this site but those solutions did not work for me. What should i write to execute my main.py python script?
I am getting the error using pip in my docker image.
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y \
software-properties-common
RUN add-apt-repository universe
RUN apt-get install -y \
python3.6 \
python3-pip
ENV PYTHONUNBUFFERED 1
RUN mkdir /api
WORKDIR /api
COPY . /api/
RUN pip install pipenv
RUN ls
RUN pipenv sync
I installed python 3.6 and pip3 but getting
Step 9/11 : RUN pip install pipenv
---> Running in b184de4eb28e
/bin/sh: 1: pip: not found
To run pip for python3 use pip3, not pip.
Another solution.
You can add this line (after apt-get install). It will upgrade pip to the version you need, for instance:
RUN pip3 install --upgrade pip==20.0.1
and you can then use pip install from requirements file (for instance):
RUN pip install -r requirements.txt
I'm using centos/python-36-centos7 as a base image of my application. In Dockerfile, after RUN pip install --upgrade pip, pip successfully upgrades from 9.0.1 to 18.0. Next step, at RUN pip install --no-cache-dir -r requirements.txt, docker keeps throwing error:
/bin/sh: /opt/app-root/bin/pip: /opt/app-root/bin/python3: bad interpreter: No such file or directory
The command '/bin/sh -c pip install --no-cache-dir -r requirements.txt' returned a non-zero code: 126
Operating systems: CentOS 7.2 64 bit
Docker version:18.06.0-ce, build 0ffa825
Complete Dockerfile:
FROM centos/python-36-centos7
MAINTAINER SamYu,sam_miaoyu#foxmail.com
USER root
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
COPY . /faceDetectBaseImg
COPY ./pip.conf /etc/pip.conf
WORKDIR /faceDetectBaseImg
RUN yum install -y epel-release
RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
RUN rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
RUN rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
RUN yum install -y ffmpeg
RUN yum -y install libXrender
RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt
pip.conf:
[global]
trusted-host = mirrors.aliyun.com
index-url = https://mirrors.aliyun.com/pypi/simple
UPDATES:
problem fixed by removing pip install --upgrade pip and running pip 9.0.1. I am thinking it has something to do with pip 18.0 vs CentOS7 docker images. I would still like to know if there is a fix under pip 18.0.
Problems fixed completely by pulling centOS7 image and build python from source. As a reminder, don't use the latest version of centos/python-36-centos7 as of June 2018.