Develop Reference

How to make a Dockerfile?

I need a Dockerfile to run my Python script. The script uses Selenium, so I need to load a driver for it to work. An ordinary .exe file - driver is not suitable, so according to the advice of the administrators of the hosting where the script is located I need to create a Dockerfile for the script to work properly.
The main problem is that I simply can not run my script, because I do not understand how to load the required driver on the server.
This is a sample code of what should be in the Dockerfile.
FROM python:3
RUN apt-get update -y
RUN apt-get install -y wget
RUN wget -O $HOME/geckodriver.tar.gz https://github.com/mozilla/geckodriver/releases/download/v0.23.0/geckodriver-v0.23.0-linux64.tar.gz
RUN tar xf $HOME/geckodriver.tar.gz -C $HOME
RUN cp $HOME/geckodriver /usr/local/bin/geckodriver
RUN chmod +x /usr/local/bin/geckodriver
RUN rm -f $HOME/geckodriver $HOME/geckodriver.tar.gz
This is the code used in the Python script
options = Options()
options.add_argument('headless')
options.add_argument('window-size=1920x935')
driver = webdriver.Chrome(options=options, executable_path=r"chromedriver.exe")
driver.get(f"https://www.wildberries.ru/catalog/{id}/feedbacks?imtId={imt_id}")
time.sleep(5)
big_stat = driver.find_element(by=By.CLASS_NAME, value="rating-product__numb")
I can redo this snippet of code to make it work on Firefox, if necessary.
This is what the directories of the hosting where all the files are located look like
The directories of the hosting

For getting Selenium to work with Python using a Dockerfile, here's an existing SeleniumBase Dockerfile.
For instructions on using it, see the README.
For building, it's basically this:
Non Apple M1 Mac:
docker build -t seleniumbase .
If running on an Apple M1 Mac, use this instead:
docker build --platform linux/amd64 seleniumbase .
Before building the Dockerfile, you'll need to clone SeleniumBase.
Here's what the Dockerfile currently looks like:
FROM ubuntu:18.04
#=======================================
# Install Python and Basic Python Tools
#=======================================
RUN apt-get -o Acquire::Check-Valid-Until=false -o Acquire::Check-Date=false update
RUN apt-get install -y python3 python3-pip python3-setuptools python3-dev python-distribute
RUN alias python=python3
RUN echo "alias python=python3" >> ~/.bashrc
#=================================
# Install Bash Command Line Tools
#=================================
RUN apt-get -qy --no-install-recommends install \
sudo \
unzip \
wget \
curl \
libxi6 \
libgconf-2-4 \
vim \
xvfb \
&& rm -rf /var/lib/apt/lists/*
#================
# Install Chrome
#================
RUN curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list && \
apt-get -yqq update && \
apt-get -yqq install google-chrome-stable && \
rm -rf /var/lib/apt/lists/*
#=================
# Install Firefox
#=================
RUN apt-get -qy --no-install-recommends install \
$(apt-cache depends firefox | grep Depends | sed "s/.*ends:\ //" | tr '\n' ' ') \
&& rm -rf /var/lib/apt/lists/* \
&& cd /tmp \
&& wget --no-check-certificate -O firefox-esr.tar.bz2 \
'https://download.mozilla.org/?product=firefox-esr-latest&os=linux64&lang=en-US' \
&& tar -xjf firefox-esr.tar.bz2 -C /opt/ \
&& ln -s /opt/firefox/firefox /usr/bin/firefox \
&& rm -f /tmp/firefox-esr.tar.bz2
#===========================
# Configure Virtual Display
#===========================
RUN set -e
RUN echo "Starting X virtual framebuffer (Xvfb) in background..."
RUN Xvfb -ac :99 -screen 0 1280x1024x16 > /dev/null 2>&1 &
RUN export DISPLAY=:99
RUN exec "$#"
#=======================
# Update Python Version
#=======================
RUN apt-get update -y
RUN apt-get -qy --no-install-recommends install python3.8
RUN rm /usr/bin/python3
RUN ln -s python3.8 /usr/bin/python3
#=============================================
# Allow Special Characters in Python Programs
#=============================================
RUN export PYTHONIOENCODING=utf8
RUN echo "export PYTHONIOENCODING=utf8" >> ~/.bashrc
#=====================
# Set up SeleniumBase
#=====================
COPY sbase /SeleniumBase/sbase/
COPY seleniumbase /SeleniumBase/seleniumbase/
COPY examples /SeleniumBase/examples/
COPY integrations /SeleniumBase/integrations/
COPY requirements.txt /SeleniumBase/requirements.txt
COPY setup.py /SeleniumBase/setup.py
RUN find . -name '*.pyc' -delete
RUN find . -name __pycache__ -delete
RUN pip3 install --upgrade pip
RUN pip3 install --upgrade setuptools
RUN pip3 install --upgrade setuptools-scm
RUN cd /SeleniumBase && ls && pip3 install -r requirements.txt --upgrade
RUN cd /SeleniumBase && pip3 install .
#=====================
# Download WebDrivers
#=====================
RUN wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz
RUN tar -xvzf geckodriver-v0.31.0-linux64.tar.gz
RUN chmod +x geckodriver
RUN mv geckodriver /usr/local/bin/
RUN wget https://chromedriver.storage.googleapis.com/2.44/chromedriver_linux64.zip
RUN unzip chromedriver_linux64.zip
RUN chmod +x chromedriver
RUN mv chromedriver /usr/local/bin/
#==========================================
# Create entrypoint and grab example tests
#==========================================
COPY integrations/docker/docker-entrypoint.sh /
COPY integrations/docker/run_docker_test_in_firefox.sh /
COPY integrations/docker/run_docker_test_in_chrome.sh /
RUN chmod +x *.sh
COPY integrations/docker/docker_config.cfg /SeleniumBase/examples/
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["/bin/bash"]

Why is there no console output from my docker container?

I have an application that consists of a docker container with a redis instance, another docker container with a rabbitmq server, and a third container in which I'm trying to start a number of celery workers that can interact with the redis and rabbitmq containers. I'm starting and stopping these workers via a rest api, and have checked that I'm able to do this on my host machine. However, after moving the setup to docker, it seems the workers are not behaving as expected. Whereas on my host machine (windows 10) I was able to see the reply from the rest api and console output from the workers, I can only see the response from the rest api (a log message) and no console output. It also seems that the workers are not accessing the redis and rabbitmq instances.
My docker container is built from a python3.6 (linux) base image. I have checked that everything is installed correctly, and there are no error logs. I build the image with the following dockerfile:
FROM python:3.6
WORKDIR /opt
# create a virtual environment and add it to PATH so that it is applied for
# all future RUN and CMD calls
ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# install msodbcsql17
RUN apt-get update \
&& apt-get install -y curl apt-transport-https gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
\
&& curl https://packages.microsoft.com/config/debian/9/prod.list >
/etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql17 mssql-tools
# Install Mono for pythonnet.
RUN apt-get update \
&& apt-get install --yes \
dirmngr \
clang \
gnupg \
ca-certificates \
# Dependency for pyodbc.
unixodbc-dev \
&& apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys
3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF \
&& echo "deb http://download.mono-project.com/repo/debian
stretch/snapshots/5.20 main" | tee /etc/apt/sources.list.d/mono-official-
stable.list \
&& apt-get update \
&& apt-get install --yes \
mono-devel=5.20\* \
&& rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
COPY src ./src
COPY setup.py ./setup.py
COPY config.json ./config.json
COPY Utility.dll ./Utility.dll
COPY settings_docker.ini ./settings.ini
COPY config.json ./config.json
COPY sql_config.json ./sql_config.json
RUN python3 -m venv $VIRTUAL_ENV \
# From here on, use virtual env's python.
&& venv/bin/pip install --upgrade pip \
&& venv/bin/pip install --no-cache-dir --upgrade pip setuptools wheel \
&& venv/bin/pip install --no-cache-dir -r requirements.txt \
# Dependency for pythonnet.
&& venv/bin/pip install --no-cache-dir pycparser \
&& venv/bin/pip install -U --no-cache-dir "pythonnet==2.5.1" \
# && python -m pip install --no-cache-dir "pyodbc==4.0.25" "pythonnet==2.5.1"
EXPOSE 8081
cmd python src/celery_artifacts/docker_workers/worker_app.py
And then run it with this command:
docker run --name app -p 8081:8081 app
I then attach the container to the same bridge network as the other 2:
docker network connect my_network app
Is there a way to see the same console output from my container as the one on the host?

Avoid dependency hell on docker

I build an AI application in Python involving quiet an amount of Python libraries. At this point, I would like to run my application inside of a docker container to make the AI App a service.
What are my options concerning dependencie so that all necessary libraries are downloaded automatically?
As an weak alternative, I tried this with a "requirement.txt" file on the same level as my Docker build file, but this didn't work.

Your Dockerfile will need instructions to install the requirements, e.g.
COPY requirement.txt requirement.txt
RUN pip install -r requirement.txt

Thank you for the very useful comments:
My dockerfile:
# Python 3.7.3
FROM python:3.7-slim
# Set the working directory to /app
WORKDIR /app
COPY greeter_server.py /app
COPY AspenTechClient.py /app
COPY OpcUa_pb2.py /app
COPY OpcUa_pb2_grpc.py /app
COPY helloworld_pb2.py /app
COPY helloworld_pb2_grpc.py /app
COPY Models.py /app
ADD ./requirement.txt /app
# Training & Validation data we need
RUN mkdir -p /app/output
RUN pip install -r requirement.txt
#RUN pip3 install grpcio grpcio-tools
#RUN pip install protobuf
#RUN pip install pandas
#RUN pip install scipy
#expose ports to outside container for web app access
EXPOSE 10500
# Argument to python command
CMD [ "python", "/app/greeter_server.py" ]
By the tips here, I already added the extra lines for "requirement.txt" and that works like a charm. Thank you very much!
Since I only want to run a deployment in the container, I will forseen trained models so no need for a GPU. For this I have a local machine. With an appropriate mount I deliver the .h5 to the container.
#pyeR_biz: Thank you very much for the tips about pipelines. This is something I didn't have experience with but certainly will do it in the near future.

You have several options. It depends a lot on the use case, the number of containers you will eventually build, production vs dev environment etc.
Generally if you have an AI application you will need a graphics card driver pre-installed on your host system for model training. Which means eventually you'll have to come up with a way to automate driver install or write instructions for end users to do that. For an app you might also need database drivers in the docker image, if your front or back end databases are outside the container. Here is a toned-down example of one of my uses cases with requirement being building docker for a data pipeline.
#Taken from puckel/docker-airflow
#can look up this image name on google to see which OS it is based on.
FROM python:3.6-slim-buster
LABEL maintainer="batman"
# Never prompt the user for choices on installation/configuration of packages
ENV DEBIAN_FRONTEND noninteractive
ENV TERM linux
# Set some default configuration for data pipeline management tool called airflow
ARG AIRFLOW_VERSION=1.10.9
ARG AIRFLOW_USER_HOME=/usr/local/airflow
ARG AIRFLOW_DEPS=""
ENV AIRFLOW_HOME=${AIRFLOW_USER_HOME}
# here install some linux dependencies required to run the pipeline.
# use apt-get install, apt-get auto-remove etc to reduce size of image
# curl and install sql server odbc driver for my linux
RUN set -ex \
&& buildDeps=' freetds-dev libkrb5-dev libsasl2-dev libssl-dev libffi-dev libpq-dev git' \
&& apt-get update -yqq \
&& apt-get upgrade -yqq \
&& apt-get install -yqq --no-install-recommends \
$buildDeps freetds-bin build-essential default-libmysqlclient-dev \
apt-utils curl rsync netcat locales gnupg wget \
&& useradd -ms /bin/bash -d ${AIRFLOW_USER_HOME} airflow \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \ #
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list \
&& apt-get update \
&& ACCEPT_EULA=Y apt-get install -y msodbcsql17 \
&& ACCEPT_EULA=Y apt-get install -y mssql-tools \
&& pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql,ssh${AIRFLOW_DEPS:+,}${AIRFLOW_DEPS}]==${AIRFLOW_VERSION} \
&& apt-get purge --auto-remove -yqq $buildDeps \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/* \
/usr/share/man \
/usr/share/doc \
/usr/share/doc-base
# Install all required packages in python environment from requirements.txt (I generally remove version numbers if my python version are same)
ADD ./requirements.txt /config/
RUN pip install -r /config/requirements.txt
# CLEANUP
RUN apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/* \
/usr/share/man \
/usr/share/doc \
/usr/share/doc-base
#CONFIGURATION
COPY script/entrypoint.sh /entrypoint.sh
COPY config/airflow.cfg ${AIRFLOW_USER_HOME}/airflow.cfg
# hand ownership of libraries to relevant user
RUN chown -R airflow: ${AIRFLOW_USER_HOME}
#expose ports to outside container for web app access
EXPOSE 8080 5555 8793
USER airflow
WORKDIR ${AIRFLOW_USER_HOME}
ENTRYPOINT ["/entrypoint.sh"]
CMD ["webserver"]
1) Select an appropriate base image which has the operating system you need.
2) Get your gpu drivers installed if you are training a model, not mandatory if you are serving the model

can't write dockerfile with installations of ubuntu, numpy, opencv

the plan is to deploy pretrained face recog-n model. But before i need to install some libs.
The idea behind docker is that it brings all the needed libs and builds entire 'env' without much overhead. One can just start dockerfile and it runs all other scripts in turn.
libs to install:
Ubuntu 16.04.6 LTS
Python 3.6.10 (3.5.x should be fine also)
OpenCV 3.3.
NumPy
imutils https://github.com/jrosebr1/imutils
dlib http://dlib.net/
face_recognition https://github.com/ageitgey/face_recognition
i m trying to use curl to download pkgs from URLs, but it's not working.
my dockerfile:
FROM ubuntu:16.04.6
RUN apt-get update && apt-get install -y curl bzip2
curl -o numpy
&& sudo apt-get install numpy
&& curl install imutils https://github.com/jrosebr1/imutils
&& curl install dlib https://dlib.net
&& sudo git clone https://github.com/ageitgey/face_recognition.git
&& curl python-opencv https://opencv.org/
&& echo 'export PATH="~/anaconda3/bin:$PATH"' >> ~/.bashrc \
&& ~/anaconda3/bin/conda update -n base conda \
&& rm miniconda_install.sh \
&& rm -rf /var/lib/apt/lists/* \
&& /bin/bash -c "source ~/.bashrc"
ENV PATH="~/anaconda3/bin:${PATH}"
##################################################
# Setup env for current project:
##################################################
EXPOSE 8000
RUN /bin/bash -c "conda create -y -n PYMODEL3.6"
ADD requirements.txt /tmp/setup/requirements.txt
RUN /bin/bash -c "source activate PYMODEL3.6 && pip install -r /tmp/setup/requirements.txt"
WORKDIR /Service
ADD Service /Service
ENTRYPOINT ["/bin/bash", "-c", "source activate PYMODEL3.6 && ./run.sh"]
the face model is pretrained.
there are 2 python files that do actual detection, 128d encoding and recognition.
the usage is like this:
#detect face, if there is face - encode it, return pickle
python3 encode.py --dataset dataset_id --encodings encodings.pickle
--confidence 0.9
#recognize using pickle
python3 face_recognizer.py --encodings encodings.pickle --image
dataset_webcam/3_1.jpg --confidence 0.9 --tolerance 0.5
should I include them in the dockerfile?

I would propose you to use a Dockerfile like the following, assuming you have all your requirements (numpy, imutils, etc...) inside your requirements.txt file, and your encode.py and face_recognizer.py files in your Service folder:
FROM python:3.6.10
RUN mkdir /tmp/setup
ADD requirements.txt /tmp/setup/requirements.txt
RUN pip install --no-cache-dir --upgrade setuptools && \
pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r /tmp/setup/requirements.txt
WORKDIR /Service
ADD Service /Service/
CMD ["./run.sh"]
EXPOSE 8000

Docker python output csv file

I have a script python which should output a file csv. I'm trying to have this file in the current working directory but without success.
This is my Dockerfile
FROM python:3.6.4
RUN apt-get update && apt-get install -y libaio1 wget unzip
WORKDIR /opt/oracle
RUN wget https://download.oracle.com/otn_software/linux/instantclient/instantclient-
basiclite-linuxx64.zip && \ unzip instantclient-basiclite-linuxx64.zip && rm
-f instantclient-basiclite-linuxx64.zip && \ cd /opt/oracle/instantclient*
&& rm -f jdbc occi mysql *README jar uidrvci genezi adrci && \ echo
/opt/oracle/instantclient > /etc/ld.so.conf.d/oracle-instantclient.conf &&
ldconfig
RUN pip install --upgrade pip
COPY . /app
WORKDIR /app
RUN pip install --upgrade pip
RUN pip install pystan
RUN apt-get -y update && python3 -m pip install cx_Oracle --upgrade
RUN pip install -r requirements.txt
CMD [ "python", "Main.py" ]
And run the container with the following command
docker container run -v $pwd:/home/learn/rstudio_script/output image

This is bad practice to bind a volume just to have 1 file on your container be saved onto your host.
Instead, what you should leverage is the copy command:
docker cp <containerId>:/file/path/within/container /host/path/target
You can set this command to auto execute with bash, after your docker run.
So something like:
#!/bin/bash
# this stores the container id
CONTAINER_ID=$(docker run -dit img)
docker cp $CONTAINER_ID:/some_path host_path
If you are adamant on using a bind volume, then as the others have pointed out, the issue is most likely your python script isn't outputting the csv to the correct path.

Your script Main.py is probably not trying to write to /home/learn/rstudio_script/output. The working directory in the container is /app because of the last WORKDIR directive in the Dockerfile. You can override that at runtime with --workdir but then the CMD would have to be changed as well.
One solution is to have your script write files to /output/ and then run it like this:
docker container run -v $PWD:/output/ image