Develop Reference

run two python scripts with docker compose

My folder structure looked like this:
My Dockerfile looked like this:
FROM python:3.8-slim-buster
WORKDIR /src
COPY src/requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
CMD [ "python", "main.py"]
When I ran these commands:
docker build --tag FinTechExplained_Python_Docker .
docker run free
my main.pyfile ran and gave the correct print statements as well. Now, I have added another file tests.py in the src folder. I want to run the tests.py first and then main.py.
I tried modifying the cmdwithin my docker file like this:
CMD [ "python", "test.py"] && [ "python", "main.py"]
but then it gives me the print statements from only the first test.pyfile.
I read about docker-compose and added this docker-compose.yml file to the root folder:
version: '3'
services:
main:
image: free
command: >
/bin/sh -c 'python tests.py'
main:
image: free
command: >
/bin/sh -c 'python main.py'
then I changed my docker file by removing the cmd:
FROM python:3.8-slim-buster
WORKDIR /src
COPY src/requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
Then I ran the following commands:
docker compose build
docker compose run tests
docker compose run main
When I run these commands separately, I get the correct print statements for both testsand main. However, I am not sure if I am using docker-composecorrectly or not.
Am I supposed to run both scripts separately? Or is there a way to run one after another using a single docker command?
How is my Dockerfile supposed to look like if I am running the python scripts from the docker-compose.yml instead?
Edit:
Ideally looking for solutions based on docker-compose

In the Bourne shell, in general, you can run two commands in sequence by putting && between them. It sounds like you're already aware of this.
# without Docker, at a normal shell prompt
python test.py && python main.py
The Dockerfile CMD has two syntactic forms. The JSON-array form does not run a shell, and so it is slightly more efficient and has slightly more consistent escaping rules. If it's not a JSON array then Docker automatically runs it via a shell. So for your use you can use the shell form:
CMD python test.py && python main.py
In comments to other answers you ask about providing this as an override in the docker-compose.yml file. Compose will not normally run a shell for you, so you need to explicitly specify it as part of the command: override.
command: /bin/sh -c 'python test.py && python main.py'
Your Dockerfile should generally specify a CMD and the docker-compose.yml often will not include a command:. This makes it easier to run the image in other contexts (via docker run without Compose; in Kubernetes) since you won't have to retype the command every different way you want to run the container. The entrypoint wrapper pattern highlighted in #sytech's answer is very useful in general and it's easy to add to a container that uses a CMD without an ENTRYPOINT; but it requires the Dockerfile to use CMD as a normal well-formed shell command.

You have to change CMD to ENTRYPOINT. And run the 1st script as daemon in the background using &.
ENTRYPOINT ["/docker_entrypoint.sh"]
docker_entrypoint.sh
#!/bin/bash
set -e
exec python tests.py &
exec python main.py

In general, it is a good rule of thumb that a container should only a single process and that essential process should be pid 1
Using an entrypoint can help you do multiple things at runtime and optionally run user-defined commands using exec, as according to the best practices guide.
For example, if you always want the tests to run whenever the container starts, then execute the defined command in CMD.
First, create an entrypoint script (be sure to make it executable with chmod +x):
#!/usr/bin/env bash
# always run tests first
python /src/tests.py
# then run user-defined command
exec "$#"
Then configure the dockerfile to copy the script and set it as the entrypoint:
#...
COPY entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["python", "main.py"]
Then when you build an image from this dockerfile and run it, the entrypoint will first execute the tests then run the command to run main.py
The command can also still be overridden by the user when running the image like docker run ... myimage <new command> which will still result in the entrypoint tests being executed, but the user can change the command being run.

You can achieve this by creating a bash script(let's name entrypoint.sh) which is containing the python commands. If you want, you can create background processes of those.
#!/usr/bin/env bash
set -e
python tests.py
python main.py
Edit your docker file as follows:
FROM python:3.8-slim-buster
# Create workDir
RUN mkdir code
WORKDIR code
ENV PYTHONPATH = /code
#upgrade pip if you like here
COPY requirements.txt .
RUN pip install -r requirements.txt
# Copy Code
COPY . .
RUN chmod +x entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]
In the docker compose file, add the following line to the service.
entrypoint: [ "./entrypoint.sh" ]

Have you try this in your docker-compose.yaml?
version: '3'
services:
main:
image: free
command: >
/bin/sh -c 'python3 tests.py & && python3 main.py &'
both will run in the background
then run in terminal
docker-compose up --build

Python logging while crontab in docker container

I'm running a python job which logs into a file:
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', filename='/app/logs/ups_tracking.log')
self.logger = logging.getLogger('TRACK-UPS')
When running the job manually, the log files are well created / incremented with new entries.
When running through crontab (syntax below), the logs are not written as expected.
### TRACKING UPS ###
* * * * * python /app/UPS/parcels.py
root#91067d2217e7:/app/logs# service cron status
[ ok ] cron is running.
I'm running the whole thing in a docker container, with the dockerfile below:
#Create the flask custom image
FROM python:latest
# Place your flask application on the server
COPY ./back /app/
WORKDIR /app
# Install requirements.txt
RUN /usr/local/bin/python -m pip install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt
RUN apt-get update && apt-get install -y netcat cron
COPY ./config/init.sh /tmp/init.sh
RUN chmod +x /tmp/init.sh
# Copy crontab_file file to the cron.d directory
COPY ./config/crontab_file /etc/cron.d/crontab_file
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/crontab_file
# Apply cron job
RUN crontab /etc/cron.d/crontab_file
# Start CRON service
RUN service cron start
EXPOSE 8889
ENTRYPOINT ["/tmp/init.sh"]
Am I missing something here ?
Thanks !

# Start CRON service
RUN service cron start
means crond is running only during that RUN stage.
seeing that I wonder if you are starting cron in /tmp/init.sh too ?

Missing environment variables on running python cronjob in Docker

I'm running a python script inside a docker container using crontab. Also, I set some environment variables (as database host, password, etc.) in .env file in the project's directory. If I run the script manually inside the container (python3 main.py) everything is working properly. But when the script is run by crontab the environment variables are not found (None).
I have the following setup:
Dockerfile
FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get -y install cron
RUN apt-get install -y python3-pip python-dev
WORKDIR /home/me/theservice
COPY . .
RUN chmod 0644 theservice-cron
RUN touch /var/log/theservice-cron.log
RUN chmod +x run.sh
ENTRYPOINT ./run.sh
run.sh
#!/bin/bash
crontab theservice-cron
cron -f
docker-compose.yml
version: '3.7'
services:
theservice:
build: .
env_file:
- ./.env
theservice-cron
HOME=/home/me/theservice
* * * * * python3 /home/me/theservice/main.py >> /var/log/theservice-cron.log 2>&1
#* * * * * cd /home/me/theservice && python3 main.py >> /var/log/theservice-cron.log 2>&1
I assumed that the cronjob is running in another directory and there the environment variables set in /home/me/theservice/.env are not accessible. So I tried to add HOME=/home/me/theservice line in the theservice-cron file or just to execute /home/me/theservice before running the script but it didn't help.
In the python script, I use os to access environment variables
import os
print(os.environ['db_host'])
How I can fix this problem?

I had similar problem.
I did fix it using the following:
CMD printenv > /etc/environment && cron && tail -f /var/log/theservice-cron.log
According to
https://askubuntu.com/questions/700107/why-do-variables-set-in-my-etc-environment-show-up-in-my-cron-environment, cron reads env vars from /etc/enviroment

For those fighting to get ENV variables from docker-compose into docker, simply have a shell script run at ENTRYPOINT in your Dockerfile, with
printenv > /etc/environment
again, the naming of "/etc/environment" is CRUCIAL !
And then in your crontab, have it call a shell script:
* * * * * bash -c "sh /var/www/html/cron_php.sh"
The scripts simply does :
#!/bin/bash
cd /var/www/html
php whatever.php
You will now have the docker-compose environment variables in your php cron application. It took me a full day to figure this out. Hope i save someone's trouble now !
UPDATE:
In Azure Docker (Web app) the mechanism doesn't seem to work. A small tweak is needed:
In the Dockerfile, in the ENTRYPOINT sh script, write a file (and CHMOD to execution rights chmod 770 ) /etc/environments.sh using this command:
eval $(printenv | awk -F= '{print "export " $1"=""""$2""" }' >> /etc/environments.sh)
Then, in your crontab shell where you execute php, do this:
#!/bin/bash
. /etc/environments.sh
php whatever.php
Notice the "." instead of source. Even though the Docker container is Linux using bash, source did not do the trick, the . did work.
Note: In my local Windows Docker the first solution, using /etc/envrionment worked fine. I was baffled to find out that on Azure the second fix was needed.

How do you deploy a flask or django application using jwilder/nginx-proxy?

I'm looking into moving some of our web servers to docker containers. The jwilder/nginx-proxy image looks interesting, and seems to do what we want, but how would one properly deploy a flask application in a container, and have it work with the jwilder/nginx-proxy server? To be clear, the flask application would also be running in a docker container.
In a separate, but related question, how would one do this for a django app?
It looks like there's a popular tiangolo/uwsgi-nginx-flask image, and a similar dockerfiles/django-uwsgi-nginx image. In this setup, from what I understand, the nginx-proxy container would direct traffic to the uwsgi-nginx-flask or django-uwsgi-nginx container. Is this a common way to do this?
The main thought I had was that in such a setup, we're running extra instances of nginx - one for every python/django app. Is this common? Or is it possible/beneficial/common to somehow have the nginx-proxy talk directly to uwsgi within the python app container?
I see that the nginx-proxy image has a VIRTUAL_PROTO=uwsgi option that other containers can be started with. Is this something that can be used to make things more efficient? Or is it more effort than it's worth?
Edit: Or is the nginx instance that accompanies the flask/django project beneficial, since it can be used to serve static content, without which, you would need to configure the nginx-proxy image with the location of every project's static files?

Personally, I prefer to have Django have one container, NGINX in separate container, other applications in other containers etc. For that I prefer to use docker-compose. You can checkout my implementation about using Django + NGINX + PostgreSQL in here.(I have not used jwilder/nginx-proxy, instead I have used official NGINX docker image)
But putting NGINX and Python server in same container does not sound that bad. I have used a lightweight alpine based images for deploying python, for example:
FROM nginx:mainline-alpine
# --- Python Installation ---
RUN apk add --no-cache python3 && \
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi && \
if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi && \
rm -r /root/.cache
# --- Work Directory ---
WORKDIR /usr/src/app
# --- Python Setup ---
ADD . .
RUN pip install -r app/requirements.pip
# --- Nginx Setup ---
COPY config/nginx/default.conf /etc/nginx/conf.d/
RUN chmod g+rwx /var/cache/nginx /var/run /var/log/nginx
RUN chgrp -R root /var/cache/nginx
RUN sed -i.bak 's/^user/#user/' /etc/nginx/nginx.conf
RUN addgroup nginx root
# --- Expose and CMD ---
EXPOSE 5000
CMD gunicorn --bind 0.0.0.0:5000 wsgi --chdir /usr/src/app/app & nginx -g "daemon off;"
Although it looks bit messy, but it works fine. Please checkout my full implementation at here.
Depending on how you want to deploy docker images, you can use either approaches. But using docker compose would be the best solution IMHO. And in both setups, you can use NGINX to serve your static contents(no need to configure it for each static file).

Docker, dockercloud and Cron

I have a dockerfile which automates the building of an image.
I am using the docker cloud, connected to Digital Ocean as the server.
Within my dockerfile, I get the software I need, add the relevant GitHub repository containing the python scripts I wish to run. I then start the cron scheduler and add the script with appropriate times. For example:
The cron_files.txt file looks like this:
0 12 * * * /usr/bin/python /home/dir/run_check.py
0 15 * * * /usr/bin/python /home/dir/run_push.py
In my dockerfile, I do the following:
RUN service cron start
RUN service cron status
RUN crontab -u root cron_files.txt
In the log files, I can see that cron is succesfully started.
Edit, thanks to r0manarmy for this - How to run a cron job inside a docker container?
# Add crontab file in the cron directory
ADD crontab /etc/cron.d/hello-cron
# Give execution rights on the cron job
RUN chmod 0644 /etc/cron.d/hello-cron
# Create the log file to be able to run tail
RUN touch /var/log/cron.log
# Run the command on container startup
CMD cron && tail -f /var/log/cron.log
How do I edit the above to create the crontab file from the cron_files.txt rather than the above example?
I've tried ADD crontab cron_files.txt /etc/cron.d/feeds
But this returns:
ADD crontab cron_files.txt /etc/cron.d/feeds
lstat crontab: no such file or directory
Ps. I am using FROM debian:jessie

You probably want to set cron as the CMD:
In order to do this, just use the crond command on, say, alpine linux.
Take a look at this example for ideas:
https://gist.github.com/mhubig/a01276e17496e9fd6648cf426d9ceeec