Develop Reference

run two python scripts with docker compose

My folder structure looked like this:
My Dockerfile looked like this:
FROM python:3.8-slim-buster
WORKDIR /src
COPY src/requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
CMD [ "python", "main.py"]
When I ran these commands:
docker build --tag FinTechExplained_Python_Docker .
docker run free
my main.pyfile ran and gave the correct print statements as well. Now, I have added another file tests.py in the src folder. I want to run the tests.py first and then main.py.
I tried modifying the cmdwithin my docker file like this:
CMD [ "python", "test.py"] && [ "python", "main.py"]
but then it gives me the print statements from only the first test.pyfile.
I read about docker-compose and added this docker-compose.yml file to the root folder:
version: '3'
services:
main:
image: free
command: >
/bin/sh -c 'python tests.py'
main:
image: free
command: >
/bin/sh -c 'python main.py'
then I changed my docker file by removing the cmd:
FROM python:3.8-slim-buster
WORKDIR /src
COPY src/requirements.txt requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
Then I ran the following commands:
docker compose build
docker compose run tests
docker compose run main
When I run these commands separately, I get the correct print statements for both testsand main. However, I am not sure if I am using docker-composecorrectly or not.
Am I supposed to run both scripts separately? Or is there a way to run one after another using a single docker command?
How is my Dockerfile supposed to look like if I am running the python scripts from the docker-compose.yml instead?
Edit:
Ideally looking for solutions based on docker-compose

In the Bourne shell, in general, you can run two commands in sequence by putting && between them. It sounds like you're already aware of this.
# without Docker, at a normal shell prompt
python test.py && python main.py
The Dockerfile CMD has two syntactic forms. The JSON-array form does not run a shell, and so it is slightly more efficient and has slightly more consistent escaping rules. If it's not a JSON array then Docker automatically runs it via a shell. So for your use you can use the shell form:
CMD python test.py && python main.py
In comments to other answers you ask about providing this as an override in the docker-compose.yml file. Compose will not normally run a shell for you, so you need to explicitly specify it as part of the command: override.
command: /bin/sh -c 'python test.py && python main.py'
Your Dockerfile should generally specify a CMD and the docker-compose.yml often will not include a command:. This makes it easier to run the image in other contexts (via docker run without Compose; in Kubernetes) since you won't have to retype the command every different way you want to run the container. The entrypoint wrapper pattern highlighted in #sytech's answer is very useful in general and it's easy to add to a container that uses a CMD without an ENTRYPOINT; but it requires the Dockerfile to use CMD as a normal well-formed shell command.

You have to change CMD to ENTRYPOINT. And run the 1st script as daemon in the background using &.
ENTRYPOINT ["/docker_entrypoint.sh"]
docker_entrypoint.sh
#!/bin/bash
set -e
exec python tests.py &
exec python main.py

In general, it is a good rule of thumb that a container should only a single process and that essential process should be pid 1
Using an entrypoint can help you do multiple things at runtime and optionally run user-defined commands using exec, as according to the best practices guide.
For example, if you always want the tests to run whenever the container starts, then execute the defined command in CMD.
First, create an entrypoint script (be sure to make it executable with chmod +x):
#!/usr/bin/env bash
# always run tests first
python /src/tests.py
# then run user-defined command
exec "$#"
Then configure the dockerfile to copy the script and set it as the entrypoint:
#...
COPY entrypoint.sh /docker-entrypoint.sh
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["python", "main.py"]
Then when you build an image from this dockerfile and run it, the entrypoint will first execute the tests then run the command to run main.py
The command can also still be overridden by the user when running the image like docker run ... myimage <new command> which will still result in the entrypoint tests being executed, but the user can change the command being run.

You can achieve this by creating a bash script(let's name entrypoint.sh) which is containing the python commands. If you want, you can create background processes of those.
#!/usr/bin/env bash
set -e
python tests.py
python main.py
Edit your docker file as follows:
FROM python:3.8-slim-buster
# Create workDir
RUN mkdir code
WORKDIR code
ENV PYTHONPATH = /code
#upgrade pip if you like here
COPY requirements.txt .
RUN pip install -r requirements.txt
# Copy Code
COPY . .
RUN chmod +x entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]
In the docker compose file, add the following line to the service.
entrypoint: [ "./entrypoint.sh" ]

Have you try this in your docker-compose.yaml?
version: '3'
services:
main:
image: free
command: >
/bin/sh -c 'python3 tests.py & && python3 main.py &'
both will run in the background
then run in terminal
docker-compose up --build

How to pass argument to docker container when running docker-compose up

I have the following Dockerfile:
FROM python:3.6
ADD . /
RUN pip install -r requirements.txt
ENTRYPOINT ["python3.6", "./main.py"]
Where main.py takes in run time arguments that are parsed using argparse like so:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('city')
args = parser.parse_args()
print(args.city)
I have succesfully built a docker image from my Dockerfile and run a container based on this image using:
docker run -it my-docker-image "los angeles"
This works great and argparse in python receives the param "los angeles" as the city in my python code snippet.
I am now trying to accomplish the equivalent of
docker run -it my-docker-image "los angeles"
But with a docker-compose.yml file. Right now my docker-compose.yml file looks like:
version: '3'
services:
data-worker:
image: url-to-my-docker-container-on-ecr-that-i-have-pulled-onto-my-local-machine
container_name: run-for-city
volumes:
- ~/.aws/:/root/.aws
I then try running:
docker-compose up
and it gets started but fails saying:
run-for-city | usage: main.py [-h] city
run-for-city | main.py: error: the following arguments are required: city
run-for-city exited with code 2
which makes perfect sense as i need to pass the city param when running docker-compose up but don't know exactly how to.
i tried:
docker-compose up "los angeles"
but that did not work.
Do i need to add something to my docker-compose.yml and or an argument to the docker-compose up command or what?

In Docker terminology, the extra parameter you're passing is a "command". (Anything after the image name in docker run is the command; conventionally it's an actual command to run, but if you have an ENTRYPOINT in your Dockerfile, it's passed as arguments to the entrypoint.) You can replicate this in your docker-compose.yml file:
version: '3'
services:
data-worker:
image: 123456789012.dkr...
volumes:
- ~/.aws/:/root/.aws
command: ["los angeles"] # <--
If you need to pass this at the command line, the only real way to do it is via an environment variable. Compose will substitute environment variables (note, somewhat limited syntax) and I believe it will work to say
command: ["$CITY"]
and you'd launch this with
CITY="los angeles" docker-compose up

Docker: unable to test a postgresql connection

I'm following a Python/TDD/Docker tutorial by TestDriven.io.
I build a custom image and I want to test it. I cannot (I think, I'm a noob with Docker and Python, please patience) do.
This is the image: registry.gitlab.com/sineverba/warehouse:latest. It works because I deployed to Heroku with success.
I don't want docker-compose for testing the final image, so I tried to do:
docker network create -d bridge flask-tdd-net
export DATABASE_TEST_URL=postgres://postgres:postgres#flask-tdd-net:5432/users_dev
docker run -d --name app -e "PORT=8765" -p 5002:8765 --network=flask-tdd-net registry.gitlab.com/sineverba/warehouse:latest
docker run -d --name db -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=users_dev -p 5432:5432 --network=flask-tdd-net postgres:12-alpine
I can launch a simple
docker exec app python -V and get version, for example.
But when I launch
docker exec app python -m pytest "project/tests"
I get (split down, full log here: https://pastebin.com/tYjn65ys)
self = <[AttributeError("'NoneType' object has no attribute 'drivername'") raised in repr()] SQLAlchemy object at 0x7fc74676e7f0>
app = <Flask 'project'>, sa_url = None, options = {}
def apply_driver_hacks(self, app, sa_url, options):
"""This method is called before engine creation and used to inject
driver specific hacks into the options. The `options` parameter is
a dictionary of keyword arguments that will then be used to call
the :func:`sqlalchemy.create_engine` function.
The default implementation provides some saner defaults for things
like pool sizes for MySQL and sqlite. Also it injects the setting of
`SQLALCHEMY_NATIVE_UNICODE`.
"""
> if sa_url.drivername.startswith('mysql'):
E AttributeError: 'NoneType' object has no attribute 'drivername'
I did try also (after stopping and removing containers and recreating DBs)
export DATABASE_TEST_URL=postgres://postgres:postgres#db:5432/users
So, moving name from users_dev to users.
Full repo link: https://github.com/sineverba/flask-tdd-docker/tree/add-gitlab-warehouse
Thank you in advance!
Edit
I changed the env cause link db was wrong. These are new commands, but got same error. I tried also to export both env, without success.
docker network create -d bridge flask-tdd-net
export DATABASE_TEST_URL=postgres://postgres:postgres#db:5432/users
export DATABASE_URL=postgres://postgres:postgres#db:5432/users
docker run -d --name app -e "PORT=8765" -p 5002:8765 --network=flask-tdd-net registry.gitlab.com/sineverba/warehouse:latest
docker run -d --name db -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=users -p 5432:5432 --network=flask-tdd-net postgres:12-alpine
docker exec app python -m pytest "project/tests"
docker container stop app && docker container rm app && docker container stop db && docker container rm db
Starting example
This is the testdriven.io example, from Gitlab integration (that I want not use). Only env exported for app is the DATABASE_TEST_URL
image: docker:stable
stages:
- build
- test
variables:
IMAGE: ${CI_REGISTRY}/${CI_PROJECT_NAMESPACE}/${CI_PROJECT_NAME}
build:
stage: build
services:
- docker:dind
variables:
DOCKER_DRIVER: overlay2
script:
- docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
- docker pull $IMAGE:latest || true
- docker build
--cache-from $IMAGE:latest
--tag $IMAGE:latest
--file ./Dockerfile.prod
"."
- docker push $IMAGE:latest
test:
stage: test
image: $IMAGE:latest
services:
- postgres:latest
variables:
POSTGRES_DB: users
POSTGRES_USER: runner
POSTGRES_PASSWORD: runner
DATABASE_TEST_URL: postgres://runner:runner#postgres:5432/users
script:
- pytest "project/tests" -p no:warnings
- flake8 project
- black project --check
- isort project/**/*.py --check-only
Solved
The error is the need to export the variables inside the docker command:
docker run -d --name app -e "PORT=8765" -p 5002:8765 -e "DATABASE_TEST_URL=postgres://postgres:postgres#db:5432/users" --network=flask-tdd-net registry.gitlab.com/sineverba/warehouse:latest

The error is the need to export the variables inside the docker command:
docker run -d --name app -e "PORT=8765" -p 5002:8765 -e "DATABASE_TEST_URL=postgres://postgres:postgres#db:5432/users" --network=flask-tdd-net registry.gitlab.com/sineverba/warehouse:latest

How to run different python scripts in parallel containers from same image

I am trying to do something very simple (I think), I want do build a docker image and launch two different scripts from the same image in parallel running containers.
Something as simple as
Container 1 -> print("Hello")
Container 2 -> print('World")
I did some research but some techniques seem a little over engineered and others do something like
CMD ["python", "script.py"] && ["python", "script2.py"]
which, isn't what I'm looking for.
I would love to see something like
$ docker ps
CONTAINER ID IMAGE CREATED STATUS NAMES
a7e789711e62 67759a80360c 12 hours ago Up 2 minutes MyContainer1
87ae9c5c3f84 67759a80360c 12 hours ago Up About a minute MyContainer2
But running two different scripts.
I'm still fairly new to all of this, so if this is a foolish question, I apologize in advance and thank you all for working with me.

You can do this easily using Docker Compose. Here is a simple docker-compose.yml file just to show the idea:
version: '3'
services:
app1:
image: alpine
command: >
/bin/sh -c 'while true; do echo "pre-processing"; sleep 1; done'
app2:
image: alpine
command: >
/bin/sh -c 'while true; do echo "post-processing"; sleep 1; done'
As you see both services use the same image, alpine in this example, but differ in commands they run. Execute docker-compose up and see the output:
app1_1 | pre-processing
app2_1 | post-processing
app2_1 | post-processing
app2_1 | post-processing
app1_1 | pre-processing
app2_1 | post-processing
app1_1 | pre-processing
app2_1 | post-processing
...
In your case, you just change the image to your image, say myimage:v1, and change the service commands so that the first service definition will have command: python script1.py line and the second one command: python script2.py.

If you are set on using the same image then I would suggest you set the ENTRYPOINT to python and then use docker run command to start the containers by providing the scrips as the CMD like so:
Dockerfile:
FROM python
...
ENTRYPOINT ["python"]
And the use the docker run command like so
docker run -d my_image script.py && docker run -d my_image script2.py
Which would start two containers each running separate scripts
BUT - I like to keep the images clean in terms of not having any additional scripts or packages that are not necessary for my service to work, so in this case I would simply create two separate images each having one of the scripts and then run them similarly.
Example:
FROM python
COPY script.py script.py
ENTRYPOINT ["python"]
CMD ["script.py"]
And the second image:
FROM python
COPY script2.py script2.py
ENTRYPOINT ["python"]
CMD ["script2.py"]
And then just build them as separate images and run the the same way as before

Any command you put at the end of the docker run command (or the Docker Compose command: field) replaces the CMD in the Dockerfile. I would suggest still putting in some useful default CMD, but you can always just
docker run --name hello myimage python script.py
docker run --name world myimage python script2.py

Try these steps, it should work.
Create a Dockerfile with contents:
FROM python:3.7-alpine
COPY script1.py /script1.py
COPY script2.py /script2.py
CMD ["/bin/sh"]
In script1.py
print("Hello")
In script2.py
print("World")
Build docker image docker build -t myimage:v1 .
Run the containers
$ docker run -it --rm --entrypoint python myimage:v1 /script1.py
Hello
$
$ docker run -it --rm --entrypoint python myimage:v1 /script2.py
World
$
NOTE: Here we're using same docker image myimage:v1 and just changing the entrypoint in every docker run command.
More info here.
Hope this helps.

Execute Python script inside a given docker-compose container

I have made a little python script to create a DB and some tables inside a RethinkDB
But now I'm trying to launch this python script inside my rethink container launched with docker-compose.
This is my docker-compose.yml rethink container config
# Rethink DB
rethink:
image: rethinkdb:latest
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015
I'm trying to execute the script with after launching my container
docker exec -it rethink python src/app/db-install.py
But I get this error
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "python": executable file not found in $PATH
Python is not found in me container. Is this possible to execute a python script inside a given container with docker-compose or with docker exec ?

First find out if you have python executable in the container:
docker exec -it rethink which python
If it exists, Use the absolute path provided by which command in previous step:
docker exec -it rethink /absolute/path/to/python src/app/db-install.py
If not, you can convert your python script to bash script, so you can run it without extra executables and libraries.
Or you can create a dockerfile, use base image, and install python.
dockerfile:
FROM rethinkdb:latest
RUN apt-get update && apt-get install -y python
Docker Compose file:
rethink:
build : .
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015

Docker-compose
Assuming that python is installed, try:
docker-compose run --rm MY_DOCKER_COMPOSE_SERVICE MY_PYTHON_COMMAND
For a start, you might also just go into the shell at first and run a python script from the command prompt.
docker-compose run --rm MY_DOCKER_COMPOSE_SERVICE bash
In your case, MY_DOCKER_COMPOSE_SERVICE is 'rethink', and that is not the container name here, but the name of the service (first line rethink:), and only the service is run with docker-compose run, not the container.
The MY_PYTHON_COMMAND is, in your case of Python2, python src/app/db-install.py, but in Python3 it is python -m src/app/db-install (without the ".py"), or, if you have Python3 and Python2 installed, python3 -m src/app/db-install.
Dockerfile
To be able to run this python command, the Python file needs to be in the container. Therefore, in your Dockerfile that you need to call with build: ., you need to copy your build directory to a directory in the container of your choice
COPY $PROJECT_PATH /tmp
This /tmp will be created in your build directory. If you just write ".", you do not have any subfolder and save it directly in the build directory.
When using /tmp as the subfolder, you might write at the end of your Dockerfile:
WORKDIR /tmp
Docker-compose
Or if you do not change the WORKDIR from the build (".") context to /tmp and you still want to reach /tmp, run your Python file like /tmp/db-install.py.

The rethinkdb image is based on the debian:jessie image :
https://github.com/rethinkdb/rethinkdb-dockerfiles/blob/da98484fc73485fe7780546903d01dcbcd931673/jessie/2.3.5/Dockerfile
The debian:jessie image does not come with python installed.
So you will need to create your own Dockerfile, something like :
FROM rethinkdb:latest
RUN apt-get update && apt-get install -y python
Then change your docker-compose :
# Rethink DB
rethink:
build : .
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015
build : . is the path to your Dockerfile.