Develop Reference

Docker Compose - Python Instances

I need to run the same python code, but with different initiation arguments with docker.
So under the main directory I've setup a folder called docker that contains different folders, each having same docker file but with the different arguments setup. Below is are examples of test_1 and test_2, where test_x is changed between the different folders, as well as test_1 becomes test_2 and so on:
Dockerfile found under docker/test_1 folder
FROM python:3.7
RUN mkdir /app/test_1
WORKDIR /app/test_1
COPY ./env/requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY ../ .
CMD ["python", "main.py","-t","test_1"]
Dockerfile found under docker/test_2 folder
FROM python:3.7
RUN mkdir /app/test_2
WORKDIR /app/test_2
COPY ./env/requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY ../ .
CMD ["python", "main.py","-t","test_2"]
Under the main directory I've setup a docker compose file that initiates the different containers (all running the same code) and that share a txt file in shared_folder:
services:
test_1:
container_name: test_1
build: ./docker/test_1
volumes:
- output:/app/shared_folder
restart: unless-stopped
test_2:
container_name: test_2
build: ./docker/test_2
volumes:
- output:/app/shared_folder
restart: unless-stopped
So my question here with docker, is this the right way to go about it, when setting up multiple py executions of the same code with different parameter? or is there another recommended approach. Do want to mention, they need to share the file in shared_folder, that's a requirement and all the instances have read/write access to the same file in the shared_folder (this is a must have).

It is very easy to override the Dockerfile CMD with a docker run command-line argument or Compose command:. So, I would build only one image, and I would give it a useful default CMD.
FROM python:3.7
WORKDIR /app
COPY ./env/requirements.txt ./
RUN pip install -r requirements.txt
COPY ./ ./
CMD ["./main.py"]
(Make sure your script is executable – maybe run chmod +x main.py on the host – and begins with a "shebang" line #!/usr/bin/env python3, so you don't have to explicitly name the interpreter.)
Now in your docker-compose.yml file, have both services build: the same image. You'll technically get two images out in the docker images output but they will have the same image ID and the second image build will run extremely quickly (it will come entirely from the layer cache). Use Compose command: to override the entire CMD as required.
version: '3.8'
services:
test_1:
build: .
command: ./main.py -t test_1
volumes:
- output:/app/shared_folder
restart: unless-stopped
test_2:
build: .
command: ./main.py -t test_2
volumes:
- output:/app/shared_folder
restart: unless-stopped
You could also manually run this outside of Compose if you just wanted to validate things, with the same approach
docker build -t myapp .
docker run --rm myapp \
./main.py --help
With this approach you do not need to rebuild the image for each different command you want to run or wrangle with the syntactic complexities of docker run --entrypoint.

First delete CMD ["python", "main.py","-t","test_2"] in Dockerfile and instead add entrypoint in docker-compose.yaml would be a better way to build the image for the code are all the same. if you have more container to start, it will save you a lot of time.
About the question you asked if the shared_folder you want to share is a read-only file, it is OK, if not, for instance, log files you want to put in it from instance out to the host, you should be careful about the log file name, should not be the same in the two containers.

I would definitely DRY it, use a single Dockefile and use an ARG to build them.
Here is what you could do:
In docker/Dockerfile:
FROM python:3.7
ARG FOLDER
## We need to duplicate the value of the ARG in an ENV
## because the arguments are only visible through the build
## so, it won't be accessible to our command
ENV FOLDER=$FOLDER
RUN mkdir -p /app/$FOLDER
WORKDIR /app/$FOLDER
COPY ./$FOLDER/env/requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
CMD ["sh", "-c", "python main.py -t $FOLDER"]
And in your docker-compose.yml define those build arguments:
version: "3.9"
services:
test1:
container_name: test_1
build:
context: ./docker
args:
FOLDER: test1
volumes:
- output:/app/shared_folder
restart: unless-stopped
test2:
container_name: test_2
build:
context: ./docker
args:
FOLDER: test2
volumes:
- output:/app/shared_folder
restart: unless-stopped

Ho to run python script inside docker and save results locally? [duplicate]

I built a docker image that inputs a local file, does some stuff to it, and returns an output file saved locally, but it does not work. How do I allow local, user input for files and then saving the output on the local machine?
My Dockerfile looks like this:
FROM python:3
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
EXPOSE 5000
CMD [ "python", "process.py" ]
Ideally, the terminal command would be something like this:
docker run -p 5000:5000 [name of docker] [local path to input file] [local path to save output file]
When I run, I get this error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"../test.flac\": stat ../test.flac: no such file or directory": unknown.
How can I do this?

Generally, the docker container cannot break out into the host machine.
However, you can mount a local directory from the host machine into the container. The files created in the mount point, inside the container, will also be visible on the host machine.
In the example below I am mounting the working directory from the host machine inside the container. My current directory contains an input-file.
The container cats the content of the input-file and appends it to the output-file
// The initial wiorking directory content
.
└── input-file
// Run my dummy container and ask it to cat the content of the input file into the output file
docker run -v $(pwd):/root/some-path ubuntu /bin/bash -c "cat /root/some-path/input-file >> /root/some-path/output-file"
// The outcome
.
├── input-file
└── output-file

Reading an HDF file outside a Docker container from a Python script inside a container

I have a Python script, python_script.py, that reads an HDF5 file, hdf_file.h5, on my local machine. The directory path to the files is
folder1
folder2
python_script.py
hdf_file.h5
I have the following sample code:
from pandas import read_hdf
df = read_hdf('hdf_file.h5')
When I run this code on my local machine, it works fine.
However, I need to place the Python script inside a Docker container, keep the HDF file out of the container, and have the code read the file. I want to have something like the following directory path for the container:
folder1
folder2
hdf_file.h5
docker-folder
python_script.py
requirements.txt
Dockerfile
I use the following Dockerfile:
FROM python:3
WORKDIR /project
COPY ./requirements.txt /project/requirements.txt
RUN pip install -r requirements.txt
COPY . /project
CMD [ "python", "python_script.py" ]
I am new to Docker and am having a lot of trouble figuring out how to get a Python script inside a container to read a file outside the container. What commands do I use or code changes do I make to be able to do this?

It seems you need to use docker volumes (https://docs.docker.com/storage/volumes/).
Try the following:
docker run -v path/where/lives/hdf5/:path/to/your/project/folder/your_image your_docker_image:your_tag
Where the first part before the : refers to host machine and after, the container.
Hope it helps!

Exporting files from docker volume to another directory

I have a python code which reads data from the file and do some calculation and save the result to the output file. The code also saves the logs in log file. So in my current directory, I have below files:
1. code.py --> The main python application
2. input.json --> This json file is used to take input data
3. output.json --> The output data is saved in this file.
4. logfile.log --> This file saves the log.
All the above file is inside the directory Application. Full path is /home/user/Projects/Application/. Now when I am running the code.py I am getting the expected results. So I converted the above code into docker by using below Dockerfile:
FROM python:3
ADD code.py /
ADD input.json /
ADD output.json /
ADD logfile.log /
CMD [ "python3", "./code.py" ]
When I am running the docker container, it is running fine but I cannot see the output data and logs in output.json and logfile.log. Then I searched for these file in the file system and found these files in below directory:
/var/lib/docker/overlay2/7c237c143f9f2e711832daccecdfb29abaf1e37a4714f34f34870e0ee4b1af07/diff/home/user/Projects/Application/
and all my files were in that directory. I checked for the logs and the data, it was there. Then I understood that all the files will be saved inside the docker volumes and not in our current directory.
Is there any way I can keep the files and all the data in my current directory /home/user/Projects/Application/ instead of docker because in this way it will be easy for me to check the outputs.
Thanks

The files are located under the docker overlay volume because you didn’t do volume mounting. To overcome this, you can modify your Dockerfile to look similar to this:
FROM python:3
RUN mkdir /app
ADD code.py /app
ADD input.json /app
ADD output.json /app
ADD logfile.log /app
WORKDIR /app
VOLUME /app
CMD [ "python3", "./code.py" ]
Then in your docker run command, make sure you pass this option:
-v /home/user/Projects/Application:/app
More information about container options can be found at https://www.aquasec.com/wiki/display/containers/Docker+Containers.
If you are using docker compose, you need to add:
volumes:
- /home/user/Projects/Application: /var/www/app

You may try to run your container as below: [you may not need to build your image]
docker run --rm -v /home/user/Projects/Application/:/home/user/Projects/Application/ -d python:3 /home/user/Projects/Application/code.py
-v ; bind mount local folder into your container at /home/user/Projects/Application/.
Feel free to take out --rm if you don't need that.
Please make sure code.py write logs to /home/user/Projects/Application/logfile.log
To verify files and folders are there are not by running command:
docker run --rm -v /home/user/Projects/Application/:/home/user/Projects/Application/ -d python:3 sh
This will drop you in a terminal, you can list files and make sure required files and configs are there.

Execute Python script inside a given docker-compose container

I have made a little python script to create a DB and some tables inside a RethinkDB
But now I'm trying to launch this python script inside my rethink container launched with docker-compose.
This is my docker-compose.yml rethink container config
# Rethink DB
rethink:
image: rethinkdb:latest
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015
I'm trying to execute the script with after launching my container
docker exec -it rethink python src/app/db-install.py
But I get this error
rpc error: code = 2 desc = oci runtime error: exec failed: exec: "python": executable file not found in $PATH
Python is not found in me container. Is this possible to execute a python script inside a given container with docker-compose or with docker exec ?

First find out if you have python executable in the container:
docker exec -it rethink which python
If it exists, Use the absolute path provided by which command in previous step:
docker exec -it rethink /absolute/path/to/python src/app/db-install.py
If not, you can convert your python script to bash script, so you can run it without extra executables and libraries.
Or you can create a dockerfile, use base image, and install python.
dockerfile:
FROM rethinkdb:latest
RUN apt-get update && apt-get install -y python
Docker Compose file:
rethink:
build : .
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015

Docker-compose
Assuming that python is installed, try:
docker-compose run --rm MY_DOCKER_COMPOSE_SERVICE MY_PYTHON_COMMAND
For a start, you might also just go into the shell at first and run a python script from the command prompt.
docker-compose run --rm MY_DOCKER_COMPOSE_SERVICE bash
In your case, MY_DOCKER_COMPOSE_SERVICE is 'rethink', and that is not the container name here, but the name of the service (first line rethink:), and only the service is run with docker-compose run, not the container.
The MY_PYTHON_COMMAND is, in your case of Python2, python src/app/db-install.py, but in Python3 it is python -m src/app/db-install (without the ".py"), or, if you have Python3 and Python2 installed, python3 -m src/app/db-install.
Dockerfile
To be able to run this python command, the Python file needs to be in the container. Therefore, in your Dockerfile that you need to call with build: ., you need to copy your build directory to a directory in the container of your choice
COPY $PROJECT_PATH /tmp
This /tmp will be created in your build directory. If you just write ".", you do not have any subfolder and save it directly in the build directory.
When using /tmp as the subfolder, you might write at the end of your Dockerfile:
WORKDIR /tmp
Docker-compose
Or if you do not change the WORKDIR from the build (".") context to /tmp and you still want to reach /tmp, run your Python file like /tmp/db-install.py.

The rethinkdb image is based on the debian:jessie image :
https://github.com/rethinkdb/rethinkdb-dockerfiles/blob/da98484fc73485fe7780546903d01dcbcd931673/jessie/2.3.5/Dockerfile
The debian:jessie image does not come with python installed.
So you will need to create your own Dockerfile, something like :
FROM rethinkdb:latest
RUN apt-get update && apt-get install -y python
Then change your docker-compose :
# Rethink DB
rethink:
build : .
container_name: rethink
ports:
- 58080:8080
- 58015:28015
- 59015:29015
build : . is the path to your Dockerfile.