Exporting files from docker volume to another directory - python

I have a python code which reads data from the file and do some calculation and save the result to the output file. The code also saves the logs in log file. So in my current directory, I have below files:
1. code.py --> The main python application
2. input.json --> This json file is used to take input data
3. output.json --> The output data is saved in this file.
4. logfile.log --> This file saves the log.
All the above file is inside the directory Application. Full path is /home/user/Projects/Application/. Now when I am running the code.py I am getting the expected results. So I converted the above code into docker by using below Dockerfile:
FROM python:3
ADD code.py /
ADD input.json /
ADD output.json /
ADD logfile.log /
CMD [ "python3", "./code.py" ]
When I am running the docker container, it is running fine but I cannot see the output data and logs in output.json and logfile.log. Then I searched for these file in the file system and found these files in below directory:
/var/lib/docker/overlay2/7c237c143f9f2e711832daccecdfb29abaf1e37a4714f34f34870e0ee4b1af07/diff/home/user/Projects/Application/
and all my files were in that directory. I checked for the logs and the data, it was there. Then I understood that all the files will be saved inside the docker volumes and not in our current directory.
Is there any way I can keep the files and all the data in my current directory /home/user/Projects/Application/ instead of docker because in this way it will be easy for me to check the outputs.
Thanks

The files are located under the docker overlay volume because you didn’t do volume mounting. To overcome this, you can modify your Dockerfile to look similar to this:
FROM python:3
RUN mkdir /app
ADD code.py /app
ADD input.json /app
ADD output.json /app
ADD logfile.log /app
WORKDIR /app
VOLUME /app
CMD [ "python3", "./code.py" ]
Then in your docker run command, make sure you pass this option:
-v /home/user/Projects/Application:/app
More information about container options can be found at https://www.aquasec.com/wiki/display/containers/Docker+Containers.
If you are using docker compose, you need to add:
volumes:
- /home/user/Projects/Application: /var/www/app

You may try to run your container as below: [you may not need to build your image]
docker run --rm -v /home/user/Projects/Application/:/home/user/Projects/Application/ -d python:3 /home/user/Projects/Application/code.py
-v ; bind mount local folder into your container at /home/user/Projects/Application/.
Feel free to take out --rm if you don't need that.
Please make sure code.py write logs to /home/user/Projects/Application/logfile.log
To verify files and folders are there are not by running command:
docker run --rm -v /home/user/Projects/Application/:/home/user/Projects/Application/ -d python:3 sh
This will drop you in a terminal, you can list files and make sure required files and configs are there.

Related

Running .env files within a docker container

I have been struggling to add env variables into my container for the past 3 hrs :( I have looked through the docker run docs but haven't managed to get it to work.
I have built my image using docker build -t sellers_json_analysis . which works fine.
I then go to run it with: docker run -d --env-file ./env sellers_json_analysis
As per the docs: $ docker run --env-file ./env.list ubuntu bash but I get the following error:
docker: open ./env: no such file or directory.
The .env file is in my root directory
But when running docker run --help I am unable to find anything about env variables, but it doesn't provide the following:
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
So not sure I am placing things incorrectly. I could add my variables into the dockerfile but I want to keep it as a public repo as it's a project I would like to display.
Your problem is wrong path, either use .env or ./.env, when you use ./env it mean a file named env in current directory
docker run -d --env-file .env sellers_json_analysis

Ho to run python script inside docker and save results locally? [duplicate]

I built a docker image that inputs a local file, does some stuff to it, and returns an output file saved locally, but it does not work. How do I allow local, user input for files and then saving the output on the local machine?
My Dockerfile looks like this:
FROM python:3
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . /app
EXPOSE 5000
CMD [ "python", "process.py" ]
Ideally, the terminal command would be something like this:
docker run -p 5000:5000 [name of docker] [local path to input file] [local path to save output file]
When I run, I get this error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"../test.flac\": stat ../test.flac: no such file or directory": unknown.
How can I do this?
Generally, the docker container cannot break out into the host machine.
However, you can mount a local directory from the host machine into the container. The files created in the mount point, inside the container, will also be visible on the host machine.
In the example below I am mounting the working directory from the host machine inside the container. My current directory contains an input-file.
The container cats the content of the input-file and appends it to the output-file
// The initial wiorking directory content
.
└── input-file
// Run my dummy container and ask it to cat the content of the input file into the output file
docker run -v $(pwd):/root/some-path ubuntu /bin/bash -c "cat /root/some-path/input-file >> /root/some-path/output-file"
// The outcome
.
├── input-file
└── output-file

Reading an HDF file outside a Docker container from a Python script inside a container

I have a Python script, python_script.py, that reads an HDF5 file, hdf_file.h5, on my local machine. The directory path to the files is
folder1
folder2
python_script.py
hdf_file.h5
I have the following sample code:
from pandas import read_hdf
df = read_hdf('hdf_file.h5')
When I run this code on my local machine, it works fine.
However, I need to place the Python script inside a Docker container, keep the HDF file out of the container, and have the code read the file. I want to have something like the following directory path for the container:
folder1
folder2
hdf_file.h5
docker-folder
python_script.py
requirements.txt
Dockerfile
I use the following Dockerfile:
FROM python:3
WORKDIR /project
COPY ./requirements.txt /project/requirements.txt
RUN pip install -r requirements.txt
COPY . /project
CMD [ "python", "python_script.py" ]
I am new to Docker and am having a lot of trouble figuring out how to get a Python script inside a container to read a file outside the container. What commands do I use or code changes do I make to be able to do this?
It seems you need to use docker volumes (https://docs.docker.com/storage/volumes/).
Try the following:
docker run -v path/where/lives/hdf5/:path/to/your/project/folder/your_image your_docker_image:your_tag
Where the first part before the : refers to host machine and after, the container.
Hope it helps!

can't open file 'app.py': [Errno 2] No such file or directory

Trying to get started with python & Docker, managed to get Docker working with a PHP example but struggling to get it working with a python file.
I am trying to get a simple hello world docker container running to simply print "Hello world"
Dockerfile
FROM python:2.7
WORKDIR /app
COPY . /app
EXPOSE 80
CMD [ "python", "app.py" ]
app.py
def hello():
return print("hello world")
hello()
I have run the following commands
docker build -t test .
docker run -p 80:80 test
Expected result: containerised app running on port 80
Actual result:python: can't open file './app.py': [Errno 2] No such file or directory
According to your comment, the directory structure seems to be
Dockerfile
src/
app.py
In this case, the CMD statement in your Dockerfile should be:
CMD [ "python", "src/app.py" ]
This is because of your app.py residing inside src folder with respect to the mentioned WORKDIR.
WORKDIR makes any subsequent commands run within that work dir you specified.
So, the next line COPY . /app will copy files to this directory ./app/app not ./app

How to access generated files inside Docker image

So basically I have a python script that will write to a file once it is done running. How do I access this file? My end goal is to run the docker image on jenkins and then read the xml file that the python script generates.
FROM python:3
ADD WebChecker.py /
ADD requirements.txt /
ADD sites.csv /
RUN pip install -r requirements.txt
CMD [ "python", "./WebChecker.py" ]
That is my Dockerfile. I have a print("Finished") in there and it is printing so that means everything is working fine. It's just now I need to see my output.xml file.
You should have done it now by following above comments. In case if you still stuck, you may give a try as below:
Build:
docker build -t some_tag_name_to_your_image .
After build is completed, you may run a container and get the xml file as below:
1. Write output file to bind volume
Run your container as below:
docker run -d --rm --name my_container \
-v ${WORKSPACE}:/path/to/xml/file/in/container \
some_tag_name_to_your_image
Once the xml file generated, that will be available at the Jenkins-host:${WORKSPACE}
Notes:
${WORKSPACE} is an env variable set by Jenkins. Read more env-vars here
Read more about bind mount here

Categories

Resources