Setup
Project
src
main.py
Dockerfile
Dockerfile (raw, needs to be revamped)
FROM python:3
ADD src/main.py /
RUN chmod +x main.py
RUN /usr/local/bin/python -m pip install --upgrade pip
COPY . /opt/app
RUN pip install -r /opt/app/requirements.txt
ADD / /usr/local
ENTRYPOINT [ "python", "./main.py" ]
main.py
if __name__ == '__main__':
if len(sys.argv) == 2:
main(sys.argv[1])
def main(logs_file_archive):
unzip_logs(logs_file_archive) # unzips all logs from the provided path to the folder with the same name as archive/Extracted directory
create_csv_files() # creates CSV files needed to plot graph
process_files() # populate CSV files generated with the right data
plot(logs_file_archive) # build this data representation
Actual/desired behaviour
Actual:
2022-01-17T22:05:31.547047838Z File "//./main.py", line 214, in <module>
2022-01-17T22:05:31.547210046Z main(sys.argv[1])
2022-01-17T22:05:31.547259438Z File "//./main.py", line 187, in main
2022-01-17T22:05:31.547670294Z unzip_logs(logs_file_archive)
2022-01-17T22:05:31.547732344Z File "//./main.py", line 54, in unzip_logs
2022-01-17T22:05:31.548296998Z with zipfile.ZipFile(file_path, "r") as zip_ref:
2022-01-17T22:05:31.548350898Z File "/usr/local/lib/python3.10/zipfile.py", line 1240, in __init__
2022-01-17T22:05:31.549638566Z self.fp = io.open(file, filemode)
2022-01-17T22:05:31.549692977Z FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/PerfReader/misc/archive.zip'
No such file or directory: '/Users/user/PerfReader/misc/archive.zip' is expected well... because there is no such file in the Docker machine.
Desired: container runs using Dockerfile, data processes, the plot is displayed real-time or saved as a file and transferred to host
Question/issue description
I am not entirely sure how to transfer the file specified to the Docker container. I read https://docs.docker.com/storage/volumes/ but it doesn't provide any examples so I seek examples of how volumes can be mounted.
Provided that my plot() in main.py does plot data properly, what are my options on displaying this data (output of the whole exercise is the plot)? Can I display the plot in real-time from Docker? Or is my only option is to generate a plot and then transfer it back to the host machine using matplotlib.pyplot.savefig?
First Question:
There are multiple ways to access the host files. You can use the copy or add commands like what you are doing in your Docker file and the file will be copied during building the image. Furthermore, you can use the binding mount, it allows you to access the host directory that was binded, something like you are running the container in this directory.
Second Question:
Docker doesn't support GUI or accessing the host display. However, you can allow it to do so using Xauth. Consider the following link for the steps.
Either way, I don't encourage using Docker for what you are doing especially the plotting part, python virtual environment would be more than enough.
Related
I have a Python script, python_script.py, that reads an HDF5 file, hdf_file.h5, on my local machine. The directory path to the files is
folder1
folder2
python_script.py
hdf_file.h5
I have the following sample code:
from pandas import read_hdf
df = read_hdf('hdf_file.h5')
When I run this code on my local machine, it works fine.
However, I need to place the Python script inside a Docker container, keep the HDF file out of the container, and have the code read the file. I want to have something like the following directory path for the container:
folder1
folder2
hdf_file.h5
docker-folder
python_script.py
requirements.txt
Dockerfile
I use the following Dockerfile:
FROM python:3
WORKDIR /project
COPY ./requirements.txt /project/requirements.txt
RUN pip install -r requirements.txt
COPY . /project
CMD [ "python", "python_script.py" ]
I am new to Docker and am having a lot of trouble figuring out how to get a Python script inside a container to read a file outside the container. What commands do I use or code changes do I make to be able to do this?
It seems you need to use docker volumes (https://docs.docker.com/storage/volumes/).
Try the following:
docker run -v path/where/lives/hdf5/:path/to/your/project/folder/your_image your_docker_image:your_tag
Where the first part before the : refers to host machine and after, the container.
Hope it helps!
I really need help in docker.
my docker file look like:
FROM python:3-alpine
LABEL author="alaa"
LABEL description="Dockerfile for Python script which generates emails"
RUN pip install tqdm
COPY email_generator.py /app/
CMD python3 /app/email_generator.py
my pthon code looks like:
import json # to read json files
import os # to access operation for get and changing directory
def writeTextFile(text, index):
f = open(ziel + '/email_%s.txt' % index, 'w+')
f.write(text)
f.close()
def writeHashFile(text):
f = open(ziel + '/00_Hash.json', 'w+')
f.write(str(text))
f.close()
def readJsonCordinate(fileName):
"""Read the json data."""
with open(fileName, 'r', encoding='utf-8') as f: # Opening the file
data = json.load(f) # Read the json file
return data
and so on...
my question is if i want to get files from the host system after building the image there i get this error. but if i run the code nativly on pycharm in my macOS it runs perfectly
Traceback (most recent call last):
File "/app/email_generator.py", line 112, in <module>
betreff = readJsonCordinate(quelle + '/Betreff.json')
File "/app/email_generator.py", line 22, in readJsonCordinate
with open(fileName, 'r', encoding='utf-8') as f: # Opening the file
FileNotFoundError: [Errno 2] No such file or directory: '/Users/soso/desktop/email_generator/Worterbuecher/Betreff.json'
That's probably because you haven't copied the file that the error is about into the docker.
Check if the file is present in the docker image:
docker run -it --rm --entrypoint="" <image-name>:<image-tag> /bin/sh
and type into console:
find / -iname 'Betreff.json'
to find that file inside the image, and change path in you python so it works with updated path
Or you can add as mapped dir the dir that contains this file by using -v flag this way:
docker run -v /Users/soso/desktop/email_generator/Worterbuecher/:/Users/soso/desktop/email_generator/Worterbuecher/ ... some other opitons <docker-image>:<docker-tag>
more info about docker run command look for the option of -v or --volume for details
You can't access files on VM using VM file paths.
It's because container filesystem is virtually disconnected from VM filesystem.
You can map VM directories and files to container using docker volumes.
Then python program will be able to access mapped files and directories from within container using container paths.
I'm writing a python script which to create AWS CloudFormation change-sets to simplify the execution I want to add this script to a Docker image and run it as the entrypoint.
To achieve that, I have to read the CFs template file and parameters file, both are in json format.
When I execute the script on the local shell environment everything works as expected.
When I run the docker container and specify the files. The script says that it could not find the file.
Now my question is how can I enable the container getting access to this files?
docker pull cf-create-change-set:latest
docker run cf-create-change-set:latest --template-body ./template.json --cli-input-json ./parameters.json
Traceback (most recent call last):
File "/app/cf-create-change-set.py", line 266, in <module>
with open(CLI_INPUT_JSON, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: './template.json'
Here is my dockerfile:
FROM AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/cicd/docker-base-image:iat
WORKDIR /app
# Copy app data
COPY app/requirements.txt .
COPY app/cf-create-change-set.py .
RUN pip3 install --no-cache-dir -r /app/requirements.txt
ENTRYPOINT [ "/app/cf-create-change-set.py" ]
The reason for the error is that the file does not exist inside the container while it exists in your filesystem. There are at least two possible solutions.
You can either ADD the files into the container at build stage (build your own Dockerfile) or map a local directory to a directory inside the container:
docker run -v ./templates:/templates image:tag --template-body /templates/template.json
This way when you run the container it will have the same file located at /templates/template.json as the contents of your local templates folder. Read more about bind mounts.
So basically I have a python script that will write to a file once it is done running. How do I access this file? My end goal is to run the docker image on jenkins and then read the xml file that the python script generates.
FROM python:3
ADD WebChecker.py /
ADD requirements.txt /
ADD sites.csv /
RUN pip install -r requirements.txt
CMD [ "python", "./WebChecker.py" ]
That is my Dockerfile. I have a print("Finished") in there and it is printing so that means everything is working fine. It's just now I need to see my output.xml file.
You should have done it now by following above comments. In case if you still stuck, you may give a try as below:
Build:
docker build -t some_tag_name_to_your_image .
After build is completed, you may run a container and get the xml file as below:
1. Write output file to bind volume
Run your container as below:
docker run -d --rm --name my_container \
-v ${WORKSPACE}:/path/to/xml/file/in/container \
some_tag_name_to_your_image
Once the xml file generated, that will be available at the Jenkins-host:${WORKSPACE}
Notes:
${WORKSPACE} is an env variable set by Jenkins. Read more env-vars here
Read more about bind mount here
I have a simple Python script, inside of a Docker container, that:
-Makes a call to an external API
-A license file is returned from the API
I want to be able to save the returned license file, to my directory inside of the docker container.
This is what I have done:
r = requests.post("http://api/endpoint", headers=headers, data=data, auth=("", "")
f = open('test.lic', 'w+')
f.write(r.content)
f.close()
My understanding is that this would create a test.lic file if one did not already exist, or open the existing test.lic file, and write the content of the request object to the test.lic. However this is not working. No file is being saved to my directory inside of the Docker container. If I run these lines from a python shell it works so I'm guessing it has something to do with being inside of a Docker container.
It could be that the file is getting saved, but it is in the working directory in the container.
The docker docs say:
The default working directory for running binaries within a container is the root directory (/), but the developer can set a different default with the Dockerfile WORKDIR command.
So the file may be ending up in the root directory / of your container.
You could add a print of os.getcwd() to your script (if you are able to see the output... which you might need to use the docker logs command) to verify this. Either way, the file will likely be in the location returned by os.getcwd().
In my case the file was saved to a different container. Check it if you have multiple containers in the docker-compose.