I'm writing a python script which to create AWS CloudFormation change-sets to simplify the execution I want to add this script to a Docker image and run it as the entrypoint.
To achieve that, I have to read the CFs template file and parameters file, both are in json format.
When I execute the script on the local shell environment everything works as expected.
When I run the docker container and specify the files. The script says that it could not find the file.
Now my question is how can I enable the container getting access to this files?
docker pull cf-create-change-set:latest
docker run cf-create-change-set:latest --template-body ./template.json --cli-input-json ./parameters.json
Traceback (most recent call last):
File "/app/cf-create-change-set.py", line 266, in <module>
with open(CLI_INPUT_JSON, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: './template.json'
Here is my dockerfile:
FROM AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/cicd/docker-base-image:iat
WORKDIR /app
# Copy app data
COPY app/requirements.txt .
COPY app/cf-create-change-set.py .
RUN pip3 install --no-cache-dir -r /app/requirements.txt
ENTRYPOINT [ "/app/cf-create-change-set.py" ]
The reason for the error is that the file does not exist inside the container while it exists in your filesystem. There are at least two possible solutions.
You can either ADD the files into the container at build stage (build your own Dockerfile) or map a local directory to a directory inside the container:
docker run -v ./templates:/templates image:tag --template-body /templates/template.json
This way when you run the container it will have the same file located at /templates/template.json as the contents of your local templates folder. Read more about bind mounts.
Related
Setup
Project
src
main.py
Dockerfile
Dockerfile (raw, needs to be revamped)
FROM python:3
ADD src/main.py /
RUN chmod +x main.py
RUN /usr/local/bin/python -m pip install --upgrade pip
COPY . /opt/app
RUN pip install -r /opt/app/requirements.txt
ADD / /usr/local
ENTRYPOINT [ "python", "./main.py" ]
main.py
if __name__ == '__main__':
if len(sys.argv) == 2:
main(sys.argv[1])
def main(logs_file_archive):
unzip_logs(logs_file_archive) # unzips all logs from the provided path to the folder with the same name as archive/Extracted directory
create_csv_files() # creates CSV files needed to plot graph
process_files() # populate CSV files generated with the right data
plot(logs_file_archive) # build this data representation
Actual/desired behaviour
Actual:
2022-01-17T22:05:31.547047838Z File "//./main.py", line 214, in <module>
2022-01-17T22:05:31.547210046Z main(sys.argv[1])
2022-01-17T22:05:31.547259438Z File "//./main.py", line 187, in main
2022-01-17T22:05:31.547670294Z unzip_logs(logs_file_archive)
2022-01-17T22:05:31.547732344Z File "//./main.py", line 54, in unzip_logs
2022-01-17T22:05:31.548296998Z with zipfile.ZipFile(file_path, "r") as zip_ref:
2022-01-17T22:05:31.548350898Z File "/usr/local/lib/python3.10/zipfile.py", line 1240, in __init__
2022-01-17T22:05:31.549638566Z self.fp = io.open(file, filemode)
2022-01-17T22:05:31.549692977Z FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/PerfReader/misc/archive.zip'
No such file or directory: '/Users/user/PerfReader/misc/archive.zip' is expected well... because there is no such file in the Docker machine.
Desired: container runs using Dockerfile, data processes, the plot is displayed real-time or saved as a file and transferred to host
Question/issue description
I am not entirely sure how to transfer the file specified to the Docker container. I read https://docs.docker.com/storage/volumes/ but it doesn't provide any examples so I seek examples of how volumes can be mounted.
Provided that my plot() in main.py does plot data properly, what are my options on displaying this data (output of the whole exercise is the plot)? Can I display the plot in real-time from Docker? Or is my only option is to generate a plot and then transfer it back to the host machine using matplotlib.pyplot.savefig?
First Question:
There are multiple ways to access the host files. You can use the copy or add commands like what you are doing in your Docker file and the file will be copied during building the image. Furthermore, you can use the binding mount, it allows you to access the host directory that was binded, something like you are running the container in this directory.
Second Question:
Docker doesn't support GUI or accessing the host display. However, you can allow it to do so using Xauth. Consider the following link for the steps.
Either way, I don't encourage using Docker for what you are doing especially the plotting part, python virtual environment would be more than enough.
I'm seeing the above error when I try to run my docker image. Below are the screenshots of my docker file and the directory structure.
As you have already specified the WORKDIR in the Dockerfile. Dont' copy your files to /.
Change your command to
COPY . . # If you want to copy whole folder into container
and as well CMD command to
CMD ["python", "src/main.py"]
I have a Python script, python_script.py, that reads an HDF5 file, hdf_file.h5, on my local machine. The directory path to the files is
folder1
folder2
python_script.py
hdf_file.h5
I have the following sample code:
from pandas import read_hdf
df = read_hdf('hdf_file.h5')
When I run this code on my local machine, it works fine.
However, I need to place the Python script inside a Docker container, keep the HDF file out of the container, and have the code read the file. I want to have something like the following directory path for the container:
folder1
folder2
hdf_file.h5
docker-folder
python_script.py
requirements.txt
Dockerfile
I use the following Dockerfile:
FROM python:3
WORKDIR /project
COPY ./requirements.txt /project/requirements.txt
RUN pip install -r requirements.txt
COPY . /project
CMD [ "python", "python_script.py" ]
I am new to Docker and am having a lot of trouble figuring out how to get a Python script inside a container to read a file outside the container. What commands do I use or code changes do I make to be able to do this?
It seems you need to use docker volumes (https://docs.docker.com/storage/volumes/).
Try the following:
docker run -v path/where/lives/hdf5/:path/to/your/project/folder/your_image your_docker_image:your_tag
Where the first part before the : refers to host machine and after, the container.
Hope it helps!
I'm trying to send files as an argument in python3 cli app (which uses argp arse for parsing) which is hosted in docker. But I'm getting OSError:
Error opening b'input_file.txt' when I perform docker run -t
input_file.txt
I tried:
docker run -t docker_image_name input_file.txt
My docker file has entry point as:
ENTRYPOINT [ "python", "/src/cli_app.py" ]
You're telling your python application to look for input_file.txt, but that file doesn't exist in the container. You're not passing a file as is, just an argument/parameter. Try the following to mount your local file (I'm assuming it's in your working directory) into the container:
docker run -it -v $(pwd)/input_file.txt:/tmp/input_file.txt docker_image_name /tmp/input_file.txt
docker run docker_image_name -e IP_FILE_NAME="input_file.txt"
In your python code access the filename as environment variable $IP_FILE_NAME. This is considering the input_file.txt is present in the container. Use COPY command to copy the file from the machine you use to build to the container.
COPY input_file.txt /src/input_file.txt
So basically I have a python script that will write to a file once it is done running. How do I access this file? My end goal is to run the docker image on jenkins and then read the xml file that the python script generates.
FROM python:3
ADD WebChecker.py /
ADD requirements.txt /
ADD sites.csv /
RUN pip install -r requirements.txt
CMD [ "python", "./WebChecker.py" ]
That is my Dockerfile. I have a print("Finished") in there and it is printing so that means everything is working fine. It's just now I need to see my output.xml file.
You should have done it now by following above comments. In case if you still stuck, you may give a try as below:
Build:
docker build -t some_tag_name_to_your_image .
After build is completed, you may run a container and get the xml file as below:
1. Write output file to bind volume
Run your container as below:
docker run -d --rm --name my_container \
-v ${WORKSPACE}:/path/to/xml/file/in/container \
some_tag_name_to_your_image
Once the xml file generated, that will be available at the Jenkins-host:${WORKSPACE}
Notes:
${WORKSPACE} is an env variable set by Jenkins. Read more env-vars here
Read more about bind mount here