I really need help in docker.
my docker file look like:
FROM python:3-alpine
LABEL author="alaa"
LABEL description="Dockerfile for Python script which generates emails"
RUN pip install tqdm
COPY email_generator.py /app/
CMD python3 /app/email_generator.py
my pthon code looks like:
import json # to read json files
import os # to access operation for get and changing directory
def writeTextFile(text, index):
f = open(ziel + '/email_%s.txt' % index, 'w+')
f.write(text)
f.close()
def writeHashFile(text):
f = open(ziel + '/00_Hash.json', 'w+')
f.write(str(text))
f.close()
def readJsonCordinate(fileName):
"""Read the json data."""
with open(fileName, 'r', encoding='utf-8') as f: # Opening the file
data = json.load(f) # Read the json file
return data
and so on...
my question is if i want to get files from the host system after building the image there i get this error. but if i run the code nativly on pycharm in my macOS it runs perfectly
Traceback (most recent call last):
File "/app/email_generator.py", line 112, in <module>
betreff = readJsonCordinate(quelle + '/Betreff.json')
File "/app/email_generator.py", line 22, in readJsonCordinate
with open(fileName, 'r', encoding='utf-8') as f: # Opening the file
FileNotFoundError: [Errno 2] No such file or directory: '/Users/soso/desktop/email_generator/Worterbuecher/Betreff.json'
That's probably because you haven't copied the file that the error is about into the docker.
Check if the file is present in the docker image:
docker run -it --rm --entrypoint="" <image-name>:<image-tag> /bin/sh
and type into console:
find / -iname 'Betreff.json'
to find that file inside the image, and change path in you python so it works with updated path
Or you can add as mapped dir the dir that contains this file by using -v flag this way:
docker run -v /Users/soso/desktop/email_generator/Worterbuecher/:/Users/soso/desktop/email_generator/Worterbuecher/ ... some other opitons <docker-image>:<docker-tag>
more info about docker run command look for the option of -v or --volume for details
You can't access files on VM using VM file paths.
It's because container filesystem is virtually disconnected from VM filesystem.
You can map VM directories and files to container using docker volumes.
Then python program will be able to access mapped files and directories from within container using container paths.
Related
Setup
Project
src
main.py
Dockerfile
Dockerfile (raw, needs to be revamped)
FROM python:3
ADD src/main.py /
RUN chmod +x main.py
RUN /usr/local/bin/python -m pip install --upgrade pip
COPY . /opt/app
RUN pip install -r /opt/app/requirements.txt
ADD / /usr/local
ENTRYPOINT [ "python", "./main.py" ]
main.py
if __name__ == '__main__':
if len(sys.argv) == 2:
main(sys.argv[1])
def main(logs_file_archive):
unzip_logs(logs_file_archive) # unzips all logs from the provided path to the folder with the same name as archive/Extracted directory
create_csv_files() # creates CSV files needed to plot graph
process_files() # populate CSV files generated with the right data
plot(logs_file_archive) # build this data representation
Actual/desired behaviour
Actual:
2022-01-17T22:05:31.547047838Z File "//./main.py", line 214, in <module>
2022-01-17T22:05:31.547210046Z main(sys.argv[1])
2022-01-17T22:05:31.547259438Z File "//./main.py", line 187, in main
2022-01-17T22:05:31.547670294Z unzip_logs(logs_file_archive)
2022-01-17T22:05:31.547732344Z File "//./main.py", line 54, in unzip_logs
2022-01-17T22:05:31.548296998Z with zipfile.ZipFile(file_path, "r") as zip_ref:
2022-01-17T22:05:31.548350898Z File "/usr/local/lib/python3.10/zipfile.py", line 1240, in __init__
2022-01-17T22:05:31.549638566Z self.fp = io.open(file, filemode)
2022-01-17T22:05:31.549692977Z FileNotFoundError: [Errno 2] No such file or directory: '/Users/user/PerfReader/misc/archive.zip'
No such file or directory: '/Users/user/PerfReader/misc/archive.zip' is expected well... because there is no such file in the Docker machine.
Desired: container runs using Dockerfile, data processes, the plot is displayed real-time or saved as a file and transferred to host
Question/issue description
I am not entirely sure how to transfer the file specified to the Docker container. I read https://docs.docker.com/storage/volumes/ but it doesn't provide any examples so I seek examples of how volumes can be mounted.
Provided that my plot() in main.py does plot data properly, what are my options on displaying this data (output of the whole exercise is the plot)? Can I display the plot in real-time from Docker? Or is my only option is to generate a plot and then transfer it back to the host machine using matplotlib.pyplot.savefig?
First Question:
There are multiple ways to access the host files. You can use the copy or add commands like what you are doing in your Docker file and the file will be copied during building the image. Furthermore, you can use the binding mount, it allows you to access the host directory that was binded, something like you are running the container in this directory.
Second Question:
Docker doesn't support GUI or accessing the host display. However, you can allow it to do so using Xauth. Consider the following link for the steps.
Either way, I don't encourage using Docker for what you are doing especially the plotting part, python virtual environment would be more than enough.
Introduction
I'm new to AWS and I'm trying to run Jellyfish on an AWS Lambda function with a python container image from an AWS base image.
Context
I want to achieve this by installing the software from source in my python container image (from AWS base image) and then uploading it to AWS ECR to be later used by a Lambda function.
My AWS architecture is:
biome-test-bucket-out (AWS S3 bucket with trigger) -> Lambda function -> biome-test-bucket-jf (AWS S3 bucket)
First, to install it from source I downloaded the latest release .tar.gz file locally, then uncompressed it, and copied the contents in the container.
My Dockerfile looks like this:
FROM public.ecr.aws/lambda/python:3.9
# Copy contents from latest release
WORKDIR /jellyfish-2.3.0
COPY ./jellyfish-2.3.0/ .
WORKDIR /
# Installing Jellyfish dependencies
RUN yum update -y
RUN yum install -y gcc-c++
RUN yum install -y make
# Jellyfish installation (in /bin)
RUN jellyfish-2.3.0/configure --prefix=/bin
RUN make -j 4
RUN make install
RUN chmod -R 777 /bin
# Copy function code
COPY app.py ${LAMBDA_TASK_ROOT}
CMD [ "app.lambda_handler" ]
The installation folder is /bin because it's in $PATH so that way I can just run "jellyfish..." command.
My app.py looks like this:
import subprocess
import boto3
import logging
from pathlib import Path, PurePosixPath
s3 = boto3.client('s3')
print('Loading function')
def lambda_handler(event, context):
# Declare buckets & get name of file
bucket_in = "biome-test-bucket-out"
bucket_out = "biome-test-bucket-jf"
key = event["Records"][0]["s3"]["object"]["key"]
# Paths where files will be stored
input_file_path = f"/tmp/{key}"
file = Path(key).stem
output_file_path = f"/tmp/{file}.jf"
# Download file
with open(input_file_path, 'wb') as f:
s3.download_fileobj(bucket_in, key, f)
# Run jellyfish with downloaded file
command = f"/bin/jellyfish count -C -s 100M -m 20 -t 1 {input_file_path} -o {output_file_path}"
logging.info(subprocess.check_output(command, shell=True))
# Upload file to bucket
try:
with open(output_file_path, 'rb') as f:
p = PurePosixPath(output_file_path).name
s3.upload_fileobj(f, bucket_out, p)
except Exception as e:
logging.error(e)
return False
return 0
Problem
If I build and run the image locally, everything works fine but once the image runs in Lambda I get this error:
/usr/bin/ld: cannot open output file /bin/.libs/12-lt-jellyfish: Read-only file system
That file isn't there after the installation so I'm guessing that Jellyfish creates a new file in /bin/.libs/ when it's running and that file as only read permissions. I'm not sure how to tackle this, any ideas?
Thank you.
I am trying to write a text (.txt) file to a local Desktop folder on Windows 10 after building a docker image (docker_run_test). The docker build seems to work seamlessly using "docker build -t docker_run_test .", running the command from the working directory on the Desktop where the Dockerfile and Python script reside (C:/Users/mdl518/Desktop/docker_tests/). The script is a simple print statement and writing of the print statement to a .txt file, but I cannot locate the output .txt. Below is the associated Dockerfile, and the Python script.
The Dockerfile:
FROM python:3
ADD docker_test.py ./
RUN pip install pandas
CMD ["python3","./docker_test.py"]
The Python script (docker_test.py):
import os
print("This is a Docker test.")
with open('docker_test.txt', 'w') as f:
f.write("This is a Docker test.")
I have searched the contents of the Docker image as well, but cannot locate the output .txt within the image either. Any assistance is most appreciated!
You have to mount/bind the folder where you want to see the results into the container.
Change the output filename to write in another folder, let say /output
with open('/output/docker_test.txt', 'w') as f:
And then ask Docker to bind host folder %HOMEPATH%\Desktop to container /output :
docker run -v %HOMEPATH%\Desktop:/output docker_run_test
Not sure for %HOMEPATH% syntax as I'm a Linux user.
I'm writing a python script which to create AWS CloudFormation change-sets to simplify the execution I want to add this script to a Docker image and run it as the entrypoint.
To achieve that, I have to read the CFs template file and parameters file, both are in json format.
When I execute the script on the local shell environment everything works as expected.
When I run the docker container and specify the files. The script says that it could not find the file.
Now my question is how can I enable the container getting access to this files?
docker pull cf-create-change-set:latest
docker run cf-create-change-set:latest --template-body ./template.json --cli-input-json ./parameters.json
Traceback (most recent call last):
File "/app/cf-create-change-set.py", line 266, in <module>
with open(CLI_INPUT_JSON, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: './template.json'
Here is my dockerfile:
FROM AWS_ACCOUNT_ID.dkr.ecr.AWS_REGION.amazonaws.com/cicd/docker-base-image:iat
WORKDIR /app
# Copy app data
COPY app/requirements.txt .
COPY app/cf-create-change-set.py .
RUN pip3 install --no-cache-dir -r /app/requirements.txt
ENTRYPOINT [ "/app/cf-create-change-set.py" ]
The reason for the error is that the file does not exist inside the container while it exists in your filesystem. There are at least two possible solutions.
You can either ADD the files into the container at build stage (build your own Dockerfile) or map a local directory to a directory inside the container:
docker run -v ./templates:/templates image:tag --template-body /templates/template.json
This way when you run the container it will have the same file located at /templates/template.json as the contents of your local templates folder. Read more about bind mounts.
I have a simple Python script, inside of a Docker container, that:
-Makes a call to an external API
-A license file is returned from the API
I want to be able to save the returned license file, to my directory inside of the docker container.
This is what I have done:
r = requests.post("http://api/endpoint", headers=headers, data=data, auth=("", "")
f = open('test.lic', 'w+')
f.write(r.content)
f.close()
My understanding is that this would create a test.lic file if one did not already exist, or open the existing test.lic file, and write the content of the request object to the test.lic. However this is not working. No file is being saved to my directory inside of the Docker container. If I run these lines from a python shell it works so I'm guessing it has something to do with being inside of a Docker container.
It could be that the file is getting saved, but it is in the working directory in the container.
The docker docs say:
The default working directory for running binaries within a container is the root directory (/), but the developer can set a different default with the Dockerfile WORKDIR command.
So the file may be ending up in the root directory / of your container.
You could add a print of os.getcwd() to your script (if you are able to see the output... which you might need to use the docker logs command) to verify this. Either way, the file will likely be in the location returned by os.getcwd().
In my case the file was saved to a different container. Check it if you have multiple containers in the docker-compose.