Im working on a OpenCv project and as many know, the instalation of that on windows is iritating. So, what i want to do is to run the project in a docker container and store the output to a folder on the host computer. In simple terms it is something like this:
Program python / opencv code
Build Docker image
Run Docker image --> Saves the output data somewhere
In some way - get access to the output data on host.
Now, i have been trying to find many ways of dooing this, and i will probably at a later time send it using other means. However, for development i need this slightly more direct approach. It also has somthing to do with colaboration with others.
Simple Docker file that can be used as the base:
FROM python:3
WORKDIR /usr/src/app
COPY . .
CMD [ "python", "./script.py" ]
Lets say that script.py creates a file called output.txt. I want that output.txt stored at my E: drive.
How to do this automatically - without having to do multiple comandline operations?
TLDR; How to get files from Docker container to host? Goal: File physically stored on E:
There are two ways to do this. First is to mount a docker volume.
docker run --name=name -d -v /path/in/host:/path/in/docker name
By mounting a volume like this, whatever you write in the directory you've mounted will be written in the host automatically. Check this for more information on volumes.
The second way is to copy the files from a container to the host like this
docker cp <containerId>:/file/path/within/container /host/path/target
Check docs here for more info on this.
Related
I am starting to get a hand on docker and try to containerized some of the applications I use. Thanks to the tutorial I was able to create docker images and containers but now I am trying to thing about the most efficient and practical ways to do things.
To present my use-case, I have a python code (let's call it process.py) that takes as an input a single .jpg image, does some operations on this image, and then output the processed .jpg image.
Normally I would run it through :
python process.py -i path_of_the_input_image -o path_of_the_output_image
Then, the way I do the connection input/output with my docker is the following. First I create the docker file :
FROM python:3.6.8
COPY . /app
WORKDIR /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
CMD python ./process.py -i ./input_output/input.jpg -o ./input_output/output.jpg
And then after building the image, I run docker run mapping the a local folder with the input_output folder of docker:
docker run -v C:/local_folder/:/app/input_output my_docker_image
This seems to work, but is not really practical, as I have to create locally a specific folder to mount it to the docker container. So here are the questions I am asking myself :
Is there a more practical ways of doings things ? To directly send one single input file and directly receive one single output files from the output of a docker container ?
When I run the docker image, what happens (If I understand correctly) is that it will create a docker container that will run my program once process.py once and then just sits there doing nothing. Even after finishing running process.py it will still be there listed in the command "docker ps -a". Is this behaviour expected ? Is there a way to automatically delete finished container ? Am I using docker run the right way ?
Is there a more practical way of having a container running continuously and on which I can query to run the program process.py on demand with a given input ?
I have a python code (let's call it process.py) that takes as an input a single .jpg image, does some operations on this image, and then output the processed .jpg image.
That's most efficiently done without Docker; just run the python command you already have. If your application has interesting Python library dependencies, you can install them in a virtual environment to avoid conflicts with the system Python installation.
When I run the Docker image...
...the container runs its main command (docker run command arguments, Dockerfile CMD, possibly combined with an entrypoint from the some sources), and when that command exits, the container exits. It will be listed in docker ps -a output, but as "Stopped" (probably with status 0 for a successful completion). You can docker run --rm to have the container automatically delete itself.
Is there a more practical way of having a container running continuously and on which I can query to run the program process.py on demand with a given input ?
Wrap it in a network service, like a Flask application. As long as this is running, you can use a tool like curl to do an HTTP POST with the input JPEG file as the body, and get the output JPEG file as the response. Avoid using local files and Docker together whenever that's an option (prefer network I/O for process inputs and outputs; prefer a database to local-file storage).
Why are volume mounts not practical?
I would argue that Dockerising your application is not practical, but you've chosen to do so for, presumably very good, reasons. Volume mounts are simply an extension to this. If you want to get data in/out of your container, the 'normal' way to do this is by using volume mounts as you have done. Sure, you could use docker cp to copy the files manually, but that's not really practical either.
As far as the process exiting goes, normally, once the main process exits, the container exits. docker ps -a shows stopped containers as well as running ones. You should see that it says Exited n minutes(hours, days etc) ago. This means that your container has run and exited, correctly. You can remove it with docker rm <containerid>.
docker ps (no -a) will only show the running ones, btw.
If you use the --rm flag in your Docker run command, it will be removed when it exits, so you won't see it in the ps -a output. Stopped containers can be started again, but that's rather unusual.
Another solution might be to change your script to wait for incoming files and process them as they are received. Then you can leave the container running, and it will just process them as needed. If doing this, make sure that your idle loop has a sleep or something in it to ensure that you don't consume too many resources.
I am building a webapp (a simple flask site) that uses docker. I want my development code to not reside within docker, but be executed by the development environment (using python3) I have defined in my dockerfile. I know that I can use the COPY . . syntax in a dockerfile to copy my source code into the image for execution, but that violates my aim of separating the container from my source. Is there a way to have a docker container read and execute the code that it is in the directory I run the docker container run command from?
Right now my container uses the copy company to copy all the source code into the container. It then uses the CMD command to automatically run the flask app:
CMD [ "python", "flask_app/server.py" ]
(I'm storing all my flask code in a directory called flask_app). I'm assuming this works because all this has been copied into the container (according to the specifications given in the dockerfile) and is being executed when I run the container. I would like for the container to instead access and execute flask_app/server.py without copying this information into itself -- is this possible? If so, how?
Instead of using COPY to move the code into the container, you'll use a "bind mount" (https://docs.docker.com/storage/bind-mounts/).
When you run the container, you'll do it with a command like this:
docker run --mount type=bind,source=<path_outside_container>,target=<path_inside_container> <image_tag>
For portability, I recommending putting this line in a script intended to be run from the repository root, and having the <path_outside_container> be "$(pwd)", so that it will work on other people's computers. You'll need to adjust <path_inside_container> and your CMD depending on where you want the code to live inside the container.
(Obviously you can also put whatever other options you'd like on the command, like --it --rm or -p <whatever>.)
I'm trying to create a container to run a program. I'm using a pre-configured image and now I need to run the program. However, it's a machine learning program and I need a dataset from my computer to run.
The file is too large to be copied to the container. It would be best if the program running in the container searched the dataset in a local directory of my computer, but I don't know how I can do this.
Well, I have made the shared folder from my machine appeared using docker run -it -v ~/Volumes/Data/Studies/PhD\Work/gitlab/J2/ydk-py:/ydk-py ydkdev/ydk-py in the container, but all files in folder ydk-py are not shown. This is the safe, usually-desired behavior. But for development and instance setup, it would be immensely useful to have access to an existing file structure.
docker run with -v will automatically mount sub-directories. In your case you are using relative path, which you need to use absolute path as per this documentation.
So change your command from
docker run -it -v ~/Volumes/Data/Studies/PhD\Work/gitlab/J2/ydk-py:/ydk-py ydkdev/ydk-py
to
docker run -it -v /home/<what ever user>/Volumes/Data/Studies/PhD\Work/gitlab/J2/ydk-py:/ydk-py ydkdev/ydk-py
it will work.
Make sure you have enough permissions on directory that you are trying to mount.
What's the proper development workflow for code that runs in a Docker container?
Solomon Hykes said that the "official" workflow involves building and running a new Docker image for each Git commit. That makes sense, but what if I want to test a change before committing it to the Git repo?
I can think of two ways to do it:
Run the code on a local development server (e.g., the Django development server). Edit a file; test on the dev server; make a Git commit; rebuild the Docker image with the new code; test again on the local Docker container.
Don't run a local dev server. Instead, build and run a new Docker image each time I edit a file, and then test the change on local Docker container.
Both approaches are pretty inefficient. Is there a better way?
A more efficient way is to run a new container from the latest image that was built (which then has the latest code).
You could start that container starting a bash shell so that you will be able to edit files from inside the container:
docker run -it <some image> bash -l
You would then run the application in that container to test the new code.
Another way to alter files in that container is to start it with a volume. The idea is to alter files in a directory on the docker host instead of messing with files from the command line from the container itself:
docker run -it -v /home/joe/tmp:/data <some image>
Any file that you will put in /home/joe/tmp on your docker host will be available under /data/ in the container. Change /data to whatever path is suitable for your case and hack away.
This question already has answers here:
Mount SMB/CIFS share within a Docker container
(5 answers)
Closed 7 years ago.
I have a small Python application that I'd like to run on Linux in Docker (using boot2docker for now). This application reads some data from my Windows network share, which works fine on Windows using the network path but fails on Linux. After doing some research I figured out how to mount a Windows share on Ubuntu. I'm attempting to implement the dockerfile so that it sets up the share for me but have been unsuccessful so far. Below is my current approach, which encounters operation not permitted at the mount command during the build process.
#Sample Python functionality
import os
folders = os.listdir(r"\\myshare\folder name")
#Dockerfile
RUN apt-get install cifs-utils -y
RUN mkdir -p "//myshare/folder name"
RUN mount -t cifs "//myshare/folder name" "//myshare/folder name" -o username=MyUserName,password=MyPassword
#Error at mount during docker build
#"mount: error(1): Operation not permitted"
#Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Edit
Not a duplicate of Mount SMB/CIFS share within a Docker container. The solution for that question references a fix during docker run. I can't run --privileged if the docker build process fails.
Q: What is the correct way to mount a Windows network share inside a Docker container?
Docker only abstracts away applications, whereas mounting filesystems happens at the kernel level and so can't be restricted to only happen inside the container. When using --privileged, the mount happens on the host and then it's passed through into the container.
Really the only way you can do this is to have the share available on the host (put it in /etc/fstab on a Linux machine, or mount it to a drive letter on a Windows machine) and have the host mount it, then make it available to the container as you would with any other volume.
Also bear in mind that mkdir -p "//myshare/folder name" is a semi-invalid path - most shells will condense the // into / so you may not have access to a folder called /myshare/folder name since the root directory of a Linux system is not normally where you put files. You might have better success using /mnt/myshare-foldername or similar instead.
An alternative could be to find a way to access the files without needing to mount them. For example you could use the smbclient command to transfer files between the Docker container and the SMB/CIFS share without needing to mount it, and this will work within the Docker container, just as you might use wget or curl to upload or download files, as is also commonly done in Dockerfiles.
You are correct that you can only use --privileged during docker run. You cannot perform mount operations without --privileged, ergo, you cannot perform mount operations during the docker build process.
This is probably by design: the goal is that a Dockerfile is largely self contained; anyone should be able to use your Dockerfile and other contents in the same directory to generate the same image; linking things to an external mount would violate this restriction.
However, your question says that you have an application that needs to read some data from a share so it's not clear why you need the share mounted during docker build. It sounds like you would be better off building an image that will launch your application as part of docker run, and possibly use docker volumes to access the share rather than attempting to mount it inside the container.