I created a simple docker container to run a python script from within the container:
FROM python:3
WORKDIR /usr/src/app
COPY . .
CMD ["test.py"]
ENTRYPOINT ["python3"]
I build it, docker build -t hello-demo . and then run it docker run -it hello-demo test.py and I get my output.
But what I want to do is to be able to rerun this not on my laptop but using AWS Batch. But I am not sure how to identify the container name Batch creates. When I manually build the container I specify what I am calling it, but I am not sure how to call the correct container when running my docker run command.
Any thoughts? Or am I going about this wrong?
Thanks!
D
First, distinguish the Docker image from the Docker container. You build the Docker image and the container is a running instance of the image (refer to the Docker overview for more details). When you run the Docker image locally you specify what Docker image to run and also (optionally) the name of the container.
When using AWS Batch, you create a job definition which defines what Docker image it will use. Thus, the Docker image already has to exist in some registry, be it Docker Hub, ECR or any other registry. You can also define other container properties like a command that will be run.
To actually run the Batch job, you submit it to the queue using the job definition (which specifies the Docker image as stated above).
Related
lambdaFunction = _lambda.DockerImageFunction(self, f'{client_id}-prefect-lambda-handler',
code=_lambda.DockerImageCode.from_image_asset(
directory="cumulus_devops_cdk/prefect-lambda-handler"
),
)
I am trying to create a lambda function from a docker image in CDK as shown above. The problem is that my company's CDK runs in a docker image and thus has trouble building a docker image inside of itself.
I know that the docker image works because it succeeded when I manually built and pushed the image to ECR and had CDK pull from that, however I would like to have it get built every time I CDK deploy.
Whenever I try to cdk deploy the stack I get this error
[100%] fail: docker build --tag cdkasset-d4a61d4806d68e3a7b9589a1e161b40523d2a3bc5be6506aaf6bb4b45fd5cc07 . exited with error code 1: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
How can I successfully build the docker image in cdk and have it deployed to the lambda function?
You can build and publish the image to Amazon ECR outside of the AWS CDK, and then reference that image using _lambda.DockerImageCode.fromEcr(repository, props?) [1]
[1] https://docs.aws.amazon.com/cdk/api/v1/docs/#aws-cdk_aws-lambda.DockerImageCode.html#static-fromwbrecrrepository-props
This is as Docker in Docker issue[1]. If you have access to the configuration of the outer Docker Container (ie. the one doing the bundling) you should mount the Docker socket to this image like using -v/--volume:
docker run -v /var/run/docker.sock:/var/run/docker.sock <image>
This allows <image> to use the host Docker Daemon.
[1] https://devopscube.com/run-docker-in-docker/
I have cloned a repository of an API built with python on my local machine and my goal is to be able to send requests and receive responses locally.
I'm not familiar python but the code is very readable and easy to understand, however the repository contains some dependencies and configuration files to Dockerise (and I'm not familiar with Docker and containers too) .
so what are the steps to follow in order to be able to interact with the API locally?.
Here are some files in the repository for config and requirements :
requirements.txt file :
fastapi==0.70.0
pytest==7.0.1
requests==2.27.1
uvicorn==0.15.0
Dockerfile file :
FROM tiangolo/uvicorn-gunicorn:python3.9
COPY ./requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
COPY ./app /app
i already installed Python3 and docker so what's next ?
Adjust Dockerfile
Assuming all code is in the /app directory you have already copied over all your code and installed all the dependencies required for the application.
But you are missing - at least (see disclaimer) - one essential line in the Dockerfile which is actually the most important line as it is the CMD command to tell Docker which command/ process should be executed when the container starts.
I am not familiar with the particular base image you are using (which is defined using the FROM command) but after googling I found this repo which suggests the following line, which does make a lot of sense to me as it starts a web server:
# open port 80 on the container to make it accesable from the outside
EXPOSE 80
# line as described in repo to start the web server
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "80"]
This should start the web server on port 80 using the application stored in a variable app in your main.py when the container starts.
Build and run container
When you have added that you need to build your image using docker build command.
docker build -t asmoun/my-container .
This builds an container image asmoun/my-container using the Dockerfile in the current directory, hence the .. So make sure you execute that when in the directory with the Dockerfile. This will take some time as the base image has to download and dependencies need to be installed.
You now have an image that you can run using docker run command:
docker run --name my-fastapi-container -d -p 80:80 asmoun/my-container
This will start a container called my-fastapi-container using the image asmoun/my-container in detached mode (-d option that makes sure your TTY is not attached to the container) and define a port mapping, which maps the port 80 on the host to port 80 on the container, which we have previously exposed in the Dockerfile (EXPOSE 80).
You should now see some ID getting printed to your console. This means the container has started. You can check its state using docker ps -a and you should see it marked as running. If it is, you should be able to connect to localhost:80 now. If it is not use docker logs my-fastapi-container to view the logs of the container and you'll hopefully learn more.
Disclaimer
Please be aware that this is only a minimal guide on how you should be able to get a simple FastAPI container up and running, but some parameters could well be different depending on the application (e.g. name of main.py could be server.py or similar stuff) in which case you will need to adjust some of the parameters but the overall process (1. adjust Dockerfile, 2. build container, 3. run container) should work. It's also possible that your application expects some other stuff to be present in the container which would need to be defined in the Dockerfile but neither me, nor you (presumably) know this, as the Dockerfile provided seems to be incomplete. This is just a best effort answer.
I have tried to link all relevant resources and commands so you can have a look at what some of them do and which options/ parameters might be of interest for you.
I am new to Docker and I am confused about containers and images somehow. I want to sue Docker for Tensorflow development. All I need is to have an easy way to write Jupyter Notebooks and use GPU powered Tensorflow.
I have the latest Tensorflow Jupyter Python 3 Image already. I run the Image with
docker run --rm --runtime=nvidia -v -it -p 8888:8888 tensorflow/tensorflow:latest-gpu-py3-jupyter
How can I make it so that my data when I work in that Image and add and edit my Jupyter Notebooks won't get lost after I exit the process. I know that Docker Images aren't meant to persist state but I am so new to this I just want something to work in with persistent data. Can someone help me guide me through this or point to a resource which will answer all my prayers?
I would also like to move some stuff into the Container that is going to be run so that I can access some custom Python libs because they contain some things that my Notebooks need to import!
Side questions:
--rm removes the container or whatever by default I run it without this flag still my data was lost
-v is for volumes? I tried with -v Bachelor:/app to mount a volume like so. It apparently doesn't make any difference. I don't know how to use the volume Bachelor that I created. Instead there are a multitude of unnamed volumes being created that are not usable whenever I run this
-it does also something no idea what
-p is the port number right?
Use Docker volumes:
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers
Example:
docker run --runtime=nvidia -v ${SOURCE_FOLDER}:${DEST_FOLDER} -p 8888:8888 tensorflow/tensorflow:latest-gpu-py3-jupyter
Change SOURCE_FOLDER and DEST_FOLDER accordingly (use absolute paths!).
Now if you navigate to localhost:8888 and create a notebook on DEST_FOLDER, it also should be available on SOURCE_FOLDER.
As for your side questions:
--it runs a container in interactive mode. You generally add /bin/bash after the run command, so you can start an interactive bash session inside the container.
--rm cleans the container after it exists.
Those options aren't really necessary for your use case. Just remember to use docker ps and docker rm <ID> to clean up your container after you're done.
I would like to run Python scripts in various stages of Jenkins (pipeline) jobs, abroad a wide range of agents. I want the same Python environment for all of these, so I'm considering using Docker for this purpose.
I'm considering using Docker to build an image that contains the Python environment (with installed packages, etc.), that then allows an external Python script based on argument input:
docker run my_image my_python_file.py
My question is now, how should the infrastructure be? I see that the Python docker distribution is 688MB, and transferring this image to all steps would surely be an overhead? However, they are all on the same network, so perhaps it wouldn't be a big issue.
Updates. So, my Dockerfile looks like this:
FROM python:3.6-slim-jessie
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python3"]
Then I build the image using
>docker build ./ -t my-app
which successfully builds the image and install my requirements. Then I want to start the image as daemon using
> docker run -dit my-app
Then I execute the process using
> docker exec -d {DAEMON_ID} my-script.py
Run your Docker container as a daemon process, and every time you need to run your Python script, call docker exec.
docker exec -d <your-container> <your-python-file.py>
Using a Docker agents for build is an effective way to have distributed and reproducible builds.
I see that the Python docker distribution is 688MB, and transferring this image to all steps would surely be an overhead?
You should consider using smaller Docker images. There are alpine and slim docker images for python. You should consider using these first. The size of the alpine python image is 89.2MB.
Also the most of the image layers will be cached by Docker, so you will be pulling a some layers with significantly smaller sizes.
What's the proper development workflow for code that runs in a Docker container?
Solomon Hykes said that the "official" workflow involves building and running a new Docker image for each Git commit. That makes sense, but what if I want to test a change before committing it to the Git repo?
I can think of two ways to do it:
Run the code on a local development server (e.g., the Django development server). Edit a file; test on the dev server; make a Git commit; rebuild the Docker image with the new code; test again on the local Docker container.
Don't run a local dev server. Instead, build and run a new Docker image each time I edit a file, and then test the change on local Docker container.
Both approaches are pretty inefficient. Is there a better way?
A more efficient way is to run a new container from the latest image that was built (which then has the latest code).
You could start that container starting a bash shell so that you will be able to edit files from inside the container:
docker run -it <some image> bash -l
You would then run the application in that container to test the new code.
Another way to alter files in that container is to start it with a volume. The idea is to alter files in a directory on the docker host instead of messing with files from the command line from the container itself:
docker run -it -v /home/joe/tmp:/data <some image>
Any file that you will put in /home/joe/tmp on your docker host will be available under /data/ in the container. Change /data to whatever path is suitable for your case and hack away.