I am running a dockerized python web application that has to run long tasks on certain requests (i.e. running some R scripts taking around 1 minute to complete). At the moment I put everything in one container and I am just running it just like this.
However, I think that it would be faster and cleaner to separate this 'background web app' and the R scripts one process = one container). I was therefore wondering if there is a way to run a container from within an other container (i.e being able to call docker run [...] on the host from the already-dockerized web application).
I tried to search for it and found some useful information on linking containers together, but in my case I'd more be interested in being able to create single-use containers on the fly.
I quite like this solution: Run docker inside a docker container? which basically allows you to use docker that's running on the host.
But if you really want to run docker in docker, here is the official solution using the dind image: https://blog.docker.com/2013/09/docker-can-now-run-within-docker/
Related
I'm using Lambda container images to package complicated libraries like opencv and pdf2image in Python.
Is there a way to run unit tests against it so I can get a code coverage for tools like Sonar?
With normal code, I could do the following:
python -m unittest -v
But not sure how to do that if the code is inside a container image.
I'm using bitbuckett pipelines as well.
In order to run Unit tests inside a container there are different solutions, and they mostly depends on where you are going to run the image.
Assuming you are going to run the image locally and assuming you are proficient with docker, you could build your image including development dependencies and then you could modify the entrypoint of the image accordingly in order to run your unit test suite. I suggest to bind a path to the local host in order to be able to retrieve possible junit.xml (or any test report) files. This way, you just do:
docker run --entrypoint="/path/to/my/ut/runner" -v /tmp/testout:/container/path myimage
Assuming you want to run the lambda remotely I suggest either to bind it to any APIGateway and to perform a remote API call (maybe create an endpoint just for development purposes that returns the test report), or to use additional tools like Terratest to invoke the lambda remotely without any additional infrastructure (note that this require using Terraform). Finally note that AWS Serverless Application Model documentation gives additional examples on how you could run your tests inside a lambda.
I'm developing a Python application for Machine Learning models, you can see my docker-compose file here: https://github.com/Quilograma/IES_Project/blob/main/docker-compose.yml.
The problem is while developing the application, every time I change a line in the Python code I've to kill all active containers and drop their respective images. Then call docker-compose up to see the change I've made. It takes roughly 5 minutes to pull all Docker images and install the Python libraries again, which significantly slows down the process of development.
Is there any turnaround to overcome this issue? I really want to use the containers. Thanks!
You do not need to remove any images. You need to re-build your image. This means all previous image layers (FROM python:<tag>, RUN pip install <packages...>) would be cached.
The alternative solution (but only because Python in an interpreted language) would be to mount your module as a volume. Then when you save in your host filesystem, it is automatically updated inside the container.
Personal example with Flask server and Kafka connection
The current workflow I have is that I created many images of different python setups, which people can pull from if they want to test a python script with a certain configuration. Then they build the container form the image and transfer the scripts, data, etc... from their local machine to the container. Next they run it in the container, and then they finally transfer the results back to their local machine.
I'm new to docker, so is there a better way to go about this? Something I have in mind that would be convenient is if there was a central machine or docker container where people could save their python scripts and data they need to run their tests and then run them in the image of the python environment they want to and save the results. Is this possible? I've been reading about volumes and think I can maybe do something with that, but I don't know...
I have a python script that looks like this:
def process_number(num):
# Do some processing using this number
print(num)
I want to spin up a kubernetes cluster and pass in a range of numbers, and have this script run in parallel across many machines. The number range can be hardcoded. I am unsure how to set up the dockerfile for this application, and how to deploy it to kubernetes, as most of the examples I can find are for web apps.
You need a base image that has python. You'll also need to define more of the script to ensure that this function runs over and over again, otherwise, the container will run and end very quickly which leads to crashLoopBackoff.
Before worrying about how to deploy this in kubernetes, you need to make sure you have a working script and that you can containerize it.
I recommend you containerize your application first and ensure it is running well before move to the kubernetes.
You can see some example of how containerize it in theses links:
Dockerize your Python Application
Containerize a Python App in 5 Minutes
After ensure your application in running as expect in a container, them you can move to he next step that is deploy it in kubernetes.
I strongly recommend you start with some kubernetes tutorials to understand how the components works and how you can deploy your application. You can use this links:
Local Development Environment for Kubernetes using Minikube
Getting Started With Kubernetes #1
Getting Started with Kubernetes #2
Kubernetes: An Introduction
kubernetes the hard way
k8s-intro-tutorials
If you don't have too much time, or you just need something that work for you case, take a look in Kubernetes Deployment, you can use your containerize application across multiples replicas.
To expose your service (internally and externally) you could use Kubernetes Service.
To test locally you can use the minikube installation on your machine
Hope that helps!
I'm creating microservices using Docker Containers.
Initially, I run one Docker container and it provides me with some output that I need as input for a second docker container.
So the flow of steps would be:
Run Docker container;
Get output;
Trigger running of second Docker container with previous output.
I have looked into Kubernetes, cloud functions and pub/sub on Google Cloud. Though I would like to run this locally first.
Unfortunately, I haven't found a solution, my processes are more like scripts than web-based applications.
If you are asking from a auto-devops point of view, you can try utilizing kubectl wait command for this.
Prepare yaml files for each of the containers (eg pod, deployment or statefulset)
Write a shell script, running kubectl apply -f on the files.
Use kubectl wait and jsonpath to pass info from prvious pods to the next, once it's in the ready / RUNNING state.
Find the detailed docs here.
You can have a look at Kubernetes Jobs. It might do the trick, although Kubernetes was not really made for running scripts in sequence and sharing dependencies between containers.
This thread is similar to what you need. One tool mentioned that intrigued me was brigade.