I'm using Lambda container images to package complicated libraries like opencv and pdf2image in Python.
Is there a way to run unit tests against it so I can get a code coverage for tools like Sonar?
With normal code, I could do the following:
python -m unittest -v
But not sure how to do that if the code is inside a container image.
I'm using bitbuckett pipelines as well.
In order to run Unit tests inside a container there are different solutions, and they mostly depends on where you are going to run the image.
Assuming you are going to run the image locally and assuming you are proficient with docker, you could build your image including development dependencies and then you could modify the entrypoint of the image accordingly in order to run your unit test suite. I suggest to bind a path to the local host in order to be able to retrieve possible junit.xml (or any test report) files. This way, you just do:
docker run --entrypoint="/path/to/my/ut/runner" -v /tmp/testout:/container/path myimage
Assuming you want to run the lambda remotely I suggest either to bind it to any APIGateway and to perform a remote API call (maybe create an endpoint just for development purposes that returns the test report), or to use additional tools like Terratest to invoke the lambda remotely without any additional infrastructure (note that this require using Terraform). Finally note that AWS Serverless Application Model documentation gives additional examples on how you could run your tests inside a lambda.
Related
I'm building a simple kind 'automation' on container image builder.
I wrote it in python, and use nixpacks via python.subprocess.
docker-py python's module is use to docker run nixpacks result, push it to registry, and delete nixpacks result.
Currently it's run well in my laptop.
Now i want to containerize all my script and also nixpacks, to be run inside my kubernetes cluster.
For that I need to build/have an image that have 'docker-server' inside.
the 'docker-server' it self doesn't need to be accessible from the outside world, just 'localy' by nixpacks and the python script .
My question is: which one of docker image that currently have docker-server in it?
is this enough?
Sincerely
-bino-
In this KF example https://github.com/kubeflow/examples/blob/master/financial_time_series/tensorflow_model/ml_pipeline.py an ML pipeline gets constructed that triggers Python functions via command line.
This means that all .py files that are being called (e.g. "python3 train.py --param value") should be in the directory where the process runs. What I don't understand is where exactly should I put the .py files in the context of GCP.
Should I just copy them using Cloud shell?
Or should I add git clone <repository with .py files> into my Dockerfile?
To kickstart KFP development using python, try the following tutorial: Data passing in python components
Should I just copy them using Cloud shell? Or should I add git clone <repository with .py files> into my Dockerfile?
Ideally, the files should be inside the container image (the Dockerfile method). This ensures maximum reproducibility.
For not very complex python scripts, the Lightweight python component feature allows you to create component from a python function. In this case the script code is store in the component command-line, so you do not need to upload the code anywhere.
Putting scripts somewhere remote (e.g. cloud storage or website) is possible, but can reduce reliability and reproducibility.
The current workflow I have is that I created many images of different python setups, which people can pull from if they want to test a python script with a certain configuration. Then they build the container form the image and transfer the scripts, data, etc... from their local machine to the container. Next they run it in the container, and then they finally transfer the results back to their local machine.
I'm new to docker, so is there a better way to go about this? Something I have in mind that would be convenient is if there was a central machine or docker container where people could save their python scripts and data they need to run their tests and then run them in the image of the python environment they want to and save the results. Is this possible? I've been reading about volumes and think I can maybe do something with that, but I don't know...
I'm creating microservices using Docker Containers.
Initially, I run one Docker container and it provides me with some output that I need as input for a second docker container.
So the flow of steps would be:
Run Docker container;
Get output;
Trigger running of second Docker container with previous output.
I have looked into Kubernetes, cloud functions and pub/sub on Google Cloud. Though I would like to run this locally first.
Unfortunately, I haven't found a solution, my processes are more like scripts than web-based applications.
If you are asking from a auto-devops point of view, you can try utilizing kubectl wait command for this.
Prepare yaml files for each of the containers (eg pod, deployment or statefulset)
Write a shell script, running kubectl apply -f on the files.
Use kubectl wait and jsonpath to pass info from prvious pods to the next, once it's in the ready / RUNNING state.
Find the detailed docs here.
You can have a look at Kubernetes Jobs. It might do the trick, although Kubernetes was not really made for running scripts in sequence and sharing dependencies between containers.
This thread is similar to what you need. One tool mentioned that intrigued me was brigade.
I am running a dockerized python web application that has to run long tasks on certain requests (i.e. running some R scripts taking around 1 minute to complete). At the moment I put everything in one container and I am just running it just like this.
However, I think that it would be faster and cleaner to separate this 'background web app' and the R scripts one process = one container). I was therefore wondering if there is a way to run a container from within an other container (i.e being able to call docker run [...] on the host from the already-dockerized web application).
I tried to search for it and found some useful information on linking containers together, but in my case I'd more be interested in being able to create single-use containers on the fly.
I quite like this solution: Run docker inside a docker container? which basically allows you to use docker that's running on the host.
But if you really want to run docker in docker, here is the official solution using the dind image: https://blog.docker.com/2013/09/docker-can-now-run-within-docker/