Develop Reference

run tests on a lambda container image?

I'm using Lambda container images to package complicated libraries like opencv and pdf2image in Python.
Is there a way to run unit tests against it so I can get a code coverage for tools like Sonar?
With normal code, I could do the following:
python -m unittest -v
But not sure how to do that if the code is inside a container image.
I'm using bitbuckett pipelines as well.

In order to run Unit tests inside a container there are different solutions, and they mostly depends on where you are going to run the image.
Assuming you are going to run the image locally and assuming you are proficient with docker, you could build your image including development dependencies and then you could modify the entrypoint of the image accordingly in order to run your unit test suite. I suggest to bind a path to the local host in order to be able to retrieve possible junit.xml (or any test report) files. This way, you just do:
docker run --entrypoint="/path/to/my/ut/runner" -v /tmp/testout:/container/path myimage
Assuming you want to run the lambda remotely I suggest either to bind it to any APIGateway and to perform a remote API call (maybe create an endpoint just for development purposes that returns the test report), or to use additional tools like Terratest to invoke the lambda remotely without any additional infrastructure (note that this require using Terraform). Finally note that AWS Serverless Application Model documentation gives additional examples on how you could run your tests inside a lambda.

Most secure way to store secrets while building a docker image (Python flask server)

So currently I am building a docker image for my python flask server, and the way that I am accessing the secret variable inside the code is
app.config["SECRET_KEY"] = os.environ.get("todo_secret_key") or "secret"
Now if I want to run the code without the container, I'd use export command(linux) to temporarily have the secret key in the environment, but I need it to be in the environment of the container.
Now, a few methods that I am aware of are
Pass it with -e in the docker run command docker run {imageName} -e {name}={value} (insecure as I dont want it to be in terminal logs)
Pass it in the dockerfile by specifying the environment variable (definitely insecure as this file would be public)
Apart from these methods, is there a more secure way to pass the variable
P.s It is my first time buidling an image so apologies if it is a silly question

You're trying to tie your runtime configuration to your build time configuration much too tightly if you're worrying about this as you're building your docker image!
Remember that these are discrete, if interconnected, stages.
Develop -> Build -> Test -> Deploy -> Test (Non-functional) -> repeat...
No stage should dictate what future stages do. docker build is the "Build" stage, and so should have no idea what configuration is correct in the environment for when the image is eventually deployed. How could it? That configuration could change over time!
Instead, look to configuration management to handle these sorts of decisions. Your docker scheduler (Kubernetes is common, but other flyweight executors like Docker Swarm or ECS should do this too) should have a way to read a configuration file and inject it into the environment. A full discussion of configuration management could (and has) filled textbooks.

How to update gcloud on Google Cloud Composer worker nodes?

There's a similar question here but from 2018 were the solution requires changing the base image for the workers. Another suggestion is to ssh into each node and apt-get install there. This doesn't seem useful because when auto scale spawns new nodes, you'd need to do it again and again.
Anyway, is there a reasonable way to upgrade the base gcloud in late 2020?

Because task instances run in a shared execution environment, it is generally not recommended to use the gcloud CLI within Composer Airflow tasks, when possible, to avoid state or version conflicts. For example, if you have multiple users using the same Cloud Composer environment, and either of them changes the active credentials used by gcloud, then they can unknowingly break the other's workflows.
Instead, consider using the Cloud SDK Python libraries to do what you need to do programmatically, or use the airflow.providers.google.cloud operators, which may already have what you need.
If you really need to use the gcloud CLI and don't share the environment, then you can use a BashOperator with a install/upgrade script to create a prerequisite for any tasks that need to use the CLI. Alternatively, you can build a custom Docker image with gcloud installed, and use GKEPodOperator or KubernetesPodOperator to run a Kubernetes pod to run the CLI command. That would be slower, but more reliable than verifying dependencies each time.

Best Way to Use Docker For Testing With Multiple Python Configurations

The current workflow I have is that I created many images of different python setups, which people can pull from if they want to test a python script with a certain configuration. Then they build the container form the image and transfer the scripts, data, etc... from their local machine to the container. Next they run it in the container, and then they finally transfer the results back to their local machine.
I'm new to docker, so is there a better way to go about this? Something I have in mind that would be convenient is if there was a central machine or docker container where people could save their python scripts and data they need to run their tests and then run them in the image of the python environment they want to and save the results. Is this possible? I've been reading about volumes and think I can maybe do something with that, but I don't know...

Prevent user from importing os module in jupyter notebook

I am trying to set up a jupyter notebook server so that a few members can have access and run analysis on it. But there are several API credentials that I stored as the environment variables that I don't want users to have access to. Basically I want to prevent users from importing os module in the notebook, since os.environ list all environment variables on the server. What would be a proper way to do this?

You could try to run the jupyter notebook server as a Docker container. That way your environment variables will be isolated from the container. Ipython has an available docker image, so you need to install docker if this approach works for you.
Installing Ipython Docker Image
If you need to pass environment variables for the Docker container refer to this question: Passing env variables to docker