Develop Reference

sh: flake8: not found, though I installed it with python pip

Here I'm going to use flake8 within a docker container. I installed flake8 using the following command and everything installed successfully.
$ sudo -H pip3 install flake8 // worked fine
and it's location path is-
/usr/local/lib/python3.8/dist-packages
Then I executed the following command but result was not expected.
$ sudo docker-compose run --rm app sh -c "flake8"
It says,
sh: flake8: not found
May be, the flake8 package is not installed in the correct location path. Please help me with this issue.

When you docker-compose run a container, it is in a new container in a new isolated filesystem. If you ran pip install in a debugging shell in another container, it won't be visible there.
As a general rule, never install software into a running container, unless it's for very short-term debugging. This will get lost as soon as the container exits, and it's very routine to destroy and recreate containers.
If you do need a tool like this, you should install it in your Dockerfile instead:
FROM python:3.10
...
RUN pip3 install flake8
However, for purely developer-oriented tools like linters, it may be better to not install them in Docker at all. A Docker image contains a fixed copy of your application code and is intended to be totally separate from anything on your host system. You can do day-to-day development, including unit testing and style checking, in a non-Docker virtual environment, and use docker-compose up --build to get a container-based integration-test setup.

Airflow: Module not found

I am trying to schedule a python script in airflow.
When the script start to execute it fails to find specific mysql module.
I already installed mysql-connector.
So it should not be a problem.
Please help me here.

Since you are running airflow in a docker image, the python lib you use should be install in the docker image.
For example, if you are using the image apache/airflow, you can create a new docker image based on this image, with all the libs you need:
Dockerfile:
FROM apache/airflow
RUN pip install mysql-connector
Then you build the image:
docker build -t my_custom_airflow_image <path/to/Dockerfile>
Then you can replace apache/airflow in your docker-compose file by my_custom_airflow_image, and it will work without any problem.

How to create a new docker image based on an existing image, but including more Python packages?

Let's say I've pulled the NVIDIA NGC PyTorch docker image like this:
docker pull nvcr.io/nvidia/pytorch:21.07-py3
Then I want to add these python packages: omegaconf wandb pycocotools?
How do I create a new Docker image with both the original Docker image and the additional Python packages?
Also, how do I distribute the new image throughout my organization?

Create a file named Dockerfile. Add to it the lines explained below.
Add a FROM line to specify the base image:
FROM nvcr.io/nvidia/pytorch:21.07-py3
Upgrade Pip to the latest version:
RUN python -m pip install --upgrade pip
Install the additional Python packages that you need:
RUN python -m pip install omegaconf wandb pycocotools
Altogether, the Dockerfile looks like this:
FROM nvcr.io/nvidia/pytorch:21.07-py3
RUN python -m pip install --upgrade pip
RUN python -m pip install omegaconf wandb pycocotools
In the same directory as the Dockerfile, run this command to build the new image, replacing my-new-image with a name of your choosing:
docker build -t my-new-image .
This works for me, but Pip generates a warning about installing packages as the root user. I found it best to ignore this warning. See the note at the end of this answer to understand why.
The new docker image should now appear on your system:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my-new-image latest 082f76972805 13 seconds ago 15.1GB
nvcr.io/nvidia/pytorch 21.07-py3 7beec3ff8d35 5 weeks ago 15GB
[...]
You can now run the new image ..
$ docker run --gpus all -it --rm --ipc=host my-new-image
.. and verify that it has the additional Python packages:
# python -m pip list | grep 'omegaconf\|wandb\|pycocotools'
omegaconf 2.1.1
pycocotools 2.0+nv0.5.1
wandb 0.12.1
The Docker Hub Repositories documentation details the steps necessary to:
Create a repository (possibly private)
Push an image
Add collaborators
Pull the image from the respository
NOTE: The problem of non-root users: Although it is considered "best practices" not to run a Docker container as the root Docker user, in practice non-root users can add several complications.
You could create a non-root user in your docker file with lines like this:
RUN useradd -ms /bin/bash myuser
USER myuser
ENV PATH "$PATH:/home/myuser/.local/bin"
However, if you run the container with mounted volumes using the -v flag, then myuser will be conferred access to those volumes based on whether their userid or groupid matches a user or group in the host system. You can modify the useradd commandline to specify the desired userid or groupid, but of course the resulting image will not be portable to systems that have different ids.
Additionally, there appears to be a limitation that prevents a non-root user from accessing a mounted volume that points to an fscrypt encrypted folder. However, this works fine for me with the root docker user.
For these reasons, I found it easiest to just let the container run as root.

How to update docker images

Let's say I have a below Python code:
#!/usr/bin/pyton3
import time
while(True):
print("Hello World")
time.sleep(1)
Using above Python code I have created a docker image pythondocker by using a dockerfile. Now dockerfile contains a lots of packages which it needs to install first and then build an image. After the image is build, I can easily start/stop the container.
Now my question is, for example, I have made few changes to my Python code and I want to update docker image pythondocker with the new changes. How can I achieve this.? One way is to fisrt stop the container, then delete the image and again build it. But building an image again will take some time as it will again install all the packages. Is there any way I can stop the image instead of deleting it and then apply the changes to the current image or may be build the image but without installing the packages/dependencies which are mentioned in dockerfile.

Your Dockerfile may look like this:
FROM python:2
RUN apt-get install libxxx
ADD requirememts.txt /
RUN pip install -r /requirements.txt
ADD main.py /usr/src/app
WORKDIR /usr/src/app
RUN pip install -r /usr/src/app/requirements.txt
CMD ["python", "main.py"]
You can simply run docker build -t some_tag .. Only lines below ADD main.py /usr/src/app will be re-installed / upgraded, and lines above are installed only once when you build the image for the first time.

You should build your docker images by using docker-compose file.
Just follow any tutorial for how to use docker-compose.
and then without any manually deletion you can re-build and re-run all images by using below commands.
Build all Images
docker-compose build
Build and run all containers
docker-compose up -d
I have listed some daily useful commands for docker, have a look
https://rohanjmohite.wordpress.com/2017/08/04/docker-daily-useful-commands/

Depending on how your Dockerfile is layered, you can simply build the image again (without deleting it). It will use the cache whenever possible.
Docker will use the cache when the files in the layer (line) did not change and the preceding layers (lines) did not change as well. So if your python code sits at the bottom of your Dockerfile, it should only build this layer. Which should be fast.
After that you can run your image again.

Heroku container:push always re-installs conda packages

I've followed the python-miniconda tutorial offered by Heroku in order to create my own ML server on Python, which utilizes Anaconda and its packages.
Everything seems to be in order, however each time I wish to update the scripts located at /webapp by entering
heroku container:push
A complete re-installation of the pip (or rather, Conda) dependencies is performed, which takes quite some time and seems illogical to me. My understanding of both Docker and Heroku frameworks is very shaky, so I haven't been able to find a solution which allows me to push ONLY my code while leaving the container as is without (re?)uploading an entire image.
Dockerfile:
FROM heroku/miniconda
ADD ./webapp/requirements.txt /tmp/requirements.txt
RUN pip install -qr /tmp/requirements.txt
ADD ./webapp /opt/webapp/
WORKDIR /opt/webapp
RUN conda install scikit-learn
RUN conda install opencv
CMD gunicorn --bind 0.0.0.0:$PORT wsgi

This happens because once you updated the webapp directory, you invalidate the build cache. Whatever after this line needs to be rebuild.
When building an image, Docker steps through the instructions in your Dockerfile, executing each in the order specified. As each instruction is examined, Docker looks for an existing image in its cache that it can reuse, rather than creating a new (duplicate) image.
Once the cache is invalidated, all subsequent Dockerfile commands generate new images and the cache is not used. (docs)
Hence, to take advantage of the build cache your Dockerfile needs to be defined such as
FROM heroku/miniconda
RUN conda install scikit-learn opencv
ADD ./webapp /opt/webapp/
RUN pip install -qr /opt/webapp/requirements.txt
WORKDIR /opt/webapp
CMD gunicorn --bind 0.0.0.0:$PORT wsgi
You should merge the two RUN conda commands to a single statement, to reduce number of layers in the image. Also, merge the ADD into single command and run pip requirements from a different directory.