Develop Reference

Docker django app running but cannot access the webpage

I am trying to run two separate django apps using docker (building on a linux server). The first application runs smoothly (using default ports) the second one apparently runs (it says starting development server at http://0.0.0.0:5000), I got no issues looking inside the portainer. Everything is running and no issue is there. When I try to connect to the page, it fails.
docker-compose:
version: '3'
services:
vrt:
build:
context: .
dockerfile: Dockerfile
ports:
- "5000:5000"
volumes:
- ./nuovoProgetto:/VehicleRammingTool
command: >
sh -c "python3 manage.py wait_for_db &&
python3 manage.py migrate &&
python3 manage.py runserver 0.0.0.0:5000"
env_file:
- ./.env.dev
depends_on:
- db
db:
image: postgres:14.1-alpine
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
volumes:
- db:/var/lib/postgresql/data
redis:
image: redis:alpine
celery:
restart: always
build:
context: .
command: celery -A nuovoProgetto worker --pool=solo --loglevel=info
volumes:
- ./nuovoProgetto:/VehicleRammingTool
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
depends_on:
- vrt
- redis
volumes:
db:
driver: local
Dockerfile:
FROM ubuntu:18.04
ENV http_proxy=http://++++++++++proxyhere
ENV https_proxy=http://+++++++++proxyhere
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"
RUN apt-get update
RUN apt-get install -y wget && rm -rf /var/lib/apt/lists/*
RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh
RUN python --version
RUN conda install -c conda-forge django psycopg2 celery redis-py django-leaflet django-celery- beat django-celery-results django-crispy-forms osmnx geopy geocoder pathos
RUN mkdir /VehicleRammingTool
COPY ./nuovoProgetto /VehicleRammingTool
WORKDIR /VehicleRammingTool
EXPOSE 5000
EDIT
I can cURL via command line the page using the proxy option, but still I can't get there via browser

Airflow docker-compose from another docker container on Azure Machine Learning Compute Cluster

I’m attempting to run Airflow in docker with docker-compose from inside another container. This container is created by default by an Azure Machine Learning compute cluster that I want to use to run my Airflow DAG.
The problem is that I have the following error when I try to execute docker-compose build through the ScriptRunConfig class (see script below).
error creating aufs mount to /var/lib/docker/aufs/mnt/3d54c61133023e1d3700ce9746529dc8328e739346d4123d632951f11acdf122-init: mount target=/var/lib/docker/aufs/mnt/3d54c61133023e1d3700ce9746529dc8328e739346d4123d632951f11acdf122-init data=br:/var/lib/docker/aufs/diff/3d54c61133023e1d3700ce9746529dc8328e739346d4123d632951f11acdf122-init=rw:/var/lib/docker/aufs/diff/d6e643e28d8729f8972b78bedd2de1de602879b65ff56e07e76a236d3096b709=ro+wh:/var/lib/docker/aufs/diff/8d3dde63564e16508dbf64e7baae79a6b2913b3262cb6dba7027e1fdf8bb6f8f=ro+wh:/var/lib/docker/aufs/diff/fa2e9c9ac8ff7af5f5d35e4d2081fac4bb5139d760675e9608d2b8d04f096837=ro+wh:/var/lib/docker/aufs/diff/d34747b3a1646124aa7a7a0fed4790e1264667aa58fc4de2288a10e4b68673ce=ro+wh:/var/lib/docker/aufs/diff/ea46ddfc022eb268aa428376132bc52b00b317e747d63078a38d044bae5d48ec=ro+wh:/var/lib/docker/aufs/diff/b3c155d801b49d8252a3926493328d471c8f4cfd72c553771374e3952a999d95=ro+wh:/var/lib/docker/aufs/diff/a067d04104f7a70c6194a3e742be72fc221759b18628285e7fd16a2d678120f3=ro+wh:/var/lib/docker/aufs/diff/3db994298571c09ee3d40abf21f657f9c8650a6fe0ea2c6c6c7590eb7c6c712f=ro+wh:/var/lib/docker/aufs/diff/273fc331f9ebae1d0a01963d46bf9edca6d91a089d4791066cb170954eb7609c=ro+wh:/var/lib/docker/aufs/diff/419a894bbee2b9b8ec05deed26dcfc21f234276e06d765d03ed939b918d3908f=ro+wh:/var/lib/docker/aufs/diff/e91d472c4e53f2a2eae97aca59f7dcacdf57a4b22d64991c348528e4081500d6=ro+wh:/var/lib/docker/aufs/diff/c23cc64e903b5254c19446e8ddc6db0253cbd19e239860c1dc95440ca65aae94=ro+wh:/var/lib/docker/aufs/diff/5794fefdefed444bf17de20f6c3ecf93743cccef83169c94aba62ec902c8380f=ro+wh,dio,xino=/dev/shm/aufs.xino: invalid argument
What I tried so far:
Discarded option : Using a docker:dind custom image where I installed curl and docker-compose. The problem is that it fails when building the image because:
it seems that Azure Machine Learning adds other steps to the Dockerfile to setup the conda environment needed for my project … but conda is not installed by default in the docker:dind image.
The Official docker image is based on Alpine Linux but Azure Machine Learning is only compatible with systems specifications including Ubuntu, Conda … as per the documentation
The option presented in this question: Using an image already available to Azure Machine Learning where I installed the Docker engine and docker-compose. The image is built successfully but I get the error shown above during the execution of my script by the compute cluster
Here's the Dockerfile for the custom Image (ubuntu os) used by the compute cluster to setup the environment. It is referred to as Dockerfile_cluster in the python script below.
FROM mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04:20211029.v1
USER root
# install docker
RUN apt-get update -y \
&& apt-get install -y \
ca-certificates \
curl \
gnupg \
lsb-release \
&& mkdir -p /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
&& echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null \
&& apt-get update -y \
&& apt-get install -y docker-ce docker-ce-cli containerd.io
# install docker-compose
RUN curl -L https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
RUN chmod +x /usr/local/bin/docker-compose
RUN ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
And my docker-compose.yml file (Different dockerfile than the first one) :
services:
postgres:
image: postgres:13
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- "5434:5432"
init_db:
build:
context: .
dockerfile: Dockerfile
command: bash -c "airflow db init && airflow db upgrade"
env_file: .env
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- postgres
scheduler:
build:
context: .
dockerfile: Dockerfile
restart: on-failure
command: bash -c "airflow scheduler"
env_file: .env
depends_on:
- postgres
ports:
- "8080:8793"
volumes:
- ./airflow_dags:/opt/airflow/dags
- ./data:/opt/airflow/data
- ./.git:/opt/airflow/.git
- ./conf:/opt/airflow/conf
- ./airflow_logs:/opt/airflow/logs
- /var/run/docker.sock:/var/run/docker.sock
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
webserver:
build:
context: .
dockerfile: Dockerfile
hostname: webserver
restart: always
env_file: .env
depends_on:
- postgres
command: bash -c "airflow users create -r Admin -u admin -e admin#example.com -f admin -l user -p admin && airflow webserver"
volumes:
- ./airflow_dags:/opt/airflow/dags
- ./data:/opt/airflow/data
- ./.git:/opt/airflow/.git
- ./conf:/opt/airflow/conf
- ./airflow_logs:/opt/airflow/logs
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "5000:8080"
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 32
And finally the script I use to submit my experiment to the compute cluster.
from azureml.core import Workspace, Experiment, Environment, ScriptRunConfig
from azureml.core.authentication import ServicePrincipalAuthentication
# Create an environment from conda reqs
env = Environment.from_conda_specification(name = "env_name", file_path = "./src/conda.yml")
# Use custom image from Dockerfile
env.docker.base_image = None
env.docker.base_dockerfile = "./Dockerfile_cluster"
# Instantiate ServicePrincipalAuth object
svc_pr = ServicePrincipalAuthentication(tenant_id=tenant_id,
service_principal_id=sp_id,
service_principal_password=sp_pwd)
# Instantiate AML Workspace object
ws = Workspace(
subscription_id=sub_id,
resource_group=rg_name,
workspace_name=ws_name,
auth=svc_pr
)
command ="bash -c 'service docker start && docker-compose build --no-cache'".split()
experiment = Experiment(workspace=ws, name='exp-test')
config = ScriptRunConfig(source_directory='.', command = command, compute_target='cpt-cluster', environment=env)
# Submit experiment
run = experiment.submit(config)
aml_url = run.get_portal_url()
print(aml_url)
run.wait_for_completion(show_output=True)
In the script above, I only tried to build first before running the services.
Normally, the commands I use to run my Airflow DAG locally or using a compute instance are:
docker-compose build –no-cache
docker-compose up postgres
In another terminal:
docker-compose up init_db
docker-compose up scheduler webserver
Thank you very much for your help.

Uknown command bash or /bin/bash on Docker Ubuntu:20.04 with docker-compose version 3.5

I am setting up docker with the Django application but the bash command is not working with docker Ubuntu:20.04 and docker-compose version 3.5.
My docker version is Docker version 20.10.7, build f0df350 and docker-compose version is Docker Compose version 2.0.0-beta.4
Can anyone help me to resolve the issue?
Below are my docker files:
Dockerfile:
FROM ubuntu:20.04
ENV PYTHONUNBUFFERED 1
RUN apt-get -y update
RUN apt-get install -y --no-install-recommends default-libmysqlclient-dev
RUN apt-get install -y gcc git libc-dev python3-dev python3-pip
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN mkdir /app
WORKDIR /app
ADD . /app
RUN pip install --upgrade pip && pip install -r requirements.txt
EXPOSE 8000
ENTRYPOINT [ "/app/manage.py" ]
docker-compose.yml
version: '3.5'
services:
db:
image: mysql:5.6
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: "mydb"
MYSQL_ROOT_PASSWORD: "root"
volumes:
- mysql_data:/var/lib/mysql
restart: always
networks:
default:
aliases:
- app-db
django:
build: .
command: bash -c "while true; do runserver 0.0.0.0:8000; sleep 10; done"
stdin_open: true
tty: true
volumes:
- .:/app
depends_on:
- db
ports:
- "8000:8000"
restart: always
environment:
MYSQL_DATABASE: "mydb"
MYSQL_USER: "root"
MYSQL_ROOT_PASSWORD: "root"
MYSQL_HOST: "app-db"
volumes:
mysql_data: {}
I am getting an error on command while running docker-compose up --build:
bash -c "while true; do runserver 0.0.0.0:8000; sleep 10; done"
Error:
Unknown command: 'bash'
Thanks in advance

When you run the container, the ENTRYPOINT and CMD are combined together into a single command; it doesn't matter if the command part is overridden by Docker Compose, the container still runs only a single command this way. So in effect you're asking for the main container process to be
/app/manage.py bash -c "while true; do runserver 0.0.0.0:8000; sleep 10; done"
and the complaint is that the Django runner doesn't understand manage.py bash as a subcommand.
In your Dockerfile itself, you probably want the default command to be to launch the server. Having ENTRYPOINT as an arbitrary "half of the command" tends to be a little more confusing, and leads to needing to override that too; it's probably better to just put this as the standard container CMD.
# No ENTRYPOINT, but
CMD ["/app/manage.py", "runserver", "0.0.0.0:8000"]
You don't need to put the restart loop into the container command since Docker already allows you to specify a restart policy for containers. You should be able to trim the docker-compose.yml section down to:
django:
build: .
# command: is built into the image
# don't usually need stdin_open: or tty:
# don't overwrite the image code with volumes:
depends_on:
- db
ports:
- "8000:8000"
restart: always # replaces the "while true ... done" shell loop
environment: *as-in-the-question

Use python packages from another docker container

How can I use python packages from another container?
ydk-py is set up with everything that I need, including all python packages and their dependencies.
I want to use those python packages in my django application. However python imports packages installed in my main container, web, and not ydk-py.
docker-compose:
version: '3.7'
services:
web:
container_name: webserver
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code/
ports:
- 8000:8000
env_file:
- .env.dev
depends_on:
- db
db:
container_name: database
image: postgres:13.0-alpine
volumes:
- postgres_data:/var/lib/postgresql/data/
env_file:
- .env.dev
ydk-py:
container_name: ydk-py
image: ydkdev/ydk-py:latest
tty: true
volumes:
postgres_data:
Dockerfile:
FROM python:3.6.12-alpine
WORKDIR /code
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
RUN apk update && apk add jpeg-dev zlib-dev postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip
COPY ./requirements.txt /code/requirements.txt
RUN pip install -r requirements.txt
COPY ./entrypoint.sh /code/entrypoint.sh
COPY . /code
ENTRYPOINT ["sh", "/code/entrypoint.sh"]

you should be able to use ydk-py as your base image to build your application
FROM ydkdev/ydk-py:latest
...

package in requirements.txt, but not seen in docker

I have valid requirements.txt file, but docker doesn't install one of packages listed in requirements
Docker version 18.09.2
python 3.7.3
requirements.txt
django==2.2.2
celery==4.2.1
selenium==3.141.0
BeautifulSoup4==4.7.1
redis==3.2.0
Dockerfile
FROM python:3.7
ENV PYTHONUNBUFFERED 1
ENV DJANGO_ENV dev
ENV DJANGO_ENV dev
COPY ./requirements.txt /code/requirements.txt
RUN pip3 install --upgrade pip
RUN pip3 install -r /code/requirements.txt
COPY . /code/
WORKDIR /code/
EXPOSE 8000
docker_compose.yml
version: '3'
services:
db:
image: postgres:9.6.5
volumes:
- postgres_data:/var/lib/postgresql/data/
redis:
image: "redis:alpine"
web:
build: .
command: bash -c "python /code/manage.py migrate --noinput && python /code/manage.py runserver 0.0.0.0:8000"
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
- redis
celery:
build: .
command: celery -A google_scraper worker -l info
volumes:
- .:/code
depends_on:
- db
- redis
volumes:
postgres_data:
actual error while I try to run celery:
ImportError: Missing redis library (pip install redis)
When I run bash for docker, and install it by manual pip3 install redis - it works fine, but still it doesn't solve problem of building it, I don't have idea where I make mistake.

I wonder how come Docker did not complain about a missing /code directory? Are you sure Docker have successfully built the image? If it did, please explain to me how the COPY ./requirements.txt /code/requirements.txt worked? I checked, the python:3.7 image does not have that directory in its root file-system...