Develop Reference

flask app service can't connect to postgres service within docker-compose

I am trying to run flask, postgres and nginx services with following docker-compose:
version: '3.6'
services:
postgres:
image: postgres:10.5
container_name: postgres
hostname: postgres
user: postgres
ports:
- "5432:5432"
networks:
- db-tier
environment:
CUSTOM_CONFIG: /etc/postgres/postgresql.conf
volumes:
- ./postgres/sql/create_tables.sql:/docker-entrypoint-initdb.d/create_tables.sql
- ./postgres/postgresql.conf:/etc/postgres/postgresql.conf
command: postgres -c config_file=/etc/postgres/postgresql.conf
restart: always
app:
image: python/app:0.0.1
container_name: app
hostname: app
build:
context: .
dockerfile: Dockerfile
depends_on:
- postgres
networks:
- app-tier
- db-tier
stdin_open: true
nginx:
image: nginx:1.22
container_name: nginx-reverse-proxy-flask
ports:
- "8080:80"
depends_on:
- app
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
networks:
- app-tier
networks:
app-tier:
driver: bridge
db-tier:
driver: bridge
This is what app.config["SQLALCHEMY_DATABASE_URI"] equals to postgresql://denis:1234Five#postgres:5432/app
The error after docker-compose up is:
app | psycopg2.OperationalError: could not connect to server: Connection refused
app | Is the server running on host "postgres" (192.168.32.2) and accepting
app | TCP/IP connections on port 5432?
What could cause this type of error? I double checked the container name of postgres service, and it running with this name postgres why flask app doesn't "see" it?

It could be issue with no propper startup probe in your postgres container and docker-compose.yml
Look at for reference how to setup them at Docker - check if postgres is ready
So after postgres starts as container, docker starts your app, but postgres inside of container is not ready yet and you get your error

This issue was resolved with postgres pg_isready adding following lines to postgres service:
healthcheck:
test: ["CMD-SHELL", "sh -c 'pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}'"]
interval: 10s
timeout: 3s
retries: 3
Took as solution from here Safe ways to specify postgres parameters for healthchecks in docker compose

Error -2 connecting to redis://redis:6379:6379. Name or service not known

I was hoping to get some insight to what I am missing, currently trying to run a docker-compose config with python (walrus for db wrapper) and redis image, but I keep receiving the same error:
redis.exceptions.ConnectionError: Error -2 connecting to redis://redis:6379. Name or service not known.
I tried different solutions on stack overflow to fix this but still nothing is working.
Here are the related docker-compose config:
version: '3.3'
services:
redis:
image: redis:latest
container_name: redis
ports:
- "6379:6379"
command: ["redis-server"]
entrypoint: redis-server --appendonly yes
consumers:
build: ./consumers
container_name: consumers
environment:
- REDIS_HOST=redis://redis
command: "./run.sh"
depends_on:
- redis
restart: always
tty: true
networks:
default:
driver: bridge
Dockerfile:
FROM python:3.10
WORKDIR /consumers
# Copy Dependencies
COPY requirements.txt requirements.txt
COPY run.sh .
# Install Dependencies
RUN pip install -r requirements.txt
COPY . .
ENV REDIS_HOST=redis://redis
RUN chmod a+x run.sh
# Run executable consumer.py
CMD [ "./run.sh"]
And connection in python using walrus to redis:
rdb = Database(host=os.getenv("REDIS_HOST", "localhost"), port=6379)
Locally without docker the setup works fine. Any direction in this case would be really appreciated.
Thank you

The following configuration made it work, removed entrypoint, created a new custom network and exposed port. REDIS_HOST was modified to redis i.e. container name. All together made it work but while trying only one of these the problem persisted.
version: '3.5'
services:
redis:
image: redis:latest
container_name: redis
ports:
- "6379:6379"
expose:
- 6379:6379
command: ["redis-server"]
networks:
- connections
consumers-g1:
build: ./consumers
container_name: consumers-g1
environment:
- REDIS_HOST=redis
command: "./run.sh"
expose:
- 6379:6379
networks:
- connections
restart: always
tty: true
networks:
connections:
name: connections
driver: bridge

How two diffrent docker container api communicate with each other requests?

In one container there Django app running which expose to host port 8000
and another container flask app is running which expose to the host port 8001. So there is a condition where the flask app API end needs to communicate with the Django app API end.
Code
req = requests.get('http://192.168.43.66:8000/api/some')
Error
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.43.66', port=8000): Max retries exceeded with url: /api/user (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc6f96942e0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
But if I change the request URL to some other API end which not running on the container it gets the response.
And each Django API end is working fine if I want to access it through some other external source like postman or browser.
Here is the docker-compose file content
Django App docker-compose.yml
version: '3.8'
services:
backend:
build:
context: .
dockerfile: Dockerfile
command: python3 manage.py runserver 0.0.0.0:8000
ports:
- 8000:8000
volumes:
- .:/app
depends_on:
- db
db:
image: mysql:5.7.22
restart: always
environment:
MYSQL_DATABASE: admin
MYSQL_USER: root
MYSQL_PASSWORD: root
MYSQL_ROOT_PASSWORD: root
volumes:
- .dbdata:/var/lib/mysql
ports:
- 33066:3306
flask app docker-compose.yml
version: '3.8'
services:
backend:
build:
context: .
dockerfile: Dockerfile
network: 'host'
command: 'python3 main.py'
ports:
- 8001:5000
volumes:
- .:/app
depends_on:
- db
db:
image: mysql:5.7.22
restart: always
environment:
MYSQL_ROOT_HOST: '%'
MYSQL_DATABASE: main
MYSQL_USER: root
MYSQL_PASSWORD: root
MYSQL_ROOT_PASSWORD: root
volumes:
- .dbdata:/var/lib/mysql
ports:
- 33067:3306
Note: in the project YAML is well formatted so that's not an error.

when you expose a port from container, the host can access it but it doesn't mean other containers can access it too.
you either have to set the network mode to host (which won't work on windows, this is only possible if you're running them on linux).
OR you can run it in a docker-compose and define your own network, here is an example:
version : '3.4'
services:
UI:
container_name: django_app
image: django_image
networks:
my_net:
ipv4_address: 172.26.1.1
ports:
- "8080:8000"
api:
container_name: flask_app
image: flask_image
networks:
my_net:
ipv4_address: 172.26.1.2
networks:
my_net:
ipam:
driver: default
config:
- subnet: 172.26.0.0/16
now your django app can access your flask app on 172.26.1.2
EDIT :
now that you have added your docker-compose files too, you should not create apps in two different docker-compose (that's why you were getting the conflicting ip range error, you were building two networks with same range).
place everything in a single docker-compose, give them ip addresses and you should be fine.
you could make your apps to read ip address of services they relate on from environment variables and then pass these env variables to your container for more flexibility.

Python urllib.urlretrieve() error in Docker

I recently developed a Python script to retrieve medical images from Open-I in Python. This script worked locally, but now that I have put this script in a Docker container the script errors when I try to retrieve the image. The output in the console only says: error. Can someone please explain to me what I am doing wrong?
import urllib
urllib.urlretrieve('https://openi.nlm.nih.gov/'+record['imgLarge'], path+str(filename))
Docker compose file:
version: '3.2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
tty: true
networks:
- retriever
kafka:
image: wurstmeister/kafka:latest
ports:
- target: 9094
published: 9094
protocol: tcp
mode: host
environment:
HOSTNAME_COMMAND: "docker info | grep ^Name: | cut -d' ' -f 2"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_CREATE_TOPICS: "rontgen_images"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- retriever
tty: true
python:
build:
context: .
dockerfile: DockerfilePython
tty: true
networks:
- retriever
networks:
retriever:
driver: bridge

Can not connect to the container based on postgres

I am new to docker container. I am trying to unit test my Flask application on Circle CI automatically. However it can not connect to postgres container. It works in my local computer (macOS Sierra). Let me know if you need more information to solve this issue. Thank you!!
docker-compose.yml
version: '3'
services:
web:
container_name: web
build: ./web
ports:
- "5000:5000"
depends_on:
- postgres
volumes:
- ./web/.:/app
tty: true
postgres:
container_name: postgres
build: ./db
ports:
- "5432:5432"
config.yml
version: 2
jobs:
build:
machine: true
working_directory: ~/repo
steps:
- checkout
- run:
name: Install Docker Compose
command: |
sudo curl -L https://github.com/docker/compose/releases/download/1.16.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
- run:
name: Start container and veryfy it's working
command: |
set -x
cd ~/repo/docker
docker-compose up --build -d
- run:
name: Run test
command: |
cd ~/repo/docker
docker-compose run web python tests/test_therapies.py
Circle Ci build log
connection = pool._invoke_creator(self)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "postgres" (172.18.0.2) and accepting
TCP/IP connections on port 5432?
----------------------------------------------------------------------
Ran 1 test in 0.025s
FAILED (errors=1)
Exited with code 1

I think the problem is that the postgres service is not fully up when your web app starts. Based on your comment about it working after adding the sleep timer, it seems this is the problem.
You can run a container called dadarek/wait-for-dependencies as a mechanism to wait for services to be up(in your case, postgres).
Here is how you can implement it:
1). Add a new service to your docker-compose.yml
waitfordb:
image: dadarek/wait-for-dependencies
depends_on:
- postgres
command: postgres:5432
Your docker-compose.yml should look now look like this:
version: '3'
services:
waitfordb:
image: dadarek/wait-for-dependencies
depends_on:
- postgres
command: postgres:5432
web:
container_name: web
build: ./web
ports:
- "5000:5000"
depends_on:
- waitfordb
- postgres
volumes:
- ./web/.:/app
tty: true
postgres:
container_name: postgres
build: ./db
ports:
- "5432:5432"
2). Startup compose
docker-compose run --rm waitfordb
docker-compose up -d web postgres
The result is that your web service should now wait for port 5432 to be up in your postgres container, before trying to start up.