Develop Reference

How to connect 2 docker compontens within the same docker-compose.yaml

I am new to the docker world and I have some issues regarding how to connect 2 docker services tougher.
I am using https://memgraph.com/ as my database and when I am running it locally I am running it like this
docker run -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform
I wrote my program which is going to connect to the database using mgclient and when I am running it locally everything is working fine.
Now I am trying to put in inside the docker container and running it using docker-compose.yaml
My docker-compose.yaml is:
version: "3.5"
services:
memgraph:
image: memgraph/memgraph-platform:2.1.0
container_name: memgraph_container
restart: unless-stopped
ports:
- "7687:7687"
- "3000:3000"
my_app:
image: memgraph_docker
container_name: something
restart: unless-stopped
command: python main.py
and when I am trying to run it with this command:
docker-compose up
I am getting an error regarding the connection to the server. Could anyone tell me what I am missing regarding the docker-compose.yaml?

How does your my_app connect to the database?
Are you using a connection string of the form localhost:7687 (or perhaps localhost:3000)? This would work locally because you are publishing (--publish=7687:7687 --publish=3000:3000) the container's ports 7687 and 3000 to the host port's (using the same ports).
NOTE You can remap ports when your docker run. For example, you could --publish=9999:7686 and then you would need to use port 9999 on your localhost to access the container's port 7687.
When you combine the 2 containers using Docker Compose, each container is given a name that matches the service name. In this case, your Memgraph database is called memgraph (matching the service name).
Using Docker Compose, localhost takes on a different mean. From my_app, localhost is my_app. So, using localhost under Docker Compose, my_app would try connecting to itself not the database.
Under Docker Compose, for my_app (the name for your app), you need to refer to Memgraph by its service name (memgraph). The ports will be unchanged as both 7687 and 3000 (whichever is correct).
NOTE The ports statement in your Docker Compose config is possibly redundant *unless you want to be able to access the database from your (local)host (which you may for debugging). From a best practice standpoint, once my_app is able to access the database correctly, you don't need to expose the database's ports to the host.
Update
It is good practice to externalize configuration (from your app). So that you can configure your app dynamically. An easy way to do this is to use environment variables.
For example:
main.py:
import os
conn = connect(
host=os.getenv("HOST"),
port=os.getenv("PORT"),
)
Then, when you run under e.g. Docker, you need to set these values:
docker run ... --env=HOST="localhost" --env=PORT="7687" ...
And under Docker Compose, you can:
version: "3.5"
services:
memgraph:
image: memgraph/memgraph-platform:2.1.0
container_name: memgraph_container
restart: unless-stopped
my_app:
image: memgraph_docker
container_name: something
restart: unless-stopped
command: python main.py
environment:
HOST: memgraph
PORT: 7687

Can't connect python (Flask) with Mysql container using docker-compose [duplicate]

I am trying to run integration tests (in python) which depend on mysql. Currently they depend on SQL running locally, but I want them to depend on a MySQL running in docker.
Contents of Dockerfile:
FROM continuumio/anaconda3:4.3.1
WORKDIR /opt/workdir
ADD . /opt/workdir
RUN python setup.py install
Contents of Docker Compose:
version: '2'
services:
mysql:
image: mysql:5.6
container_name: test_mysql_container
environment:
- MYSQL_ROOT_PASSWORD=test
- MYSQL_DATABASE=My_Database
- MYSQL_USER=my_user
- MYSQL_PASSWORD=my_password
volumes:
- db_data:/var/lib/mysql
restart: always
expose:
- "3306"
my_common_package:
image: my_common_package
depends_on:
- mysql
restart: always
links:
- mysql
volumes:
db_data:
Now, I try to run the tests in my package using:
docker-compose run my_common_package python testsql.py
and I receive the error
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on
'localhost' ([Errno 99] Cannot assign requested address)")

docker-compose will by default create virtual network were all the containers/services in the compose file can reach each other by an IP address. By using links, depends_on or network aliases they can reach each other by host name. In your case the host name is the service name, but this can be overridden. (see: docs)
Your script in my_common_package container/service should then connect to mysql on port 3306 according to your setup. (not localhost on port 3306)
Also note that using expose is only necessary if the Dockerfile for the service don't have an EXPOSE statement. The standard mysql image already does this.
If you want to map a container port to localhost you need to use ports, but only do this if it's necessary.
services:
mysql:
image: mysql:5.6
container_name: test_mysql_container
environment:
- MYSQL_ROOT_PASSWORD=test
- MYSQL_DATABASE=My_Database
- MYSQL_USER=my_user
- MYSQL_PASSWORD=my_password
volumes:
- db_data:/var/lib/mysql
ports:
- "3306:3306"
Here we are saying that port 3306 in the mysql container should be mapped to localhost on port 3306.
Now you can connect to mysql using localhost:3306 outside of docker. For example you can try to run your testsql.py locally (NOT in a container).
Container to container communication will always happen using the host name of each container. Think of containers as virtual machines.
You can even find the network docker-compose created using docker network list:
1b1a54630639 myproject_default bridge local
82498fd930bb bridge bridge local
.. then use docker network inspect <id> to look at the details.
Assigned IP addresses to containers can be pretty random, so the only viable way for container to container communication is using hostnames.

Python-Docker container cannot connect to MariaDB-Docker container

I'm trying to get my dockerized python-script to get data from an also dockerized mariadb.
I know this should be possible with networks or links. However, due to links being deprecated (According to the Docker documentation), I'd rather not use links.
docker-compose:
version: "3.7"
services:
[...]
mariadb:
build: ./db
container_name: maria_db
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: user
MYSQL_PASSWORD: user
restart: always
networks:
- logrun_to_mariadb
[...]
logrun_engine:
build: ./logrun_engine
container_name: logrun_engine
restart: always
networks:
- logrun_to_mariadb
networks:
logrun_to_mariadb:
external: false
name: logrun_to_mariadb
The logrun_engine container executes a python-script on startup:
import mysql.connector as mariadb
class DBConnector:
def __init__(self, dbname):
self.mariadb_connection = mariadb.connect(host='mariadb', port='3306', user='root', password='root', database=dbname)
self.cursor = self.mariadb_connection.cursor()
def get_Usecases(self):
self.cursor.execute("SELECT * FROM Test")
tests = []
for test in self.cursor:
print(test)
print("Logrun-Engine running...")
test = DBConnector('test_db')
test.get_Usecases()
Whenever I run docker-compose up -d, my logrun_engine logs are full of the error message:
_mysql_connector.MySQLInterfaceError: Can't connect to MySQL server on 'mariadb' (111)
When I run the python script locally and connect to a local mariadb, it works with no problems, so the script should be correct.
Most answers I found concerning this error-message are that the people used localhost or 127.0.0.1 instead of the docker container, which I already have.
I tried with bridged networks, host networks, links etc. but apparently I haven't found the correct thing yet.
Any idea how to connect these two containers?

OK, so I was just too impatient and didn't let mysql start up properly before querying the database, thanks #DanielFarrel for pointing that out.
When I added a 10sec delay in the python script before querying the database, it magically worked...

Sleep maybe one solution. However, it may be problematic in case db goes up slowly.
As an alternative you can use agent that will make sure db is up before connecting to it similar to solution here.
Run:
docker-compose up -d agent
After agent is up you are sure db is up and you app may run:
docker-compose up -d logrun_engine
The solution does use --links, however it can be easily modified to use docker networks.

sharing data over docker compose between two python servers

sharing a buffer of doubles between two python webservers(collector and calculator) over docker-compose
I am trying to simply send a buffer or an array of integers from a python server called collector to another one called calculator. calculator server should perfom simple mathimatical algorithim. This is all a trial. collector and calculator python scripts are runned in docker-compose in two containers and designed to be connected to the same network.
collector python script
app=Flask(__name__)
#app.route('/')
def index():
d={"my_number": list(range(10))}
return jsonify(d)
calculator python script
import requests
r=requests.get('https://collector:5000')
app = Flask(__name__)
#app.route('/')
def index():
numbers_array = r.json()["my_numbers"]
x=numbers_array[1] + numbers_array[2]
return '{}'.format(x)
docker-compose.yml
services:
collector:
build: .
env_file:
- collector.env
ports:
- '5000:5000'
volumes:
- '.:/app'
networks:
- my_network
calculator:
build: ./calculator
depends_on:
- collector
env_file:
- calculator.env
ports:
- '5001:5000'
volumes:
- './calculator:/app'
networks:
- my_network
networks:
my_network:
driver: bridge
Dockerfile for both images is the same
FROM python:2.7-slim
RUN mkdir /app
WORKDIR /app
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY . .
LABEL maintainer="Mahmoud KD"
VOLUME ["/app/public"]
CMD flask run --host=0.0.0.0 --port=5000
when I run the docker-compose up --build, the first server,collector is achievable on my computer host and is working fine. The second server, calculator, fails to connect to collector via request.get. I tried to ping collector from calculator container while the docker-compose is running the two containers and the ping didn't function, it says " executable file not found in PATH: unknown". it seems that the connection of the two containers is not established although while doing inspection of my_network it shows the two containers. Can any body tell me what I am doing wrong. I am very grateful...

Use expose instead
one app on port 5000
other on port 5001
docker-compose:
app1:
expose:
- 5000
app2:
expose:
- 5001
make sure you run apps with ip=0.0.0.0
If you want to access app 2 from hostmachine, forward ports
app2:
expose:
- 5001
ports:
- 80:5001
Explanation:
Expose only reveales ports inside docker world. So if you expose container's A port 8888, all other containers will be able to access that container at that port. But you will never reach it from host machine.
Standard procedure is that you forward only one port, that is 80 from security reasons and the rest of traffic is unreachable from outside world
Also change dockerfile. You dont want hardcoded ports
Edit:
Also get rid of this
volumes:
- '.:/app'
It may actually cause extra troubles
Working Example: - it works, but the provided app contains errors
docker-compose.yml
version: '3.5'
services:
collector:
container_name: collector
build:
context: collector/.
ports:
- '80:5555'
expose:
- '5555'
calculator:
container_name: calculator
build:
context: calculator/.
depends_on:
- collector
expose:
- 6666
ports:
- '81:6666'
volumes:
- './calculator:/app'
You can access both endpoints on ports 80 and 81. Communication between both endpoints are hidden from us and its on 5555 and 6666. If you close 81(or 80), you can access the other endpoint only as 'proxy'

One file to start all services... mongodb, redis, node ,angular and python

Well, my question is. How to create a file which can start node angular, python main_worker.py, MongoDB and redis? I really do not know where to start.
I just wanna start my web program without opening 7 consoles to start each service like python worker angular node and databases.

I know about angular and MongoDB others are not, will it be your help? try the following ways but you need one console
"scripts": {
"dev": "concurrently \"mongod\" \"ng serve --proxy-config proxy.conf.json --open\" \"tsc -w -p server\" \"nodemon dist/server/app.js\"",
"prod": "concurrently \"mongod\" \"ng build --aot --prod && tsc -p server && node dist/server/app.js\""
},

You can use Docker Compose to start all your services with a single command:
docker-compose up
Learn more about it here: https://docs.docker.com/compose/reference/up/
You will need to create a docker-compose.yml in your project which will looks something like:
version: "3.5"
services:
mongodb:
container_name: mongo
hostname: mongo
image: mongo
restart: always
volumes:
- mongo_data:/var/lib/mongo/data
networks:
- your-app-network
ports:
- 27017:27017
environment:
- YOUR_VARIABLE:value
redis:
container_name: redis
hostname: redis
image: redis
restart: always
volumes:
- rediso_data:/var/lib/redis/data
networks:
- your-app-network
ports:
- 6380:6380
environment:
- YOUR_VARIABLE:value
volumes:
mongo_data:
redis_data:
networks:
go-app:
name: your-app-network
Note, the sample above is not ready to use docker compose file. It just shows you the idea how you do it. You will have to edit it and add some variable and settings specific to your application as well as add more services like node.js, python, etc.