I am trying to run integration tests (in python) which depend on mysql. Currently they depend on SQL running locally, but I want them to depend on a MySQL running in docker.
Contents of Dockerfile:
FROM continuumio/anaconda3:4.3.1
WORKDIR /opt/workdir
ADD . /opt/workdir
RUN python setup.py install
Contents of Docker Compose:
version: '2'
services:
mysql:
image: mysql:5.6
container_name: test_mysql_container
environment:
- MYSQL_ROOT_PASSWORD=test
- MYSQL_DATABASE=My_Database
- MYSQL_USER=my_user
- MYSQL_PASSWORD=my_password
volumes:
- db_data:/var/lib/mysql
restart: always
expose:
- "3306"
my_common_package:
image: my_common_package
depends_on:
- mysql
restart: always
links:
- mysql
volumes:
db_data:
Now, I try to run the tests in my package using:
docker-compose run my_common_package python testsql.py
and I receive the error
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on
'localhost' ([Errno 99] Cannot assign requested address)")
docker-compose will by default create virtual network were all the containers/services in the compose file can reach each other by an IP address. By using links, depends_on or network aliases they can reach each other by host name. In your case the host name is the service name, but this can be overridden. (see: docs)
Your script in my_common_package container/service should then connect to mysql on port 3306 according to your setup. (not localhost on port 3306)
Also note that using expose is only necessary if the Dockerfile for the service don't have an EXPOSE statement. The standard mysql image already does this.
If you want to map a container port to localhost you need to use ports, but only do this if it's necessary.
services:
mysql:
image: mysql:5.6
container_name: test_mysql_container
environment:
- MYSQL_ROOT_PASSWORD=test
- MYSQL_DATABASE=My_Database
- MYSQL_USER=my_user
- MYSQL_PASSWORD=my_password
volumes:
- db_data:/var/lib/mysql
ports:
- "3306:3306"
Here we are saying that port 3306 in the mysql container should be mapped to localhost on port 3306.
Now you can connect to mysql using localhost:3306 outside of docker. For example you can try to run your testsql.py locally (NOT in a container).
Container to container communication will always happen using the host name of each container. Think of containers as virtual machines.
You can even find the network docker-compose created using docker network list:
1b1a54630639 myproject_default bridge local
82498fd930bb bridge bridge local
.. then use docker network inspect <id> to look at the details.
Assigned IP addresses to containers can be pretty random, so the only viable way for container to container communication is using hostnames.
Related
I am new to the docker world and I have some issues regarding how to connect 2 docker services tougher.
I am using https://memgraph.com/ as my database and when I am running it locally I am running it like this
docker run -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform
I wrote my program which is going to connect to the database using mgclient and when I am running it locally everything is working fine.
Now I am trying to put in inside the docker container and running it using docker-compose.yaml
My docker-compose.yaml is:
version: "3.5"
services:
memgraph:
image: memgraph/memgraph-platform:2.1.0
container_name: memgraph_container
restart: unless-stopped
ports:
- "7687:7687"
- "3000:3000"
my_app:
image: memgraph_docker
container_name: something
restart: unless-stopped
command: python main.py
and when I am trying to run it with this command:
docker-compose up
I am getting an error regarding the connection to the server. Could anyone tell me what I am missing regarding the docker-compose.yaml?
How does your my_app connect to the database?
Are you using a connection string of the form localhost:7687 (or perhaps localhost:3000)? This would work locally because you are publishing (--publish=7687:7687 --publish=3000:3000) the container's ports 7687 and 3000 to the host port's (using the same ports).
NOTE You can remap ports when your docker run. For example, you could --publish=9999:7686 and then you would need to use port 9999 on your localhost to access the container's port 7687.
When you combine the 2 containers using Docker Compose, each container is given a name that matches the service name. In this case, your Memgraph database is called memgraph (matching the service name).
Using Docker Compose, localhost takes on a different mean. From my_app, localhost is my_app. So, using localhost under Docker Compose, my_app would try connecting to itself not the database.
Under Docker Compose, for my_app (the name for your app), you need to refer to Memgraph by its service name (memgraph). The ports will be unchanged as both 7687 and 3000 (whichever is correct).
NOTE The ports statement in your Docker Compose config is possibly redundant *unless you want to be able to access the database from your (local)host (which you may for debugging). From a best practice standpoint, once my_app is able to access the database correctly, you don't need to expose the database's ports to the host.
Update
It is good practice to externalize configuration (from your app). So that you can configure your app dynamically. An easy way to do this is to use environment variables.
For example:
main.py:
import os
conn = connect(
host=os.getenv("HOST"),
port=os.getenv("PORT"),
)
Then, when you run under e.g. Docker, you need to set these values:
docker run ... --env=HOST="localhost" --env=PORT="7687" ...
And under Docker Compose, you can:
version: "3.5"
services:
memgraph:
image: memgraph/memgraph-platform:2.1.0
container_name: memgraph_container
restart: unless-stopped
my_app:
image: memgraph_docker
container_name: something
restart: unless-stopped
command: python main.py
environment:
HOST: memgraph
PORT: 7687
I would like to have a python flask application that runs with a postgresql database (psycopg2). So I made this docker-compose file:
version: "3"
services:
web:
depends_on:
- database
container_name: web
build:
context: "."
dockerfile: "docker/Dockerfile.web"
ports:
- 5000:5000
volumes:
- database:/var/run/postgresql
database:
container_name: database
environment:
POSTGRES_PASSWORD: "password"
POSTGRES_USER: "user"
POSTGRES_DB: "products"
image: postgres
expose:
- 5432
volumes:
- database:/var/run/postgresql
volumes:
database:
In my app.py I try to connect to postgres like this:
conn = psycopg2.connect(database="products", user="user", password="password", host="database", port="5432")
When I run docker-compose up I get the following error:
"Is the server running on host "database" (172.21.0.2) and accepting TCP/IP connections on port 5432?"
I don't know where I have mistaken here.
The container "database" exposes its port 5432.
Both containers are on the same network which is "web_app_default".
The socket file existes in /var/run/postgresql directory on "web" container.
Any ideas ?
Thanks for replies and have a nice day.
I think what happened is that even though you have the flag depends_on set to database, that only means that the web container will start after database container starts. However, for the first time, the database will generally take quite some time to set up and when your web server is up, the database is still not ready to accept the connection.
2 ways to work around the problem here:
Easy way with no change in code: run docker-compose up -d (detach mode) and wait for the database to finish initializing. Then run docker-compose up -d again and your web container will now be able to connect to the database.
Second way is to update the web container with restart: always so docker-compose will keep trying to restart your web container until it runs successfully (until the database is ready to accept connection)
version: "3"
services:
web:
depends_on:
- database
...
restart: always
...
I have 5 containers but I want to give different HTTP proxy for all containers
For example, my DB container port is 8989
I added to path ~/.docker/config.json
{
"proxies":{
"default":{
"httpProxy":"http://127.0.0.1:8989",
}
}
}
Should I do for each container and how can I test this worked or not?
This is not done at the Docker configuration level, instead with port maps
Docker Run
Using Docker Run command you must do it via the flag -p or --publish:
docker run --publish 127.0.0.1:8989:8989 db
So this will map the port 127.0.0.1:8989 on the host machine to the container 0.0.0.0:8989, therefore you need to ensure your container is listening on 0.0.0.0, and not on 127.0.0.1.
Docker Compose
In docker compose you can use ports inside a service to define the ports maps:
services:
db1:
image: database-image-name
ports:
# Database reachable from the internet
- "8901:8989/tcp"
# Database only reachable via localhost
- "127.0.0.1:9901:8989/tcp"
db2:
image: database-image-name
ports:
# Database reachable from the internet
- "8902:8989/tcp"
# Database only reachable via localhost
- "127.0.0.1:9902:8989/tcp"
NOTE: this is not a full docker-compose.yml file example
If you only need to access the database from another container in the same network, then you can remove the ports from your docker composer file, and reference the database from the other container by using it's service name, that in this example is db1 or `db2.
I'm trying to get my dockerized python-script to get data from an also dockerized mariadb.
I know this should be possible with networks or links. However, due to links being deprecated (According to the Docker documentation), I'd rather not use links.
docker-compose:
version: "3.7"
services:
[...]
mariadb:
build: ./db
container_name: maria_db
expose:
- 3306
environment:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: user
MYSQL_PASSWORD: user
restart: always
networks:
- logrun_to_mariadb
[...]
logrun_engine:
build: ./logrun_engine
container_name: logrun_engine
restart: always
networks:
- logrun_to_mariadb
networks:
logrun_to_mariadb:
external: false
name: logrun_to_mariadb
The logrun_engine container executes a python-script on startup:
import mysql.connector as mariadb
class DBConnector:
def __init__(self, dbname):
self.mariadb_connection = mariadb.connect(host='mariadb', port='3306', user='root', password='root', database=dbname)
self.cursor = self.mariadb_connection.cursor()
def get_Usecases(self):
self.cursor.execute("SELECT * FROM Test")
tests = []
for test in self.cursor:
print(test)
print("Logrun-Engine running...")
test = DBConnector('test_db')
test.get_Usecases()
Whenever I run docker-compose up -d, my logrun_engine logs are full of the error message:
_mysql_connector.MySQLInterfaceError: Can't connect to MySQL server on 'mariadb' (111)
When I run the python script locally and connect to a local mariadb, it works with no problems, so the script should be correct.
Most answers I found concerning this error-message are that the people used localhost or 127.0.0.1 instead of the docker container, which I already have.
I tried with bridged networks, host networks, links etc. but apparently I haven't found the correct thing yet.
Any idea how to connect these two containers?
OK, so I was just too impatient and didn't let mysql start up properly before querying the database, thanks #DanielFarrel for pointing that out.
When I added a 10sec delay in the python script before querying the database, it magically worked...
Sleep maybe one solution. However, it may be problematic in case db goes up slowly.
As an alternative you can use agent that will make sure db is up before connecting to it similar to solution here.
Run:
docker-compose up -d agent
After agent is up you are sure db is up and you app may run:
docker-compose up -d logrun_engine
The solution does use --links, however it can be easily modified to use docker networks.
I'm trying to learn how to use docker and am having some troubles. I'm using a docker-compose.yaml file for running a python script that connects to a mysql container and I'm trying to use ddtrace to send traces to datadog. I'm using the following image from this github page from datadog
ddagent:
image: datadog/docker-dd-agent
environment:
- DD_BIND_HOST=0.0.0.0
- DD_API_KEY=invalid_key_but_this_is_fine
ports:
- "127.0.0.1:8126:8126"
And my docker-compose.yaml looks like
version: "3"
services:
ddtrace-test:
build: .
volumes:
- ".:/app"
links:
- ddagent
ddagent:
image: datadog/docker-dd-agent
environment:
- DD_BIND_HOST=0.0.0.0
- DD_API_KEY=<my key>
ports:
- "127.0.0.1:8126:8126"
So then I'm running the command docker-compose run --rm ddtrace-test python test.py, where test.py looks like
from ddtrace import tracer
#tracer.wrap('test', 'test')
def foo():
print('running foo')
foo()
And when I run the command, I'm returned with
Starting service---reprocess_ddagent_1 ... done
foo
cannot send spans to localhost:8126: [Errno 99] Cannot assign requested address
I'm not sure what this error means. When I use my key and run from local instead of over a docker image, it works fine. What could be going wrong here?
Containers are about isolation so in container "localhost" means inside container so ddtrace-test cannot find ddagent inside his container. You have 2 ways to fix that:
Put network_mode: host in ddtrace-test so he will bind to host's network interface, skipping network isolation
Change ddtrace-test to use "ddagent" host instead of localhost as in docker-compose services can be accessed using theirs names