I am trying to connect to MySql DB using a python script ingested via docker. I have the following compose file:
version: '3.9'
services:
mysql_db:
image: mysql:latest
restart: unless-stopped
environment:
MYSQL_DATABASE: ${MY_SQL_DATABASE}
MYSQL_USER: ${MY_SQL_USER}
MYSQL_PASSWORD: ${MY_SQL_PASSWORD}
MYSQL_ROOT_PASSWORD: ${MY_SQL_ROOT_PASSWORD}
ports:
- '3306:3306'
volumes:
- ./mysql-data:/var/lib/mysql
adminer:
image: adminer:latest
restart: unless-stopped
ports:
- 8080:8080
ingestion-python:
build:
context: .
dockerfile: ingestion.dockerfile
depends_on:
- mysql_db
Adminer connects to MySql with success. Then I created the following ingestion script to automate a criação de uma tabela. My ingestion script is:
from dotenv import load_dotenv
import os
import pandas as pd
from sqlalchemy import create_engine
def main():
load_dotenv('.env')
user = os.environ.get('MY_SQL_USER')
password = os.environ.get('MY_SQL_PASSWORD')
host = os.environ.get('MY_SQL_HOST')
port = os.environ.get('MY_SQL_PORT')
db = os.environ.get('MY_SQL_DATABASE')
table_name = os.environ.get('MY_SQL_TABLE_NAME')
print(f'mysql+pymysql://{user}:{password}#{host}:{port}/{db}')
engine = create_engine(f'mysql+pymysql://{user}:{password}#{host}:{port}/{db}')
df = pd.read_csv('./data/data.parquet', encoding='ISO-8859-1', on_bad_lines='skip', engine='python')
df.to_sql(name=table_name, con=engine, if_exists='append')
if __name__ == '__main__':
main()
When I run my docker compose (docker-compose up -d) file I get:
2023-02-14 08:58:59 sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'mysql_db' ([Errno 111] Connection refused)")
2023-02-14 08:58:59 (Background on this error at: https://sqlalche.me/e/20/e3q8)
The credentials and connections are retrieved from my .env file:
#MYSQL CONFIG
MY_SQL_DATABASE = test_db
MY_SQL_USER = data
MY_SQL_PASSWORD = random
MY_SQL_ROOT_PASSWORD = root
#PYTHON INGESTION
MY_SQL_HOST = mysql_db
MY_SQL_PORT = 3306
MY_SQL_TABLE_NAME = test_table
Why I can't connect to MySql DB using my python script?
This is most likely a timing problem - your ingestion container is starting before the database in the mysql container is ready. The depends_on only waits for the start of the mysql container, not on the database actually being ready to accept connections.
You might want to check the log outputs from the containers to see when the database is actually ready to accept connections, and include some delay into the ingestion container. Another option would be to try opening the connection in a loop with enough retries and some timeout between retries so that you can start as soon as the database is ready.
You should set the hostname in your docker compose file:
mysql_db:
hostname: "mysql_db"
image: mysql:latest
restart: unless-stopped
environment:
MYSQL_DATABASE: ${MY_SQL_DATABASE}
MYSQL_USER: ${MY_SQL_USER}
MYSQL_PASSWORD: ${MY_SQL_PASSWORD}
MYSQL_ROOT_PASSWORD: ${MY_SQL_ROOT_PASSWORD}
ports:
- '3306:3306'
volumes:
- ./mysql-data:/var/lib/mysql
But as fallback you can also try the default hostname:port exposed in docker as connection string since you don't have a network set up:
MY_SQL_HOST = host.docker.internal
MY_SQL_PORT = 3306
MY_SQL_TABLE_NAME = test_table
Related
I have a project that using kafka, zookeeper, debezium to capture data changes from mysql database. Actually, I quite new to kafka and I just follow debezium tutorial,so I face some troubles when trying to use kafka-python client to print message value. Here is my docker-compose file and config file:
My docker-compose file
version: '2'
services:
zookeeper:
image: quay.io/debezium/zookeeper:1.8
ports:
- 2181:2181
- 2888:2888
- 3888:3888
kafka:
image: quay.io/debezium/kafka:1.8
ports:
- 9092:9092
links:
- zookeeper
environment:
- ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
mysql:
image: quay.io/debezium/example-mysql:1.8
ports:
- 3306:3306
environment:
- MYSQL_ROOT_PASSWORD=debezium
- MYSQL_USER=mysqluser
- MYSQL_PASSWORD=mysqlpw
connect:
image: quay.io/debezium/connect:1.8
ports:
- 8083:8083
links:
- kafka
- mysql
environment:
- BOOTSTRAP_SERVERS=kafka:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
Config file
{
"name": "inventory-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "debezium",
"database.password": "dbz",
"database.server.id": "184054",
"database.server.name": "dbserver1",
"database.include.list": "inventory",
"database.history.kafka.bootstrap.servers": "kafka:9092",
"database.history.kafka.topic": "schema-changes.inventory",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": false,
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable": false
}
}
Python code
from kafka-python import KafkaConsumer
import json
import os
import logging
import sys
from google.cloud import bigquery
consumer = KafkaConsumer(
"dbserver1.inventory.customers",
auto_offset_reset="from-beginning",
group_id='console-consumer-47348',
bootstrap_servers=["kafka:9092"],
api_version = (2,0,2),
value_deserializer=lambda x: json.loads(x.decode('utf-8')),
max_poll_interval_ms=5000,
max_poll_records=1,
)
for message in consumer:
msg = message.value
print('{}'.format(msg))
Thank you very much!
I am having trouble getting my docker-compose.yml file to run, I am running the command docker-compose -f docker-compose.yml and am getting the error "TypeError: You must specify a directory to build in path
[7264] Failed to execute script docker-compose"
The file I have this in is called Labsetup and inside the file I have
docker-compose.yml,dockerClient.yml,dokcerServer.yml,and
A file called Volumes that have Server.py and Client.py
To this point I have edit my docker-compose.yml file and tried running "docker-compose -f docker-compose.yml up" to run the yml below is my docker-compose.yml
version: "3"
services:
# new code to execute python scripts in /volumes
server:
image: 85345aa61801 # from DockerServer.yml
container_name: server
build: DockerServer.yml
command: python3 ./volumes/Server.py
ports:
- 8080:8080
client:
image: 700c22b0d9d6 # from DockerClient.yml
container_name: client
build: DockerClient.yml
command: python3 ./volumes/Client.py
network_mode: host
depends_on:
- server
# below is original code for containers
malicious-router:
image: handsonsecurity/seed-ubuntu:large
container_name: malicious-router-10.9.0.111
tty: true
cap_add:
- ALL
sysctls:
- net.ipv4.ip_forward=1
- net.ipv4.conf.all.send_redirects=0
- net.ipv4.conf.default.send_redirects=0
- net.ipv4.conf.eth0.send_redirects=0
privileged: true
volumes:
- ./volumes:/volumes
networks:
net-10.9.0.0:
ipv4_address: 10.9.0.111
command: bash -c "
ip route add 192.168.60.0/24 via 10.9.0.11 &&
tail -f /dev/null "
HostB1:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.5
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.5
command: bash -c "python3 Server.py"
HostB2:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.6
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.6
command: bash -c "python3 Client.py"
HostB3:
image: handsonsecurity/seed-ubuntu:large
container_name: host-192.168.60.7
tty: true
cap_add:
- ALL
volumes:
- ./volumes:/volumes
networks:
net-192.168.60.0:
ipv4_address: 192.168.60.7
command: bash -c "python3 Client.py 8080"
Router:
image: handsonsecurity/seed-ubuntu:large
container_name: router
tty: true
cap_add:
- ALL
sysctls:
- net.ipv4.ip_forward=1
volumes:
- ./volumes:/volumes
networks:
net-10.9.0.0:
ipv4_address: 10.9.0.11
net-192.168.60.0:
ipv4_address: 192.168.60.11
command: bash -c "
ip route del default &&
ip route add default via 10.9.0.1 &&
tail -f /dev/null "
networks:
net-192.168.60.0:
name: net-192.168.60.0
ipam:
config:
- subnet: 192.168.60.0/24
net-10.9.0.0:
name: net-10.9.0.0
ipam:
config:
- subnet: 10.9.0.0/24
My DockerServer.yml
# Dockerfile.
# FROM ubuntu:20.04 # FROM python:latest
# ... most recent "image" created since DockerServer build = f48ea80eae5a <-- needed in FROM
# after running command: docker build -f DockerServer.yml .
# ... it output this upon completion ...
# ---> Running in 533f2cbd3135
# Removing intermediate container 533f2cbd3135 ---> 85345aa61801 Successfully built 85345aa61801
# FROM requires this syntax:
# FROM [--platform=<platorm>] <image> [AS <name>]
FROM python3:3.8.5 85345aa61801 AS serverImg
ADD Server.py /volumes/
WORKDIR /volumes/
My DokcerClient.yml
# Dockerfile.
# FROM ubuntu:20.04 server # python:latest #FROM ubuntu:latest ?
# step1/3 image --> f48ea80eae5a ... Successfully built 700c22b0d9d6
# image needed for 2nd arg
# after running command: docker build -f DockerClient.yml
# ... it output this upon completion ...
# Removing intermediate container 29037f3d23fb ---> 700c22b0d9d6 Successfully built 700c22b0d9d6
# FROM requires this syntax:
# FROM [--platform=<platorm>] <image> [AS <name>]
FROM python3:3.8.5 700c22b0d9d6 AS clientImg
ADD Client.py /volumes/
WORKDIR /volumes/
I am creating a project for a networking class that connects users to chat on a terminal. I have two python codes one being called Server.py that starts the server and tells who is in the chat room and another called Client.py that asks the user to put in a name and they will be able to chat. I am running this all on a virtual box Ubuntu20.04.
Code for Server.py
import socket
import _thread
import threading
from datetime import datetime
class Server():
def __init__(self):
# For remembering users
self.users_table = {}
# Create a TCP/IP socket and bind it the Socket to the port
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_address = ('localhost', 8080)
self.socket.bind(self.server_address)
self.socket.setblocking(1)
self.socket.listen(10)
print('Starting up on {} port {}'.format(*self.server_address))
self._wait_for_new_connections()
def _wait_for_new_connections(self):
while True:
connection, _ = self.socket.accept()
_thread.start_new_thread(self._on_new_client, (connection,))
def _on_new_client(self, connection):
try:
# Declare the client's name
client_name = connection.recv(64).decode('utf-8')
self.users_table[connection] = client_name
print(f'{self._get_current_time()} {client_name} joined the room !!')
while True:
data = connection.recv(64).decode('utf-8')
if data != '':
self.multicast(data, owner=connection)
else:
return
except:
print(f'{self._get_current_time()} {client_name} left the room !!')
self.users_table.pop(connection)
connection.close()
def _get_current_time(self):
return datetime.now().strftime("%H:%M:%S")
def multicast(self, message, owner=None):
for conn in self.users_table:
data = f'{self._get_current_time()} {self.users_table[owner]}: {message}'
conn.sendall(bytes(data, encoding='utf-8'))
if __name__ == "__main__":
Server()
Code for Client.py
import sys
import socket
import threading
class Client():
def __init__(self, client_name):
client_name = input("Choose Your Nickname:")
# Create a TCP/IP socket and connect the socket to the port
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_address = ('localhost', 8080)
self.socket.connect(self.server_address)
self.socket.setblocking(1)
print("{} is connected to the server, you may now chat.".format(client_name))
self.client_name = client_name
send = threading.Thread(target=self._client_send)
send.start()
receive = threading.Thread(target=self._client_receive)
receive.start()
def _client_send(self):
self.socket.send(bytes(self.client_name, encoding='utf-8'))
while True:
try:
c = input()
sys.stdout.write("\x1b[1A\x1b[2K") # Delete previous line
self.socket.send(bytes(c, encoding='utf-8'))
except:
self.socket.close()
return
def _client_receive(self):
while True:
try:
print(self.socket.recv(1024).decode("utf-8"))
except:
self.socket.close()
return
if __name__ == "__main__":
client_name = sys.argv[1]
Client(client_name)
Edit: I found a example online and followed it and changed a few
[11/26/21]seed#VM:~/.../Labsetup$ docker build -f DockerServer.yml .
Sending build context to Docker daemon 33.28kB
Step 1/3 : FROM python:latest
latest: Pulling from library/python
647acf3d48c2: Pull complete
b02967ef0034: Pull complete
e1ad2231829e: Pull complete
5576ce26bf1d: Pull complete
a66b7f31b095: Pull complete
05189b5b2762: Pull complete
af08e8fda0d6: Pull complete
287d56f7527b: Pull complete
dc0580965fb6: Pull complete
Digest: sha256:f44726de10d15558e465238b02966a8f83971fd85a4c4b95c263704e6a6012e9
Status: Downloaded newer image for python:latest
---> f48ea80eae5a
Step 2/3 : ADD Server.py /volumes/
---> df60dab46969
Step 3/3 : WORKDIR /volumes/
---> Running in 533f2cbd3135
Removing intermediate container 533f2cbd3135
---> 85345aa61801
Successfully built 85345aa61801
[11/26/21]seed#VM:~/.../Labsetup$ docker build -f DockerClient.yml .
Sending build context to Docker daemon 33.28kB
Step 1/3 : FROM python:latest
---> f48ea80eae5a
Step 2/3 : ADD Client.py /volumes/
---> 76af351c1c7a
Step 3/3 : WORKDIR /volumes/
---> Running in 29037f3d23fb
Removing intermediate container 29037f3d23fb
---> 700c22b0d9d6
Successfully built 700c22b0d9d6
I am trying to perform a table creation using pyodbc on a SQL Server 2017 database hosted using Docker. I'm also using a network so that I can connect to it later from another Docker image. However, I get the following error
pyodbc.OperationalError: ('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)')
This is how I went about creating the connection.
To create and run the DB server,
docker run --name mssqldocker -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<password>' -e 'MSSQL_PID=Express' -p 7000:7000 --network=lambda-local-mssql -v <my_path> -d mcr.microsoft.com/mssql/server:2017-latest-ubuntu
I also tried adding
-h "mssqldocker"
to the command for running the Docker image and then using "mssqldocker" instead of localhost, but to no avail, since mismatched hostnames seem to be the recurring theme when using DBs and Docker together. Also tried adding in \sqlexpress without effect as well. The Python code is as follows
import pyodbc
import sql_clauses
from settings import ENDPOINT, PORT, USERNAME, PASSWORD
cnxn = pyodbc.connect(
'DRIVER={ODBC Driver 17 for SQL Server}' +
';SERVER=' + ENDPOINT + ';UID=' + USERNAME +
';PWD=' + PASSWORD)
cursor = db.cursor()
cursor.execute(create_database(dbname))
cnxn.commit()
cnxn.close()
print("Database created")
The settings file is as follows
ENDPOINT="localhost"
PORT = 7000
USERNAME="SA"
PASSWORD=<password>
In your docker run command you specify -p 7000:7000. This translates in "map the host port 7000 (first 7000 - published) to the container port 7000 (the second 7000 - exposed)". If you have MSSQL running on a different port inside your container (which probably you do) then you have to change that second 7000 to the correct port.
Once you do that you should be able to connect to MSSQL from host using "localhost:7000". This applies if your python application runs directly on host.
If your python project also runs in a container, you need to make sure it runs on the same network as the mssql container (--network=lambda-local-mssql) and then you need to connect using "mssqldocker:mssql_exposed_port". In this case localhost and 7000 (the first part of `-p 7000:...) are not valide anymore since you are on a docker managed network.
The sample code in the question is incomplete and uses variables that are not defined.
A very simple working example is:
# docker compose file
version: "3.9"
services:
<Some name>:
image: mcr.microsoft.com/mssql/server:2019-latest # Or whatever version you want
container_name: <Some name>
restart: unless-stopped
ports:
- "1433:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=<Some Password>
- MSSQL_PID=Developer
- MSSQL_AGENT_ENABLED=True
import pandas as pd
import pyodbc
cnxn = pyodbc.connect(
'DRIVER={ODBC Driver 17 for SQL Server}' +
';SERVER=' + 'localhost,1433' + ';UID=' + 'sa' +
';PWD=' + '<Some password' +
';database=<some DB name>') # database here is optional if you want to specify it below in the query.
df = pd.read_sql('some query like select * from table', cnxn)
cnxn.commit()
cnxn.close()
print(df)
here's my docker compose
version: '2.1'
services:
db:
restart: always
image: nikitph/portcastdbimage:latest
ports:
- "5432:5432"
environment:
- DEBUG = false
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
scraper:
build: .
restart: always
links:
- db
environment:
- DB_HOST = db
- BAR = FOO
depends_on:
db:
condition: service_healthy
command: [ "python3", "./cycloneprocess.py" ]
Now from what I have gleaned from stack overflow, there are two options to access this db from a different container
a) use env variable
self.connection = psycopg2.connect(host=os.environ["DB_HOST"], user=username, password=password, dbname=database)
print(os.environ["DB_HOST"]) gives me 'db'. i dont know if thats expected
b) directly use the 'db'
self.connection = psycopg2.connect(host='db', user=username, password=password, dbname=database)
none of them seem to be working as no data gets populated. everything works locally so i m quiet confident my code is accurate All variables like user etc have been checked and rechecked & they work locally. Would really appreciate any help. Everything is on same network btw.
I'm using docker, selenium, and Django.
I just realised i was doing my tests on my production database ; while i wanted to test on StaticLiveServerTestCase self-generated database.
I tried to follow that tutorial
#override_settings(ALLOWED_HOSTS=['*'])
class BaseTestCase(StaticLiveServerTestCase):
host = '0.0.0.0'
#classmethod
def setUpClass(cls):
super().setUpClass()
cls.host = socket.gethostbyname(socket.gethostname())
cls.selenium = webdriver.Remote(
command_executor='http://hub:4444/wd/hub',
desired_capabilities=DesiredCapabilities.CHROME,
)
cls.selenium.implicitly_wait(5)
#classmethod
def tearDownClass(cls):
cls.selenium.quit()
super().tearDownClass()
class MyTest(BaseTestCase):
def test_simple(self):
self.selenium.get(self.live_server_url)
I've no error trying to connect to the chrome-hub, but when i try to print my page_source, i'm not on my django app but on a chrome error message. Here is a part :
<div class="error-code" jscontent="errorCode" jstcache="7">ERR_CONNECTION_REFUSED</div>
I'm using docker-compose 1. Selenium.yml:
chrome:
image: selenium/node-chrome:3.11.0-dysprosium
volumes:
- /dev/shm:/dev/shm
links:
- hub
environment:
HUB_HOST: hub
HUB_PORT: '4444'
hub:
image: selenium/hub:3.11.0-dysprosium
ports:
- "4444:4444"
expose:
- "4444"
app:
links:
- hub
I guess i did something wrong in my docker-compose file, but i don't manage to figure out what.
Thanks in advance !
PS : live_server_url = http://localhost:8081
You need to put container_name of the container that is running Django/the tests as host when using docker-compose, i.e.
host = 'app'
For a more detailed discussion see this question