I am trying to perform a table creation using pyodbc on a SQL Server 2017 database hosted using Docker. I'm also using a network so that I can connect to it later from another Docker image. However, I get the following error
pyodbc.OperationalError: ('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)')
This is how I went about creating the connection.
To create and run the DB server,
docker run --name mssqldocker -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<password>' -e 'MSSQL_PID=Express' -p 7000:7000 --network=lambda-local-mssql -v <my_path> -d mcr.microsoft.com/mssql/server:2017-latest-ubuntu
I also tried adding
-h "mssqldocker"
to the command for running the Docker image and then using "mssqldocker" instead of localhost, but to no avail, since mismatched hostnames seem to be the recurring theme when using DBs and Docker together. Also tried adding in \sqlexpress without effect as well. The Python code is as follows
import pyodbc
import sql_clauses
from settings import ENDPOINT, PORT, USERNAME, PASSWORD
cnxn = pyodbc.connect(
'DRIVER={ODBC Driver 17 for SQL Server}' +
';SERVER=' + ENDPOINT + ';UID=' + USERNAME +
';PWD=' + PASSWORD)
cursor = db.cursor()
cursor.execute(create_database(dbname))
cnxn.commit()
cnxn.close()
print("Database created")
The settings file is as follows
ENDPOINT="localhost"
PORT = 7000
USERNAME="SA"
PASSWORD=<password>
In your docker run command you specify -p 7000:7000. This translates in "map the host port 7000 (first 7000 - published) to the container port 7000 (the second 7000 - exposed)". If you have MSSQL running on a different port inside your container (which probably you do) then you have to change that second 7000 to the correct port.
Once you do that you should be able to connect to MSSQL from host using "localhost:7000". This applies if your python application runs directly on host.
If your python project also runs in a container, you need to make sure it runs on the same network as the mssql container (--network=lambda-local-mssql) and then you need to connect using "mssqldocker:mssql_exposed_port". In this case localhost and 7000 (the first part of `-p 7000:...) are not valide anymore since you are on a docker managed network.
The sample code in the question is incomplete and uses variables that are not defined.
A very simple working example is:
# docker compose file
version: "3.9"
services:
<Some name>:
image: mcr.microsoft.com/mssql/server:2019-latest # Or whatever version you want
container_name: <Some name>
restart: unless-stopped
ports:
- "1433:1433"
environment:
- ACCEPT_EULA=Y
- SA_PASSWORD=<Some Password>
- MSSQL_PID=Developer
- MSSQL_AGENT_ENABLED=True
import pandas as pd
import pyodbc
cnxn = pyodbc.connect(
'DRIVER={ODBC Driver 17 for SQL Server}' +
';SERVER=' + 'localhost,1433' + ';UID=' + 'sa' +
';PWD=' + '<Some password' +
';database=<some DB name>') # database here is optional if you want to specify it below in the query.
df = pd.read_sql('some query like select * from table', cnxn)
cnxn.commit()
cnxn.close()
print(df)
Related
Been stuck on this for a few days. Seen quite a few stack overflow posts on this which hasn't resolved for me and read the microsoft and pyodbc docs also but seems like my issue on this may be niche and would like some help.
Goal: I want to connect to sql server via a python script using pyodbc. I've built a docker-compose.yml which for the sql server image essentially points at a Dockerfile with build: ., then the dockerfile runs a setup.sql script is run so I create a db, user and login automatically instead of creating it every time I spin up the container.
On my laptop (checked ODBC data source admin) I have ODBC Driver 17 for SQL Server, and that's why I've used in in below code:
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER=localhost;DATABASE=testdb;UID=kafkaUser;PWD=Kafka_Us3R!;')
(I've seen on other posts that using trusted_connection=yes; should resolve this, but sadly doesn't for me.) NOTE: I get this error whether I use sa, kafkaUser, or kafkaLogin with their designated passwords. I've just copied and pasted above line after last try before resulting to posting here.
Dockerfile:
FROM mcr.microsoft.com/mssql/server
WORKDIR /topics
# Env vars for sql server
ENV ACCEPT_EULA Y
ENV MSSQL_SA_PASSWORD <YourStrong!Passw0rd>
ENV MSSQL_PID Developer
ENV MSSQL_HOST localhost
ENV MSSQL_USER kafkaUser
ENV MSSQL_PASSWORD Kafka_Us3R!
ENV MSSQL_DATABASE testdb
EXPOSE 1433:1433
COPY topics/proposal-created-hl/setup.sql setup.sql
COPY topics/proposal-created-hl/setup_database.sh setup_database.sh
COPY topics/proposal-created-hl/entrypoint.sh entrypoint.sh
RUN /opt/mssql/bin/sqlservr & ./setup_database.sh
setup.sql
-- MSSQL file for local testing with docker container
IF NOT EXISTS ( SELECT * FROM sys.databases WHERE name = 'testdb' )
CREATE DATABASE [testdb];
GO
USE [testdb]
GO
IF NOT EXISTS ( SELECT name FROM sys.server_principals WHERE name = 'kafkaLogin' )
BEGIN
CREATE LOGIN [kafkaLogin] WITH PASSWORD = 'Kafka_Us3R!', CHECK_EXPIRATION = OFF, CHECK_POLICY = OFF;
ALTER SERVER ROLE [sysadmin] ADD MEMBER [kafkaLogin];
ALTER LOGIN [kafkaLogin] WITH DEFAULT_DATABASE = [testdb];
CREATE USER [kafkaUser] FOR LOGIN [kafkaLogin];
END
GO
Error:
pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'kafkaUser'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'kafkaUser'. (18456)")
setup_database.sh:
# Resource: https://stackoverflow.com/questions/58309452/docker-initialize-database-tables-and-records-in-sql-server
#!/usr/bin/env bash
# Wait for database to startup
sleep 20
/opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P '<YourStrong!Passw0rd>' -i setup.sql
I have checked this also while getting in the container (via docker exec) that the db, user and login have been created. So what detail am I missing here that has caused me to get this error? I tried the solutions of various posts of people coming across the same error but this hasn't resolved for me. Is there something else I need to look at?
Just to say that I am new to docker...
I have a docker-compose application (I like/need the docker-compose way to run multiple containers) that runs a mysql container and a python app, and I would like the python app to run every minute. I tried an infinite loop in the python code, but docker seems to prevent it from launching, so I tried with cron, but without any success (although it might not have any impact, I run docker on Windows, which seems less straightforward than on Linux :(, and use VS Code for development). Therefore, I would like to know if there is a way to use docker-compose and run the python app every minute, and if yes, how to configure cron.
my setup is the following:
docker installed on windows.
I run everything from VS Code' terminal, without any docker extension
A pythonmain.py file (for example, I used emails because I could not see any logs nor print once I run the code from cron. Most likely because cron does not launch the python app):
`
import mysql.connector as mysql
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
print("Starting Python Docker")
# EMAIL
mail_content = "Hello"
#The mail addresses and password
sender_address = 'email1#mail.com'
sender_pass = 'pwd'
receiver_address = 'email2#mail.com'
#Setup the MIME
message = MIMEMultipart()
message['From'] = sender_address
message['To'] = receiver_address
message['Subject'] = 'New data email.' #The subject line
#The body and the attachments for the mail
message.attach(MIMEText(mail_content, 'plain'))
#Create SMTP session for sending the mail
session = smtplib.SMTP('smtp.office365.com', 587) #use gmail with port
session.starttls() #enable security
session.login(sender_address, sender_pass) #login with mail_id and password
text = message.as_string()
session.sendmail(sender_address, receiver_address, text)
session.quit()
print('Mail Sent')
#Database
db = mysql.connect(host = 'mydb', user = 'root', password = 'root', port = 3306)
print("db connection passed")
cursor = db.cursor()
cursor.execute("CREATE DATABASE IF NOT EXISTS emails")
cursor.execute("SHOW DATABASES")
#connect to the right database
db = mysql.connect(
host = "mydb",
user = "root",
passwd = "root",
database = "emails"
)
cursor = db.cursor()
## creating a table in database
cursor.execute("CREATE TABLE IF NOT EXISTS emailscount (id VARCHAR(255), count INT)")
## defining the Query for charger data
query = "INSERT INTO emailscount (id, count) VALUES (%s, %s)"
values = ("123",1)
cursor.execute(query, values)
db.commit()
print(cursor.rowcount, "data inserted")
`
a dockerfile:
`
FROM python:3.9
COPY . .
RUN pip install mysql-connector-python requests
# for implementation without cron:
# CMD ["python", "./pythonmain.py"]
# trial with cron, but python app does not launch:
RUN apt-get update
RUN apt-get install -y cron
RUN chmod +x pythonmain.py
RUN crontab crontab
CMD ["cron", "-f"]
`
a crontab file:
`
*/1 * * * * python ./pythonmain.py
`
A docker-compose file:
`
version: "3"
services:
db:
container_name: mydb
image: mysql:5.7
ports:
- "33061:3306"
environment:
MYSQL_ROOT_PASSWORD: root
pythonapp:
container_name: pythonapp
depends_on:
- "db"
links:
- "db"
build: ./
ports:
- "5001:5000"
`
I would like the pythonapp to run every minute after I run the "docker-compose up" command in the terminal. But it does not work (the python app never runs, although all containers seem to be created), and I am not sure it can run as it is with the docker-compose command.
What works perfectly is when the dockerfile has the CMD ["python", "./pythonmain.py"] without everything related to cron, then the python app runs as it should, but of course, just once.
I tried as well to just run the python container from cron using docker build and docker run (not docker-compose up), but without any success neither, the code does not run (i.e. I was not able to replicate most of the examples/tutorials on cron for docker). Furthermore, using docker on windows, I could not find any log of error, even in the AppData\Local\Docker\wsl\data folder (which is empty).
Obviously, I am completely new to docker, so any help would be amazing!
Thanks a lot in advance !!
I developed a simple python Azure function app using pyodbc to select a few rows from a public IP MS SQL server. My function app runs fine on my laptop, but it doesn't work when I publish it on Azure cloud (I used Consumption - serverless plan, linux environment). Thru the logging, I knows that it always gets stuck at the pyodbc.connect(...) command and time-out.
#...
conn_str = f'Driver={driver};Server={server},{port};Database={database};Uid={user};Pwd={password};Encrypted=yes;TrustServerCertificate=no;Connection Timeout=30'
sql_query = f'SELECT * FROM {table_name}'
try:
conn = pyodbc.connect(conn_str) # always time-out here if running on Azure cloud!!!
logging.info(f'Inventory API - connected to {server}, {port}, {user}.')
except Exception as error:
logging.info(f'Inventory API - connection error: {repr(error)}.')
else:
with conn.cursor() as cursor:
cursor.execute(sql_query)
logging.info(f'Inventory API - executed query: {sql_query}.')
data = []
for row in cursor:
data.append({'Sku' : row.Sku, 'InventoryId' : row.InventoryId, 'LocationId' : row.LocationId, 'AvailableQuantity' : row.AvailableQuantity})
#...
The logging captured:
Inventory API - connection error: OperationalError('HYT00', '[HYT00] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0) (SQLDriverConnect)').
I already include the pyodbc in the requirements.txt file. I also allows all outboundIpAddresses and possibleOutboundIpAddresses of my function app on my SQL server firewall. My function app does not have any network restriction on Azure cloud (or at least it said so on the network settings).
my config file:
driver={ODBC Driver 17 for SQL Server}
server=I tried both IP and full internet host name, both didn't work
Could someone give me a hint? Thanks.
Workarounds to fix the PYODBC connection Error
Try to use a below format
import pyodbc
server = 'tcp:myserver.database.windows.net'
database = 'mydb'
username = 'myusername'
password = 'mypassword'
conn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
If you are using Domain Name add TCP with domain name server = 'tcp:myserver.database.windows.net' otherwise use the IP server = '129.0.0.1’
If you are using port use like this 'tcp:myserver.database.windows.net,1233’ or '129.0.0.1,1233'
Try to remove additional properties like Connection_Timeout, Trusted_certificate and all and check now.
Refer here
I put the following snippet into my function to check the outbound IP, and found out that Azure use a few outbound IPs that are not listed in the [outboundIpAddresses] and [possibleOutboundIpAddresses] (documented in this MS link)
import requests
#...
outbound_ip_response = requests.request('GET', 'https://checkip.amazonaws.com')
logging.info(f'Inventory API - main()- outbound ip = {outbound_ip_response.text}')
So, I followed the instructions in this link to setup a static outbound IP for my function app and allowed this IP to access my SQL server. It worked.
Goal: Connect to remote MSSQL 2016 server via Python.
Main approach: Closely followed tutorial in https://github.com/mkleehammer/pyodbc/wiki/Connecting-to-SQL-Server-from-Mac-OSX .
Problem: Able to connect via tsql, but isql is not working. Errors
[S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
[37000][unixODBC][FreeTDS][SQL Server]Login failed for user 'DOMAIN\user-p'
[37000][unixODBC][FreeTDS][SQL Server]Cannot open database "TIT_BI_OPERATIONS" requested by the login. The login failed.
Things tried:
Different ODBC drivers 13.1, 17, FreeTDS
Inclusion/exclusion of escape character in the user name.
Host name vs host ip.
Settings:
odbc.ini
[ODS_DSN]
Description = Connection to ODS MS_SQL 2016
Driver = FreeTDS
Servername = ODS_DSN
Port = 40000
Database = TIT_BI_OPERATIONS
odbcinst.ini
[FreeTDS]
Driver=/usr/local/lib/libtdsodbc.so
Setup=/usr/local/lib/libtdsodbc.so
UsageCount=1
freetds.conf
[ODS_DSN]
host = 164.10.17.77
port = 40000
tds version = 7.4
client charset = UTF-8
Notes:
Even though, its not very promising to run python without connecting through tsql and isql first, i still tried without success. Using pyodbc, pypodbc, sqlalchemy.
Most errors in the form: Login failed for user 'DOMAIN\user-p'
For ODBC driver 13: Can't open lib '/usr/local/lib/libmsodbcsql.13.dylib'
I am able to connect via SQL PRO STUDIO, using exact same credentials.
If you have any thoughts which direction to go to climb out of this connection problem, it would be greatly appreciated. Thank you!
If you're using Windows domain auth, you'll have to use FreeTDS. Oddly enough, Windows domain auth isn't supported by the Microsoft ODBC Driver, only FreeTDS.
Since you can connect with the tsql command, that means FreeTDS is working. I'd recommend connecting directly from Python explicitly. Try a connection string like this:
import pyodbc
con = pyodbc.connect(
r"DRIVER={FreeTDS};"
r"SERVER=164.10.17.77;"
r"PORT=40000;"
r"DATABASE=TIT_BI_OPERATIONS;"
f"UID=DOMAIN\\user-p;"
f"PWD=yourpassword;"
r"TDS_Version=7.3;"
)
cursor = con.cursor();
cursor.execute("SELECT 'this' AS that")
for row in cursor.fetchall():
print(row)
Note that you do need two backslashes in the UID field to connect with Windows domain auth; that is not a typo!
sqlalchemy, a db connection module for Python, uses SQL Authentication (database-defined user accounts) by default. If you want to use your Windows (domain or local) credentials to authenticate to the SQL Server, the connection string must be changed.
By default, as defined by sqlalchemy, the connection string to connect to the SQL Server is as follows:
sqlalchemy.create_engine('mssql://*username*:*password*#*server_name*/*database_name*')
This, if used using your Windows credentials, would throw an error similar to this:
sqlalchemy.exc.DBAPIError: (Error) ('28000', "[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for us
er '***S\\username'. (18456)") None None
In this error message, the code 18456 identifies the error message thrown by the SQL Server itself. This error signifies that the credentials are incorrect.
In order to use Windows Authentication with sqlalchemy and mssql, the following connection string is required:
ODBC Driver:
engine = sqlalchemy.create_engine('mssql://*server_name*/*database_name*?trusted_connection=yes')
SQL Express Instance:
engine = sqlalchemy.create_engine('mssql://*server_name*\\SQLEXPRESS/*database_name*?trusted_connection=yes')
If you're using a trusted connection/AD and not using username/password, or otherwise see the following:
SAWarning: No driver name specified; this is expected by PyODBC when using >DSN-less connections
"No driver name specified; "
Then this method should work:
from sqlalchemy import create_engine
server = <your_server_name>
database = <your_database_name>
engine = create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=ODBC+Driver+13+for+SQL+Server')
A more recent response if you want to connect to the MSSQL DB from a different user than the one you're logged with on Windows. It works as well if you are connecting from a Linux machine with FreeTDS installed.
The following worked for me from both Windows 10 and Ubuntu 18.04 using Python 3.6 & 3.7:
import getpass
from sqlalchemy import create_engine
password = getpass.getpass()
eng_str = fr'mssql+pymssql://{domain}\{username}:{password}#{hostip}/{db}'
engine = create_engine(eng_str)
What changed was to add the Windows domain before \username.
You'll need to install the pymssql package.
Create Your SqlAlchemy Connection URL From Your pyodbc Connection String OR Your Known Connection Parameters
I found all the other answers to be educational, and I found the SqlAlchemy Docs on connection strings helpful too, but I kept failing to connect to MS SQL Server Express 19 where I was using no username or password and trusted_connection='yes' (just doing development at this point).
Then I found THIS method in the SqlAlchemy Docs on Connection URLs built from a pyodbc connection string (or just a connection string), which is also built from known connection parameters (i.e. this can simply be thought of as a connection string that is not necessarily used in pyodbc). Since I knew my pyodbc connection string was working, this seemed like it would work for me, and it did!
This method takes the guesswork out of creating the correct format for what you feed to the SqlAlchemy create_engine method. If you know your connection parameters, you put those into a simple string per the documentation exemplified by the code below, and the create method in the URL class of the sqlalchemy.engine module does the correct formatting for you.
The example code below runs as is and assumes a database named master and an existing table named table_one with the schema shown below. Also, I am using pandas to import my table data. Otherwise, we'd want to use a context manager to manage connecting to the database and then closing the connection like HERE in the SqlAlchemy docs.
import pandas as pd
import sqlalchemy
from sqlalchemy.engine import URL
# table_one dictionary:
table_one = {'name': 'table_one',
'columns': ['ident int IDENTITY(1,1) PRIMARY KEY',
'value_1 int NOT NULL',
'value_2 int NOT NULL']}
# pyodbc stuff for MS SQL Server Express
driver='{SQL Server}'
server='localhost\SQLEXPRESS'
database='master'
trusted_connection='yes'
# pyodbc connection string
connection_string = f'DRIVER={driver};SERVER={server};'
connection_string += f'DATABASE={database};'
connection_string += f'TRUSTED_CONNECTION={trusted_connection}'
# create sqlalchemy engine connection URL
connection_url = URL.create(
"mssql+pyodbc", query={"odbc_connect": connection_string})
""" more code not shown that uses pyodbc without sqlalchemy """
engine = sqlalchemy.create_engine(connection_url)
d = {'value_1': [1, 2], 'value_2': [3, 4]}
df = pd.DataFrame(data=d)
df.to_sql('table_one', engine, if_exists="append", index=False)
Update
Let's say you've installed SQL Server Express on your linux machine. You can use the following commands to make sure you're using the correct strings for the following:
For the driver: odbcinst -q -d
For the server: sqlcmd -S localhost -U <username> -P <password> -Q 'select ##SERVERNAME'
pyodbc
I think that you need to put:
"+pyodbc" after mssql
try this:
from sqlalchemy import create_engine
engine = create_engine("mssql+pyodbc://user:password#host:port/databasename?driver=ODBC+Driver+17+for+SQL+Server")
cnxn = engine.connect()
It works for me
Luck!
If you are attempting to connect:
DNS-less
Windows Authentication for a server not locally hosted.
Without using ODBC connections.
Try the following:
import sqlalchemy
engine = sqlalchemy.create_engine('mssql+pyodbc://' + server + '/' + database + '?trusted_connection=yes&driver=SQL+Server')
This avoids using ODBC connections and thus avoids pyobdc interface errors from DPAPI2 vs DBAPI3 conflicts.
I would recommend using the URL creation tool instead of creating the url from scratch.
connection_url = sqlalchemy.engine.URL.create("mssql+pyodbc",database=databasename, host=servername, query = {'driver':'SQL Server'})
engine = sqlalchemy.create_engine(connection_url)
See this link for creating a connection string with SQL Server Authentication (non-domain, uses username and password)