Develop Reference

How to do df.to_sql using SQL Server in Azure

I can do a df.to_slq on my local instance of SQL Server just fine. I am getting stuck when trying to do the same df.to_sll using Python and Azure SQL Server. I thought it would essentially be done like this.
import urllib.parse
params = urllib.parse.quote_plus(
'Driver=%s;' % '{ODBC Driver 17 for SQL Server}' +
'Server=%s,1433;' % 'ryan-server.database.windows.net' +
'Database=%s;' % 'ryan_sql_db' +
'Uid=%s;' % 'UN' +
'Pwd={%s};' % 'PW' +
'Encrypt=no;' +
'TrustServerCertificate=no;'
)
from sqlalchemy.engine import create_engine
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
connection = engine.connect()
connection
all_data.to_sql('health', engine, if_exists='append', chunksize=100000, method=None,index=False)
That is giving me this error.
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n (10060) (SQLExecDirectW); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10060)')
[SQL: INSERT INTO health ([0], [Facility_BU_ID], [Code_Type], [Code], [Description], [UB_Revenue_Code], [UB_Revenue_Description], [Gross_Charge], [Cash_Charge], [Min_Negotiated_Rate], [Max_Negotiated_Rate], etc., etc., etc.
I found this link today:
https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-sql-server?view=sql-server-ver15
I tried to do something similar, like this.
import pyodbc
import pandas as pd
df = all_data
# server = 'myserver,port' # to specify an alternate port
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute(all_data.to_sql('health', cnxn, if_exists='append', chunksize=100000, method=None,index=False))
cnxn.commit()
cursor.close()
When I run that, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
What I'm really hoping to to is df.to_sql, not Insert Into. I am working in Spyder and trying to send the data from my local machine to the cloud.

I read the two links below, and got it working.
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-set-database-firewall-rule-azure-sql-database?view=azuresqldb-current
https://www.virtual-dba.com/blog/firewalls-database-level-azure-sql/
Basically, you need to open your command window on your local machine, enter 'ipconfig', and grab two IP addresses. Then, enter those into SQL Server in Azure.
EXECUTE sp_set_database_firewall_rule
N'health',
'192.0.1.1',
'192.0.0.5';
Finally, run the small script below, in SQL Server, to confirm that the changes were made correctly.
USE [ryan_sql_db]
GO
SELECT * FROM sys.database_firewall_rules
ORDER BY modify_date DESC

SQL Alchemy with SQL Server

I wasn't sure what to title my post, if you have a better idea, feel free to edit the title.
I have not used SQL Alchemy before and the documentation that I have looked at located in the following places, is not helpful:
Connecting to SQL Database Using SQL Alchemy in Python
Tutorial Point
Here is the code I am using:
import sqlalchemy as sal
from sqlalchemy import create_engine
#Here are the parameters I am using:
- server = 'Q-20/fake_example'
- database = 'AdventureWorks2017'
- driver = 'ODBC Driver 17 for SQL Server'
- trusted_connection='yes'
DATABASE_CONNECTION = 'mssql+pyodbc://#server = ' + server + '/database = ' + database + '?trusted_connection = ' + trusted_connection + '&driver=' + driver
engine = sal.create_engine(DATABASE_CONNECTION)
All of that seems to work fine without any problems; however, when I add this line:
connection=engine.connect()
I get the following error message:
sqlalchemy.exc.OperationalError: (pyodbc.OperationalError) ('08001',
'[08001] [Microsoft][ODBC Driver 17 for SQL Server]Named Pipes
Provider: Could not open a connection to SQL Server [53]. (53)
(SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL
Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17
for SQL Server]Invalid connection string attribute (0); [08001]
[Microsoft][ODBC Driver 17 for SQL Server]A network-related or
instance-specific error has occurred while establishing a connection
to SQL Server. Server is not found or not accessible. Check if
instance name is correct and if SQL Server is configured to allow
remote connections. For more information see SQL Server Books Online.
(53)')
I am not sure what is wrong with what I am doing, does anyone have any suggestions?
What I have tried so far:
I have confirmed that SQL Server is configured to allow remote connections. I did this check by following the instructions here
Removing the "#" sign before the server, but this just generated the same error message.

I figured out part of what I needed to do. I needed to change my parameters.
Old Parameters:
server = 'Q-20/fake_example'
database = 'AdventureWorks2017'
driver = 'ODBC Driver 17 for SQL Server'
trusted_connection='yes'
New Parameters:
server = 'Q-20'
database = 'AdventureWorks2017'
driver = 'SQL+SERVER+NATIVE+CLIENT+11.0'
trusted_connection='yes'
This is what my code ultimately looked like:
database_connection = 'mssql+pyodbc://Q-20/AdventureWorks2017?trusted_connection=yes&driver=SQL+SERVER+NATIVE+CLIENT+11.0'

Connecting to an Azure database using SQLAlchemy in Python

I am trying to connect to an Azure database using SQLAlchemy in Python.
My code is the following:
engine_azure = \
create_engine('mssql+pyodbc://{Server admin login}:{password}#{Server name}.database.windows.net:1433/{AdventureWorksLT}', echo=True)
I get the following message:
C:\ProgramData\Anaconda3\lib\site-packages\sqlalchemy\connectors\pyodbc.py:92: SAWarning: No driver name specified; this is expected by PyODBC when using DSN-less connections
"No driver name specified; "
Then I run the following code:
print(engine_azure.table_names())
I get the following message:
DBAPIError: (pyodbc.Error) ('01S00', '[01S00] [Microsoft][ODBC Driver Manager] Invalid connection string attribute (0) (SQLDriverConnect)')

There are 2 issues with your connection string:
As per the SQLAlchemy documentation: The delimeters must be URL escaped when using a pass-through exact pyodbc string.
And you do not specify the sql driver name either.
You can use the code below, which works fine at my side:
import pyodbc
from sqlalchemy import create_engine
import urllib
params = urllib.parse.quote_plus \ # urllib.parse.quote_plus for python 3
(r'Driver={ODBC Driver 13 for SQL Server};Server=tcp:yourDBServerName.database.windows.net,1433;Database=dbname;Uid=username#dbserverName;Pwd=xxx;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;')
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = create_engine(conn_str,echo=True)
print('connection is ok')
print(engine_azure.table_names())
Test result:
And for the connection string, you can get it by going to azure portal -> your database -> connection strings(select the ODBC in this case):

This is what i use in Python3:
params = urllib.parse.quote_plus(
'Driver=%s;' % driver +
'Server=tcp:%s,1433;' % server +
'Database=%s;' % database +
'Uid=%s;' % username +
'Pwd={%s};' % password +
'Encrypt=yes;' +
'TrustServerCertificate=no;' +
'Connection Timeout=30;')
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)

Python3 snippet that I am using with ODBC Driver 17 for SQL Server. Cost me some time to figure it all out, especially the driver version and params.
import urllib
from sqlalchemy import create_engine
driver = "{ODBC Driver 17 for SQL Server}"
server = "<server-name>.database.windows.net"
database = "<db-name>"
user = "<db-user>"
password = "<db-password>"
conn = f"""Driver={driver};Server=tcp:{server},1433;Database={database};
Uid={user};Pwd={password};Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;"""
params = urllib.parse.quote_plus(conn)
conn_str = 'mssql+pyodbc:///?autocommit=true&odbc_connect={}'.format(params)
engine = create_engine(conn_str, echo=True)
engine.execute("SELECT 1")
Furthermore, I needed to install the following drivers/tools on macOS:
brew install msodbcsql17 mssql-tools

Step 1: Install Azure SQL DB Drivers
Install the new version of SQL DB Drivers using official documentation:
Linux, MacOS, Windows
Major update to previous answers: use the last supported version of DB driver ODBC Driver 17 for SQL Server instead of outdated versions ODBC Driver 13 for SQL Server or versions without explicitly defined a version, e.g. SQL Server.
Step 2: Install sqlalchemy package
Just print in the terminal: pip install SQLAlchemy
Step 3: Wrapping specific DB logic in AzureDbConnection class
from dataclasses import dataclass
from typing import Dict, Any, Iterable
from pandas import DataFrame
from sqlalchemy import create_engine, inspect
import urllib
#dataclass(frozen=True)
class ConnectionSettings:
"""Connection Settings."""
server: str
database: str
username: str
password: str
driver: str = '{ODBC Driver 18 for SQL Server}'
timeout: int = 30
class AzureDbConnection:
"""
Azure SQL database connection.
"""
def __init__(self, conn_settings: ConnectionSettings, echo: bool = False) -> None:
conn_params = urllib.parse.quote_plus(
'Driver=%s;' % conn_settings.driver +
'Server=tcp:%s.database.windows.net,1433;' % conn_settings.server +
'Database=%s;' % conn_settings.database +
'Uid=%s;' % conn_settings.username +
'Pwd=%s;' % conn_settings.password +
'Encrypt=yes;' +
'TrustServerCertificate=no;' +
'Connection Timeout=%s;' % conn_settings.timeout
)
conn_string = f'mssql+pyodbc:///?odbc_connect={conn_params}'
self.db = create_engine(conn_string, echo=echo)
def connect(self) -> None:
"""Estimate connection."""
self.conn = self.db.connect()
def get_tables(self) -> Iterable[str]:
"""Get list of tables."""
inspector = inspect(self.db)
return [t for t in inspector.get_table_names()]
def dispose(self) -> None:
"""Dispose opened connections."""
self.conn.close()
self.db.dispose()
Major update to previous answers: do not forget to close the connection and dispose of the DB engine explicitly as soon as it stops being needed.
Enjoy!
Set connection settings and credentials using Azure DB blade on Azure Portal:
conn_settings = ConnectionSettings(
server='<db_server_name>',
database='<db_name>',
username='<user_name>',
password='***')
Open DB connection:
db_conn = AzureDbConnection(conn_settings)
db_conn.connect()
Test connection (for example, get available tables list), do other stuff, and close it finally:
try:
for t in db_conn.get_tables():
print(t)
# Do another DB-related stuff:
# ...
finally:
db_conn.dispose()

None of the solutions posted so far worked for me.
Instead, I had to specify Driver ({SQL Server}) which worked perfectly.
params = urllib.parse.quote_plus("DRIVER={SQL Server};SERVER=sqlhost.database.windows.net;DATABASE=pythonSQL;UID=username#sqldb;PWD=password56789")
conn_str = 'mssql+pyodbc:///?odbc_connect={}'.format(params)
engine_azure = create_engine(conn_str,echo=True)
Source:
https://gist.github.com/timmyreilly/f4a351eda5dd45aa9d56411d27573d7c

Connecting to Azure SQL Server with AAD authentication using PyODBC

Environment:
Docker image: python:2.7.14-stretch (which means Debian 9 as underlying OS)
Installed SQL Server ODBC driver for Debian 9 (https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017)
using pyodbc module
Connecting to Azure SQL Server with AAD authentication.
Issue:
First connection to DB is made perfectly fine. However any next connection to same DB or other DB is failed with below error:
Exception::: ('08001', u'[08001] [unixODBC][Microsoft][ODBC Driver 17
for SQL Server]SSL Provider:
[error:140A90F1:lib(20):func(169):reason(241)] (-1)
(SQLDriverConnect)')
Cannot connect to DB
Please note that this is issue with only when using AAD autnethcation mode. If I use simple sql server user, everything runs fine.
Below is the example code snippet.
db_list = [TEST_DB1, TEST_DB2]
sql_conn = None
for db in db_list:
try:
conn_string = 'DRIVER={ODBC Driver 17 for SQL Server};' \
'SERVER=' + <db_url> + \
';DATABASE=' + <db_name> + \
';UID=' + <db_username> + \
';PWD=' + <db_password> + \
';Authentication=ActiveDirectoryPassword'
print conn_string
sql_conn = pyodbc.connect(conn_string)
except Exception as e:
print "Exception:::", e
print 'Cannot connect to DB' + str(sys.exc_info()[0])
return None
sql_conn.cursor().execute(<some SQL Query>)
sql_conn.close()

Unable to connect pyODBC with SQL Server 2008 Express R2

I am using following code to connect with SQL 2008 R2:
cnxnStoneedge = pyodbc.connect("DRIVER={SQL Server};SERVER=127.0.0.1;DATABASE=myDB;UID=admin;PWD=admin")
Which gives error:
Error: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect)')
args = ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNE...t exist or access denied. (17) (SQLDriverConnect)')
with_traceback = <built-in method with_traceback of Error object>
I am not sure whether SQL connecting via named Pipes or TCP, I did enable IP Address. Attaching Screen Shot

The following test code works for me to connect Python 2.7.5 with SQL Server 2008 R2 Express Edition:
# -*- coding: utf-8 -*-
import pyodbc
connStr = (
r'Driver={SQL Server};' +
r'Server=(local)\SQLEXPRESS;' +
r'Database=myDb;' +
r'Trusted_Connection=Yes;'
)
db = pyodbc.connect(connStr)
cursor1 = db.execute('SELECT [word] FROM [vocabulary] WHERE [ID]=5')
while 1:
row = cursor1.fetchone()
if not row:
break
print row.word
cursor1.close()
db.close()
and the following connection string also works for me because my \SQLEXPRESS instance is listening on port 52865:
connStr = (
r'Driver={SQL Server};' +
r'Server=127.0.0.1,52865;' +
r'Database=myDb;' +
r'Trusted_Connection=Yes;'
)