Develop Reference

Python connect to Microsoft SQL Server with sqlalchemy for df.to_sql: target machine actively refused connection

Insert pandas df into local Microsoft SQL Server database table using df.to_sql
Created connection_url for sqlalchemy engine:
connection_url = URL.create(
"mssql+pyodbc",
username="",
password="",
host="localhost",
port=1433,
database="priority",
query={
"driver": "ODBC Driver 17 for SQL Server",
"authentication": "ActiveDirectoryIntegrated",
},
)
Used connection_url to create engine:
engine = sqlalchemy.create_engine(connection_url)
Attempted to insert df into SQL database table 'test1' with df.to_sql:
df.to_sql('test1', engine, if_exists='replace')
Received error
OperationalError: (pyodbc.OperationalError) ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: No connection could be made because the target machine actively refused it.\r\n (10061) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout expired (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online. (10061)')
What I've tried
Opened the SQL Port 1433
Ran the following in the cmd (administrator mode)
netsh advfirewall firewall add rule name= "SQL Port" dir=in action=allow protocol=TCP localport=1433
Checked the port has been added
netsh firewall show state
Which gave the following:
Ports currently open on all network interfaces:
Port Protocol Version Program
1433 TCP Any (null)
Replaced 'localhost' with '127.0.0.1'
Replaced 'localhost' with the hostname of SQL Server
I found the hostname of SQL Server by executing the following SQL query
SELECT HOST_NAME() AS HostName
Which returned the following (changed the name for security)
LAPTOP-NumDig
[Edit update] 4. Updated ODBC driver to driver 17(https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16).
[Edit update] 5. Replaced "authentication": "ActiveDirectoryIntegrated" in query with "Trusted_Connection": "yes"
Prior connections have worked with pyodbc package
I have been able to connect to the SQL database 'priority' and created the table 'test1' using the pyodbc package in Python.
Connect to SQL
conn = pyodbc.connect('Driver={SQL Server};'
'Server=LAPTOP-NumDig;'
"Database=priority;"
'UID=;' # username
'PWD=;' # password
)
cursor = conn.cursor()
Create table called 'test1' in 'priority' database
cursor.execute("""
CREATE TABLE test1 (
PersonID int,
LastName varchar(255),
FirstName varchar(255),
Address varchar(255),
City varchar(255)
);""")
Let me know your thoughts
What I'm wanting to do is insert a df as a table in SQL. Any suggestions would be great :)
FYI - Software & package versions:
VSCode 1.71.2
Python 3.9.13
Microsoft SQL Server 2019 on Windows 10 Home 10.0 (Build 22000: )
sqlalchemy 1.4.41
pyodbc 4.0.34

I found a solution that uses an alternative approach that works: passing through the exact Pyodbc string to sqlalchemy (https://pydoc.dev/sqlalchemy/latest/sqlalchemy.dialects.mssql.pyodbc.html).
import sqlalchemy
from sqlalchemy.engine import URL
connection_string = "DRIVER={SQL Server};SERVER=LAPTOP-NumDig;DATABASE=priority;UID=;PWD="
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
engine = sqlalchemy.create_engine(connection_url)
df.to_sql('test1', engine, if_exists='replace')
Note, this way of inserting took a long time to upload a df into an SQL table.
[Edit update] To make the df.to_sql() run faster, include the chunksize parameter and method='multi'. (https://towardsdatascience.com/dramatically-improve-your-database-inserts-with-a-simple-upgrade-6dfa672f1424).
I used the following code to speed up the pandas df to SQL table upload:
df.to_sql('test1', engine, if_exists='replace', chunksize=20, method='multi')

pyodbc connection failing in sqlalchemy but working through direct pyodbc connection

Why does this workc(I get a result set back):
sql_server = 'myserver.database.windows.net'
sql_database = 'pv'
sql_username = 'sqladmin'
sql_password = 'password1'
sql_driver= '{ODBC Driver 17 for SQL Server}'
with pyodbc.connect('DRIVER='+sql_driver+';SERVER=tcp:'+sql_server+';DATABASE='+sql_database+';UID='+sql_username+';PWD='+ sql_password) as conn:
with conn.cursor() as cursor:
cursor.execute("SELECT TOP 3 SAPPHIRE_CASE_ID FROM PV_ALL_SUBMISSIONS_SL")
row = cursor.fetchone()
while row:
print (str(row[0]))
row = cursor.fetchone()
But this fails:
import pyodbc
sql_engine = sqlalchemy.create_engine(f'mssql+pyodbc://{sql_username}:{sql_password}#{sql_server}/{sql_database}?driver=ODBC+Driver+17+for+SQL+Server')
df.to_sql('PV_ALL_CLOSED_CASES_SL', con=sql_engine, if_exists='append')
Error is:
OperationalError: (pyodbc.OperationalError) ('08001', '[08001]
[Microsoft][ODBC Driver 17 for SQL Server]Named Pipes Provider: Could
not open a connection to SQL Server [53]. (53) (SQLDriverConnect);
[08001] [Microsoft][ODBC Driver 17 for SQL Server]Login timeout
expired (0); [08001] [Microsoft][ODBC Driver 17 for SQL Server]A
network-related or instance-specific error has occurred while
establishing a connection to SQL Server. Server is not found or not
accessible. Check if instance name is correct and if SQL Server is
configured to allow remote connections. For more information see SQL
Server Books Online. (53)') (Background on this error at:
https://sqlalche.me/e/14/e3q8)
While I know one is doing a read and the other a write, my issue seems to be just establishing a connection one way vs another, when using the same connection details. It isn't an Azure firewall issue as I am able to connect and run a select statment via the first method, but when using create_engine() of sqlalchemy, it fails to make the connection - but I am pretty sure the connection string is correct.
It is the same variables for server, user name and password being used in both connections.
I think the issue is that the real password as an "#" symbol in it, and so this interferes with the latter connection string.

Thanks to #Larnu, this worked:
from sqlalchemy.engine import URL
connection_string = f"DRIVER={sql_driver};SERVER={sql_server};DATABASE={sql_database};UID={sql_username};PWD={sql_password}"
connection_url = URL.create("mssql+pyodbc", query={"odbc_connect": connection_string})
sql_engine = sqlalchemy.create_engine(connection_url)
I dont have to url encode when I use a cx_Oracle connection, but hey it works now.

How to do df.to_sql using SQL Server in Azure

I can do a df.to_slq on my local instance of SQL Server just fine. I am getting stuck when trying to do the same df.to_sll using Python and Azure SQL Server. I thought it would essentially be done like this.
import urllib.parse
params = urllib.parse.quote_plus(
'Driver=%s;' % '{ODBC Driver 17 for SQL Server}' +
'Server=%s,1433;' % 'ryan-server.database.windows.net' +
'Database=%s;' % 'ryan_sql_db' +
'Uid=%s;' % 'UN' +
'Pwd={%s};' % 'PW' +
'Encrypt=no;' +
'TrustServerCertificate=no;'
)
from sqlalchemy.engine import create_engine
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
connection = engine.connect()
connection
all_data.to_sql('health', engine, if_exists='append', chunksize=100000, method=None,index=False)
That is giving me this error.
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n (10060) (SQLExecDirectW); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10060)')
[SQL: INSERT INTO health ([0], [Facility_BU_ID], [Code_Type], [Code], [Description], [UB_Revenue_Code], [UB_Revenue_Description], [Gross_Charge], [Cash_Charge], [Min_Negotiated_Rate], [Max_Negotiated_Rate], etc., etc., etc.
I found this link today:
https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-sql-server?view=sql-server-ver15
I tried to do something similar, like this.
import pyodbc
import pandas as pd
df = all_data
# server = 'myserver,port' # to specify an alternate port
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute(all_data.to_sql('health', cnxn, if_exists='append', chunksize=100000, method=None,index=False))
cnxn.commit()
cursor.close()
When I run that, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
What I'm really hoping to to is df.to_sql, not Insert Into. I am working in Spyder and trying to send the data from my local machine to the cloud.

I read the two links below, and got it working.
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-set-database-firewall-rule-azure-sql-database?view=azuresqldb-current
https://www.virtual-dba.com/blog/firewalls-database-level-azure-sql/
Basically, you need to open your command window on your local machine, enter 'ipconfig', and grab two IP addresses. Then, enter those into SQL Server in Azure.
EXECUTE sp_set_database_firewall_rule
N'health',
'192.0.1.1',
'192.0.0.5';
Finally, run the small script below, in SQL Server, to confirm that the changes were made correctly.
USE [ryan_sql_db]
GO
SELECT * FROM sys.database_firewall_rules
ORDER BY modify_date DESC

Need help connecting to SQL Server from Python Flask-Appbuilder

I'm new to Python + Flask + Flask Appbuilder but I am a professional Java developer. I've been working on a small app that I initially used SqlLite and now I want to move into SQL Server, which will be the production database.
I can't seem to get the connection right.
I have tried using a DSN but I get an error message indicating there is a mismatch between the driver and something else (Python?). The searches on this error seem to indicate the driver is 32 bit and Python is 64. Still I can't get that working so I thought I'd try to connect directly. I'd prefer not using a DSN anyway. I've searched the web and can't find an example that works for me.
I have imported pyodbc. This is the current way I'm trying to connect:
params = urllib.quote_plus("DRIVER={SQL Server};SERVER=devsql07:1433;DATABASE=DevOpsSnippets;UID=<user>;PWD=<password>")
SQLALCHEMY_DATABASE_URI = "mssql+pyodbc:///?odbc_connect=%s" % params
This produces the following error message:
2016-02-17 07:11:38,115:ERROR:flask_appbuilder.security.sqla.manager:DB Creation and initialization failed: (pyodbc.Error) ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]Invalid connection. (14) (SQLDriverConnect); [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (ParseConnectParams()). (14)')
Can anyone help me get this connection correct?
I really appreciate any help.

if you are using pyodbc you should be able to connect this way
import pyodbc
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=yourServer;DATABASE=yourDatabase;UID=;PWD=')
#putting to use
SQL = "select Field1, Field2 from someTable"
cursor = cnxn.cursor()
cursor.execute(SQL)
row = cursor.fetchall()
for r in row:
print r[0] #field1
print r[1] #field2

The port should be specified through a comma. Specify the connection string as
DRIVER={SQL Server};SERVER=devsql07,1433;DATABASE.....

Cannot establish connection to sql-server using pyodbc on Windows 7

I'm using ActivePython 2.7.2.5 on Windows 7.
While trying to connect to a sql-server database with the pyodbc module using the below code, I receive the subsequent Traceback. Any ideas on what I'm doing wrong?
CODE:
import pyodbc
driver = 'SQL Server'
server = '**server-name**'
db1 = 'CorpApps'
tcon = 'yes'
uname = 'jnichol3'
pword = '**my-password**'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=server;DATABASE=db1;UID=uname;PWD=pword;Trusted_Connection=yes')
cursor = cnxn.cursor()
cursor.execute("select * from appaudit_q32013")
rows = cursor.fetchall()
for row in rows:
print row
TRACEBACK:
Traceback (most recent call last):
File "pyodbc_test.py", line 9, in <module>
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=server;DATABASE=db1;UID=uname;PWD=pword;Trusted_Connection=yes')
pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC SQL Server Driver][DBNETLIB]SQL Server does not exist or access denied. (17) (SQLDriverConnect); [01000] [Microsoft][ODBC SQL Server Driver][DBNETLIB]ConnectionOpen (Connect()). (53)')

You're using a connection string of 'DRIVER={SQL Server};SERVER=server;DATABASE=db1;UID=uname;PWD=pword;Trusted_Connection=yes', you're trying to connect to a server called server, a database called db1, etc. It doesn't use the variables you set before, they're not used.
It's possible to pass the connection string parameters as keyword arguments to the connect function, so you could use:
cnxn = pyodbc.connect(driver='{SQL Server}', host=server, database=db1,
trusted_connection=tcon, user=uname, password=pword)

I had the same error message and in my case the issue was the [SQL Server] drivers required TLS 1.0 which is disabled on my server. Changing to the newer version of the SNAC, SQL Server Native Client 11.0 fixed the problem.
So my connection string looks like:
cnxn = pyodbc.connect(driver='{SQL Server Native Client 11.0}',
host=server, database=db1, trusted_connection=tcon,
user=uname, password=pword)

I had faced this error due to another reason.
It was because my server had a "port" apart from the address.
I could fix that by assigning the following value to "Server" parameter of the connection string.
"...;Server=<server_name>,<port#>;..."
Note that it is a 'comma' and not 'colon'/'period'

cnxn = pyodbc.connect(driver='{SQL Server}', host=server, database=db1,
user=uname, password=pword)
print(cnxn)
I removed "Trusted_Connection" part and it worked for me.

Different security risks exist with either method. If you use Sql Server authentication you expose your userid/password in the code. But at least you process with the same credentials. If you use Windows authentication you have to insure all the possible users are setup with the right permission in the Sql server. With Sql authentication you can setup just one user but multiple people can use that one Sql User permissions wise.

I had the same issue today. I was using localhost in the connectionstring. Got rid of the issue by replacing localhost woth 'server name',. My db and application are running in the same machine.
If you don't have server name
go to Sql server management studio and execute below query, which will give you the server name.
SELECT ##SERVERNAME
The connection string look as below
conn = pyodbc.connect('Driver={SQL Server};'
'Server=myServerName;'
'Database=mydb;'
'Trusted_Connection=yes;')