Develop Reference

How to do df.to_sql using SQL Server in Azure

I can do a df.to_slq on my local instance of SQL Server just fine. I am getting stuck when trying to do the same df.to_sll using Python and Azure SQL Server. I thought it would essentially be done like this.
import urllib.parse
params = urllib.parse.quote_plus(
'Driver=%s;' % '{ODBC Driver 17 for SQL Server}' +
'Server=%s,1433;' % 'ryan-server.database.windows.net' +
'Database=%s;' % 'ryan_sql_db' +
'Uid=%s;' % 'UN' +
'Pwd={%s};' % 'PW' +
'Encrypt=no;' +
'TrustServerCertificate=no;'
)
from sqlalchemy.engine import create_engine
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
connection = engine.connect()
connection
all_data.to_sql('health', engine, if_exists='append', chunksize=100000, method=None,index=False)
That is giving me this error.
OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\r\n (10060) (SQLExecDirectW); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10060)')
[SQL: INSERT INTO health ([0], [Facility_BU_ID], [Code_Type], [Code], [Description], [UB_Revenue_Code], [UB_Revenue_Description], [Gross_Charge], [Cash_Charge], [Min_Negotiated_Rate], [Max_Negotiated_Rate], etc., etc., etc.
I found this link today:
https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-sql-server?view=sql-server-ver15
I tried to do something similar, like this.
import pyodbc
import pandas as pd
df = all_data
# server = 'myserver,port' # to specify an alternate port
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db'
username = 'UN'
password = 'PW'
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# Insert Dataframe into SQL Server:
for index, row in df.iterrows():
cursor.execute(all_data.to_sql('health', cnxn, if_exists='append', chunksize=100000, method=None,index=False))
cnxn.commit()
cursor.close()
When I run that, I get this error.
DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
What I'm really hoping to to is df.to_sql, not Insert Into. I am working in Spyder and trying to send the data from my local machine to the cloud.

I read the two links below, and got it working.
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-set-database-firewall-rule-azure-sql-database?view=azuresqldb-current
https://www.virtual-dba.com/blog/firewalls-database-level-azure-sql/
Basically, you need to open your command window on your local machine, enter 'ipconfig', and grab two IP addresses. Then, enter those into SQL Server in Azure.
EXECUTE sp_set_database_firewall_rule
N'health',
'192.0.1.1',
'192.0.0.5';
Finally, run the small script below, in SQL Server, to confirm that the changes were made correctly.
USE [ryan_sql_db]
GO
SELECT * FROM sys.database_firewall_rules
ORDER BY modify_date DESC

The SQL contains 0 parameter markers, but 1 parameters were supplied | pyodbc

I am trying to send a data from my code to SQL Server, but i got this error:
ProgrammingError:
('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near 'emotion'. (102) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
My Code:
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=DESKTOP-T7OFQV6\SQLEXPRESS1;'
'Database=VidgaEmotionRecognition;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('''INSERT INTO FaceEmotion emotion ?''', index_pred_as_int)
index_pred is a value from previous code snippet.
Structure of FaceEmotion table is like this:
Column: emotion - nvarchar(50) - allows nulls
Column: _id - int - key identity

Getting incorrect syntax for SQL statement in python

When I run below code which incorporates TOP ? in a select statement and I am unable to resolve.
Code:
cnt = self.getCount()
#cnt = 2
query = "select distinct top ? id, indicator from [TABLE] ORDER BY id ASC"
connsql = self.sql_connection()
resultset = connsql.cursor().execute(query, cnt).fetchall()
connsql.cursor().commit()
connsql.close()
I get this syntax error:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '#P1'. (102) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")

How to drop and create database using sql sanitizing

cursor.execute("DROP DATABASE ?", (databasename,))
I am using python3 with pyodbc driver. Only facing issue while create and delete database. other operations like select are working fine.
Getting below error:
pyodbc.ProgrammingError: ('42000', u"[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near '#P1'. (102) (SQLExecDirectW)")

In order to sanitize your data you can use SQL Server QUOTENAME to returns a Unicode string with the delimiters added to make the input string a valid SQL Server delimited identifier.
You also need to set autocommit=True in your pyodbc connection to allow dropping of databases.
conn = pyodbc.connect("DRIVER={SQL Server};"
"SERVER="+server+";"
"UID="+username+";"
"PWD="+password,
autocommit=True)
cursor = conn.cursor()
your_database_name = "YOUR_DB_NAME"
sql_drop = (
"DECLARE #sql AS NVARCHAR(MAX);"
"SET #sql = 'DROP DATABASE ' + QUOTENAME(?);"
"EXEC sp_executesql #sql"
)
cursor.execute(sql_drop, your_database_name)

Creating stored procedure in MS SQL through pyodbc

I am trying to create a store procedure in master in mssql through python code. The following is my code:
import pyodbc
conn = pyodbc.connect("driver={SQL Server};server=localhost; database=master; trusted_connection=true",
autocommit=True)
cursor = conn.cursor()
sqlcommand = """
USE master
GO
CREATE PROCEDURE sp_myCustomSystemProc
AS
BEGIN
PRINT 'myCustomCode'
END
GO
EXEC sp_ms_marksystemobject 'sp_myCustomSystemProc'
"""
cursor.execute(sqlcommand)
cursor.commit()
conn.commit()
After running this python code, I am getting this error:
Traceback (most recent call last):
File "auto_complete.py", line 27, in <module>
cursor.execute(sqlcommand)
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Incorrect syntax near 'GO'. (102) (SQLExecDirectW);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
'CREATE/ALTER PROCEDURE' must be the first statement in a query batch. (111);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Incorrect syntax near 'GO'. (102);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Incorrect syntax near 'sp_myCustomSystemProc'. (102)")
Can anyone please help me to resolve this?

Since your connection string already specifies the master database (i.e. database=master;), simply remove
USE master
GO
from your query.