Develop Reference

executemany with pyodbc, stored procedures and SQL Server

I would be grateful to know what is the easiest way to diagnose this error, as it does not seem to be easy to display what SQL is being executed via pyodbc.
My stored procedure looks like this:
ALTER PROCEDURE [dbo].[s_populate_Test_sp]
#TestDateTime DATETIME,
#TestInt INT,
#TestMoney MONEY,
#TestVarChar500 VARCHAR(500),
#TestFloat FLOAT
AS
SET NOCOUNT ON
INSERT INTO [dbo].[tbl_Test_sp2] (test_datetime, test_int, test_money, test_varchar500, test_float)
VALUES (#TestDateTime, #TestInt, #TestMoney, #TestVarChar500, #TestFloat)
I can execute this stored procedure once successfully using raw text (the commented code below), but I am having difficulty with executemany:
import os
import pyodbc
import datetime
def test_sp():
# Constants
dir_path = os.path.dirname(os.path.realpath(__file__))
# Connect
server = 'xxx'
db2 = 'xxx'
conn_str = 'DRIVER={SQL Server};SERVER=' + server + \
';DATABASE=' + db2 + ';Trusted_Connection=yes'
conn=pyodbc.connect(conn_str, autocommit=False)
cursor = conn.cursor()
cursor.fast_executemany = True
for row in range(10000):
# sql = '''EXEC [dbo].[s_populate_Test_sp] #TestDateTime = '2020-01-01 13:00',
# #TestInt = 999,
# #TestMoney = '£12.34',
# #TestVarChar500 = 'Hello My Name is Jon',
# #TestFloat = 1.234567
# '''
# cursor.execute(sql)
sql = '''exec s_populate_Test_sp (#TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?)'''
values = ['2020-01-01 13:00', 999, '£12.34', 'Hello My Name is Jon', 1.234567]
cursor.executemany(sql, [values])
conn.commit()
if __name__ == '__main__':
test_sp()
Unfortunately this yields a rather cryptic error message:
ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '#TestDateTime'. (102) (SQLExecute); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
I can't find a way of displaying the SQL before it gets executed, so it's all a bit trial and error at the moment.
Many thanks

Per the comments, the answer was to remove the parentheses, the £ sign and use a datetime.datetime object:
sql = '''exec s_populate_Test_sp #TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?'''
values = [datetime.datetime.now(), 999, 12.34, 'Hello My Name is Jon', 1.234567]
It is frustrating that it is so slow. Over my firm's VPN it can do 400 records per minute and on a virtual machine close to the server it can do 9,000 records per minute. I suspect this is one of the limitations of pyodbc and SQL Server, and I will have to do bcp or something like that for larger datasets.

How do I issue a SQL query using pyodbc that involves python variables as the parameters?

I am writing a program that searches a database and inputs the data into a excel file based on user selection from two calendars (on the GUI). I have a start and end date, I am passing those variables as parameters into a function for my SQL query.
How do I format the SQL query to select the values from the given dates?
Here is my code so far,
def load_database(startDate, endDate):
conn = pyodbc.connect('Driver={ODBC Driver 17 for SQL Server};'
'Server=DESKTOP-KVR7GNJ\SQLBULKROLL;'
'Database=db_MasterRoll;'
'UID=sa;'
'PWD=Z28#6420d')
wb = Workbook()
ws = wb.active
ws.title = "Star Ledger"
cursor = conn.cursor()
cursor.execute('SELECT ID, BarCode, DateTm, EntryDoor FROM dbo.tab_Rolls WHERE (DateTm >= #startDate) AND (DateTm <= #endDate)')
a button invokes this code by running the function "call_db_code()":
def call_db_code():
firstDate = start_dateCalendar.selection_get()
print(firstDate)
finalDate = end_dateCalendar.selection_get()
print(finalDate)
load_database(firstDate, finalDate)

Use ? as placeholders in the query, and provide a tuple of values to substitute for them.
cursor.execute('SELECT ID, BarCode, DateTm, EntryDoor FROM dbo.tab_Rolls WHERE (DateTm >= ?) AND (DateTm <= ?)', (startDate, endDate))
See the examples in the cursor.execute() documentation.

pyodbc not recognising parameter placeholder in multi-line string

Why is pyodbc throwing this error:
pyodbc.ProgrammingError: ('The SQL contains 0 parameter markers, but 1 parameters were supplied', 'HY000')
When I do have a parameter marker in the query?
It seems it only complains when I use a multiline string in python.
i.e. this is causing an error:
sql = """
SELECT a.r_object_id, a.object_name, a.document_status, a.document_id, a.document_status, b.r_version_label, a.r_link_cnt
from pharma_document_sp a, pharma_document_rp b where a.r_object_id = b.r_object_id and b.r_version_label = 'LATEST APPROVED' and a.document_id = ?
"""
cursor_cara.execute(sql, doc_id)
But this is OK:
sql = "SELECT a.r_object_id, a.object_name, a.document_status, a.document_id, a.document_status, b.r_version_label, a.r_link_cnt "
sql = sql + " from pharma_document_sp a, pharma_document_rp b where a.r_object_id = b.r_object_id and b.r_version_label = 'LATEST APPROVED' and a.document_id = ?"
cursor_cara.execute(sql, doc_id)
I am using:
pyodbc version 4.0.27, SQL ODBC Server Driver 17, Python 3.7.4

pyodbc the sql contains 0 parameter markers but 1 parameters were supplied' 'hy000'

I am using Python 3.6, pyodbc, and connect to SQL Server.
I am trying make connection to a database, then creating a query with parameters.
Here is the code:
import sys
import pyodbc
# connection parameters
nHost = 'host'
nBase = 'base'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
print("connection successfull")
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
# if run WITHOUT parameters THEN everything is OK
ask = input ('Go WITHOUT parameters y/n ?')
if ask == 'y':
# SQL without parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN '2017-03-01T00:00:00.000' AND '2017-03-01T01:00:00.000'
''')
# SQL without parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
# if run WITH parameters THEN ERROR
ask = input ('Go WITH parameters y/n ?')
if ask == 'y':
# parameters start
STARTDATE = "'2017-03-01T00:00:00.000'"
ENDDATE = "'2017-03-01T01:00:00.000'"
# parameters end
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN :STARTDATE AND :ENDDATE
''', {"STARTDATE": STARTDATE, "ENDDATE": ENDDATE})
# SQL with parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
When I run the program without parameters in SQL, it works.
When I try running it with parameters, an error occurred.

Parameters in an SQL statement via ODBC are positional, and marked by a ?. Thus:
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN ? AND ?
''', STARTDATE, ENDDATE)
# SQL with parameters stop
Plus, it's better to avoid passing dates as strings. Let pyodbc take care of that using Python's datetime:
from datetime import datetime
...
STARTDATE = datetime(year=2017, month=3, day=1)
ENDDATE = datetime(year=2017, month=3, day=1, hour=0, minute=0, second=1)
then just pass the parameters as above. If you prefer string parsing, see this answer.

If you're trying to use pd.to_sql() like me I fixed the problem by passing a parameter called chunksize.
df.to_sql("tableName", engine ,if_exists='append', chunksize=50)
hope this helps

i tryied and have a lot of different errors: 42000, 22007, 07002 and others
The work version is bellow:
import sys
import pyodbc
import datetime
# connection parameters
nHost = 'host'
nBase = 'DBname'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
STARTDATE = '11/2/2017'
ENDDATE = '12/2/2017'
params = (STARTDATE, ENDDATE)
# SQL with parameters start
sql = ('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN CAST(? as datetime) AND CAST(? as datetime)
''')
# SQL with parameters stop
# print result to console start
query = cursor.execute(sql, params)
row = query.fetchone()
while row:
print (row)
row = query.fetchone()
# print result to console stop
say = input ('everething is ok, you can close console')

I fixed this issue with code if you are using values through csv.
for i, row in read_csv_data.iterrows():
cursor.execute('INSERT INTO ' + self.schema + '.' + self.table + '(first_name, last_name, email, ssn, mobile) VALUES (?,?,?,?,?)', tuple(row))

I had a similar issue. Saw that downgrading the version of PyODBC to 4.0.6 and SQLAlchemy to 1.2.9 fixed the error,using Python 3.6

How to use for loops to do multiple SQL statements in python

I have SQl statement List, when running the a single statement it work running the loop it gives:
pyodbc.ProgrammingError: ('42000', "[42000] [MySQL][ODBC 5.1 Driver][mysqld-5.5.8]You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'Sql_2' at line 1 (1064) (SQLExecDirectW)")
SQl = """Select something"""
SQl_2 = """Select something"""
SQl_3 = """Select something"""
Sqls= ('Sql','Sql_2','Sql_3')
for x in Sqls:
print x
use = Sql_2
# use = x
cxn = pyodbc.connect('DSN=MySQL;PWD=xxx')
csr = cxn.cursor()
csr.execute(use)
fetch = csr.fetchall()

Your tuple should be
Sqls = (Sql,Sql_2,Sql_3)
instead of
Sqls = ('Sql','Sql_2','Sql_3')

You should additionally move connecting to the database as well as creating a cursor out of the for-loop, since it's unneeded overhead.
SQl = """Select something"""
SQl_2 = """Select something"""
SQl_3 = """Select something"""
Sqls = (Sql, Sql_2, Sql_3)
cxn = pyodbc.connect('DSN=MySQL;PWD=xxx')
csr = cxn.cursor()
for x in Sqls:
print x
use = Sql_2
# use = x
csr.execute(use)
fetch = csr.fetchall()