I'm trying to execute multiple queries in SQL Server using the pymssql library.
This is my code:
cur = conn.cursor()
cur.execute("DECLARE #begin_time datetime, #end_time datetime, #from_lsn binary(10), #to_lsn binary(10);
SET #begin_time = DATEADD(day, -1, GETDATE()) ;
SET #end_time = GETDATE();
SET #from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', #begin_time);
SET #to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', #end_time);
SELECT * FROM cdc.fn_cdc_get_net_changes_dbo_users(#from_lsn, #to_lsn, 'all');")
output = cur.fetchall()
print(output)
conn.close()
The code is running fine and fetching the result, however when I'm calculating the date using Python library and passing it to the code, I'm getting an error.
Sample code
from datetime import datetime, timedelta
end_date = datetime.now()
start_date = datetime.now() + timedelta(hours=-1)
cur = conn.cursor()
query = f"""DECLARE #begin_time datetime, #end_time datetime, #from_lsn binary(10), #to_lsn binary(10);
SET #begin_time = {start_date};
SET #end_time = {end_date};
SET #from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', #begin_time);
SET #to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', #end_time);
SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_users (#from_lsn, #to_lsn, 'all');"""
print(query)
cur.execute(query)
output = cur.fetchall()
print(output)
conn.close()
Error:
ProgrammingError: (102, Incorrect syntax near '11'.
DB-Lib error message 20018, severity 15:
General SQL Server error: Check messages from the SQL Server
I'm not sure what I'm doing wrong here. Would really appreciate if someone can help me and explain the issue.
Consider actual SQL parameterization of the time variables and not string interpolation or concatenation with F-strings which generally is not safe or efficient for passing values from application layer to backend database. The library, pymssql, supports parameters. Python's datetime.datetime should translate to MSSQL's DATETIME.
# PREPARED STATEMENTS WITH %s PLACEHOLDERS
query = """DECLARE #begin_time datetime, #end_time datetime, #from_lsn binary(10), #to_lsn binary(10);
SET #begin_time = %s;
SET #end_time = %s;
SET #from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', #begin_time);
SET #to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', #end_time);
SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_users (#from_lsn, #to_lsn, 'all');
"""
print(query)
# EXECUTE QUERY WITH BINDED PARAMS
cur.execute(query, [start_date, end_date])
In fact, you can shorten the query since parameters do not need declaration:
# PREPARED STATEMENTS WITH %s PLACEHOLDERS
query = """DECLARE #from_lsn binary(10), #to_lsn binary(10);
SET #from_lsn = sys.fn_cdc_map_time_to_lsn('smallest greater than or equal', %s);
SET #to_lsn = sys.fn_cdc_map_time_to_lsn('largest less than or equal', %s);
SELECT * FROM cdc.fn_cdc_get_all_changes_dbo_users (#from_lsn, #to_lsn, 'all');
"""
By the way, please note pymmsql is no longer a maintained library. Consider pyodbc for most secure, updated DB-API for Python-SQL Server connections. But note the parameter placeholder for pyodbc is qmarks ? and not %s (unlike most Python DB-APIs).
Related
I would be grateful to know what is the easiest way to diagnose this error, as it does not seem to be easy to display what SQL is being executed via pyodbc.
My stored procedure looks like this:
ALTER PROCEDURE [dbo].[s_populate_Test_sp]
#TestDateTime DATETIME,
#TestInt INT,
#TestMoney MONEY,
#TestVarChar500 VARCHAR(500),
#TestFloat FLOAT
AS
SET NOCOUNT ON
INSERT INTO [dbo].[tbl_Test_sp2] (test_datetime, test_int, test_money, test_varchar500, test_float)
VALUES (#TestDateTime, #TestInt, #TestMoney, #TestVarChar500, #TestFloat)
I can execute this stored procedure once successfully using raw text (the commented code below), but I am having difficulty with executemany:
import os
import pyodbc
import datetime
def test_sp():
# Constants
dir_path = os.path.dirname(os.path.realpath(__file__))
# Connect
server = 'xxx'
db2 = 'xxx'
conn_str = 'DRIVER={SQL Server};SERVER=' + server + \
';DATABASE=' + db2 + ';Trusted_Connection=yes'
conn=pyodbc.connect(conn_str, autocommit=False)
cursor = conn.cursor()
cursor.fast_executemany = True
for row in range(10000):
# sql = '''EXEC [dbo].[s_populate_Test_sp] #TestDateTime = '2020-01-01 13:00',
# #TestInt = 999,
# #TestMoney = '£12.34',
# #TestVarChar500 = 'Hello My Name is Jon',
# #TestFloat = 1.234567
# '''
# cursor.execute(sql)
sql = '''exec s_populate_Test_sp (#TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?)'''
values = ['2020-01-01 13:00', 999, '£12.34', 'Hello My Name is Jon', 1.234567]
cursor.executemany(sql, [values])
conn.commit()
if __name__ == '__main__':
test_sp()
Unfortunately this yields a rather cryptic error message:
ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '#TestDateTime'. (102) (SQLExecute); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")
I can't find a way of displaying the SQL before it gets executed, so it's all a bit trial and error at the moment.
Many thanks
Per the comments, the answer was to remove the parentheses, the £ sign and use a datetime.datetime object:
sql = '''exec s_populate_Test_sp #TestDateTime = ?, #TestInt = ?, #TestMoney = ?, #TestVarChar500 = ?, #TestFloat = ?'''
values = [datetime.datetime.now(), 999, 12.34, 'Hello My Name is Jon', 1.234567]
It is frustrating that it is so slow. Over my firm's VPN it can do 400 records per minute and on a virtual machine close to the server it can do 9,000 records per minute. I suspect this is one of the limitations of pyodbc and SQL Server, and I will have to do bcp or something like that for larger datasets.
I've a script that makes a query to my database on MySQL. And my doubt is if I can pass any parameter to that query through Python.
For example, on the following script I want to calculate the date_filter using Python and then apply that filter on the query.
now = dt.datetime.now()
date_filter = now - timedelta(days=3)
dq_connection = mysql.connector.connect(user='user', password='pass', host='localhost', database='db')
engine = create_engine('localhost/db')
cursor = connection.cursor(buffered=True)
query = ('''
SELECT *
FROM myTable
WHERE date >= ''' + date_filter + '''
''')
I try it on that way but I got the following error:
builtins.TypeError: can only concatenate str (not "datetime.datetime") to str
It is possible to apply the filter like that?
Thanks!
Yes, you can do it. To avoid sql injections, the best way is not using the python formatting facilities, but the sql parameters & placeholders (see that you don´t need the single quotes ' as the placeholder does the job and converts the type of the variable):
now = dt.datetime.now()
date_filter = now - timedelta(days=3)
dq_connection = mysql.connector.connect(user='user', password='pass', host='localhost', database='db')
engine = create_engine('localhost/db')
cursor = db_connection.cursor(buffered=True)
query = "SELECT * FROM myTable WHERE date >=%s"
cursor.execute(query,(date_filter,))
Also, you had a mistake in your cursor, it should be db_connection.cursor. The last comma after date_filter is ok because you need to send a tuple.
In case you need more than one paremeter, you can place more than one placeholder:
query = "SELECT * FROM myTable WHERE date >=%s and date<=%s"
cursor.execute(query,(date_filter,other_date))
You can just do something like:
WHERE date >= ''' + str(date_filter) + '''
to represent the date as string as not a datetime object.
You can try with this:
date_f = str(date_filter)
query = ('''
SELECT *
FROM myTable
WHERE date >= "{}"
'''.format(date_f))
Error when using timedelta from datetime.now() in SQL Server where clause
python 3.6
yesterday = datetime.now() - timedelta(days=1)
sql = "SELECT submit_dt, api_job_name, job_status, xml_record_count, x_successful_number, x_failed_number, " \
f"job_run_time, mf_job_name FROM JOB_LOG where submit_dt > {yesterday}"
try:
db = Database()
db.cursor.execute(sql)
rows = db.cursor.fetchall()
SQL ODBC Error: Incorrect syntax near '22' --- which is the time part of the datetime.
I've tried wrapping it in '' but then get convert from string error.
Consider parameterizing your query without any need of string conversion of datetime or string interpolation including F-strings.
yesterday = datetime.now() - timedelta(days=1)
sql = """SELECT submit_dt, api_job_name, job_status, xml_record_count,
x_successful_number, x_failed_number,
job_run_time, mf_job_name
FROM JOB_LOG
WHERE submit_dt > ?"""
try:
db = Database()
db.cursor.execute(sql, yesterday)
rows = db.cursor.fetchall()
The error was due to including the microseconds in the compare value. I was able to use:
yesterday_sql = yesterday.strftime("%Y-%m-$d %H:%M:%S")
I'm using pyodbc together with QODBC to construct an ODBC query.
I'm having trouble inserting datestamp parameters. Here you can see the escalation starting from the literal version (1) to string-format version (2) to error-state versions. (Note DateFrom & DateTo):
sql = "sp_report ProfitAndLossStandard show Amount_Title, Text, Label, Amount parameters DateFrom = {d'2018-02-12'}, DateTo = {d'2018-02-18'}, SummarizeColumnsBy='TotalOnly', ReturnRows='All'"
sql = "sp_report ProfitAndLossStandard show Amount_Title, Text, Label, Amount parameters DateFrom = %s, DateTo = %s, SummarizeColumnsBy='TotalOnly', ReturnRows='All'" % (q_startdate, q_enddate)
Subsequent attempts with the insertion syntax ?, cursor.execute(sql, (q_startdate), (q_enddate)) and the variables:
q_startdate = ("{d'%s'}" % dates[0])
q_enddate = ("{d'%s'}" % dates[1])
sql = "sp_report ProfitAndLossStandard show Amount_Title, Text, Label, Amount parameters DateFrom = ?, DateTo = ?, SummarizeColumnsBy='TotalOnly', ReturnRows='All'"
>>> ('HY004', '[HY004] [Microsoft][ODBC Driver Manager] SQL data type out of range (0) (SQLBindParameter)')
q_startdate = (dates[0])
q_enddate = (dates[1])
sql = "sp_report ProfitAndLossStandard show Amount_Title, Text, Label, Amount parameters DateFrom = {d'?'}, DateTo = {d'?'}, SummarizeColumnsBy='TotalOnly', ReturnRows='All'"
>>> ('42000', "[42000] [QODBC] [sql syntax error] Expected lexical element not found: = {d'?'} (11015) (SQLPrepare)")
Reading the pyodbc Wiki page on inserting data, I don't read about any speed bumps with insertion strings. This must have something to do with how pyodbc processes (escapes) the datestamp.
How do you parameterize datestamp--Especially with the qodbc flavor of datestamp.
It is almost never necessary to use ODBC escape sequences like {d'2018-02-12'} in a pyodbc parameterized query. If the parameter value is a true Python date object
q_startdate = date(2018, 2, 12)
then pyodbc will inform the ODBC driver that the parameter value is a SQL_TYPE_DATE as shown in the ODBC trace log
[ODBC][2984][1532535987.825823][SQLBindParameter.c][217]
Entry:
Statement = 0x1f1a6b0
Param Number = 1
Param Type = 1
C Type = 91 SQL_C_TYPE_DATE
SQL Type = 91 SQL_TYPE_DATE
Col Def = 10
Scale = 0
Rgb Value = 0x1f3ac78
Value Max = 0
StrLen Or Ind = 0x1f3ac58
and we can just use a bare parameter placeholder in our SQL command text
... parameters DateFrom = ?, ...
I am using Python 3.6, pyodbc, and connect to SQL Server.
I am trying make connection to a database, then creating a query with parameters.
Here is the code:
import sys
import pyodbc
# connection parameters
nHost = 'host'
nBase = 'base'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
print("connection successfull")
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
# if run WITHOUT parameters THEN everything is OK
ask = input ('Go WITHOUT parameters y/n ?')
if ask == 'y':
# SQL without parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN '2017-03-01T00:00:00.000' AND '2017-03-01T01:00:00.000'
''')
# SQL without parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
# if run WITH parameters THEN ERROR
ask = input ('Go WITH parameters y/n ?')
if ask == 'y':
# parameters start
STARTDATE = "'2017-03-01T00:00:00.000'"
ENDDATE = "'2017-03-01T01:00:00.000'"
# parameters end
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN :STARTDATE AND :ENDDATE
''', {"STARTDATE": STARTDATE, "ENDDATE": ENDDATE})
# SQL with parameters stop
# print result to console start
row = res.fetchone()
while row:
print (row)
row = res.fetchone()
# print result to console stop
When I run the program without parameters in SQL, it works.
When I try running it with parameters, an error occurred.
Parameters in an SQL statement via ODBC are positional, and marked by a ?. Thus:
# SQL with parameters start
res = cursor.execute('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN ? AND ?
''', STARTDATE, ENDDATE)
# SQL with parameters stop
Plus, it's better to avoid passing dates as strings. Let pyodbc take care of that using Python's datetime:
from datetime import datetime
...
STARTDATE = datetime(year=2017, month=3, day=1)
ENDDATE = datetime(year=2017, month=3, day=1, hour=0, minute=0, second=1)
then just pass the parameters as above. If you prefer string parsing, see this answer.
If you're trying to use pd.to_sql() like me I fixed the problem by passing a parameter called chunksize.
df.to_sql("tableName", engine ,if_exists='append', chunksize=50)
hope this helps
i tryied and have a lot of different errors: 42000, 22007, 07002 and others
The work version is bellow:
import sys
import pyodbc
import datetime
# connection parameters
nHost = 'host'
nBase = 'DBname'
nUser = 'user'
nPasw = 'pass'
# make connection start
def sqlconnect(nHost,nBase,nUser,nPasw):
try:
return pyodbc.connect('DRIVER={SQL Server};SERVER='+nHost+';DATABASE='+nBase+';UID='+nUser+';PWD='+nPasw)
except:
print ("connection failed check authorization parameters")
con = sqlconnect(nHost,nBase,nUser,nPasw)
cursor = con.cursor()
# make connection stop
STARTDATE = '11/2/2017'
ENDDATE = '12/2/2017'
params = (STARTDATE, ENDDATE)
# SQL with parameters start
sql = ('''
SELECT * FROM TABLE
WHERE TABLE.TIMESTAMP BETWEEN CAST(? as datetime) AND CAST(? as datetime)
''')
# SQL with parameters stop
# print result to console start
query = cursor.execute(sql, params)
row = query.fetchone()
while row:
print (row)
row = query.fetchone()
# print result to console stop
say = input ('everething is ok, you can close console')
I fixed this issue with code if you are using values through csv.
for i, row in read_csv_data.iterrows():
cursor.execute('INSERT INTO ' + self.schema + '.' + self.table + '(first_name, last_name, email, ssn, mobile) VALUES (?,?,?,?,?)', tuple(row))
I had a similar issue. Saw that downgrading the version of PyODBC to 4.0.6 and SQLAlchemy to 1.2.9 fixed the error,using Python 3.6