I have a docker image with a script which reads an iql file and executes it in impala. Removing the file reference and running a simple script works fine (ie SELECT 1;), but whenever I try to execute the script from a file I get an error
pyodbc.Error: ('HY000', '[HY000] [Cloudera][ImpalaODBC] (110) Error while executing a query in Impala: [HY000] : ParseException: Syntax error in line 2:\n\n^\nEncountered: EOF\nExpected: ALTER, COMMENT, COMPUTE, COPY, CREATE, DELETE, DESCRIBE, DROP, EXPLAIN, GRANT, INSERT, INVALIDATE, LOAD, REFRESH, REVOKE, SELECT, SET, SHOW, TRUNCATE, UPDATE, UPSERT, USE, VALUES, WITH\n\nCAUSED BY: Exception: Syntax error\n (110) (SQLExecDirectW)'
I have tried removing DriverManagerEncoding=UTF-16 from the cloudera.impalaodbc.ini file and various types of sql query (including changing the file extension to isql, iusql) but always get the same error.
Any ideas?
Python function:
def process_data_to_impala():
conn = pyodbc.connect(DSN='some_DSN', autocommit=True)
crsr = conn.cursor()
with open('/path/to/sql/file.iql','r') as inserts:
for statement in inserts:
crsr.execute(statement)
print(crsr.fetchall())
The problem relates to the way the python code is looping through the statement and the way ';' ends the query.
A better way to do it for a single query in a file is
with open('file/location.iql') as script:
statement = script.read()
print(statement)
crsr = conn.cursor()
crsr.execute(statement)
And when you have multiple queries in a file and you need to loop:
with open('file/location.iql') as script:
sqlscript = script.read()
for statement in sqlscript.split(';'):
print(statement)
crsr = conn.cursor()
crsr.execute(statement)
When reading multiple queries from one script (ie Drop table, create table) you must remove ';' from the last query. This is because the split creates a final blank query after the last ';'. See this question for more info.
I hope this helps somebody in the future!
Related
I can't figure out what's wrong with the following code,
The syntax IS ok (checked with SQL Management Studio), i have access as i should so that works too.. but for some reason as soon as i try to create a table via PyODBC then it stops working.
import pyodbc
def SQL(QUERY, target = '...', DB = '...'):
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + target + DB+';UID=user;PWD=pass')
cursor = cnxn.cursor()
cursor.execute(QUERY)
cpn = []
for row in cursor:
cpn.append(row)
return cpn
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
It fails with:
Traceback (most recent call last):
File "test_sql.py", line 25, in <module>
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
File "test_sql.py", line 20, in SQL
for row in cursor:
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Anyone have any idea to why this is?
I got a "SQL Server" driver installed (it's default), running Windows 7 against a Windows 2008 SQL Server environment (Not a express database).
Just in case some lonely net nomad comes across this issue, the solution by Torxed didn't work for me. But the following worked for me.
I was calling an SP which inserts some values into a table and then returns some data back. Just add the following to the SP :
SET NOCOUNT ON
It'll work just fine :)
The Python code :
query = "exec dbo.get_process_id " + str(provider_id) + ", 0"
cursor.execute(query)
row = cursor.fetchone()
process_id = row[0]
The SP :
USE [DBNAME]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[GET_PROCESS_ID](
#PROVIDER_ID INT,
#PROCESS_ID INT OUTPUT
)
AS
BEGIN
SET NOCOUNT ON
INSERT INTO processes(provider_id) values(#PROVIDER_ID)
SET #PROCESS_ID= SCOPE_IDENTITY()
SELECT #PROCESS_ID AS PROCESS_ID
END
Using the "SET NOCOUNT ON" value at the top of the script will not always be sufficient to solve the problem.
In my case, it was also necessary to remove this line:
Use DatabaseName;
Database was SQL Server 2012,
Python 3.7,
SQL Alchemy 1.3.8
Hope this helps somebody.
I got this because I was reusing a cursor that I was looping over:
rows = cursor.execute(...)
for row in rows:
# run query that returns nothing
cursor.execute(...)
# next iteration of this loop will throw 'Previous SQL' error when it tries to fetch next row because we re-used the cursor with a query that returned nothing
Use 2 different cursors instead
rows = cursor1.execute(...)
for row in rows:
cursor2.execute(...)
or get all results of the first cursor before using it again:
Use 2 different cursors instead
rows = cursor.execute(...)
for row in list(rows):
cursor.execute(...)
As others covered, SET NOCOUNT ON will take care of extra resultsets inside a stored procedure, however other things can also cause extra output that NOCOUNT will not prevent (and pyodbc will see as a resultset) such as forgetting to remove a print statement after debugging your stored procedure.
As Travis and others have mentioned, other things can also cause extra output that SET NOCOUNT ON will not prevent.
I had SET NOCOUNT ON at the start of my procedure but was receiving warning messages in my results set.
I set ansi warnings off at the beginning of my script in order to remove the error messages.
SET ANSI_WARNINGS OFF
Hopefully this helps someone.
If your stored procedure calls RAISERROR, pyodbc may create a set for that message.
CREATE PROCEDURE some_sp
AS
BEGIN
RAISERROR ('Some error!', 1, 1) WITH NOWAIT
RETURN 777
END
In python, you need to skip the first sets until you find one containing some results (see https://github.com/mkleehammer/pyodbc/issues/673#issuecomment-631206107 for details).
sql = """
SET NOCOUNT ON;
SET ANSI_WARNINGS OFF;
DECLARE #ret int;
EXEC #ret = some_sp;
SELECT #ret as ret;
"""
cursor = con.cursor()
cursor.execute(sql)
rows = None
#this section will only return the last result from the query
while cursor.nextset():
try:
rows = cursor.fetchall()
except Exception as e:
print("Skipping non rs message: {}".format(e))
continue
row = rows[0]
print(row[0]) # 777.
I think the root cause of the issue described above might be related with the fact that you receive the same error message when you execute for example a DELETE query which will not return a result. So if you run
result = cursor.fetchall()
you get this error, because a DELETE operation by definition does not return anything. Try to catch the exception as recommended here: How to check if a result set is empty?
In case your SQL is not Stored Proc.
usage of 'xyz != NULL' in query, will give the same error i.e. "pyodbc.ProgrammingError: No results. Previous SQL was not a query."
Use 'is not null' instead.
First off:
if you're running a Windows SQL Server 2008, use the "Native Client" that is included with the installation of the SQL software (it gets installed with the database and Toolkits so you need to install the SQL Management applicaton from Microsoft)
Secondly:
Use "Trusted_Connection=yes" in your SQL connection statement:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 10.0};SERVER=ServerAddress;DATABASE=my_db;Trusted_Connection=yes')
This should do the trick!
I have solved this problem by splitting the use database and sql query into two execute statements.
Quick rundown:
The idea here is to read some data in from a csv file, and use that as the list in the NOT IN part of my sql query. I'm connecting to the db (.mdb) with the code below.
Note LP is the tuple/list I'm trying to pass, IRdb is the path to the db
constr = r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};Dbq=' + IRdb
conn = pyodbc.connect(constr, autocommit=True)
cur = conn.cursor()
IRsql='''SELECT IRRPResults.RRPName, IRRPResults.PointName, IRRPResults.RiskPerTime FROM IRRPResults
WHERE IRRPResults.PointName<>?
AND IRRPResults.RRPName NOT LIKE ? AND IRRPResults.PointName NOT IN ?'''
cur.execute(IRsql,('Total',r'Conn%',LP))
The issue:
Everything works fine except for the execute statement (which did work before i added the NOT IN part). I've tried passing LP as string, tuple, and list, but nothing seems to be working. I get the following error
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Microsoft Access Driver] In operator without () in query expression 'IRRPResults.PointName<>Pa_RaM000 AND IRRPResults.RRPName NOT LIKE Pa_RaM001 AND IRRPResults.PointName NOT IN Pa_RaM002'. (-3100) (SQLExecDirectW)")
Any help would be greatly appreciated.
For anyone interested, or for that guy reading this 2 years from now with the same issue, or my future self when I forget what i did, I've figured out a solution or two.
The first work around was to simply use .format on the sql string to insert LP directly before it gets passed to the execute statement.
IRsql='''SELECT IRRPResults.RRPName, IRRPResults.PointName, IRRPResults.RiskPerTime FROM IRRPResults
WHERE IRRPResults.PointName<>?
AND IRRPResults.RRPName NOT LIKE ? AND IRRPResults.PointName NOT IN {}'''.format(LP)
cur.execute(IRsql,('Total',r'Conn%'))
The other solution, I got from this question, is a little more elegant and clever in that it builds a string of '?' markers for each element in LP. Then LP gets passed as a tuple/list to the execute statement.
placeholders=','.join('?'*len(LP))
IRsql='''SELECT IRRPResults.RRPName, IRRPResults.PointName, IRRPResults.RiskPerTime FROM IRRPResults
WHERE IRRPResults.PointName<>?
AND IRRPResults.RRPName NOT LIKE ? AND IRRPResults.PointName NOT IN ({})'''.format(placeholders)
cur.execute(IRsql,('Total',r'Conn%',*LP))
I'm trying to use OPENJSON in a Python script to import some basic JSON into a SQL database. I initially tried with a more complex JSON file, but simplified it for the sake of this post. Here's what I have:
sql_statement = "declare #json nvarchar(max) = '{\"name\":\"James\"}'; SELECT * FROM OPENJSON(#json) WITH (name nvarchar(20))"
cursor.execute(sql_statement)
cursor.commit()
connection.close()
The error I receive:
pypyodbc.ProgrammingError: (u'42000', u"[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]Incorrect syntax near the keyword 'with'. If
this statement is a common table expression, an xmlnamespaces clause
or a change tracking context clause, the previous statement must be
terminated with a semicolon.")
Any thoughts on why I'm seeing this error? I was successfully able to execute other SQL queries with the same pypyodbc / database configuration.
The problem could be that your database is running in an older compatibility level, where OPEN JSON is not available.
To find the compatibility level of your database, run following SQL statement:
SELECT compatibility_level FROM sys.databases WHERE name = 'your_db_name';
If the result is 120 or lower, you'll need to update your compatibility level to 130, by running:
ALTER DATABASE your_db_name SET COMPATIBILITY_LEVEL = 130;
Note: In case your database is actually Azure SQL DB, you should check the version as well, as OPEN JSON is not available for versions prior to 12.x
I have a MySQL stored procedure that is executed from Python (wrapped in Django). I get the error "commands out of sync; you can't run this command now" when I try to execute the second statement. I cannot commit the transaction at this point. This is only an issue when I call a procedure. What to do?
cursor.callproc('my_mysql_procedure', [some_id,])
result = cursor.fetchall()
for r in result:
do something
cursor.execute("select * from some_table")
result = cursor.fetchall()
EDIT: I've been asked to post the MySQL procedure. I have made it super-simple and I still see the same problem
delimiter $$
create procedure my_mysql_procedure(p_page_id int)
begin
select 1
from dual;
end$$
delimiter ;
Thanks to JoshuaBoshi for his answer, which solved the problem. After calling the procedure, I had to close the cursor and open it again before using it to execute another statement:
cursor.close()
cursor = connection.cursor()
The cursor can be closed immediately after fetchall(). The result set still remains and can be looped through.
The main cause of this is results that are not taken from the cursor before a new query is made. There may be multiple result sets.
To stop the error you must ensure you consume the result set each time with .nextset. If it produces multiple result sets- you may even need to do a few of them.
cursor.callproc('my_mysql_procedure', [some_id,])
result = cursor.fetchall()
for r in result:
do something
cursor.nextset()
cursor.execute("select * from some_table")
result = cursor.fetchall()
In my case, I found that this can actually be an indicator of other problems in the sql queries that aren't picked out as python errors until a subsequent query is made. They produced multiple result sets.
Applying this to my example below (where I saw the same error),
>>> import MySQLdb
>>> conn = MySQLdb.connect(passwd="root", db="test")
>>> cur = conn.cursor()
>>> cur.execute("insert into foo values (1););")
1L
>>> cur.nextset()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 107, in nextset
nr = db.next_result()
_mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')' at line 1
Here - because a parsing error (some trusted input data, which was munged with code) lead to that semicolon and then a new statement - this produced multiple result sets. It will not produce an error for this statement, but for the next one that attempts to run a command on the cursor.
I've made a github repo - https://github.com/odmsolutions/mysql_python_out_of_sync_demo - to demonstrate and test this.
Original answer:
Take a look at https://github.com/farcepest/MySQLdb1/issues/28 for details on how I was able to reliably reproduce this with 3 lines of code:
Minimal case to reproduce this:
(assume you have a blank db, and have created only a connection to the db, called conn)
>>> conn.query("Create table foo(bar int(11))")
>>> conn.query("insert into foo values (1););")
>>> conn.query("insert into foo values (2););")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
_mysql_exceptions.ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
It is bad syntax which was generated and from the error I couldn’t tell that that was the problem.
Try examining the last query or procedure as Bukzor suggests - and try running it in a raw mysql client to see the real problem.
This is not a python issue, but a mysql issue. Do the same from perl and you will get the same error. In general, I see that message when I run a mysql console, then kill it from another console, then try to run a query from the killed console.
Something is killing your connection between statements. It's likely an error in your procedure. Check your mysql error log.
I ran into this error when running the following Python (2.7) code:
def _execute(self, query, params, is_destructive):
try:
cursor = self.con.cursor()
cursor.execute(query, params)
if is_destructive:
self.con.commit()
return cursor.fetchall()
finally:
cursor.close()
The problem stemmed from the self.con.commit() call. We were defensive, and coded to commit destructive (write) operations before closing the cursor. MySql did not like this. Removing the commit() called solved the problem and the script was still executed via cursor.fetchall()
Since I have googled this few times now, landing on the same SO question (here), and none of the answers are helpful, here is protip for myself in the future:
this error is a source of unfetched response from MySQL. That is, we send request, and got something back but didn't care to read it.
req = "UPDATE data SET h1='qwertyuioasdasdasda' WHERE id=1;commit;"
In my case, after removing commit I was able to see this error:
Warning: (1265L, "Data truncated for column 'h1' at row 1")
It was raised because I was trying to write h1 which was too long for data type. However, commit prevents me from getting this information back.
I was getting this error when i used a query to select one column data.. yet did not receive the error when I was selecting multiple columns.
THIS ONE DID NOT WORK
mycursor.execute("SELECT PropID FROM for_sale WHERE redfin_rent_est is NULL")
THIS ONE DID WORK which did not create the commands out of sync error.
mycursor.execute("SELECT PropID, web_url FROM for_sale WHERE redfin_rent_est is NULL")
So apparently its caused my errors in syntax also? I'm still unsure on the cause.
As #Mark Gerolimatos
mentioned in this thread`s comments, you cannot execute strings containing multiple sql commands, or even commands followed by a comment.
This will NOT work:
cursor.execute('SELECT 1; SELECT 2;')
cursor.execute('SELECT 1; -- myComment')
This is fine:
cursor.execute('SELECT 1;')
I can't figure out what's wrong with the following code,
The syntax IS ok (checked with SQL Management Studio), i have access as i should so that works too.. but for some reason as soon as i try to create a table via PyODBC then it stops working.
import pyodbc
def SQL(QUERY, target = '...', DB = '...'):
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + target + DB+';UID=user;PWD=pass')
cursor = cnxn.cursor()
cursor.execute(QUERY)
cpn = []
for row in cursor:
cpn.append(row)
return cpn
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
It fails with:
Traceback (most recent call last):
File "test_sql.py", line 25, in <module>
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
File "test_sql.py", line 20, in SQL
for row in cursor:
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Anyone have any idea to why this is?
I got a "SQL Server" driver installed (it's default), running Windows 7 against a Windows 2008 SQL Server environment (Not a express database).
Just in case some lonely net nomad comes across this issue, the solution by Torxed didn't work for me. But the following worked for me.
I was calling an SP which inserts some values into a table and then returns some data back. Just add the following to the SP :
SET NOCOUNT ON
It'll work just fine :)
The Python code :
query = "exec dbo.get_process_id " + str(provider_id) + ", 0"
cursor.execute(query)
row = cursor.fetchone()
process_id = row[0]
The SP :
USE [DBNAME]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[GET_PROCESS_ID](
#PROVIDER_ID INT,
#PROCESS_ID INT OUTPUT
)
AS
BEGIN
SET NOCOUNT ON
INSERT INTO processes(provider_id) values(#PROVIDER_ID)
SET #PROCESS_ID= SCOPE_IDENTITY()
SELECT #PROCESS_ID AS PROCESS_ID
END
Using the "SET NOCOUNT ON" value at the top of the script will not always be sufficient to solve the problem.
In my case, it was also necessary to remove this line:
Use DatabaseName;
Database was SQL Server 2012,
Python 3.7,
SQL Alchemy 1.3.8
Hope this helps somebody.
I got this because I was reusing a cursor that I was looping over:
rows = cursor.execute(...)
for row in rows:
# run query that returns nothing
cursor.execute(...)
# next iteration of this loop will throw 'Previous SQL' error when it tries to fetch next row because we re-used the cursor with a query that returned nothing
Use 2 different cursors instead
rows = cursor1.execute(...)
for row in rows:
cursor2.execute(...)
or get all results of the first cursor before using it again:
Use 2 different cursors instead
rows = cursor.execute(...)
for row in list(rows):
cursor.execute(...)
As others covered, SET NOCOUNT ON will take care of extra resultsets inside a stored procedure, however other things can also cause extra output that NOCOUNT will not prevent (and pyodbc will see as a resultset) such as forgetting to remove a print statement after debugging your stored procedure.
As Travis and others have mentioned, other things can also cause extra output that SET NOCOUNT ON will not prevent.
I had SET NOCOUNT ON at the start of my procedure but was receiving warning messages in my results set.
I set ansi warnings off at the beginning of my script in order to remove the error messages.
SET ANSI_WARNINGS OFF
Hopefully this helps someone.
If your stored procedure calls RAISERROR, pyodbc may create a set for that message.
CREATE PROCEDURE some_sp
AS
BEGIN
RAISERROR ('Some error!', 1, 1) WITH NOWAIT
RETURN 777
END
In python, you need to skip the first sets until you find one containing some results (see https://github.com/mkleehammer/pyodbc/issues/673#issuecomment-631206107 for details).
sql = """
SET NOCOUNT ON;
SET ANSI_WARNINGS OFF;
DECLARE #ret int;
EXEC #ret = some_sp;
SELECT #ret as ret;
"""
cursor = con.cursor()
cursor.execute(sql)
rows = None
#this section will only return the last result from the query
while cursor.nextset():
try:
rows = cursor.fetchall()
except Exception as e:
print("Skipping non rs message: {}".format(e))
continue
row = rows[0]
print(row[0]) # 777.
I think the root cause of the issue described above might be related with the fact that you receive the same error message when you execute for example a DELETE query which will not return a result. So if you run
result = cursor.fetchall()
you get this error, because a DELETE operation by definition does not return anything. Try to catch the exception as recommended here: How to check if a result set is empty?
In case your SQL is not Stored Proc.
usage of 'xyz != NULL' in query, will give the same error i.e. "pyodbc.ProgrammingError: No results. Previous SQL was not a query."
Use 'is not null' instead.
First off:
if you're running a Windows SQL Server 2008, use the "Native Client" that is included with the installation of the SQL software (it gets installed with the database and Toolkits so you need to install the SQL Management applicaton from Microsoft)
Secondly:
Use "Trusted_Connection=yes" in your SQL connection statement:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 10.0};SERVER=ServerAddress;DATABASE=my_db;Trusted_Connection=yes')
This should do the trick!
I have solved this problem by splitting the use database and sql query into two execute statements.