I'm pretty new to python and postgresql in general but I'm having problems transferring data from one server into another. Currently I have code to run where I pull data and set that to a variable IT, when I try inserting IT into another Postgresql server, I run into errors.
First I used:
cur = con.cursor()
#Connect cursor to local server
IT = DataPull()
#Pulls the data from the remote Postgresql server and set it equal to IT
command2 = (
"""
INSERT INTO gr_data.it (column_name1,column_name2,column_name3,column_name4,column_name5,column_name6,column_name7) VALUES(?,?,?,?,?,?,?)
""")
cur.execute(command2, IT)
But I end up getting the error:
psycopg2.ProgrammingError: syntax error at or near ","
LINE 2:...column_name4,column_name5,column_name6,column_name7 VALUES<?,?,?,?,?....
^
So I figured it had to do with the question marks, I googled around and found that maybe they should be changed to "%s". Then I received this error:
TypeError: not all arguments converted during string formatting
Any help?
Here's an example of how you should perform the insert using variables:
cur.execute("INSERT INTO test (num, data) VALUES (%s, %s)", (100, "abc'def"))
It is taken from official docs you can find here
Related
I can't figure out what's wrong with the following code,
The syntax IS ok (checked with SQL Management Studio), i have access as i should so that works too.. but for some reason as soon as i try to create a table via PyODBC then it stops working.
import pyodbc
def SQL(QUERY, target = '...', DB = '...'):
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + target + DB+';UID=user;PWD=pass')
cursor = cnxn.cursor()
cursor.execute(QUERY)
cpn = []
for row in cursor:
cpn.append(row)
return cpn
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
It fails with:
Traceback (most recent call last):
File "test_sql.py", line 25, in <module>
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
File "test_sql.py", line 20, in SQL
for row in cursor:
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Anyone have any idea to why this is?
I got a "SQL Server" driver installed (it's default), running Windows 7 against a Windows 2008 SQL Server environment (Not a express database).
Just in case some lonely net nomad comes across this issue, the solution by Torxed didn't work for me. But the following worked for me.
I was calling an SP which inserts some values into a table and then returns some data back. Just add the following to the SP :
SET NOCOUNT ON
It'll work just fine :)
The Python code :
query = "exec dbo.get_process_id " + str(provider_id) + ", 0"
cursor.execute(query)
row = cursor.fetchone()
process_id = row[0]
The SP :
USE [DBNAME]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[GET_PROCESS_ID](
#PROVIDER_ID INT,
#PROCESS_ID INT OUTPUT
)
AS
BEGIN
SET NOCOUNT ON
INSERT INTO processes(provider_id) values(#PROVIDER_ID)
SET #PROCESS_ID= SCOPE_IDENTITY()
SELECT #PROCESS_ID AS PROCESS_ID
END
Using the "SET NOCOUNT ON" value at the top of the script will not always be sufficient to solve the problem.
In my case, it was also necessary to remove this line:
Use DatabaseName;
Database was SQL Server 2012,
Python 3.7,
SQL Alchemy 1.3.8
Hope this helps somebody.
I got this because I was reusing a cursor that I was looping over:
rows = cursor.execute(...)
for row in rows:
# run query that returns nothing
cursor.execute(...)
# next iteration of this loop will throw 'Previous SQL' error when it tries to fetch next row because we re-used the cursor with a query that returned nothing
Use 2 different cursors instead
rows = cursor1.execute(...)
for row in rows:
cursor2.execute(...)
or get all results of the first cursor before using it again:
Use 2 different cursors instead
rows = cursor.execute(...)
for row in list(rows):
cursor.execute(...)
As others covered, SET NOCOUNT ON will take care of extra resultsets inside a stored procedure, however other things can also cause extra output that NOCOUNT will not prevent (and pyodbc will see as a resultset) such as forgetting to remove a print statement after debugging your stored procedure.
As Travis and others have mentioned, other things can also cause extra output that SET NOCOUNT ON will not prevent.
I had SET NOCOUNT ON at the start of my procedure but was receiving warning messages in my results set.
I set ansi warnings off at the beginning of my script in order to remove the error messages.
SET ANSI_WARNINGS OFF
Hopefully this helps someone.
If your stored procedure calls RAISERROR, pyodbc may create a set for that message.
CREATE PROCEDURE some_sp
AS
BEGIN
RAISERROR ('Some error!', 1, 1) WITH NOWAIT
RETURN 777
END
In python, you need to skip the first sets until you find one containing some results (see https://github.com/mkleehammer/pyodbc/issues/673#issuecomment-631206107 for details).
sql = """
SET NOCOUNT ON;
SET ANSI_WARNINGS OFF;
DECLARE #ret int;
EXEC #ret = some_sp;
SELECT #ret as ret;
"""
cursor = con.cursor()
cursor.execute(sql)
rows = None
#this section will only return the last result from the query
while cursor.nextset():
try:
rows = cursor.fetchall()
except Exception as e:
print("Skipping non rs message: {}".format(e))
continue
row = rows[0]
print(row[0]) # 777.
I think the root cause of the issue described above might be related with the fact that you receive the same error message when you execute for example a DELETE query which will not return a result. So if you run
result = cursor.fetchall()
you get this error, because a DELETE operation by definition does not return anything. Try to catch the exception as recommended here: How to check if a result set is empty?
In case your SQL is not Stored Proc.
usage of 'xyz != NULL' in query, will give the same error i.e. "pyodbc.ProgrammingError: No results. Previous SQL was not a query."
Use 'is not null' instead.
First off:
if you're running a Windows SQL Server 2008, use the "Native Client" that is included with the installation of the SQL software (it gets installed with the database and Toolkits so you need to install the SQL Management applicaton from Microsoft)
Secondly:
Use "Trusted_Connection=yes" in your SQL connection statement:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 10.0};SERVER=ServerAddress;DATABASE=my_db;Trusted_Connection=yes')
This should do the trick!
I have solved this problem by splitting the use database and sql query into two execute statements.
I am trying to update SQL Server table through Python. But unfortunately it does not update.
I get message successful but no data was updated.
If I call the same SQL script from within SQL Server, it updates correctly.
Let me show you my script: this is my Python code:
PredString = '99'
conn = pymssql.connect(server="MyServer", database="MyDB", port="1433", user="****", password="******")
dfUpdate = pd.read_sql("EXEC UpdatePredictions '" + PredString + "'", conn)
conn.close()
print(dfUpdate)
This is the SQL Server stored procedure:
alter procedure UpdatePredictions
(#PredString varchar(max))
as
begin
update MyTable
set PredMths = #PredString
select 'Updated.'
end
When I run Python code I get "Updated" but actually no record was updated
But when I call from SQL Server:
EXEC UpdatePredictions '99'
I get message "Updated" and records are actually updated
What am I doing wrong here? How can I get Python to update the table?
Thanks to the guys who commented the answer.
As no one have made it as an answer, I will so I can mark it, so other people can find the answer easily in the future.
the problem was that Python connection wasn't committing update statement.
therefore I have to add this line after sending the update
conn.commit()
I am currently trying to use pyodbc to select data from a table within Database A and insert it into a table within Database B. I was able to establish connections with both databases, so I know there is no error there. Additionally, my first cursor.execute command (line #9) works as I was able to print all the data.
The issue I am running into is when I try and insert the data from the first cursor.execute command into Database B. There are a few questions on SO regarding this same error, however I have checked to ensure I am not committing on of those errors. All the data types are accepted within SQL Server, I have the correct number of parameters and parameter markers, and I have ensured that the columns within my Python code match both the input and output tables. I am completely stuck and would greatly appreciate any help.
The specific error I am getting is:
('HYC00', '[HYC00] [Microsoft][ODBC SQL Server Driver]Optional feature
not implemented (0) (SQLBindParameter)')
Please see my code below:
import pyodbc
import time
cnxn1 = pyodbc.connect(r"DRIVER={SQL Server Native Client 11.0};SERVER='Server';" + \
"DATABASE='DatabaseA';Trusted_Connection=Yes")
cursor1 = cnxn1.cursor()
cnxn2 = pyodbc.connect(r"DRIVER={SQL Server};SERVER='Server'," + \
"user='Username', password='Password', database='DatabaseB'")
cursor2 = cnxn2.cursor()
SQL = cursor1.execute("select * from table.DatabaseA")
SQL2 = """insert into table.DatabaseB([col1], [col2], [col3], [col4],[col5], [col6], [col7],
[col8], [col9], [col10], [col11], [col12], [col13], [col14],
[col15], [col16],[col17], [col18], [col19], [col20], [col21],
[col22], [col23], [col24], [col25], [col26], [col27], [col28],
[col29], [col30], [col31])
values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"""
for row in cursor1.fetchall():
cursor2.execute(SQL2,row)
In regard to the last two lines of code, I have also tried the following with no success:
for row in SQL:
cursor2.execute(SQL2,row)
My set-up:
MySQL server.
host running a python script.
(1) and (2) are different machines on the network.
The python script generates data which must be stored in a MySQL-database.
I use this (example-)code to achieve that:
def function sqldata(date,result):
con = mdb.connect('sql.lan', 'demouser', 'demo', 'demo')
with con:
cur = con.cursor()
cur.execute('INSERT INTO tabel(titel, nummer) VALUES( %s, %s)',(date, result))
The scipt generates one data-point approx. every minute. So this means that a new connection is opened and closed every minute. I'm wondering if it would be a better idea to open the connection at the start of the script and only close it when the script terminates. Effectively leaving the connection open indefinately.
This then obviously begs the question how to handle/recover when the SQL-server "leaves" the network (e.g. due to a reboot) for a while.
While typing my question this question appeared in the "Similar Questions" section. It is, however, from 2008 and possibly outdated and the 4 answers it received seem to contradict with each other.
What are the current insights in this matter?
Well the referred answer is right in it's point, but maybe not answering all your questions. I can not provide a full running python script for you here, but let me explain how i would go along with it:
Rule 1: Generally most mysql functions return values, that you should always check so that you can react on unwanted behavior.
Rule 2: Open a connection at the beginning of your script and use this one and only connection throughout your script.
Obviously you could check if there is an existing connection in your sqldata function, and if not then you could open a new one to the global con object.
if not con:
con = mdb.connect('sql.lan', 'demouser', 'demo', 'demo')
And if there is a connection already, you could check it's "up status" by performing a simple query with fixed expected result that you can check to see if the sql server is running.
if con:
cur = con.cursor()
returned = cur.execute('SELECT COUNT(*) FROM tabel')
if returned.with_rows:
....
Basically you could avoid this, because if you don't get a cursor back, and you check that first before using it, then you already know if the server is alive or not.
So CHECK, CHECK and CHECK. You should check everything you get back from a function to have a good error handling. Just using a connection or using a cursor without checking it first, can lead you talking to a NIL object and crashing your script.
And the last BIG HINT i can give you is to use multiple row inserts. You can actually insert hundreds of rows, if you just add the values comma seperated to your insert string:
# consider result would be filled like this
result = '("First Song",1),("Second Song",2),("Third Song",3)'
# then this will insert 3 rows with one call
returned = cur.execute('INSERT INTO tabel (titel, nummer) VALUES %s',(date, result), multi=True)
# since literally it will execute
returned = cur.execute('INSERT INTO tabel (titel, nummer) VALUES ("First Song",1),("Second Song",2),("Third Song",3)', multi=True)
# and now you can check returned for any error
if returned:
....
I can't figure out what's wrong with the following code,
The syntax IS ok (checked with SQL Management Studio), i have access as i should so that works too.. but for some reason as soon as i try to create a table via PyODBC then it stops working.
import pyodbc
def SQL(QUERY, target = '...', DB = '...'):
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=' + target + DB+';UID=user;PWD=pass')
cursor = cnxn.cursor()
cursor.execute(QUERY)
cpn = []
for row in cursor:
cpn.append(row)
return cpn
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
It fails with:
Traceback (most recent call last):
File "test_sql.py", line 25, in <module>
print SQL("CREATE TABLE dbo.Approvals (ID SMALLINT NOT NULL IDENTITY PRIMARY KEY, HostName char(120));")
File "test_sql.py", line 20, in SQL
for row in cursor:
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Anyone have any idea to why this is?
I got a "SQL Server" driver installed (it's default), running Windows 7 against a Windows 2008 SQL Server environment (Not a express database).
Just in case some lonely net nomad comes across this issue, the solution by Torxed didn't work for me. But the following worked for me.
I was calling an SP which inserts some values into a table and then returns some data back. Just add the following to the SP :
SET NOCOUNT ON
It'll work just fine :)
The Python code :
query = "exec dbo.get_process_id " + str(provider_id) + ", 0"
cursor.execute(query)
row = cursor.fetchone()
process_id = row[0]
The SP :
USE [DBNAME]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER procedure [dbo].[GET_PROCESS_ID](
#PROVIDER_ID INT,
#PROCESS_ID INT OUTPUT
)
AS
BEGIN
SET NOCOUNT ON
INSERT INTO processes(provider_id) values(#PROVIDER_ID)
SET #PROCESS_ID= SCOPE_IDENTITY()
SELECT #PROCESS_ID AS PROCESS_ID
END
Using the "SET NOCOUNT ON" value at the top of the script will not always be sufficient to solve the problem.
In my case, it was also necessary to remove this line:
Use DatabaseName;
Database was SQL Server 2012,
Python 3.7,
SQL Alchemy 1.3.8
Hope this helps somebody.
I got this because I was reusing a cursor that I was looping over:
rows = cursor.execute(...)
for row in rows:
# run query that returns nothing
cursor.execute(...)
# next iteration of this loop will throw 'Previous SQL' error when it tries to fetch next row because we re-used the cursor with a query that returned nothing
Use 2 different cursors instead
rows = cursor1.execute(...)
for row in rows:
cursor2.execute(...)
or get all results of the first cursor before using it again:
Use 2 different cursors instead
rows = cursor.execute(...)
for row in list(rows):
cursor.execute(...)
As others covered, SET NOCOUNT ON will take care of extra resultsets inside a stored procedure, however other things can also cause extra output that NOCOUNT will not prevent (and pyodbc will see as a resultset) such as forgetting to remove a print statement after debugging your stored procedure.
As Travis and others have mentioned, other things can also cause extra output that SET NOCOUNT ON will not prevent.
I had SET NOCOUNT ON at the start of my procedure but was receiving warning messages in my results set.
I set ansi warnings off at the beginning of my script in order to remove the error messages.
SET ANSI_WARNINGS OFF
Hopefully this helps someone.
If your stored procedure calls RAISERROR, pyodbc may create a set for that message.
CREATE PROCEDURE some_sp
AS
BEGIN
RAISERROR ('Some error!', 1, 1) WITH NOWAIT
RETURN 777
END
In python, you need to skip the first sets until you find one containing some results (see https://github.com/mkleehammer/pyodbc/issues/673#issuecomment-631206107 for details).
sql = """
SET NOCOUNT ON;
SET ANSI_WARNINGS OFF;
DECLARE #ret int;
EXEC #ret = some_sp;
SELECT #ret as ret;
"""
cursor = con.cursor()
cursor.execute(sql)
rows = None
#this section will only return the last result from the query
while cursor.nextset():
try:
rows = cursor.fetchall()
except Exception as e:
print("Skipping non rs message: {}".format(e))
continue
row = rows[0]
print(row[0]) # 777.
I think the root cause of the issue described above might be related with the fact that you receive the same error message when you execute for example a DELETE query which will not return a result. So if you run
result = cursor.fetchall()
you get this error, because a DELETE operation by definition does not return anything. Try to catch the exception as recommended here: How to check if a result set is empty?
In case your SQL is not Stored Proc.
usage of 'xyz != NULL' in query, will give the same error i.e. "pyodbc.ProgrammingError: No results. Previous SQL was not a query."
Use 'is not null' instead.
First off:
if you're running a Windows SQL Server 2008, use the "Native Client" that is included with the installation of the SQL software (it gets installed with the database and Toolkits so you need to install the SQL Management applicaton from Microsoft)
Secondly:
Use "Trusted_Connection=yes" in your SQL connection statement:
cnxn = pyodbc.connect('DRIVER={SQL Server Native Client 10.0};SERVER=ServerAddress;DATABASE=my_db;Trusted_Connection=yes')
This should do the trick!
I have solved this problem by splitting the use database and sql query into two execute statements.