I'm trying to generate & execute SQL statements via pyodbc. I expect multiple SQL statements, all of which start with the same SELECT & FROM but have a different value in the WHERE. The value in my WHERE clause is derived from looping through a table - each distinct value the SQL script finds in the table, I need Python to generate another SQL statement with this value as the WHERE clause.
I'm almost there with this, I'm just struggling to get pyodbc to put my query strings in formats that SQL likes. My code so far:
import pyodbc
cn = pyodbc.connect(connection info)
cursor = cn.cursor()
result = cursor.execute('SELECT distinct searchterm_name FROM table1')
for row in result:
sql = str("SELECT * from table2 WHERE table1.searchterm_name = {c}".format(c=row)),
#print sql
This code generates an output like this, where "name here" is based on the value found in table1.
('SELECT * from ifb_person WHERE searchterm_name = (u\'name here\', )',)
I just need to remove all the crap surrounding the query & where clause so it looks like this. Then I can pass it into another cursor.execute()
SELECT * from ifb_person WHERE searchterm_name = 'name here'
EDIT
for row in result:
cursor.execute("insert into test (searchterm_name) SELECT searchterm_name FROM ifb_person WHERE searchterm_name = ?",
(row[0],))
This query fails with the error pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Basically what I am trying to do is get Python to generate a fresh SQL statement for every result it finds in table1. The second query is running searches against the table ifb_person and inserting the results to a table "test". I want to run separate SQL statements for every result found in table1
pyodbc allows us to iterate over a Cursor object to return the rows, during which time the Cursor object is still "in use", so we cannot use the same Cursor object to perform other operations. For example, this code will fail:
crsr = cnxn.cursor()
result = crsr.execute("SELECT ...") # result is just a reference to the crsr object
for row in result:
# we are actually iterating over the crsr object
crsr.execute("INSERT ...") # this clobbers the previous crsr object ...
# ... so the next iteration of the for loop fails with " Previous SQL was not a query."
We can work around that by using fetchall() to retrieve all the rows into result ...
result = crsr.execute("SELECT ...").fetchall()
# result is now a list of pyodbc.Row objects and the crsr object is no longer "in use"
... or use a different Cursor object in the loop
crsr_select = cnxn.cursor()
crsr_insert = cnxn.cursor()
crsr_select.execute("SELECT ...")
for row in crsr_select:
crsr_insert.execute("INSERT ...")
Related
I have a MS access DB and want to work with it from Python. The aim is to have a table, "units", which includes everything and in order to achieve that I would like to insert information to a table "units_temp" and then join these to tables.
The code is not complete yet but at the moment I am struggling with populating a random ID (purpose is only to not be forced to change ID in the code manually every time I want to try the code before every functionality is in place).
import pyodbc
from random import randint
random_id = randint(0,10000)
conn = pyodbc.connect(r'Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\Users\aaa.bbb\Documents\Python Scripts\DB\Test_db.accdb;')
cursor = conn.cursor()
mySql_insert_query='''INSERT INTO Units (client_id,client_first_name,client_last_name,units_ordered,product_price_per_unit,product_name) VALUES (%s,%s,%s,%s,%s,%s)'''
cursor.execute('''
INSERT INTO Units (client_id,client_first_name,client_last_name,units_ordered,product_price_per_unit,product_name)
VALUES ('124','aa','bb','2','500','phones')
''')
recordTuple = (random_id,'aa','bb','99','900','random')
cursor.execute(mySql_insert_query,recordTuple)
JoinQuery = "SELECT Units.client_id from Units INNER JOIN Units_temp on (Units.client_id=Units_temp.Client_id)"
cursor.execute(JoinQuery)
conn.commit()
I get the following error
ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in query expression '%s'. (-3100) (SQLPrepare)")
pyodbc uses ? as the parameter placeholder, not %s, so your query string should be
mySql_insert_query='''INSERT INTO Units (client_id,client_first_name, … ) VALUES (?,?, … )'''
then you execute it with
cursor.execute(mySql_insert_query,recordTuple)
as before.
It's the line
mySql_insert_query='''INSERT INTO Units (client_id,client_first_name,client_last_name,units_ordered,product_price_per_unit,product_name) VALUES (%s,%s,%s,%s,%s,%s)'''
You have no values for the %s parameter. Replace these with values as you do in the next line.
Also, I seriously doubt, that these fields should be text and not numbers:
units_ordered,product_price_per_unit
I'm trying to delete all the entries from a table but are not able to do it.
Does not matter if it is TRUNCATE, or DELETE keyword. The same error occurs
import pyodbc
conn = pyodbc.connect(
r'Driver={SQL Server};'
r'Server=' + ip + '\SQLEXPRESS;'
r'Database=...;'
r'UID=...;'
r'PWD=...;', timeout=5)
cursor = conn.cursor()
data = cursor.execute("TRUNCATE TABLE table_name")
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Setting autocommit to True does not work. Parametrizing it also does not work. The connection is right because SELECT clause works well and returns the right value. With truncating and deleting it does not work at all. The DDBB is still intact.
When excecuting from the pycharm's Python Console i get the folowwing error whenever i try to access the data object (f.e. print(data.fetchval()):
Traceback (most recent call last):
File "", line 1, in
pyodbc.ProgrammingError: No results. Previous SQL was not a query.
I've read before i might have to do with how the DDBB table is indexed and its private key, but i'm not able to explain it.
I was hoping on getting the number of rows affected.
When we execute a single SQL statement via Cursor.execute, the server can return one of three things:
zero or more rows of data in a result set (for a SELECT statement), or
an integer row count (for DML statements like UPDATE, DELETE, etc.), or
an error.
We retrieve information from a result set via the pyodbc methods .fetchall(), .fetchone(), .fetchval(), etc.. We retrieve row counts using the cursor's rowcount attribute.
crsr = cnxn.cursor()
crsr.execute("DROP TABLE IF EXISTS so64124053")
crsr.execute("CREATE TABLE so64124053 (id int primary key, txt varchar(10))")
crsr.execute("INSERT INTO so64124053 (id, txt) VALUES (1, 'foo')")
print(crsr.rowcount) # 1
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 1
crsr.execute("INSERT INTO so64124053 (id, txt) VALUES (2, 'bar')")
print(crsr.rowcount) # 1
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 2
Note that TRUNCATE is a special case because it doesn't bother counting the rows it removes from the table; it just returns a row count of -1 …
crsr.execute("TRUNCATE TABLE so64124053")
print(crsr.rowcount) # -1
… however the rows are indeed removed
print(crsr.execute("SELECT COUNT(*) AS n FROM so64124053").fetchval()) # 0
I have a list comprised of several queries which are executed by a for loop. I would like to prompt the user to enter the origin (ilink) that will be utilized by the fourth query in the list.
The script runs fine when the origin is manually defined within the query. I have tried the following syntax which have all failed:
cursor.execute(lines, ilink)
cursor.execute(lines, [ilink])
cursor.execute(lines, (ilink))
I have also run the script with each query defined in its own cursor.execute(query) which accepts the argument, but does not pass any results due to multiple cursors.
import MySQLdb
ilink = raw_input("Choose and ilink to query (include 199N):" )
db = MySQLdb.connect(host="host",user="user",passwd="pass")
queries = [
"""USE monthly_audit;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_order_ids;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_internalselect;""",
"""CREATE TEMPORARY TABLE monthly_audit.tmp_order_ids AS
(SELECT DISTINCT order_id AS orders
FROM ng_tradeserver_db_history.fix_execution_reports_201906
WHERE FROM_UNIXTIME(TIMESTAMP/1000000) >= '2019-06-19 16:59:59'
AND FROM_UNIXTIME(TIMESTAMP/1000000) <= '2019-06-20 23:59:59'
AND TargetCompID = %s);""",]
cursor = db.cursor()
for lines in queries:
lines.split(",")
cursor.execute(lines, [ilink])
results = cursor.fetchall()
**This is only the relevant snippet of sql, total query is over 500 lines*
I expect the script to run the set of queries and return the results of said query to be stored in a csv. I am currently getting the following error when executing:
_mysql_exceptions.ProgrammingError: not all arguments converted during string formatting
I'm not sure if I understand your questions correct, but you can try using fstrings. I believe the quotes cause the problems during the string formatting.
Example:
query = f'''select ID, lat, lon from tbl order by st_distance(tbl.geom,st_setsrid(st_makepoint({lon},{lat}), 4326)) asc limit 1;'''
cursor.execute(query)
In this query the {lon}, {lat} are variables. Have a look at the docs for f strings https://docs.python.org/3/whatsnew/3.6.html
I would like to get names from one db and initiate a new table with it. I want to add more analysis, but this is my starting point where I'm already struggling and I have no idea where I made the mistake.
mydb = db_login()
# get team
team = pd.read_sql('SELECT * FROM team', con=mydb)
names = team.name.to_list()
this will output something like ['name1', 'name2' ...]
mycursor = mydb.cursor()
mycursor.execute("DROP TABLE IF EXISTS team_analyse")
mycursor.execute("CREATE TABLE team_analyse (name VARCHAR(50))") #todo add all needed columns
sqlFormula = "INSERT INTO team_analyse (name) VALUES (%s)" #todo initial team commit
mycursor.executemany(sqlFormula, names)
mydb.commit()
In the end I get the following error:
mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
You should be using a single INSERT INTO ... SELECT here:
INSERT INTO team_analyse (name)
SELECT name
FROM team
Your updated Python script:
mycursor = mydb.cursor()
sql = """INSERT INTO team_analyse (name)
SELECT name
FROM team"""
mycursor.execute(sql)
The error message you are seeing is telling you that you passed a list as the parameters to bind to the statement, but not all parameters were used. Actually, the single insert statement you were trying to run only has a single parameter. But in any case, it is unnecessary to bring the result set from the team table into Python's memory. Instead, use my suggestion and let MySQL do the heavy lifting.
I usually use R to do SQL queries by using ODBC to link to a SQL database. The code generally looks like this:
library(RODBC)
ch<-odbcConnect('B1P HANA',uid='****',pwd='****')
myOffice <- c(0)
office_clause = ""
if (myOffice != 0) {
office_clause = paste(
'AND "_all"."/BIC/ZSALE_OFF" IN (',paste(myOffice, collapse=", "),')'
)
}
a <- sqlQuery(ch,paste(' SELECT "_all"."CALDAY" AS "ReturnDate FROM "SAPB1P"."/BIC/AZ_RT_A212" "_all"
WHERE "_all"."CALDAY"=20180101
',office_clause,'
GROUP BY "_all"."CALDAY
'))
The workflow is:
odbcConnect is to link R and SQL using ODBC.
myOffice is an array for achieving data from R. Those data will be used as filter conditions in WHERE clause in SQL.
a stores the query result from SQL database.
So, how to do all of these in Python, i.e., do SQL queries in Python by using ODBC to link SQL database and Python? I am new to Python. All I know is like:
import pyodbc
conn = pyodbc.connect(r'DSN=B1P HANA;UID=****;PWD=****')
Then I do not know how to continue. And I cannot find an overall example online. Could anyone help by providing a comprehensive example? From link SQL database in Python unitl retrieving the result?
Execute SQL from python
Instantiate a Cursor and use the execute method of the Cursor class to execute any SQL statement.
cursor = cnxn.cursor()
Select
You can use fetchall, fetchone, and fetchmany to retrieve rows returned from SELECT statements:
import pyodbc
cursor = cnxn.cursor()
cnxn = pyodbc.connect('DSN=myDSN;UID=***;PWD=***')
cursor.execute("SELECT Col1, Col2 FROM MyTable WHERE Col1= 'SomeValue'")
rows = cursor.fetchall()
for row in rows:
print(row.Col1, row.Col2 )
You can provide parameterized queries in a sequence or in the argument list:
cursor.execute("SELECT Col1, Col2, Col3, ... FROM MyTable WHERE Col1 = ?", 'SomeValue',1)
Insert
INSERT commands also use the execute method; however, you must subsequently call the commit method after an insert or you will lose your changes:
cursor.execute("INSERT INTO MyTable (Col1) VALUES ('SomeValue')")
cnxn.commit()
Update and Delete
As with an insert, you must also call commit after calling execute for an update or delete:
cursor.execute("UPDATE MyTable SET Col1= 'SomeValue'")
cnxn.commit()
Metadata Discovery
You can use the getinfo method to retrieve data such as information about the data source and the capabilities of the driver. The getinfo method passes through input to the ODBC SQLGetInfo method.
cnxn.getinfo(pyodbc.SQL_DATA_SOURCE_NAME)