I have a very large table in mysql that I'm accessing through a python script, and I'd like the output of my select query to be stored as a list. Here's what I have:
import MySQLdb
db = MySQLdb.connect(host='', user='', passwd='', db='')
cursor = db.cursor()
sqlselect = "SELECT desig FROM table WHERE num=0;"
desigs = cursor.execute(sqlselect)
print desigs
But this just gives me the number of rows in set which is nearly 250,000. Instead, I'd like it to print a list of each 'desig'. How can I do this?
cursor.execute() will return the number of rows modified or retrieved.
Once you've executed the query, run cursor.fetchall(), which will return you an array containing each row in your query results. That is, you get an array of arrays.
Edit: list! list of lists!! argh
Related
I have a list comprised of several queries which are executed by a for loop. I would like to prompt the user to enter the origin (ilink) that will be utilized by the fourth query in the list.
The script runs fine when the origin is manually defined within the query. I have tried the following syntax which have all failed:
cursor.execute(lines, ilink)
cursor.execute(lines, [ilink])
cursor.execute(lines, (ilink))
I have also run the script with each query defined in its own cursor.execute(query) which accepts the argument, but does not pass any results due to multiple cursors.
import MySQLdb
ilink = raw_input("Choose and ilink to query (include 199N):" )
db = MySQLdb.connect(host="host",user="user",passwd="pass")
queries = [
"""USE monthly_audit;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_order_ids;""",
"""DROP TEMPORARY TABLE IF EXISTS monthly_audit.tmp_internalselect;""",
"""CREATE TEMPORARY TABLE monthly_audit.tmp_order_ids AS
(SELECT DISTINCT order_id AS orders
FROM ng_tradeserver_db_history.fix_execution_reports_201906
WHERE FROM_UNIXTIME(TIMESTAMP/1000000) >= '2019-06-19 16:59:59'
AND FROM_UNIXTIME(TIMESTAMP/1000000) <= '2019-06-20 23:59:59'
AND TargetCompID = %s);""",]
cursor = db.cursor()
for lines in queries:
lines.split(",")
cursor.execute(lines, [ilink])
results = cursor.fetchall()
**This is only the relevant snippet of sql, total query is over 500 lines*
I expect the script to run the set of queries and return the results of said query to be stored in a csv. I am currently getting the following error when executing:
_mysql_exceptions.ProgrammingError: not all arguments converted during string formatting
I'm not sure if I understand your questions correct, but you can try using fstrings. I believe the quotes cause the problems during the string formatting.
Example:
query = f'''select ID, lat, lon from tbl order by st_distance(tbl.geom,st_setsrid(st_makepoint({lon},{lat}), 4326)) asc limit 1;'''
cursor.execute(query)
In this query the {lon}, {lat} are variables. Have a look at the docs for f strings https://docs.python.org/3/whatsnew/3.6.html
I'm new to SQL and psycopg2. I'm playing around a bit and try to find our how to display the results of a query. I have a small script where I make a connection to the database and create a cursor to run the query.
from psycopg2 import connect
conn = connect(host="localhost", user="postgres", dbname="portfolio",
password="empty")
cur = conn.cursor()
cur.execute("SELECT * FROM portfolio")
for record in cur:
print("ISIN: {}, Naam: {}".format(record[0], record[1]))
print(cur.fetchmany(3))
cur.close()
conn.close()
If I run this code, the first print is fine, but the second print-statement returns [].
If I run only one of the two print-statements, I get a result every time.
Can someone explain me why?
The cursor loops over the results and returns one at a time. When it has returned all of them, it can't return any more. This is precisely like when you loop over the lines in a file (there are no more lines once you reach the end of the file) or even looping over a list (there are no more entries in the list after the last one).
If you want to manipulate the results in Python, you should probably read them into a list, which you can then traverse as many times as you like, or search, sort, etc, or access completely randomly.
cur.execute("SELECT * FROM portfolio")
result = cur.fetchall()
for record in result:
print("ISIN: {}, Naam: {}".format(record[0], record[1]))
print(result[0:3]))
I have an Oracle DB with over 5 Million rows with columns of type varchar and blob. In order to connect to the database and read the records I use python 3.6 with a JDBC driver and the library JayDeBeApi. What I am trying to achieve is to read each row, perform some
operations on the records (use a regex for example) and then store the new record values in a new table. I don't want to load all records in the memory, so what I want to do is to consequently fetch them from the database, store the fetched data, process it and then add it to the other table.
Currently I fetch all the records at once instead for example first 1000, then the next 1000 and so on. This is what I have so far:
statement = "... a select statement..."
connection= dbDriver.connect(jclassname,[driver_url,username,password],jars,)
cursor = connection.cursor()
cursor.execute(statement)
fetched = cursor.fetchall()
for result in fetched:
preprocess(result)
cursor.close()
How could I modify my code to fetch consequently and where to put the second statement which inserts the new values in the other table?
As you said, fetchall() is a bad idea in this case, as it loads all the data into the memory.
In order to avoid that you can iterate over cursor object itself:
cur.execute("SELECT * FROM test")
for row in cur: # iterate over result set row by row
do_stuff_with_row(row)
cur.close()
I'm trying to generate & execute SQL statements via pyodbc. I expect multiple SQL statements, all of which start with the same SELECT & FROM but have a different value in the WHERE. The value in my WHERE clause is derived from looping through a table - each distinct value the SQL script finds in the table, I need Python to generate another SQL statement with this value as the WHERE clause.
I'm almost there with this, I'm just struggling to get pyodbc to put my query strings in formats that SQL likes. My code so far:
import pyodbc
cn = pyodbc.connect(connection info)
cursor = cn.cursor()
result = cursor.execute('SELECT distinct searchterm_name FROM table1')
for row in result:
sql = str("SELECT * from table2 WHERE table1.searchterm_name = {c}".format(c=row)),
#print sql
This code generates an output like this, where "name here" is based on the value found in table1.
('SELECT * from ifb_person WHERE searchterm_name = (u\'name here\', )',)
I just need to remove all the crap surrounding the query & where clause so it looks like this. Then I can pass it into another cursor.execute()
SELECT * from ifb_person WHERE searchterm_name = 'name here'
EDIT
for row in result:
cursor.execute("insert into test (searchterm_name) SELECT searchterm_name FROM ifb_person WHERE searchterm_name = ?",
(row[0],))
This query fails with the error pyodbc.ProgrammingError: No results. Previous SQL was not a query.
Basically what I am trying to do is get Python to generate a fresh SQL statement for every result it finds in table1. The second query is running searches against the table ifb_person and inserting the results to a table "test". I want to run separate SQL statements for every result found in table1
pyodbc allows us to iterate over a Cursor object to return the rows, during which time the Cursor object is still "in use", so we cannot use the same Cursor object to perform other operations. For example, this code will fail:
crsr = cnxn.cursor()
result = crsr.execute("SELECT ...") # result is just a reference to the crsr object
for row in result:
# we are actually iterating over the crsr object
crsr.execute("INSERT ...") # this clobbers the previous crsr object ...
# ... so the next iteration of the for loop fails with " Previous SQL was not a query."
We can work around that by using fetchall() to retrieve all the rows into result ...
result = crsr.execute("SELECT ...").fetchall()
# result is now a list of pyodbc.Row objects and the crsr object is no longer "in use"
... or use a different Cursor object in the loop
crsr_select = cnxn.cursor()
crsr_insert = cnxn.cursor()
crsr_select.execute("SELECT ...")
for row in crsr_select:
crsr_insert.execute("INSERT ...")
I notice that the auto_convert_lobs argument does not work as expected if I am iterating a result using cx_Oracle.Cursor object.
From the documentation I read, cx_Oracle will convert Clobs into String by default : http://docs.sqlalchemy.org/en/latest/dialects/oracle.html#lob-objects
The query I'm iterating is a function that returns a cursor instead of rows :
SELECT returns_a_cursor() FROM dual
-- pseudo code of returns_a_cursor():
FUNCTION returns_a_cursor() RETURN SYS_REFCURSOR
sql := 'SELECT ...';
OPEN cursor FOR sql;
RETURN cursor;
The only reason I'm using a function instead of a direct query is only for readability and organization.
My python code looks like so
engine = create_engine('oracle+cx_oracle://...')
conn = engine.connect()
result = conn.execute('SELECT returns_a_cursor() FROM dual').fetchone()
for row in result[0]:
print row
This 'cursor' method will return my CLOB data types to cx_Oracle.LOB data instead of string.
If I change my code to execute a query without the function, it will be able to convert the CLOB data to string.
engine = create_engine('oracle+cx_oracle://...')
conn = engine.connect()
result = conn.execute('SELECT a_clob_column FROM a_table')
for row in result:
print row
Is there a way that I can keep the auto_convert_lobs setting in my use case ?