I'm working with Python and JayDeBeApi to connect to a Oracle-type database.
In the SELECT's statements I need to get about 10+ thousand of records.
In the first time I done by using the "fetchAll()" method, but this loads my memory and I wouldn't like to this to happen.
I get the cursor by using the code below:
def do_select(sql, db_conn):
resultSet = None
try:
cursor = db_conn.cursor()
cursor.execute(sql)
resultSet = {
"cursor": cursor,
"columns": cursor.description
}
except Exception as error:
print("An error occurred" + str(error))
return resultSet
And instead of using this type of code:
resultSet = self.do_select(sql, self.get_db_conn())
rows = resultSet["cursor"].fetchAll()
for row in rows:
# Do something...
I would like to do something like this:
resultSet = self.do_select(sql, self.get_db_conn())
while resultSet.next():
entire_row_tuple = resultSet.getCurrent() #I don't know if this is possible in python
#Do something with entire_row_tuple...
Is this possible in python? Or, does exists a better way instead of using "fetchAll()" method?
Thank you
Related
I am writing a script in python 3.x using mysqlconnector.
What I am trying to achieve right now is to check if there is a record inside my db which may be a duplicate to the one I am analyzing right now.
I came up with such code:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
isDuplicate = cursor.execute(("SELECT destination FROM {0} WHERE destination = '{1}';")
.format(db_name, data['destination']))
print(cursor.statement)
self.commit()
print(isDuplicate is None)
Though I still get isDuplicate as None object. I tried to check via cursor.statement what statement is being passed to my db: it turned out that while in script I get None obj while passed in db that query works fine.
I also tried SELECT COUNT(1) FROM db_name which also gave me different results.
I am out of ideas: maybe you guys can help me out?
Update:
The solution that works for me was:
q = ("SELECT * FROM {0} WHERE destination = %s AND countryCode = %s AND prefix = %s")
.format(db_name)
cursor.execute(q, (data['destination'], data['country_code'], data['prefix']))
self.cnx.commit()
isDoubled = cursor.fetchone()
So at the end of the day it was all about fetching data from the cursor :)
Maybe the reason of your issue is the way you use execute() method.
Try to make some changes and see what is printed out:
def fill_data(self, db_name, data):
cursor = self.cnx.cursor(buffered=True)
q = 'SELECT count(*) FROM {} WHERE destination = %s'.format(db_name)
duplicate_count = cursor.execute(q, (data['destination'], )).fetchall()
print(duplicate_count)
Why should I provide query parameters this way? (article is on psql, but the core principles are the same as in mysql)
update
If you are still receiving "NoneType" object has no atribute "fetchall", then the error is probably here:
cursor = self.cnx.cursor(buffered=True)
Looks like you are not creating cursor at all. I can take a look at it if you post some code about cnx creation.
I am trying to fetch data from AWS MariaDB:
cursor = self._cnx.cursor()
stmt = ('SELECT * FROM flights')
cursor.execute(stmt)
print(cursor.rowcount)
# prints 2
for z in cursor:
print(z)
# Does not iterate
row = cursor.fetchone()
# row is None
rows = cursor.fetchall()
# throws 'No result set to fetch from.'
I can verify that table contains data using MySQL Workbench. Am I missing some step?
EDIT: re 2 answers:
res = cursor.execute(stmt)
# res is None
EDIT:
I created new Python project with a single file:
import mysql.connector
try:
cnx = mysql.connector.connect(
host='foobar.rds.amazonaws.com',
user='devuser',
password='devpasswd',
database='devdb'
)
cursor = cnx.cursor()
#cursor = cnx.cursor(buffered=True)
cursor.execute('SELECT * FROM flights')
print(cursor.rowcount)
rows = cursor.fetchall()
except Exception as exc:
print(exc)
If I run this code with simple cursor, fetchall raises "No result set to fetch from". If I run with buffered cursor, I can see that _rows property of cursor contains my data, but fetchall() returns empty array.
Your issue is that cursor.execute(stmt) returns an object with results and you're not storing that.
results = cursor.execute(stmt)
print(results.fetchone()) # Prints out and pops first row
For the future googlers with the same Problem I found a workaround which may help in some cases:
I didn't find the source of the problem but a solution which worked for me.
In my case .fetchone() also returned none whatever I did on my local(on my own Computer) Database. I tried the exact same code with the Database on our companies server and somehow it worked. So I copied the complete server Database onto my local Database (by using database dumps) just to get the server settings and afterwards I also could get data from my local SQL-Server with the code which didn't work before.
I am a SQL-newbie but maybe some crazy setting on my local SQL-Server prevented me from fetching data. Maybe some more experienced SQL-user knows this setting and can explain.
I am connecting to mysql database via mysql connector and running a simple query to pull a list of IDs. I need to loop over that list and pass them into some other code. For some reason I am getting a list of tuples. Is this expected behavior? If not, what am I doing wrong?
Here is the snippet of my code:
import mysql.connector
conn = mysql.connector.connect(host='127.0.0.1', database='t', user='r', password='pwd')
cursor = conn.cursor()
query = ( "select id from T where updated < '%s'" % (run_date) )
cursor.execute(query)
for row in cursor:
print (row)
cursor.close()
I am getting the following back (from an INT field in d/b):
(Decimal('991837'),)
(Decimal('991838'),)
(Decimal('991839'),)
(Decimal('991871'),)
(Decimal('991879'),)
(Decimal('991899'),)
(Decimal('992051'),)
(Decimal('992299'),)
(Decimal('992309'),)
if you want to access just the data in the row you need to go into the dictionary
first you must make it true in the cursor
cur = db.cursor( buffered=True , dictionary=True)
then the result will be like this :
{'Decimal' : '991837'}
i'm sure the Decimal is your row name
so when you need to access to the value do this
import mysql.connector
conn = mysql.connector.connect(host='127.0.0.1', database='t', user='r', password='pwd')
cursor = conn.cursor()
query = ( "select id from T where updated < '%s'" % (run_date) )
cursor.execute(query)
for row in cursor:
print (row['Decimal'])
cursor.close()
i hope it works for i was looking for this solution for the past 2 days and no answers
the only way i debugged i opened the debugger and print out all the variables
have fun with Python :)
Yes, this is expected behavior. Using the cursor as an iterable is basically equivalent to looping over it using the fetchone() method. From the documentation for fetchone() (emphasis mine):
This method retrieves the next row of a query result set and returns a
single sequence, or None if no more rows are available. By default,
the returned tuple consists of data returned by the MySQL server,
converted to Python objects. If the cursor is a raw cursor, no such
conversion occurs;
I am retrieving a single row from a single column in my database. The pymssql documentation exclusively uses loops to access the data in a cursor.
conn = pymssql.connect(server, user, password, "tempdb")
cursor = conn.cursor()
cursor.execute('SELECT %s', 'Foo')
#This works but it's ugly
for row in cursor:
print row[0]
break
#The following throws an error
print cursor[0][0]
conn.close()
Is there a way to access the data inside the cursor object without a for loop?
You can use cursor.fetchone()[0] in place of cursor[0][0].
However, if nothing is returned from the query you're going to get an exception. Better to do something "ugly" like this:
row = cursor.fetchone()
if row:
print row[0]
I am trying to use the following additional arguments in the mysql.connector.connect(), however when I run the following code it doesn't seem to have any effect.
import mysql.connector
cnx = mysql.connector.connect(
....
raw='true',
use_unicode='false'
)
cursor = cnx.cursor()
query = ("...")
cursor.execute(query)
result = cursor.fetchall()
print result
[(datetime.date(2015, 4, 5), 1243), ...
where the results has a column of MySQL datetime format values
but even with these arguments the output is still in python datetime.date(YYYY,M,D) format
same issue also when calling col = cursor.column_names, returns unicode instead of strings even though "raw" is set to true.
print cursor.colun_names = (u'string_1', u'string_2')
is there some other configuration needed? i dont want to have to write code to convert these every time its used. thanks!
I think when Python evaluates something like:
if use_unicode:
// do something
Variable use_unicode returns True, because string 'false' evaluated to True. Try to use Boolean values.
import mysql.connector
cnx = mysql.connector.connect(
....
raw=True,
use_unicode=False
)
cursor = cnx.cursor()
query = ("...")
cursor.execute(query)
result = cursor.fetchall()
print result
raw=True means:
A MySQLCursorRaw cursor skips the conversion from MySQL data types to
Python types when fetching rows. A raw cursor is usually used to get
better performance or when you want to do the conversion yourself.
So you get ByteArray as expected. I don't know what exactly you are trying to do, but when I work with MySQL in Python, I prefer to work with dictionary data structure.
cursor = cnx.cursor(dictionary=True)
And then you will get result as dictionary. If you want to view it as string, just do print str(dict)
import mysql.connector
cnx = mysql.connector.connect(user=self.username, password=self.password, host=self.host, database=self.database);
cursor = cnx.cursor(dictionary=True)
query = ("...")
cursor.execute(query)
result = cursor.fetchall()
print str(result)