Here's a shortened version of my script:
import MySQLdb
src_db = MySQLdb.connect(**some_connection)
src_cursor = src_db.cursor()
v = src_cursor.execute('SELECT node_id FROM stats WHERE time_unit >= 1388534400')
v ends up being of long type, which I cannot understand. I expect to have a generator that would return 1-element tuples (I ask only for one column). And it returns a long value, being number of rows returned from db. Why?
When I try to iterate through it:
node_ids = {int(x[0]) for x in v}
I get following error:
TypeError: 'long' object is not iterable
You need to read the python database API specification, PEP-249.
Basically, after you've executed a query with a cursor object, you then query the cursor object for the results.
cursor.execute(my_sql)
for record in cursor.fetchall():
# do stuff
src_cursor.execute(SELECT_QUERY) will return the number of rows matching your query.
To iterate through the result of the query:
for row in src_cursor.fetchall():
To get one row at a time:
row = src_cursor.fetchone()
Related
Is there an elegant way of getting a single result from an SQLite SELECT query when using Python?
for example:
conn = sqlite3.connect('db_path.db')
cursor=conn.cursor()
cursor.execute("SELECT MAX(value) FROM table")
for row in cursor:
for elem in row:
maxVal = elem
is there a way to avoid those nested fors and get the value directly? I've tried
maxVal = cursor[0][0]
without any success.
I think you're looking for Cursor.fetchone() :
cursor.fetchone()[0]
Or you could write a wrapper function that, given SQL, returns a scalar result:
def get_scalar_result(conn, sql):
cursor=conn.cursor()
cursor.execute(sql)
return cursor.fetchone()[0]
I apologize for the possibly less than syntactically correct Python above, but I hope you get the idea.
Be careful, accepted answer might cause TypeError!
Due to fetchone() documentation:
Fetches the next row of a query result set, returning a single sequence, or None when no more data is available.
So with some SQL queries cursor.fetchone()[0] could turn into None[0] which leads to raising TypeError exception.
Better way to get first row or None is:
first_row = next(cursor, [None])[0]
If SQL query is empty, next will use default value [None] and get first element from that list without raising exceptions.
If you're not using pysqlite which has the built in cursor.fetchone
cursor.execute("select value from table order by value desc limit 1")
Sequence unpacking can be used to extract the scalar value from the result tuple.
By iterating over the cursor (or cursor.fetchall)if there are multiple rows:
for result, in cursor:
print(result)
Or using cursor.fetchone if there is a single row in the resultset:
result, = cur.fetchone()
print(result)
In both cases the trailing comma after result unpacks the element from the single-element tuple. This is the same as the more commonly seen
a, b = (1, 2)
except the tuples only have one element:
a, = (1,)
select count(*) from ... groupy by ... returns None instead of 0,
so fetchone()[0] would lead to an exception.
Therefore
def get_scalar_from_sql(sqlcur, sqlcmd):
# select count(*) from .... groupy by ... returns None instead of 0
sqlcur.execute(sqlcmd)
scalar = 0
tuple_or_None = sqlcur.fetchone()
if not tuple_or_None is None:
(scalar,) = tuple_or_None
return scalar
or you can try :
cursor.execute("SELECT * FROM table where name='martin'")
Is there an elegant way of getting a single result from an SQLite SELECT query when using Python?
for example:
conn = sqlite3.connect('db_path.db')
cursor=conn.cursor()
cursor.execute("SELECT MAX(value) FROM table")
for row in cursor:
for elem in row:
maxVal = elem
is there a way to avoid those nested fors and get the value directly? I've tried
maxVal = cursor[0][0]
without any success.
I think you're looking for Cursor.fetchone() :
cursor.fetchone()[0]
Or you could write a wrapper function that, given SQL, returns a scalar result:
def get_scalar_result(conn, sql):
cursor=conn.cursor()
cursor.execute(sql)
return cursor.fetchone()[0]
I apologize for the possibly less than syntactically correct Python above, but I hope you get the idea.
Be careful, accepted answer might cause TypeError!
Due to fetchone() documentation:
Fetches the next row of a query result set, returning a single sequence, or None when no more data is available.
So with some SQL queries cursor.fetchone()[0] could turn into None[0] which leads to raising TypeError exception.
Better way to get first row or None is:
first_row = next(cursor, [None])[0]
If SQL query is empty, next will use default value [None] and get first element from that list without raising exceptions.
If you're not using pysqlite which has the built in cursor.fetchone
cursor.execute("select value from table order by value desc limit 1")
Sequence unpacking can be used to extract the scalar value from the result tuple.
By iterating over the cursor (or cursor.fetchall)if there are multiple rows:
for result, in cursor:
print(result)
Or using cursor.fetchone if there is a single row in the resultset:
result, = cur.fetchone()
print(result)
In both cases the trailing comma after result unpacks the element from the single-element tuple. This is the same as the more commonly seen
a, b = (1, 2)
except the tuples only have one element:
a, = (1,)
select count(*) from ... groupy by ... returns None instead of 0,
so fetchone()[0] would lead to an exception.
Therefore
def get_scalar_from_sql(sqlcur, sqlcmd):
# select count(*) from .... groupy by ... returns None instead of 0
sqlcur.execute(sqlcmd)
scalar = 0
tuple_or_None = sqlcur.fetchone()
if not tuple_or_None is None:
(scalar,) = tuple_or_None
return scalar
or you can try :
cursor.execute("SELECT * FROM table where name='martin'")
Is there an elegant way of getting a single result from an SQLite SELECT query when using Python?
for example:
conn = sqlite3.connect('db_path.db')
cursor=conn.cursor()
cursor.execute("SELECT MAX(value) FROM table")
for row in cursor:
for elem in row:
maxVal = elem
is there a way to avoid those nested fors and get the value directly? I've tried
maxVal = cursor[0][0]
without any success.
I think you're looking for Cursor.fetchone() :
cursor.fetchone()[0]
Or you could write a wrapper function that, given SQL, returns a scalar result:
def get_scalar_result(conn, sql):
cursor=conn.cursor()
cursor.execute(sql)
return cursor.fetchone()[0]
I apologize for the possibly less than syntactically correct Python above, but I hope you get the idea.
Be careful, accepted answer might cause TypeError!
Due to fetchone() documentation:
Fetches the next row of a query result set, returning a single sequence, or None when no more data is available.
So with some SQL queries cursor.fetchone()[0] could turn into None[0] which leads to raising TypeError exception.
Better way to get first row or None is:
first_row = next(cursor, [None])[0]
If SQL query is empty, next will use default value [None] and get first element from that list without raising exceptions.
If you're not using pysqlite which has the built in cursor.fetchone
cursor.execute("select value from table order by value desc limit 1")
Sequence unpacking can be used to extract the scalar value from the result tuple.
By iterating over the cursor (or cursor.fetchall)if there are multiple rows:
for result, in cursor:
print(result)
Or using cursor.fetchone if there is a single row in the resultset:
result, = cur.fetchone()
print(result)
In both cases the trailing comma after result unpacks the element from the single-element tuple. This is the same as the more commonly seen
a, b = (1, 2)
except the tuples only have one element:
a, = (1,)
select count(*) from ... groupy by ... returns None instead of 0,
so fetchone()[0] would lead to an exception.
Therefore
def get_scalar_from_sql(sqlcur, sqlcmd):
# select count(*) from .... groupy by ... returns None instead of 0
sqlcur.execute(sqlcmd)
scalar = 0
tuple_or_None = sqlcur.fetchone()
if not tuple_or_None is None:
(scalar,) = tuple_or_None
return scalar
or you can try :
cursor.execute("SELECT * FROM table where name='martin'")
I have the following code:
query = """
SELECT Coalesce((SELECT sp.param_value
FROM sites_params sp
WHERE sp.param_name = 'ci'
AND sp.site_id = s.id
ORDER BY sp.id DESC
LIMIT 1), -1) AS ci
FROM sites s
WHERE s.deleted = 0
AND s.id = 10
"""
site = db_session.execute(query)
# print site
# <sqlalchemy.engine.result.ResultProxy object at 0x033E63D0>
site = db_session.execute(query).fetchone()
print site # (u'375')
print list(site) # [u'375']
Why does SQLAlchemy return tuples, not dicts, for this query? I want to use the following style to access the results of the query:
print site.ci
# u'375'
This is an old question, but still relevant today. Getting SQL Alchemy to return a dictionary is very useful, especially when working with RESTful based APIs that return JSON.
Here is how I did it using the db_session in Python 3:
resultproxy = db_session.execute(query)
d, a = {}, []
for rowproxy in resultproxy:
# rowproxy.items() returns an array like [(key0, value0), (key1, value1)]
for column, value in rowproxy.items():
# build up the dictionary
d = {**d, **{column: value}}
a.append(d)
The end result is that the array a now contains your query results in dictionary format.
As for how this works in SQL Alchemy:
Thedb_session.execute(query) returns a ResultProxy object
The ResultProxy object is made up of RowProxy objects
The RowProxy object has an .items() method that returns key, value tuples of all the items in the row, which can be unpacked as key, value in a for operation.
And here a one-liner alternative:
[{column: value for column, value in rowproxy.items()} for rowproxy in resultproxy]
From the docs:
class sqlalchemy.engine.RowProxy(parent, row, processors, keymap)
Proxy values from a single cursor row.
Mostly follows “ordered dictionary” behavior, mapping result values to the string-based column name, the integer position of the result in the row, as well as Column instances which can be mapped to the original Columns that produced this result set (for results that correspond to constructed SQL expressions).
has_key(key)
Return True if this RowProxy contains the given key.
items()
Return a list of tuples, each tuple containing a key/value pair.
keys()
Return the list of keys as strings represented by this RowProxy.
Link: http://docs.sqlalchemy.org/en/latest/core/connections.html#sqlalchemy.engine.RowProxy.items
Did you take a look at the ResultProxy docs?
It describes exactly what #Gryphius and #Syed Habib M suggest, namely to use site['ci'].
The ResultProxy does not "return a tuple" as you claim - it is (not surprisingly) a proxy that behaves (e.g. prints) like a tuple but also supports dictionary-like access:
From the docs:
Individual columns may be accessed by their integer position,
case-insensitive column name, or by schema.Column object. e.g.:
row = fetchone()
col1 = row[0] # access via integer position
col2 = row['col2'] # access via name
col3 = row[mytable.c.mycol] # access via Column object.
I've built a simple class to work like a database interface in our processes. Here it goes:
from sqlalchemy import create_engine
class DBConnection:
def __init__(self, db_instance):
self.db_engine = create_engine('your_database_uri_string')
self.db_engine.connect()
def read(self, statement):
"""Executes a read query and returns a list of dicts, whose keys are column names."""
data = self.db_engine.execute(statement).fetchall()
results = []
if len(data)==0:
return results
# results from sqlalchemy are returned as a list of tuples; this procedure converts it into a list of dicts
for row_number, row in enumerate(data):
results.append({})
for column_number, value in enumerate(row):
results[row_number][row.keys()[column_number]] = value
return results
You can easily convert each result row to a dictionary by using dict(site).
Then site['ci'] would be available if ci column is exists.
In order to have site.ci (according to https://stackoverflow.com/a/22084672/487460):
from collections import namedtuple
Site = namedtuple('Site', site.keys())
record = Site(*site)
This may help solve the OPs question. I think the problem he was having is that the row object only contained column values, but not the column names themselves, as is the case with ORM queries where the results have a dict attribute with both keys and values.
python sqlalchemy get column names dynamically?
The easiest way that I found is using list comprehension with calling dict() func on every RowProxy:
site = db_session.execute(query)
result = [dict(row) for row in site]
Based on Essential SQLAlchemy book:
A ResultProxy is a wrapper around a DBAPI cursor object, and its main
goal is to make it easier to use and manipulate the results of a
statement
Simple select example:
from sqlalchemy.sql import select
stmnt = select([cookies])
result_proxy = connection.execute(stmnt)
results = result_proxy.fetchall()
Results going to be like this:
# ID, cookie_name, quantity, amount
[
(1, u'chocolate chip', 12, Decimal('0.50')),
(2, u'dark chocolate chip', 1, Decimal('0.75')),
(3, u'peanut butter', 24, Decimal('0.25')),
(4, u'oatmeal raisin', 100, Decimal('1.00'))
]
It makes handling query results easier by allowing access using an index, name, or Column object.
Accessing cookie_name in different ways:
first_row = results[0]
first_row[1]
first_row.cookie_name
first_row[cookies.c.cookie_name]
These all result in u'chocolate chip' and they each reference the exact same data element in the first record of our results variable. This flexibility in access is only part of the power of the ResultProxy.
We can also leverage the ResultProxy as an iterable:
result_proxy = connection.execute(stmnt)
for record in result_proxy:
print(record.cookie_name)
This method uses list comprehensions, it receives a sql alchemy rowset object and returns the same items as a list of dictionaries:
class ResultHelper():
#classmethod
def resultproxy_to_list(cls, sql_alchemy_rowset):
return [{tuple[0]: tuple[1] for tuple in rowproxy.items()}
for rowproxy in sql_alchemy_rowset]
As you call db.execute(sql).fetchall(), you can easily use the following function to parse the return data to a dict:
def query_to_dict(ret):
if ret is not None:
return [{key: value for key, value in row.items()} for row in ret if row is not None]
else:
return [{}]
Here is my code in Python:
queryuniq = "SELECT COUNT(distinct src_ip), COUNT(distinct video_id)FROM video"
cur.execute(queryuniq)
uniq = []
uniq = cur.fetchall()
print uniq
ip = str(uniq[0])
video = str(uniq[1])
fd2.write("There are %d ip addresses and %d video in total" %(int(ip), int(video)))
This is the value of "uniq" variable I got:
((2052L, 163581L),)
And this error message:
fd2.write("There are %d ip addresses in total" %(int(ip)))
ValueError: invalid literal for int() with base 10: '((2052L,),)'
video = str(uniq[1])
IndexError: tuple index out of range
I just simply want to count the distinct items in a column in the database, and print the INT value in a file.
Can anyone explain why the SELECT command return a weird data format like ((2052L, 163581L),) ? Don't understand why there is a "L"after the number..
How can I solve this problem? Many thanks!
uniq is a tuple of tuples (each entry at the outer level represents a database row, within which there is a tuple of column values).
You query always returns one row. Therefore the outer tuple always contains one element, and you could fix your code by replacing:
uniq = cur.fetchall()
with
uniq = cur.fetchall()[0]
Also, the conversions from int to string and then back to int are unnecessary.
To summarize, the following is a tidied up version of your code:
queryuniq = "SELECT COUNT(distinct src_ip), COUNT(distinct video_id)FROM video"
cur.execute(queryuniq)
uniq = cur.fetchall()[0]
ip, video = uniq
fd2.write("There are %d ip addresses and %d video in total" %(ip, video))
There several things wrong with your code.
Firstly, cur.fetchall() - as the name implies - fetches all the results from the query. Since Python does not know that your query only returns a single row, it still returns a tuple of all rows. So uniq[0] does not refer to the first field in the row, it refers to the first row in the result.
Since you know you only want one row, you could use cur.fetchone().
Secondly, why are you converting the results to strings then converting them back to ints? That seems pointless. They are in the correct format already - L just means they are 'long ints'.