I'm trying to create generator object for the list of records with the data from mysql database, so I'm passing the mysql cursor object to the function as parameter.
My issue here is if the "if block" containing yield records is commented then cust_records function works perfectly fine but if I uncomment the line then the function is not working.
Not sure if this is not the way to yield the list object in Python 3
My code so far:
def cust_records(result_set) :
block_id = None
records = []
i = 0
for row in result_set :
records.append((row['id'], row, smaller_ids))
if records :
yield records
The point of generators is lazy evaluation, so storing all records in a list and yielding the list makes no sense at all. If you want to retain lazy evalution (which is IMHO preferable, specially if you have to work on arbitrary datasets that might get huge), you want to yield each record, ie:
def cust_records(result_set) :
for row in result_set :
yield (row['id'], row, smaller_ids)
# then
def example():
cursor.execute(<your_sql_query_here>)
for record in cust_records(cursor):
print(record)
else (if you really want to consume as much memory as possible) just male cust_record a plain function:
def cust_records(result_set) :
records = []
for row in result_set :
records.append((row['id'], row, smaller_ids))
return records
I'm trying to write a generic method for querying one mongodb collection (A) about ids from a separate mongodb collection (B). Here's what I have so far:
def getOtherCollInfo(self, otherCollObj, queryField, outputField="_id"):
selfIdList = self.getIds() # gets a set of ids from whole collection (B)
returned_dict = {}
for selfId in selfIdList:
curs_obj = otherCollObj.find({queryField : str(selfId)}).distinct(outputField) #otherCollObj is the 'collection (A)
temp_list=[]
for obj in curs_obj:
temp_list.append(obj)
returned_dict[selfId]=temp_list
return returned_dict
This works fine for collection A where the query_field looks like:
542de00c763f4a7f558be12f
When attempting this method on a third collection (C), it fails (I think) because the query_field is just the hex sting:
ObjectId('542de00c763f4a7f558be12f')
Is there a way to test the formatting of the id so I can make the method more generic?
i have a query which returns me a Viewobject with all the entries i want to process. I know i can iterate over this view Object so that i can use the single entries for my purposes.
Now i want to extract only the first and the last row. The first row is no problem because i can just iterate and break the loop after the first item.
Now my question is, how to get the last element from the View.
I tried by:
for row in result_rows:
rowvalue = row[3].value
diagdata = rowvalue[models.DIAGDATA]
if models.ODOMETER in diagdata:
start_mileage = diagdata[models.ODOMETER]
start_mileage_found = True
break
row = result_rows[len(result_rows)]
rowvalue = row[3].value
diagdata = rowvalue[models.DIAGDATA]
if models.ODOMETER in diagdata:
end_mileage = diagdata[models.ODOMETER]
end_mileage_found = True
The second value i obviously wont get, because view has neither a length nor can i access the rows by a index. Has anyone an idea how to get the last element?
You might run another request but with descending=True option, so that the server will stream results in reverse order.
Or you can convert iterator to array which basically the same a iterate through all values. I'm not a python expert, but it seems like list(result_rows) will do it for you. And when you are doing len(...) it probably doing it for you implicitly. There is rows_returned method to get the number of rows without turning it to list.
I have the following code:
query = """
SELECT Coalesce((SELECT sp.param_value
FROM sites_params sp
WHERE sp.param_name = 'ci'
AND sp.site_id = s.id
ORDER BY sp.id DESC
LIMIT 1), -1) AS ci
FROM sites s
WHERE s.deleted = 0
AND s.id = 10
"""
site = db_session.execute(query)
# print site
# <sqlalchemy.engine.result.ResultProxy object at 0x033E63D0>
site = db_session.execute(query).fetchone()
print site # (u'375')
print list(site) # [u'375']
Why does SQLAlchemy return tuples, not dicts, for this query? I want to use the following style to access the results of the query:
print site.ci
# u'375'
This is an old question, but still relevant today. Getting SQL Alchemy to return a dictionary is very useful, especially when working with RESTful based APIs that return JSON.
Here is how I did it using the db_session in Python 3:
resultproxy = db_session.execute(query)
d, a = {}, []
for rowproxy in resultproxy:
# rowproxy.items() returns an array like [(key0, value0), (key1, value1)]
for column, value in rowproxy.items():
# build up the dictionary
d = {**d, **{column: value}}
a.append(d)
The end result is that the array a now contains your query results in dictionary format.
As for how this works in SQL Alchemy:
Thedb_session.execute(query) returns a ResultProxy object
The ResultProxy object is made up of RowProxy objects
The RowProxy object has an .items() method that returns key, value tuples of all the items in the row, which can be unpacked as key, value in a for operation.
And here a one-liner alternative:
[{column: value for column, value in rowproxy.items()} for rowproxy in resultproxy]
From the docs:
class sqlalchemy.engine.RowProxy(parent, row, processors, keymap)
Proxy values from a single cursor row.
Mostly follows “ordered dictionary” behavior, mapping result values to the string-based column name, the integer position of the result in the row, as well as Column instances which can be mapped to the original Columns that produced this result set (for results that correspond to constructed SQL expressions).
has_key(key)
Return True if this RowProxy contains the given key.
items()
Return a list of tuples, each tuple containing a key/value pair.
keys()
Return the list of keys as strings represented by this RowProxy.
Link: http://docs.sqlalchemy.org/en/latest/core/connections.html#sqlalchemy.engine.RowProxy.items
Did you take a look at the ResultProxy docs?
It describes exactly what #Gryphius and #Syed Habib M suggest, namely to use site['ci'].
The ResultProxy does not "return a tuple" as you claim - it is (not surprisingly) a proxy that behaves (e.g. prints) like a tuple but also supports dictionary-like access:
From the docs:
Individual columns may be accessed by their integer position,
case-insensitive column name, or by schema.Column object. e.g.:
row = fetchone()
col1 = row[0] # access via integer position
col2 = row['col2'] # access via name
col3 = row[mytable.c.mycol] # access via Column object.
I've built a simple class to work like a database interface in our processes. Here it goes:
from sqlalchemy import create_engine
class DBConnection:
def __init__(self, db_instance):
self.db_engine = create_engine('your_database_uri_string')
self.db_engine.connect()
def read(self, statement):
"""Executes a read query and returns a list of dicts, whose keys are column names."""
data = self.db_engine.execute(statement).fetchall()
results = []
if len(data)==0:
return results
# results from sqlalchemy are returned as a list of tuples; this procedure converts it into a list of dicts
for row_number, row in enumerate(data):
results.append({})
for column_number, value in enumerate(row):
results[row_number][row.keys()[column_number]] = value
return results
You can easily convert each result row to a dictionary by using dict(site).
Then site['ci'] would be available if ci column is exists.
In order to have site.ci (according to https://stackoverflow.com/a/22084672/487460):
from collections import namedtuple
Site = namedtuple('Site', site.keys())
record = Site(*site)
This may help solve the OPs question. I think the problem he was having is that the row object only contained column values, but not the column names themselves, as is the case with ORM queries where the results have a dict attribute with both keys and values.
python sqlalchemy get column names dynamically?
The easiest way that I found is using list comprehension with calling dict() func on every RowProxy:
site = db_session.execute(query)
result = [dict(row) for row in site]
Based on Essential SQLAlchemy book:
A ResultProxy is a wrapper around a DBAPI cursor object, and its main
goal is to make it easier to use and manipulate the results of a
statement
Simple select example:
from sqlalchemy.sql import select
stmnt = select([cookies])
result_proxy = connection.execute(stmnt)
results = result_proxy.fetchall()
Results going to be like this:
# ID, cookie_name, quantity, amount
[
(1, u'chocolate chip', 12, Decimal('0.50')),
(2, u'dark chocolate chip', 1, Decimal('0.75')),
(3, u'peanut butter', 24, Decimal('0.25')),
(4, u'oatmeal raisin', 100, Decimal('1.00'))
]
It makes handling query results easier by allowing access using an index, name, or Column object.
Accessing cookie_name in different ways:
first_row = results[0]
first_row[1]
first_row.cookie_name
first_row[cookies.c.cookie_name]
These all result in u'chocolate chip' and they each reference the exact same data element in the first record of our results variable. This flexibility in access is only part of the power of the ResultProxy.
We can also leverage the ResultProxy as an iterable:
result_proxy = connection.execute(stmnt)
for record in result_proxy:
print(record.cookie_name)
This method uses list comprehensions, it receives a sql alchemy rowset object and returns the same items as a list of dictionaries:
class ResultHelper():
#classmethod
def resultproxy_to_list(cls, sql_alchemy_rowset):
return [{tuple[0]: tuple[1] for tuple in rowproxy.items()}
for rowproxy in sql_alchemy_rowset]
As you call db.execute(sql).fetchall(), you can easily use the following function to parse the return data to a dict:
def query_to_dict(ret):
if ret is not None:
return [{key: value for key, value in row.items()} for row in ret if row is not None]
else:
return [{}]
I have code along these lines:
classinstance.col1 = queryresult.col1
classinstance.col2 = queryresult.col2
classinstance.col3 = queryresult.col3
classinstance.col4 = queryresult.col4
Which adds variables to the classinstance and assigns the values of the queryresult column with the same name as the variable.
I am hoping to make my code a little more flexible, and not need to identify the columns by name. To this end, I was wondering if there was some way to do a loop over all the columns, rather than handle each one individually. Something like this (This is psuedocode rather than actual code, since I'm not sure what it should actually look like):
for each var in vars(queryresult):
classinstance.(var.name) = var.value
Is this possible? What does it require? Is there some fundamental misunderstanding on my part?
I'm assuming there's only one row in the result for the following example (built with help from comments here). The key component here is zip(row.cursor_description, row) used to get column names from pyodbc.Row object.
# convert row to an object, assuming row variable contains query result
rowdict = { key[0]:value for (key, value) in zip(row.cursor_description, row) }
# loop through keys (equivalent to column names) and set class instance values
# assumes existing instance of class is variable classinstance
for column in rowdict.keys():
classinstance[column] = rowdict[column]