Python put Database records in Namedtuple - python

I'm trying to write some code in python (2.7) doing this:
Open a database in sqlite
Do a query to the database, getting some results. The database has more than one table and i need the records from different tables:
database is like this:
data.db ---> [table1[col1, col2, col3], table2[col1, col2, col3]]
Iterate through the results
Do something on the results (for exaple in a record there is a date that need decoding)
Store all the records in a namedtuple for further access
Since now i have achieved part 1, 2, 3 and 4, but i can't figure out how to store the results in a namedtuple. Assume that on every iteration i store the data that i need in the namedtuple in temporary variables:
for result in results:
var1 = table1.col1
var2 = table2.col1
var3 = table3.col1
(now i want to do something with the variable, but thats' not the problem, and store the 3 variables in a namedtuple)
contacts = namedtuple('contacts', 'Z_PK ZFULLNAME ZPHONE ZTEXT ZDATE')
if has_contacts_sqlite:
# reading contacts from database
# 1st step: ZWAPHONE table
query = "SELECT * FROM ZWAPHONE"
self.tempcur.execute(query)
tempcontacts = self.tempcur.fetchall()
for contact in tempcontacts:
id = contact.Z_PK
contact_key = contact.ZCONTACT
favorite_key = contact.ZFAVORITE
status_key = contact.ZSTATUS
phonenum = contact.ZPHONE
# 2nd step: name from ZWACONTACT table
query = "SELECT * FROM ZWACONTACT WHERE Z_PK=?;"
self.tempcur.execute(query, [contact_key])
contact_entry = self.tempcur.fetchone()
if contact_entry == None:
name = "N/A"
else:
name = contact_entry.ZFULLNAME
# 3rd step: status from ZWASTATUS table
query = "SELECT * FROM ZWASTATUS WHERE Z_PK=?;"
self.tempcur.execute(query, [status_key])
status_entry = self.tempcur.fetchone()
if status_entry == None:
text = "N/A"
date = "N/A"
else:
text = status_entry.ZTEXT
date = self.formatDate(status_entry.ZDATE)
#print ("%s" %(id))
#print ("%s" %(name.encode(sys.stdout.encoding, errors='replace')))
#print ("%s" %(phonenum.encode(sys.stdout.encoding, errors='replace')))
#print ("%s" %(text.encode(sys.stdout.encoding, errors='replace')))
#print ("%s" %(date))
contact = contacts(id, name, phonenum, text, date)
print contacts
for line in contacts:
print line


A namedtuple() is nothing but a class generated with a factory function:
SomeRowResult = namedtuple('SomeRowResult', 'var1 var2 var3')
Here SomeRowResult is a class object (a subclass of tuple), and calling it will create instances of the class:
for result in results:
result = SomeRowResult(table1.col1, table2.col1, table3.col1)
If you wanted to have a list of these results, you need to explicitly build that list:
all_results = []
for result in results:
result = SomeRowResult(table1.col1, table2.col1, table3.col1)
all_results.append(result)

Related

how to assign variable value with condition

I am work with python. I have code like this :
def return_auth_notificatio():
shiporderid = all_Orderid
cursor.execute("select *from return_auth_notification")
email_record = cursor.fetchall()
for row in email_record:
Orderid = row[1]
Customercomment =row[4]
Amountreturn =row[5]
condition_return_auth_notificatio=(Customercomment,Amountreturn if shiporderid == Orderid else None)
assign_return_auth_notificatio=(condition_return_auth_notificatio if !Null )
return(assign_return_auth_notificatio)
data=return_auth_notificatio()
all_Customercomment=data[0]
all_Amountreurn=data[1]
I want this variable Customercomment,Amountreturn if records whose Orderid matches all_Orderid

Put the condition in the query instead of the loop.
def return_auth_notificatio():
customer_comments = []
amount_returns = []
cursor.execute("""
select Customercomment, Amountreturn
from return_auth_notification
WHERE Orderid = %s""", (all_Orderid,))
for comment, amount in cursor.fetchall():
customer_comments.append(comment)
amount_returns.append(amount)
return customer_comments, amount_returns
all_Customercomment, all_Amountreturn = return_auth_notificatio()
The placeholder %s assumes you're using MySQL. If it's SQLite use ? instead.

List index out of range for Python

I have a question regarding the following exercise:
def addstock():
time = datetime.now().strftime("%B %d, %Y")
hour = datetime.now().strftime("%I:%M%p")
query = 'SELECT TotalStock FROM Stocks WHERE name = ? ORDER BY MovementID DESC LIMIT 1'
parameters = (name.get(),)
lastrecord = run_query(query, parameters)
print(float(lastrecord.fetchall()[0][0]))
print(float(quantity.get()))
totalstock = (float(lastrecord.fetchall()[0][0])) + (float(quantity.get()))
query = 'SELECT precio FROM product WHERE name = ?'
precio = run_query(query, parameters)
pricequant = precio.fetchall()[0]
pricequantity = pricequant * quantities
query = 'SELECT precio FROM product WHERE name = ?'
parameters = (name.get(),)
precio = run_query(query, parameters)
priceforall = pricequant * totalstock
In this function, I print lastrecord.fetchall()[0][0] and quantity.get to make sure they are float. So the program prints in that case: 5.0 for lastrecord.fetchall and quantity.get
Up to now, no problem, but when I try to us them up, it gives me an error of List Index Out Of Range, so program do not find the value of lastrecord.fetchall()[0][0] which 2 lines before I was able to print successfully. Can someone explain why?

According to documentation:
The method fetches all (or all remaining) rows of a query result set and returns a list of tuples. If no more rows are available, it returns an empty list.
When you first time used lastrecord.fetchall()[0][0] all the records of lastrecord curser are fetched, so on the second call on totalstock = (float(lastrecord.fetchall()[0][0])) + (float(quantity.get())) there is no more rows left for the curser. If you want to reuse the fetched data, store it, then use it anytime you want, like this:
all_records = lastrecord.fetchall()
// ...
print(float(all_records[0][0]))
// ...
totalstock = (float(all_records[0][0])) + (float(quantity.get()))

Passing multiple arguments in SQL - Python

I am trying to pass two arguments into a SQL statement as below:
cursor.execute(f"""select * from table
where product_name = '{prod_name}' and date = '{sale_date}'"""")
I am trying to have this run through a loop for several combination so I am trying to see how I can have this altered accordingly.
prod_name = ['prod_a','prod_b']
sale_date = ['2020-01-01','2020-02-01']
I know how to pass one argument through a loop but I am not sure how to pass more than one argument together at the same.

It's a security danger to add variables directly to your SQL query. cursor.execute provides sanitizing as long as you pass the arguments as the second argument of the function call.
Example:
cursor.execute("select * form table where product_name = '%s' and date = '%s'", (prod_name, sale_date))
To loop through multiple lists at once you can do the following (assuming the lists have the same amount of values):
for i in range(len(prod_name)):
cursor.execute("select * form table where product_name = '%s' and date = '%s'", (prod_name[i], sale_date[i]))
By looping through a range I get the numbers of 0 - len(prod_name) and as I loop with the index of i I can use that to retrieve the first item in both lists.
Sam Mason had a good comment about using the zip function which combines iterators and can be used like so:
for args in zip(prod_name, sale_date):
cursor.execute("select * form table where product_name = '%s' and date = '%s'", args)

try this :
results = ()
dc = ['103,4770634', '42,427752', '64,10122045', '42,13603629', '42,25516425', '103,2748102', '42,1966402', '42,30262834', '42,6667711', '18,13737683', '42,28921168', '42,26076925', '103,3733654', '42,23313527', '64,3307344', '103,3973533', '42,6360982', '48,11846077', '103,3775309', '64,10122050', '42,1965119', '103,4265810', '103,3971645', '103,4962583', '103,689615', '42,22834366', '103,761655', '95,1184', '64,9594482', '42,22855603', '48,8654764', '103,4226756', '42,23366982', '103,3897036', '42,11339650', '101,6369', '42,25830920', '103,5009291', '42,29238961', '59,6299475', '42,22931663', '42,25839056', '43,11864458', '43,41346192', '103,4261645', '42,3747082', '103,4795050', '42,9417503', '103,4245623', '42,61431911']
try:
sql = "SELECT * FROM tbl1 WHERE id1 in (%s) AND id2 in (%s)"
in_ids = ', '.join(map(lambda x: '%s', dc))
in_ids = in_ids % tuple(dc)
sql = sql % (in_ids, in_ids)
cursor.execute(sql)
res = cursor.fetchall()
results = results + res
except Exception, e:
print e

How do I use variables as attribute names when using python mysql connector

I am trying to use variables in a python function to try and retrieve attributes with mysql connector
It seems to work only when I specify the name of the attribute in the query itself
def insert(ids, added_attribute):
insert = ''
if len(ids) > 0:
#insert scpecified attributes wanted
insert += ' AND (%s = %s' %(added_attribute, ids[0])
#for loop for more than one specified specific attribute
for id_index in range(1, len(ids)):
insert += ' OR %s = %s' %(added_attribute, ids[id_index])
insert += ')'#close parenthesis on query insert
return insert
def get(name, attributes = 0, ids = []):
cursor = conn.cursor()
#insert specific ids
insert = insert(ids, "id")
query = 'SELECT %s FROM (TABLE) WHERE (name = %s%s)'
cursor.execute(query, (attributes, name, insert))
data = cursor.fetchall()
cursor.close()
return data
I keep getting null as a return value

Try this...
query = 'SELECT {} FROM (TABLE) WHERE (name = {}{})'
cursor.execute(query.format(attributes, name, insert))
{} is replacing %s here and to call the variables you just need to add .format() with the vars you want inserted in order.

Data extraction and transform efficiency

I've got a Python script that connects to a MySQL database and executes a number of nested SELECT queries. It's basically a giant for loop. The database is structured such that Businesses have Menus, Menus have Sections, and Sections have Items. The script queries all the Businesses, and for each Business, it queries all of its Menus, and so on. It builds a big dictionary along the way that it then spits out as JSON.
It looks something like this:
#!/usr/bin/env python
from bottle import route, run
import mysql.connector
import json
import collections
import datetime
def getBusinesses():
conn = mysql.connector.connect(user="APIUser", password="abc123", host="12.34.56.78", port="54321", database="businesses")
cursor = conn.cursor()
objects = {}
businessesQuery = ("SELECT * FROM business")
cursor.execute(businessesQuery)
businessRows = cursor.fetchall()
businessObjects = []
for businessRow in businessRows:
print businessRow[0]
businessDict = collections.OrderedDict()
businessDict['id'] = businessRow[0]
businessDict['business_name'] = businessRow[1]
businessDict['business_address1'] = businessRow[2]
businessDict['business_address2'] = businessRow[3]
businessDict['business_city'] = businessRow[4]
businessDict['business_state'] = businessRow[5]
businessDict['business_zip'] = businessRow[6]
businessObjects.append(businessDict)
menuQuery = ("SELECT * FROM menu WHERE business_id = %s" % businessRow[0])
cursor.execute(menuQuery)
menuRows = cursor.fetchall()
menuObjects = []
for menuRow in menuRows:
menuDict = collections.OrderedDict()
menuDict['id'] = menuRow[0]
menuDict['menu_name'] = menuRow[1]
menuDict['menu_description'] = menuRow[2]
menuDict['menu_note'] = menuRow[3]
menuDict['business_id'] = menuRow[4]
menuObjects.append(menuDict)
businessDict['menus'] = menuObjects
for menuIdx, menuRow in enumerate(menuRows):
sectionQuery = ("SELECT * FROM menu_section WHERE menu_id = %s" % menuRow[0])
cursor.execute(sectionQuery)
sectionRows = cursor.fetchall()
sectionObjects = []
for sectionIdx, sectionRow in enumerate(sectionRows):
sectionDict = collections.OrderedDict()
sectionDict['id'] = sectionRow[0]
sectionDict['section_name'] = sectionRow[1]
sectionDict['section_note'] = sectionRow[2]
sectionDict['section_description'] = sectionRow[3]
sectionDict['menu_id'] = sectionRow[4]
sectionObjects.append(sectionDict)
businessDict['menus'][menuIdx]['sections'] = sectionObjects
itemQuery = ("SELECT * FROM menu_item WHERE section_id = %s" % sectionRow[0])
cursor.execute(itemQuery)
itemRows = cursor.fetchall()
itemObjects = []
for itemIdx, itemRow in enumerate(itemRows):
itemDict = collections.OrderedDict()
itemDict['id'] = itemRow[0]
itemDict['item_name'] = itemRow[1]
itemDict['item_description'] = itemRow[2]
itemDict['item_note'] = itemRow[3]
itemDict['item_price'] = itemRow[4]
itemDict['section_id'] = itemRow[5]
itemObjects.append(itemDict)
businessDict['menus'][menuIdx]['sections'][sectionIdx]['items'] = itemObjects
objects['businesses'] = businessObjects
return objects
#route('/test')
def index():
return json.dumps(getBusinesses())
run(host='192.168.1.70', port=7070)
I want to know if this is an efficient way of doing things. When I deployed my database remotely (WebFaction) and ran the Bottle server locally, it took almost 40 seconds to return a few hundred rows. So it seems like something is amiss. I have a gut feeling that there could be a better way of doing this. Just not sure what that way is!

if I had to venture a guess: notice the rough structure of your code is:
def getBusinesses():
businessesQuery = ("SELECT * FROM business")
businessRows = cursor.fetchall()
businessObjects = []
for businessRow in businessRows:
menuQuery = ("SELECT * FROM menu WHERE business_id = %s" % businessRow[0])
menuRows = cursor.fetchall()
for menuIdx, menuRow in enumerate(menuRows):
sectionQuery = ("SELECT * FROM menu_section WHERE menu_id = %s" % menuRow[0])
cursor.execute(sectionQuery)
sectionRows = cursor.fetchall()
sectionObjects = []
for sectionIdx, sectionRow in enumerate(sectionRows):
itemQuery = ("SELECT * FROM menu_item WHERE section_id = %s" % sectionRow[0])
itemRows = cursor.fetchall()
That is, you execute nearly identical queries in a loop for menu, menu_section and especially menu_item. Also, you're using fetchall() to return the full contents of the result set but examine each element only once, in a loop, where you create another list of objects.
what you might want instead is something more like:
businesses = []
cursor.execute("select * from business")
row = cursor.fetchone()
while row is not None:
business.append(...(row))
row = cursor.fetchone()
cursor.execute("select * from menu")
row = cursor.fetchone()
while row is not None:
business[row['business_id']].menus.append(...(row))
row = cursor.fetchone()
cursor.execute("select menu.business_id, menu_section.*"
" from menu_section"
" join menu on menu.id = menu_section.menu_id")
row = cursor.fetchone()
while row is not None:
business[row['business_id']][row['menu_id']].sections.append(...(row))
row = cursor.fetchone()
cursor.execute("select menu.business_id, menu_section.menu_id, menu_item.*"
" from menu_item"
" join menu_section on menu_section.id = menu_item.section_id"
" join menu on menu.id = menu_section.menu_id")
row = cursor.fetchone()
while row is not None:
business[row['business_id']][row['menu_id']][row['section_id'].items.append(...(row))
row = cursor.fetchone()
so that you're issuing a much smaller number of queries, and only loading the amount of data you can process in one go.

Categories

Resources