I have product information in to different objects, product description and product highlights. both of these objects have product_id, so I can associate them together.
decription_items is a list of dictionary, e.g:
[
{'product_id': '123', 'description': 'desc1', 'price': '$40' },
{'product_id': '124', 'description': 'desc2', 'price': '$50' },
{'product_id': '125', 'description': 'desc3', 'price': '$99' },
]
product_highlight_dict is a dictionary of (product_id, ProductHighlight)
{
'123': <product_123_highligh>,
'124': <product_124_highligh>,
'125': <product_125_highligh>,
}
and finally ProductHighlight is a class:
class ProductHighlight:
def __init__(self, product_id, location, area):
self.product_id = product_id
self.location = location
self.area = area
What I want to do is to merge these two types and write them to an xml document, in the following code, I can merge the two types:
for description_item in self.decription_items:
product_id = .get('product_id')
if product_id:
product_highlight = spider.product_highlight_dict.get(product_id)
# I don't know how to combine description_item and
# product_highlight and write them to an xml
Update
I use the following code to write product_highlight_dict to an xml. I don't know how to include description_item in the following logic?
highlights = []
for k in self.product_highlight_dict:
highlights.append(vars(self.product_highlight_dict[k]))
xml = dicttoxml.dicttoxml(highlights, custom_root='product_highlights')
file = open('filename', "wb")
file.write(xml)
file.close()
You can use description_items to build a dictionary with description and price information for every product:
product_data = {}
for description_item in description_items:
product_id = description_item["product_id"]
product_data[product_id] = description_item
Then you can use it in your code like this
highlights = []
for product_id, product_highlight in self.product_highlight_dict.items():
highlight = vars(product_highlight)
if product_id in product_data:
highlight.update(product_data[product_id])
highlights.append(highlight)
Related
def read_data(service_client):
data = list_data(domain, realm) # This returns a data frame
building_data = []
building_names = {}
all_buildings = {}
for elem in data.iterrows():
building = elem[1]['building_name']
region_id = elem[1]['region_id']
bandwith = elem[1]['bandwith']
building_id = elem[1]['building_id']
return {
'Building': building,
'Region Id': region_id,
'Bandwith': bandwith,
'Building Id': building_id,
}
Basically I am able to return a single dictionary value upon a iteration here in this example. I have tried printing it as well and others.
I am trying to find a way to store multiple dictionary values on each iteration and return it, instead of just returning one.. Does anyone know any ways to achieve this?
You may replace your for-loop with the following to get all dictionaries in a list.
naming = {
'building_name': 'Building',
'region_id': 'Region Id',
'bandwith': 'Bandwith',
'building_id': 'Building Id',
}
return [
row[list(naming.values())].to_dict()
for idx, row in data.rename(naming, axis=1).iterrows()
]
I am trying to search for all users in an sql database whose first names are "blah" and return that data to my html through an ajax call. I have this functioning with a single user like this:
user = db.execute(
'SELECT * FROM user WHERE genres LIKE ?', (str,)
).fetchone()
user_details = {
'first': user['first'],
'last': user['last'],
'email': user['email']
}
y = json.dumps(user_details)
return jsonify(y)
Now for multiple users I want the struct to look something like this:
users{
user1_details = {
'first': user['first'],
'last': user['last'],
'email': user['email']
}
user2_details = {
'first': user2['first'],
'last': user2['last'],
'email': user2['email']
}
user3_details = {
'first': user3['first'],
'last': user3['last'],
'email': user3['email']
}
}
generating each user_details in a loop. I know I can use fetchall() to find all the users, but how do I concatenate the details?
Fetch all the rows after the query, then structure the results as you'd like.
Example:
db = mysql.connection.cursor()
# query
db.execute('SELECT * FROM user')
# returned columns
header = [x[0] for x in db.description]
# returned rows
results = db.fetchall()
#data to be returned
users_object = {}
#structure results
for result in results:
users_object[result["user_id"]] = dict(zip(header,result))
return jsonify(users_object)
As you can see in under "#structure results", you just loop through the results and insert the data for each row into the users_object with key equal to "user_id" for example.
If you want the results in an array just convert users_object into an array e.g. users_array and append the dict to the array within the loop instead
The keys in the desired users dictionary do not seem particularly useful so you could instead build a list of user dicts. It's easy to go directly from fetchall() to such a list:
result = db.execute('SELECT * FROM user WHERE genres LIKE ?', (str,))
users = [{'first': first, 'last': last, 'email': email} for first, last, email in result.fetchall()]
return jsonify(users)
To return a dict containing the user list:
return jsonify({'users': users})
I am learning python and coding. I am trying one web scraping example. I download the currency exchange data from a website and I want to compute average exchange rate for each currency over a 50 days period. The problem is that I am unable to do the following.
I get results from first function which should be in form of a dictionary and then pass these dictionaries to another function as argument and to perform averaging of those values. I am unable to pass correctly dict values to another function.
my code is as follow
import os
import webbrowser
import requests as rq
import sys
from bs4 import BeautifulSoup
from xml.etree import ElementTree as ET
def saveData(path, date):
session = rq.session()
url = 'https://www.bnm.md/en/official_exchange_rates?get_xml=1&date=' + date
datastore = session.get(url)
with open(path, 'wb') as f:
f.write(datastore.content)
data = ET.fromstring(datastore.content)
'''
elements = {}
for element in data.iter():
if element.tag in ('Name', 'Value'):
elements[element.tag] = element.text
print 'elements:', elements
# Here I want to combine those all dictionaries in variable so that i can pass it as argument to another function
return elements
'''
# i replace the above triple quote code with the following below code
elements = {}
for tag, text in data.items():
if tag in ('Name', 'Value'):
elements.setdefault(tag, [])
elements[tag].append(text)
return elements
def computeAverage(elements): # I want to pass function saveData() results who are in dictioanry form to this function but I am unable to solve this issue.
print elements
def main():
dates = ['20.04.2016', '21.04.2016', '22.04.2016']
paths = []
for date in dates:
path = '/home/robbin/Desktop/webscrape/{}.xml'.format(date)
paths.append(path)
data3 = {}
for path, date in zip(paths, dates):
data2 = saveData(path, date)
print 'data2: ', data2
for k, v in data2.items():
data3.setdefault(k, [])
data3[k].append(v)
print 'data3: ', data3
computeAverage(data3)
if __name__ == '__main__':
main()
Also I am getting the results from saveData() function as dictionaries like this and it repeat every dictionary for the next item too which is wrong.
elements: {'Name': 'Euro'}
elements: {'Name': 'Euro', 'Value': '22.4023'}
elements: {'Name': 'US Dollar', 'Value': '22.4023'}
elements: {'Name': 'US Dollar', 'Value': '19.7707'}
elements: {'Name': 'Russian Ruble', 'Value': '19.7707'}
elements: {'Name': 'Russian Ruble', 'Value': '0.3014'}
elements: {'Name': 'Romanian Leu', 'Value': '0.3014'}
elements: {'Name': 'Romanian Leu', 'Value': '4.9988'}
Also what I tried to get results like this but failed
elements: {'Name': 'Euro', 'Value': '22.4023'}
elements: {'Name': 'US Dollar', 'Value': '19.7707'}
elements: {'Name': 'Russian Ruble', 'Value': '0.3014'}
elements: {'Name': 'Romanian Leu', 'Value': '4.9988'}
Updates:-------------
elements = []
for element in data.iter():
if element.tag in ('Name', 'Value'):
elements.append(element.text)
# print 'elements: ', elements
return elements
and in the main function() i make
for path, date in zip(paths, dates):
data = saveData(path, date)
# print 'data from main: ', data
computeAverage(data)
and the output of "print 'data from main: ', data" looks like this
['Euro', '22.4023', 'US Dollar', '19.7707', 'Russian Ruble', '0.3014', 'Romanian Leu', '4.9988',.........'Special Drawing Rights', '27.8688']
['Euro', '22.4408', 'US Dollar', '19.7421', 'Russian Ruble', '0.3007', 'Romanian Leu', '5.0012',.....'Special Drawing Rights', '27.8606']
I am newbie to coding and if someone help me regarding these two problems. I would be really thankful.
First of all, I agree with #Prakhar Verma.
Second, you didn't mention clearly what you want. But I can assume that you want to merge the data that you got from the 'saveData' function and then calculate average. So, here is the missing code.
data3 = {}
for path, date in zip(paths, dates):
data2 = saveData(path, date)
for k, v in data2.items():
# you can move this line after declaring the data3 dict if keys returned by saveData are fixed i.e. name, value
data3.setdefault(k, [])
data3[k].append(v)
computeAverage(data3)
Update to saveData function:
elements = {}
for tag, text in data.items():
if tag in ('Name', 'Value'):
elements.setdefault(tag, [])
elements[tag].append(text)
===================================================
Update 2:
def saveData(path, date):
#session = rq.session()
url = 'https://www.bnm.md/en/official_exchange_rates?get_xml=1&date=' + date
datastore = rq.get(url)
with open(path, 'wb') as f:
f.write(datastore.content)
data = ET.fromstring(datastore.content)
# i replace the above triple quote code with the following below code
elements = {}
for element in data.iter():
tag = element.tag
text = element.text
if tag in ('Name', 'Value'):
elements.setdefault(tag, [])
elements[tag].append(text)
return elements
def main():
dates = ['20.03.2016', '21.03.2016', '22.03.2016']
paths = []
for date in dates:
#please edit this
path = '{}.xml'.format(date)
paths.append(path)
data3 = {}
for path, date in zip(paths, dates):
data2 = saveData(path, date)
for k, v in data2.items():
data3.setdefault(k, [])
data3[k].append(v)
computeAverage(data3)
The 'saveData' function is returning data but you are not saving it in any variable. So what you need to do is save the data when it's returned from 'saveData' function and then send it as a parameter to 'computeAverage' function.
Please go through the basics of coding and follow any programming tutorial. :)
I want to save an array of objects passed from javascript through ajax to me database. This is my view code:
data2 = json.loads(request.raw_get_data)
for i in data2:
print(key)
obj = ShoppingCart(quantity = i.quantity , user_id = 3, datetime = datetime.now(), product_id = i.pk)
obj.save()
return render_to_response("HTML.html",RequestContext(request))
After the first line, i get this in my dictionary:
[{'model': 'Phase_2.product', 'fields': {'name': 'Bata', 'category': 2, 'quantity': 1, 'subcategory': 1, 'count': 2, 'price': 50}, 'imageSource': None, 'pk': 1}]
(Only one object in the array right now)
I want to be able access individual fields like quantity, id, etc in order to save the data to my database. When i debug this code, it gives a name error on 'i'. I also tried accessing the fields like this: data2[0].quantity but it gives this error: {AttributeError}dict object has no attribute quantity.
Edited code:
for i in data2:
name = i["fields"]["name"]
obj = ShoppingCart(quantity = i["fields"]["quantity"] , user_id = 3, datetime = datetime.now(), product_id = i["fields"]["pk"])
obj.save()
It might help you to visualise the returned dict with proper formatting:
[
{
'model': 'Phase_2.product',
'fields': {
'name': 'Bata',
'category': 2,
'quantity': 1,
'subcategory': 1,
'count': 2,
'price': 50
},
'imageSource': None,
'pk': 1
}
]
The most likely reason for your error is that you are trying to access values of of the inner 'fields' dictionary as if they belong to the outer i dictionary.
i.e.
# Incorrect
i["quantity"]
# Gives KeyError
# Correct
i["fields"]["quantity"]
Edit
You have the same problem in your update:
# Incorrect
i["fields"]["pk"]
# Correct
i["pk"]
The "pk" field is in the outer dictionary, not the inner "fields" dictionary.
You may try:
i['fields']['quantity']
The json.loads() returns you a dictionary, which should be accessed by key.
I have a dataframe that I need to transform into JSON. I think it would be easier to first turn it into a dictionary, but I can't figure out how. I need to transform it into JSON so that I can visualize it with js.d3
Here is what the data looks like currently:
NAME, CATEGORY, TAG
Ex1, Education, Books
Ex2, Transportation, Bus
Ex3, Education, Schools
Ex4, Education, Books
Ex5, Markets, Stores
Here is what I want the data to look like:
Data = {
Education {
Books {
key: Ex1,
key: Ex2
}
Schools {
key: Ex3
}
}
Transportation {
Bus {
key: Ex2
}
}
Markets {
Stores {
key: Ex5
}
}
(I think my JSON isn't perfect here, but I just wanted to convey the general idea).
This code is thanks to Brent Washburne's very helpful answer above. I just needed to remove the tags column because for now it was too messy (many of the rows had more than one tag separated by commas). I also added a column (of integers) which I wanted connected to the names. Here it is:
import json, string
import pprint
def to_json(file):
data = {}
for line in open(file):
fields = map(string.strip, line.split(','))
categories = data.get(fields[1], [])
to_append = {}
to_append[fields[0]] = fields[3]
categories.append(to_append)
data[fields[1]] = categories
return json.dumps(data)
print to_json('data.csv')
You can't use 'key' as a key more than once, so the innermost group is a list:
import json, string
def to_json(file):
data = {}
for line in open(file):
fields = map(string.strip, line.split(','))
categories = data.get(fields[1], {})
tags = categories.get(fields[2], [])
tags.append(fields[0])
categories[fields[2]] = tags
data[fields[1]] = categories
return json.dumps(data)
print to_json('data.csv')
Result:
{"Markets": {"Stores": ["Ex5"]}, "Education": {"Schools": ["Ex3"], "Books": ["Ex1", "Ex4"]}, "Transportation": {"Bus": ["Ex2"]}}