JSON: iterate over a list of nested dictionaries - python

I have a huge list of dictionaries appended to the list from JSON in the manner below. I would like to access the "raw" from each dictionary and store it entire in a list of dictionary or a one huge dictionary. The final goal is to access the keys in the raw and convert them to dataframe columns using pandas.
results = [{
'FirstSentences': None,
'PrintableUri': '',
'hasHtmlVersion': False,
'hasMobileHtmlVersion': False,
'isRecommendation': False,
'isTopResult': False,
'parentResult': None,
'percentScore': 100.0,
'printableUriHighlights': [],
'rankingInfo': None,
'rating': 3.0,
'raw': {'distance': 1892760.0,
'distancekm': 1892.76,
'distancemi': 1176.11,
'objecttype': 'Account',
'objecttypename': 'Account',
'plocepp': 'False',
'plochpp': 'False',
'plocsdp': 'False'}}]
The code and the error I'm getting is as below:
response = [x['raw'] for x in results]
File "<ipython-input-90-64993f9dcd67>", line 1, in <module>
response = [x['raw'] for x in results]
TypeError: list indices must be integers, not str
I have searched a lot of answers here but couldn't find the solution to my problem. Thanks a lot for the help in advance.

The key is to iterate through the list and for each element in the list, index the dictionary key 'raw'.
One simple way is to iterate over the list and and access the key 'raw' in the dictionary.
Simple for loop method:
response = []
for d in results:
response.append(d['raw'])
List comprehension method
response = [d['raw'] for d in results]

Related

python updating dictionary with list as the value type

I'm trying to iterate through data extracted from a file and store them in a dictionary based on each data's id
These are the id (str) for the data : "sensor", "version", "frame", "function"
And the data are in hexadecimal string.
What I bascially start with is a huge list of tuples in a form of id and data (that i extracted from a file)
example_list = [("sensor", 245), ("frame", 455), ("frame", 77)] and so on
This example_list stores all the data, so it has information of data for all the id.
I want to make a dictionary with id as key and list of data as value so when done iterating through the example_list, I have list of values for specific id (so I can iterate through the value list to get all the data for a specific id (the key))
To start, all values (list) will start with an empty list
my_dict = {"sensor": [], "frame": [], "version": [], "function": []}
Then, as I iterate through example_list, if the id is in my_dict as a key, I append the value to the values list in my_dict
for itm in example_list:
if itm[0] in my_dict:
tmp = my_dict[itm[0]] # since itm[0] is the id
tmp.append(itm[1])
my_dict[itm[0]] = tmp # update the list
When I tried this, it seems like the final my_dict's value list has the value of the lastest data
What I mean by this is if
example_list = [("sensor", 245), ("frame", 455), ("frame", 77)]
then
my_dict = {"sensor": [245], "frame": [77], "version": [], "function": []}
I may be wrong about this interpretation (since the data I'm reading is really big), but when I printed my_dict in the end of function, each value list had only one data inside, which is far off from what I expected (list of data instead of just one)
I tried searching and people used update function to update the dictionary but that one also didn't seem to work and gave me somehting unhashable error/warning.
Any way to implement what I want to do?
try doing it like so:
for itm in example_list:
if itm[0] in my_dict:
my_dict[itm[0]].append(itm[1])
Your code is working as required. To simplify, as you've already instantiated the dict with empty lists:
for i,j in example_list:
my_dict[i].append(j)
print(my_dict)
Output:
{'sensor': [245], 'frame': [455, 77], 'version': [], 'function': []}
What you want to do is:
for itm in example_list:
if itm[0] in my_dict.keys(): # have to look if keys match
my_dict[itm[0]].append(itm[1]) # # directly access the key-value pair
Your problem was that you created a new list and appended your item to it each time the loop was run, therefore the old data was deleted everytime.

I want to get the randomly picked key value from the dictionaries list but I got a type error.(note the list is long so putting a index is difficult)

def data_change(account):
data_name = data["name"]
data_desc = data["description"]
data_country = data["country"]
return f"{data_name}, is a {data_desc}, from {data_country}"
print(f"option A : {data_change(data_a)}")
The above code is data I want to process for the random data to display.
the list dictionary below are the first 2 example data
data = [
{
'name': 'Instagram',
'follower_count': 346,
'description': 'Social media platform',
'country': 'United States'
},
{
'name': 'Cristiano Ronaldo',
'follower_count': 215,
'description': 'Footballer',
'country': 'Portugal'
}]
and the error display is
TypeError: list indices must be integers or slices, not str
on the line: data_name = data["name"]
yes, I searched for numerous solutions but it didn't get my problem solved.
like from this link
https://www.learndatasci.com/solutions/python-typeerror-list-indices-must-be-integers-or-slices-not-str/#:~:text=This%20type%20error%20occurs%20when,using%20the%20int()%20function.
if u want to want the full file for the code ask me ok. it is a work in progress
Data is a list and not a dictionary so you use zero based index to enumerate the items in the list. You could enumerate the items via a for loop like this:
def data_change(data):
ans = []
for account in data:
data_name = account["name"]
data_desc = account["description"]
data_country = account["country"]
ans.append(f"{data_name}, is a {data_desc}, from {data_country}")
return "\n".join(ans)
Your variable data is a list with two dictionaries in it.
With data_name = data["name"] you try to access this list but name is not an integer, so you get the TypeError.
Your list only has two entries, so you can access it with data[0]["name"] or data[1]["name"].
Edit: Iterating over the dicts in your list as Mazimia stated, seems like a good idea.

how do I build dictionary from string

I am trying to build dictionary from string can someone help me to build the efficient way.
I have api which takes string with comma separate value and returns if value if API has matching values
example:
# my paramaters to my API car, bike, moter-cycle, airplane, boat
# www.api.com?words=car,bike,moter-cycle,airplane,boat
field_json = json.loads(response.text)
data = field_json['response']
list1=""
for i in data['docs']:
list[i['keyword_s']]= i['url_s']
list1 = str(list)
print(list)
return list1
from above I just get
{'bike':'http://bikes.com/bikes',
'boat':'http://boat.com/boat'}
if API find bike and boat as matching I would need my dictonary as
{'car':'none',
'bike':'http://bikes.com/bikes',
'moter-cycle': 'none',
'airplane':'none',
'boat':'http://boat.com/boat'
}
You don't need to involve strings, just fill in hole values in your returned api response
dict_a = {'bike':'http://bikes.com/bikes', 'boat':'http://boat.com/boat'}
keywords = ['car','bike','motorcycle', 'airplane', 'boat']
dict_b = {k: None for k in keywords}
dict_b.update(dict_a)
print(dict_b)
Output
{'car': None, 'bike': 'http://bikes.com/bikes', 'motorcycle': None, 'airplane': None, 'boat': 'http://boat.com/boat'}

Getting TypeError: list indices must be integers, not str when trying to read data from dictionary

In below program, I have "data" which contains a dictionary and list. I am trying to read the value of "role" but getting the error:
import json
data = {"users":[{"user_id":"11w","device_id":"AQ","role":"OWN","links":{"rel":"self","href":"test_link"}}]}
k= json.loads(data)
role= k["users"]["role"]
print role
Error : TypeError: list indices must be integers, not str
You have a dictionary which contains a list of dictionaries, not a dictionary which contains a dictionary. And lists can only be indexed with integers (and slices, but they also contain integers).
To get the role of the first (and only one in this example) user, just use this:
role = k["users"][0]["role"]
print role
# OWN
Or, if you have multiple users, iterate over them:
for user in k["users"]:
print user["role"]
Looking at this line by line:
data = {"users":[{"user_id":"11w","device_id":"AQ","role":"OWN","links":{"rel":"self","href":"test_link"}}]}
data now holds a dictionary of a list of a dictionary.
k = json.loads(data)
Gives a json TypeError as json.loads needs to be passed a string, not a python data structure like data.
role = k["users"]["role"]
As you have discovered this doesn't work. Let's find the data step by step:
print(data["users"])
[{'user_id': '11w', 'device_id': 'AQ', 'role': 'OWN', 'links': {'rel': 'self', 'href': 'test_link'}}]
Note that this is a list ("[...]") not a dictionary. To access this list members you use integers, not strings. So next step extract the first (and only) member of this list:
print(data["users"][0])
{'user_id': '11w', 'device_id': 'AQ', 'role': 'OWN', 'links': {'rel': 'self', 'href': 'test_link'}}
Now we have the nested dictionary which we can lookup by key string:
print(data["users"][0]["role"])
'OWN'
Finally we have the answer you are looking for.

Python efficiently getting a specific value from a list of dictionaries

I am using an API to get results from a database that is returned in JSON format. I'm then trying to just extract certain keys/values and append them to a blank list. The field I want is eissn and the API returns results like this:
[
{'type': 'doi', 'id': '10.16472/j.chinatobacco.2017.045'},
{'type': 'pissn', 'id': '1004-5708'},
{'type': 'eissn', 'id': '1004-5708'}
]
I want to extract the eissn id which i have tried using this code:
deissnlist = []
for i in range(0,lengthD):
deissnlist.append(doajresult[i]['bibjson']['identifier'][3]
['id'])
Where lengthD/i are the number of results returned from the API. The problem is that the eissn ID is not always the 3rd dictionary in the list. Is there any way I can search the tuples and only get the eissn ID without using another for loop (and thus going from O(n) to O(n^2)? The expected output is just the single value of the eissn id (in this case 1004-5708)
I tried converting the list to a dictionary but that didn't work because it treated each tuple as it's own dictionary. Thanks!
I may have misunderstood question and over simplified; but, if you are looking to simply extract a list of the ids from that sample input, this is how I would do it:
input = [
{'type': 'doi', 'id': '10.16472/j.chinatobacco.2017.045'},
{'type': 'pissn', 'id': '1004-5708'},
{'type': 'eissn', 'id': '1004-5708'}
]
result = [x.get('id') for x in input]
print(result)
['10.16472/j.chinatobacco.2017.045', '1004-5708', '1004-5708']
Or for just the ids where type is eissn:
result = [x.get('id') for x in input if x.get('type') == 'eissn']

Categories

Resources