Develop Reference

How get extract unique dictionary from list of dictionary with preference of value

I have dictionary below
test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product' },
{ 'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product' },
{ 'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester' },
{ 'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester' },
{ 'id': '230', 'Name': 'abc', 'Email': 'deyan#gmail.com', 'role': 'Tester' },
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' },
{ 'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester' } ]
I need to extract unique email from above list dict
I need to give give role preference Product then to Tester
My Code is below
dict([(d['Email'], d) for d in test]).values()
My Out:
dict_values([{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'},
{'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester'}])
Here in my out
{'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester'}
has to replace with
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' }
because "Product" have higher preference.
How to update my code? dict([(d['Email'], d) for d in test]).values()

Here is in case you would like to insist on using dictionaries.
We go from one row to another. Check if the email is already in the new dictionary as key.
If not, we add this as a new one.
If so, we check our new row. If our new role is "product", we will delete what was already in the dictionary, and add the new row.
new_dict = {}
for row in test:
if row["Email"] not in new_dict.keys():
new_dict.update({row["Email"]: row})
else:
if row["role"]=="Product":
new_dict.pop(row["Email"])
new_dict.update({row["Email"]: row})

Perhaps you could try it with two loops; once to get the unique emails, and second time to make sure to prioritize "Product".
It wasn't clear what happens if there is no "Product" for duplicate "Emails", so in the loop below, the first email is selected in that case.
tmp = {}
for d in test:
tmp.setdefault(d['Email'], []).append(d)
out = []
for k, lst in tmp.items():
if len(lst) == 1:
out.append(lst[0])
else:
for d in lst:
if d['role'] == 'Product':
out.append(d)
break
else:
out.append(lst[0])
Output:
[{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'Account': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'Account': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'},
{'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product'}]

Make it to a data frame and drop_duplicates by Email after sorting the column role.
test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product' },
{ 'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product' },
{ 'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester' },
{ 'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester' },
{ 'id': '230', 'Name': 'abc', 'Email': 'deyan#gmail.com', 'role': 'Tester' },
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' },
{ 'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester' } ]
df = pd.DataFrame(test)
df1 = df.sort_values(by = ["Email", "role"], ascending = True)
res_df = df1.drop_duplicates(["Email"])
output_list = []
for i in res_df.values :
output_list.append(dict([("id", i[0]), ("Name", i[1]), ("Email", i[2]), ("role", i[3])]))
> output_list
[{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'}]

Remove item in JSON if key has value

I have tried everything I can possible come up with, but the value wont go away.
I have a JSON user and if user['permissions'] have key permission = "DELETE PAGE" remove that index of del user['permissions'][1] (in this example)
I want to have a list of possible values as "DELETE PAGE" and so on. If value in key, then delete that index.
Then return the users json without those items found.
I have tried del user['permission][x] and .pop() and so on but it is still there.
{
'id': 123,
'name': 'My name',
'description': 'This is who I am',
'permissions': [{
'id': 18814,
'holder': {
'type': 'role',
'parameter': '321',
'projectRole': {
'name': 'Admin',
'id': 1,
}
},
'permission': 'VIEW PAGE'
}, {
'id': 18815,
'holder': {
'type': 'role',
'parameter': '123',
'projectRole': {
'name': 'Moderator',
'id': 2
}
},
'permission': 'DELETE PAGE'
}]
}

Here's the code:
perm = a['permissions']
for p in perm:
if p['permission'] == 'DELETE PAGE':
perm.remove(p)
print(a)
Output:
{'id': 123, 'name': 'My name', 'description': 'This is who I am', 'permissions': [{'id': 18814, 'holder': {'type': 'role', 'parameter': '321', 'projectRole': {'name': 'Admin', 'id': 1}}, 'permission': 'VIEW PAGE'}]}

How do I access a specific value in a nested Python dictionary?

I am trying to figure out how to filter for the dictionaries that have a status of "awaiting_delivery". I am not sure how to do this (or if it is impossible). I am new to python and programming. I am using Python 3.8.5 on VS Code on Ubuntu 20.04. The data below is sample data that I created that resembles json data from an API. Any help on how to filter for "status" would be great. Thank you.
nested_dict = {
'list_data': [
{
'id': 189530,
'total': 40.05,
'user_data': {
'id': 1001,
'first_name': 'jane',
'last_name': 'doe'
},
'status': 'future_delivery'
},
{
'id': 286524,
'total': 264.89,
'user_data': {
'id': 1002,
'first_name': 'john',
'last_name': 'doe'
},
'status': 'awaiting_delivery'
},
{
'id': 368725,
'total': 1054.98,
'user_data': {
'id': 1003,
'first_name': 'chris',
'last_name': 'nobody'
},
'status': 'awaiting_delivery'
},
{
'id': 422955,
'total': 4892.78,
'user_data': {
'id': 1004,
'first_name': 'mary',
'last_name': 'madeup'
},
'status': 'future_delivery'
}
],
'current_page': 1,
'total': 2,
'first': 1,
'last': 5,
'per_page': 20
}
#confirm that nested_dict is a dictionary
print(type(nested_dict))
#create a list(int_list) from the nested_dict dictionary
int_list = nested_dict['list_data']
#confirm that int_list is a list
print(type(int_list))
#create the int_dict dictionary from the int_list list
for int_dict in int_list:
print(int_dict)
#this is my attempt at filtering the int_dict dictionar for all orders with a status of awaiting_delivery
for order in int_dict:
int_dict.get('status')
print(order)
Output from Terminal Follows:
<class 'dict'>
<class 'list'>
{'id': 189530, 'total': 40.05, 'user_data': {'id': 1001, 'first_name': 'jane', 'last_name': 'doe'}, 'status': 'future_delivery'}
{'id': 286524, 'total': 264.89, 'user_data': {'id': 1002, 'first_name': 'john', 'last_name': 'doe'}, 'status': 'awaiting_delivery'}
{'id': 368725, 'total': 1054.98, 'user_data': {'id': 1003, 'first_name': 'chris', 'last_name': 'nobody'}, 'status': 'awaiting_delivery'}
{'id': 422955, 'total': 4892.78, 'user_data': {'id': 1004, 'first_name': 'mary', 'last_name': 'madeup'}, 'status': 'future_delivery'}
id
total
user_data
status

You can obtain a filtered list of dicts by doing conditional list comprehension on your list of dicts:
# filter the data
list_data_filtered = [entry for entry in nested_dict['list_data']
if entry['status'] == 'awaiting_delivery']
# print out the results
for entry in list_data_filtered:
print(entry)
# results
# {'id': 286524, 'total': 264.89, 'user_data': {'id': 1002, 'first_name': 'john', 'last_name': 'doe'}, 'status': 'awaiting_delivery'}
# {'id': 368725, 'total': 1054.98, 'user_data': {'id': 1003, 'first_name': 'chris', 'last_name': 'nobody'}, 'status': 'awaiting_delivery'}

Changing JSON to Dataframe in python [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I want to change the API json response to dataframe by making columns under data to dataframe. Note it also has some nested parameters under data (message) I want to make it individual columns.
{
'success': True,
'code': 200,
'data': [
{
'id': 342964769,
'type': 'ios',
'create_time': 1567591650,
'open_count': 2,
'environment': 'production',
'campaign_id': 12713145,
'project_id': 1758,
'error': 0,
'sent_count': 3,
'message': {
'timestamp': '1567591643',
'badge': '',
'alert': "'I pulled pints and cut turf here back in the day' - Mike Pence speaks to "
"small crowd in Doonbeg",
'sound': 'default',
'articleId': '38465289',
'category': 'news/',
'id': '342964769',
'pushId': 'fireabse-5d6f8cdb8c3e9',
'title': 'Independent.ie',
'content-available': '1',
'xpush': 'yes',
'cid': '12713145'
},
'error_message': None
}, {
'id': 342964771,
'type': 'android',
'create_time': 1567591650,
'open_count': 0,
'environment': 'production',
'campaign_id': 12713145,
'project_id': 1758,
'error': 0,
'sent_count': 0,
'message': None,
'error_message': None
}
]
}

It's what you want?
def dict_pop(d, *args):
v = d.pop(*args)
return v if v else {}
resp = [{**dict_pop(i,'message'), **i} for i in resp['data']]
resp = pd.DataFrame(resp)

You can flatten the dictionaries by removing the message level and making each entry of the dictionary part of the parent dict:
import pandas as pd
import copy
data = {
'success': True,
'code': 200,
'data': [
{
'id': 342964769,
'type': 'ios',
'create_time': 1567591650,
'open_count': 2,
'environment': 'production',
'campaign_id': 12713145,
'project_id': 1758,
'error': 0,
'sent_count': 3,
'message': {
'timestamp': '1567591643',
'badge': '',
'alert': "'I pulled pints and cut turf here back in the day' - Mike Pence speaks to "
"small crowd in Doonbeg",
'sound': 'default',
'articleId': '38465289',
'category': 'news/',
'id': '342964769',
'pushId': 'fireabse-5d6f8cdb8c3e9',
'title': 'Independent.ie',
'content-available': '1',
'xpush': 'yes',
'cid': '12713145'
},
'error_message': None
}, {
'id': 342964771,
'type': 'android',
'create_time': 1567591650,
'open_count': 0,
'environment': 'production',
'campaign_id': 12713145,
'project_id': 1758,
'error': 0,
'sent_count': 0,
'message': None,
'error_message': None
}
]
}
processed = []
for dat in data["data"]:
new_dat = copy.deepcopy(dat) # only important if the original data matters to you
if "message" in new_dat and new_dat["message"]:
message = new_dat.pop("message")
new_dat.update(message)
processed.append(new_dat)
df = pd.DataFrame(processed)
print(df.columns)
Output:
Index(['id', 'type', 'create_time', 'open_count', 'environment', 'campaign_id',
'project_id', 'error', 'sent_count', 'error_message', 'timestamp',
'badge', 'alert', 'sound', 'articleId', 'category', 'pushId', 'title',
'content-available', 'xpush', 'cid', 'message'],
dtype='object')

How to build a recursive dictionary tree from an ordered adjacency list

I've been trying to figure this out all day and Im at my wits end. Maybe I'm just getting to old for this.
I'm trying to build a tree for the load_bulk feature on django-treebeard as specified here
To save you looking, it should look like this:
data = [{'data':{'desc':'1'}},
{'data':{'desc':'2'}, 'children':[
{'data':{'desc':'21'}},
{'data':{'desc':'22'}},
{'data':{'desc':'23'}, 'children':[
{'data':{'desc':'231'}},
]},
{'data':{'desc':'24'}},
]},
{'data':{'desc':'3'}},
{'data':{'desc':'4'}, 'children':[
{'data':{'desc':'41'}},
]},
]
'data' holds the record, and if it has children, 'children' is a list of more 'data' dicts (that can also contain a list of children and so on recursively)
I get the data as an ordered list (ordered as in depth first, not by id):
e.g:
[
{'id': 232, 'name': 'jon', 'parent': 'None'}
{'id': 3522, 'name': 'dave', 'parent': '232'}
{'id': 2277, 'name': 'alice', 'parent': '3522'}
{'id': 119, 'name': 'gary', 'parent': '232'}
{'id': 888, 'name': 'gunthe', 'parent': '119'}
{'id': 750, 'name': 'beavis', 'parent': 'None'}
{'id': 555, 'name': 'urte', 'parent': '750'}
]
How can I transform it into a treebeard compliant dictionary that would look like this (typo's excepted):
[
{'data': {'id': 232, 'name': 'jon', 'parent': 'None'},
'children': [
{'data': {'id': 3522, 'name': 'dave', 'parent': '232'},
'children': [
{'data': {'id': 2277, 'name': 'alice', 'parent': '3522'}}
]
}
{'data': {'id': 119, 'name': 'gary', 'parent': '232'},
'children': [
{'id': 888, 'name': 'gunthe', 'parent': '119'}
]
}
]
{'data': {'id': 750, 'name': 'beavis', 'parent': 'None'},
'children': [
{'id': 555, 'name': 'urte', 'parent': '750'}
]
}
]
I guess I need some kind of recursion function seeing as its a recursive structure but all my attempts have failed. My brain doesnt do recursion so good.
I did a lot of searching and found mostly solutions pertaining to lists or other structures that i cant mould to fit. I'm a relative noob. ps i had more fun manually typing out the example than i did the rest of day (apart from dinner time).

Maybe there are better ways, but here is one solution:
users = [
{
'id': 232,
'name': 'jon',
'parent': None
},
{
'id': 3522,
'name': 'dave',
'parent': 232
},
{
'id': 2277,
'name': 'alice',
'parent': 3522
},
{
'id': 119,
'name': 'gary',
'parent': 232
},
{
'id': 888,
'name': 'gunthe',
'parent': 119
},
{
'id': 750,
'name': 'beavis',
'parent': None
},
{
'id': 555,
'name': 'urte',
'parent': 750
}
]
users_map = {}
for user in users:
users_map[user['id']] = user
users_tree = []
for user in users:
if user['parent'] is None:
users_tree.append(user)
else:
parent = users_map[user['parent']]
if 'childs' not in parent:
parent['childs'] = []
parent['childs'].append(user)
print(users_tree)
#user as {data: user, childs: []}
users_map = {}
for user in users:
users_map[user['id']] = {'data': user, 'childs': []}
users_tree = []
for user in users:
if user['parent'] is None:
users_tree.append(users_map[user['id']])
else:
parent = users_map[user['parent']]
parent['childs'].append(users_map[user['id']])
print(users_tree)