I've trying to convert list to tuples to customized list to dictionaries. I'm able toto divide admin, pack owner, submitter, consumer and read only. Please check below code and output
from collections import defaultdict
role_details = ['po','sub', 'cons', 'admin','read']
lst = [('name1', 'email1', 'psid1', 'new1', 11, '1', 'po'),
('name2', 'email2', 'psid2', 'new2', 12, '2', 'sub'),
('name3', 'email3', 'psid3', 'new3', 13, '3', 'sub'),
('name4', 'email4', 'psid4', 'new4', 14, '4', 'po'),
('name5', 'email5', 'psid5', 'new5', 15, '5', 'cons')]
by_role = defaultdict(list)
for name, email, psid, new, id1, id2, role_name in lst:
by_role[role_name].append({"name": name, "email": email, "psid": psid})
print({"add_sub": dict(by_role)})
output:
{'add_sub':
{'po': [{'name': 'name1', 'email': 'email1', 'psid': 'psid1'}, {'name': 'name4', 'email': 'email4', 'psid': 'psid4'}],
'sub': [{'name': 'name2', 'email': 'email2', 'psid': 'psid2'}, {'name': 'name3', 'email': 'email3', 'psid': 'psid3'}],
'cons': [{'name': 'name5', 'email': 'email5', 'psid': 'psid5'}]
}
}
but if any of the role details record not exit in the lst i'm trying to pass an empty list. Please check the expected output
expected output:
{'add_sub':
{'admin': [],
'po': [{'name': 'name1', 'email': 'email1', 'psid': 'psid1'}, {'name': 'name4', 'email': 'email4', 'psid': 'psid4'}],
'read': [],
'sub': [{'name': 'name2', 'email': 'email2', 'psid': 'psid2'}, {'name': 'name3', 'email': 'email3', 'psid': 'psid3'}],
'cons': [{'name': 'name5', 'email': 'email5', 'psid': 'psid5'}]
}
}
I'm new to python plz suggest me with logic. thank you
You can also create the dict with all the keys you need beforehand. In this case you don't even need the defaultdict, unless you need its functionality later.
#from collections import defaultdict
role_details = ['pack owner','submitter', 'consumer', 'admin','read only']
lst = [('name1', 'email1', 'psid1', 'new1', 11, '1', 'pack owner'),
('name2', 'email2', 'psid2', 'new2', 12, '2', 'submitter'),
('name3', 'email3', 'psid3', 'new3', 13, '3', 'submitter'),
('name4', 'email4', 'psid4', 'new4', 14, '4', 'pack owner'),
('name5', 'email5', 'psid5', 'new5', 15, '5', 'consumer')]
#by_role = defaultdict(list)
by_role = {k: [] for k in role_details}
for name, email, psid, new, id1, id2, role_name in lst:
by_role[role_name].append({"name": name, "email": email, "psid": psid})
print({"add_sub": by_role})
{'add_sub':
{'pack owner': [{'name': 'name1', 'email': 'email1', 'psid': 'psid1'}, {'name': 'name4', 'email': 'email4', 'psid': 'psid4'}],
'submitter': [{'name': 'name2', 'email': 'email2', 'psid': 'psid2'}, {'name': 'name3', 'email': 'email3', 'psid': 'psid3'}],
'consumer': [{'name': 'name5', 'email': 'email5', 'psid': 'psid5'}],
'admin': [],
'read only': []}
}
Also note that you don't need the dict() call in the print, since by_role is already a dict.
If you do need the defaultdict, you can do the same thing with this line:
from collections import defaultdict
# ...code...
by_role = defaultdict(list, {k: [] for k in role_details})
by_role is a defaultdict(list), so you'll get an empty list if you try to access the by_role['admin'] or by_role['read only'].
In fact, all you need to do is try to access those keys once, and they get added to the defaultdict, so you can iterate over role_details and do that:
for name, email, psid, new, id1, id2, role_name in lst:
by_role[role_name].append({"name": name, "email": email, "psid": psid})
for role_name in role_details:
_ = by_role[role_name] # Try to access every key, and don't do anything with it.
Then, you should have your expected output:
{'add_sub': {
'pack owner': [
{'name': 'name1', 'email': 'email1', 'psid': 'psid1'},
{'name': 'name4', 'email': 'email4', 'psid': 'psid4'}
],
'submitter': [
{'name': 'name2', 'email': 'email2', 'psid': 'psid2'},
{'name': 'name3', 'email': 'email3', 'psid': 'psid3'}
],
'consumer': [{'name': 'name5', 'email': 'email5', 'psid': 'psid5'}],
'admin': [],
'read only': []
}
}
Related
I have this list of dictionary and I would like to get those with the same exact value of 'name' and 'school' into a new list and also getting their 'age' merged into a list as well and the rest of the dictionary that is not identical to just add into the list as per usual..
Here is an example of the list of dictionary
[{'name': 'Jane', 'age':12, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'SMU'},{'name': 'Jane', 'age':14, 'school': 'SIT'}, {'name': 'Jane', 'age':16, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'NUS'}]
and I would like it to make it into something like this..
[{'name': 'Jane', 'age': [12,14,16], 'school': 'SIT'}, {'name': 'John', 'age': 13, 'school': 'SMU'}, {'name': 'John', 'age':13, 'school': 'NUS'}]
using Python.. please help!
tried using counter, loops but still can't get it to work..
You could use itertools.groupby().
Example:
import itertools
from pprint import pprint
data = [{'name': 'Jane', 'age':12, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'SMU'},{'name': 'Jane', 'age':14, 'school': 'SIT'}, {'name': 'Jane', 'age':16, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'NUS'}]
keyfunc = lambda x: (x["name"], x["school"])
# needs to be sorted to use groupby
data.sort(key=keyfunc)
output = []
for k,v in itertools.groupby(data, key=keyfunc):
this_group = {
"name": k[0],
"school": k[1],
"age": [i["age"] for i in v],
}
output.append(this_group)
pprint(output)
The output is:
[{'age': [12, 14, 16], 'name': 'Jane', 'school': 'SIT'},
{'age': [13], 'name': 'John', 'school': 'NUS'},
{'age': [13], 'name': 'John', 'school': 'SMU'}]
If you wish to go with the solution based on a buffer dictionary, please check out the dict.setdefault() method.
Example:
buffer = {}
for i in data:
buffer.setdefault((i["name"], i["school"]), []).append(i["age"])
For reference:
https://docs.python.org/3/library/itertools.html#itertools.groupby
https://docs.python.org/3/library/stdtypes.html#dict.setdefault
x = [{'name': 'Jane', 'age':12, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'SMU'},{'name': 'Jane', 'age':14, 'school': 'SIT'}, {'name': 'Jane', 'age':16, 'school': 'SIT'}, {'name': 'John', 'age':13, 'school': 'NUS'}]
new_x = {}
for r in x:
if r['name'] in new_x.keys():
if not isinstance(new_x[r['name']]['age'], list):
new_x[r['name']]['age'] = [new_x[r['name']]['age']]
if r['age'] not in new_x[r['name']]['age']:
new_x[r['name']]['age'].append(r['age'])
else:
new_x[r['name']] = {'age': r['age'], 'school': r['school']}
z = [v.update(name=k) for k, v in new_x.items()]
z = [v for k, v in new_x.items()]
Here is a universal solution to your problem. Only name and school are considered "special". All other keys, like age are converted to list when a new value has to be added.
l = [
{"name": "Jane", "age": 12, "school": "SIT"},
{"name": "John", "age": 13, "school": "SMU"},
{"name": "Jane", "age": 14, "school": "SIT"},
{"name": "Jane", "age": 16, "school": "SIT"},
{"name": "John", "age": 13, "school": "NUS"},
]
r = {}
for x in l:
id = f"{x['name']}-{x['school']}"
if id in r:
for k,v in x.items():
if k not in ["name", "school"]:
if k in r[id]:
if isinstance(r[id][k], list):
r[id][k].append(v)
else:
r[id][k] = [r[id][k], v]
else:
r[id][k] = v
else:
r[id] = x
result = [x for x in r.values()]
I have dictionary below
test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product' },
{ 'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product' },
{ 'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester' },
{ 'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester' },
{ 'id': '230', 'Name': 'abc', 'Email': 'deyan#gmail.com', 'role': 'Tester' },
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' },
{ 'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester' } ]
I need to extract unique email from above list dict
I need to give give role preference Product then to Tester
My Code is below
dict([(d['Email'], d) for d in test]).values()
My Out:
dict_values([{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'},
{'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester'}])
Here in my out
{'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester'}
has to replace with
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' }
because "Product" have higher preference.
How to update my code? dict([(d['Email'], d) for d in test]).values()
Here is in case you would like to insist on using dictionaries.
We go from one row to another. Check if the email is already in the new dictionary as key.
If not, we add this as a new one.
If so, we check our new row. If our new role is "product", we will delete what was already in the dictionary, and add the new row.
new_dict = {}
for row in test:
if row["Email"] not in new_dict.keys():
new_dict.update({row["Email"]: row})
else:
if row["role"]=="Product":
new_dict.pop(row["Email"])
new_dict.update({row["Email"]: row})
Perhaps you could try it with two loops; once to get the unique emails, and second time to make sure to prioritize "Product".
It wasn't clear what happens if there is no "Product" for duplicate "Emails", so in the loop below, the first email is selected in that case.
tmp = {}
for d in test:
tmp.setdefault(d['Email'], []).append(d)
out = []
for k, lst in tmp.items():
if len(lst) == 1:
out.append(lst[0])
else:
for d in lst:
if d['role'] == 'Product':
out.append(d)
break
else:
out.append(lst[0])
Output:
[{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'Account': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'Account': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'},
{'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product'}]
Make it to a data frame and drop_duplicates by Email after sorting the column role.
test = [ { 'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product' },
{ 'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product' },
{ 'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester' },
{ 'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester' },
{ 'id': '230', 'Name': 'abc', 'Email': 'deyan#gmail.com', 'role': 'Tester' },
{ 'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product' },
{ 'id': '230', 'Name': 'Sn', 'Email': 'deyan#gmail.com', 'role': 'Tester' } ]
df = pd.DataFrame(test)
df1 = df.sort_values(by = ["Email", "role"], ascending = True)
res_df = df1.drop_duplicates(["Email"])
output_list = []
for i in res_df.values :
output_list.append(dict([("id", i[0]), ("Name", i[1]), ("Email", i[2]), ("role", i[3])]))
> output_list
[{'id': '195', 'Name': 'i', 'Email': 'chdtn#gmail.com', 'role': 'Product'},
{'id': '219', 'Name': 'umar', 'Email': 'ddhi#gmail.com', 'role': 'Product'},
{'id': '74', 'Name': 'Are', 'Email': 'ddhit#gmail.com', 'role': 'Tester'},
{'id': '220', 'Name': 'Sc', 'Email': 'deyan#gmail.com', 'role': 'Product'},
{'id': '24', 'Name': 'Mee', 'Email': 'huul#gmail.com', 'role': 'Tester'}]
I am trying to figure out how to filter for the dictionaries that have a status of "awaiting_delivery". I am not sure how to do this (or if it is impossible). I am new to python and programming. I am using Python 3.8.5 on VS Code on Ubuntu 20.04. The data below is sample data that I created that resembles json data from an API. Any help on how to filter for "status" would be great. Thank you.
nested_dict = {
'list_data': [
{
'id': 189530,
'total': 40.05,
'user_data': {
'id': 1001,
'first_name': 'jane',
'last_name': 'doe'
},
'status': 'future_delivery'
},
{
'id': 286524,
'total': 264.89,
'user_data': {
'id': 1002,
'first_name': 'john',
'last_name': 'doe'
},
'status': 'awaiting_delivery'
},
{
'id': 368725,
'total': 1054.98,
'user_data': {
'id': 1003,
'first_name': 'chris',
'last_name': 'nobody'
},
'status': 'awaiting_delivery'
},
{
'id': 422955,
'total': 4892.78,
'user_data': {
'id': 1004,
'first_name': 'mary',
'last_name': 'madeup'
},
'status': 'future_delivery'
}
],
'current_page': 1,
'total': 2,
'first': 1,
'last': 5,
'per_page': 20
}
#confirm that nested_dict is a dictionary
print(type(nested_dict))
#create a list(int_list) from the nested_dict dictionary
int_list = nested_dict['list_data']
#confirm that int_list is a list
print(type(int_list))
#create the int_dict dictionary from the int_list list
for int_dict in int_list:
print(int_dict)
#this is my attempt at filtering the int_dict dictionar for all orders with a status of awaiting_delivery
for order in int_dict:
int_dict.get('status')
print(order)
Output from Terminal Follows:
<class 'dict'>
<class 'list'>
{'id': 189530, 'total': 40.05, 'user_data': {'id': 1001, 'first_name': 'jane', 'last_name': 'doe'}, 'status': 'future_delivery'}
{'id': 286524, 'total': 264.89, 'user_data': {'id': 1002, 'first_name': 'john', 'last_name': 'doe'}, 'status': 'awaiting_delivery'}
{'id': 368725, 'total': 1054.98, 'user_data': {'id': 1003, 'first_name': 'chris', 'last_name': 'nobody'}, 'status': 'awaiting_delivery'}
{'id': 422955, 'total': 4892.78, 'user_data': {'id': 1004, 'first_name': 'mary', 'last_name': 'madeup'}, 'status': 'future_delivery'}
id
total
user_data
status
You can obtain a filtered list of dicts by doing conditional list comprehension on your list of dicts:
# filter the data
list_data_filtered = [entry for entry in nested_dict['list_data']
if entry['status'] == 'awaiting_delivery']
# print out the results
for entry in list_data_filtered:
print(entry)
# results
# {'id': 286524, 'total': 264.89, 'user_data': {'id': 1002, 'first_name': 'john', 'last_name': 'doe'}, 'status': 'awaiting_delivery'}
# {'id': 368725, 'total': 1054.98, 'user_data': {'id': 1003, 'first_name': 'chris', 'last_name': 'nobody'}, 'status': 'awaiting_delivery'}
I have a dictionary as follows:
a = {'name': 'Test', 'lastName': 'Test', 'scores': ['1', '2'], 'subjects': ['te','re'] }
I have tried nested loops, but I'm not sure if that's the best approach.
As an output I need a list of dictionaries for each score and subject :
result1 = { 'name':'Test', 'lastName': 'Test', 'score': '1', 'subjects': 'te'}
result2 = { 'name':'Test', 'lastName': 'Test', 'score': '2', 'subjects': 're'}
How to best iterate through the lists and create such dictionary? The number of scores and subjects will always match.
Any help would be appreciated.
Here is a function which unzip your dictionary.
We first use next to find some list value in the dictionary, its length is the expected output size.
Note that this will fail if the dictionary contains no list at all.
def unzip_dict(d):
# Find one of the list in the dictionary to read its length
length = len(next(value for value in d.values() if isinstance(value, list)))
output = []
# Unzip the dictionary
for i in range(length):
output.append({k: v[i] if isinstance(v, list) else v for k, v in d.items()})
return output
a = {'name': 'Test', 'lastName': 'Test', 'scores': ['1', '2'], 'subjects': ['te', 're']}
print(unzip_dict(a))
Output
[{'lastName': 'Test', 'name': 'Test', 'scores': '1', 'subjects': 'te'},
{'lastName': 'Test', 'name': 'Test', 'scores': '2', 'subjects': 're'}]
Try this:
# setup data
a = {'name': 'Test', 'lastName': 'Test', 'scores': ['1', '2'], 'subjects': ['te','re'] }
# create list of dictionary
out_list = []
for value in a.get('scores'):
for item in a.get('subjects'):
out_list.append({'name': 'Test', 'lastName': 'Test', 'scores':value, 'subjects':item})
Output:
{'name': 'Test', 'lastName': 'Test', 'scores': '1', 'subjects': 'te'}
{'name': 'Test', 'lastName': 'Test', 'scores': '1', 'subjects': 're'}
{'name': 'Test', 'lastName': 'Test', 'scores': '2', 'subjects': 'te'}
{'name': 'Test', 'lastName': 'Test', 'scores': '2', 'subjects': 're'}
You don't need nested for loops, a single for loop is sufficient:
def foo(a):
finarr=[]
for i in range(len(a['scores'])):
fnarr={}
fnarr['name']=a['name']
fnarr['lastName']=a['lastName']
fnarr['score']=a['scores'][i]
fnarr['subject']=a['subjects'][i]
finarr.append(fnarr)
return finarr
Output:
[{'name': 'Test', 'lastName': 'Test', 'score': '1', 'subject': 'te'},
{'name': 'Test', 'lastName': 'Test', 'score': '2', 'subject': 're'}]
you can try this:
res1 = {}
res2= {}
for k,v in a.items():
if(k == "scores"):
res1[k] = v[0]
res2[k] = v[1]
elif(k=="subjects"):
res1[k] = v[0]
res2[k] = v[1]
else:
res1[k] = v
res2[k] = v
print(res1)
print(res2)
you can also take a look to defaultdict i think that would help you on your Task
You can utilize zip to attach scores and subjects to each other and then add it to a new list.
d = {'name': 'Test', 'lastName': 'Test', 'scores': ['1', '2'], 'subjects': ['te','re'] }
template = {'name': d['name'], 'lastName': d['lastName']}
res = []
for subject, score in zip(d['subjects'], d['scores']):
template.update({'subjects': subject, 'scores': score})
res.append(template)
print(res)
I have a data in a list like below:
[
{'id': 1, 'first_name': 'Jeanette', 'last_name': 'Penddreth', 'email': 'jpenddreth0#census.gov', 'gender': 'Female', 'ip_address': '26.58.193.2'},
{'id': 2, 'first_name': 'Giavani', 'last_name': 'Frediani', 'email': 'gfrediani1#senate.gov', 'gender': 'Male', 'ip_address': '229.179.4.212'},
{'id': 3, 'first_name': 'Noell', 'last_name': 'Bea', 'email': 'nbea2#imageshack.us', 'gender': 'Female', 'ip_address': '180.66.162.255'},
{'id': 4, 'first_name': 'Willard', 'last_name': 'Valek', 'email': 'wvalek3#vk.com', 'gender': 'Male', 'ip_address': '67.76.188.26'}
]
I am loading the data into the dynamoDb. It is failing with the error type: <class 'list'>, valid types: <class 'dict'>: ParamValidationError.
How do I convert the above list into a dictionary?
EDIT Code used:
import boto3
import json
s3_client=boto3.client('s3')
dynamodb=boto3.resource('dynamodb')
def lambda_handler(event, context):
bucket=event['Records'][0]['s3']['bucket']['name']
json_filename=event['Records'][0]['s3']['object']['key']
json_object=s3_client.get_object(Bucket=bucket,Key=json_filename)
jsonFileReader=json_object['Body'].read()
jsonDictionary=json.loads(jsonFileReader)
table=dynamodb.Table('EMPLOYEE_DETAILS')
table.put_item(Item=jsonDictionary)
return 'Done'
I'm not familiar with dynamodb, but I imagine this will work. If your JSON is a list, then you need to iterate through the items in your list, adding each one to the table.
Replace the line:
table.put_item(Item=jsonDictionary)
with:
if type(jsonDictionary) == type([]):
# It is a list - iterate through it
for item in jsonDictionary:
table.put_item(Item=item)
else:
table.put_item(Item=jsonDictionary)