To be honest, it's too easy for me to make in JS or Perl, but i've completely stuck with that in Python because of coplexed tools for dealing with dicts/lists. So, what i need:
i have an array of dicts:
[
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_1", "k_id": 4},
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_2", "k_id": 5},
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_3", "k_id": 6},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_1", "k_id": 7},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_2", "k_id": 8},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_3", "k_id": 9},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_4", "k_id": 10},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_1", "k_id": 11},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_2", "k_id": 12},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_3", "k_id": 13},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_4", "k_id": 14}
]
and i need to make that:
[
{
"id": 1,
"name": "Res1",
"type": "resource",
"indicators": [
{"name": "Ind1_1","id": 4},
{"name": "Ind1_2","id": 5},
{"name": "Ind1_3","id": 6}
]
},
{
"id": 2,
"name": "Res2",
"type": "service",
"indicators": [
{"name": "Ind2_1","id": 7},
{"name": "Ind2_2","id": 8},
{"name": "Ind2_3","id": 9},
{"name": "Ind2_4","id": 10}
]
},
{
"id": 3,
"name": "Res3",
"type": "service",
"indicators": [
{"name": "Ind3_1","id": 11},
{"name": "Ind3_2","id": 12},
{"name": "Ind3_3","id": 13},
{"name": "Ind3_4","id": 14}
]
}
]
Can you help me with that?
itertools to the rescue:
import itertools
# Assuming your original list is `l`
# if it does not come in order, you need to do this line first, and will probably be less efficient.
l = sorted(l, key=lambda x:(x["id"], x["name"], x["type"]))
d = []
for k, g in itertools.groupby(l, lambda x: (x["id"], x["name"], x["type"])):
d.append({i:v for i, v in zip(["id", "name", "type"], k)})
d[-1]["indicator"] = [{y.split('_')[1]:e[y] for y in ["k_id", "k_name"]} for e in list(g)]
d becomes:
[{'id': 1,
'indicator': [{'id': 4, 'name': 'Ind1_1'},
{'id': 5, 'name': 'Ind1_2'},
{'id': 6, 'name': 'Ind1_3'}],
'name': 'Res1',
'type': 'resource'},
{'id': 2,
'indicator': [{'id': 7, 'name': 'Ind2_1'},
{'id': 8, 'name': 'Ind2_2'},
{'id': 9, 'name': 'Ind2_3'},
{'id': 10, 'name': 'Ind2_4'}],
'name': 'Res2',
'type': 'service'},
{'id': 3,
'indicator': [{'id': 11, 'name': 'Ind3_1'},
{'id': 12, 'name': 'Ind3_2'},
{'id': 13, 'name': 'Ind3_3'},
{'id': 14, 'name': 'Ind3_4'}],
'name': 'Res3',
'type': 'service'}]
You can use a mapping dict to map ids to corresponding sub-lists, so that as you iterate through the list (named l in this example), you can append a new entry to the output list if the id is not found in the mapping, or append the entry to the existing sub-list if id is found in the mapping:
mapping = {}
output = []
for d in l:
i = {'name': d.pop('k_name'), 'id': d.pop('k_id')}
if d['id'] in mapping:
mapping[d['id']].append(i)
else:
output.append({**d, 'indicators': [i]})
mapping[d['id']] = output[-1]['indicators']
output becomes:
[{'id': 1, 'name': 'Res1', 'type': 'resource', 'indicators': [{'name': 'Ind1_1', 'id': 4}, {'name': 'Ind1_2', 'id': 5}, {'name': 'Ind1_3', 'id': 6}]}, {'id': 2, 'name': 'Res2', 'type': 'service', 'indicators': [{'name': 'Ind2_1', 'id': 7}, {'name': 'Ind2_2', 'id': 8}, {'name': 'Ind2_3', 'id': 9}, {'name': 'Ind2_4', 'id': 10}]}, {'id': 3, 'name': 'Res3', 'type': 'service', 'indicators': [{'name': 'Ind3_1', 'id': 11}, {'name': 'Ind3_2', 'id': 12}, {'name': 'Ind3_3', 'id': 13}, {'name': 'Ind3_4', 'id': 14}]}]
Related
I have a list of jsons as mentioned below
[
{
"files": 0,
"data": [
{"name": "RFC", "value": "XXXXXXX", "attId": 01},
{"name": "NOMBRE", "value": "JOSE", "attId": 02},
{"name": "APELLIDO PATERNO", "value": "MONTIEL", "attId": 03},
{"name": "APELLIDO MATERNO", "value": "MENDOZA", "attId": 04},
{"name": "FECHA NACIMIENTO", "value": "1989-02-04", "attId": 05}
],
"dirId": 1,
"docId": 4,
"structure": {
"name": "personales",
"folioId": 22
}
},
{
"files": 0,
"data": [
{"name": "CALLE", "value": "AMOR", "attId": 06},
{"name": "No. EXTERIOR", "value": "4", "attId": 07},
{"name": "No. INTERIOR", "value": "2", "attId": 08},
{"name": "C.P.", "value": "55060", "attId": 09},
{"name": "ENTIDAD", "value": "ESTADO DE MEXICO", "attId": 10},
{"name": "MUNICIPIO", "value": "ECATEPEC", "attId": 11},
{"name": "COLONIA", "value": "INDUSTRIAL", "attId": 12}
],
"dirId": 1,
"docId": 4,
"structure": {
"name": "direccion",
"folioId": 22
}
}
]
I need to convert this list of jsons into separate individual jsons and execute them separately.
how to achieve this using pyspark or python?
Something like this, just one loop should work;
PN: There are few leading zeros in your input raw data...
# Assuming given list in your question is stored in lst
for item in lst:
print(item,"\n")
# item here will behave as individual dict/json
# Do your coding here as required
# Output for items;
{'files': 0, 'data': [{'name': 'RFC', 'value': 'XXXXXXX', 'attId': '01'}, {'name': 'NOMBRE', 'value': 'JOSE', 'attId': '02'}, {'name': 'APELLIDO PATERNO', 'value': 'MONTIEL', 'attId': '03'}, {'name': 'APELLIDO MATERNO', 'value': 'MENDOZA', 'attId': '04'}, {'name': 'FECHA NACIMIENTO', 'value': '1989-02-04', 'attId': '05'}], 'dirId': 1, 'docId': 4, 'structure': {'name': 'personales', 'folioId': 22}}
{'files': 0, 'data': [{'name': 'CALLE', 'value': 'AMOR', 'attId': '06'}, {'name': 'No. EXTERIOR', 'value': '4', 'attId': '07'}, {'name': 'No. INTERIOR', 'value': '2', 'attId': '08'}, {'name': 'C.P.', 'value': '55060', 'attId': '09'}, {'name': 'ENTIDAD', 'value': 'ESTADO DE MEXICO', 'attId': '10'}, {'name': 'MUNICIPIO', 'value': 'ECATEPEC', 'attId': '11'}, {'name': 'COLONIA', 'value': 'INDUSTRIAL', 'attId': '12'}], 'dirId': 1, 'docId': 4, 'structure': {'name': 'direccion', 'folioId': 22}}
i have my sample data as
b = [{"id": 1, "name": {"d_name": "miranda", "ingredient": "orange"}, "score": 1.123},
{"id": 20, "name": {"d_name": "limca", "ingredient": "lime"}, "score": 4.231},
{"id": 3, "name": {"d_name": "coke", "ingredient": "water"}, "score": 4.231},
{"id": 2, "name": {"d_name": "fanta", "ingredient": "water"}, "score": 4.231},
{"id": 3, "name": {"d_name": "dew", "ingredient": "water & sugar"}, "score": 2.231}]
i need to sort such that score ASC, name DESC, id ASC (by relational db notation).
So far, i have implemented
def sort_func(e):
return (e['score'], e['name']['d_name'], e['id'])
a = b.sort(key=sort_func, reverse=False)
This works for score ASC, name ASC, id ASC.
but for score ASC, name DESC, id ASC if i try to sort by name DESC it throws error. because of unary - error in -e['name']['d_name'].
How can i approach this problem, from here ? Thanks,
Edit 1:
i need to make this dynamic such that there can be case such as e['name'['d_name'] ASC, e['name']['ingredient'] DESC. How can i handle this type of dynamic behaviour ?
You can sort by -score, name, -id with reverse=True:
from pprint import pprint
b = [
{
"id": 1,
"name": {"d_name": "miranda", "ingredient": "orange"},
"score": 1.123,
},
{
"id": 20,
"name": {"d_name": "limca", "ingredient": "lime"},
"score": 4.231,
},
{
"id": 3,
"name": {"d_name": "coke", "ingredient": "water"},
"score": 4.231,
},
{
"id": 2,
"name": {"d_name": "fanta", "ingredient": "water"},
"score": 4.231,
},
{
"id": 3,
"name": {"d_name": "dew", "ingredient": "water & sugar"},
"score": 2.231,
},
]
pprint(
sorted(
b,
key=lambda k: (-k["score"], k["name"]["d_name"], -k["id"]),
reverse=True,
)
)
Prints:
[{'id': 1,
'name': {'d_name': 'miranda', 'ingredient': 'orange'},
'score': 1.123},
{'id': 3,
'name': {'d_name': 'dew', 'ingredient': 'water & sugar'},
'score': 2.231},
{'id': 20, 'name': {'d_name': 'limca', 'ingredient': 'lime'}, 'score': 4.231},
{'id': 2, 'name': {'d_name': 'fanta', 'ingredient': 'water'}, 'score': 4.231},
{'id': 3, 'name': {'d_name': 'coke', 'ingredient': 'water'}, 'score': 4.231}]
I've got a list of records in which the details contain some doubles. In the list of dicts below you see that the first 3 records (with id 1, 2 and 3) have the same "count" for all the details with a dir "s" (even though their respective detail id's differ). I would like to remove all records from the root list, for which all the counts of the details with a dir "s" are the same as the counts of the details with a dir "s" in a previous record. So from the list below I would want the records with ids 2 and 3 to be removed from the records list.
I've been writing nested loops for a while, but I can't really find a way of doing this. Plus, my code constantly becomes this complete mess real quick.
What would be a logical and Pythonic way of doing this?
records = [
{
'id': 1,
'details': [
{"id": 10, "dir": "s", "count": "1"},
{"id": 20, "dir": "u", "count": "6"},
{"id": 30, "dir": "s", "count": "1"}
]
},
{
'id': 2,
'details': [
{"id": 40, "dir": "s", "count": "1"},
{"id": 50, "dir": "u", "count": "7"},
{"id": 60, "dir": "s", "count": "1"}
]
},
{
'id': 3,
'details': [
{"id": 70, "dir": "s", "count": "1"},
{"id": 80, "dir": "u", "count": "8"},
{"id": 90, "dir": "s", "count": "1"}
]
},
{
'id': 4,
'details': [
{"id": 100, "dir": "s", "count": "999"},
{"id": 110, "dir": "up", "count": "6"},
{"id": 120, "dir": "s", "count": "999"}
]
},
]
Use a set and the key based on the two elements of the dict that you consider the definition of a 'duplicate'.
Simple example to uniquify:
seen=set()
for di in records:
for sdi in di['details']:
key=(sdi['dir'], sdi['count'])
if key not in seen:
seen.add(key)
print(sdi)
else:
# deal with the duplicate?
pass
Prints:
{'id': 10, 'dir': 's', 'count': '1'}
{'id': 20, 'dir': 'u', 'count': '6'}
{'id': 50, 'dir': 'u', 'count': '7'}
{'id': 80, 'dir': 'u', 'count': '8'}
{'id': 100, 'dir': 's', 'count': '999'}
{'id': 110, 'dir': 'up', 'count': '6'}
Giving a first pass the what I think you mean:
seen=set()
new_rec=[]
for di in records:
new_di={}
new_di['id']=di['id']
new_li=[]
for sdi in di['details']:
key=(sdi['dir'], sdi['count'])
if key not in seen:
seen.add(key)
new_li.append(sdi)
else:
# deal with the duplicate?
pass
new_di['details']=new_li
new_rec.append(new_di)
Which results in:
[ { 'id': 1,
'details': [ { 'id': 10,
'dir': 's',
'count': '1'},
{ 'id': 20,
'dir': 'u',
'count': '6'}]},
{ 'id': 2,
'details': [ { 'id': 50,
'dir': 'u',
'count': '7'}]},
{ 'id': 3,
'details': [ { 'id': 80,
'dir': 'u',
'count': '8'}]},
{ 'id': 4,
'details': [ { 'id': 100,
'dir': 's',
'count': '999'},
{ 'id': 110,
'dir': 'up',
'count': '6'}]}]
This question already has answers here:
Python: create a nested dictionary from a list of parent child values
(3 answers)
Closed 3 years ago.
I have a list of dictionaries that I got from the database in parent-child relationship:
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
I want to convert it to a nested dictionary such as
[ { "id": 1, "parent_id": 0, "name": "Wood", "price": 0, "children": [ { "id": 2, "parent_id": 1, "name": "Mango", "price": 18, "children": [ { "id": 3, "parent_id": 2, "name": "Table", "price": 342 }, { "id": 4, "parent_id": 2, "name": "Box", "price": 340, "children": [ { "id": 5, "parent_id": 4, "name": "Pencil", "price": 240 } ] } ] } ] }, { "id": 6, "parent_id": 0, "name": "Electronic", "price": 20, "children": [ { "id": 7, "parent_id": 6, "name": "TV", "price": 350 }, { "id": 8, "parent_id": 6, "name": "Mobile", "price": 300, "children": [ { "id": 9, "parent_id": 8, "name": "Iphone", "price": 0, "children": [ { "id": 10, "parent_id": 9, "name": "Iphone 10", "price": 400 } ] } ] } ] } ]
You can do this recursively, starting from the root nodes (where parent_id = 0) going downwards. But before your recursive calls, you can group nodes by their parent_id so that accessing them in each recursive call can be done in constant time:
levels = {}
for n in data:
levels.setdefault(n['parent_id'], []).append(n)
def build_tree(parent_id=0):
nodes = [dict(n) for n in levels.get(parent_id, [])]
for n in nodes:
children = build_tree(n['id'])
if children: n['children'] = children
return nodes
tree = build_tree()
print(tree)
Output
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0,'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400}]}]}]}]
Code is documented inline. Ignoring the corner cases like circular relations etc.
# Actual Data
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
# Create Parent -> child links using dictonary
data_dict = { r['id'] : r for r in data}
for r in data:
if r['parent_id'] in data_dict:
parent = data_dict[r['parent_id']]
if 'children' not in parent:
parent['children'] = []
parent['children'].append(r)
# Helper function to get all the id's associated with a parent
def get_all_ids(r):
l = list()
l.append(r['id'])
if 'children' in r:
for c in r['children']:
l.extend(get_all_ids(c))
return l
# Trimp the results to have a id only once
ids = set(data_dict.keys())
result = []
for r in data_dict.values():
the_ids = set(get_all_ids(r))
if ids.intersection(the_ids):
ids = ids.difference(the_ids)
result.append(r)
print (result)
Output:
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0, 'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400}]}]}]}]
I worked out a VERY SHORT solution, I believe it isn't the most efficient algorithm, but it does the job, will need a hell of optimization to work on very large data sets.
for i in range(len(data)-1, -1, -1):
data[i]["children"] = [child for child in data if child["parent_id"] == data[i]["id"]]
for child in data[i]["children"]:
data.remove(child)
Here is the complete explanation:
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
# Looping backwards,placing the lowest child
# into the next parent in the heirarchy
for i in range(len(data)-1, -1, -1):
# Create a dict key for the current parent in the loop called "children"
# and assign to it a list comprehension that loops over all items in the data
# to get the elements which have a parent_id equivalent to our current element's id
data[i]["children"] = [child for child in data if child["parent_id"] == data[i]["id"]]
# since the child is placed inside our its parent already, we will
# remove it from its actual position in the data
for child in data[i]["children"]:
data.remove(child)
# print the new data structure
print(data)
And here is the output:
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342, 'children': []}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240, 'children': []}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350, 'children': []}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0, 'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400, 'children': []}]}]}]}]
Given I have a list of dictionaries I want to create new list of lists grouping the dictionaries by the values of "price":
dicts = [
{ "name": "item1", "price": 10 },
{ "name": "item2", "price": 5 },
{ "name": "item3", "price": 10 },
{ "name": "item4", "price": 12 },
{ "name": "item5", "price": 12 },
{ "name": "item6", "price": 5 }
]
Should create:
grouped_dicts = [
[{ "name": "item1", "price": 10 },
{ "name": "item3", "price": 10 }],
[{ "name": "item2", "price": 5 },
{ "name": "item6", "price": 5 }],
[{ "name": "item4", "price": 12 },
{ "name": "item5", "price": 12 }]
]
Is there a nice way of doing this?
Thanks
As an alternative to the answer above, here is how you do it without an additional import:
d = {}
for item in dicts:
d.setdefault(item['price'], []).append(item)
list(d.values())
Out:
[[{'name': 'item1', 'price': 10}, {'name': 'item3', 'price': 10}],
[{'name': 'item2', 'price': 5}, {'name': 'item6', 'price': 5}],
[{'name': 'item4', 'price': 12}, {'name': 'item5', 'price': 12}]]
You could use deaultdict
from collections import defaultdict
d=defaultdict(list)
for item in dicts:
d[list(item.values())[-1]].append(item)
Output:
defaultdict(list,
{10: [{'name': 'item1', 'price': 10},
{'name': 'item3', 'price': 10}],
5: [{'name': 'item2', 'price': 5}, {'name': 'item6', 'price': 5}],
12: [{'name': 'item4', 'price': 12},
{'name': 'item5', 'price': 12}]})
If you just need the list then just extract the values of defaultdict
list(d.values())
Output:
[[{'name': 'item1', 'price': 10}, {'name': 'item3', 'price': 10}],
[{'name': 'item2', 'price': 5}, {'name': 'item6', 'price': 5}],
[{'name': 'item4', 'price': 12}, {'name': 'item5', 'price': 12}]]