Develop Reference

How to remove doubles by nested attributes in Python?

I've got a list of records in which the details contain some doubles. In the list of dicts below you see that the first 3 records (with id 1, 2 and 3) have the same "count" for all the details with a dir "s" (even though their respective detail id's differ). I would like to remove all records from the root list, for which all the counts of the details with a dir "s" are the same as the counts of the details with a dir "s" in a previous record. So from the list below I would want the records with ids 2 and 3 to be removed from the records list.
I've been writing nested loops for a while, but I can't really find a way of doing this. Plus, my code constantly becomes this complete mess real quick.
What would be a logical and Pythonic way of doing this?
records = [
{
'id': 1,
'details': [
{"id": 10, "dir": "s", "count": "1"},
{"id": 20, "dir": "u", "count": "6"},
{"id": 30, "dir": "s", "count": "1"}
]
},
{
'id': 2,
'details': [
{"id": 40, "dir": "s", "count": "1"},
{"id": 50, "dir": "u", "count": "7"},
{"id": 60, "dir": "s", "count": "1"}
]
},
{
'id': 3,
'details': [
{"id": 70, "dir": "s", "count": "1"},
{"id": 80, "dir": "u", "count": "8"},
{"id": 90, "dir": "s", "count": "1"}
]
},
{
'id': 4,
'details': [
{"id": 100, "dir": "s", "count": "999"},
{"id": 110, "dir": "up", "count": "6"},
{"id": 120, "dir": "s", "count": "999"}
]
},
]

Use a set and the key based on the two elements of the dict that you consider the definition of a 'duplicate'.
Simple example to uniquify:
seen=set()
for di in records:
for sdi in di['details']:
key=(sdi['dir'], sdi['count'])
if key not in seen:
seen.add(key)
print(sdi)
else:
# deal with the duplicate?
pass
Prints:
{'id': 10, 'dir': 's', 'count': '1'}
{'id': 20, 'dir': 'u', 'count': '6'}
{'id': 50, 'dir': 'u', 'count': '7'}
{'id': 80, 'dir': 'u', 'count': '8'}
{'id': 100, 'dir': 's', 'count': '999'}
{'id': 110, 'dir': 'up', 'count': '6'}
Giving a first pass the what I think you mean:
seen=set()
new_rec=[]
for di in records:
new_di={}
new_di['id']=di['id']
new_li=[]
for sdi in di['details']:
key=(sdi['dir'], sdi['count'])
if key not in seen:
seen.add(key)
new_li.append(sdi)
else:
# deal with the duplicate?
pass
new_di['details']=new_li
new_rec.append(new_di)
Which results in:
[ { 'id': 1,
'details': [ { 'id': 10,
'dir': 's',
'count': '1'},
{ 'id': 20,
'dir': 'u',
'count': '6'}]},
{ 'id': 2,
'details': [ { 'id': 50,
'dir': 'u',
'count': '7'}]},
{ 'id': 3,
'details': [ { 'id': 80,
'dir': 'u',
'count': '8'}]},
{ 'id': 4,
'details': [ { 'id': 100,
'dir': 's',
'count': '999'},
{ 'id': 110,
'dir': 'up',
'count': '6'}]}]

Convert list of dictionaries into a nested dictionary [duplicate]

This question already has answers here:
Python: create a nested dictionary from a list of parent child values
(3 answers)
Closed 3 years ago.
I have a list of dictionaries that I got from the database in parent-child relationship:
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
I want to convert it to a nested dictionary such as
[ { "id": 1, "parent_id": 0, "name": "Wood", "price": 0, "children": [ { "id": 2, "parent_id": 1, "name": "Mango", "price": 18, "children": [ { "id": 3, "parent_id": 2, "name": "Table", "price": 342 }, { "id": 4, "parent_id": 2, "name": "Box", "price": 340, "children": [ { "id": 5, "parent_id": 4, "name": "Pencil", "price": 240 } ] } ] } ] }, { "id": 6, "parent_id": 0, "name": "Electronic", "price": 20, "children": [ { "id": 7, "parent_id": 6, "name": "TV", "price": 350 }, { "id": 8, "parent_id": 6, "name": "Mobile", "price": 300, "children": [ { "id": 9, "parent_id": 8, "name": "Iphone", "price": 0, "children": [ { "id": 10, "parent_id": 9, "name": "Iphone 10", "price": 400 } ] } ] } ] } ]

You can do this recursively, starting from the root nodes (where parent_id = 0) going downwards. But before your recursive calls, you can group nodes by their parent_id so that accessing them in each recursive call can be done in constant time:
levels = {}
for n in data:
levels.setdefault(n['parent_id'], []).append(n)
def build_tree(parent_id=0):
nodes = [dict(n) for n in levels.get(parent_id, [])]
for n in nodes:
children = build_tree(n['id'])
if children: n['children'] = children
return nodes
tree = build_tree()
print(tree)
Output
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0,'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400}]}]}]}]

Code is documented inline. Ignoring the corner cases like circular relations etc.
# Actual Data
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
# Create Parent -> child links using dictonary
data_dict = { r['id'] : r for r in data}
for r in data:
if r['parent_id'] in data_dict:
parent = data_dict[r['parent_id']]
if 'children' not in parent:
parent['children'] = []
parent['children'].append(r)
# Helper function to get all the id's associated with a parent
def get_all_ids(r):
l = list()
l.append(r['id'])
if 'children' in r:
for c in r['children']:
l.extend(get_all_ids(c))
return l
# Trimp the results to have a id only once
ids = set(data_dict.keys())
result = []
for r in data_dict.values():
the_ids = set(get_all_ids(r))
if ids.intersection(the_ids):
ids = ids.difference(the_ids)
result.append(r)
print (result)
Output:
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0, 'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400}]}]}]}]

I worked out a VERY SHORT solution, I believe it isn't the most efficient algorithm, but it does the job, will need a hell of optimization to work on very large data sets.
for i in range(len(data)-1, -1, -1):
data[i]["children"] = [child for child in data if child["parent_id"] == data[i]["id"]]
for child in data[i]["children"]:
data.remove(child)
Here is the complete explanation:
data = [
{"id":1, "parent_id": 0, "name": "Wood", "price": 0},
{"id":2, "parent_id": 1, "name": "Mango", "price": 18},
{"id":3, "parent_id": 2, "name": "Table", "price": 342},
{"id":4, "parent_id": 2, "name": "Box", "price": 340},
{"id":5, "parent_id": 4, "name": "Pencil", "price": 240},
{"id":6, "parent_id": 0, "name": "Electronic", "price": 20},
{"id":7, "parent_id": 6, "name": "TV", "price": 350},
{"id":8, "parent_id": 6, "name": "Mobile", "price": 300},
{"id":9, "parent_id": 8, "name": "Iphone", "price": 0},
{"id":10, "parent_id": 9, "name": "Iphone 10", "price": 400}
]
# Looping backwards,placing the lowest child
# into the next parent in the heirarchy
for i in range(len(data)-1, -1, -1):
# Create a dict key for the current parent in the loop called "children"
# and assign to it a list comprehension that loops over all items in the data
# to get the elements which have a parent_id equivalent to our current element's id
data[i]["children"] = [child for child in data if child["parent_id"] == data[i]["id"]]
# since the child is placed inside our its parent already, we will
# remove it from its actual position in the data
for child in data[i]["children"]:
data.remove(child)
# print the new data structure
print(data)
And here is the output:
[{'id': 1, 'parent_id': 0, 'name': 'Wood', 'price': 0, 'children': [{'id': 2, 'parent_id': 1, 'name': 'Mango', 'price': 18, 'children': [{'id': 3, 'parent_id': 2, 'name': 'Table', 'price': 342, 'children': []}, {'id': 4, 'parent_id': 2, 'name': 'Box', 'price': 340, 'children': [{'id': 5, 'parent_id': 4, 'name': 'Pencil', 'price': 240, 'children': []}]}]}]}, {'id': 6, 'parent_id': 0, 'name': 'Electronic', 'price': 20, 'children': [{'id': 7, 'parent_id': 6, 'name': 'TV', 'price': 350, 'children': []}, {'id': 8, 'parent_id': 6, 'name': 'Mobile', 'price': 300, 'children': [{'id': 9, 'parent_id': 8, 'name': 'Iphone', 'price': 0, 'children': [{'id': 10, 'parent_id': 9, 'name': 'Iphone 10', 'price': 400, 'children': []}]}]}]}]

Python, reorganize array of dicts

To be honest, it's too easy for me to make in JS or Perl, but i've completely stuck with that in Python because of coplexed tools for dealing with dicts/lists. So, what i need:
i have an array of dicts:
[
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_1", "k_id": 4},
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_2", "k_id": 5},
{"id": 1, "name": "Res1", "type": "resource", "k_name": "Ind1_3", "k_id": 6},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_1", "k_id": 7},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_2", "k_id": 8},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_3", "k_id": 9},
{"id": 2, "name": "Res2", "type": "service", "k_name": "Ind2_4", "k_id": 10},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_1", "k_id": 11},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_2", "k_id": 12},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_3", "k_id": 13},
{"id": 3, "name": "Res3", "type": "service", "k_name": "Ind3_4", "k_id": 14}
]
and i need to make that:
[
{
"id": 1,
"name": "Res1",
"type": "resource",
"indicators": [
{"name": "Ind1_1","id": 4},
{"name": "Ind1_2","id": 5},
{"name": "Ind1_3","id": 6}
]
},
{
"id": 2,
"name": "Res2",
"type": "service",
"indicators": [
{"name": "Ind2_1","id": 7},
{"name": "Ind2_2","id": 8},
{"name": "Ind2_3","id": 9},
{"name": "Ind2_4","id": 10}
]
},
{
"id": 3,
"name": "Res3",
"type": "service",
"indicators": [
{"name": "Ind3_1","id": 11},
{"name": "Ind3_2","id": 12},
{"name": "Ind3_3","id": 13},
{"name": "Ind3_4","id": 14}
]
}
]
Can you help me with that?

itertools to the rescue:
import itertools
# Assuming your original list is `l`
# if it does not come in order, you need to do this line first, and will probably be less efficient.
l = sorted(l, key=lambda x:(x["id"], x["name"], x["type"]))
d = []
for k, g in itertools.groupby(l, lambda x: (x["id"], x["name"], x["type"])):
d.append({i:v for i, v in zip(["id", "name", "type"], k)})
d[-1]["indicator"] = [{y.split('_')[1]:e[y] for y in ["k_id", "k_name"]} for e in list(g)]
d becomes:
[{'id': 1,
'indicator': [{'id': 4, 'name': 'Ind1_1'},
{'id': 5, 'name': 'Ind1_2'},
{'id': 6, 'name': 'Ind1_3'}],
'name': 'Res1',
'type': 'resource'},
{'id': 2,
'indicator': [{'id': 7, 'name': 'Ind2_1'},
{'id': 8, 'name': 'Ind2_2'},
{'id': 9, 'name': 'Ind2_3'},
{'id': 10, 'name': 'Ind2_4'}],
'name': 'Res2',
'type': 'service'},
{'id': 3,
'indicator': [{'id': 11, 'name': 'Ind3_1'},
{'id': 12, 'name': 'Ind3_2'},
{'id': 13, 'name': 'Ind3_3'},
{'id': 14, 'name': 'Ind3_4'}],
'name': 'Res3',
'type': 'service'}]

You can use a mapping dict to map ids to corresponding sub-lists, so that as you iterate through the list (named l in this example), you can append a new entry to the output list if the id is not found in the mapping, or append the entry to the existing sub-list if id is found in the mapping:
mapping = {}
output = []
for d in l:
i = {'name': d.pop('k_name'), 'id': d.pop('k_id')}
if d['id'] in mapping:
mapping[d['id']].append(i)
else:
output.append({**d, 'indicators': [i]})
mapping[d['id']] = output[-1]['indicators']
output becomes:
[{'id': 1, 'name': 'Res1', 'type': 'resource', 'indicators': [{'name': 'Ind1_1', 'id': 4}, {'name': 'Ind1_2', 'id': 5}, {'name': 'Ind1_3', 'id': 6}]}, {'id': 2, 'name': 'Res2', 'type': 'service', 'indicators': [{'name': 'Ind2_1', 'id': 7}, {'name': 'Ind2_2', 'id': 8}, {'name': 'Ind2_3', 'id': 9}, {'name': 'Ind2_4', 'id': 10}]}, {'id': 3, 'name': 'Res3', 'type': 'service', 'indicators': [{'name': 'Ind3_1', 'id': 11}, {'name': 'Ind3_2', 'id': 12}, {'name': 'Ind3_3', 'id': 13}, {'name': 'Ind3_4', 'id': 14}]}]

Sorting output of Python file by location, increment by common id

There's multiple instances of data that share the same location id's, for example in the output below there are many 3s:
121 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/121/location'}}
122 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/122/location'}}
120 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/120/location'}}
119 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/119/location'}}
191 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/191/location'}}
190 {'data': {'id': 52, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/190/location'}}
193 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/193/location'}}
187 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/187/location'}}
189 {'data': {'id': 52, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/189/location'}}
186 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/186/location'}}
198 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/198/location'}}
196 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/196/location'}}
199 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/199/location'}}
201 {'data': {'id': 3, 'type': 'location'}, 'links': {'self': 'http://localhost:2510/api/v2/jobs/201/location'}}
I'd like to sort these all in the style of:
{'data': {'id': 3, 'type': 'location'} 15
{'data': {'id': 4, 'type': 'location'} 6
{'data': {'id': 5, 'type': 'location'} 0
{'data': {'id': 6, 'type': 'location'} 11
Is there a way to adapt that Python script to output the data like that?
Actually it's coming from this JSON file, that looks like so:
{
"links": {
"self": "http://localhost:2510/api/v2/jobs?skills=data%20science"
},
"data": [
{
"id": 121,
"type": "job",
"attributes": {
"title": "Data Scientist",
"date": "2014-01-22T15:25:00.000Z",
"description": "Data scientists are in increasingly high demand amongst tech companies in London. Generally a combination of business acumen and technical skills are sought. Big data experience ..."
},
"relationships": {
"location": {
"links": {
"self": "http://localhost:2510/api/v2/jobs/121/location"
},
"data": {
"type": "location",
"id": 3
}
},
"country": {
"links": {
"self": "http://localhost:2510/api/v2/jobs/121/country"
},
"data": {
"type": "country",
"id": 1
}
},
"skills": {
"links": {
and parsed using the following Python script:
import json
from pprint import pprint
with open('data.json') as data_file:
data = json.load(data_file)
for item in data["data"]:
print(item['id'], item['relationships']['location'])
This is the full data file in my GitHub.

If I understand correctly, you have a list of items with a structure like this:
...
{{'data': {'id': 3, 'type': 'location'} ... }
{{'data': {'id': 3, 'type': 'location'} ... }
{{'data': {'id': 4, 'type': 'location'} ... }
...
And you want to count the number of items with each unique combination of id and type, and print the results in sorted order?
You could use the general counting dictionary pattern:
counts = dict()
for item in data['data']:
# here I assume the items you are looking for are locations
# for it to be a key, it has to be immutable, so make it a tuple
val = item['relationships']['location']['data']
location_tuple = (val['id'], val['type'])
if location_tuple in counts:
counts[location_tuple] += 1
else:
counts[location_tuple] = 1
# print them out in order, first send to list of tuples and sort
results = counts.items()
results.sort() # will sort on first item, which will be id
# results come in like so: ((3, location), 15)
for item in results:
print 'id:', item[0][0], 'type:', item[0][1], 'count:' item[1]
Basic idea here is that you can use the dictionary to count using tuples as keys of all the distinct things you want to count, and then use items to get it as a list of tuples, which can be sorted. Tuples are sorted by first-element, second-element, and so on, recursively, so take care when building tuples to put your first sort key in the first position, and so on, or you'll have to make adjustments to your sort call. You may have to adjust what I have depending on what you want to extract and print.

Put data in database (SQLite for example) then "GROUP BY".

List indices must be integers, not dict

I'm making a script for the game and i need help about grabbing dictionary information out of json.
So far I've dealt with list and I didn't had problem grabbing that information.
req['result']['trades'][key]['output']
This is my try of grabbing the info after which i get error list indices must be integers, not dict.
Any help greatly appreciated.
json print, i have removed some account sensitive data.
{
'fue_step': 30,
'type': 'm',
'rubies': xxx,
'might': xxx,
'race': 'xxx',
'result': {
'trades': [
{
'id': 13,
'requirement': ['VenomTrapPart', 1],
'token_cost': 1,
'output': ['VulcansBlessingRare', 1],
'name': 'VenomVulcan',
'enabled': True
},
{
'id': 14,
'requirement': ['EternalRune', 50],
'token_cost': 1,
'output': ['ColossalMite500TroopPrizeItem', 1],
'name': 'EternalColossal',
'enabled': True
},
{
'id': 86,
'requirement':['RenameProclamation', 25],
'token_cost': 1,
'output': ['TimeTrickstersBag', 1],
'name': 'RenameTrick',
'enabled': True
},
{
'id': 95,
'requirement': ['FireDragonBlaze', 1],
'token_cost': 1,
'output': ['FireTroop1000TroopPrizeItem', 10],
'name': 'BlazeFireTroop',
'enabled': True
},
{
'id': 100,
'requirement': ['StoneDragonQuake', 1],
'token_cost': 1,
'output': ['StoneTroop10kTroopPrizeItem', 1],
'name': 'StoneDQStoneTroop',
'enabled': True
},
{
'id': 113,
'requirement': ['VulcansBlessing15Elite', 3],
'token_cost': 1,
'output': ['VulcansBlessing16Elite', 1],
'name': '15 --> 16 Elite',
'enabled': True
},
{
'id': 114,
'requirement': ['LunaPowder', 25],
'token_cost': 1,
'output': ['IceTroop50kTroopPrizeItem', 1],
'name': 'Luna Powder --> Soul Reapers',
'enabled': True
},
{
'id': 115,
'requirement': ['GreaterCrystalDefense', 5],
'token_cost': 1,
'output': ['MasterCrystalDefense', 1],
'name': 'Greater Defense --> Master Defense',
'enabled': True
},
{
'id': 116,
'requirement': ['NanoCanisters', 3],
'token_cost': 1,
'output': ['TestroniusDeluxe', 1],
'name': 'Nano Canisters for Deluxes',
'enabled': True
},
{
'id': 117,
'requirement': ['VulcansBlessing16Common', 3],
'token_cost': 1,
'output': ['VulcansBlessing17Common', 1],
'name': "Common +16's for Common +17",
'enabled': True
}
],
'trades_today': 0,
'doubloons_left': 9567,
'free_trades_left': 1,
'success': True,
'auto_confirm': True,
'trade_reset_date':1450771200
},
'client_cachebreaker': '1449879464'

I edited your edit to include indentation in your JSON so that it is more readable. As soon as the edit is approved, you'll be able to see it.
Here is an example of how to access a value deep within the object:
print req['result']['trades'][4]['name']
Output:
StoneDQStoneTroop
If you want to access dicts within a list by their values, here is a post that explains a few ways to do so. This would allow you, for example, to find a dict in req['result']['trades'] by its 'id' value instead of by its index within the 'trades' list.