Develop Reference

Remove duplicates with low score in list of dicts

I currently have the following list of dicts:
[
{'id': '1', 'sim': 0.81},
{'id': '1', 'sim': 0.72},
{'id': '2', 'sim': 0.85},
{'id': '2', 'sim': 0.81},
{'id': '2', 'sim': 0.72}
]
I'd like to remove the duplicates which have not the highest sim and get the following:
[
{'id': '1', 'sim': 0.81},
{'id': '2', 'sim': 0.85},
]

One way to go would be to use pandas :
import pandas as pd
d = [
{'id': '1', 'sim': 0.81},
{'id': '1', 'sim': 0.72},
{'id': '2', 'sim': 0.85},
{'id': '2', 'sim': 0.81},
{'id': '2', 'sim': 0.72}
]
df = pd.DataFrame(d)
df = df.groupby(['id'], sort=False)['sim'].max()
Then you can keep using it as a Dataframe, or going back to nested dictionaries depending on what you need.

sims_list = [
{'id': '1', 'sim': 0.81},
{'id': '1', 'sim': 0.72},
{'id': '2', 'sim': 0.85},
{'id': '2', 'sim': 0.81},
{'id': '2', 'sim': 0.72}
]
result = []
for each_sim in sims_list:
for each_result in result:
if each_result["id"] == each_sim["id"]:
each_result["sim"] = max(each_result["sim"], each_sim["sim"])
break
else:
result.append(each_sim)
print(result)
Output
[{'id': '1', 'sim': 0.81}, {'id': '2', 'sim': 0.85}]

Merging two lists of dictionaries using common id key

I have the following two lists of dictionaries:
list_1 = [
{'id': '1', 'name': 'Johnny Johson1'},
{'id': '2', 'name': 'Johnny Johson2'},
{'id': '1', 'name': 'Johnny Johson1'},
{'id': '3', 'name': 'Johnny Johson3'},
]
list_2 = [
{'id': '1', 'datetime': '2020-01-06T12:30:00.000Z'},
{'id': '2', 'datetime': '2020-01-06T14:00:00.000Z'},
{'id': '1', 'datetime': '2020-01-06T15:30:00.000Z'},
{'id': '3', 'datetime': '2020-01-06T15:30:00.000Z'},
]
Essentially, I would like no loss of data even on duplicate IDs, as they represent different events (there is a sepearate ID for that, but for the purpose of demonstrating the problem, is not needed). If there are any IDs in one list, not in the other, then disregard that ID all together.
Ideally, I would like to end up with the following (from the amalgamation of the two lists):
list_3 = [
{'id': '1', 'name': 'Johnny Johson1', 'datetime': '2020-01-06T12:30:00.000Z'},
{'id': '2', 'name': 'Johnny Johson2', 'datetime': '2020-01-06T14:00:00.000Z'},
{'id': '1', 'name': 'Johnny Johson1', 'datetime': '2020-01-06T15:30:00.000Z'},
{'id': '3', 'name': 'Johnny Johson3', 'datetime': '2020-01-06T15:30:00.000Z'},
]

You can use the following list comprehension, which uses the double asterisk keyword argent unpacking syntax, evaluated on both lists using pairwise elements obtained with zip(). This has the effect of combining the two dictionaries into one.
list_3 = [{**x, **y} for x, y in zip(list_1, list_2)]
Output:
>>> list3
[{'id': '1', 'name': 'Johnny Johson1', 'datetime': '2020-01-06T12:30:00.000Z'},
{'id': '2', 'name': 'Johnny Johson2', 'datetime': '2020-01-06T14:00:00.000Z'},
{'id': '1', 'name': 'Johnny Johson1', 'datetime': '2020-01-06T15:30:00.000Z'},
{'id': '3', 'name': 'Johnny Johson3', 'datetime': '2020-01-06T15:30:00.000Z'}]
Note that this approach requires at least Python 3.5.

Update JSON format from other JSON file

I have two files which are a and b. I want to import certain information from data b to data a with the unique id from every response.
data
a= [{'id':'abc23','name':'aa','age':'22',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}},
{'id':'abc25','name':'bb','age':'32',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}},
{'id':'abc60','name':'cc','age':'24',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}}]
b=[{'id':'abc23','read':'2','speak':'abc','write':'2'},
{'id':'abc25','read':'3','speak':'def','write':'3'},
{'id':'abc60','read':'5','speak':'dgf','write':'1'}]
Code that I used to import from b to a :
from pprint import pprint
for dest in a:
for source in b:
if source['id'] == dest['id']:
dest['data'].update(source)
pprint(a)
Output from the code that i used :
[{ 'age': '22',
'data': {'id': 'abc23', 'read': '2', 'speak': 'abc', 'write': '2'},
'id': 'abc23',
'name': 'aa',
'responses': {'a': 1, 'b': 2}},
{ 'age': '32',
'data': {'id': 'abc25', 'read': '3', 'speak': 'def', 'write': '3'},
'id': 'abc25',
'name': 'bb',
'responses': {'a': 1, 'b': 2}},
{ 'age': '24',
'data': {'id': 'abc60', 'read': '5', 'speak': 'dgf', 'write': '1'},
'id': 'abc60',
'name': 'cc',
'responses': {'a': 1, 'b': 2}}]
But... This is the output that I want:
[{'age': '22',
'data': {'read': '2', 'speak': 'abc'},
'id': 'abc23',
'name': 'aa',
'responses': {'a': 1, 'b': 2}},
{'age': '32',
'data': {'read': '3', 'speak': 'def'},
'id': 'abc25',
'name': 'bb',
'responses': {'a': 1, 'b': 2}},
{'age': '24',
'data': {'read': '5', 'speak': 'dgf'},
'id': 'abc60',
'name': 'cc',
'responses': {'a': 1, 'b': 2}}]

It can't work the way you want with your code.
You do
dest['data'].update(source)
where source is
{'id':'abc23','read':'2','speak':'abc','write':'2'}
and dest['data'] is {'read':'','speak':''}.
When you update it will add all key-value pairs to dest['data'] and preserve the ones that won't be overwritten.
from pprint import pprint
for dest in a:
for source in b:
if source['id'] == dest['id']:
dest['data'] = {k: v for k, v in source.items() if k in dest.get('data', {})}
pprint(a)
This one will look for all the fields that are 'updateable' for each case. You might want to hardcode it, depending on your use case.

This is one approach by changing b to a dict for easy lookup.
Ex:
a= [{'id':'abc23','name':'aa','age':'22',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}},
{'id':'abc25','name':'bb','age':'32',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}},
{'id':'abc60','name':'cc','age':'24',
'data':{'read':'','speak':''},
'responses':{'a':1,'b':2}}]
b=[{'id':'abc23','read':'2','speak':'abc','write':'2'},
{'id':'abc25','read':'3','speak':'def','write':'3'},
{'id':'abc60','read':'5','speak':'dgf','write':'1'}]
b = {i.pop('id'): i for i in b} #Convert to dict key = ID & value = `read`, `speak`, `write`
for i in a:
i['data'].update(b[i['id']]) #Update list
print(a)
Output:
[{'age': '22',
'data': {'read': '2', 'speak': 'abc', 'write': '2'},
'id': 'abc23',
'name': 'aa',
'responses': {'a': 1, 'b': 2}},
{'age': '32',
'data': {'read': '3', 'speak': 'def', 'write': '3'},
'id': 'abc25',
'name': 'bb',
'responses': {'a': 1, 'b': 2}},
{'age': '24',
'data': {'read': '5', 'speak': 'dgf', 'write': '1'},
'id': 'abc60',
'name': 'cc',
'responses': {'a': 1, 'b': 2}}]

How to sort list of dictrionaries in the right way Python

I have list as follows:
data = [
{'items': [
{'key': u'3', 'id': 1, 'name': u'Typeplaatje'},
{'key': u'2', 'id': 2, 'name': u'Aanduiding van het chassisnummer '},
{'key': u'1', 'id': 3, 'name': u'Kilometerteller: Kilometerstand '},
{'key': u'5', 'id': 4, 'name': u'Inschrijvingsbewijs '},
{'key': u'4', 'id': 5, 'name': u'COC of gelijkvormigheidsattest '}
], 'id': 2, 'key': u'B', 'name': u'Onderdelen'},
{'items': [
{'key': u'10', 'id': 10, 'name': u'Koppeling'},
{'key': u'7', 'id': 11, 'name': u'Differentieel '},
{'key': u'9', 'id': 12, 'name': u'Cardanhoezen '},
{'key': u'8', 'id': 13, 'name': u'Uitlaat '},
{'key': u'6', 'id': 15, 'name': u'Batterij'}
], 'id': 2, 'key': u'B', 'name': u'Onderdelen'}
]
And I want to sort items by key.
Thus the wanted result is as follows:
res = [
{'items': [
{'key': u'1', 'id': 3, 'name': u'Kilometerteller: Kilometerstand '},
{'key': u'2', 'id': 2, 'name': u'Aanduiding van het chassisnummer '},
{'key': u'3', 'id': 1, 'name': u'Typeplaatje'},
{'key': u'4', 'id': 5, 'name': u'COC of gelijkvormigheidsattest '},
{'key': u'5', 'id': 4, 'name': u'Inschrijvingsbewijs '},
], 'id': 2, 'key': u'B', 'name': u'Onderdelen'},
{'items': [
{'key': u'6', 'id': 15, 'name': u'Batterij'},
{'key': u'7', 'id': 11, 'name': u'Differentieel '},
{'key': u'8', 'id': 13, 'name': u'Uitlaat '},
{'key': u'9', 'id': 12, 'name': u'Cardanhoezen '},
{'key': u'10', 'id': 10, 'name': u'Koppeling'}
], 'id': 2, 'key': u'B', 'name': u'Onderdelen'}
]
I've tried as follows:
res = []
for item in data:
new_data = {
'id': item['id'],
'key': item['key'],
'name': item['name'],
'items': sorted(item['items'], key=lambda k : k['key'])
}
res.append(new_data)
print(res)
The first is sorted fine, but the second one not.
What am I doing wrong and is there a better way of doing it?

Your sort is wrong in the second case because the keys are strings, and strings are sorted by their first character which is '1' if your key is '10'. A slight modification to your sorting function would do the trick:
'items': sorted(item['items'], key=lambda k : int(k['key'])
I'm doing an int because you want to sort them as if they are numbers. Here it is in your code:
res = []
for item in data:
new_data = {
'id': item['id'],
'key': item['key'],
'name': item['name'],
'items': sorted(item['items'], key=lambda k : int(k['key']) )
}
res.append(new_data)
print(res)
And here's the result:
[{'id': 2,
'items': [{'id': 3, 'key': '1', 'name': 'Kilometerteller: Kilometerstand '},
{'id': 2, 'key': '2', 'name': 'Aanduiding van het chassisnummer '},
{'id': 1, 'key': '3', 'name': 'Typeplaatje'},
{'id': 5, 'key': '4', 'name': 'COC of gelijkvormigheidsattest '},
{'id': 4, 'key': '5', 'name': 'Inschrijvingsbewijs '}],
'key': 'B',
'name': 'Onderdelen'},
{'id': 2,
'items': [{'id': 15, 'key': '6', 'name': 'Batterij'},
{'id': 11, 'key': '7', 'name': 'Differentieel '},
{'id': 13, 'key': '8', 'name': 'Uitlaat '},
{'id': 12, 'key': '9', 'name': 'Cardanhoezen '},
{'id': 10, 'key': '10', 'name': 'Koppeling'}],
'key': 'B',
'name': 'Onderdelen'}]

You need to replace the old items in the data with the sorted items based on key numerically instead of string sort. So use int(item['key']) in sort like,
>>> data
[{'items': [{'key': '1', 'id': 3, 'name': 'Kilometerteller: Kilometerstand '}, {'key': '2', 'id': 2, 'name': 'Aanduiding van het chassisnummer '}, {'key': '3', 'id': 1, 'name': 'Typeplaatje'}, {'key': '4', 'id': 5, 'name': 'COC of gelijkvormigheidsattest '}, {'key': '5', 'id': 4, 'name': 'Inschrijvingsbewijs '}], 'id': 2, 'key': 'B', 'name': 'Onderdelen'}, {'items': [{'key': '6', 'id': 15, 'name': 'Batterij'}, {'key': '7', 'id': 11, 'name': 'Differentieel '}, {'key': '8', 'id': 13, 'name': 'Uitlaat '}, {'key': '9', 'id': 12, 'name': 'Cardanhoezen '}, {'key': '10', 'id': 10, 'name': 'Koppeling'}], 'id': 2, 'key': 'B', 'name': 'Onderdelen'}]
>>>
>>> for item in data:
... item['items'] = sorted(item['items'], key=lambda x: int(x['key']))
...
>>> import pprint
>>> pprint.pprint(data)
[{'id': 2,
'items': [{'id': 3, 'key': '1', 'name': 'Kilometerteller: Kilometerstand '},
{'id': 2, 'key': '2', 'name': 'Aanduiding van het chassisnummer '},
{'id': 1, 'key': '3', 'name': 'Typeplaatje'},
{'id': 5, 'key': '4', 'name': 'COC of gelijkvormigheidsattest '},
{'id': 4, 'key': '5', 'name': 'Inschrijvingsbewijs '}],
'key': 'B',
'name': 'Onderdelen'},
{'id': 2,
'items': [{'id': 15, 'key': '6', 'name': 'Batterij'},
{'id': 11, 'key': '7', 'name': 'Differentieel '},
{'id': 13, 'key': '8', 'name': 'Uitlaat '},
{'id': 12, 'key': '9', 'name': 'Cardanhoezen '},
{'id': 10, 'key': '10', 'name': 'Koppeling'}],
'key': 'B',
'name': 'Onderdelen'}]

So list comes with a handy method called sort which sorts itself inplace. I'd use that to your advantage:
for d in data:
d['items'].sort(key=lambda x: int(x['key']))
Results:
[{'id': 2,
'items': [{'id': 3, 'key': '1', 'name': 'Kilometerteller: Kilometerstand '},
{'id': 2, 'key': '2', 'name': 'Aanduiding van het chassisnummer '},
{'id': 1, 'key': '3', 'name': 'Typeplaatje'},
{'id': 5, 'key': '4', 'name': 'COC of gelijkvormigheidsattest '},
{'id': 4, 'key': '5', 'name': 'Inschrijvingsbewijs '}],
'key': 'B',
'name': 'Onderdelen'},
{'id': 2,
'items': [{'id': 15, 'key': '6', 'name': 'Batterij'},
{'id': 11, 'key': '7', 'name': 'Differentieel '},
{'id': 13, 'key': '8', 'name': 'Uitlaat '},
{'id': 12, 'key': '9', 'name': 'Cardanhoezen '},
{'id': 10, 'key': '10', 'name': 'Koppeling'}],
'key': 'B',
'name': 'Onderdelen'}]

Join on non-unique second id - Python

I am trying to join a dictionary to another dictionary. I have two keys; one that is unique and another which is not unique. I want to join information on the non-unique key and leave all information as it is one the unique key, i.e. the number of unique id's has to stay the same.
Any ideas to how I can achieve this?
This is the first dictionary:
names = [
{'id': '1', 'name': 'Peter', 'category_id': '25'},
{'id': '2', 'name': 'Jim', 'category_id': '20'},
{'id': '3', 'name': 'Toni', 'category_id': '20'}
]
This is the second dictionary:
categories = [
{'category_id': '25', 'level': 'advanced'},
{'category_id': '20', 'level': 'beginner'}
]
And this is what I am trying to achieve:
all = [
{'id': '1', 'name': 'Peter', 'category_id': '25', 'level': 'advanced'},
{'id': '2', 'name': 'Jim', 'category_id': '20', 'level': 'beginner'},
{'id': '3', 'name': 'Toni', 'category_id': '20', 'level': 'beginner'}
]
EDIT:
names = [
{'id': '1', 'name': 'Peter', 'category_id': '25'},
{'id': '2', 'name': 'Jim', 'category_id': '20'},
{'id': '3', 'name': 'Toni', 'category_id': '20'}
]
categories = [
{'category_id': '25', 'level': 'advanced'},
{'category_id': '20', 'level': 'beginner'}
]
def merge_lists(l1, l2, key):
merged = {}
for item in l1+l2:
if item[key] in merged:
merged[item[key]].update(item)
else:
merged[item[key]] = item
return merged.values()
courses = merge_lists(names, categories, 'category_id')
print(courses)
gives:
([{'id': '1', 'name': 'Peter', 'category_id': '25', 'level': 'advanced'},
{'id': '3', 'name': 'Toni', 'category_id': '20', 'level': 'beginner'}])

Create a mapping from category_id to additional field(s), then combine the dictionaries in a loop, e.g:
cat = {d["category_id"]: d for d in categories}
res = []
for name in names:
x = name.copy()
x.update(cat[name["category_id"]])
res.append(x)
In Python 3.5+ you can use the cool new syntax:
cat = {d["category_id"]: d for d in categories}
res = [{**name, **cat[name["category_id"]]} for name in names]

Consider what you really want to do: add the level associated with each category to the names dict. So first, create a mapping from the categories to the associated levels:
cat_dict = {d['category_id']: d['level'] for d in categories}
It's then a trivial transformation on each dict in the names list:
for d in names:
d['level'] = cat_dict[d['category_id']]
The resulting names list is:
[{'category_id': '25', 'id': '1', 'level': 'advanced', 'name': 'Peter'},
{'category_id': '20', 'id': '2', 'level': 'beginner', 'name': 'Jim'},
{'category_id': '20', 'id': '3', 'level': 'beginner', 'name': 'Toni'}]