Develop Reference

python remove duplicate element from list

In python i am getting a list in a variable like :
[ {'store_id': '321', 'first_name': 'A', 'name': 'A'},
{'second_id': '322', 'first_name': 'B', 'name': 'B', },
{'second_id': '323', 'second_name': 'c', 'name': 'c', },
{'second_id': '324', 'second_name': 'A', 'name': 'A', },
]
what i actually want is i want a list without duplicating the name . if it occur once then i want to remove t and create a new list with distinct data.i am stuck here i want all data in new list. how can i remove the duplicate data from the list .
in my case i want a ne list like :
Either
{'second_id': '322', 'first_name': 'B', 'name': 'B', },
{'second_id': '323', 'second_name': 'c', 'name': 'c', },
{'second_id': '324', 'second_name': 'A', 'name': 'A', },
]
Or
[ {'store_id': '321', 'first_name': 'A', 'name': 'A'},
{'second_id': '322', 'first_name': 'B', 'name': 'B', },
{'second_id': '323', 'second_name': 'c', 'name': 'c', },
]
And the code after that i am getting this is given below:
result = {}
data = request.POST
teamName = []
First = Test.objects.filter(d=data.get('id')).values(
'first_id','first_name').annotate(id=F('first_id'),name=F('first_name')).distinct()
Second = Test.objects.filter(id=data.get('id')).values(
'second_id','second_name').annotate(id=F('second_id'),name=F('second_name')).distinct()
combined_results = list(chain(First, Second))
for team in combined_results:
team['text'] = team['name']
team['id'] = team['id']
teamName.append(team)
if not combined_results:
result['status'] = False
result['data'] = ['Data not found']
else:
result['status'] = True
result['data'] = teamName
return JsonResponse(result)

This should give you the second form
names = set()
newList = []
for d in mylist:
if d['name'] in names:
continue
else:
newList.append(d)
names.add(d['name'])
print(newList)
Output:
[{'store_id': '321', 'first_name': 'A', 'name': 'A'},
{'second_id': '322', 'first_name': 'B', 'name': 'B'},
{'second_id': '323', 'second_name': 'c', 'name': 'c'}]
EDIT:
If you want the first form, you will have to sort your original list in descending order of store_id/second_id using:
mylist = sorted(mylist, key=lambda x: x.get('store_id') or x.get('second_id'), reverse=True)
and then filter the list as earlier.

using list comprehension:
lst = [ {'store_id': '321', 'first_name': 'A', 'name': 'A'},
{'second_id': '322', 'first_name': 'B', 'name': 'B', },
{'second_id': '323', 'second_name': 'c', 'name': 'c', },
{'second_id': '324', 'second_name': 'A', 'name': 'A', },
]
uniqueLst = []
lst2 = []
[[lst2.append(x), uniqueLst.append(x['name'])] for x in lst if x['name'] not in uniqueLst]
print(lst2)

I have tried to solve your issue, It may not be the best approach but it does the job.
lets suppose you have this list
orignal_list = [
{'store_id': '321', 'first_name': 'A', 'name': 'A'},
{'second_id': '322', 'first_name': 'B', 'name': 'B', },
{'second_id': '323', 'second_name': 'c', 'name': 'c', },
{'second_id': '324', 'second_name': 'A', 'name': 'A', },
]
so we can make a function which can take original list and return a filtered list.
def filter_list(original_list):
# temp list which will contains the filtered records
temp_list = list()
# loop over original_list
for original_item in original_list:
# get values of dict expect id because every element have different id
original_values = [*original_item.values()][1:]
flag = True
# loop over temp_list
for temp_items in temp_list:
# get values expect the id
temp_values = [*temp_items.values()][1:]
# check if sample/record/item already exists
if original_values == temp_values:
flag = False
if flag:
temp_list.append(original_item)
return temp_list
I hope this will solve your problem I have tried to explain the code using comments,
let me know if you did not understand any thing.

I did something similar with sets. Since A is a set of elements:
for i in A:
if i not in B:
B.append(i)

Returning value from list within a dict

Attempting to return a specific value of a key that's in a list within a dict. I've tried iterating over items, searching specific ID's and not quite getting the result I'm after.
{
'activeAccount': True,
'country': 'USA',
'state': 'CA',
'users': [
{
'id': 'A',
'firstName': 'Tom',
'lastName': 'Cruise',
'creditScore': '713',
'balance': '65897.22',
'debts': '12414.12',
'savings': '15231.23'
},
{
'id': 'B',
'firstName': 'Jon',
'lastName': 'Snow',
'creditScore': '648',
'balance': '12366.23',
'debts': '522',
'savings': '121588'
},
{
'id': 'C',
'firstName': 'Rick',
'lastName': 'Sanchez',
'creditScore': '655',
'balance': '556425.33',
'debts': '0',
'savings': '125122.23'
},
{
'id': 'D',
'firstName': 'Monty',
'lastName': 'Python',
'creditScore': '815',
'balance': '4512699.13',
'debts': '4.25',
'savings': '5499865.12'
}
]
}
How would I retrieve a specific value of, for example, the credit score of users[2] to return the value?
655
I've tried the solution of cScore = (dict['users'][2]['creditScore']), but this isn't ideal in situations of the objects changing order, etc.

You have to iterate through the list of dicts, looking for one with an id of 2. Then you return the credit score from that dict. Use this starter code:
for account in data_base:
if account['id'] == 2:
score = account['creditScore']
break
Your underlying problem is that you chose a data representation that doesn't match your data access needs. If you always reference your accounts by id, then change your list to a dict, keyed by that value. If you look up accounts through various means, then you need to investigate some data base format that suits your use cases.

The dictionary value of 'users' is a list. So you have to iterate through the list and search for the specific id.
Here's how you can do it.
for i in cust['users']:
if i['id'] == '2':
print (i['creditScore'])
I searched for 'id' == 'D'. The output was 815
The dictionary is defined as :
cust = {
'activeAccount': True,
'country': 'USA',
'state': 'CA',
'users': [
{
'id': 'A',
'firstName': 'Tom',
'lastName': 'Cruise',
'creditScore': '713',
'balance': '65897.22',
'debts': '12414.12',
'savings': '15231.23'
},
{
'id': 'B',
'firstName': 'Jon',
'lastName': 'Snow',
'creditScore': '648',
'balance': '12366.23',
'debts': '522',
'savings': '121588'
},
{
'id': 'C',
'firstName': 'Rick',
'lastName': 'Sanchez',
'creditScore': '655',
'balance': '556425.33',
'debts': '0',
'savings': '125122.23'
},
{
'id': 'D',
'firstName': 'Monty',
'lastName': 'Python',
'creditScore': '815',
'balance': '4512699.13',
'debts': '4.25',
'savings': '5499865.12'
}
]
}

If you need to find the user by ID within of a list of users you can use a filter function:
dict = {
'activeAccount': True,
'country': 'USA',
'state': 'CA',
'users': [
{
'id': 'A',
'firstName': 'Tom',
'lastName': 'Cruise',
'creditScore': '713',
'balance': '65897.22',
'debts': '12414.12',
'savings': '15231.23'
},
{
'id': 'B',
'firstName': 'Jon',
'lastName': 'Snow',
'creditScore': '648',
'balance': '12366.23',
'debts': '522',
'savings': '121588'
},
{
'id': 'C',
'firstName': 'Rick',
'lastName': 'Sanchez',
'creditScore': '655',
'balance': '556425.33',
'debts': '0',
'savings': '125122.23'
},
{
'id': 'D',
'firstName': 'Monty',
'lastName': 'Python',
'creditScore': '815',
'balance': '4512699.13',
'debts': '4.25',
'savings': '5499865.12'
}
]
}
# get the user with ID:"D"
user = list(filter(lambda u: u["id"] == "D", dict["users"]))[0]
# get the value
cScore = user["creditScore"] # <-- Output: 815
Description
lambda u: u["id"] == "D" means: return True if the property "id" of a given element is "D".
It could be rewritten as a plain function:
def check_D(user):
return user["id"] == "D"
user = list(filter(check_D, dict["users"]))[0]
filter( <condition> , dict["users"]) means: to iterate through the list dict["users"] and to return all elements that meet the condition (the "lambda" in this case, or it could be any function that return True or False).
user = list( ... )[0] since a filter function returns an object, it's need to convert the object into a list and to get a first element of the list.
Another option
You can built a list comprehension with a condition (id == "D") and take a first element of the list:
creditScore = [usr["creditScore"] for usr in dict["users"] if usr["id"] == "D"][0]
print(creditScore) # <-- Output: 815

How to optimize a nested for loop with filtering in python

I'm trying to optimize a nested for-loop with filtering, the code looks like:
user_ids = ['A', 'B', 'C']
all_dict_1 = [
{
'id': 'all',
'user_id': 'B',
},
{
'id': 'foo',
'user_id': 'B',
},
{
'id': 'bar',
'user_id': 'A',
},
{
'id': 'bar',
'user_id': 'D',
},
]
all_dict_2 = [
{
'id': 'all',
'percentage': 0.2,
},
{
'id': 'foo',
'percentage': 0.3,
},
]
def _filter(dict_1, dict_2, user_ids):
if str(dict_1['user_id']) in user_ids:
if dict_2['id'] == 'all':
dict_1['percentage'] = dict_2['percentage']
return dict_1
if dict_1['id'] == dict_2['id']:
dict_1['percentage'] = dict_2['percentage']
return dict_1
return None
hits = [_filter(x, y, user_ids) for x in all_dict_1 for y in all_dict_2]
hits = [i for i in hits if i] # Removing None values
the all_dict_1 list is particularly long (thousands of objects), so the function takes more than 1s to run 😕
Are there any libraries or technics to make it quicker?

The logic in your question can be reduced to the following list-comprehension, which should be slightly faster:
>>> hits = [{**x, 'percentage': y['percentage']}
for x in all_dict_1 for y in all_dict_2
if x['user_id'] in user_ids and
(y['id'] == 'all' or x['id'] == y['id'])]
>>> hits
[{'id': 'all', 'user_id': 'B', 'percentage': 0.2},
{'id': 'foo', 'user_id': 'B', 'percentage': 0.2},
{'id': 'foo', 'user_id': 'B', 'percentage': 0.3},
{'id': 'bar', 'user_id': 'A', 'percentage': 0.2}]

Make user_ids a set to speed up item in user_ids tests. Filter with this first, since it rejects entries that you don't have to process at all. Use filter to avoid repeated global name lookups.
user_ids = {'A', 'B', 'C'}
filtered_dict_1 = filter(
lambda item, ids=user_ids: item['user_id'] in ids,
all_dict_1
)
Change all_dict_2 into an actual dict to allow O(1) access instead of O(n) scanning. When iterating over your entries to change them, directly access the required percentage or use an explicit default.
all_dict_2 = {
'foo': 0.3,
}
def add_percentage(item, default=0.2, percentages=all_dict_2):
item["percentage"] = percentages.get(item['id'], default)
return item
Apply the transformation using map to avoid repeated lookups of your transformation function.
hits = list(map(add_percentage, filtered_dict_1))

Add dictionary item in list to a dictionary

Hi guys I'm pretty lost with this simple problem. I have a dictionary and a list of dictionaries in python and I want to loop over the list to add each dictionary to the first dictionary but somehow it just adds the last dictionary with the solution I came up with. I'm using Python 3.6.5
This is what I've tried:
res = []
dictionary = {"id": 1, "name": "Jhon"}
dictionary_2 = [
{"surname": "Doe", "email": "jd#example.com"},
{"surname": "Morrison", "email": "jm#example.com"},
{"surname": "Targetson", "email": "jt#example.com"}
]
for info in dictionary_2:
aux_dict = dictionary
aux_dict["extra"] = info
res.append(aux_dict)
print(res)
What I expect is:
[{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Doe', 'email': 'jd#example.com'}},
{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Morrison', 'email': 'jm#example.com'}},
{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Targetson', 'email': 'jt#example.com'}}]
And this is what I get
[{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Targetson', 'email': 'jt#example.com'}},
{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Targetson', 'email': 'jt#example.com'}},
{'id': 1, 'name': 'Jhon', 'extra': {'surname': 'Targetson', 'email': 'jt#example.com'}}]
This is probably a duplicate of some other question but I can't manage to find it

This is because you keep adding the same aux_dict to res.
What you probably want to do is make a copy of dictionary; just assigning it to aux_dict does not make a copy.
This is how you make a (shallow) copy:
aux_dict = dictionary.copy()
That would be sufficient in your case.

You can achieve this in one line using list comprehension and dict constructor:
dictionary = {"id": 1, "name": "Jhon"}
dictionary_2 = [
{"surname": "Doe", "email": "jd#example.com"},
{"surname": "Morrison", "email": "jm#example.com"},
{"surname": "Targetson", "email": "jt#example.com"}
]
# ...
res = [dict(dictionary, extra=item) for item in dictionary_2]
# ...
print(res)

Appending/Merging in Python

I have the following structure:
[
{
u'123456': {'name': "Bill"},
u'234567': {'name': "Dave"},
u'345678': {'name': "Tom"}
},
]
During a for loop new items are added to the list using the extend function. Unfortunately this results in the following structure:
[
{
u'123456': {'name': "Bill"},
u'234567': {'name': "Dave"},
u'345678': {'name': "Tom"}
},
{
u'555555': {'name': "Steve"},
u'666666': {'name': "Michael"},
u'777777': {'name': "George"}
}
]
The intended result is actually a flat structure such in the following:
[
{
u'123456': {'name': "Bill"},
u'234567': {'name': "Dave"},
u'345678': {'name': "Tom"},
u'555555': {'name': "Steve"},
u'666666': {'name': "Michael"},
u'777777': {'name': "George"}
}
]
Is it possible to append to the list so that the structure gets built in a flat way.
or
Is it possible to flatten after the loop has finished?

If your list is named l you could use l[0].update(new_dict).
Example:
l = [{u'123456': {'name': "Bill"}}]
l[0].update({u'234567': {'name': "Dave"}})
print(l)
Nice formatted output is:
[
{
u'123456': {'name': 'Bill'},
u'234567': {'name': 'Dave'}
}
]

Where you currently have something like this:
mylist.extend(newdict)
You should use this:
mylist[0].update(newdict)

You can add the items of both dictionaries together:
>>> mylist = [
{
u'123456': {'name': "Bill"},
u'234567': {'name': "Dave"},
u'345678': {'name': "Tom"}
},
]
>>> mydict = {
u'555555': {'name': "Steve"},
u'666666': {'name': "Michael"},
u'777777': {'name': "George"}
}
>>> [dict(mylist[0].items() + mydict.items())]
[{u'123456': {'name': 'Bill'}, u'555555': {'name': 'Steve'}, u'777777': {'name': 'George'}, u'666666': {'name': 'Michael'}, u'345678': {'name': 'Tom'}, u'234567': {'name': 'Dave'}}]
Although it's more clean to just do .update():
>>> mylist[0].update(mydict)

You can use .update(), however this will overwrite values if you'll have duplicated keys.
def flatten(results):
newresult = {}
for subdict : results:
newresult.update(subdict)
return [newresult]