Hey guys I was scoping out the code from another post here on stackoverflow and I noticed something about a "for loop". If you change the list using "pop" or "remove" it messes up the indexes of that internal loop. It will skip whole items if you pop/remove item from the list. The way I got around it was to actually make a copy of the list to use in the "for loop" while I manipulated the other list. Im new to python.
I added to his list. My program removes anyone weighing over 180 or anyone named joe. At first I just used d_list when I notice this issue. Then I just "temp_list = d_list" which I thought made a separate copy but I guess doesn't. Then I used the copy attribute for list to make it work. This way I would not manipulate the list the "for loop" was using.
My question is, is that normal and did I go about it right to fix it? To me if the data was huge you would not want to make a copy of data. The other alternative I come up with is using a while loop instead of the outer for loop.
d_list = [ {'id':1, 'Name': 'Hannah', 'weight':150}, {'id':2, 'Name':'Andrew', 'weight':200}, {'id':3, 'Name':'Joe', 'weight':180},
{'id':4, 'Name':'Joe', 'weight':180}, {'id':5, 'Name':'Steve', 'weight':200}, {'id':6, 'Name':'Joe', 'weight':180},
{'id':7, 'Name':'George', 'weight':180}]
temp_list = d_list
#temp_list = d_list.copy()
print(d_list)
i = 0
for item in temp_list: # may make a while loop
print(item, "i = ", i, end="[")
for k, v in item.items():
print(end="*")
if (k == "weight") and (v > 180):
d_list.pop(i)
print('^popped^', i, end="") # <-- pop but you need an index
i -= 1
elif (k == "Name") and (v == "Joe"):
d_list.remove(item) # <-- remove just uses item to find and remove
print("^removed^", i, end="")
i -= 1
i += 1
print("]")
print(d_list)
print("i = ", i)
Because of the problems you identified, this kind of thing is best done by making a new list of the qualifying elements. Also, it's silly to scan all keys and values; dictionaries are meant to be used by looking up the key:
newlist = []
for item in d_list:
if item["weight"] <= 180 and item["Name"] != "Joe":
newlist.append(item)
You can then free up the old list if you're worried about "wasting" space:
del d_list
temp_list = d_list creates a reference so any changes in either list will be reflected in both so that would definitely not work. temp_list = d_list.copy() creates a shallow copy which would work as would temp_list = d_list[:] but a better approach to avoid any copying at all would be to use reversed and just remove the elements from the list:
for item in reversed(d_list):
if item.get("weight", 0) > 180 or item.get("Name") == "Joe":
d_list.remove(item)
i -= 1
If you wanted to pop you could start at the end using a range in reverse:
for i in range(len(d_list) -1 , -1, - 1):
item = d_list[i]
if item.get("weight", 0) > 180 or item.get("Name") == "Joe":
i -= 1
A third option is a list comprehension using d_list[:] to mutate the original object/list:
d_list[:] = [d for d in d_list if d.get("weight", 0) <= 180 and d.get("Name") != "Joe"]
Or combine it with a generator expression:
d_list[:] = (d for d in d_list if d.get("weight", 0) <= 180 and d.get("Name") != "Joe")
All approaches will give you the same output. Using dict.get instead of iterating over all the items is also a more efficient solution, we do two lookups per iteration instead of looking at all the keys and values in each dict.
Some timings using python3:
In [14]: %%timeit
d_list = [{'id': 1, 'Name': 'Hannah', 'weight': 150}, {'id': 2, 'Name': 'Andrew', 'weight': 200},
{'id': 3, 'Name': 'Joe', 'weight': 180},
{'id': 4, 'Name': 'Joe', 'weight': 180}, {'id': 5, 'Name': 'Steve', 'weight': 200},
{'id': 6, 'Name': 'Joe', 'weight': 180},
{'id': 7, 'Name': 'George', 'weight': 180}]
for item in reversed(d_list):
if item.get("weight", 0) > 180 or item.get("Name") == "Joe":
d_list.remove(item)
....:
100000 loops, best of 3: 4.35 µs per loop
In [15]: %%timeit
d_list = [{'id': 1, 'Name': 'Hannah', 'weight': 150}, {'id': 2, 'Name': 'Andrew', 'weight': 200},
{'id': 3, 'Name': 'Joe', 'weight': 180},
{'id': 4, 'Name': 'Joe', 'weight': 180}, {'id': 5, 'Name': 'Steve', 'weight': 200},
{'id': 6, 'Name': 'Joe', 'weight': 180},
{'id': 7, 'Name': 'George', 'weight': 180}]
for i in range(len(d_list) - 1, -1, - 1): # may make a while loop
item = d_list[i]
if item.get("weight", 0) > 180 or item.get("Name") == "Joe":
d_list.pop(i)
....:
....:
100000 loops, best of 3: 4.48 µs per loop
In [16]: %%timeit
....: d_list = [{'id': 1, 'Name': 'Hannah', 'weight': 150}, {'id': 2, 'Name': 'Andrew', 'weight': 200},
....: {'id': 3, 'Name': 'Joe', 'weight': 180},
....: {'id': 4, 'Name': 'Joe', 'weight': 180}, {'id': 5, 'Name': 'Steve', 'weight': 200},
....: {'id': 6, 'Name': 'Joe', 'weight': 180},
....: {'id': 7, 'Name': 'George', 'weight': 180}]
....: d_list[:] = (d for d in d_list if d.get("weight", 0) <= 180 and d.get("Name") != "Joe")
....:
100000 loops, best of 3: 3.23 µs per loop
In [17]: %%timeit
d_list = [{'id': 1, 'Name': 'Hannah', 'weight': 150}, {'id': 2, 'Name': 'Andrew', 'weight': 200},
{'id': 3, 'Name': 'Joe', 'weight': 180},
{'id': 4, 'Name': 'Joe', 'weight': 180}, {'id': 5, 'Name': 'Steve', 'weight': 200},
{'id': 6, 'Name': 'Joe', 'weight': 180},
{'id': 7, 'Name': 'George', 'weight': 180}]
d_list[:] = [d for d in d_list if d.get("weight", 0) <= 180 and d.get("Name") != "Joe"]
....:
100000 loops, best of 3: 2.98 µs per loop
So the list comp is the fastest followed by the gen exp. If you know the keys always exist then accessing with d["weight"] etc.. will also be faster again
Related
I would really appreciate any help on the below. I am looking to create a set of values with 1 name compiling all duplicates, with a second dict value to total another value from a list of dicts. i have compiled the below code as an example:
l = [{'id': 1, 'name': 'apple', 'price': '100', 'year': '2000', 'currency': 'eur'},
{'id': 2, 'name': 'apple', 'price': '150', 'year': '2071', 'currency': 'eur'},
{'id': 3, 'name': 'apple', 'price': '1220', 'year': '2076', 'currency': 'eur'},
{'id': 4, 'name': 'cucumber', 'price': '90000000', 'year': '2080', 'currency': 'eur'},
{'id': 5, 'name': 'pear', 'price': '1000', 'year': '2000', 'currency': 'eur'},
{'id': 6, 'name': 'apple', 'price': '150', 'year': '2022', 'currency': 'eur'},
{'id': 9, 'name': 'apple', 'price': '100', 'year': '2000', 'currency': 'eur'},
{'id': 10, 'name': 'grape', 'price': '150', 'year': '2022', 'currency': 'eur'},
]
new_list = []
for d in l:
if d['name'] not in new_list:
new_list.append(d['name'])
print(new_list)
price_list = []
for price in l:
if price['price'] not in price_list:
price_list.append(price['price'])
print(price_list)
The out put i am hoping to achieve is:
[{'name': 'apple'}, {'price': <The total price for all apples>}]
Use a dictionary whose keys are the names and values are the list of prices. Then calculate the averages of each list.
d = {}
for item in l:
d.setdefault(item['name'], []).append(int(item['price']))
for name, prices in d.items()
d[name] = sum(prices)
print(d)
Actually, I thought this was the same as yesterday's question, where you wanted the average. If you just want the total, you don't need the lists. Use a defaultdict containing integers, and just add the price to it.
from collections import defaultdict
d = defaultdict(int)
for item in l:
d[item['name']] += int(item['price'])
print(d)
This method only requires one loop:
prices = {}
for item in l:
prices.update({item['name']: prices.get(item['name'], 0) + int(item['price'])})
print(prices)
Just for fun I decided to also implement the functionality with the item and price dictionaries separated as asked in the question, which gave the following horrendous code:
prices = []
for item in l:
# get indices of prices of corresponding items
price_idx = [n+1 for n, x in enumerate(prices) if item['name'] == x.get('name') and n % 2 == 0]
if not price_idx:
prices.append({'name': item['name']})
prices.append({'price': int(item['price'])})
else:
prices[price_idx[0]].update({'price': prices[price_idx[0]]['price'] + int(item['price'])})
print(prices)
And requires the following function to retrieve prices:
def get_price(name):
for n, x in enumerate(prices):
if n % 2 == 0 and x['name'] == name:
return prices2[n+1]['price']
Which honestly completely defeats the point of having a data structure. But if it answers your question, there you go.
This could be another one:
result = {}
for item in l:
if item['name'] not in result:
result[item['name']] = {'name': item['name'], 'price': 0}
result[item['name']]['price'] += int(item['price'])
I would like to add an id key to a list of dictionaries, where each id represents the enumerated nested dictionary.
Current list of dictionaries:
current_list_d = [{'id': 0, 'name': 'Paco', 'age': 18} #all id's are 0
{'id': 0, 'name': 'John', 'age': 20}
{'id': 0, 'name': 'Claire', 'age': 22}]
Desired output:
output_list_d = [{'id': 1, 'name': 'Paco', 'age': 18} #id's are counted/enumerated
{'id': 2, 'name': 'John', 'age': 20}
{'id': 3, 'name': 'Claire', 'age': 22}]
My code:
for d in current_list_d:
d["id"]+=1
You could use a simple for loop with enumerate and update in-place the id keys in the dictionaries:
for new_id, d in enumerate(current_list_d, start=1):
d['id'] = new_id
current_list_d
[{'id': 1, 'name': 'Paco', 'age': 18},
{'id': 2, 'name': 'John', 'age': 20},
{'id': 3, 'name': 'Claire', 'age': 22}]
You can use a variable.
id_val = 1
for dict in current_list_d :
dict["id"] = id_val
id_val+=1
This question already has answers here:
Getting the difference (delta) between two lists of dictionaries
(4 answers)
Closed 3 years ago.
I have two dictionaries like below:
prev = [
{ 'id': 0, 'name': 'a' },
{ 'id': 1, 'name': 'b' },
{ 'id': 2, 'name': 'c' }
]
current = [
{ 'id': 1, 'name': 'b' },
{ 'id': 2, 'name': 'c' },
{ 'id': 3, 'name': 'e' },
{ 'id': 4, 'name': 'f' }
]
I want to get the difference of them, the result should be like below:
result = [
{ 'id': 3, 'name': 'e' },
{ 'id': 4, 'name': 'f' }
]
Only the difference of those two should appear in result list, mu solution is like below
common = []
for c in current:
for p in prev:
if c['name'] == p['name']:
common.append(c)
print(common)
I'm trying to find the common items between two and then subtract it from current list but I don't know hot to handle it. If I am using the wrong procedure to resolve the issue, is there another way I can find these two diffs?
I tried to search a lot but all the results I found were just comparing two list of integers, which in my case is list of dictionaries.
Also note that the id key is just for separating those items, let's compare by name, let's consider I want to remove commons from current and remain the rest just in current list. By that I mean that I don't need name: a and name: b from prev list.
Simple
From the data you posted, you can compare the whole dicts, so just find dicts in current that are not in prev:
new = [d for d in current if d not in prev]
print(new) # -> [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
Complex
If your real-world data might have differing ids, the solution needs to get more complex.
Since only the names are important, make a set of common names. Then you can loop over the dicts and check whether the name is in the common set.
prev = [{'id': 0, 'name': 'a'}, {'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}]
current = [{'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}, {'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
prev_names, current_names = [{d['name'] for d in x} for x in (prev, current)] # [{'c', 'b', 'a'}, {'c', 'b', 'f', 'e'}]
common_names = prev_names & current_names # {'b', 'c'}
new = [d for d in current if d['name'] not in common_names]
print(new) # -> [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
This is also easy to adapt to getting names in prev that are not common:
old = [d for d in prev if d['name'] not in common_names]
print(old) # -> [{'id': 0, 'name': 'a'}]
This will do the job
prev = [ { 'id': 0, 'name': 'a' }, { 'id': 1, 'name': 'b' }, { 'id': 2, 'name': 'c' } ]
current = [ { 'id': 1, 'name': 'b' }, { 'id': 2, 'name': 'c' }, { 'id': 3, 'name': 'e' }, { 'id': 4, 'name': 'f' } ]
common = []
for c in current:
if not any(c['id'] == p['id'] and c['name'] == p['name'] for p in prev):
common.append(c)
print(common)
Return True if any element of the iterable is true. If the iterable is empty, return False
Alse, as #wjandrea noted in the comments, this
new = [c for c in current if c not in prev]
is also a fair and nice answer. But note that it only works if comparing the whole dicts
If I understood correctly, you want only the items that appear in current and did not appear in prev.
Something like this should work
prev_names = set(map(lambda x: x['name'], prev))
new_items = [item for item in current if item['name'] not in prev_names]
new_items # [{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
Code
import itertools
list(itertools.filterfalse(lambda x: x in prev, current))
Output:
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
Based on all of the answers here is a little benchmark
import timeit
import itertools
prev = [{'id': 0, 'name': 'a'}, {'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}]
current = [{'id': 1, 'name': 'b'}, {'id': 2, 'name': 'c'}, {'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
def f1():
prev_names = set(map(lambda x: x['name'], prev))
new_items = [item for item in current if item['name'] not in prev_names]
return new_items
def f2():
common = []
for c in current:
if not any(c['id'] == p['id'] and c['name'] == p['name'] for p in prev):
common.append(c)
return common
def f3():
return list(itertools.filterfalse(lambda x: x in prev, current))
print(f1())
print(timeit.timeit("f1()", setup="from __main__ import f1"))
print(f2())
print(timeit.timeit("f2()", setup="from __main__ import f2"))
print(f3())
print(timeit.timeit("f3()", setup="from __main__ import f3"))
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
0.8235890520736575
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
2.0767332719406113
[{'id': 3, 'name': 'e'}, {'id': 4, 'name': 'f'}]
0.864271447993815
I have an unknown number of lists of product results as dictionary entries that all have the same keys. I'd like to generate a new list of products that appear in all of the old lists.
'what products are available in all cities?'
given:
list1 = [{'id': 1, 'name': 'bat', 'price': 20.00}, {'id': 2, 'name': 'ball', 'price': 12.00}, {'id': 3, 'name': 'brick', 'price': 19.00}]
list2 = [{'id': 1, 'name': 'bat', 'price': 18.00}, {'id': 3, 'name': 'brick', 'price': 11.00}, {'id': 2, 'name': 'ball', 'price': 17.00}]
list3 = [{'id': 1, 'name': 'bat', 'price': 16.00}, {'id': 4, 'name': 'boat', 'price': 10.00}, {'id': 3, 'name': 'brick', 'price': 15.00}]
list4 = [{'id': 1, 'name': 'bat', 'price': 14.00}, {'id': 2, 'name': 'ball', 'price': 9.00}, {'id': 3, 'name': 'brick', 'price': 13.00}]
list...
I want a list of dicts in which the 'id' exists in all of the old lists:
result_list = [{'id': 1, 'name': 'bat}, {'id': 3, 'name': 'brick}]
The values that aren't constant for a given 'id' can be discarded, but the values that are the same for a given 'id' must be in the results list.
If I know how many lists I've got, I can do:
results_list = []
for dict in list1:
if any(dict['id'] == d['id'] for d in list2):
if any(dict['id'] == d['id'] for d in list3):
if any(dict['id'] == d['id'] for d in list4):
results_list.append(dict)
How can I do this if I don't know how many lists I've got?
Put the ids into sets and then take the intersection of the sets.
list1 = [{'id': 1, 'name': 'steve'}, {'id': 2, 'name': 'john'}, {'id': 3, 'name': 'mary'}]
list2 = [{'id': 1, 'name': 'jake'}, {'id': 3, 'name': 'tara'}, {'id': 2, 'name': 'bill'}]
list3 = [{'id': 1, 'name': 'peter'}, {'id': 4, 'name': 'rick'}, {'id': 3, 'name': 'marci'}]
list4 = [{'id': 1, 'name': 'susan'}, {'id': 2, 'name': 'evan'}, {'id': 3, 'name': 'tom'}]
lists = [list1, list2, list3, list4]
sets = [set(x['id'] for x in lst) for lst in lists]
intersection = set.intersection(*sets)
print(intersection)
Result:
{1, 3}
Note that we call the class method set.intersection rather than the instance method set().intersection, since the latter takes intersections of its arguments with the empty set set(), and of course the intersection of anything with the empty set is empty.
If you want to turn this back into a list of dicts, you can do:
result = [{'id': i, 'name': None} for i in intersection]
print(result)
Result:
[{'id': 1, 'name': None}, {'id': 3, 'name': None}]
Now, if you also want to hold onto those attributes which are the same for all instances of a given id, you'll want to do something like this:
list1 = [{'id': 1, 'name': 'bat', 'price': 20.00}, {'id': 2, 'name': 'ball', 'price': 12.00}, {'id': 3, 'name': 'brick', 'price': 19.00}]
list2 = [{'id': 1, 'name': 'bat', 'price': 18.00}, {'id': 3, 'name': 'brick', 'price': 11.00}, {'id': 2, 'name': 'ball', 'price': 17.00}]
list3 = [{'id': 1, 'name': 'bat', 'price': 16.00}, {'id': 4, 'name': 'boat', 'price': 10.00}, {'id': 3, 'name': 'brick', 'price': 15.00}]
list4 = [{'id': 1, 'name': 'bat', 'price': 14.00}, {'id': 2, 'name': 'ball', 'price': 9.00}, {'id': 3, 'name': 'brick', 'price': 13.00}]
lists = [list1, list2, list3, list4]
sets = [set(x['id'] for x in lst) for lst in lists]
intersection = set.intersection(*sets)
all_keys = set(lists[0][0].keys())
result = []
for ident in intersection:
res = [dic for lst in lists
for dic in lst
if dic['id'] == ident]
replicated_keys = []
for key in all_keys:
if len(set(dic[key] for dic in res)) == 1:
replicated_keys.append(key)
result.append({key: res[0][key] for key in replicated_keys})
print(result)
Result:
[{'id': 1, 'name': 'bat'}, {'id': 3, 'name': 'brick'}]
What we do here is:
Look at each id in intersection and grab each dict corresponding to that id.
Find which keys have the same value in all of those dicts (one of which is guaranteed to be id).
Put those key-value pairs into result
This code assumes that:
Each dict in list1, list2, ... will have the same keys. If this assumption is false, let me know - it shouldn't be difficult to relax.
What better way (most elegant) to count the number of items {'id': n}?
'childs': {
1: [{'id': 1}, {'id': 2}, {'id': 3}],
2: [{'id': 4}],
3: [{'id': 5}, {'id': 6},]
}
>>> 6
You should answer to the comments to have a proper answer.
In the meantime, with d being the dict, I'd go with:
sum(len(x) for x in d['childs'].itervalues())
If you don't want to use for, you can do:
sum(map(len, d['childs'].itervalues()))
Or a twisted:
reduce(lambda x, y: x + len(y), d['childs'].itervalues(), 0)
But really the first version is how you would do it. It's classic Python.
As far as I understood your problem, the main task is to count the total occurrence of key "id" whenever it has value of "n". Here the dictionary name is "Childs".
count = 0
for key,value in Childs.iteritems():
if Childs[key]["id"]:
count += 1
print count
Hope it helps!!
cat = {1: [{'id': 1}, {'id': 2}, {'id': 3}], 2: [{'id': 4}], 3: [{'id': 5}, {'id': 6},]}
sum(len(x) for x in cat.values())
myDict = {'childs': {
1: [{'id': 1}, {'id': 2}, {'id': 3}],
2: [{'id': 4}],
3: [{'id': 5}, {'id': 6},]
}}
count = 0
for key in myDict["childs"]:
for item in myDict["childs"][key]:
if checkIfItemIsValid(item):
count += 1
print count
d = {
1: [{'id': 1}, {'id': 2}, {'id': 3}],
2: [{'id': 4}],
3: [{'id': 5}, {'id': 6},]
}
sum(len(i) for i in d.itervalues())
# 6
But this assumes that every member of that child value list is actually an item you want to count.
Here is one other way without using for:
sum(map(len, d.itervalues()))
In case some items may not have the 'id' key and you are only trying to count those which have that key:
sum(1 for obj in i for i in d.itervalues() if 'id' in obj)
or
len([obj for obj in i for i in d.itervalues() if 'id' in obj])