Develop Reference

Cartesian (cross) products and np.unique()

Depending on your few on my approach this is either a question about using np.unique() on awkward1 arrays or a call for a better approach:
Let a and b be two awkward1 arrays of the same outer length (number of events) but different inner lengths. For example:
a = [[1, 2], [3] , [] , [4, 5, 6]]
b = [[7] , [3, 5], [6], [8, 9]]
Let f: (x, y) -> z be a function that acts on two numbers x and y and results in the number z. For example:
f(x, y):= y - x
The idea is to compare every element in a with every element in b via f for each event and filter out the matches of a and b pairs that survive some cut applied to f. For example:
f(x, y) < 4
My approach for this is:
a = ak.from_iter(a)
b = ak.from_iter(b)
c = ak.cartesian({'x':a, 'y':b})
#c= [[{'x': 1, 'y': 7}, {'x': 2, 'y': 7}], [{'x': 3, 'y': 3}, {'x': 3, 'y': 5}], [], [{'x': 4, 'y': 8}, {'x': 4, 'y': 9}, {'x': 5, 'y': 8}, {'x': 5, 'y': 9}, {'x': 6, 'y': 8}, {'x': 6, 'y': 9}]]
i = ak.argcartesian({'x':a, 'y':b})
#i= [[{'x': 0, 'y': 0}, {'x': 1, 'y': 0}], [{'x': 0, 'y': 0}, {'x': 0, 'y': 1}], [], [{'x': 0, 'y': 0}, {'x': 0, 'y': 1}, {'x': 1, 'y': 0}, {'x': 1, 'y': 1}, {'x': 2, 'y': 0}, {'x': 2, 'y': 1}]]
diff = c['y'] - c['x']
#diff= [[6, 5], [0, 2], [], [4, 5, 3, 4, 2, 3]]
cut = diff < 4
#cut= [[False, False], [True, True], [], [False, False, True, False, True, True]]
new = c[cut]
#new= [[], [{'x': 3, 'y': 3}, {'x': 3, 'y': 5}], [], [{'x': 5, 'y': 8}, {'x': 6, 'y': 8}, {'x': 6, 'y': 9}]]
new_i = i[cut]
#new_i= [[], [{'x': 0, 'y': 0}, {'x': 0, 'y': 1}], [], [{'x': 1, 'y': 0}, {'x': 2, 'y': 0}, {'x': 2, 'y': 1}]]
It is possible that pairs with the same element from a but different elements from b survive the cut. (e.g. {'x': 3, 'y': 3} and {'x': 3, 'y': 5})
My goal is to group those pairs with the same element from a together and therefore reshape the new array into:
new = [[], [{'x': 3, 'y': [3, 5]}], [], [{'x': 5, 'y': 8}, {'x': 6, 'y': [8, 9]}]]
My only idea how to achieve this is to create a list of the indexes from a that are still present after the cut by using new_i:
i = new_i['x']
#i= [[], [0, 0], [], [1, 2, 2]]
However, I need a unique version of this list to make every index appear only once. This could be achieved with np.unique() in NumPy. But doesn't work in awkward1:
np.unique(i)
<__array_function__ internals> in unique(*args, **kwargs)
TypeError: no implementation found for 'numpy.unique' on types that implement __array_function__: [<class 'awkward1.highlevel.Array'>]
My question:
Is their a np.unique() equivalent in awkward1 and/or would you recommend a different approach to my problem?

Okay, I still don't know how to use np.unique() on my arrays, but I found a solution for my own problem:
In my previous approach I used the following code to pair up booth arrays.
c = ak.cartesian({'x':a, 'y':b})
#c= [[{'x': 1, 'y': 7}, {'x': 2, 'y': 7}], [{'x': 3, 'y': 3}, {'x': 3, 'y': 5}], [], [{'x': 4, 'y': 8}, {'x': 4, 'y': 9}, {'x': 5, 'y': 8}, {'x': 5, 'y': 9}, {'x': 6, 'y': 8}, {'x': 6, 'y': 9}]]
However, with the nested = True parameter from ak.cartesian() I get a list grouped by the elements of a:
c = ak.cartesian({'x':a, 'y':b}, axis = 1, nested = True)
#c= [[[{'x': 1, 'y': 7}], [{'x': 2, 'y': 7}]], [[{'x': 3, 'y': 3}, {'x': 3, 'y': 5}]], [], [[{'x': 4, 'y': 8}, {'x': 4, 'y': 9}], [{'x': 5, 'y': 8}, {'x': 5, 'y': 9}], [{'x': 6, 'y': 8}, {'x': 6, 'y': 9}]]]
After the cut I end up with:
new = c[cut]
#new= [[[], []], [[{'x': 3, 'y': 3}, {'x': 3, 'y': 5}]], [], [[], [{'x': 5, 'y': 8}], [{'x': 6, 'y': 8}, {'x': 6, 'y': 9}]]]
I extract the y values and reduce the most inner layer of the nested lists of new to only one element:
y = new['y']
#y= [[[], []], [[3, 5]], [], [[], [8], [8, 9]]]
new = ak.firsts(new, axis = 2)
#new= [[None, None], [{'x': 3, 'y': 3}], [], [None, {'x': 5, 'y': 8}, {'x': 6, 'y': 8}]]
(I tried to use ak.firsts() with axis = -1 but it seems to be not implemented yet.)
Now every most inner entry in new belongs to exactly one element from a. By replacing the current y of new with the previously extracted y I end up with my desired result:
new['y'] = y
#new= [[None, None], [{'x': 3, 'y': [3, 5]}], [], [None, {'x': 5, 'y': [8]}, {'x': 6, 'y': [8, 9]}]]
Anyway, should you know a better solution, I'd be pleased to hear it.

Merge values of list of dictionaries whose values are dictionaries

I have a list of dictionaries like
[
{'a': {'q': 1}, 'b': {'r': 2}, 'c': {'s': 3}},
{'a': {'t': 4}, 'b': {'u': 5}, 'c': {'v': 6}},
{'a': {'w': 7}, 'b': {'x': 8}, 'c': {'z': 9}}
]
and I want the output to be
{
'a': {'q': 1, 't': 4, 'w': 7},
'b': {'r': 2, 'u': 5, 'x': 8},
'c': {'s': 3, 'v': 6, 'z': 9}
}

There are several ways of doing this, one with usage of collections.defaultdict:
import collections
result = collections.defaultdict(dict)
lst = [
{'a': {'q': 1}, 'b': {'r': 2}, 'c': {'s': 3}},
{'a': {'t': 4}, 'b': {'u': 5}, 'c': {'v': 6}},
{'a': {'w': 7}, 'b': {'x': 8}, 'c': {'z': 9}}
]
for dct in lst:
for key, value in dct.items():
result[key].update(value)
print(result)

Permutation of items with quantities

Using python I have to get all the permutations of given subset using python.
I used itertools.permutation but result is a bit different.
Think of a machine and it has a maximum capacity, and we have products can be produced together, and we have to fill the capacity of machine.
Output format is not important, I used a dictionary to describe it. I will make a calculation after getting this combinations.
For example :
products = {'x','y','z','a'}
machine_capcacity = 8
#required output as follows:
{'x':5,'y':1,'z':1,'a':1}
{'x':4,'y':2,'z':1,'a':1}
{'x':4,'y':1,'z':2,'a':1}
{'x':4,'y':1,'z':1,'a':2}
{'x':3,'y':3,'z':1,'a':1}
{'x':3,'y':1,'z':3,'a':1}
{'x':3,'y':1,'z':1,'a':3}
{'x':3,'y':2,'z':2,'a':1}
{'x':3,'y':2,'z':1,'a':2}
{'x':3,'y':1,'z':2,'a':2}
{'x':2,'y':4,'z':1,'a':1}
# ...
{'x':6,'y':1,'z':1} # This can't be in results,since need at least 1 element of product
{'x':4,'y':1,'z':1,'a':1} # This can't be in results,since we need to fill the capacity
And we dont want repeating elements:
{'x':5,'y':1,'z':1,'a':1}
and
{'a':1,'y':1,'z':1,'x':5}
is same thing for us.

Here is a solution not relying on itertools since it's getting contrived with all the constraints (a product yielding unique results and a minimum of 1 appearance per product):
products = {'x','y','z','a'}
machine_capacity=8
def genCap(capacity = machine_capacity,used = 0):
if used == len(products)-1: yield capacity,None
else:
for i in range(1,2+capacity-len(products)+used):
yield i,genCap(capacity-i,used+1)
def printCaps(caps,current = []):
if caps is None:
print(dict(zip(products,current)))
return
for i in caps:
printCaps(i[1],current+[i[0]])
printCaps(genCap())
might be optimize-able with tail recursion and the like. Looks almost like groupby, but I can't see an easy way to use that.
For posterity I leave my old solution - product repeats counts, so filtering it becomes a problem of it's own:
You confused product with permutation. Here is a quick solution using itertools product, and the Counter collection to create the output you want:
from collections import Counter
from itertools import product
products = {'x','y','z','a'}
machine_capacity=8
for x in filter(lambda x: len(x) == len(products),
map(Counter,product(products,repeat=machine_capacity))):
print(dict(x))
Note both product and map are lazy, so they won't be evaluated until you need them. Counter provides the output you want, and converting to dict cleans it up. Note no order is guaranteed anywhere. The filter is used to make sure all your products appear at least once (length of counter equals that of products) - and it is also lazy, so only evaluated when you need it.

You can use a recursive function to find all possible combinations of the values in range(machine_capacity) that both sum to 8 and are unique. Then, the elements in products can be mapped to each element in the sublists of the combinations found:
products = ['x','y','z','a']
machine_capacity = 8
def combinations(d, current = []):
if len(current) == len(products):
yield current
else:
for i in range(machine_capacity):
if sum(current+[i]) <= machine_capacity:
yield from combinations(d, current+[i])
data = [dict(zip(products, i)) for i in filter(lambda x:sum(x) == 8 and len(x) == len(set(x)), combinations(machine_capacity))]
Output:
[{'a': 5, 'x': 0, 'z': 2, 'y': 1}, {'a': 4, 'x': 0, 'z': 3, 'y': 1}, {'a': 3, 'x': 0, 'z': 4, 'y': 1}, {'a': 2, 'x': 0, 'z': 5, 'y': 1}, {'a': 5, 'x': 0, 'z': 1, 'y': 2}, {'a': 1, 'x': 0, 'z': 5, 'y': 2}, {'a': 4, 'x': 0, 'z': 1, 'y': 3}, {'a': 1, 'x': 0, 'z': 4, 'y': 3}, {'a': 3, 'x': 0, 'z': 1, 'y': 4}, {'a': 1, 'x': 0, 'z': 3, 'y': 4}, {'a': 2, 'x': 0, 'z': 1, 'y': 5}, {'a': 1, 'x': 0, 'z': 2, 'y': 5}, {'a': 5, 'x': 1, 'z': 2, 'y': 0}, {'a': 4, 'x': 1, 'z': 3, 'y': 0}, {'a': 3, 'x': 1, 'z': 4, 'y': 0}, {'a': 2, 'x': 1, 'z': 5, 'y': 0}, {'a': 5, 'x': 1, 'z': 0, 'y': 2}, {'a': 0, 'x': 1, 'z': 5, 'y': 2}, {'a': 4, 'x': 1, 'z': 0, 'y': 3}, {'a': 0, 'x': 1, 'z': 4, 'y': 3}, {'a': 3, 'x': 1, 'z': 0, 'y': 4}, {'a': 0, 'x': 1, 'z': 3, 'y': 4}, {'a': 2, 'x': 1, 'z': 0, 'y': 5}, {'a': 0, 'x': 1, 'z': 2, 'y': 5}, {'a': 5, 'x': 2, 'z': 1, 'y': 0}, {'a': 1, 'x': 2, 'z': 5, 'y': 0}, {'a': 5, 'x': 2, 'z': 0, 'y': 1}, {'a': 0, 'x': 2, 'z': 5, 'y': 1}, {'a': 1, 'x': 2, 'z': 0, 'y': 5}, {'a': 0, 'x': 2, 'z': 1, 'y': 5}, {'a': 4, 'x': 3, 'z': 1, 'y': 0}, {'a': 1, 'x': 3, 'z': 4, 'y': 0}, {'a': 4, 'x': 3, 'z': 0, 'y': 1}, {'a': 0, 'x': 3, 'z': 4, 'y': 1}, {'a': 1, 'x': 3, 'z': 0, 'y': 4}, {'a': 0, 'x': 3, 'z': 1, 'y': 4}, {'a': 3, 'x': 4, 'z': 1, 'y': 0}, {'a': 1, 'x': 4, 'z': 3, 'y': 0}, {'a': 3, 'x': 4, 'z': 0, 'y': 1}, {'a': 0, 'x': 4, 'z': 3, 'y': 1}, {'a': 1, 'x': 4, 'z': 0, 'y': 3}, {'a': 0, 'x': 4, 'z': 1, 'y': 3}, {'a': 2, 'x': 5, 'z': 1, 'y': 0}, {'a': 1, 'x': 5, 'z': 2, 'y': 0}, {'a': 2, 'x': 5, 'z': 0, 'y': 1}, {'a': 0, 'x': 5, 'z': 2, 'y': 1}, {'a': 1, 'x': 5, 'z': 0, 'y': 2}, {'a': 0, 'x': 5, 'z': 1, 'y': 2}]

Permuting a order of items in a list in python

I have a list of dictionary items
[{'x': 0, 'y': 0}, {'x': 1, 'y': 0}, {'x': 2, 'y': 2}]
I want to have an array of "array of dictionaries" with all the maximum permutation order of the list for example for the above array it would be (3 factorial ways)
[[{'x': 0, 'y': 0}, {'x': 1, 'y': 0}, {'x': 2, 'y': 2}],
[{'x': 0, 'y': 0}, {'x': 2, 'y': 2}, {'x': 1, 'y': 0}],
[{'x': 1, 'y': 0}, {'x': 0, 'y': 0}, {'x': 2, 'y': 2}],
[{'x': 1, 'y': 0}, {'x': 2, 'y': 2}, {'x': 0, 'y': 0}],
[{'x': 2, 'y': 2}, {'x': 1, 'y': 0}, {'x': 0, 'y': 0}],
[{'x': 2, 'y': 2}, {'x': 0, 'y': 0}, {'x': 1, 'y': 0}]]

itertools can do permutations
#!python2
import itertools
yourlist = [{'x': 0, 'y': 0}, {'x': 1, 'y': 0}, {'x': 2, 'y': 2}]
for seq in itertools.permutations(yourlist):
print seq
'''
({'y': 0, 'x': 0}, {'y': 0, 'x': 1}, {'y': 2, 'x': 2})
({'y': 0, 'x': 0}, {'y': 2, 'x': 2}, {'y': 0, 'x': 1})
({'y': 0, 'x': 1}, {'y': 0, 'x': 0}, {'y': 2, 'x': 2})
({'y': 0, 'x': 1}, {'y': 2, 'x': 2}, {'y': 0, 'x': 0})
({'y': 2, 'x': 2}, {'y': 0, 'x': 0}, {'y': 0, 'x': 1})
({'y': 2, 'x': 2}, {'y': 0, 'x': 1}, {'y': 0, 'x': 0})
'''

Despite the comments, if you are still messed with how to solve your issue, consider the following.
Strategy: Make use of permutations from itertoolswhich returns a list of tuples in this case. Then, iterating through to convert list of tuples to list of lists to match with your required output.
Here is how you could do:
>>> import itertools
>>> lst = [{'x': 0, 'y': 0}, {'x': 1, 'y': 0}, {'x': 2, 'y': 2}]
>>> [list(elem) for elem in list(itertools.permutations(lst))]
[[{'x': 0, 'y': 0}, {'x': 1, 'y': 0}, {'x': 2, 'y': 2}],
[{'x': 0, 'y': 0}, {'x': 2, 'y': 2}, {'x': 1, 'y': 0}],
[{'x': 1, 'y': 0}, {'x': 0, 'y': 0}, {'x': 2, 'y': 2}],
[{'x': 1, 'y': 0}, {'x': 2, 'y': 2}, {'x': 0, 'y': 0}],
[{'x': 2, 'y': 2}, {'x': 0, 'y': 0}, {'x': 1, 'y': 0}],
[{'x': 2, 'y': 2}, {'x': 1, 'y': 0}, {'x': 0, 'y': 0}]]

comparing dicts with additional values

I have 2 lists of python dicts: realList and expectedList
I want realList to be considered equal to expectedList if there is one, and only one, "sub-dict" on realList for each dict of expectedList
By sub-dict I mean a dict with at least the same key/values, but which might have additional key/values.
so, for example:
realDict = [{'a': 1, 'b': 2}, {'a': 2, 'b': 3}]
==
expectedDict = [{'a': 1}, {'a': 2, 'b': 3}]
realDict = [{'a': 1, 'b': 2}, {'a': 2, 'b': 3}]
!=
expectedDict = [{'a': 2}, {'a': 2, 'b': 3}]
realDict = [{'a': 1, 'b': 2}, {'a': 1, 'b': 3}]
!=
expectedDict = [{'a': 1}]
Any modules to help? Only idea I can think of is iterating over expectedDict and removing a dict from it and from realDict whenever they match. Then, in the end, both must be [].

This works, but I'm unsure how performant it is:
def complete_subset(real_list, expected_list):
real_set_list = [set(d.items()) for d in real_list]
expected_set_list = [set(d.items()) for d in expected_list]
while len(real_set_list):
real_len = len(real_set_list)
i = 0
for real_set in real_set_list:
for expected_set in expected_set_list:
if not len(expected_set - real_set):
real_set_list.remove(real_set)
expected_set_list.remove(expected_set)
i = i + 1
if i == real_len:
break
return (not len(real_set_list)) and (not len(expected_set_list))
Here are my tests:
print complete_subset([{'a': 1, 'b': 2}, {'a': 2, 'b': 3}], [{'a': 1}, {'a': 2, 'b': 3}]) == True
print complete_subset([{'a': 1, 'b': 2}, {'a': 2, 'b': 3}], [{'a': 2}, {'a': 2, 'b': 3}]) == False
print complete_subset([{'a': 1, 'b': 2}, {'a': 2, 'b': 3}], [{'a': 1}]) == False
print complete_subset([{'a': 1, 'b': 2}, {'a': 1, 'b': 2}], [{'a': 1}, {'b': 2}]) == True
print complete_subset([
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
], [
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
{'a': 1, 'b': 2, 'c': 3, 'd': 4},
{'a': 1, 'b': 2, 'c': 3},
{'a': 1, 'b': 2},
{'a': 1},
{'a': 1, 'c': 3, 'd': 4, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'c': 3, 'e': 5, 'f': 6, 'h': 8, 'i': 9, 'j': 10},
{'a': 1, 'b': 2, 'd': 4, 'e': 5, 'g': 7, 'h': 8, 'j': 10},
]) == True

In case your two lists are to be compared element by element, you can do it this way.
def compare(realDict, expectedDict):
if len(readDict) != len(expectedDict):
return False
for d1, d2 in zip(realDict, expectedDict):
for key in d2:
if key not in d1 or d1[key] != d2[key]:
return False
return True