Develop Reference

finding where 2d list overlaps by value - python

One numpy 2d-array looks like this:
[[0 1 2]
[1 5 0]]
Another numpy 2d array which looks like this:
[[0 0 0 0 0 1 1 1 1 1 1 2 2 2 2 2 2]
[0 1 3 4 8 0 1 3 6 7 8 0 1 2 3 6 8]]
I want to get just the places where they "overlap":
[[0 2]
[1 0]]
without using a for loop

You can use intersect1d.
I called n1 the first array and n2 the second one.
The result is not exactly what you expected, but I believe it's correct.
intersection = np.intersect1d(n1, n2)
print(intersection)
[0 1 2]

Related

Given a matrix M n*n (containing only 0 and 1), I want to build the matrix that contains a 1 in position (i, j) if and only if there is at least a 1 in the bottom-right submatrix M[i:n, j:n]
Please note that I know there are optimal algorithm to compute this, but for performance reasons, I'm looking for a solution using numpy (so the algorithm is fully compiled)
Example:
Given this matrix:
0 0 0 0 1
0 0 1 0 0
0 0 0 0 1
1 0 1 0 0
I'm looking for a way to compute this matrix:
0 0 0 0 1
0 0 1 1 1
0 0 1 1 1
1 1 1 1 1
Thanks

Using numpy, you can accumulate the maximum value over each axis:
import numpy as np
M = np.array([[0,0,0,0,1],
[0,0,1,0,0],
[0,0,0,0,1],
[1,0,1,0,0]])
M = np.maximum.accumulate(M)
M = np.maximum.accumulate(M,axis=1)
print(M)
[[0 0 0 0 1]
[0 0 1 1 1]
[0 0 1 1 1]
[1 1 1 1 1]]
Note: This matches your example result (presence of 1 in top-left quadrant). Your explanations of the logic would produce a different result however
If we go with M[i:n,j:n] (bottom-right):
M = np.array([[0,0,0,0,1],
[0,0,1,0,0],
[0,0,0,0,1],
[1,0,1,0,0]])
M = np.maximum.accumulate(M[::-1,:])[::-1,:]
M = np.maximum.accumulate(M[:,::-1],axis=1)[:,::-1]
print(M)
[[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 0 0]]
It is essentially the same approach except with reversed accumulation on the axes

I'm looking to speed up my code that takes ~80 milliseconds for 300 sets to generate multiset_permutations from sympy. Ideally this would take only a few milliseconds; also the more items, the slower it gets.
What can I do to make my code faster? Multi-threading? Or convert to C? Any help here on speeding this up would be greatly appreciated.
import numpy as np
from time import monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_indices = 5
num_items = 300
indices = np.array([list(multiset_permutations(list(range(num_indices)))) for _ in range(num_items)])
print(indices)
[[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
...
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]
[[0 1 2 3 4]
[0 1 2 4 3]
[0 1 3 2 4]
...
[4 3 1 2 0]
[4 3 2 0 1]
[4 3 2 1 0]]]
print('Multiset Perms:', milli_time() - start_time, 'milliseconds')
Multiset Perms: 88 milliseconds
** Code Update to Reduce extra computations by 2/3 **
import itertools
import numpy as np
from time import time, monotonic
from sympy.utilities.iterables import multiset_permutations
milli_time = lambda: int(round(monotonic() * 1000))
start_time = milli_time()
num_colors = 5
color_range = list(range(num_colors))
total_media = 300
def all_perms(elements):
if len(elements) <= 1:
yield elements # Only permutation possible = no permutation
else:
# Iteration over the first element in the result permutation:
for (index, first_elmt) in enumerate(elements):
other_elmts = elements[:index]+elements[index+1:]
for permutation in all_perms(other_elmts):
yield [first_elmt] + permutation
multiset = list(multiset_permutations(color_range))
# multiset = list(itertools.permutations(color_range))
# multiset = list(all_perms(color_range))
_range = range(total_media)
perm_indices = np.array([multiset for _ in _range])
print('Multiset Perms:', milli_time() - start_time)
Multiset Perms: 34 milliseconds

First of all, you do not need to recompute the permutations.
Moreover, np.array([multiset for _ in _range]) is expensive because Numpy have to transform multiset total_media times. You can solve that using np.array([multiset]).repeat(total_media, axis=0).
Finally, sympy is not the fastest implementation to perform such a computation. A faster implementation consists in using itertools instead:
num_colors = 5
total_media = 300
color_range = list(range(num_colors))
multiset = list(set(itertools.permutations(color_range)))
perm_indices = np.array([multiset], dtype=np.int32).repeat(total_media, axis=0)
However, this itertools-based implementation do not preserve the order of the permutations. If this is important, you can use np.sort on the Numpy array converted from multiset (with a specific axis and before applying repeat).
On my machine, this takes about 0.15 ms.

I have problem with execution of np.argpartition
I have nd.array
example = np.array([[5,6,7,3,4],[1,2,3,7,5],[6,7,4,2,3],[1,2,3,5,9],[2,3,6,1,2,]])
out: [[5 6 7 3 4]
[1 2 3 7 5]
[6 7 4 2 3]
[1 2 3 5 9]
[2 3 6 1 2]]
I can get indices for sorted array by np.argsort
print(np.argsort(example))
out:
[[3 4 0 1 2]
[0 1 2 4 3]
[3 4 2 0 1]
[0 1 2 3 4]
[3 0 4 1 2]]
I want to use np.argsort to economy some time for executing, because I need only 3 sorted element in each row of this array. I use this code to do it:
print(np.argpartition(example, 3, axis=1))
out: [[3 4 0 1 2]
[1 0 2 4 3]
[3 4 2 0 1]
[1 0 2 3 4]
[3 4 0 1 2]]
I expect that the first three indices of each row will match the indices in the sorted array, but this is not the caseю That doesn't work . I don't understand what I did wrong.

np.argpartition(example, k, axis=1) does not return sorted array for first k elements. It only returns indices such that only (k+1)th element is sorted. If you see in your output, only the 4th element matches with argsort()
If you want first three sorted elements, you have to give a list for k parameter
index_array = np.argpartition(example, [0,1,2], axis=1)
print(np.take_along_axis(example,index_array, axis=1)) ##this will give you first 3 sorted elements

I need to change value for items in a numpy array on the basis of their neighbours values.
More specifically, let's suppose that I have just 3 possible values for each item in a numpy representing an image. Let's suppose my numpy is the following one:
[[1,1,1,1,1,1,1],
[1,1,1,1,1,1,1],
[1,1,1,2,1,1,1],
[1,1,1,2,1,1,1],
[1,1,1,1,1,1,1],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3]
]
What I want is:
Since the size of group of contiguous items containing the value 2 in such example is less than (3 x 3) matrix, I need to assign them the value of neighbour items: in such case 1!
Resulting numpy has to be
[[1,1,1,1,1,1,1],
[1,1,1,1,1,1,1],
[1,1,1,**1**,1,1,1],
[1,1,1,**1**,1,1,1],
[1,1,1,1,1,1,1],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3]
]
What I would like to have is that the 'spurious' elements (only two cells containing the value 2 in an area with a predominance of 1 values) are eliminated and uniformed to the area in which they appear. I hope I have explained. Thanks for any information you can give me. Thanks a lot.

In image processing this operations are called morphological filtering. In your case you can use an opening.
import numpy as np
from skimage.morphology.grey import opening
from skimage.morphology import square
a = np.array(
[[1,1,1,1,1,1,1],
[1,1,1,1,1,1,1],
[1,1,1,2,1,1,1],
[1,1,1,2,1,1,1],
[1,1,1,1,1,1,1],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3],
[3,3,3,3,3,3,3]
])
opening(a, square(3))
Out:
[[1 1 1 1 1 1 1]
[1 1 1 1 1 1 1]
[1 1 1 1 1 1 1]
[1 1 1 1 1 1 1]
[1 1 1 1 1 1 1]
[3 3 3 3 3 3 3]
[3 3 3 3 3 3 3]
[3 3 3 3 3 3 3]]

I'm trying to change values in matrix a with given index matrix d and matrix e.
And the matrix should always be symmetrical.
What I come up with is to overwrite the primal matrix with given index, and try to make it symmetrical, then go for another overwrite, until all the given index matrix have been gone through. It's not efficient.
But I'm stuck with how make it symmetrical.
For example:
a = np.ones([4,4],dtype=np.object) #the primal matrix
d = np.array([[1],
[2],
[0],
[0]]) #the first index matrix
a[np.arange(a.shape[0])[:,None],d] =2 #the element change to 2 with the indexes shown in d matrix
Now the result is:
a = np.array([[1 2 1 1]
[1 1 2 1]
[2 1 1 1]
[2 1 1 1]])
After making it symmetrical (if a[ i ][ j ] was selected in d matrix, a[ j ][ i ] should also be changed to 2, how to do this part).
The expected output should be :
a = np.array([[1 2 2 2]
[2 1 2 1]
[2 2 1 1]
[2 1 1 1]])
Then, for another overwrite again:
e = np.array([[0],[2],[1],[1]])
a[np.arange(a.shape[0])[:,None],e] =3
Now the result is:
a = np.array([[3 2 2 2]
[2 1 3 1]
[2 3 1 1]
[2 3 1 1]])
Make it symmetrical, (I don't know how to do this part) the final output should be : (overwrite the values if they were given 2 or 1 before)
a = np.array([[3 2 2 2]
[2 1 3 3]
[2 3 1 1]
[2 3 1 1]])
What should I do to get symmetrical matrix?
And, is there anyway to change the primal matrix a directly to get the final result? In a more efficient way?
Thanks in advance !!

You can simply switch the first and second indices and apply the change, the result would be symmetrical:
a[np.arange(a.shape[0])[:,None], d] = 2
a[d, np.arange(a.shape[0])[:,None]] = 2
output:
[[1 2 2 2]
[2 1 2 1]
[2 2 1 1]
[2 1 1 1]]
Same with any number of other changes:
a[np.arange(a.shape[0])[:,None], e] = 3
a[e, np.arange(a.shape[0])[:,None]] = 3
output:
[[3 2 2 2]
[2 1 3 3]
[2 3 1 1]
[2 3 1 1]]