I'm sure this question is Googleable, but I don't know what keywords to use. I'm curious about a specific case, but also about how to do it in general. Lets say I have a RGB image as an array of shape (width, height, 3) and I want to find all the pixels where the red channel is greater than 100. I feel like image > [100, 0, 0] should give me an array of indices (and would if I was comparing a scalar and using a greyscale image) but this compares each element with the list. How do I compare over the first two dimensions where each "element" is the last dimension?
To detect for red-channel only, you can do something like this -
np.argwhere(image[:,:,0] > threshold)
Explanation :
Compare the red-channel with the threshold to give us a boolean array of same shape as the input image without the third axis (color channel).
Use np.argwhere to get the indices of successful matches.
For a case when you want to see if any channel is above some threshold, use .any(-1) (any elements that satisfy the condition along the last axis/color channel).
np.argwhere((image > threshold).any(-1))
Sample run
Input image :
In [76]: image
Out[76]:
array([[[118, 94, 109],
[ 36, 122, 6],
[ 85, 91, 58],
[ 30, 2, 23]],
[[ 32, 47, 50],
[ 1, 105, 141],
[ 91, 120, 58],
[129, 127, 111]]], dtype=uint8)
In [77]: threshold
Out[77]: 100
Case #1: Red-channel only
In [69]: np.argwhere(image[:,:,0] > threshold)
Out[69]:
array([[0, 0],
[1, 3]])
In [70]: image[0,0]
Out[70]: array([118, 94, 109], dtype=uint8)
In [71]: image[1,3]
Out[71]: array([129, 127, 111], dtype=uint8)
Case #2: Any-channel
In [72]: np.argwhere((image > threshold).any(-1))
Out[72]:
array([[0, 0],
[0, 1],
[1, 1],
[1, 2],
[1, 3]])
In [73]: image[0,1]
Out[73]: array([ 36, 122, 6], dtype=uint8)
In [74]: image[1,1]
Out[74]: array([ 1, 105, 141], dtype=uint8)
In [75]: image[1,2]
Out[75]: array([ 91, 120, 58], dtype=uint8)
Faster alternative to np.any in np.einsum
np.einsum could be tricked to perform np.any's work and as it turns out is a tad faster.
Thus, boolean_arr.any(-1) would be equivalent to np.einsum('ijk->ij',boolean_arr).
Here are the associated runtimes across various datasizes -
In [105]: image = np.random.randint(0,255,(30,30,3)).astype('uint8')
...: %timeit np.argwhere((image > threshold).any(-1))
...: %timeit np.argwhere(np.einsum('ijk->ij',image>threshold))
...: out1 = np.argwhere((image > threshold).any(-1))
...: out2 = np.argwhere(np.einsum('ijk->ij',image>threshold))
...: print np.allclose(out1,out2)
...:
10000 loops, best of 3: 79.2 µs per loop
10000 loops, best of 3: 56.5 µs per loop
True
In [106]: image = np.random.randint(0,255,(300,300,3)).astype('uint8')
...: %timeit np.argwhere((image > threshold).any(-1))
...: %timeit np.argwhere(np.einsum('ijk->ij',image>threshold))
...: out1 = np.argwhere((image > threshold).any(-1))
...: out2 = np.argwhere(np.einsum('ijk->ij',image>threshold))
...: print np.allclose(out1,out2)
...:
100 loops, best of 3: 5.47 ms per loop
100 loops, best of 3: 3.69 ms per loop
True
In [107]: image = np.random.randint(0,255,(3000,3000,3)).astype('uint8')
...: %timeit np.argwhere((image > threshold).any(-1))
...: %timeit np.argwhere(np.einsum('ijk->ij',image>threshold))
...: out1 = np.argwhere((image > threshold).any(-1))
...: out2 = np.argwhere(np.einsum('ijk->ij',image>threshold))
...: print np.allclose(out1,out2)
...:
1 loops, best of 3: 833 ms per loop
1 loops, best of 3: 640 ms per loop
True
Related
Is there a more numpythonic way to do this?
#example arrays
arr = np.array([0, 1, 2, 3, 4, 5, 6, 7], dtype=np.float32)
values = np.array([0.2, 3.0, 1.5])
#get the indices where each value falls between values in arr
between = [np.nonzero(i > arr)[0][-1] for i in values]
For sorted arr, we can use np.searchsorted for performance -
In [67]: np.searchsorted(arr,values)-1
Out[67]: array([0, 2, 1])
Timings on large dataset -
In [81]: np.random.seed(0)
...: arr = np.unique(np.random.randint(0,10000, 10000))
...: values = np.random.randint(0,10000, 1000)
# #Andy L.'s soln
In [84]: %timeit np.argmin(values > arr[:,None], axis=0) - 1
10 loops, best of 3: 28.2 ms per loop
# Original soln
In [82]: %timeit [np.nonzero(i > arr)[0][-1] for i in values]
100 loops, best of 3: 8.68 ms per loop
# From this post
In [83]: %timeit np.searchsorted(arr,values)-1
10000 loops, best of 3: 57.8 µs per loop
Use broadcast and argmin
np.argmin(values > arr[:,None], axis=0) - 1
Out[32]: array([0, 2, 1], dtype=int32)
Note: I assume arr is monotonic increasing as in the sample
I need to compute the diagonals of XMX^T without a for-loop, or in other words, replacing the following for loop:
X = nump.random.randn(10000, 100)
M = numpy.random.rand(100, 100)
out = numpy.zeros(10000)
for n in range(10000):
out[n] = np.dot(np.dot(X[n, :], M), X[n, :])
I know somehow I should be using numpy.einsum, but I have not been able to figure out how?
Many thanks!
Sure there is an np.einsum way, like so -
np.einsum('ij,ij->i',X.dot(M),X)
This abuses the fast matrix-multiplication at the first level with X.dot(M) and then uses np.einsum to keep the first axis and sum reduces the second axis.
Runtime test -
This section compares all the approaches posted thus far to solve the problem.
In [132]: # Setup input arrays
...: X = np.random.randn(10000, 100)
...: M = np.random.rand(100, 100)
...:
...: def original_app(X,M):
...: out = np.zeros(10000)
...: for n in range(10000):
...: out[n] = np.dot(np.dot(X[n, :], M), X[n, :])
...: return out
...:
In [133]: np.allclose(original_app(X,M),np.einsum('ij,ij->i',X.dot(M),X))
Out[133]: True
In [134]: %timeit original_app(X,M) # Original solution
10 loops, best of 3: 97.8 ms per loop
In [135]: %timeit np.dot(X, np.dot(M,X.T)).trace()# #Colonel Beauvel's solution
1 loops, best of 3: 2.24 s per loop
In [136]: %timeit np.einsum('ij,jk,ik->i', X, M, X) # #hpaulj's solution
1 loops, best of 3: 442 ms per loop
In [137]: %timeit np.einsum('ij,ij->i',X.dot(M),X) # Proposed in this post
10 loops, best of 3: 28.1 ms per loop
Here is a simpler example:
M = array([[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]])
X = array([[ 0, 4, 8],
[ 1, 5, 9],
[ 2, 6, 10],
[ 3, 7, 11]])
What you are looking for - the sum of diagonal elements - is more commonly known as the trace in maths. You can obtain the trace of your matrix product, without loop, by:
In [102]: np.dot(X, np.dot(M,X.T)).trace()
Out[102]: 692
In [210]: X=np.arange(12).reshape(4,3)
In [211]: M=np.ones((3,3))
In [212]: out=np.zeros(4)
In [213]: for n in range(4):
out[n]= np.dot(np.dot(X[n,:],M), X[n,:])
.....:
In [214]: out
Out[214]: array([ 9., 144., 441., 900.])
One einsum approach:
In [215]: np.einsum('ij,jk,ik->i', X, M, X)
Out[215]: array([ 9., 144., 441., 900.])
Comparing the other einsum:
In [218]: timeit np.einsum('ij,jk,ik->i', X, M, X)
100000 loops, best of 3: 8.98 µs per loop
In [219]: timeit np.einsum('ij,ij->i',X.dot(M),X)
100000 loops, best of 3: 11.9 µs per loop
This is a bit faster, but results may diff with your larger size.
einsum does save calculating a lot of unnecessary values (cf. to the diagonal or trace approaches).
Similar use of einsum - Combine Einsum Expresions
Lets say I have an tensor of the following form:
import numpy as np
a = np.array([ [[1,2],
[3,4]],
[[5,6],
[7,3]]
])
# a.shape : (2,2,2) is a tensor containing 2x2 matrices
indices = np.argmax(a, axis=2)
#print indices
for mat in a:
max_i = np.argmax(mat,axis=1)
# Not really working I would like to
# change 4 in the first matrix to -1
# and 3 in the last to -1
mat[max_i] = -1
print a
Now what I would like to do is to use indices as a mask on a to replace every max element with say -1. Is there a numpy way of doing this ? so far all I have figured out is using for loops.
Here's one way using linear indexing in 3D -
m,n,r = a.shape
offset = n*r*np.arange(m)[:,None] + r*np.arange(n)
np.put(a,indices + offset,-1)
Sample run -
In [92]: a
Out[92]:
array([[[28, 59, 26, 70],
[57, 28, 71, 49],
[33, 6, 10, 90]],
[[24, 16, 83, 67],
[96, 16, 72, 56],
[74, 4, 71, 81]]])
In [93]: indices = np.argmax(a, axis=2)
In [94]: m,n,r = a.shape
...: offset = n*r*np.arange(m)[:,None] + r*np.arange(n)
...: np.put(a,indices + offset,-1)
...:
In [95]: a
Out[95]:
array([[[28, 59, 26, -1],
[57, 28, -1, 49],
[33, 6, 10, -1]],
[[24, 16, -1, 67],
[-1, 16, 72, 56],
[74, 4, 71, -1]]])
Here's another way with linear indexing again, but in 2D -
m,n,r = a.shape
a.reshape(-1,r)[np.arange(m*n),indices.ravel()] = -1
Runtime tests and verify output -
In [156]: def vectorized_app1(a,indices): # 3D linear indexing
...: m,n,r = a.shape
...: offset = n*r*np.arange(m)[:,None] + r*np.arange(n)
...: np.put(a,indices + offset,-1)
...:
...: def vectorized_app2(a,indices): # 2D linear indexing
...: m,n,r = a.shape
...: a.reshape(-1,r)[np.arange(m*n),indices.ravel()] = -1
...:
In [157]: # Generate random 3D array and the corresponding indices array
...: a = np.random.randint(0,99,(100,100,100))
...: indices = np.argmax(a, axis=2)
...:
...: # Make copies for feeding into functions
...: ac1 = a.copy()
...: ac2 = a.copy()
...:
In [158]: vectorized_app1(ac1,indices)
In [159]: vectorized_app2(ac2,indices)
In [160]: np.allclose(ac1,ac2)
Out[160]: True
In [161]: # Make copies for feeding into functions
...: ac1 = a.copy()
...: ac2 = a.copy()
...:
In [162]: %timeit vectorized_app1(ac1,indices)
1000 loops, best of 3: 311 µs per loop
In [163]: %timeit vectorized_app2(ac2,indices)
10000 loops, best of 3: 145 µs per loop
You can use indices to index into the last dimension of a provided that you also specify index arrays into the first two dimensions as well:
import numpy as np
a = np.array([[[1, 2],
[3, 4]],
[[5, 6],
[7, 3]]])
indices = np.argmax(a, axis=2)
print(repr(a[range(a.shape[0]), range(a.shape[1]), indices]))
# array([[2, 3],
# [2, 7]])
a[range(a.shape[0]), range(a.shape[1]), indices] = -1
print(repr(a))
# array([[[ 1, -1],
# [ 3, 4]],
# [[ 5, 6],
# [-1, -1]]])
Say I have two 3 dimensional matrices, like so (taken from this matlab example http://www.mathworks.com/help/matlab/ref/dot.html):
A = cat(3,[1 1;1 1],[2 3;4 5],[6 7;8 9])
B = cat(3,[2 2;2 2],[10 11;12 13],[14 15; 16 17])
If I want to take pairwise dot products along the third dimension, I could do so like this in matlab:
C = dot(A,B,3)
Which would give the result:
C =
106 140
178 220
What would be the equivalent operation in numpy, preferably a vectorized option, to avoid having to write a double for loop through the entire array. I can't seem to make sense of what np.tensordot or np.inner are supposed to do, but they might be options.
In [169]:
A = np.dstack([[[1, 1],[1 ,1]],[[2 ,3],[4, 5]],[[6, 7],[8, 9]]])
B = np.dstack([[[2, 2],[2, 2]],[[10, 11],[12, 13]],[[14, 15], [16, 17]]])
c=np.tensordot(A, B.T,1)
np.vstack([np.diag(c[:,i,i]) for i in range(A.shape[0])]).T
Out[169]:
array([[106, 140],
[178, 220]])
But surprisingly it is the slowest:
In [170]:
%%timeit
c=np.tensordot(A, B.T,1)
np.vstack([np.diag(c[:,i,i]) for i in range(A.shape[0])]).T
10000 loops, best of 3: 95.2 µs per loop
In [171]:
%timeit np.einsum('i...,i...',a,b)
100000 loops, best of 3: 6.93 µs per loop
In [172]:
%timeit inner1d(A,B)
100000 loops, best of 3: 4.51 µs per loop
Using np.einsum:
In [9]: B = np.array([[[2, 2],[2, 2]],[[10, 11],[12, 13]],[[14, 15],[16, 17]]])
In [10]: A = np.array([[[1, 1],[1, 1]],[[2, 3],[4, 5]],[[6, 7],[8, 9]]])
In [11]: np.einsum('i...,i...',A,B)
Out[11]:
array([[106, 140],
[178, 220]])
Or here's another fun one:
In [37]: from numpy.core.umath_tests import inner1d
In [38]: inner1d(A,B)
Out[38]:
array([[106, 140],
[178, 220]])
Edit in response to #flebool's comment, inner1d works for both (2,2,3) and (3,2,2) shaped arrays:
In [41]: A = dstack([[[1, 1],[1 ,1]],[[2 ,3],[4, 5]],[[6, 7],[8, 9]]])
In [42]: B = dstack([[[2, 2],[2, 2]],[[10, 11],[12, 13]],[[14, 15], [16, 17]]])
In [43]: inner1d(A,B)
Out[43]:
array([[106, 140],
[178, 220]])
Here's a solution:
A = dstack([[[1, 1],[1 ,1]],[[2 ,3],[4, 5]],[[6, 7],[8, 9]]])
B = dstack([[[2, 2],[2, 2]],[[10, 11],[12, 13]],[[14, 15], [16, 17]]])
C = einsum('...k,...k',A,B)
Basically dstack concatenates along the third axis, (docs), and then you use the powerful einstein summation tool einsum provided by numpy (docs)
Suppose I have a numpy array
a = np.array([0, 8, 25, 78, 68, 98, 1])
and a mask array b = [0, 1, 1, 0, 1]
Is there an easy way to get the following array:
[8, 25, 68] - which is first, second and forth element from the original array. Which sounds like a mask for me.
The most obvious way I have tried is a[b], but this does not yield a desirable result.
After this I tried to look into masked operations in numpy but it looks like it guides me in the wrong direction.
If a and b are both numpy arrays and b is strictly 1's and 0's:
>>> a[b.astype(np.bool)]
array([ 8, 25, 68])
It should be noted that this is only noticeably faster for extremely small cases, and is much more limited in scope then #falsetru's answer:
a = np.random.randint(0,2,5)
%timeit a[a==1]
100000 loops, best of 3: 4.39 µs per loop
%timeit a[a.astype(np.bool)]
100000 loops, best of 3: 2.44 µs per loop
For the larger case:
a = np.random.randint(0,2,5E6)
%timeit a[a==1]
10 loops, best of 3: 59.6 ms per loop
%timeit a[a.astype(np.bool)]
10 loops, best of 3: 56 ms per loop
>>> a = np.array([0, 8, 25, 78, 68, 98, 1])
>>> b = np.array([0, 1, 1, 0, 1])
>>> a[b == 1]
array([ 8, 25, 68])
Alternative using itertools.compress:
>>> import itertools
>>> list(itertools.compress(a, b))
[8, 25, 68]