I am trying to conduct something similar to searchsorted, but in the case where the array is not completely monotonic. Say I have a scalar, c and a 1D array x, I want to find the indices i of all elements such that x[i] < c <= x[i + 1]. Importantly, x is not completely monotonic.
The following code works, but I just would like to know if this is the most efficient way to do this, or if there is a simper way:
x = np.array([1,2,3,1,2,3,1,2,3])
c = 2.5
t = c > x[:-1]
u = c <= x[1:]
v = t*u
i = v.nonzero()[0]
Or in one line of code:
i = ( (c > x[:-1]) * (c <= x[1:] ).nonzero()[0]
Is this the most efficient way to recover these indices?
Two additional questions.
Is there an easy way to extend this to the case where c is a 1D array and x is a 2D array, where c has as many elements as "rows" in x, and I perform this search for each element of c in the corresponding "row" of x?
My ultimate goal is to do this with a three dimensional case. That is, suppose c is still a 1D vector with n elements. Now, let x be a 3D array, with dimensions j by n by k. Is there a way to do #1 above for each "submatrix" in x? Basically, performing #1 above j times.
For example:
x1 = np.array([1,2,3,1,2,3],[1,2,3,1,2,3],[1,2,3,1,2,3])
x2 = x1 + 1
x = np.array([x1,x2])
c = np.array([1.5,2.5,3.5])
Under #1 above, when we compare c and x1, we would get: [[0,4],[1,5],[]]
When we compare c and x2, we would get: [[],[0,4],[1,5]]
Finally, under #2, I would like to get:
[[[0,4],[1,5],[]],
[[],[0,4],[1,5]]]
We could compare once to give us the boolean mask and re-use it with negation to get the other comparison array and also use slicing -
m = c > x
i = np.flatnonzero( m[:-1] & ~m[1:] )
We can extend it to x as 2D and c as 1D case with a loop, but do minimal computations with it by pre-computing on the masks generation in a vectorized manner, like so -
m = c[:,None] > x
m2 = m[:,:-1] & ~m[:,1:]
i = [np.flatnonzero( mi ) for mi in m2]
On such task, numpy make too much comparisons. You can win a 5X factor with Numba. No difficulties to adapt for 3 dimensions.
#numba.njit
def ind(x,c):
res = empty_like(x)
i=j=0
while i < x.size-1:
if x[i]<c and c<=x[i+1]:
res[j]=i
j+=1
i+=1
return res[:j]
Related
I have a matrix X, and the labels to each vector of that matrix as a np array Y. If the value of Y is +1 I want to multiply the vector of matrix X by another vector W. If the value of Y is -1 I want to multiply the vector of matrix X by another vector Z.
I tried the following loop:
for i in X:
if Y[i] == 1:
np.sum(np.multiply(i, np.log(w)) + np.multiply((1-i), np.log(1-w)))
elif Y[i] == -1:
np.sum(np.multiply(i, np.log(z)) + np.multiply((1-i), np.log(1-z)))
IndexError: arrays used as indices must be of integer (or boolean) type
i is the index of X, but I'm not sure how to align the index of X to the value in that index of Y.
How can I do this?
Look into np.where, it is exactly what you need:
res = np.where(Y == +1, np.dot(X, W), np.dot(X, Z))
This assumes that Y can take only value +1 and -1. If that's not the case you can adapt the script above but we need to know what do you expect as result when Y takes a different value.
Also, try to avoid explicit for loops when using numpy, the resulting code will be thousands of times faster.
While it is better to use a no-iteration approach, I think you need a refresher on basic Python iteration.
Make an array:
In [57]: x = np.array(['one','two','three'])
Your style of iteration:
In [58]: for i in x: print(i)
one
two
three
to iterate on indices use range (or arange):
In [59]: for i in range(x.shape[0]): print(i,x[i])
0 one
1 two
2 three
enumerate is a handy way of getting both indices and values:
In [60]: for i,v in enumerate(x): print(i,v)
0 one
1 two
2 three
Your previous question had the same problem (and correction):
How to iterate over a numpy matrix?
Is there a way to efficiently compare multiple arrays which are broadcast together? For example:
a = np.arange( 0, 9).reshape(3,3)
b = np.arange( 9, 18).reshape(3,3)
c = np.arange(18, 27).reshape(3,3)
If I were to broadcast these as follows:
abc = a[:,:,None,None,None,None] + b[None,None,:,:,None,None] + c[None,None,None,None,:,:]
Then each element of abc is equal to a_ij + b_kl + c_mn where ij, kl, and mn index the respective arrays. What I would like instead is to get min(a_ij, b_kl, c_mn), or ideally, max(a_ij, b_kl, c_mn) - min(a_ij, b_kl, c_mn). Is there an efficient way in which I can do this?
I could, of course, broadcast temporary arrays as:
Abc = a[:,:,None,None,None,None] + 0 * b[None,None,:,:,None,None] + 0 * c[None,None,None,None,:,:]
aBc = 0 * a[:,:,None,None,None,None] + b[None,None,:,:,None,None] + 0 * c[None,None,None,None,:,:]
abC = 0 * a[:,:,None,None,None,None] + 0 * b[None,None,:,:,None,None] + c[None,None,None,None,:,:]
and then find the min/max from these arrays, however, these arrays can get quite large. It would be better if there were some way to do it in one step.
And as an additional note, these arrays are guaranteed to be broadcastable, but not necessarily have the same shape (for example, (1, 3) and (3, 3)).
You can store an intermediate array (smaller than your final result anyway) by doing the operation on a and b:
temp = np.minimum.outer(a.ravel(), b.ravel())
res = np.minimum.outer(temp.ravel(), c.ravel())
and then repeat that same operation with c. minimum computes element-wise min of 2 arrays. Since it is a ufunc, you can use outer to apply that operation to all pairs of values for those 2 arrays.
You can reshape res as you prefer.
Edit # 1
Thanks to P. Panzer comment, you do not need to use 1D arrays with ufunc.outer, which results in even simpler code:
temp = np.minimum.outer(a, b)
res = np.minimum.outer(temp, c)
So suppose i have two numpy ndarrays whose elements are matrices. I need element-wise multiplication for these two arrays, however, there should be matrix multiplication between the two matrix elements. Of course i would be able to implement this with for loops but i was looking to solve this problem without using an explicit for loop. How do i implement this?
EDIT: This for-loop does what I want to do. I'm on python 2.7
n = np.arange(8).reshape(2,2,1,2)
l = np.arange(1,9).reshape(2,2,2,1)
k = np.zeros((2,2))
for i in range(len(n)):
for j in range(len(n[i])):
k[i][j] = np.asscalar(n[i][j].dot(l[i][j]))
print k
Assuming your arrays of matrices are given as n+2 dimensional arrays A and B. What you want to achieve is as simple as C = A#B
Example
outer_dims = 2,3,4
inner_dims = 4,5,6
A = np.random.randint(0,10,(*outer_dims, *inner_dims[:2]))
B = np.random.randint(0,10,(*outer_dims, *inner_dims[1:]))
C = A#B
# check
for I in np.ndindex(outer_dims):
assert (C[I] == A[I]#B[I]).all()
UPDATE: Py2 version; thanks # hpaulj, Divakar
A = np.random.randint(0,10, outer_dims + inner_dims[:2])
B = np.random.randint(0,10, outer_dims + inner_dims[1:])
C = np.matmul(A,B)
# check
for I in np.ndindex(outer_dims):
assert (C[I] == np.matmul(A[I],B[I])).all()
If I understand correctly, this might work:
import numpy as np
a = np.array([[1,1],[1,0]])
b = np.array([[3,4],[5,4]])
x = np.array([[a,b],[b,a]])
y = np.array([[a,a],[b,b]])
result = np.array([_x # _y for _x, _y in zip(x,y)])
What is the best way to exclude exact one NumPy array entry from an operation?
I have an array x containing n values and want to exclude the i-th entry when I call numpy.prod(x). I know about MaskedArray, but is there another/better way?
I think the simplest would be
np.prod(x[:i]) * np.prod(x[i+1:])
This should be fast and also works when you don't want to or can't modify x.
And in case x is multidimensional and i is a tuple:
x_f = x.ravel()
i_f = np.ravel_multi_index(i, x.shape)
np.prod(x_f[:i_f]) * np.prod(x_f[i_f+1:])
You could use np.delete whch removes an element from a one-dimensional array:
import numpy as np
x = np.arange(1, 5)
i = 2
y = np.prod(np.delete(x, i)) # gives 8
I don't think there is any better way, honestly. Even without knowing the NumPy functions, I would do it like:
#I assume x is array of len n
temp = x[i] #where i is the index of the value you don't want to change
x = x * 5
#...do whatever with the array...
x[i] = temp
If I understand correctly, your problem is one dimensional? Even if not, you can do this the same way.
EDIT:
I checked the prod function and in this case I think you can just replace the value u don't want to use with 1 (using temp approach I've given you above) and later just put in the right value. It is just a in-place change, so it's kinda efficient. The second way you can do this is just to divide the result by the x[i] value (assuming it's not 0, as commenters said).
As np.prod is taking the product of all the elements in an array, if we want to exclude one element from the solution, we can set that element to 1 first in order to ignore it (as p * 1 = p).
So:
>>> n = 10
>>> x = np.arange(10)
>>> i = 0
>>> x[i] = 1
>>> np.prod(x)
362880
which, we can see, works:
>>> 1 * 2 * 3 * 4 * 5 * 6 * 7 * 8 * 9
362880
You could use a list comprehension to index all the points but 1:
i = 2
np.prod(x[[val for val in range(len(x)) if val != i]])
or use a set difference:
np.prod(x[list(set(range(len(x)) - {i})])
Say I have 4 numpy arrays A,B,C,D , each the size of (256,256,1792).
I want to go through each element of those arrays and do something to it, but I need to do it in chunks of 256x256x256-cubes.
My code looks like this:
for l in range(7):
x, y, z, t = 0,0,0,0
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
t += D[m,n,o] * constant
x += A[m,n,o] * D[m,n,o] * constant
y += B[m,n,o] * D[m,n,o] * constant
z += C[m,n,o] * D[m,n,o] * constant
final = (x+y+z)/t
doOutput(final)
The code works and outputs exactly what I want, but its awfully slow. I've read online that those kind of nested for loops should be avoided in python. What is the cleanest solution to it? (right now I'm trying to do this part of my code in C and somehow import it via Cython or other tools, but I'd love a pure python solution)
Thanks
Add on
Willem Van Onsem's Solution to the first part seems to work just fine and I think I comprehend it. But now I want to modify my values before summing them. It looks like
(within the outer l loop)
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
R += (D[m,n,o] * constant * (A[m,n,o]**2
+ B[m,n,o]**2 + C[m,n,o]**2)/t - final**2)
doOutput(R)
I obviously can't just square the sum x = (A[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()**2*constant since (A²+B²) != (A+B)²
How can I redo this last for loops?
Since you update t with every element of m in range(a.shape[0]), n in range(a.shape[1]) and o in range(256*l,256*(l+1)), you can substitute:
for m in range(a.shape[0]):
for n in range(a.shape[1]):
for o in range(256*l,256*(l+1)):
t += D[m,n,o]
With:
t += D[:a.shape[0],:a.shape[1],256*l:256*(l+1)].sum()
The same for the other assignments. So you can rewrite your code to:
for l in range(7):
Dsub = D[:a.shape[0],:a.shape[1],256*l:256*(l+1)]
x = (A[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
y = (B[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
z = (C[:a.shape[0],:a.shape[1],256*l:256*(l+1)]*Dsub).sum()*constant
t = Dsub.sum()*constant
final = (x+y+z)/t
doOutput(final)
Note that the * in numpy is the element-wise multiplication, not the matrix product. You can do the multiplication before the sum, but since the sum of a multiplications with a constant is equal to the multiplication of that constant with the sum, I think it is more efficient to do this out of the loop.
If a.shape[0] is equal to D.shape[0], etc. You can use : instead of :a.shape[0]. Based on your question, that seems to be the case. so:
# only when `a.shape[0] == D.shape[0], a.shape[1] == D.shape[1] (and so for A, B and C)`
for l in range(7):
Dsub = D[:,:,256*l:256*(l+1)]
x = (A[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
y = (B[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
z = (C[:,:,256*l:256*(l+1)]*Dsub).sum()*constant
t = Dsub.sum()*constant
final = (x+y+z)/t
doOutput(final)
Processing the .sum() on the numpy level will boost performance since you do not convert values back and forth and with .sum(), you use a tight loop.
EDIT:
Your updated question does not change much. You can simply use:
m,n,_* = a.shape
lo,hi = 256*l,256*(l+1)
R = (D[:m,:n,lo:hi]*constant*(A[:m,:n,lo:hi]**2+B[:m,:n,lo:hi]**2+D[:m,:n,lo:hi]**2)/t-final**2)).sum()
doOutput(R)