Suppose you have an array:
a =
[ 0,1,0]
[-1,2,1]
[3,-4,2]
And lets say you add 20 to everything
b =
[ 20, 21, 20]
[ 19, 22, 21]
[ 23, 16, 22]
Now lets say I want to add the resulting b to the original array a but only in cases where a < 0 i.e at the index [0,1] and [1,2] where a = -1, -4 respectively getting the value 0 otherwise. Ultimately leading to a matrix as such:
c =
[ 0, 0, 0]
[ 18, 0, 0]
[ 0, 12, 0]
18 = 19 (from b) + -1 (from a)
12 = 16 (from b) + -4 (from a)
And assume that I want to be able to extend this to any operation (not just add 20), so that you can't just filter all values < 20 from matrix c. So I want to use matrix a as a mask toward matrix c, zeroing the i, j where a[i,j] < 0.
I'm having a tough time finding a concise example of how to do this in numpy with python. I was hoping you may be able to direct me to the correct implementation of such a method.
What I am struggling to get is this into a mask and only performing operations on the retained values, finally resulting in c.
Thanks for the help in advance.
Probably something like:
(a + b)*(a<0)
should work unless you have very strong requirements concerning the number of intermediate arrays.
You can do this through a combination of boolean indexing and broadcasting. Working example below,
import numpy as np
a = np.array([[ 0,1,0],[-1,2,1],[3,-4,2]])
b = a+20
c = np.zeros(a.shape)
c[a<0] = b[a<0] + a[a<0]
which gives c as
array([[ 0., 0., 0.],
[ 18., 0., 0.],
[ 0., 12., 0.]])
The only important line in the code snippet above is the last one. Because the entries of a, b, and c are all aligned, we can say we want only the corresponding indices of c where a<0 to be assigned to the sum of the entries in b and a where a<0.
Here is another way to get the same result:
c = np.where(a < 0, a + b, 0)
Although this is slightly more verbose than Thomas Baruchel's solution, I found the method signature similar to the ternary operation (a < 0 ? a + b : 0), which makes it easier for me to understand what it is doing right away. Also, this is still a one-liner which makes it elegant enough in my opinion.
reference: numpy.where
Perhaps not the cleanest solution, but how about this?:
def my_mask(a, b, threshold=0):
c = numpy.zeros(a.shape)
idx = np.where(a < threshold)
for ii in idx:
c[ii[1], ii[0]] = a[ii[1], ii[0]] + b[ii[1], ii[0]]
return c
The solution using numpy.zeros_like function:
import numpy as np
# the initial array
a = [[ 0,1,0],
[-1,2,1],
[3,-4,2]]
a = np.array(a)
b = a + 20 # after adding 20 to each element
c = np.zeros_like(a) # resulting matrix (filled with zeros by default)
neg = a < 0 # indeces of negative values
c[neg] = b[neg] + a[neg] # overriding the needed elements
print(c)
The output:
[[ 0 0 0]
[18 0 0]
[ 0 12 0]]
Related
Context
I have the following example-arrays in numpy:
import numpy as np
# All arrays in this example have the shape (15,)
# Note: All values > 0 are unqiue!
a = np.array([8,5,4,-1,-1, 7,-1,-1,12,11,-1,-1,14,-1,-1])
reference = np.array([0,1,2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14])
lookup = np.array([3,6,0,-2,-2,24,-2,-2,24,48,-2,-2,84,-2,-2])
My goal is to find the elements inside the reference in a, then get the index in a and use it to extract the corresponding elements in lookup.
Finding out the matching elements and their indices works with np.flatnonzero( np.isin() ).
I can also lookup the correspodning values:
# Example how to find the index
np.flatnonzero( np.isin( reference, a) )
# -> array([ 4, 5, 7, 8, 11, 12, 14])
# Example how to find corresponding values:
lookup[ np.flatnonzero( np.isin( a, reference) ) ]
# -> array([ 3, 6, 0, 24, 24, 48, 84], dtype=int64)
Problem
I want to fill an array z with the values I looked up, following the reference.
This means, that e.g. the 8th element of z corresponds to the 8th element in the lookup-value for the 8th element in reference (= 8). This value would be 3 (reference[8] -> a[0] because a==8 here -> lookup[0] -> 3).
z = np.zeros(reference.size)
z[np.flatnonzero(np.isin(reference, a))] = ? -> numpy-array of correctly ordered lookup_values
The expected outcome for z would be:
z = [ 0 0 0 0 0 6 0 24 3 0 0 48 24 0 84]
I cannot get my head around this; I have to avoid for-loops due to performance reasons and would need a pure numpy-solution (best without udfs).
How can I fill z according with the lookup-values at the correct position?
Note: As stated in the code above, all values a > 0 are unique. Thus, there is no need to take care about the duplicated values for a < 0.
You say that you 'have to avoid for-loops due to performance reasons', so I assume that your real-world datastructure a is going to be large (thousands or millions of elements?). Since np.isin(reference, a) performs a linear search in a for every element of reference, your runtime will be O(len(reference) * len(a)).
I would strongly suggest using a dict for a, allowing lookup in O(1) per element of reference, and loop in python using for. For sufficiently large a this will outperform the 'fast' linear search performed by np.isin.
The most natural way I can think of is to just treat a and lookup as a dictionary:
In [82]: d = dict(zip(a, lookup))
In [83]: np.array([d.get(i, 0) for i in reference])
Out[83]: array([ 0, 0, 0, 0, 0, 6, 0, 24, 3, 0, 0, 48, 24, 0, 84])
This does have a bit of memory overhead but nothing crazy if reference isn't too large.
I actually had an enlightenment.
# Initialize the result
# All non-indexed entries shall be 0
z = np.zeros(reference.size, dtype=np.int64)
Now evaluate which elements in a are relevant:
mask = np.flatnonzero(np.isin(a, reference))
# Short note: If we know that any positive element of a is a number
# Which has to be in the reference, we can also shorten this to
# a simple boolean mask. This will be significantly faster to process.
mask = (a > 0)
Now the following trick: All values a > 0 are unique. Additionally, their value corresponds to the position in reference (e.g. 8 in a shall correspond to the 8th position in reference. Thus, we can use the values as index themselves:
z[ a[mask] ] = lookup[mask]
This results in the desired outcome:
z = [ 0 0 0 0 0 6 0 24 3 0 0 48 24 0 84]
Suppose:
A=np.array([1,2,0,-4])
B=np.array([1,1,1,1])
C=np.array([1,2,3,4])
With fancy indexing I can assign a scalar value to C wherever A > 0.
C[A > 0]= 1
But is there anyway to get something like C = B/A wherever A > 0 while preserving the original values of C for A <= 0 with fancy indexing ? If I try something like
C[A > 0] = B/A
I get an error like:
<input>:1: RuntimeWarning: divide by zero encountered in true_divide
Traceback (most recent call last):
File "<input>", line 1, in <module>
ValueError: NumPy boolean array indexing assignment cannot assign 4 input values to the 2 output values where the mask is true
I can get the result with a for loop or making copies of A & C where :
D = np.copy(A)
E = np.copy(C)
D[ D <= 0]= 1
E=B/A
E[A <=0] = C
or set C=Run(A,B) where
def Run(A,B):
C=np.zeros(A.shape[0],A.shape[1])
for i in range(len(A)):
if A[i] != O:
C[i] = A[i]/B[i]
else:
C[i] = C[i]
But i was just wondering if there was a more direct way to do it without adding so many steps if i am looping millions of times. Thanks.
You can index the operands: C[A > 0] = B[A > 0] / A[A > 0]. You might want to compute A > 0 once, and reuse it, e.g.
mask = A > 0
C[mask] = B[mask] / A[mask]
A more efficient alternative is to use the where parameter of np.divide or np.floor_divide. For example,
In [19]: A = np.array([1, 2, 0, -4])
In [20]: B = np.array([1, 1, 1, 1])
In [21]: C = np.array([1, 2, 3, 4])
In [22]: np.floor_divide(B, A, where=A > 0, out=C)
Out[22]: array([1, 0, 3, 4])
In [23]: C
Out[23]: array([1, 0, 3, 4])
I had to use floor_divide because all the arrays are integer arrays, and numpy.divide creates a floating point array, so that function will complain about the type mismatch if the out array is an integer array. If you want a floating point result, C should be an array of floating point values:
In [24]: C = np.array([1., 2., 3., 4.])
In [25]: np.divide(B, A, where=A > 0, out=C)
Out[25]: array([1. , 0.5, 3. , 4. ])
In [26]: C
Out[26]: array([1. , 0.5, 3. , 4. ])
Using Python (3.7.7) and numpy (1.17.4), I am working with medium sized 2d numpy arrays (from 5000x80 up to 200,000x120). For a given array, I want to calculate the Hadamard product between all possbible uniqe pairs of column-vectors of that array.
I have:
A A
[a,b,c,d] [a,b,c,d]
[1,2,3,4] [1,2,3,4]
[4,5,6,7] * [4,5,6,7]
[7,8,9,1] [7,8,9,1]
and I want to get:
[a*b, ac, ad, bc, bd, cd]
[ 2., 3., 4., 6., 8., 12.]
[20., 24., 28., 30., 35., 42.]
[56., 63., 7., 72., 8., 9.]
I already have a solution from a colleague using np.kron which I adapated a bit:
def hadamard_kron(A: np.ndarray) -> :
"""Returns the hadamard products of all unique pairs of all columns,
and return indices signifying which columns constitute a given pair.
"""
n = raw_inputs.shape[0]
ind1 = (np.kron(np.arange(0, n).reshape((n, 1)), np.ones((n, 1)))).squeeze().astype(int)
ind2 = (np.kron(np.ones((n, 1)), np.arange(0, n).reshape((n, 1)))).squeeze().astype(int)
xmat2 = np.kron(raw_inputs, np.ones((n, 1))) * np.kron(np.ones((n, 1)), raw_inputs)
hadamard_inputs = xmat2[ind2 > ind1, :]
ind1_ = ind1[ind1 < ind2]
ind2_ = ind2[ind1 < ind2]
return hadamard_A, ind1_, ind2_
hadamard_A, first_pair_members, second_pair_members = hadamard_kron(a.transpose())
Note that hadamard_A is what I want, but transposed (which is also what I want for further processing). Also, ind1_ (ind2_) gives the indices for the objects which feature as the first (second) element in the pair for which the hadamard product is calculated. I need those as well.
However, I feel this code is too inefficient: it takes to long and since I call this function several times during my algorithm, I was wondering whether there is a cleverer solution? Am I overlooking some numpy/scipy tools I could cleverly combine for this task?
Thanks all! :)
Approach #1
Simplest one with np.triu_indices -
In [45]: a
Out[45]:
array([[1, 2, 3, 4],
[4, 5, 6, 7],
[7, 8, 9, 1]])
In [46]: r,c = np.triu_indices(a.shape[1],1)
In [47]: a[:,c]*a[:,r]
Out[47]:
array([[ 2, 3, 4, 6, 8, 12],
[20, 24, 28, 30, 35, 42],
[56, 63, 7, 72, 8, 9]])
Approach #2
Memory-efficient one for large arrays -
m,n = a.shape
s = np.r_[0,np.arange(n-1,-1,-1).cumsum()]
out = np.empty((m, n*(n-1)//2), dtype=a.dtype)
for i,(s0,s1) in enumerate(zip(s[:-1], s[1:])):
out[:,s0:s1] = a[:,i,None] * a[:,i+1:]
Approach #3
Masking based one -
m,n = a.shape
mask = ~np.tri(n,dtype=bool)
m3D = np.broadcast_to(mask, (m,n,n))
b1 = np.broadcast_to(a[...,None], (m,n,n))
b2 = np.broadcast_to(a[:,None,:], (m,n,n))
out = (b1[m3D]* b2[m3D]).reshape(m,-1)
Approach #4
Extend approach #2 for a numba one -
from numba import njit
def numba_app(a):
m,n = a.shape
out = np.empty((m, n*(n-1)//2), dtype=a.dtype)
return numba_func(a,out,m,n)
#njit
def numba_func(a,out,m,n):
for p in range(m):
I = 0
for i in range(n):
for j in range(i+1,n):
out[p,I] = a[p,i] * a[p,j]
I += 1
return out
Then, leverage parallel processing (as pointed out in comments by #max9111), like so -
from numba import prange
def numba_app_parallel(a):
m,n = a.shape
out = np.empty((m, n*(n-1)//2), dtype=a.dtype)
return numba_func_parallel(a,out,m,n)
#njit(parallel=True)
def numba_func_parallel(a,out,m,n):
for p in prange(m):
I = 0
for i in range(n):
for j in range(i+1,n):
out[p,I] = a[p,i] * a[p,j]
I += 1
return out
Benchmarking
Using benchit package (few benchmarking tools packaged together; disclaimer: I am its author) to benchmark proposed solutions.
import benchit
in_ = [np.random.rand(5000, 80), np.random.rand(10000, 100), np.random.rand(20000, 120)]
funcs = [ehsan, app1, app2, app3, numba_app, numba_app_parallel]
t = benchit.timings(funcs, in_, indexby='shape')
t.rank()
t.plot(logx=False, save='timings.png')
Conclusion : Numba ones seem to be doing pretty well and app2 among NumPy ones.
Another equivalent approach to Divakar's first approach:
r,c = np.triu_indices(A.shape[1],1)
np.einsum('ij,ik->ijk',A,A)[:,r,c]
output:
[[ 2 3 4 6 8 12]
[20 24 28 30 35 42]
[56 63 7 72 8 9]]
I am able to do it in two lines for the numpy module:
x=np.zeros(10)
x[4]=1
However, I was wondering if its possible to combine the two together
There are multiple ways to do this. For example, np.arange(10) == 4 gives you an array of all False values except for one True at position 4.
Under the covers, NumPy's bool values are just 0 and 1 as uint8 (just like Python's bool values are 0 and 1, although of a unique integral type), so you can just use it as-is in any expression:
>>> np.arange(10) == 4
array([False, False, False, False, True, False, False, False, False, False], dtype=bool)
>>> np.arange(10) * 1
array([0, 0, 0, 0, 1, 0, 0, 0, 0, 0])
>>> np.arange(10) + 23
array([23, 23, 23, 23, 24, 23, 23, 23, 23, 23])
… or view it as uint8 instead of bool:
>>> (np.arange(10) == 4).view(np.uint8)
array([0, 0, 0, 0, 1, 0, 0, 0, 0, 0], dtype=uint8)
… or, if you want normal int values, you can convert it:
>>> (np.arange(10) == 4).astype(int)
array([0, 0, 0, 0, 1, 0, 0, 0, 0, 0])
And so on.
However, this seems a lot less readable, and it's also about 20% slower in a quick test, so unless you're doing this for code-golfing reasons, why?
x = numpy.array([0,0,0,0,1,0,0,0,0,0])
:P
b=np.array([int(x==4) for x in range(10)])
print(b)
[0 0 0 0 1 0 0 0 0 0]
Use this
np.where(np.arange(10)==4,1,0)
I don't that it is possible to combine:
x=np.zeros(10) #1 create an array of values
x[4]=1 #2 assign at least one value of an array a different value
in one line.
Naive:
x = np.zeros(10)[4] = 1 # Fails
fails as value of x is 1 due how python handles chained assignment. Both x and element 4 in array of zeros are assigned value 1.
Therefore, we need to first create an array of zeros, and then assign element assign element 4 a value of 1, and these two cannot be done in one line.
If you need to do this to multiple elements:
my_vect = np.zeros(42)
# set index 1 and 3 to value 1
my_vect[np.array([1,3])] = 1
Create a null vector of size 10 but the fifth value which is 1.
import numpy
x = np.zeros(10, dtype=int)
if x[5] == 0:
x[5] = 1
print(x)
array = numpy.eye( array_size )[ element_in_array_which_should_be_1 - 1 ]
So to create a null vector of size 10 but the fifth value being 1 in one line is
array = numpy.eye( 10 ) [ 5 - 1 ]
===> array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0.])
:)
I have an algorithm with plain Python loops which I would like to optimize
for speed.
Starting from an array indicating bin indices, I want summed up values for those bins.
More detailed: I start from an index array pointing to a vector of values and values for the same index should be summed up. The plain and slow Python version is like:
import numpy
ix = numpy.array([0 , 1 , 1 , 4 ])
values = numpy.array([10, 20, 30, 40])
# this models bin assignment:
# 10 belongs to bin 0
# 20 and 30 belong to bin 1
# 40 belongs to bin 4
summed = numpy.zeros_like(values)
for i in ix:
summed[i] += values[ix[i]]
print summed
[10, 50, 0, 0, 40]
This is quite slow and I ask if anybody can give me a hint how to
vectorize this.
You can use numpy.bincount():
>>> numpy.bincount(ix, values)
array([ 10., 50., 0., 0., 40.])
Bit of a hack but this gives the same result
indicator = np.arange(5)[None, ...] == ix[..., None]
summed = np.sum(values[..., None] * indicator, axis=0)