Extract n columns with highest sum in NumPy array - python

Imagine I have a NumPy matrix with 100 rows and 1000 columns.
How do I get a new matrix composed by the n columns that have the highest sums in the original matrix?

You can use np.argsort as done by #NPE here. Here's an example on two smaller arrays:
def nlargest_cols(a, n):
return a[:, sorted(a.sum(axis=0).argsort()[-n:][::-1])]
# `a` is a 3x4 array with column sums getting
# larger from left to right.
a = np.arange(12).reshape(3,4)
# `b` is `a` rotated 2 turns.
b = np.rot90(a, 2)
print(nlargest_cols(a, 2))
# [[ 2 3]
# [ 6 7]
# [10 11]]
print(nlargest_cols(b, 3))
# [[11 10 9]
# [ 7 6 5]
# [ 3 2 1]]

Related

Fastest way to fill numpy array with new arrays from function

I have a function f(a) that takes one entry from a testarray and returns an array with 5 values:
f(testarray[0])
#Output: array([[0, 1, 5, 3, 2]])
Since f(testarray[0]) is the result of an experiment, I want to run this function f for each entry of the testarray and store each result in a new NumPy array. I always thought this would be quite simple by just taking an empty NumPy array with the length of the testarray and save the results the following way:
N = 1000 #Number of entries of the testarray
test_result = np.zeros([N, 5], dtype=int)
for i in testarray:
test_result[i] = f(i)
When I run this, I don't receive any error message but nonsense results (half of the test_result is empty while the rest is filled with implausible values). Since f() works perfectly for a single entry of the testarray I suppose that something of the way of how I save the results in the test_result is wrong. What am I missing here?
(I know that I could save the results as list and then append an empty list, but this method is too slow for the large number of times I want to run the function).
Since you don't seem to understand indexing, stick with this approach
alist = [f(i) for i in testarray]
arr = np.array(alist)
I could show how to use row indices and testarray values together, but that requires more explanation.
Your problem may could be reproduced by the following small example:
testarray = np.array([5, 6, 7, 3, 1])
def f(x):
return np.array([x * i for i in np.arange(1, 6)])
f(testarray[0])
# [ 5 10 15 20 25]
test_result = np.zeros([len(testarray), 5], dtype=int) # len(testarray) or testarray.shape[0]
So, as hpaulj mentioned in the comments, you must be careful how to use indexing:
for i in range(len(testarray)):
test_result[i] = f(testarray[i])
# [[ 5 10 15 20 25]
# [ 6 12 18 24 30]
# [ 7 14 21 28 35]
# [ 3 6 9 12 15]
# [ 1 2 3 4 5]]
There will be another condition where the testarray is a specified index array that contains shuffle integers from 0 to N to full fill the zero array i.e. test_result. For this condition we can create a reproducible example as:
testarray = np.array([4, 3, 0, 1, 2])
def f(x):
return np.array([x * i for i in np.arange(1, 6)])
f(testarray[0])
# [ 4 8 12 16 20]
test_result = np.zeros([len(testarray), 5], dtype=int)
So, using your loop will get the following result:
for i in testarray:
test_result[i] = f(i)
# [[ 0 0 0 0 0]
# [ 1 2 3 4 5]
# [ 2 4 6 8 10]
# [ 3 6 9 12 15]
# [ 4 8 12 16 20]]
As it can be understand from this loop, if the index array be not from 0 to N, some rows in the zero array will left zero (unchanged):
testarray = np.array([4, 2, 4, 1, 2])
for i in testarray:
test_result[i] = f(i)
# [[ 0 0 0 0 0] # <--
# [ 1 2 3 4 5]
# [ 2 4 6 8 10]
# [ 0 0 0 0 0] # <--
# [ 4 8 12 16 20]]

Substitute row in Numpy if a condition is met - Variation

I am still figuring out Numpy syntax! I have something that works but there must be a more concise way to perform this task. In the example below, I replace selected rows of an array with new entries, where the condition is just on one element.
import numpy as np
big_array = np.random.randint(10, size=(5, 2)) # multi-dimension array
print(big_array)
bad_values = np.less_equal(big_array[:,0], 4) # condition value in one dimension
bad_rows = np.nonzero(bad_values)[0] # indexes to change, e.g. rows
print(f'these are the rows to replace {bad_rows}')
new_rows = np.random.randint(10, size=((bad_rows.size),2))+10 # smaller multi-dim array
np.put(big_array[:,0],bad_rows,y[:,0]) # should be a single line to combine this
np.put(big_array[:,1],bad_rows,y[:,1]) # with this?
print(big_array)
sample output that I want might look like
[[2 4]
[5 9]
[6 6]
[6 7]
[0 6]]
these are the rows to replace [0 4]
[[16 17]
[ 5 9]
[ 6 6]
[ 6 7]
[18 17]]
I don't know how to format put for arguments with different dimensions. This seems like it should be a one-liner. (If I try where I get length issues broadcasting.) What am I missing?

Take input in a function, divide the array in equal parts and print child array with highest value Python 3.6

I am trying to write a function which will take array as an input.
The array will always be divisible by 4. The array needs to be split into 4 parts equally.
Let's call these 4 arrays parts "One" "Two" "Three" "Four".
The array name needs to be the output which has the maximum difference between it's any 2 elements:
e.g
ONE [2 ,-3 , 3]
TWO [1 ,10, 8]
THREE [2 ,5, 13]
FOUR [-5, 3 ,-18]
The output should be "Four" because -18--5 = 21 which is the maximum.
import numpy as np
def solution(T):
arr = T
newarr = np.array_split(arr, 4)
print('[%s]' % ', '.join(map(str, newarr)))
The code below will print [ -5 3 -18], which is the sub-array that has the highest difference between its largest and smallest number.
import numpy as np
def solution(T):
arr = T
newarr = np.array_split(arr, 4)
arrayIdx = np.argmax(np.amax(newarr, axis=1)-np.amin(newarr, axis=1))
return newarr[arrayIdx]
array = [2 ,-3 , 3, 1 ,10, 8, 2 ,5, 13, -5, 3 ,-18]
print(solution(array))

How to efficiently compute logsumexp of upper triangle in a nested loop?

I have a nested for loop that iterates over rows of the weight matrix and applies logsumexp to the upper triangular portion of the outer addition matrix from these weights rows. It is very slow so I'm trying to figure out how to speed this up by either vectorizing or taking out the loops in lieu of matrix operations.
'''
Wm: weights matrix, nxk
W: updated weights matrix, nxn
triu_inds: upper triangular indices of Wxy outer matrix
'''
for x in range(n-1):
wx = Wm[x, :]
for y in range(x+1, n):
wy = Wm[y, :]
Wxy = np.add.outer(wx, wy)
Wxy = Wxy[triu_inds]
W[x, y] = logsumexp(Wxy)
logsumexp: computes the log of the sum of exponentials of an input array
a: [1, 2, 3]
logsumexp(a) = log( exp(1) + exp(2) + exp(3) )
The input data Wm is a weights matrix of nxk dimensions. K represents a patients sensor locations and n represents all such possible sensor locations. The values in Wm are basically how close a patients sensor is to a known sensor.
example:
Wm = [1 2 3]
[4 5 6]
[7 8 9]
[10 11 12]
wx = [1 2 3]
wy = [4 5 6]
Wxy = [5 6 7]
[6 7 8]
[7 8 9]
triu_indices = ([0, 0, 1], [1, 2, 2])
Wxy[triu_inds] = [6, 7, 8]
logsumexp(Wxy[triu_inds]) = log(exp(6) + exp(7) + exp(8))
You can perform the outer product on the full matrix Wm and then swap the axes corresponding to columns in operand 1 and rows in operand 2 in order to apply the triangle indices to the columns. The resulting matrix is filled for all combinations of rows, so you need to select the upper triangle part.
W = logsumexp(
np.add.outer(Wm, Wm).swapaxes(1, 2)[(slice(None),)*2 + triu_inds],
axis=-1 # Perform summation over last axis.
)
W = np.triu(W, k=1)

Needing to assess smaller 3D arrays in larger 3D array with Numpy

I have to take a random integer 50x50x50 array and determine which contiguous 3x3x3 cube within it has the largest sum.
It seems like a lot of splitting features in Numpy don't work well unless the smaller cubes are evenly divisible into the larger one. Trying to work through the thought process I made a 48x48x48 cube that is just in order from 1 to 110,592. I then was thinking of reshaping it to a 4D array with the following code and assessing which of the arrays had the largest sum? when I enter this code though it splits the array in an order that is not ideal. I want the first array to be the 3x3x3 cube that would have been in the corner of the 48x48x48 cube. Is there a syntax that I can add to make this happen?
import numpy as np
arr1 = np.arange(0,110592)
arr2=np.reshape(arr1, (48,48,48))
arr3 = np.reshape(arr2, (4096, 3,3,3))
arr3
output:
array([[[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[ 12, 13, 14],
[ 15, 16, 17]],
[[ 18, 19, 20],
[ 21, 22, 23],
[ 24, 25, 26]]],
desired output:
array([[[[ 0, 1, 2],
[ 48, 49, 50],
[ 96, 97, 98]],
etc etc
Solution
There's a live version of this solution online you can try for yourself
There's a simple (kind of) solution to your original problem of finding the maximum 3x3x3 subcube in a 50x50x50 cube that's based on changing the input array's strides. This solution is completely vectorized (meaning no looping), and so should get the best possible performance out of Numpy:
import numpy as np
def cubecube(arr, cshape):
strides = (*arr.strides, *arr.strides)
shape = (*np.array(arr.shape) - cshape + 1, *cshape)
return np.lib.stride_tricks.as_strided(arr, shape=shape, strides=strides)
def maxcube(arr, cshape):
cc = cubecube(arr, cshape)
ccsums = cc.sum(axis=tuple(range(-arr.ndim, 0)))
ix = np.unravel_index(np.argmax(ccsums), ccsums.shape)[:arr.ndim]
return ix, cc[ix]
The maxcube function takes an array and the shape of the subcubes, and returns a tuple of (first-index-of-largest-cube, largest-cube). Here's an example of how to use maxcube:
shape = (50, 50, 50)
cshape = (3, 3, 3)
# set up a 50x50x50 array
arr = np.arange(np.prod(shape)).reshape(*shape)
# set one of the subcubes as the largest
arr[37, 26, 11] = 999999
ix, cube = maxcube(arr, cshape)
print('first index of largest cube: {}'.format(ix))
print('largest cube:\n{}'.format(cube))
which outputs:
first index of largest cube: (37, 26, 11)
largest cube:
[[[999999 93812 93813]
[ 93861 93862 93863]
[ 93911 93912 93913]]
[[ 96311 96312 96313]
[ 96361 96362 96363]
[ 96411 96412 96413]]
[[ 98811 98812 98813]
[ 98861 98862 98863]
[ 98911 98912 98913]]]
In depth explanation
A cube of cubes
What you have is a 48x48x48 cube, but what you want is a cube of smaller cubes. One can be converted to the other by altering its strides. For a 48x48x48 array of dtype int64, the stride will originally be set as (48*48*8, 48*8, 8). The first value of each non-overlapping 3x3x3 subcube can be iterated over with a stride of (3*48*48*8, 3*48*8, 3*8). Combine these strides to get the strides of the cube of cubes:
# Set up a 48x48x48 array, like in OP's example
arr = np.arange(48**3).reshape(48,48,48)
shape = (16,16,16,3,3,3)
strides = (3*48*48*8, 3*48*8, 3*8, 48*48*8, 48*8, 8)
# restride into a 16x16x16 array of 3x3x3 cubes
arr2 = np.lib.stride_tricks.as_strided(arr, shape=shape, strides=strides)
arr2 is a view of arr (meaning that they share data, so no copy needs to be made) with a shape of (16,16,16,3,3,3). The ijkth 3x3 cube in arr can be accessed by passing the indices to arr2:
i,j,k = 0,0,0
print(arr2[i,j,k])
Output:
[[[ 0 1 2]
[ 48 49 50]
[ 96 97 98]]
[[2304 2305 2306]
[2352 2353 2354]
[2400 2401 2402]]
[[4608 4609 4610]
[4656 4657 4658]
[4704 4705 4706]]]
You can get the sums of all of the subcubes by just summing across the inner axes:
sumOfSubcubes = arr2.sum(3,4,5)
This will yield a 16x16x16 array in which each value is the sum of a non-overlapping 3x3x3 subcube from your original array. This solves the specific problem about the 48x48x48 array that the OP asked about. Restriding can also be used to find all of the overlapping 3x3x3 cubes, as in the cubecube function above.
Your thought process with the 48x48x48 cube goes in the right direction insofar that there are 48³ different contiguous 3x3x3 cubes within the 50x50x50 array, though I don't understand why you would want to reshape it.
What you could do is add all 27 values of each 3x3x3 cube to a 48x48x48 dimensional array by going through all 27 permutations of adjacent slices and find the maximum over it. The found entry will give you the index tuple coordinate_index of the cube corner that is closest to the origin of your original array.
import numpy as np
np.random.seed(0)
array_shape = np.array((50,50,50), dtype=int)
cube_dim = np.array((3,3,3), dtype=int)
original_array = np.random.randint(array_shape)
reduced_shape = array_shape - cube_dim + 1
sum_array = np.zeros(reduced shape, dtype=int)
for i in range(cube_dim[0]):
for j in range(cube_dim[1]):
for k in range(cube_dim[2]):
sum_array += original_array[
i:-cube_dim[0]+1+i, j:-cube_dim[1]+1+j, k:-cube_dim[2]+1+k
]
flat_index = np.argmax(sum_array)
coordinate_index = np.unravel_index(flat_index, reduced_shape)
This method should be faster than looping over each of the 48³ index combinations to find the desired cube as it uses in place summation but in turn requires more memory. I'm not sure about it, but defining an (48³, 27) array with slices and using np.sum over the second axis could be even faster.
You can easily change the above code to find a cuboid with arbitrary side lengths instead.
This is a solution without many numpy functions, just numpy.sum. First define a squared matrix and then the size of the cube cs you are going to perform the summation within.
Just change cs to adjust the cube size and find other solutions. Following #Divakar suggestion, I have used a 4x4x4 array and I also store the location where the cube is location (just the vertex of the cube's origin)
import numpy as np
np.random.seed(0)
a = np.random.randint(0,9,(4,4,4))
print(a)
cs = 2 # Cube size
my_sum = 0
idx = None
for i in range(a.shape[0]-cs+2):
for j in range(a.shape[1]-cs+2):
for k in range(a.shape[2]-cs+2):
cube_sum = np.sum(a[i:i+cs, j:j+cs, k:k+cs])
print(cube_sum)
if cube_sum > my_sum:
my_sum = cube_sum
idx = (i,j,k)
print(my_sum, idx) # 42 (0, 0, 0)
This 3D array a is
[[[5 0 3 3]
[7 3 5 2]
[4 7 6 8]
[8 1 6 7]]
[[7 8 1 5]
[8 4 3 0]
[3 5 0 2]
[3 8 1 3]]
[[3 3 7 0]
[1 0 4 7]
[3 2 7 2]
[0 0 4 5]]
[[5 6 8 4]
[1 4 8 1]
[1 7 3 6]
[7 2 0 3]]]
And you get my_sum = 42 and idx = (0, 0, 0) for cs = 2. And my_sum = 112 and idx = (1, 0, 0) for cs = 3
Here is a cumsum based fast solution:
import numpy as np
nd = 3
cs = 3
N = 50
# create indices [cs-1:, ...], [:, cs-1:, ...], ...
fromcsm = *zip(*np.where(np.identity(nd, bool), np.s_[cs-1:], np.s_[:])),
# create indices [cs:, ...], [:, cs:, ...], ...
fromcs = *zip(*np.where(np.identity(nd, bool), np.s_[cs:], np.s_[:])),
# create indices [:cs, ...], [:, :cs, ...], ...
tocs = *zip(*np.where(np.identity(nd, bool), np.s_[:cs], np.s_[:])),
# create indices [:-cs, ...], [:, :-cs, ...], ...
tomcs = *zip(*np.where(np.identity(nd, bool), np.s_[:-cs], np.s_[:])),
# create indices [cs-1, ...], [:, cs-1, ...], ...
atcsm = *zip(*np.where(np.identity(nd, bool), cs-1, np.s_[:])),
def windowed_sum(a):
out = a.copy()
for i, (fcsm, fcs, tcs, tmcs, acsm) \
in enumerate(zip(fromcsm, fromcs, tocs, tomcs, atcsm)):
out[fcs] -= out[tmcs]
out[acsm] = out[tcs].sum(axis=i)
out = out[fcsm].cumsum(axis=i)
return out
This returns the sums over all the sub cubes. We can then use argmax and unravel_index to get the offset of the maximum cube. Example:
np.random.seed(0)
a = np.random.randint(0,9,(N,N,N))
s = windowed_sum(a)
idx = np.unravel_index(np.argmax(s,), s.shape)

Categories

Resources