Develop Reference

Improving distribution array element distribution

I am trying to write a command in Python that would calculate the distribution.
For an example the call:
n=10
x = [1]*3 # returns [1,1,1]
y = [s/sum(a)*n for s in x] # returns [3.3,3.3,3.3]
The problem is that this way, the program I am using would round down the values to 3 and basically, the sum of the array would be 9 instead of 10. How could I improve the call so that I get whole values like [4.0,3.0,3.0]?

The problem is to find which elements to round up while rounding all the rest down. One way to decide is that the element with the largest fractional part gets rounded up but no element is rounded up more than once.
import numpy as np
def distribute(weights, total):
# calculate the distribution as you have done
dist = np.array(weights) / sum(weights) * total
# separate the fractional parts
fracs, dist = np.modf(dist)
# adjust if necessary
while dist.sum() != total:
# find the index of the maximum value of the fractional parts
max_idx = np.argmax(fracs)
# increment just that one value
dist[max_idx] += 1
# zero out that fractional part so we don't use it twice
fracs[max_idx] = 0
return dist
>>> distribute([1]*3, 10)
array([4., 3., 3.])
>>> distribute([1, 2, 3], 16)
array([3., 5., 8.])
>>> distribute([1, 2, 3, 1, 1], 21)
array([3., 5., 8., 3., 2.])

Add 2 column vector to ndarray [duplicate]

I have two numpy arrays of different shapes, but with the same length (leading dimension). I want to shuffle each of them, such that corresponding elements continue to correspond -- i.e. shuffle them in unison with respect to their leading indices.
This code works, and illustrates my goals:
def shuffle_in_unison(a, b):
assert len(a) == len(b)
shuffled_a = numpy.empty(a.shape, dtype=a.dtype)
shuffled_b = numpy.empty(b.shape, dtype=b.dtype)
permutation = numpy.random.permutation(len(a))
for old_index, new_index in enumerate(permutation):
shuffled_a[new_index] = a[old_index]
shuffled_b[new_index] = b[old_index]
return shuffled_a, shuffled_b
For example:
>>> a = numpy.asarray([[1, 1], [2, 2], [3, 3]])
>>> b = numpy.asarray([1, 2, 3])
>>> shuffle_in_unison(a, b)
(array([[2, 2],
[1, 1],
[3, 3]]), array([2, 1, 3]))
However, this feels clunky, inefficient, and slow, and it requires making a copy of the arrays -- I'd rather shuffle them in-place, since they'll be quite large.
Is there a better way to go about this? Faster execution and lower memory usage are my primary goals, but elegant code would be nice, too.
One other thought I had was this:
def shuffle_in_unison_scary(a, b):
rng_state = numpy.random.get_state()
numpy.random.shuffle(a)
numpy.random.set_state(rng_state)
numpy.random.shuffle(b)
This works...but it's a little scary, as I see little guarantee it'll continue to work -- it doesn't look like the sort of thing that's guaranteed to survive across numpy version, for example.

Your can use NumPy's array indexing:
def unison_shuffled_copies(a, b):
assert len(a) == len(b)
p = numpy.random.permutation(len(a))
return a[p], b[p]
This will result in creation of separate unison-shuffled arrays.

X = np.array([[1., 0.], [2., 1.], [0., 0.]])
y = np.array([0, 1, 2])
from sklearn.utils import shuffle
X, y = shuffle(X, y, random_state=0)
To learn more, see http://scikit-learn.org/stable/modules/generated/sklearn.utils.shuffle.html

Your "scary" solution does not appear scary to me. Calling shuffle() for two sequences of the same length results in the same number of calls to the random number generator, and these are the only "random" elements in the shuffle algorithm. By resetting the state, you ensure that the calls to the random number generator will give the same results in the second call to shuffle(), so the whole algorithm will generate the same permutation.
If you don't like this, a different solution would be to store your data in one array instead of two right from the beginning, and create two views into this single array simulating the two arrays you have now. You can use the single array for shuffling and the views for all other purposes.
Example: Let's assume the arrays a and b look like this:
a = numpy.array([[[ 0., 1., 2.],
[ 3., 4., 5.]],
[[ 6., 7., 8.],
[ 9., 10., 11.]],
[[ 12., 13., 14.],
[ 15., 16., 17.]]])
b = numpy.array([[ 0., 1.],
[ 2., 3.],
[ 4., 5.]])
We can now construct a single array containing all the data:
c = numpy.c_[a.reshape(len(a), -1), b.reshape(len(b), -1)]
# array([[ 0., 1., 2., 3., 4., 5., 0., 1.],
# [ 6., 7., 8., 9., 10., 11., 2., 3.],
# [ 12., 13., 14., 15., 16., 17., 4., 5.]])
Now we create views simulating the original a and b:
a2 = c[:, :a.size//len(a)].reshape(a.shape)
b2 = c[:, a.size//len(a):].reshape(b.shape)
The data of a2 and b2 is shared with c. To shuffle both arrays simultaneously, use numpy.random.shuffle(c).
In production code, you would of course try to avoid creating the original a and b at all and right away create c, a2 and b2.
This solution could be adapted to the case that a and b have different dtypes.

Very simple solution:
randomize = np.arange(len(x))
np.random.shuffle(randomize)
x = x[randomize]
y = y[randomize]
the two arrays x,y are now both randomly shuffled in the same way

James wrote in 2015 an sklearn solution which is helpful. But he added a random state variable, which is not needed. In the below code, the random state from numpy is automatically assumed.
X = np.array([[1., 0.], [2., 1.], [0., 0.]])
y = np.array([0, 1, 2])
from sklearn.utils import shuffle
X, y = shuffle(X, y)

from np.random import permutation
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data #numpy array
y = iris.target #numpy array
# Data is currently unshuffled; we should shuffle
# each X[i] with its corresponding y[i]
perm = permutation(len(X))
X = X[perm]
y = y[perm]

Shuffle any number of arrays together, in-place, using only NumPy.
import numpy as np
def shuffle_arrays(arrays, set_seed=-1):
"""Shuffles arrays in-place, in the same order, along axis=0
Parameters:
-----------
arrays : List of NumPy arrays.
set_seed : Seed value if int >= 0, else seed is random.
"""
assert all(len(arr) == len(arrays[0]) for arr in arrays)
seed = np.random.randint(0, 2**(32 - 1) - 1) if set_seed < 0 else set_seed
for arr in arrays:
rstate = np.random.RandomState(seed)
rstate.shuffle(arr)
And can be used like this
a = np.array([1, 2, 3, 4, 5])
b = np.array([10,20,30,40,50])
c = np.array([[1,10,11], [2,20,22], [3,30,33], [4,40,44], [5,50,55]])
shuffle_arrays([a, b, c])
A few things to note:
The assert ensures that all input arrays have the same length along
their first dimension.
Arrays shuffled in-place by their first dimension - nothing returned.
Random seed within positive int32 range.
If a repeatable shuffle is needed, seed value can be set.
After the shuffle, the data can be split using np.split or referenced using slices - depending on the application.

you can make an array like:
s = np.arange(0, len(a), 1)
then shuffle it:
np.random.shuffle(s)
now use this s as argument of your arrays. same shuffled arguments return same shuffled vectors.
x_data = x_data[s]
x_label = x_label[s]

There is a well-known function that can handle this:
from sklearn.model_selection import train_test_split
X, _, Y, _ = train_test_split(X,Y, test_size=0.0)
Just setting test_size to 0 will avoid splitting and give you shuffled data.
Though it is usually used to split train and test data, it does shuffle them too.
From documentation
Split arrays or matrices into random train and test subsets
Quick utility that wraps input validation and
next(ShuffleSplit().split(X, y)) and application to input data into a
single call for splitting (and optionally subsampling) data in a
oneliner.

This seems like a very simple solution:
import numpy as np
def shuffle_in_unison(a,b):
assert len(a)==len(b)
c = np.arange(len(a))
np.random.shuffle(c)
return a[c],b[c]
a = np.asarray([[1, 1], [2, 2], [3, 3]])
b = np.asarray([11, 22, 33])
shuffle_in_unison(a,b)
Out[94]:
(array([[3, 3],
[2, 2],
[1, 1]]),
array([33, 22, 11]))

One way in which in-place shuffling can be done for connected lists is using a seed (it could be random) and using numpy.random.shuffle to do the shuffling.
# Set seed to a random number if you want the shuffling to be non-deterministic.
def shuffle(a, b, seed):
np.random.seed(seed)
np.random.shuffle(a)
np.random.seed(seed)
np.random.shuffle(b)
That's it. This will shuffle both a and b in the exact same way. This is also done in-place which is always a plus.
EDIT, don't use np.random.seed() use np.random.RandomState instead
def shuffle(a, b, seed):
rand_state = np.random.RandomState(seed)
rand_state.shuffle(a)
rand_state.seed(seed)
rand_state.shuffle(b)
When calling it just pass in any seed to feed the random state:
a = [1,2,3,4]
b = [11, 22, 33, 44]
shuffle(a, b, 12345)
Output:
>>> a
[1, 4, 2, 3]
>>> b
[11, 44, 22, 33]
Edit: Fixed code to re-seed the random state

Say we have two arrays: a and b.
a = np.array([[1,2,3],[4,5,6],[7,8,9]])
b = np.array([[9,1,1],[6,6,6],[4,2,0]])
We can first obtain row indices by permutating first dimension
indices = np.random.permutation(a.shape[0])
[1 2 0]
Then use advanced indexing.
Here we are using the same indices to shuffle both arrays in unison.
a_shuffled = a[indices[:,np.newaxis], np.arange(a.shape[1])]
b_shuffled = b[indices[:,np.newaxis], np.arange(b.shape[1])]
This is equivalent to
np.take(a, indices, axis=0)
[[4 5 6]
[7 8 9]
[1 2 3]]
np.take(b, indices, axis=0)
[[6 6 6]
[4 2 0]
[9 1 1]]

If you want to avoid copying arrays, then I would suggest that instead of generating a permutation list, you go through every element in the array, and randomly swap it to another position in the array
for old_index in len(a):
new_index = numpy.random.randint(old_index+1)
a[old_index], a[new_index] = a[new_index], a[old_index]
b[old_index], b[new_index] = b[new_index], b[old_index]
This implements the Knuth-Fisher-Yates shuffle algorithm.

Shortest and easiest way in my opinion, use seed:
random.seed(seed)
random.shuffle(x_data)
# reset the same seed to get the identical random sequence and shuffle the y
random.seed(seed)
random.shuffle(y_data)

most solutions above work, however if you have column vectors you have to transpose them first. here is an example
def shuffle(self) -> None:
"""
Shuffles X and Y
"""
x = self.X.T
y = self.Y.T
p = np.random.permutation(len(x))
self.X = x[p].T
self.Y = y[p].T

With an example, this is what I'm doing:
combo = []
for i in range(60000):
combo.append((images[i], labels[i]))
shuffle(combo)
im = []
lab = []
for c in combo:
im.append(c[0])
lab.append(c[1])
images = np.asarray(im)
labels = np.asarray(lab)

I extended python's random.shuffle() to take a second arg:
def shuffle_together(x, y):
assert len(x) == len(y)
for i in reversed(xrange(1, len(x))):
# pick an element in x[:i+1] with which to exchange x[i]
j = int(random.random() * (i+1))
x[i], x[j] = x[j], x[i]
y[i], y[j] = y[j], y[i]
That way I can be sure that the shuffling happens in-place, and the function is not all too long or complicated.

Just use numpy...
First merge the two input arrays 1D array is labels(y) and 2D array is data(x) and shuffle them with NumPy shuffle method. Finally split them and return.
import numpy as np
def shuffle_2d(a, b):
rows= a.shape[0]
if b.shape != (rows,1):
b = b.reshape((rows,1))
S = np.hstack((b,a))
np.random.shuffle(S)
b, a = S[:,0], S[:,1:]
return a,b
features, samples = 2, 5
x, y = np.random.random((samples, features)), np.arange(samples)
x, y = shuffle_2d(train, test)

Summing an array along different dim each time with different slice range

Suppose I have an array b of shape (3, 10, 3) and another array v = [8, 9, 4] of shape (3,), see below. For each of the 3 arrays of shape (10, 3) in b, I need to sum a number of rows as determined by v, i.e. for i = 0, 1, 2 I need to get np.sum(b[i, 0:v[i]], axis=0). My solution (shown below) uses a for loop which is inefficient I guess. I wonder if there is an efficient (vectorized) way to do what I have described above.
NB: my actual arrays have more dimension, these arrays are for illustration.
v = np.array([8,9,4])
b = np.array([[[0., 1., 0.],
[0., 0., 1.],
[0., 0., 1.],
[0., 0., 1.],
[1., 0., 0.],
[1., 0., 0.],
[0., 0., 1.],
[1., 0., 0.],
[0., 1., 0.],
[1., 0., 0.]],
[[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
[0., 0., 1.],
[1., 0., 0.],
[1., 0., 0.],
[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.],
[0., 1., 0.]],
[[1., 0., 0.],
[1., 0., 0.],
[1., 0., 0.],
[0., 0., 1.],
[0., 1., 0.],
[0., 1., 0.],
[1., 0., 0.],
[1., 0., 0.],
[0., 0., 1.],
[1., 0., 0.]]])
n=v.shape[0]
vv=np.zeros([n, n])
for i in range(n):
vv[i]=np.sum( b[i,0:v[i]],axis=0)
Output:
vv
array([[3., 1., 4.],
[4., 2., 3.],
[3., 0., 1.]])
Edit:
Below is more an actual example of the arrays v and b.
v= np.random.randint(0,300, size=(32, 98,3))
b = np.zeros([98, 3, 300, 3])
for i in range(3):
for j in range(98):
b[j,i] = np.random.multinomial(1,[1./3, 1./3, 1./3], 300)
v.shape
Out[292]: (32, 98, 3)
b.shape
Out[293]: (98, 3, 300, 3)
I need to do the same thing as before, so the final result is an array of shape (32,98,3,3). Note that I have to do the above at each iteration that is why I'm looking for an efficient implementation.

This is a performance comparison of the different methods presented in the answers:
sliced_reduce
sliced_sum
sliced_sum_numba
reduce_cumulative (original idea here)
baseline - The "classic" Python for loop (see below).
Notes on performance
sliced_reduce reverses the order of index pairs from ascending to descending to turn the computation of superfluous elements to no-ops; this way however the array is not traversed in memory layout order and seems to slow down the method by ~30%.
reduce_cumulative performs a number of unnecessary add operations which depends on the distribution of start and stop indices. For the OP example where start indices are all zero and stop indices are uniformly distributed this will be in average twice as many operations as strictly necessary. For other distributions (e.g. non-zero start indices) this fraction might very well change and hence degrade the performance as compared to other methods. Please check your own case.
[Disclaimer] As with all performance estimations, these are rough guidelines to give a broad overview but they don't save you from running the performance tests yourself for your specific use case on your specific machine to be absolutely certain to select the best option.
Using the example dimensions from the OP:
In [15]: np.random.seed(0)
In [16]: b = np.random.randint(0, 1000, size=(98, 3, 300, 3))
In [17]: v = np.random.randint(-299, 300, size=(32, 98, 3))
In [18]: %timeit sliced_reduce(b, np.zeros_like(v), v, np.add, axis=2)
11.3 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [19]: %timeit sliced_sum(b, np.zeros_like(v), v, axis=2)
54.9 ms ± 153 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [20]: %timeit sliced_sum_numba(b, np.zeros_like(v), v, 2)
16.3 ms ± 609 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [21]: %timeit reduce_cumulative(b, np.zeros_like(v), v, np.add, axis=2)
2.05 ms ± 30.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [22]: %timeit baseline(b, np.zeros_like(v), v, axis=2)
79 ms ± 625 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
Baseline implementation:
def baseline(a, i, j, axis=None):
if axis is None:
axis = len(i.shape)
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
m = len(i.shape) - axis
result = np.empty(i.shape + a.shape[axis+1:], dtype=a.dtype)
for k in np.ndindex(i.shape):
result[k] = np.sum(a[k[m:] + (slice(i[k], j[k]),)], axis=0)
return result
Performance plots
Besides the timings for the OP's specific example case it is instructive to check how the algorithms scale with the size of the data and index arrays. Here we can separate the shapes into three different components:
Leading dimensions of the index array (those that are not present in the data array). In the OP example this is (32,).
Common dimensions of the index and the data array (the dimensions after the leading ones up to the reduced axis). In the OP example this is (98, 3).
The size of the axis to be reduced. In the OP example this is 300.
(The trailing dimensions of the data array are handled similarly by all algorithms and hence no particular scaling is to be expected.)
Hence we can create performance plots for three different cases: Varying the size of the leading dimension(s), the common dimensions and the size of the axis to be reduced. Boundaries are chosen from 1 to N where N is the largest power of 2 such that no involved array has more than 5,000,000 elements (input, index, output; intermediary arrays might be larger (such as for sliced_reduce)).
For the code see below.
Leading dimensions
Common dimensions
Reduced dimension
Code
from string import ascii_lowercase as symbols
import numba
import numpy as np
import perfplot
np.random.seed(0)
def sliced_reduce(a, i, j, ufunc=np.add, axis=2):
indices = np.tile(
np.repeat(
np.arange(np.prod(a.shape[:axis])) * a.shape[axis],
2
),
np.prod(i.shape[:len(i.shape) - axis])
)
indices[::2] += (a.shape[axis] + i.ravel()) % a.shape[axis]
indices[1::2] += (a.shape[axis] + j.ravel()) % a.shape[axis]
indices = indices.reshape(-1, 2)[::-1].ravel() # This seems to be counter-effective, please check for your own case.
result = ufunc.reduceat(a.reshape(-1, *a.shape[axis+1:]), indices)[::2] # Select only even to odd.
result[indices[::2] == indices[1::2]] = ufunc.reduce([])
return result[::-1].reshape(*(i.shape + a.shape[axis+1:]))
def sliced_sum(a, i, j, axis=2):
l = len(i.shape) - axis
m = len(i.shape) - l
n = len(a.shape) - axis - 1
leading = symbols[:l]
common = symbols[l:l+m]
summation = symbols[l+m]
trailing = symbols[l+m+1:l+m+1+n]
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
indices, i, j = np.broadcast_arrays(np.arange(a.shape[axis]),
np.expand_dims(i, -1), np.expand_dims(j, -1))
active_elements = (i <= indices) & (indices < j)
return np.einsum(f'{leading + common + summation},{common + summation + trailing}->{leading + common + trailing}',
active_elements, a)
def sliced_sum_numba(a, i, j, axis=2):
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
m = np.prod(i.shape[:len(i.shape) - axis], dtype=int)
n = np.prod(i.shape[len(i.shape) - axis:], dtype=int)
a_flat = a.reshape(-1, *a.shape[axis:])
i_flat = i.ravel()
j_flat = j.ravel()
result = np.empty((m*n,) + a.shape[axis+1:], dtype=a.dtype)
numba_sum(a_flat, i_flat, j_flat, m, n, result)
return result.reshape(*(i.shape + a.shape[axis+1:]))
#numba.jit(parallel=True, nopython=True)
def numba_sum(a, i, j, m, n, out):
for index in numba.prange(m*n):
out[index] = np.sum(a[index % n, i[index]:j[index]], axis=0)
def reduce_cumulative(a, i, j, ufunc=np.add, axis=2):
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
a = np.insert(a, 0, 0, axis)
c = ufunc.accumulate(a, axis=axis)
pre = np.ix_(*(range(x) for x in i.shape))
l = len(i.shape) - axis
return c[pre[l:] + (j,)] - c[pre[l:] + (i,)]
def baseline(a, i, j, axis=2):
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
m = len(i.shape) - axis
result = np.empty(i.shape + a.shape[axis+1:], dtype=a.dtype)
for k in np.ndindex(i.shape):
result[k] = np.sum(a[k[m:] + (slice(i[k], j[k]),)], axis=0)
return result
a = np.random.randint(0, 1000, size=(98, 3, 300, 3))
j = np.random.randint(-299, 300, size=(32, 98, 3))
i = np.zeros_like(j)
check = [f(a, i, j) for f in [sliced_reduce, sliced_sum, sliced_sum_numba, reduce_cumulative, baseline]]
assert all(np.array_equal(check[0], x) for x in check[1:])
perfplot.show(
# Leading dimensions:
# setup = lambda n: (np.random.randint(0, 1000, size=(98, 3, 300, 3)),
# np.zeros((n, 98, 3), dtype=int),
# np.random.randint(-299, 300, size=(n, 98, 3))),
# Common dimensions:
# setup = lambda n: (np.random.randint(0, 1000, size=(n, 3, 300, 3)),
# np.zeros((32, n, 3), dtype=int),
# np.random.randint(-299, 300, size=(32, n, 3))),
# Reduced dimension:
setup = lambda n: (np.random.randint(0, 1000, size=(98, 3, n, 3)),
np.zeros((32, 98, 3), dtype=int),
np.random.randint(-n+1, n, size=(32, 98, 3))),
kernels=[
lambda a: sliced_reduce(*a),
lambda a: sliced_sum(*a),
lambda a: sliced_sum_numba(*a),
lambda a: reduce_cumulative(*a),
lambda a: baseline(*a),
],
labels=['sliced_reduce', 'sliced_sum', 'sliced_sum_numba', 'reduce_cumulative', 'baseline'],
# n_range=[2 ** k for k in range(13)], # Leading dimensions.
# n_range=[2 ** k for k in range(11)], # Common dimensions.
n_range=[2 ** k for k in range(2, 13)], # Reduced dimension.
# xlabel='Size of leading dimension',
# xlabel='Size of first common dimension (second is 3)',
xlabel='Size of reduced dimension',
)

The following function allows for summing a given axis with varying slices indicated by start and stop arrays. It uses np.einsum under the hood together with an appropriately computed coefficient array that indicates which elements in the input array should participate in the sum (using coefficients 1 and 0). Relying on einsum makes the implementation compatible with other packages such as PyTorch or TensorFlow (with minor changes). It doubles the number of necessary computations since for each add operation comes an additional multiply operation with the coefficient array.
from string import ascii_lowercase as symbols
import numpy as np
def sliced_sum(a, i, j, axis=None):
"""Sum an array along a given axis for varying slices `a[..., i:j, ...]` where `i` and `j` are arrays themselves.
Parameters
----------
a : array
The array to be summed over.
i : array
The start indices for the summation axis. Must have the same shape as `j`.
j : array
The stop indices for the summation axis. Must have the same shape as `i`.
axis : int, optional
Axis to be summed over. Defaults to `len(i.shape)`.
Returns
-------
array
Shape `i.shape + a.shape[axis+1:]`.
Notes
-----
The shapes of `a` and `i`, `j` must match up to the summation axis.
That means `a.shape[:axis] == i.shape[len(i.shape) - axis:]``.
`i` and `j` can have additional leading dimensions and `a` can have additional trailing dimensions.
"""
if axis is None:
axis = len(i.shape)
# Compute number of leading, common and trailing dimensions.
l = len(i.shape) - axis # Number of leading dimensions.
m = len(i.shape) - l # Number of common dimensions.
n = len(a.shape) - axis - 1 # Number of trailing dimensions.
# Select the corresponding symbols for `np.einsum`.
leading = symbols[:l]
common = symbols[l:l+m]
summation = symbols[l+m]
trailing = symbols[l+m+1:l+m+1+n]
# Convert negative indices.
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
# Compute the "active" elements, i.e. the ones that should participate in the summation.
# "active" elements have a coefficient of 1 (True), others are 0 (False).
indices, i, j = np.broadcast_arrays(np.arange(a.shape[axis]),
np.expand_dims(i, -1), np.expand_dims(j, -1))
active_elements = (i <= indices) & (indices < j)
return np.einsum(f'{leading + common + summation},{common + summation + trailing}->{leading + common + trailing}',
active_elements, a)
For the examples in the OP it can be used in the following way:
# 1. example:
b = np.random.randint(0, 1000, size=(3, 10, 3))
v = np.random.randint(-9, 10, size=3) # Indexing into `b.shape[1]`.
result = sliced_sum(b, np.zeros_like(v), v)
# 2. example:
b = np.random.randint(0, 1000, size=(98, 3, 300, 3))
v = np.random.randint(-299, 300, size=(32, 98, 3)) # Indexing into `b.shape[2]`; one additional leading dimension for `v`.
result = sliced_sum(b, np.zeros_like(v), v, axis=2)

Another option is to use Numba to speed up the loop. This avoids unnecessary computations and memory allocation and is fully compatible with all numpy functions (i.e. also prod etc. work similarly).
import numba
import numpy as np
def sliced_sum_numba(a, i, j, axis=None):
"""Sum an array along a given axis for varying slices `a[..., i:j, ...]` where `i` and `j` are arrays themselves.
Parameters
----------
a : array
The array to be summed over.
i : array
The start indices for the summation axis. Must have the same shape as `j`.
j : array
The stop indices for the summation axis. Must have the same shape as `i`.
axis : int, optional
Axis to be summed over. Defaults to `len(i.shape)`.
Returns
-------
array
Shape `i.shape + a.shape[axis+1:]`.
Notes
-----
The shapes of `a` and `i`, `j` must match up to the summation axis.
That means `a.shape[:axis] == i.shape[len(i.shape) - axis:]``.
`i` and `j` can have additional leading dimensions and `a` can have additional trailing dimensions.
"""
if axis is None:
axis = len(i.shape)
# Convert negative indices.
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
# Operate on a flattened version of the array (dimensions up to `axis` are flattened).
m = np.prod(i.shape[:len(i.shape) - axis], dtype=int) # Elements in leading dimensions.
n = np.prod(i.shape[len(i.shape) - axis:], dtype=int) # Elements in common dimensions.
a_flat = a.reshape(-1, *a.shape[axis:])
i_flat = i.ravel()
j_flat = j.ravel()
result = np.empty((m*n,) + a.shape[axis+1:], dtype=a.dtype)
numba_sum(a_flat, i_flat, j_flat, m, n, result)
return result.reshape(*(i.shape + a.shape[axis+1:]))
#numba.jit(parallel=True, nopython=True)
def numba_sum(a, i, j, m, n, out):
for index in numba.prange(m*n):
out[index] = np.sum(a[index % n, i[index]:j[index]], axis=0)
For the examples in the OP it can be used in the following way:
# 1. example:
b = np.random.randint(0, 1000, size=(3, 10, 3))
v = np.random.randint(-9, 10, size=3) # Indexing into `b.shape[1]`.
result = sliced_sum_numba(b, np.zeros_like(v), v)
# 2. example:
b = np.random.randint(0, 1000, size=(98, 3, 300, 3))
v = np.random.randint(-299, 300, size=(32, 98, 3)) # Indexing into `b.shape[2]`; one additional leading dimension for `v`.
result = sliced_sum_numba(b, np.zeros_like(v), v, axis=2)

Another idea, brought up by this answer (hence community wiki), is to use np.cumsum and then select the rows corresponding to the slice indices. One can deal with zero indices by inserting an additional zero-row at the beginning of the axis that is to be reduced. This approach performs unnecessary computations since it computes the full cumulative sum, beyond the final index. In case the stop indices are uniformly distributed along the axis (with median input_array.shape[axis]//2) this will in average perform twice as many add operations as necessary. Nevertheless this approach seems to perform quite well compared to other methods (at least for the dimensions indicated by the OP).
def reduce_cumulative(a, i, j, ufunc, axis=None):
if axis is None:
axis = len(i.shape)
i = (a.shape[axis] + i) % a.shape[axis]
j = (a.shape[axis] + j) % a.shape[axis]
a = np.insert(a, 0, 0, axis) # Insert zeros to account for zero indices.
c = ufunc.accumulate(a, axis=axis)
pre = np.ix_(*(range(x) for x in i.shape)) # Indices for dimensions prior to `axis`.
l = len(i.shape) - axis # Number of leading dimensions in `i` and `j`.
return c[pre[l:] + (j,)] - c[pre[l:] + (i,)]

The following function allows for reducing a given axis with varying slices indicated by start and stop arrays. It uses np.ufunc.reduceat under the hood together with appropriately reshaped versions of the input array and the indices. It avoids unnecessary computations but allocates an intermediary array two times the size of the final output array (the computation of the discarded values are however no-ops).
def sliced_reduce(a, i, j, ufunc, axis=None):
"""Reduce an array along a given axis for varying slices `a[..., i:j, ...]` where `i` and `j` are arrays themselves.
Parameters
----------
a : array
The array to be reduced.
i : array
Start indices for the reduced axis. Must have the same shape as `j`.
j : array
Stop indices for the reduced axis. Must have the same shape as `i`.
ufunc : function
The function used for reducing the indicated axis.
axis : int, optional
Axis to be reduced. Defaults to `len(i.shape)`.
Returns
-------
array
Shape `i.shape + a.shape[axis+1:]`.
Notes
-----
The shapes of `a` and `i`, `j` must match up to the reduced axis.
That means `a.shape[:axis] == i.shape[len(i.shape) - axis:]``.
`i` and `j` can have additional leading dimensions and `a` can have additional trailing dimensions.
"""
if axis is None:
axis = len(i.shape)
indices = np.tile(
np.repeat(
np.arange(np.prod(a.shape[:axis])) * a.shape[axis],
2 # Repeat two times to have start and stop indices next to each other.
),
np.prod(i.shape[:len(i.shape) - axis]) # Perform summation for each element of additional axes.
)
# Add `a.shape[axis]` to account for negative indices.
indices[::2] += (a.shape[axis] + i.ravel()) % a.shape[axis]
indices[1::2] += (a.shape[axis] + j.ravel()) % a.shape[axis]
# Now indices are sorted in ascending order but this will lead to unnecessary computation when reducing
# from odd to even indices (since we're only interested in even to odd indices).
# Hence we reverse the order of index pairs (need to reverse the result as well then).
indices = indices.reshape(-1, 2)[::-1].ravel()
result = ufunc.reduceat(a.reshape(-1, *a.shape[axis+1:]), indices)[::2] # Select only even to odd.
# In case start and stop index are equal (i.e. empty slice) `reduceat` will select the element
# corresponding to the start index. Need to supply the correct default value in this case.
result[indices[::2] == indices[1::2]] = ufunc.reduce([])
return result[::-1].reshape(*(i.shape + a.shape[axis+1:])) # Reverse order and reshape.
For the examples in the OP it can be used in the following way:
# 1. example:
b = np.random.randint(0, 1000, size=(3, 10, 3))
v = np.random.randint(-9, 10, size=3) # Indexing into `b.shape[1]`.
result = sliced_reduce(b, np.zeros_like(v), v, np.add)
# 2. example:
b = np.random.randint(0, 1000, size=(98, 3, 300, 3))
v = np.random.randint(-299, 300, size=(32, 98, 3)) # Indexing into `b.shape[2]`; one additional leading dimension for `v`.
result = sliced_reduce(b, np.zeros_like(v), v, np.add, axis=2)
Notes
Reversing the order of flat index pairs in order to have even < odd and thus shortcut every second computation with a no-op doesn't seem to be a good idea (probably because the flattened array is not traversed in memory layout order anymore). Removing this part and using the flat indices in ascending order gives a performance increase of about 30% (also for the perfplots, though not included there).

For what it's worth, here's a one-liner. No promises that this is the most efficient version, because it does a lot more addition than necessary:
In [25]: b.cumsum(axis=1)[np.arange(b.shape[0]), v-1]
Out[25]:
array([[3., 1., 4.],
[4., 2., 3.],
[3., 0., 1.]])
(Also be aware that it doesn't correctly handle a 0 in v.)

Python get get average of neighbours in matrix with na value

I have very large matrix, so dont want to sum by going through each row and column.
a = [[1,2,3],[3,4,5],[5,6,7]]
def neighbors(i,j,a):
return [a[i][j-1], a[i][(j+1)%len(a[0])], a[i-1][j], a[(i+1)%len(a)][j]]
[[np.mean(neighbors(i,j,a)) for j in range(len(a[0]))] for i in range(len(a))]
This code works well for 3x3 or small range of matrix, but for large matrix like 2k x 2k this is not feasible. Also this does not work if any of the value in matrix is missing or it's like na
This code works well for 3x3 or small range of matrix, but for large matrix like 2k x 2k this is not feasible. Also this does not work if any of the value in matrix is missing or it's like na. If any of the neighbor values is na then skip that neighbour in getting the average

Shot #1
This assumes you are looking to get sliding windowed average values in an input array with a window of 3 x 3 and considering only the north-west-east-south neighborhood elements.
For such a case, signal.convolve2d with an appropriate kernel could be used. At the end, you need to divide those summations by the number of ones in kernel, i.e. kernel.sum() as only those contributed to the summations. Here's the implementation -
import numpy as np
from scipy import signal
# Inputs
a = [[1,2,3],[3,4,5],[5,6,7],[4,8,9]]
# Convert to numpy array
arr = np.asarray(a,float)
# Define kernel for convolution
kernel = np.array([[0,1,0],
[1,0,1],
[0,1,0]])
# Perform 2D convolution with input data and kernel
out = signal.convolve2d(arr, kernel, boundary='wrap', mode='same')/kernel.sum()
Shot #2
This makes the same assumptions as in shot #1, except that we are looking to find average values in a neighborhood of only zero elements with the intention to replace them with those average values.
Approach #1: Here's one way to do it using a manual selective convolution approach -
import numpy as np
# Convert to numpy array
arr = np.asarray(a,float)
# Pad around the input array to take care of boundary conditions
arr_pad = np.lib.pad(arr, (1,1), 'wrap')
R,C = np.where(arr==0) # Row, column indices for zero elements in input array
N = arr_pad.shape[1] # Number of rows in input array
offset = np.array([-N, -1, 1, N])
idx = np.ravel_multi_index((R+1,C+1),arr_pad.shape)[:,None] + offset
arr_out = arr.copy()
arr_out[R,C] = arr_pad.ravel()[idx].sum(1)/4
Sample input, output -
In [587]: arr
Out[587]:
array([[ 4., 0., 3., 3., 3., 1., 3.],
[ 2., 4., 0., 0., 4., 2., 1.],
[ 0., 1., 1., 0., 1., 4., 3.],
[ 0., 3., 0., 2., 3., 0., 1.]])
In [588]: arr_out
Out[588]:
array([[ 4. , 3.5 , 3. , 3. , 3. , 1. , 3. ],
[ 2. , 4. , 2. , 1.75, 4. , 2. , 1. ],
[ 1.5 , 1. , 1. , 1. , 1. , 4. , 3. ],
[ 2. , 3. , 2.25, 2. , 3. , 2.25, 1. ]])
To take care of the boundary conditions, there are other options for padding. Look at numpy.pad for more info.
Approach #2: This would be a modified version of convolution based approach listed earlier in Shot #1. This is same as that earlier approach, except that at the end, we selectively replace
the zero elements with the convolution output. Here's the code -
import numpy as np
from scipy import signal
# Inputs
a = [[1,2,3],[3,4,5],[5,6,7],[4,8,9]]
# Convert to numpy array
arr = np.asarray(a,float)
# Define kernel for convolution
kernel = np.array([[0,1,0],
[1,0,1],
[0,1,0]])
# Perform 2D convolution with input data and kernel
conv_out = signal.convolve2d(arr, kernel, boundary='wrap', mode='same')/kernel.sum()
# Initialize output array as a copy of input array
arr_out = arr.copy()
# Setup a mask of zero elements in input array and
# replace those in output array with the convolution output
mask = arr==0
arr_out[mask] = conv_out[mask]
Remarks: Approach #1 would be the preferred way when you have fewer number of zero elements in input array, otherwise go with Approach #2.

This is an appendix to comments under #Divakar's answer (rather than an independent answer).
Out of curiosity I tried different 'pseudo' convolutions against the scipy convolution. The fastest one was the % (modulus) wrapping one, which surprised me: obviously numpy does something clever with its indexing, though obviously not having to pad will save time.
fn3 -> 9.5ms, fn1 -> 21ms, fn2 -> 232ms
import timeit
setup = """
import numpy as np
from scipy import signal
N = 1000
M = 750
P = 5 # i.e. small number -> bigger proportion of zeros
a = np.random.randint(0, P, M * N).reshape(M, N)
arr = np.asarray(a,float)"""
fn1 = """
arr_pad = np.lib.pad(arr, (1,1), 'wrap')
R,C = np.where(arr==0)
N = arr_pad.shape[1]
offset = np.array([-N, -1, 1, N])
idx = np.ravel_multi_index((R+1,C+1),arr_pad.shape)[:,None] + offset
arr[R,C] = arr_pad.ravel()[idx].sum(1)/4"""
fn2 = """
kernel = np.array([[0,1,0],
[1,0,1],
[0,1,0]])
conv_out = signal.convolve2d(arr, kernel, boundary='wrap', mode='same')/kernel.sum()
mask = arr == 0.0
arr[mask] = conv_out[mask]"""
fn3 = """
R,C = np.where(arr == 0.0)
arr[R, C] = (arr[(R-1)%M,C] + arr[R,(C-1)%N] + arr[R,(C+1)%N] + arr[(R+1)%M,C]) / 4.0
"""
print(timeit.timeit(fn1, setup, number = 100))
print(timeit.timeit(fn2, setup, number = 100))
print(timeit.timeit(fn3, setup, number = 100))

Using numpy and scipy.ndimage, you can apply a "footprint" that defines where you look for the neighbours of each element and apply a function to those neighbours:
import numpy as np
import scipy.ndimage as ndimage
# Getting neighbours horizontally and vertically,
# not diagonally
footprint = np.array([[0,1,0],
[1,0,1],
[0,1,0]])
a = [[1,2,3],[3,4,5],[5,6,7]]
# Need to make sure that dtype is float or the
# mean won't be calculated correctly
a_array = np.array(a, dtype=float)
# Can specify that you want neighbour selection to
# wrap around at the borders
ndimage.generic_filter(a_array, np.mean,
footprint=footprint, mode='wrap')
Out[36]:
array([[ 3.25, 3.5 , 3.75],
[ 3.75, 4. , 4.25],
[ 4.25, 4.5 , 4.75]])

Finding the max and min in a tuple of tuples

I'm new to python and having some problems finding the minimum and maximum values for a tuple of tuples. I need them to normalise my data. So, basically, I have a list that is a row of 13 numbers, each representing something. Each number makes a column in a list, and I need the max and min for each column. I tried indexing/iterating through but keep getting an error of
max_j = max(j)
TypeError: 'float' object is not iterable
any help would be appreciated!
The code is (assuming data_set_tup is a tuple of tuples, eg ((1,3,4,5,6,7,...),(5,6,7,3,6,73,2...)...(3,4,5,6,3,2,2...)) I also want to make a new list using the normalised values.
normal_list = []
for i in data_set_tup:
for j in i[1:]: # first column doesn't need to be normalised
max_j = max(j)
min_j = min(j)
normal_j = (j-min_j)/(max_j-min_j)
normal_list.append(normal_j)
normal_tup = tuple(normal_list)

You can transpose rows to columns and vice versa with zip(*...). (Use list(zip(*...)) in Python 3)
cols = zip(*data_set_tup)
normal_cols = [cols[0]] # first column doesn't need to be normalised
for j in cols[1:]:
max_j = max(j)
min_j = min(j)
normal_cols.append(tuple((k-min_j)/(max_j-min_j) for k in j)
normal_list = zip(*normal_cols)

This really sounds like a job for the non-builtin numpy module, or maybe the pandas module, depending on your needs.
Adding an extra dependency on your application should not be done lightly, but if you do a lot of work on matrix-like data, then your code will likely be both faster and more readable if you use one of the above modules throughout your application.
I do not recommend converting a list of lists to a numpy array and back again just to get this single result -- it's better to use the pure python method of Jannes answer. Also, seeing that you're a python beginner, numpy may be overkill right now. But I think your question deserves an answer pointing out that this is an option.
Here's a step-by-step console illustration of how this would work in numpy:
>>> import numpy as np
>>> a = np.array([[1,3,4,5,6],[5,6,7,3,6],[3,4,5,6,3]], dtype=float)
>>> a
array([[ 1., 3., 4., 5., 6.],
[ 5., 6., 7., 3., 6.],
[ 3., 4., 5., 6., 3.]])
>>> min = np.min(a, axis=0)
>>> min
array([1, 3, 4, 3, 3])
>>> max = np.max(a, axis=0)
>>> max
array([5, 6, 7, 6, 6])
>>> normalized = (a - min) / (max - min)
>>> normalized
array([[ 0. , 0. , 0. , 0.66666667, 1. ],
[ 1. , 1. , 1. , 0. , 1. ],
[ 0.5 , 0.33333333, 0.33333333, 1. , 0. ]])
So in actual code:
import numpy as np
def normalize_by_column(a):
min = np.min(a, axis=0)
max = np.max(a, axis=0)
return (a - min) / (max - min)

We have nested_tuple = ((1, 2, 3), (4, 5, 6), (7, 8, 9)).
First of all we need to normalize it. Pythonic way:
flat_tuple = [x for row in nested_tuple for x in row]
Output: [1, 2, 3, 4, 5, 6, 7, 8, 9] # it's a list
Move it to tuple: tuple(flat_tuple), get max value: max(flat_tuple), get min value: min(flat_tuple)