Develop Reference

Sort array with repeated values

I have to order an array with values from 0 to 9 that are repeated and obtain the vector initial index. The input array is:
[3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4]
I would like to obtain the following order:
array([0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9], dtype=uint8)
Instead of:
array([0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 9, 9])
which is given by:
import numpy as np
a = [3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4]
np.argsort(a)
Is there a way to manipulate this function?

l = [1,2,3,4,5,6,7,8,9,4,3,5]
l_oredered = []
while len(l) != 0:
unique_nums = list(set(l))
unique_nums.sort()
l_oredered.extend(unique_nums)
for num in unique_nums:
l.remove(num)
print(l_oredered)
This will result with:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 3, 4, 5]
You can apply the thinking with NumPy or convert the final result into a NumPy array.

For an array with a small number of elements #Mr.O's answer is faster. The code below is faster if there are more than around 100 ints in arr.
import numpy as np
def sort_groups( arr ):
ct = np.ones( len(arr), dtype = np.int64 )
for i in set( arr ):
ct[arr == i] = ct[arr == i ].cumsum()
# ct calculates a rank for each int in arr
tosort = ( arr.max() + 1 ) * ct + arr
# tosort ranks by ct first then a if ct's are equal
return arr[ np.argsort( tosort ) ]
a = np.array([3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4])
sort_groups( a )
# array([0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9])
Breaking the function out to see what's happening:
arr = a
ct = np.ones( len(arr), dtype = np.int64 )
for i in set( arr ):
ct[arr == i] = ct[arr == i ].cumsum()
arr, ct
# (array([3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4]),
# array([1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 1, 2, 2, 2, 2, 2, 2, 2]))
tosort = ( arr.max() + 1 ) * ct + arr # Assumes arr is > 0
tosort
# array([13, 11, 12, 10, 14, 15, 16, 17, 21, 20, 19, 25, 23, 29, 22, 27, 26, 24])
arr[ np.argsort( tosort ) ]
array([0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9])

Very interesting task! Here is my attempt at to solve the problem
import numpy as np
def groupsort(a: np.ndarray):
uniques, counts = np.unique(a, return_counts=True)
min_count = np.min(counts)
counts -= min_count
n_easy = min_count * len(uniques)
# Pre allocate array
values = np.empty(n_easy + counts.sum(), dtype=a.dtype)
# Set easy values
temp = values[:n_easy].reshape(min_count, len(uniques))
temp[:] = uniques
# Set hard values
i = n_easy
while np.any(mask := counts > 0): # Python 3.8 syntax
masksum = mask.sum()
values[i : i + masksum] = uniques[mask]
counts -= mask
i += masksum
return values
a = np.array(list(range(4)) * 2 + [0, 1, 2, 0, 1, 2, 0, 1, 1, 1])
np.random.shuffle(a)
print(a)
# [3 0 1 0 1 0 2 1 2 1 0 2 1 1 2 0 3 1]
print(groupsort(a))
# [0 1 2 3 0 1 2 3 0 1 2 0 1 2 0 1 1 1]
# Your input
a = np.array([3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4])
print(groupsort(a))
# [0 1 2 3 4 5 6 7 9 0 1 2 3 4 5 6 7 9]
The idea is to split the problem in two cases. One easy case and one hard case. The easy case is to handle inputs like this: a = [0,1,2,3,0,1,2,3], where the counts for each unique value are equal. Then you can simply count the number n of a specific value (e.g. 0), then just do list(range(max(a))) * n.
The hard case is to handle inputs such as a = [1,1,1,1,1,0,0,0,2,2]. Then the idea is to get the counts of each value, in this case counts = [3,5,2,0]. Then do:
values = np.empty(counts.sum())
i = 0
while np.any(mask := counts > 0): # Python 3.8 syntax
masksum = mask.sum()
values[i : i + masksum] = uniques[mask]
counts -= mask
i += masksum
In my solution you see I have combined the two solutions to optimize for speed. Assuming np.unique has a linear average time complexity, then this algorithm also has a linear average runtime complexity.

Here's a 2-liner:
unique, counts = np.unique(a, return_counts=True)
b = [x for y in [[u for i, u in enumerate(unique) if counts[i] > n] for n in range(counts.max())] for x in y]
Output:
>>> b
[0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9, 1, 5, 9]
#^ reset ^ reset ^ reset

I prefer using np.bincount instead of np.unique, np.sort or np.argsort because it's much faster in cases maximum item of data is small.
def count_out(arr, N):
bins = np.bincount(arr, minlength=N)
threshold_idx = np.unique(bins[bins!=0])
counts = np.diff(threshold_idx, prepend=0)
mask = (bins >= threshold_idx[:, None])
full_mask = np.repeat(mask, counts, axis=0)
blocks = np.repeat([np.arange(N)], np.sum(counts), axis=0)
return blocks[full_mask]
N = 10
X = np.array([3, 5, 3, 9, 9, 9, 9, 0, 0, 6, 8, 8, 7, 0, 5, 9, 7, 8, 1, 5, 8, 8, 1, 0, 7, 1, 9])
print(X)
print(count_out(X, N))
>>> [3 5 1 3 9 7 5 9 0 9 9 0 0 6 8 8 8 9 7 0 5 9 7 8 3 1 5 8 8 1 0 7 1 9 9 8]
>>> [0 1 3 5 6 7 8 9 0 1 3 5 7 8 9 0 1 3 5 7 8 9 0 1 5 7 8 9 0 8 9 8 9 8 9 9]
The key idea is to find counts of how many times does each block repeat. Then create unique mask for each block:
Blocks:
[[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]]
Unique masks:
[[1 1 0 1 0 1 1 1 1 1]
[1 1 0 1 0 1 0 1 1 1]
[1 1 0 0 0 1 0 1 1 1]
[1 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 0 0 0 0 1]]
Finally, reconstruct all the masks by the counts we've got.
Counts: [1 2 1 1 2 1]
Full masks:
[[1 1 0 1 0 1 1 1 1 1]
[1 1 0 1 0 1 0 1 1 1]
[1 1 0 1 0 1 0 1 1 1]
[1 1 0 0 0 1 0 1 1 1]
[1 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 0 0 0 1 1]
[0 0 0 0 0 0 0 0 0 1]]
By the way, it seems this can be optimised further. At first, creation of repetitive blocks is redundant since there should be a way to create a pointer to one single block. Secondly, it's slow in case full mask is sparse. In this case you should consider implementing your own way to repeat blocks with no masking.
I hope it's helpful for you at the current point.

IIUC, you want to have a duplicated, sorted, array.
Remove the duplicated values using numpy.unique, sort, and tile to the expected size:
a = np.array([3, 1, 2, 0, 4, 5, 6, 7, 1, 0, 9, 5, 3, 9, 2, 7, 6, 4])
b = np.unique(a)
b = np.tile(b, len(a)//len(b))
output:
array([0, 1, 2, 3, 4, 5, 6, 7, 9, 0, 1, 2, 3, 4, 5, 6, 7, 9])

Multidimensional cumulative sum in numpy

I want to be able to calculate the cumulative sum of a large n-dimensional numpy array. The value of each element in the final array should be the sum of all elements which have indices greater than or equal to the current element.
2D: xᶦʲ = ∑xᵐⁿ ∀ m ≥ i and n ≥ j
3D: xᶦʲᵏ = ∑xᵐⁿᵒ ∀ m ≥ i and n ≥ j and o ≥ k
Examples in 2D:
1 1 0 2 1 0
1 1 1 -> 5 3 1
1 1 1 8 5 2
1 2 3 6 5 3
4 5 6 -> 21 16 9
7 8 9 45 33 18
Example in 3D:
1 1 1 3 2 1
1 1 1 6 4 2
1 1 1 9 6 3
1 1 1 6 4 2
1 1 1 -> 12 8 4
1 1 1 18 12 6
1 1 1 9 6 3
1 1 1 18 12 6
1 1 1 27 18 9

Flip along the last axis, cumsum along the same, flip it back and finally cumsum along the second last axis onwards until the first axis -
def multidim_cumsum(a):
out = a[...,::-1].cumsum(-1)[...,::-1]
for i in range(2,a.ndim+1):
np.cumsum(out, axis=-i, out=out)
return out
Sample 2D case run -
In [107]: a
Out[107]:
array([[1, 1, 0],
[1, 1, 1],
[1, 1, 1]])
In [108]: multidim_cumsum(a)
Out[108]:
array([[2, 1, 0],
[5, 3, 1],
[8, 5, 2]])
Sample 3D case run -
In [110]: a
Out[110]:
array([[[1, 1, 1],
[1, 1, 1],
[1, 1, 1]],
[[1, 1, 1],
[1, 1, 1],
[1, 1, 1]],
[[1, 1, 1],
[1, 1, 1],
[1, 1, 1]]])
In [111]: multidim_cumsum(a)
Out[111]:
array([[[ 3, 2, 1],
[ 6, 4, 2],
[ 9, 6, 3]],
[[ 6, 4, 2],
[12, 8, 4],
[18, 12, 6]],
[[ 9, 6, 3],
[18, 12, 6],
[27, 18, 9]]])

For those who want a "numpy-like" cumsum where the top-left corner is smallest:
def multidim_cumsum(a):
out = a.cumsum(-1)
for i in range(2,a.ndim+1):
np.cumsum(out, axis=-i, out=out)
return out
Modified from #Divakar (thanks to him!)

Here is a general solution. I'm going by the description, not the examples, i.e. order of vertical display is top down not bottom up:
import itertools as it
import functools as ft
ft.reduce(np.cumsum, it.chain((a[a.ndim*(np.s_[::-1],)],), range(a.ndim)))[a.ndim*(np.s_[::-1],)]
Or in-place:
for i in range(a.ndim):
b = a.swapaxes(0, i)[::-1]
b.cumsum(axis=0, out=b)

numpy.tile did not work as Matlab repmat

According to What is the equivalent of MATLAB's repmat in NumPy, I tried to build 3x3x5 array from 3x3 array using python.
In Matlab, this work as I expected.
a = [1,1,1;1,2,1;1,1,1];
a_= repmat(a,[1,1,5]);
size(a_) = 3 3 5
But for numpy.tile
b = numpy.array([[1,1,1],[1,2,1],[1,1,1]])
b_ = numpy.tile(b, [1,1,5])
b_.shape = (1, 3, 15)
If I want to generate the same array as in Matlab, what is the equivalent?
Edit 1
The output I would expect to get is
b_(:,:,1) =
1 1 1
1 2 1
1 1 1
b_(:,:,2) =
1 1 1
1 2 1
1 1 1
b_(:,:,3) =
1 1 1
1 2 1
1 1 1
b_(:,:,4) =
1 1 1
1 2 1
1 1 1
b_(:,:,5) =
1 1 1
1 2 1
1 1 1
but what #farenorth and the numpy.dstack give is
[[[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]]
[[1 1 1 1 1]
[2 2 2 2 2]
[1 1 1 1 1]]
[[1 1 1 1 1]
[1 1 1 1 1]
[1 1 1 1 1]]]

NumPy functions are not, in general, 'drop-in' replacements for matlab functions. Often times there are subtle difference to how the 'equivalent' functions are used. It does take time to adapt, but I've found the transition to be very worthwhile.
In this case, the np.tile documentation indicates what happens when you are trying to tile an array to higher dimensions than it is defined,
numpy.tile(A, reps)
Construct an array by repeating A the number of times given by reps.
If reps has length d, the result will have dimension of max(d, A.ndim).
If A.ndim < d, A is promoted to be d-dimensional by prepending new axes. So a shape (3,) array is promoted to (1, 3) for 2-D replication, or shape (1, 1, 3) for 3-D replication. If this is not the desired behavior, promote A to d-dimensions manually before calling this function.
In this case, your array is being cast to a shape of [1, 3, 3], then being tiled. So, to get your desired behavior just be sure to append a new singleton-dimension to the array where you want it,
>>> b_ = numpy.tile(b[..., None], [1, 1, 5])
>>> print(b_.shape)
(3, 3, 5)
Note here that I've used None (i.e. np.newaxis) and ellipses notation to specify a new dimension at the end of the array. You can find out more about these capabilities here.
Another option, which is inspired by the OP's comment would be:
b_ = np.dstack((b, ) * 5)
In this case, I've used tuple multiplication to 'repmat' the array, which is then constructed by np.dstack.
As #hpaulj indicated, Matlab and NumPy display matrices differently. To replicate the Matlab output you can do something like:
>>> for idx in xrange(b_.shape[2]):
... print 'b_[:, :, {}] = \n{}\n'.format(idx, str(b_[:, :, idx]))
...
b_[:, :, 0] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 1] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 2] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 3] =
[[1 1 1]
[1 2 1]
[1 1 1]]
b_[:, :, 4] =
[[1 1 1]
[1 2 1]
[1 1 1]]
Good luck!

Let's try the comparison, taking care to diversify the shapes and values.
octave:7> a=reshape(0:11,3,4)
a =
0 3 6 9
1 4 7 10
2 5 8 11
octave:8> repmat(a,[1,1,2])
ans =
ans(:,:,1) =
0 3 6 9
1 4 7 10
2 5 8 11
ans(:,:,2) =
0 3 6 9
1 4 7 10
2 5 8 11
numpy equivalent - more or less:
In [61]: a=np.arange(12).reshape(3,4)
In [62]: np.tile(a,[2,1,1])
Out[62]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]])
numpy again, but with order F to better match the MATLAB Fortran-derived layout
In [63]: a=np.arange(12).reshape(3,4,order='F')
In [64]: np.tile(a,[2,1,1])
Out[64]:
array([[[ 0, 3, 6, 9],
[ 1, 4, 7, 10],
[ 2, 5, 8, 11]],
[[ 0, 3, 6, 9],
[ 1, 4, 7, 10],
[ 2, 5, 8, 11]]])
I'm adding the new numpy dimension at the start, because in many ways it better replicates the MATLAB practice of adding it at the end.
Try adding the new dimension at the end. The shape is (3,4,5), but you might not like the display.
np.tile(a[:,:,None],[1,1,2])
Another consideration - what happens when you flatten the tile?
octave:10> repmat(a,[1,1,2])(:).'
ans =
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
with the order F a
In [78]: np.tile(a[:,:,None],[1,1,2]).flatten()
Out[78]:
array([ 0, 0, 3, 3, 6, 6, 9, 9, 1, 1, 4, 4, 7, 7, 10, 10, 2,
2, 5, 5, 8, 8, 11, 11])
In [79]: np.tile(a,[2,1,1]).flatten()
Out[79]:
array([ 0, 3, 6, 9, 1, 4, 7, 10, 2, 5, 8, 11, 0, 3, 6, 9, 1,
4, 7, 10, 2, 5, 8, 11])
with a C order array:
In [80]: a=np.arange(12).reshape(3,4)
In [81]: np.tile(a,[2,1,1]).flatten()
Out[81]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11])
This last one matches the Octave layout.
So does:
In [83]: a=np.arange(12).reshape(3,4,order='F')
In [84]: np.tile(a[:,:,None],[1,1,2]).flatten(order='F')
Out[84]:
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 0, 1, 2, 3, 4,
5, 6, 7, 8, 9, 10, 11])
Confused yet?

getting mean of value in matrix changes matrix

I am getting mean of ambiguate elements in matrix
import pandas as pd
ds2 = [[ 4, 1],
[ 5, 3],
[ 6, 1],
[ 7, 2],
[ 7, 2],
[ 8, 2],
[12, 1],
[ 9, 3],
[12, 1],
[13, 2],
[22, 3]]
ds2= pd.DataFrame(ds2)
print type(ds2)
print ds2
ds2 = ds2.groupby(0).mean()
print type(ds2)
print ds2
output:
<class 'pandas.core.frame.DataFrame'>
0 1
0 4 1
1 5 3
2 6 1
3 7 2
4 8 2
5 9 3
6 12 1
7 13 2
8 22 3
<class 'pandas.core.frame.DataFrame'>
1
0
4 1
5 3
6 1
7 2
8 2
9 3
12 1
13 2
22 3
Type remains same, but the way matrix looks change, is there any way to persist matrix view even after processing?

Pass param as_index=False to the groupby method:
In [140]:
ds2 = [[ 4, 1],
[ 5, 3],
[ 6, 1],
[ 7, 2],
[ 7, 2],
[ 8, 2],
[12, 1],
[ 9, 3],
[12, 1],
[13, 2],
[22, 3]]
ds2= pd.DataFrame(ds2)
ds2.groupby(0, as_index=False).mean()
Out[140]:
0 1
0 4 1
1 5 3
2 6 1
3 7 2
4 8 2
5 9 3
6 12 1
7 13 2
8 22 3
By default any columns passed will be used to form the index.
From the docs:
as_index : boolean, default True
For aggregated output, return object
with group labels as the index. Only relevant for DataFrame input.
as_index=False is effectively “SQL-style” grouped output

Mask array based on other array?

I have a numpy array arrayBig
10 5 27 30 34
2 34 23 2 3
2 3 43 12 23
2 24 34 2 34
And I have a numpy array arraySmall
1 0 0
0 1 1
1 1 0
What I want is a numpy array arrayNew
34 0 0
0 43 12
24 34 0
I know my arraySmall has the shape (3,3) and is located at index (1 1) in arrayBig. How can I get arrayNew with Numpy?

>>> import numpy as np
>>> arrayBig = np.array([
... [10, 5, 27, 30, 34],
... [2, 34, 23, 2, 3],
... [2, 3, 43, 12, 23],
... [2, 24, 34, 2, 34],
... ])
>>> arraySmall = np.array([
... [1, 0, 0],
... [0, 1, 1],
... [1, 1, 0],
... ])
>>> arrayBig[1:4, 1:4] * arraySmall
array([[34, 0, 0],
[ 0, 43, 12],
[24, 34, 0]])

I recently learned about advanced boolean indexing. I'm not sure if it's any better than the other answer but you can do:
>>> a = np.array([[1,2,3],[4,5,6]])
>>> b = np.array([[1,0],[0,1]])
>>> c = b == 0
>>> d = a[0:b.shape[0],0:b.shape[1]]
>>> d[c] = 0
>>> d
array([[1, 0],
[0, 5]])