Develop Reference

Repmat operation in python

I want to calculate the mean of a 3D array along two axes and subtract this mean from the array.
In Matlab I use the repmat function to achieve this as follows
% A is an array of size 100x50x100
mean_A = mean(mean(A,3),1); % mean_A is 1D of length 50
Am = repmat(mean_A,[100,1,100]) % Am is 3D 100x50x100
flc_A = A - Am % flc_A is 3D 100x50x100
Now, I am trying to do the same with python.
mean_A = numpy.mean(numpy.mean(A,axis=2),axis=0);
gives me the 1D array. However, I cannot find a way to copy this to form a 3D array using numpy.tile().
Am I missing something or is there another way to do this in python?

You could set keepdims to True in both cases so the resulting shape is broadcastable and use np.broadcast_to to broadcast to the shape of A:
np.broadcast_to(np.mean(np.mean(A,2,keepdims=True),axis=0,keepdims=True), A.shape)
Note that you can also specify a tuple of axes along which to take the successive means:
np.broadcast_to(np.mean(A,axis=tuple([2,0]), keepdims=True), A.shape)

numpy.tile is not the same with Matlab repmat. You could refer to this question. However, there is an easy way to repeat the work you have done in Matlab. And you don't really have to understand how numpy.tile works in Python.
import numpy as np
A = np.random.rand(100, 50, 100)
# keep the dims of the array when calculating mean values
B = np.mean(A, axis=2, keepdims=True)
C = np.mean(B, axis=0, keepdims=True) # now the shape of C is (1, 50, 1)
# then simply duplicate C in the first and the third dimensions
D = np.repeat(C, 100, axis=0)
D = np.repeat(D, 100, axis=2)
D is the 3D array you want.

How to flatten an array to a matrix in Numpy?

I am looking for an elegant way to flatten an array of arbitrary shape to a matrix based on a single parameter that specifies the dimension to retain. For illustration, I would like
def my_func(input, dim):
# code to compute output
return output
Given for example an input array of shape 2x3x4, output should be for dim=0 an array of shape 12x2; for dim=1 an array of shape 8x3; for dim=2 an array of shape 6x8. If I want to flatten the last dimension only, then this is easily accomplished by
input.reshape(-1, input.shape[-1])
But I would like to add the functionality of adding dim (elegantly, without going through all possible cases + checking with if conditions, etc.). It might be possible by first swapping dimensions, so that the dimension of interest is trailing and then applying the operation above.
Any help?

We can permute axes and reshape -
# a is input array; axis is input axis/dim
np.moveaxis(a,axis,-1).reshape(-1,a.shape[axis])
Functionally, it's basically pushing the specified axis to the back and then reshaping keeping that axis length to form the second axis and merging rest of the axes to form the first axis.
Sample runs -
In [32]: a = np.random.rand(2,3,4)
In [33]: axis = 0
In [34]: np.moveaxis(a,axis,-1).reshape(-1,a.shape[axis]).shape
Out[34]: (12, 2)
In [35]: axis = 1
In [36]: np.moveaxis(a,axis,-1).reshape(-1,a.shape[axis]).shape
Out[36]: (8, 3)
In [37]: axis = 2
In [38]: np.moveaxis(a,axis,-1).reshape(-1,a.shape[axis]).shape
Out[38]: (6, 4)

Sum simultaneously 3 dimensions of 4-dimensional matrix

I have 4d matrix, mat4. Instead of using np.sum(mat, axis=) and defining axis 3 times for sum three dimensions, is there a way I can sum many dimensions of matrix, simultaneously?
#Sum `mat4` except `axis=0`
mat4 = np.random.rand(2,3,4,5)
matsum = np.sum(mat4, axis=3)
matsum = np.sum(matsum, axis=2)
matsum = np.sum(matsum, axis=1)
print matsum.shape
>> (2L,)

axis keyword can be either an int or a tuple, so
you can simply use
np.sum(mat, axis=(1, 2, 3))
From np.sum docs:
If axis is a tuple of ints, a sum is performed on all of the axes
specified in the tuple instead of a single axis or all the axes as before.

Using an ND array to select on a dimension

How do I use an ndarray matrix to select elements of an ndarray?
Here's an example of what I mean.
a = np.arange(9)
b = np.arange(5)
c = np.arange(12)
A, B, C = np.meshgrid(a, b, c, indexing='ij')
Now, for each value of a, c, I want the b that minimizes A+C=B. Get the indices:
idx = np.abs(A+C-B).argmin(axis=1)
Clearly, idx has shape (9, 12) - it contains the index of b for each of the 9 a, and each of the 12 c.
Now, I would like to select the matrices with the "optimized b". That is, something along the lines of
B[:, idx, :]
that supposedly has shape (9, 1, 12) - because for each of the other combinations, it has only one value of b - the minimizing one. Now, B[:, idx, :] instead gives me the mesh of all potential combinations with shape (9, 9, 12, 12). I also tried
B[np.arange(B.shape[0]), idx, np.arange(B.shape[2])]
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (9,) (9,12) (12,)
How do I get that specific type of matrix I described above?

You just need to add an axis there with np.newaxis/None to trigger advanced-indexing -
B[np.arange(B.shape[0])[:,None], idx, np.arange(B.shape[2])]
The idea basically is to map the rows of idx with the first indexing array of np.arange(B.shape[0]) and as such we need to add an axis there. For mapping the columns of idx, we already have np.arange(B.shape[2]) aligned along the columns of it.
Alternative to np.newaxis
Another way to add that new axis would be with reshaping Thus, we could replace B[np.arange(B.shape[0])[:,None] with np.arange(B.shape[0]).reshape(-1,1).
Further optimization
We could optimize the codes, by using open arrays to replace the huge arrays created by meshgrid, like so -
A0, B0, C0 = np.ix_(a,b,c)
idx = np.abs(A0+C0-B0).argmin(axis=1)
Thus, get the final output, like so -
B[np.arange(len(a))[:,None], idx, np.arange(len(c))]
Just to give ourselves the idea of memory saving here -
In [47]: A.nbytes + B.nbytes + C.nbytes
Out[47]: 12960
whereas A0, B0, C0 are views into the input arrays a, b, c respectively and as such don't occupy any additional memory, i.e. absolutely free -
In [49]: np.shares_memory(a,A0)
Out[49]: True
For completeness sake, a direct way to get idx would be -
np.abs(a[:,None,None]+c-b[:,None]).argmin(axis=1)

clear authoritative explanation of numpy axis numbers?

I am getting confused by contradictory explanations of what exactly the term axis means in numpy and how these constructs are numbered.
Here's one explanation:
Axes are defined for arrays with more than one dimension.
A 2-dimensional array has two corresponding axes:
the first running vertically downwards across rows (axis 0), and
the second running horizontally across columns (axis 1).
So, in this 3x4 matrix ...
>>> b = np.arange(12).reshape(3,4)
>>> b
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
(axis 0) is the 3 rows
(axis 1) is the 4 columns
So the rule might be ...
In an MxN matrix, (axis 0) is M and (axis 1) is N.
Is this correct?
So, in a 3 dimensional matrix AxBxC
(axis 0) is A
(axis 1) is B
(axis 2) is C
Is this correct?

Everything you said is correct, with the exception of
Axes are defined for arrays with more than one dimension.
Axes are also defined for one dimensional arrays - there is just one of them (i.e. axis 0).
One intuitive way to think about axes is to consider what happens when you apply a reduction operation over one axis, such as summation. For example, suppose I have some array x:
x = np.arange(60).reshape(3, 4, 5)
If I compute x.sum(0) I am "collapsing" x over the first dimension (i.e. axis 0), so I end up with a (4, 5) array. Likewise, x.sum(1) gives me a (3, 5) array and x.sum(2) gives me a (3, 4) array.
An integer index into a single axis of x will also give me an output with one fewer axis. For example, x[0, :, :] gives me the first "row" of x, which has shape (4, 5), x[:, 0, :] gives me the first "column" with shape (3, 5), and x[:, :, 0] gives me the first slice in the third dimension of x with shape (3, 4).

If someone need a clear idea, here is the picture:

A smart way to remember this is that
axis =0 collapses the rows
Whilst
axis=1 collapses the columns
a three 3*4 array when operated upon with sum function and axis =0 would yield 1*4 output that is all the rows would be collapsed and the aggregation would be done column-wise.
The same function when performed with axis=1 would collapse the columns and yield 3*1 output with aggregation along rows.
the image link would further help assimilating this concept.
Example for understanding

Although it is possible to imagine this in 3D, I personally feel it is difficult to imagine when we go to 4D or 5D... So I decide to give up but rather think about this in an implementation perspective. Basically, it has N-number of nested for loop, and if we want to reduce one specific axis, we just work on the for loop of that axis. For example, if given a 3x3x3 tensor, axis = 0 is the for loop of a[i][x][x], axis = 1 is to loop a[x][i][x], axis = 2 is to loop a[x][x][i]. 4D, 5D, ... should have the same way.
def my_reduce_max(a, axis=0):
b = [[-1 for _ in range(3)] for _ in range(3)]
for j in range(3):
for k in range(3):
tmp_max = -1
for i in range(3):
if axis == 0:
get_value = a[i][j][k]
elif axis == 1:
get_value = a[j][i][k]
else:
get_value = a[j][k][i]
tmp_max = max(get_value, tmp_max)
b[j][k] = tmp_max
return b
a = np.arange(27).reshape((3,3,3))
print(a)
my_reduce_max(a, 2)