Develop Reference

Python loop to populate a matrix

I'm trying to populate an array in python more efficiently. I have a 5x3 matrix A that I am transforming into a 3x3 matrix (Z) by calculating z11, z12, ..., z33 independently. The code below works, but it's clunky and I'm hoping to automate this into a loop so that it will take an A matrix of any size (n x m) and transform it into a Z matrix of size (m x m). If someone could help me out I would greatly appreciate it!
import numpy as np
A = np.array([[1,0,0],
[0,1,0],
[0,1,1],
[0,0,-1],
[0,0,1]])
A1=A[:,0]
A2=A[:,1]
A3=A[:,2]
C = np.array([-2,-2, -9,-6,-4])
X = np.array([-4,-4,-8])
z11 = (sum(A1*A1))*(C[0]/X[0])
z12 = (sum(A1*A2))*(C[0]/X[1])
z13 = (sum(A1*A3))*(C[0]/X[2])
z21 = (sum(A2*A1))*(C[1]/X[0])
z22 = (sum(A2*A2))*(C[1]/X[1])
z23 = (sum(A2*A3))*(C[1]/X[2])
z31 = (sum(A3*A1))*(C[2]/X[0])
z32 = (sum(A3*A2))*(C[2]/X[1])
z33 = (sum(A3*A3))*(C[2]/X[2])
Z = np.array([[z11,z12,z13],
[z21,z22,z23],
[z31,z32,z33]])

We can use the broadcasting to achieve the same. First let's increase A by one dimension using A[:, None] and then multiply it with A. Since shape of A[:, None] is (3, 1, 5) and shape of A is (3, 5), numpy first repeats(intuitively) the array corresponding to dimension where both array don't match and then does the multiplication. This way each column of A gets multiplied with every other column(to makes sure that columns are multiplied, I have used transpose) Then we can take sum along the last axis and multiply with C[:, None] to achieve the desired output.
Use:
m = A.shape[1]
B = A[:, None].T * A.T
Z = np.sum(B, axis = -1).astype(float)*C[:m, None]/X
Output:
>>> Z
array([[0.5 , 0. , 0. ],
[0. , 1. , 0.25 ],
[0. , 2.25 , 3.375]])

Mahalanabois distance in python returns matrix instead of distance

This should be a simple question, either I am missing information, or I have mis-coded this.
I am trying to implement Mahalanabois distance in python which I am following from the formula in python.
My code is as follows:
a = np.array([[1, 3, 5]])
b = np.array([[4, 5, 6]])
X = np.empty((0,3), float)
X = np.vstack([X, [2,3,4]])
X = np.vstack([X, a])
X = np.vstack([X, b])
n = ((a-b).T)*(np.cov(X)**-1)*(a-b)
dist = np.sqrt(n)
dist returns a 3x3 array but should I not be expecting a single number representing the distance?
dist = array([[ 1.5 , 1.73205081, 1.22474487],
[ 1.73205081 , 2. , 1.41421356],
[ 1.22474487 , 1.41421356, 1. ]])
Wikipedia does not suggest (to me) that it should return a matrix. Googling implementations of mahalanbois distance in python I have not found something to compare it to.

From wiki page you could see, that a and b are vectors but in your case they are arrays. So you need reverse transposing. And also there should be matrix multiplication. In numpy * means element-wise multiplication, for matrix you should use np.dot function or .dot method of the np.array. For your case answer is:
n = (a-b).dot((np.cov(X)**-1).dot((a-b).T))
dist = np.sqrt(n)
In [54]: n
Out[54]: array([[ 25.]])
In [55]: dist
Out[55]: array([[ 5.]])
EDIT
As #roadrunner66 noticed you should use inverse matrix instead of inverse matrix of element. Usually np.linalg.inv works for that cases but for that you've got Singular Error and you need to use np.linalg.pinv:
n = (a-b).dot((np.linalg.pinv(np.cov(X))).dot((a-b).T))
dist = np.sqrt(n)
In [90]: n
Out[90]: array([[ 1.77777778]])
In [91]: dist
Out[91]: array([[ 1.33333333]])

Complicated numpy array multiplications

I have two arrays:
a = [[a11,a12],
[a21,a22]]
b = [[b11,b12],
[b21,b22]]
What I would like to do is build up a matrix as follows:
xx = np.mean(a[:,0]*b[:,0])
xy = np.mean(a[:,0]*b[:,1])
yx = np.mean(a[:,1]*b[:,0])
yy = np.mean(a[:,1]*b[:,1])
and return an array c such that
c = [[xx,xy],
yx,yy]]
Is there a nice pythonic way to do this in numpy? Because at the moment I have done it by hand, exactly as above, so the dimensions of the output array are coded in by hand, rather than determined as according to the size of the input arrays a and b.

Is there an error in your third element? If, as seems reasonable, you want yx = np.mean(a[:,1]*b[:,0]) instead of yx = np.mean(b[:,1]*a[:,0]), then you can try the following:
a = np.random.rand(2, 2)
b = np.random.rand(2, 2)
>>> c
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])
>>> np.mean(a.T[:, None, :]*b.T, axis=-1)
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])
It will actually be faster to avoid the intermediate array and express your result as a matrix multiplication:
>>> np.dot(a.T, b) / a.shape[0]
array([[ 0.26951488, 0.19019219],
[ 0.31008754, 0.1793523 ]])

Efficiently generating a Cauchy matrix from two Numpy arrays

A Cauchy matrix (Wikipedia article) is a matrix determined by two vectors (arrays of numbers). Given two vectors x and y, the Cauchy matrix C generated by them is defined entry-wise as
C[i][j] := 1/(x[i] - y[j])
Given two Numpy arrays x and y, what is an efficient way to generate a Cauchy matrix?

This is the most efficient way I found, using array broadcasting to take advantage of vectorization.
1.0 / (x.reshape((-1,1)) - y)
Edit: #HYRY and #shx2 have suggested that, instead of x.reshape((-1,1)), which makes a copy, you can use x[:,np.newaxis], which returns a view of the same array. #HYRY also suggests 1.0/np.subtract.outer(x,y), which is slightly slower for me but maybe more explicit.
Example:
>>> x = numpy.array([1,2,3,4]) #x
>>> y = numpy.array([5,6,7]) #y
>>>
>>> #transpose x, to nx1
... x = x.reshape((-1,1))
>>> x
array([[1],
[2],
[3],
[4]])
>>>
>>> #array of differences x[i] - y[j]
... #an nx1 array minus a 1xm array is an nxm array
... diff_matrix = x-y
>>> diff_matrix
array([[-4, -5, -6],
[-3, -4, -5],
[-2, -3, -4],
[-1, -2, -3]])
>>>
>>> #apply the multiplicative inverse to each entry
... cauchym = 1.0/diff_matrix
>>> cauchym
array([[-0.25 , -0.2 , -0.16666667],
[-0.33333333, -0.25 , -0.2 ],
[-0.5 , -0.33333333, -0.25 ],
[-1. , -0.5 , -0.33333333]])
I tried a few other methods, all of which were significantly slower.
This is the naive approach, which costs list comprehension:
cauchym = numpy.array([[ 1.0/(x_i-y_j) for y_j in y] for x_i in x])
This one generates the matrix as a 1-dimensional array (saving the cost of nested Python lists) and reshapes it to a matrix afterward. It also moves the division to a single Numpy operation:
cauchym = 1.0/numpy.array([(x_i-y_j) for x_i in x for y_j in y]).reshape([len(x),len(y)])
Using numpy.repeat and numpy.tile (which respectively tile the array horizontally and vertically). This way makes unnecessary copies:
lenx = len(x)
leny = len(y)
xm = numpy.repeat(x,leny) #the i'th row is s_i
ym = numpy.tile(y,lenx)
cauchym = (1.0/(xm-ym)).reshape([lenx,leny]);

I created a function hope it helps u to understand in a better way.
# Creating a function in order to form a cauchy matrix
def cauchy_matrix(arr1,arr2):
"""
Enter two arrays in order to get a cauchy matrix.The input array should be a 1-D array.
arr1 = First 1-D array
arr2 = Second 1-D array
It returns the cauchy matrix having shape equal to m*n, where m is size of arr1 and n is size of arr2.
"""
my_list = []
try:
for i in range(len(arr1)):
for j in range(len(arr2)):
z = 1/(arr1[i]-arr2[j])
my_list.append(z)
return np.array(my_list).reshape(arr1.shape[0],arr2.shape[0])
except ZeroDivisionError:
print("Check if both the arrays has '0' as one of it's element. One array can have a zero but both the arrays having '0' is not acceptable!")

How to normalize a 2-dimensional numpy array in python less verbose?

Given a 3 times 3 numpy array
a = numpy.arange(0,27,3).reshape(3,3)
# array([[ 0, 3, 6],
# [ 9, 12, 15],
# [18, 21, 24]])
To normalize the rows of the 2-dimensional array I thought of
row_sums = a.sum(axis=1) # array([ 9, 36, 63])
new_matrix = numpy.zeros((3,3))
for i, (row, row_sum) in enumerate(zip(a, row_sums)):
new_matrix[i,:] = row / row_sum
There must be a better way, isn't there?
Perhaps to clearify: By normalizing I mean, the sum of the entrys per row must be one. But I think that will be clear to most people.

Broadcasting is really good for this:
row_sums = a.sum(axis=1)
new_matrix = a / row_sums[:, numpy.newaxis]
row_sums[:, numpy.newaxis] reshapes row_sums from being (3,) to being (3, 1). When you do a / b, a and b are broadcast against each other.
You can learn more about broadcasting here or even better here.

Scikit-learn offers a function normalize() that lets you apply various normalizations. The "make it sum to 1" is called L1-norm. Therefore:
from sklearn.preprocessing import normalize
matrix = numpy.arange(0,27,3).reshape(3,3).astype(numpy.float64)
# array([[ 0., 3., 6.],
# [ 9., 12., 15.],
# [ 18., 21., 24.]])
normed_matrix = normalize(matrix, axis=1, norm='l1')
# [[ 0. 0.33333333 0.66666667]
# [ 0.25 0.33333333 0.41666667]
# [ 0.28571429 0.33333333 0.38095238]]
Now your rows will sum to 1.

I think this should work,
a = numpy.arange(0,27.,3).reshape(3,3)
a /= a.sum(axis=1)[:,numpy.newaxis]

In case you are trying to normalize each row such that its magnitude is one (i.e. a row's unit length is one or the sum of the square of each element in a row is one):
import numpy as np
a = np.arange(0,27,3).reshape(3,3)
result = a / np.linalg.norm(a, axis=-1)[:, np.newaxis]
# array([[ 0. , 0.4472136 , 0.89442719],
# [ 0.42426407, 0.56568542, 0.70710678],
# [ 0.49153915, 0.57346234, 0.65538554]])
Verifying:
np.sum( result**2, axis=-1 )
# array([ 1., 1., 1.])

I think you can normalize the row elements sum to 1 by this:
new_matrix = a / a.sum(axis=1, keepdims=1).
And the column normalization can be done with new_matrix = a / a.sum(axis=0, keepdims=1). Hope this can hep.

You could use built-in numpy function:
np.linalg.norm(a, axis = 1, keepdims = True)

it appears that this also works
def normalizeRows(M):
row_sums = M.sum(axis=1)
return M / row_sums

You could also use matrix transposition:
(a.T / row_sums).T

Here is one more possible way using reshape:
a_norm = (a/a.sum(axis=1).reshape(-1,1)).round(3)
print(a_norm)
Or using None works too:
a_norm = (a/a.sum(axis=1)[:,None]).round(3)
print(a_norm)
Output:
array([[0. , 0.333, 0.667],
[0.25 , 0.333, 0.417],
[0.286, 0.333, 0.381]])

Use
a = a / np.linalg.norm(a, ord = 2, axis = 0, keepdims = True)
Due to the broadcasting, it will work as intended.

Or using lambda function, like
>>> vec = np.arange(0,27,3).reshape(3,3)
>>> import numpy as np
>>> norm_vec = map(lambda row: row/np.linalg.norm(row), vec)
each vector of vec will have a unit norm.

We can achieve the same effect by premultiplying with the diagonal matrix whose main diagonal is the reciprocal of the row sums.
A = np.diag(A.sum(1)**-1) # A