Develop Reference

python PCA dimensionality reduction

i'm learning using PCA to finish dimensionality reduction (Python3.6) but i've got very similar but different results when using different methods here's my code
from numpy import *
from sklearn.decomposition import PCA
data_set = [[-1., -2.],
[-1., 0.],
[0., 0.],
[2., 1.],
[0., 1.]]
# 1
pca_sk = PCA(n_components=1)
newmat = pca_sk.fit_transform(data_set)
print(newmat)
# 2
meanVals = mean(data_set, axis=0)
meanRemoved = data_set - meanVals
covMat = cov(meanRemoved, rowvar=0)
eigVals, eigVects = linalg.eig(mat(covMat))
eigValInd = argsort(eigVals)
eigValInd = eigValInd[:-(1 + 1):-1]
redEigVects = eigVects[:, eigValInd]
lowDDataMat = meanRemoved * redEigVects
print(lowDDataMat)
the first one output
[[ 2.12132034]
[ 0.70710678]
[-0. ]
[-2.12132034]
[-0.70710678]]
but anothor output
[[-2.12132034]
[-0.70710678]
[ 0. ]
[ 2.12132034]
[ 0.70710678]]
why dose it happen

Back-Propagation of y = x / sum(x, dim=0) where size of tensor x is (H,W)

Q1.
I'm trying to make my custom autograd function with pytorch.
But I had a problem with making analytical back propagation with y = x / sum(x, dim=0)
where size of tensor x is (Height, Width) (x is 2-dimensional).
Here's my code
class MyFunc(torch.autograd.Function):
#staticmethod
def forward(ctx, input):
ctx.save_for_backward(input)
input = input / torch.sum(input, dim=0)
return input
#staticmethod
def backward(ctx, grad_output):
input = ctx.saved_tensors[0]
H, W = input.size()
sum = torch.sum(input, dim=0)
grad_input = grad_output * (1/sum - input*1/sum**2)
return grad_input
I used (torch.autograd import) gradcheck to compare Jacobian matrix,
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.randn(3,3,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)
and the result was
Please someone help me to get correct back propagation result
Thanks!
Q2.
Thanks for answers!
Because of your help, I could implement back propagation in case of (H,W) tensor.
However, while I implemented back propagation in case of (N,H,W) tensor, I got a problem.
I think the problem would be initializing new tensor.
Here's my new code
import torch
import torch.nn as nn
import torch.nn.functional as F
class MyFunc(torch.autograd.Function):
#staticmethod
def forward(ctx, input):
ctx.save_for_backward(input)
N = input.size(0)
for n in range(N):
input[n] /= torch.sum(input[n], dim=0)
return input
#staticmethod
def backward(ctx, grad_output):
input = ctx.saved_tensors[0]
N, H, W = input.size()
I = torch.eye(H).unsqueeze(-1)
sum = input.sum(1)
grad_input = torch.zeros((N,H,W), dtype = torch.double, requires_grad=True)
for n in range(N):
grad_input[n] = ((sum[n] * I - input[n]) * grad_output[n] / sum[n]**2).sum(1)
return grad_input
Gradcheck code is
from torch.autograd import gradcheck
func = MyFunc.apply
input = (torch.rand(2,2,2,dtype=torch.double,requires_grad=True))
test = gradcheck(func, input)
print(test)
and result is
enter image description here
I don't know why the error occurs...
Your help will be very helpful for me to implement my own convolutional network.
Thanks! Have a nice day.

Let's look an example with a single column, for instance: [[x1], [x2], [x3]].
Let sum be x1 + x2 + x3, then normalizing x will give y = [[y1], [y2], [y3]] = [[x1/sum], [x2/sum], [x3/sum]]. You're looking for dL/dx1, dL/x2, and dL/x3 - we'll just write them as: dx1, dx2, and dx3. Same for all dL/dyi.
So dx1 is equal to dL/dy1*dy1/dx1 + dL/dy2*dy2/dx1 + dL/dy3*dy3/dx1. That's because x1 contributes to all ouput element on the corresponding column: y1, y2, and y3.
We have:
dy1/dx1 = d(x1/sum)/dx1 = (sum - x1)/sum²
dy2/dx1 = d(x2/sum)/dx1 = -x2/sum²
similarly, dy3/dx1 = d(x3/sum)/dx1 = -x3/sum²
Therefore dx1 = (sum - x1)/sum²*dy1 - x2/sum²*dy2 - x3/sum²*dy3. Same for dx2 and dx3. As a result, the Jacobian is [dxi]_i = (sum - xi)/sum² and [dxi]_j = -xj/sum² (for all j different to i).
In your implementation, you seem to be missing all non-diagonal components.
Keeping the same one-column example, with x1=2, x2=3, and x3=5:
>>> x = torch.tensor([[2.], [3.], [5.]])
>>> sum = input.sum(0)
tensor([10])
The Jacobian will be:
>>> J = (sum*torch.eye(input.size(0)) - input)/sum**2
tensor([[ 0.0800, -0.0200, -0.0200],
[-0.0300, 0.0700, -0.0300],
[-0.0500, -0.0500, 0.0500]])
For an implementation with multiple columns, it's a bit trickier, more specifically for the shape of the diagonal matrix. It's easier to keep the column axis last so we don't have to bother with broadcastings:
>>> x = torch.tensor([[2., 1], [3., 3], [5., 5]])
>>> sum = x.sum(0)
tensor([10., 9.])
>>> diag = sum*torch.eye(3).unsqueeze(-1).repeat(1, 1, len(sum))
tensor([[[10., 9.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[10., 9.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[10., 9.]]])
Above diag has a shape of (3, 3, 2) where the two columns are on the last axis. Notice how we didn't need to broadcast sum.
What I wouldn't have done is: torch.eye(3).unsqueeze(0).repeat(len(sum), 1, 1). Since with this kind of shape - (2, 3, 3) - you will have to use sum[:, None, None], and will need further broadcasting down the road...
The Jacobian is simply:
>>> J = (diag - x)/sum**2
tensor([[[ 0.0800, 0.0988],
[-0.0300, -0.0370],
[-0.0500, -0.0617]],
[[-0.0200, -0.0123],
[ 0.0700, 0.0741],
[-0.0500, -0.0617]],
[[-0.0200, -0.0123],
[-0.0300, -0.0370],
[ 0.0500, 0.0494]]])
You can check the results by backpropagating through the operation using an arbitrary dy vector (not with torch.ones though, you'll get 0s because of J!). After backpropagating, x.grad should equal to torch.einsum('abc,bc->ac', J, dy).

Your Jacobian is not accurate: It is a 4d tensor, you only computed a 2D slice of it.
You neglected the second row of the Jacobian:

Answer for Q2.
I implemented back propagation myself for many batch case.
I used unsqueeze function and it worked.
size of input : (N,H,W) (N is batch size)
forward:
out = input / torch.sum(input, dim=1).unsqueeze(1)
backward:
diag = torch.eye(input.size(1), dtype=torch.double, requires_grad=True).unsqueeze(-1)
sum = input.sum(1)
grad_input = ((sum.unsqueeze(1).unsqueeze(1) * diag - input.unsqueeze(1)) * grad_out.unsqueeze(1) / (sum**2).unsqueeze(1).unsqueeze(1)).sum(2)

What is the most efficient way to compute a Kronecker Product in TensorFlow?

I am interested in implementing this paper on Kronecker Recurrent Units in TensorFlow.
This involves the computation of a Kronecker Product. TensorFlow does not have an operation for Kronecker Products. I am looking for an efficient and robust way to compute this.
Does this exist, or would I need to define a TensorFlow op manually?

If you will read the math definition of conv2d_transpose and see what Kronecker product calculates, you will see that with the appropriate size of stides for conv2d_tranpose (width, height of the second matrix), it does the same thing.
Moreover you even have batching of Kronecker's product out of the box with conv2d_transpose.
Here is an example of you which calculates the Kronecker's product for matrices from wiki.
import tensorflow as tf
a = [[1, 2], [3, 4]]
b = [[0, 5], [6, 7]]
i, k, s = len(a), len(b), len(b)
o = s * (i - 1) + k
a_tf = tf.reshape(tf.constant(a, dtype=tf.float32), [1, i, i, 1])
b_tf = tf.reshape(tf.constant(b, dtype=tf.float32), [k, k, 1, 1])
res = tf.squeeze(tf.nn.conv2d_transpose(a_tf, b_tf, (1, o, o, 1), [1, s, s, 1], "VALID"))
with tf.Session() as sess:
print sess.run(res)
Notice that in the case of a non-square matrix, you will need to calulcate more dimensions in the lines:
i, k, s = len(a), len(b), len(b)
o = s * (i - 1) + k
and use them properly as your strides/outputs arguments.

TensorFlow 1.7+ provides the function kronecker_product in tf.contrib.kfac.utils.kronecker_product:
a = tf.eye(3)
b = tf.constant([[1., 2.], [3., 4.]])
kron = tf.contrib.kfac.utils.kronecker_product(a, b)
tf.Session().run(kron)
Output:
array([[1., 2., 0., 0., 0., 0.],
[3., 4., 0., 0., 0., 0.],
[0., 0., 1., 2., 0., 0.],
[0., 0., 3., 4., 0., 0.],
[0., 0., 0., 0., 1., 2.],
[0., 0., 0., 0., 3., 4.]], dtype=float32)

Here's the utility I use for this. See kronecker_test for example of usage
def fix_shape(tf_shape):
return tuple(int(dim) for dim in tf_shape)
def concat_blocks(blocks, validate_dims=True):
"""Takes 2d grid of blocks representing matrices and concatenates to single
matrix (aka ArrayFlatten)"""
if validate_dims:
col_dims = np.array([[int(b.shape[1]) for b in row] for row in blocks])
col_sums = col_dims.sum(1)
assert (col_sums[0] == col_sums).all()
row_dims = np.array([[int(b.shape[0]) for b in row] for row in blocks])
row_sums = row_dims.sum(0)
assert (row_sums[0] == row_sums).all()
block_rows = [tf.concat(row, axis=1) for row in blocks]
return tf.concat(block_rows, axis=0)
def chunks(l, n):
"""Yield successive n-sized chunks from l."""
for i in range(0, len(l), n):
yield l[i:i + n]
from tensorflow.python.framework import ops
original_shape_func = ops.set_shapes_for_outputs
def disable_shape_inference():
ops.set_shapes_for_outputs = lambda _: _
def enable_shape_inference():
ops.set_shapes_for_outputs = original_shape_func
def kronecker(A, B, do_shape_inference=True):
"""Kronecker product of A,B.
turn_off_shape_inference: if True, makes 10x10 kron go 2.4 sec -> 0.9 sec
"""
Arows, Acols = fix_shape(A.shape)
Brows, Bcols = fix_shape(B.shape)
Crows, Ccols = Arows*Brows, Acols*Bcols
temp = tf.reshape(A, [-1, 1, 1])*tf.expand_dims(B, 0)
Bshape = tf.constant((Brows, Bcols))
# turn off shape inference
if not do_shape_inference:
disable_shape_inference()
# [1, n, m] => [n, m]
slices = [tf.reshape(s, Bshape) for s in tf.split(temp, Crows)]
# import pdb; pdb.set_trace()
grid = list(chunks(slices, Acols))
assert len(grid) == Arows
result = concat_blocks(grid, validate_dims=do_shape_inference)
if not do_shape_inference:
enable_shape_inference()
result.set_shape((Arows*Brows, Acols*Bcols))
return result
def kronecker_test():
A0 = [[1,2],[3,4]]
B0 = [[6,7],[8,9]]
A = tf.constant(A0)
B = tf.constant(B0)
C = kronecker(A, B)
sess = tf.Session()
C0 = sess.run(C)
Ct = [[6, 7, 12, 14], [8, 9, 16, 18], [18, 21, 24, 28], [24, 27, 32, 36]]
Cnp = np.kron(A0, B0)
check_equal(C0, Ct)
check_equal(C0, Cnp)

Try the following solution, see if it works for you:
def tf_kron(a,b):
a_shape = [a.shape[0].value,a.shape[1].value]
b_shape = [b.shape[0].value,b.shape[1].value]
return tf.reshape(tf.reshape(a,[a_shape[0],1,a_shape[1],1])*tf.reshape(b,[1,b_shape[0],1,b_shape[1]]),[a_shape[0]*b_shape[0],a_shape[1]*b_shape[1]])

How about something like this:
def kron(x, y):
"""Computes the Kronecker product of two matrices.
Args:
x: A matrix (or batch thereof) of size m x n.
y: A matrix (or batch thereof) of size p x q.
Returns:
z: Kronecker product of matrices x and y of size mp x nq
"""
with tf.name_scope('kron'):
x = tf.convert_to_tensor(x, dtype_hint=tf.float32)
y = tf.convert_to_tensor(y, dtype_hint=x.dtype)
def _maybe_expand(x):
xs = tf.pad(
tf.shape(x),
paddings=[[tf.maximum(2 - tf.rank(x), 0), 0]],
constant_values=1)
x = tf.reshape(x, xs)
_, mx, nx = tf.split(xs, num_or_size_splits=[-1, 1, 1])
return x, mx, nx
x, mx, nx = _maybe_expand(x)
y, my, ny = _maybe_expand(y)
x = x[..., :, tf.newaxis, :, tf.newaxis]
y = y[..., tf.newaxis, :, tf.newaxis, :]
z = x * y
bz = tf.shape(z)[:-4]
z = tf.reshape(z, tf.concat([bz, mx * my, nx * ny], axis=0))
return z
This solution:
supports batches
supports broadcasting
works in xla
clearly shows the relationship between numpy broadcasting and kronecker products.

Converting a list of points to a numpy 2D array

I'm using genfromtxt to import essentially a 2D array that has all its values listed in a text file of the form (x's and y's are integers):
x1 y1 z1
x2 y2 z2
: : :
I'm using the for loop below but I'm pretty sure there must be a one line way to do it. What would be a more efficient way to do this conversion?
raw = genfromtxt(file,skip_header = 6)
xrange = ( raw[:,0].min() , raw[:,0].max() )
yrange = ( raw[:,1].min() , raw[:,1].max() )
Z = zeros(( xrange[1] - xrange[0] +1 , yrange[1] - yrange[0] +1 ))
for row in raw:
Z[ row[0]-xrange[0] , row[1]-yrange[0] ] = row[2]

You can replace the for loop with the following:
xidx = (raw[:,0]-xrange[0]).astype(int)
yidx = (raw[:,1]-yrange[0]).astype(int)
Z[xidx, yidx] = raw[:,2]

To import a matrix from a file you can just split the lines and then convert to int.
[[int(i) for i in j.split()] for j in open('myfile').readlines()]
of course, I'm supposing your file contains only the matrix.
At the end, you can convert this 2-D array to numpy.

You may try something like this:
>>> Z = zeros((3, 3))
>>> test = array([[0, 1, 2], [1, 1, 6], [2, 0, 4]])
>>> Z[test[:, 0:2].T.tolist()]
array([ 0., 0., 0.])
>>> Z[test[:, 0:2].T.tolist()] = test[:, 2]
>>> Z
array([[ 0., 2., 0.],
[ 0., 6., 0.],
[ 4., 0., 0.]])
In your case:
Z[(raw[:, 0:2] - minimum(raw[:, 0:2], axis=0)).T.tolist()] = raw[:, 2]

You could also go with numpy.searchsorted which will also allow for non-equally spaced / float data:
raw = genfromtxt(file,skip_header = 6)
xvalues = numpy.sorted(set(raw[:,0]))
xidx = numpy.searchsorted(xvalues, raw[:,0])
yvalues = numpy.sorted(set(raw[:,1]))
yidx = numpy.searchsorted(yvalues, raw[:,1])
Z = numpy.zeros((len(xvalues), len(yvalues)))
Z[xidx, yidx] = raw[:,2]
Otherwise, I would be following Simon's answer.

How do I standardize a matrix?

Basically, take a matrix and change it so that its mean is equal to 0 and variance is 1. I'm using numpy's arrays so if it can already do it it's better, but I can implement it myself as long as I can find an algorithm.
edit: nvm nimrodm has a better implementation

The following subtracts the mean of A from each element (the new mean is 0), then normalizes the result by the standard deviation.
import numpy as np
A = (A - np.mean(A)) / np.std(A)
The above is for standardizing the entire matrix as a whole, If A has many dimensions and you want to standardize each column individually, specify the axis:
import numpy as np
A = (A - np.mean(A, axis=0)) / np.std(A, axis=0)
Always verify by hand what these one-liners are doing before integrating them into your code. A simple change in orientation or dimension can drastically change (silently) what operations numpy performs on them.

import scipy.stats as ss
A = np.array(ss.zscore(A))

from sklearn.preprocessing import StandardScaler
standardized_data = StandardScaler().fit_transform(your_data)
Example:
>>> import numpy as np
>>> from sklearn.preprocessing import StandardScaler
>>> data = np.random.randint(25, size=(4, 4))
>>> data
array([[17, 12, 4, 17],
[ 1, 16, 19, 1],
[ 7, 8, 10, 4],
[22, 4, 2, 8]])
>>> standardized_data = StandardScaler().fit_transform(data)
>>> standardized_data
array([[ 0.63812398, 0.4472136 , -0.718646 , 1.57786412],
[-1.30663482, 1.34164079, 1.55076242, -1.07959124],
[-0.57735027, -0.4472136 , 0.18911737, -0.58131836],
[ 1.24586111, -1.34164079, -1.02123379, 0.08304548]])
Works well on large datasets.

Use sklearn.preprocessing.scale.
http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html
Here is an example.
>>> from sklearn import preprocessing
>>> import numpy as np
>>> X_train = np.array([[ 1., -1., 2.],
... [ 2., 0., 0.],
... [ 0., 1., -1.]])
>>> X_scaled = preprocessing.scale(X_train)
>>> X_scaled
array([[ 0. ..., -1.22..., 1.33...],
[ 1.22..., 0. ..., -0.26...],
[-1.22..., 1.22..., -1.06...]])
http://scikit-learn.org/stable/modules/preprocessing.html#standardization-or-mean-removal-and-variance-scaling

import numpy as np
A = np.array([[1,2,6], [3000,1000,2000]]).T
A_means = np.mean(A, axis=0)
A_centr = A - A_means
A_norms = np.linalg.norm(A_centr, axis=0)
A_std = A_centr / A_norms