How could I get covariance matrix in tensorflow? Like numpy.cov() in numpy.
For example, I want to get covariance matrix of tensor A, now I have to use numpy instead
A = sess.run(model.A, feed)
cov = np.cov(np.transpose(A))
Is there anyway to get cov by tensorflow instead of numpy?
It is differnet from the problem how to compute covariance in tensorflow, where their problem is to compute covariance for two vector, while mine is to compute covariance matrix of a matrix(a 2D tensor) effectively using tensorflow API
This is months late but anyway posting for completeness.
import numpy as np
import tensorflow as tf
def tf_cov(x):
mean_x = tf.reduce_mean(x, axis=0, keep_dims=True)
mx = tf.matmul(tf.transpose(mean_x), mean_x)
vx = tf.matmul(tf.transpose(x), x)/tf.cast(tf.shape(x)[0], tf.float32)
cov_xx = vx - mx
return cov_xx
data = np.array([[1., 4, 2], [5, 6, 24], [15, 1, 5], [7,3,8], [9,4,7]])
with tf.Session() as sess:
print(sess.run(tf_cov(tf.constant(data, dtype=tf.float32))))
## validating with numpy solution
pc = np.cov(data.T, bias=True)
print(pc)
Answering from 2019. Tensorflow probability now supports effortless correlation and covariance.
https://www.tensorflow.org/probability/api_docs/python/tfp/stats/covariance
x = tf.random_normal(shape=(100, 2, 3))
y = tf.random_normal(shape=(100, 2, 3))
# cov[i, j] is the sample covariance between x[:, i, j] and y[:, i, j].
cov = tfp.stats.covariance(x, y, sample_axis=0, event_axis=None)
# cov_matrix[i, m, n] is the sample covariance of x[:, i, m] and y[:, i, n]
cov_matrix = tfp.stats.covariance(x, y, sample_axis=0, event_axis=-1)
Equivalent to np.cov(data):
import tensorflow as tf
import numpy as np
data = np.array([[1., 4, 2], [5, 6, 24], [15, 1, 5], [7,3,8], [9,4,7]])
def tf_conv(x):
x = x - tf.expand_dims(tf.reduce_mean(x, axis=1), 1)
fact = tf.cast(tf.shape(x)[1] - 1, tf.float32)
return tf.matmul(x, tf.conj(tf.transpose(x))) / fact
with tf.Session() as sess:
print(sess.run(tf_cov(tf.constant(data, dtype=tf.float32))))
Following up on #Souradeep Nanda, if you experiment with it you'll find that tfp.stats.covariance is only have the value (elementwise) for np.cov(..., rowvar=False), so you will have to multiply by 2 after the calculation. (This applies for v0.11.1, tested on 2x2 matrix).
For 3x3 matrix, the values are NOT EQUIVALENT so perhaps you might want to stay using np.cov. This applies too if you're not using rowvar=False for np.cov() version. I am not sure why.
We can use tfp aka tensorflow-probability to computer cov matrix:
import tensorflow-probability as tfp
x=tf.random.normal(shape=(3,3))
cov = tfp.stats.covariance(x)
## which are same as:
np_cov = np.cov(tf.transpose(x_zero),bias=True)
Related
Given an integer k and a symmetric matrix A (as tf.Variable), how to compute the k-th power of A
tf.matmul(A, tf.matmul(A, tf.matmul(A, ...)... )
most efficiently in TensorFlow?
Using tf.while_loop should be quite efficient:
import tensorflow as tf
k = 3
A = tf.Variable([[1, -3], [2, 5]])
result = tf.Variable(A)
i = tf.constant(0)
c = lambda i: tf.less(i, k - 1)
def body(i):
result.assign(tf.matmul(A, result))
return [tf.add(i, 1)]
_ = tf.while_loop(c, body, [i])
print(result)
<tf.Variable 'Variable:0' shape=(2, 2) dtype=int32, numpy= array([[-41, -75], [ 50, 59]], dtype=int32)>
This could be one approach to tackle this problem.
i. Convert matrix A into numpy ndarray(let's say B)
ii. Compute k-th power of B using: np.linalg.matrix_power(B, k)
iii. Convert the result back into tf.Variable
Here is a working code for the above-mentioned approach
import tensorflow as tf
import numpy as np
k = 2
A = tf.Variable([[1, -3], [2, 5]])
B = A.numpy()
M = np.linalg.matrix_power(B, k)
power = tf.Variable(M)
print(power)
You can also do this with fractions. If you need it to stay in tensor form use the aproximate solution with the taylor seiries of (1+x)^n and integer powers with 1 being the identity matrix
Not really sure what you will do with it but you can probably find something cool.
I'm trying to compute the cosine similarity between 350k sentences using tensorflow.
My sentences are first vectorisd using sklearn:
doc = df['text']
vec = TfidfVectorizer(binary=False,norm='l2',use_idf=False,smooth_idf=False,lowercase=True,stop_words='english',min_df=1,max_df=1.0,max_features=None,ngram_range=(1, 1))
X = vec.fit_transform(doc)
print(X.shape)
print(type(X))
This works very well and I get sparse matrix back, I have then tried in two ways to convert my sparse matrix to a dense one.
(1) I tried this:
dense = X.toarray()
This only works with a small amount of data (around 10k sentences), but then fails on the actual computation.
(2) I have been trying to convert the output X this way, but get the same error message when doing the first step K:
K = tf.convert_to_tensor(X, dtype=None, dtype_hint=None, name=None)
Y = tf.sparse.to_dense(K, default_value=None, validate_indices=True, name=None)
Any tips/ tricks to solve this mystery would be greatly appreciated. Also happy to consider batching my computations if that should be more efficient in terms of size?
You need to make a TensorFlow sparse matrix from your SciPy one. Since your matrix seems to be in CSR format, you can do it as follows:
import numpy as np
import scipy.sparse
import tensorflow as tf
def sparse_csr_to_tf(csr_mat):
indptr = tf.constant(csr_mat.indptr, dtype=tf.int64)
elems_per_row = indptr[1:] - indptr[:-1]
i = tf.repeat(tf.range(csr_mat.shape[0], dtype=tf.int64), elems_per_row)
j = tf.constant(csr_mat.indices, dtype=tf.int64)
indices = np.stack([i, j], axis=-1)
data = tf.constant(csr_mat.data)
return tf.sparse.SparseTensor(indices, data, csr_mat.shape)
# Test
m = scipy.sparse.csr_matrix([
[0, 0, 1, 0],
[0, 0, 0, 0],
[2, 0, 3, 4],
], dtype=np.float32)
tf_mat = sparse_csr_to_tf(m)
tf.print(tf.sparse.to_dense(tf_mat))
# [[0 0 1 0]
# [0 0 0 0]
# [2 0 3 4]]
I am trying to generalise the example given in How to use a MultiVariateNormal distribution in the latest version of Tensorflow to a normal distribution in 2-D but with more than one batch. When I run the following:
from tensorflow_probability import distributions as tfd
import tensorflow as tf
tf.compat.v1.enable_eager_execution()
mu = [[1, 2],
[-1,-2]]
cov = [[1, 3./5],
[3./5, 2]]
cov = [cov, cov] # for demonstration purpose, use same cov for both batches
mvn = tfd.MultivariateNormalFullCovariance(
loc=mu,
covariance_matrix=cov)
# generate the pdf
X, Y = tf.meshgrid(tf.range(-3, 3, 0.1), tf.range(-3, 3, 0.1))
idx = tf.concat([tf.reshape(X, [-1, 1]), tf.reshape(Y,[-1,1])], axis =1)
prob = tf.reshape(mvn.prob(idx), tf.shape(X))
I get an Incompatible shapes error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [3600,2] vs. [2,2] [Op:Sub] name: MultivariateNormalFullCovariance/log_prob/affine_linear_operator/inverse/sub/
My understanding of the documentation (https://www.tensorflow.org/api_docs/python/tf/contrib/distributions/MultivariateNormalFullCovariance) is that to compute the pdf, one needs a [n_observation, n_dimensions] tensor (which is the case in this example: idx.shape = TensorShape([Dimension(3600), Dimension(2)])). Did I get my maths wrong?
You need to add a batch axis to the idx tensor in the second-to-last position, since 60x60 can't broadcast against the mvn.batch_shape of (2,).
# TF/TFP Imports
!pip install --quiet tfp-nightly tf-nightly
import tensorflow.compat.v2 as tf
tf.enable_v2_behavior()
import tensorflow_probability as tfp
tfd = tfp.distributions
mu = [[1, 2],
[-1, -2]]
cov = [[1, 3./5],
[3./5, 2]]
cov = [cov, cov] # for demonstration purpose, use same cov for both batches
mvn = tfd.MultivariateNormalFullCovariance(
loc=mu, covariance_matrix=cov)
print(mvn.batch_shape, mvn.event_shape)
# generate the pdf
X, Y = tf.meshgrid(tf.range(-3, 3, 0.1), tf.range(-3, 3, 0.1))
print(X.shape)
idx = tf.stack([X, Y], axis=-1)[..., tf.newaxis, :]
print(idx.shape)
probs = mvn.prob(idx)
print(probs.shape)
output:
(2,) (2,) # mvn.batch_shape, mvn.event_shape
(60, 60) # X.shape
(60, 60, 1, 2) # idx.shape == X.shape + (1 "broadcast against batch", 2 "event")
(60, 60, 2) # probs.shape == X.shape + (2 "mvn batch shape")
I am using python 3 with tensorflow
I have a matrix, each row is a vector, I want to get a distance matrix - that is computer using the l2 norm loss, each value in the matrix will be a distance between two vectors
e.g
Dij = l2_distance(M(i,:), Mj(j,:))
Thanks
edit:
this is not a duplicate of that other question is about computing the norm for the each row of a matrix, I need the pairwise norm distance between each row to every other row.
This answer shows how to compute the pair-wise sum of squared differences between a collection of vectors. By simply post-composing with the square root, you arrive at your desired pair-wise distances:
M = tf.constant([[0, 0], [2, 2], [5, 5]], dtype=tf.float64)
r = tf.reduce_sum(M*M, 1)
r = tf.reshape(r, [-1, 1])
D2 = r - 2*tf.matmul(M, tf.transpose(M)) + tf.transpose(r)
D = tf.sqrt(D2)
with tf.Session() as sess:
print(sess.run(D))
# [[0. 2.82842712 7.07106781]
# [2.82842712 0. 4.24264069]
# [7.07106781 4.24264069 0. ]]
You can write a TensorFlow operation based on the formula of Euclidean distance (L2 loss).
distance = tf.sqrt(tf.reduce_sum(tf.square(tf.subtract(x1, x2))))
Sample would be
import tensorflow as tf
x1 = tf.constant([1, 2, 3], dtype=tf.float32)
x2 = tf.constant([4, 5, 6], dtype=tf.float32)
distance = tf.sqrt(tf.reduce_sum(tf.square(tf.subtract(x1, x2))))
with tf.Session() as sess:
print(sess.run(distance))
As pointed out by #fuglede, if you want to output the pairwise distances, then we can use
tf.sqrt(tf.square(tf.subtract(x1, x2)))
I would like to convert a NumPy array to a unit vector. More specifically, I am looking for an equivalent version of this normalisation function:
def normalize(v):
norm = np.linalg.norm(v)
if norm == 0:
return v
return v / norm
This function handles the situation where vector v has the norm value of 0.
Is there any similar functions provided in sklearn or numpy?
If you're using scikit-learn you can use sklearn.preprocessing.normalize:
import numpy as np
from sklearn.preprocessing import normalize
x = np.random.rand(1000)*10
norm1 = x / np.linalg.norm(x)
norm2 = normalize(x[:,np.newaxis], axis=0).ravel()
print np.all(norm1 == norm2)
# True
I agree that it would be nice if such a function were part of the included libraries. But it isn't, as far as I know. So here is a version for arbitrary axes that gives optimal performance.
import numpy as np
def normalized(a, axis=-1, order=2):
l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
l2[l2==0] = 1
return a / np.expand_dims(l2, axis)
A = np.random.randn(3,3,3)
print(normalized(A,0))
print(normalized(A,1))
print(normalized(A,2))
print(normalized(np.arange(3)[:,None]))
print(normalized(np.arange(3)))
This might also work for you
import numpy as np
normalized_v = v / np.sqrt(np.sum(v**2))
but fails when v has length 0.
In that case, introducing a small constant to prevent the zero division solves this.
As proposed in the comments one could also use
v/np.linalg.norm(v)
To avoid zero division I use eps, but that's maybe not great.
def normalize(v):
norm=np.linalg.norm(v)
if norm==0:
norm=np.finfo(v.dtype).eps
return v/norm
If you have multidimensional data and want each axis normalized to its max or its sum:
def normalize(_d, to_sum=True, copy=True):
# d is a (n x dimension) np array
d = _d if not copy else np.copy(_d)
d -= np.min(d, axis=0)
d /= (np.sum(d, axis=0) if to_sum else np.ptp(d, axis=0))
return d
Uses numpys peak to peak function.
a = np.random.random((5, 3))
b = normalize(a, copy=False)
b.sum(axis=0) # array([1., 1., 1.]), the rows sum to 1
c = normalize(a, to_sum=False, copy=False)
c.max(axis=0) # array([1., 1., 1.]), the max of each row is 1
If you don't need utmost precision, your function can be reduced to:
v_norm = v / (np.linalg.norm(v) + 1e-16)
You mentioned sci-kit learn, so I want to share another solution.
sci-kit learn MinMaxScaler
In sci-kit learn, there is a API called MinMaxScaler which can customize the the value range as you like.
It also deal with NaN issues for us.
NaNs are treated as missing values: disregarded in fit, and maintained
in transform. ... see reference [1]
Code sample
The code is simple, just type
# Let's say X_train is your input dataframe
from sklearn.preprocessing import MinMaxScaler
# call MinMaxScaler object
min_max_scaler = MinMaxScaler()
# feed in a numpy array
X_train_norm = min_max_scaler.fit_transform(X_train.values)
# wrap it up if you need a dataframe
df = pd.DataFrame(X_train_norm)
Reference
[1] sklearn.preprocessing.MinMaxScaler
There is also the function unit_vector() to normalize vectors in the popular transformations module by Christoph Gohlke:
import transformations as trafo
import numpy as np
data = np.array([[1.0, 1.0, 0.0],
[1.0, 1.0, 1.0],
[1.0, 2.0, 3.0]])
print(trafo.unit_vector(data, axis=1))
If you work with multidimensional array following fast solution is possible.
Say we have 2D array, which we want to normalize by last axis, while some rows have zero norm.
import numpy as np
arr = np.array([
[1, 2, 3],
[0, 0, 0],
[5, 6, 7]
], dtype=np.float)
lengths = np.linalg.norm(arr, axis=-1)
print(lengths) # [ 3.74165739 0. 10.48808848]
arr[lengths > 0] = arr[lengths > 0] / lengths[lengths > 0][:, np.newaxis]
print(arr)
# [[0.26726124 0.53452248 0.80178373]
# [0. 0. 0. ]
# [0.47673129 0.57207755 0.66742381]]
If you want to normalize n dimensional feature vectors stored in a 3D tensor, you could also use PyTorch:
import numpy as np
from torch import FloatTensor
from torch.nn.functional import normalize
vecs = np.random.rand(3, 16, 16, 16)
norm_vecs = normalize(FloatTensor(vecs), dim=0, eps=1e-16).numpy()
If you're working with 3D vectors, you can do this concisely using the toolbelt vg. It's a light layer on top of numpy and it supports single values and stacked vectors.
import numpy as np
import vg
x = np.random.rand(1000)*10
norm1 = x / np.linalg.norm(x)
norm2 = vg.normalize(x)
print np.all(norm1 == norm2)
# True
I created the library at my last startup, where it was motivated by uses like this: simple ideas which are way too verbose in NumPy.
Without sklearn and using just numpy.
Just define a function:.
Assuming that the rows are the variables and the columns the samples (axis= 1):
import numpy as np
# Example array
X = np.array([[1,2,3],[4,5,6]])
def stdmtx(X):
means = X.mean(axis =1)
stds = X.std(axis= 1, ddof=1)
X= X - means[:, np.newaxis]
X= X / stds[:, np.newaxis]
return np.nan_to_num(X)
output:
X
array([[1, 2, 3],
[4, 5, 6]])
stdmtx(X)
array([[-1., 0., 1.],
[-1., 0., 1.]])
For a 2D array, you can use the following one-liner to normalize across rows. To normalize across columns, simply set axis=0.
a / np.linalg.norm(a, axis=1, keepdims=True)
If you want all values in [0; 1] for 1d-array then just use
(a - a.min(axis=0)) / (a.max(axis=0) - a.min(axis=0))
Where a is your 1d-array.
An example:
>>> a = np.array([0, 1, 2, 4, 5, 2])
>>> (a - a.min(axis=0)) / (a.max(axis=0) - a.min(axis=0))
array([0. , 0.2, 0.4, 0.8, 1. , 0.4])
Note for the method. For saving proportions between values there is a restriction: 1d-array must have at least one 0 and consists of 0 and positive numbers.
A simple dot product would do the job. No need for any extra package.
x = x/np.sqrt(x.dot(x))
By the way, if the norm of x is zero, it is inherently a zero vector, and cannot be converted to a unit vector (which has norm 1). If you want to catch the case of np.array([0,0,...0]), then use
norm = np.sqrt(x.dot(x))
x = x/norm if norm != 0 else x