Develop Reference

Understanding numpy vectorization

I came across performing calculation for euclidian distance using numpy vectorization, here. Calculation done is:
>>> tri = np.array([[1, 1],
... [3, 1],
... [2, 3]])
>>> np.sum(tri**2, axis=1) ** 0.5 # Or: np.sqrt(np.sum(np.square(tri), 1))
array([1.4142, 3.1623, 3.6056])
So, to understand, I tried:
>>> np.sum(tri**2, axis=1)
array([ 2, 10, 13])
So basically, tri**2 is squaring each element: [[1,1],[9,1],[4,9]]. Next, we sum each sub-array element to get [1+1, 9+1, 4+9] = [2,10,13]
Then we take square root of each of them.
But I didnt get where are we doing the subtraction qi-pi as in the formula? Also I felt we should be getting single value: √((1-1)^2+(9-1)^2+(4-9)^2)=9.43
Am I missing some maths here or python / numpy understanding?

Assuming you have two vectors p and q represented as np.array:
dist = np.sqrt(np.sum((q - p) ** 2))
There is also np.linalg.norm which computes the same thing:
assert np.isclose(dist, np.linalg.norm(q - p))

python tensorflow l2 loss over axis

I am using python 3 with tensorflow
I have a matrix, each row is a vector, I want to get a distance matrix - that is computer using the l2 norm loss, each value in the matrix will be a distance between two vectors
e.g
Dij = l2_distance(M(i,:), Mj(j,:))
Thanks
edit:
this is not a duplicate of that other question is about computing the norm for the each row of a matrix, I need the pairwise norm distance between each row to every other row.

This answer shows how to compute the pair-wise sum of squared differences between a collection of vectors. By simply post-composing with the square root, you arrive at your desired pair-wise distances:
M = tf.constant([[0, 0], [2, 2], [5, 5]], dtype=tf.float64)
r = tf.reduce_sum(M*M, 1)
r = tf.reshape(r, [-1, 1])
D2 = r - 2*tf.matmul(M, tf.transpose(M)) + tf.transpose(r)
D = tf.sqrt(D2)
with tf.Session() as sess:
print(sess.run(D))
# [[0. 2.82842712 7.07106781]
# [2.82842712 0. 4.24264069]
# [7.07106781 4.24264069 0. ]]

You can write a TensorFlow operation based on the formula of Euclidean distance (L2 loss).
distance = tf.sqrt(tf.reduce_sum(tf.square(tf.subtract(x1, x2))))
Sample would be
import tensorflow as tf
x1 = tf.constant([1, 2, 3], dtype=tf.float32)
x2 = tf.constant([4, 5, 6], dtype=tf.float32)
distance = tf.sqrt(tf.reduce_sum(tf.square(tf.subtract(x1, x2))))
with tf.Session() as sess:
print(sess.run(distance))
As pointed out by #fuglede, if you want to output the pairwise distances, then we can use
tf.sqrt(tf.square(tf.subtract(x1, x2)))

how to get covariance matrix in tensorflow?

How could I get covariance matrix in tensorflow? Like numpy.cov() in numpy.
For example, I want to get covariance matrix of tensor A, now I have to use numpy instead
A = sess.run(model.A, feed)
cov = np.cov(np.transpose(A))
Is there anyway to get cov by tensorflow instead of numpy?
It is differnet from the problem how to compute covariance in tensorflow, where their problem is to compute covariance for two vector, while mine is to compute covariance matrix of a matrix(a 2D tensor) effectively using tensorflow API

This is months late but anyway posting for completeness.
import numpy as np
import tensorflow as tf
def tf_cov(x):
mean_x = tf.reduce_mean(x, axis=0, keep_dims=True)
mx = tf.matmul(tf.transpose(mean_x), mean_x)
vx = tf.matmul(tf.transpose(x), x)/tf.cast(tf.shape(x)[0], tf.float32)
cov_xx = vx - mx
return cov_xx
data = np.array([[1., 4, 2], [5, 6, 24], [15, 1, 5], [7,3,8], [9,4,7]])
with tf.Session() as sess:
print(sess.run(tf_cov(tf.constant(data, dtype=tf.float32))))
## validating with numpy solution
pc = np.cov(data.T, bias=True)
print(pc)

Answering from 2019. Tensorflow probability now supports effortless correlation and covariance.
https://www.tensorflow.org/probability/api_docs/python/tfp/stats/covariance
x = tf.random_normal(shape=(100, 2, 3))
y = tf.random_normal(shape=(100, 2, 3))
# cov[i, j] is the sample covariance between x[:, i, j] and y[:, i, j].
cov = tfp.stats.covariance(x, y, sample_axis=0, event_axis=None)
# cov_matrix[i, m, n] is the sample covariance of x[:, i, m] and y[:, i, n]
cov_matrix = tfp.stats.covariance(x, y, sample_axis=0, event_axis=-1)

Equivalent to np.cov(data):
import tensorflow as tf
import numpy as np
data = np.array([[1., 4, 2], [5, 6, 24], [15, 1, 5], [7,3,8], [9,4,7]])
def tf_conv(x):
x = x - tf.expand_dims(tf.reduce_mean(x, axis=1), 1)
fact = tf.cast(tf.shape(x)[1] - 1, tf.float32)
return tf.matmul(x, tf.conj(tf.transpose(x))) / fact
with tf.Session() as sess:
print(sess.run(tf_cov(tf.constant(data, dtype=tf.float32))))

Following up on #Souradeep Nanda, if you experiment with it you'll find that tfp.stats.covariance is only have the value (elementwise) for np.cov(..., rowvar=False), so you will have to multiply by 2 after the calculation. (This applies for v0.11.1, tested on 2x2 matrix).
For 3x3 matrix, the values are NOT EQUIVALENT so perhaps you might want to stay using np.cov. This applies too if you're not using rowvar=False for np.cov() version. I am not sure why.

We can use tfp aka tensorflow-probability to computer cov matrix:
import tensorflow-probability as tfp
x=tf.random.normal(shape=(3,3))
cov = tfp.stats.covariance(x)
## which are same as:
np_cov = np.cov(tf.transpose(x_zero),bias=True)

zip like function in Tensorflow? Tensorflow tensor operation

My question is about the tensor operation in Tensorflow.
Let's say:
import tensorflow as tf
import numpy as np
a = tf.Variable(np.random.random([10, 3, 3]))
b = tf.Variable(np.random.random([10, 3, 3]))
def some_function(m,n):
# just as an example
return tf.add(m, n)
This works in Tensorflow but it requires to know the dimension in advanced. However, it is very likely that the first dimension of the Tensor is None.
c = []
for i in range(10):
c.append(some_function(a[i], b[i]))
c = tf.stack(c)
So I wonder if there is a zip-like function in Tensorflow? Then we can do:
# TypeError: zip argument #1 must support iteration
c = []
for i, j in zip(a,b):
c.append(some_function(i,j))
c = tf.stack(c)
Maybe we can use some function like tf.map_fn or tf.scan? But I am not sure. Really thank you, guys.

Tensor objects are not iterable, which explains why your third code sample fails. So, to answer your question, there is no zip-like function in TensorFlow.
You can indeed use tf.map_fn to apply a function to a sequence of tensors. The problem you pose in your example code can be solved in the following fashion:
def some_function(tensor):
return tf.reduce_sum(tensor)
c = tf.stack([a, b], axis=1)
d = tf.map_fn(some_function, c, dtype=tf.float32)
yields a Tensor d whose value is [20., 6., 6.].

You can use tf.transpose like this
>>> a = tf.constant([1, 2, 3])
>>> b = tf.constant([4, 5, 6])
>>> tf.transpose([a, b])
<tf.Tensor: shape=(3, 2), dtype=int32, numpy=
array([[1, 4],
[2, 5],
[3, 6]], dtype=int32)>

For those of you using JavsScript, this is #bachr's answer in tensorflow.js (node):
const a = tf.tensor([1, 3, 5, 7])
const b = tf.tensor([2, 4, 6, 8])
const zip = tf.transpose(tf.stack([a, b]))
zip.print()
// Tensor
// [[1, 2],
// [3, 4],
// [5, 6],
// [7, 8]]

Compute pairwise distance in a batch without replicating tensor in Tensorflow?

I want to compute the pairwise square distance of a batch of feature in Tensorflow. I have a simple implementation using + and * operations by
tiling the original tensor :
def pairwise_l2_norm2(x, y, scope=None):
with tf.op_scope([x, y], scope, 'pairwise_l2_norm2'):
size_x = tf.shape(x)[0]
size_y = tf.shape(y)[0]
xx = tf.expand_dims(x, -1)
xx = tf.tile(xx, tf.pack([1, 1, size_y]))
yy = tf.expand_dims(y, -1)
yy = tf.tile(yy, tf.pack([1, 1, size_x]))
yy = tf.transpose(yy, perm=[2, 1, 0])
diff = tf.sub(xx, yy)
square_diff = tf.square(diff)
square_dist = tf.reduce_sum(square_diff, 1)
return square_dist
This function takes as input two matrices of size (m,d) and (n,d) and compute the squared distance between each row vector. The output is a matrix of size (m,n) with element 'd_ij = dist(x_i, y_j)'.
The problem is that I have a large batch and high dim features 'm, n, d' replicating the tensor consume a lot of memory.
I'm looking for another way to implement this without increasing the memory usage and just only store the final distance tensor. Kind of double looping the original tensor.

You can use some linear algebra to turn it into matrix ops. Note that what you need matrix D where a[i] is the ith row of your original matrix and
D[i,j] = (a[i]-a[j])(a[i]-a[j])'
You can rewrite that into
D[i,j] = r[i] - 2 a[i]a[j]' + r[j]
Where r[i] is squared norm of ith row of the original matrix.
In a system that supports standard broadcasting rules you can treat r as a column vector and write D as
D = r - 2 A A' + r'
In TensorFlow you could write this as
A = tf.constant([[1, 1], [2, 2], [3, 3]])
r = tf.reduce_sum(A*A, 1)
# turn r into column vector
r = tf.reshape(r, [-1, 1])
D = r - 2*tf.matmul(A, tf.transpose(A)) + tf.transpose(r)
sess = tf.Session()
sess.run(D)
result
array([[0, 2, 8],
[2, 0, 2],
[8, 2, 0]], dtype=int32)

Using squared_difference:
def squared_dist(A):
expanded_a = tf.expand_dims(A, 1)
expanded_b = tf.expand_dims(A, 0)
distances = tf.reduce_sum(tf.squared_difference(expanded_a, expanded_b), 2)
return distances
One thing I noticed is that this solution using tf.squared_difference gives me out of memory (OOM) for very large vectors, while the approach by #YaroslavBulatov doesn't. So, I think decomposing the operation yields a smaller memory footprint (which I thought squared_difference would handle better under the hood).

Here is a more general solution for two tensors of coordinates A and B:
def squared_dist(A, B):
assert A.shape.as_list() == B.shape.as_list()
row_norms_A = tf.reduce_sum(tf.square(A), axis=1)
row_norms_A = tf.reshape(row_norms_A, [-1, 1]) # Column vector.
row_norms_B = tf.reduce_sum(tf.square(B), axis=1)
row_norms_B = tf.reshape(row_norms_B, [1, -1]) # Row vector.
return row_norms_A - 2 * tf.matmul(A, tf.transpose(B)) + row_norms_B
Note that this is the square distance. If you want to change this to the Euclidean distance, perform a tf.sqrt on the result. If you want to do that, don't forget to add a small constant to compensate for the floating point instabilities: dist = tf.sqrt(squared_dist(A, B) + 1e-6).

If you want compute other method , then change the order of the tf modules.
def compute_euclidean_distance(x, y):
size_x = x.shape.dims[0]
size_y = y.shape.dims[0]
for i in range(size_x):
tile_one = tf.reshape(tf.tile(x[i], [size_y]), [size_y, -1])
eu_one = tf.expand_dims(tf.sqrt(tf.reduce_sum(tf.pow(tf.subtract(tile_one, y), 2), axis=1)), axis=0)
if i == 0:
d = eu_one
else:
d = tf.concat([d, eu_one], axis=0)
return d