Develop Reference

Tensor multiplication in Keras

I have two tensors of size
A <tf.Tensor 'sequential_12/my_layer_56/add:0' shape=(?, 300, 2) dtype=float32>
and
B <tf.Tensor 'input_82:0' shape=(?, 2, 2) dtype=float32>
Now, I would like to multiply them in the sense of the usual matrix row-column product to obtain
A * B of size (?, 300, 2), so I would be doing the matrix product only over the second and third dimension. How can I achieve this?
I tried to use tf.tensordot with different axes specifications, but it did not work so far. For example I tried
tf.tensordot(A,B,axes=[[2], [0]])
but this produces a tensor of the following form
<tf.Tensor 'Tensordot_10:0' shape=(?, 300, 2, 2) dtype=float32>

Maybe try tf.matmul:
import tensorflow as tf
samples = 1
A = tf.random.normal((samples, 300, 2))
B = tf.random.normal((samples, 2, 2))
print(tf.matmul(A, B).shape)
# (1, 300, 2)

Tensorflow/Keras: how to get the output of an Input layer by channels?

I have an Input layer that looks like this:
>>>inp = tf.keras.Input(shape=(107, 3))
>>>print(inp)
Tensor("input_25:0", shape=(None, 107, 3), dtype=float32)
Since the shape is (None, 107, 3), I want to take each (None, 107, 1) to use it for separate layers. How do I do that?

According to a related GitHub issue, you can use tf.keras.layers.Lambda to split the input tensor by channel.
import tensorflow as tf
tfkl = tf.keras.layers
inp = tf.keras.Input(shape=(107, 3))
x0 = tfkl.Lambda(lambda x : x[..., 0])(x)
x1 = tfkl.Lambda(lambda x : x[..., 1])(x)
x2 = tfkl.Lambda(lambda x : x[..., 2])(x)
The ... is an Ellipsis and fills in the slices to get the last slice only.

Keras Lambda layer: multiply a tensor with a matrix of constant

I'm writing a Lambda layer in Keras to compute the multiplication of a tensor and a matrix of constant. But the size went wrong in the output.
In the code, jdes is a tensor with dimension TensorShape([Dimension(None), Dimension(100)]). Further, six_title_embedding is the matrix of constants and it has shape (6, 100).
def cosine_distance(input):
jd = K.l2_normalize(input, axis=-1)
jt_six = K.l2_normalize(six_title_embedding, axis=-1)
return jd * jt_six
distance = Lambda(cosine_distance, output_shape=(None, 6,100))(jdes)
result = Dense(1, activation='sigmoid')(distance)
In the output, I was expecting result to have shape (None, 6, 1) but now it's (6, 1) so the number of batch_size is lost during the computation of the lambda layer. The shape of distance is now TensorShape([Dimension(6), Dimension(100)]). Should this be (None, 6, 100) so that result can have (None, 6, 1)?

You need to create an additional axis for jdes so that when multiplying it with constant tensor, they could be broadcasted to the same shape:
def cosine_distance(input):
jd = K.l2_normalize(input, axis=-1)
jt_six = K.l2_normalize(six_title_embedding, axis=-1)
jd = K.expand_dims(jd, axis=1) # now it would have a shape of (None, 1, 100)
return jd * jt_six # the result would be (None, 6, 100)

How to build a tensor from 2 scalars in Tensorflow?

I have two scalars resulting from the following operations:
a = tf.reduce_sum(tensor1), b = tf.matmul(tf.transpose(tensor2), tensor3) this is a dot product since tensor2 and tensor3 have the same dimensions (1-D vectors). Since these tensors have shape [None, dim1] it becomes difficult to deal with the shapes.
I want to build a tensor that has shape (2,1) using a and b.
I tried tf.Tensor([a,b], dtype=tf.float64, value_index=0) but raises the error
TypeError: op needs to be an Operation: [<tf.Tensor 'Sum_5:0' shape=() dtype=float32>, <tf.Tensor 'MatMul_67:0' shape=(?, ?) dtype=float32>]
Any easier way to build that tensor/vector?

This would do probably. Change axis based on what you need
a = tf.constant(1)
b = tf.constant(2)
c = tf.stack([a,b],axis=0)
Output:
array([[1],
[2]], dtype=int32)

You can use concat or stack to achieve this:
import tensorflow as tf
t1 = tf.constant([1])
t2 = tf.constant([2])
c = tf.reshape(tf.concat([t1, t2], 0), (2, 1))
with tf.Session() as sess:
print sess.run(c)
In a similar way you can achieve it with tf.stack.

tensorflow efficient way for tensor multiplication

I have two tensors in tensorflow, the first tensor is 3-D, and the second is 2D. And I want to multiply them like this:
x = tf.placeholder(tf.float32, shape=[sequence_length, batch_size, hidden_num])
w = tf.get_variable("w", [hidden_num, 50])
b = tf.get_variable("b", [50])
output_list = []
for step_index in range(sequence_length):
output = tf.matmul(x[step_index, :, :], w) + b
output_list.append(output)
output = tf.pack(outputs_list)
I use a loop to do multiply operation, but I think it is too slow. What would be the best way to make this process as simple/clean as possible?

You could use batch_matmul. Unfortunately it doesn't seem batch_matmul supports broadcasting along the batch dimension, so you have to tile your w matrix. This will use more memory, but all operations will stay in TensorFlow
a = tf.ones((5, 2, 3))
b = tf.ones((3, 1))
b = tf.reshape(b, (1, 3, 1))
b = tf.tile(b, [5, 1, 1])
c = tf.batch_matmul(a, b) # use tf.matmul in TF 1.0
sess = tf.InteractiveSession()
sess.run(tf.shape(c))
This gives
array([5, 2, 1], dtype=int32)

You could use map_fn, which scans a function along the first dimension.
x = tf.placeholder(tf.float32, shape=[sequence_length, batch_size, hidden_num])
w = tf.get_variable("w", [hidden_num, 50])
b = tf.get_variable("b", [50])
def mul_fn(current_input):
return tf.matmul(current_input, w) + b
output = tf.map_fn(mul_fn, x)
I used this at one point to implement a softmax scan along a sequence.