Develop Reference

KAGGLE BERT : ValueError: Shapes (None, 2) and (None, 3) are incompatible

I'm trying to follow a kaggle for the BERT model : https://www.kaggle.com/code/ludovicocuoghi/twitter-sentiment-analysis-with-bert-roberta/notebook
And at the step near the end I get this error :
ValueError: Shapes (None, 2) and (None, 3) are incompatible
The cell :
history_bert = model.fit([train_input_ids,train_attention_masks], y_train, validation_data=([val_input_ids,val_attention_masks], y_valid), epochs=4, batch_size=32)
I tried many things but I didnt find any solution If someone cane enlight me it'll be useful !
the shape of differents variables :
train_input_ids SHAPE: (640740, 128)
train_attention_masks SHAPE: (640740, 128)
y_train SHAPE: (640740, 2)
val_input_ids SHAPE: (71194, 128)
val_attention_masks SHAPE: (71194, 128)
y_valid SHAPE: (71194, 2)
In my dataset Sentiment has only 2 value 0 / 1 idk if this may have an impact on the error
I switched the variable's place in the functions fit.
I search for similar errors on differents forums to find a solution.

Tensor multiplication in Keras

I have two tensors of size
A <tf.Tensor 'sequential_12/my_layer_56/add:0' shape=(?, 300, 2) dtype=float32>
and
B <tf.Tensor 'input_82:0' shape=(?, 2, 2) dtype=float32>
Now, I would like to multiply them in the sense of the usual matrix row-column product to obtain
A * B of size (?, 300, 2), so I would be doing the matrix product only over the second and third dimension. How can I achieve this?
I tried to use tf.tensordot with different axes specifications, but it did not work so far. For example I tried
tf.tensordot(A,B,axes=[[2], [0]])
but this produces a tensor of the following form
<tf.Tensor 'Tensordot_10:0' shape=(?, 300, 2, 2) dtype=float32>

Maybe try tf.matmul:
import tensorflow as tf
samples = 1
A = tf.random.normal((samples, 300, 2))
B = tf.random.normal((samples, 2, 2))
print(tf.matmul(A, B).shape)
# (1, 300, 2)

TensorFlow metric: top-N accuracy

I'm using add_metric trying to create a custom metric that computes top 3 accuracy for a classifier. Here's as far as I got:
def custom_metrics(labels, predictions):
# labels => Tensor("fifo_queue_DequeueUpTo:422", shape=(?,), dtype=int64)
# predictions => {
# 'logits': <tf.Tensor 'dnn/logits/BiasAdd:0' shape=(?, 26) dtype=float32>,
# 'probabilities': <tf.Tensor 'dnn/head/predictions/probabilities:0' shape=(?, 26) dtype=float32>,
# 'class_ids': <tf.Tensor 'dnn/head/predictions/ExpandDims:0' shape=(?, 1) dtype=int64>,
# 'classes': <tf.Tensor 'dnn/head/predictions/str_classes:0' shape=(?, 1) dtype=string>
# }
Looking at the implementation of existing tf.metrics, everything is implemented using tf ops. How could I implement top 3 accuracy?

If you want to implement it yourself tf.nn.in_top_k is very useful - it returns a boolean array which indicates if target is within the top k predictions. You just have to take the mean of the result:
def custom_metrics(labels, predictions):
return tf.metrics.mean(tf.nn.in_top_k(predictions=predictions, targets=labels, k=3))
You can also import it:
from tf.keras.metrics import top_k_categorical_accuracy

using rnn_cell inside of tf.while getting ValueError: The two structures don't have the same number of elements

Given data = tf.placeholder(tf.float32, [2, None, 3]) (batch_size * time_step * feature_size), Ideally I want do tf.unstack(data, axis = 1) to get a number of tensors each of which has the shape of [2,3] so later feed them to a rnn with a for loop like
for rnn_input in rnn_inputs:
state = rnn_cell(rnn_input, state)
Using high-level API like tf.nn.dynamic_rnn is off the table so I create a work around like
import tensorflow as tf
data = tf.placeholder(tf.float32, [2, None, 3])
step_number = tf.placeholder(tf.int32, None)
loop_counter_inital = tf.constant(0)
initi_state = tf.zeros([2,3], tf.float32)
def while_condition(loop_counter, rnn_states):
return loop_counter < step_number
def while_body(loop_counter, rnn_states):
loop_counter_current = loop_counter
current_states = tf.gather_nd(data, tf.stack([tf.range(0, 2), tf.zeros([2], tf.int32)+loop_counter_current], axis=1))
cell = tf.nn.rnn_cell.BasicRNNCell(3)
rnn_states = cell(current_states, rnn_states)
return [loop_counter_current, rnn_states]
_, _states = tf.while_loop(while_condition, while_body,
loop_vars=[loop_counter_inital, initi_state],
shape_invariants=[loop_counter_inital.shape, tf.TensorShape([2, 3])])
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print (sess.run(_states, feed_dict={data:[[[3,1,6],[4,1,2]],[[5,8,1],[0,5,2]]], step_number:2 }))
The idea is to loop through each row in each of the 2D tensor of data to get the features for each time step. I got a error
First structure (2 elements): [<tf.Tensor 'while/Identity:0' shape=() dtype=int32>, <tf.Tensor 'while/Identity_1:0' shape=(2, 3) dtype=float32>]
Second structure (3 elements): [<tf.Tensor 'while/Identity:0' shape=() dtype=int32>, (<tf.Tensor 'while/basic_rnn_cell/Tanh:0' shape=(2, 3) dtype=float32>, <tf.Tensor 'while/basic_rnn_cell/Tanh:0' shape=(2, 3) dtype=float32>)]
There seems to be some related posts. None actually worked. Can anyone help?

You need to know every BasicRNNCell will implement call() with the signature (output, next_state) = call(input, state). This means that your result is a list of shapes ((?,unit),(?,unit)). So you need to do as follows.
rnn_states = cell(current_states, rnn_states)[1]
You also made a mistake here. You forgot to add 1 to loop_counter_current.
return [loop_counter_current+1, rnn_states]
Add
First structure represents the initial value of parameter loop_vars you passed in, which contains the initial values of loop_counter_inital and initi_state. So its structure corresponds to the following.
[
<tf.Tensor 'while/Identity:0' shape=() dtype=int32> #---> loop_counter_inital
, <tf.Tensor 'while/Identity_1:0' shape=(2, 3) dtype=float32> #---> initi_state
]
Second structure represents the parameter loop_vars after a loop. Its results correspond to the following based on previous errors.
[
<tf.Tensor 'while/Identity:0' shape=() dtype=int32> #---> loop_counter_inital
, (<tf.Tensor 'while/basic_rnn_cell/Tanh:0' shape=(2, 3) dtype=float32> #---> output
, <tf.Tensor 'while/basic_rnn_cell/Tanh:0' shape=(2, 3) dtype=float32>) #---> initi_state
]

How to print all tf.Tensors of a model in a loop?

I have the list of all tensors in my model which had the float32 type:
import os
import os.path
import tensorflow as tf
from tensorflow.python.platform import gfile
import numpy
numpy.set_printoptions(threshold=numpy.nan)
with tf.Session() as sess:
model_filename = 'MY_pb_file.pb'
with gfile.FastGFile(model_filename, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
_= tf.import_graph_def(graph_def,name='')
from pprint import pprint
pprint([out for op in tf.get_default_graph().get_operations() if op.type != 'Placeholder' for out in op.values() if out.dtype == tf.float32])
which gives me all the list:
<tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/add:0' shape=(16,) dtype=float32>, <tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/Rsqrt:0' shape=(16,) dtype=float32>,
<tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/mul:0' shape=(16,) dtype=float32>,
<tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/mul_1:0' shape=(?, 64, 64, 16) dtype=float32>,
<tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/mul_2:0' shape=(16,) dtype=float32>,
<tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/sub:0' shape=(16,) dtype=float32>, <tf.Tensor 'MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/add_1:0' shape=(?, 64, 64, 16) dtype=float32>,
...
At this point I can use sess.run('NAME') to see its value:
>>> sess.run('MobilenetV1/MobilenetV1/Conv2d_1_pointwise/BatchNorm/batchnorm/mul:0')
array([ 0.51656026, 29.6620369 , 0.48722425, 7.73186255,
-9.51173401, 0.60846734, 0.21111809, 0.23865609,
23.85105324, 1.04134226, 28.59620476, 35.79195023,
0.34110394, 0.5557093 , 10.39805031, 10.99952412], dtype=float32)
However, I would like print all tf.Tensor values in a loop. How can I do that?
Apparently, some needs a dictionary to be defined:
sess.run('MobilenetV1/MobilenetV1/Conv2d_1_pointwise/Relu6:0')
For example:
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'input' with dtype float and shape [?,128,128,3]
[[Node: input = Placeholder[dtype=DT_FLOAT, shape=[?,128,128,3], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

When you request the value of a tensor, Tensorflow computes that value in the graph as tensor values are not usually retained across different calls to sess.run (that's what variables are for). It seems like the operations necessary to compute the value of the tensor you requested need input from one of your input placeholders (named input in the error statement), thus you must feed a value for that placeholder through the feed dictionary in sess.run.
Following your comment, consider this example:
import tensorflow as tf
a = tf.constant(4)
b = tf.constant(3)
c = tf.placeholder(tf.int32, [], 'c')
d = a + b
e = a + c
Requesting tensor d works fine:
with tf.Session() as sess:
print(sess.run(d)) # prints 7
However, requesting e throws the same error you report:
with tf.Session() as sess:
print(sess.run(e))
which prints
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'c' with dtype int32
[[Node: c = Placeholder[dtype=DT_INT32, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
This happens because in order to evaluate e we must compute a + c and if we don't feed a value to c, this is not possible. So, e.g., this works:
with tf.Session() as sess:
print(sess.run(e, feed_dict={c: 1})) # prints 5
Evaluating d worked fine because the computation path necessary to evaluate d does not involve a placeholder. So, to fix your problem, you should feed a value for the placeholder called 'input' in your call to sess.run('MobilenetV1/MobilenetV1/Conv2d_1_pointwise/Relu6:0').