Develop Reference

KAGGLE BERT : ValueError: Shapes (None, 2) and (None, 3) are incompatible

I'm trying to follow a kaggle for the BERT model : https://www.kaggle.com/code/ludovicocuoghi/twitter-sentiment-analysis-with-bert-roberta/notebook
And at the step near the end I get this error :
ValueError: Shapes (None, 2) and (None, 3) are incompatible
The cell :
history_bert = model.fit([train_input_ids,train_attention_masks], y_train, validation_data=([val_input_ids,val_attention_masks], y_valid), epochs=4, batch_size=32)
I tried many things but I didnt find any solution If someone cane enlight me it'll be useful !
the shape of differents variables :
train_input_ids SHAPE: (640740, 128)
train_attention_masks SHAPE: (640740, 128)
y_train SHAPE: (640740, 2)
val_input_ids SHAPE: (71194, 128)
val_attention_masks SHAPE: (71194, 128)
y_valid SHAPE: (71194, 2)
In my dataset Sentiment has only 2 value 0 / 1 idk if this may have an impact on the error
I switched the variable's place in the functions fit.
I search for similar errors on differents forums to find a solution.

InvalidArgumentError: required broadcastable shapes [Op:Add] tensorflow model

I am a using CLIP model. Where I have two models. One model output is (20, 128, 256) and the other one output is (20, 256).
image_model_output = (20, 256)
text_model_output = (20, 128, 256)
I use the following to calculate this
logits = (tf.matmul(caption_embeddings, image_embeddings, transpose_b=True))
so it will be like `(20, 256) * (256, 128, 20)`
it's ouput will be `(20, 128, 20)`
Similarly I calculate like this
images_similarity = tf.matmul(
image_embeddings, image_embeddings, transpose_b=True
)
(Output)--> (20, 256) * (256, 20) = (20,20)
and this
captions_similarity = tf.matmul(
caption_embeddings, caption_embeddings, transpose_b=True
)
(Output)--> (20, 128, 256) * (256, 128, 20) = (20, 128, 128)
The problem arises here
targets = keras.activations.softmax(
(captions_similarity + images_similarity) / (2 * self.temperature)
)
So do I need to change the activation function or there is any way to add these 3d matrices with different shapes?
Sorry to technically explain like this but people with solid deep learning and machine learning backgorund will understand.
NOTE: After adding axis 1 like this tf.expand_dims(image_embeddings, axis=1) the below part runs successfully
targets = keras.activations.softmax(
(captions_similarity + images_similarity) / (2 * self.temperature)
)
However after this there is a loss funtion like below
captions_loss = keras.losses.categorical_crossentropy(
y_true=targets, y_pred=logits, from_logits=True
)
which generates this error
ValueError: Shapes (2, 128, 128) and (2, 128, 1) are incompatible
Is it possible to solve this error?

To handle the above error I used a different loss funtion. I changed the code like below.
captions_loss = keras.losses.categorical_crossentropy(
y_true=targets, y_pred=logits, from_logits=True
)
To
captions_loss = keras.losses.kl_divergence(
y_true=targets, y_pred=logits
)
To save time of developers I have answered to my own. I am available to discuss on it further if someone is interested.

Invalid argument: indices[124,0] = 2629 is not in [0, 64) (multi-input model)

I am following Deep Learning with Python section 7.1.2 Multi-input models. Here on code of Listing 7.1, I am facing following errors:
InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: indices[124,0] = 2629 is not in [0, 64)
[[node functional_11/embedding_8/embedding_lookup (defined at E:/Studies/PythonCode_DLBook/Codes/Chap7_Code2.py:30) ]]
[[functional_11/embedding_9/embedding_lookup/_16]]
(1) Invalid argument: indices[124,0] = 2629 is not in [0, 64)
[[node functional_11/embedding_8/embedding_lookup (defined at E:/Studies/PythonCode_DLBook/Codes/Chap7_Code2.py:30) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_29208]
Errors may have originated from an input operation.
Input Source operations connected to node functional_11/embedding_8/embedding_lookup:
functional_11/embedding_8/embedding_lookup/26947 (defined at C:\Users\abdul\anaconda3\envs\PIAIC\lib\contextlib.py:113)
Input Source operations connected to node functional_11/embedding_8/embedding_lookup:
functional_11/embedding_8/embedding_lookup/26947 (defined at C:\Users\abdul\anaconda3\envs\PIAIC\lib\contextlib.py:113)
Function call stack:
train_function -> train_function
The code used is:
from tensorflow.keras.models import Model
from tensorflow.keras import layers
from tensorflow.keras import Input
import numpy as np
text_vocabulary_size = 10000
question_vocabulary_size = 10000
answer_vocabulary_size = 500
text_input = Input(shape=(100,), dtype='int32', name='text')
embedded_text = layers.Embedding(64, text_vocabulary_size)(text_input)
encoded_text = layers.LSTM(32)(embedded_text)
question_input = Input(shape=(100,),dtype='int32',name='question')
embedded_question = layers.Embedding(32, question_vocabulary_size)(question_input)
encoded_question = layers.LSTM(16)(embedded_question)
concatenated = layers.concatenate([encoded_text, encoded_question],axis=-1)
answer = layers.Dense(answer_vocabulary_size,
activation='softmax')(concatenated)
model = Model([text_input, question_input], answer)
model.compile(optimizer='rmsprop',loss='categorical_crossentropy',metrics=['acc'])
num_samples = 1000
max_length = 100
text = np.random.randint(1, text_vocabulary_size,size=(num_samples, max_length))
question = np.random.randint(1, question_vocabulary_size,size=(num_samples, max_length))
answers = np.random.randint(0, 1,size=(num_samples, answer_vocabulary_size))
model.fit([text, question], answers, epochs=10, batch_size=128)
model.fit({'text': text, 'question': question}, answers,epochs=10, batch_size=128)
I do realize that error is on embedded_text layer because its input mismatch with the data shape coming in it.
However I don't know how to solve this problem, in-fact I don't know at the moment how to set/check for input data shapes and data shapes between different layers.
So it would be really helpful if someone shows how to check layer shapes while designing the model and how to resolve these kind of issues.

For TF, a common method is to use model.summary() to check for the output shape at each layer of the network. Running your code returns
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
text (InputLayer) [(None, 100)] 0
__________________________________________________________________________________________________
question (InputLayer) [(None, 100)] 0
__________________________________________________________________________________________________
embedding (Embedding) (None, 100, 10000) 1000000 text[0][0]
__________________________________________________________________________________________________
embedding_1 (Embedding) (None, 100, 10000) 1000000 question[0][0]
__________________________________________________________________________________________________
lstm (LSTM) (None, 100) 4040400 embedding[0][0]
__________________________________________________________________________________________________
lstm_1 (LSTM) (None, 100) 4040400 embedding_1[0][0]
__________________________________________________________________________________________________
concatenate (Concatenate) (None, 200) 0 lstm[0][0]
lstm_1[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 500) 100500 concatenate[0][0]
==================================================================================================
Total params: 10,181,300
Trainable params: 10,181,300
Non-trainable params: 0
So this will be the first step to troubleshoooting, if you would like to see the expected input shape to each layer, model.get_config() is one way to do it. I would refer you to the question here.
Moreover, I would suggest that you read the documentations for layers.LSTM and layers.Embedding to get a solid grasp of the parameters you are passing in and the layer you are creating. Hope this helps with the troubleshooting process :)

any way to rescale and stack features maps with different shape in tensorflow?

I intend to use the concepts of skip connection in my experiment. Basically, in my pipeline, features maps that comes after Conv2D are going to be stacked or concatenated. But, features maps are in different shape and try to stack them together into one tensor gave me error. Does anyone knows any possible way of doing this correctly in tensorflow? Any thoughts or ideas to make this happen? Thanks
idea flowchart
here is the pipeline flowchart I want to do it:
my case is little different because I got extra building block is used after Conv2D and its output now is feature maps of 15x15x64 and so on. I want to stack those features map into one then use it to Conv2D again.
my attempt:
this is my reproducible attempt:
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout, Activation, Conv2D, Flatten, MaxPool2D, BatchNormalization
inputs = tf.keras.Input(shape=(32, 32, 3))
x = inputs
x = Conv2D(32, (3, 3), input_shape=(32,32,3))(x)
x = BatchNormalization(axis=-1)(x)
x = Activation('relu')(x)
fm1 = MaxPooling2D(pool_size=(2,2))(x)
x = Conv2D(32,(3, 3), input_shape=(15,15,32))(fm1)
x = BatchNormalization(axis=-1)(x)
x = Activation('relu')(x)
fm2 = MaxPooling2D(pool_size=(2,2))(x)
concatted = tf.keras.layers.Concatenate(axis=1)([fm1, fm2])
but this way I ended up with following error: ValueError: A Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 15, 15, 32), (None, 6, 6, 32)]. I am not sure what would be correct way to stack features maps with different shape. How can we make this right? Any possible thoughts?
desired output
in my actual model, I got shape of features maps are TensorShape([None, 15, 15, 128]) and TensorShape([None, 6, 6, 128]). I need to find way to merge them or stack them into one. Ideally, shape of concatenated or stacked feature maps' shape would be: [None, 21,21,128]. Is there any way of stacking them into one? Any idea?

What you're trying to achieve doesn't work mathematically. Let me illustrate. Take the simple 1D problem (like 1D convolution). You have a (None, 64, 128) (fm1) sized output and a (None, 32, 128) (fm2) output that you want to concatenate. Then,
concatted = tf.keras.layers.Concatenate(axis=1)([fm1, fm2])
works totally fine, giving you an output of size (None, 96, 128).
Let's come to the 2D problem. Now you got two tensors (None, 15, 15, 128) and (None, 6, 6, 128) and want to end up with a (None, 21, 21, 128) sized output. Well the math doesn't work here. To understand why, reduce this to 1D format. Then you got
fm1 -> (None, 225, 128)
fm2 -> (None, 36, 128)
By concat you get,
concatted -> (None, 261, 128)
If the math works you should get (None, 441, 128) which is reshape-able to (None, 21, 21, 128). So this cannot be achieved unless you pad the edges of the smaller with 441-261 = 180 on the reshaped tensor. And then reshape it to the desired shape. Following is an example of how you can do it,
concatted = tf.keras.layers.Lambda(
lambda x: K.reshape(
K.concatenate(
[K.reshape(x[0], (-1, 225, 128)),
tf.pad(
K.reshape(x[1], (-1, 36, 128)), [(0,0), (0, 180), (0,0)]
)
], axis=1
), (-1, 21, 21, 128))
)([fm1, fm2])
Important: But I can't guaranttee the performance of your model this just solves your problem mathematically. In a machine learning perspective, I wouldn't advice this. Best way would be making sure the outputs are compatible in sizes for concatenation. Few ways would be,
Not reduce the size of convolution outputs (stride = 0 and padding='same')
Use transpose convolution operation to size-up the smaller one

Python Array Reshaping Issue to Array with Shape (None, 192)

I have this error and I'm not sure how do I reshape where there's a dimension with None.
Exception: Error when checking : expected input_1 to have shape (None, 192) but got array with shape (192, 1)
How do I reshape an array to (None, 192)?
I've the array accuracy with shape (12, 16) and I did accuracy.reshape(-1) that gives (192,). However this is not (None, 192).

In keras/keras/engine/training.py
def standardize_input_data(data, names, shapes=None,
check_batch_dim=True,
exception_prefix=''):
...
# check shapes compatibility
if shapes:
for i in range(len(names)):
...
for j, (dim, ref_dim) in enumerate(zip(array.shape, shapes[i])):
if not j and not check_batch_dim:
# skip the first axis
continue
if ref_dim:
if ref_dim != dim:
raise Exception('Error when checking ' + exception_prefix +
': expected ' + names[i] +
' to have shape ' + str(shapes[i]) +
' but got array with shape ' +
str(array.shape))
Comparing that with the error
Error when checking : expected input_1 to have shape (None, 192) but got array with shape (192, 1)
So it is comparing (None, 192) with (192, 1), and skipping the 1st axis; that is comparing 192 and 1. If array has shape (n, 192) it probably would pass.
So basically, what ever is generating the (192,1) shape, as opposed to (1,192) or a broadcastable (192,) is causing the error.
I'm adding keras to the tags on the guess that this is the problem module.
Searching other keras tagged SO questions:
Exception: Error when checking model target: expected dense_3 to have shape (None, 1000) but got array with shape (32, 2)
Error: Error when checking model input: expected dense_input_6 to have shape (None, 784) but got array with shape (784L, 1L)
Dimensions not matching in keras LSTM model
Getting shape dimension errors with a simple regression using Keras
Deep autoencoder in Keras converting one dimension to another i
I don't know enough about keras to understand the answers, but there's more to it than simply reshaping your input array.