Develop Reference

Tensorflow-datasets: Cannot batch tensors of different shapes error even after resize?

Having some trouble with the tensorflow-datasets module. Using the stanford_dogs dataset, I resize images to [180,180], yet when the model is trained, from the error message, it appears tensorflow is trying to load the images in their original sizes.
What am I doing wrong?
Code to replicate error (and the error) below. Dataset is around 750mb. Can copy paste it into google colab and run to replicate.
import io
import numpy as np
import tensorflow as tf
import tensorflow_datasets as tfds
def _normalize_img(img, label):
img = tf.cast(img, tf.float32) / 255.
img = tf.image.resize(img,[180,180])
return (img, label)
train_dataset, test_dataset = tfds.load(name="stanford_dogs", split=['train', 'test'], as_supervised=True)
train_dataset = train_dataset.shuffle(1024).batch(32)
train_dataset = train_dataset.map(_normalize_img)
test_dataset = test_dataset.batch(32)
test_dataset = test_dataset.map(_normalize_img)
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(64,2,padding='same',activation='relu',input_shape=(180,180,3)),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Conv2D(32,2,padding='same',activation='relu'),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(120,activation='softmax')
])
model.compile(
optimizer=tf.keras.optimizers.Adam(0.001),
loss='sparse_categorical_crossentropy')
history = model.fit(
train_dataset,
epochs=5)
Fails with the error:
InvalidArgumentError: Cannot batch tensors with different shapes in component 0. First element had shape [278,300,3] and element 1 had shape [375,500,3].
[[node IteratorGetNext (defined at <ipython-input-29-15023f95f627>:39) ]] [Op:__inference_train_function_4908]

You encounter this error because the tf.data.Dataset API cannot create a batch of tensors with different shapes. As the batch function will return Tensors of shape (batch, height, width, channels), the height, width and channels values must be constant throughout the dataset. You can read more about why in the Introduction to Tensors guide.
Batching after resizing will solve your issue :
train_dataset = train_dataset.shuffle(1024)
train_dataset = train_dataset.map(_normalize_img)
# we batch once every image is the same size
train_dataset = train_dataset.batch(32)

Combine feature and labels to correctly produce tf dataset for model.fit (Using tf.data.Dataset.from_tensor_slices)

I have created a model that takes an input of shape (None, 512). Below is the summary of my model
shape of training feature
train_ids.shape
(10, 512)
shape of the training response variable
indus_cat_train.shape
(10, 49)
My model runs perfectly if I use
history = model.fit(
train_ids, indus_cat_train, epochs=2, validation_data=(
valid_ids, indus_cat_valid))
However my actual dataset is very large and feeding the completed dataset all at once is consuming so much RAM and shut down all the process.
I want to feed all data in batches or one by one. In order to complete this task, I tried out tf.data.Dataset.from_tensor_slices function
# training data
tf_train_data = tf.data.Dataset.from_tensor_slices((train_ids, indus_cat_train))
# validation data
tf_valid_data = tf.data.Dataset.from_tensor_slices((valid_ids, indus_cat_valid))
The above code is running fine and upon inspection, it is giving the desired shape
for elem in t:
print(elem[0].shape) # for features
print(elem[1].shape) # for response
print output
(512,) # for features
(49,) # for response variable
# terminating all other output to save space
However on-calling model.fit on tf_train_dataset the model gives me an error
bert_history = model.fit(
tf_train_data, epochs=2, validation_data=tf_valid_data)
WARNING:tensorflow:Model was constructed with shape (None, 512) for input Tensor("input_ids_1:0", shape=(None, 512), dtype=int32), but it was called on an input with incompatible shape (512, 1).
Sharing model code for further understanding as asked by Prateek
# training data
tf_train_data = tf.data.Dataset.from_tensor_slices((train_ids, indus_cat_train))
# validation data
tf_valid_data = tf.data.Dataset.from_tensor_slices((valid_ids, indus_cat_valid))
# model downloaded from bert
bert_model_name = "uncased_L-12_H-768_A-12"
bert_ckpt_dir = "bert_model"
bert_ckpt_file = os.path.join(bert_ckpt_dir, "bert_model.ckpt")
bert_config_file = os.path.join(bert_ckpt_dir, "bert_config.json")
# creating tokenizer
tokenizer = FullTokenizer(vocab_file=os.path.join(bert_ckpt_dir, "vocab.txt"))
# create function for model
def create_model(max_seq_len, bert_ckpt_file, n_classes):
with tf.io.gfile.GFile(bert_config_file, "r") as reader:
# get bert configurations
bert_configurations = StockBertConfig.from_json_string(reader.read())
bert_params = map_stock_config_to_params(bert_configurations)
bert_params_adapter_size = None
bert = BertModelLayer.from_params(bert_params, name="bert")
input_ids = keras.layers.Input(shape=(max_seq_len,), dtype="int32",
name="input_ids")
bert_output = bert(input_ids)
print("bert shape", bert_output.shape)
cls_out = keras.layers.Lambda(lambda seq: seq[:, 0, :])(bert_output)
cls_out = keras.layers.Dropout(0.5)(cls_out)
logits = keras.layers.Dense(units=765, activation="tanh")(cls_out)
logits = keras.layers.Dropout(0.5)(logits)
logits = keras.layers.Dense(
units=n_classes, activation="softmax")(logits)
model = keras.Model(inputs=input_ids, outputs=logits)
model.build(input_shape=(None, max_seq_len))
load_stock_weights(bert, bert_ckpt_file)
return model
n_cats = 49 #number of output categories
model = create_model(max_seq_len=512, bert_ckpt_file=bert_ckpt_file,
n_classes=n_cats)
model.summary()
optimizer = tf.keras.optimizers.Adam( learning_rate=learning_rate, epsilon=1e-08)
loss = tf.keras.losses.CategoricalCrossentropy()metric = tf.keras.metrics.CategoricalCrossentropy( name='categorical_crossentropy')model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
bert_history = model.fit( tf_train_data, epochs=2, validation_data=tf_valid_data)

I have solved it using dataset.batch. tf.data.Dataset was missing the batch size arguments as a result of which supplied tensors are not batched, i.e. I was getting shape (512,1) instead of (512,) and (49,1) instead of (49,)
batch_size = 2
tf_train_data = tf.data.Dataset.from_tensor_slices((train_ids,
indus_cat_train)).batch(batch_size)
tf_valid_data = tf.data.Dataset.from_tensor_slices((valid_ids,
indus_cat_valid)).batch(batch_size)
bert_history = model.fit(
tf_train_data, epochs=2, validation_data=tf_valid_data)

Keras: Trying to model.predict() gives "ValueError: Tensor's shape is not compatible with supplied shape"

I'm following the TensorFlow Keras tutorial for text generation. The training part works perfectly, but when I try to predict the next token, I get an error.
Here's all the important code:
Making the vocabulary and dataset.
vocab = sorted(set(text))
char2index = { c:i for i, c in enumerate(vocab) }
index2char = np.array(vocab)
chars_to_int = np.array([char2index[c] for c in text])
char_dataset = tf.data.Dataset.from_tensor_slices(chars_to_int)
sequences = char_dataset.batch(seq_length + 1, drop_remainder=True)
def split_input_and_target(sequence):
input_ = sequence[:-1]
target_ = sequence[1:]
return input_, target_
dataset = sequences.map(split_input_and_target)
dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)
Building the model
(important part here is that BATCH_SIZE = 64):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(len(vocab), EMBEDDING_DIM,
batch_input_shape=[BATCH_SIZE, None]))
# here are a few more layers
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")
model.fit(dataset, epochs=EPOCHS)
Actually trying to generate text (this one was copied almost directly from the tutorial after I started getting desperate):
num_tokens = 100
seed = "some text"
input_eval = [char2index[c] for c in seed]
input_eval = tf.expand_dims(input_eval, 0)
text_generated = []
model.reset_states()
for i in range(num_tokens):
predictions = model(input_eval)
predictions = tf.squeeze(predictions, 0)
# more stuff
Then, I first get a warning:
WARNING:tensorflow:Model was constructed with shape (64, None) for input Tensor("embedding_14_input:0", shape=(64, None), dtype=float32), but it was called on an input with incompatible shape (1, 9).
Then it gives me an error:
---->3 predictions = model(input_eval)
...
ValueError: Tensor's shape (9, 64, 256) is not compatible with supplied shape [9, 1, 256]
The second number, 64, is my batch size. If I change BATCH_SIZE to 1, everything works and all is fine, but this is obviously not the solution I am hoping for.

(I somehow managed to miss a step in the tutorial despite reading it several times over the past few hours.)
Here's the relevant passage:
To keep this prediction step simple, use a batch size of 1.
Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.
To run the model with a different batch_size, we need to rebuild the model and restore the weights from the checkpoint.
tf.train.latest_checkpoint(checkpoint_dir)
model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
model.build(tf.TensorShape([1, None]))
I hope my silly mistake will help somebody to remember to reload the model in the future!

Behavior difference when building TF Keras RNN with two different methods

I am building a RNN text generator, mostly going from the Tensorflow docs here.
My question, I have defined the model two ways:
Method (1):
model = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab_size, embed_dim,
batch_input_shape=[batch_size, None]),
tf.keras.layers.GRU(rnn_units, return_sequences=True, stateful=True,
recurrent_initializer='glorot_uniform'),
tf.keras.layers.Dense(vocab_size)
])
Method (2):
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(vocab_size, embed_dim,
batch_input_shape=[BATCH_SIZE, None]))
model.add(tf.keras.layers.GRU(rnn_units, return_sequences=True,
stateful=True,
recurrent_initializer='glorot_uniform'))
model.add(tf.keras.layers.Dense(vocab_size))
In my mind, these both do the same thing. However when generating text with:
def generate_text(model, start_string, length=1000):
# converting start string to numbers (vectorisation)
input_eval = [char2idx[s] for s in start_string]
input_eval = tf.expand_dims(input_eval, 0)
# initialise empty string to store results
text = []
model.reset_states()
for i in range(length):
predictions = model(input_eval)
# remove batch dimension
predictions = tf.squeeze(predictions, 0)
# use categorical distribution to predict character returned by model
predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()
# we pass the predicted character as the next input to the model
# along with previous hidden state
input_eval = tf.expand_dims([predicted_id], 0)
# append predicted text
text.append(idx2char[predicted_id])
return (start_string + ''.join(text))
Which I pass:
print(generate_text(model, start_string=u'From '))
Method (1) works perfectly, but method (2) throws the following error:
WARNING:tensorflow:Model was constructed with shape Tensor("embedding_1_input:0", shape=(64, None), dtype=float32) for input (64, None), but it was re-called on a Tensor with incompatible shape (1, 5).
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-33-eb814780c9fe> in <module>()
----> 1 print(generate_text(model, start_string=u'From ', length=PRINT))
14 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py in set_shape(self, shape)
1086 raise ValueError(
1087 "Tensor's shape %s is not compatible with supplied shape %s" %
-> 1088 (self.shape, shape))
1089
1090 # Methods not supported / implemented for Eager Tensors.
ValueError: Tensor's shape (5, 64, 1024) is not compatible with supplied shape [5, 1, 1024]
If anyone could help me understand what the difference is between these two methods that would be amazing, thankyou!
Edit:
Including model saving and loading code. I use this to save the model (with a batch size 64) and then load with a batch size of 1 for text generation.
Saving weights:
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
filepath='./training_checkpoints/ckpt_{epoch}',
save_weights_only=True
)
Loading weights into new model (batch size = 1):
model = build_model(len(vocab_char), EMBED_DIM, UNITS, 1)
model.load_weights(tf.train.latest_checkpoint('./training_checkpoints'))
model.build(tf.TensorShape([1, None]))
model.summary()

fine tune a model using Keras Functional API

I am using VGG16 to finetune it on my dataset.
Here's the model:
def finetune(self, aux_input):
model = applications.VGG16(weights='imagenet', include_top=False)
# return model
drop_5 = Input(shape=(7, 7, 512))
flatten = Flatten()(drop_5)
# aux_input = Input(shape=(1,))
concat = Concatenate(axis=1)([flatten, aux_input])
fc1 = Dense(512, kernel_regularizer=regularizers.l2(self.weight_decay))(concat)
fc1 = Activation('relu')(fc1)
fc1 = BatchNormalization()(fc1)
fc1_drop = Dropout(0.5)(fc1)
fc2 = Dense(self.num_classes)(fc1_drop)
top_model_out = Activation('softmax')(fc2)
top_model = Model(inputs=drop_5, outputs=top_model_out)
output = top_model(model.output)
complete_model = Model(inputs=[model.input, aux_input], outputs=output)
return complete_model
I have two inputs to the model. In the above function, I'm using Concatenate for the flattened array and my aux_input.
I'm not sure if this would work with imagenet weights.
When I run this, I get an error:
ValueError: Graph disconnected: cannot obtain value for tensor
Tensor("aux_input:0", shape=(?, 1), dtype=float32) at layer
"aux_input". The following previous layers were accessed without
issue: ['input_2', 'flatten_1']
Not sure where am I going wrong.
If it matters, this is fit function:
model.fit(x={'input_1': x_train, 'aux_input': y_aux_train}, y=y_train, batch_size=batch_size,
epochs=maxepoches, validation_data=([x_test, y_aux_test], y_test),
callbacks=[reduce_lr, tensorboard], verbose=2)
But, I get an error before this fit function when I call model.summary().

The problem is that you are using aux_input in your top_model but you don't specify it as an input in your definition of top_model. Try replacing your definition of top_model and output with the following:
top_model = Model(inputs=[drop_5, aux_input], outputs=top_model_out)
output = top_model([model.output, aux_input])