Develop Reference

Tensorflow : Manually selecting the batch when training deep neural networks

# x_train.shape[0] = 54000
model.fit(
x_train, y_train,
batch_size = 128,
epochs = 12,
validation_data = (x_val, y_val)
)
When I am using this fit() method to train a neural network:
batch_size = 128 means that I randomly pick 54000 // 128 batches of size 128 in my training dataset every epoch.
Are those batches chosen with replacement? I suspect from the docs they're not but I'd like confirmation.
Can I manually choose my batches? I would like to focus on specific images and not others for a given batch, by choosing them personally instead of letting randomness choose for me.

Are those batches chosen with replacement?
In each individual epoch, no. Of course the entire dataset is used again in the next epoch.
Can I manually choose my batches? I would like to focus on specific images and not others for a given batch, by choosing them personally instead of letting randomness choose for me.
You should create a custom dataset for this, and leave the rest of the training loop (data loader, model etc.) unchanged.
But be aware that the samples in a minibatch are supposed to be random.

Progress bar not shown during training, Python

I'm trying to train a 2D Unet, for the segmentation task.
I execute this line of code:
model.fit(training_generator, epochs = params["nEpoches"],
validation_data=validation_generator, verbose = 1, use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])
Where
training_generator = Istance of DataGenerator(x_training, y_train_flat, **params), with the image and the masks array as parameters of this class.
epochs = 2
validation_generator = Istance of DataGenerator(x_validation, y_validation_flat, **params), with validation data.
callbacks_list = checkPoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, mode='min', period=1)
callbacks_list = checkPoint
With the verbose=1 parameter I think I should see a progress bar showing the training status for each epoch, but the only thing I see is Epoch 1/2, without any bar. So I can't say if the training process is going on or if it's stucked somewhere.

According to Tensorflow documentation,
steps_per_epoch:-
Integer or None. Total number of steps (batches of samples) before
declaring one epoch finished and starting the next epoch. When
training with input tensors such as TensorFlow data tensors, the
default None is equal to the number of samples in your dataset divided
by the batch size, or 1 if that cannot be determined. If x is a
tf.data dataset, and 'steps_per_epoch' is None, the epoch will run
until the input dataset is exhausted. When passing an infinitely
repeating dataset, you must specify the steps_per_epoch argument.
validation_steps:-
Only relevant if validation_data is provided and is a tf.data dataset.
Total number of steps (batches of samples) to draw before stopping
when performing validation at the end of every epoch. If
'validation_steps' is None, validation will run until the
validation_data dataset is exhausted. In the case of an infinitely
repeated dataset, it will run into an infinite loop. If
'validation_steps' is specified and only part of the dataset will be
consumed, the evaluation will start from the beginning of the dataset
at each epoch. This ensures that the same validation samples are used
every time.
In your case, training progress is going on, as rightly mentioned by #Kaveh, it does not know how much steps it should have for one epoch and ran into an infinite loop. Check your batch size and add steps_per_epoch and validation_steps to the model.fit() as shown below will resolve your issue.
model.fit(training_generator,
steps_per_epoch = len(training_generator) // training_generator.batch_size,
epochs = params["nEpoches"],
validation_data=validation_generator,
validation_steps=len(validation_generator) // validation_generator.batch_size,
verbose = 1,
use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])
For more information you can refer here

All training samples are not loading during training

I'm just starting with NLP. I loaded the 'imdb_reviews' dataset from tensorflow_datasets.
There were 25000 testing samples, but when I run I only train for 782 samples. I didn't use batch_size, just loaded entire dataset at once as you can see
The other hyperparameters are:
vocab_size = 10000
input_length = 120
embedding_dims = 16
Can anyone tell me what I'm doing wrong ?

By default the fit method of tf.keras.model will set the batch size to be 32.
https://www.tensorflow.org/api_docs/python/tf/keras/Model
As 32*782 = 25,024 it probably just drops the last batch.

tensorflow 2.0, model.fit() : Your input ran out of data

I am absolutely new to TensorFlow and Keras, and I am trying to make my way around trying out some code that I am finding online.
In particular I am using the fashion-MNIST - consisting of 60000 examples and test set of 10000 examples. Each of them is a 28x28 grayscale image.
I am following this tutorial "https://towardsdatascience.com/building-your-first-neural-network-in-tensorflow-2-tensorflow-for-hackers-part-i-e1e2f1dfe7a0", and I have no problem until the definition of
history = model.fit(
train_dataset.repeat(),
epochs=10,
steps_per_epoch=500,
validation_data=val_dataset.repeat(),
validation_steps=2)
As long as I understood, I need to use train_dataset.repeat() as input dataset because otherwise I won't have enough training example using those values for the hyperparameters (epochs, steps_per_epochs).
My question is: how can I avoid to have to use .repeat()?
How do I need to change the hyperparameters?
I am coping the code here, for simplicity:
def preprocess(x,y):
x = tf.cast(x,tf.float32) / 255.0
y = tf.cast(y, tf.float32)
return x,y
def create_dataset(xs, ys, n_classes=10):
ys = tf.one_hot(ys, depth=n_classes)
return tf.data.Dataset.from_tensor_slices((xs, ys)).map(preprocess).shuffle(len(ys)).batch(128)
model.compile(optimizer = 'adam', loss =tf.losses.CategoricalCrossentropy(from_logits= True), metrics =['accuracy'])
history1 = model.fit(train_dataset.repeat(),
epochs=10,
steps_per_epoch=500,
validation_data=val_dataset.repeat(),
validation_steps=2)
Thanks!

If you don't want to use .repeat() you need to have your model passing thought your entire data only one time per epoch.
In order to do that you need to calculate how many steps it will take for your model to pass throught the entire dataset, the calcul is easy :
steps_per_epoch = len(train_dataset) // batch_size
So with a train_dataset of 60 000 sample and a batch_size of 128, you need to have 468 steps per epoch.
By setting this parameter like that you make sure that you do not exceed the size of your dataset.

I encountered the same problem and here is what I found.
Documentation of tf.keras.Model.fit: "If x is a tf.data dataset, and 'steps_per_epoch' is None, the epoch will run until the input dataset is exhausted."
In other words, we don't need to specify 'steps_per_epoch' if we use the tf.data.dataset as the training data, and tf will figure out how many steps are there. Meanwhile, tf will automatically repeat the dataset when the next epoch begins, so you can specify any 'epoch'.
When passing an infinitely repeating dataset (e.g. dataset.repeat()), you must specify the steps_per_epoch argument.

What does nb_epoch in neural network stands for?

i'm currently beginning to discover Keras library for deap learning, it seems that in the training phase a centain number of epoch is chosen, but i don't know on which assumption is this choice based on.
In the Mnist dataset the number of epochs chosen is 4 :
model.fit(X_train, Y_train,
batch_size=128, nb_epoch=4,
show_accuracy=True, verbose=1,
validation_data=(X_test, Y_test))
Could someone tell me why and how do we choose a correct number of epochs ?

Starting Keras 2.0, nb_epoch argument has been renamed to epochs everywhere.
Neural networks are trained iteratively, making multiple passes over entire dataset. Each pass over entire dataset is referred to as epoch.
There are two possible ways to choose an optimum number of epochs:
1) Set epochs to a large number, and stop training when validation accuracy or loss stop improving: so-called early stopping
from keras.callbacks import EarlyStopping
early_stopping = EarlyStopping(monitor='val_loss', patience=4, mode='auto')
model.fit(X_train, Y_train,
batch_size=128, epochs=500,
show_accuracy=True, verbose=1,
validation_data=(X_test, Y_test),callbacks = [early_stopping])
2) Consider number of epochs as a hyperparameter and select the best value based on a set of trials (runs) on a grid of epochs values

it seems you might be using old version of keras ,nb_epoch refers to number of epochs which has been replaced by epoch
if you look here you will see that it has been deprecated.
One epoch means that you have trained all dataset(all records) once,if you have 384 records,one epoch means that you have trained your model for all on all 384 records.
Batch size means the data you model uses on single iteration,in that case,128 batch size means that at once,your model takes 128 and do some a single forward pass and backward pass(backpropation)[This is called one iteration]
.it
To break it down with this example,one iteration,your model takes 128 records[1st batch] from your whole 384 to be trained and do a forward pass and backward pass(back propagation).
on second batch,it takes from 129 to 256 records and do another iteration.
then 3rd batch,from 256 to 384 and performs the 3rd iteration.
In this case,we say that it has completed one epoch.
the number of epoch tells the model the number it has to repeat all those processes above then stops.
There is no correct way to choose a number of epoch,its something that is done by experimenting,usually when the model stops to learn(loss is not going down anymore) you usually decrease the learning rate,if it doesn't go down after that and the results seems to be more or less as you expected then you select at that epoch where the model stopped to learn
I hope it helps

In neural networks, an epoch is equivalent to training the network using each data once.
The number of epochs, nb_epoch, is hence how many times you re-use your data during training.