Develop Reference

tensorflow callbacks.EarlyStopping not working

I'm trying to understand why tf.keras.callbacks.EarlyStopping isn't behaving as expected.
In the simple reproducible example below, I'd expect training to stop after validation loss has been consistently decreasing for 3 epochs. You can see that is the case from epoch 5.
import tensorflow as tf
from tensorflow.keras import models, layers
import numpy as np
tf.random.set_seed(42)
# make 100 400-dimensional vectors, representing features:
x = np.full((100, 400), 7)
x = np.expand_dims(x, 1)
# make 100 800-dimensional vectors, representing labels:
y = np.full((100, 800), 1)
y = np.expand_dims(y, 1)
model = models.Sequential([
layers.Input((None,400)),
layers.Dense(800)
])
model.compile(loss='mse', optimizer='adam')
model.summary()
history = model.fit(x=x,
y=y,
epochs=100,
validation_split=0.2,
callbacks=tf.keras.callbacks.EarlyStopping(monitor='val_loss', verbose=1, patience=3)
)
Why does the model keeps training until the 100th epoch?

Why my Keras model is not loading through all 5 epochs as specified?

train_length = 972
from keras.callbacks import ModelCheckpoint, EarlyStopping
history = model.fit(datagenerator.flow(x_train,y_train, batch_size = 32) ,
steps_per_epoch = 30 , epochs = 5, validation_data = datagenerator.flow(x_test, y_test))
I will also provide the construction of the model itself below.
from keras.models import Sequential
from keras.models import Model
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau, TensorBoard
from keras import optimizers, losses, activations, models
from keras.layers import Convolution2D, Dense, Input, Flatten, Dropout, MaxPooling2D, BatchNormalization, GlobalAveragePooling2D, Concatenate
from keras import applications
input_shape = (128,128, 3)
base_model = keras.applications.InceptionResNetV2(weights='imagenet',
include_top=False,
input_shape=(128,128,3))
base_model.trainable = False
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(2048, activation='relu')(x)
x= BatchNormalization()(x)
x=Dropout(0.3)(x)
x= Dense(64,activation='relu')(x)
x= BatchNormalization()(x)
x=Dropout(0.3)(x)
predictions = Dense(2, activation='softmax')(x)
for layer in base_model.layers:
layer.trainable = False
model=Sequential()
model = Model(inputs=base_model.input, outputs=predictions)
model.compile(loss='sparse_categorical_crossentropy',
optimizer='RMSprop',
metrics=['accuracy'])
I am not sure how to implement validation steps, which I heard is a solution, but I get errors when I do so. How can I edit my code such that it goes through all 5 epochs instead of halting after just one? I will post an image of what it looks like after I run it.
Here is the link: https://imgur.com/BJJPZng

here is a way to determine the batch_size and steps.
length=len(x_train)
b_max= 80 # set this based on how much your memory can hold
batch_size=sorted([int(length/n) for n in range(1,length+1) if length % n ==0 and
length/n<=b_max],reverse=True)[0]
steps=int(length/batch_size)
do the same with x_test to get the validation batch size and validation steps.
For example if length=972 and b_max=80 then result is batch_size=54 and steps=18
if you set b_max=40 you get batch size=36 and steps=27.
These values ensure you got through your training set or validation exactly once per epoch. Note if length is a prime number, batch_size=1 and steps=length

Multi class classification using Bi-LSTM and glove

I am doing multi class classification using BI LSTM and glove embeddings, when I train my model, on the prediction (model.predict) I get not proper results as below, the results are not between 0 and 1, can anyone help me please?
I also used one hot encoded.
3916/3916 [==============================] - 17s 4ms/step
[[9.9723792e-01 1.6954101e-03 1.0665554e-03]
[1.6794224e-01 8.6485274e-02 7.4557245e-01]
[9.4370516e-03 1.0848863e-03 9.8947805e-01]
...
[1.3264662e-02 9.7078091e-01 1.5954463e-02]
[1.2019513e-02 9.8711687e-01 8.6356810e-04]
[8.1863362e-01 1.5828104e-01 2.3085352e-02]]
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM,Dense, Dropout,Bidirectional
from tensorflow.keras.layers import SpatialDropout1D
from tensorflow.keras.layers import Embedding
from tensorflow.keras.preprocessing.text import Tokenizer
embedding_vector_length = 100
model_2 = Sequential()
model_2.add(Embedding(len(tokenizer.word_index) + 1, embedding_vector_length,
input_length=409,name="Bi-LSTM") )
model_2.add(SpatialDropout1D(0.3))
model_2.add(Bidirectional(LSTM(64, return_sequences=False, recurrent_dropout=0.4)))
model_2.add(Dropout(0.5))
model_2.add(Dense(3,activation='softmax'))
model_2.compile(loss='categorical_crossentropy',optimizer='adam',
metrics=['accuracy'])
print(model_2.summary())
model_2.layers[0].set_weights([embedding_matrix])
model_2.layers[0].trainable = False
print(model_2.summary())
from keras.callbacks import EarlyStopping
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=3)
history_2=model_2.fit(x_train, y_train,
batch_size=400,
epochs=30,
validation_data=(x_val, y_val),
callbacks=[es])
#We save this model so that we can use in own web app

If the printed matrix is your model.predict() results - they are between 0 an 1 (you need to take exponential part into account)

Same model produces consistently different accuracies in Keras and Tensorflow

I'm trying to implement the same model in Keras, and in Tensorflow using Keras layers, using custom data. The two models produce consistently different accuracies over many times of training (keras ~71%, tensorflow ~65%). I want tensorflow to do as well as keras so I can go into the tensorflow iterations to tweak some lower level algorithms.
Here's my original Keras code:
from keras.layers import Dense, Dropout, Input
from keras.models import Model, Sequential
from keras import backend as K
input_size = 2000
num_classes = 4
num_industries = 22
num_aux_inputs = 3
main_input = Input(shape=(input_size,),name='text_vectors')
x = Dense(units=64, activation='relu', name = 'dense1')(main_input)
drop1 = Dropout(0.2,name='dropout1')(x)
auxiliary_input = Input(shape=(num_aux_inputs,), name='aux_input')
x = keras.layers.concatenate([drop1,auxiliary_input])
x = Dense(units=64, activation='relu',name='dense2')(x)
drop2 = Dropout(0.1,name='dropout2')(x)
x = Dense(units=32, activation='relu',name='dense3')(drop2)
main_output = Dense(units=num_classes,
activation='softmax',name='main_output')(x)
model = Model(inputs=[main_input, auxiliary_input],
outputs=main_output)
model.compile(loss=keras.losses.categorical_crossentropy, metrics= ['accuracy'],optimizer=keras.optimizers.Adadelta())
history = model.fit([train_x,train_x_auxiliary], train_y, batch_size=128, epochs=20, verbose=1, validation_data=([val_x,val_x_auxiliary], val_y))
loss, accuracy = model.evaluate([val_x,val_x_auxiliary], val_y, verbose=0)
Here's I moved the keras layers to tensorflow following this article:
import tensorflow as tf
from keras import backend as K
import keras
from keras.layers import Dense, Dropout, Input # Dense layers are "fully connected" layers
from keras.metrics import categorical_accuracy as accuracy
from keras.objectives import categorical_crossentropy
tf.reset_default_graph()
sess = tf.Session()
K.set_session(sess)
input_size = 2000
num_classes = 4
num_industries = 22
num_aux_inputs = 3
x = tf.placeholder(tf.float32, shape=[None, input_size], name='X')
x_aux = tf.placeholder(tf.float32, shape=[None, num_aux_inputs], name='X_aux')
y = tf.placeholder(tf.float32, shape=[None, num_classes], name='Y')
# build graph
layer = Dense(units=64, activation='relu', name = 'dense1')(x)
drop1 = Dropout(0.2,name='dropout1')(layer)
layer = keras.layers.concatenate([drop1,x_aux])
layer = Dense(units=64, activation='relu',name='dense2')(layer)
drop2 = Dropout(0.1,name='dropout2')(layer)
layer = Dense(units=32, activation='relu',name='dense3')(drop2)
output_logits = Dense(units=num_classes, activation='softmax',name='main_output')(layer)
loss = tf.reduce_mean(categorical_crossentropy(y, output_logits))
acc_value = tf.reduce_mean(accuracy(y, output_logits))
correct_prediction = tf.equal(tf.argmax(output_logits, 1), tf.argmax(y, 1), name='correct_pred')
optimizer = tf.train.AdadeltaOptimizer(learning_rate=1.0, rho=0.95,epsilon=tf.keras.backend.epsilon()).minimize(loss)
init = tf.global_variables_initializer()
sess.run(init)
epochs = 20 # Total number of training epochs
batch_size = 128 # Training batch size
display_freq = 300 # Frequency of displaying the training results
num_tr_iter = int(len(y_train) / batch_size)
with sess.as_default():
for epoch in range(epochs):
print('Training epoch: {}'.format(epoch + 1))
# Randomly shuffle the training data at the beginning of each epoch
x_train, x_train_aux, y_train = randomize(x_train, x_train_auxiliary, y_train)
for iteration in range(num_tr_iter):
start = iteration * batch_size
end = (iteration + 1) * batch_size
x_batch, x_aux_batch, y_batch = get_next_batch(x_train, x_train_aux, y_train, start, end)
# Run optimization op (backprop)
feed_dict_batch = {x: x_batch, x_aux:x_aux_batch, y: y_batch,K.learning_phase(): 1}
optimizer.run(feed_dict=feed_dict_batch)
I also implemented the whole model from scratch in tensorflow, but it also is a ~65% accuracy, so I decided to try this Keras-layers-within-TF set up to identify problems.
I've looked up posts on similar problems with Keras and Tensorflow, and have tried the following which didn't help in my case:
Keras's dropout layer is only active in the training phase, so I did the same in my tf code by setting keras.backend.learning_phase().
Keras and Tensorflow have different variable initializations. I've tried initializing my weights in tensorflow these following 3 ways, which is supposed to be the same as Keras's weight initialization, but they also didn't affect the accuracies:
initer = tf.glorot_uniform_initializer()
initer = tf.contrib.layers.xavier_initializer()
initer = tf.random_normal(shape) * (np.sqrt(2.0/(shape[0] + shape[1])))
The optimizer in the two versions are set to be exactly the same! Though it doesn't look like the accuracy depends on the optimizer - I tried using different optimizers in both keras and tf and the accuracies each converge to the same.
Help!

It seems to me that this is most probably the weight initialization problem. What I would suggest you to do is to initialize keras layers and before training get the layer weights and initialize tf layers with those values.
I have ran into that kind of problems and it solved problems for me but it was a long time ago and I don't know if they made those initializers the same. At that time tf and keras initializations were not the same obviously.

I checked with initializers,seed, parameters and hyperparameters but accuracy is different.
I checked the code for Keras and they randomly shuffle the batch of images and then fed into the network, so this shuffling is different across different engines. So we need to figure out a way in which we can fed the same set of batch images to the network in order to get same accuracy

Convolutional Neural Net-Keras-val_acc Keyerror 'acc'

I am trying to implement CNN by Theano. I used Keras library. My data set is 55 alphabet images, 28x28.
In the last part I get this error:
train_acc=hist.history['acc']
KeyError: 'acc'
Any help would be much appreciated. Thanks.
This is part of my code:
from keras.models import Sequential
from keras.models import Model
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D, MaxPooling2D
from keras.optimizers import SGD, RMSprop, adam
from keras.utils import np_utils
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.cm as cm
from urllib.request import urlretrieve
import pickle
import os
import gzip
import numpy as np
import theano
import lasagne
from lasagne import layers
from lasagne.updates import nesterov_momentum
from nolearn.lasagne import NeuralNet
from nolearn.lasagne import visualize
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from PIL import Image
import PIL.Image
#from Image import *
import webbrowser
from numpy import *
from sklearn.utils import shuffle
from sklearn.cross_validation import train_test_split
from tkinter import *
from tkinter.ttk import *
import tkinter
from keras import backend as K
K.set_image_dim_ordering('th')
%%%%%%%%%%
batch_size = 10
# number of output classes
nb_classes = 6
# number of epochs to train
nb_epoch = 5
# input iag dimensions
img_rows, img_clos = 28,28
# number of channels
img_channels = 3
# number of convolutional filters to use
nb_filters = 32
# number of convolutional filters to use
nb_pool = 2
# convolution kernel size
nb_conv = 3
%%%%%%%%
model = Sequential()
model.add(Convolution2D(nb_filters, nb_conv, nb_conv,
border_mode='valid',
input_shape=(1, img_rows, img_clos)))
convout1 = Activation('relu')
model.add(convout1)
model.add(Convolution2D(nb_filters, nb_conv, nb_conv))
convout2 = Activation('relu')
model.add(convout2)
model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adadelta')
%%%%%%%%%%%%
hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=True, verbose=1, validation_data=(X_test, Y_test))
hist = model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
show_accuracy=True, verbose=1, validation_split=0.2)
%%%%%%%%%%%%%%
train_loss=hist.history['loss']
val_loss=hist.history['val_loss']
train_acc=hist.history['acc']
val_acc=hist.history['val_acc']
xc=range(nb_epoch)
#xc=range(on_epoch_end)
plt.figure(1,figsize=(7,5))
plt.plot(xc,train_loss)
plt.plot(xc,val_loss)
plt.xlabel('num of Epochs')
plt.ylabel('loss')
plt.title('train_loss vs val_loss')
plt.grid(True)
plt.legend(['train','val'])
print (plt.style.available) # use bmh, classic,ggplot for big pictures
plt.style.use(['classic'])
plt.figure(2,figsize=(7,5))
plt.plot(xc,train_acc)
plt.plot(xc,val_acc)
plt.xlabel('num of Epochs')
plt.ylabel('accuracy')
plt.title('train_acc vs val_acc')
plt.grid(True)
plt.legend(['train','val'],loc=4)
#print plt.style.available # use bmh, classic,ggplot for big pictures
plt.style.use(['classic'])

In a not-so-common case (as I expected after some tensorflow updates), despite choosing metrics=["accuracy"] in the model definitions, I still got the same error.
The solution was: replacing metrics=["acc"] with metrics=["accuracy"] everywhere. In my case, I was unable to plot the parameters of the history of my training. I had to replace
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']
to
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']

Your log variable will be consistent with the metrics when you compile your model.
For example, the following code
model.compile(loss="mean_squared_error", optimizer=optimizer)
model.fit_generator(gen,epochs=50,callbacks=ModelCheckpoint("model_{acc}.hdf5")])
will gives a KeyError: 'acc' because you didn't set metrics=["accuracy"] in model.compile.
This error also happens when metrics are not matched. For example
model.compile(loss="mean_squared_error",optimizer=optimizer, metrics="binary_accuracy"])
model.fit_generator(gen,epochs=50,callbacks=ModelCheckpoint("model_{acc}.hdf5")])
still gives a KeyError: 'acc' because you set a binary_accuracy metric but asking for accuracy later.
If you change the above code to
model.compile(loss="mean_squared_error",optimizer=optimizer, metrics="binary_accuracy"])
model.fit_generator(gen,epochs=50,callbacks=ModelCheckpoint("model_{binary_accuracy}.hdf5")])
it will work.

You can use print(history.history.keys()) to find out what metrics you have and what they are called. In my case also, it was called "accuracy", not "acc"

In my case switching from
metrics=["accuracy"]
to
metrics=["acc"]
was the solution.

from keras source :
warnings.warn('The "show_accuracy" argument is deprecated, '
'instead you should pass the "accuracy" metric to '
'the model at compile time:\n'
'`model.compile(optimizer, loss, '
'metrics=["accuracy"])`')
The right way to get the accuracy is indeed to compile your model like this:
model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=["accuracy"])
does it work?

Make sure to check this "breaking change":
Metrics and losses are now reported under the exact name specified by the user (e.g. if you pass metrics=['acc'], your metric will be reported under the string "acc", not "accuracy", and inversely metrics=['accuracy'] will be reported under the string "accuracy".

If you are using Tensorflow 2.3 then you can specify like this
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss=tf.keras.losses.CategoricalCrossentropy(), metrics=[tf.keras.metrics.CategoricalAccuracy(name="acc")])

In the New version of TensorFlow, some things have changed so we have to replace it with :
acc = history.history['accuracy']

print(history.history.keys())
output--
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])
so you need to change "acc" to "accuracy" and "val_acc" to "val_accuracy"

For Practice
3.5-classifying-movie-reviews.ipynb
Change
acc = history.history['acc']
val_acc = history.history['val_acc']
To
acc = history.history['binary_accuracy']
val_acc = history.history['val_binary_accuracy']
&
Change
acc_values = history_dict['acc']
val_acc_values = history_dict['val_acc']
To
acc_values = history_dict['binary_accuracy']
val_acc_values = history_dict['val_binary_accuracy']
================
Practice
3.6-classifying-newswires.ipynb
Change
acc = history.history['acc']
val_acc = history.history['val_acc']
To
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']