I am using this CNN architecture :
def createModel():
model = Sequential()
model.add(Conv2D(96, (11, 11), strides=(4,4), padding='same', activation='relu', input_shape=(224,224,3))) #, input_shape=input_shape
model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2), padding='same'))
model.add(Conv2D(256, (5, 5), strides=(1,1), padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2), padding='same'))
model.add(Conv2D(384, (3, 3), strides=(1,1), padding='same', activation='relu'))
model.add(Conv2D(384, (3, 3), strides=(1,1), padding='same', activation='relu'))
model.add(Conv2D(256, (3, 3), strides=(1,1), padding='same', activation='relu'))
model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(4096, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
return model
At the total, it makes 226,065,795 trainable parameters... I train my model on a quite small dataset. Indeed, I have three classes, each one contains 1000 images, that are split between training and validation set. I used OneHotEncoding.
However, when I train the model, the overall accuracy tends to a very low result : 32%. I am using categorical_crossentropy loss function. When looking deeper, it seems that the model always predicts 1,0,0 vector, i.e. the first class and never the others, even for the trained data that thus have been shown to the model.
What's the problem there ? Why do I have poor results ? My dataset is balanced tough. Is it because my number of parameters is too high compared to the amount of data? I don't understand...
EDIT : removing Dropout still keeps the bad thing
Related
I have built a simple CNN using tensorflow-gpu 2.4.0 for cifar10
# Set constent
num_classes = len(label_names) # The number of classes
input_shape = (32, 32, 3) # The input shape 32 * 32 pixels, 3 RGB brightness for each pixel
# Construct a sequential model
model = Sequential()
# Starting add layers to the model
# First VGG blockes
model.add(Conv2D(16, (3, 3), padding='same', activation='relu', input_shape=input_shape))
model.add(BatchNormalization()) # BatchNormalization after convolutional layer
model.add(Conv2D(16, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization()) # BatchNormalization after convolutional layer
model.add(MaxPooling2D((2, 2))) # MaxPooling2D to keep important features as well as shirnk info size
model.add(Dropout(0.1)) # Drop some portion of nodes to increase generality of model
# Second VGG blockes
model.add(Conv2D(32, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(32, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(0.2))
# Third VGG blockes
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(0.3))
# Fourth VGG blockes
model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
model.add(BatchNormalization())
model.add(MaxPooling2D((2, 2)))
model.add(Dropout(0.4))
# Flat and dense layer
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
# Final softmax layer
model.add(Dense(num_classes, activation='softmax'))
# The optimizer gives the learning rate lr and some other parameters.
opt = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
# Model complie
model.compile(loss='categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
And while I am training this model:
#Train model
epochs = 100
batch_size = 128
# create data generator
datagen = ImageDataGenerator(horizontal_flip=True,
width_shift_range=5,
height_shift_range=5,
rotation_range=20)
history = model.fit(datagen.flow(x_training, y_training, batch_size=batch_size),
epochs=epochs,
validation_data=(x_validate, y_validate),
verbose=1,
shuffle=True)
The RAM of my computer keeps grow until the training process crashes. Adding python garbage collector for at end of each epoch did not work. Disabling eager detection did not work.
I using
tensorflow-gpu 2.4.0
CUDA 11.0
cuDNN 8.0
I'm trying to implement CNN presented in this paper "A Novel Machine Learning Aided Antenna Selection Scheme for MIMO Internet of Things" which is found in this article.
In my implementation, I follow all instructions therein, however, my binary-crossentropy loss function is not decreasing, i.e., it is constant over all epochs. I have been trying: change learning rate, normalize data set (min max, standard scale), change the number of epochs/batch_size, change the optimizer however nothing is making effect.
Have anyone any suggestions?
matcont = hdf5storage.loadmat('train.mat')
# matcont = sio.loadmat('train')
Input = (matcont['in'])
Output = (matcont['out'])
model = Sequential()
model.add(Conv2D(16, (2, 2), padding="same",input_shape=(1,1,8)))
model.add(Activation("relu"))
# model.add(Dropout(0.2))
model.add(Conv2D(16, (2, 2), padding="same"))
model.add(Activation("relu"))
# model.add(Dropout(0.2))
# model.add(Conv2D(16, (2, 2), padding="same"))
# model.add(Activation("relu"))
# model.add(Dropout(0.2))
# model.add(Conv2D(16, (2, 2), padding="same"))
# model.add(Activation("relu"))
# model.add(Dropout(0.2))
# model.add(Conv2D(16, (2, 2), padding="same"))
# model.add(Activation("relu"))
# model.add(Dropout(0.2))
# model.add(Conv2D(16, (2, 2), padding="same"))
# model.add(Activation("relu"))
# model.add(Dropout(0.2))
# model.add(Conv2D(16, (2, 2), padding="same"))
# model.add(Activation("relu"))
model.add(Flatten())
model.add(Dense(256,activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(8,activation='sigmoid'))
model.summary()
adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, decay=0.01)
model.compile(adam,loss='binary_crossentropy',metrics = ['accuracy'])
model.fit(Input,Output,epochs=100,validation_split=0.11,batch_size=100)
You first want to verify that your model is able to conquer the data. The way to do that is to focus on minimizing the training loss by dropping regularization and increasing the model size until it overfits the data. The model should be able to get an almost "perfect score" on the training data.
I'm building a CNN to control a vehicle within a video game. The network takes a screenshot as input and uses controller values as targets. For now I'm just using two controller values as targets: steering which is a value between -1 and 1, as well as throttle, which is between 0 and 1. I have rounded the values of steering into 7 values, and throttle into 4 values giving me 28 distinct classes which I will balance (reason for rounding was difficulty balancing un-binned classes).
My question is whether I should train the network with a single value target 1-27 (one for each case), or whether I should use the two rounded controller values as targets (an array: [steering, throttle])? I understand that both create 28 target classes, but does the structure of the target output affect the performance of the network? Is one of these options noticeably better than the other?
The model for preliminary testing:
'''
model = Sequential()
model.add(Conv2D(24, kernel_size=(5, 5), strides=(2, 2), activation='relu', input_shape= INPUT_SHAPE))
model.add(Conv2D(36, kernel_size=(5, 5), strides=(2, 2), activation='relu'))
model.add(Conv2D(48, kernel_size=(5, 5), strides=(2, 2), activation='relu'))
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(Flatten())
model.add(Dense(1164, activation='relu'))
drop_out = 1 - keep_prob
model.add(Dropout(drop_out))
model.add(Dense(100, activation='relu'))
model.add(Dropout(drop_out))
model.add(Dense(50, activation='relu'))
model.add(Dropout(drop_out))
model.add(Dense(10, activation='relu'))
model.add(Dropout(drop_out))
model.add(Dense(OUT_SHAPE, activation='softsign'))
'''
I want to add some extra information to a CNN as gender, age, a vector...
My CNN have as inputs matrices that represent voice histograms with dimensions 125x64. Since they are from different persons, I would like to add that information to the model. Besides, I would like to add some vector 125x1 who represents the pitch or the energy of the voice (getting from feature extraction) but I think is not a good idea to attach it to the histogram.
model = Sequential()
model.add(Conv2D(32, (3, 3), padding='valid', strides=1,
input_shape=input_shape, activation='relu'))
model.add(MaxPooling2D(pool_size=(4, 3), strides=(1, 3)))
model.add(Conv2D(32, (1, 3), padding='valid', strides=1,
input_shape=input_shape, activation='relu'))
model.add(MaxPooling2D(pool_size=(1, 3), strides=(1, 3)))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='adadelta',
metrics=['accuracy'])
It indeed doesn't make much sense to add that data to the histogram. Keras has an explanation in their own documentation to how you use multiple inputs in a model: https://keras.io/getting-started/functional-api-guide/. The paragraph Multi-input and multi-output models seems to be what you're looking for.
I've used Floyd hub to to train the following model and saved it
# Create the model
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(3, 32, 32), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(Dropout(0.2))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(1024, activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))
# Compile model
epochs = 50
adammax = keras.optimizers.Adamax(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(loss='categorical_crossentropy', optimizer=adammax, metrics=['accuracy'])
print(model.summary())
When I try to load it on my PC, it's working fine. But when I load it on the Raspberry Pi I get the following error. I tried also to save just the weights and load them, but it didn't work and I got the same error. I am using the same version of Tensorflow as Floyd hub on the Raspberry Pi.
As mentioned above, you're passing T=DT_INT64, while that is not one of the supported kernels for this op. You could see if the int64 version is just not shipped in the .so file, write the op kernel yourself, or try casting to tf.int32 right before this op in the python code. The last one worked well for me.