Develop Reference

Is it normal for a CNN model to get ~20% accuracy with a dataset that contains 60-80 images per class?

I'm working with this Kaggle chess pieces dataset but after I coded my model and ran it, it only achieved about 20% accuracy and stalled there. Is this normal if each class has less than 100 images to train? I did image augmentation as well. If this is the case, around how many images do I need for datasets like this?
This is my model structure:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32, (3,3), activation='relu', input_shape=(150, 150, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(6, activation = "softmax")
])
model.compile(
loss = "sparse_categorical_crossentropy",
optimizer = "rmsprop",
metrics = ["accuracy"]
)

Looks like this person got up to around 40% validation accuracy: https://www.kaggle.com/code/diegofreitasholanda/chess-pieces-image-classification
But in general, yes that is a small dataset, and it will be hard to learn a good, generalizable network well, especially when the images all look very different from one another (I see some real pictures, others are clip art, etc).

multilabel text clasification with 1D CNN

I'm working on authorship detection from text task, I'm doing this by using a data frame consisting of symbol n-gram that I created using about 110k of 147 different authors' texts and TfidfVectorizer. Example of data, I encode author name strings using sklearn label encoder, which converts strings to numbers Example of labels
Data seperated into: (99817, 1000) (11091, 1000) (99817,) (11091,)
Using model below my best results were after 7-8 iterations: loss: 0.7225 -
accuracy: 0.8070 - val_loss: 1.3828 - val_accuracy: 0.6777, after that model starts to overfit.
Model:
model = Sequential(
[
Dense(300, activation="relu", input_shape=(Data_train.shape[-1],)),
Dense(750, activation="relu"),
BatchNormalization(),
Dropout(0.5),
Dense(147, activation="softmax"),
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
history = model.fit(Data_train,
Labels_train,
epochs=10,
shuffle=True,
callbacks=[early_stopping])
I want to try to solve this task with CNN network, I have found similar example of my task using 1D network and tried to use it:
vocab_size = 1000
maxlen = 1000
batch_size = 32
embedding_dims = 10
filters = 16
kernel_size = 3
hidden_dims = 250
epochs = 10
early_stopping = EarlyStopping(patience=0.1)
model = Sequential(
[
Embedding(vocab_size, embedding_dims, input_length=maxlen),
Dropout(0.5),
Conv1D(filters, kernel_size, padding='valid', activation='relu'),
MaxPooling1D(),
BatchNormalization(),
Conv1D(filters, kernel_size, padding='valid', activation='relu'),
MaxPooling1D(),
Flatten(),
Dense(hidden_dims, activation='relu'),
Dropout(0.5),
Dense(147, activation='softmax')
]
)
model.compile(optimizer='adam',
loss=keras.losses.SparseCategoricalCrossentropy(),
metrics=['accuracy'])
model.fit(Data_train, Labels_train,
batch_size=batch_size,
epochs=epochs,
validation_split=0.2,
callbacks=[early_stopping])
I only manage to get these results, from the first iteration to the 5, accuracy and val_accuracy reach ~0.07 and stay the same, after 5 iterations this is what I got:
loss: 4.5621 - accuracy: 0.0701 - val_loss: 4.5597 - val_accuracy:
0.0702
Could someone help me to improve these models to get better results, especially CNN? any suggestions are welcome, if I need to provide anything more please let me know, thank you.

I have managed to solve my issue, instead of using an embedding layer, I transformed my data and labels adding additional dimensions and passing input shape directly to the convolutional layer
Data_train_tmp = tf.expand_dims(Data_train, -1)
Labels_train_tmp = tf.expand_dims(Labels_train, 1)
Conv1D(62, 5, activation='relu', input_shape=(Data_train.shape[1],1)),

Not able to find a proper CNN

I am using Keras Tensorflow in Colab and I am working on the oxford_flowers102 dataset. Task is image classification. With quite many categories (102) and not so many images per class. I tried to build different neural networks, starting from simple one to more complex ones, with and without image augmentation, dropout, hyper parameter tuning, batch size adjustment, optimizer adjustment, image resizing size .... however, I was not able to find a good CNN which gives me an accetable val_accuracy and finally a good test accuracy. Up to now my max val_accuracy I was able to get was poor 0.3x. I am pretty sure that it is possible to get better results, I am somehow just not finding the right CNN setup. My code so far:
import tensorflow as tf
from keras.models import Model
import tensorflow_datasets as tfds
import tensorflow_hub as hub
# update colab tensorflow_datasets to current version 3.2.0,
# otherwise tfds.load will lead to error when trying to load oxford_flowers102 dataset
!pip install tensorflow_datasets --upgrade
# restart runtime
oxford, info = tfds.load("oxford_flowers102", with_info=True, as_supervised=True)
train_data=oxford['train']
test_data=oxford['test']
validation_data=oxford['validation']
IMG_SIZE = 224
def format_example(image, label):
image = tf.cast(image, tf.float32)
image = image*1/255.0
image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
return image, label
train = train_data.map(format_example)
validation = validation_data.map(format_example)
test = test_data.map(format_example)
BATCH_SIZE = 32
SHUFFLE_BUFFER_SIZE = 1000
train_batches = train.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)
test_batches = test.batch(BATCH_SIZE)
validation_batches = validation.batch(BATCH_SIZE)
First model I tried:
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(102)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_batches, validation_data=validation_batches, epochs=20)
Epoch 20/20 32/32 [==============================] - 4s 127ms/step -
loss: 2.9830 - accuracy: 0.2686 - val_loss: 4.8426 - val_accuracy:
0.0637
When I run it for more epochs, it overfits, val_loss goes up, val_accuracy does not go up.
Second model (very simple one):
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(IMG_SIZE, IMG_SIZE, 3)),
tf.keras.layers.Dense(128,activation='relu'),
tf.keras.layers.Dense(102)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_batches, validation_data=validation_batches, epochs=20)
Does not work at all, loss stays at 4.6250.
Third model:
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(102)
])
base_learning_rate = 0.0001
model.compile(optimizer=tf.optimizers.RMSprop(lr=base_learning_rate),
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_batches, validation_data=validation_batches, epochs=20)
Model overfits. Val_accuracy not above 0.15.
I added dropout layers to this model (trying differet rates) and also adjusted the kernels. However, no real improvement. Also tried adam optimizer.
Fourth model:
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(128, (3,3), activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)),
tf.keras.layers.Dropout(0.4),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(256, (3,3), activation='relu'),
tf.keras.layers.Dropout(0.4),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dropout(0.4),
tf.keras.layers.Dense(102)
])
model.compile(optimizer='adam',
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_batches, validation_data=validation_batches, epochs=20)
Same problem again, no good val_accuracy. Also tried it with RMSprop optimizer. Not able to get a val_accuracy higher than 0.2.
Fifth model:
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 3)),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(64, (2,2), activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(102)
])
base_learning_rate = 0.001
model.compile(optimizer=tf.optimizers.RMSprop(lr=base_learning_rate),
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
history = model.fit(train_batches, validation_data=validation_batches, epochs=250)
val_accuracy at the highest around 0.3x. Also tried it with adam.
When I tried it with transfer learning, using Mobilenet I immediately got 0.7x within 10 epochs. So I wondered why I am not able to get close to this with a self-built CNN? I do not expect 0.8 or to beat Mobilenet. But where is my mistake? How would a self-built CNN look like with which I can get lets say 0.6-0.7 val_accuracy?

You can try to use some predefined CNN with optimizing its parameters using some metaheuristic optimizers such as Grey Wolf optimizer or PSO, etc....

It's not entirely clear from your question: are you concerned that your model architecture is inferior to that of say MobileNet's, or that your performance is not comparable to that of transfer learning with MobileNet?
In response to the first, in general, the popular architectures such as ResNet, MobileNet, AlexNet are very cleverly crafted networks and so are likely to better represent data than a hand-defined network unless you do something very clever yourself.
In response to the second, the more complex a model gets, the more data it needs to train it well so that it is not underfit or overfit to the data. This poses a problem on datasets such as your (with a few thousand images) because it is difficult for a complex CNN to learn meaningful rules (kernels) for extracting information from images in general without instead learning rules for memorizing the limited set of training inputs. In summary, you want a larger model to make more accurate predictions, but this in turn requires more data, which sometimes you don't have. I suspect that if you used an untrained MobileNet versus your untrained network on the oxford flowers102 dataset, you'd see similarly poor performance.
Enter transfer learning. By pretraining relatively large models on relatively huge datsets (most are pretrained on ImageNet which has millions of images), the model is able to learn to extract relevant information from arbitrary images much better than it would be on a smaller dataset. These general rules for feature extraction apply to your smaller dataset as well, so with just a bit of fine-tuning the transfer learning model will likely far outperform any model trained solely on your dataset.

Tensorflow image classification binary crossentropy loss is negative

I'm new to Tensorflow. I followed some tutorials with a provided dataset and wanted to try something on my own. I decided I'd try to classify Magic the Gathering sets. Each card has a symbol in different colors on it: Black, Gold and so on.
The colors don't matter, just the different symbols. So I created a dataset of 3 different sets (so 3 different symbols) and got around 15'000 images like this. Some are a little bit rotated, some have an X and Y offset, just to get some different images.
Then I adapted the tutorial on the tensorflow website for image classification. Instead of two classes I wanted to try three:
batch_size = 250
epochs = 3
IMG_HEIGHT = 55
IMG_WIDTH = 55
train_image_generator = ImageDataGenerator(rescale=1./255)
validation_image_generator = ImageDataGenerator(rescale=1./255)
train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
directory=train_dir,
shuffle=True,
target_size=(IMG_HEIGHT, IMG_WIDTH),
class_mode='binary')
val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,
directory=validation_dir,
target_size=(IMG_HEIGHT, IMG_WIDTH),
class_mode='binary')
model = Sequential([
Conv2D(16, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
MaxPooling2D(),
Conv2D(32, 3, padding='same', activation='relu'),
MaxPooling2D(),
Conv2D(64, 3, padding='same', activation='relu'),
MaxPooling2D(),
Flatten(),
Dense(512, activation='relu'),
Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
history = model.fit_generator(
train_data_gen,
steps_per_epoch=total_train // batch_size,
epochs=epochs,
validation_data=val_data_gen,
validation_steps=total_val // batch_size,
callbacks=[cp_callback]
)
But my loss is negative and I don't get a good accuracy after training. What did I mess up? Is the model used in the tutorial not good for my usecase? Or is there an error in the code because I used three instead of two classes?

The model from the tutorial was used for binary classification (only two classes, cat or dog). You on the other hand want to classify 3 classes not 2. Therefore you have to adapt the architecture a little bit. Your last layer should be:
Dense(3, activation='softmax')
Three neurons because you have three classes and softmax activation because you want your outputs to be valid probabilities. To compile the model, use categorical_crossentropy instead of binary_crossentropy and make sure your labels are one-hot-encoded. Also for your ImageDataGenerator you should pass class_mode=categorical to the .flow_from_directory() function.

How to improve my CNN ? high and constant validation error

I am working on a problem for predicting a score of how fat cows are, based on images of cows.
I applied a CNN to estimate the value which is between 0-5 ( the dataset i have, contains only values between 2.25 and 4 )
I am using 4 CNN layers and 3 Hidden layers.
I actualy have 2 problems :
1/ I got 0.05 training error, but after 3-5 epochs the validation error remains at about 0.33.
2/ The value predicted by my NN are between 2.9 and 3.3 which is too narrow compared with the dataset range. Is it normal ?
How can i improve my model ?
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(512, 424,1)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(input_shape=(512, 424)),
tf.keras.layers.Dense(256, activation=tf.nn.relu),
tf.keras.layers.Dense(128, activation=tf.nn.relu),
tf.keras.layers.Dense(64, activation=tf.nn.relu),
tf.keras.layers.Dense(1, activation='linear')
])
Learning Curve:
Prediction:

This seems to be the case of Overfitting. You can
Shuffle the Data, by using shuffle=True in cnn_model.fit. Code is shown below:
history = cnn_model.fit(x = X_train_reshaped,
y = y_train,
batch_size = 512,
epochs = epochs, callbacks=[callback],
verbose = 1, validation_data = (X_test_reshaped, y_test),
validation_steps = 10, steps_per_epoch=steps_per_epoch, shuffle = True)
Use Early Stopping. Code is shown below
callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=15)
Use Regularization. Code for Regularization is shown below (You can try l1 Regularization or l1_l2 Regularization as well):
from tensorflow.keras.regularizers import l2
Regularizer = l2(0.001)
cnn_model.add(Conv2D(64,3, 3, input_shape = (28,28,1), activation='relu', data_format='channels_last',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
cnn_model.add(Dense(units = 10, activation = 'sigmoid',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
You can try using BatchNormalization.
Perform Image Data Augmentation using ImageDataGenerator. Refer this link for more info about that.
If the Pixels are not Normalized, Dividing the Pixel Values with 255 also helps.
Finally, if there still no change, you can try using Pre-Trained Models like ResNet or VGG Net, etc..