Develop Reference

Evaluation accuracy stays the same while test accuracy increases just fine

I have tried over and over with different approaches to building this model however I continue to run into this issue where my training accuracy steadily increases just fine but my validation and evaluation accuracy remains very low (55% - 65%).
Epoch 95/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6326 - accuracy: 0.8057 - val_loss: 2.0461 - val_accuracy: 0.5985
Epoch 96/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6485 - accuracy: 0.7990 - val_loss: 1.9512 - val_accuracy: 0.5909
Epoch 97/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6263 - accuracy: 0.8032 - val_loss: 2.0344 - val_accuracy: 0.5682
Epoch 98/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6249 - accuracy: 0.7990 - val_loss: 2.0183 - val_accuracy: 0.5682
Epoch 99/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6189 - accuracy: 0.8007 - val_loss: 2.0818 - val_accuracy: 0.5758
Epoch 100/100
119/119 [==============================] - 0s 2ms/step - loss: 0.6261 - accuracy: 0.8024 - val_loss: 2.0591 - val_accuracy: 0.5833
18/18 [==============================] - 0s 1ms/step - loss: 2.2385 - accuracy: 0.5628
EVAL:
[2.238506317138672, 0.5628318786621094]
The entire script is as follows:
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from keras import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import Adam
import numpy as np
from keras.utils import to_categorical
def count_classes(y: list):
counts = {}
for i in y:
if i in counts.keys():
counts[i] += 1
else:
counts[i] = 1
return counts
dataset = pd.read_csv('Dementia-data.csv')
X= dataset.iloc[:,1:]
y= dataset.iloc[:,0] # roughly 2562 input variables
X.head(2)
#standardizing the input feature
X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle = True, random_state = 0, test_size=0.2)
print("TRAIN:")
print(count_classes(list(y_train)))
print("TEST:")
print(count_classes(list(y_test)))
sc = StandardScaler()
scaler = sc.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
optimizer = Adam(lr=0.00005)
classifier = Sequential()
#First Hidden Layer
classifier.add(Dense(32, activation='relu', kernel_initializer='random_normal', kernel_regularizer=regularizers.l2(0.005), input_dim=2562))
#Second Hidden Layer
classifier.add(Dropout(0.2))
classifier.add(Dense(64, activation='relu', kernel_initializer='random_normal', kernel_regularizer=regularizers.l2(0.005)))
#Output Layer
classifier.add(Dropout(0.2))
classifier.add(Dense(64, activation='relu', kernel_initializer='random_normal'))
classifier.add(Dense(7, activation='softmax', kernel_initializer='random_normal'))
#Compiling the neural network
classifier.compile(optimizer =optimizer,loss='sparse_categorical_crossentropy', metrics =['accuracy'])
#Fitting the data to the training dataset
classifier.fit(X_train,y_train, batch_size=20, epochs=200, validation_split=0.1, shuffle=True)
eval_model=classifier.evaluate(X_test, y_test)
print("EVAL:")
print(eval_model)
I have tried all sorts of things to try to combat overfittings such as regulators, dropouts, and data splitting because that seems to be the lead cause for problems like this one. I do not know if I am missing something.
The class distribution for the test and train data is as follows, it doesn't look like there is any inconsistencies between the 2 datasets in terms of ratios of classses:
TRAIN:
{2: 822, 4: 136, 6: 229, 5: 184, 3: 76, 1: 57}
TEST:
{2: 199, 6: 59, 5: 45, 1: 26, 3: 15, 4: 33}
The plotted accuracy for training and testing:
The plotted loss for training and testing, the loss clearly increases for the testing data:
I have been struggling with this for weeks now and I'd truly appreciate any help to get this working. I have also tried using learning rates between 0.05 and 0.00005 with little to no improvement.

Deep learning train large numbers

When I try to create model to predict this data. I can't get good loss. How can I optimize it?
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
X = np.array([32878265.2, 39635188.8, 738222697.41, 33921812.23, 39364408, 50854015, 50938146.63, 54062184.4, 32977734, 27267164, 30673902.72])
y = np.array([80712, 111654, 127836.61, 128710, 147907, 152862, 154962, 138503, 140238, 105121, 113211.8])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1)
scaler = preprocessing.StandardScaler().fit(X_train.reshape(-1, 1))
X_scaled = scaler.transform(X_train.reshape(-1, 1))
tf.random.set_seed(42)
model = tf.keras.Sequential([
# tf.keras.layers.Dense(10),
tf.keras.layers.Dense(1, input_shape=[1]),
tf.keras.layers.Dense(1),
])
model.compile(loss=tf.keras.losses.mae,
optimizer=tf.keras.optimizers.Adam(),
metrics=["mae"])
model.fit(X_scaled, y_train, epochs=5000,
validation_data=(X_test, y_test))
Epoch 2466/5000 1/1 [==============================] - 0s 29ms/step -
loss: 38000588.0000 - mae: 38000588.0000 - val_loss: 28384532.0000 -
val_mae: 28384532.0000 Epoch 2467/5000 1/1
[==============================] - 0s 31ms/step - loss: 38000588.0000
mae: 38000588.0000 - val_loss: 28384536.0000 - val_mae: 28384536.0000 Epoch 2468/5000 1/1 [==============================] - 0s 41ms/step - loss: 38000588.0000 - mae: 38000588.0000 - val_loss:
28384540.0000 - val_mae: 28384540.0000 Epoch 2469/5000 1/1 [==============================] - 0s 41ms/step - loss: 38000588.0000
mae: 38000588.0000 - val_loss: 28384536.0000 - val_mae: 28384536.0000

Your NN model is just a linear regression. When you plot the data, you see that you have an outlier which is the main problem for a good prediction:
I guess, you typed a digit too much.

What is the calculation process of loss functions in multi-class multi-label classification problems using deep learning?

Dataset description:
(1) X_train: (6000,4) shape
(2) y_train: (6000,4) shape
(3) X_validation: (2000,4) shape
(4) y_validation: (2000,4) shape
(5) X_test: (2000,4) shape
(6) y_test: (2000,4) shape
Relationship between X and Y is shown here
For single label classification, the activation function of the last layer is Softmax and the loss function is categorical_crossentrop.
And I know the mathematical calculation method for the loss function.
And for multi-class multi-label classification problems, the activation function of the last layer is sigmoid, and the loss function is binary_crossentrop.
I want to know how the mathematical calculation method of the loss function works
It would be a great help to me if you let me know.
def MinMaxScaler(data):
numerator = data - np.min(data)
denominator = np.max(data) - np.min(data)
return numerator / (denominator + 1e-5)
kki = pd.read_csv(filename,names=['UE0','UE1','UE2','UE3','selected_UE0','selected_UE1','selected_UE2','selected_UE3'])
print(kki)
def LoadData(file):
xy = np.loadtxt(file, delimiter=',', dtype=np.float32)
print("Data set length:", len(xy))
tr_set_size = int(len(xy) * 0.6)
xy[:, 0:-number_of_UEs] = MinMaxScaler(xy[:, 0:-number_of_UEs]) #number_of_UES : 4
X_train = xy[:tr_set_size, 0: -number_of_UEs] #6000 row
y_train = xy[:tr_set_size, number_of_UEs:number_of_UEs*2]
X_valid = xy[tr_set_size:int((tr_set_size/3) + tr_set_size), 0:-number_of_UEs]
y_valid = xy[tr_set_size:int((tr_set_size/3) + tr_set_size), number_of_UEs:number_of_UEs *2]
X_test = xy[int((tr_set_size/3) + tr_set_size):, 0:-number_of_UEs]
y_test = xy[int((tr_set_size/3) + tr_set_size):, number_of_UEs:number_of_UEs*2]
print("Training X shape:", X_train.shape)
print("Training Y shape:", y_train.shape)
print("validation x shape:", X_valid.shape)
print("validation y shape:", y_valid.shape)
print("Test X shape:", X_test.shape)
print("Test Y shape:", y_test.shape)
return X_train, y_train, X_valid, y_valid, X_test, y_test, tr_set_size
X_train, y_train, X_valid, y_valid, X_test, y_test, tr_set_size = LoadData(filename)
model = Sequential()
model.add(Dense(64,activation='relu', input_shape=(X_train.shape[1],)))
model.add(Dense(46, activation='relu'))
model.add(Dense(24, activation='relu'))
model.add(Dense(12, activation='relu'))
model.add(Dense(4, activation= 'sigmoid'))
model.compile( loss ='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
hist = model.fit(X_train, y_train, epochs=5, batch_size=1, verbose= 1, validation_data=(X_valid, y_valid), callbacks= es)
This is a learning process, and even if epochs are repeated,
Accuracy does not improve.
Epoch 1/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.2999 - accuracy: 0.5345 - val_loss: 0.1691 - val_accuracy: 0.5465
Epoch 2/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.1554 - accuracy: 0.4883 - val_loss: 0.1228 - val_accuracy: 0.4710
Epoch 3/10
6000/6000 [==============================] - 14s 2ms/step - loss: 0.1259 - accuracy: 0.4710 - val_loss: 0.0893 - val_accuracy: 0.4910
Epoch 4/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.1094 - accuracy: 0.4990 - val_loss: 0.0918 - val_accuracy: 0.5540
Epoch 5/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0967 - accuracy: 0.5223 - val_loss: 0.0671 - val_accuracy: 0.5405
Epoch 6/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0910 - accuracy: 0.5198 - val_loss: 0.0836 - val_accuracy: 0.5380
Epoch 7/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0870 - accuracy: 0.5348 - val_loss: 0.0853 - val_accuracy: 0.5775
Epoch 8/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0859 - accuracy: 0.5518 - val_loss: 0.0515 - val_accuracy: 0.6520
Epoch 9/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0792 - accuracy: 0.5508 - val_loss: 0.0629 - val_accuracy: 0.4350
Epoch 10/10
6000/6000 [==============================] - 13s 2ms/step - loss: 0.0793 - accuracy: 0.5638 - val_loss: 0.0632 - val_accuracy: 0.6270

Mistake 1 -
The shape of y_train, y_validation and y_test should be (6000,), (2000,) and (2000,) respectively.
Mistake 2 -
For multi-class classification, the loss should be categorical_crossentropy and activation should be a softmax. So, change these two lines, like this:
model.add(Dense(4, activation= 'softmax'))
model.compile(loss ='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
Suggestion -
Why are you splitting data by yourself? Use scikit-learn train_test_split. This code will give you proper splits:
from sklearn.model_selection import train_test_split
x, x_test, y, y_test = train_test_split(xtrain, labels, test_size=0.2, train_size=0.8)
x_train, x_validation, y_train, y_validation = train_test_split(x, y, test_size = 0.25, train_size =0.75)

Regression model output looks inconsistent

Some context about my project: I intend to study various parameters about bullets and how they affect the ballistics coefficient (i.e. bullet performance) of the projectile. I have different parameters, such as weight, caliber, sectional density, etc. I feel that I did this all wrong though; I am just reading through tutorials and applying what I feel could be useful and relevant in my project.
The output of my regression model looks a bit off to me; the trained model continuously outputs 0.0201 as MSE throughout the model.fit() part of my program.
Also, the model.predict(X) seems to have an accuracy of 100%, however, this does not seem right; I borrowed some code from a tutorial describing Keras models to display the model output while displaying the expected output.
This is the program constructing the model and training it
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.utils import shuffle
import tensorflow as tf
from tensorflow.keras.callbacks import TensorBoard
from pandas.plotting import scatter_matrix
import time
name = 'Bullet Database Analysis v2-{}'.format(int(time.time()))
tensorboard = TensorBoard(log_dir='logs/{}'.format(name))
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)
df = pd.read_csv('Bullet Optimization\ShootForum Bullet DB_2.csv')
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
dataset = df.values
X = dataset[:,0:12]
X = np.asarray(X).astype(np.float32)
y = dataset[:,13]
y = np.asarray(y).astype(np.float32)
X_train, X_val_and_test, y_train, y_val_and_test = train_test_split(X, y, test_size=0.3, shuffle=True)
X_val, X_test, y_val, y_test = train_test_split(X_val_and_test, y_val_and_test, test_size=0.5)
from keras.models import Sequential
from keras.layers import Dense, BatchNormalization
model = Sequential(
[
#2430 is the shape of X_train
#BatchNormalization(axis=-1, momentum = 0.1),
Dense(2430, activation='relu'),
Dense(32, activation='relu'),
Dense(1),
]
)
model.compile(loss='mse', metrics=['mse'])
history = model.fit(X_train, y_train,
batch_size=64,
epochs=20,
validation_data=(X_val, y_val),
#callbacks = [tensorboard]
)
# plt.plot(history.history['loss'],'r')
# plt.plot(history.history['val_loss'],'m')
plt.plot(history.history['mse'],'b')
plt.show()
model.summary()
model.save("Bullet Optimization\Bullet Database Analysis.h5")
Here is my code, loading my previously trained model via h5
import numpy as np
import tensorflow as tf
from tensorflow import keras
from keras.models import load_model
import pandas as pd
df = pd.read_csv('Bullet Optimization\ShootForum Bullet DB_2.csv')
model = load_model('Bullet Optimization\Bullet Database Analysis.h5')
dataset = df.values
X = dataset[:,0:12]
y = dataset[:,13]
model.fit(X,y, epochs=10)
#predictions = np.argmax(model.predict(X), axis=-1)
predictions = model.predict(X)
# summarize the first 5 cases
for i in range(5):
print('%s => %d (expected %d)' % (X[i].tolist(), predictions[i], y[i]))
This is the output
Epoch 1/10
2021-03-09 10:38:06.372303: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-03-09 10:38:07.747241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
109/109 [==============================] - 2s 4ms/step - loss: 0.0201 - mse: 0.0201
Epoch 2/10
109/109 [==============================] - 1s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 3/10
109/109 [==============================] - 0s 4ms/step - loss: 0.0201 - mse: 0.0201
Epoch 4/10
109/109 [==============================] - 0s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 5/10
109/109 [==============================] - 1s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 6/10
109/109 [==============================] - 1s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 7/10
109/109 [==============================] - 1s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 8/10
109/109 [==============================] - 0s 4ms/step - loss: 0.0201 - mse: 0.0201
Epoch 9/10
109/109 [==============================] - 1s 5ms/step - loss: 0.0201 - mse: 0.0201
Epoch 10/10
109/109 [==============================] - 0s 4ms/step - loss: 0.0201 - mse: 0.0201
[0.314, 7.9756, 100.0, 100.0, 31.4, 0.00314, 318.4713376, 6.480041472000001, 0.51, 12.95400001, 4.067556004, 0.145] => 0 (expected 0)
[0.358, 9.0932, 148.0, 148.0, 52.983999999999995, 0.002418919, 413.4078212, 9.590461379, 0.635, 16.12900002, 5.774182006, 0.165] => 0 (expected 0)
[0.313, 7.9502, 83.0, 83.0, 25.979, 0.003771084, 265.1757188, 5.378434422000001, 0.504, 12.80160001, 4.006900804, 0.121] => 0 (expected 0)
[0.251, 6.3754, 50.0, 50.0, 12.55, 0.00502, 199.20318730000002, 3.2400207360000004, 0.4, 10.16000001, 2.5501600030000002, 0.113] => 0 (expected 0)
[0.251, 6.3754, 50.0, 50.0, 12.55, 0.00502, 199.20318730000002, 3.2400207360000004, 0.41, 10.41400001, 2.613914003, 0.113] => 0 (expected 0)
Here is a link to my training dataset. Within my code, I used train_test_split to create both the test and train dataset.
Lastly, is there a way within Tensorboard to visualize the model fitting with the dataset? I really feel that although my model is training, it is not making any significant fitting even though the MSE error is reduced.

Because you have nan values in your dataset. Before splitting up you can check it with df.isna().sum(). These can have a negative impact on your network. Here I just simply dropped them (df.dropna(inplace = True, axis = 0)) but you can use some imputation techniques to replace them.
Also 2430 neurons can be overkill for this data, start with less neurons.
model = tf.keras.models.Sequential(
[
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(1),
]
)
Here is the last epoch:
Epoch 20/20
27/27 [==============================] - 0s 8ms/step - loss: 8.2077e-04 - mse: 8.2077e-04 -
val_loss: 8.5023e-04 - val_mse: 8.5023e-04
While doing regression, calculating accuracy straight forward is not a valid option. You can use model.evaluate(X_test, y_test) or when you get predictions by model.predict, you can use other regression metrics to compute how close your predictions are.

Loss is very big from the beginning, and almost not decreasing in tensorflow

I am trying to perform linear regression in tensorflow. I don't understand why from the beginning my loss is so big. What is more, the decrease in loss is very small during training.
Could you please tell me if there are any mistakes in my code that may cause this problem?
If it is important, I used Kaggle dataset - "House price prediction" - which can be found at kaggle at /shree1992/housedata.
I paste my code below:
import sys
import tensorflow as tf
import numpy as np
import pandas as pd
np.set_printoptions(threshold=sys.maxsize)
import numpy as np
import matplotlib.pyplot as plt
# Loading and inspecting the data:
data = pd.read_csv('dataset/data.csv')
print(data.head(1))
data = data.drop('street', 1)
data = data.drop('statezip', 1)
print(data.head(1))
# Encoding categorical data - street, city, statezip, country:
data = pd.get_dummies(data, columns=["city", "country"])
data = data.values
X = np.array(data[:, 2:], dtype=np.float)
Y = np.array(data[:, 1], dtype=np.float)
print(X.shape)
# it had 4600 observations, 4659 features before dropping two columns - (4600, 57)
#after dropping 'street' and 'statezip' - (4600, 57)
# Splitting the data into training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)
N, D = X_train.shape
# Scaling the data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
# Creating a model
model = tf.keras.models.Sequential([
tf.keras.layers.Input(shape=(D,)),
tf.keras.layers.Dense(1)
])
model.compile(optimizer=tf.keras.optimizers.SGD(0.001, 0.9), loss='mse')
# model.compile(optimizer='adam', loss='mse')
# learning rate scheduler
def schedule(epoch, lr):
if epoch >= 50:
return 0.0001
return 0.001
scheduler = tf.keras.callbacks.LearningRateScheduler(schedule)
# Train the model
r = model.fit(X_train, y_train, epochs=200, callbacks=[scheduler])
Below you can find a fragment of results of running this code:
Epoch 38/200
101/101 [==============================] - 0s 464us/step - loss: 270232453120.0000
Epoch 39/200
101/101 [==============================] - 0s 464us/step - loss: 270549336064.0000
Epoch 40/200
101/101 [==============================] - 0s 464us/step - loss: 270402207744.0000
Epoch 41/200
101/101 [==============================] - 0s 619us/step - loss: 270913732608.0000
Epoch 42/200
101/101 [==============================] - 0s 464us/step - loss: 271168536576.0000
Epoch 43/200
101/101 [==============================] - 0s 464us/step - loss: 270916206592.0000
Epoch 44/200
101/101 [==============================] - 0s 464us/step - loss: 272359129088.0000
Epoch 45/200
101/101 [==============================] - 0s 619us/step - loss: 271552167936.0000
Epoch 46/200
101/101 [==============================] - 0s 464us/step - loss: 272676618240.0000
Epoch 47/200
101/101 [==============================] - 0s 619us/step - loss: 272397254656.0000
Epoch 48/200
101/101 [==============================] - 0s 464us/step - loss: 270996291584.0000
Epoch 49/200
101/101 [==============================] - 0s 619us/step - loss: 271571435520.0000
Epoch 50/200
101/101 [==============================] - 0s 464us/step - loss: 272347545600.0000
Epoch 51/200
101/101 [==============================] - 0s 619us/step - loss: 269679640576.0000