In order to analyze data, I need loss for each of output dimensions, instead I get only one loss which i suspect is a mean of the losses for all output dimensions.
Any help to understand what is the loss I get and how to get separate loss for each output:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from scipy import stats
from keras import models
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras import optimizers
from sklearn.model_selection import KFold
siz=100000
inp0=np.random.randint(100, 1000000 , size=(siz,3))
rand0=np.random.randint(-100, 100 , size=(siz,2))
a1=0.2;a2=0.8;a3=2.5;a4=2.6;a5=1.2;a6=0.3
oup1=np.dot(inp0[:,0],a1)+np.dot(inp0[:,1],a2)+np.dot(inp0[:,2],a3)\
+rand0[:,0]
oup2=np.dot(inp0[:,0],a4)+np.dot(inp0[:,1],a5)+np.dot(inp0[:,2],a6)\
+rand0[:,1]
oup_tot=np.concatenate((oup1.reshape(siz,1), oup2.reshape(siz,1)),\
axis=1)
normzer_inp = MinMaxScaler()
inp_norm = normzer_inp.fit_transform(inp0)
normzer_oup = MinMaxScaler()
oup_norm = normzer_oup.fit_transform(oup_tot)
X=inp_norm
Y=oup_norm
kfold = KFold(n_splits=2, random_state=None, shuffle=False)
opti_SGD = SGD(lr=0.01, momentum=0.9)
model1 = Sequential()
for train, test in kfold.split(X, Y):
model = Sequential()
model.add(Dense(64, input_dim=X.shape[1], activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(Y.shape[1], activation='linear'))
model.compile(loss='mean_squared_error', optimizer=opti_SGD)
history = model.fit(X[train], Y[train], \
validation_data=(X[test], Y[test]), \
epochs=100,batch_size=2048, verbose=2)
I get:
Epoch 1/100
- 0s - loss: 0.0864 - val_loss: 0.0248
Epoch 2/100
- 0s - loss: 0.0218 - val_loss: 0.0160
Epoch 3/100
- 0s - loss: 0.0125 - val_loss: 0.0091
I would like to know what is the loss i got now and how to get losses for each output dimension.
Pass a list of functions to the metrics argument in the compile function. See here: https://keras.io/metrics/#custom-metrics
import keras.backend as K
...
def loss_first_dim(y_true, y_pred):
return K.mean(K.square(y_pred[:, 0] - y_true[:, 0]))
model.compile(optimizer='rmsprop',
loss='binary_crossentropy',
metrics=[loss_first_dim])
...
Related
I'm having some problems which I cannot solve on my own. I'm pretty new to ML and Sequential Models of Keras.
Problem:
I only get NaN for loss and accuracy during fit()
Further when I try to predict, I'm just getting NaNs for the prediction.
My data is defined as datas(85802, 223) inclusive target
*UPDATE:
The problem was, that I did new NaN values after clearing the NaNs.
datas.dropna(inplace=True) <===== This have switched
labels = datas.columns
datas['target'] = datas['close'].shift(-4) <==== With this
Pretty dumb issue... Thank you guys!
import pandas as pd
import numpy as numpy
from keras.layers import Dense, Dropout, Activation, Flatten,Reshape
from keras.layers import Conv1D, MaxPooling1D, LeakyReLU
from keras.utils import np_utils
from keras.layers import GRU,CuDNNGRU
from keras.callbacks import CSVLogger, ModelCheckpoint
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers.experimental import preprocessing
import matplotlib.pyplot as plt
from tensorflow import keras
from tensorflow.keras.models import Sequential
import tensorflow as tf
import h5py
import os
import pandas as pd
datas = pd.read_csv("D:/freqtrade/user_data/ML/New_Approach_16_08_2021/csv/ADA_USDT.csv")
datas.drop(columns=["date"], inplace=True)
print(datas.isna().sum())
datas.dropna(inplace=True)
labels = datas.columns
datas['target'] = datas['close'].shift(-4)
target = datas['target']
datas = datas[:].values
X_train, X_test, y_train, y_test = train_test_split(datas, target, test_size=0.33)
X_train = tf.convert_to_tensor(X_train, dtype=tf.float32)
X_test = tf.convert_to_tensor(X_test, dtype=tf.float32)
def build_and_compile_model():
model = Sequential()
model.add(Dense(200, input_shape=(222,), activation="relu"))
# model.add(Dropout(0.2))
model.add(Dense(180, activation="tanh"))
model.add(Dense(100, activation="relu"))
model.add(Dense(64, activation='relu'))
model.add(Dense(12, activation='relu'))
model.add(Dense(1))
model.compile(loss='mse', metrics=['accuracy'],
optimizer='adam')
return model
dnn_model = build_and_compile_model()
history = dnn_model.fit(
X_train, y_train,
validation_split=0.2,
verbose=1, epochs=2, batch_size=32)
dnn_model.summary()
history.model.save('dnn_model2')
X_test.columns = labels
test_predictions = history.model.predict(X_test)
print(test_predictions)
1438/1438 [==============================] - 6s 4ms/step - loss: nan - accuracy: 0.0000e+00 - val_loss: nan - val_accuracy: 0.0000e+00
Epoch 2/2
1438/1438 [==============================] - 6s 4ms/step - loss: nan - accuracy: 0.0000e+00 -
My goal is to train a convolutional neural network to recognise the images present in the mnist sign language dataset. Here is my attempt to process the data and train the model
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
import random
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D, Activation, Dropout, Flatten, Dense
import cv2
import keras
import sys
import tensorflow as tf
from keras import optimizers
import json
train_df = pd.read_csv("data/sign_mnist_train.csv")
test_df = pd.read_csv("data/sign_mnist_test.csv")
X = np.array(train_df.drop(["label"], axis=1))
y = np.array(train_df[["label"]])
X = X.reshape(-1, 28, 28, 1)
X = tf.cast(X, tf.float32)
model = Sequential()
model.add(Conv2D(28, (3,3), activation = 'relu'))
model.add(MaxPooling2D((2,2)))
model.add(Flatten())
model.add(Dense(24, activation = 'softmax'))
model.compile(optimizer='RMSprop',
loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(X, y, epochs=10, validation_split=0.2)
and after running this I get this result
Epoch 1/10
687/687 [==============================] - 4s 6ms/step - loss: 174.9729 - accuracy: 0.0438 - val_loss: 174.6281 - val_accuracy: 0.0382
Epoch 2/10
687/687 [==============================] - 2s 3ms/step - loss: 174.9779 - accuracy: 0.0433 - val_loss: 174.6281 - val_accuracy: 0.0382
Epoch 3/10
687/687 [==============================] - 2s 3ms/step - loss: 174.9777 - accuracy: 0.0433 - val_loss: 174.6281 - val_accuracy: 0.0382
and this continues for the remaining 7 epochs. My model is slightly different from what I have provided (for brevity) but this sequential model has the same issue, which makes me suspect that the issue must come before the model = Sequential() line. Furthermore, I have tried countless combinations of optimizers/loss and all those do is make the accuracy/loss converge to slightly different numbers, so I doubt that's the problem.
One of potential is that you use loss='binary_crossentropy' rather than loss='CategoricalCrossentropy'.
Besides, you defined the split datasets for training and testing, but you again defined it as model.fit(X, y, epochs=10, validation_split=0.2) to split datasets with 20% for validation and 80% for training.
I am using tf.random.set_seed to assure the reproducibility of my experiments but getting different results in terms of loss after training my model multiple times. I am monitoring the learning curve of each experiment using Tensorboard, but I am getting different values of loss and accuracy.
Providing the solution here (Answer Section), even though it is present in the Comment Section, for the benefit of the community.
To reproduce same results, you can create function as below and pass seeds directly to the layers as mentioned Daniel in the comments.
def reset_random_seeds():
os.environ['PYTHONHASHSEED']=str(2)
tf.random.set_seed(2)
np.random.seed(2)
random.seed(2)
Please refer complete code in below, which reproduced same results
import os
####*IMPORANT*: Have to do this line *before* importing tensorflow
os.environ['PYTHONHASHSEED']=str(2)
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers
import random
import pandas as pd
import numpy as np
def reset_random_seeds():
os.environ['PYTHONHASHSEED']=str(2)
tf.random.set_seed(2)
np.random.seed(2)
random.seed(2)
#make some random data
reset_random_seeds()
NUM_ROWS = 1000
NUM_FEATURES = 10
random_data = np.random.normal(size=(NUM_ROWS, NUM_FEATURES))
df = pd.DataFrame(data=random_data, columns=['x_' + str(ii) for ii in range(NUM_FEATURES)])
y = df.sum(axis=1) + np.random.normal(size=(NUM_ROWS))
def run(x, y):
reset_random_seeds()
model = keras.Sequential([
keras.layers.Dense(40, input_dim=df.shape[1], activation='relu'),
keras.layers.Dense(20, activation='relu'),
keras.layers.Dense(10, activation='relu'),
keras.layers.Dense(1, activation='linear')
])
NUM_EPOCHS = 100
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x, y, epochs=NUM_EPOCHS, verbose=0)
predictions = model.predict(x).flatten()
loss = model.evaluate(x, y) #This prints out the loss by side-effect
#With Tensorflow 2.0 this is now reproducible!
run(df, y)
run(df, y)
run(df, y)
Output:
32/32 [==============================] - 0s 2ms/step - loss: 0.5633
32/32 [==============================] - 0s 2ms/step - loss: 0.5633
32/32 [==============================] - 0s 2ms/step - loss: 0.5633
for tensorflow 2, use tf.random.set_seed(seed) instead
I'm testing simple networks on Keras with TensorFlow backend and I ran into an issue with using sigmoid activation function
The network isn't learning for first 5-10 epochs, and then everything is fine.
I tried using initializers and regularizers, but that only made it worse.
I use the network like this:
import numpy as np
import keras
from numpy import expand_dims
from keras.preprocessing.image import ImageDataGenerator
from matplotlib import pyplot
# load the image
(x_train, y_train), (x_val, y_val), (x_test, y_test) = netowork2_ker.load_data_shared()
# expand dimension to one sample
x_train = expand_dims(x_train, 2)
x_train = np.reshape(x_train, (50000, 28, 28))
x_train = expand_dims(x_train, 3)
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)
datagen = ImageDataGenerator(
rescale=1./255,
width_shift_range=[-1, 0, 1],
height_shift_range=[-1, 0, 1],
rotation_range=10)
epochs = 20
batch_size = 50
num_classes = 10
model = keras.Sequential()
model.add(keras.layers.Conv2D(64, (3, 3), padding='same',
input_shape=x_train.shape[1:],
activation='sigmoid'))
model.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(keras.layers.Conv2D(100, (3, 3),
activation='sigmoid'))
model.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(100,
activation='sigmoid'))
#model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(num_classes,
activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),
steps_per_epoch=len(x_train) / batch_size, epochs=epochs,
verbose=2, shuffle=True)
With the code above I get results like these:
Epoch 1/20
- 55s - loss: 2.3098 - accuracy: 0.1036
Epoch 2/20
- 56s - loss: 2.3064 - accuracy: 0.1038
Epoch 3/20
- 56s - loss: 2.3068 - accuracy: 0.1025
Epoch 4/20
- 56s - loss: 2.3060 - accuracy: 0.1079
...
For 7 epochs (different every time) and then the loss rapidly goes downward and i achieve 0.9623 accuracy in 20 epochs.
But if I change activation from sigmoid to relu it works great and gives me 0.5356 accuracy in the first epoch.
This issue makes sigmoid almost unusable for me and I'd like to know, I can do something about it. Is this a bug or am I doing something wrong?
Activation function suggestion：
In practice, the sigmoid non-linearity has recently fallen out of favor and it is rarely ever used. ReLU is the most common choice, if there are a large fraction of “dead” units in network, try Leaky ReLU and tanh. Never use sigmoid.
Reasons for not using the sigmoid：
A very undesirable property of the sigmoid neuron is that when the neuron’s activation saturates at either tail of 0 or 1, the gradient at these regions is almost zero. In addition, Sigmoid outputs are not zero-centered.
Im a new in deeplearning. I have a X with 34 dimension, which are some stock technical indicator data. And Y is label, which is binary(1,-1) represent the stock is uptrend or downtrend.Here is my code.
import pandas as pd
import numpy as np
data = pd.read_csv('data/data_week.csv')
data.dropna(inplace=True)
x = data.loc[:, 'bbands_upperband':'turn_std_5']
y = data['label']
from keras.models import Sequential
from keras.layers import Dense, Activation
model = Sequential()
model.add(Dense(512, activation='relu', input_dim=34))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(200, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(1, activation='relu'))
model.compile(optimizer='rmsprop',
loss='binary_crossentropy',
metrics=['accuracy'])
model.fit(x, y, epochs=2, batch_size=200)
231200/1041021 [=====>........................] - ETA: 59s - loss: 0.7098 - acc: 0.0086
232000/1041021 [=====>........................] - ETA: 59s - loss: 0.7087 - acc: 0.0086
However, the accuracy is less than 1%. I think this must has something wrong.
If you know please tell me and thank you very much!
For a binary classification model you should use sigmoid activation function in your last dense layer
model.add(Dense(1, activation='sigmoid'))
also, your classes must be (0,1) not (-1,1)