Develop Reference

Binary classification using 04-zoo.csv data

In the zoo data, there are several types of each sample.
For making the binary classification, type 0,1,2,3 gets label 0 and type 4,5,6 gets label 1.
It's too hard for me to make split the types into two labels. So, I adjusted the data just adding the labels like below screen capture.
and I built the model like below.
import tensorflow as to
import numpy as np
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense, Dropout
zoo = np.loadtxt('~\\data-04-zoo.csv', delimiter=',', dtype=np.float32)
features = zoo[:, 0:-1]
labels = zoo[:, [-1]]
X_train, X_test, Y_train, Y_test = train_test_split(features, labels, test_size=0.25, shuffle=True, random_state=77)
print('X_train shape:', X_train.shape)
print('X_test shape:', X_test.shape)
print('y_train shape:', Y_train.shape)
print('y_test shape:', Y_test.shape)
model = Sequential()
model.add(Dense(40, input_dim=17, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(40, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])
model.fit(X_train, Y_train, validation_split=0.25, shuffle=True, epochs=20, batch_size=2)
score = model.evaluate(X_test, Y_test, batch_size=2)
print(score)
My professor said that model can't be taught.
What's the problem?? I don't know how to adjust the codes.
The Result

Keras Look Ahead not training with RAdam

I'm attempting to use look ahead with RAdam as the inner optimizer. I've tried Look Ahead with Adam as the inner optimizer and seperately I've used RAdam as the optimizer with good results. However, when I try to use Look Ahead with RAdam accuracy actually gets worse during training.
I've verified that training goes well when these optimizers are run separately. I've also tried with and without warmup using RAdam.
from keras_lookahead import Lookahead
from keras_radam import RAdam
batch_size = 128
num_classes = 10
epochs = 20
Ranger=Lookahead(RAdam(total_steps=10000, warmup_proportion=0.1, min_lr=1e-5))
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))
model.summary()
model.compile(loss='categorical_crossentropy',
optimizer=Ranger,
metrics=['accuracy'])
history = model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test))
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
The model trains but accuracy starts poor (.0851) and gets worse (.0836). Any ideas why

Adding prior belief into a neural Network

I am busy with a classification problem, with three classes. One of the classes is never predicted/classified. I would like to know if there s anyway to inject a prior belief into my neural network, be design or not.
My football prediction model predicts [Draws , Home Win , Away Win]. My classes are pretty balanced (40% , 30 % , 30%). The class [Draw] that accounts for 40% of the data is the one the my NN never predicts. My dataset contains 1900 samples.
I am using a deep NN with 2 to 4 hidden layers.
My code of my best model(based on training/val loss) is as follows:
X_all = df.copy()
train_cols = ['a_line0','a_line1','a_line2','a_line3','a_line4','a_line5',
'a_line6','a_line7','a_line8','a_line9','a_line10','h_line0',
'h_line1','h_line2','h_line3','h_line4','h_line5','h_line6',
'h_line7','h_line8','h_line9','h_line10','odds0','odds1','odds2']
x = X_all[train_cols]
x_v = x.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x_v)
x = pd.DataFrame(x_scaled)
y = X_all['result']
ohe = OneHotEncoder(n_values=3,categories='auto')
y = ohe.fit_transform(y.reshape(-1,1))
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)
for lr,ep in [(0.001,300)]:
model = Sequential()
model.add(Dense(25, input_dim=25, activation='relu'))
model.add(Dense(36, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(12, activation='relu'))
model.add(Dense(3, activation='sigmoid'))
adam = kr.optimizers.Adam(lr=lr, decay=1e-6)
model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
model.fit(X_train, y_train, epochs=ep, batch_size=10,verbose = 0)
_, accuracy = model.evaluate(X_test, y_test)
_, accuracy1 = model.evaluate(X_train, y_train)
print('Testing Accuracy: %.2f' % (accuracy*100),'Train Accuracy: %.2f' % (accuracy1*100), 'learning rate : ', lr)
I apologise if the code is a bit messy.
My model also overfits by +- 16% (52% vs 68%) on this config of my network.

Since you are in a multi-class single-label setting (i.e. your labels are mutually exclusive), you should not use sigmoid as activation in your final layer; change it to
model.add(Dense(3, activation='softmax'))
Also, dropout should not be used by default; remove it for starters, and only add it if it improves the result.

Predict a text with bag of word approach

I am trying text classification using the bag of word model. Everything works fine till I use the test set for testing and evaluation of accuracy but how we can check the class of a single statement.
I have a data frame with 2 classes labels and body.
cout_vect = CountVectorizer()
final_count = cout_vect.fit_transform(df['body'].values.astype('U'))
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from keras.utils import np_utils
X_train, X_test, y_train, y_test = train_test_split(final_count, df['label'], test_size = .3, random_state=25)
model = Sequential()
model.add(Dense(264, input_dim=X_train.shape[1], activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(3, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
y_train = np_utils.to_categorical(y_train, num_classes=3)
y_test = np_utils.to_categorical(y_test, num_classes=3)
model.fit(X_train, y_train, epochs=50, batch_size=32)
model.evaluate(x=X_test, y=y_test, batch_size=None, verbose=1, sample_weight=None)
Now I want to predict this statement using my model. How to do this
I tried converting my statement to vector using the count vectorizer but according to the bag of word approach, it is just an 8 dimension vector.
x = "Your account balance has been deducted for 4300"
model.predict(x, batch_size=None, verbose=0, steps=None)

You need to do this:
# First transform the sentence to bag-of-words according to the already learnt vocabulary
x = cout_vect.transform([x])
# Then send the feature vector to the predict
print(model.predict(x, batch_size=None, verbose=0, steps=None))
You have not shown how you "I tried converting my statement to vector using the count vectorizer but according to the bag of word approach, it is just an 8 dimension vector.", but I'm guessing you did this:
cout_vect.fit_transform([x])
If you call fit() (or fit_transform()), the vectorizer will forget all the previous training and only remember the current vocab, hence you only got a feature vector of size 8, whereas your previous vector was of higher size.

Shape mismatch in LSTM in keras

I am trying to run a LSTM using Keras on my custom features set. I have train and test features in separate files. Each csv file contains 11 columns with last column as class label. There are total 40 classes in my dataset. The problem is I am not able to figure out the correct input_shape to the first layer. I had explored all the stackoverflow and github but still not able to solve this
Below is my complete code.
import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
numpy.random.seed(7)
train_dataset = numpy.loadtxt("train.csv", delimiter=",")
X_train = train_dataset[:, 0:10]
y_train = train_dataset[:, 10]
test_dataset = numpy.loadtxt("test.csv", delimiter=",")
X_test = test_dataset[:, 0:10]
y_test = test_dataset[:, 10]
model = Sequential()
model.add(LSTM(32, return_sequences=True, input_shape=X_train.shape))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(32))
model.add(Dense(1, activation='softmax'))
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=10, epochs=1)
score, acc = model.evaluate(X_test, y_test, batch_size=10)
print('Test score:', score)
print('Test accuracy:', acc * 100)
Whatever I change in input_shape parameter wither I get error in first LSTM layer of in fit method.

you don't have a time dimension in your input.
Input for RNN should be (batch_size, time_step, features) while your input has dimension (batch_size, features).
If you want to use your 10 columns one at a time you should reshape the array with
numpy.reshape(train_dataset, (-1, train_dataset.shape[1], 1))
Try this code:
train_dataset = numpy.loadtxt("train.csv", delimiter=",")
train_dataset = numpy.reshape(train_dataset, (-1, train_dataset.shape[1], 1))
X_train = train_dataset[:, 0:10]
y_train = train_dataset[:, 10]
test_dataset = numpy.loadtxt("test.csv", delimiter=",")
test_dataset = numpy.reshape(test_dataset, (-1, train_dataset.shape[1], 1))
X_test = test_dataset[:, 0:10]
y_test = test_dataset[:, 10]
model = Sequential()
model.add(LSTM(32, return_sequences=True, input_shape=(X_train.shape[1], 1)))
model.add(LSTM(32, return_sequences=True))
model.add(LSTM(32))
model.add(Dense(1, activation='softmax'))
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['accuracy'])
model.fit(X_train, y_train, batch_size=10, epochs=1)
score, acc = model.evaluate(X_test, y_test, batch_size=10)
print('Test score:', score)
print('Test accuracy:', acc * 100)