PCA Implementation on a Convolutional Neural Network - python

After applying PCA on MNIST data, I identified CNN model and layers. After fitting CNN model (X_train_PCA, Y_train) I end up with dimension problem at evaluation phase. Here is the message
"ValueError: Error when checking input: expected conv2d_1_input to have shape (1, 10, 10) but got array with shape (1, 28, 28)". When I try to reshape X_test into 10X10 format, I got a very low score
First I applied min-max regularization, and then PCA to X_train. Then, I produced validation data from X_train. The problem is; I can fit the data in 100 dimension format(after applying PCA), my input data becomes 10X10. When I try to get score from fitted model using X_test which is still (10000, 1, 28, 28)). I get an errors as mentioned above. How can I solve dimension problem. I also tried to transform X_test with minmaxscaler and PCA. No change in score
pca_3D = PCA(n_components=100)
X_train_pca = pca_3D.fit_transform(X_train)
X_train_pca.shape
cnn_model_1_scores = cnn_model_1.evaluate(X_test, Y_test, verbose=0)
# Split the data into training, validation and test sets
X_train1 = X_pca_proj_3D[:train_size]
X_valid = X_pca_proj_3D[train_size:]
Y_train1 = Y_train[:train_size]
Y_valid = Y_train[train_size:]
# We need to convert the input into (samples, channels, rows, cols) format
X_train1 = X_train1.reshape(X_train1.shape[0], 1, 10,
10).astype('float32')
X_valid = X_valid.reshape(X_valid.shape[0], 1, 10, 10).astype('float32')
X_test = X_test.reshape(X_test.shape[0], 1, 28, 28).astype('float32')
X_train1.shape, X_valid.shape, X_test.shape
((51000, 1, 10, 10), (9000, 1, 10, 10), (10000, 1, 28, 28))
#create model
cnn_model_1=Sequential()
#1st Dense Layer
cnn_model_1.add(Conv2D(32, kernel_size=(5,5),
data_format="channels_first",
input_shape=(1,10,10),
activation='relu'))
#Max-Pooling
cnn_model_1.add(MaxPooling2D(pool_size=(2,2)))
#Max pooling is a sample-based discretization process. The objective is to
down-sample an input representation (image, hidden-layer output matrix,
etc.), reducing its dimensionality
# the number of layers, remains unchanged in the pooling operation
#cnn_model_1.add(BatchNormalization())
#Dropout
cnn_model_1.add(Flatten())
#cnn_model_1.add(BatchNormalization())
#2nd Dense Layer
cnn_model_1.add(Dense(128, activation='relu'))
#final softmax layer
cnn_model_1.add(Dense(10, activation='softmax'))
# print a summary and check if you created the network you intended
cnn_model_1.summary()
#Compile Model
cnn_model_1.compile(loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy'])
#Fit the model
cnn_model_1_history=cnn_model_1.fit(X_train1, Y_train1,
validation_data=(X_valid, Y_valid), epochs=5, batch_size=100, verbose=2)
# Final evaluation of the model
cnn_model_1_scores = cnn_model_1.evaluate(X_test, Y_test, verbose=0)
print("Baseline Test Accuracy={0:.2f}% (categorical_crossentropy) loss=
{1:.2f}".format(cnn_model_1_scores[1]*100, cnn_model_1_scores[0]))
cnn_model_1_scores

I solved the problem, updating the post to give intuition for other coders to debug their code. First, I applied PCA on X_test data and after getting low score I tried without applying. As #Scott suggested, this was wrong. After carefully checking my code, I saw that I forgot to change X_test to X_test_pca after applying PCA on test data while constructing CNN model. I also fitted PCA on X_train while applying PCA on X_test data.

Related

Why are LSTM prediction shape not as expected?

I am having a time series dataset with the timestamp and one value (in total 2 columns only) and training an LSTM to predict the values for each hour.
So I prepare the data set to the model is like below :
Take the last 5 previous values of the value as X and observed value for the hour as y.
then I split the train and test for each X and y.
So I have the train and data sets with the below shape after scaling it with a min-max scaler.
print(train_X.shape,train_y.shape,test_X.shape,test_y.shape)
(16195, 5) (16195,) (8716, 5) (8716,)
then I build the model by
model = Sequential()
model.add(LSTM(5, input_shape=(n_steps,n_features),recurrent_dropout=0.2,return_sequences=True))
model.add(BatchNormalization())
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
I fit the model and predict by
history = model.fit(train_X, train_y, epochs=10, batch_size=64,validation_data=(test_X, test_y),shuffle=False)
#predict the instances
predicted = model.predict(test_X)
I have now predicted in the shape of (8716, 5, 1).
which is not correct I guess because the prediction should be the same as test_y shape which is (8716,).
So when I reshape to inverse scale
predicted=yhat.reshape(predicted.shape[0], -1).reshape(-1, 1)
inverse_predictions= scaler_y.inverse_transform(predicted)
This gives the shape as (43580, 1) which is wrong because the predicted is having the dimension (8716, 5, 1) instead of (8716,).
I am not sure which part is causing the error. Any help is appreciated.
You can delete return_sequences=True, that should fix the issue.
Alternatively you can use a flattening layer but I don't think this is what you would want to do here.

Keras - Number of samples error with multidimensional input

I'm trying to construct a (very simple) Keras model as a baseline for a project. I have a list of 3459 numpy arrays of shape (2, 6, 15) as input, and a list of target values (ints as numpy arrays with shape ()). When I try to train the model I get this error:
"ValueError: Number of samples 2 is less than samples required for specified batch_size 32 and steps 108."
The model so far is extremely simple, but I'm having no luck getting it to train:
input = Input(shape=(2, 6, 15))
x = Dense(64, activation='relu')(input)
x = Dense(64, activation='relu')(x)
output = Dense(1)(x)
model = Model(inputs=input, outputs=output)
model.compile(optimizer='adam', loss='mean_squared_error',
metrics=['accuracy'])
hist = model.fit(
X_train,
y_train,
batch_size=32,
epochs=10,
validation_data=(X_test, y_test),
steps_per_epoch=(len(X_train) // 32),
validation_steps=(len(X_test) // 32))
I'm currently loading the data from pickle files, and I suspect that the issue might be the array structure of the individual training cases. When looking at one of the arrays in the X_train it has a structure [[[...]...], [[...]...]], and I suspect the code is confusing the outer brackets as the batch container, so it's reading a batch size of 2 as input instead. Just a theory, but I don't know how to address that to check for myself.
The error is indeed due to the way your data is being generated/loaded; there are no errors if you train the model on random tensors with the specified shapes.

Keras official example of LSTM classifier using real values for training target?

From the official example in Keras docs, the stacked LSTM classifier is trained using categorical_crossentropy as a loss function, as expected. https://keras.io/getting-started/sequential-model-guide/#examples
But the y_train values are seeded using numpy.random.random() which outputs real numbers, versus 0,1 binary classification ( which is typical )
Are the y_train values being promoted to 0,1 values under the hood?
Can you even train this loss function against real values between 0,1 ?
How is accuracy then calculated ?
Confusing.. no?
from keras.models import Sequential
from keras.layers import LSTM, Dense
import numpy as np
data_dim = 16
timesteps = 8
num_classes = 10
# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
input_shape=(timesteps, data_dim))) # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True)) # returns a sequence of vectors of dimension 32
model.add(LSTM(32)) # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))
model.compile(loss='categorical_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# Generate dummy training data
x_train = np.random.random((1000, timesteps, data_dim))
y_train = np.random.random((1000, num_classes))
# Generate dummy validation data
x_val = np.random.random((100, timesteps, data_dim))
y_val = np.random.random((100, num_classes))
model.fit(x_train, y_train,
batch_size=64, epochs=5,
validation_data=(x_val, y_val))
For this example, the y_train and y_test are not the one-hot encoding anymore, but the probabilities of each classes. So it is still applicable for cross-entropy. And we can treat the one-hot encoding as the special case of the probabilities vector.
y_train[0]
array([0.30172708, 0.69581121, 0.23264601, 0.87881279, 0.46294832,
0.5876406 , 0.16881395, 0.38856604, 0.00193709, 0.80681196])

Keras: Wrong Input Shape in LSTM Neural Network

I am trying to train an LSTM recurrent neural network, for sequence classification.
My data has the following formart:
Input: [1,5,2,3,6,2, ...] -> Output: 1
Input: [2,10,4,6,12,4, ...] -> Output: 1
Input: [4,1,7,1,9,2, ...] -> Output: 2
Input: [1,3,5,9,10,20, ...] -> Output: 3
.
.
.
So basically I want to provide a sequence as an input and get an integer as an output.
Each input sequence has length = 2000 float numbers, and I have around 1485 samples for training
The output is just an integer from 1 to 10
This is what I tried to do:
# Get the training numpy 2D array for the input (1485X 2000).
# Each element is an input sequence of length 2000
# eg: [ [1,2,3...], [4,5,6...], ... ]
x_train = get_training_x()
# Get the training numpy 2D array for the outputs (1485 X 1).
# Each element is an integer output for the corresponding input from x_train
# eg: [ 1, 2, 3, ...]
y_train = get_training_y()
# Create the model
model = Sequential()
model.add(LSTM(100, input_shape=(x_train.shape)))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
model.fit(x_train, y_train, nb_epoch=3, batch_size=64)
I get the following error:
Error when checking input: expected lstm_1_input to have 3 dimensions, but got array with shape (1485, 2000)
I tried using this instead:
model.add(LSTM(100, input_shape=(1485, 1, 2000)))
But got the another error this time:
ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=4
Can anyone explain what is my input shape? and what am I doing wrong?
Thanks
try reshaping your training data to:
x_train=x_train.reshape(x_train.shape[0], 1, x_train.shape[1])
input_shape=(None, x_train.shape[1], 1), where None is the batch size, x_train.shape[1] is the length of each sequence of features, and 1 is each feature length. (Not sure if batch size is necessary for Sequential model).
And then reshape your data into x_train = x_train.reshape(-1, x_train.shape[1], 1).
Given the format of your input and output, you can use parts of the approach taken by one of the official Keras examples. More specifically, since you are not creating a binary classifier, but rather predicting an integer, you can use one-hot encoding to encode y_train using to_categorical().
# Number of elements in each sample
num_vals = x_train.shape[1]
# Convert all samples in y_train to one-hot encoding
y_train = to_categorical(y_train)
# Get number of possible values for model inputs and outputs
num_x_tokens = np.amax(x_train) + 1
num_y_tokens = y_train.shape[1]
model = Sequential()
model.add(Embedding(num_x_tokens, 100))
model.add(LSTM(100))
model.add(Dense(num_y_tokens, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=64,
epochs=3)
The num_x_tokens in the code above would be the maximum size of the element in one of your input samples (e.g. if you have two samples [1, 7, 2] and [3, 5, 4] then num_x_tokens is 7). If you use numpy you can find this with np.amax(x_train). Similarly, num_y_tokens is the number of categories you have in y_train.
After training, you can run predictions using the code below. Using np.argmax effectively reverses to_categorical in this configuration.
model_out = model.predict(x_test)
model_out = np.argmax(model_out, axis=1)
You can import to_categorical using from keras.utils import to_categorical, Embedding using from keras.layers import Embedding, and numpy using import numpy as np.
Also, you don't have to do print(model.summary()). model.summary() is enough to print out the summary.
EDIT
If it is the case that the input is of the form [[0.12, 0.31, ...], [0.22, 0.95, ...], ...] (say, generated with x_train = np.random.rand(num_samples, num_vals)) then you can use x_train = np.reshape(x_train, (num_samples, num_vals, 1)) to change the shape of the array to input it into the LSTM layer. The code to train the model in that case would be:
num_samples = x_train.shape[0]
num_vals = x_train.shape[1] # Number of elements in each sample
# Reshape for what LSTM expects
x_train = np.reshape(x_train, (num_samples, num_vals, 1))
y_train = to_categorical(y_train)
# Get number of possible values for model outputs
num_y_tokens = y_train.shape[1]
model = Sequential()
model.add(LSTM(100, input_shape=(num_vals, 1)))
model.add(Dense(num_y_tokens, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=64,
epochs=3)
The num_vals is the length of each sample array in x_train. np.reshape(x_train, (num_samples, num_vals, 1)) changes each sample from [0.12, 0.31, ...] form to [[0.12], [0.31], ...] form, which is the shape that LSTM then takes (input_shape=(num_vals, 1)). The extra 1 seems strange in this case, but it is necessary to add an extra dimension to the input for the LSTM since it expects each sample to have at least two dimensions, typically called (timesteps, data_dim), or in this case (num_vals, 1).
To see how else LSTMs are used in Keras you can refer to:
Keras Sequential model guide (has several LSTM examples)
Keras examples (look for *.py files with lstm in their name)

Specifying Dense using keras library

I slightly misunderstand how to create a simple Sequence for my data.
The data has the following dimensions:
X_train.shape
(2369, 12)
y_train.shape
(2369,)
X_test.shape
(592, 12)
y_test.shape
(592,)
This is how I create the model:
batch_size = 128
nb_epoch = 20
in_out_neurons = X_train.shape[1]
dimof_middle = 100
model = Sequential()
model.add(Dense(batch_size, batch_input_shape=(None, in_out_neurons)))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(batch_size))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(in_out_neurons))
model.add(Activation('linear'))
# I am solving the regression problem, not the classification one
model.compile(loss="mean_squared_error", optimizer="rmsprop")
history = model.fit(X_train, y_train,
batch_size=batch_size, nb_epoch=nb_epoch,
verbose=1, validation_data=(X_test, y_test))
The error message:
Exception: Error when checking model input: expected dense_input_14 to
have shape (None, 1) but got array with shape (2369, 12)รง
The error is:
Error when checking model target: expected activation_42 to have shape
(None, 12) but got array with shape (2369, 1)
This error occurs at line:
model.add(Dense(in_out_neurons))
How to change Dense to make it work?
Another question is how to add a simple autoencoder in order to initialize weights of ANN?
One of your problems is that you seem to misunderstand what a batch is.
A batch is the number of training samples computed at a time, so instead of computing one training sample from X_train at a time you use, for example, 100 at a time. The important bit here is that this has nothing to do with your model.
So when you write
model.add(Dense(batch_size, batch_input_shape=(None, in_out_neurons)))
then you create a fully connected layer with an output size of one batch. That does not make a lot of sense.
Another problem is that your model's output is 12 neurons while your Y is only one value/neuron. Your model looks like this:
|
v
[128]
[128]
[ 12]
|
v
Then what fit() does is, it inputs a matrix of shape (128, 12) ((batch size, X_train.shape[1])) into the model and attempts to compare the output of shape (128,12) from the last layer to the corresponding Y values of the batch (shape (128,1)).

Categories

Resources