I am learning to use Keras functional API and I have managed to build and compile a model. But when I call the model.fit passing the data X and labels y, I got an error. It seems I still haven't got the idea of how it works.
The task is classifying sentences into 6 types, and the code goes:
X_ = ... # shape: (2787, 100) each row a sentence and each column a feature
y_= ... # shape: (2787,)
word_matrix_weights= ... # code to initiate a lookup matrix for vocabulary embeddings. shape: (9825,300)
deep_inputs = Input(shape=(100,))
embedding = Embedding(9825, 300, input_length=100,
weights=[word_matrix_weights], trainable=False)(deep_inputs)
flat = Flatten()(embedding)
hidden = Dense(6, activation="softmax")(flat)
model = Model(inputs=deep_inputs, outputs=hidden)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x=X_,y=y_,epochs=100, batch_size=10, verbose=0) #error here
The last line generates an error:
File "/home/zz/Programs/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1555, in fit
batch_size=batch_size)
File "/home/zz/Programs/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1413, in _standardize_user_data
exception_prefix='target')
File "/home/zz/Programs/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 154, in _standardize_input_data
str(array.shape))
ValueError: Error when checking target: expected dense_1 to have shape (None, 6) but got array with shape (2878, 1)
Any suggestions, please?
You have a Dense layer with 6 units and softmax activation as the last layer. So its output would be of shape (?,6) where each of those 6 values indicates the probability of belonging to corresponding class. Since you have used categorical_crossentropy as the loss function, the labels (i.e. y_) should have the same shape (i.e. (2787,6)) as well. You can one-hot encode y_ by using to_categorical method:
from keras.utils import to_categorical
y_ = to_categorical(y_)
This one-hot encodes the labels, i.e. converts 3 to [0,0,0,1,0,0] (assuming label numbers start from zero).
If you don't want to one-hot encode your labels you can change the loss argument to 'sparse_categorical_crossentropy'.
Related
I am building a prediction model for sequence data using conv1d layer provided by Keras. This is how I did
input_layer = Input(shape=(500,))
layer = Conv1D(128,5,activation="relu")(input_layer)
layer = MaxPooling1D(pool_size=2)(layer)
layer = Flatten()(layer)
layer = Dense(128, activation='relu')(layer)
output_layer = Dense(10, activation='softmax')(layer)
classifier = Model(input_layer, output_layer)
classifier.summary()
classifier.compile(optimizer=optimizers.Adam(), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
return classifier
However, am facing the following error:
Traceback (most recent call last):
File "train.py", line 71, in <module>
classifier = create_cnn_model()
File "train.py", line 60, in create_cnn_model
layer = Conv1D(128,5, activation="relu")(input_layer)
File "C:\Python368\lib\site-packages\keras\backend\tensorflow_backend.py", line 75, in symbolic_fn
_wrapper
return func(*args, **kwargs)
File "C:\Python368\lib\site-packages\keras\engine\base_layer.py", line 446, in __call__
self.assert_input_compatibility(inputs)
File "C:\Python368\lib\site-packages\keras\engine\base_layer.py", line 342, in assert_input_compat
ibility
str(K.ndim(x)))
ValueError: Input 0 is incompatible with layer conv1d_1: expected ndim=3, found ndim=2
I think the input_shape in the first layer is not setup right. How to set it up?
Right, conv layers need 3 dimensional input.
I am assuming you have a univariate time series with 500 samples.
You need to write a function to split the time series into steps.
For example:
x y
[t-n,...,t-2,t-1] t
So you are basically using the last n values to predict the next value in your series.
Then your input shape will be [len(x), n, 1]
The TensorFlow documentation for tf.nn.sparse_softmax_cross_entropy_with_logits explicitly declares that I should not apply softmax to the inputs of this op:
This op expects unscaled logits, since it performs a softmax on logits
internally for efficiency. Do not call this op with the output of
softmax, as it will produce incorrect results.
However if I use cross entropy without softmax it gives me unexpected results. According to CS231n course the expected loss value is around 2.3 for CIFAR-10:
For example, for CIFAR-10 with a Softmax classifier we would expect
the initial loss to be 2.302, because we expect a diffuse probability
of 0.1 for each class (since there are 10 classes), and Softmax loss
is the negative log probability of the correct class so: -ln(0.1) =
2.302.
However without softmax I get much bigger values, for example 108.91984.
What exactly am I doing wrong with sparse_softmax_cross_entropy_with_logits? The TF code is shown below.
import tensorflow as tf
import numpy as np
from tensorflow.python import keras
(_, _), (x_test, y_test) = keras.datasets.cifar10.load_data()
x_test = np.reshape(x_test, [-1, 32, 32, 3])
y_test = np.reshape(y_test, (10000,))
y_test = y_test.astype(np.int32)
x = tf.placeholder(dtype=tf.float32, shape=(None, 32, 32, 3))
y = tf.placeholder(dtype=tf.int32, shape=(None,))
layer = tf.layers.Conv2D(filters=16, kernel_size=3)(x)
layer = tf.nn.relu(layer)
layer = tf.layers.Flatten()(layer)
layer = tf.layers.Dense(units=1000)(layer)
layer = tf.nn.relu(layer)
logits = tf.layers.Dense(units=10)(layer)
# If this line is uncommented I get expected value around 2.3
# logits = tf.nn.softmax(logits)
loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
logits=logits)
loss = tf.reduce_mean(loss, name='cross_entropy')
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
res = sess.run(loss, feed_dict={x: x_test[0:256], y: y_test[0:256]})
print("loss: ", res)
# Expected output is value close to 2.3
# Real outputs are 108.91984, 72.82324, etc.
The issue is not in the lines
# If this line is uncommented I get expected value around 2.3
# logits = tf.nn.softmax(logits)
Images in cifar10 dataset are in RGB, thus pixel values are in range [0, 256). If you divide your x_test by 255
x_test = np.reshape(x_test, [-1, 32, 32, 3]).astype(np.float32) / 255
the values will be rescaled to [0,1] and tf.nn.sparse_softmax_cross_entropy_with_logits will return expected values
I'm attempting to follow along on what I'm thinking is the 5th or 6th simple introductory tutorial for keras that almost but never quite works.
Stripping everything out, I appear to come down to a problem with the format of my input. I read in an array of images, and extract two types, images of sign language ones and images of sign language zeros. I then set up an array of ones and zeros to correspond to what the images actually are, then make sure of sizes and types.
import numpy as np
from subprocess import check_output
print(check_output(["ls", "../data/keras/"]).decode("utf8"))
## load dataset of images of sign language numbers
x = np.load('../data/keras/npy_dataset/X.npy')
# Get the zeros and ones, construct a list of known values (Y)
X = np.concatenate((x[204:409], x[822:1027] ), axis=0) # from 0 to 204 is zero sign and from 205 to 410 is one sign
Y = np.concatenate((np.zeros(205), np.ones(205)), axis=0).reshape(X.shape[0],1)
# test shape and type
print("X shape: " , X.shape)
print("X class: " , type(X))
print("Y shape: " , Y.shape)
print("Y type: " , type(Y))
This gives me:
X shape: (410, 64, 64)
X class: <class 'numpy.ndarray'>
Y shape: (410, 1)
Y type: <class 'numpy.ndarray'>
which is all good. I then load the relevant bits from Keras, using Tensorflow as the backend and try to construct a classifier.
# get the relevant keras bits.
from keras.models import Sequential
from keras.layers import Convolution2D
# construct a classifier
classifier = Sequential() # initialize neural network
classifier.add(Convolution2D(32, (3, 3), input_shape=(410, 64, 64), activation="relu", data_format="channels_last"))
classifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
classifier.fit(X, Y, batch_size=32, epochs=10, verbose=1)
This results in:
ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (410, 64, 64)
This SO question, I think, suggests that my input shape needs to be altered to have a 4th dimension added to it - though it also says it's the output shape that needs to altered, I haven't been able to find anywhere to specify an output shape, so I'm assuming it is meant that I should alter the input shape to input_shape=(1, 64, 64, 1).
If I change my input shape however, then I immeadiately get this:
ValueError: Input 0 is incompatible with layer conv2d_1: expected ndim=4, found ndim=5
Which this github issue suggests is because I no longer need to specify the number of samples. So I'm left with the situation of using one input shape and getting one error, or changing it and getting another error.
Reading this and this made me think I might need to reshape my data to include information about the channels in X, but if I add in
X = X.reshape(X.shape[0], 64, 64, 1)
print(X.shape)
Then I get
ValueError: Error when checking target: expected conv2d_1 to have 4 dimensions, but got array with shape (410, 1)
If I change the reshape to anything else, i.e.
X = X.reshape(X.shape[0], 64, 64, 2)
Then I get a message saying it's unable to reshape the data, so I'm obviously doing something wrong with that, if that is, indeed, the problem.
I have read the suggested Conv2d docs which shed exactly zero light on the matter for me. Anyone else able to?
At first I used the following data sets (similar to your case):
import numpy as np
import keras
X = np.random.randint(256, size=(410, 64, 64))
Y = np.random.randint(10, size=(410, 1))
x_train = X[:, :, :, np.newaxis]
y_train = keras.utils.to_categorical(Y, num_classes=10)
And then modified your code as follows to work:
from keras.models import Sequential
from keras.layers import Convolution2D, Flatten, Dense
classifier = Sequential() # initialize neural network
classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 1), activation="relu", data_format="channels_last"))
classifier.add(Flatten())
classifier.add(Dense(10, activation='softmax'))
classifier.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
classifier.fit(x_train, y_train, batch_size=32, epochs=10, verbose=1)
Changed the shape of X from 410 x 64 x 64 to 410 x 64 x 64 x 1 (with channel 1).
input_shape be the shape of a sample data, that is, 64 x 64 x 1.
Changed the shape of Y using keras.utils.to_categorical() (one-hot encoding with num_classes=10).
Before compiling, Flatten() and Dense() were applied because you want categorical_crossentropy.
I'm trying to load the bottleneck_features that I obtained from running resnet50 into a top layer model. I ran predict_generator on resnet and saved the resultant bottleneck_features to a npy file. I am unable to fit the model I have created because of the following error:
Traceback (most recent call last):
File "Labeled_Image_Recognition.py", line 119, in <module>
callbacks=[checkpointer])
File "/home/dillon/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/models.py", line 963, in fit
validation_steps=validation_steps)
File "/home/dillon/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/engine/training.py", line 1630, in fit
batch_size=batch_size)
File "/home/dillon/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/engine/training.py", line 1490, in _standardize_user_data
_check_array_lengths(x, y, sample_weights)
File "/home/dillon/anaconda3/envs/tensorflow/lib/python3.6/site-packages/keras/engine/training.py", line 220, in _check_array_lengths
'and ' + str(list(set_y)[0]) + ' target samples.')
ValueError: Input arrays should have the same number of samples as target arrays. Found 940286 input samples and 14951 target samples.
I'm not really sure what it means. I have 940286 total images in my train dir and there are 14951 total subdirs that these images are separated into. My two hypotheses are:
It is possible that I am not formatting the train_data and train_labels correctly.
I set up the model incorrectly
Any guidance into the right direction would be much appreciated!
Here is the code:
# Constants
num_train_dirs = 14951 #This is the total amount of classes I have
num_valid_dirs = 13168
def load_labels(path):
targets = os.listdir(path)
labels = np_utils.to_categorical(targets, len(targets))
return labels
def create_model(train_data):
model = Sequential()
model.add(Flatten(input_shape=train_data.shape[1:]))
model.add(Dense(num_train_dirs, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_train_dirs, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
return model
train_data = np.load(open('bottleneck_features/bottleneck_features_train.npy', 'rb'))
train_labels = load_labels(raid_train_dir)
valid_data = np.load(open('bottleneck_features/bottleneck_features_valid.npy', 'rb'))
valid_labels = train_labels
model = create_model(train_data)
model.summary()
checkpointer = ModelCheckpoint(filepath='weights/first_try.hdf5', verbose=1, save_best_only=True)
print("Fitting model...")
model.fit(train_data, train_labels,
epochs=50,
batch_size=100,
verbose=1,
validation_data=(valid_data, valid_labels),
callbacks=[checkpointer])
In case of supervised learning the number of input samples (X) must match the number of output (labels) samples (Y).
For example: if we want to fit (learn) a NN to recognize handwritten digits and we feed 10.000 images (X) to our model, then we should also pass 10.000 labels (Y).
In your case those numbers don't match.
I am working on image classification tasks and decided to use Lasagne + Nolearn for neural networks prototype.
All standard examples like MNIST numbers classification run well, but problems appear when I try to work with my own images.
I want to use 3-channel images, not grayscale.
And there is the code where I'm trying to get arrays from images:
img = Image.open(item)
img = ImageOps.fit(img, (256, 256), Image.ANTIALIAS)
img = np.asarray(img, dtype = 'float64') / 255.
img = img.transpose(2,0,1).reshape(3, 256, 256)
X.append(img)
Here is the code of NN and its fitting:
X, y = simple_load("new")
X = np.array(X)
y = np.array(y)
net1 = NeuralNet(
layers=[ # three layers: one hidden layer
('input', layers.InputLayer),
('hidden', layers.DenseLayer),
('output', layers.DenseLayer),
],
# layer parameters:
input_shape=(None, 65536), # 96x96 input pixels per batch
hidden_num_units=100, # number of units in hidden layer
output_nonlinearity=None, # output layer uses identity function
output_num_units=len(y), # 30 target values
# optimization method:
update=nesterov_momentum,
update_learning_rate=0.01,
update_momentum=0.9,
regression=True, # flag to indicate we're dealing with regression problem
max_epochs=400, # we want to train this many epochs
verbose=1,
)
net1.fit(X, y)
I recieve exceptions like this one:
Traceback (most recent call last):
File "las_mnist.py", line 39, in <module>
net1.fit(X[i], y[i])
File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 266, in fit
self.train_loop(X, y)
File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 273, in train_loop
X, y, self.eval_size)
File "/usr/local/lib/python2.7/dist-packages/nolearn/lasagne.py", line 377, in train_test_split
kf = KFold(y.shape[0], round(1. / eval_size))
IndexError: tuple index out of range
So, in which format do you "feed" your networks with image data?
Thanks for answers or any tips!
If you're doing classification you need to modify a couple of things:
In your code you have set regression = True. To do classification remove this line.
Ensure that your input shape matches the shape of X if want to input 3 distinct channels
Because you are doing classification you need the output to use a softmax nonlinearity (at the moment you have the identity which will not help you with classification)
X, y = simple_load("new")
X = np.array(X)
y = np.array(y)
net1 = NeuralNet(
layers=[ # three layers: one hidden layer
('input', layers.InputLayer),
('hidden', layers.DenseLayer),
('output', layers.DenseLayer),
],
# layer parameters:
input_shape=(None, 3, 256, 256), # TODO: change this
hidden_num_units=100, # number of units in hidden layer
output_nonlinearity=lasagne.nonlinearities.softmax, # TODO: change this
output_num_units=len(y), # 30 target values
# optimization method:
update=nesterov_momentum,
update_learning_rate=0.01,
update_momentum=0.9,
max_epochs=400, # we want to train this many epochs
verbose=1,
)
I also asked it in lasagne-users forum and Oliver Duerr helped me a lot with code sample:
https://groups.google.com/forum/#!topic/lasagne-users/8ZA7hr2wKfM