Develop Reference

Why do we reshape data?

I have got a dataset which consists of 100,000 rows and 12 columns, where each column stands of a certain input to train a sequential GRU model to predict only 1 output. The following is the code for the model:
model = Sequential()
model.add(GRU(units=70, return_sequences=True, input_shape=(1,12),activity_regularizer=regularizers.l2(0.0001)))
model.add(GRU(units=50, return_sequences=True,dropout=0.1))
model.add(GRU(units=30, dropout=0.1))
model.add(Dense(units=5))
model.add(Dense(units=3))
model.add(Dense(units=1, activation='relu'))
model.compile(loss=['mae'], optimizer=Adam(lr=0.0001),metrics=['mse'])
model.summary()
history=model.fit(X_train, y_train, batch_size=1000,epochs=30,validation_split=0.1, verbose=1)
However, before that I had to transform the training dataset from 2D to 3D using x_train=x.reshape(-1,1,12) and the output from 1D to 2D using y_train=y.reshape(-1,1). That is the part I really don't understand, why not just keep them as they are?

You had to describe your data in order to be decisive.
But since each layers output is the input of the next layer, their shape must be equal. In the incomplete example you gave your labels need to be a single value for each sample and I think that's why reshape was used.

Error while using Convolutional layers on CPU

Can I use Convolutional layers of keras without gpu support? I am getting errors when I use it on Colab with runtime as None.
My code looks like this:
model = tf.keras.Sequential()
model.add(layers.Conv1D(1,5, name='conv1', padding="same", activation='relu',data_format="channels_first", input_shape=(1,2048)))
# model.add(layers.LSTM(5, activation='tanh'))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(num_classes, activation='softmax'))
#model.summary()
model.compile(loss=tf.keras.losses.categorical_crossentropy,
optimizer=tf.keras.optimizers.SGD(lr=0.001, momentum=0.9),
metrics=['accuracy'])
x_train = train_value
y_train = train_label
x_test = test_value
y_test = test_label
print(np.shape(x_train)) #shape of x train is (4459, 1, 2048)
print(np.shape(x_test)) #shape of test is (1340,1,2048)
history = model.fit(x_train, y_train,
batch_size=100,
epochs=30,
verbose=1,
validation_data=(x_test, y_test)
)
It is running fine on GPU but gives following error on CPU:
InvalidArgumentError: Conv2DCustomBackpropFilterOp only supports NHWC.
[[{{node
training/SGD/gradients/gradients/conv1/conv1d_grad/Conv2DBackpropFilter}}]]
UnimplementedError: The Conv2D op currently only supports the NHWC
tensor format on the CPU. The op was given the format: NCHW [[{{node
conv1_1/conv1d}}]]
I have figured out that the problem is with the format of Input Data. My input data are vectors of size (1,2048). Can you please guide me on how to convert these vectors to NHWC format?
I would really appreciate it, if someone can clear this up for me.
Thanks in advance.

Per the Keras documentation
data_format: A string, one of "channels_last" (default) or "channels_first". The ordering of the dimensions in the inputs. "channels_last" corresponds to inputs with shape (batch, steps, channels) (default format for temporal data in Keras) while "channels_first" corresponds to inputs with shape (batch, channels, steps)
Now Keras in TensorFlow appears to implement Conv1D in terms of a Conv2D operator - basically forming an "image" with 1 row, W columns, and then your C "channels". That's why your getting error messages about image shapes when you don't have image data.
In the docs above "channels" are the number of data items per time step (e.g. perhaps you have 5 sensor readings at each time step so you'd have 5 channels). From your answers above I believe you're passing tensors with shape (n, 1, 2048) where n is your batch size. So, with channels_last TensorFlow thinks that means you have n examples in your batch each with a sequence length of 1 and 2048 data items per time step - that is only a single time step with 2048 data items per observation (e.g. 2048 sensor readings taken at each time step) in which case the convolution won't be doing a convolution - it'd be equivalent to a single dense layer taking all 2048 numbers as input.
I think in reality you have only a single data item per time step and you have 2048 time steps. That explains why passing channels_first improves your accuracy - now TensorFlow understand that your data represents 1 data item samples 2048 times and it can do a convolution over that data.
To fix you can just tf.reshape(t, (1, 2048, 1)) - and remove the channels_first (that code assumes you're doing batches of size 1 and your tensor is named t). Now it's in the format (n, s, 1) where n is the batch size (1 here), s is the number of time steps (2048), and 1 indicates one data point per time step. You can now run the same model on the GPU or CPU.

Why did I receive input error in a simple Keras Functional API?

I hope that together we can solve my problem, I will try to summarise it and paste in some codes.
At our research we are using TensorFlow/Keras for our DNN, to classify images (convolutional network). It was a quite simple Sequential model, but now we are trying to add more inputs, so I started to change the network to the Functional API (honestly, this is the first time that I used it). I recreated the original convolutional network and everything was working fine.
So I generated the necessary additional data into simple text files (one for each image) and these will be the second input in our model. This is were something went wrong, because I've got an error and now I am stuck with the solution.
To recreate the error, I made a new simple python file without the convolution part, just to try out the model's text file part. The input is 5000 text files, containing only a float number. After the preprocessing, we will have 4000 for train and additional 1000 for testing, both are stored in a numpy array. The train and test arrays were split through sklearn.model_selection.train_test_split.
In the original multi-channel network, I tried to concatenate the convolutional part with the text file and after it some dense layers are coming. But no concatenate here, just the train for the data stored in the text files.
Here are the shapes of the input and label arrays:
X_train.shape
(4000,)
y_train.shape
(4000, 5)
The - very - simple network:
inputA = Input(shape=(X_train.shape[0],))
x = Flatten()(inputA)
x = Dense(256, activation='relu')(x)
x = Dense(256, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(5, activation='softmax')(x)
x = Model(inputs=inputA, outputs = x)
Compile:
x.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
Model fit:
x.fit(X_train, y_train,
validation_data=(X_test, y_test),
batch_size=batch,
verbose=1,
epochs=epoch_number)
And the received error message:
ValueError: Error when checking input: expected input_8 to have shape (4000,) but got array with shape (1,)
The question is, what did I wrong? The previous codes are working just fine in the Sequential model, but here no. Could somebody help me solve this mystery?
Best wishes,
Tamas

It's because of this line:
inputA = Input(shape=(X_train.shape[0],))
Keras expects the number of features, not the number of samples. In your case, the input_shape=(1,).

How to make simple data recall in keras neural networks

I'm trying the get a hang of keras and I'm trying to get basic time series prediction working. My input is a list of random ints between 0 and 10 such as:[1,3,2,4,7,5,9,0] and my labels are the same as the input but delayed such as: [X,X,1,3,2,4,7,5] and I'm trying to have my model learn this relationship of remembering past data points.
My code is:
labels = keras.utils.to_categorical(output, num_keys)
model = keras.Sequential([
keras.layers.LSTM(10),
keras.layers.Dense(10, activation='relu'),
keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer=tf.train.AdamOptimizer(),
loss=tf.keras.losses.categorical_crossentropy,
metrics=['accuracy'])
model.fit(input, labels, epochs=30, verbose=2,shuffle=False)
and I get the error:ValueError: Please provide as model inputs either a single array or a list of arrays. You passed: x=[7, 6,...
I've tried reformating my input with:
input=numpy.array([[i,input[i]]for i in range(len(input))])
input=numpy.reshape(input,input.shape+(1,))
and adding input_shape=input.shape[1:] to my LSTM layer and that throws no errors but the accuracy is no better then just blind guessing
This seems like that kind of thing that could be trivial but I'm clearly missing something.

With keras.layers.LSTM(10), you need to include the input data shape: keras.layers.LSTM(10, input_shape = (input.shape[1], input.shape[2])).
Keras is expecting the input data shaped as [instances, time, predictors] and since you don't have any additional predictors, you may need to reshape your input data to input.reshape(input.shape[0], input.shape[1], 1).
Keras will infer the data shapes for the next layers, but the first layer needs the input shape defined.

How to handle variable sized input in CNN with Keras?

I am trying to perform the usual classification on the MNIST database but with randomly cropped digits.
Images are cropped the following way : removed randomly first/last and/or row/column.
I would like to use a Convolutional Neural Network using Keras (and Tensorflow backend) to perform convolution and then the usual classification.
Inputs are of variable size and i can't manage to get it to work.
Here is how I cropped digits
import numpy as np
from keras.utils import to_categorical
from sklearn.datasets import load_digits
digits = load_digits()
X = digits.images
X = np.expand_dims(X, axis=3)
X_crop = list()
for index in range(len(X)):
X_crop.append(X[index, np.random.randint(0,2):np.random.randint(7,9), np.random.randint(0,2):np.random.randint(7,9), :])
X_crop = np.array(X_crop)
y = to_categorical(digits.target)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_crop, y, train_size=0.8, test_size=0.2)
And here is the architecture of the model I want to use
from keras.layers import Dense, Dropout
from keras.layers.convolutional import Conv2D
from keras.models import Sequential
model = Sequential()
model.add(Conv2D(filters=10,
kernel_size=(3,3),
input_shape=(None, None, 1),
data_format='channels_last'))
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(10, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
model.summary()
model.fit(X_train, y_train, epochs=100, batch_size=16, validation_data=(X_test, y_test))
Does someone have an idea on how to handle variable sized input in my neural network?
And how to perform classification?

TL/DR - go to point 4
So - before we get to the point - let's fix some problems with your network:
Your network will not work because of activation: with categorical_crossentropy you need to have a softmax activation:
model.add(Dense(10, activation='softmax'))
Vectorize spatial tensors: as Daniel mentioned - you need to, at some stage, switch your vectors from spatial (images) to vectorized (vectors). Currently - applying Dense to output from a Conv2D is equivalent to (1, 1) convolution. So basically - output from your network is spatial - not vectorized what causes dimensionality mismatch (you can check that by running your network or checking the model.summary(). In order to change that you need to use either GlobalMaxPooling2D or GlobalAveragePooling2D. E.g.:
model.add(Conv2D(filters=10,
kernel_size=(3, 3),
input_shape=(None, None, 1),
padding="same",
data_format='channels_last'))
model.add(GlobalMaxPooling2D())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(10, activation='softmax'))
Concatenated numpy arrays need to have the same shape: if you check the shape of X_crop you'll see that it's not a spatial matrix. It's because you concatenated matrices with different shapes. Sadly - it's impossible to overcome this issue as numpy.array need to have a fixed shape.
How to make your network train on examples of different shape: The most important thing in doing this is to understand two things. First - is that in a single batch every image should have the same size. Second - is that calling fit multiple times is a bad idea - as you reset inner model states. So here is what needs to be done:
a. Write a function which crops a single batch - e.g. a get_cropped_batches_generator which given a matrix cuts a batch out of it and crops it randomly.
b. Use train_on_batch method. Here is an example code:
from six import next
batches_generator = get_cropped_batches_generator(X, batch_size=16)
losses = list()
for epoch_nb in range(nb_of_epochs):
epoch_losses = list()
for batch_nb in range(nb_of_batches):
# cropped_x has a different shape for different batches (in general)
cropped_x, cropped_y = next(batches_generator)
current_loss = model.train_on_batch(cropped_x, cropped_y)
epoch_losses.append(current_loss)
losses.append(epoch_losses.sum() / (1.0 * len(epoch_losses))
final_loss = losses.sum() / (1.0 * len(losses))
So - a few comments to code above: First, train_on_batch doesn't use nice keras progress bar. It returns a single loss value (for a given batch) - that's why I added logic to compute loss. You could use Progbar callback for that also. Second - you need to implement get_cropped_batches_generator - I haven't written a code to keep my answer a little bit more clear. You could ask another question on how to implement it. Last thing - I use six to keep compatibility between Python 2 and Python 3.

Usually, a model containing Dense layers cannot have variable size inputs, unless the outputs are also variable. But see the workaround and also the other answer using GlobalMaxPooling2D - The workaround is equivalent to GlobalAveragePooling2D. These are layers that can eliminiate the variable size before a Dense layer and suppress the spatial dimensions.
For an image classification case, you may want to resize the images outside the model.
When my images are in numpy format, I resize them like this:
from PIL import Image
im = Image.fromarray(imgNumpy)
im = im.resize(newSize,Image.LANCZOS) #you can use options other than LANCZOS as well
imgNumpy = np.asarray(im)
Why?
A convolutional layer has its weights as filters. There is a static filter size, and the same filter is applied to the image over and over.
But a dense layer has its weights based on the input. If there is 1 input, there is a set of weights. If there are 2 inputs, you've got twice as much weights. But weights must be trained, and changing the amount of weights will definitely change the result of the model.
As #Marcin commented, what I've said is true when your input shape for Dense layers has two dimensions: (batchSize,inputFeatures).
But actually keras dense layers can accept inputs with more dimensions. These additional dimensions (which come out of the convolutional layers) can vary in size. But this would make the output of these dense layers also variable in size.
Nonetheless, at the end you will need a fixed size for classification: 10 classes and that's it. For reducing the dimensions, people often use Flatten layers, and the error will appear here.
A possible fishy workaround (not tested):
At the end of the convolutional part of the model, use a lambda layer to condense all the values in a fixed size tensor, probably taking a mean of the side dimensions and keeping the channels (channels are not variable)
Suppose the last convolutional layer is:
model.add(Conv2D(filters,kernel_size,...))
#so its output shape is (None,None,None,filters) = (batchSize,side1,side2,filters)
Let's add a lambda layer to condense the spatial dimensions and keep only the filters dimension:
import keras.backend as K
def collapseSides(x):
axis=1 #if you're using the channels_last format (default)
axis=-1 #if you're using the channels_first format
#x has shape (batchSize, side1, side2, filters)
step1 = K.mean(x,axis=axis) #mean of side1
return K.mean(step1,axis=axis) #mean of side2
#this will result in a tensor shape of (batchSize,filters)
Since the amount of filters is fixed (you have kicked out the None dimensions), the dense layers should probably work:
model.add(Lambda(collapseSides,output_shape=(filters,)))
model.add(Dense.......)
.....
In order for this to possibly work, I suggest that the number of filters in the last convolutional layer be at least 10.
With this, you can make input_shape=(None,None,1)
If you're doing this, remember that you can only pass input data with a fixed size per batch. So you have to separate your entire data in smaller batches, each batch having images all of the same size. See here: Keras misinterprets training data shape