Develop Reference

Time series classification using CNN

I am trying to build a convolutional neural network which classifies time series data into two classes. For the time being I only have a small dataset so what I need first is to augment my datasets so I can feed them into a network.
For the data augmentation task, I found some very helpful methods at https://github.com/uchidalab/time_series_augmentation repository. What I have tried so far is to add some gaussian noise to my data, a permutation method, a time warping, a window slice and a window warp methods. These methods are being applied on a (batches, batch_rows, channels)=(354, 400, 3) dataset to generate a (1770, 400, 3) dataset (including train and test datasets and their corresponding labels).
Given the fact that I have a limited number of inputs, I would like to know if you have any suggestions for a 1D CNN structure for a good performance over these datasets.
What I have tried so far is this network:
verbose, epochs, batch_size = 0, 10, 8
n_timesteps, n_features, n_outputs = trainX.shape[1], trainX.shape[2], trainy.shape[1]
model = Sequential()
model.add(Conv1D(filters=16, kernel_size=3, activation='relu', input_shape=(n_timesteps,n_features)))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(n_outputs, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# fit network
model.fit(trainX, trainy, epochs=epochs, batch_size=batch_size, verbose=verbose)
# evaluate model
_, accuracy = model.evaluate(testX, testy, batch_size=batch_size, verbose=0)
No matter the changes I make in the parameters and the hyperparameters, I always get an accuracy around 50%, meaning that a binary classifier does not exists.
I would really appreciate if anyone can tell me what probably is the problem. Does this happens due to poor data quality produced by the augmentation methods? Or is it has to do with the network itself?
Thanks in advance

If it's a classification between two classes, you should use binary_crossentropy as loss function.

Validation accuracy is low and not increasing while training accuracy is increasing

I am a newbie to Keras and machine learning in general. I’m trying to build a classification model using the Sequential model. After some experiments, I see that my validation accuracy behavior is very low and not increasing, although the training accuracy works well. I added regularization parameters to the layers and dropouts also in between the layers. Still, the behavior exists. Here’s my code.
from keras.regularizers import l2
model = keras.models.Sequential()
model.add(keras.layers.Conv1D(filters=32, kernel_size=1, strides=1, padding="SAME", activation="relu", input_shape=[512,1],kernel_regularizer=keras.regularizers.l2(l=0.1))) # 一定要加 input shape
keras.layers.Dropout=0.35
model.add(keras.layers.MaxPool1D(pool_size=1,activity_regularizer=l2(0.01)))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(256, activation="softmax",activity_regularizer=l2(0.01)))
model.compile(loss="sparse_categorical_crossentropy",
optimizer="adam",
metrics=["accuracy"])
Ahistory = model.fit(train_x, trainy, epochs=300,
validation_split = 0.2,
batch_size = 16)
And here is the final results I got.
What is the reason behind this.? How do I fine-tune the model.?

How do I decrease loss on keras sequential model

I am hoping to get some guidance on what steps I should take next in my attempt to model a certain system. It contains 3 independent variables, 24 dependent variables, and about 21,000 rows. In my attempts to model, I cannot get the accuracy above about 50% or the loss less than about 6500. I've been using variations on the following code:
EPOCHS = 30
#OPTIMIZER = 'adam'
#OPTIMIZER = 'adagrad'
BATCH_SIZE = 10
OUTPUT_UNITS = len(y.columns)
print(f'OUTPUT_UNITS: {OUTPUT_UNITS}')
model = Sequential()
model.add(Dense(8, activation='relu', input_dim=3)) # 3 X parameters, with eng_speed removed
#model.add(Dense(8, activation='relu', input_dim=4)) # 4 X parameters
model.add(Dense(32, activation='relu' ))
#model.add(Dense(64, activation='relu' ))
#model.add(Dense(12, activation='relu' ))
model.add(Dense(OUTPUT_UNITS)) # number of predicted (y) values (labels)
model.summary()
adadelta = optimizers.Adadelta()
adam = optimizers.Adam(lr=0.001)
model.compile(optimizer=adadelta, loss='mse', metrics=['accuracy'])
#model.compile(optimizer=opt, loss='mse', metrics=['accuracy'])
history = model.fit(x=X_train, y=y_train, epochs=EPOCHS, batch_size=BATCH_SIZE)
I've tried removing and adding layers, changing the size of them, different optimizers, learning rates, etc. The following two graphs are typical of what I've been seeing--they both flatten very quickly, then don't improve:
I'm obviously new at this and would appreciate it if someone pointed me in the right direction: an approach to try, something to read up on, whatever. Thanks in advance.

Since (according to your mse loss and your regression tag) you are in a regression setting, accuracy is meaningless (it is only used in classification settings); please see own answer in What function defines accuracy in Keras when the loss is mean squared error (MSE)?
Given that, there is in principle absolutely no reason to consider a loss of 6500 as "high", and hence needing improvement...

Can a neural network be configured to output a matrix in Keras?

I am working with predicting a time series in 3 dimensions and would like to know if it is possible to configure a model to output a martix in Keras.
Currently, I have 3 regression models I train one after the other, one for predicting each output dimension. With a prediction horizon of 10 samples for example, each model is outputting a 10x1 vector. However, it seems like this could be done much more efficiently with a single model.
Thank you

I have found a much better way to do this using the Keras core layer Reshape. For a prediction horizon by predicted variables sized output, add a Reshape layer after the dense layer with that shape
from keras.layers import Dense, Reshape, Sequential, LSTM
model = Sequential()
model.add(LSTM(100, activation='relu', return_sequences=True, input_shape=(n_steps_in, n_features)))
model.add(LSTM(100, activation='relu'))
model.add(Dense(n_steps_out*n_features))
model.add(Reshape((n_steps_out,n_features)))
model.compile(optimizer='adam', loss='mse')

I figured out a pretty easy work around. I just reshape the targets on the way in and reshape the predictions on the way out.
input_data = input_data.reshape((num_simulations,input_samples*3))
target_data = target_data.reshape((num_simulations,horizon*3))
model.fit(input_data, target_data, validation_split=0.2, epochs=epochs,
batch_size=batch_size, verbose=0, shuffle=True)
prediction = model.predict(input_data, batch_size=batch_size)
prediction = prediction.reshape((num_simulations,horizon,3))

MLP with keras for prediction

I try to create a neural network with keras (backened tensorflow).
I have 4 Input and 2 Output variables:
not available
I want to do predictions to a Testset not available.
This is my Code:
from keras import optimizers
from keras.models import Sequential
from keras.layers import Dense
import numpy
numpy.random.seed(7)
dataset = numpy.loadtxt("trainingsdata.csv", delimiter=";")
X = dataset[:,0:4]
Y = dataset[:,4:6]
model = Sequential()
model.add(Dense(4, input_dim=4, init='uniform', activation='sigmoid'))
model.add(Dense(3, init='uniform', activation='sigmoid'))
model.add(Dense(2, init='uniform', activation='linear'))
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['accuracy'])
model.fit(X, Y, epochs=150, batch_size=10, verbose=2)
testset = numpy.loadtxt("testdata.csv", delimiter=";")
Z = testset[:,0:4]
predictions = model.predict(Z)
print(predictions)
When I run the script, the accuracy is 1.000 after every epoch and I get as result always the same output for every input pair:
[-5.83297 68.2967]
[-5.83297 68.2967]
[-5.83297 68.2967]
...
Has anybody an idea what the fault in my code is?

I suggest you normalize / standardize your data before feeding it to your model and then check if your model starts to learn.
Have a look at scikit-learn's StandardScaler.
And look into this SO thread to learn how to correctly fit_transform your training data and only transform your test data.
There is also this tutorial that makes use of scikit-learn's data preprocessing pipeline: http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/

Neural networks have a tough time if the scale of the input variables is too different from each other. Having 10, 1000, 100000 as the same inputs causes the gradients to collapse towards whatever the large value is. The other values effectively don't provide any information.
One method is to simply rescale the input variables by a constant. You can simply divide the 206000 by 100000. Try getting all of the variables to be at around the same number of digits. Large numbers are a bit harder than small numbers, for networks.