I am building a model based on this code for noise suppression. My problem with the vanilla implementation is that it loads all data at once, which is not the best idea when the training data gets really large; my input file, denoted in the linked code as training.h5, is over 30 GB.
I decided to instead go with tf.data interface that should allow me to work with large data sets; my problem here is that I don't know how to properly shape TFRecordDataset so that it meets what's required by the Model API.
If you check model.fit(x_train, [y_train, vad_train], it essentially requires the following:
x_train, shape [nb_sequences, window, 42]
y_train, shape [nb_sequences, window, 22]
vad_train, shape [nb_sequences, window, 1]
window one typically fixes (in the code: 2000), so the only variable nb_sequences that stems from how large is your data set. However, with tf.data, we don't supply x and y, but only x (see Model API docs).
Saving tfrecord to file
In an effort to make the code reproducible, I created the input file with the following code:
writer = tf.io.TFRecordWriter(path='example.tfrecord')
for record in data:
feature = {}
feature['X'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[:42]))
feature['y'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[42:64]))
feature['vad'] = tf.train.Feature(float_list=tf.train.FloatList(value=[record[64]]))
example = tf.train.Example(features=tf.train.Features(feature=feature))
serialized = example.SerializeToString()
writer.write(serialized)
writer.close()
data is our training data with shape [10000, 65]. My example.tfrecord is available here. It's 3 MB, in reality it would be 30 GB+.
You might notice that in the linked code, numpy array has shape [x, 87], while mine is [x, 65]. That's OK - the remainder is not used anywhere.
Loading the dataset with tf.data.TFRecordDataset
I would like to use tf.data to load "on demand" the data with some prefetching, there's no need to keep it all in memory. My attempt:
import datetime
import numpy as np
import h5py
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GRU
from tensorflow.keras import regularizers
from tensorflow.keras.constraints import Constraint
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras import backend as K
from tensorflow.keras.layers import concatenate
def load_dataset(path):
def _parse_function(example_proto):
keys_to_features = {
'X': tf.io.FixedLenFeature([42], tf.float32),
'y': tf.io.FixedLenFeature([22], tf.float32),
'vad': tf.io.FixedLenFeature([1], tf.float32)
}
features = tf.io.parse_single_example(example_proto, keys_to_features)
return (features['X'], (features['y'], features['vad']))
dataset = tf.data.TFRecordDataset(path).map(_parse_function)
return dataset
def my_crossentropy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.binary_crossentropy(y_pred, y_true), axis=-1)
def mymask(y_true):
return K.minimum(y_true + 1., 1.)
def msse(y_true, y_pred):
return K.mean(mymask(y_true) * K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)
def mycost(y_true, y_pred):
return K.mean(mymask(y_true) * (10 * K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true))) + K.square(
K.sqrt(y_pred) - K.sqrt(y_true)) + 0.01 * K.binary_crossentropy(y_pred, y_true)), axis=-1)
def my_accuracy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.equal(y_true, K.round(y_pred)), axis=-1)
class WeightClip(Constraint):
'''Clips the weights incident to each hidden unit to be inside a range
'''
def __init__(self, c=2.0):
self.c = c
def __call__(self, p):
return K.clip(p, -self.c, self.c)
def get_config(self):
return {'name': self.__class__.__name__,
'c': self.c}
def build_model():
reg = 0.000001
constraint = WeightClip(0.499)
main_input = Input(shape=(None, 42), name='main_input')
tmp = Dense(24, activation='tanh', name='input_dense', kernel_constraint=constraint, bias_constraint=constraint)(
main_input)
vad_gru = GRU(24, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='vad_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(tmp)
vad_output = Dense(1, activation='sigmoid', name='vad_output', kernel_constraint=constraint,
bias_constraint=constraint)(vad_gru)
noise_input = concatenate([tmp, vad_gru, main_input])
noise_gru = GRU(48, activation='relu', recurrent_activation='sigmoid', return_sequences=True, name='noise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(noise_input)
denoise_input = concatenate([vad_gru, noise_gru, main_input])
denoise_gru = GRU(96, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='denoise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(
denoise_input)
denoise_output = Dense(22, activation='sigmoid', name='denoise_output', kernel_constraint=constraint,
bias_constraint=constraint)(denoise_gru)
model = Model(inputs=main_input, outputs=[denoise_output, vad_output])
model.compile(loss=[mycost, my_crossentropy],
metrics=[msse],
optimizer='adam', loss_weights=[10, 0.5])
return model
model = build_model()
dataset = load_dataset('example.tfrecord')
My dataset has now the following shape:
<MapDataset shapes: ((42,), ((22,), (1,))), types: (tf.float32, (tf.float32, tf.float32))>
which I thought is what Model API expects (spoiler: it doesn't).
model.fit(dataset.batch(10))
gives following error:
ValueError: Error when checking input: expected main_input to have 3 dimensions, but got array with shape (None, 42)
Makes sense, I don't have the window here. At the same time it seems like it's not getting correct shape expected by Model(inputs=main_input, outputs=[denoise_output, vad_output]).
How to modify load_dataset so that it matches what's expected by the Model API for the tf.data?
Given that your model has 1 input and 2 outputs, your tf.data.Dataset should have two entries:
1) Input array of shape (window, 42)
2) Tuple of two arrays each of shape (window, 22) and (window, 1)
EDIT: Updated answer - you already return two element tuple
I just noticed that your dataset has these two entries (similar to those described above) and the only thing that differs is the shape.
The only operations you need to perfom is to batch your data twice:
First - to restore the window parameter.
Second - to pass a batch to a model.
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
model.fit(dataset.batch(window_size).batch(batch_size)
And that should work.
Below is an old answer, where I wrongfully assumed your dataset shape:
Old Answer, where I assumed you are returning three element tuple:
Assuming that you are starting from three element tuple of shapes (42,), (22,) and (1,), this can be achieved in the same batching operations, enriched with a custom_reshape function to return two-element tuple:
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
dataset = dataset.batch(window_size).batch(batch_size)
# Change output format
def custom_reshape(x, y, vad):
return x, (y, vad)
dataset = dataset.map(custom_reshape)
In short, given this dataset shape, you could just call:
model.fit(dataset.batch(window_size).batch(10).map(custom_reshape)
and it should work too.
Best of luck. And sorry again for the fuss.
Related
I'm trying to follow the tutorial given here.
This tutorial trains a Keras model using a genetic algorithm, with the PyGAD package. I'm interested in the binary classification case. My input matrix is of dimension 10000x20. Hence, I've created the following model using Keras:
input_layer = tensorflow.keras.layers.Input(20)
dense_layer1 = tensorflow.keras.layers.Dense(500, activation="relu")(input_layer)
dense_layer2 = tensorflow.keras.layers.Dense(500, activation="relu")(dense_layer1)
output_layer = tensorflow.keras.layers.Dense(1, activation="softmax")(dense_layer2)
model = tensorflow.keras.Model(inputs=input_layer, outputs=output_layer)
keras_ga = pygad.kerasga.KerasGA(model=model,
num_solutions=10)
However, when I go to run the algorithm, using ga_instance.run(), I get the error:
ValueError: Shapes (10000,) and (10000, 1) are incompatible
I can't figure out why I'm getting this error? I want my Keras model to have 2 hidden layers, each with 500 hidden nodes and 1 output node.
I think the problem is related to how each output is represented in the array. if you have a single output for 10000 instances, then this is an example of preparing the data that works with PyGAD. Its shape is (1000, 1).
numpy.random.uniform(0, 1, (1000, 1))
Here is a code that works but for a simple network architecture because, based on the fitness function you used, the fitness sometimes is NaN.
As I do not have the same data you used, I generated the input/output data randomly.
import tensorflow.keras
import pygad.kerasga
import numpy
import pygad
def fitness_func(solution, sol_idx):
global data_inputs, data_outputs, keras_ga, model
model_weights_matrix = pygad.kerasga.model_weights_as_matrix(model=model,
weights_vector=solution)
model.set_weights(weights=model_weights_matrix)
predictions = model.predict(data_inputs)
cce = tensorflow.keras.losses.CategoricalCrossentropy()
solution_fitness = 1.0 / (cce(data_outputs, predictions).numpy() + 0.00000001)
# print("solution_fitness", cce(data_outputs, predictions).numpy(), solution_fitness)
return solution_fitness
def callback_generation(ga_instance):
print("Generation = {generation}".format(generation=ga_instance.generations_completed))
print("Fitness = {fitness}".format(fitness=ga_instance.best_solution(ga_instance.last_generation_fitness)[1]))
data_inputs = numpy.random.uniform(0, 1, (1000, 20))
data_outputs = numpy.random.uniform(0, 1, (1000, 1))
# create model
from tensorflow.keras.layers import Dense, Dropout
l1_rate=1e-6
l2_rate = 1e-6
input_layer = tensorflow.keras.layers.InputLayer(20)
dense_layer1 = tensorflow.keras.layers.Dense(10, activation="relu",kernel_regularizer=tensorflow.keras.regularizers.l1_l2(l1=l1_rate, l2=l2_rate))
output_layer = tensorflow.keras.layers.Dense(1, activation="sigmoid")
model = tensorflow.keras.Sequential()
model.add(input_layer)
model.add(dense_layer1)
model.add(Dropout(0.2))
model.add(output_layer)
keras_ga = pygad.kerasga.KerasGA(model=model,
num_solutions=10)
# Run pygad
num_generations = 30
num_parents_mating = 5
initial_population = keras_ga.population_weights
ga_instance = pygad.GA(num_generations=num_generations,
num_parents_mating=num_parents_mating,
initial_population=initial_population,
fitness_func=fitness_func,
on_generation=callback_generation)
ga_instance.run()
Thanks for using PyGAD!
I know there are several questions about this here, but I haven't found one which fits exactly my problem.
I'm trying to fit an LSTM with data from Pandas DataFrames but getting confused about the format I have to provide them.
I created a small code snipped which shall show you what I try to do:
import pandas as pd, tensorflow as tf, random
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
targets = pd.DataFrame(index=pd.date_range(start='2019-01-01', periods=300, freq='D'))
targets['A'] = [random.random() for _ in range(len(targets))]
targets['B'] = [random.random() for _ in range(len(targets))]
features = pd.DataFrame(index=targets.index)
for i in range(len(features)) :
features[str(i)] = [random.random() for _ in range(len(features))]
model = Sequential()
model.add(LSTM(units=targets.shape[1], input_shape=features.shape))
model.compile(optimizer='adam', loss='mae')
model.fit(features, targets, batch_size=10, epochs=10)
this results to:
ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [10, 300]
which I expect relates to the dimensions of the features DataFrame provided. I guess that once fixed this the next error would mention the targets DataFrame.
As far as I understand, 'units' parameter of my first layer defines the output dimensionality of this model. The inputs have to have a 3D shape, but I don't know how to create them out of the 2D world of the Data Frames.
I hope you can help me understanding the reshape mechanism in Python and how to use them in combination with Pandas DataFrames. (I'm quite new to Python and came from R)
Thankls in advance
Lets looks at the few popular ways in LSTMs are used.
Many to Many
Example: You have a sentence (composed of words in sequence). Give these sequence of words you would like to predict the Parts of speech (POS) of each word.
So you have n words and you feed each word per timestep to the LSTM. Each LSTM timestep (also called LSTM unwrapping) will produce and output. The word is represented by a a set of features normally word embeddings. So the input to LSTM is of size bath_size X time_steps X features
Keras code:
inputs = keras.Input(shape=(10,3))
lstm = keras.layers.LSTM(8, input_shape = (10, 3), return_sequences = True)(inputs)
outputs = keras.layers.TimeDistributed(keras.layers.Dense(5, activation='softmax'))(lstm)
model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(loss='categorical_crossentropy', optimizer='adam')
X = np.random.randn(4,10,3)
y = np.random.randint(0,2, size=(4,10,5))
model.fit(X, y, epochs=2)
print (model.predict(X).shape)
Many to One
Example: Again you have a sentence (composed of words in sequence). Give these sequence of words you would like to predict sentiment of the sentence if it is positive or negative.
Keras code
inputs = keras.Input(shape=(10,3))
lstm = keras.layers.LSTM(8, input_shape = (10, 3), return_sequences = False)(inputs)
outputs =keras.layers.Dense(5, activation='softmax')(lstm)
model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(loss='categorical_crossentropy', optimizer='adam')
X = np.random.randn(4,10,3)
y = np.random.randint(0,2, size=(4,5))
model.fit(X, y, epochs=2)
print (model.predict(X).shape)
Many to multi-headed
Example: You have a sentence (composed of words in sequence). Give these sequence of words you would like to predict sentiment of the sentence as well the author of the sentence.
This is multi-headed model where one head will predict the sentiment and another head will predict the author. Both the heads share the same LSTM backbone.
Keras code
inputs = keras.Input(shape=(10,3))
lstm = keras.layers.LSTM(8, input_shape = (10, 3), return_sequences = False)(inputs)
output_A = keras.layers.Dense(5, activation='softmax')(lstm)
output_B = keras.layers.Dense(5, activation='softmax')(lstm)
model = keras.Model(inputs=inputs, outputs=[output_A, output_B])
model.compile(loss='categorical_crossentropy', optimizer='adam')
X = np.random.randn(4,10,3)
y_A = np.random.randint(0,2, size=(4,5))
y_B = np.random.randint(0,2, size=(4,5))
model.fit(X, [y_A, y_B], epochs=2)
y_hat_A, y_hat_B = model.predict(X)
print (y_hat_A.shape, y_hat_B.shape)
What you are looking for is Many to Multi head model where your predictions for A will be made by one head and another head will make predictions for B
The input data for the LSTM has to be 3D.
If you print the shapes of your DataFrames you get:
targets : (300, 2)
features : (300, 300)
The input data has to be reshaped into (samples, time steps, features). This means that targets and features must have the same shape.
You need to set a number of time steps for your problem, in other words, how many samples will be used to make a prediction.
For example, if you have 300 days and 2 features the time step can be 3. So that three days will be used to make one prediction (you can choose this arbitrarily). Here is the code for reshaping your data (with a few more changes):
import pandas as pd
import numpy as np
import tensorflow as tf
import random
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense
data = pd.DataFrame(index=pd.date_range(start='2019-01-01', periods=300, freq='D'))
data['A'] = [random.random() for _ in range(len(data))]
data['B'] = [random.random() for _ in range(len(data))]
# Choose the time_step size.
time_steps = 3
# Use numpy for the 3D array as it is easier to handle.
data = np.array(data)
def make_x_y(ts, data):
"""
Parameters
ts : int
data : numpy array
This function creates two arrays, x and y.
x is the input data and y is the target data.
"""
x, y = [], []
offset = 0
for i in data:
if offset < len(data)-ts:
x.append(data[offset:ts+offset])
y.append(data[ts+offset])
offset += 1
return np.array(x), np.array(y)
x, y = make_x_y(time_steps, data)
print(x.shape, y.shape)
nodes = 100 # This is the width of the network.
out_size = 2 # Number of outputs produced by the network. Same size as features.
model = Sequential()
model.add(LSTM(units=nodes, input_shape=(x.shape[1], x.shape[2])))
model.add(Dense(out_size)) # For the output a Dense (fully connected) layer is used.
model.compile(optimizer='adam', loss='mae')
model.fit(x, y, batch_size=10, epochs=10)
Well, just to finalize this issue I would like to provide one solution I have meanwhile worked on. The class TimeseriesGenerator in tf.keras.... enabled me quite easy to provide the data in the right shape to an LSTM model
from keras.preprocessing.sequence import TimeseriesGenerator
import numpy as np
window_size = 7
batch_size = 8
sampling_rate = 1
train_gen = TimeseriesGenerator(X_train.values, y_train.values,
length=window_size, sampling_rate=sampling_rate,
batch_size=batch_size)
valid_gen = TimeseriesGenerator(X_valid.values, y_valid.values,
length=window_size, sampling_rate=sampling_rate,
batch_size=batch_size)
test_gen = TimeseriesGenerator(X_test.values, y_test.values,
length=window_size, sampling_rate=sampling_rate,
batch_size=batch_size)
There are many other ways on implementing generators e.g. using the more_itertools which provides the function windowed, or making use of tensorflow.Dataset and its function window.
For me the TimeseriesGenerator was sufficient to feed the tests I did.
In case you would like to see an example modeling the DAX based on some stocks I'm sharing a notebook on Github.
I've just started exploring TensorFlow and I'm facing an issue regarding performance. As a starting example, I tried implementing a model to simulate a logic gate. Let's say there are two inputs A and B and one output Y. Suppose Y depended only on B and not on A. That means that the following are valid examples:
[0, 0] -> 0
[1, 0] -> 0
[0, 1] -> 1
[1, 1] -> 1
I created training sets for this data and created a model that uses a DenseFeatures layer using two features A and B. This layer feeds into a Dense(128, 'relu') layer, which feeds into a Dense(16, 'relu') layer, which finally feeds into a Dense(1, 'sigmoid') layer.
Training this NN works fine and the predictions are perfect. However, I noticed that on my MacBook, each prediction takes about 250ms. This is too much, since my final goal is to use such a NN to test hundreds of predictions each second.
So I stripped the network down to DenseFeatures([A, B]) -> Dense(8, 'relu') -> Dense(1, 'sigmoid'), however predictions for this NN still takes the same about of time. I was expecting that the execution speed depends on the complexity of the model. I can see that this is not the case here? What am I doing wrong?
Also, I had read that TensorFlow uses floating point math for accuracy but this has a penalty hit in terms of performance and if we convert our data to use integer math, it would speed things up. However, I have no idea of how to achieve that.
I would really appreciate if someone can help me understand why predictions for such a simple logic gate and such a simple NN is taking this long. And how can I speed it up.
For reference, here is my code in python:
import random
from typing import Any, List
import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split
from tensorflow import feature_column
from tensorflow.keras import layers
class Input:
def __init__(self, data: List[int]):
self.data = data
class Output:
def __init__(self, value: float):
self.value = value
class Item:
def __init__(self, input: Input, output: Output):
self.input = input
self.output = output
DATA: List[Item] = []
for i in range(10000):
x = Input([random.randint(0, 1), random.randint(0, 1)])
y = Output(x.data[1])
DATA.append(Item(x, y))
BATCH_SIZE = 5
DATA_TRAIN, DATA_TEST = train_test_split(DATA, shuffle=True, test_size=0.2)
DATA_TRAIN, DATA_VAL = train_test_split(DATA_TRAIN, shuffle=True, test_size=0.2/0.8)
def toDataSet(data: List[Item], shuffle: bool, batch_size: int):
a = {
'a': [x.input.data[0] for x in data],
'b': [x.input.data[1] for x in data],
}
b = [x.output.value for x in data]
return tf.data.Dataset.from_tensor_slices((a, b)).shuffle(buffer_size=len(data)).batch(BATCH_SIZE)
DS_TRAIN = toDataSet(DATA_TRAIN, True, 5)
DS_VAL = toDataSet(DATA_VAL, True, 5)
DS_TEST = toDataSet(DATA_TEST, True, 5)
FEATURES = []
FEATURES.append(a)
FEATURES.append(b)
feature_layer = tf.keras.layers.DenseFeatures(FEATURES)
model = tf.keras.models.load_model('MODEL.H5')
model = tf.keras.Sequential([
feature_layer,
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(DS_TRAIN, validation_data=DS_VAL, epochs=10)
loss, accuracy = model.evaluate(DS_TEST)
for i in range(1000):
val = model.predict([np.array([random.randint(0, 1)]), np.array([random.randint(0, 1)])])
Since you are only using integers, change the input of the model to use 8-bit signed integers. You can do this by changing the datatype in your input layer by adding the dtype parameter. This will vastly improve processing speed since you won't be wasting calculations.
I encountered a strange error when using tf.keras in python with a dataset that consists out of tensors.
What I do is create a dataset from a csv file and modify it as it contains a time series. In the end, I end up with a dataset that has tuples of input and output tensor.
These are fed into a tf.keras model and here is where it becomes interesting. Fit calls a function in training.py of the keras enginge which is called _standardize_user_data. This should return the input, targets, and so on for fit_loop, which in turn checks something on it and crashes with
if issparse is not None and issparse(ins[i]) and not
K.is_sparse(feed[i]):
IndexError: list index out of range
in line 187 of training_arrays.py. This is due to the fact _standardize_user_data returns empty lists, with the reasoning that if tensors are the input, then it everything should be set up already, which it apparently isn't. that Here is some small MWE.
#!/usr/bin/env python3
import argparse
import glob
import logging
import numpy as np
import tensorflow as tf
from tensorflow import keras
def read_dataset(filename, columns, field_defaults, input_size, output_size, stride, input_features):
def decode_csv(row):
fields = tf.decode_csv(row, record_defaults=field_defaults, field_delim=',')
all_columns = dict(zip(columns, fields))
return all_columns
def split_window(window):
inputs = tf.reshape(tf.concat(window['value'][0:input_size], axis=1), [input_size, input_features])
outputs = tf.reshape(tf.concat(window['value'][input_size:input_size + output_size], axis=1),
[output_size, input_features])
return inputs, outputs
dataset = tf.data.TextLineDataset(filenames=filename)
dataset = dataset.map(decode_csv)
dataset = dataset.apply(tf.contrib.data.sliding_window_batch(window_size=input_size + output_size, stride=stride))
dataset = dataset.map(split_window)
dataset = dataset.repeat()
return dataset
if __name__ == "__main__":
COLUMNS = ['value']
FIELD_DEFAULTS = [[0.0]]
INPUT_FEATURES = 1
epochs = 1
steps = 1
INPUT_SIZE = 4
OUTPUT_SIZE = 2
STRIDE = 1
input_train = "./data/"
input_train_list = glob.glob(input_train + "*")
model = keras.Sequential()
model.add(tf.keras.layers.Dense(OUTPUT_SIZE, activation=None))
set = read_dataset(input_train_list, COLUMNS, FIELD_DEFAULTS, INPUT_SIZE, OUTPUT_SIZE, STRIDE, INPUT_FEATURES)
model.compile(optimizer=tf.train.AdamOptimizer(0.01), loss='mse', metrics=['mse'])
model.fit(set, epochs=epochs, steps_per_epoch=steps)
The data is in the form of:
0.047910000000000785
3.0999999999892225e-05
0.0160979999999995
2.9000000000500847e-05
0.01716599999999957
2.800000000036107e-05
2.9999999999752447e-05
0.019235000000000113
Does anybody know if this is intended or if something goes wrong? Does my data generation give the wrong output? Or could it be that there is a bug?
I'm thankful for any help!
I'm trying to create a simple weighted loss function.
Say, I have input dimensions 100 * 5, and output dimensions also 100 * 5. I also have a weight matrix of the same dimension.
Something like the following:
import numpy as np
train_X = np.random.randn(100, 5)
train_Y = np.random.randn(100, 5)*0.01 + train_X
weights = np.random.randn(*train_X.shape)
Defining the custom loss function
def custom_loss_1(y_true, y_pred):
return K.mean(K.abs(y_true-y_pred)*weights)
Defining the model
from keras.layers import Dense, Input
from keras import Model
import keras.backend as K
input_layer = Input(shape=(5,))
out = Dense(5)(input_layer)
model = Model(input_layer, out)
Testing with existing metrics works fine
model.compile('adam','mean_absolute_error')
model.fit(train_X, train_Y, epochs=1)
Testing with our custom loss function doesn't work
model.compile('adam',custom_loss_1)
model.fit(train_X, train_Y, epochs=10)
It gives the following stack trace:
InvalidArgumentError (see above for traceback): Incompatible shapes: [32,5] vs. [100,5]
[[Node: loss_9/dense_8_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_9/dense_8_loss/Abs, loss_9/dense_8_loss/mul/y)]]
Where is the number 32 coming from?
Testing a loss function with weights as Keras tensors
def custom_loss_2(y_true, y_pred):
return K.mean(K.abs(y_true-y_pred)*K.ones_like(y_true))
This function seems to do the work. So, probably suggests that a Keras tensor as a weight matrix would work. So, I created another version of the loss function.
Loss function try 3
from functools import partial
def custom_loss_3(y_true, y_pred, weights):
return K.mean(K.abs(y_true-y_pred)*K.variable(weights, dtype=y_true.dtype))
cl3 = partial(custom_loss_3, weights=weights)
Fitting data using cl3 gives the same error as above.
InvalidArgumentError (see above for traceback): Incompatible shapes: [32,5] vs. [100,5]
[[Node: loss_11/dense_8_loss/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_11/dense_8_loss/Abs, loss_11/dense_8_loss/Variable/read)]]
I wonder what I'm missing! I could have used the notion of sample_weight in Keras; but then I'd have to reshape my inputs to a 3d vector.
I thought that this custom loss function should really have been trivial.
In model.fit the batch size is 32 by default, that's where this number is coming from. Here's what's happening:
In custom_loss_1 the tensor K.abs(y_true-y_pred) has shape (batch_size=32, 5), while the numpy array weights has shape (100, 5). This is an invalid multiplication, since the dimensions don't agree and broadcasting can't be applied.
In custom_loss_2 this problem doesn't exist because you're multiplying 2 tensors with the same shape (batch_size=32, 5).
In custom_loss_3 the problem is the same as in custom_loss_1, because converting weights into a Keras variable doesn't change their shape.
UPDATE: It seems you want to give a different weight to each element in each training sample, so the weights array should have shape (100, 5) indeed.
In this case, I would input your weights' array into your model and then use this tensor within the loss function:
import numpy as np
from keras.layers import Dense, Input
from keras import Model
import keras.backend as K
from functools import partial
def custom_loss_4(y_true, y_pred, weights):
return K.mean(K.abs(y_true - y_pred) * weights)
train_X = np.random.randn(100, 5)
train_Y = np.random.randn(100, 5) * 0.01 + train_X
weights = np.random.randn(*train_X.shape)
input_layer = Input(shape=(5,))
weights_tensor = Input(shape=(5,))
out = Dense(5)(input_layer)
cl4 = partial(custom_loss_4, weights=weights_tensor)
model = Model([input_layer, weights_tensor], out)
model.compile('adam', cl4)
model.fit(x=[train_X, weights], y=train_Y, epochs=10)