My goal is to create a custom loss function that calculates the loss based on y_true, y_pred and the tensor of the models input layer:
import numpy as np
from tensorflow import keras as K
input_shape = (16, 16, 1)
input = K.layers.Input(input_shape)
dense = K.layers.Dense(16)(input)
output = K.layers.Dense(1)(dense)
model = K.Model(inputs=input, outputs=output)
def CustomLoss(y_true, y_pred):
return K.backend.sum(K.backend.abs(y_true - model.input * y_pred))
model.compile(loss=CustomLoss)
model.fit(np.ones(input_shape), np.zeros(input_shape))
However, this code fails with the following error message:
TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.
How can I pass the input tensor of my model to the loss function?
Tensorflow Version: 2.4.1
Python Version: 3.8.8
You can use add_loss to pass external layers to your loss. Here an example:
import numpy as np
from tensorflow import keras as K
def CustomLoss(y_true, y_pred, input_l):
return K.backend.sum(K.backend.abs(y_true - input_l * y_pred))
input_shape = (16, 16, 1)
n_sample = 10
X = np.random.uniform(0,1, (n_sample,) + input_shape)
y = np.random.uniform(0,1, (n_sample,) + input_shape)
inp = K.layers.Input(input_shape)
dense = K.layers.Dense(16)(inp)
out = K.layers.Dense(1)(dense)
target = K.layers.Input(input_shape)
model = K.Model(inputs=[inp,target], outputs=out)
model.add_loss( CustomLoss( target, out, inp ) )
model.compile(loss=None, optimizer='adam')
model.fit(x=[X,y], y=None, epochs=3)
To use the model in inference mode (removing the target from inputs)
final_model = K.Model(model.input[0], model.output)
final_model.predict(X)
I am building a model based on this code for noise suppression. My problem with the vanilla implementation is that it loads all data at once, which is not the best idea when the training data gets really large; my input file, denoted in the linked code as training.h5, is over 30 GB.
I decided to instead go with tf.data interface that should allow me to work with large data sets; my problem here is that I don't know how to properly shape TFRecordDataset so that it meets what's required by the Model API.
If you check model.fit(x_train, [y_train, vad_train], it essentially requires the following:
x_train, shape [nb_sequences, window, 42]
y_train, shape [nb_sequences, window, 22]
vad_train, shape [nb_sequences, window, 1]
window one typically fixes (in the code: 2000), so the only variable nb_sequences that stems from how large is your data set. However, with tf.data, we don't supply x and y, but only x (see Model API docs).
Saving tfrecord to file
In an effort to make the code reproducible, I created the input file with the following code:
writer = tf.io.TFRecordWriter(path='example.tfrecord')
for record in data:
feature = {}
feature['X'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[:42]))
feature['y'] = tf.train.Feature(float_list=tf.train.FloatList(value=record[42:64]))
feature['vad'] = tf.train.Feature(float_list=tf.train.FloatList(value=[record[64]]))
example = tf.train.Example(features=tf.train.Features(feature=feature))
serialized = example.SerializeToString()
writer.write(serialized)
writer.close()
data is our training data with shape [10000, 65]. My example.tfrecord is available here. It's 3 MB, in reality it would be 30 GB+.
You might notice that in the linked code, numpy array has shape [x, 87], while mine is [x, 65]. That's OK - the remainder is not used anywhere.
Loading the dataset with tf.data.TFRecordDataset
I would like to use tf.data to load "on demand" the data with some prefetching, there's no need to keep it all in memory. My attempt:
import datetime
import numpy as np
import h5py
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import GRU
from tensorflow.keras import regularizers
from tensorflow.keras.constraints import Constraint
from tensorflow.keras.callbacks import ModelCheckpoint
from tensorflow.keras import backend as K
from tensorflow.keras.layers import concatenate
def load_dataset(path):
def _parse_function(example_proto):
keys_to_features = {
'X': tf.io.FixedLenFeature([42], tf.float32),
'y': tf.io.FixedLenFeature([22], tf.float32),
'vad': tf.io.FixedLenFeature([1], tf.float32)
}
features = tf.io.parse_single_example(example_proto, keys_to_features)
return (features['X'], (features['y'], features['vad']))
dataset = tf.data.TFRecordDataset(path).map(_parse_function)
return dataset
def my_crossentropy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.binary_crossentropy(y_pred, y_true), axis=-1)
def mymask(y_true):
return K.minimum(y_true + 1., 1.)
def msse(y_true, y_pred):
return K.mean(mymask(y_true) * K.square(K.sqrt(y_pred) - K.sqrt(y_true)), axis=-1)
def mycost(y_true, y_pred):
return K.mean(mymask(y_true) * (10 * K.square(K.square(K.sqrt(y_pred) - K.sqrt(y_true))) + K.square(
K.sqrt(y_pred) - K.sqrt(y_true)) + 0.01 * K.binary_crossentropy(y_pred, y_true)), axis=-1)
def my_accuracy(y_true, y_pred):
return K.mean(2 * K.abs(y_true - 0.5) * K.equal(y_true, K.round(y_pred)), axis=-1)
class WeightClip(Constraint):
'''Clips the weights incident to each hidden unit to be inside a range
'''
def __init__(self, c=2.0):
self.c = c
def __call__(self, p):
return K.clip(p, -self.c, self.c)
def get_config(self):
return {'name': self.__class__.__name__,
'c': self.c}
def build_model():
reg = 0.000001
constraint = WeightClip(0.499)
main_input = Input(shape=(None, 42), name='main_input')
tmp = Dense(24, activation='tanh', name='input_dense', kernel_constraint=constraint, bias_constraint=constraint)(
main_input)
vad_gru = GRU(24, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='vad_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(tmp)
vad_output = Dense(1, activation='sigmoid', name='vad_output', kernel_constraint=constraint,
bias_constraint=constraint)(vad_gru)
noise_input = concatenate([tmp, vad_gru, main_input])
noise_gru = GRU(48, activation='relu', recurrent_activation='sigmoid', return_sequences=True, name='noise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(noise_input)
denoise_input = concatenate([vad_gru, noise_gru, main_input])
denoise_gru = GRU(96, activation='tanh', recurrent_activation='sigmoid', return_sequences=True, name='denoise_gru',
kernel_regularizer=regularizers.l2(reg), recurrent_regularizer=regularizers.l2(reg),
kernel_constraint=constraint, recurrent_constraint=constraint, bias_constraint=constraint)(
denoise_input)
denoise_output = Dense(22, activation='sigmoid', name='denoise_output', kernel_constraint=constraint,
bias_constraint=constraint)(denoise_gru)
model = Model(inputs=main_input, outputs=[denoise_output, vad_output])
model.compile(loss=[mycost, my_crossentropy],
metrics=[msse],
optimizer='adam', loss_weights=[10, 0.5])
return model
model = build_model()
dataset = load_dataset('example.tfrecord')
My dataset has now the following shape:
<MapDataset shapes: ((42,), ((22,), (1,))), types: (tf.float32, (tf.float32, tf.float32))>
which I thought is what Model API expects (spoiler: it doesn't).
model.fit(dataset.batch(10))
gives following error:
ValueError: Error when checking input: expected main_input to have 3 dimensions, but got array with shape (None, 42)
Makes sense, I don't have the window here. At the same time it seems like it's not getting correct shape expected by Model(inputs=main_input, outputs=[denoise_output, vad_output]).
How to modify load_dataset so that it matches what's expected by the Model API for the tf.data?
Given that your model has 1 input and 2 outputs, your tf.data.Dataset should have two entries:
1) Input array of shape (window, 42)
2) Tuple of two arrays each of shape (window, 22) and (window, 1)
EDIT: Updated answer - you already return two element tuple
I just noticed that your dataset has these two entries (similar to those described above) and the only thing that differs is the shape.
The only operations you need to perfom is to batch your data twice:
First - to restore the window parameter.
Second - to pass a batch to a model.
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
model.fit(dataset.batch(window_size).batch(batch_size)
And that should work.
Below is an old answer, where I wrongfully assumed your dataset shape:
Old Answer, where I assumed you are returning three element tuple:
Assuming that you are starting from three element tuple of shapes (42,), (22,) and (1,), this can be achieved in the same batching operations, enriched with a custom_reshape function to return two-element tuple:
window_size = 1
batch_size = 10
dataset = load_dataset('example.tfrecord')
dataset = dataset.batch(window_size).batch(batch_size)
# Change output format
def custom_reshape(x, y, vad):
return x, (y, vad)
dataset = dataset.map(custom_reshape)
In short, given this dataset shape, you could just call:
model.fit(dataset.batch(window_size).batch(10).map(custom_reshape)
and it should work too.
Best of luck. And sorry again for the fuss.
I have created an RNN with the Keras functional API in TensorFlow 2.0 where the following piece of code workes
sum_input = keras.Input(shape=(UNIT_SIZE, 256,), name='sum')
x = tf.unstack(sum_input,axis=2, num=256)
t_sum = x[0]
for i in range(len(x) - 1):
t_sum = keras.layers.Add()([t_sum, x[i+1]])
sum_m = keras.Model(inputs=sum_input, outputs=t_sum, name='sum_model')
I then had to changed to Tensorflow 1.13 which gives me the following error
ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer` (thus holding past layer metadata). Found: Tensor("add_254/add:0", shape=(?, 40), dtype=float32)
I don't understand why the output tensor is not from a Tensorflow layer, since t_sum is the output from keras.layers.Add.
I have tried to wrap parts of the code into keras.layers.Lambda as suggested in ValueError: Output tensors to a Model must be the output of a TensorFlow Layer
, but it doesn't seem to work for me.
The problem is not with Add() layer but with tf.unstack() - it is not an instance of keras.layers.Layer(). You can just wrap it up as custom layer:
import tensorflow as tf
class Unstack(tf.keras.layers.Layer):
def __init__(self):
super(Unstack, self).__init__()
def call(self, inputs, num=256):
return tf.unstack(inputs, axis=2, num=num)
x = Unstack()(sum_input)
or, instead of subclassing, you can do it using Lambda layer:
x = tf.keras.layers.Lambda(lambda t: tf.unstack(t, axis=2, num=256))(sum_input)
I would like to apply dropout to the outputs from an RNN. For example, in Tensorflow 1.8.0, I could do this:
import tensorflow as tf
import tensorflow.contrib.eager as tfe
tfe.enable_eager_execution()
x = tf.random_uniform((10, 5, 3))
gru_cell1 = tf.contrib.rnn.GRUCell(2)
gru_cell1 = tf.contrib.rnn.DropoutWrapper(gru_cell1, output_keep_prob=0.5)
cell = tf.contrib.rnn.MultiRNNCell([gru_cell1])
init_state = cell.zero_state(10, tf.float32)
cell_output, _ = tf.nn.dynamic_rnn(cell, x,
initial_state=init_state, time_major=False)
cell_output
How can I achieve the same thing using the Keras API?
I have thought of the following two ways but they were unsuccessful:
import tensorflow as tf
import tensorflow.contrib.eager as tfe
tfe.enable_eager_execution()
# Attempt 1
x = tf.random_uniform((10, 5, 3))
gru_layer = tf.keras.layers.GRU(2, return_sequences=True, input_shape=(10, 5, 3))
gru_layer = tf.keras.layers.Dropout(0.5)(gru_layer)
# Gives the following error:
# ValueError: Attempt to convert a value (<tensorflow.python.keras._impl.keras.layers.recurrent.GRU object
# at 0x000001C520681F60>) with an unsupported type (<class 'tensorflow.python.keras._impl.keras.layers.recurrent.GRU'>)
# to a Tensor.
# Attempt 2
x = tf.random_uniform((10, 5, 3))
gru_layer = tf.keras.layers.GRU(2, return_sequences=True, input_shape=(10, 5, 3))
gru_layer = tf.keras.layers.TimeDistributed(tf.keras.layers.Dropout(0.4))(gru_layer)
# Gives the following error:
# ValueError: as_list() is not defined on an unknown TensorShape.
To get the model output, without training, like you're doing in the TF code, the following code should work. Indeed, you need an Input layer, and to hook each layer to the previous one, and a Model as well:
import numpy as np
from keras.models import Model
from keras.layers import Dropout, GRU, Input
x = np.random.randn(10, 5, 3)
inputs = Input(shape=(5, 3))
gru_layer = GRU(2, return_sequences=True)(inputs)
gru_layer = Dropout(0.5)(gru_layer)
model = Model(inputs=inputs, outputs=gru_layer)
output = model.predict(x)
i'm trying to use the tensorflow unique function (https://www.tensorflow.org/api_docs/python/tf/unique) in a keras lambda layer.
Code below:
def unique_idx(x):
output = tf.unique(x)
return output[1]
then
inp1 = Input(batch_shape(None, 1))
idx = Lambda(unique_idx)(inp1)
model = Model(inputs=inp1, outputs=idx)
when I now use **model.compile(optimizer='Adam', loss='mean_squared_error')**
I get the error:
ValueError: Tensor conversion requested dtype int32 for Tensor with
dtype float32: 'Tensor("lambda_9_sample_weights_1:0", shape=(?,),
dtype=float32)'
Does anybody know whats the error here or a different way of using the tensorflow function?
A keras model expects a float32 as output, but the indices returned from tf.unique is a int32. A casting fixes your problem.
Another issue is that unique expects a flatten array. reshape fixes this one.
import tensorflow as tf
from keras import Input
from keras.layers import Lambda
from keras.engine import Model
def unique_idx(x):
x = tf.reshape(x, [-1])
u, indices = tf.unique(x)
return tf.cast(indices, tf.float32)
x = Input(shape=(1,))
y = Lambda(unique_idx)(x)
model = Model(inputs=x, outputs=y)
model.compile(optimizer='adam', loss='mse')