I am trying to implement a custom loss function in Keras with TF backend based on the Laplacian of two images.
def blur_loss(y_true, y_pred):
#weighting of blur loss
alpha = 1
mae = losses.mean_absolute_error(y_true, y_pred)
lapKernel = K.constant([0, 1, 0, 1, -4, 1, 0, 1, 0],shape = [3, 3])
trueLap = K.conv2d(y_true, lapKernel)
predLap = K.conv2d(y_pred, lapKernel)
trueBlur = K.var(trueLap)
predBlur = K.var(predLap)
blurLoss = alpha * K.abs(trueBlur - predBlur)
loss = (1-alpha) * mae + alpha * blurLoss
return loss
When I try to compile the model I get this error
Traceback (most recent call last):
File "kitti_train.py", line 65, in <module>
model.compile(loss='mean_absolute_error', optimizer='adam', metrics=[blur_loss])
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 924, in compile
handle_metrics(output_metrics)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 921, in handle_metrics
mask=masks[i])
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/engine/training.py", line 450, in weighted
score_array = fn(y_true, y_pred)
File "/home/ubuntu/prednet/blur_loss.py", line 14, in blur_loss
trueLap = K.conv2d(y_true, lapKernel)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 3164, in conv2d
data_format='NHWC')
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 655, in convolution
num_spatial_dims, strides, dilation_rate)
File "/home/ubuntu/.virtualenvs/dl4cv/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 483, in _get_strides_and_dilation_rate
(len(dilation_rate), num_spatial_dims))
ValueError: len(dilation_rate)=2 but should be 0
After reading other questions, my understanding is that this problem stems from the compilation using placeholder tensors for y_true and y_pred. I've tried checking if the inputs are placeholders and replacing them with zero tensors, but this gives me other errors.
How do I use a convolution (the image processing function, not a layer) in my loss function without getting these errors?
The problem here was a misunderstanding of the conv2d function which is not simply a 2-dimensional convolution. It is a batched 2-d convolution of multiple channels. So while you might expect a *2d function to accept 2-dimensional tensors, the input should actually 4 dimensions (batch_size, height, width, channels) and the filter should also be 4 dimensions (filter_height, filter_width, input_channels, output_channels). Details can be found in the TF docs
Related
I understand that there are several options for kernel_constraint in Conv2D in Keras: max_norm, non_neg or unit_norm..
But what I needed is to set the anchor (center) position in the filter kernels to be zero.
For example, if we have a filter kernel with its size of (width, height) = (5, 5), and that we have 3 channels in the input. I need to constrain the anchor (center) point of this kernel for every channel to be 0, like w(2,2,:)=0, assuming we put the channel dimension the 3rd dimension. If there are multiple filters, the anchor position of each filter should be with zero. How could I implement this?
I assume a custom kernel constraint is needed. This link gives suggest how to create one class inheriting from Constraint: https://github.com/keras-team/keras/issues/8196. This shows how in-built constraints are implemented:
https://github.com/keras-team/keras/blob/master/keras/constraints.py
But still, I do not know how the dimensions of w are manipulated, and how to set the desired position to be zero. Any help is appreciated. Thanks.
Update:
Daniel Möller's answer was tried. The error message is as follows:
raise ValueError('An operation has None for gradient. '
ValueError: An operation has None for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.
Since Daniel can run that without problem on his side, to inspect what goes wrong in my program, I post my simplified code here. My data have 8 channels, but it shouldn't matter how many you have.
from keras.layers import Input, Conv2D
from keras.models import Model, optimizers
import numpy as np
import tensorflow as tf
from keras import backend as K
from keras.callbacks import ModelCheckpoint
class ZeroCenterConv2D(Conv2D):
def __init__(self, filters, kernel_size, **kwargs):
super(ZeroCenterConv2D, self).__init__(filters, kernel_size, **kwargs)
def call(self, inputs):
assert self.kernel_size[0] % 2 == 1, "Error: the kernel size is an even number"
assert self.kernel_size[1] % 2 == 1, "Error: the kernel size is an even number"
centerX = (self.kernel_size[0] - 1) // 2
centerY = (self.kernel_size[1] - 1) // 2
kernel_mask = np.ones(self.kernel_size + (1, 1))
kernel_mask[centerX, centerY] = 0
kernel_mask = K.constant(kernel_mask)
customKernel = self.kernel * kernel_mask
outputs = K.conv2d(
inputs,
customKernel,
strides=self.strides,
padding=self.padding,
data_format=self.data_format,
dilation_rate=self.dilation_rate)
if self.activation is not None:
return self.activation(outputs)
return outputs
size1 = 256
size2 = 256
input_img = Input(shape=(size1, size2, 8))
conv1 = ZeroCenterConv2D(8, (5, 5), padding='same', activation='relu')(input_img)
autoencoder = Model(input_img, conv1)
adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-8)
autoencoder.compile(optimizer=adam, loss='mean_squared_error')
import scipy.io
A = scipy.io.loadmat('data_train')
x_train = A['data']
x_train = np.reshape(x_train, (1, 256, 256, 8))
from keras.callbacks import TensorBoard
autoencoder.fit(x_train, x_train,
epochs=5,
batch_size=1,
shuffle=False,
validation_data=(x_train, x_train),
callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])
decoded_imgs = autoencoder.predict(x_train)
When conv1 = ZeroCenterConv2D... was replaced by the conventional conv1 = Conv2D..., everything works.
The full error message:
Connected to pydev debugger (build 181.4668.75)
/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
Traceback (most recent call last):
File "/snap/pycharm-community/60/helpers/pydev/pydevd.py", line 1664, in <module>
main()
File "/snap/pycharm-community/60/helpers/pydev/pydevd.py", line 1658, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/snap/pycharm-community/60/helpers/pydev/pydevd.py", line 1068, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/allen/autotion/temptest", line 62, in <module>
callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])
File "/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/keras/engine/training.py", line 1682, in fit
self._make_train_function()
File "/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/keras/engine/training.py", line 992, in _make_train_function
loss=self.total_loss)
File "/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/keras/optimizers.py", line 445, in get_updates
grads = self.get_gradients(loss, params)
File "/home/allen/kerasProject/keras/venv/py2.7/local/lib/python2.7/site-packages/keras/optimizers.py", line 80, in get_gradients
raise ValueError('An operation has `None` for gradient. '
ValueError: An operation has `None` for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.
Process finished with exit code 1
A further update: adding "bias" part in the code in Daniel's answer (already done), problems solved!
You need a custom Conv2D layer for that, where you change its call method to apply the zero at the center.
class ZeroCenterConv2D(Conv2D):
def __init__(self, filters, kernel_size, **kwargs):
super(ZeroCenterConv2D, self).__init__(filters, kernel_size, **kwargs)
def call(self, inputs):
assert self.kernel_size[0] % 2 == 1, "Error: the kernel size is an even number"
assert self.kernel_size[1] % 2 == 1, "Error: the kernel size is an even number"
centerX = (self.kernel_size[0] - 1) // 2
centerY = (self.kernel_size[1] - 1) // 2
kernel_mask = np.ones(self.kernel_size + (1, 1))
kernel_mask[centerX, centerY] = 0
kernel_mask = K.variable(kernel_mask)
customKernel = self.kernel * kernel_mask
outputs = K.conv2d(
inputs,
customKernel,
strides=self.strides,
padding=self.padding,
data_format=self.data_format,
dilation_rate=self.dilation_rate)
if self.use_bias:
outputs = K.bias_add(
outputs,
self.bias,
data_format=self.data_format)
if self.activation is not None:
return self.activation(outputs)
return outputs
This will not replace the actual weights, though, but the center ones will never be used.
When you use layer.get_weights() of model.get_weights(), you will see the center weights as they were initialized (not as zeros).
I am trying to see what my data looks like after going through a convolutional layer in my Keras model. I am using the Theano backend. The code I have has been cobbled together from the Keras Github:
def get_layer0_outputs(model, test_data):
output = model.layers[0].output
inputs = [K.learning_phase()] + model.inputs
func = K.function(inputs, [output])
return func([0] + [test_data])
What I'm trying to do here is compile a function for the first layer (a Conv2D layer) in my network. The test_data argument is an np.ndarray. My model is loaded correctly, and I have already trained it with decent accuracy.
However, when I call this function, I get a cryptic stacktrace:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/theano/compile/function_module.py", line 884, in __call__
self.fn() if output_subset is None else\
File "/usr/local/lib/python3.5/dist-packages/theano/gof/op.py", line 872, in rval
r = p(n, [x[0] for x in i], o)
File "/usr/local/lib/python3.5/dist-packages/theano/tensor/nnet/abstract_conv.py", line 1626, in perform
conv_out = self.conv(img, kern, mode="valid", dilation=self.filter_dilation)
File "/usr/local/lib/python3.5/dist-packages/theano/tensor/nnet/abstract_conv.py", line 1531, in conv
dilated_kern[n, im0, ...],
IndexError: index 1 is out of bounds for axis 1 with size 1
What does this mean? Am I calling my function incorrectly?
Your function works for me using the following model:
a = Input(shape=(224,224,3))
b = Conv2D(8, 3, strides=(2,2))(a)
model = Model(inputs=a, outputs=b)
model.compile(optimizer='sgd', loss='mse')
def get_layer0_outputs(model, test_data):
output = model.layers[0].output
inputs = [K.learning_phase()] + model.inputs
func = K.function(inputs, [output])
return func([0] + [test_data])
print get_layer0_outputs(model, np.zeros((1, 224, 224, 3)))[0].shape
Note that layer 0 is an Input layer not a Conv2D, but the code also works for layer 1. I'm using the tensorflow backend so I don't know if the difference is your model or the theano backend.
I'm trying to expand the tflearn example for linear regression by increasing the number of columns to 21.
from trafficdata import X,Y
import tflearn
print(X.shape) #(1054, 21)
print(Y.shape) #(1054,)
# Linear Regression graph
input_ = tflearn.input_data(shape=[None,21])
linear = tflearn.single_unit(input_)
regression = tflearn.regression(linear, optimizer='sgd', loss='mean_square',
metric='R2', learning_rate=0.01)
m = tflearn.DNN(regression)
m.fit(X, Y, n_epoch=1000, show_metric=True, snapshot_epoch=False)
print("\nRegression result:")
print("Y = " + str(m.get_weights(linear.W)) +
"*X + " + str(m.get_weights(linear.b)))
However, tflearn complains:
Traceback (most recent call last):
File "linearregression.py", line 16, in <module>
m.fit(X, Y, n_epoch=1000, show_metric=True, snapshot_epoch=False)
File "/usr/local/lib/python3.5/dist-packages/tflearn/models/dnn.py", line 216, in fit
callbacks=callbacks)
File "/usr/local/lib/python3.5/dist-packages/tflearn/helpers/trainer.py", line 339, in fit
show_metric)
File "/usr/local/lib/python3.5/dist-packages/tflearn/helpers/trainer.py", line 818, in _train
feed_batch)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 975, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (64,) for Tensor 'TargetsData/Y:0', which has shape '(21,)'
I found the shape (64, ) comes from the default batch size of tflearn.regression().
Do I need to transform the labels (Y)? In what way?
Thanks!
I tried to do the same. I made these changes to get it to work
# linear = tflearn.single_unit(input_)
linear = tflearn.fully_connected(input_, 1, activation='linear')
My guess is that with features >1 you cannot use tflearn.single_unit(). You can add additional fully_connected layers, but the last one must have only 1 neuron because Y.shape=(?,1)
You have 21 features. Therefore, you cannot use linear = tflearn.single_unit(input_)
Instead try this: linear = tflearn.fully_connected(input_, 21, activation='linear')
The error you get is because your labels, i.e., Y has a shape of (1054,).
You have to first preprocess it.
Try using the code given below before # linear regression graph:
Y = np.expand_dims(Y,-1)
Now before regression = tflearn.regression(linear, optimizer='sgd', loss='mean_square',metric='R2', learning_rate=0.01), type the below code:
linear = tflearn.fully_connected(linear, 1, activation='linear')
Is there a way to reshape a TF tensor inside of a custom Keras loss function? I'm defining this custom loss function for a convolutional neural network?
def custom_loss(x, x_hat):
"""
Custom loss function for training background extraction networks (autoencoders)
"""
#flatten x, x_hat before computing mean, median
shape = x_hat.get_shape().as_list()
batch_size = shape[0]
image_size = np.prod(shape[1:])
x = tf.reshape(x, [batch_size, image_size])
x_hat = tf.reshape(x_hat, [batch_size, image_size])
B0 = reduce_median(tf.transpose(x_hat))
# I divide by sigma in the next step. So I add a small float32 to F0
# so as to prevent sigma from becoming 0 or Nan.
F0 = tf.abs(x_hat - B0) + 1e-10
sigma = tf.reduce_mean(tf.sqrt(F0 / 0.5), axis=0)
background_term = tf.reduce_mean(F0 / sigma, axis=-1)
bce = binary_crossentropy(x, x_hat)
loss = bce + background_term
return loss
In addition to computing the standard binary_crossentropy an additional background_term is added into the loss. This term incentives the network to predict images close the median of a batch. Since the outputs of the CNN are 2d and reduce_median works better with 1d arrays I have to reshape the images into 1d arrays. When I try to train this network I get the error
Traceback (most recent call last):
File "stackoverflow.py", line 162, in <module>
autoencoder = build_conv_autoencoder(lambda_W, input_shape, num_filters, optimizer, custom_loss)
File "stackoverflow.py", line 136, in build_conv_autoencoder
autoencoder.compile(optimizer, loss, metrics=[mean_squared_error])
File "/usr/local/lib/python3.5/dist-packages/keras/models.py", line 594, in compile
**kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 667, in compile
sample_weight, mask)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 318, in weighted
score_array = fn(y_true, y_pred)
File "stackoverflow.py", line 26, in custom_loss
x = tf.reshape(x, [batch_size, image_size])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2448, in reshape
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 494, in apply_op
raise err
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 491, in apply_op
preferred_dtype=default_dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 710, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 441, in <listcomp>
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/compat.py", line 65, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got None
It seems like Keras is calling custom_loss before the TensorFlow graph is instantiated. This makes batch_size None instead of the actual value. Is there a proper way to reshape tensors inside loss functions to this error is avoided? You can look at the full code here .
Is there a proper way to reshape tensors...
If you are using Keras you should use the K.reshape(x,shape) method, which is a wrapper for tf.reshape(x,shape) as we can see in the docs.
I also notice you are using get_shape() to obtain your tensor shape, when on Keras you can do this with K.int_shape(x) as also mentioned in the docs, like this:
shape = K.int_shape(x_hat)
Besides that there are several other operations you do directly calling your Tensorflow import, instead of the Keras Backend (like tf.abs(), tf.reduce_mean(), tf.transpose(), etc.). You should consider using its corresponding wrappers in the keras backend to have uniform notation and guarantee a more regular behaviour. Also, by using the Keras backend you are giving your program compatibility with both Theano and Tensorflow, so it is a big plus you should consider.
Additionally, some TypeError may appear when working with tensors with undefined dimension(s). Please take a look at this question where they explain about reshaping tensors with undefined dimensions. Also, for its equivalent in Keras, check this other question, where in an answer I explain how to achieve that using Keras with Tensorflow as backend.
...Now regarding your code. Basically, as you have some undefined dimensions, you can pass the value -1 to have it infer the shape no matter what size it could be (it is explained in the first linked question, but can also be seen in the docs). Something like:
x = tf.reshape(x, [-1, image_size])
Or using Keras backend:
x = K.reshape(x, [-1, image_size])
I have a model network implemented in keras with theano as the backend which currently initialises with random weights for the filters:
# Combine and reshape for convolution
seq = concat(embeddings)
cshape = (config.window_size, sum(f.output_dim for f in features))
seq = Reshape((1,)+cshape)(seq)
# Convolutions
conv_outputs = []
for filter_size, filter_num in zip(config.filter_sizes, config.filter_nums):
conv = Convolution2D(filter_num, filter_size, cshape[1], activation='relu')(seq)
cout = Flatten()(conv)
conv_outputs.append(cout)
seq = concat(conv_outputs)
But now I want to be able to load in previously generated weights which were stored as numpy arrays.
This is the altered code which attempts to read in the arrays:
# Combine and reshape for convolution
seq = concat(embeddings)
cshape = (config.window_size, sum(f.output_dim for f in features))
seq = Reshape((1,)+cshape)(seq)
# Convolutions
conv_outputs = []
for filter_size, filter_num in zip(config.filter_sizes, config.filter_nums):
filters = np.load('path/to/filters/size-%d.npy' % filter_size)
conv = Convolution2D(filter_num, filter_size, cshape[1], weights=filters)(seq)
cout = Flatten()(conv)
conv_outputs.append(cout)
seq = concat(conv_outputs)
When I try to run the augmented script, I run into the following error:
Traceback (most recent call last):
File "conv.py", line 78, in <module>
weights=filters)(seq)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 458, in __call__
self.build(input_shapes[0])
File "/usr/local/lib/python2.7/dist-packages/keras/layers/convolutional.py", line 324, in build
self.set_weights(self.initial_weights)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 846, in set_weights
' weights. Provided weights: ' + str(weights))
Exception: You called `set_weights(weights)` on layer "convolution2d_1" with a weight list of length 3, but the layer was expecting 2 weights.
I've tried searching online but I can't figure out why keras is throwing this error up.