In keras / tensorflow it is often quite simple to describe layers directly as functions that map their input to an output, like so:
def resnet_block(x, kernel_size):
ch = x.shape[-1]
out = Conv2D(ch, kernel_size, strides = (1,1), padding='same', activation='relu')(x)
out = Conv2D(ch, kernel_size, strides = (1,1), padding='same', activation='relu')(out)
out = Add()([x,out])
return out
whereas subclassing Layer to get something like
r = ResNetBlock(kernel_size=(3,3))
y = r(x)
is a little more cumbersome (or even a lot more cumbersome for more complex examples).
Since keras seems perfectly happy to construct the underlying weights of its layers when they're being called for the first time, I was wondering if it was possible to just wrap functions such as the one above and let keras figure things out once there are inputs, i.e. I would like it to look like this:
r = FunctionWrapperLayer(lambda x:resnet_block(x, kernel_size=(3,3)))
y = r(x)
I've made an attempt at implementing FunctionWrapperLayer, which looks as follows:
class FunctionWrapperLayer(Layer):
def __init__(self, fn):
super(FunctionWrapperLayer, self).__init__()
self.fn = fn
def build(self, input_shape):
shape = input_shape[1:]
inputs = Input(shape)
outputs = self.fn(inputs)
self.model = Model(inputs=inputs, outputs=outputs)
self.model.compile()
def call(self, x):
return self.model(x)
This looks like it might work, however I've run into some bizarre issues whenever I use activations, e.g. with
def bad(x):
out = tf.keras.activations.sigmoid(x)
out = Conv2D(1, (1,1), strides=(1,1), padding='same')(out)
return out
x = tf.constant(tf.reshape(tf.range(48,dtype=tf.float32),[1,4,-1,1])
w = FunctionWrapperLayer(bad)
w(x)
I get the following error
FailedPreconditionError: Error while reading resource variable _AnonymousVar34 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar34/class tensorflow::Var does not exist.
[[node conv2d_6/BiasAdd/ReadVariableOp (defined at <ipython-input-33-fc380d9255c5>:12) ]] [Op:__inference_keras_scratch_graph_353]
What this suggests to me is that there is something inherently wrong with initializing models like that in the build method. Maybe someone has a better idea as to what might be going on there or how else to get the functionality I would like.
Update:
As mentioned by jr15, the above does work when the function involved only uses keras layers. However, the following ALSO works, which has me a little puzzled:
i = Input(x.shape[1:])
o = bad(i)
model = Model(inputs=i, outputs=o)
model(x)
Incidentally, model.submodules yields
(<tensorflow.python.keras.engine.input_layer.InputLayer at 0x219d80c77c0>,
<tensorflow.python.keras.engine.base_layer.TensorFlowOpLayer at 0x219d7afc820>,
<tensorflow.python.keras.layers.convolutional.Conv2D at 0x219d7deafa0>)
meaning the activation is automatically turned into a "TensorFlowOpLayer" when doing it like that.
Another update:
Looking at the original error message, it seems like the activation isn't the only culprit. If I remove the convolution and use the wrapper everything works as well and again I find a "TensorFlowOpLayer" when inspecting the submodules.
You solution actually works! The trouble you're running into is that tf.keras.activations.sigmoid is not a Layer, but a plain Tensorflow function. To make it work, use keras.layers.Activation("sigmoid")(x) instead. For the more general case, where you want to use some Tensorflow function as a layer, you can wrap it in a Lambda layer like so:
out = keras.layers.Lambda(lambda x: tf.some_function(x))(out)
See the docs for more info: https://keras.io/api/layers/core_layers/lambda/
With Tensorflow 2.4 it apparently just works now. The submodules now show a "TFOpLambda" layer.
To anybody interested, here is some slightly improved wrapper code that also accommodates multi-input models:
class FunctionWrapperLayer(Layer):
def __init__(self, fn):
super(FunctionWrapperLayer, self).__init__()
self.fn = fn
def build(self, input_shapes):
super(FunctionWrapperLayer, self).build(input_shapes)
if type(input_shapes) is list:
inputs = [Input(shape[1:]) for shape in input_shapes]
else:
inputs = Input(input_shapes[1:])
outputs = self.fn(inputs)
self.fn_model = Model(inputs=inputs, outputs=outputs)
self.fn_model.compile()
def call(self, x):
return self.fn_model(x)
Related
i built and trained my LSTM model for a regression task and everything works fine. i would like to use the fast_gradient_method function from cleverhans (or any other cleverhans function as the issue stands for any other attack).
i don't understand how am i supposed to pass the model to the function. from cleverhans:
:param model_fn: a callable that takes an input tensor and returns the model logits
whatever input i give to the function (the model itself, the weights i get with get_weights, the weights of the "stage" right before the dense layer...), i get this error:
TypeError: 'module' object is not callable
what would be the correct input to make it work?
in the only working example i found, the following line of code is used to define logits_model and then pass it as :param model_fn:, but i still get the error above
logits_model = tf.keras.Model(model.input,model.layers[-1].output)
to pass a valid model, it should be defined in the following way:
(it is just an example)
"make" is only needed for model.summary() to work, I found the code in another SO post that I can't seem to find right now
class modSubclass(Model):
def __init__(self):
super(modSubclass, self).__init__()
self.lstm1 = GRU(hidden_size1, activation='relu',return_sequences=True,input_shape=(input_size,1))
self.lstm2 = GRU(hidden_size2, activation='relu')
self.dense1 = Dense(K, activation='relu')
def call(self,x):
x = self.lstm1(x)
x = self.lstm2(x)
x = self.dense1(x)
return x
def make(self, input_shape):
'''
This method makes the command "model.summary()" work.
input_shape: (H,W,C), do not specify batch B
'''
x = tf.keras.layers.Input(shape=input_shape)
model = tf.keras.Model(inputs=[x], outputs=self.call(x), name='actor')
print(model.summary())
return model
I am performing research where I created a custom Activation Function layer, and want to incorporate it in a model using the functional API.
Problem is that when I use the functional API, it decomposes my layer into the series of tf.math operations I utilize.
With minimal code,
import tensorflow as tf
from tensorflow.python.keras.engine.base_layer import Layer
class CustomAct(Layer):
def __init__(self):
super(CustomAct, self).__init__()
# do some stuff here
#tf_utils.shape_type_conversion
def build(self, input_shape):
# do some stuff here
def call(self, x):
# performs abs((x^3)/(x))^(-(x^3)/(x))
# MODEL CREATION
inputs = tf.keras.Input(shape=(32, 32, 3))
x = tf.keras.layers.Conv2D(4, (3, 3))(inputs)
x = tf.keras.layers.BatchNormalization()(x)
x = CustomAct()(x)
x = tf.keras.layers.Dense(10, activation="sigmoid")(x)
model = tf.keras.Model(inputs, x)
The CustAct works fine using Sequential, but whenever I use the functional API, it decomposes the CustAct into the original math functions, as we can see from the model summary below:
I think the problem is that when using the functional API, you pass the previous input, which then calls the call() function, which performs my mathematics. Is there another function I need to implement or something to get rid of this problem?
Compare the following code snippets. I implemented a simple keras model like this
inp = layers.Input((10,2))
x = layers.Flatten()(inp)
x = layers.Dense(5)(x)
m = models.Model(inputs=inp, outputs=x)
For one reason or another, I need to have my model in an objective way. So no problem, it's easy to reimplement that into:
class MyModel(tf.keras.Model):
def __init__(self, inp_shape, out_size = 5):
super(MyModel, self).__init__()
self.inp = layers.InputLayer(input_shape=inp_shape)
self.flatten = layers.Flatten()
self.dense = layers.Dense(out_size)
def call(self, a):
x = self.inp(a)
x = self.flatten(x)
x = self.dense(x)
return x
However in the second case when I try to run:
m = MyModel((10,2))
m.summary()
I get:
ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
I don't quite get why? Shouldn't the above be equivalent?
The reason for this is that when you create an object of this model you are just creating its layers and not its graph. So in short the output from layer 1 is not going in layer 2 cause those are entirely separate attributes of the class but when you call the model those separate attributes combines and form the graph.
When you define a model in tf. keras with subclassed API, you need to build the model first by calling build or run the model on some data.
m = MyModel((10,2))
m.build(input_shape=(10, 2)) # < -- build the model
m.summary()
That said, you don't also need to define self.inp while building the model with subclassed API. The .summary() may not look right to you for the subclassed model, you may need to check this instead.
I would like to implement a custom tf.keras layer called MyLayer which has three inputs and contains a sub layer which in turn has three inputs, like in the figure below:
I assume that the right thing to do would be to create a MyLayer class that extends tf.keras.layers.Layer and implement the __init__, build and call methods, as mentioned in the official documentation.
Now, the examples provided in the documentation are relative to pretty simple layers that are composed of several sublayers connected in a sequential manner, that is one after the other. For instance, the MLPBlock layer consists of 3 linear layers ordered sequentially.
In general, however, sublayers are not ordered sequentially, but can form branches. This suggests that those layers could be run in parallel, since they are not connected to one another.
Going back to the custom layer I would like to implement, you can see that Layer1, Layer2 and Layer3 could be run in parallel. Once their outputs are computed, they can be fed to Layer4. The point is: how do I run them in parallel? I couldn't find any "ParallelCombinator" or things like that among the available Keras layers.
If I were to follow the examples provided in the documentation, I would write something along these lines:
class MyLayer(keras.layers.Layer):
def __init__(self, ...):
super(MyLayer, self).__init__()
self.layer_1 = Layer1(...)
self.layer_2 = Layer2(...)
self.layer_3 = Layer3(...)
self.layer_4 = Layer4(...)
def call(self, inputs):
tmp_1 = self.layer_1(inputs[0])
tmp_2 = self.layer_2(inputs[1])
tmp_3 = self.layer_3(inputs[2])
return self.layer4([tmp_1, tmp_2, tmp_3])
This, however, would imply that Layer1, Layer2 and Layer3 are run sequentially, not in parallel.
One possible solution that I came up with involves structuring MyLayer as a tf.keras.Model built with Keras's functional API rather than as a subclass of tf.keras.Layer, like so:
def MyLayer(...):
input_1 = tf.keras.layers.Input(...)
input_2 = tf.keras.layers.Input(...)
input_3 = tf.keras.layers.Input(...)
layer_1 = Layer1(...)(input_1)
layer_2 = Layer2(...)(input_2)
layer_3 = Layer3(...)(input_3)
output_1 = Layer4(...)([layer_1, layer_2, layer_3])
return tf.keras.Model(inputs=[input_1, input_2, input_3], outputs=output_1)
if __name__ == '__main__':
my_layer = MyLayer(...)
input_1 = ...
input_2 = ...
input_3 = ...
output = my_layer([input_1, input_2, input_3])
The reason why I think this would work is that I assume that when I feed some inputs to a tf.keras.Model, as in output = my_layer([input_1, input_2, input_3]), the layers that can be run in parallel are effectively run in parallel (or are they?). This solution, however, feels like a hack to me, as MyLayer is supposed to be a layer, not a model. In fact, a tf.keras.Model instance exposes methods like fit(...) that aren't meant to be called on a layer.
Does anybody know what's the best approach to implement MyLayer?
I want to write some custom Keras Layers and do some advanced calculations in the layer, for example with Numpy, Scikit, OpenCV...
I know there are some math functions in keras.backend that can operate on tensors, but i need some more advanced functions.
However, i have no clue how to implement this correctly, i get the error message:
You must feed a value for placeholder tensor 'input_1' with dtype float and shape [...]
Here is my custom layer:
class MyCustomLayer(Layer):
def __init__(self, **kwargs):
super(MyCustomLayer, self).__init__(**kwargs)
def call(self, inputs):
"""
How to implement this correctly in Keras?
"""
nparray = K.eval(inputs) # <-- does not work
# do some calculations here with nparray
# for example with Numpy, Scipy, Scikit, OpenCV...
result = K.variable(nparray, dtype='float32')
return result
def compute_output_shape(self, input_shape):
output_shape = tuple([input_shape[0], 256, input_shape[3]])
return output_shape # (batch, 256, channels)
The error appears here in this dummy model:
inputs = Input(shape=(96, 96, 3))
x = MyCustomLayer()(inputs)
x = Flatten()(x)
x = Activation("relu")(x)
x = Dense(1)(x)
predictions = Activation("sigmoid")(x)
model = Model(inputs=inputs, outputs=predictions)
Thanks for all hints...
TD;LR You should not mix Numpy inside Keras layers. Keras uses Tensorflow underneath because it has to track all the computations to be able to compute the gradients in the backward phase.
If you dig in Tensorflow, you will see that it almost covers all the Numpy functionality (or even extends it) and if I remember correctly, Tensorflow functionality can be accessed through the Keras backend (K).
What are the advance calculations/functions you need?
i think that this kinda process should apply before the model because the process does not contain variables so it cant be optimized.
K.eval(inputs) does not work beacuse you are trying to evaluate a placeholder not variable placeholders has not values for evaluate. if you want get values you should feed it or you can make a list from tensors one by one with tf.unstack()
nparray = tf.unstack(tf.unstack(tf.unstack(inputs,96,0),96,0),3,0)
your call function is wrong because returns a variable you should return a constant:
result = K.constant(nparray, dtype='float32')
return result