I am performing research where I created a custom Activation Function layer, and want to incorporate it in a model using the functional API.
Problem is that when I use the functional API, it decomposes my layer into the series of tf.math operations I utilize.
With minimal code,
import tensorflow as tf
from tensorflow.python.keras.engine.base_layer import Layer
class CustomAct(Layer):
def __init__(self):
super(CustomAct, self).__init__()
# do some stuff here
#tf_utils.shape_type_conversion
def build(self, input_shape):
# do some stuff here
def call(self, x):
# performs abs((x^3)/(x))^(-(x^3)/(x))
# MODEL CREATION
inputs = tf.keras.Input(shape=(32, 32, 3))
x = tf.keras.layers.Conv2D(4, (3, 3))(inputs)
x = tf.keras.layers.BatchNormalization()(x)
x = CustomAct()(x)
x = tf.keras.layers.Dense(10, activation="sigmoid")(x)
model = tf.keras.Model(inputs, x)
The CustAct works fine using Sequential, but whenever I use the functional API, it decomposes the CustAct into the original math functions, as we can see from the model summary below:
I think the problem is that when using the functional API, you pass the previous input, which then calls the call() function, which performs my mathematics. Is there another function I need to implement or something to get rid of this problem?
Related
i built and trained my LSTM model for a regression task and everything works fine. i would like to use the fast_gradient_method function from cleverhans (or any other cleverhans function as the issue stands for any other attack).
i don't understand how am i supposed to pass the model to the function. from cleverhans:
:param model_fn: a callable that takes an input tensor and returns the model logits
whatever input i give to the function (the model itself, the weights i get with get_weights, the weights of the "stage" right before the dense layer...), i get this error:
TypeError: 'module' object is not callable
what would be the correct input to make it work?
in the only working example i found, the following line of code is used to define logits_model and then pass it as :param model_fn:, but i still get the error above
logits_model = tf.keras.Model(model.input,model.layers[-1].output)
to pass a valid model, it should be defined in the following way:
(it is just an example)
"make" is only needed for model.summary() to work, I found the code in another SO post that I can't seem to find right now
class modSubclass(Model):
def __init__(self):
super(modSubclass, self).__init__()
self.lstm1 = GRU(hidden_size1, activation='relu',return_sequences=True,input_shape=(input_size,1))
self.lstm2 = GRU(hidden_size2, activation='relu')
self.dense1 = Dense(K, activation='relu')
def call(self,x):
x = self.lstm1(x)
x = self.lstm2(x)
x = self.dense1(x)
return x
def make(self, input_shape):
'''
This method makes the command "model.summary()" work.
input_shape: (H,W,C), do not specify batch B
'''
x = tf.keras.layers.Input(shape=input_shape)
model = tf.keras.Model(inputs=[x], outputs=self.call(x), name='actor')
print(model.summary())
return model
I am trying to implement a custom PCA layer for my model being developed using Model Subclassing API. This is how I have defined the layer.
class PCALayer(tf.keras.layers.Layer):
def __init__(self):
super(PCALayer, self).__init__()
self.pc = pca
def call(self, input_tensor, training=False):
x = K.constant(self.pc.transform(input_tensor))
return x
The pca itself is from sklearn.decomposition.PCA and has been fit with the needed data and not transformed.
Now, this is how I have added the layer to my model
class ModelSubClassing(tf.keras.Model):
def __init__(self, initizlizer):
super(ModelSubClassing, self).__init__()
# define all layers in init
# Layer of Block 1
self.pca_layer = PCALayer()
self.dense1 = tf.keras.layers.Dense(...)
self.dense2 = tf.keras.layers.Dense(...)
self.dense3 = tf.keras.layers.Dense(...)
def call(self, input_tensor, training=False):
# forward pass: block 1
x = self.pca_layer(input_tensor)
x = self.dense1(x)
x = self.dense2(x)
return self.dense3(x)
When I compile the model there is no error. However, when I fit the model, I get the following error:
NotImplementedError: Cannot convert a symbolic Tensor (model_sub_classing_1/Cast:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
Can anyone help me please...
self.pc.transform which comes from sklearn is expecting a numpy array, but you provide a tf tensor. When the layer is built, it passes a symbolic tensor to build the graph etc, and this tensor cannot be converted to a numpy array. The answer is in error :
you're trying to pass a Tensor to a NumPy call, which is not supported
Compare the following code snippets. I implemented a simple keras model like this
inp = layers.Input((10,2))
x = layers.Flatten()(inp)
x = layers.Dense(5)(x)
m = models.Model(inputs=inp, outputs=x)
For one reason or another, I need to have my model in an objective way. So no problem, it's easy to reimplement that into:
class MyModel(tf.keras.Model):
def __init__(self, inp_shape, out_size = 5):
super(MyModel, self).__init__()
self.inp = layers.InputLayer(input_shape=inp_shape)
self.flatten = layers.Flatten()
self.dense = layers.Dense(out_size)
def call(self, a):
x = self.inp(a)
x = self.flatten(x)
x = self.dense(x)
return x
However in the second case when I try to run:
m = MyModel((10,2))
m.summary()
I get:
ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.
I don't quite get why? Shouldn't the above be equivalent?
The reason for this is that when you create an object of this model you are just creating its layers and not its graph. So in short the output from layer 1 is not going in layer 2 cause those are entirely separate attributes of the class but when you call the model those separate attributes combines and form the graph.
When you define a model in tf. keras with subclassed API, you need to build the model first by calling build or run the model on some data.
m = MyModel((10,2))
m.build(input_shape=(10, 2)) # < -- build the model
m.summary()
That said, you don't also need to define self.inp while building the model with subclassed API. The .summary() may not look right to you for the subclassed model, you may need to check this instead.
In keras / tensorflow it is often quite simple to describe layers directly as functions that map their input to an output, like so:
def resnet_block(x, kernel_size):
ch = x.shape[-1]
out = Conv2D(ch, kernel_size, strides = (1,1), padding='same', activation='relu')(x)
out = Conv2D(ch, kernel_size, strides = (1,1), padding='same', activation='relu')(out)
out = Add()([x,out])
return out
whereas subclassing Layer to get something like
r = ResNetBlock(kernel_size=(3,3))
y = r(x)
is a little more cumbersome (or even a lot more cumbersome for more complex examples).
Since keras seems perfectly happy to construct the underlying weights of its layers when they're being called for the first time, I was wondering if it was possible to just wrap functions such as the one above and let keras figure things out once there are inputs, i.e. I would like it to look like this:
r = FunctionWrapperLayer(lambda x:resnet_block(x, kernel_size=(3,3)))
y = r(x)
I've made an attempt at implementing FunctionWrapperLayer, which looks as follows:
class FunctionWrapperLayer(Layer):
def __init__(self, fn):
super(FunctionWrapperLayer, self).__init__()
self.fn = fn
def build(self, input_shape):
shape = input_shape[1:]
inputs = Input(shape)
outputs = self.fn(inputs)
self.model = Model(inputs=inputs, outputs=outputs)
self.model.compile()
def call(self, x):
return self.model(x)
This looks like it might work, however I've run into some bizarre issues whenever I use activations, e.g. with
def bad(x):
out = tf.keras.activations.sigmoid(x)
out = Conv2D(1, (1,1), strides=(1,1), padding='same')(out)
return out
x = tf.constant(tf.reshape(tf.range(48,dtype=tf.float32),[1,4,-1,1])
w = FunctionWrapperLayer(bad)
w(x)
I get the following error
FailedPreconditionError: Error while reading resource variable _AnonymousVar34 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar34/class tensorflow::Var does not exist.
[[node conv2d_6/BiasAdd/ReadVariableOp (defined at <ipython-input-33-fc380d9255c5>:12) ]] [Op:__inference_keras_scratch_graph_353]
What this suggests to me is that there is something inherently wrong with initializing models like that in the build method. Maybe someone has a better idea as to what might be going on there or how else to get the functionality I would like.
Update:
As mentioned by jr15, the above does work when the function involved only uses keras layers. However, the following ALSO works, which has me a little puzzled:
i = Input(x.shape[1:])
o = bad(i)
model = Model(inputs=i, outputs=o)
model(x)
Incidentally, model.submodules yields
(<tensorflow.python.keras.engine.input_layer.InputLayer at 0x219d80c77c0>,
<tensorflow.python.keras.engine.base_layer.TensorFlowOpLayer at 0x219d7afc820>,
<tensorflow.python.keras.layers.convolutional.Conv2D at 0x219d7deafa0>)
meaning the activation is automatically turned into a "TensorFlowOpLayer" when doing it like that.
Another update:
Looking at the original error message, it seems like the activation isn't the only culprit. If I remove the convolution and use the wrapper everything works as well and again I find a "TensorFlowOpLayer" when inspecting the submodules.
You solution actually works! The trouble you're running into is that tf.keras.activations.sigmoid is not a Layer, but a plain Tensorflow function. To make it work, use keras.layers.Activation("sigmoid")(x) instead. For the more general case, where you want to use some Tensorflow function as a layer, you can wrap it in a Lambda layer like so:
out = keras.layers.Lambda(lambda x: tf.some_function(x))(out)
See the docs for more info: https://keras.io/api/layers/core_layers/lambda/
With Tensorflow 2.4 it apparently just works now. The submodules now show a "TFOpLambda" layer.
To anybody interested, here is some slightly improved wrapper code that also accommodates multi-input models:
class FunctionWrapperLayer(Layer):
def __init__(self, fn):
super(FunctionWrapperLayer, self).__init__()
self.fn = fn
def build(self, input_shapes):
super(FunctionWrapperLayer, self).build(input_shapes)
if type(input_shapes) is list:
inputs = [Input(shape[1:]) for shape in input_shapes]
else:
inputs = Input(input_shapes[1:])
outputs = self.fn(inputs)
self.fn_model = Model(inputs=inputs, outputs=outputs)
self.fn_model.compile()
def call(self, x):
return self.fn_model(x)
I am trying to apply one idea proposed by Rusu et al. in https://arxiv.org/pdf/1511.06295.pdf, which consists in training a NN changing the output layer according to the class of the input, i.e., provided that we know the id of the input, we would pick the corresponding output layer. This way, all the hidden layers would be trained with all the data, but each output layer would only be trained with its corresponding type of input data.
This is meant to achieve good results in a transfer learning framework.
How can I implement this "change of the last layer" in tensorflow 2.0?
If you use model subclassing, you can actually define you forward pass.
class MyModel(tf.keras.Model):
def __init__(self):
super(Model, self).__init__()
self.block_1 = BlockA()
self.block_2 = BlockB()
self.global_pool = layers.GlobalAveragePooling2D()
self.classifier = Dense(num_classes)
def call(self, inputs):
if condition:
x = self.block_1(inputs)
else:
x = self.block_2(inputs)
x = self.global_pool(x)
return self.classifier(x)
You'll still have the backprop part to figure out, but I think it's fairly easy if you use a multioutput model and train all your "last layers" at the same time.