I am using a convolutional neural net (CNN) called make_unet from here. It works and code is able to run with this CNN. But I know that in deep learning you have to initialize weights for optimization of the neural network.
The documentation in Keras clearly indicates the use of a kernel_initializer for weight initialization. However, I do not see any kernel_initializer in the make_unet function I am using.
Anyone who can provide some insight would be appreciated.
In Keras initialisers are passed on a per-layer basis via arguments kernel_initializer and bias_initializer, e.g.
Dense(64, kernel_initializer='random_uniform', bias_initializer='zeros')
All built-in layers come with a sensible default initialiser. For example, all convolutional layers use kernel_initializer='glorot_uniform', bias_initializer='zeros'. Keras gives you many alternative options. You can also create your custom initialisers.
Related
I wanted to add one tanh layer to embedding layer with keras functional api:
x=layers.Embedding(vocab_size, 8, input_length=max_length)(input)
output=keras.activations.tanh(x)
model = Model(inputs=input, outputs=output)
model.compile(optimizer='rmsprop',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(data, labels)
but system told me I must use keras layers ,not tensor. I searched a lot keras tutorials. There is only one way to solve this problem:
model.add(Activation('tanh'))
but it is Sequential model which I don't want to use.Is there some ways to solve this with functional api?
With the functional api it's almost the same as the Sequential model:
output = Activation('tanh')(x)
I want to implement i-RevNet on MNIST dataset on keras and generate the original 28*28 input images from the output of i-RevNet, but i don't have a clue. Online resources I can find are all based on tensorflow.
important is this paper https://arxiv.org/pdf/1802.07088.pdf - i-REVNET: DEEP INVERTIBLE NETWORKS and this git https://github.com/jhjacobsen/pytorch-i-revnet
when reading the above paper critical components in i-RevNets are homeomorphic layers, on the link between topology and neural nets cf http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ - Neural Networks, Manifolds, and Topology ( search for 'homeomorphic' )
in https://github.com/jhjacobsen/pytorch-i-revnet homeomorphic layers are implemented in class irevnet_block(nn.Module), note that there are NO operations that discard information like maxpooling, averaging, ... ( with exception of the output layer ), only batch normalization ( https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c ) is applied, the ReLUs are also locally strictly linear.
in Where do I call the BatchNormalization function in Keras? is how to implement this in keras, simply stack the layers into a homeomorphic layer:
homeomorphic layer -> NO POOLING, ... LAYERS
model.add(Dense(64, init='uniform'))
model.add(Activation('relu'))
model.add(BatchNormalization())
the rest of the code in https://github.com/jhjacobsen/pytorch-i-revnet/blob/master/models/iRevNet.py like i.e. def inverse(self, x) or def forward(self, x) can be reproduced using the keras functions in https://keras.io/layers/merge/ . Cf https://github.com/jhjacobsen/pytorch-i-revnet/blob/master/models/model_utils.py on the merge and split functions, they use torch.cat and torch.split whichs keras equivalents are in https://keras.io/layers/merge/
Update: Found it: The class is tf.keras.layers.Activation; needs to be called with argument activation='relu'....
Trying to access tf.keras.layers.ReLU gives the error:
AttributeError: module
'tensorflow.tools.api.generator.api.keras.layers' has no attribute
'ReLU'.
In the docs, version master has such a layer. Version 1.8 (and 1.9) only seems to have leaky relu, PReLU, and other derivatives.
Right now I'm using ThresholdedReLU with theta of 0.0, I hope this results in a standard ReLU. But there must be a simple 'ReLU' layer as well?
Where can I find keras' ReLU layer in tensorflow 1.8?
I want a keras layer class, i.e., not tf.keras.backend.relu.
It feels as if I'm overlooking something completely obvious. I haven't used keras before, so, sorry if this is a super stupid question.
For simple activation layers you can just use the Activation layer.
Activation('relu')
Available activations can be found here
I'm building a U-net using tensorflow on top of a keras.application model as encoder. However, I'd like to change the padding of some layers from 'valid' to 'same'. How can I achieve that without instancing the entire network?
Example:
encoder = tf.contrib.keras.applications.Xception(
include_top=False, weights='imagenet',
input_tensor=inputs, pooling=None)
encoder.layers[1].padding = 'same' # this does not work, does not update the tensor
The easier way I've found was to change the original Keras file and introduce padding= 'same' in the convolutions arguments during the model instantiation.
I'm trying to replicate the code in this blog article How convolutional neural networks see the world
It works well in a CNN where there's no dropout layer but when there's one (or more) dropout layers, I can't directly use the layer.output line because it expects a learning phase.
When I use the recommend way to extract the output of a layer :
get_layer_output = K.function([model.layers[0].input, K.learning_phase()],
[model.layers[layer_index].output])
layer_output = get_3rd_layer_output([input_img, 0])[0]
The problem is that I can't put a placeholder in input_img because it expects "real" data but if I put directly "real" data then the rest of the code doesn't work (creating the loss, gradients and iterating needs a placeholder).
Is there a way I can make this work?
I'm using the Tensorflow backend.
EDIT : I solved my issue by using the K.set_learning_phase() method before doing anything like building my model (I had to start from a new environment and I used the method right after the imports).