I'm trying to use tensorflow's MobileNet v2.
I don't understand why, but it seems that the last fully connected layers, with the output categories (dimensionality 1000) layer is missing and I'm left with what seems to be just the embeddings after some convolutional layer.
Any idea on why this is happening? How can I add, or where can I find the pre-trained fully connected layers block?
Here is the code:
image = np.array(PIL.Image.open("amsterdam.jpg"))
image = np.expand_dims(image,0)
IMG_SIZE = image.shape[1:3]
IMG_SHAPE = IMG_SIZE + (3,)
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
tf.keras.utils.plot_model(base_model,to_file='model.png', show_shapes=True)
Here you can see the structure of the neural network as I plotted it with tf.keras.utils.plot_model:
Any idea on how to fix this?
include_top=False: return the model without dense layers for classification. You can add your own dense layers.
include_top=True: return the entire model.
If you want to get also dense layers for classification, use include_top=True as the default is. When you set include_top=False the model will not return the dense layers, in order to let you make your own dense layers and make your own classification to suit your needs.
I wanted to add one tanh layer to embedding layer with keras functional api:
x=layers.Embedding(vocab_size, 8, input_length=max_length)(input)
output=keras.activations.tanh(x)
model = Model(inputs=input, outputs=output)
model.compile(optimizer='rmsprop',loss='categorical_crossentropy',metrics=['accuracy'])
model.fit(data, labels)
but system told me I must use keras layers ,not tensor. I searched a lot keras tutorials. There is only one way to solve this problem:
model.add(Activation('tanh'))
but it is Sequential model which I don't want to use.Is there some ways to solve this with functional api?
With the functional api it's almost the same as the Sequential model:
output = Activation('tanh')(x)
I want to implement i-RevNet on MNIST dataset on keras and generate the original 28*28 input images from the output of i-RevNet, but i don't have a clue. Online resources I can find are all based on tensorflow.
important is this paper https://arxiv.org/pdf/1802.07088.pdf - i-REVNET: DEEP INVERTIBLE NETWORKS and this git https://github.com/jhjacobsen/pytorch-i-revnet
when reading the above paper critical components in i-RevNets are homeomorphic layers, on the link between topology and neural nets cf http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ - Neural Networks, Manifolds, and Topology ( search for 'homeomorphic' )
in https://github.com/jhjacobsen/pytorch-i-revnet homeomorphic layers are implemented in class irevnet_block(nn.Module), note that there are NO operations that discard information like maxpooling, averaging, ... ( with exception of the output layer ), only batch normalization ( https://towardsdatascience.com/batch-normalization-in-neural-networks-1ac91516821c ) is applied, the ReLUs are also locally strictly linear.
in Where do I call the BatchNormalization function in Keras? is how to implement this in keras, simply stack the layers into a homeomorphic layer:
homeomorphic layer -> NO POOLING, ... LAYERS
model.add(Dense(64, init='uniform'))
model.add(Activation('relu'))
model.add(BatchNormalization())
the rest of the code in https://github.com/jhjacobsen/pytorch-i-revnet/blob/master/models/iRevNet.py like i.e. def inverse(self, x) or def forward(self, x) can be reproduced using the keras functions in https://keras.io/layers/merge/ . Cf https://github.com/jhjacobsen/pytorch-i-revnet/blob/master/models/model_utils.py on the merge and split functions, they use torch.cat and torch.split whichs keras equivalents are in https://keras.io/layers/merge/
I am using Keras for computing a simple sequence classification neural network. I played with the different module and I found that there are two way to create Sequential neural network.
The first way is to use Sequential API. This is the most common way which I found in a lot of tutorial/documentation.
Here is the code :
# Sequential Neural Network using Sequential()
model = Sequential()
model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu', input_shape=(27 , 300,)))
model.add(MaxPooling1D(pool_size=2))
model.add(LSTM(100))
model.add(Dense(len(7, activation='softmax'))
model.summary()
The second ways is to build de sequential neural network from "scratch" with the Model API. Here is the code.
# Sequential neural network using Model()
inputs = Input(shape=(27 , 300))
x = Conv1D(filters=32, kernel_size=3, padding='same', activation='relu')(inputs)
x = MaxPooling1D(pool_size=2)(x)
x = LSTM(100)(x)
predictions = Dense(7, activation='softmax')(x)
model = Model(inputs=inputs, outputs=predictions)
model.summary()
I trained it both with a fixed seed (np.random.seed(1337)), with the same training data and my output are different...
Knowing that the only difference in the summary is the first layer of inputs with the Model API.
Is there anyone that knows why this neural network are different ?
And if there are not, why did i get different results ?
Thanks
You setup the random seed only in numpy and not in tensorflow (in case it's the backend of keras in your case). Try to add this in your code:
from numpy.random import seed
seed(1337)
from tensorflow import set_random_seed
set_random_seed(1337)
the detailed article about this topic here
tf.keras.backend.clear_session()
tf.random.set_seed(seed_value)
You can use above code block and run the loaded model for some iterations and check if the error still persists .
I was facing the same issue for reproducibiity,it worked for me .
As mentioned by andrey, over and above these 2 seed setter, you need to setup the Python Hash Environment
import os
os.environ['PYTHONHASHSEED']=str(seed_value)
you can still add one more block to force TensorFlow to use single thread.
( if you are using multicore)
Multiple threads are a potential source of non-reproducible results.
session_conf = tf.ConfigProto(
intra_op_parallelism_threads=1,
inter_op_parallelism_threads=1)
sess = tf.Session(config=session_conf)
AFAIK, we still need to create a model and add layers to the model using the functional API. Why people say that the functional AI is used to create non-sequential neural network?
Why people said it is used to non-sequential neural network?
The thing is that with the Sequential Model you are defining a model step by step (sequentially), when you call the .add() method. Whereas, on the Functional API (specifically the Model class) you have more freedom, as you can define different layers that receive different inputs and then instantiate a model with the Model creator using any of those layers (not necessarily in a step by step, or sequential way).
In other words, when calling model = Sequential() you are in that moment instantiating your model object (for which you then add layers and constraints). In the Functional API, you create layers and then instantiate your model by calling model = Model(inputs=in, outputs=out) with your desired input and output layer(s). As you see, both approaches can be equivalent, for example, these two are the same:
from keras.models import Sequential, Model
from keras.layers import Input, Dense, Activation
#---Using the Sequential model
model = Sequential() #Object is instantiated here
model.add(Dense(32, input_dim=784))
model.add(Activation('relu'))
#---Or using the Functional API
a = Input(shape=(784,))
b = Dense(32, activation='relu')(a)
model = Model(inputs=a, outputs=b) #Object is instantiated here
Considering only this, then choosing which way to go depends more on your personal style and coding preferences. Now, there is a major advantage of using the Functional API over the Sequential model, which is that you can share or reuse layers across different models.
When compiling and fitting a model, all its associated layers will be compiled and trained. Thus, any other model that shares such layers will also reflect those changes. This gives you the freedom to do many things, as obtaining sub-models of your network, redefine them, obtain its relative outputs, merge them into more complex models, etc., without having to train again for each of those sub-models.
To make it clearer, here is an example (based on this Keras Autoencoder Blog post) that illustrates what was discussed in the last paragraph:
from keras.layers import Input, Dense
from keras.models import Model
#Create an autoencoder, along with its encoder and decoder model
input_img = Input(shape=(784,))
encoded = Dense(32, activation='relu')(input_img)
decoded = Dense(784, activation='sigmoid')(encoded)
#Here we define our autoencoder model: image -> encoding -> decoded image
autoencoder = Model(input_img, decoded)
#Now here is the advantage of the Funcional API
#We can reuse those layers to obtain an encoder model (image -> encoding)
#as well as a decoder model (encoding -> image)
#but compile all three by just compiling and fitting the Autoencoder model
encoder = Model(input_img, encoded) #see how the 'encoded' layer is output
# create a placeholder for an encoded (32-dimensional) input
encoded_input = Input(shape=(32,))
# retrieve the last layer of the autoencoder model
decoder_layer = autoencoder.layers[-1]
# create the decoder model
decoder = Model(encoded_input, decoder_layer(encoded_input))
#compile and fit with your data
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')
autoencoder.fit(X,Y,...)
After this, you will be able to make predictions on the encoder and decoder model individually (to visualize your encoding, for example), as well as make predictions with the autoencoder model as a whole. At this point, doing the following things are equivalent:
#obtain reconstructed representation directly
autoencoder_imgs = autoencoder.predict(x_test)
#obtain reconstructed representation by joining encoder and decoder models
encoder_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)
Hope this helps. Personally, I always try to use the Functional API instead, regardless if I want or not to reuse or recycle layers, as I find it more verbose, but that is up to you to decide.
Well, "sequential" is not really the most correct term, but it's the name chosen by Keras developers. All models work in sequence, of course.
The difference is:
The Sequential model is a straight line. You keep adding layers, every new layer takes the output of the previous layer. You cannot make creative graphs with branches.
The functoinal API Model is completely free to have as many ramifications, inputs and outputs as you need.
Example with the Sequential model:
from keras.models import Sequential
from keras.layers import *
#you create a model
model = Sequential()
#and you add layers
model.add(SomeKerasLayer(...))
model.add(AnotherKerasLayer(...))
#as you can see, this model is a straight line, you only add layers "sequentially"
Example with the functional API Model:
Now here we start creating really fancy models.
from keras.models import Model
from keras.layers import *
We start by defining input tensors. And we can have any number of inputs! (The sequential model is limited to one input, that you define in the first layer with input_shape).
input1 = Input(inputShape1)
#We can have more inputs if we want!
input2 = Input(inputShape2)
input3 = Input(inputShape3)
We work by creating layers and "calling layers with input tensors".
When we call a layer with an input tensor, we get an output tensor.
And we can create whatever path we want.
#Example: two separate layers taking two separate inputs:
output1 = SomeLayer(...)(input1)
output2 = AnotherLayer(...)(input2)
We can join two branches with different options, such as add, multiply, concatenate, etc.:
#joining the previous tensors output1 and output2
joined1_2 = Concatenate()([output1,output2])
We can reuse the same layers with different inputs, getting different outputs:
aLayer = AKerasLayer(...) #notice I'm creating this layer but not calling it yet
#calling the same layer with two different inputs
output1 = aLayer(joined1_2)
output2 = aLayer(input3)
And finally, we can define the model with as many inputs and outputs we want:
model = Model([input1,input2,input3],[output1, output2])
Reusing and associating models
Both models, Sequential and functional API, can be used as if they were layers.
You can call a model with and input tensor and get an output tensor, just like you would do when creating a functional API model:
input1 = Input(shape)
output1 = anExistingSequentialModel(input1)
output2 = anExistingFunctionalModel(input1)
newModel = Model(input1,[output1,output2])
And you can add models in sequential models as well (beware of the branches, it's best to have one input and one output for the added model, since this is a Sequential one)
seqModel = Sequential()
seqModel.add(anotherModel)