I currently have a keras model which uses an Embedding layer. Something like this:
input = tf.keras.layers.Input(shape=(20,) dtype='int32')
x = tf.keras.layers.Embedding(input_dim=1000,
output_dim=50,
input_length=20,
trainable=True,
embeddings_initializer='glorot_uniform',
mask_zero=False)(input)
This is great and works as expected. However, I want to be able to send text to my model, have it preprocess the text into integers, and continue normally.
Two issues:
1) The Keras docs say that Embedding layers can only be used as the first layer in a model: https://keras.io/layers/embeddings/
2) Even if I could add a Lambda layer before the Embedding, I'd need it to keep track of certain state (like a dictionary mapping specific words to integers). How might I go about this stateful preprocessing?
In short, I need to modify the underlying Tensorflow DAG, so when I save my model and upload to ML Engine, it'll be able to handle my sending it raw text.
Thanks!
Here are the first few layers of a model which uses a string input:
input = keras.layers.Input(shape=(1,), dtype="string", name='input_1')
lookup_table_op = tf.contrib.lookup.index_table_from_tensor(
mapping=vocab_list,
num_oov_buckets=num_oov_buckets,
default_value=-1,
)
lambda_output = Lambda(lookup_table_op.lookup)(input)
emb_layer = Embedding(int(number_of_categories),int(number_of_categories**0.25))(lambda_output)
Then you can continue the model as you normally would after an embedding layer. This is working for me and the model trains fine from string inputs.
It is recommended that you do the string -> int conversion in some preprocessing step to speed up the training process. Then after the model is trained you create a second keras model that just converts string -> int and then combine the two models to get the full string -> target model.
Related
I am trying to train a model for emotion recognition, which uses one of VGG's layer's output as an input.
I could manage what I want by running the prediction in a first step, saving the extracted features and then using them as input to my network, but I am looking for a way to do the whole process at once.
The second model uses a concatenated array of feature maps as input (I am working with video data), so I am not able to simply wire it to the output of VGG.
I tried to use a map operation as depicted in the tf.data.dataset API documentations this way :
def trimmed_vgg16():
vgg16 = tf.keras.applications.vgg16.VGG16(input_shape=(224,224,3))
trimmed = tf.keras.models.Model(inputs=vgg16.get_input_at(0),
outputs=vgg16.layers[-3].get_output_at(0))
return trimmed
vgg16 = trimmed_vgg16()
def _extract_vgg_features(images, labels):
pred = vgg16_model.predict(images, batch_size=batch_size, steps=1)
return pred, labels
dataset = #load the dataset (image, label) as usual
dataset = dataset.map(_extract_vgg_features)
But I'm getting this error : Tensor Tensor("fc1/Relu:0", shape=(?, 4096), dtype=float32) is not an element of this graph which is pretty explicit. I'm stuck here, as I don't see a good way of inserting the trained model in the same graph and getting predictions "on the fly".
Is there a clean way of doing this or something similar ?
Edit: missed a line.
Edit2: added details
You should be able to connect the layers by first creating the vgg16 and then retrieving the output of the model as such and afterward you can use that tensor as an input to your own network.
vgg16 = tf.keras.applications.vgg16.VGG16(input_shape=(224,224,3))
network_input = vgg16.get_input_at(0)
vgg16_out = vgg16.layers[-3].get_output_at(0) # use this tensor as input to your own network
So, I got this multi-input model with 6 identical inputs of same shape. Right now If I have to use this model, I have to multiply my input data with total numbers of input layer, i.e. 6. I was wondering if I can add another layer on top of this and can pass single input that will connect with all these 6 inputs. I'm not sure how to accomplish this! Any thoughts?
Issue was something like this: I have a "BASE" multi-input model, where all inputs are identical, as this "BASE" model was just a combination of multiple models which happens to share identical type input! Now, when using this "BASE" model for classification, I had to provide [input_data x "total_inputs"] for each input layer, which is something I don't wanted to do, say when classifying millions of sentences!
So, the ideal solution was to just have a single input which is connected with all "BASE" model inputs!!
Alrighty, so here is how it's done:
Create a new top_model which will take single input and generate multiple identical outputs. This can be done with Lambda layer.
single_input = layers.Input(input_shape)
multi_output = layers.Lambda(lambda x: [x] * total_numbers_of_base_inputs)(single_input)
top_model = Model(inputs=single_input, outputs=multi_output)
Use the top_model input and your multi_input_base_model like below to create new single_input model.
new_model = Model(inputs=top_model.input, outputs=multi_input_base_model(top_model.output))
I am using the new tf.estimator.WarmStartSettings to initialize my network from a previous checkpoint. I now want to run the same network on a new data source, with other vocabs to use for the embeddings.
This snippet from the documentation page of WarmStartSettings seems to describe my use case:
Warm-start all weights but the embedding parameters corresponding to
sc_vocab_file have a different vocab from the one used in the current
model:
vocab_info = ws_util.VocabInfo(
new_vocab=sc_vocab_file.vocabulary_file,
new_vocab_size=sc_vocab_file.vocabulary_size,
num_oov_buckets=sc_vocab_file.num_oov_buckets,
old_vocab="old_vocab.txt"
)
ws = WarmStartSettings(
ckpt_to_initialize_from="/tmp",
var_name_to_vocab_info={
"input_layer/sc_vocab_file_embedding/embedding_weights": vocab_info
})
tf.estimator.VocabInfo allows to specify the old and new vocab with their respective sizes. However, when I try to use the WarmStartSettings as shown above with 2 vocabs of different sizes, I get the following error:
ValueError: Shape of variable input_layer/sc_vocab_file_embedding/embedding_weights
((1887, 30)) doesn't match with shape of tensor
input_layer/sc_vocab_file_embedding/embedding_weights ([537, 30]) from checkpoint reader.
Why does VocabInfo allow to provide separate sizes for the vocabs if their size has to match anyway?
I have an interesting question.
I am using Pipeline object to run a ML task.
This is how my Pipeline object looks like.
jpsa_mlp.pipeline.getStages()
Out[244]:
[StringIndexer_479d82259c10308d0587,
Tokenizer_4c5ca5ea35544bb835cb,
StopWordsRemover_4641b68e77f00c8fbb91,
CountVectorizer_468c96c6c714b1000eef,
IDF_465eb809477c6c986ef9,
MultilayerPerceptronClassifier_4a67befe93b015d5bd07]
All the estimators and transformers inside this pipeline object have been coded as part of class methods with JPSA being class object.
Now I want to put a method for hyper parameter tuning. So I use below:
self.paramGrid = ParamGridBuilder()\
.addGrid(self.pipeline.getStages()[5].layers, [len(self.pipeline.getStages()[3].vocab),10,3])\
.addGrid(self.pipeline.getStages()[5].maxIter, [100,300])\
.build()
The problem is for a Neural Network classifier one of the hyper parameter is basically the hidden layer size. The layers attribute of MLP classifier requires the size of input layer, hidden and output layer. Input and Output is fixed (based on data we have). So I wanted to put input layer size as the size of my feature vector. However I don't know the size of my feature vector because the estimator inside the pipeline object to create feature vectors (Count Vectorizer, IDF) have not been fit yet to the data.
The pipeline object will fit the data during cross validation by using a cross validator object of Spark. Then only I would be able to have CountVectorizerModel to know the feature vector size.
If I had Countvectorizer materialized then I can use either the countvectorizerModel.vocab to get the length of the feature vector and use that as a parameter for input layer value in layers attribute of mlp.
SO then how do I add hyper parameters for Layers for mlp (both the hidden and input layer size)?
You can find out that information from your dataframe schema metadata.
Scala code:
val length = datasetAfterPipe.schema(datasetAfterPipe.schema.fieldIndex("columnName"))
.metadata.getMetadata("ml_attr").getLong("num_attrs")
Since is requested PySpark code:
u can se them "navigating" metadata: datasetAfterPipe.schema["features"].metadata["ml_attr"]
here is sample output (xxx is all features made into features columns and the end results is the size):
Out:
{'attrs': {'numeric': [{'idx': xxxxxxx }]}, 'num_attrs': 337}
so u slice metadata:
lenFeatureVect = datasetAfterPipe.schema["features"].metadata["ml_attr"]["num_attrs"]
print('Len feature vector:', lenFeatureVect)
Out:
337
Note: if u have "scaled features" then u need to use "pre-Scaled" column
"features" in order to get attributes info (assuming u scale after vectorizing otherwise is not getting applied limitations if u feed original columns) since u feed feature
vectors to that step into Pipeline.
I want to create a multi-layered bidirectional LSTM in Tensorflow. Currently my single-layered model looks like:
cell_fw = tf.contrib.rnn.LSTMCell(hidden_size)
cell_bw = tf.contrib.rnn.LSTMCell(hidden_size)
(self.out_fw, self.out_bw), _ = tf.nn.bidirectional_dynamic_rnn(cell_fw, cell_bw, input, ...)
In order to turn this into a multi-layered I suspect I can not simply wrap a few LSTMCells with MultiRNNCells like so:
multi_cell_fw = tf.contrib.rnn.MultiRNNCell([cell_fw] * num_layers, ...)
and feed them into the bidirectional_dynamic_rnn since both forward and backward LSTMs in each layer need the output of both the forward and backward directions of the preceding layer. Currently my solution is to create my bidirectional_dynamic_rnns in a loop, feeding in the concatenated output of LSTMs of the preceding layers.
However, it's not very clean and frankly I'm not sure if it's correct, though it does work on a toy dataset. Is there a better way that's comparably elegant to using something like MultiRNNCell?
I'm using Tensorflow API r1.0.
Just do:
multi_cell_fw = tf.contrib.rnn.MultiRNNCell([cell_fw for _ in range(num_layers)], ...)
That should work.