Develop Reference

Layer concatenate was called with an input that isn't a symbolic tensor

I have two models (model_a,model_b) of similar structure (VGG16 architecture with a replaced top block). I need to concatenate the outputs of the last layers of both models in order to send as input to an attention mechanism.
I run the following line of code for concatenation:
merged = Concatenate()([model_a.layers[-1].layers[-1], model_b.layers[-1].layers[-1]])
(model_a.layers[-1] is the top block which is a Sequential object, model_a.layers[-1].layers[-1] is a Dense layer.)
However, I receive the following error when I try to do so:
Layer concatenate_8 was called with an input that isn't a symbolic
tensor. Received type: < class 'keras.layers.core.Dense' >. Full input:
[< keras.layers.core.Dense object at 0x >,
< keras.layers.core.Dense object at 0x >]. All inputs to the
layer should be tensors.
I noticed that similar issues are fixed by redefining the last layer by specifying the input layer for it, but I'm not sure how that solution would help here since I'm using predefined and pre-trained models.

Use the .output attribute of the model to access the symbolic tensor.
merged = Concatenate()(
[model_a.layers[-1].layers[-1].output, model_b.layers[-1].layers[-1].output])
The tensorflow.keras.layers.Layer documentation states the following about Layer.output
output: Retrieves the output tensor(s) of a layer.
Only applicable if the layer has exactly one output, i.e. if it is connected to one incoming layer.
Regarding the comment
Actually, when I tried to finally combine inputs and outputs using the Model() method, I had an error - Graph disconnected: Cannot get value of tensor Tensor Flatten.. , however when I changed model_a.layers[-1].layers[-1].output to model_a.output and similarly for model_b, the issue was resolved. Any idea why so?
It's difficult to say without seeing the model code, but you can compare the values of model_a.layers[-1].layers[-1] to model_a.output.

Keras Lambda Layer Before Embedding: Use to Convert Text to Integers

I currently have a keras model which uses an Embedding layer. Something like this:
input = tf.keras.layers.Input(shape=(20,) dtype='int32')
x = tf.keras.layers.Embedding(input_dim=1000,
output_dim=50,
input_length=20,
trainable=True,
embeddings_initializer='glorot_uniform',
mask_zero=False)(input)
This is great and works as expected. However, I want to be able to send text to my model, have it preprocess the text into integers, and continue normally.
Two issues:
1) The Keras docs say that Embedding layers can only be used as the first layer in a model: https://keras.io/layers/embeddings/
2) Even if I could add a Lambda layer before the Embedding, I'd need it to keep track of certain state (like a dictionary mapping specific words to integers). How might I go about this stateful preprocessing?
In short, I need to modify the underlying Tensorflow DAG, so when I save my model and upload to ML Engine, it'll be able to handle my sending it raw text.
Thanks!

Here are the first few layers of a model which uses a string input:
input = keras.layers.Input(shape=(1,), dtype="string", name='input_1')
lookup_table_op = tf.contrib.lookup.index_table_from_tensor(
mapping=vocab_list,
num_oov_buckets=num_oov_buckets,
default_value=-1,
)
lambda_output = Lambda(lookup_table_op.lookup)(input)
emb_layer = Embedding(int(number_of_categories),int(number_of_categories**0.25))(lambda_output)
Then you can continue the model as you normally would after an embedding layer. This is working for me and the model trains fine from string inputs.
It is recommended that you do the string -> int conversion in some preprocessing step to speed up the training process. Then after the model is trained you create a second keras model that just converts string -> int and then combine the two models to get the full string -> target model.

CNTK with python - activation for each layer

I am using the python API of CNTK to train some CNN that I save using the save_model function.
Now I want to run some analysis on my network afterwards. Specifically I want to take a look at the activations of each layer. Obviously I can run my network on some data called img like this:
model.eval(img)
But that will only give me the output of the last Layer in my Network. Is there some easy way to also get the output from the previous layers?

Actually, there is even an example provided for that task: https://github.com/Microsoft/CNTK/tree/master/Examples/Image/FeatureExtraction
Let me give you a short overview about the essential steps:
Important is the name of your node, of which you want to get the output.
# get the node in the graph of which you desire the output
node_in_graph = loaded_model.find_by_name(node_name)
output_nodes = combine([node_in_graph.owner])
# evaluate the node e.g. using a minibatch_source
mb = minibatch_source.next_minibatch(1)
output = output_nodes.eval(mb[features_si])
# access the values as a one dimensional vector
out_values = output[0].flatten()
desired_output = out_values[np.newaxis]
Basically you just do the same like you do anyways with the difference that you retrieve an intermediate node.

CNTK & python: How to pass input data to the eval func?

With CNTK I have created a network with 2 input neurons and 1 output neuron.
A line in the training file looks like
|features 1.567518 2.609619 |labels 1.000000
Then the network was trained with brain script. Now I want to use the network for predicting values. For example: Input data is [1.82, 3.57]. What ist the output from the net?
I have tried Python with the following code, but here I am new. Code does not work. So my question is: How to pass the input data [1.82, 3.57] to the eval function?
On stackoverflow there are some hints, here and here, but this is too abstract for me.
Thank you.
import cntk as ct
import numpy as np
z = ct.load_model("LR_reg.dnn", ct.device.cpu())
input_data= np.array([1.82, 3.57], dtype=np.float32)
pred = z.eval({ z.arguments[0] : input_data })
print(pred)

Here's the most defensive way of doing it. CNTK can be forgiving if you omit some of this when the network is specified with V2 constructs. Not sure about a network that was created with V1 code.
Basically you need a pair of braces for each axis. Which axes exist in Brainscript? There's a batch axis, a sequence axis and then the static axes of your network. You have one dimensional data so that means the following should work:
input_data= np.array([[[1.82, 3.57]]], dtype=np.float32)
This specifies a batch of one sequence, of length one, containing one 1d vector of two elements. You can also try omitting the outermost braces and see if you are getting the same result.
Update based on more information from the comment below, we should not forget that the V1 code also saved the part of the network that computes things like loss and accuracy. If we provide only the features, CNTK will complain that the labels have not been provided. There are two ways to deal with this issue. One possibility is to provide some fake labels, so that the network can evaluate these auxiliary operations. Another possibility is to identify the prediction and use that. If the prediction was called 'p' in V1, this python code
p = z.find_by_name('p')
should create a CNTK function that only needs the features in order to compute the prediction.

How to get feature vector column length in Spark Pipeline

I have an interesting question.
I am using Pipeline object to run a ML task.
This is how my Pipeline object looks like.
jpsa_mlp.pipeline.getStages()
Out[244]:
[StringIndexer_479d82259c10308d0587,
Tokenizer_4c5ca5ea35544bb835cb,
StopWordsRemover_4641b68e77f00c8fbb91,
CountVectorizer_468c96c6c714b1000eef,
IDF_465eb809477c6c986ef9,
MultilayerPerceptronClassifier_4a67befe93b015d5bd07]
All the estimators and transformers inside this pipeline object have been coded as part of class methods with JPSA being class object.
Now I want to put a method for hyper parameter tuning. So I use below:
self.paramGrid = ParamGridBuilder()\
.addGrid(self.pipeline.getStages()[5].layers, [len(self.pipeline.getStages()[3].vocab),10,3])\
.addGrid(self.pipeline.getStages()[5].maxIter, [100,300])\
.build()
The problem is for a Neural Network classifier one of the hyper parameter is basically the hidden layer size. The layers attribute of MLP classifier requires the size of input layer, hidden and output layer. Input and Output is fixed (based on data we have). So I wanted to put input layer size as the size of my feature vector. However I don't know the size of my feature vector because the estimator inside the pipeline object to create feature vectors (Count Vectorizer, IDF) have not been fit yet to the data.
The pipeline object will fit the data during cross validation by using a cross validator object of Spark. Then only I would be able to have CountVectorizerModel to know the feature vector size.
If I had Countvectorizer materialized then I can use either the countvectorizerModel.vocab to get the length of the feature vector and use that as a parameter for input layer value in layers attribute of mlp.
SO then how do I add hyper parameters for Layers for mlp (both the hidden and input layer size)?

You can find out that information from your dataframe schema metadata.
Scala code:
val length = datasetAfterPipe.schema(datasetAfterPipe.schema.fieldIndex("columnName"))
.metadata.getMetadata("ml_attr").getLong("num_attrs")

Since is requested PySpark code:
u can se them "navigating" metadata: datasetAfterPipe.schema["features"].metadata["ml_attr"]
here is sample output (xxx is all features made into features columns and the end results is the size):
Out:
{'attrs': {'numeric': [{'idx': xxxxxxx }]}, 'num_attrs': 337}
so u slice metadata:
lenFeatureVect = datasetAfterPipe.schema["features"].metadata["ml_attr"]["num_attrs"]
print('Len feature vector:', lenFeatureVect)
Out:
337
Note: if u have "scaled features" then u need to use "pre-Scaled" column
"features" in order to get attributes info (assuming u scale after vectorizing otherwise is not getting applied limitations if u feed original columns) since u feed feature
vectors to that step into Pipeline.