Develop Reference

Keras_tuner nonsense ValueError: Unknown initializer: relu

I am running a hyperparameter search on a new version keras tuner for a NN, and I get an error that 1) didn't exist in the old one, 2) doesn't make sense.
ValueError: Unknown initializer: relu. Please ensure this object is passed to the custom_objects argument.
I don't get why 'relu' is passed as initializer, the code breaks at the definition of learning rate (as per keras tuner documentation):
hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=hp_learning_rate),
loss=self.loss_function,
metrics=['sparse_categorical_accuracy', 'accuracy'])
However, it works with
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
loss=self.loss_function,
metrics=['sparse_categorical_accuracy', 'accuracy'])
I am not specifying any weight initializers, so they all should be default ones.
Any idea why this could be happening?
thank you very much in advance!

How do i specify to a model what to take as input of a custom loss function?

I'm having issues in understanding/implementing a custom loss function in my model.
I have a keras model which is composed by 3 sub models as you can see here in the model architecture,
Now, I'd like to use the outputs of model and model_2 in my custom loss function.
I understand that in the loss function definition I can write:
def custom_mse(y_true, y_pred):
*calculate stuff*
return loss
But how do I tell the model to take its 2 outputs as inputs of the loss function?
Maybe, and i hope so, it's super trivial but I didn't find anything online, if you could help me it'd be fantastic.
Thanks in advance
Context:
model and model_2 are the same pretrained model, a binary classifier, which predicts the interaction between 2 inputs (of image-like type).
model_1 is a generative model which will edit one of the inputs.
Therefore:
complete_model = Model(inputs=[input_1, input_2], outputs=[out_model, out_model2])
opt = *an optimizer*
complete_model.compile(loss=custom_mse,
??????,
optimizer = opt,
metrics=['whatever'])
The main goal is to compare the prediction with the edited input against the one with the un-edited input, therefore the model will outputs the 2 interactions, which i need to use in the loss function.
EDIT:
Thank you Andrey for the solution,
Now however i can't manage to implement together the 2 loss functions, namely the one with add_loss(func) and a classic binary_crossentropy in model.complie(loss='binary_crossentropy', ...).
Can I maybe add an add_loss specifying model_2.output and the label? If yes do you know how?
They work by themselves but not together, when i try to run the code they raise
ValueError: Shapes must be equal rank, but are 0 and 4 From merging shape 0 with other shapes. for '{{node AddN}} = AddN[N=2, T=DT_FLOAT](binary_crossentropy/weighted_loss/value, complete_model/generator/tf_op_layer_SquaredDifference_3/SquaredDifference_3)' with input shapes: [], [?,500,400,1].

You can add loss with compile() only for standard loss function signature (y_true, y_pred). You can not use it because your signature is something like (y_true, (y_pred1, y_pred2)). Use add_loss() API instead. See here: https://keras.io/api/losses/

TFX Evaluator problem with a keras multioutput model

I am using a very simple keras model in TFX, to solve a regression problem.
It seems that TFX wants that you use a keras model with named outputs, so I made:
output = {key: tf.keras.layers.Dense(1, name = key)(x)
for key in _transformed_names(_LABEL_KEYS)}
model(inputs, outputs)
I don't understand how the evaluator maps the label names of my dataset with the output names of my model.
In my code I set up the label_keys and prediction_keys argument in tfma.ModelSpec with a list of form :
[["model output name", "Label key in my Dataset"]]
It seems that the proto message is created correctly, but when I run the Evaluator I get the following error:
ValueError: unable to prepare labels and predictions because the labels and/or predictions are dicts with unrecognized keys. If a multi-output keras model (or estimator) was used check that an output_name was provided. If an estimator was used check that common prediction keys were provided (e.g. logistic, probabilities, etc)
If I try to provide a single label key and a single prediction key using the label_key and prediction_key arguments I get the following error:
TypeError: update_state() takes from 2 to 3 positional arguments but 4 were given [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/ComputePerSlice/ComputeUnsampledMetrics/CombinePerSliceKey/WindowIntoDiscarding']
I have tried in all possible ways but nothing.
Is there a way to use a model with no named outputs (an Dense output layer with more than one node) ? Or a way to solve this problem ?
P.S. Is there a Tutorials of a TFX pipeline with a multiple outputs keras model ?
Thanks.

in eval_config,setting
options=Options(include_default_metrics=BoolValue(value=False))
e.g:
eval_config = tfma.EvalConfig(
model_specs = [...],
slicing_specs=[tfma.SlicingSpec(),...],
metrics_specs=[...],
options=Options(include_default_metrics=BoolValue(value=False))
)
evaluator = Evaluator(
...
eval_config=eval_config
)

How to use layer normalization in tensorflow 1.12?

I am stuck with tensorflow 1.12, and I need to use layer normalization. I can't find some examples of this, and as I am new to tensorflow I am unable to figure out where I am going wrong.
tf.contrib.layers.layer_norm is the function that I want to include in my tf.keras.Sequential() like this -
self.module = K.Sequential([
tf.contrib.layers.layer_norm(trainable=True),
K.layers.Activation(self.activation),
K.layers.Dense(units=self.output_size, activation=None, kernel_initializer=self.initializer)
])
I also tried using
self.ln = tf.contrib.layers.layer_norm(trainable=True)
### and in call()
self.ln(self.module)
In all the cases, it throws the error at the line defining tf.contrib.layers.layer_norm(trainable=True)-
TypeError: layer_norm() missing 1 required positional argument: 'inputs'
I understand that the inputs need to be given as the argument to layernorm, but if I want it to trainable, it can only be defined in __init__(). Where am I going wrong?
I use mainly PyTorch, so it is quite obvious that I am not able to grasp the ideology of tf. Any suggestions will be very helpful!

Sequential needs to be initialized by a list of Layer instances, such as tf.keras.layers.Activation, tf.keras.layers.Dense. tf.contrib.layers.layer_norm is functional instead of Layer instance.
There is a third party implementation of layer normalization in keras style - keras-layer-normalization. But I haven't tested in tensorflow.

Keras Embedding ,where is the "weights" argument?

I have seen such kind of code as follow:
embed_word = Embedding(params['word_voc_size'], params['embed_dim'], weights=[word_embed_matrix], input_length = params['word_max_size']
, trainable=False, mask_zero=True)
When I look up the document in Keras website [https://faroit.github.io/keras-docs/2.1.5/layers/embeddings/][1]
I didnt see weights argument,
keras.layers.Embedding(input_dim, output_dim, embeddings_initializer='uniform', embeddings_regularizer=None, activity_regularizer=None, embeddings_constraint=None, mask_zero=False, input_length=None)
So I am confused,why we can use the argument weights which was not defined the in Keras document?
My keras version is 2.1.5. Hope someone can help me.

Keras' Embedding layer subclasses the Layer class (every Keras layer does this). The weights attribute is implemented in this base class, so every subclass will allow to set this attribute through a weights argument. This is also why you won't find it back in the documentation or the implementation of the Embedding layer itself.
You can check the base layer implementation here (Ctrl + F for 'weight').