Develop Reference

Eager execution inside lambda layer in Tensorflow

I have Tensorflow 2.9.1 installed at my laptop, and according to the documentation the eager execution should be enabled by default. I have a problem while trying to convert Tensor object to Numpy Array inside a model. I keep getting 'Tensor' object has no attribute 'numpy'
I wanted to have some Lambda Layers inside my model and do some operations using Numpy, but the eager execution seems to be disabled inside the model. I tried to run tf.executing_eagerly() inside model and it returned false. On the otherhand, when I tried to run tf.executing_eagerly() outside the mode, I got true.
Could someone clear my confusion here?
import keras
import tensorflow as tf
from keras import layers, models
import numpy as np
import matplotlib.pyplot as plt
tf.config.run_functions_eagerly(True)
def do_something(input_tensor):
a = add_one(input_tensor.numpy)
b = minus_one(a)
c = tf.convert_to_tensor(b, dtype=tf.float32)
return c
def add_one(input):
return input + 1.0
def minus_one(input):
return input - 1.0
encoding_dim = 32
input_img = layers.Input(shape=(784,))
encoded = layers.Dense(encoding_dim, activation='relu')(input_img)
simulation_layer = layers.Lambda(do_something, name="channel_simulation")(encoded)
decoded = layers.Dense(784, activation='sigmoid')(encoded)
autoencoder = models.Model(input_img, decoded)
encoder = models.Model(input_img, encoded)
encoded_input = layers.Input(shape=(encoding_dim,))
decoder_layer = autoencoder.layers[-1]
decoder = models.Model(encoded_input, decoder_layer(encoded_input))
autoencoder.compile(optimizer='adam', loss='binary_crossentropy', run_eagerly=True)

Incompatible shape sizes using PyGAD

I'm trying to follow the tutorial given here.
This tutorial trains a Keras model using a genetic algorithm, with the PyGAD package. I'm interested in the binary classification case. My input matrix is of dimension 10000x20. Hence, I've created the following model using Keras:
input_layer = tensorflow.keras.layers.Input(20)
dense_layer1 = tensorflow.keras.layers.Dense(500, activation="relu")(input_layer)
dense_layer2 = tensorflow.keras.layers.Dense(500, activation="relu")(dense_layer1)
output_layer = tensorflow.keras.layers.Dense(1, activation="softmax")(dense_layer2)
model = tensorflow.keras.Model(inputs=input_layer, outputs=output_layer)
keras_ga = pygad.kerasga.KerasGA(model=model,
num_solutions=10)
However, when I go to run the algorithm, using ga_instance.run(), I get the error:
ValueError: Shapes (10000,) and (10000, 1) are incompatible
I can't figure out why I'm getting this error? I want my Keras model to have 2 hidden layers, each with 500 hidden nodes and 1 output node.

I think the problem is related to how each output is represented in the array. if you have a single output for 10000 instances, then this is an example of preparing the data that works with PyGAD. Its shape is (1000, 1).
numpy.random.uniform(0, 1, (1000, 1))
Here is a code that works but for a simple network architecture because, based on the fitness function you used, the fitness sometimes is NaN.
As I do not have the same data you used, I generated the input/output data randomly.
import tensorflow.keras
import pygad.kerasga
import numpy
import pygad
def fitness_func(solution, sol_idx):
global data_inputs, data_outputs, keras_ga, model
model_weights_matrix = pygad.kerasga.model_weights_as_matrix(model=model,
weights_vector=solution)
model.set_weights(weights=model_weights_matrix)
predictions = model.predict(data_inputs)
cce = tensorflow.keras.losses.CategoricalCrossentropy()
solution_fitness = 1.0 / (cce(data_outputs, predictions).numpy() + 0.00000001)
# print("solution_fitness", cce(data_outputs, predictions).numpy(), solution_fitness)
return solution_fitness
def callback_generation(ga_instance):
print("Generation = {generation}".format(generation=ga_instance.generations_completed))
print("Fitness = {fitness}".format(fitness=ga_instance.best_solution(ga_instance.last_generation_fitness)[1]))
data_inputs = numpy.random.uniform(0, 1, (1000, 20))
data_outputs = numpy.random.uniform(0, 1, (1000, 1))
# create model
from tensorflow.keras.layers import Dense, Dropout
l1_rate=1e-6
l2_rate = 1e-6
input_layer = tensorflow.keras.layers.InputLayer(20)
dense_layer1 = tensorflow.keras.layers.Dense(10, activation="relu",kernel_regularizer=tensorflow.keras.regularizers.l1_l2(l1=l1_rate, l2=l2_rate))
output_layer = tensorflow.keras.layers.Dense(1, activation="sigmoid")
model = tensorflow.keras.Sequential()
model.add(input_layer)
model.add(dense_layer1)
model.add(Dropout(0.2))
model.add(output_layer)
keras_ga = pygad.kerasga.KerasGA(model=model,
num_solutions=10)
# Run pygad
num_generations = 30
num_parents_mating = 5
initial_population = keras_ga.population_weights
ga_instance = pygad.GA(num_generations=num_generations,
num_parents_mating=num_parents_mating,
initial_population=initial_population,
fitness_func=fitness_func,
on_generation=callback_generation)
ga_instance.run()
Thanks for using PyGAD!

Tensorflow Hub shape not defined

I am trying to load in a pre-trained embedding model but no matter what shape I give, I cannot find the input shape. It seems like a pretty popular choice, but I cannot find any indicator on the tensorflow hub page on what input shape to use. The input sequence is supposed to have variable-length, so I use an input shape of none. Keras automatically provides the batch size
embedding_url = 'https://tfhub.dev/google/universal-sentence-encoder-large/5'
embedding_layer = hub.KerasLayer(embedding_url)
premises = tf.keras.Input(shape=(None,))
conclusion = tf.keras.Input(shape=(None,))
x1 = embedding_layer(premises)
x2 = embedding_layer(conclusion)
model = tf.keras.Model(inputs=[premises, conclusion], outputs=[x1, x2])
Here is the error I get
ValueError: Python inputs incompatible with input_signature:
inputs: (
Tensor("input_5:0", shape=(None, None), dtype=float32))
input_signature: (
TensorSpec(shape=<unknown>, dtype=tf.string, name=None))

You can use Functional API with KerasLayer by keeping the input shape parameter as an empty tuple.
Code:
import tensorflow_hub as hub
import tensorflow as tf
embedding_url = 'https://tfhub.dev/google/universal-sentence-encoder-large/5'
premises = tf.keras.layers.Input(shape=(), name="Input1", dtype=tf.string)
conclusion = tf.keras.layers.Input(shape=(), name="Input2", dtype=tf.string)
embedding_layer = hub.KerasLayer(embedding_url)
x1 = embedding_layer(premises)
x2 = embedding_layer(conclusion)
model = tf.keras.Model(inputs=[premises, conclusion], outputs=[x1, x2])
tf.keras.utils.plot_model(model, 'my_first_model.png', show_shapes=True)
Your model looks like this:

Keras (Tensorflow) - name array_ops not defined

I'm having an issue with Keras/Tensorflow deserializing a model. Basically this is an implementation of a convolutional neural network on text, which requires a dimension to be added at an early stage. The error message is this:
File
"/usr/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/backend.py",
line 2231, in expand_dims NameError: name 'array_ops' is not defined
The code causing this error message:
import numpy as np
from docopt import docopt
import tensorflow as tf
from vdcnn import utils
if __name__ == '__main__':
arguments = docopt(__doc__, version='1.0')
model = tf.keras.models.load_model(arguments["--checkpoint"])
print(type(model))
proc = utils.Preprocessor(padding_size=256)
data, labels, test_data, test_labels = proc.process_document(arguments["--data"])
for i in range(len(test_data)):
test_vec = test_data[i]
prediction = model.predict(x=test_vec[np.newaxis])
predlabel = utils.labels_in_order[np.argmax(prediction)]
truthlabel = utils.labels_in_order[np.argmax(test_labels[i])]
print("Truth: {} \t Predicted: {}".format(truthlabel, predlabel)
The code that calls "expand_dims" uses a Keras Lambda wrapper around the Tensorflow function:
...
inputs = tf.keras.Input(shape=(self.sequence_max_length,), dtype='int32', name='inputs')
embedding = tf.keras.layers.Embedding(self.num_quantized_chars, self.embedding_size, input_length=self.sequence_max_length)(inputs)
embedding = tf.keras.layers.Lambda(tf.expand_dims, arguments={'axis' : -1, 'name' : 'embedding_expanded'})(embedding)
conv0 = tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=[1, self.embedding_size], padding='same', kernel_initializer='he_normal')(embedding)
conv0 = tf.keras.layers.Activation('relu')(conv0)
...
And, just for kicks, the line it's referencing in the tensorflow libs:
from tensorflow.python.ops import array_ops
[two thousand lines of crap]
def expand_dims(x, axis=-1):
"""Adds a 1-sized dimension at index "axis".
Arguments:
x: A tensor or variable.
axis: Position where to add a new axis.
Returns:
A tensor with expanded dimensions.
"""
return array_ops.expand_dims(x, axis)
I'm using Python 3.6 and Tensorflow 1.5, and this error occurs on both OSX 10.11.6 and RHEL 7. I've tried using various permutations of tf.keras, tf.keras.backend, just keras without tf, and all of it basically calls the exact same code, although sometimes it complains about "gen_array_ops" instead of "array_ops" with the same problem.
Anyone have any thoughts?

The issue was this: https://github.com/keras-team/keras/issues/8123#issuecomment-354857044
On top of that, it required reinstalling everything on all machines and using keras directly instead of tf.keras to get the proper error message, apparently because of how Keras uses object serialization and the way Python "tracebacks" work.

Add Tensorflow pre-processing to existing Keras model (for use in Tensorflow Serving)

I would like to include my custom pre-processing logic in my exported Keras model for use in Tensorflow Serving.
My pre-processing performs string tokenization and uses an external dictionary to convert each token to an index for input to the Embedding layer:
from keras.preprocessing import sequence
token_to_idx_dict = ... #read from file
# Custom Pythonic pre-processing steps on input_data
tokens = [tokenize(s) for s in input_data]
token_idxs = [[token_to_idx_dict[t] for t in ts] for ts in tokens]
tokens_padded = sequence.pad_sequences(token_idxs, maxlen=maxlen)
Model architecture and training:
model = Sequential()
model.add(Embedding(max_features, 128, input_length=maxlen))
model.add(LSTM(128, activation='sigmoid'))
model.add(Dense(n_classes, activation='softmax'))
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
model.fit(x_train, y_train)
Since the model will be used in Tensorflow Serving, I want to incorporate all pre-processing logic into the model itself (encoded in the exported model file).
Q: How can I do so using the Keras library only?
I found this guide explains how to combine Keras and Tensorflow. But I'm still unsure how to export everything as one model.
I know Tensorflow has built-in string splitting, file I/O, and dictionary lookup operations.
Pre-processing logic using Tensorflow operations:
# Get input text
input_string_tensor = tf.placeholder(tf.string, shape={1})
# Split input text by whitespace
splitted_string = tf.string_split(input_string_tensor, " ")
# Read index lookup dictionary
token_to_idx_dict = tf.contrib.lookup.HashTable(tf.contrib.lookup.TextFileInitializer("vocab.txt", tf.string, 0, tf.int64, 1, delimiter=","), -1)
# Convert tokens to indexes
token_idxs = token_to_idx_dict.lookup(splitted_string)
# Pad zeros to fixed length
token_idxs_padded = tf.pad(token_idxs, ...)
Q: How can I use these Tensorflow pre-defined pre-processing operations and my Keras layers together to both train and then export the model as a "black box" for use in Tensorflow Serving?

I figured it out, so I'm going to answer my own question here.
Here's the gist:
First, (in separate code file) I trained the model using Keras only with my own pre-processing functions, exported the Keras model weights file and my token-to-index dictionary.
Then, I copied just the Keras model architecture, set the input as the pre-processed tensor output, loaded the weights file from the previously trained Keras model, and sandwiched it between the Tensorflow pre-processing operations and the Tensorflow exporter.
Final product:
import tensorflow as tf
from keras import backend as K
from keras.models import Sequential, Embedding, LSTM, Dense
from tensorflow.contrib.session_bundle import exporter
from tensorflow.contrib.lookup import HashTable, TextFileInitializer
# Initialize Keras with Tensorflow session
sess = tf.Session()
K.set_session(sess)
# Token to index lookup dictionary
token_to_idx_path = '...'
token_to_idx_dict = HashTable(TextFileInitializer(token_to_idx_path, tf.string, 0, tf.int64, 1, delimiter='\t'), 0)
maxlen = ...
# Pre-processing sub-graph using Tensorflow operations
input = tf.placeholder(tf.string, name='input')
sparse_tokenized_input = tf.string_split(input)
tokenized_input = tf.sparse_tensor_to_dense(sparse_tokenized_input, default_value='')
token_idxs = token_to_idx_dict.lookup(tokenized_input)
token_idxs_padded = tf.pad(token_idxs, [[0,0],[0,maxlen]])
token_idxs_embedding = tf.slice(token_idxs_padded, [0,0], [-1,maxlen])
# Initialize Keras model
model = Sequential()
e = Embedding(max_features, 128, input_length=maxlen)
e.set_input(token_idxs_embedding)
model.add(e)
model.add(LSTM(128, activation='sigmoid'))
model.add(Dense(num_classes, activation='softmax'))
# Load weights from previously trained Keras model
weights_path = '...'
model.load_weights(weights_path)
K.set_learning_phase(0)
# Export model in Tensorflow format
# (Official tutorial: https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/serving_basic.md)
saver = tf.train.Saver(sharded=True)
model_exporter = exporter.Exporter(saver)
signature = exporter.classification_signature(input_tensor=model.input, scores_tensor=model.output)
model_exporter.init(sess.graph.as_graph_def(), default_graph_signature=signature)
model_dir = '...'
model_version = 1
model_exporter.export(model_dir, tf.constant(model_version), sess)
# Input example
with sess.as_default():
token_to_idx_dict.init.run()
sess.run(model.output, feed_dict={input: ["this is a raw input example"]})

The accepted answer is super helpful, however it uses an outdated Keras API as #Qululu mentioned, and an outdated TF Serving API (Exporter), and it does not show how to export the model so that its input is the original tf placeholder (versus Keras model.input, which is post preprocessing). Following is a version that works well as of TF v1.4 and Keras 2.1.2:
sess = tf.Session()
K.set_session(sess)
K._LEARNING_PHASE = tf.constant(0)
K.set_learning_phase(0)
max_features = 5000
max_lens = 500
dict_table = tf.contrib.lookup.HashTable(tf.contrib.lookup.TextFileInitializer("vocab.txt",tf.string, 0, tf.int64, TextFileIndex.LINE_NUMBER, vocab_size=max_features, delimiter=" "), 0)
x_input = tf.placeholder(tf.string, name='x_input', shape=(None,))
sparse_tokenized_input = tf.string_split(x_input)
tokenized_input = tf.sparse_tensor_to_dense(sparse_tokenized_input, default_value='')
token_idxs = dict_table.lookup(tokenized_input)
token_idxs_padded = tf.pad(token_idxs, [[0,0],[0, max_lens]])
token_idxs_embedding = tf.slice(token_idxs_padded, [0,0], [-1, max_lens])
model = Sequential()
model.add(InputLayer(input_tensor=token_idxs_embedding, input_shape=(None, max_lens)))
...REST OF MODEL...
model.load_weights("model.h5")
x_info = tf.saved_model.utils.build_tensor_info(x_input)
y_info = tf.saved_model.utils.build_tensor_info(model.output)
prediction_signature = tf.saved_model.signature_def_utils.build_signature_def(inputs={"text": x_info}, outputs={"prediction":y_info}, method_name=tf.saved_model.signature_constants.PREDICT_METHOD_NAME)
builder = saved_model_builder.SavedModelBuilder("/path/to/model")
legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')
init_op = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
sess.run(init_op)
# Add the meta_graph and the variables to the builder
builder.add_meta_graph_and_variables(
sess, [tag_constants.SERVING],
signature_def_map={
signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
prediction_signature,
},
legacy_init_op=legacy_init_op)
builder.save()
UPDATE Doing pre-processing for inference with Tensorflow is a CPU op, and is not carried out efficiently if the model is deployed on a GPU server. The GPU stalls really bad, and the throughput is very low. Therefore, we ditched this for efficient pre-processing in the client process, instead.