Develop Reference

Transfer Learning Trainable Model Throws Errors On saving

I have downloaded strong texta pretrained model, and im trying to transfer learn it.
therefore I'm loading the model which is saved as a 'xray_model.h5' file, and set it as untrainable:
model = tf.keras.models.load_model('xray_model.h5')
model.trainable = False
later I take the start layer and end layer and build my addings on it:
base_input = model.layers[0].input
base_output = model.get_layer(name="flatten").output
base_output = build_model()(base_output)
new_model = keras.Model(inputs=base_input, outputs=base_output)
since I want to train my layers (and after some games, I realized that I might need to train the old layers too) I want to set the model as trainable:
for i in range(len(new_model.layers)):
new_model._layers[i].trainable = True
BUT, when I start training it, with the callback:
METRICS = ['accuracy',
tf.keras.metrics.Precision(name='precision'),
tf.keras.metrics.Recall(name='recall'),
lr_metric]
reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=2, min_lr=0.00001, verbose=1)
save_callback = tf.keras.callbacks.ModelCheckpoint("new_xray_model.h5",
save_best_only=True,
monitor='accuracy')
history = new_model.fit(train_generator,
verbose=1,
steps_per_epoch=BATCH_SIZE,
epochs=EPOCHS,
validation_data=test_generator,
callbacks=[save_callback, reduce_lr])
I get the next error:
File "C:\Users\jm10o\AppData\Local\Programs\Python\Python38\lib\site-packages\h5py\_hl\group.py", line 373, in __setitem__
h5o.link(obj.id, self.id, name, lcpl=lcpl, lapl=self._lapl)
File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5o.pyx", line 202, in h5py.h5o.link
OSError: Unable to create link (name already exists)
Process finished with exit code 1
I noticed that it happens only when I'm trying to further train the model which I loaded.
I couldn't find any solution for it.

The problem came from the Model_checkpoint callback. for each epoch, you save the model with the same name.
use the following format
ModelCheckpoint('your_model_name{epoch:0d}.h5',
monitor='accuracy')

Tensorflow TypeError: Can not convert a NoneType into a Tensor or Operation

I am learning TensorFlow and was going through this step-by-step guide. The below code is the exact same as on the website. However, when running it, I get an error when trying to fit the model. The full traceback I get is as follows:
Traceback (most recent call last):
File "C:\users\name\desktop\python ml tutorial\embedding.py", line 49, in <module>
model.fit(x=padded_docs, y=labels, epochs=50, verbose=0)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\training.py", line 1213, in fit
self._make_train_function()
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\training.py", line 316, in _make_train_function
loss=self.total_loss)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 506, in get_updates
return [self.apply_gradients(grads_and_vars)]
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 441, in apply_gradients
kwargs={"name": name})
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\distribute\distribute_lib.py", line 1917, in merge_call
return self._merge_call(merge_fn, args, kwargs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\distribute\distribute_lib.py", line 1924, in _merge_call
return merge_fn(self._strategy, *args, **kwargs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\optimizer_v2\optimizer_v2.py", line 494, in _distributed_apply
with ops.control_dependencies(update_ops):
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 5257, in control_dependencies
return get_default_graph().control_dependencies(control_inputs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\func_graph.py", line 356, in control_dependencies
return super(FuncGraph, self).control_dependencies(filtered_control_inputs)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 4691, in control_dependencies
c = self.as_graph_element(c)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3610, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Users\name\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3699, in _as_graph_element_locked
(type(obj).__name__, types_str))
TypeError: Can not convert a NoneType into a Tensor or Operation.
And the full code is below:
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define document
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([1,1,1,1,1,0,0,0,0,0])
# integer-encode the documents
vocab_size = 50
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print(encoded_docs)
# padding
max_length = 4
padded_docs = pad_sequences(encoded_docs, maxlen = max_length, padding = 'post')
print(padded_docs)
# define model
model = Sequential()
model.add(Embedding(vocab_size, 8, input_length=max_length))
model.add(Flatten())
model.add(Dense(1, activation='sigmoid'))
# compile the model
model.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics=['accuracy'])
# summarize
print(model.summary())
# fit the model
model.fit(x=padded_docs, y=labels, epochs=50, verbose=0)
# evaluate the model
loss, accuracy = model.evaluate(x=padded_docs, y=labels, verbose=0)
print("Accuracy: {}".format(accuracy))
What's going on here? The article was originally written back in 2017 but its last revision and update was just a week ago. I imagine they are constantly tweaking TensorFlow since it is still state-of-the-art and needs a lot of improvement.
Any ideas on how to circumvent this?
Edit:
I began trying to figure out where the script could have gone wrong. I will be listing what I found here, hopefully it will help us spot something:
I found that in ops.py's control_dependencies() function, control_inputs parameter has the following values: Tensor("Adam/gradients/gradients/loss/dense_1_loss/binary_crossentropy/logistic_loss_grad/Reshape_1:0", shape=(None, 1), dtype=float32), Tensor("Adam/gradients/gradients/loss/dense_1_loss/binary_crossentropy/logistic_loss/Log1p_grad/mul:0", shape=(None, 1), dtype=float32), and None. When it becomes None, the program crashes.

ValueError: Passing a dictionary input to a Sequential Model which doesn't have FeatureLayer as the first layer is an error

I've tried running the following code, but got this error:
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py",
line 819, in fit
use_multiprocessing=use_multiprocessing)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py",
line 235, in fit
use_multiprocessing=use_multiprocessing)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py",
line 593, in _process_training_inputs
use_multiprocessing=use_multiprocessing)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py",
line 706, in _process_inputs
use_multiprocessing=use_multiprocessing)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py",
line 702, in init
x = standardize_function(x)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py",
line 660, in standardize_function
standardize(dataset, extract_tensors_from_dataset=False)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py",
line 2346, in _standardize_user_data
all_inputs, y_input, dict_inputs = self._build_model_with_inputs(x, y)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py",
line 2572, in _build_model_with_inputs
self._set_inputs(cast_inputs)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py",
line 2647, in _set_inputs
inputs = self._set_input_attrs(inputs)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\training\tracking\base.py",
line 457, in _method_wrapper
result = method(self, *args, **kwargs)
File
"C:\Users\TomerK\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\keras\engine\training.py",
line 2681, in _set_input_attrs
raise ValueError('Passing a dictionary input to a Sequential Model '
ValueError: Passing a dictionary input to a Sequential Model which
doesn't have FeatureLayer as the first layer is an error.
Code:
# -*- coding: utf-8 -*-
import os
#os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import tensorflow_datasets as tfds
try:
model = keras.models.load_model("passrockmodel.h5")
except:
print('\nDownloading Train Dataset...\n')
train_dataset = tfds.load(name="rock_you", split="train[:75%]")
assert isinstance(train_dataset, tf.data.Dataset)
print('\nDownloading Test Dataset...\n')
test_dataset = tfds.load("rock_you", split='train[-25%:]')
assert isinstance(test_dataset, tf.data.Dataset)
model = tf.keras.Sequential([
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid'),
])
model.compile(
loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model.fit(train_dataset, epochs=20)
model.save("passrockmodel.h5")
test_loss, test_accuracy = model.evaluate(test_dataset)
print('\nPredicting...\n')
predictions = model.predict(test_dataset)
print(predictions[0])

I've had your problem yesterday. Here's what solved it for me:
your first layer should be of type tf.keras.layers.DenseFeatures
This first layer must be instantiated with an array of tf.feature_column objects. It happens that all my columns were numeric so my array was:
featureColumns = [tf.feature_column.numeric_column(columnNames[i], normalizer_fn= lambda x: (x - mean[i])/std[i]) for i in range(len(columnNames[:-1]))]
Note: the normalizer_fn arg is very useful as you can see, as well. It can eliminate the need of any additional normalization preprocessing layer should you need it.
And so my layer became:
layers.DenseFeatures(feature_columns=featureColumns, trainable=True)
I believe this should solve the error mentioned above in your question. Which is quoted as
ValueError: Passing a dictionary input to a Sequential Model which
doesn't have FeatureLayer as the first layer is an error.

Loading Weights - NoneType Found

I am working on an LSTM for a final project. I've been following TensorFlow's tutorial here: https://www.tensorflow.org/tutorials/sequences/text_generation for most of it, especially for how to save and load the models. However, it's coming up with this error:
Traceback (most recent call last):
File "D:\xxx\Documents\Class Coding\Artificial Intelligence\Shelley>\Writerbot.py", line 187, in
restore_progress()
File "D:\xxx\Documents\Class Coding\Artificial Intelligence\Shelley\Writerbot.py", line 141, in restore_progress
shelley.load_weights(weights)
File "C:\Users\xxx\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\network.py", line 1508, in load_weights
if _is_hdf5_filepath(filepath):
File "C:\Users\xxx\AppData\Roaming\Python\Python36\site-packages\tensorflow\python\keras\engine\network.py", line 1648, in _is_hdf5_filepath
return filepath.endswith('.h5') or filepath.endswith('.keras')
AttributeError: 'NoneType' object has no attribute 'endswith'
And here is my code related to loading and restoring weights, as best as I can tell, since the rest of the error's coming from keras:
def create_shelley(vocab, embedding, numunits, batch):
"""This is what actually creates a neural network."""
shelley = tf.keras.Sequential([
tf.keras.layers.Embedding(vocab, embedding,
batch_input_shape=[batch, None]),
lstm(numunits,
return_sequences=True,
recurrent_initializer='glorot_uniform',
stateful=True),
tf.keras.layers.Dense(vocab)
])
return shelley
def train():
"""We create weight checkpoints as we train our neural network on files fed into it."""
checkpoints = 'D:\\xxx\\Documents\\Class Coding\\Artificial Intelligence\\Shelley\\trainingcheckpoints'
prefix = os.path.join(checkpoints, "ckpt_{epoch}")
callback=tf.keras.callbacks.ModelCheckpoint(
filepath=prefix,
save_weights_only=True)
print(epochsteps)
history = shelley.fit(botfeed.repeat(), epochs=epochs, steps_per_epoch=epochsteps, callbacks=[callback])
def restore_progress():
"""Load the most recent weight checkpoint."""
trainingcheckpoints = "D:\\Robin Pegau\\Documents\\Class Coding\\Artificial Intelligence\\Shelley\\trainingcheckpoints\\checkpoint"
weights = tf.train.latest_checkpoint(trainingcheckpoints)
shelley = create_shelley(vocab, embed, totalunits, batch = 1)
shelley.load_weights(weights)
shelley.build(tf.TensorShape([1, None]))
restore_progress()
There is a "checkpoint" file that has no filetype. There are also files that look like "ckpt_[x].index" and "ckpt_[x].data-00000-of-00001
Thank you all for your help in advance.

Unknown: KeyError: 'pyfunc_0'

I am exporting a savedModel which takes a string placeholder as the input tensor. I injected a graph to preprocess this string tensor so that it can be passed into the model. However, I am using py_func to perform my python string operations on the tensor.
Here input_text is the input tensor in the savedModel signature. I created another placeholder with default input_ints which is initialized with result of performing py_func on input_text. I initially had input_text as an operation (input_ints =tf.py_func(preprocess, [input_text], tf.int64)) but then tf.nn.dynamic_rnn was not accepting a tensor with unspecified shape.
# Create the graph object
with tf.name_scope('inputs'):
input_text = tf.placeholder(tf.string, name="input_text")
input_ints = tf.placeholder_with_default(
tf.py_func(preprocess, [input_text], tf.int64), shape=[None, None])
def lstm_cell():
# Your basic LSTM cell
lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size, reuse=tf.get_variable_scope().reuse)
# Add dropout to the cell
return tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)
# def create_rnn():
with tf.name_scope("Embeddings"):
embedding = tf.Variable(tf.random_uniform((vocab_size, embed_size), -1, 1))
embed = tf.nn.embedding_lookup(embedding, input_ints)
with tf.name_scope("RNN_layers"):
cell = tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(lstm_layers)])
initial_state = cell.zero_state(batch_size, tf.float32)
with tf.name_scope("RNN_forward"):
outputs, final_state = tf.nn.dynamic_rnn(cell, embed, initial_state=initial_state)
with tf.name_scope('predictions'):
predictions = tf.contrib.layers.fully_connected(outputs[:, -1], 1, activation_fn=tf.sigmoid)
Now using the above implementation, I can export the model properly but when restoring the model, I get the following error:
2017-11-23 17:29:14.600184: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: KeyError: 'pyfunc_0'
Traceback (most recent call last):
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
status, run_metadata)
File "/Users/sakibarrahman/anaconda/lib/python3.6/contextlib.py", line 89, in __exit__
next(self.gen)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.UnknownError: KeyError: 'pyfunc_0'
[[Node: inputs/PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_INT64], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_inputs/input_text_0_0)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "neural_load_model.py", line 85, in <module>
result = sess.run(output_tensor, {input_tensor: "Charter Communications, Inc. (CHTR) Stock Rating Reaffirmed by Goldman Sachs Group, Inc. (The)"})
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: KeyError: 'pyfunc_0'
[[Node: inputs/PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_INT64], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_inputs/input_text_0_0)]]
Caused by op 'inputs/PyFunc', defined at:
File "neural_load_model.py", line 74, in <module>
model = tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], import_path)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/saved_model/loader_impl.py", line 216, in load
saver = tf_saver.import_meta_graph(meta_graph_def_to_load, **saver_kwargs)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1698, in import_meta_graph
**kwargs)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 656, in import_scoped_meta_graph
producer_op_list=producer_op_list)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 313, in import_graph_def
op_def=op_def)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/Users/sakibarrahman/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
UnknownError (see above for traceback): KeyError: 'pyfunc_0'
[[Node: inputs/PyFunc = PyFunc[Tin=[DT_STRING], Tout=[DT_INT64], token="pyfunc_0", _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_inputs/input_text_0_0)]]
I have looked at this issue posted on Github but I am not sure as to how to implement this. Also, I am just loading the model and passing in a string for input and not using 'freeze_graph'.
My code for saving the model:
saver = tf.train.Saver()
#Define new functions
def preprocess(text):
.
.
.
tf.reset_default_graph()
.
.
.
#Define new placeholder that was not in the original model graph
#Define new placeholder with default value initialized with py_func that was not in the original model graph
with tf.name_scope('inputs'):
input_text = tf.placeholder(tf.string, name="input_text")
input_ints = tf.placeholder_with_default(
tf.py_func(preprocess, [input_text], tf.int64), shape=[None, None])
.
.
.
#Define placeholders and ops that I need and were in the original graph
saver = tf.train.Saver()
#Serving the model
with tf.Session() as sess:
#Restore from old checkpoint
saver.restore(sess, import_path)
print ('Exporting trained model to %s'%(export_path))
builder = saved_model_builder.SavedModelBuilder(export_path)
original_assets_directory = export_path + '/assets'
original_assets_filename = "vocabulary.pickle"
original_assets_filepath = write_vocab(original_assets_directory,
original_assets_filename)
# Set up the assets collection.
assets_filepath = tf.constant(original_assets_filepath)
tf.add_to_collection(tf.GraphKeys.ASSET_FILEPATHS, assets_filepath)
filename_tensor = tf.Variable(
original_assets_filename,
name="vocab_tensor",
trainable=False,
collections=[])
assign_filename_op = filename_tensor.assign(original_assets_filename)
# Build the signature_def_map.
classification_inputs = utils.build_tensor_info(input_text)
classification_outputs_classes = utils.build_tensor_info(predictions)
classification_signature = signature_def_utils.build_signature_def(
inputs={signature_constants.CLASSIFY_INPUTS: classification_inputs},
outputs={
signature_constants.CLASSIFY_OUTPUT_CLASSES:
classification_outputs_classes,
},
method_name=signature_constants.CLASSIFY_METHOD_NAME)
legacy_init_op = tf.group(
tf.tables_initializer(), name='legacy_init_op')
#add the sigs to the servable
builder.add_meta_graph_and_variables(
sess, [tag_constants.SERVING],
signature_def_map={
signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:
classification_signature
},
assets_collection=tf.get_collection(tf.GraphKeys.ASSET_FILEPATHS),
legacy_init_op=tf.group(assign_filename_op))
print ("added meta graph and variables")
builder.save()
print("model saved")
My code for loading the model. Not defining the function or the placeholders leads to the 'pyfunc_0' error:
#Define preprocess function
def preprocess(text_bin):
#Define new placeholders
with tf.name_scope('inputs'):
input_text = tf.placeholder(tf.string, name="input_text")
input_ints = tf.placeholder_with_default(
tf.py_func(preprocess, [input_text], tf.int64), shape=[None, None])
with tf.Session(graph=tf.Graph()) as sess:
# restore save model
model = tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], import_path)
print("model restored")
loaded_graph = tf.get_default_graph()
# get necessary tensors by name
input_tensor_name = model.signature_def[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs[signature_constants.CLASSIFY_INPUTS].name
input_tensor = loaded_graph.get_tensor_by_name(input_tensor_name)
output_tensor_name = model.signature_def[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY].outputs[signature_constants.CLASSIFY_OUTPUT_CLASSES].name
output_tensor = loaded_graph.get_tensor_by_name(output_tensor_name)
result = sess.run(output_tensor, {input_tensor: "Some String"})
print (result)
Update:
Defining the functions and placeholders when loading the savedModel seems to work. However, I don't know why they are not being added to the graph prior to using the builder to save the model

It looks like your model has a custom layer. You can follow the model code and find that. So, you can define that function before graph loading. Also, the function definition order is important.

The preprocess function that was being used was not really part of the graph, so py_func() wouldn't know which function to use when loading the savedModel. There is currently no easy way to do preprocessing within Tensorflow Serve flow. It has to be done on the client side before using the model, or a custom op may have to be created so that it can be a part of the model. The other alternative may be to create a custom servable.