Develop Reference

Converting Mobilenet Model to TFLite changes input size

right now I'm trying to convert a SavedModel to TFLite for use on a raspberry pi. The model is MobileNet Object Detection trained on a custom dataset. The SavedModel works perfectly, and retains the same shape of (1, 150, 150, 3). However, when I convert it to a TFLite model using this code:
import tensorflow as tf
saved_model_dir = input("Model dir: ")
# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) # path to the SavedModel directory
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
And run this code to run the interpreter:
import numpy as np
import tensorflow as tf
from PIL import Image
from os import listdir
from os.path import isfile, join
from random import choice, random
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
input_shape = input_details[0]['shape']
print(f"Required input shape: {input_shape}")
I get an input shape of [1 1 1 3], therefore I can't use a 150x150 image as input.
I'm using Tensorflow 2.4 on Python 3.7.10 with Windows 10.
How would I fix this?

You can rely on TFLite converter V1 API to set input shapes. Please check out the input_shapes argument in https://www.tensorflow.org/api_docs/python/tf/compat/v1/lite/TFLiteConverter.

How about calling resize_tensor_input() before calling allocate_tensors() ?
interpreter.resize_tensor_input(0, [1, 150, 150, 3], strict=True)
interpreter.allocate_tensors()

tflite quantization how to change the input dtype

see possible solution at the end of the post
I am trying to fully quantize the keras-vggface model from rcmalli to run on an NPU. The model is a Keras model (not tf.keras).
When using TF 1.15 for quantization with:
print(tf.version.VERSION)
num_calibration_steps=5
converter = tf.lite.TFLiteConverter.from_keras_model_file('path_to_model.h5')
#converter.post_training_quantize = True # This only makes the weight in8 but does not initialize model quantization
def representative_dataset_gen():
for _ in range(num_calibration_steps):
pfad='path_to_image(s)'
img=cv2.imread(pfad)
# Get sample input data as a numpy array in a method of your choosing.
yield [img]
converter.representative_dataset = representative_dataset_gen
tflite_quant_model = converter.convert()
open("quantized_model", "wb").write(tflite_quant_model)
The model is converted but as I need full int8 quantization, I add:
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
This error message appears:
ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: input_1
clearly, the input of the model still requires float32.
Questions:
Do I have to adapt the quantization method that the input dtype is changed ? or
Do I have to change the input layer of the model to dtype int8 beforehand?
Or is that actually reporting that the model is not actually quantized?
If 1 or 2 is the answer, would you also have a best practice tip for me?
Addition:
Using :
h5_path = 'my_model.h5'
model = keras.models.load_model(h5_path)
model.save(os.getcwd() +'/modelTF2')
to save the h5 as pb with TF 2.2 and then using converter=tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
as TF 2.x tflite takes floats, and convert them to uint8s internally . I thought that could be a solution. Unfortunately, this error message appears:
tf.lite.TFLiteConverter.from_keras_model giving 'str' object has no attribute 'call'
Apparently TF2.x cannot handle pure keras models.
using tf.compat.v1.lite.TFLiteConverter.from_keras_model_file() to solve this error just repeats the error from above, as we are back again at "TF 1.15" level.
Addition 2
Another solution is to transfer the keras model to tf.keras manually. I will look into that if there is no other solution.
Regarding the comment of Meghna Natraj
To recreate the model (using TF 1.13.x) just:
pip install git+https://github.com/rcmalli/keras-vggface.git
and
from keras_vggface.vggface import VGGFace
pretrained_model = VGGFace(model='resnet50', include_top=False, input_shape=(224, 224, 3), pooling='avg') # pooling: None, avg or max
pretrained_model.summary()
pretrained_model.save("my_model.h5") #using h5 extension
The input layer is connected. Too bad, that looked like a good/easy fix.
Possible Solution
It seems to work using TF 1.15.3 I used 1.15.0 beforehand. I will check if I did something else different by accident.

A possible reason why this fails is that the model has input tensors that are not connected to the output tensor, i.,e they are probably unused.
Here is a colab notebook where I've reproduced this error. Modify the io_type at the beginning of the notebook to tf.uint8 to see an error similar to one you got.
SOLUTION
You need to manually inspect the model and to see if there are any inputs that are dangling/lost/not connected to the output and remove them.
Post a link to the model and I can try to debug it as well.

Converting saved_model to TFLite model using TF 2.0

currently I am working on converting custom object detection model (trained using SSD and inception network) to quantized TFLite model. I can able to convert custom object detection model from frozen graph to quantized TFLite model using the following code snippet (using Tensorflow 1.4):
converter = tf.lite.TFLiteConverter.from_frozen_graph(args["model"],input_shapes = {'normalized_input_image_tensor':[1,300,300,3]},
input_arrays = ['normalized_input_image_tensor'],output_arrays = ['TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1',
'TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3'])
converter.allow_custom_ops=True
converter.post_training_quantize=True
tflite_model = converter.convert()
open(args["output"], "wb").write(tflite_model)
However tf.lite.TFLiteConverter.from_frozen_graph class method is not available for Tensorflow 2.0 (refer this link). So I tried to convert the model using tf.lite.TFLiteConverter.from_saved_model class method. The code snippet is shown below:
converter = tf.lite.TFLiteConverter.from_saved_model("/content/") # Path to saved_model directory
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
The above code snippet throws the following error:
ValueError: None is only supported in the 1st dimension. Tensor 'image_tensor' has invalid shape '[None, None, None, 3]'.
I tried to pass input_shapes as argument
converter = tf.lite.TFLiteConverter.from_saved_model("/content/",input_shapes={"image_tensor" : [1,300,300,3]})
but it throws the following error:
TypeError: from_saved_model() got an unexpected keyword argument 'input_shapes'
Am I missing something? Please feel free to correct me!

I got the solution using tf.compat.v1.lite.TFLiteConverter.from_frozen_graph. This compat.v1 brings the functionality of TF1.x into TF2.x.
Following is the full code:
converter = tf.compat.v1.lite.TFLiteConverter.from_frozen_graph("/content/tflite_graph.pb",input_shapes = {'normalized_input_image_tensor':[1,300,300,3]},
input_arrays = ['normalized_input_image_tensor'],output_arrays = ['TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1',
'TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3'])
converter.allow_custom_ops=True
# Convert the model to quantized TFLite model.
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()
# Write a model using the following line
open("/content/uno_mobilenetV2.tflite", "wb").write(tflite_model)

MobileBert from TensorFlow Lite in Python

I downloaded MobileBert model from TensorFlow - the Questions and Answers model based on TensorFlow Lite for mobile devices from here:
https://www.tensorflow.org/lite/models/bert_qa/overview
The example how to use it provided only for Android. Can anybody advise how to use this model in Python (for testing purposes). I followed recommendations on how to use TensorFlow Lite API, but I need to figure out how to modify it to use for MobileBert:
import numpy as np
import tensorflow as tf
# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="mobilebert_float_20191023.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
input_data = np.array(np.random.random_sample(input_shape), dtype=np.int32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

Please take a look at this project : https://github.com/samsgates/MobileBERT
It is basically designed to run the Google's model that you mentioned. I managed to run it with TF 1.14 along with other dependencies installed (sentencepiece, bert-for-tf2, both are available from PiP). The only thing I had to tweak is to remove the tf.dtypes.cast() wrappers over the input/output tensors. E.g. for the inputs I changed
input_ids = tf.dtypes.cast(self.get_ids(stokens),tf.int32)
to
input_ids = self.get_ids(stokens).astype('int32')
And for the outputs I changed
end = tf.argmax(end_logits,output_type=tf.dtypes.int32).numpy()[0]
to simple
end = np.argmax(end_logits)
That did the trick for me. Hope this helps!

Add custom pre-process within the Tensorflow graph

I have a meta and checkpoint files from which I load the weights of a pre-trained model. This works fine.
To test this model with a new image I need to do some pre-processing for the image (converting from grayscale to RGB, etc) which is basically done using opencv library. Doing this I do get my desired output.
But now what I want to do is add this pre-processing code to tensorflow itself so that when I save this model and re-use it I can only pass the image path as an argument and I don't need to do any pre-processing before passing it to tensorflow. I want tensorflow to handle all this.
I have tried the following already
The following I have used to implement the preprocessing the images within the tensorflow itself and save the new meta and checkpoint files
graph = tf.Graph()
with graph.as_default():
def dataprocess(x):
#convert from gray to rgb,etc
return y
path = ["images/test.jpg"]
filenames = tf.constant(path)
dataset = tf.contrib.data.Dataset.from_tensor_slices((filenames))
dataset = dataset.map(
lambda path : tf.py_func(
dataprocess[path], [tf.float32]))
iterator = dataset.make_initializable_iterator()
next_element = iterator.get_next()
next_element = tf.reshape(next_element,[-1,3,224,224]) #reshape as tensorflow shows unknown
The below code I use to restore my previous model
with tf.Session(graph=graph) as sess:
sess.run(iterator.initializer)
element1 = sess.run(next_element)
saver = tf.train.import_meta_graph('./.meta')
saver.restore(sess,'./')
saver1 = tf.train.Saver()
input= graph.get_tensor_by_name('input_1:0')
output= graph.get_tensor_by_name('predictions/Sigmoid:0')
print(sess.run(output,{input:element1}))
saver1.save(sess,'/tmp/test1/')
This all works fine
Next, I use the newly created meta and checkpoint files to test for an image(path)
path = ["images/test.jpg"]
with tf.Session() as sess:
saver = tf.train.import_meta_graph('./.meta')
saver.restore(sess,'./')
graph = tf.get_default_graph()
input = graph.get_tensor_by_name('Const:0')
output= graph.get_tensor_by_name('predictions/Sigmoid:0')
print(sess.run(output,{input:path}))
Using I get the following error
InvalidArgumentError: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,3,224,224]
[[Node: input_1 = Placeholder[dtype=DT_FLOAT, shape=[?,3,224,224], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
input_1 corresponds to the above input of the graph
So I am assuming I am not passing the path to the correct place
I am new to tensorflow and there is pretty less documentation regarding this
Thank you

But now what I want to do is add this pre-processing code to
tensorflow itself so that when I save this model and re-use it I can
only pass the image path as an argument and I don't need to do any
pre-processing before passing it to tensorflow. I want tensorflow to
handle all this.
Unfortunately this is not possible if you use the tf.py_func operation. When you save the graph the python code inside the tf.py_func will not be saved as it is not part of the graph. (See py_func limitations here.) The only way to make the pre-processing part of the graph would be to rewrite it with tensorflow code without using the tf.py_func.