right now I'm trying to convert a SavedModel to TFLite for use on a raspberry pi. The model is MobileNet Object Detection trained on a custom dataset. The SavedModel works perfectly, and retains the same shape of (1, 150, 150, 3). However, when I convert it to a TFLite model using this code:
import tensorflow as tf
saved_model_dir = input("Model dir: ")
# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) # path to the SavedModel directory
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
And run this code to run the interpreter:
import numpy as np
import tensorflow as tf
from PIL import Image
from os import listdir
from os.path import isfile, join
from random import choice, random
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
input_shape = input_details[0]['shape']
print(f"Required input shape: {input_shape}")
I get an input shape of [1 1 1 3], therefore I can't use a 150x150 image as input.
I'm using Tensorflow 2.4 on Python 3.7.10 with Windows 10.
How would I fix this?
You can rely on TFLite converter V1 API to set input shapes. Please check out the input_shapes argument in https://www.tensorflow.org/api_docs/python/tf/compat/v1/lite/TFLiteConverter.
How about calling resize_tensor_input() before calling allocate_tensors() ?
interpreter.resize_tensor_input(0, [1, 150, 150, 3], strict=True)
interpreter.allocate_tensors()
Related
At the End is a benefit analysis with Hi Guys,
at the moment I'm stuck with the conversion of an .pb Modell to a fully quantized integer TFLite model in TF2. I used a pre-trained model (SSD MobileNet v2 320x320) from the TensorFlow 2 Detection Model Zoo and pre-trained it on a small own dataset just to test the Workflow (I'm new to Tensorflow and machine learning in general).
The .pb Modell is working on my Computer.
Now I want to use the model on a Raspberry Pi with a Coral USB Edge TPU to perform Object Detection. Therefore the model has to be fully quantized to Integer Values.
I tried to use the code from the official TensorflowLite documentary, but had a few Errors with the tf2.3.1 version. Therefore I upgraded to tf-nightly=dev2.4.0-dev20200920 which solved some Errors.
I'm able to create an optimized float16 and Dynamic range quantified TFLite model. Or at least I get no errors and a TFLite file with a reduced size. But when I'm trying to convert the model to an int8 model with a representative dataset and I get a Runtime Error. Right now I'm not sure if my code to create a representative_dataset is wrong or the Problem is inside the TFLite scripts in TF2.
Help is very much appreciated because I'm stuck on this problem for days now and can't find any solution.
RuntimeError: Quantization not yet supported for op: 'CUSTOM'.
You can find the Python Script, 2 images and the original saved_model.pb file in the following Google Drive Folder here.
You have to adjust the path saved_model_dir and images_path in the script tflite_full_int.py.
Here is my Code:
import tensorflow as tf
import os
import numpy as np
import cv2
saved_model_dir = '**YOURPATH**/modelandimage/my_model/saved_model'
images_path = '**YOURPATH**/modelandimage/new'
def rep_data():
for f_name in os.listdir(images_path):
file_path = os.path.normpath(os.path.join(images_path, f_name))
img = cv2.imread(file_path)
img = cv2.resize(img, (320, 320))
img = img / 255.0
img = np.reshape(img, (1, 320, 320, 3))
img = img.astype(np.float32)
test = tf.dtypes.as_dtype(img)
yield[img]
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = rep_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_quant_model_full_int = converter.convert()
with open('model_quant_int.tflite', 'wb') as f:
f.write(tflite_quant_model_full_int)
System:
Linux: Ubuntu 18.04.5
tf-nightly=dev2.4.0-dev20200920
Cuda=11.0, Cudnn=8.0.2
Python=3.7.9
Cheers Guys
I am trying to deploy this mask-rcnn model in android. I was able to load the keras weight, freeze the model and convert it to .tflite model using tflite 1.13 toco using this script.
It seems this model uses some tf_ops that is not supported in tflite. Thus I had to use
converter.target_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,tf.lite.OpsSet.SELECT_TF_OPS]
to convert the model. Now when I try to infer this model using python interpreter, I get segmentation error in interpreter.invoke() and the python script crashes.
def run_tf_model(model_path="mask_rcnn_coco.tflite"):
interpreter = tf.lite.Interpreter(model_path)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()[0]
output_details = interpreter.get_output_details()[0]
print(" input_details", input_details)
print("output_details",output_details)
# Test model on random input data.
input_shape = input_details['shape']
print("input_shape tflite",input_shape)
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details['index'])
print(output_data)
Thus I am unable to find out if my converted model has been converted correctly or not.
P.S. I am planning to use this model in android, but I have little experience with android(java or kotlin) tflite api. If any one can point out any resources for learning about that would be also be helpful.
Edit: I also tried to run the inference on android with java api. But get the following error tensorflow/lite/kernels/gather.cc:80 0 <= axis && axis < NumDimensions(input).
Detailed in this tensorflow issue
You can verify your custom trained tflite model using TFLite python interpreter. Reference
import numpy as np
import tensorflow as tf
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test the model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)
I downloaded MobileBert model from TensorFlow - the Questions and Answers model based on TensorFlow Lite for mobile devices from here:
https://www.tensorflow.org/lite/models/bert_qa/overview
The example how to use it provided only for Android. Can anybody advise how to use this model in Python (for testing purposes). I followed recommendations on how to use TensorFlow Lite API, but I need to figure out how to modify it to use for MobileBert:
import numpy as np
import tensorflow as tf
# Load TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="mobilebert_float_20191023.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
input_data = np.array(np.random.random_sample(input_shape), dtype=np.int32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)
Please take a look at this project : https://github.com/samsgates/MobileBERT
It is basically designed to run the Google's model that you mentioned. I managed to run it with TF 1.14 along with other dependencies installed (sentencepiece, bert-for-tf2, both are available from PiP). The only thing I had to tweak is to remove the tf.dtypes.cast() wrappers over the input/output tensors. E.g. for the inputs I changed
input_ids = tf.dtypes.cast(self.get_ids(stokens),tf.int32)
to
input_ids = self.get_ids(stokens).astype('int32')
And for the outputs I changed
end = tf.argmax(end_logits,output_type=tf.dtypes.int32).numpy()[0]
to simple
end = np.argmax(end_logits)
That did the trick for me. Hope this helps!
I want to use this TF Hub asset:
https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3
Versions:
Version: 1.15.0-dev20190726
Eager mode: False
Hub version: 0.5.0
GPU is available
Code
feature_extractor_url = "https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3"
feature_extractor_layer = hub.KerasLayer(module,
input_shape=(HEIGHT, WIDTH, CHANNELS))
I get:
ValueError: Importing a SavedModel with tf.saved_model.load requires a 'tags=' argument if there is more than one MetaGraph. Got 'tags=None', but there are 2 MetaGraphs in the SavedModel with tag sets [[], ['train']]. Pass a 'tags=' argument to load this SavedModel.
I tried:
module = hub.Module("https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/3",
tags={"train"})
feature_extractor_layer = hub.KerasLayer(module,
input_shape=(HEIGHT, WIDTH, CHANNELS))
But when I try to save the model I get:
tf.keras.experimental.export_saved_model(model, tf_model_path)
# model.save(h5_model_path) # Same error
NotImplementedError: Can only generate a valid config for `hub.KerasLayer(handle, ...)`that uses a string `handle`.
Got `type(handle)`: <class 'tensorflow_hub.module.Module'>
Tutorial here
It's been a while, but assuming you have migrated to the TF2, this can easily be accomplished with the most recent model version as follows:
import tensorflow as tf
import tensorflow_hub as hub
num_classes=10 # For example
m = tf.keras.Sequential([
hub.KerasLayer("https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/5", trainable=True)
tf.keras.layers.Dense(num_classes, activation='softmax')
])
m.build([None, 224, 224, 3]) # Batch input shape.
# train as needed
m.save("/some/output/path")
Please update this question if that doesn't work for you. I believe your issue arose from mixing hub.Module with hub.KerasLayer. The model version you were using was in TF1 Hub format, so within TF1 it is meant to be used exclusively with hub.Module, and not mixed with hub.KerasLayer. Within TF2, hub.KerasLayer can load TF1 Hub format models directly from their URL for composition in larger models, but they cannot be fine-tuned.
Please refer to this compatibility guide for more information
You should use tf.keras.models.save_model(model,'NeuralNetworkModel')
You will get saved model in a folder that can be used later in your sequential nework
When bringing a keras model to production tensorflow serve is often used as a REST API. It has some drawbacks, as the image data is expecting the same Input format as the network input layer e.g. a array with shape (300,300,3) in json. The only way to make this working seems to be wraping the tensorflow serve API into another service.
How is it possible to make tensorflow serve delivering a keras model, which accepts base64 encoded images, without wrapping it in another API?
I found a solution for this and here is a more detailed explanation:
import tensorflow as tf
sess = tf.Session() # get the tensorflow session to reuse it in keras
from keras import backend as K
from keras.models import load_model
K.set_session(sess)
K.set_learning_phase(0) # make sure we disable dropout and other training specific layers
string_inp = tf.placeholder(tf.string, shape=(None,)) #string input for the base64 encoded image
imgs_map = tf.map_fn(
tf.image.decode_image,
string_inp,
dtype=tf.uint8
) # decode jpeg
imgs_map.set_shape((None, None, None, 3))
imgs = tf.image.resize_images(imgs_map, [300, 300]) # resize images
imgs = tf.reshape(imgs, (-1, 300, 300, 3)) # reshape them
img_float = tf.cast(imgs, dtype=tf.float32) / 255 - 0.5 # and make them to floats
model = load_model('keras.h5', compile=False) # load the model
output = model(img_float) # use the image tensor as input for keras
# ...(save to savedModel format and load in tensorflow serve)