Develop Reference

Convert a torchscript model into Core ML

I am trying to convert a Detectron 2 model into Core ML. I have scripted it and now I have a model of type 'torch.jit._script.RecursiveScriptModule'. If I try and convert it into Core ML using this code:
mlmodel = ct.converters.convert(
torchscipt_model,
inputs=[ct.TensorType(shape=(1, 3, 64, 64))],
)
I get the following error:
RuntimeError: Unknown type bool encountered in graph lowering. This
type is not supported in ONNX export.
I scripted the model using the following code (as illustrated in Detectron's documentation)
model = build_model(cfg)
model.eval()
fields = {"pred_boxes": Boxes, "scores": torch.Tensor, "pred_classes": torch.Tensor, "pred_masks": torch.Tensor, "proposal_boxes": Boxes, "objectness_logits": torch.Tensor}
torchscipt_model = scripting_with_instances(model, fields)
Do you know any other way to convert my Detectron2 model into Core ML?
Thank you so much for your help!

Error when converting a tf model to TFlite model

I am currently building a model to use it onto my nano 33 BLE sense board to predict weather by mesuring Humidity, Pressure, Temperature, I have 5 classes.
I have used a kaggle dataset to train on it.
df_labels = to_categorical(df.pop('Summary'))
df_features = np.array(df)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df_features, df_labels, test_size=0.15)
normalize = preprocessing.Normalization()
normalize.adapt(X_train)
activ_func = 'gelu'
model = tf.keras.Sequential([
normalize,
tf.keras.layers.Dense(units=6, input_shape=(3,)),
tf.keras.layers.Dense(units=100,activation=activ_func),
tf.keras.layers.Dense(units=100,activation=activ_func),
tf.keras.layers.Dense(units=100,activation=activ_func),
tf.keras.layers.Dense(units=100,activation=activ_func),
tf.keras.layers.Dense(units=5, activation='softmax')
])
model.compile(optimizer='adam',#tf.keras.optimizers.Adagrad(lr=0.001),
loss='categorical_crossentropy',metrics=['acc'])
model.summary()
model.fit(x=X_train,y=y_train,verbose=1,epochs=15,batch_size=32, use_multiprocessing=True)
Then the model is trained, I want to convert it into a tflite model when I run the command convert I get the following message :
# Convert the model to the TensorFlow Lite format without quantization
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model to disk
open("gesture_model.tflite", "wb").write(tflite_model)
import os
basic_model_size = os.path.getsize("gesture_model.tflite")
print("Model is %d bytes" % basic_model_size)
<unknown>:0: error: failed while converting: 'main': Ops that can be supported by the flex runtime (enabled via setting the -emit-select-tf-ops flag):
tf.Erf {device = ""}
For your information I use google colab to design the model.
If anyone has any idea or solution to this issue, I would be glad to hear it !

This often happens when you have not set the converter's supported Operations.
Here is an example:
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
This list of supported operations are constantly changing so in case the error still appears you can also try to set the experimental converter features as follow:
converter.experimental_new_converter = True

I solved the problem ! It was the activation function 'gelu' not yet supported by TFlite. I changed it to 'relu' and no more problem.

Tensorflow (TF2) quantization to full integer error with TFLiteConverter RuntimeError: Quantization not yet supported for op: 'CUSTOM'

At the End is a benefit analysis with Hi Guys,
at the moment I'm stuck with the conversion of an .pb Modell to a fully quantized integer TFLite model in TF2. I used a pre-trained model (SSD MobileNet v2 320x320) from the TensorFlow 2 Detection Model Zoo and pre-trained it on a small own dataset just to test the Workflow (I'm new to Tensorflow and machine learning in general).
The .pb Modell is working on my Computer.
Now I want to use the model on a Raspberry Pi with a Coral USB Edge TPU to perform Object Detection. Therefore the model has to be fully quantized to Integer Values.
I tried to use the code from the official TensorflowLite documentary, but had a few Errors with the tf2.3.1 version. Therefore I upgraded to tf-nightly=dev2.4.0-dev20200920 which solved some Errors.
I'm able to create an optimized float16 and Dynamic range quantified TFLite model. Or at least I get no errors and a TFLite file with a reduced size. But when I'm trying to convert the model to an int8 model with a representative dataset and I get a Runtime Error. Right now I'm not sure if my code to create a representative_dataset is wrong or the Problem is inside the TFLite scripts in TF2.
Help is very much appreciated because I'm stuck on this problem for days now and can't find any solution.
RuntimeError: Quantization not yet supported for op: 'CUSTOM'.
You can find the Python Script, 2 images and the original saved_model.pb file in the following Google Drive Folder here.
You have to adjust the path saved_model_dir and images_path in the script tflite_full_int.py.
Here is my Code:
import tensorflow as tf
import os
import numpy as np
import cv2
saved_model_dir = '**YOURPATH**/modelandimage/my_model/saved_model'
images_path = '**YOURPATH**/modelandimage/new'
def rep_data():
for f_name in os.listdir(images_path):
file_path = os.path.normpath(os.path.join(images_path, f_name))
img = cv2.imread(file_path)
img = cv2.resize(img, (320, 320))
img = img / 255.0
img = np.reshape(img, (1, 320, 320, 3))
img = img.astype(np.float32)
test = tf.dtypes.as_dtype(img)
yield[img]
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = rep_data
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_quant_model_full_int = converter.convert()
with open('model_quant_int.tflite', 'wb') as f:
f.write(tflite_quant_model_full_int)
System:
Linux: Ubuntu 18.04.5
tf-nightly=dev2.4.0-dev20200920
Cuda=11.0, Cudnn=8.0.2
Python=3.7.9
Cheers Guys

tflite quantization how to change the input dtype

see possible solution at the end of the post
I am trying to fully quantize the keras-vggface model from rcmalli to run on an NPU. The model is a Keras model (not tf.keras).
When using TF 1.15 for quantization with:
print(tf.version.VERSION)
num_calibration_steps=5
converter = tf.lite.TFLiteConverter.from_keras_model_file('path_to_model.h5')
#converter.post_training_quantize = True # This only makes the weight in8 but does not initialize model quantization
def representative_dataset_gen():
for _ in range(num_calibration_steps):
pfad='path_to_image(s)'
img=cv2.imread(pfad)
# Get sample input data as a numpy array in a method of your choosing.
yield [img]
converter.representative_dataset = representative_dataset_gen
tflite_quant_model = converter.convert()
open("quantized_model", "wb").write(tflite_quant_model)
The model is converted but as I need full int8 quantization, I add:
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8 # or tf.uint8
converter.inference_output_type = tf.int8 # or tf.uint8
This error message appears:
ValueError: Cannot set tensor: Got value of type UINT8 but expected type FLOAT32 for input 0, name: input_1
clearly, the input of the model still requires float32.
Questions:
Do I have to adapt the quantization method that the input dtype is changed ? or
Do I have to change the input layer of the model to dtype int8 beforehand?
Or is that actually reporting that the model is not actually quantized?
If 1 or 2 is the answer, would you also have a best practice tip for me?
Addition:
Using :
h5_path = 'my_model.h5'
model = keras.models.load_model(h5_path)
model.save(os.getcwd() +'/modelTF2')
to save the h5 as pb with TF 2.2 and then using converter=tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
as TF 2.x tflite takes floats, and convert them to uint8s internally . I thought that could be a solution. Unfortunately, this error message appears:
tf.lite.TFLiteConverter.from_keras_model giving 'str' object has no attribute 'call'
Apparently TF2.x cannot handle pure keras models.
using tf.compat.v1.lite.TFLiteConverter.from_keras_model_file() to solve this error just repeats the error from above, as we are back again at "TF 1.15" level.
Addition 2
Another solution is to transfer the keras model to tf.keras manually. I will look into that if there is no other solution.
Regarding the comment of Meghna Natraj
To recreate the model (using TF 1.13.x) just:
pip install git+https://github.com/rcmalli/keras-vggface.git
and
from keras_vggface.vggface import VGGFace
pretrained_model = VGGFace(model='resnet50', include_top=False, input_shape=(224, 224, 3), pooling='avg') # pooling: None, avg or max
pretrained_model.summary()
pretrained_model.save("my_model.h5") #using h5 extension
The input layer is connected. Too bad, that looked like a good/easy fix.
Possible Solution
It seems to work using TF 1.15.3 I used 1.15.0 beforehand. I will check if I did something else different by accident.

A possible reason why this fails is that the model has input tensors that are not connected to the output tensor, i.,e they are probably unused.
Here is a colab notebook where I've reproduced this error. Modify the io_type at the beginning of the notebook to tf.uint8 to see an error similar to one you got.
SOLUTION
You need to manually inspect the model and to see if there are any inputs that are dangling/lost/not connected to the output and remove them.
Post a link to the model and I can try to debug it as well.

Convert Mask-RCNN model for android with tflite

I am trying to deploy this mask-rcnn model in android. I was able to load the keras weight, freeze the model and convert it to .tflite model using tflite 1.13 toco using this script.
It seems this model uses some tf_ops that is not supported in tflite. Thus I had to use
converter.target_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,tf.lite.OpsSet.SELECT_TF_OPS]
to convert the model. Now when I try to infer this model using python interpreter, I get segmentation error in interpreter.invoke() and the python script crashes.
def run_tf_model(model_path="mask_rcnn_coco.tflite"):
interpreter = tf.lite.Interpreter(model_path)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()[0]
output_details = interpreter.get_output_details()[0]
print(" input_details", input_details)
print("output_details",output_details)
# Test model on random input data.
input_shape = input_details['shape']
print("input_shape tflite",input_shape)
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details['index'])
print(output_data)
Thus I am unable to find out if my converted model has been converted correctly or not.
P.S. I am planning to use this model in android, but I have little experience with android(java or kotlin) tflite api. If any one can point out any resources for learning about that would be also be helpful.
Edit: I also tried to run the inference on android with java api. But get the following error tensorflow/lite/kernels/gather.cc:80 0 <= axis && axis < NumDimensions(input).
Detailed in this tensorflow issue

You can verify your custom trained tflite model using TFLite python interpreter. Reference
import numpy as np
import tensorflow as tf
# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test the model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
# The function `get_tensor()` returns a copy of the tensor data.
# Use `tensor()` in order to get a pointer to the tensor.
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)