I've trained a Tensorflow Lite (TFLite) model saved as a *.tflite file.
I'm writing code that lets me pick a tflite file, and a folder containing images, and then runs inference on this images using that model.
Here is what I have written:
def testModel(self, testData):
#Test any model on any dataset
model = "**path to model file**"
#Loading TFLite model and allocating tensors.
interpreter = tf.lite.Interpreter(model_path=model)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
rawImg = "**path to test images folder**"
imgNameList = glob.glob(os.path.join(os.getcwd(), rawImg) + os.sep + '*') # gets list of image names in dir
#creates dataset and dataloader from images
testDataset = SalObjDataset(img_name_list = imgNameList,lbl_name_list = [], transform=transforms.Compose([RescaleT(224),ToTensorLab(flag=0)]))
testDataloader = DataLoader(testDataset,batch_size=1,shuffle=False)
#loops through dataloader (goes through each image file)
for _, data in enumerate(testDataloader):
inputImg = data['image']
if torch.cuda.is_available():
inputImg = Variable(inputImg.cuda())
else:
inputImg = Variable(inputImg)
#rearranges dimensions in image file to match the expected input dimensions
#also changes the type to uint8 as expected
inputImg = tf.transpose(inputImg.cpu(), perm = [0,2,3,1])
inputImg = tf.cast(inputImg, tf.uint8)
interpreter.set_tensor(input_details[0]['index'], inputImg)
interpreter.invoke()
output_data = interpreter.get_tensor_details()
print(output_data)
if __name__ == '__main__':
#initialise object with the modelID of the model you want to test
#pass the testing data folder name to testModel()
#this is the folder where the model is
modelID = "model_1"
tester = ModelTrainer(modelID)
#this is the folder where the testing images are
tester.testModel("model_1/model_1/plant")
The way our it's setup, the images for each label are stored in their own subdirectory, so all images of a 'plant' would be in folder/plant/image-1.jpg.
I'm not sure if I'm using 'interpreter.set_tensor' correctly, I've gone through the documentation quite intensively and I'm still a bit confused.
I'm also not sure how to make sense of the output, I would like to somehow get a loss/accuracy value, how do I go about doing this?
My output is currently just [[255]] for each image.
Thanks!
I'm assuming you have trained a object detection Have you added necessary metadata needed for interpreter to get the Outputs according to image given below Outputs
if not make sure to use metadata writer API
use this notebook for writing metadata in which pass the labels and model which
does not have any metadata in it
https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/models/convert/metadata_writer_tutorial.ipynb
TensorFlow Lite Metadata Writer API provides an easy-to-use API to create Model Metadata for popular ML tasks supported by the TFLite Task Library. This notebook shows examples on how the metadata should be populated for the following tasks below:
Image classifiers
Object detectors
Image segmenters
Natural language classifiers
Audio classifiers
https://www.tensorflow.org/lite/models/convert/metadata_writer_tutorial
Related
I'm trying to use Tensorflow Lite Image Searcher with mobilenet_v3 to query a database for a similar image and I'm getting surprisingly bad results.
Thus, I suspect there is a mistake in one of my steps.
Here is my code:
from tflite_model_maker import searcher
from tflite_support.task import vision
# Load pretrained model:
model_name = "lite-model_imagenet_mobilenet_v3_large_100_224_feature_vector_5_metadata_1.tflite"
data_loader = searcher.ImageDataLoader.create(model_name)
# Load db images, calc feature vectors
path_db = 'dir_with_db_jpg_images'
data_loader.load_from_folder(path_db)
# Set model:
scann_options = searcher.ScaNNOptions(
distance_measure="dot_product",
tree=searcher.Tree(num_leaves=10, num_leaves_to_search=2),
score_ah=searcher.ScoreAH(2, anisotropic_quantization_threshold=0.2))
model = searcher.Searcher.create_from_data(data_loader, scann_options)
# Export as tflite model:
model.export(
export_filename="searcher.tflite",
userinfo="",
export_format=searcher.ExportFormat.TFLITE)
# Load model:
image_searcher = vision.ImageSearcher.create_from_file("searcher.tflite")
# Predict NN for query image:
image = vision.TensorImage.create_from_file('path_query_img.jpg')
result = image_searcher.search(image)
result.nearest_neighbors
Do the bad results stem from missing preprocessing steps for the input images (DB or query)?
What happens to the images once load_from_folder(path) is called before the feature vector is created? And is this different from image_searcher.search(vision.TensorImage.create_from_file('path_query_img.jpg')) which might explain the bad results.
I couldn't figure out a way to first load the DB images and then feed them to the model for it to extract the feature vector and append it to the searcher model. Do such methods exist? That would allow me to experiment more with preprocessing steps (e.g. resizing the images to the 224x244 expected image size of the network - which I tried).
Maybe there is another problem with my code?
Thank you!
I am working on an image recognition project using TensorFlow and Keras, that I would like to implement to my Android project. And I am new to Tensorflow...
I would like to find the closest match between an image to a folder with +2000 images. Images are similar in background and size, like so:
For now I have this following Python code that works okay.
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.vgg16 import VGG16, preprocess_input
from tensorflow.keras.models import Model
import numpy as np
from PIL import Image
base_model = VGG16(weights='imagenet')
model = Model(inputs=base_model.input, outputs=base_model.get_layer('fc1').output)
def extract(img):
img = img.resize((224, 224)) # Resize the image
img = img.convert('RGB') # Convert the image color space
x = image.img_to_array(img) # Reformat the image
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)
feature = model.predict(x)[0] # Extract Features
return feature / np.linalg.norm(feature)
# Iterate through images and extract Features
images = ["img1.png","img2.png","img3.png","img4.png","img5.png"...+2000 more]
all_features = np.zeros(shape=(len(images),4096))
for i in range(len(images)):
feature = extract(img=Image.open(images[i]))
all_features[i] = np.array(feature)
# Match image
query = extract(img=Image.open("image_to_match.png")) # Extract its features
dists = np.linalg.norm(all_features - query, axis=1) # Calculate the similarity (distance) between images
ids = np.argsort(dists)[:5] # Extract 5 images that have lowest distance
Now I am a bit lost to where to go from here. To my understanding I need to create a .h5 file with all extracted image features and a .tflite file containing the model.
UPDATE after answer
I can convert the model with:
# Convert the model.
base_model = VGG16(weights='imagenet')
model = Model(inputs=base_model.input, outputs=base_model.get_layer('fc1').output)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
But how can I get the extracted features to my Android project? Also, the file size of the model is +400 mb so Android doesnt allow to import it.
Hope you can help me, thanks.
From Tensorflows own site:
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Please find the below diagram for better understanding of the conversion process. The TensorFlow Lite converter takes a tensorflow/keras model and generates a tensoflow lite (.tflite) model. Even though there is a command line way of converting the model (https://www.tensorflow.org/lite/convert/index#cmdline) you are recommended to use the Python API to do the same, because it allows to add metadata(if required) and apply optimizations to the model. The following steps lets you convert your keras model to tflite.
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
Many a times when you are dealing with bigger models, your model size will be much bigger than the allowed size and might not perform as good as you want. So you have apply optimizations to make the model work. This is done using tf.lite.Optimize. It allows you to optimize your model for speed, storage etc. before tensorflow allowed a lot of manual control where you were able to specify what you need to optimize upon using tf.lite.Optimize.OPTIMIZE_FOR_LATENCY or tf.lite.Optimize.OPTIMIZE_FOR_SIZE nowadays default comes with both these optimizations. Now the conversion code becomes like this.
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT] #optimization
tflite_quant_model = converter.convert()
# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_quant_model)
What this does is a dynamic range quantization. Check the size of your model after this step.
If you want to further quantize the model, for example convert all float32 to float16 which will reduce the model size to approx. half the size as original, you can do it specifying a target spec. then your code will look like this. (understand that this will affect the model accuracy)
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT] #optimization
converter.target_spec.supported_types = [tf.float16] #target spec
tflite_fp16_model = converter.convert()
tflite_model_fp16_file = tflite_models_dir/"model_quant_f16.tflite"
tflite_model_fp16_file.write_bytes(tflite_fp16_model)
There are other types of post training quantizations also, which you can find in this page.
https://www.tensorflow.org/lite/performance/post_training_quantization
All this is post training quantizations, you may quantize the model before that also. refer to this link for the same, you can also find a lot of tutorials via search https://www.tensorflow.org/api_docs/python/tf/quantization/
After this you will have to run these tflite models to test them using python.
the steps are as below.
Load the model onto interpreters.
Test the model with sample images(s)
Evaluate the model
you can find a detailed example on this page
https://www.tensorflow.org/lite/performance/post_training_float16_quant
there are many other types of quantization as well based on the precision required and the type of edge device you are going to use. Please refer to the below link for details on how to apply them to your model.
https://www.tensorflow.org/lite/performance/model_optimization.
After quantization check your model size, this should reduce the model size to the required value if not repeat the operation will lower precisions.
You have 2 options:
Post-training quantization
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quant_model = converter.convert()
Selective builds
TensorFlow Lite enables you to reduce model binary sizes by using selective builds (Mobilenet). It says:
Selective builds skip unused operations in your model set and produce
a compact library with just the runtime and the op kernels required
for the model to run on your mobile device.
I have transformed MNIST images saved as .pt files in a folder in Google drive. I'm writing my Pytorch code in Colab.
I would like to use these files, and create a Dataset that stores these images as Tensors. How can I do this?
Transforming images during training took too long. Hence, transformed them and saved them all as .pt files. I just want to load them back as a dataset and use them in my model.
The approach you are following to save images is indeed a good idea. In such a case, you can simply write your own Dataset class to load the images.
from torch.utils.data import Dataset, DataLoader
from torch.utils.data.sampler import RandomSampler
class ReaderDataset(Dataset):
def __init__(self, filename):
# load the images from file
def __len__(self):
# return total dataset size
def __getitem__(self, index):
# write your code to return each batch element
Then you can create Dataloader as follows.
train_dataset = ReaderDataset(filepath)
train_sampler = RandomSampler(train_dataset)
train_loader = DataLoader(
train_dataset,
batch_size=args.batch_size,
sampler=train_sampler,
num_workers=args.data_workers,
collate_fn=batchify,
pin_memory=args.cuda,
drop_last=args.parallel
)
# args is a dictionary containing parameters
# batchify is a custom function that prepares each mini-batch
I am working on training my own images read from my folders. I would be thankful if you could help me for this.
I successfully read my all images from the folder and create my own onehot_encoded labels. However, in each time I run my code, it takes a lot of time to do read all images from the folders. Therefore, I want to create dataset from these images and save it like MNIST to use faster. Thus, I will not read my whole images again. Could you please help me for this?
The code is:
path = "D:/cleandata/train_data/"
loadedImages = []
labels = []
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
for i in range(len(os.listdir(path))):
imagesList = listdir(path+os.listdir(path)[i])
for image in imagesList:
image_raw_data_jpg tf.gfile.FastGFile(path+os.listdir(path)
[i]+'/'+image, 'rb').read()
raw_image =tf.image.decode_png(image_raw_data_jpg,3)
gray_resize=tf.image.resize_images(raw_image, [28, 28])
image_data =
sess.run(tf.image.rgb_to_grayscale(gray_resize))
loadedImages.append(image_data)
Here is a tutorial on how to use a TFRecords file. It shows how to create the file (containing images and labels) and read from it.
http://www.machinelearninguru.com/deep_learning/tensorflow/basics/tfrecord/tfrecord.html
Or you could just use zipfile, and include the label in the image file name, thus keeping them together (that is what I did)
I'm relatively new to ML and very much new to TensorfFlow. I've spent quite a bit of time on the TensorFlow MINST tutorial as well as https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/how_tos/reading_data to try and figure out how to read my own data, but I'm getting a bit confused.
I have a bunch of images (.png) in a directory /images/0_Non/. I'm trying to make these into a TensorFlow Data set so then I can basically run the stuff from the MINST tutorial on it as a first pass.
import tensorflow as tf
# Make a queue of file names including all the JPEG images files in the relative image directory.
filename_queue = tf.train.string_input_producer(tf.train.match_filenames_once("../images/0_Non/*.png"))
image_reader = tf.WholeFileReader()
# Read a whole file from the queue, the first returned value in the tuple is the filename which we are ignoring.
_, image_file = image_reader.read(filename_queue)
image = tf.image.decode_png(image_file)
# Start a new session to show example output.
with tf.Session() as sess:
# Required to get the filename matching to run.
tf.initialize_all_variables().run()
# Coordinate the loading of image files.
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(coord=coord)
# Get an image tensor and print its value.
image_tensor = sess.run([image])
print(image_tensor)
# Finish off the filename queue coordinator.
coord.request_stop()
coord.join(threads)
I'm having a bit of trouble understanding what's going on here. So it seems like image is a tensor and image_tensor is an numpy array?
How do I get my images into a data set? I also tried following along the Iris example which is for a CSV which brought me to here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/datasets/base.py, but wasn't sure how to get this to work for my case where I have a bunch of png's.
Thanks!
The recently added tf.data API makes it easier to do this:
import tensorflow as tf
# Make a Dataset of file names including all the PNG images files in
# the relative image directory.
filename_dataset = tf.data.Dataset.list_files("../images/0_Non/*.png")
# Make a Dataset of image tensors by reading and decoding the files.
image_dataset = filename_dataset.map(lambda x: tf.decode_png(tf.read_file(x)))
# NOTE: You can add additional transformations, like
# `image_dataset.batch(BATCH_SIZE)` or `image_dataset.repeat(NUM_EPOCHS)`
# in here.
iterator = image_dataset.make_one_shot_iterator()
next_image = iterator.get_next()
# Start a new session to show example output.
with tf.Session() as sess:
try:
while True:
# Get an image tensor and print its value.
image_array = sess.run([next_image])
print(image_tensor)
except tf.errors.OutOfRangeError:
# We have reached the end of `image_dataset`.
pass
Note that for training you will need to get labels from somewhere. The Dataset.zip() transformation is a possible way to combine together image_dataset with a dataset of labels from a different source.