I am implementing image classification using TensorFlow Inception v3 with GTX 1060 GPU. I trained around 3000 images in 9 different classifier, where each classifier contains from 100 to 500 images(100*100 px) using tensor for poets.
But when I have images more than 500, it throws out error and training fails. Same class with less images trains without error. My Image data size for each class is around 5000~10000 but only I can train data size of around 300~400. more of the data and it gives out following GPU error.
The specification as follows:
Windows 7 with GeForce GTX 1060
Anaconda 3 with python 3.5
Tensorflow Inception V3
I am novice for Tensorflow but I had faced the same problem.
My suggestion would be:
Try to rename all the images like:
TrainImage001.jpg
TrainImage002.jpg
TrainImage003.jpg
and try converting your images to jpg.
Still I am not sure but at least give it a try.. :)
Related
In Tensorflow Objecet Detection, I trained an SSD MobileNet V2 model. Now I'm trying to convert the saved_model to a frozen inference graph so I can use it in the Tello drone to detect cars, but I'm really confused as to how to do it.
I followed some tutorials but received no responses. First, I train in TF2. As noted, frozen graphs are not supported in TF2. So now I train in TF1.
I have a custom trained Mask R-CNN Inception ResNet V2 1024x1024 model from Tensorflow's model zoo. Linked here: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
When running the inference on an RTX 2080 TI, it takes 1 to 2 seconds to process an image, the results are good, and I understand my GPU is not a data center but I expected the processing time to be much faster.
Although I currently run my model through docker, I also have ran it through Anaconda on two Tesla v100 GPUs with very similar results.
Cuda info:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
The model is saved using Tensorflow's SavedModel format consisting of a directory containing the model, and frozen weights. This directory looks like this:
assets/
saved_model.pb
variables/
In the sample code, PATH_TO_SAVED_MODEL points to this directory.
I load and run my model using the following functions from TensorFlow2
# load the model from the frozen weights
import tensorflow as tf
detect_fn = tf.saved_model.load(PATH_TO_SAVED_MODEL)
# skipping code here...
# looping over frames in a video file
# The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
input_tensor = tf.convert_to_tensor(image_np)
# The model expects a batch of images, so add an axis with `tf.newaxis`.
input_tensor = input_tensor[tf.newaxis, ...]
# input_tensor = np.expand_dims(image_np, 0)
detections = detect_fn(input_tensor)
Is there a better method to load and run the model, that will be much faster or should I try to load a batch of images instead of a single image at a time?
Environment
I use tlt container version2 which support trt and pycuda:
tensorrt version : 7.0.0-1+cuda10.0
GPU Type: 1080 Ti
when I run inference for single image this part of code works perfectly:
np.copyto(inputs[0].host, img.ravel())
but when I run it for multiple images in a list (batch_size>1):
np.copyto(inputs[0].host, img_list.ravel())
it gives me an error.
Does trt model support multiple input of images in inference?
if it is then how to feed multiple images into inputs[0].host in inference part of tensorrt?
So GPUs and their "capacity" to hold images are what defines how many images it can hold of a given size. There is something called GPU memory, in case of NVIDIA A100(which is the highest capacity GPU available currently with NVIDIA) its 40GB.
So it all ultimately boils down to your GPU's memory and has got nothing to do with TensorRT, TensorRT is just an SDK
I have a research about soccer result prediction using bayesian network. But, I just a beginner on it. Then I want to start study about BN (Bayesian Network) with MNIST dataset, because my soccer dataset is just similar with MNIST, or I can say my dataset imitate the MNIST. I follow tutorial from this website https://alpha-i.co/blog/MNIST-for-ML-beginners-The-Bayesian-Way.html and I try it. But I can't get the result because there is out of memory output. I train the model from tutorial in CPU with Intel(R) Core(TM) i7-6700K CPU # 4.00GHz, NVIDIA graphic with OpenGL renderer: GeForce GTX 1060 3GB/PCIe/SSE2, and RAM 32.132,6 MB. Actually, I don't quite understand with the tutorial. I had tried ANN (Artificial Neural Network) using tensorflow with 500 data before I tried the BN and I got the result of ANN. But, why I can't get the BN result? Does BN need more memory space to run? Can I get other BN tutorials with MNIST??
I am new to tensorflow and trying to train my own object detection model.
Hardware: have tried multiple things, but biggest was a 32gb cpu. Also tried a 8gb cpu & 2gb gpu.
All running windows, python 3.6.2 & tensorflow 1.3
I've been following the guide here https://pythonprogramming.net/introduction-use-tensorflow-object-detection-api-tutorial/
Error is: (sorry, VM was shut down and that's all I have)
error message OOM
Batch Size: 3
Im starting to think my issue is the initial image size (4mb, 4288x3216), so I've been trying other resources (paperspace) and am still running out of memory on an 8gb gpu.
Ive got 571 images to train the model, and and 250 or so more for testing, all of the same size.
Any suggestions on passing the model a smaller amount of data so that I can keep it running?
As one of the comments suggested, I can resize but how will that affect my XML file?