I'm trying to accelerate my model's performance by converting it to OnnxRuntime. However, I'm getting weird results, when trying to measure inference time.
While running only 1 iteration OnnxRuntime's CPUExecutionProvider greatly outperforms OpenVINOExecutionProvider:
CPUExecutionProvider - 0.72 seconds
OpenVINOExecutionProvider - 4.47 seconds
But if I run let's say 5 iterations the result is different:
CPUExecutionProvider - 3.83 seconds
OpenVINOExecutionProvider - 14.13 seconds
And if I run 100 iterations, the result is drastically different:
CPUExecutionProvider - 74.19 seconds
OpenVINOExecutionProvider - 46.96seconds
It seems to me, that the inference time of OpenVinoEP is not linear, but I don't understand why.
So my questions are:
Why does OpenVINOExecutionProvider behave this way?
What ExecutionProvider should I use?
The code is very basic:
import onnxruntime as rt
import numpy as np
import time
from tqdm import tqdm
limit = 5
# MODEL
device = 'CPU_FP32'
model_file_path = 'road.onnx'
image = np.random.rand(1, 3, 512, 512).astype(np.float32)
# OnnxRuntime
sess = rt.InferenceSession(model_file_path, providers=['CPUExecutionProvider'], provider_options=[{'device_type' : device}])
input_name = sess.get_inputs()[0].name
start = time.time()
for i in tqdm(range(limit)):
out = sess.run(None, {input_name: image})
end = time.time()
inference_time = end - start
print(inference_time)
# OnnxRuntime + OpenVinoEP
sess = rt.InferenceSession(model_file_path, providers=['OpenVINOExecutionProvider'], provider_options=[{'device_type' : device}])
input_name = sess.get_inputs()[0].name
start = time.time()
for i in tqdm(range(limit)):
out = sess.run(None, {input_name: image})
end = time.time()
inference_time = end - start
print(inference_time)
The use of ONNX Runtime with OpenVINO Execution Provider enables the inferencing of ONNX models using ONNX Runtime API while the OpenVINO toolkit runs in the backend.
This accelerates ONNX model's performance on the same hardware compared to generic acceleration on Intel® CPU, GPU, VPU and FPGA.
Generally, CPU Execution Provider works best with small iteration since its intention is to keep the binary size small. Meanwhile, the OpenVINO Execution Provider is intended for Deep Learning inference on Intel CPUs, Intel integrated GPUs, and Intel® MovidiusTM Vision Processing Units (VPUs).
This is why the OpenVINO Execution Provider outperforms the CPU Execution Provider during larger iterations.
You should choose Execution Provider that would suffice your own requirements. If you going to execute complex DL with large iteration, then go for OpenVINO Execution Provider. For a simpler use case, where you need the binary size to be smaller with smaller iterations, you can choose the CPU Execution Provider instead.
For more information, you may refer to this ONNX Runtime Performance Tuning
Regarding non-linear time, it might be the case that there is some preparation that happens when you first run the model with OpenVINO - perhaps the model is first compiled to OpenVINO when you first call sess.run. I observed a similar effect when using TFLite. For these scenarios, it makes sense to discard the time of the first iteration when benchmarking. There also tends to be quite a bit of variance so running >10 or ideally >100 iterations is a good idea.
Related
I am training a deep RL agent using tensorflow on a custom environment and have noticed that calling the environment.step method in my training loop is much slower than just calling it outside.
Here is psuedo code which explains this
class agent():
........
def train():
for i in batch_size:
start = time.time()
nextstate, reward, done = env.step()
agent.timecounter += time.time - start
For the agent I will find that calling env.step takes up ~.5 seconds out of the ~.8 seconds wall time for a single batch. Instead of calling inside the agent training loop, if I just call the code normally, I don't see this overhead.
start = time.time()
for i in batch_size:
out = env.step()
print(time.time() - start)
I will find that it only takes ~.01 seconds to run the environment.step for an equivalent amount of times. What is causing this extra overhead? It's making training for me very slow.
I tried with gym's cartpole environment and it does not have this issue. My custom environment has several dictionary attributes. Is this possibly what's causing the issue? I don't see any other differences between my code and gym's cartpole.
Full code to reproduce on your machine is available at https://github.com/ronan-keane/havsim/tree/DL3
in scripts/meng assignments/control 1/traintest.py
I managed to answer my own question. The issue was that the input from the neural net to the custom python code was being converted to a tf.float32. This would then cause the python code to convert everything to tensor, and all the math operations in the pure python code would then be acting on newly converted tensors. This added the huge 50x overhead.
Issue was fixed by just converting the output of the NN to a regular python float.
We ran inference over 1280*720 RGB images using Faster RCNN from TensorFlow model zoo, trained over the COCO dataset and got some results
Test case 1:
Created a tf session using tf.Session(graph=tf.Graph())
Batch Size = 4, GPU use = 100% (as default)
Time Taken for inference of 4 images = 0.32 secs
Test case 2:
Then we restricted the GPU usage for each TF session to a fraction using gpu_options
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.40)
tf.Session(graph=tf.Graph(), config=tf.ConfigProto(gpu_options=gpu_options))
For a batch size of 2, the time taken for inference was 0.16secs, which is understandable because this is linear.
Test case 3:
Ran 2 instances of inferences as two different python processes. Here, both the batch sizes were 2.
The inference slowed down considerably, and the final value was 0.32secs, which is same as the case of one process running inference over the batch size of 4.
Batch Size = 2, GPU use = 40%, Processes = 2
Time Taken = 0.32 secs
Hence, in Case 1 and Case 3, time taken is the same.
Q.1 Is there any way to reduce the time taken?
Q.2 Where is the bottleneck? In 1st case also, it doesn’t seem that the entire GPU memory is being utilised. Therefore, we had thought that if we were to divide the inference into two, more efficient utilisation of resources could be done. Where are we going wrong in our understanding?
Question:
There seems to be a slight idling time between inferences when using keras with a tensorflow back end shown in nvprofiler. The attached image of my terminal shows that keras takes ~80ms for each inference, while a check with nvprofiler shows that the calculation takes 62ms (including host to device copy and device to host copy) and it is basically idling the rest of the time. In addition, there is a difference in the average inference time (between 80 ~ 150ms) each time i run the code below and i am unsure why that is the case.
I was wondering if there is any way to speed up the inference time
by reducing this idling time.
Another question is am i missing some steps like clearing the GPU
memory that cause the difference in inference time.
The input to my model is (1,800,700,36) which is basically a point cloud in a voxelized grid. The output is 2 matrix (1,200,175,1) & (1x200,175,6).
Setup:
Ubuntu 16.04
Intel(R) Core(TM) i7-7820X CPU
GeForce GTX 980
Python 2.7
Tensorflow 1.12.0
Keras 2.2.4
Attempts reduce inference idle time
Allowing the memory to dynamically grow on the GPU
config.gpu_options.allow_growth = True
Limit the maximum gpu usage
config.gpu_options.per_process_gpu_memory_fraction = 0.66
Set learning phase to test
keras.backend.set_learning_phase(0)
Change batch size to None and step size to 1. Changing batch size to 1 and step size to None makes inference slower
model.predict(input_data, batch_size=None, verbose=1, steps=1)
Terminal Output
Output of keras verbose in terminal that shows each step taking ~80ms
Same code, Output of keras verbose in terminal that shows each step taking ~100ms
Images of NV profiler
As NV profiler is very long, I have split the image to 3 parts and have minimized process threads that are not running anything during that period. The images shows that the actual calculation takes ~60ms and the GPU is basically doing nothing for 20ms. This idle changes from run to run and sometimes it goes up to 70ms.
Part 1, time between the green lines is not doing anything
Part 2, time between the green lines is not doing anything
Part 3, time between the green lines is not doing anything
Code
import numpy as np
import tensorflow as tf
import keras.backend.tensorflow_backend
from keras.backend.tensorflow_backend import set_session
from keras.models import load_model
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.66
sess = tf.Session(config=config)
set_session(sess)
keras.backend.set_learning_phase(0)
losses = {
"binary_crossentropy" : "binary_crossentropy",
"huber_loss" : losses.huber_loss,
}
model = load_model('/home/louis/pixor/models/city_trainin_set/pixor_model_10_0.008.hdf5', custom_objects=losses)
obj = massage_own_label(base_dir,data)
input_data = obj.process_input_data(obj.load_pc_data(0))
for i in range(0,100):
out_class, out_labels = model.predict(input_data, batch_size=None, verbose=1, steps=1)
I was trying to find out if GPU tensor operations are actually faster than CPU ones. So, I wrote this particular code below to implement a simple 2D addition of CPU tensors and GPU cuda tensors successively to see the speed difference:
import torch
import time
###CPU
start_time = time.time()
a = torch.ones(4,4)
for _ in range(1000000):
a += a
elapsed_time = time.time() - start_time
print('CPU time = ',elapsed_time)
###GPU
start_time = time.time()
b = torch.ones(4,4).cuda()
for _ in range(1000000):
b += b
elapsed_time = time.time() - start_time
print('GPU time = ',elapsed_time)
To my surprise, the CPU time was 0.93 sec and the GPU time was as high as 63 seconds. Am I doing the cuda tensor operation properly or is the concept of cuda tensors works faster only in very highly complex operations, like in neural networks?
Note: My GPU is NVIDIA 940MX and torch.cuda.is_available() call returns True.
GPU acceleration works by heavy parallelization of computation. On a GPU you have a huge amount of cores, each of them is not very powerful, but the huge amount of cores here matters.
Frameworks like PyTorch do their to make it possible to compute as much as possible in parallel. In general matrix operations are very well suited for parallelization, but still it isn't always possible to parallelize computation!
In your example you have a loop:
b = torch.ones(4,4).cuda()
for _ in range(1000000):
b += b
You have 1000000 operations, but due to the structure of the code it impossible to parallelize much of these computations. If you think about it, to compute the next b you need to know the value of the previous (or current) b.
So you have 1000000 operations, but each of these has to be computed one after another. Possible parallelization is limited to the size of your tensor. This size though is not very large in your example:
torch.ones(4,4)
So you only can parallelize 16 operations (additions) per iteration.
As the CPU has few, but much more powerful cores, it is just much faster for the given example!
But things change if you change the size of the tensor, then PyTorch is able to parallelize much more of the overall computation. I changed the iterations to 1000 (because I did not want to wait so long :), but you can put in any value you like, the relation between CPU and GPU should stay the same.
Here are the results for different tensor sizes:
#torch.ones(4,4) - the size you used
CPU time = 0.00926661491394043
GPU time = 0.0431208610534668
#torch.ones(40,40) - CPU gets slower, but still faster than GPU
CPU time = 0.014729976654052734
GPU time = 0.04474186897277832
#torch.ones(400,400) - CPU now much slower than GPU
CPU time = 0.9702610969543457
GPU time = 0.04415607452392578
#torch.ones(4000,4000) - GPU much faster then CPU
CPU time = 38.088677167892456
GPU time = 0.044649362564086914
So as you see, where it is possible to parallelize stuff (here the addition of the tensor elements), GPU becomes very powerful. GPU time is not changing at all for the given calculations, the GPU can handle much more! (as long as it doesn't run out of memory :)
I'm running a series of neural networks (Keras library using a Tensorflow backend), and I have the following results for the time it took to train each neural network in Jupyter Notebook:
ELAPSED TIME: 2.7005105018615723
0
ELAPSED TIME: 2.4810903072357178
1
ELAPSED TIME: 2.801435708999634
2
ELAPSED TIME: 2.6753993034362793
3
ELAPSED TIME: 2.8625667095184326
4
ELAPSED TIME: 2.5828065872192383
5
while later on you have:
ELAPSED TIME: 5.062163829803467
0
ELAPSED TIME: 5.162402868270874
1
ELAPSED TIME: 5.301288366317749
2
ELAPSED TIME: 5.386904001235962
3
ELAPSED TIME: 6.126806020736694
4
The program consists of a function that trains a separate neural network model on their respective datasets, and only exports their final training accuracy (saved to another file).
I thought the reason why it took longer for the latter networks to train was because the program was consuming too much memory, so I would delete the models (with the del keyword) after having obtained their training accuracy, but that doesn't seem to be doing much.
If I were to restart the Jupyter Notebook kernel, the time to run each network would shorten back to about 2 seconds (original duration), but it takes longer for the latter models to run.
What could be possible reasons for this, and what solutions could be implemented?
.
NOTE: I did not include any code because it would make this post more dense, but if necessary I can upload it.
Are you running on an NVIDA GPU? If so it's possible some part of the old model is still on the GPU. Try running nvidia-smi while the slow model is running and see if anything else is using up GPU memory/resources.
If that doesn't work you can also run the tensorflow timeline and compare between the slow and fast runs. More info on how to generate the timeline in Keras here: https://github.com/tensorflow/tensorflow/issues/9868 and the code I pasted below for creating the timeline was taken from that link
from tensorflow.python.client import timeline
# Make your keras model
# ...
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
model.compile(loss='MSE', optimizer='Adam', options=run_options, run_metadata=run_metadata)
# Run model in your usual way
# ...
trace = timeline.Timeline(step_stats=run_metadata.step_stats)
with open('timeline.ctf.json', 'w') as f:
f.write(trace.generate_chrome_trace_format())
For more info on the tensorflow timeline see https://stackoverflow.com/a/37774470/2826818 The timeline will allow you to see how much time each operation is taking and determine which one is causing the slow down.
You can clear your session after you are done with each model, this fixed the issue for me.
from keras import backend as K
for model in models_to_train:
model.fit(X, y)
# save necessary metrics
K.clear_session()
(Also I was getting a segmentation fault until I update tensoflow-gpu to 1.9.0)