I am trying to use a TensorRT engine for inference in a python class that inherits from multiprocessing. The engine works in a standalone python script on my system, but now while integrating it into the codebase, the multiprocessing used in the class seems to be causing problems.
I am not getting any errors. It just skips everything after the line self.runtime = trt.Runtime(self.trt_logger). My debugger from vscode does not go into the function either.
In the docs the following is mentioned, that I do not fully understand:
The TensorRT builder may only be used by one thread at a time. If you
need to run multiple builds simultaneously, you will need to create
multiple builders. The TensorRT runtime can be used by multiple
threads simultaneously, so long as each object uses a different
execution context.
The following parts of my code are started, joined and terminated from another file:
# more imports
import logging
import multiprocessing
import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
class MyClass(multiprocessing.Process):
def __init__(self, messages):
multiprocessing.Process.__init__(self)
# other stuff
self.exit = multiprocessing.Event()
def load_tensorrt_model(self, config):
'''Load tensorrt model with engine'''
logging.debug('Start')
# Reading the config parameters related to the engine
engine_file = config['trt_engine']['trt_folder'] + os.path.sep + config['trt_engine']['engine_file']
class_names_file = config['trt_engine']['trt_folder'] + os.path.sep + config['trt_engine']['class_names_file']
# Verify if all the necessary files are present, if so load the detection network
if os.path.exists(engine_file) and os.path.exists(class_names_file):
try:
logging.debug('In try statement')
self.trt_logger = trt.Logger()
f = open(engine_file, 'rb')
logging.debug('I can get here, but no further')
self.runtime = trt.Runtime(self.trt_logger)
logging.debug('Cannot get here')
self.engine = self.runtime.deserialize_cuda_engine(f.read())
# More stuff
I have found someone with a multithreading problem, but as of now I was unable to use this to solve my problem.
Any help is appreciated.
System specs:
Python 3.6.9
Jetson NX
Jetpack 4.4.1
L4T 32.4.4
Tensorrt 7.1.3.0-1
Cuda10.2
Ubuntu 18.04
same problem. It seems pycuda autoinit not working well under a multi process scenario.
try to replace import pycuda.autoinit with
cuda.init()
self.cuda_context = cuda.Device(0).make_context()
When I run the code below it shows me the error.
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
I have been searching solution on the online. The problem is the code below working in old version of torch (0.4.1). I want to know whether it is possible to modify or replace this code for working in the new version of pytorch.
from torch.utils.ffi import _wrap_function
from ._nms import lib as _lib, ffi as _ffi
__all__ = []
def _import_symbols(locals):
for symbol in dir(_lib):
fn = getattr(_lib, symbol)
if callable(fn):
locals[symbol] = _wrap_function(fn, _ffi)
else:
locals[symbol] = fn
__all__.append(symbol)
_import_symbols(locals())
I am facing the same problem have just seen some useful information in:
https://pytorch.org/tutorials/advanced/cpp_extension.html
https://pytorch.org/docs/stable/cpp_extension.html
To avoid downgrade the version of PyTorch, you should consider to use the following libraries while finding more details in the above links:
from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CppExtension
anyone knows if there is a method to prevent tensorflow from polluting standard error with gpus' memory allocation log?.
I noted that when the following command is executed:
with tf.Session() as sess:
tensorflow prints on standard error a log about memory and gpu resources allocation. Something like:
I tensorflow/core/common_runtime/local_device.cc:25] Local device intra op parallelism threads: 48
Graphics Device pciBusID 0000:02:00.0
Free memory: 11.75GiB
...
For important reasons, I wanna avoid this printing.
This was recently fixed, and should be available if you upgrade to TensorFlow 0.12 or later.
To disable all logging output from TensorFlow, set the following environment variable before launching Python:
$ export TF_CPP_MIN_LOG_LEVEL=3
$ python ...
You can also adjust the verbosity by changing the value of TF_CPP_MIN_LOG_LEVEL:
0 = all messages are logged (default behavior)
1 = INFO messages are not printed
2 = INFO and WARNING messages are not printed
3 = INFO, WARNING, and ERROR messages are not printed
You can set an environment variable before launching Python as described in the first answer, or you can add the following lines to your Python code:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
Change 3 to values (0, 1, 2, 3) according to the messages you want avoid.
If using TensorFlow 2.0+, make sure to put those lines before import tensorflow to be effective.
Defaults to 0, so all logs are shown. Set TF_CPP_MIN_LOG_LEVEL to 1 to filter out INFO logs, 2 to additionall filter out WARNING, 3 to additionally filter out ERROR.
In TensorFlow 2.0, this does not work for all logging messages (I still get tf.function retracing warnings, for instance).
To make sure everything is turned off, do this
import os
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
import logging
import tensorflow as tf
logger = tf.get_logger()
logger.setLevel(logging.ERROR) # or logging.INFO, logging.WARNING, etc.
Note 1: If you leave out the os.environ["TF_CPP_MIN_LOG_LEVEL"] line, some info messages are still printed (like the This TensorFlow binary is optimized with... that runs on startup.)
Note 2: in TF 1.0 you can also manually set the verbosity with tf.logging.set_verbosity(tf.logging.ERROR) but TF 2.0 does not have tf.logging.
I am using the RasaNLUHttpInterpreter as stated here to start my server. I give the class all the 4 parameters required (model-name, token, server-name, and project-name). However, I always get the error, that apparently I am handing over 5 arguments (what I don't really do).
The error occurred since I updated my Rasa-Core and NLU to the latest version. However, as in docs, I feel that I use the method correctly. Does anyone have an idea what I am doing wrong or what's happening here?
Here is my run-server.py where I use the RasaNLUHttpInterpreter:
import os
from os import environ as env
from gevent.pywsgi import WSGIServer
from server import create_app
from rasa_core import utils
from rasa_core.interpreter import RasaNLUHttpInterpreter
utils.configure_colored_logging("DEBUG")
user_input_dir = "/app/nlu/" + env["RASA_NLU_PROJECT_NAME"] + "/user_input"
if not os.path.exists(user_input_dir):
os.makedirs(user_input_dir)
nlu_interpreter = RasaNLUHttpInterpreter(
'model_20190702-103405', None, 'http://rasa-nlu:5000', 'test_project')
app = create_app(
model_directory = env["RASA_CORE_MODEL_PATH"],
cors_origins="*",
loglevel = "DEBUG",
logfile = "./logs/rasa_core.log",
interpreter = nlu_interpreter)
http_server = WSGIServer(('0.0.0.0', 5005), app)
http_server.serve_forever()
I am using:
rasa_nlu~=0.15.1
rasa_core==0.14.5
as already mentioned here I have analyzed the problem in detail.
First of all, the method calls and the given link belong to a rasa version that is deprecated. After updating to the latest rasa version which splits up core and nlu, the project was refactored to fit the according documentation.
After rebuilding the bot with the exact same setup, no errors were thrown and the bot worked as expected.
We came to the conclusion that this must have been a particular problem on threxx's workstation.
If someone else might come to this point, he is welcome to post here such that we could help him.
By debugging information I mean what TensorFlow shows in my terminal about loaded libraries and found devices etc. not Python errors.
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: Graphics Device
major: 5 minor: 2 memoryClockRate (GHz) 1.0885
pciBusID 0000:04:00.0
Total memory: 12.00GiB
Free memory: 11.83GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:717] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Graphics Device, pci bus id: 0000:04:00.0)
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:51] Creating bin of max chunk size 1.0KiB
...
You can disable all debugging logs using os.environ :
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
Tested on tf 0.12 and 1.0
In details,
0 = all messages are logged (default behavior)
1 = INFO messages are not printed
2 = INFO and WARNING messages are not printed
3 = INFO, WARNING, and ERROR messages are not printed
2.0 Update (10/8/19)
Setting TF_CPP_MIN_LOG_LEVEL should still work (see below in v0.12+ update), but there was a reported issue for version 2.0 until 2.3.z fixed in 2.4 and later. If setting TF_CPP_MIN_LOG_LEVEL does not work for you (again, see below), try doing the following to set the log level:
import tensorflow as tf
tf.get_logger().setLevel('INFO')
In addition, please see the documentation on tf.autograph.set_verbosity which sets the verbosity of autograph log messages - for example:
# Can also be set using the AUTOGRAPH_VERBOSITY environment variable
tf.autograph.set_verbosity(1)
v0.12+ Update (5/20/17), Working through TF 2.0+:
In TensorFlow 0.12+, per this issue, you can now control logging via the environmental variable called TF_CPP_MIN_LOG_LEVEL; it defaults to 0 (all logs shown) but can be set to one of the following values under the Level column.
Level | Level for Humans | Level Description
-------|------------------|------------------------------------
0 | INFO | [Default] Print all messages
1 | WARNING | Filter out INFO messages
2 | ERROR | Filter out INFO & WARNING messages
3 | NONE | Filter out all messages
See the following generic OS example using Python:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # or any {'0', '1', '2'}
import tensorflow as tf
You can set this environmental variable in the environment that you run your script in. For example, with bash this can be in the file ~/.bashrc, /etc/environment, /etc/profile, or in the actual shell as:
TF_CPP_MIN_LOG_LEVEL=2 python my_tf_script.py
To be thorough, you call also set the level for the Python tf_logging module, which is used in e.g. summary ops, tensorboard, various estimators, etc.
# append to lines above
tf.logging.set_verbosity(tf.logging.ERROR) # or any {DEBUG, INFO, WARN, ERROR, FATAL}
For 1.14 you will receive warnings if you do not change to use the v1 API as follows:
# append to lines above
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR) # or any {DEBUG, INFO, WARN, ERROR, FATAL}
**For Prior Versions of TensorFlow or TF-Learn Logging (v0.11.x or lower):**
View the page below for information on TensorFlow logging; with the new update, you're able to set the logging verbosity to either DEBUG, INFO, WARN, ERROR, or FATAL. For example:
tf.logging.set_verbosity(tf.logging.ERROR)
The page additionally goes over monitors which can be used with TF-Learn models. Here is the page.
This doesn't block all logging, though (only TF-Learn). I have two solutions; one is a 'technically correct' solution (Linux) and the other involves rebuilding TensorFlow.
script -c 'python [FILENAME].py' | grep -v 'I tensorflow/'
For the other, please see this answer which involves modifying source and rebuilding TensorFlow.
For compatibility with Tensorflow 2.0, you can use tf.get_logger
import logging
tf.get_logger().setLevel(logging.ERROR)
I have had this problem as well (on tensorflow-0.10.0rc0), but could not fix the excessive nose tests logging problem via the suggested answers.
I managed to solve this by probing directly into the tensorflow logger. Not the most correct of fixes, but works great and only pollutes the test files which directly or indirectly import tensorflow:
# Place this before directly or indirectly importing tensorflow
import logging
logging.getLogger("tensorflow").setLevel(logging.WARNING)
To anyone still struggling to get the os.environ solution to work as I was, check that this is placed before you import tensorflow in your script, just like mwweb's answer:
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # or any {'0', '1', '2'}
import tensorflow as tf
I solved with this post Cannot remove all warnings #27045 , and the solution was:
import logging
logging.getLogger('tensorflow').disabled = True
I am using Tensorflow version 2.3.1 and none of the solutions above have been fully effective.
Until, I find this package.
Install like this:
with Anaconda,
python -m pip install silence-tensorflow
with IDEs,
pip install silence-tensorflow
And add to the first line of code:
from silence_tensorflow import silence_tensorflow
silence_tensorflow()
That's It!
As TF_CPP_MIN_LOG_LEVEL didn't work for me you can try:
tf.logging.set_verbosity(tf.logging.WARN)
Worked for me in tensorflow v1.6.0
Usual python3 log manager works for me with tensorflow==1.11.0:
import logging
logging.getLogger('tensorflow').setLevel(logging.INFO)
for tensorflow 2.1.0, following code works fine.
import tensorflow as tf
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
To add some flexibility here, you can achieve more fine-grained control over the level of logging by writing a function that filters out messages however you like:
logging.getLogger('tensorflow').addFilter(my_filter_func)
where my_filter_func accepts a LogRecord object as input [LogRecord docs] and
returns zero if you want the message thrown out; nonzero otherwise.
Here's an example filter that only keeps every nth info message (Python 3 due
to the use of nonlocal here):
def keep_every_nth_info(n):
i = -1
def filter_record(record):
nonlocal i
i += 1
return int(record.levelname != 'INFO' or i % n == 0)
return filter_record
# Example usage for TensorFlow:
logging.getLogger('tensorflow').addFilter(keep_every_nth_info(5))
All of the above has assumed that TensorFlow has set up its logging state already. You can ensure this without side effects by calling tf.logging.get_verbosity() before adding a filter.
Yeah, I'm using tf 2.0-beta and want to enable/disable the default logging. The environment variable and methods in tf1.X don't seem to exist anymore.
I stepped around in PDB and found this to work:
# close the TF2 logger
tf2logger = tf.get_logger()
tf2logger.error('Close TF2 logger handlers')
tf2logger.root.removeHandler(tf2logger.root.handlers[0])
I then add my own logger API (in this case file-based)
logtf = logging.getLogger('DST')
logtf.setLevel(logging.DEBUG)
# file handler
logfile='/tmp/tf_s.log'
fh = logging.FileHandler(logfile)
fh.setFormatter( logging.Formatter('fh %(asctime)s %(name)s %(filename)s:%(lineno)d :%(message)s') )
logtf.addHandler(fh)
logtf.info('writing to %s', logfile)
I was struggling from this for a while, tried almost all the solutions here but could not get rid of debugging info in TF 1.14, I have tried following multiple solutions:
import os
import logging
import sys
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # FATAL
stderr = sys.stderr
sys.stderr = open(os.devnull, 'w')
import tensorflow as tf
tf.get_logger().setLevel(tf.compat.v1.logging.FATAL)
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
logging.getLogger('tensorflow').setLevel(tf.compat.v1.logging.FATAL)
sys.stderr = stderr
import absl.logging
logging.root.removeHandler(absl.logging._absl_handler)
absl.logging._warn_preinit_stderr = False
The debugging info still showed up, what finally helped was restarting my pc (actually restarting the kernel should work). So if somebody has similar problem, try restart kernel after you set your environment vars, simple but might not come in mind.
If you only need to get rid of warning outputs on the screen, you might want to clear the console screen right after importing the tensorflow by using this simple command (Its more effective than disabling all debugging logs in my experience):
In windows:
import os
os.system('cls')
In Linux or Mac:
import os
os.system('clear')
None of the solutions above could solve my problem in Jupyter Notebook, so I use the following snippet code bellow from Cicoria, and issues solved.
import warnings
with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=FutureWarning)
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.preprocessing.text import Tokenizer
print('Done')
Most of the answers here work, but you have to use them every time you open a new session (e.g. with JupyterLab). To make the changes stick, you have to set the environment variable.
Linux:
export TF_CPP_MIN_LOG_LEVEL="3"
(Also add the above line to .bashrc to make the change permanent, not just for the session)
Windows:
setx TF_CPP_MIN_LOG_LEVEL "3"
Both set the environment variables for the user.
After testing various suggestions so that they could also silence the resulting executable built with PyInstaller, I came up with this setting:
import logging
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
logging.getLogger('tensorflow').setLevel(logging.ERROR)
import tensorflow as tf
The line
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
will silent the warning about rebuilding TensorFlow:
I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA.
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
The line
logging.getLogger('tensorflow').setLevel(logging.ERROR)
will silent the warning about AutoGraph:
WARNING:tensorflow:AutoGraph is not available in this environment: functions lack code information. This is typical of some environments like the interactive Python shell. See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/autograph/g3doc/reference/limitations.md#access-to-source-code for more information.
The key point is to place these two before importing Tensorflow—despite Pylint's warning!
tensorflow 2.11.0
In Jupyter notebooks, you can use the %env magic command:
%env TF_CPP_MIN_LOG_LEVEL=3
import tensorflow as tf