EDIT: The reason that the :root: was appearing was because I typed logging.error(...) instead of logger.error. This caused the program to default to handler root. This changed the general formatting which includes the handler name. Correcting to logger and just adding the error name to the message seems to create the correct output.
OLD TEXT
With the logging package I am trying log a custom error. When doing so the error message shows up, but the custom error exception appears to show up as root. For example a simple case is shown below running on python 3.6.6
Input looks like:
import logging
import logging.config
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
HANDLER = logging.FileHandler('test.txt')
logger.addHandler(HANDLER)
FORMATTER = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
HANDLER.setFormatter(FORMATTER)
class FailedToDoSimpleTaskError(Exception):
pass
def fail_todo_thing():
raise FailedToDoSimpleTaskError('It was a good attempt though')
try:
fail_todo_thing()
except FailedToDoSimpleTaskError as err:
logging.error(err)
Output looks like:
__main__-ERROR:root:It was a good attempt though
What I am trying to understand is why it shows up with :root:, and if there is any way to have it show up instead with :FailedToDoSimpleTaskError: ?
As you stated in your edit, if you correct logging.error(err) with logger.error(err) you get the correct output __main__ - ERROR - It was a good attempt though.
The reason for root appearing in your message is explained under logging documentation. In that documentation you can find out about changing the format of displayed messages.
Since you did use logging.error(err) you were not invoking logger which had specified the format you wanted. Hence the root word appeared.
You should have also noticed that when using logging.error(err) you would get the message on the console and not added to your log file. And if you try to use logging.info no message would have been written to the console, as the defaul log level is WARNING and the level you had set as INFO was for the HANDLER named logger. So if you try for example logging.info("Info test") nothing would appear in the log console nor in your log file. However using logger.info("Info test") you would get the message "Info test" written in your log file.
Related
I have a basic config for the logging module with debug level - now I want to create another logger with error level only. How can I do that?
The problem is that the root handler is called in addition to the error-handler - this is something I want to avoid.
import logging
fmt = '%(asctime)s:%(funcName)s:%(lineno)d:%(levelname)s:%(name)s:%(message)s'
logging.basicConfig(level=logging.DEBUG, format=fmt)
logger = logging.getLogger('Temp')
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.error('boo')
The above code prints boo twice while I expect once only, and I have no idea what to do with this annoying issue...
In [4]: logger.error('boo')
boo
2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
In [5]: logger.handlers
Out[5]: [<StreamHandler stderr (ERROR)>]
Some basics about the logging module
logger: a person who receives the log string, sorts it by a predefined level, then uses his own handler if any to process the log and, by default passes the log to his superior.
root logger: the superior of superiors, does all the things that a normal logger does but doesn't pass the received log to anyone else.
handler: a private contractor of a logger, who actually does anything with the log, eg. formats the log, writes it to a file or stdout, or sends it through tcp/udp.
formatter: a theme, a design that the handler applies to the log.
basicConfig: a shortcut way to config the root logger. This is useful when you want him to do all the job and all his lower rank loggers would just pass the log to him.
With no argument, basicConfig sets root logger's level to WARNING and add a StreamHandler that output the log to stderr.
What you did
You created a format and used a shortcut basicConfig to config the root logger. You want the root logger to do all the actual logging things
You created a new low-rank logger Temp
You want it to accept logs with only ERROR level and above.
You created another StreamHandler. Which output to stdout by default.
You want it to handle only ERROR level and above
Oh, you assigned it to Temp logger, which made 5. redundant since the level is set in 3.
Oh wait, thought you just want the root logger to do the job since 1.!
You logged an ERROR with your logger.
What happened
Your Temp logger accepted a string boo at ERROR level. Then told its handler to process the string. Since this handler didn't have any formatter assigned to it, it outputted the string as-is to stdout: boo
After that, Temp logger passed the string boo to his superior, the root logger.
The root logger accepted the log since the log level is ERROR > WARNING.
The root logger then told its handler to process the string boo.
This handler applies the format string to boo. Added timestamp, added location, added the name of logger that passed the log, etc.
Finally it outputted the result to stderr: 2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
Make it right
Since your code does exactly what you tell it to do, you have to tell it as much detail as possible.
Only use basicConfig when you are lazy. By removing basicConfig line, your problem solved.
Use logger = logging.getLogger('__name__') so that the logger has the name of the module. Looking at the log and know exactly which import path that it came from.
Decide if a logger should keep the log on its own or pass it up the chain with the propagate property. In your case, logger.propagate = False also solves the problem.
Use a dictConfig file so you don't get messed with the config code.
In practice, you should not add handlers to your logger, and just let the logger pass the log all the way to the root and let the root do the actual logging. Why?
Someone else uses your code as a module, can control the logging. For example, not output to stdout but to tcp/udp, output with a different format, etc.
You can turn off the logging from a specific logger entirely, by propagating=False.
You know exactly all the handlers and formatters in the code if you only added them to the root logger. You have centralized control over the logging.
I'm using logging to log some information.
I used both FileHandler and StreamHandler in my logger, to output the message in the console and saved to a specific file.
In the console, to highlight some important message, I used \033. But this word also add into my log file and shown a weird symbol. Following is my currently usage example
logger = logging.getLogger('SayHello')
file_hdlr = logging.FileHandler('test.log')
file_hdlr.setFormatter(logging.Formatter('%(message)s'))
console_hdlr = logging.StreamHandler()
console_hdlr.setFormatter(logging.Formatter('%(message)s'))
logger.addHandler(file_hdlr)
logger.addHandler(console_hdlr)
logger.setLevel(logging.INFO)
logger.info('\033[1;31mHello World\033[0m')
Output in console:
Hello World (in red)
Output in log file:
[1;31mHello World[0m
How can I ignore the \033 and other color code in my file handler? Should I override the FileHandler class? Thanks a lot!
After several hours tracing, I found that FileHandler also inherited from StreamHandler. FileHandler.emit call StreamHandler.emit to write message into the file.
To ignore the control sequences in the message, I have to override both FileHandler and StreamHandler, says CustomFileHandler and CustomStreamHandler, respectively.
CustomFileHandler.emit call CustomStreamHandler.emit to write the message, where we have to remove the control sequences in it. With the help of regular expression, this should be easy to find them.
I set up a basic python logger that writes to a log file and to stdout. When I run my python program locally, log messages with logging.info appear as expected in the file and in the console. However, when I run the same program remotely via ssh -n user#server python main.py neither the console nor the file show any logging.info messages.
This is the code used to set up the logger:
def setup_logging(model_type, dataset):
file_name = dataset + "_" + model_type + time.strftime("_%Y%m%d_%H%M")
logging.basicConfig(
level=logging.INFO,
format="[%(levelname)-5.5s %(asctime)s] %(message)s",
datefmt='%H:%M:%S',
handlers=[
logging.FileHandler("log/{0}.log".format(file_name)),
logging.StreamHandler()
])
I already tried the following things:
Sending a message to logging.warning: Those appear as expected on the root logger. However, even without setting up the logger and falling back to the default logging.info messages do not show up.
The file and folder permissions seem to be alright and an empty file is created on disk.
Using print works as usual as well
If you look into the source code of basicConfig function, you will see that the function is applied only when there are no handlers on the root logger:
_acquireLock()
try:
force = kwargs.pop('force', False)
if force:
for h in root.handlers[:]:
root.removeHandler(h)
h.close()
if len(root.handlers) == 0:
handlers = kwargs.pop("handlers", None)
if handlers is None:
...
I think, one of the libraries you use configures logging on import. And as you see from the sample above, one of the solutions is to use force=True argument.
A possible disadvantage is that several popular data-science libraries keep a reference to the loggers they configure, so that when you reconfigure logging yourselves their old loggers with the handlers are still there and do not see your changes. In which case you will also need to clean the handlers for those loggers as well.
My python code generates a log file using logging framework and all INFO messages are captured in the log file. I integrated my program with ROBOT framework and now the log file is not generated. Instead the INFO messages are printed in the log.html. I understand this is because robot existing logger is being called and hence INFO are directed to log.html. I don't want the behavior to change, I still want the user defined log file to be generated separately with just the INFO level messages.
How can I achieve this?
Python Code --> Logging Library --> "Log File"
RobotFramework --> Python Code --> Logging Library --> by default "log.html"
When you run using python code it will allow you set log file name.
But when you run using robotframework, the file is by default set to log.html (since robot uses the same logging library internally that you are using) so your logging function is overridden by that of robotframework.
That is why you see it in log.html instead of your file.
You can also refer Robot Framework not creating file or writing to it
Hope it helps!
The issue has been fixed now, which was a very minor one. But am still analyzing it deeper, will update when I am clear on the exact cause.
This was the module that I used,
def call_logger(logger_name, logFile):
level = logging.INFO
l = logging.getLogger(logger_name)
if not getattr(l, 'handler_set', None):
formatter = logging.Formatter('%(asctime)s : %(message)s')
fileHandler = logging.FileHandler(logFile, mode = 'a')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
l.setLevel(level)
l.addHandler(fileHandler)
l.addHandler(streamHandler)
l.handler_set = True
When I changed the parameter "logFile" to a different name "log_file" it worked.
Looks like "logFile" was a built in robot keyword.
I am currently writing an API and an application which uses the API. I have gotten suggestions from people stating that I should perform logging using handlers in the application and use a "logger" object for logging from the API.
In light of the advice I received above, is the following implementation correct?
class test:
def __init__(self, verbose):
self.logger = logging.getLogger("test")
self.logger.setLevel(verbose)
def do_something(self):
# do something
self.logger.log("something")
# by doing this i get the error message "No handlers could be found for logger "test"
The implementation i had in mind was as follows:
#!/usr/bin/python
"""
....
....
create a logger with a handler
....
....
"""
myobject = test()
try:
myobject.do_something()
except SomeError:
logger.log("cant do something")
I'd like to get my basics strong, I'd be grateful for any help and suggestions for code you might recommend I look up.
Thanks!
It's not very clear whether your question is about the specifics of how to use logging or about logging exceptions, but if the latter, I would agree with Adam Crossland that log-and-swallow is a pattern to be avoided.
In terms of the mechanics of logging, I would make the following observations:
You don't need to have a logger as an instance member. It's more natural to declare loggers at module level using logger = logging.getLogger(__name__), and this will also work as expected in sub-packages.
Your call logger.log("message") would likely fail anyway because the log method has a level as the first argument, rather than a message.
You should declare handlers, and if your usage scenario is fairly simple you can do this in your main method or if __name__ == '__main__': clause by adding for example
logging.basicConfig(filename='/tmp/myapp.log', level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(name)s %(message)s')
and then elsewhere in your code, just do for example
import logging
logger = logging.getLogger(__name__)
once at the top of each module where you want to use logging, and then
logger.debug('message with %s', 'arguments') # or .info, .warning, .error etc.
in your code wherever needed.
The danger with the pattern that you are thinking about is that you may end up effectively hiding exceptions by putting them in a log. Many exceptions really should crash your program because they represent a problem that needs to be fixed. Generally, it is more useful to be able to step into code with a debugger to find out what caused the exception.
If there are cases that an exception represents an expected condition that does not affect the stability of the app or the correctness of its behavior, doing nothing but writing a notation to the log is OK. But be very, very careful about how you use this.
I usually do the following:
import logging
import logging.config
logging.config.fileConfig('log.congig')
# for one line log records
G_LOG = logging.getLogger(__name__)
# for records with stacktraces
ST_LOG = logging.getLogger('stacktrace.' + __name__)
try:
# some code
G_LOG.info('some message %s %s', param1, param2)
except (StandardError,):
message = 'some message'
G_LOG.error(message)
# exc_info appends stacktrace to the log message
ST_LOG.error(message, exc_info=True)
Format of the config file can be seen in the python manual