Develop Reference

How to avoid root handler being called from the custom logger in Python?

I have a basic config for the logging module with debug level - now I want to create another logger with error level only. How can I do that?
The problem is that the root handler is called in addition to the error-handler - this is something I want to avoid.
import logging
fmt = '%(asctime)s:%(funcName)s:%(lineno)d:%(levelname)s:%(name)s:%(message)s'
logging.basicConfig(level=logging.DEBUG, format=fmt)
logger = logging.getLogger('Temp')
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.error('boo')
The above code prints boo twice while I expect once only, and I have no idea what to do with this annoying issue...
In [4]: logger.error('boo')
boo
2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
In [5]: logger.handlers
Out[5]: [<StreamHandler stderr (ERROR)>]

Some basics about the logging module
logger: a person who receives the log string, sorts it by a predefined level, then uses his own handler if any to process the log and, by default passes the log to his superior.
root logger: the superior of superiors, does all the things that a normal logger does but doesn't pass the received log to anyone else.
handler: a private contractor of a logger, who actually does anything with the log, eg. formats the log, writes it to a file or stdout, or sends it through tcp/udp.
formatter: a theme, a design that the handler applies to the log.
basicConfig: a shortcut way to config the root logger. This is useful when you want him to do all the job and all his lower rank loggers would just pass the log to him.
With no argument, basicConfig sets root logger's level to WARNING and add a StreamHandler that output the log to stderr.
What you did
You created a format and used a shortcut basicConfig to config the root logger. You want the root logger to do all the actual logging things
You created a new low-rank logger Temp
You want it to accept logs with only ERROR level and above.
You created another StreamHandler. Which output to stdout by default.
You want it to handle only ERROR level and above
Oh, you assigned it to Temp logger, which made 5. redundant since the level is set in 3.
Oh wait, thought you just want the root logger to do the job since 1.!
You logged an ERROR with your logger.
What happened
Your Temp logger accepted a string boo at ERROR level. Then told its handler to process the string. Since this handler didn't have any formatter assigned to it, it outputted the string as-is to stdout: boo
After that, Temp logger passed the string boo to his superior, the root logger.
The root logger accepted the log since the log level is ERROR > WARNING.
The root logger then told its handler to process the string boo.
This handler applies the format string to boo. Added timestamp, added location, added the name of logger that passed the log, etc.
Finally it outputted the result to stderr: 2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
Make it right
Since your code does exactly what you tell it to do, you have to tell it as much detail as possible.
Only use basicConfig when you are lazy. By removing basicConfig line, your problem solved.
Use logger = logging.getLogger('__name__') so that the logger has the name of the module. Looking at the log and know exactly which import path that it came from.
Decide if a logger should keep the log on its own or pass it up the chain with the propagate property. In your case, logger.propagate = False also solves the problem.
Use a dictConfig file so you don't get messed with the config code.
In practice, you should not add handlers to your logger, and just let the logger pass the log all the way to the root and let the root do the actual logging. Why?
Someone else uses your code as a module, can control the logging. For example, not output to stdout but to tcp/udp, output with a different format, etc.
You can turn off the logging from a specific logger entirely, by propagating=False.
You know exactly all the handlers and formatters in the code if you only added them to the root logger. You have centralized control over the logging.

Why does logging a custom error in python appear as root?

EDIT: The reason that the :root: was appearing was because I typed logging.error(...) instead of logger.error. This caused the program to default to handler root. This changed the general formatting which includes the handler name. Correcting to logger and just adding the error name to the message seems to create the correct output.
OLD TEXT
With the logging package I am trying log a custom error. When doing so the error message shows up, but the custom error exception appears to show up as root. For example a simple case is shown below running on python 3.6.6
Input looks like:
import logging
import logging.config
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
HANDLER = logging.FileHandler('test.txt')
logger.addHandler(HANDLER)
FORMATTER = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
HANDLER.setFormatter(FORMATTER)
class FailedToDoSimpleTaskError(Exception):
pass
def fail_todo_thing():
raise FailedToDoSimpleTaskError('It was a good attempt though')
try:
fail_todo_thing()
except FailedToDoSimpleTaskError as err:
logging.error(err)
Output looks like:
__main__-ERROR:root:It was a good attempt though
What I am trying to understand is why it shows up with :root:, and if there is any way to have it show up instead with :FailedToDoSimpleTaskError: ?

As you stated in your edit, if you correct logging.error(err) with logger.error(err) you get the correct output __main__ - ERROR - It was a good attempt though.
The reason for root appearing in your message is explained under logging documentation. In that documentation you can find out about changing the format of displayed messages.
Since you did use logging.error(err) you were not invoking logger which had specified the format you wanted. Hence the root word appeared.
You should have also noticed that when using logging.error(err) you would get the message on the console and not added to your log file. And if you try to use logging.info no message would have been written to the console, as the defaul log level is WARNING and the level you had set as INFO was for the HANDLER named logger. So if you try for example logging.info("Info test") nothing would appear in the log console nor in your log file. However using logger.info("Info test") you would get the message "Info test" written in your log file.

robot framework info message not printing

My python code generates a log file using logging framework and all INFO messages are captured in the log file. I integrated my program with ROBOT framework and now the log file is not generated. Instead the INFO messages are printed in the log.html. I understand this is because robot existing logger is being called and hence INFO are directed to log.html. I don't want the behavior to change, I still want the user defined log file to be generated separately with just the INFO level messages.
How can I achieve this?

Python Code --> Logging Library --> "Log File"
RobotFramework --> Python Code --> Logging Library --> by default "log.html"
When you run using python code it will allow you set log file name.
But when you run using robotframework, the file is by default set to log.html (since robot uses the same logging library internally that you are using) so your logging function is overridden by that of robotframework.
That is why you see it in log.html instead of your file.
You can also refer Robot Framework not creating file or writing to it
Hope it helps!

The issue has been fixed now, which was a very minor one. But am still analyzing it deeper, will update when I am clear on the exact cause.
This was the module that I used,
def call_logger(logger_name, logFile):
level = logging.INFO
l = logging.getLogger(logger_name)
if not getattr(l, 'handler_set', None):
formatter = logging.Formatter('%(asctime)s : %(message)s')
fileHandler = logging.FileHandler(logFile, mode = 'a')
fileHandler.setFormatter(formatter)
streamHandler = logging.StreamHandler()
streamHandler.setFormatter(formatter)
l.setLevel(level)
l.addHandler(fileHandler)
l.addHandler(streamHandler)
l.handler_set = True
When I changed the parameter "logFile" to a different name "log_file" it worked.
Looks like "logFile" was a built in robot keyword.

Logging Handlers Empty - Why Logging TimeRoatingFileHandler doesn't work

So I do logging.config.fileConfig to setup my logging from a file config that has console and file handler. Then I do logging.getLogger(name) to get my logger and log. At certain times I want the filehandler's filename to change i.e. log rotate (I can't use time rotator because of some issues with Windows platform) so to do that I call logger.handlers - it shows an empty list, so I cant close them!! However when I step through the debugger, its clearly not empty (well of course without it I wouldn't be able to log right)
Not sure whats going on here, any gotchas that I'm missing?
Appreciate any help. Thanks.

It seems you need to correctly get the root logger:
logger = logging.getLogger(__name__)
handlers = logger.handlers[:]
print('module {}'.format(handlers))
print('module {}'.format(logger.hasHandlers()))
logger = logging.getLogger('root')
handlers = logger.handlers[:]
print('root {}'.format(handlers))
print('root {}'.format(logger.hasHandlers()))
logger = logging.getLogger()
handlers = logger.handlers[:]
print('blank {}'.format(handlers))
print('blank {}'.format(logger.hasHandlers()))
output:
module []
module True
root []
root True
blank [<logging.handlers.RotatingFileHandler object at 0x108d82898>,
<logging.StreamHandler object at 0x108d826d8>]
blank True

Firstly the issue is that, if you use a config file to initialise logging with file and console handlers, then it does not populate logging.handlers list, so you can not iterate over it and close+flush the streams prior to opening new one with a new logging file name.
If you want to use TimeRotatingFileHandler or RotatingFileHandler, it sits under logging/handler.py and when it tries to do a roll over, it only closes its own stream, as it has no idea what streams the parent logging (mostly singleton) class may have open. And so when you do a roll over, there is a file lock (file filehandler) and boom it all fails.
So the solution (for me) is to initialise logging programatically and use addHandlers on logging, which also populates logging.handlers [], which I then use to iterate over my console/file handler and close them prior to manually rotating the file.
It to me looks like an obvious bug with the logging class, and if its working on unix - it really shouldn't.
Thanks everyone, especially #falsetru for your help.

You can use RotatingFileHandler (not TimedRotatingFileHandler).
Calling doRollover of the handler will rotate the log files.

Maybe there is no such name as 'TimeRoatingFileHandler' because you missed 'd' in word 'Timed'. So it must be: 'TimedRoatingFileHandler'

FileHandler not sending output to either location I want

First time playing around with the logging module. I have two questions really:
All of the examples I have found on creating a FileHandler just use a dummy file name, but I want my log file to have a certain title format.
log = time.strftime('./logs/'+'%H:%M:%S %d %b %Y', time.localtime())+'.log'
logger = logging.getLogger('myproject')
formatter = logging.Formatter('%(asctime)s %(message)s', '%H:%M:%S %d %b %Y')
handler = logging.FileHandler(log)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
This seems to work at first, in that the log file is created, but when I kill the script and examine it, there is nothing in it. And before you suggest using with open(log) as f:, using f in place of log in handler = logging.FileHandler(log) gives me errors about the file not even being created. What am I doing wrong to not have the log file written to? Is it because I'm trying to be too clever with the filename?
The second question may subsume the first somehow. What I would really like is a way to write to both the console and the log file concurrently, since this script takes a long time and I'd like to gauge progress. I have seen that there are answers on how to do this, such as here, but all of these seem to assume that I want different levels of logging for the two different outputs; I don't. Incidentally, I do realize that if my script didn't dynamically generate the name of the log file, I could do something like python myscript.py | tee log.file to avoid making different handlers.
I feel I am having trouble answering my second question because of the variables involved in the first, but I am foggy right now due to illness and subsequent lack of sleep. If someone could explain to me like a five-year-old how to write to both a console and log file with identical output, I would greatly appreciate it.
Many thanks.

Your code is configuring a logger named 'myproject':
logger = logging.getLogger('myproject')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
If you use methods on that logger, everything will work:
logger.info('foo bar')
logging.getLogger('myproject').info('foo bar')
But if you just use the module-level functions, it will not:
logging.info('foo bar')
Why? Because the module-level functions use the default root logger, which is not the one you've configured. See the docs for details on how this works.
Generally, the way you deal with this is either to create a module-level logger object for each module, like this:
logger = logging.getLogger(__name__)
… or a class or instance attribute logger, something like this:
self.logger = logging.getLogger('{}.{}'.format(__name__, cls.__name__))
And then do everything through that logger or self.logger object, not through the module-level functions.
So, why are the module-level functions even there? For convenience in simple programs, mainly. If that sounds appropriate for you, and you want to use them, you can; you just have to configure the root logger instead of a different one. Change the first line to this:
logger = logging.getLogger()
Now when you configure logger, you're also configuring the module-level functions.