Develop Reference

How to avoid root handler being called from the custom logger in Python?

I have a basic config for the logging module with debug level - now I want to create another logger with error level only. How can I do that?
The problem is that the root handler is called in addition to the error-handler - this is something I want to avoid.
import logging
fmt = '%(asctime)s:%(funcName)s:%(lineno)d:%(levelname)s:%(name)s:%(message)s'
logging.basicConfig(level=logging.DEBUG, format=fmt)
logger = logging.getLogger('Temp')
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.error('boo')
The above code prints boo twice while I expect once only, and I have no idea what to do with this annoying issue...
In [4]: logger.error('boo')
boo
2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
In [5]: logger.handlers
Out[5]: [<StreamHandler stderr (ERROR)>]

Some basics about the logging module
logger: a person who receives the log string, sorts it by a predefined level, then uses his own handler if any to process the log and, by default passes the log to his superior.
root logger: the superior of superiors, does all the things that a normal logger does but doesn't pass the received log to anyone else.
handler: a private contractor of a logger, who actually does anything with the log, eg. formats the log, writes it to a file or stdout, or sends it through tcp/udp.
formatter: a theme, a design that the handler applies to the log.
basicConfig: a shortcut way to config the root logger. This is useful when you want him to do all the job and all his lower rank loggers would just pass the log to him.
With no argument, basicConfig sets root logger's level to WARNING and add a StreamHandler that output the log to stderr.
What you did
You created a format and used a shortcut basicConfig to config the root logger. You want the root logger to do all the actual logging things
You created a new low-rank logger Temp
You want it to accept logs with only ERROR level and above.
You created another StreamHandler. Which output to stdout by default.
You want it to handle only ERROR level and above
Oh, you assigned it to Temp logger, which made 5. redundant since the level is set in 3.
Oh wait, thought you just want the root logger to do the job since 1.!
You logged an ERROR with your logger.
What happened
Your Temp logger accepted a string boo at ERROR level. Then told its handler to process the string. Since this handler didn't have any formatter assigned to it, it outputted the string as-is to stdout: boo
After that, Temp logger passed the string boo to his superior, the root logger.
The root logger accepted the log since the log level is ERROR > WARNING.
The root logger then told its handler to process the string boo.
This handler applies the format string to boo. Added timestamp, added location, added the name of logger that passed the log, etc.
Finally it outputted the result to stderr: 2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
Make it right
Since your code does exactly what you tell it to do, you have to tell it as much detail as possible.
Only use basicConfig when you are lazy. By removing basicConfig line, your problem solved.
Use logger = logging.getLogger('__name__') so that the logger has the name of the module. Looking at the log and know exactly which import path that it came from.
Decide if a logger should keep the log on its own or pass it up the chain with the propagate property. In your case, logger.propagate = False also solves the problem.
Use a dictConfig file so you don't get messed with the config code.
In practice, you should not add handlers to your logger, and just let the logger pass the log all the way to the root and let the root do the actual logging. Why?
Someone else uses your code as a module, can control the logging. For example, not output to stdout but to tcp/udp, output with a different format, etc.
You can turn off the logging from a specific logger entirely, by propagating=False.
You know exactly all the handlers and formatters in the code if you only added them to the root logger. You have centralized control over the logging.

How to check if a logger exists

I've had to extend the existing logging library to add some features. One of these is a feature which lets a handler listen to any existing log without knowing if it exists beforehand. The following allows the handler to listen, but does not check if the log exists:
def listen_to_log(target, handler):
logging.getLogger(target).addHandler(handler)
The issue is that I don't want any handler to listen to a log which isn't being logged to, and want to raise a ValueError if the log doesn't already exist. Ideally I'd do something like the following:
def listen_to_log(target, handler):
if not logging.logExists(target):
raise ValueError('Log not found')
logging.getLogger(target).addHandler(handler)
So far I have been unable to find a function like logging.logExists, is there such a function, or a convenient workaround?

Workaround that works for me and possibly you:
When you create a logger for your own code, you will almost certainly create a logger with handlers (file handler and/or console handler).
When you have not yet created a logger and you get the 'root' logger by
logger = logging.getLogger()
then this logger will have no handlers yet.
Therefore, I always check whether the logger above results in a logger that has handlers by
logging.getLogger().hasHandlers()
So I create a logger only if the root logger does not have handlers:
logger = <create_my_logger> if not logging.getLogger().hasHandlers() else logging.getLogger()
The "create_my_logger" snippet represents my code/function that returns a logger (with handlers).

WARNING. This is not documented. It may change without notice.
The logging module internally uses a single Manager object to hold the hierarchy of loggers in a dict accessible as:
import logging
logging.Logger.manager.loggerDict
All loggers except the root logger are stored in that dict under their names.
There are few examples on the net:
http://code.activestate.com/lists/python-list/621740/
and
https://michaelgoerz.net/notes/use-of-the-logging-module-in-python.html (uses Python 2 print)

Of course, if you attach a handler to the root logger, you will (generally) get all messages (unless a library author specifically chooses not to propagate to root handlers).
Why should you care if a handler listens to a log which hasn't been created yet? You can apply filters to a handler that ignore certain events.
This doesn't specifically answer your question, because I see your question as an instance of an XY Problem.

FileHandler not sending output to either location I want

First time playing around with the logging module. I have two questions really:
All of the examples I have found on creating a FileHandler just use a dummy file name, but I want my log file to have a certain title format.
log = time.strftime('./logs/'+'%H:%M:%S %d %b %Y', time.localtime())+'.log'
logger = logging.getLogger('myproject')
formatter = logging.Formatter('%(asctime)s %(message)s', '%H:%M:%S %d %b %Y')
handler = logging.FileHandler(log)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
This seems to work at first, in that the log file is created, but when I kill the script and examine it, there is nothing in it. And before you suggest using with open(log) as f:, using f in place of log in handler = logging.FileHandler(log) gives me errors about the file not even being created. What am I doing wrong to not have the log file written to? Is it because I'm trying to be too clever with the filename?
The second question may subsume the first somehow. What I would really like is a way to write to both the console and the log file concurrently, since this script takes a long time and I'd like to gauge progress. I have seen that there are answers on how to do this, such as here, but all of these seem to assume that I want different levels of logging for the two different outputs; I don't. Incidentally, I do realize that if my script didn't dynamically generate the name of the log file, I could do something like python myscript.py | tee log.file to avoid making different handlers.
I feel I am having trouble answering my second question because of the variables involved in the first, but I am foggy right now due to illness and subsequent lack of sleep. If someone could explain to me like a five-year-old how to write to both a console and log file with identical output, I would greatly appreciate it.
Many thanks.

Your code is configuring a logger named 'myproject':
logger = logging.getLogger('myproject')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
If you use methods on that logger, everything will work:
logger.info('foo bar')
logging.getLogger('myproject').info('foo bar')
But if you just use the module-level functions, it will not:
logging.info('foo bar')
Why? Because the module-level functions use the default root logger, which is not the one you've configured. See the docs for details on how this works.
Generally, the way you deal with this is either to create a module-level logger object for each module, like this:
logger = logging.getLogger(__name__)
… or a class or instance attribute logger, something like this:
self.logger = logging.getLogger('{}.{}'.format(__name__, cls.__name__))
And then do everything through that logger or self.logger object, not through the module-level functions.
So, why are the module-level functions even there? For convenience in simple programs, mainly. If that sounds appropriate for you, and you want to use them, you can; you just have to configure the root logger instead of a different one. Change the first line to this:
logger = logging.getLogger()
Now when you configure logger, you're also configuring the module-level functions.

Why am I not receiving log INFO messages in python?

So I'm running the following code from the command line python:
import logging
rootLog = logging.getLogger(__name__)
rootLog.setLevel(logging.INFO)
rootLog.warning("This is a root warning")
rootLog.info("This is root info")
def info():
log = rootLog.getChild("info")
log.info("This is info")
log.warning("This is a warning")
info()
I'm expecting to see all four log messages on the console, but I'm only seeing the warnings. What is going on? Am I misunderstanding something?
EDIT:
I discovered by adding logging.basicConfig() at the beginning of the script that I'll get the output that I expected. This is strange, because the python documentation on logging states:
The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically if no handlers are defined for the root logger.

The bit you quote from the manual is under Module-Level Functions and applies only if the module function
logging.debug()
is literally called. Since you are calling an instance method with rootLog.info() basicConfig isn't getting called for you and you are probably talking to a null logger. The documentation is kinda confusing there.
Use loggging.basicConfig() and things should work.

Ok, I dug through the code in the logging module, and I think I figured it out, at least partially. What you are seeing is happening because what you are calling rootLogger isn't actually the root logger. It is true that if no handlers have been added to the root logger (the true root logger) when you call one of the log methods directly on logging, it calls basicConfig, but calling a log method on an instance of Logger as you are doing here does not actually call basicConfig at all. That's actually irrelevant in this case anyway though. :) I'm not sure where the handler that is getting called here is being created, but I can almost guarantee it's attached to the true root logger. The true root logger is initialized by default to WARNING. Try doing logging.root.setLevel(logging.INFO) and see if you get what you expect. You should also see what you want if you manually attach a logger to your rootLogger.

python logging specific level only

I'm logging events in my python code uing the python logging module. I have 2 logging files I wish to log too, one to contain user information and the other a more detailed log file for devs. I've set the the two logging files to the levels I want (usr.log = INFO and dev.log = ERROR) but cant work out how to restrict the logging to the usr.log file so only the INFO level logs are written to the log file as opposed to INFO plus everthing else above it e.g. INFO, WARNING, ERROR and CRITICAL.
This is basically my code:-
import logging
logger1 = logging.getLogger('')
logger1.addHandler(logging.FileHandler('/home/tmp/usr.log')
logger1.setLevel(logging.INFO)
logger2 = logging.getLogger('')
logger2.addHandler(logging.FileHandler('/home/tmp/dev.log')
logger2.setLevel(logging.ERROR)
logging.critical('this to be logged in dev.log only')
logging.info('this to be logged to usr.log and dev.log')
logging.warning('this to be logged to dev.log only')
Any help would be great thank you.

I am in general agreement with David, but I think more needs to be said. To paraphrase The Princess Bride - I do not think this code means what you think it means. Your code has:
logger1 = logging.getLogger('')
...
logger2 = logging.getLogger('')
which means that logger1 and logger2 are the same logger, so when you set the level of logger2 to ERROR you actually end up setting the level of logger1 at the same time. In order to get two different loggers, you would need to supply two different logger names. For example:
logger1 = logging.getLogger('user')
...
logger2 = logging.getLogger('dev')
Worse still, you are calling the logging module's critical(), info() and warning() methods and expecting that both loggers will get the messages. This only works because you used the empty string as the name for both logger1 and logger2 and thus they are not only the same logger, they are also the root logger. If you use different names for the two loggers as I have suggested, then you'll need to call the critical(), info() and warning() methods on each logger individually (i.e. you'll need two calls rather than just one).
What I think you really want is to have two different handlers on a single logger. For example:
import logging
mylogger = logging.getLogger('mylogger')
handler1 = logging.FileHandler('usr.log')
handler1.setLevel(logging.INFO)
mylogger.addHandler(handler1)
handler2 = logging.FileHandler('dev.log')
handler2.setLevel(logging.ERROR)
mylogger.addHandler(handler2)
mylogger.setLevel(logging.INFO)
mylogger.critical('A critical message')
mylogger.info('An info message')
Once you've made this change, then you can use filters as David has already mentioned. Here's a quick sample filter:
class MyFilter(object):
def __init__(self, level):
self.__level = level
def filter(self, logRecord):
return logRecord.levelno <= self.__level
You can apply the filter to each of the two handlers like this:
handler1.addFilter(MyFilter(logging.INFO))
...
handler2.addFilter(MyFilter(logging.ERROR))
This will restrict each handler to only write out log messages at the level specified.

First: this is a rather odd thing to want to do, and strikes me as a slight misuse of the logging system. I can't imagine any situation in which it makes sense to notify the user about the normal operation of the program but not about things that are more important. The logging levels should be used to indicate importance; if you have messages that are only of interest to developers, you should be using some other mechanism to distinguish them (such as which logger you send them to).
That being said, you can implement arbitrary filtering of log records by creating a Filter subclass whose filter method implements your desired criteria and install it on the appropriate handler.