How can I change root logger level in it's submodule
#main.py
logging.basicConfig(filename=filename,format='%(asctime)s %(levelname)s %(message)s',filemode='w')
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
After certain event I want to change all the logging to CRITICAL
sub_logger = logging.getLogger('jdm_health')
#submodule.py
if event:
logger.setLevel(logging.CRITICAL)#root logger
Related
In my main.py file I initialize the logger as follows:
# File: main.py
import logging
logger = logging.getLogger(__name__)
import submodule
#...
def main():
logger.debug(f'Test')
if __name__ == '__main__':
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('[%(levelname)s | %(name)s] %(message)s')
# The handler h1 logs only debug and info to stdout
h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)
# The handler h2 logs only warning, error and exception to stderr
h2 = logging.StreamHandler()
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
logger.addHandler(h1)
logger.addHandler(h2)
main()
When I use the logger in the main script, everything works correctly.
But when I use it in a submodule like this:
# File: submodule.py
import logging
logger = logging.getLogger(__name__)
#...
logger.info('Test 1')
logger.error('Test 2')
I should see the following output (or something like this):
[DEBUG | __main__] Test
[INFO | submodule] Test 1
[ERROR | submodule] Test 2
But I get this output:
[DEBUG | __main__] Test
Test 2
To me it looks like the logger in submodules did not use the config of the logger in the main module. How to fix that?
This is because
logger = logging.getLogger(__name__)
will return different loggers in the main module and the submodule (because __name__ is a different string). You are now configuring only the logger for the main module.
But loggers are hierarchial, so the easiest way forward is to configure the root logger with the handlers:
root_logger = logging.getLogger('')
...
root_logger.addHandler(h1)
Every other logger is a child of the root logger so messages will "bubble up" by default to the root and use the handlers and levels configured there.
Some simple codes to illustrate my question:
/src
-- t1.py
-- t2.py
-- test.py
The test.py
import logging
import t1
import t2
def main():
# I need to set logging level for each module, so can't call logging.basicConfig
# logging.basicConfig(level=logging.DEBUG)
t1.foo()
t2.foo2()
if __name__ == "__main__":
main()
t1.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo():
logger.info('foo log message')
t2.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo2():
logger.info('foo2 log message')
When I run python3 test.py I won't get logs. But if I call logging.basicConfig(level=logging.DEBUG) in test.py I will get logs.
So what did I do wrong ?
--- update ---
What I can add to the answer I got is that the root logger's default level is 'WARNING'. So I did not get any output.
logger.setLevel(logging.DEBUG) sets level of messages processable by this logger.
logging.basicConfig(level=logging.DEBUG) does more than that. It creates a handler for the root logger which prints the logging records to stdout.
An example from the cookbook:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
code:
import logging
logging.root.setLevel(logging.NOTSET)
logger = logging.getLogger('name')
logger.setLevel(logging.NOTSET)
def func():
logger.info('1')
print(logging.getLevelName(logger.getEffectiveLevel()))
print(logger.handlers)
**some code**
logger.info('2')
print(logging.getLevelName(logger.getEffectiveLevel()))
print(logger.handlers)
logger.setLevel(logging.NOTSET)
logger.info('3')
print(logging.getLevelName(logger.getEffectiveLevel()))
print(logger.handlers)
output:
1
NOTSET
[]
WARNING
[]
WARNING
[]
I assume some code change the level of the logger (which I'm sure not the logger with name 'name').
But setting the level after some code does not work.
How should I set it back?
You are only setting the level on the logger itself. If a loggers level is NOTSET the effective level includes all the parent loggers, since logs propagate up the logger hierarchy. If the root logger has level WARNING and the 'name' logger has level NOTSET the effective level of the 'name' logger is WARNING. ('root' is an ancestor of every logger)
The root logger can be accessed with any of these: logging.root or logging.getLogger() or logging.getLogger('root'). The level can be set the same way it is set on any logger:
root = logging.getLogger()
root.setLevel(logging.NOTSET)
I want to log to a single log file from main and all sub modules.
The log messages send from a main file, where I define the logger, work as expected. But the ones send from a call to an imported function are missing.
It is working if I use logging.basicConfig as in Example 1 below.
But the second example which allows for more custom settings does not work.
Any ideas why?
# in the submodule I have this code
import logging
logger = logging.getLogger(__name__)
EXAMPLE 1 - Working
Here I create two handlers and just pass them to basicConfig:
# definition of root looger in main module
formatter = logging.Formatter(fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s", datefmt="%Y.%m.%d %H:%M:%S")
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
handler2 = logging.StreamHandler(stream=None)
handler2.setFormatter(formatter)
handler2.setLevel(logging.DEBUG)
logging.basicConfig(handlers=[handler, handler2], level=logging.DEBUG)
logger = logging.getLogger(__name__)
EXAMPLE 2 - Not working
Here I create two handlers and addHandler() them to the root logger:
# definition of root looger in main module
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
#handler.setLevel(logging.ERROR)
logger.addHandler(handler)
handler = logging.StreamHandler(stream=None)
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
You need to configure the (one and only) root logger in the main module of your software. This is done by calling
logger = logging.getLogger() #without arguments
instead of
logger = logging.getLogger(__name__)
(Python doc on logging)
The second example creates a separate, child logger with the name of your script.
If there are no handlers defined in the submodules, the log message is being passed down to the root logger to handle it.
A related question can be found here:
Python Logging - How to inherit root logger level & handler
I would like to log my python script that uses elasticsearch-py. In particular, I want to have three logs:
General log: log INFO and above both to the stdout and to a file.
ES log: only ES related messages only to a file.
ES tracing log: Extended ES logging (curl queries and their output for instance) only to a file.
Here is what I have so far:
import logging
import logging.handlers
es_logger = logging.getLogger('elasticsearch')
es_logger.setLevel(logging.INFO)
es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log',
maxBytes=0.5*10**9,
backupCount=3)
es_logger.addHandler(es_logger_handler)
es_tracer = logging.getLogger('elasticsearch.trace')
es_tracer.setLevel(logging.DEBUG)
es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log',
maxBytes=0.5*10**9,
backupCount=3)
es_tracer.addHandler(es_tracer_handler)
logger = logging.getLogger('mainLog')
logger.setLevel(logging.DEBUG)
# create file handler
fileHandler = logging.handlers.RotatingFileHandler('top-camps.log',
maxBytes=10**6,
backupCount=3)
fileHandler.setLevel(logging.INFO)
# create console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setLevel(logging.INFO)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
consoleHandler.setFormatter(formatter)
fileHandler.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(consoleHandler)
logger.addHandler(fileHandler)
My problem is that INFO messages of es_logger are displayed also on the terminal. As a matter of fact the log messages are saved to the right files!
If I remover the part related to logger, then the ES logging works fine, i.e. only saved to the corresponding files. But then I don't have the other part.... What is it that I'm doing wrong with the last part of the settings?
Edit
Possible hint: In the sources of elasticsearch-py there's a logger named logger. Could it be that it conflicts with mine? I tried to change the name of logger to main_logger in the lines above but it didn't help.
Possible hint 2: If I replace logger = logging.getLogger('mainLog') with logger = logging.getLogger(), then the format of the output to the console of es_logger changes and becomes identical to the one defined in the snippet.
I think you are being hit by the somewhat confusing logger hierarchy propagation. Everything that is logged in "elasticsearch.trace" that passes the loglevel of that logger, will propagate first to the "elasticsearch" logger and then to the root ("") logger. Note that once the message passes the loglevel of the "elasticsearch.trace" logger, the loglevels of the parents ("elasticsearch" and root) are not checked, but all messages will be sent to the handlers. (The handlers themselves have log levels that do apply.)
Consider the following example that illustrates the issue, and a possible solution:
import logging
# The following line will basicConfig() the root handler
logging.info('DUMMY - NOT SEEN')
ll = logging.getLogger('foo')
ll.setLevel('DEBUG')
ll.addHandler(logging.StreamHandler())
ll.debug('msg1')
ll.propagate = False
ll.debug('msg2')
Output:
msg1
DEBUG:foo:msg1
msg2
You see that "msg1" is logged both by the "foo" logger, and its parent, the root logger (as "DEBUG:foo:msg1"). Then, when propagation is turned off ll.propagate = False before "msg2", the root logger no longer logs it. Now, if you were to comment out the first line (logging.info("DUMMY..."), then the behavior would change so that the root logger line would not be shown. This is because the logging module top level functions info(), debug() etc. configure the root logger with a handler when no handler has yet been defined. That is also why you see different behavior in your example when you modify the root handler by doing logger = logging.getLogger().
I can't see in your code that you would be doing anything to the root logger, but as you see, a stray logging.info() or the like in your code or library code would cause a handler to be added.
So, to answer your question, I would set logger.propagate = False to the loggers where it makes sense for you and where you want propagation, check that the log level of the handlers themselves are as you want them.
Here is an attempt:
es_logger = logging.getLogger('elasticsearch')
es_logger.propagate = False
es_logger.setLevel(logging.INFO)
es_logger_handler=logging.handlers.RotatingFileHandler('top-camps-base.log',
maxBytes=0.5*10**9,
backupCount=3)
es_logger.addHandler(es_logger_handler)
es_tracer = logging.getLogger('elasticsearch.trace')
es_tracer.propagate = False
es_tracer.setLevel(logging.DEBUG)
es_tracer_handler=logging.handlers.RotatingFileHandler('top-camps-full.log',
maxBytes=0.5*10**9,
backupCount=3)
es_tracer.addHandler(es_tracer_handler)
logger = logging.getLogger('mainLog')
logger.propagate = False
logger.setLevel(logging.DEBUG)
# create file handler
fileHandler = logging.handlers.RotatingFileHandler('top-camps.log',
maxBytes=10**6,
backupCount=3)
fileHandler.setLevel(logging.INFO)
# create console handler
consoleHandler = logging.StreamHandler()
consoleHandler.setLevel(logging.INFO)
# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
consoleHandler.setFormatter(formatter)
fileHandler.setFormatter(formatter)
# add the handlers to logger
logger.addHandler(consoleHandler)
logger.addHandler(fileHandler)