I want logger to print INFO messages from all my code but not from 3rd party libraries. This is discussed in multiple places, but the suggested solution does not work for me. Here is my simulation of external library, extlib.py:
#!/usr/bin/env python3
from logging import info
def f():
info("i am extlib f()")
My module:
#!/usr/bin/env python3
import logging
from logging import info
import extlib
logging.basicConfig(level=logging.INFO)
info("I am mymodule")
extlib.f()
Output:
INFO:root:I am mymodule
INFO:root:i am extlib f()
My attempt to only enable INFO for local module:
#!/usr/bin/env python3
import logging
from logging import info
import extlib
logging.getLogger(__name__).setLevel(logging.INFO)
info("I am mymodule")
extlib.f()
Output: nothing
Desired output:
INFO:root:I am mymodule
What am I doing wrong?
The problem is that they aren't using the logger class in the external library. If they were then you could filter it out. I'm not certain you can stop them from logging information since they are using the info function call. Here is a workaround though.
Suppressing output of module calling outside library
Update:
here's how you do loggers
import logging
# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
Related
Some simple codes to illustrate my question:
/src
-- t1.py
-- t2.py
-- test.py
The test.py
import logging
import t1
import t2
def main():
# I need to set logging level for each module, so can't call logging.basicConfig
# logging.basicConfig(level=logging.DEBUG)
t1.foo()
t2.foo2()
if __name__ == "__main__":
main()
t1.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo():
logger.info('foo log message')
t2.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo2():
logger.info('foo2 log message')
When I run python3 test.py I won't get logs. But if I call logging.basicConfig(level=logging.DEBUG) in test.py I will get logs.
So what did I do wrong ?
--- update ---
What I can add to the answer I got is that the root logger's default level is 'WARNING'. So I did not get any output.
logger.setLevel(logging.DEBUG) sets level of messages processable by this logger.
logging.basicConfig(level=logging.DEBUG) does more than that. It creates a handler for the root logger which prints the logging records to stdout.
An example from the cookbook:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
I'm not very familiar with python logging and I am trying to get it to log output to the console. I've gotten it to work but it seems that the output is seen twice in the console and I'm not sure why. I've looked at the other questions asked here about similar situations but I could not find anything that helped me.
I have three modules, lets call them ma, mb, mc and a main. The main imports these 3 modules and calls their functions.
In each module I set up a logger like so:
ma.py
import logging
logger = logging.getLogger('test.ma') #test.mb for mb.py/test.mc for mc.py
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#right before the end of each module, after I'm finished logging what I need.
logger.removeHandler(console_log)
mb.py
import logging
logger = logging.getLogger('test.mb')
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#end of file
logger.removeHandler(console_log)
mc.py
import logging
logger = logging.getLogger('test.mc')
logger.setLevel(logging.DEBUG)
console_log = logging.StreamHandler()
console_log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
...
...
#end of file
logger.removeHandler(console_log)
The main problem I'm having is that output is printing twice and for some parts of the program it is not formatted. Any help would appreciated, thanks!
A better approach to using a global logger is to use a function that wraps the call to logging.getLogger:
import logging
def get_logger(name):
logger = logging.getLogger(name)
if not logger.handlers:
# Prevent logging from propagating to the root logger
logger.propagate = 0
console = logging.StreamHandler()
logger.addHandler(console)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
console.setFormatter(formatter)
return logger
def main():
logger = get_logger(__name__)
logger.setLevel(logging.DEBUG)
logger.info("This is an info message")
logger.error("This is an error message")
logger.debug("This is a debug message")
if __name__ == '__main__':
main()
Output
2017-03-09 12:02:41,083 - __main__ - This is an info message
2017-03-09 12:02:41,083 - __main__ - This is an error message
2017-03-09 12:02:41,083 - __main__ - This is a debug message
Note: Using a function also allows us to set logger.propagate to False to prevent log messages from being sent to the root logger. This is almost certainly the cause of the duplicated output you are seeing.
Update: After calling get_logger, the desired level must be set by calling logger.setLevel. If this is not done, then only error messages will be seen.
To log in multiple modules, to one global file, create a logging instance and then get that instance in whichever modules you like. So for example in main, i would have:
import logging
logging.basicConfig(filename='logfile.log', level=logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(message)s')
logger = logging.getLogger('Global Log')
Then, in every module that I want to be able to append to this log, I have:
import logging
logger = logging.getLogger('Global Log')
Now everytime you access the logger in your module, you are accessing the same logging instance. This way, you should only have to set up and format the logger once.
More about logging instance in the docs:
https://docs.python.org/2/library/logging.html#logging.getLogger
In my case I was seeing logging twice due to the way /dev/log redirects records to the rsyslog daemon I have locally running.
Using a
handler = logging.handlers.SysLogHandler(address='/run/systemd/journal/syslog')
solved my problem of double logging.
I have a program in which I wrote logs both as info and debug.
Since the debug contains also calls to slow functions, my program is running slow even if I set debugging to INFO.
Is it possible to completely skip those line from computation?
in the next example 10 seconds have to pass before the info log is executed.
import logging.handlers
import sys
import time
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging_stream_handler = logging.StreamHandler(sys.stdout)
logging_stream_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s --- %(message)s'))
logger.addHandler(logging_stream_handler)
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
logger.info("ccc")
You could check if the logger is enabled for such a level with the isEnabledFor method:
if logger.isEnabledFor(logging.DEBUG):
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
You shouldn't have logging inside debug commands. If you must, then in order to skip it you must branch your code.
import logging.handlers
import sys
import time
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logging_stream_handler = logging.StreamHandler(sys.stdout)
logging_stream_handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s --- %(message)s'))
logger.addHandler(logging_stream_handler)
if logger.isEnabledFor(logging.DEBUG):
logger.debug("aaa", time.sleep(10))
logger.debug("bbb")
logger.info("ccc")
In my programs, I generally want to log to both a file and to the screen. If I import a module, I want the calls to logger within that module's functions/classes to also result in a log to a file and the screen. Test_module_A does this, and test_module_B logs to the file but not the screen. Is this because the loggers in the modules propagate up to the root logger which through basicConfig is only setup to log to the file? I just want to make sure I'm using logging properly, as I am going to rewrite all my code to use this type of logging rather than "print." Going forward, I am only going to use logging and not print, as I figure logging is more flexible. Is this a good idea?
main.py
import logging
logger = logging.getLogger(__name__)
task_file='temp.txt'
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(filename=task_file, format=format_str, datefmt=datefmt_str,
level=logging.INFO)
console = logging.StreamHandler()
#console.setLevel(logging.INFO)
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("from main.py")
import test_module_A
import test_module_B
test_module_A.py
import logging
logger = logging.getLogger(__name__)
format_str = '%(asctime)s %(module)s %(levelname)s: %(message)s'
datefmt_str = '%m/%d/%Y %I:%M:%S%p'
console = logging.StreamHandler()
formatter = logging.Formatter(format_str)
formatter.datefmt = datefmt_str
console.setFormatter(formatter)
logger.addHandler(console)
logger.info("upon import of test_module_A")
test_module_B.py
import logging
logger = logging.getLogger(__name__)
logger.info("upon import of test_module_B")
Upon running main.py:
#screen output
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
# test.txt
05/06/2014 12:36:33AM main INFO: from main.py
05/06/2014 12:36:33AM test_module_A INFO: upon import of test_module_A
05/06/2014 12:36:33AM test_module_B INFO: upon import of test_module_B
It's not good practice to configure logging (add handlers, etc.) in each module: the way to do it is to just have
logger = logging.getLogger(__name__)
and actual logging calls in each module as required, and only have the configuration done in one place (called from the main script early on)
if __name__ == '__main__':
logging.basicConfig(level=..., format=..., ...) #or other configuration code
handler = logging.FileHandler(...)
# set formatters, filters etc.
logging.getLogger().addHandler(handler) # add to root logger
main()
The basicConfig() call adds to the root logger a handler which writes to sys.stderr, and together with the FileHandler added explicitly, the logs from all modules should be written to both screen and file.
I have my main script that interprets cli commands with argparse and then starts the application by calling the according stuff form an other module (made by myself).
My question now is how I can attach a handler to the logger from that module. The logger is retrieved with
logger = logging.getLogger(__name__)
I hence put following in my main script:
consoleHandler = logging.StreamHandler()
logger = logging.getLogger('MyModule')
logger.addHandler(consoleHandler)
However there is 0 logging output from 'MyModule'. Log levels are correct, eg there should be an output.
In MyModule I have the following:
logging.getLogger(__name__).addHandler(logging.NullHandler())
However removing that makes no difference.
So how can I correctly attach a handler to the logger of MyModule?
The best example of adding handler to logger is the one from Docs, See below.
Did you added consoleHandler with setLevel() and the Formatter() ?
As for this line:
logging.getLogger(__name__).addHandler(logging.NullHandler())
Why are you using it this way? I would recommend to follow the next code for adding an handler the right way.
import logging
# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
The reason you're getting no output is most likely that you haven't set a level on the logger, so it will default to WARNING. Try doing logger.setLevel(logging.DEBUG) and see if that causes output to be produced. I know you said that log levels are correct, but I can't see any other reason why no output would be produced. You can try posting a small self-contained script that demonstrates the problem, if you like.
NullHandler only needs to be added for library modules to cater for their use in an application that doesn't configure logging. It is not needed in modules of an application that configures logging.
In general, you can add a handler to the root logger and it will be used by loggers in all modules.
Update: The comment to Kobi K's answer indicates that a level has been set on the handler - this is not enough, as you need to set a level on the logger, whose level is checked first. Only if the event is allowed through by the logger's level will the handlers (and their levels) come into play.