In my main.py file I initialize the logger as follows:
# File: main.py
import logging
logger = logging.getLogger(__name__)
import submodule
#...
def main():
logger.debug(f'Test')
if __name__ == '__main__':
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('[%(levelname)s | %(name)s] %(message)s')
# The handler h1 logs only debug and info to stdout
h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)
# The handler h2 logs only warning, error and exception to stderr
h2 = logging.StreamHandler()
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
logger.addHandler(h1)
logger.addHandler(h2)
main()
When I use the logger in the main script, everything works correctly.
But when I use it in a submodule like this:
# File: submodule.py
import logging
logger = logging.getLogger(__name__)
#...
logger.info('Test 1')
logger.error('Test 2')
I should see the following output (or something like this):
[DEBUG | __main__] Test
[INFO | submodule] Test 1
[ERROR | submodule] Test 2
But I get this output:
[DEBUG | __main__] Test
Test 2
To me it looks like the logger in submodules did not use the config of the logger in the main module. How to fix that?
This is because
logger = logging.getLogger(__name__)
will return different loggers in the main module and the submodule (because __name__ is a different string). You are now configuring only the logger for the main module.
But loggers are hierarchial, so the easiest way forward is to configure the root logger with the handlers:
root_logger = logging.getLogger('')
...
root_logger.addHandler(h1)
Every other logger is a child of the root logger so messages will "bubble up" by default to the root and use the handlers and levels configured there.
Related
Some simple codes to illustrate my question:
/src
-- t1.py
-- t2.py
-- test.py
The test.py
import logging
import t1
import t2
def main():
# I need to set logging level for each module, so can't call logging.basicConfig
# logging.basicConfig(level=logging.DEBUG)
t1.foo()
t2.foo2()
if __name__ == "__main__":
main()
t1.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo():
logger.info('foo log message')
t2.py
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
def foo2():
logger.info('foo2 log message')
When I run python3 test.py I won't get logs. But if I call logging.basicConfig(level=logging.DEBUG) in test.py I will get logs.
So what did I do wrong ?
--- update ---
What I can add to the answer I got is that the root logger's default level is 'WARNING'. So I did not get any output.
logger.setLevel(logging.DEBUG) sets level of messages processable by this logger.
logging.basicConfig(level=logging.DEBUG) does more than that. It creates a handler for the root logger which prints the logging records to stdout.
An example from the cookbook:
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
I want to log to a single log file from main and all sub modules.
The log messages send from a main file, where I define the logger, work as expected. But the ones send from a call to an imported function are missing.
It is working if I use logging.basicConfig as in Example 1 below.
But the second example which allows for more custom settings does not work.
Any ideas why?
# in the submodule I have this code
import logging
logger = logging.getLogger(__name__)
EXAMPLE 1 - Working
Here I create two handlers and just pass them to basicConfig:
# definition of root looger in main module
formatter = logging.Formatter(fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s", datefmt="%Y.%m.%d %H:%M:%S")
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
handler2 = logging.StreamHandler(stream=None)
handler2.setFormatter(formatter)
handler2.setLevel(logging.DEBUG)
logging.basicConfig(handlers=[handler, handler2], level=logging.DEBUG)
logger = logging.getLogger(__name__)
EXAMPLE 2 - Not working
Here I create two handlers and addHandler() them to the root logger:
# definition of root looger in main module
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
#handler.setLevel(logging.ERROR)
logger.addHandler(handler)
handler = logging.StreamHandler(stream=None)
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
You need to configure the (one and only) root logger in the main module of your software. This is done by calling
logger = logging.getLogger() #without arguments
instead of
logger = logging.getLogger(__name__)
(Python doc on logging)
The second example creates a separate, child logger with the name of your script.
If there are no handlers defined in the submodules, the log message is being passed down to the root logger to handle it.
A related question can be found here:
Python Logging - How to inherit root logger level & handler
I have put the following in my config.py:
import time
import logging
#logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S', level=logging.INFO)
logFormatter = logging.Formatter('%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
rootLogger = logging.getLogger()
rootLogger.setLevel(logging.INFO)
fileHandler = logging.FileHandler("{0}.log".format(time.strftime('%Y%m%d%H%M%S')))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
and then I am doing
from config import *
in all of my scripts and imported files.
Unfortunately, this causes multiple log files created.
How to fix this? I wan't centralized config.py with logging configured both to console and file.
Case 1: Independent Scripts / Programs
In case we are talking about multiple, independent scripts, that should have logging set up in the same way: I would say, each independent application should have its own log. If you definitively do not want this, you would have to
make sure that all applications have the same log file name (e.g. create a function in config.py, with a parameter "timestamp", which is provided by your script(s)
specify the append filemode for the fileHandler
make sure that config.py is not called twice somewhere, as you would add the log handlers twice, which would result in each log message being printed twice.
Case 2: One big application consisting of modules
In case we are talking about one big application, consisting of modules, you could adopt a structure like the following:
config.py:
def set_up_logging():
# your logging setup code
module example (some_module.py):
import logging
def some_function():
logger = logging.getLogger(__name__)
[...]
logger.info('sample log')
[...]
main example (main.py)
import logging
from config import set_up_logging
from some_module import some_function
def main():
set_up_logging()
logger = logging.getLogger(__name__)
logger.info('Executing some function')
some_function()
logger.info('Finished')
if __name__ == '__main__':
main()
Explanation:
With the call to set_up_logging() in main() you configure your applications root logger
each module is called from main(), and get its logger via logger = logging.getLogger(__name__). As the modules logger are in the hierarchy below the root logger, those loggings get "propagated up" to the root logger and handled by the handlers of the root logger.
For more information see Pythons logging module doc and/or the logging cookbook
I'm having trouble getting child loggers named properly in python (2.7). I have the following file structure:
-mypackage
-__init__.py
-main.py
-log
-__init__.py
-logfile.log
-src
-__init__.py
-logger.py
-otherfile.py
The contents of main.py are:
import logging
import src.logger
from src.otherfile import Foo
logger = logging.getLogger(__name__)
logger.info('Logging from main')
foo = Foo()
The contents of otherfile.py are:
import logging
class Foo():
def __init__(self):
self.logger = logging.getLogger(__name__)
self.logger.info('Logging from class in otherfile')
The contents of logger.py are:
import os
import logging
logdir = os.path.dirname(__file__)
logfile = os.path.join(logdir, '../log/controller.log')
logger = logging.getLogger('__main__')
logger.setLevel(logging.DEBUG)
fh = logging.FileHandler(logfile)
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - $(name)s - %(levelname)s: %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
logger.info('logging from logger.py')
I used logging.getLogger(__name__) in each file based on the docs. The exception is logger.py, where I name the logger __main__ so that it would run from top down and not try to derive everything from a buried file.
When I run main.py, it logs correctly from logger.py and main.py, but the logger from otherfile.py isn't derived correctly from the main logger.
How do I get the logger in otherfile.py to derive from my main logger?
In logger.py you are configuring the "__main__" logger. I was tricked by the fact that in main.py you use __name__. Since you are invoking python main.py, __name__ evaluates to "__main__". Right.
This can become a problem since when imported (instead of executed), main.py's logger won't be "__main__" but "main". It can be fixed by making your package executable: rename main.py to __main__.py and running your package like this:
python -m mypackage
This way, logger names (actually module __name__'s) will remain consistent.
That said, in no way the logger that you configure in logger.py is a parent of the logger in otherfile.py. The real parent of that logger is called "mypackage" but you haven't configured it, so it's logs are invisible.
You have several choices, you can configure (set log level, handler and formatter):
the root logger : logger = logging.getLogger()
mypackage's logger : logger = logger.getLogger(__name__) in mypackage.__init__
... or go down to the level of granularity you wish.
You can easily create hierarchies of loggers by calling separating levels of loggers by a dot ("."). For example, the logger returned by calling logging.getLogger('__main__.' + __name__) inherits all properties from the logger returned by logging.getLogger('__main__'). This behaviour is described here: https://docs.python.org/2/library/logging.html#module-level-functions.
I am having some difficulties using python's logging. I have two files, main.py and mymodule.py. Generally main.py is run, and it will import mymodule.py and use some functions from there. But sometimes, I will run mymodule.py directly.
I tried to make it so that logging is configured in only 1 location, but something seems wrong.
Here is the code.
# main.py
import logging
import mymodule
logger = logging.getLogger(__name__)
def setup_logging():
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
if __name__ == '__main__':
setup_logging()
logger.info('calling mymodule.myfunc()')
mymodule.myfunc()
and the imported module:
# mymodule.py
import logging
logger = logging.getLogger(__name__)
def myfunc():
msg = 'myfunc is being called'
logger.info(msg)
print('done with myfunc')
if __name__ == '__main__':
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info('myfunc was executed directly')
myfunc()
When I run the code, I see this output:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
done with myfunc
But I expect to see this:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
INFO: 2016-07-14 18:15:09 myfunc(8) -- myfunc is being called
done with myfunc
Anybody have any idea why the second logging.info call doesn't print to screen? thanks in advance!
Loggers exist in a hierarchy, with a root logger (retrieved with logging.getLogger(), no arguments) at the top. Each logger inherits configuration from its parent, with any configuration on the logger itself overriding the inherited configuration. In this case, you are never configuring the root logger, only the module-specific logger in main.py. As a result, the module-specific logger in mymodule.py is never configured.
The simplest fix is probably to use logging.basicConfig in main.py to set options you want shared by all loggers.
Chepner is correct. I got absorbed into this problem. The problem is simply in your main script
16 log = logging.getLogger() # use this form to initialize the root logger
17 #log = logging.getLogger(__name__) # never use this one
If you use line 17, then your imported python modules will not log any messages
In you submodule.py
import logging
logger = logging.getLogger()
logger.debug("You will not see this message if you use line 17 in main")
Hope this posting can help someone who got stuck on this problem.
While the logging-package is conceptually arranged in a namespace hierarchy using dots as separators, all loggers implicitly inherit from the root logger (like every class in Python 3 silently inherits from object). Each logger passes log messages on to its parent.
In your case, your loggers are incorrectly chained. Try adding print(logger.name) in your both modules and you'll realize, that your instantiation of logger in main.py is equivalent to
logger = logging.getLogger('__main__')
while in mymodule.py, you effectively produce
logger = logging.getLogger('mymodule')
The call to log INFO-message from myfunc() passes directly the root logger (as the logger in main.py is not among its ancestors), which has no handler set up (in this case the default message dispatch will be triggered, see here)