Develop Reference

Python logger not printing debug messages, despite explicit handler

I have a logger in on of my files which has a handler attached to it and it's level has been set to debug. Despite that, when running my program, the debug statement is not printed to the console. The root logger is still set to warning, but I understood that if I add a handler to the logger, the log is passed to that handler and logged before being passed to the parent loggers (which is eventually a null logger). It doesn't seem that is the case. For context here is the code in the file:
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
class OpenBST:
app_data_folder = Path(user_data_dir(appname=lib_info.lib_name,
appauthor="HydrOffice"))
def __init__(self,
progress: CliProgress = CliProgress(use_logger=True),
app_data_path: Path = app_data_folder) -> None:
app_data_path.mkdir(exist_ok=True, parents=True)
self.progress = progress
self._prj = None
self._app_info = OpenBSTInfo(app_data_path=app_data_path)
self.current_project = None
logging.debug("App instance started")
And below is where it's called in an example script:
from pathlib import Path
from hyo2.openbst.lib.openbst import OpenBST
logging.basicConfig()
logger = logging.getLogger(__name__)
project_directory = Path(os.path.expanduser("~/Documents/openbst_projects"))
project_name = "test_project"
# Create App instance
obst = OpenBST()
Why doesn't the logger.debug('App instance started') not print out to the console?
EDIT:
The code below includes the suggestion from #Jesse R
__init__ was modified as such:
class OpenBST:
app_data_folder = Path(user_data_dir(appname=lib_info.lib_name,
appauthor="HydrOffice"))
def __init__(self,
progress: CliProgress = CliProgress(use_logger=True),
app_data_path: Path = app_data_folder) -> None:
app_data_path.mkdir(exist_ok=True, parents=True)
logger = logging.getLogger(__name__)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
self.progress = progress
self._prj = None
self._app_info = OpenBSTInfo(app_data_path=app_data_path)
self.current_project = None
logger.debug("App instance started")
No output is generated (exit code 0).
My understanding was a handler attached to a logger would execute before passing log up the chain (where the root is still set to warning).

You call logging.debug("App instance started"), which is not part of the logger that you declare from getLogger. You can set the debug level universally for logging with
logging.basicConfig(level=logging.DEBUG)
also calling logger = logging.getLogger(__name__) outside of the class does not inherit correctly, since you're not passing it but instead use logging. You can create a new logger by moving that declaration inside of the class.
For Example:
import logging
class SampleClass:
def __init__(self):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.info('will log')
logging.info('will not log')
SampleClass()
Running:
$ python logtest.py
INFO:__main__:will log

How to configure logging to all scripts in the prject?

I have put the following in my config.py:
import time
import logging
#logging.basicConfig(format='%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S', level=logging.INFO)
logFormatter = logging.Formatter('%(asctime)s %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
rootLogger = logging.getLogger()
rootLogger.setLevel(logging.INFO)
fileHandler = logging.FileHandler("{0}.log".format(time.strftime('%Y%m%d%H%M%S')))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
and then I am doing
from config import *
in all of my scripts and imported files.
Unfortunately, this causes multiple log files created.
How to fix this? I wan't centralized config.py with logging configured both to console and file.

Case 1: Independent Scripts / Programs
In case we are talking about multiple, independent scripts, that should have logging set up in the same way: I would say, each independent application should have its own log. If you definitively do not want this, you would have to
make sure that all applications have the same log file name (e.g. create a function in config.py, with a parameter "timestamp", which is provided by your script(s)
specify the append filemode for the fileHandler
make sure that config.py is not called twice somewhere, as you would add the log handlers twice, which would result in each log message being printed twice.
Case 2: One big application consisting of modules
In case we are talking about one big application, consisting of modules, you could adopt a structure like the following:
config.py:
def set_up_logging():
# your logging setup code
module example (some_module.py):
import logging
def some_function():
logger = logging.getLogger(__name__)
[...]
logger.info('sample log')
[...]
main example (main.py)
import logging
from config import set_up_logging
from some_module import some_function
def main():
set_up_logging()
logger = logging.getLogger(__name__)
logger.info('Executing some function')
some_function()
logger.info('Finished')
if __name__ == '__main__':
main()
Explanation:
With the call to set_up_logging() in main() you configure your applications root logger
each module is called from main(), and get its logger via logger = logging.getLogger(__name__). As the modules logger are in the hierarchy below the root logger, those loggings get "propagated up" to the root logger and handled by the handlers of the root logger.
For more information see Pythons logging module doc and/or the logging cookbook

python logging - message not showing up in child

I am having some difficulties using python's logging. I have two files, main.py and mymodule.py. Generally main.py is run, and it will import mymodule.py and use some functions from there. But sometimes, I will run mymodule.py directly.
I tried to make it so that logging is configured in only 1 location, but something seems wrong.
Here is the code.
# main.py
import logging
import mymodule
logger = logging.getLogger(__name__)
def setup_logging():
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
if __name__ == '__main__':
setup_logging()
logger.info('calling mymodule.myfunc()')
mymodule.myfunc()
and the imported module:
# mymodule.py
import logging
logger = logging.getLogger(__name__)
def myfunc():
msg = 'myfunc is being called'
logger.info(msg)
print('done with myfunc')
if __name__ == '__main__':
# only cofnigure logger if script is main module
# configuring logger in multiple places is bad
# only top-level module should configure logger
if not len(logger.handlers):
logger.setLevel(logging.DEBUG)
# create console handler with a higher log level
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(levelname)s: %(asctime)s %(funcName)s(%(lineno)d) -- %(message)s', datefmt = '%Y-%m-%d %H:%M:%S')
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info('myfunc was executed directly')
myfunc()
When I run the code, I see this output:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
done with myfunc
But I expect to see this:
$>python main.py
INFO: 2016-07-14 18:13:04 <module>(22) -- calling mymodule.myfunc()
INFO: 2016-07-14 18:15:09 myfunc(8) -- myfunc is being called
done with myfunc
Anybody have any idea why the second logging.info call doesn't print to screen? thanks in advance!

Loggers exist in a hierarchy, with a root logger (retrieved with logging.getLogger(), no arguments) at the top. Each logger inherits configuration from its parent, with any configuration on the logger itself overriding the inherited configuration. In this case, you are never configuring the root logger, only the module-specific logger in main.py. As a result, the module-specific logger in mymodule.py is never configured.
The simplest fix is probably to use logging.basicConfig in main.py to set options you want shared by all loggers.

Chepner is correct. I got absorbed into this problem. The problem is simply in your main script
16 log = logging.getLogger() # use this form to initialize the root logger
17 #log = logging.getLogger(__name__) # never use this one
If you use line 17, then your imported python modules will not log any messages
In you submodule.py
import logging
logger = logging.getLogger()
logger.debug("You will not see this message if you use line 17 in main")
Hope this posting can help someone who got stuck on this problem.

While the logging-package is conceptually arranged in a namespace hierarchy using dots as separators, all loggers implicitly inherit from the root logger (like every class in Python 3 silently inherits from object). Each logger passes log messages on to its parent.
In your case, your loggers are incorrectly chained. Try adding print(logger.name) in your both modules and you'll realize, that your instantiation of logger in main.py is equivalent to
logger = logging.getLogger('__main__')
while in mymodule.py, you effectively produce
logger = logging.getLogger('mymodule')
The call to log INFO-message from myfunc() passes directly the root logger (as the logger in main.py is not among its ancestors), which has no handler set up (in this case the default message dispatch will be triggered, see here)

add custom handler with Python logging

I have been working on this almost all day couldn't figure what I am missing. I am trying to add a custom handler to emit all log data into a GUI session. It works but the handler doesn't extend to the submodules and just emits records from the main module. Here is a small snippet I tried
I have two files
# main.py
import logging
import logging_two
def myapp():
logger = logging.getLogger('myapp')
logging.basicConfig()
logger.info('Using myapp')
ch = logging.StreamHandler()
logger.addHandler(ch)
logging_two.testme()
print logger.handlers
myapp()
Second module
#logging_two
import logging
def testme():
logger = logging.getLogger('testme')
logger.info('IN test me')
print logger.handlers
I would expect the logger in logging_two.testme to have the handler I have added in the main module. I looked at the docs to me it seems this should work but I am not sure if I got it wrong?
the result I get is
[]
[<logging.StreamHandler object at 0x00000000024ED240>]

In myapp() you are adding the handler to the logger named 'myapp'. Since testme() is getting the logger named 'testme' it does not have the handler since it is a different part of the logging hierarchy.
If you just have logger = logger.getLogger() in myapp() then it would work since you are adding the handler to the root of the hierarchy.
Check out the python logging docs.

Python - Avoid passing logger reference between functions?

I have a simple Python script that uses the in-built logging.
I'm configuring logging inside a function. Basic structure would be something like this:
#!/usr/bin/env python
import logging
import ...
def configure_logging():
logger = logging.getLogger("my logger")
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
def count_parrots():
...
logger.debug??
if __name__ == '__main__':
logger = configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()
I can call logger fine from inside __main__. However, how do I call logger from inside the count_parrots() function? What's the most pythonic way of handling configuring a logger like this?

You can either use the root (default) logger, and thus the module level functions logging.debug, ... or get your logger in the function using it.
Indeed, the getLogger function is a factory-like function with a registry (singleton like), i.e. it always returns the same instance for the given logger name.
You can thus get your logger in count_parrots by simply using
logger = logging.getLogger("my logger")
at the beginning. However, the convention is to use a dotted hierarchical name for your logger. See http://docs.python.org/library/logging.html#logging.getLogger
EDIT:
You can use a decorator to add the logging behaviour to your individual functions, for example:
def debug(loggername):
logger = logging.getLogger(loggername)
def log_(enter_message, exit_message=None):
def wrapper(f):
def wrapped(*args, **kargs):
logger.debug(enter_message)
r = f(*args, **kargs)
if exit_message:
logger.debug(exit_message)
return r
return wrapped
return wrapper
return log_
my_debug = debug('my.logger')
#my_debug('enter foo', 'exit foo')
def foo(a, b):
return a+b
you can "hardcode" the logger name and remove the top-level closure and my_debug.

You can just do :
logger = logging.getLogger("my logger")
in your count_parrots() method. When you pass the name that was used earlier (i.e. "my logger") the logging module would return the same instance that was created corresponding to that name.
Update: From the logging tutorial
(emphais mine)
getLogger() returns a reference to a
logger instance with the specified
name if it is provided, or root if
not. The names are period-separated
hierarchical structures. Multiple
calls to getLogger() with the same
name will return a reference to the
same logger object.

The typical way to handle logging is to have a per-module logger stored in a global variable. Any functions and methods within that module then just reference that same logger instance.
This is discussed briefly in the intro to the advance logging tutorial in the documentation:
http://docs.python.org/howto/logging.html#advanced-logging-tutorial
You can pass logger instances around as parameters, but doing so is typically rare.

I got confused by how global variables work in Python. Within a function you only need to declare global logger if you were doing something like logger = logging.getLogger("my logger") and hoping to modify the global logger.
So to modify your example, you can create a global logger object at the start of the file. If your module can be imported by another one, you should add the NullHandler so that if the importer of the library doesn't want logging enabled, they don't have any issues with your lib (ref).
#!/usr/bin/env python
import logging
import ...
logger = logging.getLogger("my logger").addHandler(logging.NullHandler())
def configure_logging():
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
def count_parrots():
...
logger.debug('counting parrots')
...
return parrots
if __name__ == '__main__':
configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()

If you don't need the log messages on your console, you can use in a minimalist way.
Alternatively you can use tail -f myapp.log to see the messages on the console.
import logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', \
filename='myapp.log', \
level=logging.INFO)
def do_something():
logging.info('Doing something')
def main():
logging.info('Started')
do_something()
logging.info('Finished')
if __name__ == '__main__':
main()

You can give logger as argument to count_parrots() Or, what I would do, create class parrots and use logger as one of its method.