The Problem:
Given a logging config and a logger that employs that config, I see log messages from the script in which the log handler is configured, but not from the root logger, to which the same handler is assigned.
Details:
(Using Python 2.7)
I have a module my_mod which instantiates a logger. my_mod has a function my_command which logs some messages using that logger. my_mod exists inside of a library my_lib, so I don't want to configure the logger with any handlers; as recommended, I want to leave the log handling to the developer using my_mod. my_mod looks like:
import logging
LOGGER = logging.getLogger(__name__)
def my_command():
LOGGER.debug("This is a log message from module.py")
print "This is a print statement from module.py"
I also have a python script my_script.py, which uses my_mod.my_command. my_script.py instantiates a logger, and in this case I do have handlers and formatters configured. my_script.py configures handlers and formatters using fileConfig and a config file that lives alongside my_script.py:
import os
import logging
import logging.config
from my_mod.module import my_command
logging.config.fileConfig('{0}/logging.cfg'.format(
os.path.dirname(os.path.realpath(__file__))))
LOGGER = logging.getLogger(__name__)
LOGGER.debug("This is a log message from script.py")
my_command()
From what I can tell, my config file appears to be set up correctly...
[loggers]
keys=root,script
[handlers]
keys=consoleHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_script]
level=DEBUG
handlers=consoleHandler
qualname=script
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)
[formatter_simpleFormatter]
format=%(asctime)s [%(levelname)s] %(name)s: %(message)s
datefmt=
...but when I run my_script.py I get only the log line from my_script.py, and not the one from my_mod.my_command. I know that my_command is working, though, because the print statement in my_command after the debug log statement successfully prints to the console:
20:27 $ python script.py
2015-06-15 20:27:54,488 [DEBUG] __main__: This is a log message from script.py
This is a print statement from module.py
What am I doing wrong?
NOTE: The example shows using debug, but even when I keep logging.cfg specifying level=DEBUG (I also tried level=NOTSET) for root logger and call LOGGER.info(message) in my_command, nothing is logged to the console.
A potential problem is that you are importing the module before you set up the logger configuration. That way, the module requests a logger before the logging is setup.
Looking a fileConfig()'s documentation, the reason subsequent logging to the pre-obtained loggers fails is the default value for its disable_existing_loggers argument:
logging.config.fileConfig(fname, defaults=None, disable_existing_loggers=True)
If you change your code to
logging.config.fileConfig(
'{0}/logging.cfg'.format(os.path.dirname(os.path.realpath(__file__))),
disable_existing_loggers=False
)
the problem should go away.
Note that existing loggers are only disabled when they are not explicitly named in the configuration file. For example:
import logging
import logging.config
lFooBefore = logging.getLogger('foo')
lScriptBefore = logging.getLogger('script')
logging.config.fileConfig('logger.ini')
lFooBefore.debug('Does not log')
lScriptBefore.debug('Does log')
logging.getLogger('foo').debug('Does also not log')
logging.getLogger('bar').debug('Does log')
No idea why the default value for disable_existing_loggers is the way it is ...
Related
I am trying to configure multiple loggers within my application so that loggers in a particular sub-module of my application will send messages across the HTTPHandler and other areas of my application will not. Essentially the longer processing sections will send input to the HTTPHandler in order to inform the user of the processing that is going on. Ideally i would like to do this based upon the module name and configure the logger based upon the module name. e.g. qualname=application.processing.*
However I cannot even seem to do basic control of loggers based upon a hard coded name.
I've attempted configuration both through fileConfig and dictConfig as you can see in the code below.
import logging
import logging.config
import ruamel.yaml as yaml
def main():
#logging_config_file = "logging.yaml"
# try:
# with open(logging_config_file) as configuration_file:
# logger_dict = {}
# logger_dict = yaml.load(configuration_file)
# logging.config.dictConfig( logger_dict )
# except Exception as y:
# print( "Unable to configure logging (", y, ") attempting Config file setup")
try:
logging.config.fileConfig( "logging.conf")
except Exception as y:
print( "Unable to configure logging (", y, ") attempting Config file setup")
logger = logging.getLogger(__name__)
logger.debug( "in main" )
logger2 = logging.getLogger( "ONLY_ME")
logger2.info( "using only me")
return 0
if __name__ == '__main__':
main()
[loggers]
keys=root, webApp
[handlers]
keys=consoleHandler,fileHandler,httpHandler
[formatters]
keys=
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_webApp]
level=DEBUG
qualname='ONLY_ME'
handlers=fileHandler,httpHandler
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
args=(sys.stdout,)
[handler_fileHandler]
class=FileHandler
level=DEBUG
args=('webAppLogging.log',)
propagate=0
[handler_httpHandler]
class=handlers.HTTPHandler
level=INFO
args=("localhost:8080","/","GET",)
propagate=0
version: 1
disable_existing_loggers: false
handlers:
console_handler:
class: logging.StreamHandler
stream: ext://sys.stdout
file_handler:
class: logging.FileHandler
filename: webAppLogging.log
propagate: false
http_handler:
level: INFO
class: logging.handlers.HTTPHandler
host: localhost:8080
url: /
method: GET
propagate: false
root:
handlers: [console_handler]
level: DEBUG
loggers:
webApp:
qualname: ONLY_ME
level: DEBUG
handlers: [file_handler, http_handler]
I would expect with the code and the configuration that on console I would see:
'in main'
and in the file and the http message i would see:
'using only me'
Instead i see in console:
'in main'
'using only me'
and I see nothing in webAppLogging.log nor do i see any http message traffic.
If i set the handlers into the root logger then i see messages in the log and in the http message traffic
I feel like I am missing something simple here. Any insight is appreciated.
Found the issue(s). In both cases it was the syntax of the config/yaml file in regards to what logging is expecting. In the case of the .config
qualname='ONLY_ME'
should be
qualname=ONLY_ME
the '' was throwing it off
In the case of the yaml it appears the the logger name is acting as the "qualname" from the config and the "qualname:" doesn't appear to do anything. changing
webApp: to ONLY_ME: seems to affect the behavior.
In terms of the desired effect children will inherit their parent logger if they dont' have a specific one defined for them. e.g. onlyme.process and onlyme.display will both use the common logger of "onlyme" if it is defined either in the configuration or earlier in the program.
So I am trying to implement logging within my Python program. The goal is to set it up so that a log file is created and everything the program does through it's various modules is logged (based on logging level). This is what my current code looks like:
Text File for Log Configuration:
#logging.conf
[loggers]
keys=root,MainLogger
[handlers]
keys=consoleHandler
[formatters]
keys=consoleFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_MainLogger]
level=DEBUG
handlers=consoleHandler
qualname=MainLogger
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=consoleFormatter
args=(sys.stdout,)
[formatter_consoleFormatter]
format=%(asctime)s | %(levelname)-8s | %(filename)s-%(funcName)s-%lineno)04d | %(message)s
External Module to Test Logs:
#test.py
import logging
logger = logging.getLogger(__name__)
def testLog():
logger.debug("Debug Test")
logger.info("Info Test")
logger.warning("Warning Test")
logger.error("Error Test")
Main file:
#__init__.py
import logging
import logging.config
from datetime import datetime
logging.config.fileConfig('logging.conf', disable_existing_loggers = False)
logger = logging.getLogger('MainLogger')
fileHandler = logging.FileHandler('{:%Y-%m-%d}.log'.format(datetime.now()))
formatter = logging.Formatter('%(asctime)s | %(levelname)-8s | %(lineno)04d | %(message)s')
fileHandler.setFormatter(formatter)
logger.addHandler(fileHandler)
if __name__ == "__main__":
import test
logger.debug("Debug Test")
test.testLog()
Currently, all log messages are currently being displayed withing the IDLE3 shell when I run __init__.py and the log file is being created. However within the log file itself the only message being recording is the "Debug Test" from __init__.py. None of the messages from the test.py module are being recorded in the log file.
What is my problem?
In test.py it grabs a logger object before you configure it later in your __init__.py. Make sure you configure the logging module first before grabbing any logger instance.
I set up logging throughout my python package using a logconfig.ini file.
[loggers]
keys=extracts,root
[formatters]
keys=simple,detailed
[handlers]
keys=file_handler
[formatter_simple]
format=%(module)s - %(levelname)s - %(message)s
datefmt=%Y-%m-%d %H:%M:%S
[formatter_detailed]
format=%(asctime)s %(name)s:%(lineno)s %(levelname)s %(message)s
datefmt=%Y-%m-%d %H:%M:%S
[handler_file_handler]
class=logging.handlers.RotatingFileHandler
level=DEBUG
formatter=detailed
args=('/ebs/logs/foo.log', 'a', 100000000, 3)
[logger_extracts]
level=DEBUG
handlers=file_handler
propagate=1
qualname=extracts
[logger_root]
level=NOTSET
handlers=
But whenever I run my application, I get the following warning message in prompt,
No handlers found for logger __main__
How can I fix this?
You have to call logging.basicConfig() first:
Logging HOWTO
The call to basicConfig() should come before any calls to debug(),
info() etc. As it’s intended as a one-off simple configuration
facility, only the first call will actually do anything: subsequent
calls are effectively no-ops.
Or all logging.info('Starting logger for...') which will call logging.basicConfig() automatically. So something like:
import logging
logging.info('Starting logger for...') # or call logging.basicConfig()
LOG = logging.getLogger(name)
The module author's reason for this behavior is here
I found my error.
It turns out the the root logger is used for main.
I just need to attach a handler to the root logger as so,
[logger_root]
level=NOTSET
handlers=file_handler
I have a logger configured from a file and would like to change the level of my logging without having to change the .conf file, but instead using inline code;
import logging.config
logging.config.fileConfig('..\\LoggingConfig\\loggingfile.conf')
logging.StreamHandler.setLevel(logging.info)
logging.debug("Debug")
logging.info("Info")
This should only print the "Info" log line to the screen. I don't know on which object to call the setLevel()! logging.StreamHandler.setLevel(logging.info) is just a stab in the dark after 30 mins searching...
The loggingfile.conf file;
[loggers]
keys=root
[logger_root]
handlers=screen
level=NOTSET
[formatter_modfunc]
format=%(module)-20s %(funcName)-25s %(levelno)-3s: %(message)s
[handlers]
keys=screen
[handler_screen]
class=StreamHandler
formatter=modfunc
level=DEBUG
args=(sys.stdout,)
qualname=screen
You need to call setLevel on your Logger instance.
LOGGER = logging.getLogger('your.module.file.name')
LOGGER.setLevel(_level)
LOGGER.info('foo')
If you are only using the basic logger, you can do it like this
logging.basicConfig(level=_level)
logging.info('foo')
See http://docs.python.org/howto/logging.html
When using logging.config.fileConfig and you want to dynamically change level for all child loggers at once, you can...
a) set level for root logger:
logging.getLogger().setLevel(logging.WARNING)
b) disable other levels
logging.disable(logging.INFO)
I am using logging module of python. How can I access the handlers defined in config file from the code. As an example, I have a logger defined and two handlers - one for screen and other for file. I want to use appropriate handler based on user preference (whether they want to log on screen or to a file). How can I dynamically add and remove handlers defined in config file from loggers defined in config file?
[loggers]
keys=root,netmap
[handlers]
keys=fil,screen
[logger_root]
level=NOTSET
handlers=
[logger_netmap]
level=INFO
handlers=fil,screen
qualname=netmap
[formatters]
keys = simple
[formatter_simple]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=
[handler_fil]
class=handlers.RotatingFileHandler
args=('file.log','a','maxBytes=10000','backupCount=5')
formatter=simple
[handler_screen]
class=StreamHandler
args = (sys.stdout,)
formatter=simple
Depending on whether user runs the program with -v or not I need to use one of File or Screen Handler. How can I add or delete fil or screen handlers from netmap logger?
Instead of having to dynamically change the config file, just use Logger.addHandler(handler).
fileHandler = logging.handlers.RotatingFileHandler('file.log', mode='a', maxBytes=10000, backupCount=5)
logger = logging.getLogger('netmap')
if LOG_TO_FILE:
logger.addHandler(fileHandler)
Then to load in formatting from perhaps the same file;
import ConfigParser
configParser = ConfigParser.ConfigParser()
config.read('config.ini')
format = config.get('formatter_simple', 'format')
fileHandler.setFormatter(format)
From the logging module's documentation it looks like logging objects have these two methods:
Logger.addHandler(hdlr)
Adds the specified handler hdlr to this logger.
Logger.removeHandler(hdlr)
Removes the specified handler hdlr from this logger.
So it seems like you ought be able to use them to add or change the netmap logger handler to be whatever you want based on what's in the config file.
Ok So I found an elegant way through a gud soul on the python google group. It works like charm. Here's the code
import logging
import getopt
import sys
import logging.config
def stop(m,handl):
consoleHand=[h for h in m.handlers if h.__class__ is handl]
print consoleHand
if consoleHand:
h1=consoleHand[0]
h1.filter=lambda x: False
logging.config.fileConfig('log.conf')
my_log = logging.getLogger('netmap')
args=sys.argv[1:]
opt,arg = getopt.gnu_getopt(args,'v')
l=''
for o,a in opt:
if o=='-v':
l='verbose'
stop(my_log,logging.handlers.RotatingFileHandler)
if not l:
stop(my_log,logging.StreamHandler)
my_log.debug('Starting.....')
my_log.warning('Unstable')
my_log.error('FIles Missing')
my_log.critical('BlowOut!!')
The config file is still the same. I can now easily activate or deactivate handlers.