I seem to be having a spot of trouble getting the logger set up in Python. I'm trying to read in values for the control of the program (oceanographic simulations, with a lot of little tuning factors which are best kept isolated in a file somewhere, as we tend to play around with them a fair bit) from a control file (JSON-based). Among these is the level to which the logging should be set. As this is something I expect people may misspell and Python is quite strict on, I'm trying to wrap the logger setup in a try-except which will default it back to INFO if there's a mistake (and give a warning), as shown:
with open(control['outFileStem'] + "_log.info", 'w'):
pass #Clear an existing logfile
#Then set up the logging
try:
logging.basicConfig(filename = control['outFileStem'] + "_log.info", level = control['logLevel'])
except ValueError:
control['logLevel'] = "INFO"
logging.basicConfig(filename = control['outFileStem'] + "_log.info", level = logging.INFO)
logging.warning('WARNING: Logging Level set to INFO')
logging.info('Control structure created successfully')
However, when I experiment with it by mangling the level in the control file, it exclusively sets the level to WARNING. This happens no matter which variant I use for the level (i.e. logging.INFO, the numerical code, calling the value in from the control object as opposed to putting it in myself, etc.) I can't figure out what's going on here.
In case it's relevant, control is a dict to which I've altered the init method to handle the reading in from the JSON file nice and cleanly.
Thanks in advance.
Note that basicConfig() (as documented) does nothing if handlers are already present on the root logger. The basicConfig() implementation sets handlers first and the level at the end, so if it fails at that point, handlers are already set and the basicConfig() call in the except clause does nothing. The simplest solution is, in the exception clause, to set the level using
logging.getLogger().setLevel(logging.INFO)
instead of calling basicConfig() again to do it.
How do you define logLevel in control (what is it's type)?
What do you think of this?
#!/usr/bin/env python
import logging
control = {"logLevel": "FAKE"}
# get log level object which is just a number
try:
log_level = getattr(logging, control["logLevel"])
except AttributeError as ex:
log_level = logging.INFO
logging.basicConfig(filename = "logfile", level = log_level)
logging.debug("debug test")
logging.info("info test")
logging.warn("warn test")
logging.error("error test")
Related
I am using .setFormatter() to set the same logging.Formatter() on each of my handlers.
Is there a way to set a global default format?
Alternatively - is it possible to iterate though the handlers already added via .addHandler() to a logger ?
Another question mentions what the format is, but not how to set it.
The intended way is to attach the formatter to each handler as you're creating them.
Since you're supposed to set up logging destinations in one, central place at the start of the main program, this isn't so taxing to do.
E.g. this is the stock logging set-up code that I use in my scripts that are to be run autonomously:
# set up logging #####################################
import sys,logging,logging.handlers,os.path
log_file=os.path.splitext(__file__)[0]+".log"
l = logging.getLogger()
l.setLevel(logging.DEBUG)
f = logging.Formatter('%(asctime)s %(process)d:%(thread)d %(name)s %(levelname)-8s %(message)s')
h=logging.StreamHandler(sys.stdout)
h.setLevel(logging.NOTSET)
h.setFormatter(f)
l.addHandler(h)
h=logging.handlers.RotatingFileHandler(log_file,maxBytes=1024**2,backupCount=1)
h.setLevel(logging.NOTSET)
h.setFormatter(f)
l.addHandler(h)
del h,f
#hook to log unhandled exceptions
def excepthook(type,value,traceback):
logging.error("Unhandled exception occured",exc_info=(type,value,traceback))
#Don't need another copy of traceback on stderr
if old_excepthook!=sys.__excepthook__:
old_excepthook(type,value,traceback)
old_excepthook = sys.excepthook
sys.excepthook = excepthook
del excepthook,log_file
# ####################################################
There are other methods but each has a downside:
Each logger has an undocumented <logger>.handlers list, but it only lists handlers connected to that logger directly. So, you need to iterate over this list for all loggers you have if you have multiple.
There is a global undocumented logging._handlerList (references to all handlers are kept there to shut them down at exit). Likewise, that is an implementation detail.
Finally, you can override a handler's init logic by
replacing __init__ methods of Handler and/or subclass (that will affect everything else that uses logging), or by
subclassing/addin a mixin to the required classes.
That is probably an overkill.
If you have a number of handlers, you're advised to configure logging using the logging.config.dictConfig() API. This other answer shows how to configure formatters for handlers; you may be able to adapt it to your needs.
// , The one line answer to "how do I set my logging level to debug?" is usually something like the following:
logging.basicConfig(level=logging.DEBUG)
However, one may be tempted to change it later, back to something like INFO.
logging.basicConfig(level=logging.INFO)
This does not work, however. logging.basicConfig(level=logging.INFO), if run again at the same scope in the same process, will not result in a change to the level of the messages logged.
The official Python documentation, near the middle of https://docs.python.org/2/howto/logging.html#logging-to-a-file, lists the following:
The call to basicConfig() should come before any calls to debug(), info() etc. As it’s intended as a one-off simple configuration facility, only the first call will actually do anything: subsequent calls are effectively no-ops.
Source: https://docs.python.org/2/howto/logging.html#logging-basic-tutorial
Neither, however, does Logger.setLevel() work for this.
The following code attempts to set a lower log level:
logging.basicConfig(level=logging.DEBUG)
logging.debug("setting logging.basicConfig(level=logging.WARNING)")
logging.debug(myLogger.getEffectiveLevel())
logging.basicConfig(level=logging.WARNING) # This does nothing.
logging.getLogger('root').setLevel(level=logging.WARNING)
logging.debug(myLogger.getEffectiveLevel())
logging.debug("logger root, currently at " \
+ str(logging.getLogger('root').getEffectiveLevel()) \
+ " is enabled for" \
+ str(logging.getLogger('root').isEnabledFor(logging.DEBUG)))
This results in the following output:
DEBUG:root:setting logging.basicConfig(level=logging.WARNING)
DEBUG:root:10
DEBUG:root:30
DEBUG:root:30is enabled forFalse
It seems like one cannot "set" a Logger to a lower level of verbosity without access to something else created with logging.basicConfig().
How does a Python script change its logging level halfway through execution?
logging.disable(logging.INFO) → disable messages at logging.INFO
and lower, until...
logging.disable(logging.NOTSET) → resume normal logging
Link: https://docs.python.org/2/library/logging.html#logging.disable
Use this to temporarily throttle logging output.
It temporarily overrides the log levels for all loggers.
If you need to shut down logging, use the creatively named logging.shutdown()
First of all, logging.basicConfig() is a constructor, which creates a StreamHandler object to connect to the root logger.
Logging levels are just integer constants, not magic properties of a Python script.
To actually change the level for that specific logger, root, you may need to access the StreamHandler created for it by logging.basicConfig().
I have a log.py module, that is used in at least two other modules (server.py and device.py).
It has these globals:
fileLogger = logging.getLogger()
fileLogger.setLevel(logging.DEBUG)
consoleLogger = logging.getLogger()
consoleLogger.setLevel(logging.DEBUG)
file_logging_level_switch = {
'debug': fileLogger.debug,
'info': fileLogger.info,
'warning': fileLogger.warning,
'error': fileLogger.error,
'critical': fileLogger.critical
}
console_logging_level_switch = {
'debug': consoleLogger.debug,
'info': consoleLogger.info,
'warning': consoleLogger.warning,
'error': consoleLogger.error,
'critical': consoleLogger.critical
}
It has two functions:
def LoggingInit( logPath, logFile, html=True ):
global fileLogger
global consoleLogger
logFormatStr = "[%(asctime)s %(threadName)s, %(levelname)s] %(message)s"
consoleFormatStr = "[%(threadName)s, %(levelname)s] %(message)s"
if html:
logFormatStr = "<p>" + logFormatStr + "</p>"
# File Handler for log file
logFormatter = logging.Formatter(logFormatStr)
fileHandler = logging.FileHandler(
"{0}{1}.html".format( logPath, logFile ))
fileHandler.setFormatter( logFormatter )
fileLogger.addHandler( fileHandler )
# Stream Handler for stdout, stderr
consoleFormatter = logging.Formatter(consoleFormatStr)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter( consoleFormatter )
consoleLogger.addHandler( consoleHandler )
And:
def WriteLog( string, print_screen=True, remove_newlines=True,
level='debug' ):
if remove_newlines:
string = string.replace('\r', '').replace('\n', ' ')
if print_screen:
console_logging_level_switch[level](string)
file_logging_level_switch[level](string)
I call LoggingInit from server.py, which initializes the file and console loggers. I then call WriteLog from all over the place, so multiple threads are accessing fileLogger and consoleLogger.
Do I need any further protection for my log file? The documentation states that thread locks are handled by the handler.
The good news is that you don't need to do anything extra for thread safety, and you either need nothing extra or something almost trivial for clean shutdown. I'll get to the details later.
The bad news is that your code has a serious problem even before you get to that point: fileLogger and consoleLogger are the same object. From the documentation for getLogger():
Return a logger with the specified name or, if no name is specified, return a logger which is the root logger of the hierarchy.
So, you're getting the root logger and storing it as fileLogger, and then you're getting the root logger and storing it as consoleLogger. So, in LoggingInit, you initialize fileLogger, then re-initialize the same object under a different name with different values.
You can add multiple handlers to the same logger—and, since the only initialization you actually do for each is addHandler, your code will sort of work as intended, but only by accident. And only sort of. You will get two copies of each message in both logs if you pass print_screen=True, and you will get copies in the console even if you pass print_screen=False.
There's actually no reason for global variables at all; the whole point of getLogger() is that you can call it every time you need it and get the global root logger, so you don't need to store it anywhere.
A more minor problem is that you're not escaping the text you insert into HTML. At some point you're going to try to log the string "a < b" and end up in trouble.
Less seriously, a sequence of <p> tags that isn't inside a <body> inside an <html> is not a valid HTML document. But plenty of viewers will take care of that automatically, or you can post-process your logs trivially before displaying them. But if you really want this to be correct, you need to subclass FileHandler and have your __init__ add a header if given an empty file and remove a footer if present, then have your close add a footer.
Getting back to your actual question:
You do not need any additional locking. If a handler correctly implements createLock, acquire, and release (and it's called on a platform with threads), the logging machinery will automatically make sure to acquire the lock when needed to make sure each message is logged atomically.
As far as I know, the documentation doesn't directly say that StreamHandler and FileHandler implement these methods, it does strongly imply it (the text you mentioned in the question says "The logging module is intended to be thread-safe without any special work needing to be done by its clients", etc.). And you can look at the source for your implementation (e.g., CPython 3.3) and see that they both inherit correctly-implemented methods from logging.Handler.
Likewise, if a handler correctly implements flush and close, the logging machinery will make sure it's finalized correctly during normal shutdown.
Here, the documentation does explain what StreamHandler.flush(), FileHandler.flush(), and FileHandler.close(). They're mostly what you'd expect, except that StreamHandler.close() is a no-op, meaning it's possible that final log messages to the console may get lost. From the docs:
Note that the close() method is inherited from Handler and so does no output, so an explicit flush() call may be needed at times.
If this matters to you, and you want to fix it, you need to do something like this:
class ClosingStreamHandler(logging.StreamHandler):
def close(self):
self.flush()
super().close()
And then use ClosingStreamHandler() instead of StreamHandler().
FileHandler has no such problem.
The normal way to send logs to two places is to just use the root logger with two handlers, each with their own formatter.
Also, even if you do want two loggers, you don't need the separate console_logging_level_switch and file_logging_level_switch maps; calling Logger.debug(msg) is exactly the same thing as calling Logger.log(DEBUG, msg). You'll still need some way to map your custom level names debug, etc. to the standard names DEBUG, etc., but you can just do one lookup, instead of doing it once per logger (plus, if your names are just the standard names with different cast, you can cheat).
This is all described pretty well in the `Multiple handlers and formatters section, and the rest of the logging cookbook.
The only problem with the standard way of doing this is that you can't easily turn off console logging on a message-by-message basis. That's because it's not a normal thing to do. Usually, you just log by levels, and set the log level higher on the file log.
But, if you want more control, you can use filters. For example, give your FileHandler a filter that accepts everything, and your ConsoleHandler a filter that requires something starting with console, then use the filter 'console' if print_screen else ''. That reduces WriteLog to almost a one-liner.
You still need the extra two lines to remove newlines—but you can even do that in the filter, or via an adapter, if you want. (Again, see the cookbook.) And then WriteLog really is a one-liner.
Python logging is thread safe:
Is Python's logging module thread safe?
http://docs.python.org/2/library/logging.html#thread-safety
So you have no problem in the Python (library) code.
The routine that you call from multiple threads (WriteLog) does not write to any shared state. So you have no problem in your code.
So you are OK.
I'm logging events in my python code uing the python logging module. I have 2 logging files I wish to log too, one to contain user information and the other a more detailed log file for devs. I've set the the two logging files to the levels I want (usr.log = INFO and dev.log = ERROR) but cant work out how to restrict the logging to the usr.log file so only the INFO level logs are written to the log file as opposed to INFO plus everthing else above it e.g. INFO, WARNING, ERROR and CRITICAL.
This is basically my code:-
import logging
logger1 = logging.getLogger('')
logger1.addHandler(logging.FileHandler('/home/tmp/usr.log')
logger1.setLevel(logging.INFO)
logger2 = logging.getLogger('')
logger2.addHandler(logging.FileHandler('/home/tmp/dev.log')
logger2.setLevel(logging.ERROR)
logging.critical('this to be logged in dev.log only')
logging.info('this to be logged to usr.log and dev.log')
logging.warning('this to be logged to dev.log only')
Any help would be great thank you.
I am in general agreement with David, but I think more needs to be said. To paraphrase The Princess Bride - I do not think this code means what you think it means. Your code has:
logger1 = logging.getLogger('')
...
logger2 = logging.getLogger('')
which means that logger1 and logger2 are the same logger, so when you set the level of logger2 to ERROR you actually end up setting the level of logger1 at the same time. In order to get two different loggers, you would need to supply two different logger names. For example:
logger1 = logging.getLogger('user')
...
logger2 = logging.getLogger('dev')
Worse still, you are calling the logging module's critical(), info() and warning() methods and expecting that both loggers will get the messages. This only works because you used the empty string as the name for both logger1 and logger2 and thus they are not only the same logger, they are also the root logger. If you use different names for the two loggers as I have suggested, then you'll need to call the critical(), info() and warning() methods on each logger individually (i.e. you'll need two calls rather than just one).
What I think you really want is to have two different handlers on a single logger. For example:
import logging
mylogger = logging.getLogger('mylogger')
handler1 = logging.FileHandler('usr.log')
handler1.setLevel(logging.INFO)
mylogger.addHandler(handler1)
handler2 = logging.FileHandler('dev.log')
handler2.setLevel(logging.ERROR)
mylogger.addHandler(handler2)
mylogger.setLevel(logging.INFO)
mylogger.critical('A critical message')
mylogger.info('An info message')
Once you've made this change, then you can use filters as David has already mentioned. Here's a quick sample filter:
class MyFilter(object):
def __init__(self, level):
self.__level = level
def filter(self, logRecord):
return logRecord.levelno <= self.__level
You can apply the filter to each of the two handlers like this:
handler1.addFilter(MyFilter(logging.INFO))
...
handler2.addFilter(MyFilter(logging.ERROR))
This will restrict each handler to only write out log messages at the level specified.
First: this is a rather odd thing to want to do, and strikes me as a slight misuse of the logging system. I can't imagine any situation in which it makes sense to notify the user about the normal operation of the program but not about things that are more important. The logging levels should be used to indicate importance; if you have messages that are only of interest to developers, you should be using some other mechanism to distinguish them (such as which logger you send them to).
That being said, you can implement arbitrary filtering of log records by creating a Filter subclass whose filter method implements your desired criteria and install it on the appropriate handler.
I want to unify the whole logging facility of my app. Any warning is raise an exception, next I catch it and pass it to the logger. But the question: Is there in logging any mute facility? Sometimes logger becomes too verbose. Sometimes for the reason of too noisy warnings, is there are any verbose limit in warnings?
http://docs.python.org/library/logging.html
http://docs.python.org/library/warnings.html
Not only are there log levels, but there is a really flexible way of configuring them. If you are using named logger objects (e.g., logger = logging.getLogger(...)) then you can configure them appropriately. That will let you configure verbosity on a subsystem-by-subsystem basis where a subsystem is defined by the logging hierarchy.
The other option is to use logging.Filter and Warning filters to limit the output. I haven't used this method before but it looks like it might be a better fit for your needs.
Give PEP-282 a read for a good prose description of the Python logging package. I think that it describes the functionality much better than the module documentation does.
Edit after Clarification
You might be able to handle the logging portion of this using a custom class based on logging.Logger and registered with logging.setLoggerClass(). It really sounds like you want something similar to syslog's "Last message repeated 9 times". Unfortunately I don't know of an implementation of this anywhere. You might want to see if twisted.python.log supports this functionality.
from the very source you mentioned.
there are the log-levels, use the wisely ;-)
LEVELS = {'debug': logging.DEBUG,
'info': logging.INFO,
'warning': logging.WARNING,
'error': logging.ERROR,
'critical': logging.CRITICAL}
This will be a problem if you plan to make all logging calls from some blind error handler that doesn't know anything about the code that raised the error, which is what your question sounds like. How will you decide which logging calls get made and which don't?
The more standard practice is to use such blocks to recover if possible, and log an error (really, if it is an error that you weren't specifically prepared for, you want to know about it; use a high level). But don't rely on these blocks for all your state/debug information. Better to sprinkle your code with logging calls before it gets to the error-handler. That way, you can observe useful run-time information about a system when it is NOT failing and you can make logging calls of different severity. For example:
import logging
from traceback import format_exc
logger = logging.getLogger() # Gives the root logger. Change this for better organization
# Add your appenders or what have you
def handle_error(e):
logger.error("Unexpected error found")
logger.warn(format_exc()) #put the traceback in the log at lower level
... #Your recovery code
def do_stuff():
logger.info("Program started")
... #Your main code
logger.info("Stuff done")
if __name__ == "__main__":
try:
do_stuff()
except Exception,e:
handle_error(e)