I am currently writing an API and an application which uses the API. I have gotten suggestions from people stating that I should perform logging using handlers in the application and use a "logger" object for logging from the API.
In light of the advice I received above, is the following implementation correct?
class test:
def __init__(self, verbose):
self.logger = logging.getLogger("test")
self.logger.setLevel(verbose)
def do_something(self):
# do something
self.logger.log("something")
# by doing this i get the error message "No handlers could be found for logger "test"
The implementation i had in mind was as follows:
#!/usr/bin/python
"""
....
....
create a logger with a handler
....
....
"""
myobject = test()
try:
myobject.do_something()
except SomeError:
logger.log("cant do something")
I'd like to get my basics strong, I'd be grateful for any help and suggestions for code you might recommend I look up.
Thanks!
It's not very clear whether your question is about the specifics of how to use logging or about logging exceptions, but if the latter, I would agree with Adam Crossland that log-and-swallow is a pattern to be avoided.
In terms of the mechanics of logging, I would make the following observations:
You don't need to have a logger as an instance member. It's more natural to declare loggers at module level using logger = logging.getLogger(__name__), and this will also work as expected in sub-packages.
Your call logger.log("message") would likely fail anyway because the log method has a level as the first argument, rather than a message.
You should declare handlers, and if your usage scenario is fairly simple you can do this in your main method or if __name__ == '__main__': clause by adding for example
logging.basicConfig(filename='/tmp/myapp.log', level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(name)s %(message)s')
and then elsewhere in your code, just do for example
import logging
logger = logging.getLogger(__name__)
once at the top of each module where you want to use logging, and then
logger.debug('message with %s', 'arguments') # or .info, .warning, .error etc.
in your code wherever needed.
The danger with the pattern that you are thinking about is that you may end up effectively hiding exceptions by putting them in a log. Many exceptions really should crash your program because they represent a problem that needs to be fixed. Generally, it is more useful to be able to step into code with a debugger to find out what caused the exception.
If there are cases that an exception represents an expected condition that does not affect the stability of the app or the correctness of its behavior, doing nothing but writing a notation to the log is OK. But be very, very careful about how you use this.
I usually do the following:
import logging
import logging.config
logging.config.fileConfig('log.congig')
# for one line log records
G_LOG = logging.getLogger(__name__)
# for records with stacktraces
ST_LOG = logging.getLogger('stacktrace.' + __name__)
try:
# some code
G_LOG.info('some message %s %s', param1, param2)
except (StandardError,):
message = 'some message'
G_LOG.error(message)
# exc_info appends stacktrace to the log message
ST_LOG.error(message, exc_info=True)
Format of the config file can be seen in the python manual
Related
I seem to be having a spot of trouble getting the logger set up in Python. I'm trying to read in values for the control of the program (oceanographic simulations, with a lot of little tuning factors which are best kept isolated in a file somewhere, as we tend to play around with them a fair bit) from a control file (JSON-based). Among these is the level to which the logging should be set. As this is something I expect people may misspell and Python is quite strict on, I'm trying to wrap the logger setup in a try-except which will default it back to INFO if there's a mistake (and give a warning), as shown:
with open(control['outFileStem'] + "_log.info", 'w'):
pass #Clear an existing logfile
#Then set up the logging
try:
logging.basicConfig(filename = control['outFileStem'] + "_log.info", level = control['logLevel'])
except ValueError:
control['logLevel'] = "INFO"
logging.basicConfig(filename = control['outFileStem'] + "_log.info", level = logging.INFO)
logging.warning('WARNING: Logging Level set to INFO')
logging.info('Control structure created successfully')
However, when I experiment with it by mangling the level in the control file, it exclusively sets the level to WARNING. This happens no matter which variant I use for the level (i.e. logging.INFO, the numerical code, calling the value in from the control object as opposed to putting it in myself, etc.) I can't figure out what's going on here.
In case it's relevant, control is a dict to which I've altered the init method to handle the reading in from the JSON file nice and cleanly.
Thanks in advance.
Note that basicConfig() (as documented) does nothing if handlers are already present on the root logger. The basicConfig() implementation sets handlers first and the level at the end, so if it fails at that point, handlers are already set and the basicConfig() call in the except clause does nothing. The simplest solution is, in the exception clause, to set the level using
logging.getLogger().setLevel(logging.INFO)
instead of calling basicConfig() again to do it.
How do you define logLevel in control (what is it's type)?
What do you think of this?
#!/usr/bin/env python
import logging
control = {"logLevel": "FAKE"}
# get log level object which is just a number
try:
log_level = getattr(logging, control["logLevel"])
except AttributeError as ex:
log_level = logging.INFO
logging.basicConfig(filename = "logfile", level = log_level)
logging.debug("debug test")
logging.info("info test")
logging.warn("warn test")
logging.error("error test")
First time playing around with the logging module. I have two questions really:
All of the examples I have found on creating a FileHandler just use a dummy file name, but I want my log file to have a certain title format.
log = time.strftime('./logs/'+'%H:%M:%S %d %b %Y', time.localtime())+'.log'
logger = logging.getLogger('myproject')
formatter = logging.Formatter('%(asctime)s %(message)s', '%H:%M:%S %d %b %Y')
handler = logging.FileHandler(log)
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
This seems to work at first, in that the log file is created, but when I kill the script and examine it, there is nothing in it. And before you suggest using with open(log) as f:, using f in place of log in handler = logging.FileHandler(log) gives me errors about the file not even being created. What am I doing wrong to not have the log file written to? Is it because I'm trying to be too clever with the filename?
The second question may subsume the first somehow. What I would really like is a way to write to both the console and the log file concurrently, since this script takes a long time and I'd like to gauge progress. I have seen that there are answers on how to do this, such as here, but all of these seem to assume that I want different levels of logging for the two different outputs; I don't. Incidentally, I do realize that if my script didn't dynamically generate the name of the log file, I could do something like python myscript.py | tee log.file to avoid making different handlers.
I feel I am having trouble answering my second question because of the variables involved in the first, but I am foggy right now due to illness and subsequent lack of sleep. If someone could explain to me like a five-year-old how to write to both a console and log file with identical output, I would greatly appreciate it.
Many thanks.
Your code is configuring a logger named 'myproject':
logger = logging.getLogger('myproject')
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
If you use methods on that logger, everything will work:
logger.info('foo bar')
logging.getLogger('myproject').info('foo bar')
But if you just use the module-level functions, it will not:
logging.info('foo bar')
Why? Because the module-level functions use the default root logger, which is not the one you've configured. See the docs for details on how this works.
Generally, the way you deal with this is either to create a module-level logger object for each module, like this:
logger = logging.getLogger(__name__)
… or a class or instance attribute logger, something like this:
self.logger = logging.getLogger('{}.{}'.format(__name__, cls.__name__))
And then do everything through that logger or self.logger object, not through the module-level functions.
So, why are the module-level functions even there? For convenience in simple programs, mainly. If that sounds appropriate for you, and you want to use them, you can; you just have to configure the root logger instead of a different one. Change the first line to this:
logger = logging.getLogger()
Now when you configure logger, you're also configuring the module-level functions.
I'm logging events in my python code uing the python logging module. I have 2 logging files I wish to log too, one to contain user information and the other a more detailed log file for devs. I've set the the two logging files to the levels I want (usr.log = INFO and dev.log = ERROR) but cant work out how to restrict the logging to the usr.log file so only the INFO level logs are written to the log file as opposed to INFO plus everthing else above it e.g. INFO, WARNING, ERROR and CRITICAL.
This is basically my code:-
import logging
logger1 = logging.getLogger('')
logger1.addHandler(logging.FileHandler('/home/tmp/usr.log')
logger1.setLevel(logging.INFO)
logger2 = logging.getLogger('')
logger2.addHandler(logging.FileHandler('/home/tmp/dev.log')
logger2.setLevel(logging.ERROR)
logging.critical('this to be logged in dev.log only')
logging.info('this to be logged to usr.log and dev.log')
logging.warning('this to be logged to dev.log only')
Any help would be great thank you.
I am in general agreement with David, but I think more needs to be said. To paraphrase The Princess Bride - I do not think this code means what you think it means. Your code has:
logger1 = logging.getLogger('')
...
logger2 = logging.getLogger('')
which means that logger1 and logger2 are the same logger, so when you set the level of logger2 to ERROR you actually end up setting the level of logger1 at the same time. In order to get two different loggers, you would need to supply two different logger names. For example:
logger1 = logging.getLogger('user')
...
logger2 = logging.getLogger('dev')
Worse still, you are calling the logging module's critical(), info() and warning() methods and expecting that both loggers will get the messages. This only works because you used the empty string as the name for both logger1 and logger2 and thus they are not only the same logger, they are also the root logger. If you use different names for the two loggers as I have suggested, then you'll need to call the critical(), info() and warning() methods on each logger individually (i.e. you'll need two calls rather than just one).
What I think you really want is to have two different handlers on a single logger. For example:
import logging
mylogger = logging.getLogger('mylogger')
handler1 = logging.FileHandler('usr.log')
handler1.setLevel(logging.INFO)
mylogger.addHandler(handler1)
handler2 = logging.FileHandler('dev.log')
handler2.setLevel(logging.ERROR)
mylogger.addHandler(handler2)
mylogger.setLevel(logging.INFO)
mylogger.critical('A critical message')
mylogger.info('An info message')
Once you've made this change, then you can use filters as David has already mentioned. Here's a quick sample filter:
class MyFilter(object):
def __init__(self, level):
self.__level = level
def filter(self, logRecord):
return logRecord.levelno <= self.__level
You can apply the filter to each of the two handlers like this:
handler1.addFilter(MyFilter(logging.INFO))
...
handler2.addFilter(MyFilter(logging.ERROR))
This will restrict each handler to only write out log messages at the level specified.
First: this is a rather odd thing to want to do, and strikes me as a slight misuse of the logging system. I can't imagine any situation in which it makes sense to notify the user about the normal operation of the program but not about things that are more important. The logging levels should be used to indicate importance; if you have messages that are only of interest to developers, you should be using some other mechanism to distinguish them (such as which logger you send them to).
That being said, you can implement arbitrary filtering of log records by creating a Filter subclass whose filter method implements your desired criteria and install it on the appropriate handler.
I am facing the following scenario :
I set DEBUG=True/False and based on this I do the logging.
Traditional way of doing this is
if DEBUG:
logger.log(whatever)
Is there a better way to write this code? Using closures/lambda functions etc.. ?
Look at the logging library's manual: http://docs.python.org/library/logging.html
you can configure directly in code, or in a config file ...
e.g.
import logging
logging.basicConfig(filename='example.log',level=logging.INFO)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
(see how it'll only display info and warning messages)
or, from a file:
import logging
import logging.config
logging.config.fileConfig('logging.conf')
# create logger
logger = logging.getLogger('simpleExample')
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
See the config section for the logging config format. http://docs.python.org/howto/logging.html#configuring-logging
The relevance of logging.getLogger(...) is that it allows you to set different log levels for different parts of your code.
Edit: if you're worried about expensive operations, it's best to use the following:
if logger.isEnabledFor(logging.DEBUG):
logger.debug('Message with %s, %s', expensive_func1(),
expensive_func2())
HTH
You are over thinking this. You have 2 lines of code that are very clear and concise. There is not much that you could do to change this without making things more complicated.
The only other short and simple way to do logging is to move the debug check into a wrapper function.
def log(msg):
if DEBUG:
logger.log(msg)
This will mean that the rest of your code will just have one line when logging a message. The only potential downside is that if you are building the message string, it will be built even when DEBUG is false. This is really only an issue if you need a highly efficient program and you are constructing very complicated log strings.
In this particular instance, you could consider using python's logging module, which will handle this (and other logging needs) for you. In its most basic incarnation:
# Call once somewhere early in your code
logging.basicConfig(level=DEBUG and logging.DEBUG or logging.INFO)
....
logging.debug(whatever)
One thing to watch is that 'whatever' will always get evaluated (unlike in your example).
I want to unify the whole logging facility of my app. Any warning is raise an exception, next I catch it and pass it to the logger. But the question: Is there in logging any mute facility? Sometimes logger becomes too verbose. Sometimes for the reason of too noisy warnings, is there are any verbose limit in warnings?
http://docs.python.org/library/logging.html
http://docs.python.org/library/warnings.html
Not only are there log levels, but there is a really flexible way of configuring them. If you are using named logger objects (e.g., logger = logging.getLogger(...)) then you can configure them appropriately. That will let you configure verbosity on a subsystem-by-subsystem basis where a subsystem is defined by the logging hierarchy.
The other option is to use logging.Filter and Warning filters to limit the output. I haven't used this method before but it looks like it might be a better fit for your needs.
Give PEP-282 a read for a good prose description of the Python logging package. I think that it describes the functionality much better than the module documentation does.
Edit after Clarification
You might be able to handle the logging portion of this using a custom class based on logging.Logger and registered with logging.setLoggerClass(). It really sounds like you want something similar to syslog's "Last message repeated 9 times". Unfortunately I don't know of an implementation of this anywhere. You might want to see if twisted.python.log supports this functionality.
from the very source you mentioned.
there are the log-levels, use the wisely ;-)
LEVELS = {'debug': logging.DEBUG,
'info': logging.INFO,
'warning': logging.WARNING,
'error': logging.ERROR,
'critical': logging.CRITICAL}
This will be a problem if you plan to make all logging calls from some blind error handler that doesn't know anything about the code that raised the error, which is what your question sounds like. How will you decide which logging calls get made and which don't?
The more standard practice is to use such blocks to recover if possible, and log an error (really, if it is an error that you weren't specifically prepared for, you want to know about it; use a high level). But don't rely on these blocks for all your state/debug information. Better to sprinkle your code with logging calls before it gets to the error-handler. That way, you can observe useful run-time information about a system when it is NOT failing and you can make logging calls of different severity. For example:
import logging
from traceback import format_exc
logger = logging.getLogger() # Gives the root logger. Change this for better organization
# Add your appenders or what have you
def handle_error(e):
logger.error("Unexpected error found")
logger.warn(format_exc()) #put the traceback in the log at lower level
... #Your recovery code
def do_stuff():
logger.info("Program started")
... #Your main code
logger.info("Stuff done")
if __name__ == "__main__":
try:
do_stuff()
except Exception,e:
handle_error(e)