So I'm running the following code from the command line python:
import logging
rootLog = logging.getLogger(__name__)
rootLog.setLevel(logging.INFO)
rootLog.warning("This is a root warning")
rootLog.info("This is root info")
def info():
log = rootLog.getChild("info")
log.info("This is info")
log.warning("This is a warning")
info()
I'm expecting to see all four log messages on the console, but I'm only seeing the warnings. What is going on? Am I misunderstanding something?
EDIT:
I discovered by adding logging.basicConfig() at the beginning of the script that I'll get the output that I expected. This is strange, because the python documentation on logging states:
The functions debug(), info(), warning(), error() and critical() will call basicConfig() automatically if no handlers are defined for the root logger.
The bit you quote from the manual is under Module-Level Functions and applies only if the module function
logging.debug()
is literally called. Since you are calling an instance method with rootLog.info() basicConfig isn't getting called for you and you are probably talking to a null logger. The documentation is kinda confusing there.
Use loggging.basicConfig() and things should work.
Ok, I dug through the code in the logging module, and I think I figured it out, at least partially. What you are seeing is happening because what you are calling rootLogger isn't actually the root logger. It is true that if no handlers have been added to the root logger (the true root logger) when you call one of the log methods directly on logging, it calls basicConfig, but calling a log method on an instance of Logger as you are doing here does not actually call basicConfig at all. That's actually irrelevant in this case anyway though. :) I'm not sure where the handler that is getting called here is being created, but I can almost guarantee it's attached to the true root logger. The true root logger is initialized by default to WARNING. Try doing logging.root.setLevel(logging.INFO) and see if you get what you expect. You should also see what you want if you manually attach a logger to your rootLogger.
Related
I have a basic config for the logging module with debug level - now I want to create another logger with error level only. How can I do that?
The problem is that the root handler is called in addition to the error-handler - this is something I want to avoid.
import logging
fmt = '%(asctime)s:%(funcName)s:%(lineno)d:%(levelname)s:%(name)s:%(message)s'
logging.basicConfig(level=logging.DEBUG, format=fmt)
logger = logging.getLogger('Temp')
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.error('boo')
The above code prints boo twice while I expect once only, and I have no idea what to do with this annoying issue...
In [4]: logger.error('boo')
boo
2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
In [5]: logger.handlers
Out[5]: [<StreamHandler stderr (ERROR)>]
Some basics about the logging module
logger: a person who receives the log string, sorts it by a predefined level, then uses his own handler if any to process the log and, by default passes the log to his superior.
root logger: the superior of superiors, does all the things that a normal logger does but doesn't pass the received log to anyone else.
handler: a private contractor of a logger, who actually does anything with the log, eg. formats the log, writes it to a file or stdout, or sends it through tcp/udp.
formatter: a theme, a design that the handler applies to the log.
basicConfig: a shortcut way to config the root logger. This is useful when you want him to do all the job and all his lower rank loggers would just pass the log to him.
With no argument, basicConfig sets root logger's level to WARNING and add a StreamHandler that output the log to stderr.
What you did
You created a format and used a shortcut basicConfig to config the root logger. You want the root logger to do all the actual logging things
You created a new low-rank logger Temp
You want it to accept logs with only ERROR level and above.
You created another StreamHandler. Which output to stdout by default.
You want it to handle only ERROR level and above
Oh, you assigned it to Temp logger, which made 5. redundant since the level is set in 3.
Oh wait, thought you just want the root logger to do the job since 1.!
You logged an ERROR with your logger.
What happened
Your Temp logger accepted a string boo at ERROR level. Then told its handler to process the string. Since this handler didn't have any formatter assigned to it, it outputted the string as-is to stdout: boo
After that, Temp logger passed the string boo to his superior, the root logger.
The root logger accepted the log since the log level is ERROR > WARNING.
The root logger then told its handler to process the string boo.
This handler applies the format string to boo. Added timestamp, added location, added the name of logger that passed the log, etc.
Finally it outputted the result to stderr: 2021-04-26 18:54:24,329:<module>:1:ERROR:Temp:boo
Make it right
Since your code does exactly what you tell it to do, you have to tell it as much detail as possible.
Only use basicConfig when you are lazy. By removing basicConfig line, your problem solved.
Use logger = logging.getLogger('__name__') so that the logger has the name of the module. Looking at the log and know exactly which import path that it came from.
Decide if a logger should keep the log on its own or pass it up the chain with the propagate property. In your case, logger.propagate = False also solves the problem.
Use a dictConfig file so you don't get messed with the config code.
In practice, you should not add handlers to your logger, and just let the logger pass the log all the way to the root and let the root do the actual logging. Why?
Someone else uses your code as a module, can control the logging. For example, not output to stdout but to tcp/udp, output with a different format, etc.
You can turn off the logging from a specific logger entirely, by propagating=False.
You know exactly all the handlers and formatters in the code if you only added them to the root logger. You have centralized control over the logging.
I've had to extend the existing logging library to add some features. One of these is a feature which lets a handler listen to any existing log without knowing if it exists beforehand. The following allows the handler to listen, but does not check if the log exists:
def listen_to_log(target, handler):
logging.getLogger(target).addHandler(handler)
The issue is that I don't want any handler to listen to a log which isn't being logged to, and want to raise a ValueError if the log doesn't already exist. Ideally I'd do something like the following:
def listen_to_log(target, handler):
if not logging.logExists(target):
raise ValueError('Log not found')
logging.getLogger(target).addHandler(handler)
So far I have been unable to find a function like logging.logExists, is there such a function, or a convenient workaround?
Workaround that works for me and possibly you:
When you create a logger for your own code, you will almost certainly create a logger with handlers (file handler and/or console handler).
When you have not yet created a logger and you get the 'root' logger by
logger = logging.getLogger()
then this logger will have no handlers yet.
Therefore, I always check whether the logger above results in a logger that has handlers by
logging.getLogger().hasHandlers()
So I create a logger only if the root logger does not have handlers:
logger = <create_my_logger> if not logging.getLogger().hasHandlers() else logging.getLogger()
The "create_my_logger" snippet represents my code/function that returns a logger (with handlers).
WARNING. This is not documented. It may change without notice.
The logging module internally uses a single Manager object to hold the hierarchy of loggers in a dict accessible as:
import logging
logging.Logger.manager.loggerDict
All loggers except the root logger are stored in that dict under their names.
There are few examples on the net:
http://code.activestate.com/lists/python-list/621740/
and
https://michaelgoerz.net/notes/use-of-the-logging-module-in-python.html (uses Python 2 print)
Of course, if you attach a handler to the root logger, you will (generally) get all messages (unless a library author specifically chooses not to propagate to root handlers).
Why should you care if a handler listens to a log which hasn't been created yet? You can apply filters to a handler that ignore certain events.
This doesn't specifically answer your question, because I see your question as an instance of an XY Problem.
// , The one line answer to "how do I set my logging level to debug?" is usually something like the following:
logging.basicConfig(level=logging.DEBUG)
However, one may be tempted to change it later, back to something like INFO.
logging.basicConfig(level=logging.INFO)
This does not work, however. logging.basicConfig(level=logging.INFO), if run again at the same scope in the same process, will not result in a change to the level of the messages logged.
The official Python documentation, near the middle of https://docs.python.org/2/howto/logging.html#logging-to-a-file, lists the following:
The call to basicConfig() should come before any calls to debug(), info() etc. As it’s intended as a one-off simple configuration facility, only the first call will actually do anything: subsequent calls are effectively no-ops.
Source: https://docs.python.org/2/howto/logging.html#logging-basic-tutorial
Neither, however, does Logger.setLevel() work for this.
The following code attempts to set a lower log level:
logging.basicConfig(level=logging.DEBUG)
logging.debug("setting logging.basicConfig(level=logging.WARNING)")
logging.debug(myLogger.getEffectiveLevel())
logging.basicConfig(level=logging.WARNING) # This does nothing.
logging.getLogger('root').setLevel(level=logging.WARNING)
logging.debug(myLogger.getEffectiveLevel())
logging.debug("logger root, currently at " \
+ str(logging.getLogger('root').getEffectiveLevel()) \
+ " is enabled for" \
+ str(logging.getLogger('root').isEnabledFor(logging.DEBUG)))
This results in the following output:
DEBUG:root:setting logging.basicConfig(level=logging.WARNING)
DEBUG:root:10
DEBUG:root:30
DEBUG:root:30is enabled forFalse
It seems like one cannot "set" a Logger to a lower level of verbosity without access to something else created with logging.basicConfig().
How does a Python script change its logging level halfway through execution?
logging.disable(logging.INFO) → disable messages at logging.INFO
and lower, until...
logging.disable(logging.NOTSET) → resume normal logging
Link: https://docs.python.org/2/library/logging.html#logging.disable
Use this to temporarily throttle logging output.
It temporarily overrides the log levels for all loggers.
If you need to shut down logging, use the creatively named logging.shutdown()
First of all, logging.basicConfig() is a constructor, which creates a StreamHandler object to connect to the root logger.
Logging levels are just integer constants, not magic properties of a Python script.
To actually change the level for that specific logger, root, you may need to access the StreamHandler created for it by logging.basicConfig().
So I do logging.config.fileConfig to setup my logging from a file config that has console and file handler. Then I do logging.getLogger(name) to get my logger and log. At certain times I want the filehandler's filename to change i.e. log rotate (I can't use time rotator because of some issues with Windows platform) so to do that I call logger.handlers - it shows an empty list, so I cant close them!! However when I step through the debugger, its clearly not empty (well of course without it I wouldn't be able to log right)
Not sure whats going on here, any gotchas that I'm missing?
Appreciate any help. Thanks.
It seems you need to correctly get the root logger:
logger = logging.getLogger(__name__)
handlers = logger.handlers[:]
print('module {}'.format(handlers))
print('module {}'.format(logger.hasHandlers()))
logger = logging.getLogger('root')
handlers = logger.handlers[:]
print('root {}'.format(handlers))
print('root {}'.format(logger.hasHandlers()))
logger = logging.getLogger()
handlers = logger.handlers[:]
print('blank {}'.format(handlers))
print('blank {}'.format(logger.hasHandlers()))
output:
module []
module True
root []
root True
blank [<logging.handlers.RotatingFileHandler object at 0x108d82898>,
<logging.StreamHandler object at 0x108d826d8>]
blank True
Firstly the issue is that, if you use a config file to initialise logging with file and console handlers, then it does not populate logging.handlers list, so you can not iterate over it and close+flush the streams prior to opening new one with a new logging file name.
If you want to use TimeRotatingFileHandler or RotatingFileHandler, it sits under logging/handler.py and when it tries to do a roll over, it only closes its own stream, as it has no idea what streams the parent logging (mostly singleton) class may have open. And so when you do a roll over, there is a file lock (file filehandler) and boom it all fails.
So the solution (for me) is to initialise logging programatically and use addHandlers on logging, which also populates logging.handlers [], which I then use to iterate over my console/file handler and close them prior to manually rotating the file.
It to me looks like an obvious bug with the logging class, and if its working on unix - it really shouldn't.
Thanks everyone, especially #falsetru for your help.
You can use RotatingFileHandler (not TimedRotatingFileHandler).
Calling doRollover of the handler will rotate the log files.
Maybe there is no such name as 'TimeRoatingFileHandler' because you missed 'd' in word 'Timed'. So it must be: 'TimedRoatingFileHandler'
I have a log.py module, that is used in at least two other modules (server.py and device.py).
It has these globals:
fileLogger = logging.getLogger()
fileLogger.setLevel(logging.DEBUG)
consoleLogger = logging.getLogger()
consoleLogger.setLevel(logging.DEBUG)
file_logging_level_switch = {
'debug': fileLogger.debug,
'info': fileLogger.info,
'warning': fileLogger.warning,
'error': fileLogger.error,
'critical': fileLogger.critical
}
console_logging_level_switch = {
'debug': consoleLogger.debug,
'info': consoleLogger.info,
'warning': consoleLogger.warning,
'error': consoleLogger.error,
'critical': consoleLogger.critical
}
It has two functions:
def LoggingInit( logPath, logFile, html=True ):
global fileLogger
global consoleLogger
logFormatStr = "[%(asctime)s %(threadName)s, %(levelname)s] %(message)s"
consoleFormatStr = "[%(threadName)s, %(levelname)s] %(message)s"
if html:
logFormatStr = "<p>" + logFormatStr + "</p>"
# File Handler for log file
logFormatter = logging.Formatter(logFormatStr)
fileHandler = logging.FileHandler(
"{0}{1}.html".format( logPath, logFile ))
fileHandler.setFormatter( logFormatter )
fileLogger.addHandler( fileHandler )
# Stream Handler for stdout, stderr
consoleFormatter = logging.Formatter(consoleFormatStr)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter( consoleFormatter )
consoleLogger.addHandler( consoleHandler )
And:
def WriteLog( string, print_screen=True, remove_newlines=True,
level='debug' ):
if remove_newlines:
string = string.replace('\r', '').replace('\n', ' ')
if print_screen:
console_logging_level_switch[level](string)
file_logging_level_switch[level](string)
I call LoggingInit from server.py, which initializes the file and console loggers. I then call WriteLog from all over the place, so multiple threads are accessing fileLogger and consoleLogger.
Do I need any further protection for my log file? The documentation states that thread locks are handled by the handler.
The good news is that you don't need to do anything extra for thread safety, and you either need nothing extra or something almost trivial for clean shutdown. I'll get to the details later.
The bad news is that your code has a serious problem even before you get to that point: fileLogger and consoleLogger are the same object. From the documentation for getLogger():
Return a logger with the specified name or, if no name is specified, return a logger which is the root logger of the hierarchy.
So, you're getting the root logger and storing it as fileLogger, and then you're getting the root logger and storing it as consoleLogger. So, in LoggingInit, you initialize fileLogger, then re-initialize the same object under a different name with different values.
You can add multiple handlers to the same logger—and, since the only initialization you actually do for each is addHandler, your code will sort of work as intended, but only by accident. And only sort of. You will get two copies of each message in both logs if you pass print_screen=True, and you will get copies in the console even if you pass print_screen=False.
There's actually no reason for global variables at all; the whole point of getLogger() is that you can call it every time you need it and get the global root logger, so you don't need to store it anywhere.
A more minor problem is that you're not escaping the text you insert into HTML. At some point you're going to try to log the string "a < b" and end up in trouble.
Less seriously, a sequence of <p> tags that isn't inside a <body> inside an <html> is not a valid HTML document. But plenty of viewers will take care of that automatically, or you can post-process your logs trivially before displaying them. But if you really want this to be correct, you need to subclass FileHandler and have your __init__ add a header if given an empty file and remove a footer if present, then have your close add a footer.
Getting back to your actual question:
You do not need any additional locking. If a handler correctly implements createLock, acquire, and release (and it's called on a platform with threads), the logging machinery will automatically make sure to acquire the lock when needed to make sure each message is logged atomically.
As far as I know, the documentation doesn't directly say that StreamHandler and FileHandler implement these methods, it does strongly imply it (the text you mentioned in the question says "The logging module is intended to be thread-safe without any special work needing to be done by its clients", etc.). And you can look at the source for your implementation (e.g., CPython 3.3) and see that they both inherit correctly-implemented methods from logging.Handler.
Likewise, if a handler correctly implements flush and close, the logging machinery will make sure it's finalized correctly during normal shutdown.
Here, the documentation does explain what StreamHandler.flush(), FileHandler.flush(), and FileHandler.close(). They're mostly what you'd expect, except that StreamHandler.close() is a no-op, meaning it's possible that final log messages to the console may get lost. From the docs:
Note that the close() method is inherited from Handler and so does no output, so an explicit flush() call may be needed at times.
If this matters to you, and you want to fix it, you need to do something like this:
class ClosingStreamHandler(logging.StreamHandler):
def close(self):
self.flush()
super().close()
And then use ClosingStreamHandler() instead of StreamHandler().
FileHandler has no such problem.
The normal way to send logs to two places is to just use the root logger with two handlers, each with their own formatter.
Also, even if you do want two loggers, you don't need the separate console_logging_level_switch and file_logging_level_switch maps; calling Logger.debug(msg) is exactly the same thing as calling Logger.log(DEBUG, msg). You'll still need some way to map your custom level names debug, etc. to the standard names DEBUG, etc., but you can just do one lookup, instead of doing it once per logger (plus, if your names are just the standard names with different cast, you can cheat).
This is all described pretty well in the `Multiple handlers and formatters section, and the rest of the logging cookbook.
The only problem with the standard way of doing this is that you can't easily turn off console logging on a message-by-message basis. That's because it's not a normal thing to do. Usually, you just log by levels, and set the log level higher on the file log.
But, if you want more control, you can use filters. For example, give your FileHandler a filter that accepts everything, and your ConsoleHandler a filter that requires something starting with console, then use the filter 'console' if print_screen else ''. That reduces WriteLog to almost a one-liner.
You still need the extra two lines to remove newlines—but you can even do that in the filter, or via an adapter, if you want. (Again, see the cookbook.) And then WriteLog really is a one-liner.
Python logging is thread safe:
Is Python's logging module thread safe?
http://docs.python.org/2/library/logging.html#thread-safety
So you have no problem in the Python (library) code.
The routine that you call from multiple threads (WriteLog) does not write to any shared state. So you have no problem in your code.
So you are OK.