Develop Reference

Guarantee all logs go to file

I need to set up logging so it will always log to a file, no matter what a user or other programmer might setLevel too. For instance I have this logger set up:
initialize_logging.py
import logging
filename="./files/logs/attribution.log", level=logging.INFO)
logger = logging.getLogger('attribution')
formatter = logging.Formatter('%(asctime)s - %(levelname)s: %(message)s')
fh = logging.FileHandler('./files/logs/attribution.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
ch = logging.StreamHandler()
ch.setLevel(logging.WARNING)
ch.setFormatter(formatter)
logger.setLevel(logging.DEBUG)
logger.addHandler(fh)
logger.addHandler(ch)
then I have this file:
main.py
%load_ext autotime
import initialize_logging
import logging
logger = logging.getLogger('attribution')
logger.setLevel(logging.WARNING)
logger.info('TEST')
The above results in nothing being logged to my file and nothing being output.
However in main, if I setLevel(logging.INFO) then everything gets written to file and to the sterrout (I'm using a jupyter notebook so it prints on screen immediatly.)
The behavior I would like is that the user of my notebook can setLevel to determine what they want to see printed on screen but no matter what all logs get sent to the log file.
How do I do this?

Create a Logger subclass that is enabled for all levels and set it as the LoggerClass. Override the isEnabledFor function to return True regardless of the Logger's level. Override the setLevel function to find all of the Logger's StreamHandlers and set their level to be the same as the Logger. The behavior you want will rely on the level set for the handlers.
logging_init.py
class AllLevels(logging.Logger):
def isEnabledFor(self,level):
"""
Is this logger enabled for level 'level'?
"""
return True
def setLevel(self,level):
super().setLevel(level)
for h in self.handlers:
if isinstance(h,logging.StreamHandler):
h.setLevel(self.level)
filename="attribution.log"
logging.setLoggerClass(AllLevels)
logger = logging.getLogger('attribution')
formatter = logging.Formatter('%(asctime)s - %(levelname)s: %(message)s')
fh = logging.FileHandler(filename)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
ch = logging.StreamHandler()
ch.setLevel(logging.WARNING)
ch.setFormatter(formatter)
logger.setLevel(logging.DEBUG)
logger.addHandler(fh)
logger.addHandler(ch)
caveat: I have not tried to think of all or any ways that this could go wrong. I only attempted to create the behavior.
Maybe instead of setting all StreamHandlers to the same level as the logger, only set a StreamHandler with a specific name.
Nothing will stop someone from inspecting the logger; finding the handlers; arbitrarily changing their levels. Maybe use a FileHandler subclass with its setLevel function overridden.
class AlwaysFileHandler(logging.FileHandler):
def setLevel(self,level):
"""
Set the logging level of this handler to DEBUG.
"""
self.level = logging.DEBUG

Logging with two handlers - one to file and one to stderr

I have the following code to create two logging handlers. The goal is to have the stream_handler write only to the sterr and the file_handler only to file.
In the code below, stream_handler writes to the file as well, so I have duplicate log entries whenever I log a message. How can I modify it to get the stream_handler not to write to the file?
def create_timed_rotating_log(
name='log',
path='logs/',
when='D',
interval=1,
backupCount=30,
form='%(asctime)s | %(name)s | %(levelname)s: %(message)s',
sterr = False,
verbose=False):
logger = logging.getLogger(name)
formatter = logging.Formatter(form)
logger.setLevel(logging.DEBUG)
if sterr:
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.DEBUG if verbose else logging.ERROR)
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
file_handler = TimedRotatingFileHandler(filename=path+name,
when=when,
interval=interval,
backupCount=backupCount)
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG if verbose else logging.ERROR)
logger.addHandler(file_handler)
return logger

Looks like you have considered that the messages go to both handlers as duplication. If you wanted the file not to receive messages and them to only go to stderr, when stderr=True, you never to put the code that adds the file handler in an else so that it is only added if stderr=False
Specifically:
if sterr:
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.DEBUG if verbose else logging.ERROR)
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
else:
file_handler = TimedRotatingFileHandler(filename=path+name,
when=when,
interval=interval,
backupCount=backupCount)
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG if verbose else logging.ERROR)
logger.addHandler(file_handler)
I had considered this possible but less likely than the more common issue of setting up handlers multiple times when how you are supposed use logging is that in the main module you add the handlers to the root logger and then only get the logger for the current module via logger = logging.getLogger(__name__). This and import logging is the only logging code that should appear in any but the module where if __name__=="__main__": occurs where right after that it should setup the logging.
The only way you will have duplicates is if you call that function in your program twice. The issue isn't these two handlers but the other two handlers from the second call to this setup function.
You need to just add the handlers once to the root logger not once for every logger in each module.

Not exactly sure why this is the case but adding the StreamHandler first causes it to also write to the file. I moved the StreamHandler to after adding the TimedRotatingFileHandler and this resolved the problem.

python.logging: Why does my non-basicConfig setting not work?

I want to log to a single log file from main and all sub modules.
The log messages send from a main file, where I define the logger, work as expected. But the ones send from a call to an imported function are missing.
It is working if I use logging.basicConfig as in Example 1 below.
But the second example which allows for more custom settings does not work.
Any ideas why?
# in the submodule I have this code
import logging
logger = logging.getLogger(__name__)
EXAMPLE 1 - Working
Here I create two handlers and just pass them to basicConfig:
# definition of root looger in main module
formatter = logging.Formatter(fmt="%(asctime)s %(name)s.%(levelname)s: %(message)s", datefmt="%Y.%m.%d %H:%M:%S")
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
handler2 = logging.StreamHandler(stream=None)
handler2.setFormatter(formatter)
handler2.setLevel(logging.DEBUG)
logging.basicConfig(handlers=[handler, handler2], level=logging.DEBUG)
logger = logging.getLogger(__name__)
EXAMPLE 2 - Not working
Here I create two handlers and addHandler() them to the root logger:
# definition of root looger in main module
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler('logger.log')
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
#handler.setLevel(logging.ERROR)
logger.addHandler(handler)
handler = logging.StreamHandler(stream=None)
handler.setFormatter(formatter)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)

You need to configure the (one and only) root logger in the main module of your software. This is done by calling
logger = logging.getLogger() #without arguments
instead of
logger = logging.getLogger(__name__)
(Python doc on logging)
The second example creates a separate, child logger with the name of your script.
If there are no handlers defined in the submodules, the log message is being passed down to the root logger to handle it.
A related question can be found here:
Python Logging - How to inherit root logger level & handler

Suspend the formatting of the logger, then go back to it

I have a logging configuration where I log to a file and to the console:
logging.basicConfig(filename=logfile, filemode='w',
level=numlevel,
format='%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
# add console messages
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console.setFormatter(consoleformatter)
logging.getLogger('').addHandler(console)
At some point in my script, i need to interact with the user by printing a summary and asking for confirmation. The summary is currently produced by prints in a loop. I would like to suspend the current format of the console logs, so that I can printout one big block of text with a question at the end and wait for user input. But i still want all of this to be logged to file!
The function that does this is in a module, where I tried the following :
logger = logging.getLogger(__name__)
def summaryfunc:
logger.info('normal logging business')
clearformatter = logging.Formatter('%(message)s')
logger.setFormatter(clearformatter)
logger.info('\n##########################################')
logger.info('Summary starts here')
Which yields the error: AttributeError: 'Logger' object has no attribute 'setFormatter'
I understand that a logger is a logger, not a handler, but i'm not sure on how to get things to work...
EDIT:
Following the answers, my problem turned into : how can i suspend logging to the console when interacting with the user, while still being able to log to file. IE: suspend only the streamHandler. Since this is happening in a module, the specifics of the handlers are defined elsewhere, so here is how i did it :
logger.debug('Normal logging to file and console')
root_logger = logging.getLogger()
stream_handler = root_logger.handlers[1]
root_logger.removeHandler(stream_handler)
print('User interaction')
logger.info('Logging to file only')
root_logger.addHandler(stream_handler)
logger.info('Back to logging to both file and console')
This relies on the streamHandler always being the second in the list returned by handlers but i believe this is the case because it's in the order I added the handlers to the root logger...

I agree with Vinay that you should use print for normal program output and only use logging for logging purpose. However, if you still want to switch formatting in the middle, then switch back, here is how to do it:
import logging
def summarize():
console_handler.setFormatter(logging.Formatter('%(message)s'))
logger.info('Here is my report')
console_handler.setFormatter(console_formatter)
numlevel = logging.DEBUG
logfile = 's2.log'
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
console_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console_handler.setFormatter(console_formatter)
logger.addHandler(console_handler)
file_handler = logging.FileHandler(filename=logfile, mode='w')
file_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
file_handler.setFormatter(file_formatter)
logger.addHandler(file_handler)
logger.info('Before summary')
summarize()
logger.info('After summary')
Discussion
The script creates a logger object and assigned to it two handlers: one for the console and one for file.
In the function summarize(), I switched in a new formatter for the console handler, do some logging, then switched back.
Again, let me remind that you should not use logging to display normal program output.
Update
If you want to suppress console logging, then turn it back on. Here is a suggestion:
def interact():
# Remove the console handler
for handler in logger.handlers:
if not isinstance(handler, logging.FileHandler):
saved_handler = handler
logger.removeHandler(handler)
break
# Interact
logger.info('to file only')
# Add the console handler back
logger.addHandler(saved_handler)
Note that I did not test the handlers against logging.StreamHandler since a logging.FileHandler is derived from logging.StreamHandler. Therefore, I removed those handlers that are not FileHandler. Before removing, I saved that handler for later restoration.
Update 2: .handlers = []
In the main script, if you have:
logger = logging.getLogger(__name__) # __name__ == '__main__'
Then in a module, you do:
logger = logging.getLogger(__name__) # __name__ == module's name, not '__main__'
The problem is, in the script, __name__ == '__main__' and in the module, __name__ == <the module's name> and not '__main__'. In order to achieve consistency, you will need to make up some name and use them in both places:
logger = logging.getLogger('MyScript')

Logging shouldn't be used to provide the actual output of your program - a program is supposed to run the same way if logging is turned off completely. So I would suggest that it's better to do what you were doing before, i.e. prints in a loop.

Python logging: Set handlers for all loggers of used modules

I have my main script that interprets cli commands with argparse and then starts the application by calling the according stuff form an other module (made by myself).
My question now is how I can attach a handler to the logger from that module. The logger is retrieved with
logger = logging.getLogger(__name__)
I hence put following in my main script:
consoleHandler = logging.StreamHandler()
logger = logging.getLogger('MyModule')
logger.addHandler(consoleHandler)
However there is 0 logging output from 'MyModule'. Log levels are correct, eg there should be an output.
In MyModule I have the following:
logging.getLogger(__name__).addHandler(logging.NullHandler())
However removing that makes no difference.
So how can I correctly attach a handler to the logger of MyModule?

The best example of adding handler to logger is the one from Docs, See below.
Did you added consoleHandler with setLevel() and the Formatter() ?
As for this line:
logging.getLogger(__name__).addHandler(logging.NullHandler())
Why are you using it this way? I would recommend to follow the next code for adding an handler the right way.
import logging
# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')

The reason you're getting no output is most likely that you haven't set a level on the logger, so it will default to WARNING. Try doing logger.setLevel(logging.DEBUG) and see if that causes output to be produced. I know you said that log levels are correct, but I can't see any other reason why no output would be produced. You can try posting a small self-contained script that demonstrates the problem, if you like.
NullHandler only needs to be added for library modules to cater for their use in an application that doesn't configure logging. It is not needed in modules of an application that configures logging.
In general, you can add a handler to the root logger and it will be used by loggers in all modules.
Update: The comment to Kobi K's answer indicates that a level has been set on the handler - this is not enough, as you need to set a level on the logger, whose level is checked first. Only if the event is allowed through by the logger's level will the handlers (and their levels) come into play.