I would like to use logging to log information (currently same information) to both console and to a file.
However, I am seeing extra unwanted information being printed to the console.
I would like to get the following output to both console and the file:
INFO - thisuser executed the pipeline at 2019-04-17 13:44:50,626
default log message
other default log message
INFO - pipeline execution completed at 2019-04-17 13:44:50,627
INFO - total time elapsed: 100.4 minutes
I am getting the expected output in the file, but console outputs the following:
INFO:start_log:thisuser
INFO - thisuser executed the pipeline at 2019-04-17 13:44:50,626
INFO:root:default log message
default log message
INFO:root:other default log message
other default log message
INFO:end_log:-
INFO - pipeline execution completed at 2019-04-17 13:44:50,627
INFO:duration_log:100.4
INFO - total time elapsed: 100.4 minutes
I would like to remove extra information (odd lines above) printed to the console. Any help will be greatly appreciated!
Below is the code I am running:
import logging
import getpass
class DispatchingFormatter:
def __init__(self, formatters, default_formatter):
self._formatters = formatters
self._default_formatter = default_formatter
def format(self, record):
formatter = self._formatters.get(record.name, self._default_formatter)
return formatter.format(record)
logging.basicConfig(level=logging.INFO)
formatter = DispatchingFormatter({
'start_log': logging.Formatter('%(levelname)s - %(message)s executed the pipeline at %(asctime)s'),
'end_log': logging.Formatter('%(levelname)s - pipeline execution completed at %(asctime)s'),
'duration_log': logging.Formatter('%(levelname)s - total time elapsed: %(message)s minutes')
},
logging.Formatter('%(message)s'),
)
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter)
f_handler = logging.FileHandler('log.txt')
f_handler.setFormatter(formatter)
logging.getLogger().addHandler(c_handler)
logging.getLogger().addHandler(f_handler)
logging.getLogger('start_log').info(f'{getpass.getuser()}')
logging.info('default log message')
logging.info('other default log message')
logging.getLogger('end_log').info('-')
time_elapsed = 100.4
logging.getLogger('duration_log').info(f'{time_elapsed}')
The pipeline will be printing ~100 lines of information (results of the analysis), which I would not like to prepend by any logging level, hence I tried to implement using multiple formatters.
I know this is somewhat hacky solution.. if someone has general suggestions for improvements, this will also be appreciated!
As the documentation for logging.basicConfig states:
Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger.
So there is already a stream handler attached. You can remove it by calling:
logging.getLogger().handlers.clear()
before you add the other handlers.
Alternatively you can use logging.getLogger().setLevel(logging.INFO) instead of basicConfig. In that case you don't have to clear the handlers since they are none attached by default.
Upon searching around (Replace default handler of Python logger) and reading up more on the documentation (https://docs.python.org/3.7/library/logging.html#), I found a working solution.
I was adding two handlers (one to print to console and another to print to file with the specified formatting) to the logger's default handler, and apparently the logger's default handler prints to the console those lines that I did not want.
A solution (perhaps a hacky one) is to first empty out the handlers by doing the following:
logging.basicConfig(level=logging.INFO, filemode='w')
# Added two lines below to first empty out (remove original handler) handlers
root_logger = logging.getLogger()
root_logger.handlers = []
Then continue on to create formatter objects I want and add two handlers that I wanted (same code as in the question):
formatter = DispatchingFormatter({
'start_log': logging.Formatter('%(levelname)s - %(message)s executed the pipeline at %(asctime)s', datefmt='%Y-%m-%d %H:%M:%S'),
'end_log': logging.Formatter('%(levelname)s - pipeline execution completed at %(asctime)s', datefmt='%Y-%m-%d %H:%M:%S'),
'duration_log': logging.Formatter('%(levelname)s - total time elapsed: %(message)s minutes')
},
logging.Formatter('%(message)s'),
)
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter)
f_handler = logging.FileHandler('log.txt', 'w+')
f_handler.setFormatter(formatter)
logging.getLogger().addHandler(c_handler)
logging.getLogger().addHandler(f_handler)
Related
Basically, I want to initialise a logger for each purpose, such that each logger will log to a file, but I notice that the content of log has been printed to console as well. I don't want that to happen, but I don't know how to adjust my code.
def getLogger(loggerName:str, fileDir:str,
level = logging.DEBUG,
format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'):
"""return a logger for logging
somehow the logDecorator function does
Args:
loggerName (str): name given to the logger
fileDir (str): directory
level ([type], optional): [description]. Defaults to logging.DEBUG.
format (str, optional): [description]. Defaults to '%(asctime)s - %(name)s - %(levelname)s - %(message)s'.
Returns:
logger: [description]
"""
myLogger = logging.getLogger(loggerName)
myLogger.setLevel(level)
myFormat = logging.Formatter(format)
myHandler = logging.FileHandler(fileDir, mode='a')
myHandler.setFormatter(myFormat)
myLogger.addHandler(myHandler)
# console only prints WARNING & ABOVE
# ideally, what I thought is that if I don't not
# add a SteamHandler, the log won't even print to console at all
# but adding a SteamHandler still prints EVERYTHING to the console
# despite the level set as WARNING
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(myFormat)
myLogger.addHandler(console)
return myLogger
I have tried this:
mainLogger = logging.getLogger()
mainLogger.setLevel(logging.WARNING) # only log WARNING and above
but It won't work as well, still printing to console.
So I am lost in my understanding of LOGGING module, I am not sure where have I done wrong.
Example of using the code:
aRandomLogger = getLogger(loggerName = 'ProgressLogger', fileDir = '../logs/ProgressLogger.log')
aRandomLogger.info('This is an information, I want it to be log to file, but not on console')
aRandomLogger.warning('this is a warning, I want it be be log to file, and shown on console')
Turns out, both logs are saved to file and to console.
It has been awhile, but I have managed to figure out the solution to this.
myLogger = logging.getLogger(loggerName)
myLogger.propagate = False
Setting propagate to False will prevent the logger from producing unwanted logs everywhere.
So I have been having this weird problem in my Spyder IDE (and only in Spyder).
I initialize a logger using the logging module, and each time I run the script the messages are printed in more and more times (once on the first try, twice on the second, thrice on the third, etc...). Any ideas why?
The code:
import logging
logger = logging.getLogger(__name__)
handle = logging.StreamHandler()
logger.addHandler(handle)
handle.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt='%(asctime)s - %(levelname)s - %(message)s')
handle.setFormatter(formatter)
logger.warning('Testing')
Thanks in advance!
EDIT: This is what I see in my console after two tries:
logger.warning('Testing')
2020-09-30 15:34:34,763 - WARNING - Testing
logger.warning('Testing')
2020-09-30 15:34:38,476 - WARNING - Testing
2020-09-30 15:34:38,476 - WARNING - Testing
The logging module uses a static logger object that lives in your python session. So, every time you run your script, addHandler will add a new handler to logger.handlers. You can add a line to your script that prints logger.handlers and check.
...
logger.addHandler(handle)
print(logger.handlers)
...
And every time you call logger.warning the message will be printed as many times as the number of handlers you have.
To fix this you need to remove the handler:
...
logger.warning('Testing')
logger.removeHandler(handle)
Another fix is to check if you already have a handler before adding a new one. This way you will only have one handler, and you won't need to remove it.
...
if not logger.handlers:
handle = logging.StreamHandler()
formatter = logging.Formatter(fmt='%(asctime)s - %(levelname)s - %(message)s')
handle.setFormatter(formatter)
handle.setLevel(logging.DEBUG)
logger.addHandler(handle)
...
You could also look into removing all handlers first.
I'm simply trying to have a python logger with a specific format that outputs log messages only to the console. I've tried many different things but I keep getting 2 lines of console output per log call.
Here is my code:
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
# Create console handler
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(asctime)s - %(name)s - %(message)s')
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
logger.info('TEST LOG info')
With output:
INFO - 2017-08-21 14:30:00,751 - my_logger - TEST LOG info
INFO:my_logger:TEST LOG info
I did exactly this and it didn't work: Disable output of root logger
Any idea what is going on? I don't care whether I use the root logger or not, I just want one line
The above code actually should be working correctly. Although my script was very lean, it was importing a non-system library which, somewhere down the line, had some logging configured which was affecting my output.
I have two log handlers in my code: a StreamHandler to write INFO level logs from the same module to stdout, and a FileHandler to write more verbose, DEBUG logs to a file. This is my code:
import sys
import logging
log = logging.getLogger('mymodule')
log.setLevel(logging.DEBUG)
logf = logging.FileHandler('file.log')
logf.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
log.addHandler(logf)
logs = logging.StreamHandler(sys.stdout)
logs.setLevel(logging.INFO)
logs.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
log.addHandler(logs)
However, I want the FileHandler to also write DEBUG info from other modules. I can achieve this if I remove the name from logging.getLogger(), but this will also affect my StreamHandler, which I only want to print output from my own module.
So is there a way to have either of the handlers use a different name?
There's a name attribute available on the base Handler class. I took a look at the Python source and doesn't look like it's used internally so you could manually set each handler to a different name.
I've spent a bit of time looking through the site at Python logger questions hoping my would be resolved there. I've set up a logger with two stream handlers that have both different formats and levels of logging, here's a functional snippet from my codebase:
import os
import time
import logging
LOG_LEVELS = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG]
TEST_RESULT_LEVELV_NUM = 51
# http://stackoverflow.com/a/11784984/196832
def status(self, message, *args, **kws):
self._log(TEST_RESULT_LEVELV_NUM, message, args, **kws)
logging.addLevelName(TEST_RESULT_LEVELV_NUM, "RESULT")
logging.Logger.result = status
def setup_logging(level=0, quiet=False, logdir=None):
logger = logging.getLogger('juju-test')
ffmt = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s: %(message)s')
cfmt = logging.Formatter('%(name)s %(levelname)s: %(message)s')
#logger.setLevel(0)
if level >= len(LOG_LEVELS):
level = len(LOG_LEVELS) - 1
if logdir:
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile = os.path.join(logdir, 'juju-test.%s.log' % int(time.time()))
fh = logging.FileHandler(logfile)
# Always at least log to INFO for file, unless DEBUG is requested
fh.setLevel(LOG_LEVELS[level if level >= 2 else 2])
fh.setFormatter(ffmt)
logger.addHandler(fh)
if not quiet:
ch = logging.StreamHandler()
ch.setLevel(LOG_LEVELS[level])
ch.setFormatter(cfmt)
logger.addHandler(ch)
return logger
I've been using an argparse to feed this, but for testing purposes if you feed the following to function:
logger = setup_logging(level=1, logdir="/tmp/oofrab/")
logger.info('Informative!')
logger.warn('Whoa buddy!')
logger.error('Look what you did.')
logger.result("They told me not to make a custom logging level, I'll show them!")
logger.debug('Lots of bugs, man')
I'd expect to see status, error, and warn in the console. Then status, error, warn and info in the log. However, I only see down to warn in both console and log file, despite selecting logging.INFO (key 2 in the LOG_LEVELS list) for the file handler. Is this expected?
I'm not using basicConfig or anything else when building the logger, why can't I have these two custom levels?
Apparently, logging.NOTSET does not mean "All levels", but rather defaults. So setting the parent logger to level 0 does only reverts it to it's default accepted levels. That being said, if I set logger.setLevel to logging.DEBUG that essentially sets logging to accept ALL levels then passes filtering on to the various handlers to filter further.
To get around this (and potential custom log levels) I've set the initial logger level to 1