Basically, I want to initialise a logger for each purpose, such that each logger will log to a file, but I notice that the content of log has been printed to console as well. I don't want that to happen, but I don't know how to adjust my code.
def getLogger(loggerName:str, fileDir:str,
level = logging.DEBUG,
format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'):
"""return a logger for logging
somehow the logDecorator function does
Args:
loggerName (str): name given to the logger
fileDir (str): directory
level ([type], optional): [description]. Defaults to logging.DEBUG.
format (str, optional): [description]. Defaults to '%(asctime)s - %(name)s - %(levelname)s - %(message)s'.
Returns:
logger: [description]
"""
myLogger = logging.getLogger(loggerName)
myLogger.setLevel(level)
myFormat = logging.Formatter(format)
myHandler = logging.FileHandler(fileDir, mode='a')
myHandler.setFormatter(myFormat)
myLogger.addHandler(myHandler)
# console only prints WARNING & ABOVE
# ideally, what I thought is that if I don't not
# add a SteamHandler, the log won't even print to console at all
# but adding a SteamHandler still prints EVERYTHING to the console
# despite the level set as WARNING
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(myFormat)
myLogger.addHandler(console)
return myLogger
I have tried this:
mainLogger = logging.getLogger()
mainLogger.setLevel(logging.WARNING) # only log WARNING and above
but It won't work as well, still printing to console.
So I am lost in my understanding of LOGGING module, I am not sure where have I done wrong.
Example of using the code:
aRandomLogger = getLogger(loggerName = 'ProgressLogger', fileDir = '../logs/ProgressLogger.log')
aRandomLogger.info('This is an information, I want it to be log to file, but not on console')
aRandomLogger.warning('this is a warning, I want it be be log to file, and shown on console')
Turns out, both logs are saved to file and to console.
It has been awhile, but I have managed to figure out the solution to this.
myLogger = logging.getLogger(loggerName)
myLogger.propagate = False
Setting propagate to False will prevent the logger from producing unwanted logs everywhere.
Related
I will start by saying the I found a similar question and none of the suggested solutions worked for me at the end of 2022.
I do not manage to do simple things like changing the severity level of the logging on Colab:
Or writing the logs to a file:
import logging
logging.basicConfig(filename='app.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s', force=True)
logging.warning('This will get logged to a file')
As Colab is very popular and so it is the logging module, I thought it makes sense to ask again.
For the first problem, the debug message does not print because the lowest severity of the root logger is set to WARNING. You can change it to DEBUG with
import logging
logging.getLogger('RootLogger').setLevel(logging.DEBUG)
For the second problem, as Colab is still on Python 3.7 and not 3.8, the force= parameter does not work yet. So you just need to manually delete the default StreamHandler that sends logging output to sys.stderr before carrying out your configuration. (code source: https://stackoverflow.com/a/49202811/9987623)
import logging
# Check current handlers
print(logging.root.handlers)
# [<StreamHandler stderr (NOTSET)>]
# Remove all handlers associated with the root logger object.
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
print(logging.root.handlers)
# []
Now, you can run your code. Here is an example:
from pathlib import Path
fmt = '%(name)s - %(levelname)s - %(message)s'
log_dir = Path.cwd().joinpath("logs")
path = Path(log_dir)
path.mkdir(exist_ok=True)
log_fname = "app.log"
log_path = log_dir.joinpath(log_fname)
logging.basicConfig(
level=logging.DEBUG,
filemode="w",
format=fmt,
filename=log_path,
)
logging.warning("this is a warning")
Next cell:
!cat ./logs/app.log
# root - WARNING - this is a warning
Next cell:
logging.debug('this is a debug message')
Next cell:
!cat ./logs/app.log
# root - WARNING - this is a warning
# root - DEBUG - this is a debug message
I have a file named helper.py
import logging
import os
from json import load
def get_config(value):
with open('config.json','r') as f:
result=load(f)[value]
return result
def get_logger(name,level):
logpath=get_config("log_path")
if not os.path.exists(logpath):
os.mkdir(logpath)
logger = logging.getLogger(name)
if not bool(logger.handlers):
formatter = logging.Formatter('%(asctime)s.%(msecs)03d - %(name)s - %(levelname)s - %(message)s',datefmt='%Y-%m-%d %H:%M:%S')
fh = logging.FileHandler(os.path.join(logpath,f'{get_config("log_file_name")}.log'),mode="w",encoding='utf-8')
fh.setFormatter(formatter)
logger.addHandler(fh)
ch = logging.StreamHandler()
ch.setFormatter(formatter)
ch.setLevel(level)
logger.addHandler(ch)
return logger
LOGGER=get_logger("MyLogger",logging.INFO)
This is config.json:
{
"save_path" : "results/",
"log_path" : "logs/",
"log_file_name" : "MyLog"
}
let's say I am using LOGGER from helper it in x.py
from helper import LOGGER
logger=LOGGER
def div(x,y):
try:
logger.info("inside div")
return x/y
except Exception as e:
logger.error(f"div failed due to {e.message if 'message' in dir(e) else e}")
I am using this LOGGER in other files by importing helper.LOGGER for logging purposes but it's not printing anything on the console nor writing in a log file
My attempt:
I tried adding sys.stdout in StreamHandler() It doesn't worked
Then I tried setting the level of fh but nothing works
I tried adding basicConfig() instead of fileHandler() but then printing to console using print() and the output of logs is not coming in the correct order
Kindly let me know where I go wrong
Any help is appreciated :)
Thanks :)
You are not setting the level on the LOGGER, which by default is warning. This is why your info level log is not appearing. The Python documentation has a flow chart illustrating when a log will be logged: https://docs.python.org/3/howto/logging.html#logging-flow
The first thing it does, is that before a logger sends a log to their handlers it checks if the level is enabled for the logger. You should add logger.setLevel(level) in your get_logger().
I would like to use logging to log information (currently same information) to both console and to a file.
However, I am seeing extra unwanted information being printed to the console.
I would like to get the following output to both console and the file:
INFO - thisuser executed the pipeline at 2019-04-17 13:44:50,626
default log message
other default log message
INFO - pipeline execution completed at 2019-04-17 13:44:50,627
INFO - total time elapsed: 100.4 minutes
I am getting the expected output in the file, but console outputs the following:
INFO:start_log:thisuser
INFO - thisuser executed the pipeline at 2019-04-17 13:44:50,626
INFO:root:default log message
default log message
INFO:root:other default log message
other default log message
INFO:end_log:-
INFO - pipeline execution completed at 2019-04-17 13:44:50,627
INFO:duration_log:100.4
INFO - total time elapsed: 100.4 minutes
I would like to remove extra information (odd lines above) printed to the console. Any help will be greatly appreciated!
Below is the code I am running:
import logging
import getpass
class DispatchingFormatter:
def __init__(self, formatters, default_formatter):
self._formatters = formatters
self._default_formatter = default_formatter
def format(self, record):
formatter = self._formatters.get(record.name, self._default_formatter)
return formatter.format(record)
logging.basicConfig(level=logging.INFO)
formatter = DispatchingFormatter({
'start_log': logging.Formatter('%(levelname)s - %(message)s executed the pipeline at %(asctime)s'),
'end_log': logging.Formatter('%(levelname)s - pipeline execution completed at %(asctime)s'),
'duration_log': logging.Formatter('%(levelname)s - total time elapsed: %(message)s minutes')
},
logging.Formatter('%(message)s'),
)
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter)
f_handler = logging.FileHandler('log.txt')
f_handler.setFormatter(formatter)
logging.getLogger().addHandler(c_handler)
logging.getLogger().addHandler(f_handler)
logging.getLogger('start_log').info(f'{getpass.getuser()}')
logging.info('default log message')
logging.info('other default log message')
logging.getLogger('end_log').info('-')
time_elapsed = 100.4
logging.getLogger('duration_log').info(f'{time_elapsed}')
The pipeline will be printing ~100 lines of information (results of the analysis), which I would not like to prepend by any logging level, hence I tried to implement using multiple formatters.
I know this is somewhat hacky solution.. if someone has general suggestions for improvements, this will also be appreciated!
As the documentation for logging.basicConfig states:
Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger.
So there is already a stream handler attached. You can remove it by calling:
logging.getLogger().handlers.clear()
before you add the other handlers.
Alternatively you can use logging.getLogger().setLevel(logging.INFO) instead of basicConfig. In that case you don't have to clear the handlers since they are none attached by default.
Upon searching around (Replace default handler of Python logger) and reading up more on the documentation (https://docs.python.org/3.7/library/logging.html#), I found a working solution.
I was adding two handlers (one to print to console and another to print to file with the specified formatting) to the logger's default handler, and apparently the logger's default handler prints to the console those lines that I did not want.
A solution (perhaps a hacky one) is to first empty out the handlers by doing the following:
logging.basicConfig(level=logging.INFO, filemode='w')
# Added two lines below to first empty out (remove original handler) handlers
root_logger = logging.getLogger()
root_logger.handlers = []
Then continue on to create formatter objects I want and add two handlers that I wanted (same code as in the question):
formatter = DispatchingFormatter({
'start_log': logging.Formatter('%(levelname)s - %(message)s executed the pipeline at %(asctime)s', datefmt='%Y-%m-%d %H:%M:%S'),
'end_log': logging.Formatter('%(levelname)s - pipeline execution completed at %(asctime)s', datefmt='%Y-%m-%d %H:%M:%S'),
'duration_log': logging.Formatter('%(levelname)s - total time elapsed: %(message)s minutes')
},
logging.Formatter('%(message)s'),
)
c_handler = logging.StreamHandler()
c_handler.setFormatter(formatter)
f_handler = logging.FileHandler('log.txt', 'w+')
f_handler.setFormatter(formatter)
logging.getLogger().addHandler(c_handler)
logging.getLogger().addHandler(f_handler)
I'm simply trying to have a python logger with a specific format that outputs log messages only to the console. I've tried many different things but I keep getting 2 lines of console output per log call.
Here is my code:
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
# Create console handler
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(asctime)s - %(name)s - %(message)s')
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
logger.info('TEST LOG info')
With output:
INFO - 2017-08-21 14:30:00,751 - my_logger - TEST LOG info
INFO:my_logger:TEST LOG info
I did exactly this and it didn't work: Disable output of root logger
Any idea what is going on? I don't care whether I use the root logger or not, I just want one line
The above code actually should be working correctly. Although my script was very lean, it was importing a non-system library which, somewhere down the line, had some logging configured which was affecting my output.
I've spent a bit of time looking through the site at Python logger questions hoping my would be resolved there. I've set up a logger with two stream handlers that have both different formats and levels of logging, here's a functional snippet from my codebase:
import os
import time
import logging
LOG_LEVELS = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG]
TEST_RESULT_LEVELV_NUM = 51
# http://stackoverflow.com/a/11784984/196832
def status(self, message, *args, **kws):
self._log(TEST_RESULT_LEVELV_NUM, message, args, **kws)
logging.addLevelName(TEST_RESULT_LEVELV_NUM, "RESULT")
logging.Logger.result = status
def setup_logging(level=0, quiet=False, logdir=None):
logger = logging.getLogger('juju-test')
ffmt = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s: %(message)s')
cfmt = logging.Formatter('%(name)s %(levelname)s: %(message)s')
#logger.setLevel(0)
if level >= len(LOG_LEVELS):
level = len(LOG_LEVELS) - 1
if logdir:
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile = os.path.join(logdir, 'juju-test.%s.log' % int(time.time()))
fh = logging.FileHandler(logfile)
# Always at least log to INFO for file, unless DEBUG is requested
fh.setLevel(LOG_LEVELS[level if level >= 2 else 2])
fh.setFormatter(ffmt)
logger.addHandler(fh)
if not quiet:
ch = logging.StreamHandler()
ch.setLevel(LOG_LEVELS[level])
ch.setFormatter(cfmt)
logger.addHandler(ch)
return logger
I've been using an argparse to feed this, but for testing purposes if you feed the following to function:
logger = setup_logging(level=1, logdir="/tmp/oofrab/")
logger.info('Informative!')
logger.warn('Whoa buddy!')
logger.error('Look what you did.')
logger.result("They told me not to make a custom logging level, I'll show them!")
logger.debug('Lots of bugs, man')
I'd expect to see status, error, and warn in the console. Then status, error, warn and info in the log. However, I only see down to warn in both console and log file, despite selecting logging.INFO (key 2 in the LOG_LEVELS list) for the file handler. Is this expected?
I'm not using basicConfig or anything else when building the logger, why can't I have these two custom levels?
Apparently, logging.NOTSET does not mean "All levels", but rather defaults. So setting the parent logger to level 0 does only reverts it to it's default accepted levels. That being said, if I set logger.setLevel to logging.DEBUG that essentially sets logging to accept ALL levels then passes filtering on to the various handlers to filter further.
To get around this (and potential custom log levels) I've set the initial logger level to 1