I have setup logging as follows:
def setUp():
LOG_FORMAT = '%(asctime)s %(levelname)-8s %(name)s %(message)s'
#LOG_FORMAT = '%(asctime)s %(name)s %(message)s'
logging.basicConfig(level=logging.DEBUG, format=LOG_FORMAT)
formatter = logging.Formatter(LOG_FORMAT)
ch = logging.StreamHandler()
ch.setLevel(logging.ERROR)
ch.setFormatter(formatter)
logging.getLogger().addHandler(ch)
LOG_FILENAME = 'file.log'
fh = logging.FileHandler(LOG_FILENAME, 'w')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logging.getLogger().addHandler(fh)
However, the console still shows DEBUG messages. Am I missing something here?
Note that setting the level to ERROR on fh works fine.
I think you need to remove the call to logging.basicConfig. That function adds another logging.StreamHandler that probably is the one that is printing the messages you don't want to be printed.
To check this you can take a look at the handlers attribute for the root logger (it's a list with all the handlers being used) and verify how many logging.StreamHandlers there are. Also, probably the message with level set to logging.ERROR are printed twice because of the two logging.StreamHandlers.
My final advice is avoid using logging.basicConfig if you're going to explicitly configure the handlers in the code.
Edit: Just for completeness, the source code of logging.BasicConfig is as follows:
if len(root.handlers) == 0:
filename = kwargs.get("filename")
if filename:
mode = kwargs.get("filemode", 'a')
hdlr = FileHandler(filename, mode)
else:
stream = kwargs.get("stream")
hdlr = StreamHandler(stream)
fs = kwargs.get("format", BASIC_FORMAT)
dfs = kwargs.get("datefmt", None)
fmt = Formatter(fs, dfs)
hdlr.setFormatter(fmt)
root.addHandler(hdlr)
level = kwargs.get("level")
if level is not None:
root.setLevel(level)
where you can see that unless filename is passed, a logging.StreamHandler is created.
From Python docs on logging.basicConfig:
Does basic configuration for the logging system by creating a
StreamHandler with a default Formatter and adding it to the root
logger.
As you set debug level of root logger to logging.DEBUG and you didn't switched off forwarding messages up to the root logger your DEBUG messages get logged by this StreamHandler created by basicConfig
Related
I ran the following using both the Python shell, and run it as a Python file from the command line. I don's see my log output at all.
import logging
formatter = logging.Formatter('%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s')
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(stream_handler)
logger.info(("info logging"))
Your logging output was almost correct with the exception of setLevel. The logging level needs to be defined on the logger instance instead of the handler instance. Your code therefore only needs a very small tweak to make it work:
import logging
formatter = logging.Formatter(
'%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s'
)
stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(stream_handler)
logger.setLevel(logging.INFO)
logger.info("info logging")
This piece of code produces the following output:
2019-08-21 15:04:55,118,118 INFO [testHandler.py:11] info logging
Note, I also removed the double brackets on the logger.info call as these are not necessary.
Use logging.basicConfig() to initialize the logging system.
You need to set the level of your logger,
logger.setLevel(logging.INFO)
Below statement can be removed from your code,
stream_handler.setLevel(logging.INFO)
A logger and a handler can have different levels. You have only set the level for the handler, not the logger itself.
import logging
formatter = logging.Formatter('%(asctime)s,%(msecs)d %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s')
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logger = logging.getLogger()
logger.setLevel(logging.INFO) # Required
logger.addHandler(stream_handler)
logger.info(("info logging"))
I can't figure out why log events are being printed to the console when I have not defined a console handler. All of the examples I read through have explicitly defined a console handler (streamhandler) in order to print messages to the console.
I want these events to be printed to a file only.
import logging
logger = logging.getLogger(__name__)
my_format = '%(asctime)-25s %(levelname)-8s LOGGER: %(name)-12s MODULE: %(module)-15s FUNCTION: %(funcName)-30s MSG: %(message)s'
my_datefmt ='%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(format=my_format, datefmt=my_datefmt, level=logging.DEBUG)
formatter = logging.Formatter(my_format, datefmt=my_datefmt)
logger.setLevel(logging.DEBUG)
handler1 = logging.FileHandler('mylog.txt')
handler1.setLevel(logging.DEBUG)
handler1.setFormatter(formatter)
logger.addHandler(handler1)
logger.debug("Why is this printed to the console")
EDIT:
It has been pointed out that I was not considering the root logger. Upon calling logging.basicConfig, a default streamhandler is added to the root logger (logger = getLogger())
The root logger's handler can be modified, however I have found that I can just prevent my logger from propagating logs up to the root logger.
This can be done like so:
import logging
logger = logging.getLogger(__name__)
my_format = '%(asctime)-25s %(levelname)-8s LOGGER: %(name)-12s MODULE: %(module)-15s FUNCTION: %(funcName)-30s MSG: %(message)s'
my_datefmt ='%m/%d/%Y %I:%M:%S%p'
logging.basicConfig(format=my_format, datefmt=my_datefmt, level=logging.DEBUG)
formatter = logging.Formatter(my_format, datefmt=my_datefmt)
logger.setLevel(logging.DEBUG)
handler1 = logging.FileHandler('mylog.txt')
handler1.setLevel(logging.DEBUG)
handler1.setFormatter(formatter)
logger.addHandler(handler1)
logger.propagate = False ####
logger.debug("Why is this printed to the console")
> ipython
import logging
logging.basicConfig?
*****************logging.basicConfig**************
Signature: logging.basicConfig(**kwargs)
Docstring:
Do basic configuration for the logging system.
This function does nothing if the root logger already has handlers
configured. It is a convenience method intended for use by simple scripts
to do one-shot configuration of the logging package.
The default behaviour is to create a StreamHandler which writes to
sys.stderr, set a formatter using the BASIC_FORMAT format string, and
add the handler to the root logger.
...
You have 2 handlers.
Hi I am trying to enable logs on python and for some reason nothing gets logged. Can some one help me what mistake I am making.
def setup():
logger = logging.getLogger('mytest')
formatter = logging.Formatter('%(asctime)s %(message)s')
fileHandler = logging.FileHandler('mylog.log')
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(formatter)
logger.addHandler(fileHandler)
logger.debug('hello world')
The following example will make it a bit more clear. The main level you need to set is for the logger object. You can check the logger level by calling the getEffectiveLevel method.
def setup():
logger = logging.getLogger('mytest')
formatter = logging.Formatter('%(asctime)s %(message)s')
fileHandler = logging.FileHandler('mylog.log')
fileHandler.setLevel(logging.DEBUG)
fileHandler.setFormatter(formatter)
logger.addHandler(fileHandler)
logger.critical("before setting Level on logger, effective level = {}".format(
logger.getEffectiveLevel()))
#THIS WON'T BE logged becuse logger level is WARN
# and DEBUG < WARN
logger.debug('logging debug before setting debug level')
logger.setLevel(logging.DEBUG)
logger.critical("after setting Level on logger, effective level = {}".format(
logger.getEffectiveLevel()))
#THIS WILL BE logged in mylog.log because DEBUG >= DEBUG
logger.debug('logging debug after setting debug level')
fileHandler.setLevel(logging.CRITICAL)
#THIS WON'T BE logged in mylog.log because fileHandler level is CRITICAL--->
# and DEBUG < CRITICAL
logger.debug("this won't be printed in mylog.log")
#THIS WILL BE logged in mylog.log BECAUSE CRITICAL >= CRITICAL--->
logger.critical("logging at critical level")
I have setup a python logging RotatingFileHandler with the logging module and created a string formatting config. Here is my test script:
class ExceptionHandler :
def __init__ ( self ) :
self.log = self.setupLog ( "testlog" )
def setupLog (self, name) :
log = logging.getLogger(name)
logging.basicConfig(format="%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s")
#by setting our logger to the DEBUG level (lowest level) we will include all other levels by default
log.setLevel(logging.DEBUG)
#setup the rotating file handler to automatically increment the log file name when the max size is reached
log.addHandler( logging.handlers.RotatingFileHandler('%s.log' % name, mode='a', maxBytes=50000, backupCount=5) )
return log
if __name__ == "__main__" :
exceptionHandler = ExceptionHandler()
exceptionHandler.log.log( 20, "Successfully completed script!" )
What I'm expecting in my testlog.log file is this output; instead this is only being printed to stdout and not my file:
INFO 2015-05-06 09:07:55,472 <module> 56 Successfully completed script!
And what I'm getting in my file is simply the following without any string formatting:
Successfully completed script!
Does anyone know what the problem is with my log setup/config?
You're seeing the correct result on the console because the log is being propogated from your new Logger named testlog to the root logger which is still writing to the console. This Logger has the correct formatting as set
by basicConfig.
However, your new Handler does not inherit that config from basicConfig() because, as the docs say, all it does is
Does basic configuration for the logging system by creating a StreamHandler with a default Formatter and adding it to the root logger.
You need to add the formatting to your handler. Something like this:
#setup the rotating file handler to automatically increment the log file name when the max size is reached
file_handler = logging.handlers.RotatingFileHandler('%s.log' % name, mode='a', maxBytes=50000, backupCount=5)
file_handler.setFormatter(logging.Formatter(fmt="%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s"))
log.addHandler(file_handler)
N.B. If you don't want to pass the messages on to the root logger - add log.propagate = False in your setupLog function
I solved my issue by changing how the formatter gets applied to the logging handler like so:
def setupLog (self, name) :
log = logging.getLogger(name)
#by setting our logger to the DEBUG level (lowest level) we will include all other levels by default
log.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler('%s.log' % name, mode='a', maxBytes=50000, backupCount=5)
#logging.basicConfig(format="%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s")
handler.setFormatter( logging.Formatter( "%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s" ) )
#setup the rotating file handler to automatically increment the log file name when the max size is reached
log.addHandler( handler )
return log
try with the logging.basicConfig before the log = logging.getLogger(name)*
def setupLog (self, name) :
logging.basicConfig(format="%(levelname)s %(asctime)s %(funcName)s %(lineno)d %(message)s")
log = logging.getLogger(name)
I've read a few posts on this but I'm still confused. I have this logging setup:
import logging
class MongoHandler(logging.Handler):
def __init__(self):
logging.Handler.__init__(self)
from pymongo import Connection
self.db = Connection('db_server').db_name
def emit(self, record):
try:
self.db.Logging.save(record.__dict__)
except:
print 'Logging Error: Unable to save log entry to db'
mh = MongoHandler()
sh = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(threadName)s - %(levelname)s - %(message)s')
sh.setFormatter(formatter)
log = logging.getLogger('DeviceMonitor_%s' % hostname)
log.addHandler(mh)
log.addHandler(sh)
log.setLevel(logging.INFO)
I want to be able to set a different level for the StreamHandler and the MongoHandler. Is that possible or do I need to have a second Logger obj?
You can set a different logging level for each logging handler but it seems you will have to set the logger's level to the "lowest". In the example below I set the logger to DEBUG, the stream handler to INFO and the TimedRotatingFileHandler to DEBUG. So the file has DEBUG entries and the stream outputs only INFO. You can't direct only DEBUG to one and only INFO to another handler. For that you'll need another logger.
logger = logging.getLogger("mylog")
formatter = logging.Formatter(
'%(asctime)s | %(name)s | %(levelname)s: %(message)s')
logger.setLevel(logging.DEBUG)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
stream_handler.setFormatter(formatter)
logFilePath = "my.log"
file_handler = logging.handlers.TimedRotatingFileHandler(
filename=logFilePath, when='midnight', backupCount=30)
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
logger.addHandler(stream_handler)
logger.info("Started");
try:
x = 14
y = 0
z = x / y
except Exception as ex:
logger.error("Operation failed.")
logger.debug(
"Encountered {0} when trying to perform calculation.".format(ex))
logger.info("Ended");
I needed a time to understand the point
Set the general logger below your subloggers (handlers) (your result of logging.getLogger())
Set your subloggers levels on an equal or superior level to your general logger
In addition to GrantVS's answer:
I had to use
logging.basicConfig(level=logging.DEBUG)
in order for it to work. Otherwise great answer, thanks!
Had the same problem but the solution didn't work for iPython as the QtConsole automatically creates a handler with no level set:
import logging
root = logging.getLogger()
root.handlers
Out: [<StreamHandler <stderr> (NOTSET)>]
As a result iPython printed both DEBUG and INFO to console in spite of having different levels for my file handler and stream handler.
This thread pointed out this issue for me: Logging module does not print in IPython
I made a helper module (helped greatly by this stack thread!) called custom_logging.py to make logging more convenient in other modules:
import logging
from pathlib import Path
import sys
def _add_stream_handler(logger: logging.Logger):
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
stream_handler.setFormatter(formatter)
stream_handler.setLevel(logging.INFO)
logger.addHandler(stream_handler)
return logger
def _add_file_handler(logger: logging.Logger, log_path: Path):
file_handler = logging.FileHandler(log_path, mode='w')
formatter = logging.Formatter(
fmt='%(asctime)s %(name)-12s %(levelname)-8s %(message)s', datefmt='%m-%d %H:%M')
file_handler.setFormatter(formatter)
file_handler.setLevel(logging.DEBUG)
logger.addHandler(file_handler)
return logger
def create_logger(root_dir: Path, caller: str) -> logging.Logger:
log_path = root_dir / 'logs' / f'{caller}.log'
logger = logging.getLogger(caller)
root = logging.getLogger()
logger.setLevel(logging.DEBUG)
# If haven't already launched handlers...
if not len(logger.handlers):
_add_file_handler(logger=logger, log_path=log_path)
_add_stream_handler(logger=logger)
logger.info('Logging started.')
# Delete the Qtconsole stderr handler
# ... as it automatically logs both DEBUG & INFO to stderr
if root.handlers:
root.handlers = []
return logger
def log_dataframe(df, logger: logging.Logger, name: str = "DataFrame") -> None:
logger.debug(
f'''{name} head:\n {df.head()}\n----------\n''')
def log_dataframes(*args, logger: logging.Logger) -> None:
for gdf in args:
logger.debug(
f'''DataFrame head:\n {gdf.head()}\n----------\n''')
Can use its functions via:
from custom_logging import create_logger, log_dataframe
Or import custom_logging and custom_logging.create_logger() etc.
Also see sections 'Multiple handlers and formatters' and 'Logging to multiple destinations' in the official logging cookbook at:
https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook