I am using the logging module in Python to write debug and error messages.
I want to write to file all messages of logging.DEBUG or greater.
However, I only want to print to the screen messages of logging.WARNING or greater.
Is this possible using just one Logger and one FileHandler?
As it has been mentioned, handlers are so easy to create and add that you're probably better off just using two handlers. If, however, for some reason you want to stick to one, the Python logging cookbook has a section describing more or less what you want to do: logging to both console and file, but at different levels (it even shows you how to do different formatting). It does it with a single StreamHandler rather than a FileHandler, though:
import logging
# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename='/temp/myapp.log',
filemode='w')
# define a Handler which writes INFO messages or higher to the sys.stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')
# Now, define a couple of other loggers which might represent areas in your
# application:
logger1 = logging.getLogger('myapp.area1')
logger2 = logging.getLogger('myapp.area2')
logger1.debug('Quick zephyrs blow, vexing daft Jim.')
logger1.info('How quickly daft jumping zebras vex.')
logger2.warning('Jail zesty vixen who grabbed pay from quack.')
logger2.error('The five boxing wizards jump quickly.')
Edit: As discussed in the comments this code still generates two handlers, but "hides" one construction through the use of basicConfig(). I would strongly encourage you to create both explicitly.
You can append lots of handlers to the same logger with different loglevel, but root loglevel must be as verbose as the most verbose handler. This example log messages to file with log.debug() and log.info() to console:
log = logging.getLogger("syslog2elastic")
log.setLevel(logging.DEBUG) # this must be DEBUG to allow debug messages through
console = logging.StreamHandler()
console.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s.%(msecs)03d - %(name)s:%(lineno)d - %(levelname)s - %(message)s", "%Y%m%d%H%M%S")
console.setFormatter(formatter)
log.addHandler(console)
fh = RotatingFileHandler(args.logfile, maxBytes=104857600, backupCount=5)
fh.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s.%(msecs)03d - %(message)s", "%Y%m%d%H%M%S")
fh.setFormatter(formatter)
log.addHandler(fh)
No. File and Screen output means you need two handlers.
You can do this in the following way, making two loggers, one for file logging, another one for console logging. Make sure to set the root logger to the most verbose of the two.
import logging
logging.getLogger().setLevel(logging.DEBUG) # This must be as verbose as the most verbose handler
formatter = logging.Formatter(
'%(asctime)s.%(msecs)03d %(levelname)s %(module)s - %(funcName)s [%(lineno)s]: %(message)s',
datefmt='%Y-%m-%d %H:%M:%S',
)
console_logging = logging.StreamHandler()
console_logging.setLevel(logging.WARNING)
console_logging.setFormatter(formatter)
logging.getLogger().addHandler(console_logging)
file_logging = logging.FileHandler('file.log')
file_logging.setLevel(logging.DEBUG)
file_logging.setFormatter(formatter)
logging.getLogger().addHandler(file_logging)
Related
I've written an init function to configure the python logging module to log to a file and the console. I want to limit the log file size using the RotatingFileHandler. The code below doesn't cause any errors and does everything I want, except it doesn't rotate the logs. I set a low file size to test things out.
How can I configure to use rotating logs and the console with different formats like below?
import logging, logging.handlers
LOG_LEVEL = logging.DEBUG
CONSOLE_LEVEL = logging.DEBUG
def init_logger(fullpath, console_level=CONSOLE_LEVEL, log_level=LOG_LEVEL):
"""
Setup the logger object
Args:
fullpath (str): full path to the log file
"""
logging.basicConfig(level=LOG_LEVEL,
format='%(asctime)s %(threadName)-10s %(name)-12s %
(levelname)-8s %(message)s',
datefmt='%m-%d-%y %H:%M:%S',
filename=fullpath,
filemode='w')
_logger = logging.getLogger('_root')
_logger.setLevel(log_level)
log_handler = logging.handlers.RotatingFileHandler(filename=fullpath,
maxBytes=50, backupCount=10)
log_handler.setLevel(log_level)
_logger.addHandler(log_handler)
console = logging.StreamHandler()
console.setLevel(console_level)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %
(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
logging.debug("Creating log file")
This is your logging's structure:
---root <- I mean real root, which you will get from logging.getLogger([""])
---NormalFileHandler <- Configged by logging.basicConfig, this handler won't rotate file.
---StreamHandler
------_root <- Your root logger, although I don't understand why you want this.
------RotatingFileHandler <- This one will rotate file.
And then you use logging.debug("Creating log file"), which is the same as calling debug on root logger: logging.getLogger().debug(...). So this log passed to StreamHandler and NormalFileHandler.
That's why you see the file isn't got rotated.
The correct configuration should be:
def init_logger(fullpath, console_level=CONSOLE_LEVEL, log_level=LOG_LEVEL):
"""
Setup the logger object
Args:
fullpath (str): full path to the log file
"""
logger = logging.getLogger('YourLogger')
logger.setLevel(log_level)
log_handler = logging.handlers.RotatingFileHandler(filename=fullpath,
maxBytes=50, backupCount=10)
log_handler.setLevel(log_level)
formatter = logging.Formatter('%(asctime)s %(threadName)-10s %(name)-12s %
(levelname)-8s %(message)s')
log_handler.setFormatter(formatter)
logger.addHandler(log_handler)
console = logging.StreamHandler()
console.setLevel(console_level)
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %
(message)s')
console.setFormatter(formatter)
logger.addHandler(console) # Or you can add it to root logger, but it is not recommended, you should use your own logger instead of root logger. Or it will cause some problems.
logger.debug("Creating log file")
And then when you want to use logger, you should use:
logger = logging.getLogger('YourLogger')
logger.info(...)
logger.debug(...)
I'm simply trying to have a python logger with a specific format that outputs log messages only to the console. I've tried many different things but I keep getting 2 lines of console output per log call.
Here is my code:
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
# Create console handler
stream_handler = logging.StreamHandler()
formatter = logging.Formatter('%(levelname)s - %(asctime)s - %(name)s - %(message)s')
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
logger.info('TEST LOG info')
With output:
INFO - 2017-08-21 14:30:00,751 - my_logger - TEST LOG info
INFO:my_logger:TEST LOG info
I did exactly this and it didn't work: Disable output of root logger
Any idea what is going on? I don't care whether I use the root logger or not, I just want one line
The above code actually should be working correctly. Although my script was very lean, it was importing a non-system library which, somewhere down the line, had some logging configured which was affecting my output.
I have two log handlers in my code: a StreamHandler to write INFO level logs from the same module to stdout, and a FileHandler to write more verbose, DEBUG logs to a file. This is my code:
import sys
import logging
log = logging.getLogger('mymodule')
log.setLevel(logging.DEBUG)
logf = logging.FileHandler('file.log')
logf.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
log.addHandler(logf)
logs = logging.StreamHandler(sys.stdout)
logs.setLevel(logging.INFO)
logs.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
log.addHandler(logs)
However, I want the FileHandler to also write DEBUG info from other modules. I can achieve this if I remove the name from logging.getLogger(), but this will also affect my StreamHandler, which I only want to print output from my own module.
So is there a way to have either of the handlers use a different name?
There's a name attribute available on the base Handler class. I took a look at the Python source and doesn't look like it's used internally so you could manually set each handler to a different name.
I've spent a bit of time looking through the site at Python logger questions hoping my would be resolved there. I've set up a logger with two stream handlers that have both different formats and levels of logging, here's a functional snippet from my codebase:
import os
import time
import logging
LOG_LEVELS = [logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG]
TEST_RESULT_LEVELV_NUM = 51
# http://stackoverflow.com/a/11784984/196832
def status(self, message, *args, **kws):
self._log(TEST_RESULT_LEVELV_NUM, message, args, **kws)
logging.addLevelName(TEST_RESULT_LEVELV_NUM, "RESULT")
logging.Logger.result = status
def setup_logging(level=0, quiet=False, logdir=None):
logger = logging.getLogger('juju-test')
ffmt = logging.Formatter('%(asctime)s %(name)s %(levelname)-8s: %(message)s')
cfmt = logging.Formatter('%(name)s %(levelname)s: %(message)s')
#logger.setLevel(0)
if level >= len(LOG_LEVELS):
level = len(LOG_LEVELS) - 1
if logdir:
if not os.path.exists(logdir):
os.makedirs(logdir)
logfile = os.path.join(logdir, 'juju-test.%s.log' % int(time.time()))
fh = logging.FileHandler(logfile)
# Always at least log to INFO for file, unless DEBUG is requested
fh.setLevel(LOG_LEVELS[level if level >= 2 else 2])
fh.setFormatter(ffmt)
logger.addHandler(fh)
if not quiet:
ch = logging.StreamHandler()
ch.setLevel(LOG_LEVELS[level])
ch.setFormatter(cfmt)
logger.addHandler(ch)
return logger
I've been using an argparse to feed this, but for testing purposes if you feed the following to function:
logger = setup_logging(level=1, logdir="/tmp/oofrab/")
logger.info('Informative!')
logger.warn('Whoa buddy!')
logger.error('Look what you did.')
logger.result("They told me not to make a custom logging level, I'll show them!")
logger.debug('Lots of bugs, man')
I'd expect to see status, error, and warn in the console. Then status, error, warn and info in the log. However, I only see down to warn in both console and log file, despite selecting logging.INFO (key 2 in the LOG_LEVELS list) for the file handler. Is this expected?
I'm not using basicConfig or anything else when building the logger, why can't I have these two custom levels?
Apparently, logging.NOTSET does not mean "All levels", but rather defaults. So setting the parent logger to level 0 does only reverts it to it's default accepted levels. That being said, if I set logger.setLevel to logging.DEBUG that essentially sets logging to accept ALL levels then passes filtering on to the various handlers to filter further.
To get around this (and potential custom log levels) I've set the initial logger level to 1
I don't know why this code prints to the screen, but not to the file? File "example1.log" is created, but nothing is written there.
#!/usr/bin/env python3
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(message)s',
handlers=[logging.FileHandler("example1.log"),
logging.StreamHandler()])
logging.debug('This message should go to the log file and to the console')
logging.info('So should this')
logging.warning('And this, too')
I have "bypassed" this problem by creating a logging object, but it keeps bugging me why basicConfig() approach failed?
PS. If I change basicConfig call to:
logging.basicConfig(level=logging.DEBUG,
filename="example2.log",
format='%(asctime)s %(message)s',
handlers=[logging.StreamHandler()])
then all logs are in the file and nothing is displayed in the console.
Try this working fine(tested in python 2.7) for both console and file
# set up logging to file
logging.basicConfig(
filename='log_file_name.log',
level=logging.INFO,
format= '[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s',
datefmt='%H:%M:%S'
)
# set up logging to console
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)
logger = logging.getLogger(__name__)
I can't reproduce it on Python 3.3. The messages are written both to the screen and the 'example2.log'. On Python <3.3 it creates the file but it is empty.
The code:
from logging_tree import printout # pip install logging_tree
printout()
shows that FileHandler() is not attached to the root logger on Python <3.3.
The docs for logging.basicConfig() say that handlers argument is added in Python 3.3. The handlers argument isn't mentioned in Python 3.2 documentation.
Another technique using the basicConfig is to setup all your handlers in the statement and retrieve them after the fact, as in...
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(module)s %(funcName)s %(message)s',
handlers=[logging.FileHandler("my_log.log", mode='w'),
logging.StreamHandler()])
stream_handler = [h for h in logging.root.handlers if isinstance(h , logging.StreamHandler)][0]
stream_handler.setLevel(logging.INFO)
More sensibly though is to construct your stream handler instance outside and configure them as standalone objects that you pass to the handlers list as in...
import logging
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(module)s %(funcName)s %(message)s',
handlers=[logging.FileHandler("my_log.log", mode='w'),
stream_handler])
In the example below, you can specify the log destination based on its level. For example, the code below lets all logs over the INFO level go to the log file, and all above ERROR level goes to the console.
import logging
logging.root.handlers = []
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO , filename='ex.log')
# set up logging to console
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
# set a format which is simpler for console use
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(message)s')
console.setFormatter(formatter)
logging.getLogger("").addHandler(console)
logging.debug('debug')
logging.info('info')
logging.warning('warning')
logging.error('error')
logging.exception('exp')
WOOAH!
I just spent about 20 minutes being baffled by this.
Eventually I worked out that the StreamHandler was outputting to stderr, not stdout (in a 'Doze DOS screen these have the same font colour!).
This resulted in me being able to run the code perfectly OK and get sensible results, but in a pytest function things going awry. Until I changed from this:
out, _ = capsys.readouterr()
assert 'test message check on console' in out, f'out was |{out}|'
to this:
_, err = capsys.readouterr()
assert 'test message check on console' in err, f'err was |{err}|'
NB the constructor of StreamHandler is
class logging.StreamHandler(stream=None)
and, as the docs say, "If stream is specified, the instance will use it for logging output; otherwise, sys.stderr will be used."
NB it seems that supplying the level keyword does not run setLevel on the handlers: you'd need to iterate on the resulting handlers and run setLevel on each, if it matters to you.
This is a ValueError if FileHandler and StreamHandler both are present in BasicConfig function
https://docs.python.org/3/library/logging.html#logging.basicConfig
See image below: