I'm using Python's logging module to log some debug strings to a file which works pretty well. Now in addition, I'd like to use this module to also print the strings out to stdout. How do I do this? In order to log my strings to a file I use following code:
import logging
import logging.handlers
logger = logging.getLogger("")
logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(
LOGFILE, maxBytes=(1048576*5), backupCount=7
)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
and then call a logger function like
logger.debug("I am written to the file")
Thank you for some help here!
Just get a handle to the root logger and add the StreamHandler. The StreamHandler writes to stderr. Not sure if you really need stdout over stderr, but this is what I use when I setup the Python logger and I also add the FileHandler as well. Then all my logs go to both places (which is what it sounds like you want).
import logging
logging.getLogger().addHandler(logging.StreamHandler())
If you want to output to stdout instead of stderr, you just need to specify it to the StreamHandler constructor.
import sys
# ...
logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))
You could also add a Formatter to it so all your log lines have a common header.
ie:
import logging
logFormatter = logging.Formatter("%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s] %(message)s")
rootLogger = logging.getLogger()
fileHandler = logging.FileHandler("{0}/{1}.log".format(logPath, fileName))
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
Prints to the format of:
2012-12-05 16:58:26,618 [MainThread ] [INFO ] my message
logging.basicConfig() can take a keyword argument handlers since Python 3.3, which simplifies logging setup a lot, especially when setting up multiple handlers with the same formatter:
handlers – If specified, this should be an iterable of already created handlers to add to the root logger. Any handlers which don’t already have a formatter set will be assigned the default formatter created in this function.
The whole setup can therefore be done with a single call like this:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[
logging.FileHandler("debug.log"),
logging.StreamHandler()
]
)
(Or with import sys + StreamHandler(sys.stdout) per original question's requirements – the default for StreamHandler is to write to stderr. Look at LogRecord attributes if you want to customize the log format and add things like filename/line, thread info etc.)
The setup above needs to be done only once near the beginning of the script. You can use the logging from all other places in the codebase later like this:
logging.info('Useful message')
logging.error('Something bad happened')
...
Note: If it doesn't work, someone else has probably already initialized the logging system differently. Comments suggest doing logging.root.handlers = [] before the call to basicConfig().
Adding a StreamHandler without arguments goes to stderr instead of stdout. If some other process has a dependency on the stdout dump (i.e. when writing an NRPE plugin), then make sure to specify stdout explicitly or you might run into some unexpected troubles.
Here's a quick example reusing the assumed values and LOGFILE from the question:
import logging
from logging.handlers import RotatingFileHandler
from logging import handlers
import sys
log = logging.getLogger('')
log.setLevel(logging.DEBUG)
format = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
ch = logging.StreamHandler(sys.stdout)
ch.setFormatter(format)
log.addHandler(ch)
fh = handlers.RotatingFileHandler(LOGFILE, maxBytes=(1048576*5), backupCount=7)
fh.setFormatter(format)
log.addHandler(fh)
Here is a complete, nicely wrapped solution based on Waterboy's answer and various other sources. It supports logging to both console and log file, allows for different log level settings, provides colorized output and is easily configurable (also available as Gist):
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# -------------------------------------------------------------------------------
# -
# Python dual-logging setup (console and log file), -
# supporting different log levels and colorized output -
# -
# Created by Fonic <https://github.com/fonic> -
# Date: 04/05/20 - 02/07/23 -
# -
# Based on: -
# https://stackoverflow.com/a/13733863/1976617 -
# https://uran198.github.io/en/python/2016/07/12/colorful-python-logging.html -
# https://en.wikipedia.org/wiki/ANSI_escape_code#Colors -
# -
# -------------------------------------------------------------------------------
# Imports
import os
import sys
import logging
# Logging formatter supporting colorized output
class LogFormatter(logging.Formatter):
COLOR_CODES = {
logging.CRITICAL: "\033[1;35m", # bright/bold magenta
logging.ERROR: "\033[1;31m", # bright/bold red
logging.WARNING: "\033[1;33m", # bright/bold yellow
logging.INFO: "\033[0;37m", # white / light gray
logging.DEBUG: "\033[1;30m" # bright/bold black / dark gray
}
RESET_CODE = "\033[0m"
def __init__(self, color, *args, **kwargs):
super(LogFormatter, self).__init__(*args, **kwargs)
self.color = color
def format(self, record, *args, **kwargs):
if (self.color == True and record.levelno in self.COLOR_CODES):
record.color_on = self.COLOR_CODES[record.levelno]
record.color_off = self.RESET_CODE
else:
record.color_on = ""
record.color_off = ""
return super(LogFormatter, self).format(record, *args, **kwargs)
# Set up logging
def set_up_logging(console_log_output, console_log_level, console_log_color, logfile_file, logfile_log_level, logfile_log_color, log_line_template):
# Create logger
# For simplicity, we use the root logger, i.e. call 'logging.getLogger()'
# without name argument. This way we can simply use module methods for
# for logging throughout the script. An alternative would be exporting
# the logger, i.e. 'global logger; logger = logging.getLogger("<name>")'
logger = logging.getLogger()
# Set global log level to 'debug' (required for handler levels to work)
logger.setLevel(logging.DEBUG)
# Create console handler
console_log_output = console_log_output.lower()
if (console_log_output == "stdout"):
console_log_output = sys.stdout
elif (console_log_output == "stderr"):
console_log_output = sys.stderr
else:
print("Failed to set console output: invalid output: '%s'" % console_log_output)
return False
console_handler = logging.StreamHandler(console_log_output)
# Set console log level
try:
console_handler.setLevel(console_log_level.upper()) # only accepts uppercase level names
except:
print("Failed to set console log level: invalid level: '%s'" % console_log_level)
return False
# Create and set formatter, add console handler to logger
console_formatter = LogFormatter(fmt=log_line_template, color=console_log_color)
console_handler.setFormatter(console_formatter)
logger.addHandler(console_handler)
# Create log file handler
try:
logfile_handler = logging.FileHandler(logfile_file)
except Exception as exception:
print("Failed to set up log file: %s" % str(exception))
return False
# Set log file log level
try:
logfile_handler.setLevel(logfile_log_level.upper()) # only accepts uppercase level names
except:
print("Failed to set log file log level: invalid level: '%s'" % logfile_log_level)
return False
# Create and set formatter, add log file handler to logger
logfile_formatter = LogFormatter(fmt=log_line_template, color=logfile_log_color)
logfile_handler.setFormatter(logfile_formatter)
logger.addHandler(logfile_handler)
# Success
return True
# Main function
def main():
# Set up logging
script_name = os.path.splitext(os.path.basename(sys.argv[0]))[0]
if (not set_up_logging(console_log_output="stdout", console_log_level="warning", console_log_color=True,
logfile_file=script_name + ".log", logfile_log_level="debug", logfile_log_color=False,
log_line_template="%(color_on)s[%(created)d] [%(threadName)s] [%(levelname)-8s] %(message)s%(color_off)s")):
print("Failed to set up logging, aborting.")
return 1
# Log some messages
logging.debug("Debug message")
logging.info("Info message")
logging.warning("Warning message")
logging.error("Error message")
logging.critical("Critical message")
# Call main function
if (__name__ == "__main__"):
sys.exit(main())
NOTE regarding Microsoft Windows:
For colorized output to work within the classic Command Prompt of Microsoft Windows, some additional code is necessary. This does NOT apply to the newer Terminal app which supports colorized output out of the box.
There are two options:
1) Use Python package colorama (filters output sent to stdout and stderr and translates escape sequences to native Windows API calls; works on Windows XP and later):
import colorama
colorama.init()
2) Enable ANSI terminal mode using the following function (enables terminal to interpret escape sequences by setting flag ENABLE_VIRTUAL_TERMINAL_PROCESSING; more info on this here, here, here and here; works on Windows 10 and later):
# Imports
import sys
import ctypes
# Enable ANSI terminal mode for Command Prompt on Microsoft Windows
def windows_enable_ansi_terminal_mode():
if (sys.platform != "win32"):
return None
try:
kernel32 = ctypes.windll.kernel32
result = kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7)
if (result == 0): raise Exception
return True
except:
return False
Either run basicConfig with stream=sys.stdout as the argument prior to setting up any other handlers or logging any messages, or manually add a StreamHandler that pushes messages to stdout to the root logger (or any other logger you want, for that matter).
Logging to stdout and rotating file with different levels and formats:
import logging
import logging.handlers
import sys
if __name__ == "__main__":
# Change root logger level from WARNING (default) to NOTSET in order for all messages to be delegated.
logging.getLogger().setLevel(logging.NOTSET)
# Add stdout handler, with level INFO
console = logging.StreamHandler(sys.stdout)
console.setLevel(logging.INFO)
formater = logging.Formatter('%(name)-13s: %(levelname)-8s %(message)s')
console.setFormatter(formater)
logging.getLogger().addHandler(console)
# Add file rotating handler, with level DEBUG
rotatingHandler = logging.handlers.RotatingFileHandler(filename='rotating.log', maxBytes=1000, backupCount=5)
rotatingHandler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
rotatingHandler.setFormatter(formatter)
logging.getLogger().addHandler(rotatingHandler)
log = logging.getLogger("app." + __name__)
log.debug('Debug message, should only appear in the file.')
log.info('Info message, should appear in file and stdout.')
log.warning('Warning message, should appear in file and stdout.')
log.error('Error message, should appear in file and stdout.')
After having used Waterboy's code over and over in multiple Python packages, I finally cast it into a tiny standalone Python package, which you can find here:
https://github.com/acschaefer/duallog
The code is well documented and easy to use. Simply download the .py file and include it in your project, or install the whole package via pip install duallog.
I have handled redirecting logs and prints to a file on disk, stdout, and stderr simultaneously, via the following module (the Gist is also available here):
import logging
import pathlib
import sys
from ml.common.const import LOG_DIR_PATH, ML_DIR
def create_log_file_path(file_path, root_dir=ML_DIR, log_dir=LOG_DIR_PATH):
path_parts = list(pathlib.Path(file_path).parts)
relative_path_parts = path_parts[path_parts.index(root_dir) + 1:]
log_file_path = pathlib.Path(log_dir, *relative_path_parts)
log_file_path = log_file_path.with_suffix('.log')
# Create the directories and the file itself
log_file_path.parent.mkdir(parents=True, exist_ok=True)
log_file_path.touch(exist_ok=True)
return log_file_path
def set_up_logs(file_path, mode='a', level=logging.INFO):
log_file_path = create_log_file_path(file_path)
logging_handlers = [logging.FileHandler(log_file_path, mode=mode),
logging.StreamHandler(sys.stdout)]
logging.basicConfig(
handlers=logging_handlers,
format='%(asctime)s %(name)s %(levelname)s %(message)s',
level=level
)
class OpenedFileHandler(logging.FileHandler):
def __init__(self, file_handle, filename, mode):
self.file_handle = file_handle
super(OpenedFileHandler, self).__init__(filename, mode)
def _open(self):
return self.file_handle
class StandardError:
def __init__(self, buffer_stderr, buffer_file):
self.buffer_stderr = buffer_stderr
self.buffer_file = buffer_file
def write(self, message):
self.buffer_stderr.write(message)
self.buffer_file.write(message)
class StandardOutput:
def __init__(self, buffer_stdout, buffer_file):
self.buffer_stdout = buffer_stdout
self.buffer_file = buffer_file
def write(self, message):
self.buffer_stdout.write(message)
self.buffer_file.write(message)
class Logger:
def __init__(self, file_path, mode='a', level=logging.INFO):
self.stdout_ = sys.stdout
self.stderr_ = sys.stderr
log_file_path = create_log_file_path(file_path)
self.file_ = open(log_file_path, mode=mode)
logging_handlers = [OpenedFileHandler(self.file_, log_file_path,
mode=mode),
logging.StreamHandler(sys.stdout)]
logging.basicConfig(
handlers=logging_handlers,
format='%(asctime)s %(name)s %(levelname)s %(message)s',
level=level
)
# Overrides write() method of stdout and stderr buffers
def write(self, message):
self.stdout_.write(message)
self.stderr_.write(message)
self.file_.write(message)
def flush(self):
pass
def __enter__(self):
sys.stdout = StandardOutput(self.stdout_, self.file_)
sys.stderr = StandardError(self.stderr_, self.file_)
def __exit__(self, exc_type, exc_val, exc_tb):
sys.stdout = self.stdout_
sys.stderr = self.stderr_
self.file_.close()
Written as a context manager, you can simply add the functionality to your python script by adding an extra line:
from logger import Logger
...
if __name__ == '__main__':
with Logger(__file__):
main()
Although the question specifically asks for a logger configuration, there is an alternative approach that does not require any changes to the logging configuration, nor does it require redirecting stdout.
A bit simplistic, perhaps, but it works:
def log_and_print(message: str, level: int, logger: logging.Logger):
logger.log(level=level, msg=message) # log as normal
print(message) # prints to stdout by default
Instead of e.g. logger.debug('something') we now call log_and_print(message='something', level=logging.DEBUG, logger=logger).
We can also expand this a little, so it prints to stdout only when necessary:
def log_print(message: str, level: int, logger: logging.Logger):
# log the message normally
logger.log(level=level, msg=message)
# only print to stdout if the message is not logged to stdout
msg_logged_to_stdout = False
current_logger = logger
while current_logger and not msg_logged_to_stdout:
is_enabled = current_logger.isEnabledFor(level)
logs_to_stdout = any(
getattr(handler, 'stream', None) == sys.stdout
for handler in current_logger.handlers
)
msg_logged_to_stdout = is_enabled and logs_to_stdout
if not current_logger.propagate:
current_logger = None
else:
current_logger = current_logger.parent
if not msg_logged_to_stdout:
print(message)
This checks the logger and its parents for any handlers that stream to stdout, and checks if the logger is enabled for the specified level.
Note this has not been optimized for performance.
For 2.7, try the following:
fh = logging.handlers.RotatingFileHandler(LOGFILE, maxBytes=(1048576*5), backupCount=7)
Related
I try using this openshift-restclient-python library. My custom Logger prints everything twice after I run into this bug.
modules/logging/Logging.py
import logging
class CustomLogger:
logger=None
def __init__(self):
if (CustomLogger.logger==None):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
logger.addHandler(stdout_handler)
CustomLogger.logger=logger
def getLogger(self):
return CustomLogger.logger
logger = CustomLogger().getLogger()
This ist my main.py:
#!/usr/bin/env python3
import sys
from modules.logging.Logging import logger
from kubernetes import client, config
from openshift.dynamic import DynamicClient
from openshift.helper.userpassauth import OCPLoginConfiguration
import warnings
warnings.filterwarnings("ignore")
apihost = 'myhost'
username = 'myuser'
password = 'insecure'
ca_cert = '/path/to/cert'
kubeConfig = OCPLoginConfiguration(ocp_username=username, ocp_password=password)
kubeConfig.host = apihost
kubeConfig.verify_ssl = False
kubeConfig.ssl_ca_cert = ca_cert
kubeConfig.get_token()
k8s_client = client.ApiClient(kubeConfig)
logger.warning("this is printed once")
dyn_client = DynamicClient(k8s_client)
logger.warning("this is printed twice")
v1_projects = dyn_client.resources.get(api_version='project.openshift.io/v1', kind='Project')
project_list = v1_projects.get()
sys.exit(0)
executing the main.py I get the following output
this is printed once
ERROR:root:load cache error: ResourceList.__init__() got an unexpected keyword argument 'base_resource_lookup'
this is printed twice
WARNING:modules.logging.Logging:this is printed twice
If I do not use my custom logger but a simple configuration as below in main.py then everything is printed once.
import logging
logging.basicConfig(level=logging.DEBUG)
I have found this answer so I also tried removing any handler but the only handler is the one that contains my customization, so I end up with a basic logger.
What am I doing wrong?
Thanks
EDIT:
There is an easier way reproducing the issue.
I still have my custom logger as posted before but my main.py now:
#!/usr/bin/env python3
import sys
from modules.logging.Logging import logger
import logging
print(logger.handlers)
print("number handlers: " +str(len(logger.handlers)))
logger.warning("this is printed once")
logging.basicConfig(level=logging.DEBUG)
logger.warning("this is printed twice")
print("number handlers: " +str(len(logger.handlers)))
for h in logger.handlers:
logger.removeHandler(h)
print("number handlers: " +str(len(logger.handlers)))
logger.warning("still get printed")
sys.exit(0)
the output:
[<StreamHandler <stderr> (DEBUG)>]
number handlers: 1
this is printed once
this is printed twice
WARNING:modules.logging.Logging:this is printed twice
number handlers: 1
number handlers: 0
WARNING:modules.logging.Logging:still get printed
The code logging.basicConfig(level=logging.DEBUG) doesn't add another handler but cause everything to be logged. I actually only want the customized logs printed by the streamingHandler. How can I revert what is done by logging.basicConfig(level=logging.DEBUG)?
Please try remove this peace of code from class CustomLogger
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
logger.addHandler(stdout_handler)
btw i was struggling with the same. Found answers using search on this website.
https://stackoverflow.com/a/26730545/15637940
https://stackoverflow.com/a/70876179/15637940
and a lot more answered questions...
I solved it that way:
class CustomLogger:
logger=None
def __init__(self):
if (CustomLogger.logger==None):
logging.basicConfig(filename='/dev/null', filemode='w', format='%(name)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
fmt = '%(asctime)s | %(message)s'
stdout_handler = logging.StreamHandler()
stdout_handler.setLevel(logging.DEBUG)
stdout_handler.setFormatter(CustomFormatter(fmt))
logger.addHandler(stdout_handler)
CustomLogger.logger=logger
def getLogger(self):
return CustomLogger.logger
logger = CustomLogger().getLogger()
It seems that the library I am using at some place logs to the RootLogger. According to this answer logging.basicConfig() is a constructor of a streamHandler that connects to the RootLogger.
If I use logger = logging.getLogger('root') instead of logger = logging.getLogger(__name__) then everything is printed once. However, in that case everything the library logs on DEBUG-Level is printed to the terminal.
The line logging.basicConfig(filename='/dev/null', filemode='w', format='%(name)s - %(levelname)s - %(message)s') causes that everything logged by the root logger is printed to /dev/null.
I am trying to use an FTP server stub during tests. I don't want the console output, but I would like to capture the logging to a file.
I want the FTP server to run in a different process, so I use multiprocessing.
My code as follows sets all logging to level WARNING:
from pyftpdlib.authorizers import DummyAuthorizer
from pyftpdlib.handlers import FTPHandler
from pyftpdlib.servers import FTPServer
import pyftpdlib.log as pyftpdliblog
import os
import logging
import multiprocessing as mp
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', '.', perm='elradfmwM')
handler = FTPHandler
handler.authorizer = authorizer
pyftpdliblog.LEVEL = logging.WARNING
logging.basicConfig(filename='pyftpd.log', level=logging.INFO)
server = FTPServer(('', 2121), handler)
def main():
p = mp.Process(target=server.serve_forever)
p.start()
if __name__ == '__main__':
main()
How do I set only the console logging to level WARNING, or even better, completely shutdown without giving up the file logging?
So, after digging inside the code, I found out the following hint:
# This is a method of FTPServer and it is called before
# server.serve_forever
def _log_start(self):
if not logging.getLogger('pyftpdlib').handlers:
# If we get to this point it means the user hasn't
# configured logger. We want to log by default so
# we configure logging ourselves so that it will
# print to stderr.
from pyftpdlib.ioloop import _config_logging
_config_logging()
So, all I had to do is to define my own appropriate handlers:
logger = logging.getLogger('pyftpdlib')
logger.setLevel(logging.INFO)
hdlr = logging.FileHandler('pyftpd.log' )
logger.addHandler(hdlr)
Now, there is file logging, but console logging will not start.
Something like this:
import logging
date_format = "%Y/%m/%d %H:%M:%S"
log_file_path = "my_file.txt"
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
# own_module_logger = logging.getLogger(__name__)
pyftpdlib_logger = logging.getLogger("pyftpdlib")
# Setup logging to file (Only pyftpdlib)
filehandler = logging.FileHandler(filename = log_file_path)
filehandler.setLevel(logging.DEBUG)
fileformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
filehandler.setFormatter(fileformatter)
pyftpdlib_logger.addHandler(filehandler)
pyftpdlib_logger.propagate = False
# Setup logging to console (All other)
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter(fmt = "%(asctime)s - %(levelname)-8s - %(name)s.%(funcName)s - %(message)s",
datefmt = date_format)
console.setFormatter(consoleformatter)
root_logger.addHandler(console)
# Do your loggings
a = logging.getLogger()
a.info('root I')
a.debug('root D')
b = logging.getLogger("pyftpdlib")
b.info('P I')
b.debug('P D')
logging.shutdown()
So loggings of pyftpdlib go to file. Everything from your module to console. One of the key thing here is the propagate!
I have a logging configuration where I log to a file and to the console:
logging.basicConfig(filename=logfile, filemode='w',
level=numlevel,
format='%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
# add console messages
console = logging.StreamHandler()
console.setLevel(logging.INFO)
consoleformatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console.setFormatter(consoleformatter)
logging.getLogger('').addHandler(console)
At some point in my script, i need to interact with the user by printing a summary and asking for confirmation. The summary is currently produced by prints in a loop. I would like to suspend the current format of the console logs, so that I can printout one big block of text with a question at the end and wait for user input. But i still want all of this to be logged to file!
The function that does this is in a module, where I tried the following :
logger = logging.getLogger(__name__)
def summaryfunc:
logger.info('normal logging business')
clearformatter = logging.Formatter('%(message)s')
logger.setFormatter(clearformatter)
logger.info('\n##########################################')
logger.info('Summary starts here')
Which yields the error: AttributeError: 'Logger' object has no attribute 'setFormatter'
I understand that a logger is a logger, not a handler, but i'm not sure on how to get things to work...
EDIT:
Following the answers, my problem turned into : how can i suspend logging to the console when interacting with the user, while still being able to log to file. IE: suspend only the streamHandler. Since this is happening in a module, the specifics of the handlers are defined elsewhere, so here is how i did it :
logger.debug('Normal logging to file and console')
root_logger = logging.getLogger()
stream_handler = root_logger.handlers[1]
root_logger.removeHandler(stream_handler)
print('User interaction')
logger.info('Logging to file only')
root_logger.addHandler(stream_handler)
logger.info('Back to logging to both file and console')
This relies on the streamHandler always being the second in the list returned by handlers but i believe this is the case because it's in the order I added the handlers to the root logger...
I agree with Vinay that you should use print for normal program output and only use logging for logging purpose. However, if you still want to switch formatting in the middle, then switch back, here is how to do it:
import logging
def summarize():
console_handler.setFormatter(logging.Formatter('%(message)s'))
logger.info('Here is my report')
console_handler.setFormatter(console_formatter)
numlevel = logging.DEBUG
logfile = 's2.log'
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
console_handler = logging.StreamHandler()
console_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
console_handler.setFormatter(console_formatter)
logger.addHandler(console_handler)
file_handler = logging.FileHandler(filename=logfile, mode='w')
file_formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s:%(funcName)s - %(message)s')
file_handler.setFormatter(file_formatter)
logger.addHandler(file_handler)
logger.info('Before summary')
summarize()
logger.info('After summary')
Discussion
The script creates a logger object and assigned to it two handlers: one for the console and one for file.
In the function summarize(), I switched in a new formatter for the console handler, do some logging, then switched back.
Again, let me remind that you should not use logging to display normal program output.
Update
If you want to suppress console logging, then turn it back on. Here is a suggestion:
def interact():
# Remove the console handler
for handler in logger.handlers:
if not isinstance(handler, logging.FileHandler):
saved_handler = handler
logger.removeHandler(handler)
break
# Interact
logger.info('to file only')
# Add the console handler back
logger.addHandler(saved_handler)
Note that I did not test the handlers against logging.StreamHandler since a logging.FileHandler is derived from logging.StreamHandler. Therefore, I removed those handlers that are not FileHandler. Before removing, I saved that handler for later restoration.
Update 2: .handlers = []
In the main script, if you have:
logger = logging.getLogger(__name__) # __name__ == '__main__'
Then in a module, you do:
logger = logging.getLogger(__name__) # __name__ == module's name, not '__main__'
The problem is, in the script, __name__ == '__main__' and in the module, __name__ == <the module's name> and not '__main__'. In order to achieve consistency, you will need to make up some name and use them in both places:
logger = logging.getLogger('MyScript')
Logging shouldn't be used to provide the actual output of your program - a program is supposed to run the same way if logging is turned off completely. So I would suggest that it's better to do what you were doing before, i.e. prints in a loop.
I have a simple Python script that uses the in-built logging.
I'm configuring logging inside a function. Basic structure would be something like this:
#!/usr/bin/env python
import logging
import ...
def configure_logging():
logger = logging.getLogger("my logger")
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
def count_parrots():
...
logger.debug??
if __name__ == '__main__':
logger = configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()
I can call logger fine from inside __main__. However, how do I call logger from inside the count_parrots() function? What's the most pythonic way of handling configuring a logger like this?
You can either use the root (default) logger, and thus the module level functions logging.debug, ... or get your logger in the function using it.
Indeed, the getLogger function is a factory-like function with a registry (singleton like), i.e. it always returns the same instance for the given logger name.
You can thus get your logger in count_parrots by simply using
logger = logging.getLogger("my logger")
at the beginning. However, the convention is to use a dotted hierarchical name for your logger. See http://docs.python.org/library/logging.html#logging.getLogger
EDIT:
You can use a decorator to add the logging behaviour to your individual functions, for example:
def debug(loggername):
logger = logging.getLogger(loggername)
def log_(enter_message, exit_message=None):
def wrapper(f):
def wrapped(*args, **kargs):
logger.debug(enter_message)
r = f(*args, **kargs)
if exit_message:
logger.debug(exit_message)
return r
return wrapped
return wrapper
return log_
my_debug = debug('my.logger')
#my_debug('enter foo', 'exit foo')
def foo(a, b):
return a+b
you can "hardcode" the logger name and remove the top-level closure and my_debug.
You can just do :
logger = logging.getLogger("my logger")
in your count_parrots() method. When you pass the name that was used earlier (i.e. "my logger") the logging module would return the same instance that was created corresponding to that name.
Update: From the logging tutorial
(emphais mine)
getLogger() returns a reference to a
logger instance with the specified
name if it is provided, or root if
not. The names are period-separated
hierarchical structures. Multiple
calls to getLogger() with the same
name will return a reference to the
same logger object.
The typical way to handle logging is to have a per-module logger stored in a global variable. Any functions and methods within that module then just reference that same logger instance.
This is discussed briefly in the intro to the advance logging tutorial in the documentation:
http://docs.python.org/howto/logging.html#advanced-logging-tutorial
You can pass logger instances around as parameters, but doing so is typically rare.
I got confused by how global variables work in Python. Within a function you only need to declare global logger if you were doing something like logger = logging.getLogger("my logger") and hoping to modify the global logger.
So to modify your example, you can create a global logger object at the start of the file. If your module can be imported by another one, you should add the NullHandler so that if the importer of the library doesn't want logging enabled, they don't have any issues with your lib (ref).
#!/usr/bin/env python
import logging
import ...
logger = logging.getLogger("my logger").addHandler(logging.NullHandler())
def configure_logging():
logger.setLevel(logging.DEBUG)
# Format for our loglines
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Setup console logging
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
# Setup file logging as well
fh = logging.FileHandler(LOG_FILENAME)
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
def count_parrots():
...
logger.debug('counting parrots')
...
return parrots
if __name__ == '__main__':
configure_logging()
logger.debug("I'm a log file")
parrots = count_parrots()
If you don't need the log messages on your console, you can use in a minimalist way.
Alternatively you can use tail -f myapp.log to see the messages on the console.
import logging
logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', \
filename='myapp.log', \
level=logging.INFO)
def do_something():
logging.info('Doing something')
def main():
logging.info('Started')
do_something()
logging.info('Finished')
if __name__ == '__main__':
main()
You can give logger as argument to count_parrots() Or, what I would do, create class parrots and use logger as one of its method.
Is there an easy way with python's logging module to send messages with a DEBUG or INFO level and the one with a higher level to different streams?
Is it a good idea anyway?
import logging
import sys
class LessThanFilter(logging.Filter):
def __init__(self, exclusive_maximum, name=""):
super(LessThanFilter, self).__init__(name)
self.max_level = exclusive_maximum
def filter(self, record):
#non-zero return means we log this message
return 1 if record.levelno < self.max_level else 0
#Get the root logger
logger = logging.getLogger()
#Have to set the root logger level, it defaults to logging.WARNING
logger.setLevel(logging.NOTSET)
logging_handler_out = logging.StreamHandler(sys.stdout)
logging_handler_out.setLevel(logging.DEBUG)
logging_handler_out.addFilter(LessThanFilter(logging.WARNING))
logger.addHandler(logging_handler_out)
logging_handler_err = logging.StreamHandler(sys.stderr)
logging_handler_err.setLevel(logging.WARNING)
logger.addHandler(logging_handler_err)
#demonstrate the logging levels
logger.debug('DEBUG')
logger.info('INFO')
logger.warning('WARNING')
logger.error('ERROR')
logger.critical('CRITICAL')
Implementation aside, I do think it is a good idea to use the logging facilities in python to output to the terminal, in particular because you can add another handler to additionally log to a file. If you set stdout to be INFO instead of DEBUG, you can even include additional DEBUG information that the user wouldn't standardly see in the log file.
Yes. You must define multiple handlers for your logging.
http://docs.python.org/library/logging.html#logging-to-multiple-destinations
http://docs.python.org/library/logging.handlers.html#module-logging.handlers
I had the same problem and wrote a custom logging handler called SplitStreamHandler:
import sys
import logging
class SplitStreamHandler(logging.Handler):
def __init__(self):
logging.Handler.__init__(self)
def emit(self, record):
# mostly copy-paste from logging.StreamHandler
try:
msg = self.format(record)
if record.levelno < logging.WARNING:
stream = sys.stdout
else:
stream = sys.stderr
fs = "%s\n"
try:
if (isinstance(msg, unicode) and
getattr(stream, 'encoding', None)):
ufs = fs.decode(stream.encoding)
try:
stream.write(ufs % msg)
except UnicodeEncodeError:
stream.write((ufs % msg).encode(stream.encoding))
else:
stream.write(fs % msg)
except UnicodeError:
stream.write(fs % msg.encode("UTF-8"))
stream.flush()
except (KeyboardInterrupt, SystemExit):
raise
except:
self.handleError(record)
Just for your convenience adding everything together with the formatter in one package:
# shared formatter, but you can use separate ones:
FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(threadName)s - %(message)s'
formatter = logging.Formatter(FORMAT)
# single app logger:
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
# 2 handlers for the same logger:
h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
# filter out everything that is above INFO level (WARN, ERROR, ...)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)
log.addHandler(h1)
h2 = logging.StreamHandler(sys.stderr)
# take only warnings and error logs
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
log.addHandler(h2)
# profit:
log.info(...)
log.debug(...)
My use case was to redirect stdout to a datafile while seeing errors on the screen during processing.
right from the updated docs, it cover this case pretty well now.
http://docs.python.org/howto/logging.html#logging-advanced-tutorial
import sys # Add this.
import logging
# create logger
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler( sys.__stdout__ ) # Add this
ch.setLevel(logging.DEBUG)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
i've mentioned on comments the two changes required from the example to make the output go to stdout. you may also use filters to redirect depending on the level.
more information to understand the changes is at
http://docs.python.org/library/logging.handlers.html#module-logging.handlers
Here's the compact solution I use (tested with Python 3.10):
import logging
import sys
root_logger = logging.getLogger()
# configure default StreamHandler to stderr
logging.basicConfig(
format="%(asctime)s | %(levelname)-8s | %(filename)s:%(funcName)s(%(lineno)d) | %(message)s",
level=logging.INFO, # overall log level; DEBUG or INFO make most sense
)
stderr_handler = root_logger.handlers[0] # get default handler
stderr_handler.setLevel(logging.WARNING) # only print WARNING+
# add another StreamHandler to stdout which only emits below WARNING
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.addFilter(lambda rec: rec.levelno < logging.WARNING)
stdout_handler.setFormatter(stderr_handler.formatter) # reuse the stderr formatter
root_logger.addHandler(stdout_handler)
You will have to associated a handler to each stream you want logged statements to be sent to.
Python has 5 logging levels - CRITICAL, ERROR, WARNING, INFO, and DEBUG.
Level
Numeric Value
Stream
CRITICAL
50
STDERROR
ERROR
40
STDERROR
WARNING
30
STDOUT
INFO
20
STDOUT
DEBUG
10
STDOUT
NOTSET
0
Let's assume you want that Critical and Error logs should go to STDERROR, while others should go to STDOUT - as shown in the table above.
Here are the steps you should follow.
Step1 Initialize the logging module
import logging, sys
log = logging.getLogger()
Step 2 Create two streams, one attached to STDERROR,
other to STDOUT
hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)
Step 3 Set log level for each handler. This tells handler to only log statements which have log level equal to or higher than the level we set. eg. if we set this to WARNING, the handler will only handle logging statements for WARNING, ERROR, and CRITICAL levels. Also make sure that the log level is set in handler only, and not with the log object, so that there is no conflict
hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')
Now here comes a catch. While hStErr will only output logging for ERROR and CRITICAL, hStOut will output for all 5 levels. Remember that setLevel only tells the minimum logging level which should be handled, so all levels which are greater will also be handled. To limit hStOut to not handle ERROR and CRITICAL, we use a filter.
Step 4 Specify a filter so that ERROR, and CRITICAL aren't handled by hStOut
hStOut.addFilter(lambda x : x.levelno < logging.ERROR)
Step 5 Add these handlers to logger
log.addHandler(hStErr)
log.addHandler(hStOut)
Here are all the pieces together.
import logging, sys
log = logging.getLogger()
hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)
hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')
hStOut.addFilter(lambda x : x.levelno < logging.ERROR)
log.addHandler(hStErr)
log.addHandler(hStOut)
log.error("error log")
log.info("info log")
Output when we run this script.
error log
info log
Pycharm IDE colors output from std error red. The following image shows that the error log statement above was sent to stderr.
If we comment the addFilter line in above script, we will see the following output.
error log
error log
info log
Note that without filter hStOut will output logging statements from both INFO and ERROR, while for INFO hStErr outputs nothing, and hStOut outputs a single statement - info log
Not necessarily a good idea (it could be confusing to see info and debug messages mixed in with normal output!), but feasible, since you can have multiple handler objects and a custom filter for each of them, in order to pick and choose which log records each handler gets to handle.
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import logging
import sys
class LessThenFilter(logging.Filter):
def __init__(self, level):
self._level = level
logging.Filter.__init__(self)
def filter(self, rec):
return rec.levelno < self._level
log = logging.getLogger()
log.setLevel(logging.NOTSET)
sh_out = logging.StreamHandler(stream=sys.stdout)
sh_out.setLevel(logging.DEBUG)
sh_out.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))
sh_out.addFilter(LessThenFilter(logging.WARNING))
log.addHandler(sh_out)
sh_err = logging.StreamHandler(stream=sys.stderr)
sh_err.setLevel(logging.WARNING)
sh_err.setFormatter(logging.Formatter('%(levelname)s: %(message)s'))
log.addHandler(sh_err)
logging.critical('x')
logging.error('x')
logging.warning('x')
logging.info('x')
logging.debug('x')