I'm trying to decipher the information contained in my logs (the logging setup is using the default formatter). The documentation states:
Do formatting for a record - if a formatter is set, use it. Otherwise, use the default formatter for the module.
However, I can't find any reference actually stating what this default format is.
The default format is located here which is:
BASIC_FORMAT = "%(levelname)s:%(name)s:%(message)s"
The Format code will tell you how you can customize it. Here is one example on how you can customize it.
import sys
import logging
logging.basicConfig(
level=logging.DEBUG,
format="[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
datefmt="%d/%b/%Y %H:%M:%S",
stream=sys.stdout)
logging.info("HEY")
Which results in:
[26/May/2013 06:41:40] INFO [root.<module>:1] HEY
import logging
print(logging.BASIC_FORMAT)
Old thread but this comes up first in my google search results for the query "python logging default format", so I thought I should add my answer.
Also some comments asked about how one could have come to discover this on their own. Here is a natural thing to do:
import logging
print(dir(logging))
BASIC_FORMAT is in there, in fact it is the first entry in the result in my case.
It's in the source of logging/__init__.py:
_defaultFormatter = Formatter()
The default formatting string is %(message)s, which is in the source as well:
if fmt:
self._fmt = fmt
else:
self._fmt = "%(message)s"
#rh0dium's and #Blender's answers contradict each other but they're both correct... just incomplete.
As of Python 3.10:
If you create a handler without specifying a formatter, the default format is "%(message)s".
If you call logging.basicConfig it sets up a handler on the root logger with a formatter that defaults to "%(levelname)s:%(name)s:%(message)s".
Note that logging.info, logging.warning etc will automatically call logging.basicConfig() if you haven't called it already.
But calling the root logger directly (like logging.getLogger().info(...)) won't call logging.basicConfig().
So the actual behaviour depends on whether you've ever called logging.basicConfig or logging.info/logging.warning/etc.
$ python3
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0] on linux
>>> import logging
>>> root = logging.getLogger()
>>> root.warning("W")
W
>>> logging.warning("W")
WARNING:root:W
>>> root.warning("W")
WARNING:root:W
>>> mylogger = logging.getLogger("mylogger")
>>> mylogger.addHandler(logging.StreamHandler())
>>> mylogger.propagate = False
>>> mylogger.warning("W")
W
The default seems to be %(levelname)s:%(name)s:%(message)s
import logging
logging.error("Some error")
print "fmt: " , logging.root.handlers[0].formatter._fmt
# ERROR:root:Some error
# fmt: %(levelname)s:%(name)s:%(message)s
Here is the example of an advanced way of logging:-
import logging
class logger_con():
def log_test(self):
"""
:create logger
:set logger level
:create console handler
:add formatter to console handler
:add console handler to logger
:add logging massage
:return:
"""
#create logger and set level
logger=logging.getLogger(logger_con.__name__)
logger.setLevel(logging.INFO)
#create console handler(we are using steamHandler which is only used to display log in console)
con_handler=logging.StreamHandler()
con_handler.setLevel(logging.INFO)
#create formatter and add formatter to con_handler
formatter=logging.Formatter('%(asctime)s : %(message)s : %(levelname)s -%(name)s',datefmt='%d%m%Y %I:%M:%S %p')
con_handler.setFormatter(formatter)
#add console handler to logger
logger.addHandler(con_handler)
logger.debug('Program debugging')
logger.info('massage conatain information')
logger.warning('warning message')
logger.error('massage contains error')
logger.critical('critical massages')
Related
I am trying to set loggers for my python code, I want to set the level of the log from the configuration file. But unable to do by me. Here the code is given below, If you noticed that in the given below code can see logger.setLevel(logging.INFO). I don't want to directly mention as a hardcoded value logging.INFO. Need to get this from the config file, is it possible?
import logging
from logging.config import fileConfig
from datetime import date
class Log:
#staticmethod
def trace():
today = date.today()
# dd/mm/YY
d1 = today.strftime("%d_%m_%Y")
# Gets or creates a logger
logger = logging.getLogger(__name__)
# set log level
logger.setLevel(logging.INFO)
# define file handler and set formatter
file_handler = logging.FileHandler('log/'+d1+'_logfile.log')
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(name)s : %(message)s')
file_handler.setFormatter(formatter)
# add file handler to logger
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
return logger
You can always use Python built-in Configuration file parser
Have the log levels in a config file and read that value. Since that value will be in string, you can define the dictionary mapping in your code. See below for an example.
import configparser
config= configparser.ConfigParser()
config.read('configfile')
log_level_info = {'logging.DEBUG': logging.DEBUG,
'logging.INFO': logging.INFO,
'logging.WARNING': logging.WARNING,
'logging.ERROR': logging.ERROR,
}
print(config['DEFAULT']['LOG_LEVEL'])
my_log_level_from_config = config['DEFAULT']['LOG_LEVEL']
my_log_level = log_level_info.get(my_log_level_from_config, logging.ERROR)
logger.setLevel(my_log_level)
Your config file would be like below:
user#Inspiron:~/code/advanced_python$ cat configfile
[DEFAULT]
LOG_LEVEL = logging.INFO
user#Inspiron:~/code/advanced_python$
If I understood correctly, you need a way to set your logging level at runtime instead of a hard-coded value. I would say you have two options.
The first solution would be to parse your configuration file, and set the level of logging accordingly. If you don't want to parse it everytime the Log class is invoked, in your main you can set a variable that you pass to the Log class.
The second one, that I also suggest, would be to set handlers with python logging class https://docs.python.org/3/library/logging.config.html
logging level (logging.INFO) is an integer value. can you pass numbers from your config file to set log level
print(logging.INFO)
print(logging.WARN)
print(logging.DEBUG)
print(logging.ERROR)
20
30
10
40
The logging module is a neat way to maintain a clean program logger, with customizable things such as message formatting. There is still a point that is not clear to me: how to specify a custom formatting for the arguments (the series of *args to be merged into the msg format string) passed to the Logger class methods (e.g. logger.info or logger.debug) before they are merged into the msg format string?.
For instance, consider the following simple case:
import logging
from termcolor import colored
# regular logging
logging.basicConfig(level="INFO", format="%(levelname)s\t%(message)s")
logging.info("Hello, %s", "world")
# same with explicitely using bold format for the argument
logging.info("Hello, %s", colored("world", attrs=["bold"]))
Resulting in the following output:
INFO Hello, world!
INFO Hello, world!
Is there a way to specify once for all the formatting for every additional argument passed to the logger method without manually changing every logging?
This section has a nice example for using logging.Formatter()
import logging
from termcolor import colored
new_format = logging.Formatter('%(levelname)s\t' + colored('%(message)s', attrs=['bold']))
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
handler.setFormatter(new_format)
logger = logging.getLogger('')
logger.setLevel(logging.INFO)
logger.addHandler(handler)
logger.info('Hello world!')
EDIT
Realized this works too
import logging
from termcolor import colored
logging.basicConfig(format='%(levelname)s:\t' + colored('%(message)s', attrs=['bold']), level=logging.INFO)
logging.info('Hello World!')
I am trying to beef up the logging in my Python scripts and I am grateful if could share best practices with me. For now I have created this little script (I should say that I run Python 3.4)
import logging
import io
import sys
def Streamhandler(stream, level, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"):
ch = logging.StreamHandler(stream)
ch.setLevel(level)
formatter = logging.Formatter(format)
ch.setFormatter(formatter)
return ch
# get the root logger
logger = logging.getLogger()
stream = io.StringIO()
logger.addHandler(Streamhandler(stream, logging.WARN))
stream_error = io.StringIO()
logger.addHandler(Streamhandler(stream_error, logging.ERROR))
logger.addHandler(Streamhandler(stream=sys.stdout, level=logging.DEBUG))
print(logger)
for h in logger.handlers:
print(h)
print(h.level)
# 'application' code # goes to the root logger!
logging.debug('debug message')
logging.info('info message')
logging.warning('warn message')
logging.error('error message')
logging.critical('critical message')
print(stream.getvalue())
print(stream_error.getvalue())
I have three handlers, 2 of them write into a io.StringIO (this seems to work). I need this to simplify testing but also to send logs via a HTTP email service. And then there is a Streamhandler for the console. However, logging.debug and logging.info messages are ignored on the console here despite setting the level explicitly low enough?!
First, you didn't set the level on the logger itself:
logger.setLevel(logging.DEBUG)
Also, you define a logger but do your calls on logging - which will call on the root logger. Not that it will make any difference in your case since you didn't specify a name for your logger, so logging.getLogger() returns the root logger.
wrt/ "best practices", it really depends on how "complex" your scripts are and of course on your logging needs.
For self-contained simple scripts with simple use cases (single known environment, no concurrent execution, simple logging to a file or stderr etc), a simple call to logging.basicConfig() and direct calls to logging.whatever() are usually good enough.
For anything more complex, it's better to use a distinct config file - either in ini format or as Python dict (using logging.dictConfig), split your script into distinct module(s) or package(s) each defining it's own named logger (with logger = logging.getLogger(__name__)) and only keep your script itself as the "runner" for your code, ie: configure logging, import modules, parse command line args and call the main function - preferably in a try/except block as to properly log any unhandled exception before crashing.
A logger has a threshold level too, you need to set it to DEBUG first:
logger.setLevel(logging.DEBUG)
In my code I get a logger from my client, then I do stuff and log my analysis to the logger.
I want to add my own prefix to the logger but I don't want to create my own formatter, just to add my prefix to the existing one.
In addition I want to remove my prefix once my code is done.
From looking at the documentation I could only find ways to create new formatter but not to modify an existing one. Is there a way to do so?
You are correct. As per Python 3 and Python 2 documentation there is no way to reset your format on the existing formatter object and you do need to create a new logging.Formatter object. However, looking at the object at runtime there is _fmt method to get the existing format and it seems tweaking it will work. I tried in 2.7 and it works. Below is the example.
Example code for python 2.7:
import logging
logger = logging.getLogger('something')
myFormatter = logging.Formatter('%(asctime)s - %(message)s')
handler = logging.StreamHandler()
handler.setFormatter(myFormatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.info("log statement here")
#Tweak the formatter
myFormatter._fmt = "My PREFIX -- " + myFormatter._fmt
logger.info("another log statement here")
Output:
2015-03-11 12:51:36,605 - log statement here
My PREFIX -- 2015-03-11 12:51:36,605 - another log statement here
This can be achieved with logging.LoggerAdapter
import logging
class CustomAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return f"[my prefix] {msg}", kwargs
logger = CustomAdapter(logging.getLogger(__name__))
Please note that only the message will be affected. But this technique can be used for more complicated cases
You can actually set the format through the 'basicConfig', it is mentioned in the Python document: https://docs.python.org/2/howto/logging-cookbook.html#context-info
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('\n new hello')
11:15:01 INFO hello
11:16:49 WARNING
new hello
Because the log is crowded, I want to explicitly insert a newline before asctime and levelname. Is this possible without modifying format?
I looked into logging module and googled a bit and could not find a viable way.
I have two solutions, the first is very easy, but the output is not very clean. The second method will produce the exact output you want, but it is a little more involved.
Method 1
To produce a blank line, just log an empty string with a new line:
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('\n')
logging.warning('new hello')
The output will have an empty info line, which is not very clean:
16:07:26 INFO hello
16:07:26 INFO
16:07:26 WARNING new hello
Method 2
In this method, I created two different handlers. The console_handler which I use most of the time. When I need a new line, I switch to a second handler, blank_handler.
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch handler, output a blank line
self.removeHandler(self.console_handler)
self.addHandler(self.blank_handler)
for i in range(how_many_lines):
self.info('')
# Switch back
self.removeHandler(self.blank_handler)
self.addHandler(self.console_handler)
def create_logger():
# Create a handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
console_handler.setFormatter(logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s"))
# Create a "blank line" handler
blank_handler = logging.StreamHandler()
blank_handler.setLevel(logging.DEBUG)
blank_handler.setFormatter(logging.Formatter(fmt=''))
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
# Save some data and add a method to logger object
logger.console_handler = console_handler
logger.blank_handler = blank_handler
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
The output is what you want to see:
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
Discussion
If you can put up with the less-than-perfect output, method 1 is the way to go. It has the advantage of being simple, least amount of effort.
The second method does the job correctly, but it is a little involved. It creates two different handlers and switch them in order to achieve your goal.
Another disadvantage of using method 2 is you have to change your code by searching for logging and replacing them with logger. You must take care replacing only relevant parts and leave such text as logging.DEBUG in tact.
Could you not add the newline after the first hello? i.e.
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello\n')
logging.info('new hello')
Which will output
2014-08-06 11:37:24,061 INFO : hello
2014-08-06 11:37:24,061 INFO : new hello
Easiest way to insert newlines that I figured out:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s\n\r%(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('new hello')
11:50:32 INFO
hello
11:50:32 INFO
new hello
Use a custom Formatter which uses different format strings at different times. You can't do this using basicConfig() - you'll have to use other parts of the logging API.
class MyFormatter(logging.Formatter):
def format(self, record):
# set self._fmt to value with or without newline,
# as per your decision criteria
# self._fmt = ...
return super(MyFormatter, self).format(record)
Or, you can call the super method, then modify the string to insert a newline before returning it (in case it's dependent on line length, say).
As an alternative to Hai Vu's Method 2 you could as well reset the handler's Formatter every time you want to log a new line:
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch formatter, output a blank line
self.handler.setFormatter(self.blank_formatter)
for i in range(how_many_lines):
self.info('')
# Switch back
self.handler.setFormatter(self.formatter)
def create_logger():
# Create a handler
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s")
blank_formatter = logging.Formatter(fmt="")
handler.setFormatter(formatter)
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
# Save some data and add a method to logger object
logger.handler = handler
logger.formatter = formatter
logger.blank_formatter = blank_formatter
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
Output
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
The advantage of this is that you have a single handler. For example you can define a FileHandler's mode-attribute to write, if you wanted to clean your log-file on every new run of your program.
If you are just looking to output some debug code in development then you may not want to spend time on this. The 5 second fix is this;
str = "\n\n\n"
log.getLogger().debug(str)
where the logger is the standard python logger
Something like this. Add \n into you logging.basicConfig between asctime and levelname
>>> logging.basicConfig(level=logging.DEBUG, format='%(asctime)s\n %(levelname)s %(message)s',datefmt='%H:%M:%S')
What about writing to the log file, without the logging service?
fn_log = 'test.log'
logging.basicConfig(filename=fn_log, level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('no empty line')
def empty_line(fn_log):
new_empty_line = open(fn_log,'a+')
new_empty_line.write('\n')
new_empty_line.close()
empty_line(fn_log)
logging.warning('hello')
Output:
09:26:00 INFO hello
11:51:05 INFO hello
11:51:05 WARNING no empty line
11:51:05 WARNING hello
Following up on Vinay Salip's helpful answer (below), I did it this way (I'm using the python3 superclass convention, but super(MyFormatter, self) works just as well) ...
class MyFormatter(logging.Formatter):
def format(self, record):
return super().format(record).replace(r'\n', '\n')
Then, I can embed newlines as follows:
logging.info('Message\\n\\n\\n\\nOther stuff')
or
logging.info(r'Message\n\n\n\nOther stuff')
If you use FileHandler or descendants thereof, these two functions may help. An added benefit is that all FileHandler type handlers attached to the logger should get the newline.
def getAllLoggerFilenames(logger):
""" Returns array of all log filenames attached to the logger. """
logFiles = [];
parent = logger.__dict__['parent'];
if parent.__class__.__name__ == 'RootLogger':
for h in logger.__dict__['handlers']:
if h.baseFilename:
logFiles.append(h.baseFilename);
else:
logFiles = getAllLoggerFilenames(parent);
return logFiles;
def logBlankLine(logger):
""" This utility method writes a blank line to the log. """
logNames = getAllLoggerFilenames(logger)
for fn in logNames:
with open(fn, 'a') as fh:
fh.write("\n")
Usage:
# We use YAML for logging config files, YMMV:
with open(logConfig, 'rt') as f:
logging.config.dictConfig(yaml.safe_load(f.read()))
logger = logging.getLogger("test.test")
logger.info("line 1")
logBlankLine(logger)
logger.info("line 2")
Output:
2019/12/22 16:33:59.152: INFO : test.test : line 1
2019/12/22 16:33:59.152: INFO : test.test : line 2
The easiest solution is to use f-strings if you are using Python 3:
logging.info( f'hello\n' )
You can try the following solution. It's simple and straightforward.
logging.debug("\b" * 20) # output blank line using escape character
logging.debug("debug message")