In my code I get a logger from my client, then I do stuff and log my analysis to the logger.
I want to add my own prefix to the logger but I don't want to create my own formatter, just to add my prefix to the existing one.
In addition I want to remove my prefix once my code is done.
From looking at the documentation I could only find ways to create new formatter but not to modify an existing one. Is there a way to do so?
You are correct. As per Python 3 and Python 2 documentation there is no way to reset your format on the existing formatter object and you do need to create a new logging.Formatter object. However, looking at the object at runtime there is _fmt method to get the existing format and it seems tweaking it will work. I tried in 2.7 and it works. Below is the example.
Example code for python 2.7:
import logging
logger = logging.getLogger('something')
myFormatter = logging.Formatter('%(asctime)s - %(message)s')
handler = logging.StreamHandler()
handler.setFormatter(myFormatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.info("log statement here")
#Tweak the formatter
myFormatter._fmt = "My PREFIX -- " + myFormatter._fmt
logger.info("another log statement here")
Output:
2015-03-11 12:51:36,605 - log statement here
My PREFIX -- 2015-03-11 12:51:36,605 - another log statement here
This can be achieved with logging.LoggerAdapter
import logging
class CustomAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return f"[my prefix] {msg}", kwargs
logger = CustomAdapter(logging.getLogger(__name__))
Please note that only the message will be affected. But this technique can be used for more complicated cases
You can actually set the format through the 'basicConfig', it is mentioned in the Python document: https://docs.python.org/2/howto/logging-cookbook.html#context-info
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
Related
I am using a library python-sonarqube-api, which shows a password in debug logs using a logger which I consider a bug.
Until it can be fixed I need to hide the password in the logs. I am considering using a filter but I am not sure how to use it without breaking current structure of all loggers in the software.
Could you suggest me some solution please?
You can solve this problem using a custom Formatter.
import logging
import re
class SensitiveFormatter(logging.Formatter):
"""Formatter that removes sensitive information in logs."""
#staticmethod
def _filter(s):
# Filter out the password with regex
# or replace etc.
# Replace here with your own regex..
return re.sub(r"ABCDEF", r"<MASKED>", s)
def format(self, record):
original = logging.Formatter.format(self, record) # call parent method
return self._filter(original)
Then, use it inside your handlers:
# Create the specific logger
mylogger = logging.getLogger("foobar")
mylogger.setLevel(logging.DEBUG)
mylogger.propagate = False
# Create the handler
streamhandler = logging.StreamHandler()
streamhandler.setLevel(logging.INFO)
# Create the specific formatter
sensitive_formatter = SensitiveFormatter(
fmt="[pid:%(process)d] - %(asctime)s - %(levelname)-8s - %(message).1000s"
)
streamhandler.setFormatter(sensitive_formatter)
mylogger.addHandler(streamhandler)
mylogger.info("This is a password: ABCDEF")
[pid:453381] - 2023-02-07 14:47:56,075 - INFO - This is a password: <MASKED>
I am trying to set loggers for my python code, I want to set the level of the log from the configuration file. But unable to do by me. Here the code is given below, If you noticed that in the given below code can see logger.setLevel(logging.INFO). I don't want to directly mention as a hardcoded value logging.INFO. Need to get this from the config file, is it possible?
import logging
from logging.config import fileConfig
from datetime import date
class Log:
#staticmethod
def trace():
today = date.today()
# dd/mm/YY
d1 = today.strftime("%d_%m_%Y")
# Gets or creates a logger
logger = logging.getLogger(__name__)
# set log level
logger.setLevel(logging.INFO)
# define file handler and set formatter
file_handler = logging.FileHandler('log/'+d1+'_logfile.log')
formatter = logging.Formatter('%(asctime)s : %(levelname)s : %(name)s : %(message)s')
file_handler.setFormatter(formatter)
# add file handler to logger
logger.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logger.addHandler(console_handler)
return logger
You can always use Python built-in Configuration file parser
Have the log levels in a config file and read that value. Since that value will be in string, you can define the dictionary mapping in your code. See below for an example.
import configparser
config= configparser.ConfigParser()
config.read('configfile')
log_level_info = {'logging.DEBUG': logging.DEBUG,
'logging.INFO': logging.INFO,
'logging.WARNING': logging.WARNING,
'logging.ERROR': logging.ERROR,
}
print(config['DEFAULT']['LOG_LEVEL'])
my_log_level_from_config = config['DEFAULT']['LOG_LEVEL']
my_log_level = log_level_info.get(my_log_level_from_config, logging.ERROR)
logger.setLevel(my_log_level)
Your config file would be like below:
user#Inspiron:~/code/advanced_python$ cat configfile
[DEFAULT]
LOG_LEVEL = logging.INFO
user#Inspiron:~/code/advanced_python$
If I understood correctly, you need a way to set your logging level at runtime instead of a hard-coded value. I would say you have two options.
The first solution would be to parse your configuration file, and set the level of logging accordingly. If you don't want to parse it everytime the Log class is invoked, in your main you can set a variable that you pass to the Log class.
The second one, that I also suggest, would be to set handlers with python logging class https://docs.python.org/3/library/logging.config.html
logging level (logging.INFO) is an integer value. can you pass numbers from your config file to set log level
print(logging.INFO)
print(logging.WARN)
print(logging.DEBUG)
print(logging.ERROR)
20
30
10
40
I want to log to 2 different files, but different things.
I was trying this:
runid = str(uuid.uuid1())
logger = logging.getLogger('logger1')
other_logger = logging.getLogger('logger2')
logger.setLevel(logging.DEBUG)
debug_handler = logging.FileHandler('log/debug.log')
info_handler = logging.FileHandler('log/info.log')
other_handler = logging.FileHandler('log/other_info.log')
debug_handler.setLevel(logging.DEBUG)
info_handler.setLevel(logging.INFO)
other_handler.setLevel(logging.INFO)
formatter = logging.Formatter(runid + ' - %(asctime)s - %(levelname)s - %(funcName)s - %(message)s')
debug_handler.setFormatter(formatter)
info_handler.setFormatter(formatter)
other_handler.setFormatter(formatter)
logger.addHandler(debug_handler)
logger.addHandler(info_handler)
other_logger.addHandler(other_handler)
logger.info('message1')
other_logger.info('message2')
But logger and other_logger are working as one and I get both messages in all files, no matter if I call in logger or other_logger.
According doc:
"Loggers have the following attributes and methods. Note that Loggers
are never instantiated directly, but always through the module-level
function logging.getLogger(name). Multiple calls to getLogger() with
the same name will always return a reference to the same Logger
object."
But the parent object is always the same, as in this small test:
import logging
log1 = logging.getLogger('hey')
log2 = logging.getLogger('you')
print log1.parent, log2.parent
enrique#enrique-mbp:$ python /tmp/test.py
<logging.RootLogger object at 0x26f0810> <logging.RootLogger object at 0x26f0810>
How can I solve this?
The problem is here:
logger.setLevel(logging.DEBUG)
You need to set the level for other_logger as well:
logger.setLevel(logging.DEBUG)
other_logger.setLevel(logging.DEBUG) # or INFO because that is the lowest level being used by a handler
Without this, other_logger remains at the default logging level, logging.WARNING, which prevents the other_logger.info('message2') line from logging anything.
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('\n new hello')
11:15:01 INFO hello
11:16:49 WARNING
new hello
Because the log is crowded, I want to explicitly insert a newline before asctime and levelname. Is this possible without modifying format?
I looked into logging module and googled a bit and could not find a viable way.
I have two solutions, the first is very easy, but the output is not very clean. The second method will produce the exact output you want, but it is a little more involved.
Method 1
To produce a blank line, just log an empty string with a new line:
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('\n')
logging.warning('new hello')
The output will have an empty info line, which is not very clean:
16:07:26 INFO hello
16:07:26 INFO
16:07:26 WARNING new hello
Method 2
In this method, I created two different handlers. The console_handler which I use most of the time. When I need a new line, I switch to a second handler, blank_handler.
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch handler, output a blank line
self.removeHandler(self.console_handler)
self.addHandler(self.blank_handler)
for i in range(how_many_lines):
self.info('')
# Switch back
self.removeHandler(self.blank_handler)
self.addHandler(self.console_handler)
def create_logger():
# Create a handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
console_handler.setFormatter(logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s"))
# Create a "blank line" handler
blank_handler = logging.StreamHandler()
blank_handler.setLevel(logging.DEBUG)
blank_handler.setFormatter(logging.Formatter(fmt=''))
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
# Save some data and add a method to logger object
logger.console_handler = console_handler
logger.blank_handler = blank_handler
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
The output is what you want to see:
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
Discussion
If you can put up with the less-than-perfect output, method 1 is the way to go. It has the advantage of being simple, least amount of effort.
The second method does the job correctly, but it is a little involved. It creates two different handlers and switch them in order to achieve your goal.
Another disadvantage of using method 2 is you have to change your code by searching for logging and replacing them with logger. You must take care replacing only relevant parts and leave such text as logging.DEBUG in tact.
Could you not add the newline after the first hello? i.e.
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello\n')
logging.info('new hello')
Which will output
2014-08-06 11:37:24,061 INFO : hello
2014-08-06 11:37:24,061 INFO : new hello
Easiest way to insert newlines that I figured out:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s\n\r%(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('new hello')
11:50:32 INFO
hello
11:50:32 INFO
new hello
Use a custom Formatter which uses different format strings at different times. You can't do this using basicConfig() - you'll have to use other parts of the logging API.
class MyFormatter(logging.Formatter):
def format(self, record):
# set self._fmt to value with or without newline,
# as per your decision criteria
# self._fmt = ...
return super(MyFormatter, self).format(record)
Or, you can call the super method, then modify the string to insert a newline before returning it (in case it's dependent on line length, say).
As an alternative to Hai Vu's Method 2 you could as well reset the handler's Formatter every time you want to log a new line:
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch formatter, output a blank line
self.handler.setFormatter(self.blank_formatter)
for i in range(how_many_lines):
self.info('')
# Switch back
self.handler.setFormatter(self.formatter)
def create_logger():
# Create a handler
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s")
blank_formatter = logging.Formatter(fmt="")
handler.setFormatter(formatter)
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
# Save some data and add a method to logger object
logger.handler = handler
logger.formatter = formatter
logger.blank_formatter = blank_formatter
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
Output
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
The advantage of this is that you have a single handler. For example you can define a FileHandler's mode-attribute to write, if you wanted to clean your log-file on every new run of your program.
If you are just looking to output some debug code in development then you may not want to spend time on this. The 5 second fix is this;
str = "\n\n\n"
log.getLogger().debug(str)
where the logger is the standard python logger
Something like this. Add \n into you logging.basicConfig between asctime and levelname
>>> logging.basicConfig(level=logging.DEBUG, format='%(asctime)s\n %(levelname)s %(message)s',datefmt='%H:%M:%S')
What about writing to the log file, without the logging service?
fn_log = 'test.log'
logging.basicConfig(filename=fn_log, level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('no empty line')
def empty_line(fn_log):
new_empty_line = open(fn_log,'a+')
new_empty_line.write('\n')
new_empty_line.close()
empty_line(fn_log)
logging.warning('hello')
Output:
09:26:00 INFO hello
11:51:05 INFO hello
11:51:05 WARNING no empty line
11:51:05 WARNING hello
Following up on Vinay Salip's helpful answer (below), I did it this way (I'm using the python3 superclass convention, but super(MyFormatter, self) works just as well) ...
class MyFormatter(logging.Formatter):
def format(self, record):
return super().format(record).replace(r'\n', '\n')
Then, I can embed newlines as follows:
logging.info('Message\\n\\n\\n\\nOther stuff')
or
logging.info(r'Message\n\n\n\nOther stuff')
If you use FileHandler or descendants thereof, these two functions may help. An added benefit is that all FileHandler type handlers attached to the logger should get the newline.
def getAllLoggerFilenames(logger):
""" Returns array of all log filenames attached to the logger. """
logFiles = [];
parent = logger.__dict__['parent'];
if parent.__class__.__name__ == 'RootLogger':
for h in logger.__dict__['handlers']:
if h.baseFilename:
logFiles.append(h.baseFilename);
else:
logFiles = getAllLoggerFilenames(parent);
return logFiles;
def logBlankLine(logger):
""" This utility method writes a blank line to the log. """
logNames = getAllLoggerFilenames(logger)
for fn in logNames:
with open(fn, 'a') as fh:
fh.write("\n")
Usage:
# We use YAML for logging config files, YMMV:
with open(logConfig, 'rt') as f:
logging.config.dictConfig(yaml.safe_load(f.read()))
logger = logging.getLogger("test.test")
logger.info("line 1")
logBlankLine(logger)
logger.info("line 2")
Output:
2019/12/22 16:33:59.152: INFO : test.test : line 1
2019/12/22 16:33:59.152: INFO : test.test : line 2
The easiest solution is to use f-strings if you are using Python 3:
logging.info( f'hello\n' )
You can try the following solution. It's simple and straightforward.
logging.debug("\b" * 20) # output blank line using escape character
logging.debug("debug message")
I'm trying to decipher the information contained in my logs (the logging setup is using the default formatter). The documentation states:
Do formatting for a record - if a formatter is set, use it. Otherwise, use the default formatter for the module.
However, I can't find any reference actually stating what this default format is.
The default format is located here which is:
BASIC_FORMAT = "%(levelname)s:%(name)s:%(message)s"
The Format code will tell you how you can customize it. Here is one example on how you can customize it.
import sys
import logging
logging.basicConfig(
level=logging.DEBUG,
format="[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
datefmt="%d/%b/%Y %H:%M:%S",
stream=sys.stdout)
logging.info("HEY")
Which results in:
[26/May/2013 06:41:40] INFO [root.<module>:1] HEY
import logging
print(logging.BASIC_FORMAT)
Old thread but this comes up first in my google search results for the query "python logging default format", so I thought I should add my answer.
Also some comments asked about how one could have come to discover this on their own. Here is a natural thing to do:
import logging
print(dir(logging))
BASIC_FORMAT is in there, in fact it is the first entry in the result in my case.
It's in the source of logging/__init__.py:
_defaultFormatter = Formatter()
The default formatting string is %(message)s, which is in the source as well:
if fmt:
self._fmt = fmt
else:
self._fmt = "%(message)s"
#rh0dium's and #Blender's answers contradict each other but they're both correct... just incomplete.
As of Python 3.10:
If you create a handler without specifying a formatter, the default format is "%(message)s".
If you call logging.basicConfig it sets up a handler on the root logger with a formatter that defaults to "%(levelname)s:%(name)s:%(message)s".
Note that logging.info, logging.warning etc will automatically call logging.basicConfig() if you haven't called it already.
But calling the root logger directly (like logging.getLogger().info(...)) won't call logging.basicConfig().
So the actual behaviour depends on whether you've ever called logging.basicConfig or logging.info/logging.warning/etc.
$ python3
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0] on linux
>>> import logging
>>> root = logging.getLogger()
>>> root.warning("W")
W
>>> logging.warning("W")
WARNING:root:W
>>> root.warning("W")
WARNING:root:W
>>> mylogger = logging.getLogger("mylogger")
>>> mylogger.addHandler(logging.StreamHandler())
>>> mylogger.propagate = False
>>> mylogger.warning("W")
W
The default seems to be %(levelname)s:%(name)s:%(message)s
import logging
logging.error("Some error")
print "fmt: " , logging.root.handlers[0].formatter._fmt
# ERROR:root:Some error
# fmt: %(levelname)s:%(name)s:%(message)s
Here is the example of an advanced way of logging:-
import logging
class logger_con():
def log_test(self):
"""
:create logger
:set logger level
:create console handler
:add formatter to console handler
:add console handler to logger
:add logging massage
:return:
"""
#create logger and set level
logger=logging.getLogger(logger_con.__name__)
logger.setLevel(logging.INFO)
#create console handler(we are using steamHandler which is only used to display log in console)
con_handler=logging.StreamHandler()
con_handler.setLevel(logging.INFO)
#create formatter and add formatter to con_handler
formatter=logging.Formatter('%(asctime)s : %(message)s : %(levelname)s -%(name)s',datefmt='%d%m%Y %I:%M:%S %p')
con_handler.setFormatter(formatter)
#add console handler to logger
logger.addHandler(con_handler)
logger.debug('Program debugging')
logger.info('massage conatain information')
logger.warning('warning message')
logger.error('massage contains error')
logger.critical('critical massages')