Basically, I want to initialise a logger for each purpose, such that each logger will log to a file, but I notice that the content of log has been printed to console as well. I don't want that to happen, but I don't know how to adjust my code.
def getLogger(loggerName:str, fileDir:str,
level = logging.DEBUG,
format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'):
"""return a logger for logging
somehow the logDecorator function does
Args:
loggerName (str): name given to the logger
fileDir (str): directory
level ([type], optional): [description]. Defaults to logging.DEBUG.
format (str, optional): [description]. Defaults to '%(asctime)s - %(name)s - %(levelname)s - %(message)s'.
Returns:
logger: [description]
"""
myLogger = logging.getLogger(loggerName)
myLogger.setLevel(level)
myFormat = logging.Formatter(format)
myHandler = logging.FileHandler(fileDir, mode='a')
myHandler.setFormatter(myFormat)
myLogger.addHandler(myHandler)
# console only prints WARNING & ABOVE
# ideally, what I thought is that if I don't not
# add a SteamHandler, the log won't even print to console at all
# but adding a SteamHandler still prints EVERYTHING to the console
# despite the level set as WARNING
console = logging.StreamHandler()
console.setLevel(logging.WARNING)
console.setFormatter(myFormat)
myLogger.addHandler(console)
return myLogger
I have tried this:
mainLogger = logging.getLogger()
mainLogger.setLevel(logging.WARNING) # only log WARNING and above
but It won't work as well, still printing to console.
So I am lost in my understanding of LOGGING module, I am not sure where have I done wrong.
Example of using the code:
aRandomLogger = getLogger(loggerName = 'ProgressLogger', fileDir = '../logs/ProgressLogger.log')
aRandomLogger.info('This is an information, I want it to be log to file, but not on console')
aRandomLogger.warning('this is a warning, I want it be be log to file, and shown on console')
Turns out, both logs are saved to file and to console.
It has been awhile, but I have managed to figure out the solution to this.
myLogger = logging.getLogger(loggerName)
myLogger.propagate = False
Setting propagate to False will prevent the logger from producing unwanted logs everywhere.
IN a Rest API server I am using logging module to logs the script run and to send the result to th client. I am using a logging with a file global handler and a single stream handler.
The Http response depends by the number of errors and critical in logger.
IN order to count logger and error I am using a custom version of logger.
Inspired by this post
I wrote the following code in order to count the times a method is called
def init_logger(lid: str):
log = MyLogger(str(lid))
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
log.addHandler(ch)
log.setLevel(logging.DEBUG)
return log
The problem is that if I init two different logger with different id the counting is overlapped as showed below
log1 = init_logger(1)
log2 = init_logger(2)
log1.info("INFO 1")
print(log2.info.called) -->1
It doesn't work because using a decorator for this is a horrible way to count logs. What if someone writes a log by using log1.log(logging.INFO, "INFO 1") instead of using the convenience shortcut?
Here is a much better way, which is also how the python documentation proposes to do such things:
import logging
from collections import Counter
def counting_filter(record):
logger = logging.getLogger(record.name)
logger.counter[record.levelno] += 1
return True
def init_logger(lid):
log = logging.getLogger(str(lid))
log.counter = Counter() # <-- added this line, but count could be stored anywhere
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
log.addHandler(ch)
log.setLevel(logging.DEBUG)
log.addFilter(counting_filter) # <-- added Filter here
return log
log1 = init_logger(1)
log2 = init_logger(2)
log1.info("info 1")
print(log1.counter[logging.INFO]) # 1
print(log2.counter[logging.INFO]) # 0
Currently I use for logging a custom logging system that works as follow:
I have a Log class that ressemble the following:
class Log:
def __init__(self):
self.script = ""
self.datetime = datetime.datetime.now().replace(second=0, microsecond=0)
self.mssg = ""
self.mssg_detail = ""
self.err = ""
self.err_detail = ""
I created a function decorator that perform a try/except on the function call, and add a message either to .mssg or .err on the Log object accordingly.
def logging(fun):
#functools.wraps(fun)
def inner(self, *args):
try:
f = fun(self, *args)
self.logger.mssg += fun.__name__ +" :ok, "
return f
except Exception as e:
self.logger.err += fun.__name__ +": error: "+str(e.args)
return inner
So usually a script is a class that is composed of multiple methods that are run sequentially.
I hence run those methods (decorated such as mentionned above) , and lastly I upload the Log object into a mysql db.
This works quite fine and alright. But now I want to modify those items so that they integrate with the "official" logging module of python.
What I dont like about that module is that it is not possible to "save" the messages onto 1 log object in order to upload/save to log only at the end of the run. Rather each logging call will write/send the message to a file etc. - which create lots of performances issues sometimes. I could usehandlers.MemoryHandler , but it still doesn't seems to perform as my original system: it is said to collect messages and flush them to another handler periodically - which is not what i want: I want to collect the messages in memory and to flush them on request with an explicit function.
Anyone has any suggestions?
Here is my idea. Use a handler to capture the log in a StringIO. Then you can grab the StringIO whenever you want. Since there was perhaps some confusion in the discussion thread - StringIO is a "file-like" interface for strings, there isn't ever an actual file involved.
import logging
import io
def initialize_logging(log_level, log_name='default_logname'):
logger = logging.getLogger(log_name)
logger.setLevel(log_level)
log_stream = io.StringIO()
if not logger.handlers:
ch = logging.StreamHandler(log_stream)
ch.setLevel(log_level)
ch.setFormatter(logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
))
logger.addHandler(ch)
logger.propagate = 0
return logger, log_stream
And then something like:
>>> logger, log_stream = initialize_logging(logging.INFO, "logname")
>>> logger.warning("Hello World!")
And when you want the log information:
>>> log_stream.getvalue()
'2017-05-16 16:35:03,501 - logname - WARNING - Hello World!\n'
At program start (in the main), you can:
instanciate your custom logger => global variable/singleton.
register a function at program end which will flush your logger.
Run your decorated functions.
To register a function you can use atexit.register function. See the page Exit handlers in the doc.
EDIT
The idea above can be simplified.
To delay the logging, you can use the standard MemoryHandler handler, described in the page logging.handlers — Logging handlers
Take a look at this GitHub project: https://github.com/tantale/python-ini-cfg-demo
And replace the INI file by this:
[formatters]
keys=default
[formatter_default]
format=%(asctime)s:%(levelname)s:%(message)s
class=logging.Formatter
[handlers]
keys=console, alternate
[handler_console]
class=logging.handlers.MemoryHandler
formatter=default
args=(1024, INFO)
target=alternate
[handler_alternate]
class=logging.StreamHandler
formatter=default
args=()
[loggers]
keys=root
[logger_root]
level=DEBUG
formatter=default
handlers=console
To log to a database table, just replace the alternate handler by your own database handler.
There is some blog/SO questions about that:
You can look at Logging Exceptions To Your SQLAlchemy Database to create a SQLAlchemyHandler
See Store Django log to database if you are using DJango.
EDIT2
Note: ORM generally support "Eager loading", for instance with SqlAlchemy
In my code I get a logger from my client, then I do stuff and log my analysis to the logger.
I want to add my own prefix to the logger but I don't want to create my own formatter, just to add my prefix to the existing one.
In addition I want to remove my prefix once my code is done.
From looking at the documentation I could only find ways to create new formatter but not to modify an existing one. Is there a way to do so?
You are correct. As per Python 3 and Python 2 documentation there is no way to reset your format on the existing formatter object and you do need to create a new logging.Formatter object. However, looking at the object at runtime there is _fmt method to get the existing format and it seems tweaking it will work. I tried in 2.7 and it works. Below is the example.
Example code for python 2.7:
import logging
logger = logging.getLogger('something')
myFormatter = logging.Formatter('%(asctime)s - %(message)s')
handler = logging.StreamHandler()
handler.setFormatter(myFormatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.info("log statement here")
#Tweak the formatter
myFormatter._fmt = "My PREFIX -- " + myFormatter._fmt
logger.info("another log statement here")
Output:
2015-03-11 12:51:36,605 - log statement here
My PREFIX -- 2015-03-11 12:51:36,605 - another log statement here
This can be achieved with logging.LoggerAdapter
import logging
class CustomAdapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return f"[my prefix] {msg}", kwargs
logger = CustomAdapter(logging.getLogger(__name__))
Please note that only the message will be affected. But this technique can be used for more complicated cases
You can actually set the format through the 'basicConfig', it is mentioned in the Python document: https://docs.python.org/2/howto/logging-cookbook.html#context-info
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)-15s %(name)-5s %(levelname)-8s IP: %(ip)-15s User: %(user)-8s %(message)s')
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('\n new hello')
11:15:01 INFO hello
11:16:49 WARNING
new hello
Because the log is crowded, I want to explicitly insert a newline before asctime and levelname. Is this possible without modifying format?
I looked into logging module and googled a bit and could not find a viable way.
I have two solutions, the first is very easy, but the output is not very clean. The second method will produce the exact output you want, but it is a little more involved.
Method 1
To produce a blank line, just log an empty string with a new line:
import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('\n')
logging.warning('new hello')
The output will have an empty info line, which is not very clean:
16:07:26 INFO hello
16:07:26 INFO
16:07:26 WARNING new hello
Method 2
In this method, I created two different handlers. The console_handler which I use most of the time. When I need a new line, I switch to a second handler, blank_handler.
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch handler, output a blank line
self.removeHandler(self.console_handler)
self.addHandler(self.blank_handler)
for i in range(how_many_lines):
self.info('')
# Switch back
self.removeHandler(self.blank_handler)
self.addHandler(self.console_handler)
def create_logger():
# Create a handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.DEBUG)
console_handler.setFormatter(logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s"))
# Create a "blank line" handler
blank_handler = logging.StreamHandler()
blank_handler.setLevel(logging.DEBUG)
blank_handler.setFormatter(logging.Formatter(fmt=''))
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(console_handler)
# Save some data and add a method to logger object
logger.console_handler = console_handler
logger.blank_handler = blank_handler
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
The output is what you want to see:
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
Discussion
If you can put up with the less-than-perfect output, method 1 is the way to go. It has the advantage of being simple, least amount of effort.
The second method does the job correctly, but it is a little involved. It creates two different handlers and switch them in order to achieve your goal.
Another disadvantage of using method 2 is you have to change your code by searching for logging and replacing them with logger. You must take care replacing only relevant parts and leave such text as logging.DEBUG in tact.
Could you not add the newline after the first hello? i.e.
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello\n')
logging.info('new hello')
Which will output
2014-08-06 11:37:24,061 INFO : hello
2014-08-06 11:37:24,061 INFO : new hello
Easiest way to insert newlines that I figured out:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s\n\r%(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.info('new hello')
11:50:32 INFO
hello
11:50:32 INFO
new hello
Use a custom Formatter which uses different format strings at different times. You can't do this using basicConfig() - you'll have to use other parts of the logging API.
class MyFormatter(logging.Formatter):
def format(self, record):
# set self._fmt to value with or without newline,
# as per your decision criteria
# self._fmt = ...
return super(MyFormatter, self).format(record)
Or, you can call the super method, then modify the string to insert a newline before returning it (in case it's dependent on line length, say).
As an alternative to Hai Vu's Method 2 you could as well reset the handler's Formatter every time you want to log a new line:
import logging
import types
def log_newline(self, how_many_lines=1):
# Switch formatter, output a blank line
self.handler.setFormatter(self.blank_formatter)
for i in range(how_many_lines):
self.info('')
# Switch back
self.handler.setFormatter(self.formatter)
def create_logger():
# Create a handler
handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt="%(name)s %(levelname)-8s: %(message)s")
blank_formatter = logging.Formatter(fmt="")
handler.setFormatter(formatter)
# Create a logger, with the previously-defined handler
logger = logging.getLogger('logging_test')
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
# Save some data and add a method to logger object
logger.handler = handler
logger.formatter = formatter
logger.blank_formatter = blank_formatter
logger.newline = types.MethodType(log_newline, logger)
return logger
if __name__ == '__main__':
logger = create_logger()
logger.info('Start reading database')
logger.info('Updating records ...')
logger.newline()
logger.info('Finish updating records')
Output
logging_test INFO : Start reading database
logging_test INFO : Updating records ...
logging_test INFO : Finish updating records
The advantage of this is that you have a single handler. For example you can define a FileHandler's mode-attribute to write, if you wanted to clean your log-file on every new run of your program.
If you are just looking to output some debug code in development then you may not want to spend time on this. The 5 second fix is this;
str = "\n\n\n"
log.getLogger().debug(str)
where the logger is the standard python logger
Something like this. Add \n into you logging.basicConfig between asctime and levelname
>>> logging.basicConfig(level=logging.DEBUG, format='%(asctime)s\n %(levelname)s %(message)s',datefmt='%H:%M:%S')
What about writing to the log file, without the logging service?
fn_log = 'test.log'
logging.basicConfig(filename=fn_log, level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s', datefmt='%H:%M:%S')
logging.info('hello')
logging.warning('no empty line')
def empty_line(fn_log):
new_empty_line = open(fn_log,'a+')
new_empty_line.write('\n')
new_empty_line.close()
empty_line(fn_log)
logging.warning('hello')
Output:
09:26:00 INFO hello
11:51:05 INFO hello
11:51:05 WARNING no empty line
11:51:05 WARNING hello
Following up on Vinay Salip's helpful answer (below), I did it this way (I'm using the python3 superclass convention, but super(MyFormatter, self) works just as well) ...
class MyFormatter(logging.Formatter):
def format(self, record):
return super().format(record).replace(r'\n', '\n')
Then, I can embed newlines as follows:
logging.info('Message\\n\\n\\n\\nOther stuff')
or
logging.info(r'Message\n\n\n\nOther stuff')
If you use FileHandler or descendants thereof, these two functions may help. An added benefit is that all FileHandler type handlers attached to the logger should get the newline.
def getAllLoggerFilenames(logger):
""" Returns array of all log filenames attached to the logger. """
logFiles = [];
parent = logger.__dict__['parent'];
if parent.__class__.__name__ == 'RootLogger':
for h in logger.__dict__['handlers']:
if h.baseFilename:
logFiles.append(h.baseFilename);
else:
logFiles = getAllLoggerFilenames(parent);
return logFiles;
def logBlankLine(logger):
""" This utility method writes a blank line to the log. """
logNames = getAllLoggerFilenames(logger)
for fn in logNames:
with open(fn, 'a') as fh:
fh.write("\n")
Usage:
# We use YAML for logging config files, YMMV:
with open(logConfig, 'rt') as f:
logging.config.dictConfig(yaml.safe_load(f.read()))
logger = logging.getLogger("test.test")
logger.info("line 1")
logBlankLine(logger)
logger.info("line 2")
Output:
2019/12/22 16:33:59.152: INFO : test.test : line 1
2019/12/22 16:33:59.152: INFO : test.test : line 2
The easiest solution is to use f-strings if you are using Python 3:
logging.info( f'hello\n' )
You can try the following solution. It's simple and straightforward.
logging.debug("\b" * 20) # output blank line using escape character
logging.debug("debug message")