I created a custom log level as below:
class MyLoggerAdapter(logging.LoggerAdapter):
def __init__(self, logger, extra):
logging.LoggerAdapter.__init__(self, logger, extra)
self.logger = logger
logging.addLevelName(25, "NOTICE")
def notice(self, msg, *args, **kwargs):
self.logger.log(25, msg, *args, **kwargs)
The issue I have is that my logger format uses %(pathname)s and %(lineno)d, and whenever I call logger.notice() those values refer to the line within the definition of the notice method, rather than the location that the method was called from. How do I add the notice log level but print the correct location of the message to the logs?
Related
I have two logging classes, that should print logs on two different files.
First class
import logging
class firstLog:
def __init__(self):
self.logger = logging.getLogger("my_logger1")
logging.basicConfig(format="%(asctime)s.%(msecs)06d:\n%(message)s\n", level=logging.DEBUG,
datefmt="%d/%m/%Y %H:%M:%S",
filename="filename1.log")
def printlog(self, msg, *args, **kwargs):
self.logger.debug(msg, *args, **kwargs)
Second class
import logging
class secondLog:
def __init__(self):
self.logger = logging.getLogger("my_logger2")
logging.basicConfig(format="%(asctime)s.%(msecs)06d:\n%(message)s\n", level=logging.DEBUG,
datefmt="%d/%m/%Y %H:%M:%S",
filename="filename2.log")
def printlog(self, msg, *args, **kwargs):
self.logger.debug(msg, *args, **kwargs)
I instantiate the two classes, but only filename1.log is written
log1 = firstLog()
log2 = secondLog()
log1.printlog("text1")
log2.printlog("text2")
What I obtain is that a new file, called filename1.log is created, and contains
04/06/2021 08:57:04.000988:
text1
04/06/2021 08:57:04.000990:
text2
but I need two separate files
logging.basicConfig() is only applied once.
If logging has been configured already, basicConfig() will do nothing unless force is set. (source)
In your case you should get separate loggers with logging.getLogger(name) and different names. Then configure the loggers manually.
logging.basicConfig() is meant to be run once at your app's startup time, and will configure the root logger.
You will need to configure the two other loggers manually if you need their output to go elsewhere (or maybe configure a "routing" handler on the root logger).
Recently I came across logging in python.
I have the following code in test.py file
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())
logger.debug("test Message")
Now, is there any way I can print the resulting Logrecord object generated by logger.debug("test Message") because it's stated in the documentation that
LogRecord instances are created automatically by the Logger every time something is logged
https://docs.python.org/3/library/logging.html#logrecord-objects
I checked saving debug into a variable and print it
test = logger.debug("test Message")
print(test)
the output is NONE
My goal is to check/view the final Logrecord object generated by logging.debug(test.py) in the same test.py by using print() This is for my own understanding.
print(LogrecordObject.__dict__)
So how to get hold of the Logrecord object generated by logger.debug("test Message")
There is no return in debug()
# Here is the snippet for the source code
def debug(self, msg, *args, **kwargs):
if self.isEnabledFor(DEBUG):
self._log(DEBUG, msg, args, **kwargs)
If you wanna get LogRecord return, you need to redefine a debug(), you can overwrite like this:
import logging
DEBUG_LEVELV_NUM = 9
logging.addLevelName(DEBUG_LEVELV_NUM, "MY_DEBUG")
def _log(self, level, msg, args, exc_info=None, extra=None, stack_info=False):
sinfo = None
fn, lno, func = "(unknown file)", 0, "(unknown function)"
if exc_info:
if isinstance(exc_info, BaseException):
exc_info = (type(exc_info), exc_info, exc_info.__traceback__)
elif not isinstance(exc_info, tuple):
exc_info = sys.exc_info()
record = self.makeRecord(self.name, level, fn, lno, msg, args,
exc_info, func, extra, sinfo)
self.handle(record)
return record
def my_debug(self, message, *args, **kws):
if self.isEnabledFor(DEBUG_LEVELV_NUM):
# Yes, logger takes its '*args' as 'args'.
record = self._log(DEBUG_LEVELV_NUM, message, args, **kws)
return record
logger = logging.getLogger(__name__)
logging.Logger.my_debug = my_debug
logging.Logger._log = _log
logger.setLevel(DEBUG_LEVELV_NUM)
logger.addHandler(logging.StreamHandler())
test = logger.my_debug('test custom debug')
print(test)
Reference:
How to add a custom loglevel to Python's logging facility
You can create a handler that instead of formatting the LogRecord instance to a string, just save it in a list to be viewed and inspected later:
import logging
import sys
# A new handler to store "raw" LogRecords instances
class RecordsListHandler(logging.Handler):
"""
A handler class which stores LogRecord entries in a list
"""
def __init__(self, records_list):
"""
Initiate the handler
:param records_list: a list to store the LogRecords entries
"""
self.records_list = records_list
super().__init__()
def emit(self, record):
self.records_list.append(record)
# A list to store the "raw" LogRecord instances
logs_list = []
# Your logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# Add the regular stream handler to print logs to the console, if you like
logger.addHandler(logging.StreamHandler(sys.stdout))
# Add the RecordsListHandler to store the log records objects
logger.addHandler(RecordsListHandler(logs_list))
if __name__ == '__main__':
logger.debug("test Message")
print(logs_list)
Output:
test Message
[<LogRecord: __main__, 10, C:/Automation/Exercises/222.py, 36, "test Message">]
Suppose I needed to systematically add some extra information for a logger, and that this logger would then be used in some classes.
I would use a pattern that would look a bit like this :
import logging
logger = logging.getLogger(__name__)
format_ = '%(asctime)s %(levelname)s [%(name)s: %(lineno)s] -- %(message)s'
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(format_))
logger.addHandler(handler)
logger.setLevel(logging.INFO)
class A(object):
def log_thing(self, msg, *args, **kwargs):
logger.info(msg, *args, **kwargs)
def do_something(self):
# Some code here
self.log_thing('Everything is ok')
if __name__ == "__main__":
a = A()
a.do_something()
However, if you run this code, you'll notice that the "lineno" will be displayed as the line where the actual code to "logger.info" is done (line 13).
Is there a simple way to prevent this behaviour, and print the linenumber of the code that called "log_thing" (the one at line 17) ? Or are "shortcut to logging" functions doomed to behave like this ?
I'm trying to create a custom logger this way:
File: logger.py
import logging
class Logger(logging.getLoggerClass()):
def warning(self, msg, *args, **kwargs):
super(Logger, self).warning(msg, *args, **kwargs)
logging.setLoggerClass(Logger)
log = logging.getLogger('test')
handler = logging.StreamHandler()
formatter = logging.Formatter('%(pathname)s')
handler.setFormatter(formatter)
log.addHandler(handler)
File: test.py
from logger import log
log.warning('')
Output:
$ python test.py
/home/dario/Desktop/logging_test/logger.py
The expected output would be:
/home/dario/Desktop/logging_test/test.py
What's even weirder, if I comment the setLoggerClass line I do get test.py, but without the full path.
What am I doing wrong? Thank you!
Tested with Python 2.7.4 and 3.3.1 on Arch Linux x86_64.
From the Python Documentation - Logging - LogRecord attributes:
%(pathname)s - Full pathname of the source file where the logging call was issued (if available).
I.e. where a logging object, (or any subclass, like Logger) was initialized.
In this case logging and your instance of it, log, is both defined and initialized in the same file test.py.
Maybe it's a bit clearer if I split up the files:
logger.py: (the class)
import logging
# This is the path `logging` gets
class Logger(logging.getLoggerClass()):
def warning(self, msg, *args, **kwargs):
super(Logger, self).warning(msg, *args, **kwargs)
log.py: (a log module which has a log object of the type Logger)
import logging
from logger import Logger
logging.setLoggerClass(Logger)
log = logging.getLogger('test')
handler = logging.StreamHandler()
# Not this file's path, but where class Logger
# creates a logging object with super()
formatter = logging.Formatter('%(pathname)s')
handler.setFormatter(formatter)
log.addHandler(handler)
test.py: (main)
from log import log
log.warning('')
I am trying to use the python logging module to create a custom log file that records other information like host name and adds it to my DB. Below are the classes I created to do this, and the Handler part was working just fine, but now that I added a custom LogRecord class, it throws this error:
/src/lib/__init__.py", line 31, in __init__
logging.LogRecord.__init__(self, *args, **kwargs)
exceptions.TypeError: __init__() takes at most 9 arguments (10 given)
And here is how I execute it
logging.setLoggerClass(MyLogger)
log = logging.getLogger('testing')
log.addHandler(MyLogHandler())
d = {'host': '192.168.0.1'}
log.warn('Hi', d)
And here are the classes. It obviously has to do with the *args, **kwargs, but when I look at it, the *args is empty, and **kwargs only contains the d variable specified above. I don't understand the problem.
class MyLogRecord(logging.LogRecord):
def __init__(self, *args, **kwargs):
logging.LogRecord.__init__(self, *args, **kwargs) //THIS IS THE LINE IT DIES ON
self.host = 'localhost'
class MyLogFormatter(logging.Formatter):
def __init__(self, fmt, datefmt=None, host=None):
logging.Formatter.__init__(self, fmt, datefmt)
self.host = host
def format(self, record):
return logging.Formatter.format(record)
class MyLogger(logging.getLoggerClass()):
def makeRecord(self, *args, **kwargs):
return MyLogRecord(*args, **kwargs)
class MyLogHandler(logging.Handler): # Inherit from logging.Handler
def __init__(self):
# run the regular Handler __init__
logging.Handler.__init__(self)
# Our custom argument
self.mongo = MongoLogger()
def setupCustomLogger(self, name, this_host):
formatter = MyLogFormatter(fmt='%(asctime)s - %(levelname)s - %(module)s - %(message)s - %(host)s')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger = logging.getLogger(name)
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)
return logger
def emit(self, record):
# record.message is the log message
self.mongo.log(record)
class MongoLogger(object):
'''Logs messages to a MongoDB fh_admin log collection.'''
def log(self, message):
##todo write log to DB
print message
The error is telling you exactly what's wrong; you are calling the constructor with too many arguments. To see what I mean, take a look at how log-records are ordinarily constructed in the default implementation of makeRecord:
def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
"""
A factory method which can be overridden in subclasses to create
specialized LogRecords.
"""
rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
if extra is not None:
for key in extra:
if (key in ["message", "asctime"]) or (key in rv.__dict__):
raise KeyError("Attempt to overwrite %r in LogRecord" % key)
rv.__dict__[key] = extra[key]
return rv
Notice how makeRecord takes an extra param that it doesn't pass directly to LogRecord? You, on the other hand, are passing that directly to LogRecord.__init__, which is causing the error.
From here, you've got two options; you could provide a more complete implementation of makeRecord, or you could try using the LoggerAdapter class which should help you achieve the same goal with less code.
Here's an example:
# Common log info to be added to all logs reported with `log_adapter`
context = {'host': 'localhost'}
log = logging.getLogger('testing')
log.addHandler(logging.StreamHandler())
d = {'host': '192.168.0.1'}
log_adapter = logging.LoggerAdapter(log, context)
log_adapter.warning('Hi', d)
If you need to calculate the value of 'host' (for example) each time something is logged, you could make context an instance of a class that looks like a dictionary. Like so:
class LogContext(object):
def __getitem__(self, key):
if key == 'host':
return 'localhost'
raise KeyError(key)
def __iter__(self):
return iter(['host'])
log_adapter = logging.LoggerAdapter(log, LogContext())
log_adapter.warning('Hi', d)
One thing to note about LoggingAdapter, it apparently doesn't define all of the handy shortcut functions as the ordinary Logger class. That's why I've called the warning method instead of warn as you did above.
More info on LoggingAdapter and adding context to your logs can be found in the python docs.
NOTE - I didn't include MyLogHandler, MyLogFormatter, or MongoLogger in my examples as they were not relevant to the issue/error.