I have logging function as follows.
logging.basicConfig(
filename = fileName,
format = "%(levelname) -10s %(asctime)s %(message)s",
level = logging.DEBUG
)
def printinfo(string):
if DEBUG:
logging.info(string)
def printerror(string):
if DEBUG:
logging.error(string)
print string
I need to login the line number, stack information. For example:
1: def hello():
2: goodbye()
3:
4: def goodbye():
5: printinfo()
---> Line 5: goodbye()/hello()
How can I do this with Python?
SOLVED
def printinfo(string):
if DEBUG:
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
logging.debug(stack_trace[:-1])
if LOG:
logging.info(string)
gives me this info which is exactly what I need.
DEBUG 2011-02-23 10:09:13,500 [
' File "/abc.py", line 553, in <module>\n runUnitTest(COVERAGE, PROFILE)\n',
' File "/abc.py", line 411, in runUnitTest\n printinfo(string)\n']
Current function name, module and line number you can do simply by changing your format string to include them.
logging.basicConfig(
filename = fileName,
format = "%(levelname) -10s %(asctime)s %(module)s:%(lineno)s %(funcName)s %(message)s",
level = logging.DEBUG
)
Most people only want the stack when logging an exception, and the logging module does that automatically if you call logging.exception(). If you really want stack information at other times then you will need to use the traceback module for extract the additional information you need.
import inspect
import traceback
def method():
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
print ''.join(stack_trace)
Use stack_trace[:-1] to avoid including method/printinfo in the stack trace.
As of Python 3.2, this can be simplified to passing the stack_info=True flag to the logging calls. However, you'll need to use one of the above answers for any earlier version.
Late answer, but oh well.
Another solution is that you can create your own formatter with a filter as specified in the docs here. This is a really great feature as you now no longer have to use a helper function (and have to put the helper function everywhere you want the stack trace). Instead, a custom formatted implements it directly into the logs themselves.
import logging
class ContextFilter(logging.Filter):
def __init__(self, trim_amount)
self.trim_amount = trim_amount
def filter(self, record):
import traceback
record.stack = ''.join(
str(row) for row in traceback.format_stack()[:-self.trim_amount]
)
return True
# Now you can create the logger and apply the filter.
logger = logging.getLogger(__name__)
logger.addFilter(ContextFilter(5))
# And then you can directly implement a stack trace in the formatter.
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s \n %(stack)s')
Note: In the above code I trim the last 5 stack frames. This is just for convenience and so that we don't show stack frames from the python logging package itself.(It also might have to be adjusted for different versions of the logging package)
Use the traceback module.
logging.error(traceback.format_exc())
Here is an example that i hope it can help you:
import inspect
import logging
logging.basicConfig(
format = "%(levelname) -10s %(asctime)s %(message)s",
level = logging.DEBUG
)
def test():
caller_list = []
frame = inspect.currentframe()
this_frame = frame # Save current frame.
while frame.f_back:
caller_list.append('{0}()'.format(frame.f_code.co_name))
frame = frame.f_back
caller_line = this_frame.f_back.f_lineno
callers = '/'.join(reversed(caller_list))
logging.info('Line {0} : {1}'.format(caller_line, callers))
def foo():
test()
def bar():
foo()
bar()
Result:
INFO 2011-02-23 17:03:26,426 Line 28 : bar()/foo()/test()
Look at traceback module
>>> import traceback
>>> def test():
>>> print "/".join( str(x[2]) for x in traceback.extract_stack() )
>>> def main():
>>> test()
>>> main()
<module>/launch_new_instance/mainloop/mainloop/interact/push/runsource/runcode/<module>/main/test
This is based on #mouad's answer but made more useful (IMO) by including at each level the filename (but not its full path) and line number of the call stack, and by leaving the stack in most-recently-called-from (i.e. NOT reversed) order because that's the way I want to read it :-)
Each entry has file:line:func() which is the same sequence as the normal stacktrace, but all on the same line so much more compact.
import inspect
def callers(self):
caller_list = []
frame = inspect.currentframe()
while frame.f_back:
caller_list.append('{2}:{1}:{0}()'.format(frame.f_code.co_name,frame.f_lineno,frame.f_code.co_filename.split("\\")[-1]))
frame = frame.f_back
callers = ' <= '.join(caller_list)
return callers
You may need to add an extra f_back if you have any intervening calls to produce the log text.
frame = inspect.currentframe().f_back
Produces output like this:
file2.py:620:func1() <= file3.py:211:func2() <= file3.py:201:func3() <= main.py:795:func4() <= file4.py:295:run() <= main.py:881:main()
I only need this stacktrace in two key functions, so I add the output of callers into the text in the logger.debug() call, like htis:
logger.debug("\nWIRE: justdoit request -----\n"+callers()+"\n\n")
Related
I created a custom logging.Logger that is used by several different objects in a script I'm running like so:
class TestLogger(logging.Logger):
def __init__(self, name, file=None):
super(TestLogger, self).__init__(name, level=logging.DEBUG)
self.log_file = file
...
def addLogFile(self, log_file):
self.log_file = log_file
self.setFormat()
# set the format of the log
def setFormat(self, default=True, end='\n'):
# remove any Handlers
self.removeStreamHandlers()
self.removeFileHandlers()
# get the log format string, default or message
format_str = DEFAULT_FORMAT if default else CUSTOM_FORMAT
std_formatter = logging.Formatter(format_str, datefmt=self.DATE_FORMAT)
# add the stream handler
console = logging.StreamHandler(sys.stdout)
console.setFormatter(std_formatter)
console.terminator = end
self.addHandler(console)
# add the file handler
if self.log_file:
file_formatter = logging.Formatter(format_str, datefmt=self.DATE_FORMAT)
logger = logging.FileHandler(self.log_file)
logger.setFormatter(file_formatter)
self.addHandler(logger)
# remove all stream handlers
def removeStreamHandlers(self):
stream_handlers = [h for h in self.handlers if isinstance(h, logging.StreamHandler)
and not isinstance(h, logging.FileHandler)]
for sh in stream_handlers:
self.removeHandler(sh)
# remove all file handlers
def removeFileHandlers(self):
file_handlers = [h for h in self.handlers if isinstance(h, logging.StreamHandler)
and isinstance(h, logging.FileHandler)]
for fh in file_handlers:
self.removeHandler(fh)
class Something:
def __init__(self):
self.log = TestLogger('Something')
...
def __del__(self):
self.log.removeFileHandlers()
self.log.removeStreamHandler()
class SomethingElse:
def __init__(self):
self.log = TestLogger('SomethingElse')
...
def __del__(self):
self.log.removeFileHandlers()
self.log.removeStreamHandler()
All of these objects are initialized and designed to share the same log file like so:
log_file = 'test.log'
s = Something()
se = SomethingElse()
s.addLogFile(log_file)
se.addLogFile(log_file)
...
del s, se
The problem seems to be that when I try to rerun my program, it throws a ResourceWarning every time I run setFormat(). It seems like the file isn't properly being closed and I'm not sure where this could be happening.
First of all, you should reuse the handlers. If you want to change the formatter just call setFormatter on the existing handlers and keep them.
If you really do want to throw away the handler and use a new one, there is a close() method on the FileHandler that is supposed to be called to clean up when the handler is done logging. So in your case you would change your code to look like this:
for fh in file_handlers:
fh.close()
self.removeHandler(fh)
I'm using a logging filter to print out my log messages including some custom fields that are not present in the usual logging framework.
For instance:
class NISARLogger(object):
def __init__(self, filename):
self.filename = filename
fid = logging.FileHandler(filename)
formatter_str = '%(asctime)s, %(levelname)s, %(pge)s, %(module)s, %(error_code)i, \
%(source)s:%(line_number)i, "%(error_name)s: %(message)s"'
formatter = logging.Formatter(formatter_str)
fid.setFormatter(formatter)
self.logger = logging.getLogger(name="NISAR")
self.logger.setLevel(logging.DEBUG)
self.logger.addHandler(fid)
def log_message(self, class_filter, message):
xfilter = class_filter()
log_funct = getattr(self.logger, xfilter.level)
self.logger.addFilter(xfilter)
log_funct(message)
def close(self):
logging.shutdown()
Everything seems to be working fine except my log looks like this:
2020-08-18 14:41:07,431, INFO, QA, misc, 100000, '../verify_rslc.py':70, "N/A: Opening file L_JOINT_00213_LINE12_RUN1_FP_12122019134617.h5 with xml spec /Users/cmoroney/Desktop/working/NISAR/src/GitHub/QualityAssurance/xml/nisar_L1_SLC.xml"
2020-08-18 14:41:07,432, INFO, QA, misc, 100000, '/Users/cmoroney/Desktop/working/NISAR/src/GitHub/QualityAssurance/quality/SLCFile.py':28, "N/A: Opening file L_JOINT_00213_LINE12_RUN1_FP_12122019134617.h5"
where there's a lot of padding between the '100000' (error code parameter) and the filename (source parameter) both of which are extra parameters passed into the logger via the 'addFilter' call. I've tried experimenting with the length of the 'source' and 'error_code' fields in the formatter_str variable but no luck. Any idea where that padding is coming from?
The extra space is coming from the whitespace in the source code itself at the start of the second line.
formatter_str = '%(asctime)s, %(levelname)s, %(pge)s, %(module)s, %(error_code)i, \
%(source)s:%(line_number)i, "%(error_name)s: %(message)s"'
Try this instead:
formatter_str = ('%(asctime)s, %(levelname)s, %(pge)s, %(module)s, %(error_code)i, '
'%(source)s:%(line_number)i, "%(error_name)s: %(message)s"')
I would like to collect info with the help of logging.
The idea is simple. I have hash_value of some data, which I want to write to log. So, I set up my logging this way:
import logging
logger.setLevel(logging.DEBUG)
logging.basicConfig(format='%(asctime)s :%(message)s', level=logging.INFO)
As you can see, now timing and some message will automatically write to log file, for example I can use it like this:
logger.info('Initial data: {}'.format(data))
But what if I want to write hash_value of my data automatically? Like it is happening with time now.
I looked through documentation and find nothing useful. There is no attribute for variable in module logging.
So I am forced to do it awry. Like this:
hash_value = hash(data)
logger.info('Initial data: {} {}'.format(hash_value, data))
I would expect from this code:
logging.basicConfig(format='%(asctime)s: %(variable)s :%(message)s', level=logging.INFO)
and
logger.info('Initial data: {}'.format(hash_value, data))
to do the job. But it does not work (and it should not basically) and I did not find the solution in documentation.
So, how to avoid this awry code:
logger.info('Initial data: {} {}'.format(hash_value, data))
which I am having now?
import logging
import sys
MY_PARAMS = ("variable1", "var2", )
class ExtraFilter(logging.Filter):
def filter(self, record):
# this one used for second, simplier handler
# to avoid duplicate of logging entries if "extra" keyword was passed.
# Check all of your custom params:
# if all of them are present - record should be filtered
# * all because if any of them is missing - there would be silent exception and record wont be logged at all
# bellow is just an example how to check.
# You can use something like this:
# if all(hasattr(record, param) for param in MY_PARAMS): return False
if hasattr(record, "variable1"):
return False
return True
# init logging
log = logging.getLogger()
# init handlers and formatters
h1 = logging.StreamHandler(sys.stdout)
f1 = logging.Formatter('%(asctime)s: %(variable1)s: %(var2)s: %(message)s')
h2 = logging.StreamHandler(sys.stdout)
f2 = logging.Formatter('%(asctime)s: %(message)s')
h1.setFormatter(f1)
h2.setFormatter(f2)
h2.addFilter(ExtraFilter())
log.addHandler(h1)
log.addHandler(h2)
# example of data:
extra = {"variable1": "test1", "var2": "test2"}
log.setLevel(logging.DEBUG)
log.debug("debug message", extra=extra)
log.info("info message")
The above code will produce following output:
2017-11-04 09:16:36,787: test1: test2: debug message
2017-11-04 09:16:36,787: info message
It is not awry code, you want to add two informations, therefore you must either pass two parameters to format or concatenate the string more "manually"
You could go with
Logging.info("initial data " + hash_value + " " + data)
Or you could change the "data" object so its "str" or the repr method adds the hash by itself (preferably the repr in this case)
Class Data():
....
def __repr__(self):
Return self.hash() + " " self.data
Which in this case will print the hash and the string version of the parameter data( or simply whatever you want to show as string) passing only one parameter in the string format.
Anyway, you could make the formating string prettier with....
Logging.info("Initial data {hash} {data}".format(hash=hash_value, data=data))
By the way, in C++ and Java you would also need to declare two "entries" for those two atributes. In java would be something like this:
LOGGING.info("Initial data {} {}", hash, data);
I am trying to build a script that sets breakpoints in winIDEA (this ide will run a project, that has different comments). The breakpoints must be set in winIDEA after a specific comment is recognised. I am somewhat new to this language and I am having problems making this script. I am not sure if what I have here is good, but I am trying to get the line where the comment is recognised, and then set a breakpoint in the program at this specific line.
import inspect
import logging
import isystem.connect as ic
import sys
connMgr = ic.ConnectionMgr()
connMgr.connectMRU('')
dbg = ic.CDebugFacade(connMgr)
bc = ic.CBreakpointController(connMgr)
exe = ic.CExecutionController(connMgr)
logging.basicConfig(
format = "%(levelname) -10s %(asctime)s %(message)s",
level = logging.DEBUG
def test():
caller_list = []
frame = inspect.currentframe()
this_frame = frame # Save current frame.
while frame.f_back:
print frame
caller_list.append('{0}()'.format(frame.f_code.co_name))
frame = frame.f_back
caller_line = this_frame.f_back.f_lineno
callers = '/'.join(reversed(caller_list))
logging.info('Line {0} : {1}'.format(caller_line, callers))
print caller_line
def foo():
test()
def bar():
foo()
test()
datafile= file(r'C:\Documents and Settings\stiral1\Desktop\function.py')
stringfile=datafile.read()
item_search="haha"
counter=0
print stringfile.find(item_search,counter)
while counter != -1:
print stringfile.find(item_search,counter)
bc.setBP(counter,r'C:\_DevTools\winIDEA\2012\Examples\Simulator\PPC\Simple\main.c')
I get instead a random line, and the position where I encounter my element in the string (I make the script that runs in ide a string at some point). I have no idea left... Help a newbie!
This is what worked up for me:
def wdSetBPComment(wdPrmStr):
assert(wdGetNrOfSubStrings(wdPrmStr) == 2)
comment = wdGetSubString_1(wdPrmStr)
function = wdGetSubString_2(wdPrmStr)
gl_tpLocation.setResourceName(function)
gl_tpLocation.setSearch(ic.E_TRUE)
gl_tpLocation.setMatchingType(ic.CTestLocation.E_MATCH_PLAIN)
gl_tpLocation.setSearchPattern(comment)
lineLocation = addrCtrl.getSourceLocation(gl_tpLocation)
wdSetBreakpoint(lineLocation.getFileName()+lineLocation.getLineNumber())
gl_bcCtrl.setBP(lineNo,fileName)
return (gl_wdOkStr)
I'm writing python package/module and would like the logging messages mention what module/class/function they come from. I.e. if I run this code:
import mymodule.utils.worker as worker
w = worker.Worker()
w.run()
I'd like to logging messages looks like this:
2010-06-07 15:15:29 INFO mymodule.utils.worker.Worker.run <pid/threadid>: Hello from worker
How can I accomplish this?
Thanks.
I tend to use the logging module in my packages/modules like so:
import logging
log = logging.getLogger(__name__)
log.info("Whatever your info message.")
This sets the name of your logger to the name of the module for inclusion in the log message. You can control where the name is by where %(name)s is found in the format string. Similarly you can place the pid with %(process)d and the thread id with %(thread)d. See the docs for all the options.
Formatting example:
import logging
logging.basicConfig(format="%(asctime)s %(levelname)s %(name)s %(process)d/%(threadName)s: %(message)s")
logging.getLogger('this.is.the.module').warning('Testing for SO')
Gives me:
2010-06-07 08:43:10,494 WARNING this.is.the.module 14980/MainThread: Testing for SO
Here is my solution that came out of this discussion. Thanks to everyone for suggestions.
Usage:
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from hierlogger import hierlogger as logger
>>> def main():
... logger().debug("test")
...
>>> main()
DEBUG:main:test
By default it will name logger as ... You can also control the depth by providing parameter:
3 - module.class.method default
2 - module.class
1 - module only
Logger instances are also cached to prevent calculating logger name on each call.
I hope someone will enjoy it.
The code:
import logging
import inspect
class NullHandler(logging.Handler):
def emit(self, record): pass
def hierlogger(level=3):
callerFrame = inspect.stack()[1]
caller = callerFrame[0]
lname = '__heirlogger'+str(level)+'__'
if lname not in caller.f_locals:
loggerName = str()
if level >= 1:
try:
loggerName += inspect.getmodule(inspect.stack()[1][0]).__name__
except: pass
if 'self' in caller.f_locals and (level >= 2):
loggerName += ('.' if len(loggerName) > 0 else '') +
caller.f_locals['self'].__class__.__name__
if callerFrame[3] != '' and level >= 3:
loggerName += ('.' if len(loggerName) > 0 else '') + callerFrame[3]
caller.f_locals[lname] = logging.getLogger(loggerName)
caller.f_locals[lname].addHandler(NullHandler())
return caller.f_locals[lname]