I want to save all following Exceptions in a file. The reason why I need this is because the IDLE for python 3.1.1 in Ubuntu raises an Exception at calltipps, but close to fast, that it isn't readble. Also I need this for testing. The best, would be if I just call a function which saves all Exception to a file. Thank you! ;)
// edit:
i had looked first for a more general way! so that you do not have to place your whole code in a function or indentation. but now that worked wery well for me. although I would be still grateful, if you find a way!
thanks!
If you have a convenient main() function (whatever it's called), then you can use the logging module:
import logging
def main():
raise Exception("Hey!")
logging.basicConfig(level=logging.DEBUG, filename='/tmp/myapp.log')
try:
main()
except:
logging.exception("Oops:")
logging.exception conveniently gets the current exception and puts the details in the log:
ERROR:root:Oops:
Traceback (most recent call last):
File "C:\foo\foo.py", line 9, in <module>
main()
File "C:\foo\foo.py", line 4, in main
raise Exception("Hey!")
Exception: Hey!
Related
Background
Consider the following minimal example:
When I save the following script and run it from terminal,
import time
time.sleep(5)
raise Exception
the code will raise an error after sleeping five seconds, leaving the following traceback.
Traceback (most recent call last):
File "test/minimal_error.py", line 4, in <module>
raise Exception
Exception
Now, say, I run the script, and during the 5-second-sleep, I add a line in the middle.
import time
time.sleep(5)
a = 1
raise Exception
After the python interpreter wakes up from the sleep and reaches the next line, raise Exception, it will raise the error, but it leaves the following traceback.
Traceback (most recent call last):
File "test/minimal_error.py", line 4, in <module>
a = 1
Exception
So the obvious problem is that it doesn't print the actual code that caused the error. Although it gives the correct line number (correctly reflecting the version of the script that is running, while understandably useless) and a proper error message, I can't really know what piece of code actually caused the error.
In real practice, I implement one part of a program, run it to see if that part is doing fine, and while it is still running, I move on to the next thing I have to implement. And when the script throws an error, I have to find which actual line of code caused the error. I usually just read the error message and try to deduce the original code that caused it. Sometimes it isn't easy to guess, so I copy the script to clipboard and rollback the code by undoing what I've written after running the script, check the line that caused error, and paste back from clipboard.
Question
Is there any understandable reason why the interpreter shows a = 1, which is line 4 of the "current" version of the code, instead of raise Exception, which is line 4 of the "running" version of the code? If the interpreter knows "line 4" caused the error and the error message is "Exception", why can't it say the command raise Exception raised it?
I'm not really sure if this question is on-topic here, but I don't think I can conclude it off-topic from what the help center says. It is about "[a] software [tool] commonly used by programmers" (the Python interpreter) and is "a practical, answerable problem that is unique to software development," I think. I don't think it's opinion-based, because there should be a reason for this choice of implementation.
(Observed the same in Python 2.7.16, 3.6.8, 3.7.2, and 3.7.3, so it doesn't seem to be version-specific, but a thing that just happens in Python.)
The immediate reason is that Python re-opens the file and reads the specified line again to print it in error messages. So why would it need to do that when it already read the file in the beginning? Because it doesn't keep the source code in memory, only the generated byte code.
In fact, Python will never hold the entire contents of the source file in memory at one time. Instead the lexer will read from the file and produce one token at a time, which the parser then parses and turns into byte code. Once the parser is done with a token, it's gone.
So the only way to get back at the original source code is to open the source file again.
I think it a classic problem which is described here.
Sleep use os system call to pause execution of that thread.
In my experience programming with Java, I have become quite fond of the stack traces it generates when my code goes awry, but I feel that the traces generated by python are a bit lacking by comparison. For example, a trace in java might look like this:
java.lang.RuntimeException
at test.package.Example.c(Example.java:20)
at test.package.Example.b(Example.java:15)
at test.package.Example.a(Example.java:10)
Whereas a python trace might look like this:
Traceback (most recent call last):
File "example.py", line 10, in <module>
a()
File "example.py", line 2, in a
b()
File "example.py", line 5, in b
c()
File "example.py", line 8, in c
raise Exception
Exception
While both of these traces convey basically the same information, I personally find that the trace from java is easier to follow.
Is there a means to change the format python uses for printing its stack traces, or would that sort of change require me to create a custom exception handler at the root of my program?
using traceback module
import traceback
try:
x= 1/0
except Exception as e:
print(e)
traceback.print_exc()
There is a means to change the format Python uses to format its stack traces, and that is that you write your own formatter instead. There is only one built-in format.
You can assign your own function to sys.excepthook and it will act as a top-level exception handler that will get access to exceptions that were about to rise uncaught and cause the program to exit. There you can make use of the traceback object to format things however you like. Triptych's answer shows how to use the traceback module to get the info for each stack frame. extract_tb returns a 4-tuple of the filename, line number, function, and source text of the offending line, so if you want to not display the source text you could just throw that away and concatenate the rest. But you'll have to do the work of constructing whatever output you want to see.
If you really want to, you can reformat exception tracebacks with the traceback.extract_tb method.
ref: https://docs.python.org/2/library/traceback.html#traceback.extract_tb
Stackoverflow posts helped me a lot with Python, however I stuck on this one. I cannot figure out how to simply skip module if it has exceptions within it. Also, if it happens I like to alter one variable so rest of code would know to skip functionality related with that module.
I have main.py which is loading my module sms.py. Here you also see my attempt which does not work:
try:
import sms
except ImportError:
print "Ok, lets skip that module"
dont_use_sms = 1
Part of sms.py which causes exception looks following:
import gammu
sm = gammu.StateMachine()
sm.ReadConfig()
try:
sm.Init() # this one to be exact
except:
raise
when I run this I get following:
Traceback (most recent call last):
File "./main.py", line 10, in <module>
import sms
File "/path/to/sms.py", line 7, in <module>
sm.Init()
gammu.ERR_DEVICENOTEXIST: {'Text': u"Error opening device, it doesn't exist.", 'Code': 4, 'Where': 'Init'}
I have tried to alter exception by putting gammu.ERR_DEVICENOTEXIST as argument, however it didn't help.
I feel that that exception should be somehow handled by sms.py and properly forwarded to main.py, but cannot figure out how.
By the way, I know what causes gammu.ERR_DEVICENOTEXIST and that is not a problem. Question is about how to continue with rest of program if it appears.
That you for suggestions.
You can also change your main.py.
Instead of:
except ImportError:
you can say:
except:
And then it should continue.
PS: Naked except statements are not good style
I have control over what goes into /usr/bin/python and can replace it with my script which calls python underneath. However, I do not have control on the programs that are written (cannot mandate a particular convention etc.)
In such a situation, what would be the best way to have python dump the stacktrace into a database in addition to displaying it on stdout? (Have your own script?)
Update:
Clarification: I meant a stacktrace that a program generates upon error:
l = [1,2,3]
l[4]
Traceback (most recent call last):
File "", line 1, in
IndexError: list index out of range
Solution: I think AKX's solution below works in redirecting stderr to a script which dumps the stacktrace into a NoSQL store. Thanks!
You can define default exception handler:
import sys
import traceback
def my_handler(typ, value, tb):
error_str = traceback.format_exception(typ, value, tb)
print 'Here you can write exception to DB: ', error_str
sys.excepthook = my_handler
print 1 / 0 # here you can execute third party code via execfile/import
Look into documentation about exception handling and generating tracebacks.
I'm working on a Python library used by third-party developers to write extensions for our core application.
I'd like to know if it's possible to modify the traceback when raising exceptions, so the last stack frame is the call to the library function in the developer's code, rather than the line in the library that raised the exception. There are also a few frames at the bottom of the stack containing references to functions used when first loading the code that I'd ideally like to remove too.
Thanks in advance for any advice!
You can remove the top of the traceback easily with by raising with the tb_next element of the traceback:
except:
ei = sys.exc_info()
raise ei[0], ei[1], ei[2].tb_next
tb_next is a read_only attribute, so I don't know of a way to remove stuff from the bottom. You might be able to screw with the properties mechanism to allow access to the property, but I don't know how to do that.
Take a look at what jinja2 does here:
https://github.com/mitsuhiko/jinja2/blob/5b498453b5898257b2287f14ef6c363799f1405a/jinja2/debug.py
It's ugly, but it seems to do what you need done. I won't copy-paste the example here because it's long.
Starting with Python 3.7, you can instantiate a new traceback object and use the .with_traceback() method when throwing. Here's some demo code using either sys._getframe(1) (or a more robust alternative) that raises an AssertionError while making your debugger believe the error occurred in myassert(False): sys._getframe(1) omits the top stack frame.
What I should add is that while this looks fine in the debugger, the console behavior unveils what this is really doing:
Traceback (most recent call last):
File ".\test.py", line 35, in <module>
myassert_false()
File ".\test.py", line 31, in myassert_false
myassert(False)
File ".\test.py", line 26, in myassert
raise AssertionError().with_traceback(back_tb)
File ".\test.py", line 31, in myassert_false
myassert(False)
AssertionError
Rather than removing the top of the stack, I have added a duplicate of the second-to-last frame.
Anyway, I focus on how the debugger behaves, and it seems this one works correctly:
"""Modify traceback on exception.
See also https://github.com/python/cpython/commit/e46a8a
"""
import sys
import types
def myassert(condition):
"""Throw AssertionError with modified traceback if condition is False."""
if condition:
return
# This function ... is not guaranteed to exist in all implementations of Python.
# https://docs.python.org/3/library/sys.html#sys._getframe
# back_frame = sys._getframe(1)
try:
raise AssertionError
except AssertionError:
traceback = sys.exc_info()[2]
back_frame = traceback.tb_frame.f_back
back_tb = types.TracebackType(tb_next=None,
tb_frame=back_frame,
tb_lasti=back_frame.f_lasti,
tb_lineno=back_frame.f_lineno)
raise AssertionError().with_traceback(back_tb)
def myassert_false():
"""Test myassert(). Debugger should point at the next line."""
myassert(False)
if __name__ == "__main__":
myassert_false()
You might also be interested in PEP-3134, which is implemented in python 3 and allows you to tack one exception/traceback onto an upstream exception.
This isn't quite the same thing as modifying the traceback, but it would probably be the ideal way to convey the "short version" to library users while still having the "long version" available.
What about not changing the traceback? The two things you request can both be done more easily in a different way.
If the exception from the library is caught in the developer's code and a new exception is raised instead, the original traceback will of course be tossed. This is how exceptions are generally handled... if you just allow the original exception to be raised but you munge it to remove all the "upper" frames, the actual exception won't make sense since the last line in the traceback would not itself be capable of raising the exception.
To strip out the last few frames, you can request that your tracebacks be shortened... things like traceback.print_exception() take a "limit" parameter which you could use to skip the last few entries.
That said, it should be quite possible to munge the tracebacks if you really need to... but where would you do it? If in some wrapper code at the very top level, then you could simply grab the traceback, take a slice to remove the parts you don't want, and then use functions in the "traceback" module to format/print as desired.
For python3, here's my answer. Please read the comments for an explanation:
def pop_exception_traceback(exception,n=1):
#Takes an exception, mutates it, then returns it
#Often when writing my repl, tracebacks will contain an annoying level of function calls (including the 'exec' that ran the code)
#This function pops 'n' levels off of the stack trace generated by exception
#For example, if print_stack_trace(exception) originally printed:
# Traceback (most recent call last):
# File "<string>", line 2, in <module>
# File "<string>", line 2, in f
# File "<string>", line 2, in g
# File "<string>", line 2, in h
# File "<string>", line 2, in j
# File "<string>", line 2, in k
#Then print_stack_trace(pop_exception_traceback(exception),3) would print:
# File "<string>", line 2, in <module>
# File "<string>", line 2, in j
# File "<string>", line 2, in k
#(It popped the first 3 levels, aka f g and h off the traceback)
for _ in range(n):
exception.__traceback__=exception.__traceback__.tb_next
return exception
This code might be of interest for you.
It takes a traceback and removes the first file, which should not be shown. Then it simulates the Python behavior:
Traceback (most recent call last):
will only be shown if the traceback contains more than one file.
This looks exactly as if my extra frame was not there.
Here my code, assuming there is a string text:
try:
exec(text)
except:
# we want to format the exception as if no frame was on top.
exp, val, tb = sys.exc_info()
listing = traceback.format_exception(exp, val, tb)
# remove the entry for the first frame
del listing[1]
files = [line for line in listing if line.startswith(" File")]
if len(files) == 1:
# only one file, remove the header.
del listing[0]
print("".join(listing), file=sys.stderr)
sys.exit(1)