I'm working on a Python library used by third-party developers to write extensions for our core application.
I'd like to know if it's possible to modify the traceback when raising exceptions, so the last stack frame is the call to the library function in the developer's code, rather than the line in the library that raised the exception. There are also a few frames at the bottom of the stack containing references to functions used when first loading the code that I'd ideally like to remove too.
Thanks in advance for any advice!
You can remove the top of the traceback easily with by raising with the tb_next element of the traceback:
except:
ei = sys.exc_info()
raise ei[0], ei[1], ei[2].tb_next
tb_next is a read_only attribute, so I don't know of a way to remove stuff from the bottom. You might be able to screw with the properties mechanism to allow access to the property, but I don't know how to do that.
Take a look at what jinja2 does here:
https://github.com/mitsuhiko/jinja2/blob/5b498453b5898257b2287f14ef6c363799f1405a/jinja2/debug.py
It's ugly, but it seems to do what you need done. I won't copy-paste the example here because it's long.
Starting with Python 3.7, you can instantiate a new traceback object and use the .with_traceback() method when throwing. Here's some demo code using either sys._getframe(1) (or a more robust alternative) that raises an AssertionError while making your debugger believe the error occurred in myassert(False): sys._getframe(1) omits the top stack frame.
What I should add is that while this looks fine in the debugger, the console behavior unveils what this is really doing:
Traceback (most recent call last):
File ".\test.py", line 35, in <module>
myassert_false()
File ".\test.py", line 31, in myassert_false
myassert(False)
File ".\test.py", line 26, in myassert
raise AssertionError().with_traceback(back_tb)
File ".\test.py", line 31, in myassert_false
myassert(False)
AssertionError
Rather than removing the top of the stack, I have added a duplicate of the second-to-last frame.
Anyway, I focus on how the debugger behaves, and it seems this one works correctly:
"""Modify traceback on exception.
See also https://github.com/python/cpython/commit/e46a8a
"""
import sys
import types
def myassert(condition):
"""Throw AssertionError with modified traceback if condition is False."""
if condition:
return
# This function ... is not guaranteed to exist in all implementations of Python.
# https://docs.python.org/3/library/sys.html#sys._getframe
# back_frame = sys._getframe(1)
try:
raise AssertionError
except AssertionError:
traceback = sys.exc_info()[2]
back_frame = traceback.tb_frame.f_back
back_tb = types.TracebackType(tb_next=None,
tb_frame=back_frame,
tb_lasti=back_frame.f_lasti,
tb_lineno=back_frame.f_lineno)
raise AssertionError().with_traceback(back_tb)
def myassert_false():
"""Test myassert(). Debugger should point at the next line."""
myassert(False)
if __name__ == "__main__":
myassert_false()
You might also be interested in PEP-3134, which is implemented in python 3 and allows you to tack one exception/traceback onto an upstream exception.
This isn't quite the same thing as modifying the traceback, but it would probably be the ideal way to convey the "short version" to library users while still having the "long version" available.
What about not changing the traceback? The two things you request can both be done more easily in a different way.
If the exception from the library is caught in the developer's code and a new exception is raised instead, the original traceback will of course be tossed. This is how exceptions are generally handled... if you just allow the original exception to be raised but you munge it to remove all the "upper" frames, the actual exception won't make sense since the last line in the traceback would not itself be capable of raising the exception.
To strip out the last few frames, you can request that your tracebacks be shortened... things like traceback.print_exception() take a "limit" parameter which you could use to skip the last few entries.
That said, it should be quite possible to munge the tracebacks if you really need to... but where would you do it? If in some wrapper code at the very top level, then you could simply grab the traceback, take a slice to remove the parts you don't want, and then use functions in the "traceback" module to format/print as desired.
For python3, here's my answer. Please read the comments for an explanation:
def pop_exception_traceback(exception,n=1):
#Takes an exception, mutates it, then returns it
#Often when writing my repl, tracebacks will contain an annoying level of function calls (including the 'exec' that ran the code)
#This function pops 'n' levels off of the stack trace generated by exception
#For example, if print_stack_trace(exception) originally printed:
# Traceback (most recent call last):
# File "<string>", line 2, in <module>
# File "<string>", line 2, in f
# File "<string>", line 2, in g
# File "<string>", line 2, in h
# File "<string>", line 2, in j
# File "<string>", line 2, in k
#Then print_stack_trace(pop_exception_traceback(exception),3) would print:
# File "<string>", line 2, in <module>
# File "<string>", line 2, in j
# File "<string>", line 2, in k
#(It popped the first 3 levels, aka f g and h off the traceback)
for _ in range(n):
exception.__traceback__=exception.__traceback__.tb_next
return exception
This code might be of interest for you.
It takes a traceback and removes the first file, which should not be shown. Then it simulates the Python behavior:
Traceback (most recent call last):
will only be shown if the traceback contains more than one file.
This looks exactly as if my extra frame was not there.
Here my code, assuming there is a string text:
try:
exec(text)
except:
# we want to format the exception as if no frame was on top.
exp, val, tb = sys.exc_info()
listing = traceback.format_exception(exp, val, tb)
# remove the entry for the first frame
del listing[1]
files = [line for line in listing if line.startswith(" File")]
if len(files) == 1:
# only one file, remove the header.
del listing[0]
print("".join(listing), file=sys.stderr)
sys.exit(1)
Related
I'm using PyRun_String() from Python C API to run python code.
Passing Py_file_input for start and for globals and locals I pass dictionary created with PyDict_New(), and for string of code str I pass my code.
For example I'm having next code:
def f():
def g():
assert False, 'TestExc'
g()
f()
Of cause this code is expected to throw an exception and show stack. I print error with PyErr_Print() and get next stack:
Traceback (most recent call last):
File "<string>", line 5, in <module>
File "<string>", line 4, in f
File "<string>", line 3, in g
AssertionError: TestExc
As one can see this exception stack is missing lines with code, for example if same script is run in pure Python interpreter then it prints next stack:
Traceback (most recent call last):
File "test.py", line 5, in <module>
f()
File "test.py", line 4, in f
g()
File "test.py", line 3, in g
assert False, 'TestExc'
AssertionError: TestExc
So it has code annotation e.g. assert False, 'TestExc' for last line of stack and f() and g() for previous lines. Also it has file name (but having file name is not very important).
Is there any way to have this code shown when using PyRun_String()? I expect that I need to use another function for example PyRun_StringFlags() which has extra parameters PyCompilerFlags * flags that I can probably use to tell compiler to save code attached to each line when compiling. But I don't see anywhere documentation for PyCompilerFlags and don't know what should I pass.
Also maybe there are other useful flags for PyCompilerFlags, for example would be nice to have test.py file name instead of <string> inside exception stack, probably this behaviour is also tweakable by some of PyCompilerFlags values?
Also I was using exec() from built-ins of Python, passing string with program's code to it. But got same exception stack without code annotation. Seems that if it is some interpreter-wide param whether to save code annotation or not.
I also tried to write special function to get current stack, using standard traceback and sys modules:
def GetStack():
import traceback, sys
frame = sys._getframe()
extracted = traceback.extract_stack(frame)
def AllFrames(f):
while f is not None:
yield f
f = f.f_back
all_frames = list(reversed(list(AllFrames(frame))))
assert len(extracted) == len(all_frames)
return [{
'file': fs.filename, 'first_line': fr.f_code.co_firstlineno,
'line': fs.lineno, 'func': fs.name, 'code': fs._line,
} for fs, fr in zip(extracted, all_frames)]
This function returns whole stack correctly, ut inside code fields there are empty strings. Looks like frame objects don't have code annotation inside theirs ._line attribute as they probably should, this might be the reason of having no code annotation inside all the functions used above.
Do you know if there is any way to provide code annotation for all stack retrieving/printing operations? Besides manually composing correct stack trace by hand. Maybe there is at least some standard module that allows to set this lines of code somehow at least manually?
Update. I found out that traceback module uses linecache.getline(self.filename, self.lineno) (see here) to get source code. Does anybody know how can I fill linecache with source text from memory with given filename without using temporary file?
Also interesting if raised exception uses traceback module to output exception to console or maybe it has its own formatting implementation?
Answering my own question. After reading source code of PyRun_String() I found out that it is impossible to annotate code-lines of exception (unless I missed something).
Because PyRun_String() sets filename to "<string>" and doesn't allow to give other name, while exception printing code tries to read file from file system, and of course doesn't find this file name.
But I found out how to use Py_CompileString() with PyEval_EvalCode() to achieve line annotation instead of using just PyRun_String().
Basically I create temporary file with the help of tempfile standard module. You can create non-temporary file too, doesn't matter. Then write source code to this file and provide file name to Py_CompileString(). After this, lines are annotated correctly.
Below code is in C++, although you can use it in C too with small tweaks (like using PyObject * instead of auto).
Important. For the sake of simplicity in my code I don't handle errors of all function, also don't do reference counting of objects. Also finally I don't delete temporary file. These all things should be done in real programs. Hence code below can't be used in production directly, without modifications.
Try it online!
#include <Python.h>
int main() {
Py_SetProgramName(L"prog.py");
Py_Initialize();
char const source[] = R"(
def f():
def g():
assert False, 'TestExc'
g()
f()
)";
auto pTempFile = PyImport_ImportModule("tempfile");
auto pDeleteFalse = PyDict_New();
PyDict_SetItemString(pDeleteFalse, "delete", Py_False);
auto pFile = PyObject_Call(
PyObject_GetAttrString(pTempFile, "NamedTemporaryFile"),
PyTuple_Pack(0), pDeleteFalse);
auto pFileName = PyObject_GetAttrString(pFile, "name");
PyObject_CallMethod(pFile, "write", "y", source);
PyObject_CallMethod(pFile, "close", nullptr);
auto pCompiled = Py_CompileString(
source, PyUnicode_AsUTF8(pFileName), Py_file_input);
auto pGlobal = PyDict_New(), pLocal = PyDict_New();
auto pEval = PyEval_EvalCode(pCompiled, pGlobal, pLocal);
PyErr_Print();
Py_FinalizeEx();
}
Output:
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 6, in <module>
f()
File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 5, in f
g()
File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 4, in g
assert False, 'TestExc'
AssertionError: TestExc
In my experience programming with Java, I have become quite fond of the stack traces it generates when my code goes awry, but I feel that the traces generated by python are a bit lacking by comparison. For example, a trace in java might look like this:
java.lang.RuntimeException
at test.package.Example.c(Example.java:20)
at test.package.Example.b(Example.java:15)
at test.package.Example.a(Example.java:10)
Whereas a python trace might look like this:
Traceback (most recent call last):
File "example.py", line 10, in <module>
a()
File "example.py", line 2, in a
b()
File "example.py", line 5, in b
c()
File "example.py", line 8, in c
raise Exception
Exception
While both of these traces convey basically the same information, I personally find that the trace from java is easier to follow.
Is there a means to change the format python uses for printing its stack traces, or would that sort of change require me to create a custom exception handler at the root of my program?
using traceback module
import traceback
try:
x= 1/0
except Exception as e:
print(e)
traceback.print_exc()
There is a means to change the format Python uses to format its stack traces, and that is that you write your own formatter instead. There is only one built-in format.
You can assign your own function to sys.excepthook and it will act as a top-level exception handler that will get access to exceptions that were about to rise uncaught and cause the program to exit. There you can make use of the traceback object to format things however you like. Triptych's answer shows how to use the traceback module to get the info for each stack frame. extract_tb returns a 4-tuple of the filename, line number, function, and source text of the offending line, so if you want to not display the source text you could just throw that away and concatenate the rest. But you'll have to do the work of constructing whatever output you want to see.
If you really want to, you can reformat exception tracebacks with the traceback.extract_tb method.
ref: https://docs.python.org/2/library/traceback.html#traceback.extract_tb
I am running a bunch of code all at once in python by copying it from my editor and pasting it into python. This code includes nested for loops. I am doing some web scraping and the program quits at different times. I suspect that this is because it doesn't have time to load. I get the following error (once again - the program scrapes different amounts of text each time):
Traceback (most recent call last):
File "<stdin>", line 35, in <module>
IndexError: list index out of range
First, what does line 35 refer to? Is this the place in the relevant inner for-loop?
Second, I think that the error might be caused by a line of code using selenium like this:
driver.find_elements_by_class_name("button")[j-1].click()
In this case, how can handle this error? What is some example code with either explicit waits or exception handling that would address the issue?
It means that [j-1] doesn't exist for a given value of j, possibly if j-1 exceeds the max number of elements in the list
You can try your code and catch an IndexError exception like this:
try:
# your code here
except IndexError:
# handle the error here
An IndexError happens when you try to access an index of a list that does not exist. For example:
>>> a = [1, 2, 3]
>>> print(a[10])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: list index out of range
It's difficult to say how you should handle the error without more detail.
When working with code snippets, it's convenient to have them open in a text editor and either
only copy-paste into a console the part you're currently working on so that all the relevant variables are in the local namespace that you can explore from the console, or
copy-paste a moderate-to-large chunk as a whole while having enabled automatic post-mortem debugger calling, e.g. with Automatically start the debugger on an exception Activestate recipe or IPython's %pdb magic, or
run a script as a whole under debugger e.g with -m pdb, IPython's %run or by using an IDE.
I want to save all following Exceptions in a file. The reason why I need this is because the IDLE for python 3.1.1 in Ubuntu raises an Exception at calltipps, but close to fast, that it isn't readble. Also I need this for testing. The best, would be if I just call a function which saves all Exception to a file. Thank you! ;)
// edit:
i had looked first for a more general way! so that you do not have to place your whole code in a function or indentation. but now that worked wery well for me. although I would be still grateful, if you find a way!
thanks!
If you have a convenient main() function (whatever it's called), then you can use the logging module:
import logging
def main():
raise Exception("Hey!")
logging.basicConfig(level=logging.DEBUG, filename='/tmp/myapp.log')
try:
main()
except:
logging.exception("Oops:")
logging.exception conveniently gets the current exception and puts the details in the log:
ERROR:root:Oops:
Traceback (most recent call last):
File "C:\foo\foo.py", line 9, in <module>
main()
File "C:\foo\foo.py", line 4, in main
raise Exception("Hey!")
Exception: Hey!
When A Python exception is thrown by code that spans multiple lines, e.g.:
myfoos = [foo("bar",
"baz",
"quux",
i) for i in range(10)]
Python will report the line number of the last line, and will show the code fragment from that line:
Traceback (most recent call last):
File "test.py", line 4, in <module>
i) for i in range(10)]
NameError: name 'foo' is not defined
Is there any way to determine what the first line is? Is there any way to catch the exception and manipulate the traceback object to be able to report something like this instead:
Traceback (most recent call last):
File "test.py", lines 1-4 in <module>
myfoos = [foo("bar",
"baz",
"quux",
i) for i in range(10)]
NameError: name 'foo' is not defined
Finding the beginning of the line will be really hard. You'll have to either parse the Python or maybe dig into the compiled byte code. There are modules in the standard library for parsing Python, but I can tell you from experience that interpreting their output is a black art. And I'm not sure the compiled byte code has the answer either...
In a try/except block you can except NameError and try setting NameError.lineno, though I'm not exactly sure if or how this works, but it's the best I've found thusfar.
try:
somecode
except NameError
NameError.lineno = [1,4]
You'll have to figure out where the statement begins and ends yourself somehow as well as which statement is raising the error.
Hope this helps