Develop Reference

Python how to bubble up exception with full stack information

See the following code
def ex_string(ex):
return "Error:{} Traceback:{}".format(ex, traceback.format_exc())
def map_values(listing):
...
obj = listing['value']
...
def process_listing(listing):
try:
listing = map_values(listing)
except Exception as ex:
do_some_handling(listing)
print ex_string(ex) # this stack shows actual line inside map_values
raise ex
def start():
for listing in listings:
try:
process_listing(listing)
save_listing(listing)
except Exception as ex:
error = ex_string(ex) #this stack shows map_values function only
log_error(error)
As above, when listing does not contain 'value' element, I don't know which line is causing issue, all I know from log is somewhere inside map_values function, how do I get the full stack trace from where the exception was first raised? In this case obj = listing['value']

Use raise with no argument inside an except clause to reraise the exception you're handling with its original traceback:
def process_listing(listing):
try:
listing = map_values(listing)
except Exception as ex:
do_some_handling(listing)
print ex_string(ex)
raise # no ex
raise with no argument means you want the exception you're handling to be considered unhandled, so it resumes propagating with the same traceback.
Using raise with one argument here means you're signalling that your exception handler has encountered an error, so Python constructs a new stack trace pointing to the raise in the exception handler.
Note that raise with an argument behaves differently on Python 3, where exceptions record their associated traceback. On Python 3, raise and raise ex will both preserve the original traceback, but raise ex will add an extra record to the traceback for the raise ex line.
Also, Python uses a strange traceback construction system where an exception has no stack trace when initially raised, and each time it bubbles up through a stack frame, an additional frame record is added to the stack trace. (A record is also added for a raise with 1 or 2 arguments, but not 0 or 3, hence the difference in the previous paragraph.) Due to this traceback construction system, a traceback won't contain any information about frames the exception didn't propagate to, and unusual exception handling patterns can cause a traceback to look nothing like what the stack looked like at the point of the initial raise.

How can I create an Exception in Python minus the last stack frame?

Not sure how possible this is, but here goes:
I'm trying to write an object with some slightly more subtle behavior - which may or may not be a good idea, I haven't determined that yet.
I have this method:
def __getattr__(self, attr):
try:
return self.props[attr].value
except KeyError:
pass #to hide the keyerror exception
msg = "'{}' object has no attribute '{}'"
raise AttributeError(msg.format(self.__dict__['type'], attr))
Now, when I create an instance of this like so:
t = Thing()
t.foo
I get a stacktrace containing my function:
Traceback (most recent call last):
File "attrfun.py", line 23, in <module>
t.foo
File "attrfun.py", line 15, in __getattr__
raise AttributeError(msg.format(self._type, attr))
AttributeError: 'Thing' object has no attribute 'foo'
I don't want that - I want the stack trace to read:
Traceback (most recent call last):
File "attrfun.py", line 23, in <module>
t.foo
AttributeError: 'Thing' object has no attribute 'foo'
Is this possible with a minimal amount of effort, or is there kind of a lot required? I found this answer which indicates that something looks to be possible, though perhaps involved. If there's an easier way, I'd love to hear it! Otherwise I'll just put that idea on the shelf for now.

You cannot tamper with traceback objects (and that's a good thing). You can only control how you process one that you've already got.
The only exceptions are: you can
substitute an exception with another or re-raise it with raise e (i.e make the traceback point to the re-raise statement's location)
raise an exception with an explicit traceback object
remove outer frame(s) from a traceback object by accessing its tb_next property (this reflects a traceback object's onion-like structure)
For your purpose, the way to go appears to be the 1st option: re-raise an exception from a handler one level above your function.
And, I'll say this again, this is harmful for yourself or whoever will be using your module as it deletes valuable diagnostic information. If you're dead set on making your module proprietary with whatever rationale, it's more productive for that goal to make it a C extension.

The traceback object is created during stack unwinding, not directly when you raise the exception, so you can not alter it right in your function. What you could do instead (though it's probably a bad idea) is to alter the top level exception hook so that it hides your function from the traceback.
Suppose you have this code:
class MagicGetattr:
def __getattr__(self, item):
raise AttributeError(f"{item} not found")
orig_excepthook = sys.excepthook
def excepthook(type, value, traceback):
iter_tb = traceback
while iter_tb.tb_next is not None:
if iter_tb.tb_next.tb_frame.f_code is MagicGetattr.__getattr__.__code__:
iter_tb.tb_next = None
break
iter_tb = iter_tb.tb_next
orig_excepthook(type, value, traceback)
sys.excepthook = excepthook
# The next line will raise an error
MagicGetattr().foobar
You will get the following output:
Traceback (most recent call last):
File "test.py", line 49, in <module>
MagicGetattr().foobar
AttributeError: foobar not found
Note that this ignores the __cause__ and __context__ members of the exception, which you would probably want to visit too if you were to implement this in real life.

You can get the current frame and any other level using the inspect module. For instance, here is what I use when I'd like to know where I'm in my code :
from inspect import currentframe
def get_c_frame(level = 0) :
"""
Return caller's frame
"""
return currentframe(level)
...
def locate_error(level = 0) :
"""
Return a string containing the filename, function name and line
number where this function was called.
Output is : ('file name' - 'function name' - 'line number')
"""
fi = get_c_frame(level = level + 2)
return '({} - {} - {})'.format(__file__,
fi.f_code,
fi.f_lineno)

Print an error message without printing a traceback and close the program when a condition is not met

I've seen similar questions to this one but none of them really address the trackback.
If I have a class like so
class Stop_if_no_then():
def __init__(self, value one, operator, value_two, then, line_or_label, line_number):
self._firstvalue = value_one
self._secondvalue = value_two
self._operator = operator
self._gohere = line_or_label
self._then = then
self._line_number = line_number
def execute(self, OtherClass):
"code comparing the first two values and making changes etc"
What I want my execute method to be able to do is if self._then is not equal to the string "THEN" (in allcaps) then I want it to raise a custom error message and terminate the whole program while also not showing a traceback.
If the error is encountered the only thing that should print out would look something like (I'm using 3 as an example, formatting is not a problem) this.
`Syntax Error (Line 3): No -THEN- present in the statement.`
I'm not very picky about it actually being an exception class object, so there's no issue in that aspect. Since I will be using this in a while loop, simple if, elif just repeats the message over and over (because obviously I am not closing the loop). I have seen sys.exit() but that also prints out a giant block of red text, unless I am not using it correctly. I don't want to catch the exception in my loop because there are other classes in the same module in which I need to implement something like this.

You can turn off the traceback by limiting its depth.
Python 2.x
import sys
sys.tracebacklimit = 0
Python 3.x
In Python 3.5.2 and 3.6.1, setting tracebacklimit to 0 does not seem to have the intended effect. This is a known bug. Note that -1 doesn't work either. Setting it to None does however seem to work, at least for now.
In Python 3.6.2 and above you should set tracebacklimit to 0 or -1, as setting it to None does not disable the traceback output.
Python 3.6.1 and bellow results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Exception
Python 3.6.2 and above results:
>>> import sys
>>> sys.tracebacklimit = 0
>>> raise Exception
Exception
>>> sys.tracebacklimit = -1
>>> raise Exception
Exception
>>> sys.tracebacklimit = None
>>> raise Exception
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Exception
Nevertheless, for better or worse, if multiple exceptions are raised, they can all still be printed. For example:
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>

You can use SystemExit exception:
except Exception as err:
raise SystemExit(err)
https://docs.python.org/3/library/exceptions.html

You can use a try: and then except Exception as inst:
What that will do is give you your error message in a variable named inst and you can print out the arguments on the error with inst.args. Try printing it out and seeing what happens, and is any item in inst.args is the one you are looking for.
EDIT Here is an example I tried with pythons IDLE:
>>> try:
open("epik.sjj")
except Exception as inst:
d = inst
>>> d
FileNotFoundError(2, 'No such file or directory')
>>> d.args
(2, 'No such file or directory')
>>> d.args[1]
'No such file or directory'
>>>
EDIT 2: as for closing the program you can always raise and error or you can use sys.exit()

The cleanest way that I know is to use sys.excepthook.
You implement a three argument function that accepts type, value, and traceback and does whatever you like (say, only prints the value) and assign that function to sys.excepthook.
Here is an example:
import sys
def excepthook(type, value, traceback):
print(value)
sys.excepthook = excepthook
raise ValueError('hello')
This is available in both python 2 and python 3.

If you want to get rid of any traceback for customs exceptions and have line number,
you can do this trick
Python 3
import sys
import inspect
class NoTraceBackWithLineNumber(Exception):
def __init__(self, msg):
try:
ln = sys.exc_info()[-1].tb_lineno
except AttributeError:
ln = inspect.currentframe().f_back.f_lineno
self.args = "{0.__name__} (line {1}): {2}".format(type(self), ln, msg),
sys.exit(self)
class MyNewError(NoTraceBackWithLineNumber):
pass
raise MyNewError("Now TraceBack Is Gone")
Will give this output, and make the raise keyword useless
MyNewError (line 16): Now TraceBack Is Gone

"Exception chaining can be disabled by using from None " - Python docs
>>> try:
... open('database.sqlite')
... except IOError:
... raise RuntimeError from None
Traceback (most recent call last):
File "<stdin>", line 4, in <module>

In general, if you want to catch any exception except SystemExit, and exit with the exception's message without the traceback, define your main function as below:
>>> import sys
>>> def main():
... try:
... # Run your program from here.
... raise RandomException # For testing
... except (Exception, KeyboardInterrupt) as exc:
... sys.exit(exc)
...
>>> main()
name 'RandomException' is not defined
$ echo $?
1
Note that in the case of multiple exceptions being raised, only one message is printed.
This answer is meant to improve upon the one by The-IT.

Catching unpickleable exceptions and re-raising

This is a followup to my question Hang in Python script using SQLAlchemy and multiprocessing. As discussed in that question, pickling exceptions is problematic in Python. This is usually not a issue, but one case when it is, is when errors occur in the python multiprocessing module. Since multiprocessing moves objects around by pickling, if an error occurs inside a multiprocessing process, the entire process may hang, as demonstrated in that question.
One possible approach is to fix all the problematic exceptions, as discussed in that question. This is not easy, since one cannot easily know in advance which exceptions may be called. An alternative approach, which was suggested by lbolla in an answer to the question, is to catch the exception, construct an equivalent harmless exception, and then rethrow.
However, I'm not sure of exactly how to do this. Consider the following code.
class BadExc(Exception):
def __init__(self, message, a):
'''Non-optional param in the constructor.'''
Exception.__init__(self, message)
self.a = a
import sys
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
raise Exception(e.__class__.__name__ + ": " +str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
This code pickles and unpickles the exception, and then throws it, giving the error
raising error
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
Exception: BadExc: bad exception error message
Credits to Glenn Maynard's answer to "“Inner exception” (with traceback) in Python?".
This has the important stuff, namely the traceback, the error message, and the exception type, so this might be the best one can do. But ideally I'd like something that looks exactly like the original exception, namely
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
__main__.BadExc: bad exception error message
or more generally, with the name of the exception in the front, rather than Exception. Is this possible?
Alternatively, instead of the BadExc class, one can use the print foo statement instead, which gives a NameError. However, this exception does not require special handling.

You can override sys.excepthook to achieve what you want. It at least works for this example, but it's pretty hacky so please test and no promises :-)
import sys
def excepthook_wrapper(type, value, traceback):
if len(value.args) == 2:
name, msg = value.args
value.args = (msg,)
sys.__excepthook__(name, value, traceback)
else:
sys.__excepthook__(type, value, traceback)
sys.excepthook = excepthook_wrapper
(Edit: I'm not really happy with this because now 'normal' Exceptions with two arguments will get handled differently too. Possible solution, 'tag' your special Exceptions by passing "PICKLED" as a first argument and then check for that, instead of checking for the length of the args.)
And then create the Exception with two arguments, the name (__module__.__class__) and the Exception message (str(e)):
try:
try:
#print foo
raise BadExc("bad exception error message", "a")
except Exception, e:
cls = e.__class__
if hasattr(cls, '__module__'):
name = '{0}.{1}'.format(cls.__module__, cls.__name__)
else:
name = cls.__name__
raise Exception(name, str(e)), None, sys.exc_info()[2]
except Exception, f:
pass
Then this:
import cPickle
a = cPickle.dumps(f)
l = cPickle.loads(a)
print "raising error"
raise sys.exc_info()[0], sys.exc_info()[1], sys.exc_info()[2]
Prints:
raising error
Traceback (most recent call last):
File "test.py", line 18, in <module>
raise BadExc("bad exception error message", "a")
__main__.BadExc: bad exception error message

How to save traceback / sys.exc_info() values in a variable?

I want to save the name of the error and the traceback details into a variable. Here's is my attempt.
import sys
try:
try:
print x
except Exception, ex:
raise NameError
except Exception, er:
print "0", sys.exc_info()[0]
print "1", sys.exc_info()[1]
print "2", sys.exc_info()[2]
Output:
0 <type 'exceptions.NameError'>
1
2 <traceback object at 0xbd5fc8>
Desired Output:
0 NameError
1
2 Traceback (most recent call last):
File "exception.py", line 6, in <module>
raise NameError
P.S. I know this can be done easily using the traceback module, but I want to know usage of sys.exc_info()[2] object here.

This is how I do it:
>>> import traceback
>>> try:
... int('k')
... except:
... var = traceback.format_exc()
...
>>> print var
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: invalid literal for int() with base 10: 'k'
You should however take a look at the traceback documentation, as you might find there more suitable methods, depending to how you want to process your variable afterwards...

sys.exc_info() returns a tuple with three values (type, value, traceback).
Here type gets the exception type of the Exception being handled
value is the arguments that are being passed to constructor of exception class
traceback contains the stack information like where the exception occurred etc.
For Example, In the following program
try:
a = 1/0
except Exception,e:
exc_tuple = sys.exc_info()
Now If we print the tuple the values will be this.
exc_tuple[0] value will be "ZeroDivisionError"
exc_tuple[1] value will be "integer division or modulo by zero" (String passed as parameter to the exception class)
exc_tuple[2] value will be "trackback object at (some memory address)"
The above details can also be fetched by simply printing the exception in string format.
print str(e)

Use traceback.extract_stack() if you want convenient access to module and function names and line numbers.
Use ''.join(traceback.format_stack()) if you just want a string that looks like the traceback.print_stack() output.
Notice that even with ''.join() you will get a multi-line string, since the elements of format_stack() contain \n. See output below.
Remember to import traceback.
Here's the output from traceback.extract_stack(). Formatting added for readability.
>>> traceback.extract_stack()
[
('<string>', 1, '<module>', None),
('C:\\Python\\lib\\idlelib\\run.py', 126, 'main', 'ret = method(*args, **kwargs)'),
('C:\\Python\\lib\\idlelib\\run.py', 353, 'runcode', 'exec(code, self.locals)'),
('<pyshell#1>', 1, '<module>', None)
]
Here's the output from ''.join(traceback.format_stack()). Formatting added for readability.
>>> ''.join(traceback.format_stack())
' File "<string>", line 1, in <module>\n
File "C:\\Python\\lib\\idlelib\\run.py", line 126, in main\n
ret = method(*args, **kwargs)\n
File "C:\\Python\\lib\\idlelib\\run.py", line 353, in runcode\n
exec(code, self.locals)\n File "<pyshell#2>", line 1, in <module>\n'

Be careful when you take the exception object or the traceback object out of the exception handler, since this causes circular references and gc.collect() will fail to collect. This appears to be of a particular problem in the ipython/jupyter notebook environment where the traceback object doesn't get cleared at the right time and even an explicit call to gc.collect() in finally section does nothing. And that's a huge problem if you have some huge objects that don't get their memory reclaimed because of that (e.g. CUDA out of memory exceptions that w/o this solution require a complete kernel restart to recover).
In general if you want to save the traceback object, you need to clear it from references to locals(), like so:
import sys, traceback, gc
type, val, tb = None, None, None
try:
myfunc()
except:
type, val, tb = sys.exc_info()
traceback.clear_frames(tb)
# some cleanup code
gc.collect()
# and then use the tb:
if tb:
raise type(val).with_traceback(tb)
In the case of jupyter notebook, you have to do that at the very least inside the exception handler:
try:
myfunc()
except:
type, val, tb = sys.exc_info()
traceback.clear_frames(tb)
raise type(val).with_traceback(tb)
finally:
# cleanup code in here
gc.collect()
Tested with python 3.7.
p.s. the problem with ipython or jupyter notebook env is that it has %tb magic which saves the traceback and makes it available at any point later. And as a result any locals() in all frames participating in the traceback will not be freed until the notebook exits or another exception will overwrite the previously stored backtrace. This is very problematic. It should not store the traceback w/o cleaning its frames. Fix submitted here.

The object can be used as a parameter in Exception.with_traceback() function:
except Exception as e:
tb = sys.exc_info()
print(e.with_traceback(tb[2]))