Develop Reference

How do I catch exceptions after a "raise from" in python?

I have code that raises exceptions from other exceptions so that we can see details about eveything that went wrong. In the example below we include information about what we are processing and what specific thing in the processing went wrong.
def process(thing):
try:
process_widgets(thing)
except Exception as e:
raise CouldNotProcess(thing) from e
def process_widgets(thing):
for widget in get_widgets(thing):
raise CouldNotProcessWidget(widget)
def do_processing():
for thing in things_to_process():
process(thing)
I am trying to change this so that process_widgets can raise a specific type of exception and do_processing can change its behaviour based on this exception. However the raise from in process is masking this exception which makes this impossible. Is there a good to let do_processing know about what went wrong in process_widgets while also doing raise from.
Ideas:
Python 3.11 has exception groups. So perhaps there is a way of adding exceptions to group and catching them with the likely confusing except* syntax.
There is a dirty trick where I do raise e from CouldNoPorcess(thing) to get both the helpful logging.
Apparently internally exception chaining works by adding __cause__ property (forming a linked list) so I could manually look through causes in the top most exception to manually implement behaviour like except* with exception groups.
def raised_from(e, type):
while e is not None:
if isinstance(e, Specific):
return True
e = e.__cause__
return False
...
try:
do_processing()
except CouldNotProcess as e:
if raised_from(e, CouldNotProcessWidget):
do_stuff()
You see this pattern quite a lot with status_codes from http.
I could use logging rather than adding information to exceptions. This hides information from the exception handling code, but works for logging. I think this is the work around that I'll use at the moment.
It's noticeable that the PEP says that exception chaining isn't quite designed for adding information to exceptions.
Update
python 3.11 has an add_note method and notes property which can be used to add information - which works for some use cases.
For this use case exception groups might be the way to go, though I am concerned that this might be a little confusing.

Is it possible to catch an exception from outside code that is already catching it?

This is a hard question to phrase, but here's a stripped-down version of the situation. I'm using some library code that accepts a callback. It has its own error-handling, and raises an error if anything goes wrong while executing the callback.
class LibraryException(Exception):
pass
def library_function(callback, string):
try:
# (does other stuff here)
callback(string)
except:
raise LibraryException('The library code hit a problem.')
I'm using this code inside an input loop. I know of potential errors that could arise in my callback function, depending on values in the input. If that happens, I'd like to reprompt, after getting helpful feedback from its error message. I imagine it looking something like this:
class MyException(Exception):
pass
def my_callback(string):
raise MyException("Here's some specific info about my code hitting a problem.")
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except MyException as e:
print(e)
continue
Of course, this doesn't work, because MyException will be caught within library_function, which will raise its own (much less informative) Exception and halt the program.
The obvious thing to do would be to validate my input before calling library_function, but that's a circular problem, because parsing is what I'm using the library code for in the first place. (For the curious, it's Lark, but I don't think my question is specific enough to Lark to warrant cluttering it with all the specific details.)
One alternative would be to alter my code to catch any error (or at least the type of error the library generates), and directly print the inner error message:
def my_callback(string):
error_str = "Here's some specific info about my code hitting a problem."
print(error_str)
raise MyException(error_str)
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except LibraryException:
continue
But I see two issues with this. One is that I'm throwing a wide net, potentially catching and ignoring errors other than in the scope I'm aiming at. Beyond that, it just seems... inelegant, and unidiomatic, to print the error message, then throw the exception itself into the void. Plus the command line event loop is only for testing; eventually I plan to embed this in a GUI application, and without printed output, I'll still want to access and display the info about what went wrong.
What's the cleanest and most Pythonic way to achieve something like this?

There seems to be many ways to achieve what you want. Though, which one is more robust - I cannot find a clue about. I'll try to explain all the methods that seemed apparent to me. Perhaps you'll find one of them useful.
I'll be using the example code you provided to demonstrate these methods, here's a fresher on how it looks-
class MyException(Exception):
pass
def my_callback(string):
raise MyException("Here's some specific info about my code hitting a problem.")
def library_function(callback, string):
try:
# (does other stuff here)
callback(string)
except:
raise Exception('The library code hit a problem.')
The simplest approach - traceback.format_exc
import traceback
try:
library_function(my_callback, 'boo!')
except:
# NOTE: Remember to keep the `chain` parameter of `format_exc` set to `True` (default)
tb_info = traceback.format_exc()
This does not require much know-how about exceptions and stack traces themselves, nor does it require you to pass any special frame/traceback/exception to the library function. But look at what this returns (as in, the value of tb_info)-
'''
Traceback (most recent call last):
File "path/to/test.py", line 14, in library_function
callback(string)
File "path/to/test.py", line 9, in my_callback
raise MyException("Here's some specific info about my code hitting a problem.")
MyException: Here's some specific info about my code hitting a problem.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "path/to/test.py", line 19, in <module>
library_function(my_callback, 'boo!')
File "path/to/test.py", line 16, in library_function
raise Exception('The library code hit a problem.')
Exception: The library code hit a problem.
'''
That's a string, the same thing you'd see if you just let the exception happen without catching. Notice the exception chaining here, the exception at the top is the exception that happened prior to the exception at the bottom. You could parse out all the exception names-
import re
exception_list = re.findall(r'^(\w+): (\w+)', tb_info, flags=re.M)
With that, you'll get [('MyException', "Here's some specific info about my code hitting a problem"), ('Exception', 'The library code hit a problem')] in exception_list
Although this is the easiest way out, it's not very context aware. I mean, all you get are class names in string form. Regardless, if that is what suits your needs - I don't particularly see a problem with this.
The "robust" approach - recursing through __context__/__cause__
Python itself keeps track of the exception trace history, the exception currently at hand, the exception that caused this exception and so on. You can read about the intricate details of this concept in PEP 3134
Whether or not you go through the entirety of the PEP, I urge you to at least familiarize yourself with implicitly chained exceptions and explicitly chained exceptions. Perhaps this SO thread will be useful for that.
As a small refresher, raise ... from is for explicitly chaining exceptions. The method you show in your example, is implicit chaining
Now, you need to make a mental note - TracebackException#__cause__ is for explicitly chained exceptions and TracebackException#__context__ is for implicitly chained exceptions. Since your example uses implicit chaining, you can simply follow __context__ backwards and you'll reach MyException. In fact, since this is only one level of nesting, you'll reach it instantly!
import sys
import traceback
try:
library_function(my_callback, 'boo!')
except:
previous_exc = traceback.TracebackException(*sys.exc_info()).__context__
This first constructs the TracebackException from sys.exc_info. sys.exc_info returns a tuple of (exc_type, exc_value, exc_traceback) for the exception at hand (if any). Notice that those 3 values, in that specific order, are exactly what you need to construct TracebackException - so you can simply destructure it using * and pass it to the class constructor.
This returns a TracebackException object about the current exception. The exception that it is implicitly chained from is in __context__, the exception that it is explicitly chained from is in __cause__.
Note that both __cause__ and __context__ will return either a TracebackException object, or None (if you're at the end of the chain). This means, you can call __cause__/__context__ again on the return value and basically keep going till you reach the end of the chain.
Printing a TracebackException object just prints the message of the exception, if you want to get the class itself (the actual class, not a string), you can do .exc_type
print(previous_exc)
# prints "Here's some specific info about my code hitting a problem."
print(previous_exc.exc_type)
# prints <class '__main__.MyException'>
Here's an example of recursing through .__context__ and printing the types of all exceptions in the implicit chain. (You can do the same for .__cause__)
def classes_from_excs(exc: traceback.TracebackException):
print(exc.exc_type)
if not exc.__context__:
# chain exhausted
return
classes_from_excs(exc.__context__)
Let's use it!
try:
library_function(my_callback, 'boo!')
except:
classes_from_excs(traceback.TracebackException(*sys.exc_info()))
That will print-
<class 'Exception'>
<class '__main__.MyException'>
Once again, the point of this is to be context aware. Ideally, printing isn't the thing you'll want to do in a practical environment, you have the class objects themselves on your hands, with all the info!
NOTE: For implicitly chained exceptions, if an exception is explicitly suppressed, it'll be a bad day trying to recover the chain - regardless, you might give __supressed_context__ a shot.
The painful way - walking through traceback.walk_tb
This is probably the closest you can get to the low level stuff of exception handling. If you want to capture entire frames of information instead of just the exception classes and messages and such, you might find walk_tb useful....and a bit painful.
import traceback
try:
library_function(my_callback, 'foo')
except:
tb_gen = traceback.walk_tb(sys.exc_info()[2])
There is....entirely too much to discuss here. .walk_tb takes a traceback object, you may remember from the previous method that the 2nd index of the returned tuple from sys.exec_info is just that. It then returns a generator of tuples of frame object and int (Iterator[Tuple[FrameType, int]]).
These frame objects have all kinds of intricate information. Though, whether or not you'll actually find exactly what you're looking for, is another story. They may be complex, but they aren't exhaustive unless you play around with a lot of frame inspection. Regardless, this is what the frame objects represent.
What you do with the frames is upto you. They can be passed to many functions. You can pass the entire generator to StackSummary.extract to get framesummary objects, you can iterate through each frame to have a look at [0].f_locals (The [0] on Tuple[FrameType, int] returns the actual frame object) and so on.
for tb in tb_gen:
print(tb[0].f_locals)
That will give you a dict of the locals for each frame. Within the first tb from tb_gen, you'll see MyException as part of the locals....among a load of other stuff.
I have a creeping feeling I have overlooked some methods, most probably with inspect. But I hope the above methods will be good enough so that no one has to go through the jumble that is inspect :P

Chase's answer above is phenomenal. For completeness's sake, here's how I implemented their second approach in this situation. First, I made a function that can search the stack for the specified error type. Even though the chaining in my example is implicit, this should be able to follow implicit and/or explicit chaining:
import sys
import traceback
def find_exception_in_trace(exc_type):
"""Return latest exception of exc_type, or None if not present"""
tb = traceback.TracebackException(*sys.exc_info())
prev_exc = tb.__context__ or tb.__cause__
while prev_exc:
if prev_exc.exc_type == exc_type:
return prev_exc
prev_exc = prev_exc.__context__ or prev_exc.__cause__
return None
With that, it's as simple as:
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except LibraryException as exc:
if (my_exc := find_exception_in_trace(MyException)):
print(my_exc)
continue
raise exc
That way I can access my inner exception (and print it for now, although eventually I may do other things with it) and continue. But if my exception wasn't in there, I simply reraise whatever the library raised. Perfect!

Python: How to catch a list of exceptions? [duplicate]

I know that I can do:
try:
# do something that may fail
except:
# do this if ANYTHING goes wrong
I can also do this:
try:
# do something that may fail
except IDontLikeYouException:
# say please
except YouAreTooShortException:
# stand on a ladder
But if I want to do the same thing inside two different exceptions, the best I can think of right now is to do this:
try:
# do something that may fail
except IDontLikeYouException:
# say please
except YouAreBeingMeanException:
# say please
Is there any way that I can do something like this (since the action to take in both exceptions is to say please):
try:
# do something that may fail
except IDontLikeYouException, YouAreBeingMeanException:
# say please
Now this really won't work, as it matches the syntax for:
try:
# do something that may fail
except Exception, e:
# say please
So, my effort to catch the two distinct exceptions doesn't exactly come through.
Is there a way to do this?

From Python Documentation:
An except clause may name multiple exceptions as a parenthesized tuple, for example
except (IDontLikeYouException, YouAreBeingMeanException) as e:
pass
Or, for Python 2 only:
except (IDontLikeYouException, YouAreBeingMeanException), e:
pass
Separating the exception from the variable with a comma will still work in Python 2.6 and 2.7, but is now deprecated and does not work in Python 3; now you should be using as.

How do I catch multiple exceptions in one line (except block)
Do this:
try:
may_raise_specific_errors():
except (SpecificErrorOne, SpecificErrorTwo) as error:
handle(error) # might log or have some other default behavior...
The parentheses are required due to older syntax that used the commas to assign the error object to a name. The as keyword is used for the assignment. You can use any name for the error object, I prefer error personally.
Best Practice
To do this in a manner currently and forward compatible with Python, you need to separate the Exceptions with commas and wrap them with parentheses to differentiate from earlier syntax that assigned the exception instance to a variable name by following the Exception type to be caught with a comma.
Here's an example of simple usage:
import sys
try:
mainstuff()
except (KeyboardInterrupt, EOFError): # the parens are necessary
sys.exit(0)
I'm specifying only these exceptions to avoid hiding bugs, which if I encounter I expect the full stack trace from.
This is documented here: https://docs.python.org/tutorial/errors.html
You can assign the exception to a variable, (e is common, but you might prefer a more verbose variable if you have long exception handling or your IDE only highlights selections larger than that, as mine does.) The instance has an args attribute. Here is an example:
import sys
try:
mainstuff()
except (KeyboardInterrupt, EOFError) as err:
print(err)
print(err.args)
sys.exit(0)
Note that in Python 3, the err object falls out of scope when the except block is concluded.
Deprecated
You may see code that assigns the error with a comma. This usage, the only form available in Python 2.5 and earlier, is deprecated, and if you wish your code to be forward compatible in Python 3, you should update the syntax to use the new form:
import sys
try:
mainstuff()
except (KeyboardInterrupt, EOFError), err: # don't do this in Python 2.6+
print err
print err.args
sys.exit(0)
If you see the comma name assignment in your codebase, and you're using Python 2.5 or higher, switch to the new way of doing it so your code remains compatible when you upgrade.
The suppress context manager
The accepted answer is really 4 lines of code, minimum:
try:
do_something()
except (IDontLikeYouException, YouAreBeingMeanException) as e:
pass
The try, except, pass lines can be handled in a single line with the suppress context manager, available in Python 3.4:
from contextlib import suppress
with suppress(IDontLikeYouException, YouAreBeingMeanException):
do_something()
So when you want to pass on certain exceptions, use suppress.

From Python documentation -> 8.3 Handling Exceptions:
A try statement may have more than one except clause, to specify
handlers for different exceptions. At most one handler will be
executed. Handlers only handle exceptions that occur in the
corresponding try clause, not in other handlers of the same try
statement. An except clause may name multiple exceptions as a
parenthesized tuple, for example:
except (RuntimeError, TypeError, NameError):
pass
Note that the parentheses around this tuple are required, because
except ValueError, e: was the syntax used for what is normally
written as except ValueError as e: in modern Python (described
below). The old syntax is still supported for backwards compatibility.
This means except RuntimeError, TypeError is not equivalent to
except (RuntimeError, TypeError): but to except RuntimeError as
TypeError: which is not what you want.

If you frequently use a large number of exceptions, you can pre-define a tuple, so you don't have to re-type them many times.
#This example code is a technique I use in a library that connects with websites to gather data
ConnectErrs = (URLError, SSLError, SocketTimeoutError, BadStatusLine, ConnectionResetError)
def connect(url, data):
#do connection and return some data
return(received_data)
def some_function(var_a, var_b, ...):
try: o = connect(url, data)
except ConnectErrs as e:
#do the recovery stuff
blah #do normal stuff you would do if no exception occurred
NOTES:
If you, also, need to catch other exceptions than those in the
pre-defined tuple, you will need to define another except block.
If you just cannot tolerate a global variable, define it in main()
and pass it around where needed...

One of the way to do this is..
try:
You do your operations here;
......................
except(Exception1[, Exception2[,...ExceptionN]]]):
If there is any exception from the given exception list,
then execute this block.
......................
else:
If there is no exception then execute this block.
and another way is to create method which performs task executed by except block and call it through all of the except block that you write..
try:
You do your operations here;
......................
except Exception1:
functionname(parameterList)
except Exception2:
functionname(parameterList)
except Exception3:
functionname(parameterList)
else:
If there is no exception then execute this block.
def functionname( parameters ):
//your task..
return [expression]
I know that second one is not the best way to do this, but i'm just showing number of ways to do this thing.

As of Python 3.11 you can take advantage of the except* clause that is used to handle multiple exceptions.
PEP-654 introduced a new standard exception type called ExceptionGroup that corresponds to a group of exceptions that are being propagated together. The ExceptionGroup can be handled using a new except* syntax. The * symbol indicates that multiple exceptions can be handled by each except* clause.
For example, you can handle multiple exceptions
try:
raise ExceptionGroup('Example ExceptionGroup', (
TypeError('Example TypeError'),
ValueError('Example ValueError'),
KeyError('Example KeyError'),
AttributeError('Example AttributeError')
))
except* TypeError:
...
except* ValueError as e:
...
except* (KeyError, AttributeError) as e:
...
For more details see PEP-654.

Should the try/except be placed in the function declaration or call?

Both these snippets do the same thing:
Try/except in function declaration:
def something():
try:
# code goes here
except:
print("Error")
sys.exit(1)
something()
Try/except in function call:
def something():
# code goes here
try:
something()
except:
print("Error")
sys.exit(1)
Is there one that is better/more Pythonic/recommended by PEP8 or is it just up to personal preference? I understand that the second method would get tedious and repetitive if the function needs to be called more than once, but assuming the function is only called once, which one should I use?

the general rule is "only catch exceptions you can handle", see here for an explanation
note that an uncaught exception (in most languages) will cause the program to exit with an unsuccessful status code (i.e. your sys.exit(1)), it will probably also print out a message saying that an exception occurred. your demo therefore is emulating default behaviour, but doing it worse
further, you're catching every exception and this is generally bad style, e.g. you'll implicitly catch SystemExit and other internal exceptions that you probably shouldn't be dealing interacting with

re-raising exception doesn't work as expected

I have this rather intricate try except block:
try:
self.sorting = sys.argv[1]
try:
test_sorting_var = int(self.sorting)
if test_sorting_var < 1:
print "Sorting column number not valid."
raise ValueError
else:
self.sorting = test_sorting_var
except ValueError:
print "There's a problem with the sorting value provided"
print "Either the column doesn't exists or the column number is invalid"
print "Please try a different sorting value or omit it."
sys.exit(1)
except:
if self.sorting not in self.output_table.column_headers:
print "Sorting column name not valid."
raise ValueError
except:
pass
Basically I'm checking:
If there's a sys.argv[1]
If so, try it as int, and see if it's less than 1
If int fails, test it as string
In both 2+3, if the tests don't succeed, I'm raising a ValueError that should be caught in the except ValueError block and it does as expected:
Sorting column number not valid.
There's a problem with the sorting value provided
Either the column doesn't exists or the column number is invalid
Please try a different sorting value or omit it.
BUT! The sys.exit(1) is not invoked and the program just continues.
How can I fix it and even make it more readable?

In the last two lines you catch any exception:
except:
pass
This includes the exception SystemExit, which is raised by sys.exit.
To fix this, only catch exceptions deriving from Exception, which
SystemExit does not:
except Exception:
pass
In general, it's (almost) never a good idea to do a bare except, always catch Exception, or if possible, something more specific.

The builtint sys.exit() raises a SystemExit-Exception. As you are catching any type of exception when you don't define the Exception to catch (except: without an Exception Type) the SystemExit gets also caught. Ultimately the function will run until the last line where you wrote pass.
Best thing to do is to always catch specific Exceptions and never ever catch all Exceptions with an except:.
Furthermore you should put the check if self.sorting is not in self.output_table.column_headers outside the try catch where you check for a valid self.sorting.

From the documentation for sys.exit:
Exit from Python. This is implemented by raising the SystemExit exception, so cleanup actions specified by finally clauses of try statements are honored, and it is possible to intercept the exit attempt at an outer level.
This means that the outer try except loop is catching the SystemExit exception and causing it to pass. You can add this exception to the outer block and call it again.

I think I would do something like this:
import sys
def MyClass(object):
def method(self, argv, ...):
# ...
if len(argv) < 2:
raise RuntimeError("Usage: {} <sorting column>".format(argv[0]))
sorting = argv[1]
try:
self.sorting = int(sorting)
except ValueError:
try:
self.sorting = self.output_table.column_headers.index(sorting)
except ValueError:
raise ValueError("Invalid sorting column '{}'.".format(sorting))
# ...
try:
# ...
obj.method(sys.argv, ...)
except Exception as e:
sys.exit(e.message)
It's okay to ask for forgiveness instead of permission when it makes things easier (for example to parse a number), but if you need to make sure if sys.argv has enough elements just check it, it will make the program flow clearer.
Avoid using sys.exit within regular code, try to use it only in the outermost levels. For the most part, it is generally better to let exceptions bubble up and catch them at top level or let them crash the program if necessary.
Do make use of exception parameters to store error information, you can decide at a later point whether to print the error, log it, show it in a popup, ...
Instead of using sys.argv directly from within a class, you can pass it as an argument to the method/constructor, it will make the code easier to test and more flexible towards the future.