In trying to eliminate potential race condition in a python module I wrote to monitor some specialized workflows, I learned about python's "easier to ask forgiveness than permission" (EAFP) coding style, and I'm now raising lots of custom exceptions with try/except blocks where I used to use if/thens.
I'm new to python and This EAFP style makes sense logically and seems make my code more robust, but something about this feels way overboard. Is is bad practice to define one or more exceptions per method?
These custom exceptions tend to be useful only to a single method and, while it feels like a functionally correct solution, it seems like a lot of code to maintain.
Here a sample method for example:
class UploadTimeoutFileMissing(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
class UploadTimeoutTooSlow(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
def check_upload(file, timeout_seconds, max_age_seconds, min_age_seconds):
timeout = time.time() + timeout_seconds
## Check until file found or timeout
while (time.time() < timeout):
time.sleep(5)
try:
filetime = os.path.getmtime(file)
filesize = os.path.getsize(file)
except OSError:
print "File not found %s" % file
continue
fileage = time.time() - filetime
## Make sure file isn't pre-existing
if fileage > max_age_seconds:
print "File too old %s" % file
continue
## Make sure file isn't still uploading
elif fileage <= min_age_seconds:
print "File too new %s" % file
continue
return(filetime, filesize)
## Timeout
try:
filetime
filesize
raise UploadTimeoutTooSlow("File still uploading")
except NameError:
raise UploadTimeoutFileMissing("File not sent")
define one or more exceptions per method
If you mean that the exception is actually defined per method as in "within the method body", then yes. That is bad practice. This is true also if you define two exceptions that would relate to the same error but you create two because two different methods raise them.
If you ask whether it is bad practice to raise more than one exception per method, then no, that is good practice. And if the errors are not of the same category, it's perfectly ok to define several exceptions per module.
In general, for larger modules you will define more than one exception. If you would work on some arithmetic library and you would define a ZeroDivisionError and an OverflowError (if they weren't already defined in python, because you can of course re-use those) that would be perfectly fine.
Is is bad practice to define one or more exceptions per method?
Yes.
One per module is more typical. It depends, of course, on the detailed semantics. The question boils down to this: "What will you really try to catch?"
If you're never going to use except ThisVeryDetailedException: in your code, then your very detailed exception isn't very helpful.
If you can do this: except Error as e: if e.some_special_case for the very few times it matters, then you can easily simplify to one exception per module and handle your special cases as attributes of the exception rather than different types of exceptions.
The common suggestions (one per module, named Error) means that your code will often look like this.
try:
something
except some_module.Error as e:
carry on
This gives you a nice naming convention: module.Error. This covers numerous sins.
On an unrelated note, if you think you've got "potential race condition" you should probably redesign things correctly or stop trying to use threads or switch to multiprocessing. If you use multiprocessing, you'll find that it's very easy to avoid race conditions.
I'm going to weigh in on this because custom exceptions are dear to my heart. I'll explain my circumstances and the reader can weigh them against their own.
I'm the pipeline architect for a visual effects company - most of what I do involves developing what I call the "Facility API" - it's a system of a great many modules which handle everything from locating things on the filesystem, managing module/tool/project configuration, to handling datatypes from various CG applications to enable collaboration.
I go to great lengths to try to ensure that Python's built-in exceptions never bubble up. Since our developers will be relying on an ecosystem of existing modules to build their own tools on top of, having the API let a generic IOError escape is counterproductive - especially since the calling routine might not even be aware that it's reading the filesystem (abstraction is a beautiful thing). If the underlying module is unable to express something meaningful about that error, more work needs to be done.
My approach to solving this is to create a facility exception class from which all other facility exceptions are derived. There are subclasses of that for specific types of task or specific host applications - which allows me to customize error handling (for instance, exceptions raised in Maya will launch a UI to aid in troubleshooting since the usual exception would be raised in an inconspicuous console and would often be missed).
All sorts of reporting is built into the facility exception class - exceptions don't appear to a user without also being reported internally. For a range of exceptions, I get an IM any time one is raised. Others simply report quietly into a database that I can query for recent (daily or weekly) reports. Each links to EXTENSIVE data captured from the user session - typically including a screenshot, stack trace, system configuration, and a whole lot more. This means I can effectively troubleshoot problems before they're reported - and have more information at my fingertips than most users are likely able to provide.
Very fine gradations in purpose are discouraged - the exceptions accept passed values (sometimes even a dictionary instead of a string, if we want to provide plenty of data for troubleshooting) to provide with their formatted output.
So no - I don't think defining an exception or two per module is unreasonable - but they need to be meaningful and add something to the project. If you're just wrapping an IOError to raise MyIOError("I got an IO error!"), then you may want to rethink that.
I don't think it's necessary to have an extremely specific exception for every possible scenario. A single UploadTimeoutError would probably be fine, and you can just customize the exception string - that's what the strings are for, after all. Note how python doesn't have a separate exception for every possible type of syntax error, just a general SyntaxError.
Also - is it actually necessary to define the __init__ and __str__ methods for each of your custom exceptions? As far as I can tell, if you're not implementing any unusual behavior, you don't need to add any code:
>>> class MyException(Exception): pass
...
>>> raise MyException("oops!")
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
MyException: oops!
>>> str(MyException("oops!"))
'oops!'
Related
This is a hard question to phrase, but here's a stripped-down version of the situation. I'm using some library code that accepts a callback. It has its own error-handling, and raises an error if anything goes wrong while executing the callback.
class LibraryException(Exception):
pass
def library_function(callback, string):
try:
# (does other stuff here)
callback(string)
except:
raise LibraryException('The library code hit a problem.')
I'm using this code inside an input loop. I know of potential errors that could arise in my callback function, depending on values in the input. If that happens, I'd like to reprompt, after getting helpful feedback from its error message. I imagine it looking something like this:
class MyException(Exception):
pass
def my_callback(string):
raise MyException("Here's some specific info about my code hitting a problem.")
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except MyException as e:
print(e)
continue
Of course, this doesn't work, because MyException will be caught within library_function, which will raise its own (much less informative) Exception and halt the program.
The obvious thing to do would be to validate my input before calling library_function, but that's a circular problem, because parsing is what I'm using the library code for in the first place. (For the curious, it's Lark, but I don't think my question is specific enough to Lark to warrant cluttering it with all the specific details.)
One alternative would be to alter my code to catch any error (or at least the type of error the library generates), and directly print the inner error message:
def my_callback(string):
error_str = "Here's some specific info about my code hitting a problem."
print(error_str)
raise MyException(error_str)
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except LibraryException:
continue
But I see two issues with this. One is that I'm throwing a wide net, potentially catching and ignoring errors other than in the scope I'm aiming at. Beyond that, it just seems... inelegant, and unidiomatic, to print the error message, then throw the exception itself into the void. Plus the command line event loop is only for testing; eventually I plan to embed this in a GUI application, and without printed output, I'll still want to access and display the info about what went wrong.
What's the cleanest and most Pythonic way to achieve something like this?
There seems to be many ways to achieve what you want. Though, which one is more robust - I cannot find a clue about. I'll try to explain all the methods that seemed apparent to me. Perhaps you'll find one of them useful.
I'll be using the example code you provided to demonstrate these methods, here's a fresher on how it looks-
class MyException(Exception):
pass
def my_callback(string):
raise MyException("Here's some specific info about my code hitting a problem.")
def library_function(callback, string):
try:
# (does other stuff here)
callback(string)
except:
raise Exception('The library code hit a problem.')
The simplest approach - traceback.format_exc
import traceback
try:
library_function(my_callback, 'boo!')
except:
# NOTE: Remember to keep the `chain` parameter of `format_exc` set to `True` (default)
tb_info = traceback.format_exc()
This does not require much know-how about exceptions and stack traces themselves, nor does it require you to pass any special frame/traceback/exception to the library function. But look at what this returns (as in, the value of tb_info)-
'''
Traceback (most recent call last):
File "path/to/test.py", line 14, in library_function
callback(string)
File "path/to/test.py", line 9, in my_callback
raise MyException("Here's some specific info about my code hitting a problem.")
MyException: Here's some specific info about my code hitting a problem.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "path/to/test.py", line 19, in <module>
library_function(my_callback, 'boo!')
File "path/to/test.py", line 16, in library_function
raise Exception('The library code hit a problem.')
Exception: The library code hit a problem.
'''
That's a string, the same thing you'd see if you just let the exception happen without catching. Notice the exception chaining here, the exception at the top is the exception that happened prior to the exception at the bottom. You could parse out all the exception names-
import re
exception_list = re.findall(r'^(\w+): (\w+)', tb_info, flags=re.M)
With that, you'll get [('MyException', "Here's some specific info about my code hitting a problem"), ('Exception', 'The library code hit a problem')] in exception_list
Although this is the easiest way out, it's not very context aware. I mean, all you get are class names in string form. Regardless, if that is what suits your needs - I don't particularly see a problem with this.
The "robust" approach - recursing through __context__/__cause__
Python itself keeps track of the exception trace history, the exception currently at hand, the exception that caused this exception and so on. You can read about the intricate details of this concept in PEP 3134
Whether or not you go through the entirety of the PEP, I urge you to at least familiarize yourself with implicitly chained exceptions and explicitly chained exceptions. Perhaps this SO thread will be useful for that.
As a small refresher, raise ... from is for explicitly chaining exceptions. The method you show in your example, is implicit chaining
Now, you need to make a mental note - TracebackException#__cause__ is for explicitly chained exceptions and TracebackException#__context__ is for implicitly chained exceptions. Since your example uses implicit chaining, you can simply follow __context__ backwards and you'll reach MyException. In fact, since this is only one level of nesting, you'll reach it instantly!
import sys
import traceback
try:
library_function(my_callback, 'boo!')
except:
previous_exc = traceback.TracebackException(*sys.exc_info()).__context__
This first constructs the TracebackException from sys.exc_info. sys.exc_info returns a tuple of (exc_type, exc_value, exc_traceback) for the exception at hand (if any). Notice that those 3 values, in that specific order, are exactly what you need to construct TracebackException - so you can simply destructure it using * and pass it to the class constructor.
This returns a TracebackException object about the current exception. The exception that it is implicitly chained from is in __context__, the exception that it is explicitly chained from is in __cause__.
Note that both __cause__ and __context__ will return either a TracebackException object, or None (if you're at the end of the chain). This means, you can call __cause__/__context__ again on the return value and basically keep going till you reach the end of the chain.
Printing a TracebackException object just prints the message of the exception, if you want to get the class itself (the actual class, not a string), you can do .exc_type
print(previous_exc)
# prints "Here's some specific info about my code hitting a problem."
print(previous_exc.exc_type)
# prints <class '__main__.MyException'>
Here's an example of recursing through .__context__ and printing the types of all exceptions in the implicit chain. (You can do the same for .__cause__)
def classes_from_excs(exc: traceback.TracebackException):
print(exc.exc_type)
if not exc.__context__:
# chain exhausted
return
classes_from_excs(exc.__context__)
Let's use it!
try:
library_function(my_callback, 'boo!')
except:
classes_from_excs(traceback.TracebackException(*sys.exc_info()))
That will print-
<class 'Exception'>
<class '__main__.MyException'>
Once again, the point of this is to be context aware. Ideally, printing isn't the thing you'll want to do in a practical environment, you have the class objects themselves on your hands, with all the info!
NOTE: For implicitly chained exceptions, if an exception is explicitly suppressed, it'll be a bad day trying to recover the chain - regardless, you might give __supressed_context__ a shot.
The painful way - walking through traceback.walk_tb
This is probably the closest you can get to the low level stuff of exception handling. If you want to capture entire frames of information instead of just the exception classes and messages and such, you might find walk_tb useful....and a bit painful.
import traceback
try:
library_function(my_callback, 'foo')
except:
tb_gen = traceback.walk_tb(sys.exc_info()[2])
There is....entirely too much to discuss here. .walk_tb takes a traceback object, you may remember from the previous method that the 2nd index of the returned tuple from sys.exec_info is just that. It then returns a generator of tuples of frame object and int (Iterator[Tuple[FrameType, int]]).
These frame objects have all kinds of intricate information. Though, whether or not you'll actually find exactly what you're looking for, is another story. They may be complex, but they aren't exhaustive unless you play around with a lot of frame inspection. Regardless, this is what the frame objects represent.
What you do with the frames is upto you. They can be passed to many functions. You can pass the entire generator to StackSummary.extract to get framesummary objects, you can iterate through each frame to have a look at [0].f_locals (The [0] on Tuple[FrameType, int] returns the actual frame object) and so on.
for tb in tb_gen:
print(tb[0].f_locals)
That will give you a dict of the locals for each frame. Within the first tb from tb_gen, you'll see MyException as part of the locals....among a load of other stuff.
I have a creeping feeling I have overlooked some methods, most probably with inspect. But I hope the above methods will be good enough so that no one has to go through the jumble that is inspect :P
Chase's answer above is phenomenal. For completeness's sake, here's how I implemented their second approach in this situation. First, I made a function that can search the stack for the specified error type. Even though the chaining in my example is implicit, this should be able to follow implicit and/or explicit chaining:
import sys
import traceback
def find_exception_in_trace(exc_type):
"""Return latest exception of exc_type, or None if not present"""
tb = traceback.TracebackException(*sys.exc_info())
prev_exc = tb.__context__ or tb.__cause__
while prev_exc:
if prev_exc.exc_type == exc_type:
return prev_exc
prev_exc = prev_exc.__context__ or prev_exc.__cause__
return None
With that, it's as simple as:
while True:
something = input('Enter something: ')
try:
library_function(my_callback, something)
except LibraryException as exc:
if (my_exc := find_exception_in_trace(MyException)):
print(my_exc)
continue
raise exc
That way I can access my inner exception (and print it for now, although eventually I may do other things with it) and continue. But if my exception wasn't in there, I simply reraise whatever the library raised. Perfect!
Let's say I want to be able to log to file every time any exception is raised, anywhere in my program. I don't want to modify any existing code.
Of course, this could be generalized to being able to insert a hook every time an exception is raised.
Would the following code be considered safe for doing such a thing?
class MyException(Exception):
def my_hook(self):
print('---> my_hook() was called');
def __init__(self, *args, **kwargs):
global BackupException;
self.my_hook();
return BackupException.__init__(self, *args, **kwargs);
def main():
global BackupException;
global Exception;
BackupException = Exception;
Exception = MyException;
raise Exception('Contrived Exception');
if __name__ == '__main__':
main();
If you want to log uncaught exceptions, just use sys.excepthook.
I'm not sure I see the value of logging all raised exceptions, since lots of libraries will raise/catch exceptions internally for things you probably won't care about.
Your code as far as I can tell would not work.
__init__ has to return None and you are trying to return an instance of backup exception. In general if you would like to change what instance is returned when instantiating a class you should override __new__.
Unfortunately you can't change any of the attributes on the Exception class. If that was an option you could have changed Exception.__new__ and placed your hook there.
the "global Exception" trick will only work for code in the current module. Exception is a builtin and if you really want to change it globally you need to import __builtin__; __builtin__.Exception = MyException
Even if you changed __builtin__.Exception it will only affect future uses of Exception, subclasses that have already been defined will use the original Exception class and will be unaffected by your changes. You could loop over Exception.__subclasses__ and change the __bases__ for each one of them to insert your Exception subclass there.
There are subclasses of Exception that are also built-in types that you also cannot modify, although I'm not sure you would want to hook any of them (think StopIterration).
I think that the only decent way to do what you want is to patch the Python sources.
This code will not affect any exception classes that were created before the start of main, and most of the exceptions that happen will be of such kinds (KeyError, AttributeError, and so forth). And you can't really affect those "built-in exceptions" in the most important sense -- if anywhere in your code is e.g. a 1/0, the real ZeroDivisionError will be raised (by Python's own internals), not whatever else you may have bound to that exceptions' name.
So, I don't think your code can do what you want (despite all the semicolons, it's still supposed to be Python, right?) -- it could be done by patching the C sources for the Python runtime, essentially (e.g. by providing a hook potentially caught on any exception even if it's later caught) -- such a hook currently does not exist because the use cases for it would be pretty rare (for example, a StopIteration is always raised at the normal end of every for loop -- and caught, too; why on Earth would one want to trace that, and the many other routine uses of caught exceptions in the Python internals and standard library?!).
Download pypy and instrument it.
I have:
a function call_vendor_code() that calls external vendor code;
an exception class MyDomainException(Exceprion).
I know that function call_vendor_code() could throw MyDomainException (because that logic in function is my). But how do I know which vendor code exceptions could be thrown in call_vendor_code() in subcalls?
Example:
def call_vendor_code():
if not vendor.package.module.make_more_love():
raise MyDomainException('Not enough love.')
Vendor code could throw any other exceptions that I don't handle. It is possible to intercept all excepions from vendor code, but i thing that it is bad solution.
def call_vendor_code():
try:
if not vendor.package.module.make_more_love():
raise MyDomainException('Not enough love.')
except Exception, e:
raise MyDomainException(e)
How to interact/not interact with other-level exceptions?
Your logic is sound.
How you handle these vendor specific exceptions, more elegantly, depends
on the type of exceptions raised by the vendor code.
Are they plain python exceptions(e.g. ValueError, TypeError, etc)? Then
you don't have much of a choice. You should wrap them under a generic
except Exception statement. In these unfortunate cases, many people
try to differentiate the type of error by parsing the exception message.
A major anti-pattern, but what can you do?
Do they raise their own vendor specific exceptions(best case scenario)?
Then you can easily differentiate and handle them more elegantly in your
code. They should have some kind of documentation about it :)
Or, if the code is available, read through the code and see for yourself.
If we're talking about vendor code that talks to some external system(some
kind of DB for example), they might have a generic VendorException with
some internal error code(1133, 623, whatever) that is stored in the exception
instance as an attribute. Again, they most probably will have a listing of
those error codes, along with their description in some kind of documentation,
that you can use to map them into your own custom exceptions.
Say I have this code:
def wait_for_x(timeout_at=None):
while condition_that_could_raise_exceptions
if timeout_at is not None and time.time() > timeout_at:
raise SOMEEXCEPTIONHERE
do_some_stuff()
try:
foo()
wait_for_x(timeout_at=time.time() + 10)
bar()
except SOMEEXCEPTIONHERE:
# report timeout, move on to something else
How do I pick an exception type SOMEEXCEPTIONHERE for the function? Is it reasonable to create a unique exception type for that function, so that there's no danger of condition_that_could_raise_exceptions raising the same exception type?
wait_for_x.Timeout = type('Timeout', (Exception,), {})
If distinguishing exceptions from wait_for_x from those from condition_that_could_raise_exceptions is important enough, then sure, define a new exception type. After all, the type is the main way of distinguishing different kinds of exceptions, and parsing the message tends to get messy pretty quickly.
Yes, you should certainly define your own exception class whenever none of the built-in exception types are appropriate. In some cases (say, if you're building some kind of HTML munger on top of LXML or BeautifulSoup) it might also be appropriate to use an exception from some other module.
Python Standard Library defines a lot of its own custom exceptions. It seems good practice to do that as well for personal modules or functions.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Python FAQ: “How fast are exceptions?”
I remember reading that Python implements a "Better to seek forgiveness than to ask permission" philosophy with regards to exceptions. According to the author, this meant Python code should use a lot of try - except clauses, rather than trying to determine ahead of time if you were about to do something that would cause an exception.
I just wrote some try - except clauses on my web app in which an exception will be raised most of the time the code is run. So, in this case, raising and catching an exception will be the norm. Is this bad from an efficiency point of view? I also remember someone telling me that catching a raised exception has a large performance overhead.
Is it unnecessarily inefficient to use try - except clauses in which you expect an exception to be raised and caught almost all of the time?
Here's the code -- its using the Django ORM to check for objects that associate users with various third party social providers.
try:
fb_social_auth = UserSocialAuth.objects.get(user=self, provider='facebook')
user_dict['facebook_id'] = fb_social_auth.uid
except ObjectDoesNotExist:
user_dict['facebook_id'] = None
try:
fs_social_auth = UserSocialAuth.objects.get(user=self, provider='foursquare')
user_dict['foursquare_id'] = fs_social_auth.uid
except ObjectDoesNotExist:
user_dict['foursquare_id'] = None
try:
tw_social_auth = UserSocialAuth.objects.get(user=self, provider='twitter')
user_dict['twitter_id'] = tw_social_auth.uid
except ObjectDoesNotExist:
user_dict['twitter_id'] = None
The first one will rarely take the exception, since right now we are enforcing "Sign In With Facebook" as the primary method for new users to join the site. But, Twitter and Foursquare are optional, in case they want to import friends or followers, and I expect most people will not.
I'm open to better ways to code this logic.
Whenever you code there is a balancing of concerns: performance, readability, correctness, extendability, maintainability, etc.
Unfortunately, it is often not possible to improve code in each of these directions at the same time. What is fast may not be as readable for instance.
One of the reasons why try..except is encouraged in Python is because you often can not anticipate all the ways your code may be used, so rather than checking if a specific condition exists, it is more general to just catch any of a certain class of error that might arise. Thus try..except may make your code more reusable.
However, it is also true that try..except is slow if the except clause is often being reached.
Is there a way to code that block so that an exception is not being raised and use try..except to catch the less frequent condition?
Or if not, for the sake of efficiency, you may choose not to use try..except. There are few hard and fast rules in programming. You have to choose your way based on your balance of concerns.
If you are attempting to optimize this function for speed, you should focus on what is likely to be the actual bottleneck. Your three database queries, each of which will cause the operating system to context switch, almost certainly take an order of magnitude longer than catching an exception. If you want to make the code as fast as possible, begin by combining all three database queries into one:
auth_objects = UserSocialAuth.objects.filter(user=self, provider__in=('facebook', 'foursquare', 'twitter'))
and then loop through the objects. The provider__in filter may be unnecessary if those three providers are the only ones in the database.
It's true that catching an exception is moderately expensive (see below for some timings) and you wouldn't want to this it in the bottleneck of your program, but in the examples you give, catching the exception is going to be a very small part of the runtime in comparison with the call to Model.objects.get which has to build a SQL query, transmit it to the database server, and wait for the database to report that there's no such object.
Some example timings. Function f2 throws and catches an exception, while f1 implements the same functionality without using exceptions.
d = dict()
def f1():
if 0 in d: return d[0]
else: return None
def f2():
try: return d[0]
except KeyError: return None
>>> timeit(f1)
0.25134801864624023
>>> timeit(f2)
2.4589600563049316
And f3 tries to get a non-existent object from the database (which is running on the same machine) via Django's ORM:
def f3():
try:
MyModel.objects.get(id=999999)
except MyModel.DoesNotExist:
pass
This takes about 400 times longer than f2 (so long that I didn't want to wait for the default number=1000000 iterations to complete):
>>> timeit(f3, number=1000)
1.0703678131103516