Initially I thought being as specific as possible is the best way to go about it:
from django.db import IntegrityError
from psycopg2.errorcodes import UNIQUE_VIOLATION
try:
m = Model.objects.create(value=m2.old_value)
except IntegrityError as e:
if e.__cause__.pgcode != UNIQUE_VIOLATION:
raise
m = Model.objects.get(value=m2.old_value)
However later I ran into the answer that says:
SQLAlchemy is an abstraction library and if it were to raise exceptions specific to the underlying dpapi adapter, it would significantly reduce the portability of code written within it.
...
However, unless your error handling is very specific to the type raised by the dbapi layer, I'd suggest that inspecting the underlying type might be unnecessary as the SQLAlchemy exception that is raised will provide enough runtime information to do what you have to do.
django.db.IntegrityError is supposedly raised in the following cases:
INTEGRITY_CONSTRAINT_VIOLATION = '23000'
RESTRICT_VIOLATION = '23001'
NOT_NULL_VIOLATION = '23502'
FOREIGN_KEY_VIOLATION = '23503'
UNIQUE_VIOLATION = '23505'
CHECK_VIOLATION = '23514'
EXCLUSION_VIOLATION = '23P01'
I'm not sure under which circumstances these particular errors occur, but the general rule is, "don't ignore exceptions." And if I choose:
try:
m = Model.objects.create(value=m2.old_value)
except IntegrityError as e:
m = Model.objects.get(value=m2.old_value)
It sounds like I might ignore an exception I didn't mean to ignore. What would you suggest?
I don't see any problem with doing what you're doing. There's certainly no reason not to add code to get a clearer idea of what kinds of errors get thrown and are catchable under various conditions. If you're not sure if this code is right as is, then it's better to be too specific rather than not specific enough. After you've run and tested your code for a while, you'll know more about the "correctness" of what you're doing here. Maybe you'll decide later to make a change. Maybe you'll decide that you can treat larger groups of exceptions in a similar way.
One important thought though... You' aren't ignoring any errors here. There's a difference between ignoring an error, which means doing nothing different when one occurs than when it does not, and catching an error and then doing something different because that error occurred. That's not ignoring the error at all, and that's what you're doing here. So the question here is "is doing a get under all cases where a create throws an IntegrityError exception the right thing to do?". If the answer is "No", then that's where you have to test for specific error conditions and decide if the right thing is to do a get or let the error propagate because something is really wrong. That's just what you're doing. If the code you present is doing that correctly, then maybe it's exactly what you want.
Bottom line...do whatever you have to do to get the code to act right in all cases. If doing that means digging into the exception the way you are, so be it.
I'm curious...what are your thoughts on not allowing an exception to be thrown at all? That is, what about checking for the pre-existence of the record before attempting the create? Is there a reason you can't do this? Also, what about reversing the two operations? Might it make sense to do the get first, and then do the create if that fails? I'm only presenting these other options to explore all the options here, both for your benefit and that of the question itself.
I haven't had a very thorough training in python and sometimes don't know the correct way of doing things. One of these things is testing if my resultQuery returned a result or not. I find myself doing this a lot:
try:
user = User.objects.get(email=email)
except DoesNotExist:
user = User()
I don't know about python but try catches in other languages are supposed to be for exceptions and not for normal program flow. How would you do this with an if else?
I guess I want something similar to:
if request.GET.get('email','') is not None:
email = request.GET['email'])
Your exception example is most commonly the preferred way to do things. Wikipedia has quite a nice description about this:
Python style calls for the use of exceptions whenever an error
condition might arise. Rather than testing for access to a file or
resource before actually using it, it is conventional in Python to
just go ahead and try to use it, catching the exception if access is
rejected.
Exceptions are often used as an alternative to the if-block
(...). A commonly-invoked motto is EAFP, or "It is Easier to Ask
for Forgiveness than Permission."
Exceptions aren't necessarily costly in Python. Again, quoting the Wikipedia example:
if hasattr(spam, 'eggs'):
ham = spam.eggs
else:
handle_error()
... versus:
try:
ham = spam.eggs
except AttributeError:
handle_error()
These two code samples have the same effect, although there will be performance differences. When spam has the attribute eggs, the EAFP sample will run faster. When spam does not have the attribute eggs (the "exceptional" case), the EAFP sample will run slower.
Specifically for Django, I would use the exception example. It is the standard way to do things in that framework, and following standards is never a bad thing :).
You can use .filter() instead of .get(), and also make use of django's optimized queryset evaluation.
You can do
qs = User.objects.filter(email=email)
if qs :
user = qs[0]
...
I agree that catching the exception is the normal pattern here. If you do need to check for whether or not a queryset returns results, the exists method is one way to do so that I don't think has been mentioned yet. The docs run through the various performance implications, most notably that if you know the queryset will be evaluated it's better to just check the boolean interpretation of the queryset.
Django defines a class EmptyQuerySet that can be used for special implementations when strict isinstance(...) testing is necessary.
A straightforward way is as follows:
try:
user = User.objects.get(email=email)
if user:
# ...
else:
# ...
except DoesNotExist:
user = User()
Django provides a get method to directly retrieve one single Model instance or will raise an Exception. In this case the normal way is to check for Exceptions such as DoesNotExist
To retrieve entire rows from a database query as you normally would, use the user.objects.filter(...) method. The returned QuerySet instance provides you with .count(), .exists() methods.
From time to time in Python, I see the block:
try:
try_this(whatever)
except SomeException as exception:
#Handle exception
else:
return something
What is the reason for the try-except-else to exist?
I do not like that kind of programming, as it is using exceptions to perform flow control. However, if it is included in the language, there must be a good reason for it, isn't it?
It is my understanding that exceptions are not errors, and that they should only be used for exceptional conditions (e.g. I try to write a file into disk and there is no more space, or maybe I do not have permission), and not for flow control.
Normally I handle exceptions as:
something = some_default_value
try:
something = try_this(whatever)
except SomeException as exception:
#Handle exception
finally:
return something
Or if I really do not want to return anything if an exception happens, then:
try:
something = try_this(whatever)
return something
except SomeException as exception:
#Handle exception
"I do not know if it is out of ignorance, but I do not like that
kind of programming, as it is using exceptions to perform flow control."
In the Python world, using exceptions for flow control is common and normal.
Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination).
In addition, the try-except-style is used to prevent the race-conditions inherent in some of the "look-before-you-leap" constructs. For example, testing os.path.exists results in information that may be out-of-date by the time you use it. Likewise, Queue.full returns information that may be stale. The try-except-else style will produce more reliable code in these cases.
"It my understanding that exceptions are not errors, they should only
be used for exceptional conditions"
In some other languages, that rule reflects their cultural norms as reflected in their libraries. The "rule" is also based in-part on performance considerations for those languages.
The Python cultural norm is somewhat different. In many cases, you must use exceptions for control-flow. Also, the use of exceptions in Python does not slow the surrounding code and calling code as it does in some compiled languages (i.e. CPython already implements code for exception checking at every step, regardless of whether you actually use exceptions or not).
In other words, your understanding that "exceptions are for the exceptional" is a rule that makes sense in some other languages, but not for Python.
"However, if it is included in the language itself, there must be a
good reason for it, isn't it?"
Besides helping to avoid race-conditions, exceptions are also very useful for pulling error-handling outside loops. This is a necessary optimization in interpreted languages which do not tend to have automatic loop invariant code motion.
Also, exceptions can simplify code quite a bit in common situations where the ability to handle an issue is far removed from where the issue arose. For example, it is common to have top level user-interface code calling code for business logic which in turn calls low-level routines. Situations arising in the low-level routines (such as duplicate records for unique keys in database accesses) can only be handled in top-level code (such as asking the user for a new key that doesn't conflict with existing keys). The use of exceptions for this kind of control-flow allows the mid-level routines to completely ignore the issue and be nicely decoupled from that aspect of flow-control.
There is a nice blog post on the indispensibility of exceptions here.
Also, see this Stack Overflow answer: Are exceptions really for exceptional errors?
"What is the reason for the try-except-else to exist?"
The else-clause itself is interesting. It runs when there is no exception but before the finally-clause. That is its primary purpose.
Without the else-clause, the only option to run additional code before finalization would be the clumsy practice of adding the code to the try-clause. That is clumsy because it risks
raising exceptions in code that wasn't intended to be protected by the try-block.
The use-case of running additional unprotected code prior to finalization doesn't arise very often. So, don't expect to see many examples in published code. It is somewhat rare.
Another use-case for the else-clause is to perform actions that must occur when no exception occurs and that do not occur when exceptions are handled. For example:
recip = float('Inf')
try:
recip = 1 / f(x)
except ZeroDivisionError:
logging.info('Infinite result')
else:
logging.info('Finite result')
Another example occurs in unittest runners:
try:
tests_run += 1
run_testcase(case)
except Exception:
tests_failed += 1
logging.exception('Failing test case: %r', case)
print('F', end='')
else:
logging.info('Successful test case: %r', case)
print('.', end='')
Lastly, the most common use of an else-clause in a try-block is for a bit of beautification (aligning the exceptional outcomes and non-exceptional outcomes at the same level of indentation). This use is always optional and isn't strictly necessary.
What is the reason for the try-except-else to exist?
A try block allows you to handle an expected error. The except block should only catch exceptions you are prepared to handle. If you handle an unexpected error, your code may do the wrong thing and hide bugs.
An else clause will execute if there were no errors, and by not executing that code in the try block, you avoid catching an unexpected error. Again, catching an unexpected error can hide bugs.
Example
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
return something
The "try, except" suite has two optional clauses, else and finally. So it's actually try-except-else-finally.
else will evaluate only if there is no exception from the try block. It allows us to simplify the more complicated code below:
no_error = None
try:
try_this(whatever)
no_error = True
except SomeException as the_exception:
handle(the_exception)
if no_error:
return something
so if we compare an else to the alternative (which might create bugs) we see that it reduces the lines of code and we can have a more readable, maintainable, and less buggy code-base.
finally
finally will execute no matter what, even if another line is being evaluated with a return statement.
Broken down with pseudo-code
It might help to break this down, in the smallest possible form that demonstrates all features, with comments. Assume this syntactically correct (but not runnable unless the names are defined) pseudo-code is in a function.
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle_SomeException(the_exception)
# Handle a instance of SomeException or a subclass of it.
except Exception as the_exception:
generic_handle(the_exception)
# Handle any other exception that inherits from Exception
# - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit
# Avoid bare `except:`
else: # there was no exception whatsoever
return something()
# if no exception, the "something()" gets evaluated,
# but the return will not be executed due to the return in the
# finally block below.
finally:
# this block will execute no matter what, even if no exception,
# after "something" is eval'd but before that value is returned
# but even if there is an exception.
# a return here will hijack the return functionality. e.g.:
return True # hijacks the return in the else clause above
It is true that we could include the code in the else block in the try block instead, where it would run if there were no exceptions, but what if that code itself raises an exception of the kind we're catching? Leaving it in the try block would hide that bug.
We want to minimize lines of code in the try block to avoid catching exceptions we did not expect, under the principle that if our code fails, we want it to fail loudly. This is a best practice.
It is my understanding that exceptions are not errors
In Python, most exceptions are errors.
We can view the exception hierarchy by using pydoc. For example, in Python 2:
$ python -m pydoc exceptions
or Python 3:
$ python -m pydoc builtins
Will give us the hierarchy. We can see that most kinds of Exception are errors, although Python uses some of them for things like ending for loops (StopIteration). This is Python 3's hierarchy:
BaseException
Exception
ArithmeticError
FloatingPointError
OverflowError
ZeroDivisionError
AssertionError
AttributeError
BufferError
EOFError
ImportError
ModuleNotFoundError
LookupError
IndexError
KeyError
MemoryError
NameError
UnboundLocalError
OSError
BlockingIOError
ChildProcessError
ConnectionError
BrokenPipeError
ConnectionAbortedError
ConnectionRefusedError
ConnectionResetError
FileExistsError
FileNotFoundError
InterruptedError
IsADirectoryError
NotADirectoryError
PermissionError
ProcessLookupError
TimeoutError
ReferenceError
RuntimeError
NotImplementedError
RecursionError
StopAsyncIteration
StopIteration
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
UnicodeError
UnicodeDecodeError
UnicodeEncodeError
UnicodeTranslateError
Warning
BytesWarning
DeprecationWarning
FutureWarning
ImportWarning
PendingDeprecationWarning
ResourceWarning
RuntimeWarning
SyntaxWarning
UnicodeWarning
UserWarning
GeneratorExit
KeyboardInterrupt
SystemExit
A commenter asked:
Say you have a method which pings an external API and you want to handle the exception at a class outside the API wrapper, do you simply return e from the method under the except clause where e is the exception object?
No, you don't return the exception, just reraise it with a bare raise to preserve the stacktrace.
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise
Or, in Python 3, you can raise a new exception and preserve the backtrace with exception chaining:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise DifferentException from the_exception
I elaborate in my answer here.
Python doesn't subscribe to the idea that exceptions should only be used for exceptional cases, in fact the idiom is 'ask for forgiveness, not permission'. This means that using exceptions as a routine part of your flow control is perfectly acceptable, and in fact, encouraged.
This is generally a good thing, as working this way helps avoid some issues (as an obvious example, race conditions are often avoided), and it tends to make code a little more readable.
Imagine you have a situation where you take some user input which needs to be processed, but have a default which is already processed. The try: ... except: ... else: ... structure makes for very readable code:
try:
raw_value = int(input())
except ValueError:
value = some_processed_value
else: # no error occured
value = process_value(raw_value)
Compare to how it might work in other languages:
raw_value = input()
if valid_number(raw_value):
value = process_value(int(raw_value))
else:
value = some_processed_value
Note the advantages. There is no need to check the value is valid and parse it separately, they are done once. The code also follows a more logical progression, the main code path is first, followed by 'if it doesn't work, do this'.
The example is naturally a little contrived, but it shows there are cases for this structure.
See the following example which illustrate everything about try-except-else-finally:
for i in range(3):
try:
y = 1 / i
except ZeroDivisionError:
print(f"\ti = {i}")
print("\tError report: ZeroDivisionError")
else:
print(f"\ti = {i}")
print(f"\tNo error report and y equals {y}")
finally:
print("Try block is run.")
Implement it and come by:
i = 0
Error report: ZeroDivisionError
Try block is run.
i = 1
No error report and y equals 1.0
Try block is run.
i = 2
No error report and y equals 0.5
Try block is run.
Is it a good practice to use try-except-else in python?
The answer to this is that it is context dependent. If you do this:
d = dict()
try:
item = d['item']
except KeyError:
item = 'default'
It demonstrates that you don't know Python very well. This functionality is encapsulated in the dict.get method:
item = d.get('item', 'default')
The try/except block is a much more visually cluttered and verbose way of writing what can be efficiently executing in a single line with an atomic method. There are other cases where this is true.
However, that does not mean that we should avoid all exception handling. In some cases it is preferred to avoid race conditions. Don't check if a file exists, just attempt to open it, and catch the appropriate IOError. For the sake of simplicity and readability, try to encapsulate this or factor it out as apropos.
Read the Zen of Python, understanding that there are principles that are in tension, and be wary of dogma that relies too heavily on any one of the statements in it.
You should be careful about using the finally block, as it is not the same thing as using an else block in the try, except. The finally block will be run regardless of the outcome of the try except.
In [10]: dict_ = {"a": 1}
In [11]: try:
....: dict_["b"]
....: except KeyError:
....: pass
....: finally:
....: print "something"
....:
something
As everyone has noted using the else block causes your code to be more readable, and only runs when an exception is not thrown
In [14]: try:
dict_["b"]
except KeyError:
pass
else:
print "something"
....:
Just because no-one else has posted this opinion, I would say
avoid else clauses in try/excepts because they're unfamiliar to most people
Unlike the keywords try, except, and finally, the meaning of the else clause isn't self-evident; it's less readable. Because it's not used very often, it'll cause people that read your code to want to double-check the docs to be sure they understand what's going on.
(I'm writing this answer precisely because I found a try/except/else in my codebase and it caused a wtf moment and forced me to do some googling).
So, wherever I see code like the OP example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
# do some more processing in non-exception case
return something
I would prefer to refactor to
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
return # <1>
# do some more processing in non-exception case <2>
return something
<1> explicit return, clearly shows that, in the exception case, we are finished working
<2> as a nice minor side-effect, the code that used to be in the else block is dedented by one level.
Whenever you see this:
try:
y = 1 / x
except ZeroDivisionError:
pass
else:
return y
Or even this:
try:
return 1 / x
except ZeroDivisionError:
return None
Consider this instead:
import contextlib
with contextlib.suppress(ZeroDivisionError):
return 1 / x
This is my simple snippet on howto understand try-except-else-finally block in Python:
def div(a, b):
try:
a/b
except ZeroDivisionError:
print("Zero Division Error detected")
else:
print("No Zero Division Error")
finally:
print("Finally the division of %d/%d is done" % (a, b))
Let's try div 1/1:
div(1, 1)
No Zero Division Error
Finally the division of 1/1 is done
Let's try div 1/0
div(1, 0)
Zero Division Error detected
Finally the division of 1/0 is done
I'm attempting to answer this question in a slightly different angle.
There were 2 parts of the OP's question, and I add the 3rd one, too.
What is the reason for the try-except-else to exist?
Does the try-except-else pattern, or the Python in general, encourage using exceptions for flow control?
When to use exceptions, anyway?
Question 1: What is the reason for the try-except-else to exist?
It can be answered from a tactical standpoint. There is of course reason for try...except... to exist. The only new addition here is the else... clause, whose usefulness boils down to its uniqueness:
It runs an extra code block ONLY WHEN there was no exception happened in the try... block.
It runs that extra code block, OUTSIDE of the try... block (meaning any potential exceptions happen inside the else... block would NOT be caught).
It runs that extra code block BEFORE the final... finalization.
db = open(...)
try:
db.insert(something)
except Exception:
db.rollback()
logging.exception('Failing: %s, db is ROLLED BACK', something)
else:
db.commit()
logging.info(
'Successful: %d', # <-- For the sake of demonstration,
# there is a typo %d here to trigger an exception.
# If you move this section into the try... block,
# the flow would unnecessarily go to the rollback path.
something)
finally:
db.close()
In the example above, you can't move that successful log line into behind the finally... block. You can't quite move it into inside the try... block, either, due to the potential exception inside the else... block.
Question 2: does Python encourage using exceptions for flow control?
I found no official written documentation to support that claim. (To readers who would disagree: please leave comments with links to evidences you found.) The only vaguely-relevant paragraph that I found, is this EAFP term:
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.
Such paragraph merely described that, rather than doing this:
def make_some_noise(speaker):
if hasattr(speaker, "quack"):
speaker.quack()
we would prefer this:
def make_some_noise(speaker):
try:
speaker.quack()
except AttributeError:
logger.warning("This speaker is not a duck")
make_some_noise(DonaldDuck()) # This would work
make_some_noise(DonaldTrump()) # This would trigger exception
or potentially even omitting the try...except:
def make_some_noise(duck):
duck.quack()
So, the EAFP encourages duck-typing. But it does not encourage using exceptions for flow control.
Question 3: In what situation you should design your program to emit exceptions?
It is a moot conversation on whether it is anti-pattern to use exception as control flow. Because, once a design decision is made for a given function, its usage pattern would also be determined, and then the caller would have no choice but to use it that way.
So, let's go back to the fundamentals to see when a function would better produce its outcome via returning a value or via emitting exception(s).
What is the difference between the return value and the exception?
Their "blast radius" are different. Return value is only available to the immediate caller; exception can be automatically relayed for unlimited distance until it is caught.
Their distribution patterns are different. Return value is by definition one piece of data (even though you could return a compound data type such as a dictionary or a container object, it is still technically one value).
The exception mechanism, on the contrary, allows multiple values (one at a time) to be returned via their respective dedicate channel. Here, each except FooError: ... and except BarError: ... block is considered as its own dedicate channel.
Therefore, it is up to each different scenario to use one mechanism that fits well.
All normal cases should better be returned via return value, because the callers would most likely need to use that return value immediately. The return-value approach also allows nesting layers of callers in a functional programming style. The exception mechanism's long blast radius and multiple channels do not help here.
For example, it would be unintuitive if any function named get_something(...) produces its happy path result as an exception. (This is not really a contrived example. There is one practice to implement BinaryTree.Search(value) to use exception to ship the value back in the middle of a deep recursion.)
If the caller would likely forget to handle the error sentinel from the return value, it is probably a good idea to use exception's characterist #2 to save caller from its hidden bug. A typical non-example would be the position = find_string(haystack, needle), unfortunately its return value of -1 or null would tend to cause a bug in the caller.
If the error sentinel would collide with a normal value in the result namespace, it is almost certain to use an exception, because you'd have to use a different channel to convey that error.
If the normal channel i.e. the return value is already used in the happy-path, AND the happy-path does NOT have sophisicated flow control, you have no choice but to use exception for flow control. People keep talking about how Python uses StopIteration exception for iteration termination, and use it to kind of justify "using exception for flow control". But IMHO this is only a practical choice in a particular situation, it does not generalize and glorify "using exception for flow control".
At this point, if you already make a sound decision on whether your function get_stock_price() would produce only return-value or also raise exceptions, or if that function is provided by an existing library so that its behavior has long be decided, you do not have much choice in writing its caller calculate_market_trend(). Whether to use get_stock_price()'s exception to control the flow in your calculate_market_trend() is merely a matter of whether your business logic requires you to do so. If yes, do it; otherwise, let the exception bubble up to a higher level (this utilizes the characteristic #1 "long blast radius" of exception).
In particular, if you are implementing a middle-layer library Foo and you happen to be making a dependency on lower-level library Bar, you would probably want to hide your implementation detail, by catching all Bar.ThisError, Bar.ThatError, ..., and map them into Foo.GenericError. In this case, the long blast radius is actually working against us, so you might hope "only if library Bar were returning its errors via return values". But then again, that decision has long been made in Bar, so you can just live with it.
All in all, I think whether to use exception as control flow is a moot point.
OP, YOU ARE CORRECT. The else after try/except in Python is ugly. it leads to another flow-control object where none is needed:
try:
x = blah()
except:
print "failed at blah()"
else:
print "just succeeded with blah"
A totally clear equivalent is:
try:
x = blah()
print "just succeeded with blah"
except:
print "failed at blah()"
This is far clearer than an else clause. The else after try/except is not frequently written, so it takes a moment to figure what the implications are.
Just because you CAN do a thing, doesn't mean you SHOULD do a thing.
Lots of features have been added to languages because someone thought it might come in handy. Trouble is, the more features, the less clear and obvious things are because people don't usually use those bells and whistles.
Just my 5 cents here. I have to come along behind and clean up a lot of code written by 1st-year out of college developers who think they're smart and want to write code in some uber-tight, uber-efficient way when that just makes it a mess to try and read / modify later. I vote for readability every day and twice on Sundays.
I was just looking at the Python FAQ because it was mentioned in another question. Having never really looked at it in detail before, I came across this question: “How fast are exceptions?”:
A try/except block is extremely efficient. Actually catching an exception is expensive. In versions of Python prior to 2.0 it was common to use this idiom:
try:
value = mydict[key]
except KeyError:
mydict[key] = getvalue(key)
value = mydict[key]
I was a little bit surprised about the “catching an exception is expensive” part. Is this referring only to those except cases where you actually save the exception in a variable, or generally all excepts (including the one in the example above)?
I’ve always thought that using such idioms as shown would be very pythonic, especially as in Python “it is Easier to Ask Forgiveness than it is to get Permission”. Also many answers on SO generally follow this idea.
Is the performance for catching Exceptions really that bad? Should one rather follow LBYL (“Look before you leap”) in such cases?
(Note that I’m not directly talking about the example from the FAQ; there are many other examples where you just look out for an exception instead of checking the types before.)
Catching exceptions is expensive, but exceptions should be exceptional (read, not happen very often). If exceptions are rare, try/catch is faster than LBYL.
The following example times a dictionary key lookup using exceptions and LBYL when the key exists and when it doesn't exist:
import timeit
s = []
s.append('''\
try:
x = D['key']
except KeyError:
x = None
''')
s.append('''\
x = D['key'] if 'key' in D else None
''')
s.append('''\
try:
x = D['xxx']
except KeyError:
x = None
''')
s.append('''\
x = D['xxx'] if 'xxx' in D else None
''')
for i,c in enumerate(s,1):
t = timeit.Timer(c,"D={'key':'value'}")
print('Run',i,'=',min(t.repeat()))
Output
Run 1 = 0.05600167960596991 # try/catch, key exists
Run 2 = 0.08530091918578364 # LBYL, key exists (slower)
Run 3 = 0.3486251291120652 # try/catch, key doesn't exist (MUCH slower)
Run 4 = 0.050621117060586585 # LBYL, key doesn't exist
When the usual case is no exception, try/catch is "extremely efficient" when compared to LBYL.
The cost depends on implementation, obviously, but I wouldn't worry about it. It's unlikely going to matter, anyway. Standard protocols raise exceptions in strangest of places (think StopIteration), so you're surrounded with raising and catching whether you like it or not.
When choosing between LBYL and EAFP, worry about readability of the code, instead of focusing on micro-optimisations. I'd avoid type-checking if possible, as it might reduce the generality of the code.
If the case where the key is not found is more than exceptional, I would suggest using the 'get' method, which provide a constant speed in all cases :
s.append('''\
x = D.get('key', None)
''')
s.append('''\
x = D.get('xxx', None)
''')
In trying to eliminate potential race condition in a python module I wrote to monitor some specialized workflows, I learned about python's "easier to ask forgiveness than permission" (EAFP) coding style, and I'm now raising lots of custom exceptions with try/except blocks where I used to use if/thens.
I'm new to python and This EAFP style makes sense logically and seems make my code more robust, but something about this feels way overboard. Is is bad practice to define one or more exceptions per method?
These custom exceptions tend to be useful only to a single method and, while it feels like a functionally correct solution, it seems like a lot of code to maintain.
Here a sample method for example:
class UploadTimeoutFileMissing(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
class UploadTimeoutTooSlow(Exception):
def __init__(self, value):
self.parameter = value
def __str__(self):
return repr(self.parameter)
def check_upload(file, timeout_seconds, max_age_seconds, min_age_seconds):
timeout = time.time() + timeout_seconds
## Check until file found or timeout
while (time.time() < timeout):
time.sleep(5)
try:
filetime = os.path.getmtime(file)
filesize = os.path.getsize(file)
except OSError:
print "File not found %s" % file
continue
fileage = time.time() - filetime
## Make sure file isn't pre-existing
if fileage > max_age_seconds:
print "File too old %s" % file
continue
## Make sure file isn't still uploading
elif fileage <= min_age_seconds:
print "File too new %s" % file
continue
return(filetime, filesize)
## Timeout
try:
filetime
filesize
raise UploadTimeoutTooSlow("File still uploading")
except NameError:
raise UploadTimeoutFileMissing("File not sent")
define one or more exceptions per method
If you mean that the exception is actually defined per method as in "within the method body", then yes. That is bad practice. This is true also if you define two exceptions that would relate to the same error but you create two because two different methods raise them.
If you ask whether it is bad practice to raise more than one exception per method, then no, that is good practice. And if the errors are not of the same category, it's perfectly ok to define several exceptions per module.
In general, for larger modules you will define more than one exception. If you would work on some arithmetic library and you would define a ZeroDivisionError and an OverflowError (if they weren't already defined in python, because you can of course re-use those) that would be perfectly fine.
Is is bad practice to define one or more exceptions per method?
Yes.
One per module is more typical. It depends, of course, on the detailed semantics. The question boils down to this: "What will you really try to catch?"
If you're never going to use except ThisVeryDetailedException: in your code, then your very detailed exception isn't very helpful.
If you can do this: except Error as e: if e.some_special_case for the very few times it matters, then you can easily simplify to one exception per module and handle your special cases as attributes of the exception rather than different types of exceptions.
The common suggestions (one per module, named Error) means that your code will often look like this.
try:
something
except some_module.Error as e:
carry on
This gives you a nice naming convention: module.Error. This covers numerous sins.
On an unrelated note, if you think you've got "potential race condition" you should probably redesign things correctly or stop trying to use threads or switch to multiprocessing. If you use multiprocessing, you'll find that it's very easy to avoid race conditions.
I'm going to weigh in on this because custom exceptions are dear to my heart. I'll explain my circumstances and the reader can weigh them against their own.
I'm the pipeline architect for a visual effects company - most of what I do involves developing what I call the "Facility API" - it's a system of a great many modules which handle everything from locating things on the filesystem, managing module/tool/project configuration, to handling datatypes from various CG applications to enable collaboration.
I go to great lengths to try to ensure that Python's built-in exceptions never bubble up. Since our developers will be relying on an ecosystem of existing modules to build their own tools on top of, having the API let a generic IOError escape is counterproductive - especially since the calling routine might not even be aware that it's reading the filesystem (abstraction is a beautiful thing). If the underlying module is unable to express something meaningful about that error, more work needs to be done.
My approach to solving this is to create a facility exception class from which all other facility exceptions are derived. There are subclasses of that for specific types of task or specific host applications - which allows me to customize error handling (for instance, exceptions raised in Maya will launch a UI to aid in troubleshooting since the usual exception would be raised in an inconspicuous console and would often be missed).
All sorts of reporting is built into the facility exception class - exceptions don't appear to a user without also being reported internally. For a range of exceptions, I get an IM any time one is raised. Others simply report quietly into a database that I can query for recent (daily or weekly) reports. Each links to EXTENSIVE data captured from the user session - typically including a screenshot, stack trace, system configuration, and a whole lot more. This means I can effectively troubleshoot problems before they're reported - and have more information at my fingertips than most users are likely able to provide.
Very fine gradations in purpose are discouraged - the exceptions accept passed values (sometimes even a dictionary instead of a string, if we want to provide plenty of data for troubleshooting) to provide with their formatted output.
So no - I don't think defining an exception or two per module is unreasonable - but they need to be meaningful and add something to the project. If you're just wrapping an IOError to raise MyIOError("I got an IO error!"), then you may want to rethink that.
I don't think it's necessary to have an extremely specific exception for every possible scenario. A single UploadTimeoutError would probably be fine, and you can just customize the exception string - that's what the strings are for, after all. Note how python doesn't have a separate exception for every possible type of syntax error, just a general SyntaxError.
Also - is it actually necessary to define the __init__ and __str__ methods for each of your custom exceptions? As far as I can tell, if you're not implementing any unusual behavior, you don't need to add any code:
>>> class MyException(Exception): pass
...
>>> raise MyException("oops!")
Traceback (most recent call last):
File "<ipython console>", line 1, in <module>
MyException: oops!
>>> str(MyException("oops!"))
'oops!'