I have been told to design a new API in my company, and I am facing a dilemma when it comes to coding practices.
My API have to do several checks before they can be run, and often require multiple levels of functions to run.
Everything is fine until here. But most of my check (sub to sub to sub) function require the main API to return, without doing anything. Almost all of my check function have to return some data which is used by the next check function, and that is where my issue is. Because of this kind of structure, I have to return a status at the end of every check function along with the processed data, and after the function is called, I have to check the status before going to the next function.
Sample code:
def check1a():
if some_process():
return True, data_positive
return False, data_negative
#data_positive and data_negative cannot be used to identify whether the check passed or not.
def check1():
stats,data = check1a()
if not status:
return False, data
status, data = check1b(data)
if not status:
return False, data
status, data = check1c(data)
if not status:
return False, data
return status, data
def mainAPI():
status, data = check1(data)
if not status:
return data
status, data = check2(data)
if not status:
return data
status, data = check3()
if not status:
return "Failed"
return data
Being a religious follower of the "DRY" concept, if feel using exceptions to run the code in the following manner would be best.
def check1a():
if some_process():
return data_positive
exception1a = Exception("Error in check 1 a")
exception.data = data_negative
raise exception
def check1():
data = check1a()
data = check1b(data)
data = check1c(data)
return data
def mainAPI():
try:
data = check1(data)
data = check2(data)
data = check3(data)
return data
except Exception as e:
return e.data #I know exceptions don't always have data, but this is an illustration of what I think I should implement
Unfortunately raising an exception in the code to do implement this kind of working is kind of shunned upon at my company.
So here are my questions.
Is it really wrong to use Exceptions in this manner?
Is there a known downside to using Exceptions in this manner?
Is there a pythonic (or even a general coding) method which allows me to implement my code, and does not require me to stop following DRY.
This may not be a great answer, someone else way be able to help more.
Try-Exception:
This an opinion based topic. If they say they don't like you using try exception like this thn they probably don't believe in the "Better ask forgiveness than permission" principle.
That being said, throwing an Exception isn't bad; HOWEVER catching a general Exception is considered bad. If a piece of software is not running as desired (i.e. in some unknown way) you want it to fail thus you should only catch the specific Exception you want to catch.
You can find a an ample list of viable exceptions here, just pick one that seems reasonable and use it: Python Programming Exceptions
If you don't want catch one of the preexisting exceptions you can always make your own:
class MyAPIException(Exception):
def __init___(self, val):
self.val = val
Exception.__init__(self, "APIException with with arguments {0}".format(self.val))
def do_stuff(a,b,c):
raise MyAPIException({
'a' : a,
'b' : b,
'c' : c,
})
try:
do_stuff(1, 2, 3)
except MyAPIException as e:
print("API Exception:", e)
Alternative:
Another way you could help with DRY could be to use a list to make your calls.
def check1():
# List of functions you want to call in order
calls = [check1a, check1b, check1c]
for index, call in enumerate(calls):
# If it is the first function we will not pass any data
status, data = call() if index == 0 else call(data)
if not status:
return False, data
return status, data
This implementation also makes it easy to implement it as a generator if you wanted to return the result of each function call.
The answer by Error - Syntactical Remorse is a good one. Use exceptions that are defined by your API to handle control flow. To further expand on the answer, your API does not need to expose such exceptions, they can be caught by your internal functions as part of the way to handle control flow:
class MyAPIException(Exception):
pass
class SensorMiscalibrated(MyAPIException):
pass
def mainAPI():
try:
data = check1(data)
data = check2(data)
data = check3(data)
return True, data
except SensorMiscalibrated as e:
return False, data
What's neat about this is that any other exception raised by check1, check2… regarding file permissions or process errors will just bubble up and be ignored by the exception handler.
Exceptions are encouraged in Python and they do not introduce a performance penalty unlike other languages that implement them in a different way.
Related
I have two functions which do the same thing basically but return differently in case of exceptions. Which one is the preferred approach?
Approach 1:
def f1():
try:
data = some_random_function()
return data["success"]
except Exception as error:
print(error)
return "failure"
Approach 2:
def f2():
try:
data = some_random_function()
return data["success"]
except Exception as error:
print(error)
return "failure"
Although both approaches are equivalent, I would recommend approach 1. It makes much clearer that 'failure' is returned only when there is an exception. Someone reading approach 2 may get a first impression that it always returns 'failure'. This is strictly from a clean code perspective.
Although both the approaches work the same way i.e. return data["success"] in case of an exception-free code and return "failure" otherwise, the two approaches differ in the way they convey the information :
A1- Return "failure" in case of an exception
A2- Return "failure" as a default return value.
So, it's my suggestion that you use the second approach as it is more clear and explicit.
As you have a generic catch-all for errors, both approaches are equivalent, because in case of an exception you'll enter the print "failure" branch.
I guess it comes down to matter of taste and maybe future maintainability, i.e. when you add more except branches to handle some errors differently, to choose approach 2.
Its all down to personal preferences. For me, approach 1 is much clearer than two but both are good solutions. I think its more than just individual preference, its more of the code base you are working with. If you are working with a team, look into code with similar structure, see what other's have done. A uniform codebase is much valuable than fancy code here and there.
Also, for languages like JavaScript and python, which have functional scope, I want to put forward a third option:
def f3():
try:
data = some_random_function()
message = "success"
except Exception as error:
print(error)
message = "failure"
return message
Lets say I have a function myFunc defined as
def myFunc(value):
return value if isinstance(value, int) else None
Now wherever in my project I use myFunc the enclosing funciton should return automatically if the value returned from myFunc is None and should continue if some integer value is returned
For example:
def dumbFunc():
# some code
# goes here..
result = myFunc('foo')
# some code
# goes here..
This funciton should automatically behave like..
def dumbFunc():
# some code
# goes here..
result = myFunc('foo')
if not result:
return
# some code
# goes here..
PS - I don't know whether this thing even possible or not.
This is simply not possible.
Apart from exceptions, you cannot give a called function the ability to impact the control flow of the calling scope. So a function call foo() can never interrupt the control flow without throwing an exception. As a consumer of the function, you (the calling function) always have the responsibility yourself to handle such cases and decide about your own control flow.
And it is a very good idea to do it like that. Just the possibility that a function call might interrupt my control flow without having a possibility to react on it first sounds like a pure nightmare. Just alone for the ability to release and cleanup resources, it is very important that the control flow is not taken from me.
Exceptions are the notable exception from this, but of course this is a deeply rooted language feature which also still gives me the ability to act upon it (by catching exceptions, and even by having finally blocks to perform clean up tasks). Exceptions are deliberately not silent but very loud, so that interruptions from the deterministic control flow are clearly visible and have a minimum impact when properly handled.
But having a silent feature that does neither give any control nor feedback would be just a terrible idea.
If myFunc is used at 100 places in my project, everywhere I need to put an if condition after it.
If your code is like that that you could just return nothing from any function that calls myFunc without having to do anything, then either you are building an unrealistic fantasy project, or you simply are not aware of the implications this can have to the calling code of the functions that would be returned that way.
ok, I'll bite.
on the one hand, this isn't really possible. if you want to check something you have to have a line in your code that checks it.
there are a few ways you could achieve something like this, but i think you may have already found the best one.
you already have this function:
def myFunc(value):
return value if isinstance(value, int) else None
I would probably have done:
def myFunc(value):
return isinstance(value, int)
but either way you could use it:
def dumb_func():
value = do_something()
if myFunc(value):
return
do_more()
return value
alternately you could use try and except
I would raise a TypeError, seeing as that seems to be what you are checking:
def myFunc(value):
if not isinstance(value, int):
raise TypeError('myFunc found that {} is not an int'.format(value))
then you can use this as such
def dumb_func():
value = do_something()
try:
myFunc(value):
Except TypeError as e:
print e # some feedback that this has happened, but no error raised
return
do_more()
return value
for bonus points you could define a custom exception (which is safer because then when you catch that specific error you know it wasn't raised by anything else in your code, also if you did that you could be lazier eg:)
Class CustomTypeError(TypeError):
pass
def dumb_func():
try:
value = do_something()
myFunc(value):
do_more()
return value
Except CustomTypeError as e:
print e # some feedback that this has happened, but no error raised
return
but none of this gets around the fact that if you want to act based on the result of a test, you have to check that result.
Python has a ternary conditional operator, and the syntax you used is right, so this will work:
def myFunc(value):
return value if isinstance(value, int) else None
def dumbFunc():
print("Works?")
result = myFunc(5)
print(result)
dumbFunc()
Result:
Works?
5
I want the function to return automatically in that case
This is not possible. To do that, you have to check the return value of myFunc() and act upon it.
PS: You could do that with a goto statement, but Python, fortunately, doesn't support this functionality.
Does Python has a feature that allows one to evaluate a function or expression and if the evaluation fails (an exception is raised) return a default value.
Pseudo-code:
evaluator(function/expression, default_value)
The evaluator will try to execute the function or expression and return the result is the execution is successful, otherwise the default_value is returned.
I know I create a user defined function using try and except to achieve this but I want to know if the batteries are already included before going off and creating a custom solution.
In order to reuse code, you can create a decorating function (that accepts a default value) and decorate your functions with it:
def handle_exceptions(default):
def wrap(f):
def inner(*a):
try:
return f(*a)
except Exception, e:
return default
return inner
return wrap
Now let's see an example:
#handle_exceptions("Invalid Argument")
def test(num):
return 15/num
#handle_exceptions("Input should be Strings only!")
def test2(s1, s2):
return s2 in s1
print test(0) # "Invalid Argument"
print test(15) # 1
print test2("abc", "b") # True
print test2("abc", 1) # Input should be Strings only!
No, the standard way to do this is with try... except.
There is no mechanism to hide or suppress any generic exception within a function. I suspect many Python users would consider indiscriminate use of such a function to be un-Pythonic for a couple reasons:
It hides information about what particular exception occurred. (You might not want to handle all exceptions, since some could come from other libraries and indicate conditions that your program can't recover from, like running out of disk space.)
It hides the fact that an exception occurred at all; the default value returned in case of an exception might coincide with a valid non-default value. (Sometimes reasonable, sometimes not really so.)
One of the principles of the Pythonic philosophy, I believe, is that "explicit is better than implicit," so Python generally avoids automatic type casting and error recovery, which are features of more "implicit- friendly"languages like Perl.
Although the try... except form can be a bit verbose, in my opinion it has a lot of advantages in terms of clearly showing where an exception may occur and what the control flow is around that exception.
When I run this code:
from nltk import NaiveBayesClassifier,classify
import USSSALoader
import random
class genderPredictor():
def getFeatures(self):
if self._loadNames() != None:
maleNames,femaleNames=self._loadNames()
else:
print "There is no training file."
return
featureset = list()
for nameTuple in maleNames:
features = self._nameFeatures(nameTuple[0])
featureset.append((features,'M'))
for nameTuple in femaleNames:
features = self._nameFeatures(nameTuple[0])
featureset.append((features,'F'))
return featureset
def trainAndTest(self,trainingPercent=0.80):
featureset = self.getFeatures()
random.shuffle(featureset)
name_count = len(featureset)
cut_point=int(name_count*trainingPercent)
train_set = featureset[:cut_point]
test_set = featureset[cut_point:]
self.train(train_set)
return self.test(test_set)
def classify(self,name):
feats=self._nameFeatures(name)
return self.classifier.classify(feats)
def train(self,train_set):
self.classifier = NaiveBayesClassifier.train(train_set)
return self.classifier
def test(self,test_set):
return classify.accuracy(self.classifier,test_set)
def getMostInformativeFeatures(self,n=5):
return self.classifier.most_informative_features(n)
def _loadNames(self):
return USSSALoader.getNameList()
def _nameFeatures(self,name):
name=name.upper()
return {
'last_letter': name[-1],
'last_two' : name[-2:],
'last_is_vowel' : (name[-1] in 'AEIOUY')
}
if __name__ == "__main__":
gp = genderPredictor()
accuracy=gp.trainAndTest()
And self._loadNames() returns None, I got this error (from random imported module):
shuffle C:\Python27\lib\random.py 285
TypeError: object of type 'NoneType' has no len()
This happend because despite I put a return statment in getFeatures(self), the flow jumps into the next class method (which is trainAndTest(self,trainingPercent=0.80)) which calls the random module (random.shuffle(featureset)).
So, I'd like to know: how to stop the procedure flow not only in the getFeatures(self) method, but in the entire class that contains it?
By the way, thanks Stephen Holiday for sharing the code.
This happend because despite I put a return statment in
getFeatures(self), the flow jumps into the next class method (which is
trainAndTest(self,trainingPercent=0.80)) which calls the random module
(random.shuffle(featureset)).
An important thing to remember is that None is a perfectly valid value. The return statement in your getFeatures() is doing exactly what it is told and returning the valid value. Only an exceptional situation, or you explicitly, will stop that flow.
Instead of asking how you can "return from the class", what you might want to look into is checking the return values of functions you call and making sure its what you expect before you proceed. There are two places you could do this:
def trainAndTest(self,trainingPercent=0.80):
featureset = self.getFeatures()
...
def _loadNames(self):
return USSSALoader.getNameList()
In the first spot, you could check if featureset is None, and react if it is None.
In the second spot, instead of blindly returning, you could check it first and react there.
Secondly. you have the option of raising exceptions. Exceptions are a situation where the code has encountered an error and can't continue. It is then the responsibility of the calling function to either handle it or let it ride up the chain. If nothing handles the exception, your application will crash. As you can see, you are getting an exception being raised from the random class because you are allowing a None to make its way into the shuffle call.
names = USSSALoader.getNameList()
if names is None:
# raise an exception?
# do something else?
# ask the user to do something?
The question at that point is, what do you want your program to do at that moment when it happens to get a None instead of a valid list? Do you want an exception similar to the one being raised by random, but more helpful and specific to your application? Or maybe you just want to call some other method that gets a default list. Is not having the names list even a situation where your application do anything other than exit? That would be an unrecoverable situation.
names = USSSALoader.getNameList()
if names is None:
raise ValueError("USSSALoader didn't return any "
"valid names! Can't continue!")
Update
From your comment, I wanted to add the specific handling you wanted. Python has a handful of built in exception types to represent various circumstances. The one you would most likely want to raise is an IOError, indicating that the file could not be found. I assume "file" means whatever file USSSALoader.getNameList() needs to use and can't find.
names = USSSALoader.getNameList()
if names is None:
raise IOError("No USSSALoader file found")
At this point, unless some function higher up the calling chain handles it, your program will terminate with a traceback error.
There is nothing like "return from the entire class". You need to organize your code so that return values are valid in the functions that get them. Those functions can test the value to determine what to do next. The class boundaries have no effect on program flow, just the namespacing of methods.
Generally what you would do here is check for validity after you call the function, e.g.:
featureset = self.getFeatures()
if not featureset:
# You could log an error message if you expected to get something, etc.
return
I have some test cases. The test cases rely on data which takes time to compute. To speed up testing, I've cached the data so that it doesn't have to be recomputed.
I now have foo(), which looks at the cached data. I can't tell ahead of time what it will look at, as that depends a lot on the test case.
If a test case fails cause it doesn't find the right cached data, I don't want it to fail - I want it to compute the data and then try again. I also don't know what exception in particular it will throw cause of missing data.
My code right now looks like this:
if cacheExists:
loadCache()
dataComputed = False
else:
calculateData()
dataComputed = True
try:
foo()
except:
if not dataComputed:
calculateData()
dataComputed = True
try:
foo()
except:
#error handling code
else:
#the same error handling code
What's the best way to re-structure this code?
I disagree with the key suggestion in the existing answers, which basically boils down to treating exceptions in Python as you would in, say, C++ or Java -- that's NOT the preferred style in Python, where often the good old idea that "it's better to ask forgiveness than permission" (attempt an operation and deal with the exception, if any, rather than obscuring your code's main flow and incurring overhead by thorough preliminary checks). I do agree with Gabriel that a bare except is hardly ever a good idea (unless all it does is some form of logging followed by a raise to let the exception propagate). So, say you have a tuple with all the exception types that you do expect and want to handle the same way, say:
expected_exceptions = KeyError, AttributeError, TypeError
and always use except expected_exceptions: rather than bare except:.
So, with that out of the way, one slightly less-repetitious approach to your needs is:
try:
foo1()
except expected_exceptions:
try:
if condition:
foobetter()
else:
raise
except expected_exceptions:
handleError()
A different approach is to use an auxiliary function to wrap the try/except logic:
def may_raise(expected_exceptions, somefunction, *a, **k):
try:
return False, somefunction(*a, **k)
except expected_exceptions:
return True, None
Such a helper may often come in useful in several different situations, so it's pretty common to have something like this somewhere in a project's "utilities" modules. Now, for your case (no arguments, no results) you could use:
failed, _ = may_raise(expected_exceptions, foo1)
if failed and condition:
failed, _ = may_raise(expected_exceptions, foobetter)
if failed:
handleError()
which I would argue is more linear and therefore simpler. The only issue with this general approach is that an auxiliary function such as may_raise does not FORCE you to deal in some way or other with exceptions, so you might just forget to do so (just like the use of return codes, instead of exceptions, to indicate errors, is prone to those return values mistakenly being ignored); so, use it sparingly...!-)
Using blanket exceptions isn't usually a great idea. What kind of Exception are you expecting there? Is it a KeyError, AttributeError, TypeError...
Once you've identified what type of error you're looking for you can use something like hasattr() or the in operator or many other things that will test for your condition before you have to deal with exceptions.
That way you can clean up your logic flow and save your exception handling for things that are really broken!
Sometimes there's no nice way to express a flow, it's just complicated. But here's a way to call foo() in only one place, and have the error handling in only one place:
if cacheExists:
loadCache()
dataComputed = False
else:
calculateData()
dataComputed = True
while True:
try:
foo()
break
except:
if not dataComputed:
calculateData()
dataComputed = True
continue
else:
#the error handling code
break
You may not like the loop, YMMV...
Or:
if cacheExists:
loadCache()
dataComputed = False
else:
calculateData()
dataComputed = True
done = False
while !done:
try:
foo()
done = True
except:
if not dataComputed:
calculateData()
dataComputed = True
continue
else:
#the error handling code
done = True
I like the alternative approach proposed by Alex Martelli.
What do you think about using a list of functions as argument of the may_raise. The functions would be executed until one succeed!
Here is the code
def foo(x):
raise Exception("Arrrgh!")
return 0
def foobetter(x):
print "Hello", x
return 1
def try_many(functions, expected_exceptions, *a, **k):
ret = None
for f in functions:
try:
ret = f(*a, **k)
except expected_exceptions, e:
print e
else:
break
return ret
print try_many((foo, foobetter), Exception, "World")
result is
Arrrgh!
Hello World
1
Is there a way to tell if you want to do foobetter() before making the call? If you get an exception it should be because something unexpected (exceptional!) happened. Don't use exceptions for flow control.