What is the best practice to run functions in sequence?
I have 4 functions, if the previous one failed, do not run the following ones. In each function, I set global error and error = 1 when exception occurs. Then in main, I just use if statement to check the value of error. I think there should be a better way to do it.
def main():
engine = conn_engine()
if error == 0:
process_sql()
if error == 0:
append_new_rows_to_prod()
if error == 0:
send_email_log()
The canonical way is to raise an exception within the function. For example:
def process_sql():
# Do stuff
if stuff_failed:
raise ProcessSQLException("Error while processing SQL")
def append_new_rows_to_prod():
# Do other stuff
if other_stuff_failed:
raise AppendRowsException("Error while appending rows")
def main():
engine = conn_engine()
try:
process_sql()
append_new_rows_to_prod()
send_email_log()
except ProcessSQLException, AppendRowsException as e:
# Handle exception, or gracefully exit
Related
I have below script which generates report for large size data.
Due to large size data request call times out.
I have added exception to handle this situation which works fine to get keep script running.
Issue I am having is after exception it goes to next project and skips the projects where it timed out.
I want it to start from same project again.
How can I achieve this ?
if __name__ = ‘__main__’
for project in AllProjectData['value']:
try:
project_name = project['name']
** code to be executed
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue
You are catching exceptions in a very odd way. I've never seen it done like this. I believe this code is catching all exceptions. For example:
try:
1/0
except:
ZeroDivisionError
pass
Works fine, but so does (it should raise IndexError):
try:
a = []
print(a[1])
except:
ZeroDivisionError
pass
So you shouldn't write except statements this way. What you should have is something along the lines:
success = False
while not success:
try:
# Your execution code
except (requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout):
continue
else:
success = True
Also you should try and not put so much code in your except statement as it is confusing as to what you're trying to catch and where. Also, you're completely missing some possibilities like a KeyError when there's no id field in project and others.
Try this -
def myfunc():
# Write the code given in question completely
# Then add this in your except statement
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
# Call your function so that it will jump back to the same function
myfunc()
# You don't need continue keyword because it jumps to the same function
The simplest way would be to use a while loop with a counter variable. Let me demonstrate:
i = 0
while i < len(AllProjectData['value']):
try:
project = AllProjectData['value'][i]
# code to be executed
i += 1 # increment counter on success
except:
requests.ConnectionError,
requests.exceptions.ReadTimeout,
requests.exceptions.Timeout,
requests.exceptions.ConnectTimeout
continue
This way, you will go to the next project only if work on the previous project was executed, as the loop variable is incremented only in the try block.
Note: I have assumed that your iterable is indexable. If it's not, just pass it inside the list() constructor.
In Python, can I use try and except when calling functions from my core function, and error checking that the called functions are succeeding? Is this a good way to structure your script with the core function calling functions and sandwiching them in try/except statements to manage errors? If the called functions throw an error or a False, will the try in the core function manage that?
def core_function():
try:
function_a()
except_Exception as e:
print(e)
try:
function_b()
except Exception as E:
print(e)
def function_a()
#this will error
print 1 + 'a'
return True
def function_b()
print 1 + 1
return True
If the called functions throw an error or a False, will the try in the core function manage that
There are basically two ways a function can report an error. By returning something that indicates an error or by raiseing an exception. try catch block in Python handles the latter. You can do something like that.
def core_function():
try:
if function_a() == False:
raise Exception('function_a failed')
if function_b() == False:
raise Exception('function_b failed')
except Exception as E:
print(e)
Read this for Conventions for error reporting: Exceptions vs returning error codes
So I am trying to import a module, and test methods from a class in that module.
Here is an example of a method.
def production_warning(self, targetenv):
if targetenv == 'prdv':
prodwarning1 = raw_input("WARNING: You are deploying to the production environment. Are you sure you want to do this? Y/N: ").upper()
if prodwarning1 == "N":
sys.exit()
prodwarning2 = raw_input("DEPLOYING TO PRDV, ARE YOU REALLY, REALLY SURE? Y/N: ").upper()
if prodwarning2 == "N":
sys.exit()
else:
return True
Here is an example of a test I am trying to write.
def production_warning():
try:
assert test.production_warning('prdv') is not errors
assert test.validate_params('fakeenv') is errors
print "Test Passed {0}/5: validate_params".format(counter)
test_db_refresh()
except:
print "Test Failed {0}/5: validate_params".format(counter)
test_db_refresh()
def db_refresh_prompt():
# assert test.db_refresh_prompt() is not errors
global counter
counter += 1
print "Test Passed {0}/5: db_refresh_prompt".format(counter)
production_warning()
db_refresh_prompt()
etc()
How do I check if an error is raised? At the end of the day I'm trying to run through all of these tests and for each function, if no exceptions are raised, print "Success". If an exception is raised, move on to the next test. People seem to keep pointing me in the direction of "calling your function will automatically raise an exception if there is one", but this will stop my test whenever an exception is thrown and I don't want that, I want to continue on to the next test.
I can work around this by doing:
def validate_params():
try:
assert test.validate_params('hackenv-re', 'test.username') is not errors
assert test.validate_params('fakeenv', 'test.username') is errors
assert test.validate_params('hackevn-re', 'vagrant') is errors
global counter
counter += 1
print "Test Passed {0}/5: validate_params".format(counter)
test_db_refresh()
except:
print "Test Failed {0}/5: validate_params".format(counter)
test_db_refresh()
but it seems like that defeats the purpose of using unittest in the first place? I thought with unittest I can just assert if an exception is raised and it returns a T/F that I can do whatever I want with.
Hope that is enough information.
Based on many of the answers given, I'm assuming there is nothing built in to unittest where I can do assertRaise (I believe this is used in Django)
Asserting that the tested code does not raise exceptions comes for free, you don't need to write code for that.
This question already has answers here:
is there a pythonic way to try something up to a maximum number of times?
(10 answers)
Closed 7 months ago.
I am writing in Python 2.7 and encounter the following situation. I would like to try calling a function three times. If all three times raise errors, I will raise the last error I get. If any one of the calls succeed, I will quit trying and continue immediately.
Here is what I have right now:
output = None
error = None
for _e in range(3):
error = None
try:
print 'trial %d!' % (_e + 1)
output = trial_function()
except Exception as e:
error = e
if error is None:
break
if error is not None:
raise error
Is there a better snippet that achieve the same use case?
use decorator
from functools import wraps
def retry(times):
def wrapper_fn(f):
#wraps(f)
def new_wrapper(*args,**kwargs):
for i in range(times):
try:
print 'try %s' % (i + 1)
return f(*args,**kwargs)
except Exception as e:
error = e
raise error
return new_wrapper
return wrapper_fn
#retry(3)
def foo():
return 1/0;
print foo()
Here is one possible approach:
def attempt(func, times=3):
for _ in range(times):
try:
return func()
except Exception as err:
pass
raise err
A demo with a print statement in:
>>> attempt(lambda: 1/0)
Attempt 1
Attempt 2
Attempt 3
Traceback (most recent call last):
File "<pyshell#18>", line 1, in <module>
attempt(lambda: 1/0)
File "<pyshell#17>", line 8, in attempt
raise err
ZeroDivisionError: integer division or modulo by zero
If you're using Python 3.x and get an UnboundLocalError, you can adapt as follows:
def attempt(func, times=3):
to_raise = None
for _ in range(times):
try:
return func()
except Exception as err:
to_raise = err
raise to_raise
This is because the err is cleared at the end of the try statement; per the docs:
When an exception has been assigned using as target, it is cleared
at the end of the except clause.
Ignoring the debug output and the ancient Python dialect, this looks good. The only thing I would change is to put it into a function, you could then simply return the result of trial_function(). Also, the error = None then becomes unnecessary, including the associated checks. If the loop terminates, error must have been set, so you can just throw it. If you don't want a function, consider using else in combination with the for loop and breaking after the first result.
for i in range(3):
try:
result = foo()
break
except Exception as error:
pass
else:
raise error
use_somehow(result)
Of course, the suggestion to use a decorator for the function still holds. You can also apply this locally, the decorator syntax is only syntactic sugar after all:
# retry from powerfj's answer below
rfoo = retry(3)(foo)
result = rfoo()
Came across a clean way of doing the retries. There is a module called retry.
First install the module using
pip install retry
Then import the module in the code.
from retry import retry
Use #retry decorator above the method, We can pass the parameters to the decorator. Some of the parameters are tries , delay , Exception.
Example
from retry import retry
#retry(AssertionError, tries=3, delay=2)
def retryfunc():
try:
ret = False
assert ret, "Failed"
except Exception as ex:
print(ex)
raise ex
The above code asserts and fails everytime, but the retry decorator retries for 3 times with a delay of 2 seconds between retries. Also this only retries on Assertion failures since we have specified the error type as AssertionError on any other error the function wont retry.
Sometimes I need the following pattern within a for loop. At times more than once in the same loop:
try:
# attempt to do something that may diversely fail
except Exception as e:
logging.error(e)
continue
Now I don't see a nice way to wrap this in a function as it can not return continue:
def attempt(x):
try:
raise random.choice((ValueError, IndexError, TypeError))
except Exception as e:
logging.error(e)
# continue # syntax error: continue not properly in loop
# return continue # invalid syntax
return None # this sort of works
If I return None than I could:
a = attempt('to do something that may diversely fail')
if not a:
continue
But I don't feel that does it the justice. I want to tell the for loop to continue (or fake it) from within attempt function.
Python already has a very nice construct for doing just this and it doesn't use continue:
for i in range(10):
try:
r = 1.0 / (i % 2)
except Exception, e:
print(e)
else:
print(r)
I wouldn't nest any more than this, though, or your code will soon get very ugly.
In your case I would probably do something more like this as it is far easier to unit test the individual functions and flat is better than nested:
#!/usr/bin/env python
def something_that_may_raise(i):
return 1.0 / (i % 2)
def handle(e):
print("Exception: " + str(e))
def do_something_with(result):
print("No exception: " + str(result))
def wrap_process(i):
try:
result = something_that_may_raise(i)
except ZeroDivisionError, e:
handle(e)
except OverflowError, e:
handle(e) # Realistically, this will be a different handler...
else:
do_something_with(result)
for i in range(10):
wrap_process(i)
Remember to always catch specific exceptions. If you were not expecting a specific exception to be thrown, it is probably not safe to continue with your processing loop.
Edit following comments:
If you really don't want to handle the exceptions, which I still think is a bad idea, then catch all exceptions (except:) and instead of handle(e), just pass. At this point wrap_process() will end, skipping the else:-block where the real work is done, and you'll go to the next iteration of your for-loop.
Bear in mind, Errors should never pass silently.
The whole idea of exceptions is that they work across multiple levels of indirection, i.e., if you have an error (or any other exceptional state) deep inside your call hierarchy, you can still catch it on a higher level and handle it properly.
In your case, say you have a function attempt() which calls the functions attempt2() and attempt3() down the call hierarchy, and attempt3() may encounter an exceptional state which should cause the main loop to terminate:
class JustContinueException(Exception):
pass
for i in range(0,99):
try:
var = attempt() # calls attempt2() and attempt3() in turn
except JustContinueException:
continue # we don't need to log anything here
except Exception, e:
log(e)
continue
foo(bar)
def attempt3():
try:
# do something
except Exception, e:
# do something with e, if needed
raise # reraise exception, so we catch it downstream
You can even throw a dummy exception yourself, that would just cause the loop to terminate, and wouldn't even be logged.
def attempt3():
raise JustContinueException()
Apart from the context I just want to answer the question in a brief fashion. No, a function cannot continue a loop it may be called in. That is because it has no information about this context. Also, it would raise a whole new class of questions like what shall happen if that function is called without a surrounding loop to handle that continue?
BUT a function can signal by various means that it wants the caller to continue any loop it currently performs. One means of course is the return value. Return False or None to signal this for example. Another way of signaling this is to raise a special Exception:
class ContinuePlease(Exception): pass
def f():
raise ContinuePlease()
for i in range(10):
try:
f()
except ContinuePlease:
continue
Maybe you want to do continuations? You could go and look at how Eric Lippert explains them (if you are ready to have your mind blown, but in Python it could look a bit like this:
def attempt(operation, continuation):
try:
operation()
except:
log('operation failed!')
continuation()
Inside your loop you could do:
attempt(attempt_something, lambda: foo(bar)) # attempt_something is a function
You could use this:
for l in loop:
attempt() and foo(bar)
but you should make sure attempt() returns True or False.
Really, though, Johnsyweb's answer is probably better.
Think that you are mapping foo on all items where attempt worked. So attempt is a filter and it's easy to write this as a generator:
def attempted( items ):
for item in items:
try:
yield attempt( item )
except Exception, e:
log(e)
print [foo(bar) for bar in attempted( items )]
I wouldn't normally post a second answer, but this is an alternative approach if you really don't like my first answer.
Remember that a function can return a tuple.
#!/usr/bin/env python
def something_that_mail_fail(i):
failed = False
result = None
try:
result = 1.0 / (i % 4)
except:
failed = True # But we don't care
return failed, result
for i in range(20):
failed, result = something_that_mail_fail(i)
if failed:
continue
for rah in ['rah'] * 3:
print(rah)
print(result)
I maintain that try ... except ... else is the way to go, and you shouldn't silently ignore errors though. Caveat emptor and all that.
Try the for loop outside the try, except block
This answer had Python 3.4 in mind however there are better ways in newer versions. Here is my suggestion
import sys
if '3.4' in sys.version:
from termcolor import colored
def list_attributes(module_name):
'''Import the module before calling this func on it.s '''
for index, method in enumerate(dir(module_name)):
try:
method = str(method)
module = 'email'
expression = module + '.' + method
print('*' * len(expression), '\n')
print( str(index).upper() + '. ',colored( expression.upper(), 'red'),
' ', eval( expression ).dir() , '...' , '\n'2 )
print('' * len(expression), '\n')
print( eval( expression + '.doc' ), '\n'*4,
'END OF DESCRIPTION FOR: ' + expression.upper(), '\n'*4)
except (AttributeError, NameError):
continue
else:
pass
finally:
pass
Edit: Removed all that stupidity I said...
The final answer was to rewrite the whole thing, so that I don't need to code like that.