Develop Reference

python flow control for conditionally retrying an operation

I'm attempting an operation in Python, and if it fails in certain ways I'd like to retry up to 10 times. If it fails in any other way I'd like it to fail immediately. After 10 retries I'd like all failures to propagate to the caller.
I've been unable to code the flow control in a satisfying way. Here's one example of the behavior (but not the style!) I want:
def run():
max_retries = 10
for retry_index in range(max_retries):
try:
result = run_operation()
except OneTypeOfError:
if retry_index < max_retries - 1:
continue
else:
raise
if result.another_type_of_error:
if retry_index < max_retries - 1:
continue
else:
raise AnotherTypeOfError()
try:
result.do_a_followup_operation()
except AThirdTypeOfError:
if retry_index < max_retries - 1:
continue
else:
raise
return result
raise Exception("We don't expect to end up here")
At first I thought I could just refactor this so the retry logic is separate from the error handling logic. The problem is that if, for example, OneTypeOfError is raised by result.do_a_followup_operation(), I don't want to retry at all in that case. I only want to retry in the specific circumstances coded above.
So I thought perhaps I could refactor this into a function which returns the result, a raised Exception (if any), and a flag indicating whether it's a retryable exception. Somehow that felt less elegant than the above to me.
I'm wondering if there are any flow control patterns in Python which might help here.

You can use a specific exception class and recursion to be a little more dry. Sth along these lines:
class Retry(Exception):
def __init__(self, e):
super(Exception, self).__init__()
self.e = e
def run(max_retries=10, exc=None):
if max_retries <= 0:
raise exc or Exception('Whatever')
try:
try:
result = run_operation()
except OneTypeOfError as e:
raise Retry(e)
if result.another_type_of_error:
raise Retry(AnotherTypeOfError())
try:
result.do_a_followup_operation()
except AThirdTypeOfError as e:
raise Retry(e)
except Retry as r:
return run(max_retries=max_retries-1, exc=r.e)
else:
return result
An iterative solution with a given number of iterations seems semantically questionable. After all, you want the whole thing to succeed and a retry is a fallback. And having run out of retries sounds like a base case to me.

If I understand this correctly, you could do it as follows by changing two things:
Counting tries_left down from 10 instead of retry_index up from 0 reads more naturally and lets you exploit that positive numbers are truthy.
If you changed (or wrapped) run_operation() such that it would already raise AnotherTypeOfError if result.another_error is true, you could combine the first two except blocks.
The code can optionally be made slightly more dense by omitting the else after raise (or after continue if you choose to test for if tries_left instead) – the control flow is diverted at that point anyway –, and by putting a simple statement on the same line as a bare if without else.
for tries_left in range(10, -1, -1):
try:
result = run_operation()
except OneTypeOfError, AnotherTypeOfError:
if not tries_left: raise
continue
try:
result.do_a_followup_operation()
except AThirdTypeOfError:
if not tries_left: raise
continue
return result

Edit: Ah I didn't realize that your code indicated you didn't use multiple except blocks, as pointed out by JETM
Here's your quick primer on ExceptionHandling:
try:
# do something
except OneTypeOfError as e:
# handle one type of error one way
print(e) # if you want to see the Exception raised but not raise it
except AnotherTypeOfError as e:
# handle another type of error another way
raise e('your own message')
except (ThirdTypeOfError, FourthTypeOfError) as e:
# handle error types 3 & 4 the same way
print(e) # if you want to see the Exception raised but not raise it
except: # DONT DO THIS!!!
'''
Catches all and any exceptions raised.
DONT DO THIS. Makes it hard to figure out what goes wrong.
'''
else:
# if the try block succeeds and no error is raisedm then do this.
finally:
'''
Whether the try block succeeds or fails and one of the except blocks is activated.
Once all those are done with, finally run this block.
This works even if your program crashed and so is great for cleaning up for example.
'''
I did this once a long time ago, and recursively called the same function in one kind of exception but not another. I also passed the retry index and max_retries variable to the function, which meant adding those as parameters.
The other way would be to place the entire thing in a for loop of max_retries and add a break in all except blocks for the exceptions where you don't want a retry.
Finally, instead of a for loop, you can put the entire thing in a while loop, insert an increment condition in except block for one type of exception and make the while condition false in except block for other exceptions.

Difference between raising exception and automatically causing an exception

Currently using the wxPython framework and my code looks like this:
Event bind:
self.frequency_grid.Bind(wx.grid.EVT_GRID_CELL_CHANGE, self.on_cell_changed)
Function that handles changed cells:
def on_cell_changed(self, event):
self.current_grid = event.GetEventObject()
try:
new_value= self.get_cell_value()
if new_value < 0:
raise AttributeError
#allow the cell to update
except AttributeError:
event.Veto()
wx.MessageBox(_("Positive values only."), "", wx.OK|wx.ICON_WARNING)
except:
wx.MessageBox(_("Invalid value for cell."), "", wx.OK|wx.ICON_WARNING)
event.Veto()
The function get_cell_value() reads the value from the current cell and converts it to an integer simply by using int(). If the user enters a character like 'a' into the cell, obviously this function fails and the exception is raised. In this case the messagebox comes out telling the user the cell has an invalid value. This is what I call the automatically caused exception, and the final exception block is executed.
In the case of negative values, I manually raise an AttributeError (just wanted to see something different from ValueError which is what happens when user inputs characters).
In this case however, the wxPython sends the EVT_GRID_CELL_CHANGE event twice, so there must be something different about manually raised exceptions.
I've separately raised a ticket about the duplicated events at http://trac.wxwidgets.org/ticket/16333 but just trying to understand how the first scenario doesn't make wxPython send 2 events compared to the second scenario.

Don't use except: if you really want to catch any possible error(even system memory or anything else non-related).
To make your code look better, I'd suggest to cast value to int with try...except block, then check for negative values.
try:
value = int(text)
except:
return MessageBox('Enter digit')
if value < 0:
return MessageBox('Enter positive digit')

There is no difference in the Exceptions, but you are processing them in a different way, at a different place in the code.
I suspect it's the veto() call that causes the extra event. But why not validate the input in one place only? i.e have your get_cell_value() function should make sure it's an integer >0.
Also, this structure is both a recipe for circular events, and a painfully annoying UI. People should be able to make a typo and go back and correct it without an annoying dialog popping up.
Maybe provide an indicator of a problem, like making the cell background red, but only go the dialog route once the user tries to move on to the next step.

After a bit more tracing, I found that creating the MessageBox causes the EVT_GRID_CELL_CHANGING event to occur, which then leads to the EVT_GRID_CELL_CHANGED event to occur, which is why I saw duplicated events.
The reason why I did not see duplicated events during the entry of a character was because a Veto() was called in the event handler for the EVT_GRID_CELL_CHANGING if an int() conversion was invalid because my handler for that event gets the grid input and tries to convert it.
In conclusion, there is no difference in Python exception handling, but however, a better wxPython demo should be implemented to prevent the duplicated message box during the demo and show other users how to better use the grid mechanism.

How to return a value and raise an Exception

I have two objectives with this try/except statement.
It needs to return a value of 1 if no problems occurred, or 0 if any problems occurred.
It needs to raise an exception and end the script.
I have the return value working. I also have the SystemExit() working. But together, they aren't working.
My Python Script (that's relevant):
except IOError:
value_to_return = 0
return value_to_return
raise SystemExit("FOOBAR")
With this, it ignores the raise SystemExit("FOOBAR") line completely. How do I go about getting a returned value and still raise SystemExit("FOOBAR")? This may be elementary to some, but I'm actually having quite a bit of difficulty with it.

Returning and raising are mutually exclusive.
Raising SystemExit will end the script. A few cleanup routines get to run, and if the caller really, really wants to, they can catch the SystemExit and cancel it, but mostly, you can think of it as stopping execution right there. The caller will never get a chance to see a return value or do anything meaningful with it.
Returning means you want the script to continue. Continuing might mean having the caller raise SystemExit, or it might mean ignoring the error, or it might mean something else. Whatever it means is up to you, as you're the one writing the code.
Finally, are you sure you should be handling this error at all? Catching an exception only to turn it into a system shutdown may not be the most useful behavior. It's not a user-friendly way to deal with problems, and it hides all the useful debugging information you'd get from a stack trace.

You can raise an error with a 'returning_value' argument to be used after the calling.
Another pythonic answer to your problem could be to make use of the error arguments in the raise and then, in your call manage the error to get the value, convert it from string and get your 'return-ish'.
def your_f():
try:
some_io_thingy_ok()
return 1
except IOError:
raise SystemExit("FOOBAR", 0)
try:
my_returning_value = your_f()
except SystemExit as err:
my_returning_value = err.args[1]
print(my_returning_value)
From Python 3 docs :
When an exception occurs, it may have an associated value, also known
as the exception’s argument. The presence and type of the argument
depend on the exception type.
The except clause may specify a variable after the exception name. The
variable is bound to an exception instance with the arguments stored
in instance.args. For convenience, the exception instance defines
str() so the arguments can be printed directly without having to reference .args. One may also instantiate an exception first before
raising it and add any attributes to it as desired.

To exit a script and return an exit status, use sys.exit():
import sys
sys.exit(value_to_return)

I think what you may be looking for is something more like this:
def some_function():
# this function should probably do some stuff, then return 1 if
# it was successful or 0 otherwise.
pass
def calling_function():
a = some_function()
if a == 1:
raise SystemExit('Get the heck outta here!')
else:
# Everything worked!
pass

You can't "raise" and "return" in the same time, so you have to add a special variable to the return value (e.g: in tuple) in case of error.
E.g:
I have a function (named "func") which counts something and I need the (partial) result even if an exception happened during the counting. In my example I will use KeyboardInterrupt exception (the user pressed CTRL-C).
Without exception handling in the function (it's wrong, in case of any exception the function doesn't give back anything):
def func():
s=0
for i in range(10):
s=s+1
time.sleep(0.1)
return s
x=0
try:
for i in range(10):
s=func()
x=x+s
except KeyboardInterrupt:
print(x)
else:
print(x)
And now I introduce a boolean return value (in a tuple, next to the original return value) to indicate if an exception happened. Because in the function I handle only the KeyboardInterrupt exception, I can be sure that's happened, so I can raise the same where I called the function:
def func():
try:
s=0
for i in range(10):
s=s+1
time.sleep(0.1)
except KeyboardInterrupt: # <- the trick is here
return s, True # <- the trick is here
return s, False # <- the trick is here
x=0
try:
for i in range(10):
s,e=func()
x=x+s
if e: # <- and here
raise KeyboardInterrupt # <- and here
except KeyboardInterrupt:
print(x)
else:
print(x)
Note: my example is python3. The time module is used (in both code above), but I haven't import it just to make it shorter. If you want to really try it, put at the beginning:
import time

i was looking for an answer without using try, use 'finally' keyword like this.. if any one knows fill me in
here is an answer for your poblem
try:
9/0#sample error "don't know how to make IOError"
except ZeroDivisionError:
value_to_return = 0
raise SystemExit("FOOBAR")
finally:return value_to_return

Why use else in try/except construct in Python?

I am learning Python and have stumbled upon a concept I can't readily digest: the optional else block within the try construct.
According to the documentation:
The try ... except statement has an optional else clause, which, when
present, must follow all except clauses. It is useful for code that
must be executed if the try clause does not raise an exception.
What I am confused about is why have the code that must be executed if the try clause does not raise an exception within the try construct -- why not simply have it follow the try/except at the same indentation level? I think it would simplify the options for exception handling. Or another way to ask would be what the code that is in the else block would do that would not be done if it were simply following the try statement, independent of it. Maybe I am missing something, do enlighten me.
This question is somewhat similar to this one but I could not find there what I am looking for.

The else block is only executed if the code in the try doesn't raise an exception; if you put the code outside of the else block, it'd happen regardless of exceptions. Also, it happens before the finally, which is generally important.
This is generally useful when you have a brief setup or verification section that may error, followed by a block where you use the resources you set up in which you don't want to hide errors. You can't put the code in the try because errors may go to except clauses when you want them to propagate. You can't put it outside of the construct, because the resources definitely aren't available there, either because setup failed or because the finally tore everything down. Thus, you have an else block.

One use case can be to prevent users from defining a flag variable to check whether any exception was raised or not(as we do in for-else loop).
A simple example:
lis = range(100)
ind = 50
try:
lis[ind]
except:
pass
else:
#Run this statement only if the exception was not raised
print "The index was okay:",ind
ind = 101
try:
lis[ind]
except:
pass
print "The index was okay:",ind # this gets executes regardless of the exception
# This one is similar to the first example, but a `flag` variable
# is required to check whether the exception was raised or not.
ind = 10
try:
print lis[ind]
flag = True
except:
pass
if flag:
print "The index was okay:",ind
Output:
The index was okay: 50
The index was okay: 101
The index was okay: 10

Preventing python definition from execution

I want to know what is the best way of checking an condition in Python definition and prevent it from further execution if condition is not satisfied. Right now i am following the below mentioned scheme but it actually prints the whole trace stack. I want it to print only an error message and do not execute the rest of code. Is there any other cleaner solution for doing it.
def Mydef(n1,n2):
if (n1>n2):
raise ValueError("Arg1 should be less than Arg2)
# Some Code
Mydef(2,1)

That is what exceptions are created for. Your scheme of raising exception is good in general; you just need to add some code to catch it and process it
try:
Mydef(2,1)
except ValueError, e:
# Do some stuff when exception is raised, e.message will contain your message
In this case, execution of Mydef stops when it encounters raise ValueError line of code, and goes to the code block under except.
You can read more about exceptions processing in the documentation.
If you don't want to deal with exceptions processing, you can gracefully stop function to execute further code with return statement.
def Mydef(n1,n2):
if (n1>n2):
return

def Mydef(n1,n2):
if (n1>n2):
print "Arg1 should be less than Arg2"
return None
# Some Code
Mydef(2,1)
Functions stop executing when they reach to return statement or they run the until the end of definition. You should read about flow control in general (not specifically to python)