I'm looking for the most pythonic way of trying a command, catching if an error occurs and retrying by running a preparatory command and then the original command. Specifically in my case I'm looking to write a table to a database, catch if a "schema does not exist" error is thrown, then trying to create a schema and retrying the write table. Then, if the write table errors again I don't want to catch it.
So far I have (schematically):
try:
write_to_table(table, schema)
except sqlalchemy.exc.ProgrammingError:
create_schema(schema)
write_to_table(table, schema)
This does what I want, but seems a bit off somehow, maybe because I'm duplicating write_to_table().
So what's the most pythonic way of doing the above?
P.S. When I say I'd like to retry, I do NOT want something like this: How to retry after exception?
Just create a reusable decorator !!
def retry(func):
'''
Decorator.
If the decorated function fails with an exception
calls it again after a random interval.
'''
def _retry(*args,**kwargs):
max_retries = 1
for i in range(max_retries):
try:
value = func(*args,**kwargs)
return value
except sqlalchemy.exc.ProgrammingError as e:
print('function:[{}({},{})] Failed with error: {} . Retrying...'.format(func.__name__,str(*args),str(kwargs),str(e)))
time.sleep(random.uniform(1,3))
print("Max_retries({}) exceeded for function {}".format(max_retries,func.__name__))
return _retry
By using the above decorator, in which you can also configure the number of retriesand the retry interval, you can do the following:
#retry
write_to_table(table, schema)
Related
TLDR: Is there a good way to cascade a heirarchy of exception handlers similar to what is possible with a series of if statements? e.g. one handler may attempt to handle a problem but throw an exception to be caught by the next handler, or the initial try block throws the second exception directly.
Basic premise: (There may be poory conceived code here, so bear with me) Also may be a duplicate, but I couldn't find it.
I am trying to verify the validity of a url with a head request. If I get a ConnectionError the url is not valid. The head request will helpfully throw a MissingSchema exception for missing "http://" so I added an exception handler to try the url with an "http://". However, if the url is still invalid it throws the expected ConnectionError. Is there a good way to pass that exception back to the exception handler that takes the ConnectionError directly from the try block? This would be similar to how you can cascade if statements. I could solve this particular example with some copy paste or recursion, but I could see both solutions becoming pretty annoying in more complex functions. Sample code below:
def checkURL(url):
try:
resp = requests.head(url)
return True # if request does not raise exception
except requests.exceptions.MissingSchema as exception:
try:
#try url with schema
resp = requests.head('http://' + url)
return True
#if url is still bad it will throw Connection Error
#I would like this to be also handled by the block below
except requests.ConnectionError as exception:
#ConnectionError == bad url
return False
I could solve this by duplicating my ConnectionError handler in the secondary try - except block, but that seems like a bad solution. Or I could recursively call checkURL('http://' + url) in the MissingSchema handler, but I could see that being problematic / inefficient also if there was more work being done in the initial try block. There's a good chance I'm missing something obvious here, but I'd appreciate any feedback.
In this case it would be easier to check the protocol "http://" or "https://" with .startswith() or a regular expression.
Nesting exception handling is rarely a good design choice.
Also your strategy for checking the url by trying the request multiple times can have serious performance issues if you are going to do a lot of checks.
Your best option here would be to check whatever you can without performing any request first, and only then have a single try block with multiple except clauses if you need to address the failures separately.
Edit: in the context of your question, where you want to retry the request in case of failures it is not really helpful and will force you to repeat the code.
If you need some mechanics that keeps retrying some operation then you need to enclose the try-except block inside a loop. For example:
for url in url_variations:
try:
request.head(url)
return url
except BlaBlaError:
continue
return None
I have the following (paraphrased) code that's subject to race conditions:
def calculate_and_cache(template, template_response):
# run a fairly slow and intensive calculation:
calculated_object = calculate_slowly(template, template_response)
cached_calculation = Calculation(calculated=calculated_object,
template=template,
template_response=template_response)
# try to save the calculation just computed:
try:
cached_calculation.save()
return cached_calculation
# if another thread beat you to saving this, catch the exception
# but return the object that was just calculated
except DatabaseError as error:
log(error)
return cached_calculation
And it's raising a DatabaseTransactionError:
TransactionManagementError: An error occurred in the current transaction.
You can't execute queries until the end of the 'atomic' block.
The docs have this to say about DTE's:
When exiting an atomic block, Django looks at whether it’s exited normally or with an exception to determine whether to commit or roll back.... If you attempt to run database queries before the rollback happens, Django will raise a TransactionManagementError.
But they also have this, much more vague thing to say about them as well:
TransactionManagementError is raised for any and all problems related to database transactions.
My questions, in order of ascending generality:
Will catching a DatabaseError actually address the race condition by letting the save() exit gracefully while still returning the object?
Where does the atomic block begin in the above code and where does it end?
What am I doing wrong and how can I fix it?
The Django docs on controlling transactions explicitly have an example of catching exceptions in atomic blocks.
In your case, you don't appear to be using the atomic decorator at all, so first you need to add the required import.
from django.db import transaction
Then you need to move the code that could raise a database error into an atomic block:
try:
with transaction.atomic():
cached_calculation.save()
return cached_calculation
# if another thread beat you to saving this, catch the exception
# but return the object that was just calculated
except DatabaseError as error:
log(error)
return cached_calculation
We've been experimenting with sqlalchemy's disconnect handling, and how it integrates with ORM. We've studied the docs, and the advice seems to be to catch the disconnect exception, issue a rollback() and retry the code.
eg:
import sqlalchemy as SA
retry = 2
while retry:
retry -= 1
try:
for name in session.query(Names):
print name
break
except SA.exc.DBAPIError as exc:
if retry and exc.connection_invalidated:
session.rollback()
else:
raise
I follow the rationale -- you have to rollback any active transactions and replay them to ensure a consistent ordering of your actions.
BUT -- this means a lot of extra code added to every function that wants to work with data. Furthermore, in the case of SELECT, we're not modifying data and the concept of rollback/re-request is not only unsightly, but a violation of the principle of DRY (don't repeat yourself).
I was wondering if others would mind sharing how they handle disconnects with sqlalchemy.
FYI: we're using sqlalchemy 0.9.8 and Postgres 9.2.9
The way I like to approach this is place all my database code in a lambda or closure, and pass that into a helper function that will handle catching the disconnect exception, and retrying.
So with your example:
import sqlalchemy as SA
def main():
def query():
for name in session.query(Names):
print name
run_query(query)
def run_query(f, attempts=2):
while attempts > 0:
attempts -= 1
try:
return f() # "break" if query was successful and return any results
except SA.exc.DBAPIError as exc:
if attempts > 0 and exc.connection_invalidated:
session.rollback()
else:
raise
You can make this more fancy by passing a boolean into run_query to handle the case where you are only doing a read, and therefore want to retry without rolling back.
This helps you satisfy the DRY principle since all the ugly boiler-plate code for managing retries + rollbacks is placed in one location.
Using exponential backoff (https://github.com/litl/backoff):
#backoff.on_exception(
backoff.expo,
sqlalchemy.exc.DBAPIError,
factor=7,
max_tries=3,
on_backoff=lambda details: LocalSession.get_main_sql_session().rollback(),
on_giveup=lambda details: LocalSession.get_main_sql_session().flush(), # flush the session
logger=logging
)
def pessimistic_insertion(document_metadata):
LocalSession.get_main_sql_session().add(document_metadata)
LocalSession.get_main_sql_session().commit()
Assuming that LocalSession.get_main_sql_session() returns a singleton.
I'm trying to handle failure on fabric, but the example I saw on the docs was too localized for my taste. I need to execute rollback actions if any of a number of actions fail. I tried, then, to use contexts to handle it, like this:
#_contextmanager
def failwrapper():
with settings(warn_only=True):
result = yield
if result.failed:
rollback()
abort("********* Failed to execute deploy! *********")
And then
#task
def deploy():
with failwrapper():
updateCode()
migrateDb()
restartServer()
Unfortunately, when one of these tasks fail, I do not get anything on result.
Is there any way of accomplishing this? Or is there another way of handling such situations?
According to my tests, you can accomplish that with this:
from contextlib import contextmanager
#contextmanager
def failwrapper():
try:
yield
except SystemExit:
rollback()
abort("********* Failed to execute deploy! *********")
As you can see I got rid of the warn_only setting as I suppose you don't need it if the rollback can be executed and you're aborting the execution anyway with abort().
Fabric raises SystemExit exception when encountering errors and warn_only setting is not used. We can just catch the exception and do the rollback.
Following on from Henri's answer, this also handles keyboard interrupts (Ctrl-C) and other exceptions:
#_contextmanager
def failwrapper():
try:
yield
except:
rollback()
raise
I've got a function that often throws an exception (SSH over 3g).
I'd like to keep trying to run function() every 10 seconds until it succeeds (doesn't throw an exception).
As I see it, there are two options:
Nesting:
def nestwrapper():
try:
output = function()
except SSHException as e:
# Try again
sleep(10)
return nestwrapper()
return output
Looping: (updated)
It's been pointed out that the previous looping code was pretty unnecessary.
def loopwrapper():
while True:
try:
return function()
except SSHException as e:
sleep(10)
Is there a preferred method of doing this?
Is there an issue with nesting and the exception stack?
I would find a loop to be cleaner and more efficient here. If this is an automation job, the recursive method could hit python recursion limit (default is 1000 iirc, can check with sys.getrecursionlimit()).
Don't use status is False for your expression, because this is an identity comparison. Use while not status.
I would probably implement it slightly differently too, because I don't see any need for the two different functions here:
def function_with_retries():
while True:
try:
output = function()
except SSHException:
sleep(10)
else:
return output
I'm not sure it makes a heck of a lot of sense to specially wrap the function call twice. the exception is probably reasonable, and you're going to the extra step of retrying on that particular exception. What I mean is that the try/except is rather tightly involved with the retrying loop.
This is the way I normally do this:
def retry_something():
while True:
try:
return something()
except SomeSpecialError:
sleep(10)
The while True: is really exactly what you're up to, you're going to loop forever, or rather, until you actually manage to something(), and then return. There's no further need for a boolean flag of success, that's indicated by the normal case of the return statement (which politely escapes the loop).
Keep it simple.
function looper(f):
while 1:
try:
return f()
except SSHException, e:
sleep(10)
output = looper(<function to call>)