One of my view in Django executes save operations on 6-7 tables. I want these transactions to be atomic I,e if the 5th or 6th transaction fail I want to rollback all the previous saves.
The view contains a try-except block to handle the exceptions raised.
It looks something like this:
#transaction.atomic
def my_view(request):
sid = transaction.savepoint()
try:
Table1.save()
Table2.save()
Table3.save()
Table4.save()
Table5.save()
Table6.save()
Table7.save() # This might fail. In case of failure I want to rollback saves from Table1 to Table6
transaction.savepoint_commit(sid)
except Exception as e:
print(str(e))
transaction.savepoint_rollback(sid)
return JsonResponse({"Status": 0, "Data": str(e)})
I've tried the above and Table7.save() has failed and Table1 to Table6 rollback has not happened.
I want to return the JSON response as {"Status": 0, Data: "Error That occurred"} in all the cases.
I don't want to re-raise the exception in except block as done in This link
What should I do to return a proper JSONResponse and rollback everything in case of failure?
As suggested in the link:
transaction.atomic will execute a transaction on the database if your
view produces a response without errors. Because you're catching the
exception yourself, it appears to Django that your view executed just
fine. If you catch the exception, you need to handle it yourself
An alternative approach is to use transaction.atomic inside with (as context manager), which will ensure a commit or rollback (again suggested on the same link and also explained here)
def my_view(request):
try:
with transaction.atomic():
Table1.save()
Table2.save()
Table3.save()
Table4.save()
Table5.save()
Table6.save()
Table7.save() # This might fail. In case of failure I want to rollback saves from Table1 to Table6
except Exception as e:
print(str(e))
return JsonResponse({"Status": 0, "Data": str(e)})
Related
We use mongouse version 0.4.2 and have a piece of code that looks like this:
from mongoengine import Document
class Job(Document):
.....
def create_and_save_job(cls, my_data):
try:
j = cls(**my_data)
j.save()
except Exception as e:
if "duplicate unique keys" in e.message:
return
else:
raise
logger.info({"id": j.id, "status": "success"})
We use sentry to capture any exception raised in our code.
We later found out that we logged a lot of id that don't actually exist in the Job collection, and sentry doesn't have any exception related to this.
(something like)
log: {"id": 1234, "status": "success"}
>> db.job.findOne({_id: ObjectId(1234)})
null
What could be causing this? Should we place the save with insert_one and reload after save?
Thanks,
try:
cursor = connection.cursor()
cursor.execute('update m_password set password = :new_password , updater_id = :updater_id_dic , update_date = :update_date_dic where user_cd = :user_cd_dic and del_flg = :del_flag_dic', queryDictionary)
# Commit
connection.commit()
# Close the cursor
cursor.close()
# Close the database connection
connection.close()
return {
'status': "success",
'errorCode': 0,
'errorMessage': json.dumps("")
}
except: cx_Oracle.IntegrityError as e:
errorObj, = e.args
return {
'status': "error",
'errorCode': errorObj.code,
'errorMessage': json.dumps(errorObj.message)
I am trying to update the value of oracle database . Database connection is alright . I can get the value from database . But update the value is not working . It shows success but value is not actually updated . Also there is no error .
also lambda log showing successfully executed . Please check this image
[1]: https://i.stack.imgur.com/PtLdy.png
I am stuck here .It will be really helpful if i get some help .
Thank you.
It's likely that you're getting an error other than an IntegrityError and don't have an explicit error handler for it. This would cause you to continue towards your this is fine return method.
In fact, updating a password shouldn't throw an integrity error at all since you're not operating on something that effects your schema or is effected by a schema constraint. According to the docs:
This exception is raised when the relational integrity of the data is affected. For example, a duplicate key was inserted or a foreign key constraint would fail.
To troubleshoot, I'd start by accepting any error here. Once you've determined what error type you've caught, you can add per exception error handling as needed.
except Exception as e:
errorObj, = e.args
return {
...
I'm trying to run over a for loop that validates objects and saves them, and I want to fail it all if at least one have failed, but only after going over all the objects. I've tried different approaches, but on all of them - even if there was an exception, at least one object was saved to DB. In the latest version, see below, I'm trying to set
transaction.set_rollback(True)
if at least on exception was raised.
try:
is_failed = False
with transaction.atomic():
for identifier, spec in spec_dict.items():
try:
spec_data = {'title':my_title,
'identifier': identifier,
'updated_by': user_id,
'created_by': user_id
}
serializer = SpecSerializer(data=spec_data)
serializer.is_valid(raise_exception=True)
serializer.save()
except DataError as DE:
print("** in DataError")
is_failed = True
pass
except ValidationError as VE:
print("** in ValidationError")
print(str(VE))
is_failed = True
pass
except Exception as Exc:
print("** inside Exception: " + str(Exc))
is_failed = True
pass
if is_failed:
transaction.set_rollback(True)
except IntegrityError:
print("** inside integrity error")
pass
Seems like the 'set_rollback' doesn't affect the transaction. Worth to mention that all our http requests are wrapped in transaction.
EDIT:
Should transaction.atomic() work for non view functions? Couldn't find answer for that
So, apparently -
transaction.atomic():
is managing the transaction for the 'default' DB by default, unless other DB is mentioned:
transaction.atomic(using='otherDB'):
Since we use more than one DB and the one that I worked on was not set as the default, I was missing the 'using'.
PostgreSQL and SQL defines a Serializable transaction isolation level. If you isolate transactions to this level, conflicting concurrent transactions abort and need retrying.
I am familiar with the concept of transaction retries from Plone / Zope world where the entire HTTP request can be replayed in the case there is a transaction conflict. How similar functionality could be achieved with SQLAlchemy (and potentially with zope.sqlalchemy)? I tried to read the documentation of zope.sqlalchemy and Zope transaction manager, but this is not obvious the me.
Specially I want something like this:
# Try to do the stuff, if it fails because of transaction conflict do again until retry count is exceeded
with transaction.manager(retries=3):
do_stuff()
# If we couldn't get the transaction through even after 3 attempts, fail with a horrible exception
So, after poking around two weeks and getting no off-the-shelf solution I came up with my own.
Here is a ConflictResolver class which provides managed_transaction function decorator. You can use the decorator to mark functions to be retryable. I.e. if there is an database conflict error when running the function, the function is run again, now with more hopes the db transaction which caused the conflict error would have finished.
The source code is here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
The unit tests to cover it are here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
Python 3.4+ only.
"""Serialized SQL transaction conflict resolution as a function decorator."""
import warnings
import logging
from collections import Counter
from sqlalchemy.orm.exc import ConcurrentModificationError
from sqlalchemy.exc import OperationalError
UNSUPPORTED_DATABASE = "Seems like we might know how to support serializable transactions for this database. We don't know or it is untested. Thus, the reliability of the service may suffer. See transaction documentation for the details."
#: Tuples of (Exception class, test function). Behavior copied from _retryable_errors definitions copied from zope.sqlalchemy
DATABASE_COFLICT_ERRORS = []
try:
import psycopg2.extensions
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((psycopg2.extensions.TransactionRollbackError, None))
# ORA-08177: can't serialize access for this transaction
try:
import cx_Oracle
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((cx_Oracle.DatabaseError, lambda e: e.args[0].code == 8177))
if not DATABASE_COFLICT_ERRORS:
# TODO: Do this when cryptoassets app engine is configured
warnings.warn(UNSUPPORTED_DATABASE, UserWarning, stacklevel=2)
#: XXX: We need to confirm is this the right way for MySQL, SQLIte?
DATABASE_COFLICT_ERRORS.append((ConcurrentModificationError, None))
logger = logging.getLogger(__name__)
class CannotResolveDatabaseConflict(Exception):
"""The managed_transaction decorator has given up trying to resolve the conflict.
We have exceeded the threshold for database conflicts. Probably long-running transactions or overload are blocking our rows in the database, so that this transaction would never succeed in error free manner. Thus, we need to tell our service user that unfortunately this time you cannot do your thing.
"""
class ConflictResolver:
def __init__(self, session_factory, retries):
"""
:param session_factory: `callback()` which will give us a new SQLAlchemy session object for each transaction and retry
:param retries: The number of attempst we try to re-run the transaction in the case of transaction conflict.
"""
self.retries = retries
self.session_factory = session_factory
# Simple beancounting diagnostics how well we are doing
self.stats = Counter(success=0, retries=0, errors=0, unresolved=0)
#classmethod
def is_retryable_exception(self, e):
"""Does the exception look like a database conflict error?
Check for database driver specific cases.
:param e: Python Exception instance
"""
if not isinstance(e, OperationalError):
# Not an SQLAlchemy exception
return False
# The exception SQLAlchemy wrapped
orig = e.orig
for err, func in DATABASE_COFLICT_ERRORS:
# EXception type matches, now compare its values
if isinstance(orig, err):
if func:
return func(e)
else:
return True
return False
def managed_transaction(self, func):
"""SQL Seralized transaction isolation-level conflict resolution.
When SQL transaction isolation level is its highest level (Serializable), the SQL database itself cannot alone resolve conflicting concurrenct transactions. Thus, the SQL driver raises an exception to signal this condition.
``managed_transaction`` decorator will retry to run everyhing inside the function
Usage::
# Create new session for SQLAlchemy engine
def create_session():
Session = sessionmaker()
Session.configure(bind=engine)
return Session()
conflict_resolver = ConflictResolver(create_session, retries=3)
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
# Both threads modify the same wallet simultaneously
w = session.query(BitcoinWallet).get(1)
w.balance += 1
# Execute the conflict sensitive code inside a managed transaction
myfunc()
The rules:
- You must not swallow all exceptions within ``managed_transactions``. Example how to handle exceptions::
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
try:
my_code()
except Exception as e:
if ConflictResolver.is_retryable_exception(e):
# This must be passed to the function decorator, so it can attempt retry
raise
# Otherwise the exception is all yours
- Use read-only database sessions if you know you do not need to modify the database and you need weaker transaction guarantees e.g. for displaying the total balance.
- Never do external actions, like sending emails, inside ``managed_transaction``. If the database transaction is replayed, the code is run twice and you end up sending the same email twice.
- Managed transaction section should be as small and fast as possible
- Avoid long-running transactions by splitting up big transaction to smaller worker batches
This implementation heavily draws inspiration from the following sources
- http://stackoverflow.com/q/27351433/315168
- https://gist.github.com/khayrov/6291557
"""
def decorated_func():
# Read attemps from app configuration
attempts = self.retries
while attempts >= 0:
session = self.session_factory()
try:
result = func(session)
session.commit()
self.stats["success"] += 1
return result
except Exception as e:
if self.is_retryable_exception(e):
session.close()
self.stats["retries"] += 1
attempts -= 1
if attempts < 0:
self.stats["unresolved"] += 1
raise CannotResolveDatabaseConflict("Could not replay the transaction {} even after {} attempts".format(func, self.retries)) from e
continue
else:
session.rollback()
self.stats["errors"] += 1
# All other exceptions should fall through
raise
return decorated_func
Postgres and Oracle conflict errors are marked as retryable by zope.sqlalchemy. Set your isolation level in the engine configuration and the transaction retry logic in pyramid_tm or Zope will work.
There are three situations when I need to handle exceptions.
When data validation raised exception
When library/module functions raised exceptions (e.g. database connection abort)
When business logic raises exception such as 500, 503, 401, 403 and 404
def library_func():
try:
...
except HTTPException:
raise TwitterServiceException("Twitter is down!")
#view_config(route_name="home", renderer="json")
#validator
#authorization
def home_view(request):
try:
tweets = library_func()
return {"tweets": tweets}
except TwitterServiceException as e:
LOG.critical(e.msg)
raise ParnterServcieError(e.msg) # this is probably a 503 error
def validator(args):
# I will show the high level of this decorator
try:
decode input as JSON
verify data format
except ValueError as err:
error = {'error': "Missing required parameters."}
except json.JSONDecodeError as err:
error = {'error': "Failed to decode the incoming JSON payload."}
if error is not None:
return HTTPBadRequest(body=json.dumps(error),
content_type='application/json')
def authorization(args):
# very similar to validator except it performs authorization and if failed
# 401 is raised with some helpful message.
The doc suggests Custom Exception Views. In my PoC above, I will tie ParnterServcieError as one. I can even generalize HTTPBadRequest and all praymid.httpexceptions using custom exception so that I no longer need to repeat json.dumps and content_type. I can set a boilerplate error body before I return request.response object.
Idea:
#view_config(context=ParnterServcieError)
def 503_service_error_view(e, request):
request.response.status = 503
request.response.json_body = {"error": e.msg}
return request.response
I can generalize one for all uncaught, unspecified exceptions (which results in 500 Internal Server Error) called 500_internal_server_error_view.
Does this seem sane and clean to people? Is my way of handling high and low level of exceptions proper and Pythonic?
I applied this strategy to ToDoPyramid and could encapsulate error handling in a single custom exception view that was repeated multiple times in the application before. Until you could even improve it, you got a great idea. Pyramid rocks.
References
Catching database connection error in ToDoPyramid