Develop Reference

SQLModel and sqlalchemy exception handling

I am trying to understand exception handling in SQLModel and SqlAlchemy.
Assume I have this code
def create_blog(blog: BlogCreate, session: Session = Depends(get_session)):
try:
result = Blog(title=blog.title, body=blog.body, author_id=blog.author_id)
session.add(result)
session.commit()
session.refresh(result)
except exc.IntegrityError:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Author with {blog.author_id} not found"
)
except:
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Something went wrong"
)
return result
Here, author_id is a foreign key. Now Suppose if I pass it some id for which author doesn't exist (and not doing try except), It is going to throw an error (in my terminal logs).
DETAIL: Key (author_id)=(5) is not present in table "author".
How can I get this message so that I can pass it to user?

Google resource manager get all exceptions - Python

im making a python script that can manage my google projects.
im having a insue with one part
when i try to exclude the project its can return to me many errors.
i did a peace of code to get this exception:
try:
# Initialize request argument(s)
request = DeleteProjectRequest(
name=project,
)
self.project_manager.delete_project(request=request)
except PermissionDenied as exc:
# GCP returns PermissionDenied whether we actually does
# not have permissions to perform the get_project call
# or when the project does not exist. Due to this reason,
# the PermissionDenied exception catch won't be deterministic.
logger.error(f"Project '{project_id}' does not exist", exc)
return False
i need to get the error message of all types of errors
i changed except PermissionDenied as exc: for except Exception as exc:
and it works but i need to call the logger only if the error is PermissionDenied and in all cases i need to call another function passing the message as parameter like it return_to_db(error_message)
my question is. how can i run only the logger if the error is PermissionDenied?

You can also catch multiple Exceptions by adding additional blocks, though it will choose the first isinstance() match (so if you put Exception first, it will be selected instead, while TypeError would be continued past)
try:
self.project_manager.delete_project(
request=DeleteProjectRequest(name=project))
except PermissionDenied as exc:
# GCP returns PermissionDenied whether we actually does
# not have permissions to perform the get_project call
# or when the project does not exist. Due to this reason,
# the PermissionDenied exception catch won't be deterministic.
logger.error(f"Project '{project_id}' does not exist", exc)
except Exception:
# FIXME other handling to go here
pass # fall to return False
else: # didn't raise
return True
# opportunity for finally: block here too
# if any Exception was raised, continue to return False
return False

You can add a condition of the instance type of the current exception in Python, example :
try:
# Initialize request argument(s)
request = DeleteProjectRequest(
name=project,
)
self.project_manager.delete_project(request=request)
except Exception as exc:
if isinstance(exc, PermissionDenied):
logger.error(f"Project '{project_id}' does not exist", exc)
return False
As expected, the logger is executed only if the exception instance is PermissionDenied.

Django Unable to rollback with try-exception block for atomic transactions

One of my view in Django executes save operations on 6-7 tables. I want these transactions to be atomic I,e if the 5th or 6th transaction fail I want to rollback all the previous saves.
The view contains a try-except block to handle the exceptions raised.
It looks something like this:
#transaction.atomic
def my_view(request):
sid = transaction.savepoint()
try:
Table1.save()
Table2.save()
Table3.save()
Table4.save()
Table5.save()
Table6.save()
Table7.save() # This might fail. In case of failure I want to rollback saves from Table1 to Table6
transaction.savepoint_commit(sid)
except Exception as e:
print(str(e))
transaction.savepoint_rollback(sid)
return JsonResponse({"Status": 0, "Data": str(e)})
I've tried the above and Table7.save() has failed and Table1 to Table6 rollback has not happened.
I want to return the JSON response as {"Status": 0, Data: "Error That occurred"} in all the cases.
I don't want to re-raise the exception in except block as done in This link
What should I do to return a proper JSONResponse and rollback everything in case of failure?

As suggested in the link:
transaction.atomic will execute a transaction on the database if your
view produces a response without errors. Because you're catching the
exception yourself, it appears to Django that your view executed just
fine. If you catch the exception, you need to handle it yourself
An alternative approach is to use transaction.atomic inside with (as context manager), which will ensure a commit or rollback (again suggested on the same link and also explained here)
def my_view(request):
try:
with transaction.atomic():
Table1.save()
Table2.save()
Table3.save()
Table4.save()
Table5.save()
Table6.save()
Table7.save() # This might fail. In case of failure I want to rollback saves from Table1 to Table6
except Exception as e:
print(str(e))
return JsonResponse({"Status": 0, "Data": str(e)})

SQLAlchemy, Serializable transactions isolation and retries in idiomatic Python way

PostgreSQL and SQL defines a Serializable transaction isolation level. If you isolate transactions to this level, conflicting concurrent transactions abort and need retrying.
I am familiar with the concept of transaction retries from Plone / Zope world where the entire HTTP request can be replayed in the case there is a transaction conflict. How similar functionality could be achieved with SQLAlchemy (and potentially with zope.sqlalchemy)? I tried to read the documentation of zope.sqlalchemy and Zope transaction manager, but this is not obvious the me.
Specially I want something like this:
# Try to do the stuff, if it fails because of transaction conflict do again until retry count is exceeded
with transaction.manager(retries=3):
do_stuff()
# If we couldn't get the transaction through even after 3 attempts, fail with a horrible exception

So, after poking around two weeks and getting no off-the-shelf solution I came up with my own.
Here is a ConflictResolver class which provides managed_transaction function decorator. You can use the decorator to mark functions to be retryable. I.e. if there is an database conflict error when running the function, the function is run again, now with more hopes the db transaction which caused the conflict error would have finished.
The source code is here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
The unit tests to cover it are here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
Python 3.4+ only.
"""Serialized SQL transaction conflict resolution as a function decorator."""
import warnings
import logging
from collections import Counter
from sqlalchemy.orm.exc import ConcurrentModificationError
from sqlalchemy.exc import OperationalError
UNSUPPORTED_DATABASE = "Seems like we might know how to support serializable transactions for this database. We don't know or it is untested. Thus, the reliability of the service may suffer. See transaction documentation for the details."
#: Tuples of (Exception class, test function). Behavior copied from _retryable_errors definitions copied from zope.sqlalchemy
DATABASE_COFLICT_ERRORS = []
try:
import psycopg2.extensions
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((psycopg2.extensions.TransactionRollbackError, None))
# ORA-08177: can't serialize access for this transaction
try:
import cx_Oracle
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((cx_Oracle.DatabaseError, lambda e: e.args[0].code == 8177))
if not DATABASE_COFLICT_ERRORS:
# TODO: Do this when cryptoassets app engine is configured
warnings.warn(UNSUPPORTED_DATABASE, UserWarning, stacklevel=2)
#: XXX: We need to confirm is this the right way for MySQL, SQLIte?
DATABASE_COFLICT_ERRORS.append((ConcurrentModificationError, None))
logger = logging.getLogger(__name__)
class CannotResolveDatabaseConflict(Exception):
"""The managed_transaction decorator has given up trying to resolve the conflict.
We have exceeded the threshold for database conflicts. Probably long-running transactions or overload are blocking our rows in the database, so that this transaction would never succeed in error free manner. Thus, we need to tell our service user that unfortunately this time you cannot do your thing.
"""
class ConflictResolver:
def __init__(self, session_factory, retries):
"""
:param session_factory: `callback()` which will give us a new SQLAlchemy session object for each transaction and retry
:param retries: The number of attempst we try to re-run the transaction in the case of transaction conflict.
"""
self.retries = retries
self.session_factory = session_factory
# Simple beancounting diagnostics how well we are doing
self.stats = Counter(success=0, retries=0, errors=0, unresolved=0)
#classmethod
def is_retryable_exception(self, e):
"""Does the exception look like a database conflict error?
Check for database driver specific cases.
:param e: Python Exception instance
"""
if not isinstance(e, OperationalError):
# Not an SQLAlchemy exception
return False
# The exception SQLAlchemy wrapped
orig = e.orig
for err, func in DATABASE_COFLICT_ERRORS:
# EXception type matches, now compare its values
if isinstance(orig, err):
if func:
return func(e)
else:
return True
return False
def managed_transaction(self, func):
"""SQL Seralized transaction isolation-level conflict resolution.
When SQL transaction isolation level is its highest level (Serializable), the SQL database itself cannot alone resolve conflicting concurrenct transactions. Thus, the SQL driver raises an exception to signal this condition.
``managed_transaction`` decorator will retry to run everyhing inside the function
Usage::
# Create new session for SQLAlchemy engine
def create_session():
Session = sessionmaker()
Session.configure(bind=engine)
return Session()
conflict_resolver = ConflictResolver(create_session, retries=3)
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
# Both threads modify the same wallet simultaneously
w = session.query(BitcoinWallet).get(1)
w.balance += 1
# Execute the conflict sensitive code inside a managed transaction
myfunc()
The rules:
- You must not swallow all exceptions within ``managed_transactions``. Example how to handle exceptions::
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
try:
my_code()
except Exception as e:
if ConflictResolver.is_retryable_exception(e):
# This must be passed to the function decorator, so it can attempt retry
raise
# Otherwise the exception is all yours
- Use read-only database sessions if you know you do not need to modify the database and you need weaker transaction guarantees e.g. for displaying the total balance.
- Never do external actions, like sending emails, inside ``managed_transaction``. If the database transaction is replayed, the code is run twice and you end up sending the same email twice.
- Managed transaction section should be as small and fast as possible
- Avoid long-running transactions by splitting up big transaction to smaller worker batches
This implementation heavily draws inspiration from the following sources
- http://stackoverflow.com/q/27351433/315168
- https://gist.github.com/khayrov/6291557
"""
def decorated_func():
# Read attemps from app configuration
attempts = self.retries
while attempts >= 0:
session = self.session_factory()
try:
result = func(session)
session.commit()
self.stats["success"] += 1
return result
except Exception as e:
if self.is_retryable_exception(e):
session.close()
self.stats["retries"] += 1
attempts -= 1
if attempts < 0:
self.stats["unresolved"] += 1
raise CannotResolveDatabaseConflict("Could not replay the transaction {} even after {} attempts".format(func, self.retries)) from e
continue
else:
session.rollback()
self.stats["errors"] += 1
# All other exceptions should fall through
raise
return decorated_func

Postgres and Oracle conflict errors are marked as retryable by zope.sqlalchemy. Set your isolation level in the engine configuration and the transaction retry logic in pyramid_tm or Zope will work.

Advice on Pyramid views exception handling

There are three situations when I need to handle exceptions.
When data validation raised exception
When library/module functions raised exceptions (e.g. database connection abort)
When business logic raises exception such as 500, 503, 401, 403 and 404
def library_func():
try:
...
except HTTPException:
raise TwitterServiceException("Twitter is down!")
#view_config(route_name="home", renderer="json")
#validator
#authorization
def home_view(request):
try:
tweets = library_func()
return {"tweets": tweets}
except TwitterServiceException as e:
LOG.critical(e.msg)
raise ParnterServcieError(e.msg) # this is probably a 503 error
def validator(args):
# I will show the high level of this decorator
try:
decode input as JSON
verify data format
except ValueError as err:
error = {'error': "Missing required parameters."}
except json.JSONDecodeError as err:
error = {'error': "Failed to decode the incoming JSON payload."}
if error is not None:
return HTTPBadRequest(body=json.dumps(error),
content_type='application/json')
def authorization(args):
# very similar to validator except it performs authorization and if failed
# 401 is raised with some helpful message.
The doc suggests Custom Exception Views. In my PoC above, I will tie ParnterServcieError as one. I can even generalize HTTPBadRequest and all praymid.httpexceptions using custom exception so that I no longer need to repeat json.dumps and content_type. I can set a boilerplate error body before I return request.response object.
Idea:
#view_config(context=ParnterServcieError)
def 503_service_error_view(e, request):
request.response.status = 503
request.response.json_body = {"error": e.msg}
return request.response
I can generalize one for all uncaught, unspecified exceptions (which results in 500 Internal Server Error) called 500_internal_server_error_view.
Does this seem sane and clean to people? Is my way of handling high and low level of exceptions proper and Pythonic?

I applied this strategy to ToDoPyramid and could encapsulate error handling in a single custom exception view that was repeated multiple times in the application before. Until you could even improve it, you got a great idea. Pyramid rocks.
References
Catching database connection error in ToDoPyramid