PostgreSQL and SQL defines a Serializable transaction isolation level. If you isolate transactions to this level, conflicting concurrent transactions abort and need retrying.
I am familiar with the concept of transaction retries from Plone / Zope world where the entire HTTP request can be replayed in the case there is a transaction conflict. How similar functionality could be achieved with SQLAlchemy (and potentially with zope.sqlalchemy)? I tried to read the documentation of zope.sqlalchemy and Zope transaction manager, but this is not obvious the me.
Specially I want something like this:
# Try to do the stuff, if it fails because of transaction conflict do again until retry count is exceeded
with transaction.manager(retries=3):
do_stuff()
# If we couldn't get the transaction through even after 3 attempts, fail with a horrible exception
So, after poking around two weeks and getting no off-the-shelf solution I came up with my own.
Here is a ConflictResolver class which provides managed_transaction function decorator. You can use the decorator to mark functions to be retryable. I.e. if there is an database conflict error when running the function, the function is run again, now with more hopes the db transaction which caused the conflict error would have finished.
The source code is here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
The unit tests to cover it are here: https://bitbucket.org/miohtama/cryptoassets/src/529c50d74972ff90fe5b61dfbfc1428189cc248f/cryptoassets/core/tests/test_conflictresolver.py?at=master
Python 3.4+ only.
"""Serialized SQL transaction conflict resolution as a function decorator."""
import warnings
import logging
from collections import Counter
from sqlalchemy.orm.exc import ConcurrentModificationError
from sqlalchemy.exc import OperationalError
UNSUPPORTED_DATABASE = "Seems like we might know how to support serializable transactions for this database. We don't know or it is untested. Thus, the reliability of the service may suffer. See transaction documentation for the details."
#: Tuples of (Exception class, test function). Behavior copied from _retryable_errors definitions copied from zope.sqlalchemy
DATABASE_COFLICT_ERRORS = []
try:
import psycopg2.extensions
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((psycopg2.extensions.TransactionRollbackError, None))
# ORA-08177: can't serialize access for this transaction
try:
import cx_Oracle
except ImportError:
pass
else:
DATABASE_COFLICT_ERRORS.append((cx_Oracle.DatabaseError, lambda e: e.args[0].code == 8177))
if not DATABASE_COFLICT_ERRORS:
# TODO: Do this when cryptoassets app engine is configured
warnings.warn(UNSUPPORTED_DATABASE, UserWarning, stacklevel=2)
#: XXX: We need to confirm is this the right way for MySQL, SQLIte?
DATABASE_COFLICT_ERRORS.append((ConcurrentModificationError, None))
logger = logging.getLogger(__name__)
class CannotResolveDatabaseConflict(Exception):
"""The managed_transaction decorator has given up trying to resolve the conflict.
We have exceeded the threshold for database conflicts. Probably long-running transactions or overload are blocking our rows in the database, so that this transaction would never succeed in error free manner. Thus, we need to tell our service user that unfortunately this time you cannot do your thing.
"""
class ConflictResolver:
def __init__(self, session_factory, retries):
"""
:param session_factory: `callback()` which will give us a new SQLAlchemy session object for each transaction and retry
:param retries: The number of attempst we try to re-run the transaction in the case of transaction conflict.
"""
self.retries = retries
self.session_factory = session_factory
# Simple beancounting diagnostics how well we are doing
self.stats = Counter(success=0, retries=0, errors=0, unresolved=0)
#classmethod
def is_retryable_exception(self, e):
"""Does the exception look like a database conflict error?
Check for database driver specific cases.
:param e: Python Exception instance
"""
if not isinstance(e, OperationalError):
# Not an SQLAlchemy exception
return False
# The exception SQLAlchemy wrapped
orig = e.orig
for err, func in DATABASE_COFLICT_ERRORS:
# EXception type matches, now compare its values
if isinstance(orig, err):
if func:
return func(e)
else:
return True
return False
def managed_transaction(self, func):
"""SQL Seralized transaction isolation-level conflict resolution.
When SQL transaction isolation level is its highest level (Serializable), the SQL database itself cannot alone resolve conflicting concurrenct transactions. Thus, the SQL driver raises an exception to signal this condition.
``managed_transaction`` decorator will retry to run everyhing inside the function
Usage::
# Create new session for SQLAlchemy engine
def create_session():
Session = sessionmaker()
Session.configure(bind=engine)
return Session()
conflict_resolver = ConflictResolver(create_session, retries=3)
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
# Both threads modify the same wallet simultaneously
w = session.query(BitcoinWallet).get(1)
w.balance += 1
# Execute the conflict sensitive code inside a managed transaction
myfunc()
The rules:
- You must not swallow all exceptions within ``managed_transactions``. Example how to handle exceptions::
# Create a decorated function which can try to re-run itself in the case of conflict
#conflict_resolver.managed_transaction
def myfunc(session):
try:
my_code()
except Exception as e:
if ConflictResolver.is_retryable_exception(e):
# This must be passed to the function decorator, so it can attempt retry
raise
# Otherwise the exception is all yours
- Use read-only database sessions if you know you do not need to modify the database and you need weaker transaction guarantees e.g. for displaying the total balance.
- Never do external actions, like sending emails, inside ``managed_transaction``. If the database transaction is replayed, the code is run twice and you end up sending the same email twice.
- Managed transaction section should be as small and fast as possible
- Avoid long-running transactions by splitting up big transaction to smaller worker batches
This implementation heavily draws inspiration from the following sources
- http://stackoverflow.com/q/27351433/315168
- https://gist.github.com/khayrov/6291557
"""
def decorated_func():
# Read attemps from app configuration
attempts = self.retries
while attempts >= 0:
session = self.session_factory()
try:
result = func(session)
session.commit()
self.stats["success"] += 1
return result
except Exception as e:
if self.is_retryable_exception(e):
session.close()
self.stats["retries"] += 1
attempts -= 1
if attempts < 0:
self.stats["unresolved"] += 1
raise CannotResolveDatabaseConflict("Could not replay the transaction {} even after {} attempts".format(func, self.retries)) from e
continue
else:
session.rollback()
self.stats["errors"] += 1
# All other exceptions should fall through
raise
return decorated_func
Postgres and Oracle conflict errors are marked as retryable by zope.sqlalchemy. Set your isolation level in the engine configuration and the transaction retry logic in pyramid_tm or Zope will work.
Related
I need to write historic data into InfluxDB (I'm using Python, which is not a must in this case, so I maybe willing to accept non-Python solutions). I set up the write API like this
write_api = client.write_api(write_options=ASYNCHRONOUS)
The Data comes from a DataFrame with a timestamp as key, so I write it to the database like this
result = write_api.write(bucket=bucket, data_frame_measurement_name=field_key, record=a_data_frame)
This call does not throw an exception, even if the InfluxDB server is down. result has a protected attribute _success that is a boolean in debugging, but I cannot access it from the code.
How do I check if the write was a success?
If you use background batching, you can add custom success, error and retry callbacks.
from influxdb_client import InfluxDBClient
def success_cb(details, data):
url, token, org = details
print(url, token, org)
data = data.decode('utf-8').split('\n')
print('Total Rows Inserted:', len(data))
def error_cb(details, data, exception):
print(exc)
def retry_cb(details, data, exception):
print('Retrying because of an exception:', exc)
with InfluxDBClient(url, token, org) as client:
with client.write_api(success_callback=success_cb,
error_callback=error_cb,
retry_callback=retry_cb) as write_api:
write_api.write(...)
If you are eager to test all the callbacks and don't want to wait until all retries are finished, you can override the interval and number of retries.
from influxdb_client import InfluxDBClient, WriteOptions
with InfluxDBClient(url, token, org) as client:
with client.write_api(success_callback=success_cb,
error_callback=error_cb,
retry_callback=retry_cb,
write_options=WriteOptions(retry_interval=60,
max_retries=2),
) as write_api:
...
if you want to immediately write data into database, then use SYNCHRONOUS version of write_api - https://github.com/influxdata/influxdb-client-python/blob/58343322678dd20c642fdf9d0a9b68bc2c09add9/examples/example.py#L12
The asynchronous write should be "triggered" by call .get() - https://github.com/influxdata/influxdb-client-python#asynchronous-client
Regards
write_api.write() returns a multiprocessing.pool.AsyncResult or multiprocessing.pool.AsyncResult (both are the same).
With this return object you can check on the asynchronous request in a couple of ways. See here: https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.AsyncResult
If you can use a blocking request, then write_api = client.write_api(write_options=SYNCRONOUS) can be used.
from datetime import datetime
from influxdb_client import WritePrecision, InfluxDBClient, Point
from influxdb_client.client.write_api import SYNCHRONOUS
with InfluxDBClient(url="http://localhost:8086", token="my-token", org="my-org", debug=False) as client:
p = Point("my_measurement") \
.tag("location", "Prague") \
.field("temperature", 25.3) \
.time(datetime.utcnow(), WritePrecision.MS)
try:
client.write_api(write_options=SYNCHRONOUS).write(bucket="my-bucket", record=p)
reboot = False
except Exception as e:
reboot = True
print(f"Reboot? {reboot}")
I want to establish a persistent sessionkey for use in all my tests:
class TestAuthorisedViews(TestCase):
#classmethod
def setUpTestData(cls):
# Establish the client and log in for all views
cls.client = Client()
cls.client.force_login(user=cls.standard_user)
# Set a session key
cls.session['mykey'] = 1
cls.session.save()
However, when used in tests I get a KeyError:
def test_view_url_accessible_by_name(self):
print('session', self.client.session['mykey'])
Results in KeyError: 'mykey'
My understanding from the docs is that this is because a new SessionStore is created every time this property is accessed.
Do I understand this correctly or is something else going on here? How can I create a session key in setUpTestData for use in subsequent tests?
I have some trouble understanding a sequence of events causing a bug in my appplication which can only be seen intermittently in the app deployed on GAE, and never when running with the local devserver.py.
All the related code snippets below (trimmed for MCV, hopefully I didn't lose anything significant) are executed during handling of the same task queue request.
The entry point:
def job_completed_task(self, _):
# running outside transaction as query is made
if not self.all_context_jobs_completed(self.context.db_key, self):
# this will transactionally enqueue another task
self.trigger_job_mark_completed_transaction()
else:
# this is transactional
self.context.jobs_completed(self)
The corresponding self.context.jobs_completed(self) is:
#ndb.transactional(xg=True)
def jobs_completed(self, job):
if self.status == QAStrings.status_done:
logging.debug('%s jobs_completed %s NOP' % (self.lid, job.job_id))
return
# some logic computing step_completed here
if step_completed:
self.status = QAStrings.status_done # includes self.db_data.put()
# this will transactionally enqueue another task
job.trigger_job_mark_completed_transaction()
The self.status setter, hacked to obtain a traceback for debugging this scenario:
#status.setter
def status(self, new_status):
assert ndb.in_transaction()
status = getattr(self, self.attr_status)
if status != new_status:
traceback.print_stack()
logging.info('%s status change %s -> %s' % (self.name, status, new_status))
setattr(self, self.attr_status, new_status)
The job.trigger_job_mark_completed_transaction() eventually enqueues a new task like this:
task = taskqueue.add(queue_name=self.task_queue_name, url=url, params=params,
transactional=ndb.in_transaction(), countdown=delay)
The GAE log for the occurence, split as it doesn't fit into a single screen:
My expectation from the jobs_completed transaction is to either see the ... jobs_completed ... NOP debug message and no task enqueued or to at least see the status change running -> done info message and a task enqueued by job.trigger_job_mark_completed_transaction().
What I'm actually seeing is both messages and no task enqueued.
The logs appears to indicate the transaction is attempted twice:
1st time it finds the status not done, so it executes the logic, sets the status to done (and displays the traceback and the info msg) and should transactionally enqueue the new task - but it doesn't
2nd time it finds the status done and just prints the debug message
My question is - if the 1st transaction attempt fails shouldn't the status change be rolled back as well? What am I missing?
I found a workaround: specifying no retries to the jobs_completed() transaction:
#ndb.transactional(xg=True, retries=0)
def jobs_completed(self, job):
This prevents the automatic repeated execution, instead causing an exception:
TransactionFailedError(The transaction could not be committed. Please
try again.)
Which is acceptable as I already have in place a back-off/retry safety net for the entire job_completed_task(). Things are OK now.
As for why the rollback didn't happen, the only thing that crosses my mind is that somehow the entity was read (and cached in my object attribute) prior to entering the transaction, thus not being considered part of the (same) transaction. But I couldn't find a code path that would do that, so it's just speculation.
I have a very unreliable API that I request using Python. I have been thinking about using requests_cache and setting expire_after to be 999999999999 like I have seen other people do.
The only problem is, I do not know if when the API works again, that if the data is updated. If requests_cache will automatically auto-update and delete the old entry.
I have tried reading the docs but I cannot really see this anywhere.
requests_cache will not update until the expire_after time has passed. In that case it will not detect that your API is back to a working state.
I note that the project has since added an option that I implemented in the past; you can now set the old_data_on_error option when configuring the cache; see the CachedSession documentation:
old_data_on_error – If True it will return expired cached response if update fails.
It would reuse existing cache data in case a backend update fails, rather than delete that data.
In the past, I created my own requests_cache session setup (plus small patch) that would reuse cached values beyond expire_after if the backend gave a 500 error or timed out (using short timeouts) to deal with a problematic API layer, rather than rely on expire_after:
import logging
from datetime import (
datetime,
timedelta
)
from requests.exceptions import (
ConnectionError,
Timeout,
)
from requests_cache.core import (
dispatch_hook,
CachedSession,
)
log = logging.getLogger(__name__)
# Stop logging from complaining if no logging has been configured.
log.addHandler(logging.NullHandler())
class FallbackCachedSession(CachedSession):
"""Cached session that'll reuse expired cache data on timeouts
This allows survival in case the backend is down, living of stale
data until it comes back.
"""
def send(self, request, **kwargs):
# this *bypasses* CachedSession.send; we want to call the method
# CachedSession.send() would have delegated to!
session_send = super(CachedSession, self).send
if (self._is_cache_disabled or
request.method not in self._cache_allowable_methods):
response = session_send(request, **kwargs)
response.from_cache = False
return response
cache_key = self.cache.create_key(request)
def send_request_and_cache_response(stale=None):
try:
response = session_send(request, **kwargs)
except (Timeout, ConnectionError):
if stale is None:
raise
log.warning('No response received, reusing stale response for '
'%s', request.url)
return stale
if stale is not None and response.status_code == 500:
log.warning('Response gave 500 error, reusing stale response '
'for %s', request.url)
return stale
if response.status_code in self._cache_allowable_codes:
self.cache.save_response(cache_key, response)
response.from_cache = False
return response
response, timestamp = self.cache.get_response_and_time(cache_key)
if response is None:
return send_request_and_cache_response()
if self._cache_expire_after is not None:
is_expired = datetime.utcnow() - timestamp > self._cache_expire_after
if is_expired:
self.cache.delete(cache_key)
# try and get a fresh response, but if that fails reuse the
# stale one
return send_request_and_cache_response(stale=response)
# dispatch hook here, because we've removed it before pickling
response.from_cache = True
response = dispatch_hook('response', request.hooks, response, **kwargs)
return response
def basecache_delete(self, key):
# We don't really delete; we instead set the timestamp to
# datetime.min. This way we can re-use stale values if the backend
# fails
try:
if key not in self.responses:
key = self.keys_map[key]
self.responses[key] = self.responses[key][0], datetime.min
except KeyError:
return
from requests_cache.backends.base import BaseCache
BaseCache.delete = basecache_delete
The above subclass of CachedSession bypasses the original send() method to instead go directly to the original requests.Session.send() method, to return existing cached value even if the timeout has passed but the backend has failed. Deletion is disabled in favour of setting the timeout value to 0, so we can still reuse that old value if a new request fails.
Use the FallbackCachedSession instead of a regular CachedSession object.
If you wanted to use requests_cache.install_cache(), make sure to pass in FallbackCachedSession to that function in the session_factory keyword argument:
import requests_cache
requests_cache.install_cache(
'cache_name', backend='some_backend', expire_after=180,
session_factory=FallbackCachedSession)
My approach is a little more comprehensive than what requests_cache implemented some time after I hacked together the above; my version will fall back to a stale response even if you explicitly marked it as deleted before.
Try to do something like that:
class UnreliableAPIClient:
def __init__(self):
self.some_api_method_cached = {} # we will store results here
def some_api_method(self, param1, param2)
params_hash = "{0}-{1}".format(param1, param2) # need to identify input
try:
result = do_call_some_api_method_with_fail_probability(param1, param2)
self.some_api_method_cached[params_hash] = result # save result
except:
result = self.some_api_method_cached[params_hash] # resort to cached result
if result is None:
raise # reraise exception if nothing cached
return result
Of course you can make simple decorator with that, up to you - http://www.artima.com/weblogs/viewpost.jsp?thread=240808
I have a python application which has lots of small database access functions, using sqlalchemy. I'm trying to avoid having lots of boilerplate session handling code around these functions.
I have numerous functions that look something like this:
def get_ticket_history(Session, ticket_id):
s = Session()
try:
rows = s.query(TicketHistory)\
.filter(TicketHistory.ticket_fk==ticket_id)\
.order_by(TicketHistory.id.desc()).all()
s.commit()
return rows
except:
s.rollback()
raise
finally:
s.close()
I am trying to refactor these functions, but not sure I have the best approach yet. The best I currently have is the following:
def execute(Session, fn, *args, **kwargs):
s = Session()
try:
ret = fn(s, *args, **kwargs)
s.commit()
return ret
except:
s.rollback()
raise
finally:
s.close()
def get_ticket_history(self, ticket_id):
def sql_fn(s):
return s.query(TicketHistory)\
.filter(TicketHistory.ticket_fk==ticket_id)\
.order_by(TicketHistory.id.desc()).all()
return execute(self.sentinel_session, sql_fn)
Is there a better or more idiomatic way of doing this? Perhaps using a decorator?
Thanks,
Jon
The SQLAlchemy docs present a possible way of doing this with context managers.
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
Copying the code snippet here for completeness:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
This session_scope can be used cleanly without repeating the boiler plate now.
class ThingOne(object):
def go(self, session):
session.query(FooBar).update({"x": 5})
class ThingTwo(object):
def go(self, session):
session.query(Widget).update({"q": 18})
def run_my_program():
with session_scope() as session:
ThingOne().go(session)
ThingTwo().go(session)
From Sql alchemy version 1.4:
The Session may be used as a context manager without the use of external helper functions.
Example From documentation
Session = sessionmaker(engine)
with Session() as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
To begin, commit transaction and close the session, below approach can be applied.
Session = sessionmaker(engine)
with Session.begin() as session:
session.add(some_object)
session.add(some_other_object)
# commits transaction, closes session
Documentation:
https://docs.sqlalchemy.org/en/14/orm/session_api.html#sqlalchemy.orm.sessionmaker
morphyn's suggestion to use a context manager is good. You could make such a context manager by applying the contextlib.contextmanager decorator to a function very much like your first get_ticket_history, replacing the code between try and except with a yield statement and renaming it, say, transaction. PEP 343 has a near-identical example of that name.
Then, use that context manager with the with statement to reimplement get_ticket_history. It looks like SQLAlchemy already provides that function, though, as method begin:
http://docs.sqlalchemy.org/en/rel_0_8/orm/session.html#autocommit-mode
transaction handling (begin, commit/rolllback) using with clause
with engine.begin() as connection:
r1 = connection.execute(table1.select())
connection.execute(table1.insert(), {"col1": 7, "col2": "this is some data"})
Old questions, but I still stumbled upon it so here is the relevant link from the docu:
https://docs.sqlalchemy.org/en/13/core/connections.html