I have a Flask application that uses SQLAlchemy (with some Marshmallow for serialization and deserialization).
I'm currently encountering some intermittent issues when trying to dump an object post-commit.
To give an example, let's say I have implemented a (multi-tenant) system for tracking system faults of some sort. This information is contained in a fault table:
class Fault(Base):
__tablename__ = "fault"
fault_id = Column(BIGINT, primary_key=True)
workspace_id = Column(Integer, ForeignKey('workspace.workspace_id'))
local_fault_id = Column(Integer)
name = Column(String)
description = Column(String)
I've removed a number of columns in the interest of simplicity, but this is the core of the model. The columns should be largely self explanatory, with workspace_id effectively representing tenant, and local_fault_id representing a tenant-specific fault sequence number, which is handled via a separate fault_sequence table.
This fault_sequence table holds a counter against workspace, and is updated by means of a simple on_fault_created() function that is executed by a trigger:
CREATE TRIGGER fault_created
AFTER INSERT
ON "fault"
FOR EACH ROW
EXECUTE PROCEDURE on_fault_created();
So - the problem:
I have a Flask endpoint for fault creation, where we create an instance of a Fault entity, add this via a scoped session (session.add(fault)), then call session.commit().
It seems that this is always successful in creating the desired entities in the database, executing the sequence update trigger etc. However, when I then try to interrogate the fault object for updated fields (after commit()), around 10% of the time I find that each key/field just points to an Exception:
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block
Which seems to boil down to the following:
(psycopg2.errors.InvalidTextRepresentation) invalid input syntax for integer: ""
[SQL: SELECT fault.fault_id AS fault_fault_id, fault.workspace_id AS fault_workspace_id, fault.local_fault_id AS fault_local_fault_id, fault.name as fault_name, fault.description as fault_description
FROM fault
WHERE fault.fault_id = %(param_1)s]
[parameters: {'param_1': 166}]
(Background on this error at: http://sqlalche.me/e/13/2j8
My question, then, is what do we think could be causing this?
I think it smells like a race condition, with the update trigger not being complete before SQLAlchemy has tried to get the updated data; perhaps local_fault_id is null, and this is resulting in the invalid input syntax error.
That said, I have very low confidence on this. Any guidance here would be amazing, as I could really do with retrieving that sequence number that's incremented/handled by the update trigger.
Thanks
Edit 1:
Some more info:
I have tried removing the update trigger, in the hope of eliminating that as a suspect. This behaviour is still intermittently evident, so I don't think it's related to that.
I have tried adopting usage of flush and refresh before the commit, and this allows me to get the values that I need - though commit still appears to 'break' the fault object.
Edit 2:
So it really seems to be more postgres than anything else. When I interrogate my database logs, this is the weirdest thing. I can copy and paste the command it says is failing, and I struggle to see how this integer value in the WHERE clause is possibly evaluating to an empty string.
This same error is reproducible with SELECT ... FROM fault WHERE fault.fault_id = '', which in no way seems to be the query making to the DB.
I am stumped.
Your sentence "This same error is reproducible with SELECT ... FROM fault WHERE fault.fault_id = '', which in no way seems to be the query making to the DB." seems to indicate that you are trying to access an object that does not have the database primary key "fault_id".
I guess, given that you did not provide the code, that you are adding the object to your session (session.add), committing (session.commit) and then using the object. As fault_id is autogenerated by the database, the fault object in the session (in memory) does not have fault_id.
I believe you can correct this with:
session.add(fault)
session.commit()
session.refresh(fault)
The refresh needs to be AFTER commit to refresh the fault object and retrieve fault_id.
If you are using async, you need
session.add(fault)
await session.commit()
await session.refresh(fault)
Related
I have a flask project that relies on flask-sqlalchemy and celery to do a lot of things. Many of the celery tasks reach out to external API's, fetch some data, and read/update data on disk. When expanding my number of tasks that are being run, I see a lot of 40001 errors such as
(psycopg2.errors.SerializationFailure) could not serialize access due to read/write dependencies among transactions
DETAIL: Reason code: Canceled on identification as a pivot, during write.
HINT: The transaction might succeed if retried.
And other similar errors, all relating to transactions failing. We are using serializable transactions in SQLAlchemy.
I've tried to rerun the tasks using the builtin celery auto-retry features, but I think an architectural change needs to be made, since the tasks are still failing, even with several retries. The reading from remote APIs and storing to database logic is mingled in the tasks and my first idea was to separate this logic out as much as possible, first reading from database and API, then sending another task that would make changes to API/database, but I'm not sure this would help. I don't want to wrap every single call to db.session.query(Object) or similar in try/catch clauses if there's a better way?
Code looks like this (heavily redacted)
#celery_app.task(
bind=True,
autoretry_for=(
HTTPError,
sqlalchemy.exc.OperationalError,
sqlalchemy.exc.IntegrityError,
),
retry_backoff=5,
retry_jitter=True,
retry_kwargs={"max_retries": 5},
)
def sync_db_objects_task(self, instance_id: int):
instance = ObjectInstance.query.get(instance_id)
if not instance:
return
# Run a query fetching objects of interest
latest_local_tickets = db.session.query(..)
if all([t.deleted for t in latest_local_tickets]):
return
remote_config = (instance.config.address, instance.config.port, instance.config.username, instance.config.password)
# determine which ticket objects are out of sync:
out_of_sync_tickets = out_of_sync_tickets(latest_local_tickets, remote_config)
successes, fails = [],[]
for result in out_of_sync_tickets:
db_object = latest_local_tickets[result.name]
try:
#Sync
#connect, update
status, reason = connect.sync(db_object)
successes.append(db_object.name)
except HTTPError as e:
status, reason = e.status_code, e.msg
fails.append(db_object.name)
db_object.update_state(status, reason)
db.session.add(db_object)
db.session.commit()
logger.info(f"successes: {successes}")
logger.info(f"fails:{fails}")
It usually fails in the call to update_state which looks like this:
#declarative_mixin
class ReplicationMixin:
status = Column(types.Text, default="unknown", nullable=False)
status_reason = Column(types.Text, nullable=True)
last_sync_date = Column(types.DateTime, default=None)
def update_state(status:str, reason:str=None):
self.status = status if status else "fail"
self.status_reason = parse_reason(reason)
self.last_sync_date = datetime.utcnow()
Like this:
[2022-07-29 09:22:11,305: ERROR/ForkPoolWorker-8] (psycopg2.errors.SerializationFailure) could not serialize access due to read/write dependencies among transactions
DETAIL: Reason code: Canceled on identification as a pivot, during write.
HINT: The transaction might succeed if retried.
[SQL: UPDATE tickets SET updated=%(updated)s, last_sync_date=%(last_sync_date)s WHERE tickets.id = %(tickets_id)s]
[parameters: {'updated': datetime.datetime(2022, 7, 29, 9, 22, 11, 304293), 'last_sync_date': datetime.datetime(2022, 7, 29, 9, 22, 11, 303875), 'tickets_id': 124}]
(Background on this error at: https://sqlalche.me/e/14/e3q8)
But i think it's a red herring, as its probably being subsequently read somewhere else.
Can I attempt to leverage db.session.refresh(object) or something else to make sure I commit and re-read data before changing it, or is there some other way to alleviate this issue?
I've been stumped for a week on this issue and I cant seem to figure out what the best way forward is. Very appreciative for any help I can get.
I'm trying to get a list of warnings after a mySQL query, using Django admin. I can see from the documentation here that it's possible to record warnings by setting connection.get_warnings to true. But I can't find anything explaining how to read those warnings.
I do not want to throw an exception - I am deleting items from the database using a DELETE IGNORE statement and want to get all instances of deletions that failed (due to external keys, etc.)
I've tried returning the result of the execute function itself (just gave me a number) and calling fetchwarnings() on the cursor (threw a "Cursor object has no attribute fetchwarnings" error).
I'm still new to both Python and Django. I'm looking through all the documentation I can find but can't find anything that works.
from django.db import connection
query = "{query here}"
connection.get_warnings = True
with connection.cursor() as cursor:
cursor.execute(query) <-- Returns a number
return cursor.fetchwarnings() <-- Throws an error
I am trying to add username, user id and many other variables after the user press /start in Telegram bot. But I am getting an error in this code
def add_user(userid, username):
userCount = c.execute(f"COUNT({userIdColumn} FROM {info_table} WHERE {userIdColumn}={userid}")
if(userCount <= 0):
c.execute(f"INSERT INTO {info_table}({userIdColumn},{usernameColumn},{isBannedColumn},{isSubscribedColumn},{userNotesColumn}) VALUES ({userid},{username},0,1,' ')")
c.commit()
db.close()
I tried declaring cursor() and even database inside the function but the errors I get are
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 20532 and this is thread id 19184.
sqlite3.OperationalError: near "COUNT": syntax error
I am also aware that there is a post about the same problem which unfortunately did not solved my problem.
My question is what is the correct declaration here?
The fix for the first error was adding check_same_thread=False as a paramter = sqlite3.connect("Your db",check_same_thread=False) and also you need to declare the cursor() inside the function
The second error was caused by wrong format as it should be SELECT COUNT
I am trying to delete features using the delete_features method on the FeatureLayer Object and I keep getting the following error: "This SqlTransaction has completed; it is no longer usable."
The code is below. The error message seems to populate in the last line where="OBJECTID >=0", but I'm not a 100 sure if this is the problem. Unfortunately I'm not very good at programming.
gis = arcgis.GIS("http://gfcgis.maps.arcgis.com", "UserName", "Password")
feature_layer_item = gis.content.search(FeatureLayer, item_type = 'Feature Service')[0]
flayers = feature_layer_item.layers
flayer = flayers[0]
flayer.delete_features(where="OBJECTID >= 0", rollback_on_failure=True)
Any help would be greatly appreciated.
Michael
This sounds like a zombie transaction. Check with your DBA if there's a query, most likely a Stored Procedure, which is being called when your code runs. This message usually shows up when the application code tries to do a commit on the DB after the SP has already committed.
That's the SQL Transaction which has already completed.
Come to find out, it was a simple syntax error causing the issue. I didn't put quotations around 'True' for the rollback_on_failure parameter.
My Python script uses an ADODB.Recordset object. I use an ADODB.Command object with a collection of ADODB.Parameter objects to update a record in the set. After that, I check the state of the recordset, and it was 1, which is adStateOpen. But when I call MyRecordset.Close(), I get an exception complaining that the operation is invalid in the set's current state. What state could an open recordset be in that would make it invalid to close it, and what can I do to fix it?
Code is scattered between a couple of files. I'll work on getting an illustration together.
Yes, that was the problem. Once I change the value of one of a recordset's ADODB.Field objects, I have to either update the recordset using ADODB.Recordset.Update() or call CancelUpdate().
The reason I'm going through all this rigarmarole of the ADODB.Command object is that ADODB.Recordset.Update() fails at random (or so it seems to me) times, complaining that "query-based update failed because row to update cannot be found". I've never been able to predict when that will happen or find a reliable way to keep it from happening. My only choice when that happens is to replace the ADODB.Recordset.Update() call with the construction of a complete update query and executing it using an ADODB.Connection or ADODB.Command object.