As part of the upgrade process, our product scripts update a stored procedure for a trigger. There are two daemons running, either of which can update the stored procedure. It seems that PostgrSQL is not serializing the DDL to upgrade the procedure. The exact error is "DB_Cursor: exception in execute: tuple concurrently updated". A Google search yields no exact matches for this error in the search results. It would appear we have a race condition. What is the best approach for avoiding or preventing such an exception? It prevents the upgrade process from succeeding and one or both of the processes (daemons) must be restarted to retry the upgrade and recover. Is there known issue with PostgreSQL? We are running PostgreSQL 9.2.5.
It seems that PostgreSQL is not serializing the DDL to upgrade the
procedure
Yes. This is mentioned from time to time on pgsql mailing lists, for example recently here:
'tuple concurrently updated' error when granting permissions
Excerpt:
We do have such locking for DDL on tables/indexes, but the theory in
the past has been that it's not worth the trouble for objects
represented by single catalog rows, such as functions or roles. You
can't corrupt the database with concurrent updates on such a row,
you'll just get a "tuple concurrently updated" error from all but the
first-to-arrive update.
If you're concurrently replacing functions bodies, this is clearly your problem.
And the proposed solution is:
In the meantime, you could consider using an application-managed
advisory lock if you really need such grants to work transparently.
If by design multiple concurrent clients can decide to perform DDL, then you really should make sure only one of them is doing it. You can do it using advisory locks.
Example in pseudocode:
function try_upgrade(db) {
if ( ! is_upgrade_needed(db) ) {
// we check it before acquiring a lock to speed up a common case of
// no upgrade available
return UPGRADE_NOT_NEEDED;
}
query_result = db->begin_transaction();
if ( query_result < 0 ) throw Error("begin failed");
query_result = db->query(
"select pg_advisory_xact_lock(?)", MAGIC_NUMBER_UPGRADE_LOCK
);
if ( query_result < 0 ) throw Error("pg_advisory_xact_lock failed");
// another client might have performed upgrade between the previous check
// and acquiring advisory lock
if ( ! is_upgrade_needed(db) ) {
query_result = db->rollback_transaction();
return UPGRADE_NOT_NEEDED;
}
perform_upgrade();
query_result = db->commit_transaction();
if ( query_result < 0 ) throw Error("commit failed");
return UPGRADE_PERFORMED;
}
Related
I have a very important webhook which call my lambda function. The issue is this webhook is hitting my lambda function thrice with same data. I don't want to process thrice. I want to exit if it's already being called. I tried to store the data (paid) in dynamo db and check if it's already present but that ain't working. it's like the db is not atomic.
I call below method before executing the code.
def check_duplicate_webhook(user_id, order_id):
try:
status = dynamodb_table_payment.get_item(Key={'user_id': user_id},
ProjectionExpression='payments.#order_id.#pay_status',
ExpressionAttributeNames={
"#order_id": order_id,
'#pay_status': "status"
})
if "Item" in status and "payments" in status['Item']:
check = status['Item']['payments'][order_id]
if check == 'paid':
return True
return False
except Exception as e:
log(e)
return False
Updating the database
dynamodb_table_payment.update_item(Key={'user_id': user_id},
UpdateExpression="SET payments.#order_id.#pay_status = :pay_status, "
"payments.#order_id.#update_date = :update_date, "
"payments.#order_id.reward = :reward_amount",
ExpressionAttributeNames={
"#order_id": attr['order_id'],
'#pay_status': "status",
'#update_date': 'updated_at'
},
ExpressionAttributeValues={
":pay_status": 'paid',
':update_date': int(time.time()),
':reward_amount': reward_amount
})
DynamoDB isn't atomic and if the three requests come very close together, it could happen that the read value isn't consistent. For financial transactions it is recommended to use DynamoDB transactions.
May I also suggest that you use Step Functions and decouple the triggering from the actual execution. The webhook will trigger a function that will register the payment for execution. A different function will execute it. You will need some orchestration in the future, if for not anything else, to implement a retry logic.
You're separating the retrieve and update, which can cause a race condition. You should be able to switch to a put_item() with condition, which will only insert once (or optionally update if the criteria are met).
You could also use a FIFO SQS queue as an intermediary between the webhook and Lambda, and let it do the deduplication. But that's a more complex solution.
It also appears that you're storing all orders for a given customer in a single record in DynamoDB. This seems like a bad idea to me: first because you need more RCUs/WCUs to be able to retrieve larger records, second because you will eventually bump up against the size limit of a DynamoDB record, and third because it makes the update logic more complex. I think you would be better to manage orders separately, using a key of (user_id, order_id).
I need to apply a query timeout at a global level in my application. The query: SET SESSION max_execution_time=1 does this with MySQL 5.7. I am using MySQL 5.6 and cannot upgrade at the moment. Any solution with SQL Alchemy would also help.
It seems there is no equivalent to max_execution_time in MySQL prior to versions 5.7.4 and 5.7.8 (the setting changed its name). What you can do is create your own periodic job that checks if queries have exceeded timeout and manually kill them. Unfortunately that is not quite the same as what the newer MySQL versions do: without inspecting the command info you'll end up killing all queries, not just read only SELECT, and it is nigh impossible to control at session level.
One way to do that would be to create a stored procedure that queries the process list and kills as required. Such stored procedure could look like:
DELIMITER //
CREATE PROCEDURE stmt_timeout_killer (timeout INT)
BEGIN
DECLARE query_id INT;
DECLARE done INT DEFAULT FALSE;
DECLARE curs CURSOR FOR
SELECT id
FROM information_schema.processlist
WHERE command = 'Query' AND time >= timeout;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
-- Ignore ER_NO_SUCH_THREAD, in case the query finished between
-- checking the process list and actually killing threads
DECLARE CONTINUE HANDLER FOR 1094 BEGIN END;
OPEN curs;
read_loop: LOOP
FETCH curs INTO query_id;
IF done THEN
LEAVE read_loop;
END IF;
-- Prevent suicide
IF query_id != CONNECTION_ID() THEN
KILL QUERY query_id;
END IF;
END LOOP;
CLOSE curs;
END//
DELIMITER ;
Alternatively you could implement all that in your application logic, but it would require separate round trips to the database for each query to be killed. What's left then is to call this periodically:
# Somewhere suitable
engine.execute(text("CALL stmt_timeout_killer(:timeout)"), timeout=30)
How and where exactly depends heavily on your actual application.
Situation: I have a live trading script which computes all sorts of stuff every x minutes in my main thread (Python). the order sending is performed through such thread. the reception and execution of such orders though is a different matter as I cannot allow x minutes to pass but I need them as soon as they come in. I initialized another thread to check for such data (execution) which is in a database table (POSTGRES SQL).
Problem(s): I cannot continuosly perform query every xx ms, get data from DB, compare table length, and then get the difference for a variety of reasons (not only guy to use such DB, perforamnce issues, etc). so I looked up some solutions and came up with this thread (https://dba.stackexchange.com/questions/58214/getting-last-modification-date-of-a-postgresql-database-table) where basically the gist of it was that
"There is no reliable, authorative record of the last modified time of a table".
Question: what can I do about it, that is: getting near instantenuous responses from a postgres sql table without overloading the whole thing using Python?
You can use notifications in postgresql:
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
import select
def dblisten(dsn):
connection = psycopg2.connect(dsn)
connection.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = connection.cursor()
cur.execute("LISTEN new_id;")
while True:
select.select([connection],[],[])
connection.poll()
events = []
while connection.notifies:
notify = connection.notifies.pop().payload
do_something(notify)
and install a trigger for each update:
CREATE OR REPLACE FUNCTION notify_id_trigger() RETURNS trigger AS $$
BEGIN
PERFORM pg_notify('new_id', NEW.ID);
RETURN new;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER data_modified AFTER insert or update on data_table for each row execute procedure notify_id_trigger();")
I have a big problem with a deadlock in an InnoDB table used with sqlalchemy.
sqlalchemy.exc.InternalError: (mysql.connector.errors.InternalError) 1213 (40001): Deadlock found when trying to get lock; try restarting transaction.
I have already serialized the access, but still get a deadlock error.
This code is executed on the first call in every function. Every thread and process should wait here, till it gets the lock. It's simplified, as selectors are removed.
# The work with the index -1 always exists.
f = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.offset == -1).first()
I have reduced my code to a minimal state. I am currently running only concurrent calls on the method next_slice. Session handling, rollback and deadloc handling are handled outside.
I get deadlocks even all access is serialized. I did tried to increment a retry counter in the offset == -1 entity as well.
def next_slice(self, s, processgroup_id, itemcount):
f = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.offset == -1).first()
#Take first matching object if available / Maybe some workers failed
item = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.processgroup_id != processgroup_id,
WorkerInProgress.processgroup_id != 'finished',
WorkerInProgress.processgroup_id != 'finished!locked',
WorkerInProgress.offset != -1
).order_by(WorkerInProgress.offset.asc()).limit(1).first()
# *****
# Some code is missing here. as it's not executed in my testcase
# Fetch the latest item and add a new one
item = s.query(WorkerInProgress).with_for_update().order_by(
WorkerInProgress.offset.desc()).limit(1).first()
new = WorkerInProgress()
new.offset = item.offset + item.count
new.count = itemcount
new.maxtries = 3
new.processgroup_id = processgroup_id
s.add(new)
s.commit()
return new.offset, new.count
I don't understand why the deadlocks are occurring.
I have reduced deadlock by fetching all items in one query, but still get deadlocks. Perhaps someone can help me.
Finally I solved my problem. It's all in the documentation, but I have to understand it first.
Always be prepared to re-issue a transaction if it fails due to
deadlock. Deadlocks are not dangerous. Just try again.
Source: http://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks-handling.html
I have solved my problem by changing the architecture of this part. I still get a lot of deadlocks, but they appear almost in the short running methods.
I have splitted my worker table to a locking and an non locking part. The actions on the locking part are now very short and no data is handling during the get_slice, finish_slice and fail_slice operations.
The transaction part with data handling are now in a non locking part and without concurrent access to table rows. The results are stored in finish_slice and fail_slice to the locking table.
Finally I have found a good description on stackoverflow too. After identifying the right search terms.
https://stackoverflow.com/a/2596101/5532934
What is the best way to deal with the
1205 "deadlock victim"
error when calling SQL Server from Python?
The issue arises when I have multiple Python scripts running, and all are attempting to update a table with a MERGE statement which adds a row if it doesn't yet exist (this query will be called millions of times in each script).
MERGE table_name as table // including UPDLOCK or ROWLOCK eventually
// results in deadlock
USING ( VALUES ( ... ) )
AS row( ... )
ON table.feature = row.feature
WHEN NOT MATCHED THEN
INSERT (...)
VALUES (...)
The scripts require immediate access to the table to access the unique id assigned to the row.
Eventually, one of the scripts raises an OperationalError:
Transaction (Process ID 52) was deadlocked on lock resources with
another process and has been chosen as the deadlock victim. Rerun the
transaction.
1) I have tried using a try-except block around the call in Python:
while True:
try:
cur.execute(stmt)
break
except OperationalError:
continue
This approach slows the process down considerably. Also, I think I might be doing this incorrectly (I think I might need to reset the connection...).
2) Use a try-catch in SQL Server (something like below...):
WHILE 1 = 1
BEGIN
BEGIN TRY
MERGE statement // see above
BREAK
END TRY
BEGIN CATCH
SELECT ERROR_NUMBER() AS ErrorNumber
ROLLBACK
CONTINUE
END CATCH;
END
3) Something else?
Thanks for your help. And let me know if you need additional details, etc.
I am using Python 2.7, SQL Server 2008, and pymssql to make the connection.