#receiver(post_save, sender=MyRequestLog)
def steptwo_launcher(sender, instance, **kwargs):
GeneralLogging(calledBy='MyRequestLog', logmsg='enter steptwo_launcher').save() # remember to remove this line
if instance.stepCode == 100:
GeneralLogging(calledBy='MyRequestLog', logmsg='step code 100 found. launch next step').save()
nextStep.delay(instance.requestId,False)
I think I just witness my code losing a race condition. The backend of my application updates the status of task one, and it writes a stepCode of 100 to the log when the next task should be started. The front end of the application polls to report the current step back to the end user.
It appears that after the backend created an entry with stepCode 100, the front request came in so soon after, that the if instance.stepCode == 100: was never found to be True. The GeneralLogging only reports one entry at the time of the suspected collision and does not launch the nextstep.
My question is to 1) Confirm this is possible, which I already suspect. and 2) A way to fix this so nextStep is not skipped due to the race condition.
This question lacks a bunch of potentially useful information (e.g. missing code, missing output), but any code of the form
if state == x:
change_state
has a potential race condition when multiple control paths hit this code.
Two of the most common ways to handle this problem are (1) locks:
with some_lock:
if state:
change_state
i.e. stop everyone else from hitting this code until we're done, and (2) queues:
queue.push(item_to_be_processed)
# somewhere else
item_to_be_processed = queue.pop()
A queue/lock implementation in a db could use select_for_update and use an extra processed field, i.e. let the "writer" process save the model with processed = False and have the "reader" process do:
from django.db import transaction
...
with transaction.atomic():
items = MyRequestLog.objects.select_for_update(skip_locked=True).filter(
stepCode=100,
processed=False
)
for item in items:
do_something_useful(item) # you might want to pull this outside of the atomic block if your application allows (so you don't keep rows locked for an extended period)
item.processed = True
item.save()
ps: check your database for support (https://docs.djangoproject.com/en/2.0/ref/models/querysets/#select-for-update)
Related
I am hitting a roadblock in my thespian implementation. I have an infinite while loop that will break if a condition is met, but I want to stay in the while loop until the condition is met. I understand that it is blocking and I am fine with not being able to receive any messages while in the infinite loop, however, it seems like other actors are not entering the loop until the previous actor has exited their infinite loop. This makes me believe that all actors are on a single thread. Any idea on workarounds?
class Actor(ActorTypeDispatcher):
def logic(self):
should_break = False
while True:
print(self.id)
if should_break:
break
**insert logic to determine if should_break is True or False**
I have instantiated 100 actors (the id is incremented from 1-100). Actors 1-5 immediately enter the break statement after an iteration, however when I print the id it is stuck on 6. The logic to break the loop for Actor where id = 6 is dependent on other Actors actions simultaneously.
Per Thespian's docs:
By default, the "simpleSystemBase" is used, which runs all Actors synchronously in the current process. While this base is very useful for debugging (see the Debugging section), it does not provide the level of concurrency usually desired at production time.
Which suggests that by default Thespian effectively runs all actors in the same thread.
Alternative bases (e.g. the multiprocTCPBase, which the docs elsewhere cite as the base to choose unless you're sure you want multiprocUDPBase or multiprocQueueBase) can be passed to the ActorSystem at startup. The default simpleSystemBase has no parallelism.
I have a big problem with a deadlock in an InnoDB table used with sqlalchemy.
sqlalchemy.exc.InternalError: (mysql.connector.errors.InternalError) 1213 (40001): Deadlock found when trying to get lock; try restarting transaction.
I have already serialized the access, but still get a deadlock error.
This code is executed on the first call in every function. Every thread and process should wait here, till it gets the lock. It's simplified, as selectors are removed.
# The work with the index -1 always exists.
f = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.offset == -1).first()
I have reduced my code to a minimal state. I am currently running only concurrent calls on the method next_slice. Session handling, rollback and deadloc handling are handled outside.
I get deadlocks even all access is serialized. I did tried to increment a retry counter in the offset == -1 entity as well.
def next_slice(self, s, processgroup_id, itemcount):
f = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.offset == -1).first()
#Take first matching object if available / Maybe some workers failed
item = s.query(WorkerInProgress).with_for_update().filter(
WorkerInProgress.processgroup_id != processgroup_id,
WorkerInProgress.processgroup_id != 'finished',
WorkerInProgress.processgroup_id != 'finished!locked',
WorkerInProgress.offset != -1
).order_by(WorkerInProgress.offset.asc()).limit(1).first()
# *****
# Some code is missing here. as it's not executed in my testcase
# Fetch the latest item and add a new one
item = s.query(WorkerInProgress).with_for_update().order_by(
WorkerInProgress.offset.desc()).limit(1).first()
new = WorkerInProgress()
new.offset = item.offset + item.count
new.count = itemcount
new.maxtries = 3
new.processgroup_id = processgroup_id
s.add(new)
s.commit()
return new.offset, new.count
I don't understand why the deadlocks are occurring.
I have reduced deadlock by fetching all items in one query, but still get deadlocks. Perhaps someone can help me.
Finally I solved my problem. It's all in the documentation, but I have to understand it first.
Always be prepared to re-issue a transaction if it fails due to
deadlock. Deadlocks are not dangerous. Just try again.
Source: http://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks-handling.html
I have solved my problem by changing the architecture of this part. I still get a lot of deadlocks, but they appear almost in the short running methods.
I have splitted my worker table to a locking and an non locking part. The actions on the locking part are now very short and no data is handling during the get_slice, finish_slice and fail_slice operations.
The transaction part with data handling are now in a non locking part and without concurrent access to table rows. The results are stored in finish_slice and fail_slice to the locking table.
Finally I have found a good description on stackoverflow too. After identifying the right search terms.
https://stackoverflow.com/a/2596101/5532934
I'm trying to use Django 1.6 transactions to avoid race conditions on a game I'm developing. The game server has one simple goal: to pair two players.
My current approach is:
User wants to play
The server checks if there is anyone else waiting to play.
If there is not, it creates a GameConnection object (that has a unique identifier - uuid4).
If there is, it gets the GameConnection identifier and deletes the GameConnection.
This is the code:
# data['nickname'] = user's choice
games = GameConnection.objects.all()
if not games:
game = GameConnection.objects.create(connection=unicode(uuid.uuid4()))
game.nick1 = data["nickname"]
game.save()
response = HttpResponse(json.dumps({'connectionId': game.connection, 'whoAmI': 1, 'nick1': game.nick1, 'nick2': ""}))
else:
game = games[0]
conn = game.connection
nick1 = game.nick1
nick2 = data["nickname"]
game.delete()
response = HttpResponse(json.dumps({'connectionId': conn, 'whoAmI': 2, 'nick1': nick1, 'nick2': nick2}))
return response
Obviously there is a race condition on the code above. As this code is not atomic, it can happen that:
A checks for game connections. Finds none.
A creates a game connection.
B checks for game connections. Finds one (A).
C checks for game connections. Finds one (A).
B gets A's connection identifier and starts a game.
C gets A's connection identifier and starts a game.
I tried do but this whole block under with transaction.atomic():, or to use the #transaction.atomic decorator. But still, I am able to reproduce the race condition.
I am sure there is something about the transaction dynamics I am missing here. Can anyone shed a light?
#Sai is on track... the key is that the lock/mutex won't occur until a write (or delete). As coded, there will always be a time between "discovery" (read) of the pending connection and "claim" (write/lock) of the pending connection, with no way to know that a connection is in the process of being claimed.
If you are using PostgreSQL (pretty sure MySQL supports it, too), you can force the lock with "select for update", which will prevent another request from getting the same row until the transaction completes:
game = GameConnection.objects.all()[:1].select_for_update()
if game:
#do something, update, delete, etc.
else:
#create
Final note - consider something other than all() to be explicit about which game might be picked up (e.g., order by a "created" timestamp or something). Hope that helps.
I have this django views.py method that aims to insert many data into the db. It loops through arrays of models and, if an object isn't already on the db, it gets inserted.
This is what the code looks like:
def update_my_db(request):
a_models = A_Model.objects.filter(my_flag=True)
for a_model in a_models:
b_model_array = []
[...] # this is where b_model_array gets filled
for index in range(len(b_model_array)):
current_b_model = b_model_array[index]
try:
b_model = B_Model.objects.get(my_field=current_b_model.my_field)
except (KeyError, B_Model.DoesNotExist):
b_model = B_Model.objects.create(field_1=current_b_model.field_1, field_2=current_b_model.field_2)
b_model.save()
return HttpResponse(response)
I have noticed after several tests that the db is only updated by the end of the last iteration, as if django awaits to do a batch insert to mysql.
The thing is: there is a possibility of any of the iterations raising an exception, making all the data gathered so far be discarded because of the error (already tested and confirmed it). When it comes to adding 400 new lines, raising an exception at loop #399 and discarding all the previous 398 lines would be extremely undesirable for me.
I understand that batching would be the best choice concerning performance, but this is a background routine, so I'm not worried about it.
Bottomline: is there a way to actually force django to update the database on every iteration?
If you're on Django 1.6, check this out: https://docs.djangoproject.com/en/dev/topics/db/transactions/
You're interested in the context manager part of that page:
from django.db import transaction
def viewfunc(request):
# This code executes in autocommit mode (Django's default).
do_stuff()
with transaction.atomic():
# This code executes inside a transaction.
do_more_stuff()
So I'm creating a django app that allows a user to add a new line of text to an existing group of text lines. However I don't want multiple users adding lines to the same group of text lines concurrently. So I created a BoolField isBeingEdited that is set to True once a user decides to append a specific group. Once the Bool is True no one else can append the group until the edit is submitted, whereupon the Bool is set False again. Works alright, unless someone decides to make an edit then changes their mind or forgets about it, etc. I want isBeingEdited to flip back to False after 10 minutes or so. Is this a job for cron, or is there something easier out there? Any suggestions?
Change the boolean to a "lock time"
To lock the model, set the Lock time to the current time.
To unlock the model, set the lock time to None
Add an "is_locked" method. That method returns "not locked" if the current time is more than 10 minutes after the lock time.
This gives you your timeout without Cron and without regular hits into the DB to check flags and unset them. Instead, the time is only checked if you are interest in wieither this model is locked. A Cron would likely have to check all models.
from django.db import models
from datetime import datetime, timedelta
# Create your models here.
class yourTextLineGroup(models.Model):
# fields go here
lock_time = models.DateTimeField(null=True)
locked_by = models.ForeignKey()#Point me to your user model
def lock(self):
if self.is_locked(): #and code here to see if current user is not locked_by user
#exception / bad return value here
pass
self.lock_time = datetime.now()
def unlock(self):
self.lock_time = None
def is_locked(self):
return self.lock_time and datetime.now() - self.lock_time < timedelta(minutes=10)
Code above assumes that the caller will call the save method after calling lock or unlock.