I'm trying to use Django 1.6 transactions to avoid race conditions on a game I'm developing. The game server has one simple goal: to pair two players.
My current approach is:
User wants to play
The server checks if there is anyone else waiting to play.
If there is not, it creates a GameConnection object (that has a unique identifier - uuid4).
If there is, it gets the GameConnection identifier and deletes the GameConnection.
This is the code:
# data['nickname'] = user's choice
games = GameConnection.objects.all()
if not games:
game = GameConnection.objects.create(connection=unicode(uuid.uuid4()))
game.nick1 = data["nickname"]
game.save()
response = HttpResponse(json.dumps({'connectionId': game.connection, 'whoAmI': 1, 'nick1': game.nick1, 'nick2': ""}))
else:
game = games[0]
conn = game.connection
nick1 = game.nick1
nick2 = data["nickname"]
game.delete()
response = HttpResponse(json.dumps({'connectionId': conn, 'whoAmI': 2, 'nick1': nick1, 'nick2': nick2}))
return response
Obviously there is a race condition on the code above. As this code is not atomic, it can happen that:
A checks for game connections. Finds none.
A creates a game connection.
B checks for game connections. Finds one (A).
C checks for game connections. Finds one (A).
B gets A's connection identifier and starts a game.
C gets A's connection identifier and starts a game.
I tried do but this whole block under with transaction.atomic():, or to use the #transaction.atomic decorator. But still, I am able to reproduce the race condition.
I am sure there is something about the transaction dynamics I am missing here. Can anyone shed a light?
#Sai is on track... the key is that the lock/mutex won't occur until a write (or delete). As coded, there will always be a time between "discovery" (read) of the pending connection and "claim" (write/lock) of the pending connection, with no way to know that a connection is in the process of being claimed.
If you are using PostgreSQL (pretty sure MySQL supports it, too), you can force the lock with "select for update", which will prevent another request from getting the same row until the transaction completes:
game = GameConnection.objects.all()[:1].select_for_update()
if game:
#do something, update, delete, etc.
else:
#create
Final note - consider something other than all() to be explicit about which game might be picked up (e.g., order by a "created" timestamp or something). Hope that helps.
Related
I am hitting a roadblock in my thespian implementation. I have an infinite while loop that will break if a condition is met, but I want to stay in the while loop until the condition is met. I understand that it is blocking and I am fine with not being able to receive any messages while in the infinite loop, however, it seems like other actors are not entering the loop until the previous actor has exited their infinite loop. This makes me believe that all actors are on a single thread. Any idea on workarounds?
class Actor(ActorTypeDispatcher):
def logic(self):
should_break = False
while True:
print(self.id)
if should_break:
break
**insert logic to determine if should_break is True or False**
I have instantiated 100 actors (the id is incremented from 1-100). Actors 1-5 immediately enter the break statement after an iteration, however when I print the id it is stuck on 6. The logic to break the loop for Actor where id = 6 is dependent on other Actors actions simultaneously.
Per Thespian's docs:
By default, the "simpleSystemBase" is used, which runs all Actors synchronously in the current process. While this base is very useful for debugging (see the Debugging section), it does not provide the level of concurrency usually desired at production time.
Which suggests that by default Thespian effectively runs all actors in the same thread.
Alternative bases (e.g. the multiprocTCPBase, which the docs elsewhere cite as the base to choose unless you're sure you want multiprocUDPBase or multiprocQueueBase) can be passed to the ActorSystem at startup. The default simpleSystemBase has no parallelism.
I want to create a Database class which can create cursors on demand.
It must be possible to use the cursors in parallel (two or more cursor can coexist) and, since we can only have one cursor per connection, the Database class must handle multiple connections.
For performance reasons we want to reuse connections as much as possible and avoid creating a new connection every time a cursor is created:
whenever a request is made the class will try to find, among the opened connections, the first non-busy connection and use it.
A connection is still busy as long as the cursor has not been consumed.
Here is an example of such class:
class Database:
...
def get_cursos(self,query):
selected_connection = None
# Find usable connection
for con in self.connections:
if con.is_busy() == False: # <--- This is not PEP 249
selected_connection = con
break
# If all connections are busy, create a new one
if (selected_connection is None):
selected_connection = self._new_connection()
self.connections.append(selected_connection)
# Return cursor on query
cur = selected_connection.cursor()
cur.execute(query)
return cur
However looking at the PEP 249 standard I cannot find any way to check whether a connection is actually being used or not.
Some implementations such as MySQL Connector offer ways to check whether a connection has still unread content (see here), however as far as I know those are not part of PEP 249.
Is there a way I can achieve what described before for any PEP 249 compliant python database API ?
Perhaps you could use the status of the cursor to tell you if a cursor is being used. Let's say you had the following cursor:
new_cursor = new_connection.cursor()
cursor.execute(new_query)
and you wanted to see if that connection was available for another cursor to use. You might be able to do something like:
if (new_cursor.rowcount == -1):
another_new_cursor = new_connection.cursor()
...
Of course, all this really tells you is that the cursor hasn't executed anything yet since the last time it was closed. It could point to a cursor that is done (and therefore a connection that has been closed) or it could point to a cursor that has just been created or attached to a connection. Another option is to use a try/catch loop, something along the lines of:
try:
another_new_cursor = new_connection.cursor()
except ConnectionError?: //not actually sure which error would go here but you get the idea.
print("this connection is busy.")
Of course, you probably don't want to be spammed with printed messages but you can do whatever you want in that except block, sleep for 5 seconds, wait for some other variable to be passed, wait for user input, etc. If you are restricted to PEP 249, you are going to have to do a lot of things from scratch. Is there a reason you can't use external libraries?
EDIT: If you are willing to move outside of PEP 249, here is something that might work, but it may not be suitable for your purposes. If you make use of the mysql python library, you can take advantage of the is_connected method.
new_connection = mysql.connector.connect(host='myhost',
database='myDB',
user='me',
password='myPassword')
...stuff happens...
if (new_connection.is_connected()):
pass
else:
another_new_cursor = new_connection.cursor()
...
#receiver(post_save, sender=MyRequestLog)
def steptwo_launcher(sender, instance, **kwargs):
GeneralLogging(calledBy='MyRequestLog', logmsg='enter steptwo_launcher').save() # remember to remove this line
if instance.stepCode == 100:
GeneralLogging(calledBy='MyRequestLog', logmsg='step code 100 found. launch next step').save()
nextStep.delay(instance.requestId,False)
I think I just witness my code losing a race condition. The backend of my application updates the status of task one, and it writes a stepCode of 100 to the log when the next task should be started. The front end of the application polls to report the current step back to the end user.
It appears that after the backend created an entry with stepCode 100, the front request came in so soon after, that the if instance.stepCode == 100: was never found to be True. The GeneralLogging only reports one entry at the time of the suspected collision and does not launch the nextstep.
My question is to 1) Confirm this is possible, which I already suspect. and 2) A way to fix this so nextStep is not skipped due to the race condition.
This question lacks a bunch of potentially useful information (e.g. missing code, missing output), but any code of the form
if state == x:
change_state
has a potential race condition when multiple control paths hit this code.
Two of the most common ways to handle this problem are (1) locks:
with some_lock:
if state:
change_state
i.e. stop everyone else from hitting this code until we're done, and (2) queues:
queue.push(item_to_be_processed)
# somewhere else
item_to_be_processed = queue.pop()
A queue/lock implementation in a db could use select_for_update and use an extra processed field, i.e. let the "writer" process save the model with processed = False and have the "reader" process do:
from django.db import transaction
...
with transaction.atomic():
items = MyRequestLog.objects.select_for_update(skip_locked=True).filter(
stepCode=100,
processed=False
)
for item in items:
do_something_useful(item) # you might want to pull this outside of the atomic block if your application allows (so you don't keep rows locked for an extended period)
item.processed = True
item.save()
ps: check your database for support (https://docs.djangoproject.com/en/2.0/ref/models/querysets/#select-for-update)
This is the code I want to implement. Simply stated, it is a small chat bot that connects to a server and joins room testroom1 with credentials user and pass. Using a library, an event is called every time somebody enters a message into the chatroom, and this bot will print their message, with formatting:
import ch
class bot(ch.RoomManager):
for x in range(0, 3):
def onMessage(self, room, user, message):
print("[{0}] {1}: {2}".format(room.name, user.name.title(), message.body))
rooms = ["testroom1"]
username = "user"
password = "pass"
bot.easy_start(rooms,username,password)
However, the key problem is this:
for x in range(0, 3):
I only want the "onMessage" function to be accessed 3 times then to not print any more messages. This doesn't seem to work, and it continues to print messages until I exit it.
If possible, I would like to limit the onMessage function with while loops further in my program, and to use it in several different cases to parse messages at different times.
The source of the ch library and basis for my simple code is here
Any insight on this issue is appreciated.
I think you have misunderstood the Python syntax slightly. The for loop in your class will define the same onMessagemethod three times, with each new iteration replacing the old one. This would be apparent if you tried to use the x (or rather self.x) within your function.
What you want, is a counter within the method or class, that is incremented with each call and then, when the max count is reached you could just have the onMessagemethod do nothing.
Try the following code, which I think does what you want:
import ch
class bot(ch.RoomManager):
messages_left = 3
def onMessage(self, room, user, message):
if self.messages_left > 0:
print("[{0}] {1}: {2}".format(room.name, user.name.title(), message.body))
self.messages_left -=1
rooms = ["testroom1"]
username = "user"
password = "pass"
bot.easy_start(rooms,username,password)
What I have done here, is to add an initializing function that defines an object variable messages_left that is decremented in the onMessagecall, until it hits zero. Further, I have added a class parameter, which allows you to control how many times you want onMessage to format the message.
I'm trying to connect to a TeamSpeak server using the QueryServer to make a bot. I've taken advice from this thread, however I still need help.
This is The TeamSpeak API that I'm using.
Before the edits, this was the summary of what actually happened in my script (1 connection):
It connects.
It checks for channel ID (and it's own client ID)
It joins the channel and starts reading everything
If someone says an specific command, it executes the command and then it disconnects.
How can I make it so it doesn't disconnect? How can I make the script stay in a "waiting" state so it can keep reading after the command is executed?
I am using Python 3.4.1.
I tried learning Threading but either I'm dumb or it doesn't work the way I thought it would. There's another "bug", once waiting for events, if I don't trigger anything with a command, it disconnects after 60 seconds.
#Librerias
import ts3
import threading
import datetime
from random import choice, sample
# Data needed #
USER = "thisisafakename"
PASS = "something"
HOST = "111.111.111.111"
PORT = 10011
SID = 1
class BotPrincipal:
def __init__(self, manejador=False):
self.ts3conn = ts3.query.TS3Connection(HOST, PORT)
self.ts3conn.login(client_login_name=USER, client_login_password=PASS)
self.ts3conn.use(sid=SID)
channelToJoin = Bot.GettingChannelID("TestingBot")
try: #Login with a client that is ok
self.ts3conn.clientupdate(client_nickname="The Reader Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
self.suscribirEvento("textchannel", ChannelToJoin)
self.ts3conn.on_event = self.manejadorDeEventos
self.ts3conn.recv_in_thread()
except ts3.query.TS3QueryError: #Name already exists, 2nd client connect with this info
self.ts3conn.clientupdate(client_nickname="The Writer Bot")
self.MyData = self.GettingMyData()
self.MoveUserToChannel(ChannelToJoin, Bot.MyData["client_id"])
def __del__(self):
self.ts3conn.close()
def GettingMyData(self):
respuesta = self.ts3conn.whoami()
return respuesta.parsed[0]
def GettingChannelID(self, nombre):
respuesta = self.ts3conn.channelfind(pattern=ts3.escape.TS3Escape.unescape(nombre))
return respuesta.parsed[0]["cid"]
def MoveUserToChannel(self, idCanal, idUsuario, passCanal=None):
self.ts3conn.clientmove(cid=idCanal, clid=idUsuario, cpw=passCanal)
def suscribirEvento(self, tipoEvento, idCanal):
self.ts3conn.servernotifyregister(event=tipoEvento, id_=idCanal)
def SendTextToChannel(self, idCanal, mensajito="Error"):
self.ts3conn.sendtextmessage(targetmode=2, target=idCanal, msg=mensajito) #This works
print("test") #PROBLEM HERE This doesn't work. Why? the line above did work
def manejadorDeEventos(sender, event):
message = event.parsed[0]['msg']
if "test" in message: #This works
Bot.SendTextToChannel(ChannelToJoin, "This is a test") #This works
if __name__ == "__main__":
Bot = BotPrincipal()
threadprincipal = threading.Thread(target=Bot.__init__)
threadprincipal.start()
Prior to using 2 bots, I tested to launch the SendTextToChannel when it connects and it works perfectly, allowing me to do anything that I want after it sends the text to the channel. The bug that made entire python code stop only happens if it's triggered by the manejadorDeEventos
Edit 1 - Experimenting with threading.
I messed it up big time with threading, getting to the result where 2 clients connect at same time. Somehow i think 1 of them is reading the events and the other one is answering. The script doesn't close itself anymore and that's a win, but having a clone connection doesn't looks good.
Edit 2 - Updated code and actual state of the problem.
I managed to make the double connection works more or less "fine", but it disconnects if nothing happens in the room for 60 seconds. Tried using Threading.timer but I'm unable to make it works. The entire question code has been updated for it.
I would like an answer that helps me to do both reading from the channel and answering to it without the need of connect a second bot for it (like it's actually doing...) And I would give extra points if the answer also helps me to understand an easy way to make a query to the server each 50 seconds so it doesn't disconnects.
From looking at the source, recv_in_thread doesn't create a thread that loops around receiving messages until quit time, it creates a thread that receives a single message and then exits:
def recv_in_thread(self):
"""
Calls :meth:`recv` in a thread. This is useful,
if you used ``servernotifyregister`` and you expect to receive events.
"""
thread = threading.Thread(target=self.recv, args=(True,))
thread.start()
return None
That implies that you have to repeatedly call recv_in_thread, not just call it once.
I'm not sure exactly where to do so from reading the docs, but presumably it's at the end of whatever callback gets triggered by a received event; I think that's your manejadorDeEventos method? (Or maybe it's something related to the servernotifyregister method? I'm not sure what servernotifyregister is for and what on_event is for…)
That manejadorDeEventos brings up two side points:
You've declared manejadorDeEventos wrong. Every method has to take self as its first parameter. When you pass a bound method, like self.manejadorDeEventos, that bound self object is going to be passed as the first argument, before any arguments that the caller passes. (There are exceptions to this for classmethods and staticmethods, but those don't apply here.) Also, within that method, you should almost certainly be accessing self, not a global variable Bot that happens to be the same object as self.
If manejadorDeEventos is actually the callback for recv_in_thread, you've got a race condition here: if the first message comes in before your main threads finishes the on_event assignment, the recv_on_thread won't be able to call your event handler. (This is exactly the kind of bug that often shows up one time in a million, making it a huge pain to debug when you discover it months after deploying or publishing your code.) So, reverse those two lines.
One last thing: a brief glimpse at this library's code is a bit worrisome. It doesn't look like it's written by someone who really knows what they're doing. The method I copied above only has 3 lines of code, but it includes a useless return None and a leaked Thread that can never be joined, not to mention that the whole design of making you call this method (and spawn a new thread) after each event received is weird, and even more so given that it's not really explained. If this is the standard client library for a service you have to use, then you really don't have much choice in the matter, but if it's not, I'd consider looking for a different library.