I have python server that waits for a global flag to be set and exits.
In a few threads, I have code that waits using zmq.Poller for
a message. It times out, prints a heartbeat message, then waits on poller
for a new message:
def timed_recv(zock, msec=5000.0):
poller = zmq.Poller()
poller.register(zock, zmq.POLLIN)
events = dict(poller.poll(msec))
data = None
if events and events.get(zock) == zmq.POLLIN:
# if a message came in time, read it.
data = zock.recv()
return data
So in the above function, I wait for 5 seconds for a message to arrive. If none does, the function returns, the calling loop prints a message and waits for a new message:
while not do_exit():
timed_recv(zock)
print "Program still here!"
sys.exit()
do_exit() checks a global flag for exitting.
Now, if the flag is set, there can be a 5 second delay between it being set, and the loop exitting. How, can I poll for both zock input, and for the global flag being set so that the loop exits quickly?
I thought I can add to the poller, a file descriptor that closes upon global flag being set. Does that seem reasonable? It seems kind of hackish.
Is there a better way to wait for global flag and POLLIN on zock?
(We are using zmq version 3.0 on debian.)
thanks.
The easiest way is to drop the use of a flag, and use another 0mq socket to convey a message. The poller can then wait on both 0mq sockets. The message could be just a single byte; it's arrival in the poller is the message, not its content.
In doing that you're heading down the road to Actor Model programming.
It's a whole lot easier if a development sticks to one programming model; mixing stuff up (e.g. 0mq and POSIX condition variables) is inviting a lot of problems.
Related
How can I make a coroutine stop with timeout?
I don't understand why asyncio.wait_for() doesn't work for me.
I have such piece of code (planning to make my implementation of telnet client):
def expect(self, pattern, timeout=20):
if type(pattern) == str:
pattern = pattern.encode('ascii', 'ignore')
return self.loop.run_until_complete(asyncio.wait_for(self.asyncxpect(pattern), timeout))
async def asyncxpect(self, pattern): #receives data in a cumulative way until match is found
regexp = re.compile(b'(?P<payload>[\s\S]*)(?P<pattern>%s)' %pattern)
self.buffer = b''
while True:
# add timeout
# add exception handling for unexpectedly closed connections
data = await self.loop.sock_recv(self.sock, 10000)
self.buffer += data
m = re.match(regexp, self.buffer)
if m:
payload = m.group('payload')
match = m.group('pattern')
return payload, match
As I thought this code, at some point (in await statement) returns control to event loop. I thought it should happen when there is no more data to receive.
And if event loop has control, it can stop with timeout.
But if server doesn't send anything useful (that matched) my code just stumbles in this loop, right at await point.
I think it is different from this problem Python asyncio force timeout, because I'm not using blocking statements like time.sleep(n).
Here is my code
When the server closes the connection, sock_recv returns an empty bytearray (b''), indicating end of file. Since you don't handle that condition, your code ends up stuck in an infinite loop processing the same buffer.
To correct it, add something like:
if data == b'':
break
...after the data = await loop.sock_recv(...) line.
But the above still doesn't explain why wait_for is unable to cancel the rogue coroutine. The problem is that await doesn't mean "pass control to the event loop", as it is sometimes understood. It means "request value from the provided awaitable object, yielding control to the event loop if (and as long as) the object indicates that it does not have a value ready." The if is crucial: if the object does have a value ready when first asked, this value will be used immediately without ever deferring to the event loop. In other words, await doesn't guarantee that the event loop will get a chance to run.
For example, the following coroutine completely blocks the event loop and prevents any other coroutine from ever running, despite its inner loop consisting of nothing but awaiting:
async def busy_loop():
while True:
await noop()
async def noop():
pass
In your example, since the socket does not block at all when it is at end-of-file, the coroutine is never suspended, and (in collusion with the above bug) your coroutine never exits.
To ensure that other tasks get a chance to run, you can add await asyncio.sleep(0) in a loop. This should not be necessary for most code, where requesting IO data will soon result in a wait, at which point the event loop will kick in. (In fact, needing to do so often indicates a design flaw.) In this case it is only in combination with the EOF-handling bug that the code gets stuck.
I am using ZeroMQ and a publish/subscribe pattern, in Python.
The server sends a message, as follows:
ZMQsocket_Publisher.send(subscription_string)
which is received by a client, that, as consequence, starts a loop, like this:
loop_condition = True
while loop_condition:
ZMQsocket_Pusher.send(strings, zmq.SNDMORE)
ZMQsocket_Pusher.send(data)
during which by default infinitely responds to the server in each iteration by sending back data to the server.
The question is:
I would like to stop the while when a particular event occurs by either changing the condition or sending a break/interrupt signal. The problem I am facing right now is that if the while is still in progress, that particular client is not able to receive a "stop" message sent in a second moment by the server.
Is there an easy way to stop the while or the execution when using this pattern?
Thank you in advance!
As I understood, you have something like this:
subscription_string = your_client_receiver_socket.recv()
strings, data = some_processing(subscription_string)
loop_condition = True
while loop_condition:
ZMQsocket_Pusher.send(strings, zmq.SNDMORE)
ZMQsocket_Pusher.send(data)
If that's the case, simply add check of stop signal in while loop:
while loop_condition:
ZMQsocket_Pusher.send(strings, zmq.SNDMORE)
ZMQsocket_Pusher.send(data)
try:
stop = your_client_receiver_socket.recv(flags=zmq.DONTWAIT)
if check_on_stop(stop):
break
except zmq.error.Again:
pass
zmq.DONTWAIT flag will say receiver to receive in non-blocking way, raising zmq.error.Again exception if it couldn't receive anything. With this you can send some stop signal from server to stop client's loop.
OK, so i have a pretty simple turn based application (game).
Each user sends a request on the server and then wait for response. The important is, that only ONE user makes the request on the server (send his actions) and all other users are just waiting, if the server sends them some data, so they must always check (in loop) if something is coming from server.
I´m using the built-in module of python "socket" and the way i manage the clients is this: For every user i create one thread, in which runs infinite loop, until the application ends, which checks for request (if it is the turn of the user) or checks if it got anything to sent to the other users. Now let´s move to Clients. Every client has again one thread with infinite loop, waiting for data from server.
The problem is that the GUI is made in PyQt4.4, where i cant get into the loop of the PyQt itself (although i have seen, that it is possible to do this with twisted, but then i would have to rewrite my code) so i have to use the thread, that means i can use classic python threading library or QThread, but QThread sadly doesn´t have any Events, which are pretty crucial because i want always wait after the message from the server for the response of the program, so i can send response to the server again. On the other hand, I am not sure, if i can use Thread from threading to emit signals. So which one is the right one to go?
Btw: is actually ok, to run the infinite client and server side loop? Because in every tutorial I have seen, the client close the connection right after he gets his answer.
EDIT:
here´s some of the code
Server side loop for connection with client:
while self.running:
if self.is_on_turn == p and self.reply is not None:
cmd = conn.recv(1024)
if cmd == '':
conn.close()
return
cmd = eval(cmd)
if self.is_on_turn != p: # User is not on turn
print "END"
conn.sendall("END")
else:
self.queue.put(cmd)
ev.wait() # Here works another program with the message and decide what to do next
ev.clear() #
conn.sendall(str(self.message))
if self.reply:
ev.wait() #
ev.clear() #
if self.reply:
r = conn.recv(1024)
if r == '':
conn.close()
return
self.queue.put(eval(r))
ev.wait() #
ev.clear() #
conn.sendall(str(self.message))
conn.close()
Client side loop:
def main_loop(self, pipe, conn, e, o): #e is event, o is bool (whether the client has to answer back to the server)
is_on_turn = conn.recv(4096)
pipe.send((is_on_turn))
while True:
if is_on_turn == h or o.value and o.value is not None:
conn.send(str(pipe.recv()))
pipe.send(eval(conn.recv(4096)))
e.wait()
e.clear()
The pipe is there, because I made it in multiprocessing at first, there should the emit signal for the PyQt instead, but as I said, I am not sure which approach to use
So the result is, that I have just used QTcpServer and QTcpSocket as sugessted by ekhumoro, which resulted in much cleaner code and easier management :)
I'm making a chat application in twisted. Suppose my server is designed in such a way that whenever it detects a client online, it sends the client all the pending-messages (those messages of that client which were cached in a python-list on the server because it was offline) one-by-one in a while loop until the list is exhausted. Something like this:
class MyChat(LineReceiver):
def connectionMade(self):
self.factory.clients.append(self)
while True:
#retrieve first message from a list of pending-messages(queue) of "self"
msg = self.retrieveFromQueue(self)
if msg != "empty":
self.transport.write(msg)
else:
break
def lineReceived(self, line):
...
def connectionLost(self, reason):
...
def retrieveFromQueue(self, who):
msglist = []
if who in self.factory.userMessages:
msglist = self.factory.userMessages[who]
if msglist != []:
msg = msglist.pop(0) #msglist is a list of strings
self.factory.userMessages[self] = msglist
return msg
else:
return "empty"
factory.userMessages = {} #dict of list of incoming messages of users who aren't online
So according to my understanding of Twisted, the while loop will block the main reactor thread and any interaction from any other client with the server will not be registered by the server. If that's the case, I want an alternate code/method to this approach which will not block the twisted thread.
Update: There may be 2000-3000 pending messages per user because of the nature of the app.
I think that https://glyph.twistedmatrix.com/2011/11/blocking-vs-running.html addresses this point.
The answer here depends on what exactly self.retrieveFromQueue(self) does. You implied it's something like:
if self.list_of_messages:
return self.list_of_messages.pop(0)
return b"empty"
If this is the case, then the answer is one thing. On the other hand, if the implementation is something more like:
return self.remote_mq_client.retrieve_queue_item(self.queue_identifier)
then the answer might be something else entirely. However, note that it's the implementation of retrieveFromQueue upon which the answer appears to hinge.
That there is a while loop isn't quite as important. The while loop reflects the fact that (to use Glyph's words), this code is getting work done.
You may decide that the amount of work this loop represents is too great to all get done at one time. If there are hundreds of millions of queued messages then copying them one by one into the connection's send buffer will probably use both a noticable amount of time and memory. In this case, you may wish to consider the producer/consumer pattern and its support in Twisted. This won't make the code any less (or more) "blocking" but it will make it run for shorter periods of time at a time.
So the questions to answer here are really:
whether or not retrieveFromQueue blocks
if it does not block, whether or not there will be so many queued messages that processing them all will cause connectionMade to run for so long that other clients notice a disruption in service
Following this example in the ØMQ docs, I'm trying to create a simple receiver. The example uses infinite loop. Everything works just fine. However, on MS Windows, when I hit CTRL+C to raise KeyboardInterrupt, the loop does not break. It seems that recv() method somehow ignores the exception. However, I'd love to exit the process by hiting CTRL+C instead of killing it. Is that possible?
In response to the #Cyclone's request, I suggest the following as a possible solution:
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL);
# any pyzmq-related code, such as `reply = socket.recv()`
A zmq.Poller object seems to help:
def poll_socket(socket, timetick = 100):
poller = zmq.Poller()
poller.register(socket, zmq.POLLIN)
# wait up to 100msec
try:
while True:
obj = dict(poller.poll(timetick))
if socket in obj and obj[socket] == zmq.POLLIN:
yield socket.recv()
except KeyboardInterrupt:
pass
# Escape while loop if there's a keyboard interrupt.
Then you can do things like:
for message in poll_socket(socket):
handle_message(message)
and the for-loop will automatically terminate on Ctrl-C. It looks like the translation from Ctrl-C to a Python KeyboardInterrupt only happens when the interpreter is active and Python has not yielded control to low-level C code; the pyzmq recv() call apparently blocks while in low-level C code, so Python never gets a chance to issue the KeyboardInterrupt. But if you use zmq.Poller then it will stop at a timeout and give the interpreter a chance to issue the KeyboardInterrupt after the timeout is complete.
Don't know if this going to work in Windows, but in Linux I did something like this:
if signal.signal(signal.SIGINT, signal.SIG_DFL):
sys.exit()
Try ctrl+break (as in the key above Page Up, I had to look it up, I don't think I've ever touched that key before)
suggested near the bottom of this thread. I haven't done anything too fancy, but this seems to work well enough in the cases I've tried.