I have written the following piece of code to handle a lost connection in twisted :
class Foo(LineReceiver):
def connectionLost(self, reason):
if reason.type != ConnectionLost:
reactor.stop()
def terminate(self):
self.transport.loseConnection()
The terminate method is called by some input/output protocol.
I had to test reason.type in the connectionLost method to avoid an error
'can't stop reactor that isn' running' when I interrupt my program with Ctrl-C instead
of calling the terminate method.
This code works but I wonder if there is a more elegant way of managing the end of a connection in twisted ?
Thanks !
The problem you're facing is that control+C is invoking an already-installed signal handler that stops the reactor. The reactor, while stopping, invokes your connectionLost method, because shutting down the reactor automatically closes all connections. In turn, your protocol tries to stop the reactor - only to find it's already been stopped!
A simple (but not entirely correct) way to avoid this condition is to look at the running attribute on the reactor before stopping it, like so:
def connectionLost(self, reason):
if reactor.running:
reactor.stop()
(Note also that you probably shouldn't check the exception type that way; you might want to use Failure.check instead, if you actually care about the type of the reason, but in most cases you shouldn't care about the type of the reason; a broken connection is just a broken connection, unless you are trying to ascertain something very specific attributes of its brokenness, like for example whether the termination should cause an SSL session termination.)
If you want to be more thorough about checking for this, you will need to monitor the reactor's state via a "before", "shutdown" system event trigger, or use the twisted.internet.task.react API introduced in Twisted 12.3.
Related
We have a complex multiservice that needs to do some fairly intricate accounting when shutting down in order to implement a "graceful" shutdown.
I'm trying to write tests for this under trial. The issue is that the reactor is effectively a process-global resource, and shutting down my service means that trial's reactor is also stopped, which (of course) makes it explode.
This is documented to be a no-no in trial, but I need some kind of workaround that allows me to write my tests. My first thought was to use a mock.Mock, but this means we're not really using a reactor that's shutting down, which isn't going to give me behavior that's faithful to the actual shutdown process.
I believe what I need is a way to separate trial's reactor from the reactor of my service-under-test. Sharing a mutable resource between the test system and the system under test is surely an anti-pattern.
There's a difference between shutting down a service and stopping a reactor. You should be able to test most of the desired behavior with myservice.stopService. To test the code that actually initiates the shutdown, just mock out reactor.stop with self.patch(reactor, 'stop', mock.Mock()), and later assert that it was called. If you want to link the two, then have your mock stop call your service's stopService.
I have made a Home Security program in Python that uses Raspberry Pi's GPIOs to sense movement and actuate the siren. The users activate/deactivate the system using a NFC Tag to a nfc reder connected also to raspberry pi.
For this I need to constantly check for nfc tags in a non blocking manner and at the same time constantly check the sensors for movement also non blocking. I need some more parallel stuff to do but I think these two are enough to make my point.
Right now I use threads which I start/stop like this - Stopping a thread after a certain amount of time -
I'm not sure if this is the optimal way but as of now the system works fine.
Now I want to extend its functionality to offer notifications through websockets. I found that this can be done with Twisted but I am confused..
Here is an example code of how I am trying to do it:
from twisted.internet import reactor
from autobahn.websocket import WebSocketServerFactory, \
WebSocketServerProtocol, \
listenWS
def thread1(stop_event):
while(not stop_event.is_set()):
stop_event.wait(4)
print "checking sensor"
# sensor_state = GPIO.input(11)
if sensor_state == 1:
# how can I call send_m("sensor detected movement") #<---
t1_stop_event.set()
t1_stop_event = Event()
t1 = Thread(target=thread1, args=(t1_stop_event,))
class EchoServerProtocol(WebSocketServerProtocol):
def onMessage(self, msg, binary):
print "received: "+msg
print "stopping thread1"
t1_stop_event.set()
def send_m(self, msg):
self.sendMessage(msg)
if __name__ == '__main__':
t1.start()
factory = WebSocketServerFactory("ws://localhost:9000")
factory.protocol = EchoServerProtocol
listenWS(factory)
reactor.run()
So how can I call the send method of the server protocol from a thread like the thread1?
As is often the case, the answer to your question about threads and Twisted is "don't use threads".
The reason you're starting a thread here appears to be so you can repeatedly check a GPIO sensor. Does checking the sensor block? I'm guessing not, since if it's a GPIO it's locally available hardware and its results will be available immediately. But I'll give you the answer both ways.
The main thing you are using threads for here is to do something repeatedly. If you want to do something repeatedly in Twisted, there is never a reason to use threads :). Twisted includes a great API for recurring tasks: LoopingCall. Your example, re-written to use LoopingCall (again, assuming that the GPIO call does not block) would look like this:
from somewhere import GPIO
from twisted.internet import reactor, task
from autobahn.websocket import WebSocketServerFactory, \
WebSocketServerProtocol, \
listenWS
class EchoServerProtocol(WebSocketServerProtocol):
def check_movement(self):
print "checking sensor"
sensor_state = GPIO.input(11)
if sensor_state == 1:
self.send_m("sensor detected movement")
def connectionMade(self):
WebSocketServerProtocol.connectionMade(self)
self.movement_checker = task.LoopingCall(self.check_movement)
self.movement_checker.start(4)
def onMessage(self, msg, binary):
self.movement_checker.stop()
def send_m(self, msg):
self.sendMessage(msg)
if __name__ == '__main__':
factory = WebSocketServerFactory("ws://localhost:9000")
factory.protocol = EchoServerProtocol
listenWS(factory)
reactor.run()
Of course, there is one case where you still need to use threads: if the GPIO checker (or whatever your recurring task is) needs to run in a thread because it is a potentially blocking operation in a library that can't be modified to make better use of Twisted, and you don't want to block the main loop.
In that case, you still want to use LoopingCall, and take advantage of another one of its features: if you return a Deferred from the function that LoopingCall is calling, then it won't call that function again until the Deferred fires. This means you can shuttle a task off to a thread and not worry about the main loop piling up queries for that thread: you can just resume the loop on the main thread automatically when the thread completes.
To give you a more concrete idea of what I mean, here's the check_movement function modified to work with a long-running blocking call that's run in a thread, instead of a quick polling call that can be run on the main loop:
def check_movement(self):
from twisted.internet.threads import deferToThread
def get_input():
# this is run in a thread
return GPIO.input(11)
def check_input(sensor_state):
# this is back on the main thread, and can safely call send_m
if sensor_state == 1:
self.send_m("sensor movement detected")
return deferToThread(get_input).addCallback(check_input)
Everything else about the above example stays exactly the same.
There are a few factors at play in your example. Short answer: study this documentation on threads in Twisted.
While you don't have to use Twisted's reactor to use protocol classes (threading and protocol implementation are decoupled), you have called reactor.run so all of the below I consider applicable to you.
Let Twisted create threads for you. Going outside the framework can get you in trouble. There are no "public" APIs for IPC messaging with the reactor (I think), so if you use Twisted, you pretty much need to go all the way.
By default, Twisted does not switch threads to call your callbacks. To delegate to a worker thread from the main reactor thread (i.e. to perform blocking I/O), you don't have to create a thread yourself, you use reactor.callInThread and it will run in a worker thread. If you never do this, everything runs in the main reactor thread, meaning for example any I/O operations will block the reactor thread and you can't receive any events until your I/O completes.
Code running in worker threads should use reactor.callFromThread to do anything that is not thread-safe. Provide a callback, which will run in the main reactor thread. You're better safe than sorry here, trust me.
All of the above applies to Deferred processing also. So don't be afraid to use partial(reactor.callFromThread, mycallback) or partial(reactor.callInThread, mycallback) instead of simply mycallback when setting up callbacks. I learned that the hard way; without that, I found that any blocking I/O that I might do in deferred callbacks was either erroring out (due to thread safety issues) or blocking the main thread.
If you're just starting out in Twisted, it's a bit of a "trust fall". Learn to let go of managing your own threads and passing messages via Queue objects, etc. Once you figure out how Deferred and the reactor work (it's called "Twisted" for a reason!), it will seem perfectly natural to you. Twisted does force you to decouple and separate concerns in a functional programming style, but once you're over that, I've found that it's very clean and works well.
One tip: I wrote some decorators to use on all my callback functions so that I didn't have to be constantly calling callInThread and callFromThread and setting up Deferred for exception handling callbacks throughout the code; my decorators enable that behavior for me. It's likely prevented bugs from forgetting to do that, and it's certainly made Twisted development more pleasant for me.
I'm running a pretty simple server in Python using Twisted. When I try to run two at the same time, this exception occurs:
twisted.internet.error.CannotListenError: Couldn't listen on 127.0.0.1:5050: [Errno 98] Address already in use.
It makes a lot of sense. How can I catch this exception?
I'd simply like to terminate the reactor and shut everything down if an existing server is running. Otherwise, I get the exception and it just hangs indefinitely until I kill the process.
You need to use an error handler callback, an errBack in Twisted lingo. You can add one to a Deferred using the addErrback method.
I have a Twisted/PyQt application that (among other things) connects to a bunch of remote resources. When the user closes the window I want to shut down all the connections, cleanly if possible, forcibly if not.
The problem is that by the time I go to close the connections, it appears that the reactor is no longer alive to let me do so.
Here's my app code:
# Create app and connect the Twisted/Qt reactors
app = QApplication(sys.argv)
qtreactor.qt4reactor.install()
# Shutdown Twisted when window is closed
#defer.inlineCallbacks
def stop():
print "="*40, "Closing connections..."
yield closeConnections()
print "="*40, "closed."
print "="*40, "Stopping reactor..."
reactor.stop()
print "="*40, "stopped."
app.connect(app, SIGNAL("lastWindowClosed()"), stop)
reactor.runReturn()
rc = app.exec_()
exit(rc)
And here's a stripped down version of my cleanup code:
#defer.inlineCallbacks
def closeConnections():
for connection in connections:
print "Closing connection #%s" % connection
yield threads.deferToThread(popen("/foo/bar/cleanup %s" % connection))
print "Connection closed."
The first print statement is reached, and the command is executed, but I never get the second one, nor do I go through the for loop multiple times.
Is my analysis correct? Is the problem that the reactor is already down, so I never hear back from threads.deferToThread? Or is there some other problem? Furthermore, how do I fix it?
Thanks,
Jonathan
I don't know exactly when that lastWindowClosed() signal fires. However, even if it fires early enough, before the reactor has shut down (preventing you from doing what you want to do), I'm sure that PyQt doesn't know what to do with the Deferred that is returned by your stop function. This means that the shutdown process will proceed merrily onward while your asynchronous cleanup code tries to run. Likely the GUI shutdown will finish before your network shutdown gets anywhere.
So, use reactor.addSystemEventTrigger('before', 'shutdown', stop) instead. I don't know if this will run slightly earlier or slightly later than lastWindowClosed(), but it will run early enough that the reactor will still be usable, and it will pay attention to the Deferred your function returns. Shutdown will be suspended, in fact, until that Deferred fires. This gives you all the time you need to do your cleanup.
Separately from all that, you shouldn't do threads.deferToThread(popen("/foo/bar/cleanup %s" % connection)):
You need to pass a callable to deferToThread, not the result of calling the callable. As written, your code runs popen in the reactor thread and passes a file object to the thread to be called (which makes no sense, of course)
Mixing threads and child processes is iffy. You might get away with it most of the time, I dunno.
reactor.spawnProcess will let you run a child process without blocking, without threads, and without worrying about mixing threads and processes. See also twisted.internet.utils.getProcessOutput if you don't need all the features of spawnProcess (which you appear not to).
UPDATE: For ease of reading, here is how to add a callback before the reactor gets shutdown:
reactor.addSystemEventTrigger('before', 'shutdown', callable)
Original question follows.
If I have a client connected to a server, and it's chilling in the reactor main loop waiting for events, when I hit CTRL-C, I get a "Connection to the other side was lost in a non-clean fashion: Connection lost." How can I set it up so that I know when a KeyboardInterrupt happens, so that I can do proper clean-up and disconnect cleanly? Or how can I implement a cleaner way to shutdown that doesn't involve CTRL-C, if possible?
If you really, really want to catch C-c specifically, then you can do this in the usual way for a Python application - use signal.signal to install a handler for SIGINT that does whatever you want to do. If you invoke any Twisted APIs from the handler, make sure you use reactor.callFromThread since almost all other Twisted APIs are unsafe for invocation from signal handlers.
However, if you're really just interested in inserting some shutdown-time cleanup code, then you probably want to use IService.stopService (or the mechanism in terms of which it is implemented,reactor.addSystemEventTrigger) instead.
If you're using twistd, then using IService.stopService is easy. You already have an Application object with at least one service attached to it. You can add another one with a custom stopService method that does your shutdown work. The method is allowed to return a Deferred. If it does, then the shutdown process is paused until that Deferred fires. This lets you clean up your connections nicely, even if that involves some more network (or any other asynchronous) operations.
If you're not using twistd, then using reactor.addSystemEventTrigger directly is probably easier. You can install a before shutdown trigger which will get called in the same circumstance IService.stopService would have been called. This trigger (just any callable object) can also return a Deferred to delay shutdown. This is done with a call to reactor.addSystemEventTrigger('before', 'shutdown', callable) (sometime before shutdown is initiated, so that it's already registered whenever shutdown does happen).
service.tac gives an example of creating and using a custom service.
wxacceptance.py gives an example of using addSystemEventTrigger and delaying shutdown by (an arbitrary) three seconds.
Both of these mechanisms will give you notification whenever the reactor is stopping. This may be due to a C-c keystroke, or it may be because someone used kill -INT ..., or it may be because somewhere reactor.stop() was called. They all lead to reactor shutdown, and reactor shutdown always processes shutdown event triggers.
I'm not sure whether you talking about a client or a server that you've written.
Anyway, nothing wrong with 'CTRL-C'.
If you're writing a server as an Application. Subclass from twisted.application.service.Service and define startService and stopService. Maintain a list of active protocol instances. Use stopService to go through them and close them gracefully.
If you've got a client, you could also subclass Service, but it could be simpler to use reactor.addSystemEventTrigger('before','shutdown',myCleanUpFunction), and close connection(s) gracefully in this function.