This is what I'm trying to accomplish. I'm making a remote call to a server for information, and I want to block to wait for the info. I created a function that returns a Deferred such that when the RPC comes in with the reply, the deferred is called. Then I have a function called from a thread that goes threads.blockingCallFromThread(reactor, deferredfunc, args).
If something goes wrong - for example, the server goes down - then the call will never un-block. I'd prefer the deferred to go off with an exception in these cases.
I partially succeeded. I have a deferred, onConnectionLost which goes off when the connection is lost. I modified my blocking call function to:
deferred = deferredfunc(args)
self.onConnectionLost.addCallback(lambda _: deferred.errback(
failure.Failure(Exception("connection lost while getting run"))))
result = threads.blockingCallFromThread(
reactor, lambda _: deferred, None)
return result
This works fine. If the server goes down, the connection is lost, and the errback is triggered. However, if the server does not go down and everything shuts down cleanly, onConnectionLost still gets fired, and the anonymous callback here attempts to trigger the errback, causing an AlreadyCalled exception to be raised.
Is there any neat way to check that a deferred has already been fired? I want to avoid wrapping it in a try/except block, but I can always resort to that if that's the only way.
There are ways, but you really shouldn't do it. Your code that is firing the Deferred should be keeping track of whether it's fired the Deferred or not in the associated state. Really, when you fire the Deferred, you should lose track of it so that it can get properly garbage collected; that way you never need to worry about calling it twice, since you won't have a reference to it any more.
Also, it looks like you're calling deferredfunc from the same thread that you're calling blockingCallFromThread. Don't do that; functions which return Deferreds are most likely calling reactor APIs, and those APIs are not thread safe. In fact, Deferred itself is not thread safe. This is why it's blockingCallFromThread, not blockOnThisDeferredFromThread. You should do blockingCallFromThread(reactor, deferredfunc, args).
If you really want errback-if-it's-been-called-otherwise-do-nothing behavior, you may want to cancel the Deferred.
Related
I'm seeing an issue in my test suite in trial where everything works fine until I get a timeout. If a test fails due to a timeout, the tearDown function never gets called, leaving the reactor unclean which in turn causes the rest of the tests to fail. I believe tearDown should be called after a timeout, does anyone know why this might happen?
You are correct that tearDown() should be called regardless of what happens in your test. From the documentation for tearDown():
This is called even if the test method raised an exception
However, there is a catch. From the same documentation:
This method will only be called if the setUp() succeeds, regardless of the outcome of the test method.
So it sounds like you perhaps start the reactor in setUp() and when it times out, this is preventing your tearDown() from running - the idea being that whatever you were trying to "set up" in setUp() was not successfully set up, so you do not want to try to tear it down. However, it would be hard to diagnose with certainty unless you provide the code of your setUp and tearDown methods, along with the code of any relevant tests.
It's rather strange because on my box, the teardown executes even if a timeout occurs. The tests should stop running if the reactor is not in a clean state, unless you use the --unclean-warnings flag. Does the test runner stop after the timeout for you? What version of Python and Twisted are you running?
As a side note, if you need to run a unique teardown for a specific test function, there's a very convenient addCleanup() callback. It comes in handy if you need to cancel callback, LoopingCall, or callLater functions so that the reactor isn't in a dirty state. addCleanup returns a Deferred so you can just chain callbacks that perform an adhoc teardown. It might be a good option to try if the class teardown isn't working for you.
PS
I've been so used to writing "well behaved" Twisted code, I don't even recall how to get into an unclean reactor state :D I swear I'm not bragging. Could you provide me a brief summary of what you're doing so that I could test it out on my end?
I found the problem, I'll put this here in case it's helpful to anyone else in the future.
I was returning a deferred from the test that had already been called (as in, deferred.callback had been called), but it still had an unfinished callback chain. From what I can see in the trial code here https://github.com/twisted/twisted/blob/twisted-16.5.0/src/twisted/trial/_asynctest.py#L92, the reactor is crashed when this happens, which explains why the tearDown doesn't get called. The solution for me was to return a deferred from the offending tests that does not have a callback chain that lives for a long time (it's callbacks do not return deferreds themselves).
I'm writing a multi-threaded application that utilizes QThreads. I know that, in order to start a thread, I need to override the run() method and call that method using the thread.start() somewhere (in my case in my GUI thread).
I was wondering, however, is it required to call the .wait() method anywhere and also am I supposed to call the .quit() once the thread finishes, or is this done automatically?
I am using PySide.
Thanks
Both answers depend on what your code is doing and what you expect from the thread.
If your logic which uses the thread needs to wait synchronously for the moment QThread finishes, then yes, you need to call wait(). However such requirement is a sign of sloppy threading model, except very specific situations like application startup and shutdown. Usage of QThread::wait() suggests creeping sequential operation, which means that you are effectively not using threads concurrently.
quit() exits QThread-internal event loop, which is not mandatory to use. A long-running thread (as opposed to one-task worker) must have an event loop of some sort - this is a generic statement, not specific to QThread. You either do it yourself (in form of some while(keepRunning) { } cycle) or use Qt-provided event loop, which you fire off by calling exec() in your run() method. The former implementation is finishable by you, because you did provide the keepRunning condition. The Qt-provided implementation is hidden from you and here goes the quit() call - which internally does nothing more than setting some sort of similar flag inside Qt.
To achieve something similar to google app engines 'deferred calls' (i.e., the request is handled, and afterwards the deferred task is handled), i experimented a little and came up with the solution to spawn a thread in which my deferred call is handled.
I am now trying to determine if this is an acceptable way.
Is it possible (according to the WSGI specification) that the process is terminated by the webserver after the actual request is handled, but before all threads run out?
(if there's a better way, that would be also fine)
WSGI does not specify the lifetime of an application process (as WSGI application is a Python callable object). You can run it in a way that is completely independent of the web server, in which case, only you control the lifetime.
There is also nothing in the WSGI that would prohibit you from spawning threads, or processes, or doing whatever the hell you want.
FWIW, also have a read of:
http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
The hooking of actions to close() of iterable is the only way within context of the WSGI specification itself for doing deferred work. That isn't in a separate thread though and would occur within the context of the actual request, albeit after the response is supposed to have been flushed back to the client. Thus your deferred action will consume that request thread until the work is complete and so that request thread would not be able to handle other requests until then.
In general, if you do use background threads, there is no guarantee that any hosting mechanism would wait until those background threads complete before shutting process down. In fact, can't even think of any standard deployment mechanism which does wait. There isn't really even a guarantee that atexit handlers will be called on process shutdown, something that the referenced documentation also briefly talks about.
Consider the following code:
df = defer.Deferred()
def hah(_): raise ValueError("4")
df.addCallback(hah)
df.callback(hah)
When it runs, that exception just gets eaten. Where did it go? How can I get it to be displayed? Doing defer.setDebugging(True) has no effect.
I ask this because other times, I get a printout saying "Unhandled error in Deferred:". How do I get that to happen in this case? I see that if I add an errback to df then the errback gets called with the exception, but all I want to do is print the error and do nothing else, and I don't want to manually add that handler to every deferred I create.
The exception is still sitting in the Deferred. There are two possible outcomes at this point:
You could add an errback to the Deferred. As soon as you do, it will get called with a Failure containing the exception that was raised.
You could let the Deferred be garbage collected (explicitly delete df, or return from the function, or lose the reference in any other way). This triggers the ''Unhandled error in Deferred'' code.
Because an errback can be added to a Deferred at any time (ie, the first point above), Deferreds don't do anything with otherwise unhandled errors right away. They don't know if the error is really unhandled, or just unhandled so far. It's only when the Deferred is garbage collected that it can be sure no one else is going to handle the exception, so that's when it gets logged.
In general, you want to be sure you have errbacks on Deferreds, precisely because it's sometimes hard to predict when a Deferred will get garbage collected. It might be a long time, which means it might be a long time before you learn about the exception if you don't have your own errback attached.
This doesn't have to be a terrible burden. Any Deferred (a) which is returned from a callback on another Deferred (b) (ie, when chaining happens) will pass its errors along to b. So (a) doesn't need extra errbacks on it for logging and reporting, only (b) does. If you have a single logical task which is complicated and involves many asynchronous operations, it's almost always the case that all of the Deferreds involved in those operations should channel their results (success or failure) to one main Deferred that represents the logical operation. You often only need special error handling behavior on that one Deferred, and that will let you handle errors from any of the other Deferreds involved.
UPDATE: For ease of reading, here is how to add a callback before the reactor gets shutdown:
reactor.addSystemEventTrigger('before', 'shutdown', callable)
Original question follows.
If I have a client connected to a server, and it's chilling in the reactor main loop waiting for events, when I hit CTRL-C, I get a "Connection to the other side was lost in a non-clean fashion: Connection lost." How can I set it up so that I know when a KeyboardInterrupt happens, so that I can do proper clean-up and disconnect cleanly? Or how can I implement a cleaner way to shutdown that doesn't involve CTRL-C, if possible?
If you really, really want to catch C-c specifically, then you can do this in the usual way for a Python application - use signal.signal to install a handler for SIGINT that does whatever you want to do. If you invoke any Twisted APIs from the handler, make sure you use reactor.callFromThread since almost all other Twisted APIs are unsafe for invocation from signal handlers.
However, if you're really just interested in inserting some shutdown-time cleanup code, then you probably want to use IService.stopService (or the mechanism in terms of which it is implemented,reactor.addSystemEventTrigger) instead.
If you're using twistd, then using IService.stopService is easy. You already have an Application object with at least one service attached to it. You can add another one with a custom stopService method that does your shutdown work. The method is allowed to return a Deferred. If it does, then the shutdown process is paused until that Deferred fires. This lets you clean up your connections nicely, even if that involves some more network (or any other asynchronous) operations.
If you're not using twistd, then using reactor.addSystemEventTrigger directly is probably easier. You can install a before shutdown trigger which will get called in the same circumstance IService.stopService would have been called. This trigger (just any callable object) can also return a Deferred to delay shutdown. This is done with a call to reactor.addSystemEventTrigger('before', 'shutdown', callable) (sometime before shutdown is initiated, so that it's already registered whenever shutdown does happen).
service.tac gives an example of creating and using a custom service.
wxacceptance.py gives an example of using addSystemEventTrigger and delaying shutdown by (an arbitrary) three seconds.
Both of these mechanisms will give you notification whenever the reactor is stopping. This may be due to a C-c keystroke, or it may be because someone used kill -INT ..., or it may be because somewhere reactor.stop() was called. They all lead to reactor shutdown, and reactor shutdown always processes shutdown event triggers.
I'm not sure whether you talking about a client or a server that you've written.
Anyway, nothing wrong with 'CTRL-C'.
If you're writing a server as an Application. Subclass from twisted.application.service.Service and define startService and stopService. Maintain a list of active protocol instances. Use stopService to go through them and close them gracefully.
If you've got a client, you could also subclass Service, but it could be simpler to use reactor.addSystemEventTrigger('before','shutdown',myCleanUpFunction), and close connection(s) gracefully in this function.