From the following guide: http://krondo.com/blog/?p=1682
Deferreds help us avoid one of the pitfalls we identified with callback programming. When we use a deferred to manage our callbacks, we simply can’t make the mistake of calling both the callback and the errback, or invoking the callback twenty-seven times. We can try, but the deferred will raise an exception right back at us, instead of passing our mistake onto the callbacks themselves
Can anyone give me a better explanation?
I noticed that it wouldn't work anyways because in most cases in this tutorial the end callback also calls reactor.stop().
But why though doesn't it make sense to call a deferred twice? Why is it any different than calling a chain of methods again?
A Deferred represents the result of a request which may be available (now or in the future) but is not definitely available now.
Results flow through Deferreds, via their callback chain. For example, in a synchronous program, you might have something like:
response_bytes = make_request(...)
response_dict = parse_json(response_bytes)
response_object = construct_object(response_dict)
return response_object.method()
Translated to code that returns a Deferred, this is:
response_bytes_later = make_request_async(...)
response_dict_later = response_bytes_later.addCallback(parse_json)
response_object_later = response_dict_later.addCallback(construct_object)
return response_object_later.addCallback(lambda response_object:
response_object.method())
Asking why you can't fire (or "call back") the Deferred returned by make_request_async is similar to asking why you can't have make_request return multiple times to cause the request to be reissued. If you want to issue the request again in the synchronous version, you have to call make_request again (and get a new result). If you want to issue the request again in the asynchronous version, you have to call make_request_async again (and get a new Deferred).
Related
I have some hooks in place, and I thought I could decorate them with #ndb.tasklet in order to use async apis inside the hooks.
e.g.
#classmethod
#ndb.tasklet
def _post_delete_hook(cls, key,future):
yield do_something_async()
This seemed to work, but every now and then I see "suspended generator" error for the code inside those hooks.
Should I be using #ndb.synctasklet instead?
An example of error:
suspended generator _post_put_hook(data_field.py:112) raised TypeError(Expected Future, received <class 'google.appengine.api.apiproxy_stub_map.UserRPC'>: <google.appengine.api.apiproxy_stub_map.UserRPC object at 0x09AA00B0>)
The code causing the error occasionally was:
t, d = yield (queue.add_async(task), queue.delete_tasks_async(taskqueue.Task(name=existing_task_name)))
Now that I've put #ndb.synctasklet it raises an actual exception.
An ndb tasklet returns a future. If calling the tasklet results in an exception, the exception will only be raised if the future's get_result method is called.
ndb.synctasklet automatically calls get_result on the futures yielded by tasklets, causing exceptions to be raised if they occurred, rather than just logged.
For the error that you are seeing, you may be able to fix it by converting the UserRPCs returned by the taskqueue async methods to tasklets.
This untested code is based on ndb.context.urlfetch (link), which converts the UserRPC produced by urlfetch.createRPC into a Future.
#ndb.tasklet
def add_async(queue, **taskqueue_kwargs):
rpc = queue.add_async(**taskqueue_kwargs)
result = yield rpc
raise ndb.Return(result)
You would need to create a tasklet for each async method that you want to use, or you could extend the taskqueue class and make the async methods tasklets.
I have a scenario with Tornado where I have a coroutine that is called from a non-coroutine or without yielding, yet I need to propagate the exception back.
Imagine the following methods:
#gen.coroutine
def create_exception(with_yield):
if with_yield:
yield exception_coroutine()
else:
exception_coroutine()
#gen.coroutine
def exception_coroutine():
raise RuntimeError('boom')
def no_coroutine_create_exception(with_yield):
if with_yield:
yield create_exception(with_yield)
else:
create_exception(with_yield)
Calling:
try:
# Throws exception
yield create_exception(True)
except Exception as e:
print(e)
will properly raise the exception. However, none of the following raise the exception :
try:
# none of these throw the exception at this level
yield create_exception(False)
no_coroutine_create_exception(True)
no_coroutine_create_exception(False)
except Exception as e:
print('This is never hit)
The latter are variants similar to what my problem is - I have code outside my control calling coroutines without using yield. In some cases, they are not coroutines themselves. Regardless of which scenario, it means that any exceptions they generate are swallowed until Tornado returns them as "future exception not received."
This is pretty contrary to Tornado's intent, their documentation basically states you need to do yield/coroutine through the entire stack in order for it to work as I'm desiring without hackery/trickery.
I can change the way the exception is raised (ie modify exception_coroutine). But I cannot change several of the intermediate methods.
Is there something I can do in order to force the exception to be raised throughout the Tornado stack, even if it is not properly yielded? Basically to properly raise the exception in all of the last three situations?
This is complicated because I cannot change the code that is causing this situation. I can only change exception_coroutine for example in the above.
What you're asking for is impossible in Python because the decision to yield or not is made by the calling function after the coroutine has finished. The coroutine must return without raising an exception so it can be yielded, and after that it is no longer possible for it to raise an exception into the caller's context in the event that the Future is not yielded.
The best you can do is detect the garbage collection of a Future, but this can't do anything but log (this is how the "future exception not retrieved" message works)
If you're curious why this isn't working, it's because no_coroutine_create_exception contains a yield statement. Therefore it's a generator function, and calling it does not execute its code, it only creates a generator object:
>>> no_coroutine_create_exception(True)
<generator object no_coroutine_create_exception at 0x101651678>
>>> no_coroutine_create_exception(False)
<generator object no_coroutine_create_exception at 0x1016516d0>
Neither of the calls above executes any Python code, it only creates generators that must be iterated.
You'd have to make a blocking function that starts the IOLoop and runs it until your coroutine finishes:
def exception_blocking():
return ioloop.IOLoop.current().run_sync(exception_coroutine)
exception_blocking()
(The IOLoop acts as a scheduler for multiple non-blocking tasks, and the gen.coroutine decorator is responsible for iterating the coroutine until completion.)
However, I think I'm likely answering your immediate question but merely enabling you to proceed down an unproductive path. You're almost certainly better off using async code or blocking code throughout instead of trying to mix them.
I am getting started with twisted as one of the libs I'll be using depends on it. In an early test I am confused about how to catch an exception thrown in a function like this:
#defer.inlineCallbacks
def read_input_status(self, address, length, callback):
assert callback
# ...
If callbackis None an AssertionError is thrown, so I tried to see it... The function is called like this:
def cb():
pass
def eb():
pass
d = task.deferLater(reactor, 1, client.read_input_status, 0x0000, 8, None)
d.addCallback(cb)
d.addErrback(eb)
I'm calling deferLater here on purpose to be able to deal with errors, as I understood that's not possible when using `callLater'. But my errback is never called.
What's weird is that when trying to debug and looking at the twisted lib's code I think I've seen a reason, why my errback is without effect. My decorated generator function (argument g below) is called by twisted's defer._inlineCallbacks implementation like this (breviated):
def _inlineCallbacks(result, g, deferred):
# ...
while 1:
try:
# ...
result = g.send(result)
except:
deferred.errback()
return deferred
I do see my exception pop up in the last section, where a deferred's errback is then called. But: that is not my deferred... If I go up one call in the debugger's call hierarchy, I see which deferred object is actually passed to _inlineCallbacks:
def unwindGenerator(*args, **kwargs):
# ...
return _inlineCallbacks(None, gen, Deferred())
Am I mistaken or is this simply a new object, empty, plain, without an callbacks/errbacks attached?
Sorry for this lengthy elaboration. Couldn't find anything immediately related, except for this SO post where I could not directly see how it solves my issue.
Thanks a lot.
[UPDATE] Please see this gist for working sample code (Python 2.7.6, Twisted 13.2.0).
Figured it out after rereading the docs about Twisted Deferred callbacks and errbacks. The issue with the code above and in the linked gist are the missing arguments for callback and errback. If I replace what's written above with the following code, the exception is caught fine and notified via errback as expected:
def cb(result):
pass
def eb(failure):
pass
deferred.defer(f,e)
function f(e):
e.put_async()
Changes to e are discarded in the SDK and presumably on production too. One obvious way to solve the problem is to store all rpc's and get_result() them, but this is not pretty.
Is there a way to make this function ndb-compatible?
This function is used in multiple parts of the code, normal requests are #ndb.toplevel, so there is no problem for them.
The function that you defer can't be an #ndb.toplevel, but you can have that call a function that is:
def f(e):
g(e)
#ndb.toplevel
def g(e):
e.put_async()
This one's a structure design problem, I guess. Back for some advice.
To start: I'm writing a module. Hence the effort of making it as usable to potential developers as possible.
Inside an object (let's call it Swoosh) I have a method which, when called, may result in either success (a new object is returned -- for insight: it's an httplib.HTTPResponse) or failure (surprising, isn't it?).
I'm having trouble deciding how to handle failures. There are two main cases here:
user supplied data that was incorrect
data was okay, but user interaction will be needed () - I need to pass back to the user a string that he or she will need to use in some way.
In (1) I decided to raise ValueError() with an appropriate description.
In (2), as I need to actually pass a str back to the user.. I'm not sure about whether it would be best to just return a string and leave it to the user to check what the function returned (httplib.HTTPResponse or str) or raise a custom exception? Is passing data through raising exceptions a good idea? I don't think I've seen this done anywhere, but on the other hand - I haven't seen much.
What would you, as a developer, expect from an object/function like this?
Or perhaps you find the whole design ridiculous - let me know, I'll happily learn.
As much as I like the approach of handling both cases with specifically-typed exceptions, I'm going to offer a different approach in case it helps: callbacks.
Callbacks tend to work better if you're already using an asynchronous framework like Twisted, but that's not their only place. So you might have a method that takes a function for each outcome, like this:
def do_request(on_success, on_interaction_needed, on_failure):
"""
Submits the swoosh request, and awaits a response.
If no user interaction is needed, calls on_success with a
httplib.HTTPResponse object.
If user interaction is needed, on_interaction_needed is
called with a single string parameter.
If the request failed, a ValueError is passed to on_failure
"""
response = sumbit_request()
if response.is_fine():
on_success(response)
elif response.is_partial()
on_interaction_needed(response.message)
else:
on_failure(ValueError(response.message))
Being Python, there are a million ways to do this. You might not like passing an exception to a function, so you maybe just take a callback for the user input scenario. Also, you might pass the callbacks in to the Swoosh initialiser instead.
But there are drawbacks to this too, such as:
Carelessness may result in spaghetti code
You're allowing your caller to inject logic into your function (eg. exceptions raised in the callback will propagate out of Swoosh)
My example here is simple, your actual function might not be
As usual, careful consideration and good documentation should avoid these problems. In theory.
I think raising an exception may actually be a pretty good idea in this case. Squashing multiple signals into a single return value of a function isn't ideal in Python, due to duck typing. It's not very Pythonic; every time you need to do something like:
result = some_function(...)
if isinstance(result, TypeA):
do_something(result)
elif isinstance(result, TypeB):
do_something_else(result)
you should be thinking about whether it's really the best design (as you're doing).
In this case, if you implement a custom exception, then the code that calls your function can just treat the returned value as a HTTPResponse. Any path where the function is unable to return something its caller can treat that way is handled by throwing an exception.
Likewise, the code that catches the exception and prompts the user with the message doesn't have to worry about the exact type of the thing its getting. It just knows that it's been explicitly instructed (by the exception) to show something to the user.
If the user interaction case means the calling code has to show a prompt, get some input and them pass control back to your function, it might be ugly trying to handle that with an exception. Eg,
try:
Swoosh.method()
except UserInteraction, ex:
# do some user interaction stuff
# pass it back to Swoosh.method()?
# did Swoosh need to save some state from the last call?
except ValueError:
pass # whatever
If this user interaction is a normal part of the control flow, it might be cleaner to pass a user-interaction function into your method in the first place - then it can return a result to the Swoosh code. For example:
# in Swoosh
def method(self, userinteractor):
if more_info_needed:
more_info = userinteractor.prompt("more info")
...
ui = MyUserInteractor(self) # or other state
Swoosh.method(ui)
You can return a tuple of (httplib.HTTPResponse, str) with the str being optionally None.
Definitely raise an exception for 1).
If you don't like returning a tuple, you can also create a "response object" i.e. an instance of a new class ( lets say SomethingResponse ) that encapsulates the HTTPResponse with optional messages to the end-user( in the simplest case, just a str).