Develop Reference

How to wrap custom future to use with asyncio in Python?

There is a lot of libraries that use their custom version of Future. kafka and s3transfer are just two examples: all their custom future-like classes have object as the superclass.
Not surprisingly, you cannot directly call asyncio.wrap_future() on such objects and can't use await with them.
What is the proper way of wrapping such futures for use with asyncio?

If the future class supports standard future features such as done callbacks and the result method, just use something like this:
def wrap_future(f):
loop = asyncio.get_event_loop()
aio_future = loop.create_future()
def on_done(*_):
try:
result = f.result()
except Exception as e:
loop.call_soon_threadsafe(aio_future.set_exception, e)
else:
loop.call_soon_threadsafe(aio_future.set_result, result)
f.add_done_callback(on_done)
return aio_future
Consider that code a template which you can customize to match the specifics of the future you are dealing with.
Intended usage is to call it from the thread that runs the asyncio event loop:
value = await wrap_future(some_foreign_future)
If you are calling it from a different thread, be sure to pass loop explicitly because asyncio.get_event_loop will fail when invoked from a thread not registered with asyncio.

How can I raise an exception through Tornado coroutines incorrectly called?

I have a scenario with Tornado where I have a coroutine that is called from a non-coroutine or without yielding, yet I need to propagate the exception back.
Imagine the following methods:
#gen.coroutine
def create_exception(with_yield):
if with_yield:
yield exception_coroutine()
else:
exception_coroutine()
#gen.coroutine
def exception_coroutine():
raise RuntimeError('boom')
def no_coroutine_create_exception(with_yield):
if with_yield:
yield create_exception(with_yield)
else:
create_exception(with_yield)
Calling:
try:
# Throws exception
yield create_exception(True)
except Exception as e:
print(e)
will properly raise the exception. However, none of the following raise the exception :
try:
# none of these throw the exception at this level
yield create_exception(False)
no_coroutine_create_exception(True)
no_coroutine_create_exception(False)
except Exception as e:
print('This is never hit)
The latter are variants similar to what my problem is - I have code outside my control calling coroutines without using yield. In some cases, they are not coroutines themselves. Regardless of which scenario, it means that any exceptions they generate are swallowed until Tornado returns them as "future exception not received."
This is pretty contrary to Tornado's intent, their documentation basically states you need to do yield/coroutine through the entire stack in order for it to work as I'm desiring without hackery/trickery.
I can change the way the exception is raised (ie modify exception_coroutine). But I cannot change several of the intermediate methods.
Is there something I can do in order to force the exception to be raised throughout the Tornado stack, even if it is not properly yielded? Basically to properly raise the exception in all of the last three situations?
This is complicated because I cannot change the code that is causing this situation. I can only change exception_coroutine for example in the above.

What you're asking for is impossible in Python because the decision to yield or not is made by the calling function after the coroutine has finished. The coroutine must return without raising an exception so it can be yielded, and after that it is no longer possible for it to raise an exception into the caller's context in the event that the Future is not yielded.
The best you can do is detect the garbage collection of a Future, but this can't do anything but log (this is how the "future exception not retrieved" message works)

If you're curious why this isn't working, it's because no_coroutine_create_exception contains a yield statement. Therefore it's a generator function, and calling it does not execute its code, it only creates a generator object:
>>> no_coroutine_create_exception(True)
<generator object no_coroutine_create_exception at 0x101651678>
>>> no_coroutine_create_exception(False)
<generator object no_coroutine_create_exception at 0x1016516d0>
Neither of the calls above executes any Python code, it only creates generators that must be iterated.
You'd have to make a blocking function that starts the IOLoop and runs it until your coroutine finishes:
def exception_blocking():
return ioloop.IOLoop.current().run_sync(exception_coroutine)
exception_blocking()
(The IOLoop acts as a scheduler for multiple non-blocking tasks, and the gen.coroutine decorator is responsible for iterating the coroutine until completion.)
However, I think I'm likely answering your immediate question but merely enabling you to proceed down an unproductive path. You're almost certainly better off using async code or blocking code throughout instead of trying to mix them.

Is this right way to call coroutine method in Tornado framework?

I have WebSocketHandler in my Tornado application.
I am not sure is this a right way to make code asynchronous.
class MyHandler(WebSocketHandler):
def open(self):
do something ...
self.my_coroutine_method()
#gen.coroutine
def my_coroutine_method(self):
user = yield db.user.find_one() # call motor asynchronous engine
self.write_message(user)

Yes, this is correct. However, in some cases simply calling a coroutine without yielding can cause exceptions to be handled in unexpected ways, so I recommend using IOLoop.current().spawn_callback(self.my_coroutine_method) when calling a coroutine from a non-coroutine like this.

Why can't we call a Twisted deferred twice?

From the following guide: http://krondo.com/blog/?p=1682
Deferreds help us avoid one of the pitfalls we identified with callback programming. When we use a deferred to manage our callbacks, we simply can’t make the mistake of calling both the callback and the errback, or invoking the callback twenty-seven times. We can try, but the deferred will raise an exception right back at us, instead of passing our mistake onto the callbacks themselves
Can anyone give me a better explanation?
I noticed that it wouldn't work anyways because in most cases in this tutorial the end callback also calls reactor.stop().
But why though doesn't it make sense to call a deferred twice? Why is it any different than calling a chain of methods again?

A Deferred represents the result of a request which may be available (now or in the future) but is not definitely available now.
Results flow through Deferreds, via their callback chain. For example, in a synchronous program, you might have something like:
response_bytes = make_request(...)
response_dict = parse_json(response_bytes)
response_object = construct_object(response_dict)
return response_object.method()
Translated to code that returns a Deferred, this is:
response_bytes_later = make_request_async(...)
response_dict_later = response_bytes_later.addCallback(parse_json)
response_object_later = response_dict_later.addCallback(construct_object)
return response_object_later.addCallback(lambda response_object:
response_object.method())
Asking why you can't fire (or "call back") the Deferred returned by make_request_async is similar to asking why you can't have make_request return multiple times to cause the request to be reissued. If you want to issue the request again in the synchronous version, you have to call make_request again (and get a new result). If you want to issue the request again in the asynchronous version, you have to call make_request_async again (and get a new Deferred).

How to catch exceptions from a twisted inlineCallbacks decorated function?

I am getting started with twisted as one of the libs I'll be using depends on it. In an early test I am confused about how to catch an exception thrown in a function like this:
#defer.inlineCallbacks
def read_input_status(self, address, length, callback):
assert callback
# ...
If callbackis None an AssertionError is thrown, so I tried to see it... The function is called like this:
def cb():
pass
def eb():
pass
d = task.deferLater(reactor, 1, client.read_input_status, 0x0000, 8, None)
d.addCallback(cb)
d.addErrback(eb)
I'm calling deferLater here on purpose to be able to deal with errors, as I understood that's not possible when using `callLater'. But my errback is never called.
What's weird is that when trying to debug and looking at the twisted lib's code I think I've seen a reason, why my errback is without effect. My decorated generator function (argument g below) is called by twisted's defer._inlineCallbacks implementation like this (breviated):
def _inlineCallbacks(result, g, deferred):
# ...
while 1:
try:
# ...
result = g.send(result)
except:
deferred.errback()
return deferred
I do see my exception pop up in the last section, where a deferred's errback is then called. But: that is not my deferred... If I go up one call in the debugger's call hierarchy, I see which deferred object is actually passed to _inlineCallbacks:
def unwindGenerator(*args, **kwargs):
# ...
return _inlineCallbacks(None, gen, Deferred())
Am I mistaken or is this simply a new object, empty, plain, without an callbacks/errbacks attached?
Sorry for this lengthy elaboration. Couldn't find anything immediately related, except for this SO post where I could not directly see how it solves my issue.
Thanks a lot.
[UPDATE] Please see this gist for working sample code (Python 2.7.6, Twisted 13.2.0).

Figured it out after rereading the docs about Twisted Deferred callbacks and errbacks. The issue with the code above and in the linked gist are the missing arguments for callback and errback. If I replace what's written above with the following code, the exception is caught fine and notified via errback as expected:
def cb(result):
pass
def eb(failure):
pass