In Python I am trying to create an API for a connected device. I want to be available for both threaded (using request) and async applications (using aiohttp).
What I've come up with is wrapping the get method of both requests and aiohttp in a decorator. This decorator is passed at init and API calls are explicitly wrapped using the passed decorator.
It works, but I'd like to know how others think of this approach ? Are there better ways or will I be running into issues later on ?
Any help appreciated !
def threaded_gett(function):
# The threaded decorator
def wrapper(*args, **kwargs):
url, params = function(*args)
response = requests.get(url, params)
_json = response.json()
return function.__self__.process_response(_json)
return wrapper
def async_gett(function):
# The async decorator
def wrapper(*args, **kwargs):
url, params = function(*args)
try:
resp = yield from function.__self__.session.get(url, params=params)
except Exception as ex:
lgr.exception(ex)
else:
_json = yield from resp.json()
yield from resp.release()
return function.__self__.process_response(_json)
# wrapping the decorator in the async coroutine decorator.
wrapper = asyncio.coroutine(wrapper)
return wrapper
class ThreadedApi(BaseApi):
def __init__(self,threaded_gett):
Base.__init(self,threaded_gett)
class AsyncApi(BaseApi):
def __init__(self,async_gett):
Base.__init(self,async_gett)
class BaseApi():
def __init__(self,get_wrapper):
self.status = get_wrapper(self.status)
def status(self):
return <status path>
Your code is not complete but yes, the approach might work in simple cases (when .process_response() is very generic and could be applied to all API calls).
Related
I'm trying to create a Python script which running Tornado Async http client with fetch and trying to get the response and print the response.body to the screen.
my class code is:
class MyAsyncHTTPClient(httpclient.AsyncHTTPClient):
#gen.coroutine
def _fetch(self, url):
print('send Asynchronous request ...['+url+"] ")
http_response = yield gen.Task(self.fetch, url)
print(http_response.body)
print('got Asynchronous response !!'+str(http_response))
raise gen.Return(http_response.body)
and I'm calling it this way:
async_http_client = MyAsyncHTTPClient()
res_body = async_http_client._fetch(url)
The issue is that I'm not sure how to deal with this code to get the returned value once it's ready.
can you please help?
Thanks!
Editing
I have also tried implementing this function like:
class MyAsyncHTTPClient(httpclient.AsyncHTTPClient):
#gen.coroutine
def _fetch(self, url):
print('send Asynchronous request ...['+url+"] "+str(self))
http_response = yield self.fetch(url)
print('got Asynchronous response !!')
return http_response.body
But I'm having the same results :(
Editing again
I have succeeded running the async class...but without the inherited object self.
I did it like that:
#gen.coroutine
def _fetch_async(self, url):
print('send Asynchronous request ...[' + url + "] ")
http_response = yield httpclient.AsyncHTTPClient().fetch(url)
#http_response = yield self.fetch(url)
print('got Asynchronous response !!')
return http_response.body
and it worked fine.
The issue is that I need to use the inherited object self, and I'm not sure what am I missing here when defining the class.
When debugging I can see that self is very "thin" with its content..
Please let me know what am I doing wrong here.
Thanks!
Asynchronous functions can only be called from other asynchronous functions. You must call _fetch like this:
#gen.coroutine
def f():
async_http_client = MyAsyncHTTPClient()
res_body = yield async_http_client._fetch(url)
If you're not doing this in the context of a tornado server, the best way to call a coroutine from your __main__ block is like this:
tornado.ioloop.IOLoop.current().run_sync(f)
I used requests-futures to grab web page asynchronously.And my machine have multicores, so I also want to grab many websites simultaneously,then I try to use concurrent.futures, it seems concurrent.futures also provide asynchronous method,so what's the difference between concurrent.futures' async and requests-futures' async?If they are same,means I could deprecate requests-futures?
requests-futures is just a very small wrapper on top of concurrent.futures. You can see this by looking at the source code (With docstrings removed for brevity):
from concurrent.futures import ThreadPoolExecutor
from requests import Session
from requests.adapters import DEFAULT_POOLSIZE, HTTPAdapter
class FuturesSession(Session):
def __init__(self, executor=None, max_workers=2, *args, **kwargs):
super(FuturesSession, self).__init__(*args, **kwargs)
if executor is None:
executor = ThreadPoolExecutor(max_workers=max_workers)
# set connection pool size equal to max_workers if needed
if max_workers > DEFAULT_POOLSIZE:
adapter_kwargs = dict(pool_connections=max_workers,
pool_maxsize=max_workers)
self.mount('https://', HTTPAdapter(**adapter_kwargs))
self.mount('http://', HTTPAdapter(**adapter_kwargs))
self.executor = executor
def request(self, *args, **kwargs):
func = sup = super(FuturesSession, self).request
background_callback = kwargs.pop('background_callback', None)
if background_callback:
def wrap(*args_, **kwargs_):
resp = sup(*args_, **kwargs_)
background_callback(self, resp)
return resp
func = wrap
return self.executor.submit(func, *args, **kwargs) # This returns a concurrent.futures.Future
When you use requests-futures, you're really using concurrent.futures.ThreadPoolExecutor, which returns a concurrent.futures.Future when you submit a task to it. If it's more convenient for you to use the API provided by requests-futures for dealing with HTTP requests, it's ok to stick with it, and even use the objects returned by it with other methods provided by the concurrent.futures module.
I have the following web app:
import bottle
app = bottle.Bottle()
#app.route('/ping')
def ping():
print 'pong'
return 'pong'
#app.hook('after_request')
def after():
print 'foo'
print bottle.response.body
if __name__ == "__main__":
app.run(host='0.0.0.0', port='9999', server='cherrypy')
Is there a way to access the response body before sending the response back?
If I start the app and I query /ping, I can see in the console that the ping() and the after() function run in the right sequence
$ python bottle_after_request.py
Bottle v0.11.6 server starting up (using CherryPyServer())...
Listening on http://0.0.0.0:9999/
Hit Ctrl-C to quit.
pong
foo
but when in after() I try to access response.body, I don't have anything.
In Flask the after_request decorated functions take in input the response object so it's easy to access it. How can I do the same in Bottle?
Is there something I'm missing?
Is there a way to access the response body before sending the response back?
You could write a simple plugin, which (depending on what you're actually trying to do with the response) might be all you need.
Here's an example from the Bottle plugin docs, which sets a request header. It could just as easily manipulate body.
from bottle import response, install
import time
def stopwatch(callback):
def wrapper(*args, **kwargs):
start = time.time()
body = callback(*args, **kwargs)
end = time.time()
response.headers['X-Exec-Time'] = str(end - start)
return body
return wrapper
install(stopwatch)
Hope that works for your purposes.
You can use plugin approach, this is what i did
from bottle import response
class BottlePlugin(object):
name = 'my_custom_plugin'
api = 2
def __init__(self, debug=False):
self.debug = debug
self.app = None
def setup(self, app):
"""Handle plugin install"""
self.app = app
def apply(self, callback):
"""Handle route callbacks"""
def wrapper(*a, **ka):
"""Encapsulate the result in the expected api structure"""
# Check if the client wants a different format
# output depends what you are returning from view
# in my case its dict with keys ("data")
output = callback(*a, **ka)
data = output["data"]
paging = output.get("paging", {})
response_data = {
data: data,
paging: paging
}
# in case if you want to update response
# e.g response code
response.status = 200
return response_data
return wrapper
So I have repeating code that I do for many GETs -- checking whether the response was cached previously and returning that if it is available.
The code I'd like to get working looks like this:
class Handler(web.RequestHandler):
#gen.coroutine
def get_cache(self):
try:
response = yield gen.Task(get_redis)
except:
logging.log()
if response:
self.finish(response)
raise gen.Return()
#gen.coroutine
#asynchronous
def get(self):
self.get_cache()
response = do_sql_get()
self.set_cache(key, response)
self.finish(response)
What's happening now is that it gets the cache if there but continues running the rest of the code in self.get. That it does this makes sense to me, but I'm not sure how to refactor it properly with it stopping as soon as self.finish is called in the self.get_cache method.
get_cache should return a value that indicates whether it finished the request or not (or it should return the cached data and leave it to the caller to finish the request). I would do one of the following:
#gen.coroutine
def serve_from_cache(self):
response = yield gen.Task(get_redis)
if response:
self.finish(response)
raise gen.Return(True)
else:
raise gen.Return(False)
#gen.coroutine
def get(self):
if (yield self.serve_from_cache()):
return
# do work
yield self.set_cache(...)
or
#gen.coroutine
def get_cache(self):
return yield gen.Task(get_redis)
#gen.coroutine
def get(self):
resp = yield self.get_cache()
if resp:
self.finish(resp)
return
# do work...
I'm trying to use AsyncHTTPClient in Tornado to do multiple callouts to a "device" available over http:
def ext_call(self, params):
device = AsyncHTTPClient()
request = HTTPRequest(...)
return partial(device.fetch, request)
#coroutine
def _do_call(self, someid):
acall = self.ext_call(params)
waitkey = str(someid)
acall(callback = (yield Callback(waitkey)))
response = yield Wait(waitkey)
raise Return(response)
def get_device_data(self, lst):
for someid in lst:
r = self._do_call(someid)
print 'response', r
But instead of HTTP responses as AsyncHTTPClient should return after .fetch, I'm getting this:
response <tornado.concurrent.TracebackFuture object at 0x951840c>
Why this is not working like examples in http://www.tornadoweb.org/en/stable/gen.html ?
Got this one solved. It appears that #coroutine has to be applied all the way down from the get/post method of your class inheriting from RequestHandler, otherwise #coroutine/yield magic does not work.
Apparently this is a case of Tornado newbiness combined with bad design on my part: according to a colleague one should not do "callback spaghetti" of nested #coroutine and yield()s, but rather move all the synchronous code out of request handler and call before or after async code it and have #coroutine call hierarchy flat rather than deep.