I started a little project using tornado and motor,
I feel some confused respect how i have to handle the access data layer if i want to have a non-bloking access
usally i separate my project with this structure
root_project
-logic
-data
--UsersDao
-handlers
--Users
-main.py
but i don't know if i do something like this the connection would be non-blocking
#gen.coroutine
#tornado.web.asynchronous
def get(self, id):
users = self.settings["User"]
result = yield from users.get(id)
self.write(json_encode(result))
self.finish()
'users' it's my UsersDao object and looks like
class UsersDao(object):
....
def get(self, user, callback=None):
try:
user = yield self._db["users"].find_one({'_id': user})
...create user object
return user
except ValueError:
pass
except OperationFailure:
pass
except Exception:
raise
In general, whenever you use yield, you're doing something asynchronous/non-blocking. So in this case the code you've posted looks correct except for the missing #gen.coroutine decorator on UsersDao.get (whenever you use yield for asynchronous stuff, you need this decorator, and you need to use yield any time you call it).
Related
There is a tricky post handler, sometimes it can take a lots of time (depending on a input values), sometimes not.
What I want is to write back whenever 1 second passes, dynamically allocating the response.
def post():
def callback():
self.write('too-late')
self.finish()
timeout_obj = IOLoop.current().add_timeout(
dt.timedelta(seconds=1),
callback,
)
# some asynchronous operations
if not self.request.connection.stream.closed():
self.write('here is your response')
self.finish()
IOLoop.current().remove_timeout(timeout_obj)
Turns out I can't do much from within callback.
Even raising an exception is suppressed by the inner context and won't be passed through the post method.
Any other ways to achieve the goal?
Thank you.
UPD 2020-05-15:
I found similar question
Thanks #ionut-ticus, using with_timeout() is much more convenient.
After some tries, I think I came really close to what i'm looking for:
def wait(fn):
#gen.coroutine
#wraps(fn)
def wrap(*args):
try:
result = yield gen.with_timeout(
dt.timedelta(seconds=20),
IOLoop.current().run_in_executor(None, fn, *args),
)
raise gen.Return(result)
except gen.TimeoutError:
logging.error('### TOO LONG')
raise gen.Return('Next time, bro')
return wrap
#wait
def blocking_func(item):
time.sleep(30)
# this is not a Subprocess.
# It is a file IO and DB
return 'we are done here'
Still not sure, should wait() decorator being wrapped in a
coroutine?
Some times in a chain of calls of a blocking_func(), there can
be another ThreadPoolExecutor. I have a concern, would this work
without making "mine" one global, and passing to the
Tornado's run_in_executor()?
Tornado: v5.1.1
An example of usage of tornado.gen.with_timeout. Keep in mind the task needs to be async or else the IOLoop will be blocked and won't be able to process the timeout:
#gen.coroutine
def async_task():
# some async code
#gen.coroutine
def get(self):
delta = datetime.timedelta(seconds=1)
try:
task = self.async_task()
result = yield gen.with_timeout(delta, task)
self.write("success")
except gen.TimeoutError:
self.write("timeout")
I'd advise to use https://github.com/aio-libs/async-timeout:
import asyncio
import async_timeout
def post():
try:
async with async_timeout.timeout(1):
# some asynchronous operations
if not self.request.connection.stream.closed():
self.write('here is your response')
self.finish()
IOLoop.current().remove_timeout(timeout_obj)
except asyncio.TimeoutError:
self.write('too-late')
self.finish()
I have a report service in tornado application.
I would like to re-use the function that creates reports from a report Json.
Meaning, in the new handler that "regenerate" existing report, I would like to reuse an existing handler that knows how to create reports from a Json.
server.py:
def create_server():
return tornado.web.Application([
(r"/task", generator.GenHandler),
(r"/task/(.+)", generator.GenHandler),
url(r"/regenerate_task", generator.GenHandler, name="regenerate_task"),
url(r"/regenerate_task/(.+)", generator.GenHandler, name="regenerate_task"),
(r"/report_status/regenerate", report_status.Regenerate)
genHandler.class:
class GenHandler(tornado.web.RequestHandler):
async def post(self):
try:
LOGGER.info(str(self.request.body))
gen_args = self.parsed_body
# create here report using the parsed body
and this is the handler I am trying to create.
It will take a saved json from DB and create a completely new report with the original report logic.
class Regenerate(tornado.web.RequestHandler):
async def post(self):
rep_id = self.request.arguments.get('rep_id')[0].decode("utf-8") if self.request.arguments.get('rep_id') \
else 0
try:
report = db_handler.get_report_by_id(rep_id)
if *REPORT IS VALID*:
return self.reverse_url("regenerate_task", report)
else:
report = dict(success=True, report_id=rep_id, report=[])
except Exception as ex:
report = dict(success=False, report_id=rep_id, report=[], error=str(ex))
finally:
self.write(report)
Right now, nothing happens. I just get the JSON I needed, but no entry for GenHandler and no report being regenerated
reverse_url returns a url for specified alias, but doesn't invoke it.
You have such problem where you have to invoke another handler because you have poor code organisation. Storing a report generation code (i.e. business logic) in handler is a bad practice and you should move it to a separate class (which is usually called a Controller in a MVC pattern and handler is a View) or at least separate method and then reuse it in your Renegate handler.
I am having problems with a delete request in Tornado. The request arrives to the server and everything in the handler work great, but it never returns the response to the client.
I have tried returning something, with only the "return" and even without the "return" and the result is always the same.
I am using Python 3.4, Tornado 4.1 and the RestClient of Firefox.
#web.asynchronous
#gen.coroutine
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
return
Tornado documentation (tornado.web.asynchronous):
If this decorator is given, the response is not finished when the method > returns. It is up to the request handler to call self.finish() to finish > the HTTP request.
You need to call tornado.web.RequestHandler.finish method. This will work:
#web.asynchronous
#gen.coroutine
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
self.finish()
return
However, you don't need asynchronous approach in this example. This will work also in the same way:
def delete(self, _id):
try:
model = Model()
model.delete(_id)
self.set_status(204)
except Exception as e:
logging.error(e)
self.set_status(500)
return
Also, if you are using #gen.coroutine decorator, you don't need to use #web.asynchronous decorator. Simply use only #gen.coroutine, it is the correct way and much more elegant.
Lastly, I think you should read this article for understanding asynchronous programming in Tornado.
I have a python application which has lots of small database access functions, using sqlalchemy. I'm trying to avoid having lots of boilerplate session handling code around these functions.
I have numerous functions that look something like this:
def get_ticket_history(Session, ticket_id):
s = Session()
try:
rows = s.query(TicketHistory)\
.filter(TicketHistory.ticket_fk==ticket_id)\
.order_by(TicketHistory.id.desc()).all()
s.commit()
return rows
except:
s.rollback()
raise
finally:
s.close()
I am trying to refactor these functions, but not sure I have the best approach yet. The best I currently have is the following:
def execute(Session, fn, *args, **kwargs):
s = Session()
try:
ret = fn(s, *args, **kwargs)
s.commit()
return ret
except:
s.rollback()
raise
finally:
s.close()
def get_ticket_history(self, ticket_id):
def sql_fn(s):
return s.query(TicketHistory)\
.filter(TicketHistory.ticket_fk==ticket_id)\
.order_by(TicketHistory.id.desc()).all()
return execute(self.sentinel_session, sql_fn)
Is there a better or more idiomatic way of doing this? Perhaps using a decorator?
Thanks,
Jon
The SQLAlchemy docs present a possible way of doing this with context managers.
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html#when-do-i-construct-a-session-when-do-i-commit-it-and-when-do-i-close-it
Copying the code snippet here for completeness:
from contextlib import contextmanager
#contextmanager
def session_scope():
"""Provide a transactional scope around a series of operations."""
session = Session()
try:
yield session
session.commit()
except:
session.rollback()
raise
finally:
session.close()
This session_scope can be used cleanly without repeating the boiler plate now.
class ThingOne(object):
def go(self, session):
session.query(FooBar).update({"x": 5})
class ThingTwo(object):
def go(self, session):
session.query(Widget).update({"q": 18})
def run_my_program():
with session_scope() as session:
ThingOne().go(session)
ThingTwo().go(session)
From Sql alchemy version 1.4:
The Session may be used as a context manager without the use of external helper functions.
Example From documentation
Session = sessionmaker(engine)
with Session() as session:
session.add(some_object)
session.add(some_other_object)
session.commit()
To begin, commit transaction and close the session, below approach can be applied.
Session = sessionmaker(engine)
with Session.begin() as session:
session.add(some_object)
session.add(some_other_object)
# commits transaction, closes session
Documentation:
https://docs.sqlalchemy.org/en/14/orm/session_api.html#sqlalchemy.orm.sessionmaker
morphyn's suggestion to use a context manager is good. You could make such a context manager by applying the contextlib.contextmanager decorator to a function very much like your first get_ticket_history, replacing the code between try and except with a yield statement and renaming it, say, transaction. PEP 343 has a near-identical example of that name.
Then, use that context manager with the with statement to reimplement get_ticket_history. It looks like SQLAlchemy already provides that function, though, as method begin:
http://docs.sqlalchemy.org/en/rel_0_8/orm/session.html#autocommit-mode
transaction handling (begin, commit/rolllback) using with clause
with engine.begin() as connection:
r1 = connection.execute(table1.select())
connection.execute(table1.insert(), {"col1": 7, "col2": "this is some data"})
Old questions, but I still stumbled upon it so here is the relevant link from the docu:
https://docs.sqlalchemy.org/en/13/core/connections.html
Would like to sort a simple query, but not sure how this works with "gen.task", as it takes a method as arg1 and param as arg2.
This works more than fine :
response, error = yield gen.Task(db.client().collection.find, {"user_id":user_id})
if response:
#blablabla
But then how do I give it the sort()?
UPDATE : This now throws a 'callback must be callable' error. Which seems to be some other issue with Tornado now.
def findsort(self, find, callback):
return callback(db.client().collection.find(find).sort({"myfield":1}))
#gen.engine
def anotherfunction(self):
response, error = yield gen.Task(self.findsort, {"user_id":user_id})
Use asyncmongo, it works perfectly with gen.
After juggling you will get something like this:
DB = asyncmongo.Client()
class MainHandler(tornado.web.RequestHandler):
#tornado.web.asynchronous
#gen.engine
def get(self):
result, error = yield gen.Task(DB.collection.find, {}, limit=50, sort=[('myfield', 1)])
And about 'callback must be callable'.. When working with gen - always describe +1 argument in functions, which is called by gen.Task.
def findsort(self, find, params, callback): #here we recieve self + 3 args, if we remove params - callback will contain {"user_id":user_id}
return callback(db.client().collection.find(find).sort({"myfield":1}))
#gen.engine
def anotherfunction(self):
response, error = yield gen.Task(self.findsort, {"user_id":user_id}) #we see 2 args, but it passes 3 args to findsort
Looks like you are trying to make db calls to mongo db asynchronous. By default pymongo is blocking but there is a separate branch called motor which makes it possible to have async queries.
See http://emptysquare.net/blog/introducing-motor-an-asynchronous-mongodb-driver-for-python-and-tornado/ for more details.
It supports the tornado.gen generator pattern too.
I'm not familiar with gen.Task but maybe you could try:
#gen.engine
def anotherfunction(self):
def findsort(find):
return db.client().collection.find(find).sort({"myfield":1})
response, error = yield gen.Task(findsort, {"user_id":user_id})
You should use asyncmongo, an async implemntation of pymongo.
gen.Task requires the function to have a callback parameter.