Develop Reference

How to paginate in Flask-SQLAlchemy for db.session joined queries?

Say, we have the following relationships:
a person can have many email addresses
a email service provider can (obviously) serve multiple email address
So, it's a many to many relationship. I have three tables: emails, providers, and users. Emails have two foreign ids for provider and user.
Now, given a specific person, I want to print all the email providers and the email address it hosts for this person, if it exists. (If the person do not have an email at Gmail, I still want Gmail be in the result. I believe otherwise I only need a left inner join to solve this.)
I figured out how to do this with the following subqueries (following the sqlalchemy tutorial):
email_subq = db.session.query(Emails).\
filter(Emails.user_id==current_user.id).\
subquery()
provider_and_email = db.session.query(Provider, email_subq).\
outerjoin(email_subq, Provider.emails).\
all()
This works okay (it returns a 4-tuple of (Provider, user_id, provider_id, email_address), all the information that I want), but I later found out this is not using the Flask BaseQuery class, so that pagination provided by Flask-SQLAlchemy does not work. Apparently db.session.query() is not the Flask-SQLAlchemy Query instance.
I tried to do Emails.query.outerjoin[...] but that returns only columns in the email table though I want both the provider info and the emails.
My question: how can I do the same thing with Flask-SQLAlchemy so that I do not have to re-implement pagination that is already there?
I guess the simplest option at this point is to implement my own paginate function, but I'd love to know if there is another proper way of doing this.

I'm not sure if this is going to end up being the long-term solution, and it does not directly address my concern about not using the Flask-SQLAlchemy's BaseQuery, but the most trivial way around to accomplish what I want is to reimplement the paginate function.
And, in fact, it is pretty easy to use the original Flask-SQLAlchemy routine to do this:
def paginate(query, page, per_page=20, error_out=True):
if error_out and page < 1:
abort(404)
items = query.limit(per_page).offset((page - 1) * per_page).all()
if not items and page != 1 and error_out:
abort(404)
# No need to count if we're on the first page and there are fewer
# items than we expected.
if page == 1 and len(items) < per_page:
total = len(items)
else:
total = query.order_by(None).count()
return Pagination(query, page, per_page, total, items)
Modified from the paginate function found around line 376: https://github.com/mitsuhiko/flask-sqlalchemy/blob/master/flask_sqlalchemy.py

Your question is how to use Flask-SQLAlchemy's Pagination with regular SQLAlchemy queries.
Since Flask-SQLAlchemy's BaseQuery object holds no state of its own, and is derived from SQLAlchemy's Query, and is really just a container for methods, you can use this hack:
from flask.ext.sqlalchemy import BaseQuery
def paginate(sa_query, page, per_page=20, error_out=True):
sa_query.__class__ = BaseQuery
# We can now use BaseQuery methods like .paginate on our SA query
return sa_query.paginate(page, per_page, error_out)
To use:
#route(...)
def provider_and_email_view(page):
provider_and_email = db.session.query(...) # any SQLAlchemy query
paginated_results = paginate(provider_and_email, page)
return render_template('...', paginated_results=paginated_results)
*Edit:
Please be careful doing this. It's really just a way to avoid copying/pasting the paginate function, as seen in the other answer. Note that BaseQuery has no __init__ method. See How dangerous is setting self.__class__ to something else?.
*Edit2:
If BaseQuery had an __init__, you could construct one using the SA query object, rather than hacking .__class__.

Hey I have found a quick fix for this here it is:
provider_and_email = Provider.query.with_entities(email_subq).\
outerjoin(email_subq, Provider.emails).paginate(page, POST_PER_PAGE_LONG, False)

I'm currently using this approach:
query = BaseQuery([Provider, email_subq], db.session())
to create my own BaseQuery. db is the SqlAlchemy instance.
Update: as #afilbert suggests you can also do this:
query = BaseQuery(provider_and_email.subquery(), db.session())

How do you init your application with SQLAlchemy?
Probably your current SQLAlchemy connection has nothing to do with flask.ext.sqalchemy and you use original sqlalchemy
Check this tutorial and check your imports, that they really come from flask.ext.sqlalchemy
http://pythonhosted.org/Flask-SQLAlchemy/quickstart.html#a-minimal-application

You can try to paginate the list with results.
my_list = [my_list[i:i + per_page] for i in range(0, len(my_list), per_page)][page]

I did this and it works:
query = db.session.query(Table1, Table2, ...).filter(...)
if page_size is not None:
query = query.limit(page_size)
if page is not None:
query = query.offset(page*page_size)
query = query.all()

I could be wrong, but I think your problem may be the .all(). By using that, you're getting a list, not a query object.
Try leaving it off, and pass your query to the pagination method like so (I left off all the subquery details for clarity's sake):
email_query = db.session.query(Emails).filter(**filters)
email_query.paginate(page, per_page)

How to set an ndb keyProperty

I'm having some trouble understanding how entities and keys work in Google App Engine NDB.
I have a post entity and a user entity. How do I set the user_key on post to user?
In the interactive console, I have this so far:
from google.appengine.ext import ndb
from app.lib.posts import Post
from app.lib.users import User
from random import shuffle
users = User.query()
posts = Post.query().fetch()
for post in posts:
post.user_key = shuffle(users)[0]
post.put()
I'm just trying to set up some seed data for development. I know this probably isn't the ideal way to set things, but my first question is:
How do I get a key from an entity (the reverse is described in the docs)
How do I set associations in ndb?

try:
post.user_key = shuffle(users)[0].key

Maybe this helps to understand the NDB. I had the same questions with you.
class Person(ndb.Expando):
pass
class Favourite(ndb.Expando):
pass
class Picture(ndb.Expando):
pass
person = Person()
person.put()
picture = Picture()
picture.put()
fav = Favourite(parent=person.key,
person=person.key,
picture=picture.key
)
fav.put()

Verify that shuffle works in this case, as User.query() returns an iter, not a list. (You can convert it to a list using shuffle( [ x for x in users ] ). Beware, this list could be looong.
NDB has some really wired behavior sometimes, so id recommend you dont store an NDB-Key, but its serialized string, which is also compatible to ext.db: post.user_key = shuffle( [ x for x in users ] ).key.urlsafe()
You could use KeyProperty for associations. If you need a more fine-graned control over your relations, you must implement them yourself.
See https://developers.google.com/appengine/docs/python/ndb/properties#structured

Pyramid resource: In plain English

I've been reading on the ways to implement authorization (and authentication) to my newly created Pyramid application. I keep bumping into the concept called "Resource". I am using python-couchdb in my application and not using RDBMS at all, hence no SQLAlchemy. If I create a Product object like so:
class Product(mapping.Document):
item = mapping.TextField()
name = mapping.TextField()
sizes = mapping.ListField()
Can someone please tell me if this is also called the resource? I've been reading the entire documentation of Pyramids, but no where does it explain the term resource in plain simple english (maybe I'm just stupid). If this is the resource, does this mean I just stick my ACL stuff in here like so:
class Product(mapping.Document):
__acl__ = [(Allow, AUTHENTICATED, 'view')]
item = mapping.TextField()
name = mapping.TextField()
sizes = mapping.ListField()
def __getitem__(self, key):
return <something>
If I were to also use Traversal, does this mean I add the getitem function in my python-couchdb Product class/resource?
Sorry, it's just really confusing with all the new terms (I came from Pylons 0.9.7).
Thanks in advance.

I think the piece you are missing is the traversal part. Is Product
the resource? Well it depends on what your traversal produces, it
could produce products.....
Perhaps it might be best to walk this through from the view back to
how it gets configured when the application is created...
Here's a typical view.
#view_config(context=Product, permission="view")
def view_product(context, request):
pass # would do stuff
So this view gets called when context is an instance of Product. AND
if the acl attribute of that instance has the "view"
permission. So how would an instance of Product become context?
This is where the magic of traversal comes in. The very logic of
traversal is simply a dictionary of dictionaries. So one way that this
could work for you is if you had a url like
/product/1
Somehow, some resource needs to be traversed by the segments of the
url to determine a context so that a view can be determined. What if
we had something like...
class ProductContainer(object):
"""
container = ProductContainer()
container[1]
>>> <Product(1)>
"""
def __init__(self, request, name="product", parent=None):
self.__name__ = name
self.__parent__ = parent
self._request = request
def __getitem__(self, key):
p = db.get_product(id=key)
if not p:
raise KeyError(key)
else:
p.__acl__ = [(Allow, Everyone,"view")]
p.__name__ = key
p.__parent__ = self
return p
Now this is covered in the documentation and I'm attempting to boil it
down to the basics you need to know. The ProductContainer is an object
that behaves like a dictionary. The "name" and "parent"
attributes are required by pyramid in order for the url generation
methods to work right.
So now we have a resource that can be traversed. How do we tell
pyramid to traverse ProductContainer? We do that through the
Configurator object.
config = Configurator()
config.add_route(name="product",
path="/product/*traverse",
factory=ProductContainer)
config.scan()
application = config.make_wsgi_app()
The factory parameter expects a callable and it hands it the current
request. It just so happens that ProductContainer.init will do
that just fine.
This might seem a little much for such a simple example, but hopefully
you can imagine the possibilities. This pattern allows for very
granular permission models.
If you don't want/need a very granular permission model such as row
level acl's you probably don't need traversal, instead you can use
routes with a single root factory.
class RootFactory(object):
def __init__(self, request):
self._request = request
self.__acl__ = [(Allow, Everyone, "view")] # todo: add more acls
#view_config(permission="view", route_name="orders")
def view_product(context, request):
order_id, product_id = request.matchdict["order_id"], request.matchdict["product_id"]
pass # do what you need to with the input, the security check already happened
config = Configurator(root_factory=RootFactory)
config.add_route(name="orders",
path="/order/{order_id}/products/{product_id}")
config.scan()
application = config.make_wsgi_app()
note: I did the code example from memory, obviously you need all the necessary imports etc. in other words this isn't going to work as a copy/paste

Have you worked through http://michael.merickel.org/projects/pyramid_auth_demo/ ? If not, I suspect it may help. The last section http://michael.merickel.org/projects/pyramid_auth_demo/object_security.html implements the pattern you're after (note the example "model" classes inherit from nothing more complex than object).

read Colander Schema from a config file or database

I googled for a while but I could not find reference on how to retrieve a Colander Schema from a config file or from a database. I think this is not difficult to implement but I might have overlooked something. Maybe somebody has done or seen something like that and might share some insights.
Here a sample for a Colander Schema:
class PageSchema(colander.MappingSchema):
title = SchemaNode(String(),
title='Page title',
description='The title of the page',
)
description = SchemaNode(String(),
title='A short description',
description='Keep it under 60 characters or so',
missing = u'',
validator=colander.Length(max=79)
)
body = colander.SchemaNode(colander.String(),
description='Tell the world',
missing = u'')

As micheal said, it might not be supported. If you really need it. Here is some pointers.
Save your schema in a database by name for example: "PageSchema". Save all of its record in the database with all the needed parameters.
You'd have to do something like that:
for row in rows:
attrinbutes[row['name']] = build_attribute(row)
schemas[schema_name] = type(schema_name, (colander.MappingSchema,), attributes)
exec('%s = schemas[schema_name]' % schema_name)
In other words, it loads all attributes and build a class using the type operator. That kind of task is pretty simple and should work as good as the habitual class syntax. The exec call is just to push the name in locals. You could probably use locals()[schema_name] = schmea or even other scopes.
That way you can load schemas from anywhere if needed. You could build yourself a factory like:
schemas.get('PageSchema') that would return a schema if possible or None if not present.
That's pretty much it!

This is not supported by colander. The one thing I know of in this area is the "limone" package which does the opposite. It is able to generate arbitrary python objects from a colander schema.

ColanderAlchemy may do what you need. It takes SQLAlchemy objects and generates a Colander schema from them. However, generating from an SQLAlchemy object isn't exactly "from a database".

Django - accessing the RequestContext from within a custom filter

I've got a filter currency, which takes a value in USD and converts it to a currency (either USD or GBP). The currency to convert to is stored in the session, but filters don't take RequestContext, so I can't grab it straight from there.
Is there a better way than passing the relevant session element into the template, and from the template into the filter as an argument? Whilst this approach is working, it seems fairly horrible, and I'm likely to end up passing the currency to (almost) every template.
My filter currently looks something like this:
def currency(value, currency):
if currency == 'usd':
val = '$%.2f' % value
return mark_safe(val)
d = Decimal(value)
val = '£%.2f' % (d*Decimal('0.63'))
return mark_safe(val)

If you create a template tag instead of a filter, you are given the context to work with (which contains the request). http://docs.djangoproject.com/en/dev/howto/custom-template-tags/#writing-custom-template-tags

I would have to agree with Adam that migrating the code to a custom tag is the best way.
However, a client needed to record the use of certain filters only when a page was published and had a HUGE inventory of templates that used the existing filter syntax. It would have been a costly undertaking to rewrite all the templates. So, I came up with this simple function that extracts the context from the call stack:
https://gist.github.com/drhoden/e05292e52fd5fc92cc3b
def get_context(max_depth=4):
import inspect
stack = inspect.stack()[2:max_depth]
context = {}
for frame_info in stack:
frame = frame_info[0]
arg_info = inspect.getargvalues(frame)
if 'context' in arg_info.locals:
context = arg_info.locals['context']
break
return context
Be sure to read my warnings, but this DOES give standard filters access to the context (when it is available) WITHOUT having to turn your filter into a tag.

This can be done using a filter. First make sure that you have "django.core.context_processors.request" in you TEMPLATE_CONTEXT_PROCESSORS. If you don't, you can add this to your settings.py file:
TEMPLATE_CONTEXT_PROCESSORS += (
"django.core.context_processors.request"
)
Then in your template, your filter will look like this (assuming your session variable is named 'currency_type'):
{{value|currency:request.session.currency_type}}
Or is something like this what you are considering fairly horrible?

A somehow less hacky solution to Daniel Rhoden's proposal is, to use threading.local(). Define a middleware class, which stores your request as a global object inside your local thread, and add that class to your MIDDLEWARE_CLASSES.
Now a template filter can easily access that request object.