Develop Reference

Control indexing for ndb.Expando model properties

I have an ndb.Model that contains an ndb.Expando model as a structured property, and I'd like to be able to control indexing, but I cannot find anything in the documentation about it.
class MyModel2(ndb.Expando):
pass
class MyModel(ndb.Model):
sub_entity = ndb.StructuredProperty(MyModel2)
The only piece of documentation I can find is the following:
You can specify the usual property options for structured properties
(except indexed)
I assume this is because the actual indexing should be specified on a per-property basis in the embedded model. However, it is not clear how to enable or disable indexing on Expando model properties.

You can use the _properties object to set if a field should be indexed or not. E.g.
a = MyModel2()
a.value = '3'
a._properties['value']._indexed = False
You can find the docs on _properties have more details.

Detect whether a specific property of an entity has been altered when 'putting' the entity?

Looking for an efficient way to check whether a specific property of a db.Model has been changed vs. the version stored in the datastore before putting it. Obviously I would like to avoid getting a fresh instance from the database! Is it possible? Something like this:
class MyClass(db.Model):
prop = db.StringProperty()
instance = MyClass()
instance.string_property = 'string'
instance.put()
instance2 = db.get(instance.key())
instance2.string_property = 'string2'
instance2.put() #would like to detect that string_property was altered!

I think thats a bad idea.
If you have a look at the code for db (you should really be using ndb now) there is a lot going on under the hood for each property/model (metaclasses are used here heavily) and depending on the class/property and or value being assigned you could start getting some real odd behavior and or bugs.
It doesn't mean you should, but consider what you are doing carefully ;-)
I would consider either using hooks (se nick johnsons blog for some ideas blog.notdot.net/2010/04/… ) or switch to ndb and use it's pre/post hooks (see model hooks https://developers.google.com/appengine/docs/python/ndb/entities#hooks )

I realise this is just a basic Python programming question:
class MyClass():
prop = 'init string'
def __setattr__(self, name, value):
print name + ' is being updated to ' + value
test = MyClass()
test.prop = 'summing else'
prop is being updated to summing else

Entity groups, ReferenceProperty or key as a string

After building a few application on the gae platform I usually use some relationship between different models in the datastore in basically every application. And often I find my self the need to see what record is of the same parent (like matching all entry with same parent)
From the beginning I used the db.ReferenceProperty to get my relations going, like:
class Foo(db.Model):
name = db.StringProperty()
class Bar(db.Model):
name = db.StringProperty()
parentFoo = db.ReferanceProperty(Foo)
fooKey = someFooKeyFromSomePlace
bars = Bar.all()
for bar in bar:
if bar.parentFoo.key() == fooKey:
// do stuff
But lately I've abandoned this approch since the bar.parentFoo.key() makes a sub query to fetch Foo each time. The approach I now use is to store each Foo key as a string on Bar.parentFoo and this way I can string compare this with someFooKeyFromSomePlace and get rid of all the subquery overhead.
Now I've started to look at Entity groups and wondering if this is even a better way to go? I can't really figure out how to use them.
And as for the two approaches above I'm wondering is there any downsides to using them? Could using stored key string comeback and bit me in the * * *. And last but not least is there a faster way to do this?

Tip:
replace...
bar.parentFoo.key() == fooKey
with...
Bar.parentFoo.get_value_for_datastore(bar) == fooKey
To avoid the extra lookup and just fetch the key from the ReferenceProperty
See Property Class

I think you should consider this as well. This will help you fetch all the child entities of a single parent.
bmw = Car(brand="BMW")
bmw.put()
lf = Wheel(parent=bmw,position="left_front")
lf.put()
lb = Wheel(parent=bmw,position="left_back")
lb.put()
bmwWheels = Wheel.all().ancestor(bmw)
For more reference in modeling. you can refer this Appengine Data modeling

I'm not sure what you're trying to do with that example block of code, but I get the feeling it could be accomplished with:
bars = Bar.all().filter("parentFoo " = SomeFoo)
As for entity groups, they are mainly used if you want to alter multiple things in transactions, since appengine restricts that to entities within the same group only; in addition, appengine allows ancestor filters ( http://code.google.com/appengine/docs/python/datastore/queryclass.html#Query_ancestor ), which could be useful depending on what it is you need to do. With the code above, you could very easily also use an ancestor query if you set the parent of Bar to be a Foo.
If your purposes still require a lot of "subquerying" as you put it, there is a neat prefetch pattern that Nick Johnson outlines here: http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine which basically fetches all the properties you need in your entity set as one giant get instead of a bunch of tiny ones, which gets rid of a lot of the overhead. However do note his warnings, especially regarding altering the properties of entities while using this prefetch method.
Not very specific, but that's all the info I can give you until you be more specific about exactly what you're trying to do here.

When you design your modules you also need to consider whether you want to be able to save this within a transaction. However only do this if you need to use transactions.
An alternative approach is to assign the parent like so:
from google.appengine.ext import db
class Foo(db.Model):
name = db.StringProperty()
class Bar(db.Model):
name = db.StringProperty()
def _save_entities( foo_name, bar_name ):
"""Save the model data"""
foo_item = Foo( name = foo_name )
foo_item.put()
bar_item = Bar( parent = foo_item, name = bar_name )
bar_item.put()
def main():
# Run the save in a transaction, if any fail this should all roll back
db.run_in_transaction( _save_transaction, "foo name", "bar name" )
# to query the model data using the ancestor relationship
for item in bar_item.gql("WHERE ANCESTOR IS :ancestor", ancestor = foo_item.key()).fetch(1000):
# do stuff

List of References in Google App Engine for Python

In Google App Engine, there is such a thing as a ListProperty that allows you to hold a list (array) of items. You may also specify the type of the item being held, for instance string, integer, or whatever.
Google App Engine also allows you to have a ReferenceProperty. A ReferenceProperty "contains" a reference to another Google App Engine Model entity. If you access a ReferenceProperty, it will automatically retrieve the actual entity that the reference points to. This is convenient, as it beats getting the key, and then getting the entity for said key.
However, I do not see any such thing as a ListReferenceProperty (or ReferenceListProperty). I would like to hold a list of references to other entities, that would automatically be resolved when I attempt to access elements within the list. The closest I can get it seems is to hold a list of db.Key objects. I can use these keys to then manually retrieve their associated entities from the server.
Is there any good solution to this? Basically, I would like the ability to have a collection of (auto-dereferencing) references to other entities. I can almost get there by having a collection of keys to other entities, but I would like it to "know" that these are key items, and that it could dereference them as a service to me.
Thank you

Step one:
Use db.ListProperty(db.Key) to create the relationship. You want the ListProp to be on the Entity that will have the fewer references in the Many to Many relationship. This will also give you a back reference. So:
class Spam
prop1 = db.String
eggs = db.List
class Eggs
prop1 = db.string
#property
def spams(self):
return Spam.all().filter('eggs', self.key())
This provides a References both ways.
Step two:
Create a utlility method that derefrences properties.
def prefetch_refprops(entities, *props):
"""Dereference Reference Properties to reduce Gets. See:
http://blog.notdot.net/2010/01/ReferenceProperty-prefetching-in-App-Engine
"""
fields = [(entity, prop) for entity in entities for prop in props]
ref_keys = [prop.get_value_for_datastore(x) for x, prop in fields]
ref_entities = dict((x.key(), x) for x in db.get(set(ref_keys)))
for (entity, prop), ref_key in zip(fields, ref_keys):
prop.__set__(entity, ref_entities[ref_key])
return entities
Usage would be:
derefrenced_spams = prefetch_refprops(Spams, models.Spam.eggs)

You're right, there's no built in ReferenceListProperty. It'd be possible to write one yourself - custom Property subclasses are generally fairly easy - but getting it right is harder than you'd think, when it comes to deferencing and caching a list of references.
You can use a db.ListProperty(db.Key), however, which allows you to store a list of keys. Then, you can load them individually or all at once using a batch db.get() operation. This requires you to do the resolution step yourself, but it also gives you more control over when you dereference entities.

Using Property Builtin with GAE Datastore's Model

I want to make attributes of GAE Model properties. The reason is for cases like to turn the value into uppercase before storing it. For a plain Python class, I would do something like:
Foo(db.Model):
def get_attr(self):
return self.something
def set_attr(self, value):
self.something = value.upper() if value != None else None
attr = property(get_attr, set_attr)
However, GAE Datastore have their own concept of Property class, I looked into the documentation and it seems that I could override get_value_for_datastore(model_instance) to achieve my goal. Nevertheless, I don't know what model_instance is and how to extract the corresponding field from it.
Is overriding GAE Property classes the right way to provides getter/setter-like functionality? If so, how to do it?
Added:
One potential issue of overriding get_value_for_datastore that I think of is it might not get called before the object was put into datastore. Hence getting the attribute before storing the object would yield an incorrect value.

Subclassing GAE's Property class is especially helpful if you want more than one "field" with similar behavior, in one or more models. Don't worry, get_value_for_datastore and make_value_from_datastore are going to get called, on any store and fetch respectively -- so if you need to do anything fancy (including but not limited to uppercasing a string, which isn't actually all that fancy;-), overriding these methods in your subclass is just fine.
Edit: let's see some example code (net of imports and main):
class MyStringProperty(db.StringProperty):
def get_value_for_datastore(self, model_instance):
vv = db.StringProperty.get_value_for_datastore(self, model_instance)
return vv.upper()
class MyModel(db.Model):
foo = MyStringProperty()
class MainHandler(webapp.RequestHandler):
def get(self):
my = MyModel(foo='Hello World')
k = my.put()
mm = MyModel.get(k)
s = mm.foo
self.response.out.write('The secret word is: %r' % s)
This shows you the string's been uppercased in the datastore -- but if you change the get call to a simple mm = my you'll see the in-memory instance wasn't affected.
But, a db.Property instance itself is a descriptor -- wrapping it into a built-in property (a completely different descriptor) will not work well with the datastore (for example, you can't write GQL queries based on field names that aren't really instances of db.Property but instances of property -- those fields are not in the datastore!).
So if you want to work with both the datastore and for instances of Model that have never actually been to the datastore and back, you'll have to choose two names for what's logically "the same" field -- one is the name of the attribute you'll use on in-memory model instances, and that one can be a built-in property; the other one is the name of the attribute that ends up in the datastore, and that one needs to be an instance of a db.Property subclass and it's this second name that you'll need to use in queries. Of course the methods underlying the first name need to read and write the second name, but you can't just "hide" the latter because that's the name that's going to be in the datastore, and so that's the name that will make sense to queries!

What you want is a DerivedProperty. The procedure for writing one is outlined in that post - it's similar to what Alex describes, but by overriding get instead of get_value_for_datastore, you avoid issues with needing to write to the datastore to update it. My aetycoon library has it and other useful properties included.