Develop Reference

ComputedProperty only updates on second put()

I have a ComputedProperty inside a StructuredProperty that does not get updated when the object is first created.
When I create the object address_components_ascii does not get saved. The field is not visible in the Datastore Viewer at all. But if I get() and then immediately put() again (even without changing anything), the ComputedProperty works as expected. The address_components field works properly.
I have tried clearing the database, and deleting the whole database folder, without success.
I am using the local dev server on windows 7. I have not tested it on GAE.
Here's the code:
class Item(ndb.Model):
location = ndb.StructuredProperty(Location)
The inner Location class:
class Location(ndb.Model):
address_components = ndb.StringProperty(repeated=True) # array of names of parent areas, from smallest to largest
address_components_ascii = ndb.ComputedProperty(lambda self: [normalize(part) for part in self.address_components], repeated=True)
The normalization function
def normalize(s):
return unicodedata.normalize('NFKD', s.decode("utf-8").lower()).encode('ASCII', 'ignore')
An example of the address_components field:
[u'114B', u'Drottninggatan', u'Norrmalm', u'Stockholm', u'Stockholm', u'Stockholms l\xe4n', u'Sverige']
and the address_components_ascii field, after the second put():
[u'114b', u'drottninggatan', u'norrmalm', u'stockholm', u'stockholm', u'stockholms lan', u'sverige']

The real problem seemed to be the order that GAE calls _prepare_for_put() on the StructuredProperty relative to the call to _pre_put_hook() of the surrounding Model.
I was writing to address_components in the Item._pre_put_hook(). I assume GAE computed the ComputedProperty of the StructuredProperty before calling the _pre_put_hook() on Item. Reading from the ComputedProperty causes its value to be recalculated.
I added this to the end of the _pre_put_hook():
# quick-fix: ComputedProperty not getting generated properly
# read from all ComputedProperties, to compute them again before put
_ = self.address_components_ascii
I'm saving the return value to a dummy variable to avoid IDE warnings.

I just tried this code on dev server and its worked. Computed property is accessible before and after put.
from google.appengine.ext import ndb
class TestLocation(ndb.Model):
address = ndb.StringProperty(repeated=True)
address_ascii = ndb.ComputedProperty(lambda self: [
part.lower() for part in self.address], repeated=True)
class TestItem(ndb.Model):
location = ndb.StructuredProperty(TestLocation)
item = TestItem(id='test', location=TestLocation(
address=['Drottninggatan', 'Norrmalm']))
assert item.location.address_ascii == ['drottninggatan', 'norrmalm']
item.put()
assert TestItem.get_by_id('test').location.address_ascii == [
'drottninggatan', 'norrmalm']

This seems to be a limitation in ndb. Simply doing a put() followed by a get() and another put() worked. It's slower, but only required when creating an object the first time.
I added this method:
def double_put(self):
return self.put().get().put()
which is a drop-in replacement for put().
When I put() a new object I call MyObject.double_put() instead of MyObject.put().

LDAP gidNumber like Auto Integer

Still working with LDAP...
The problem i submit today is this: i'm creating a posixGroup on a server LDAP using a custom method developed in python using Django framework. I attach the method code below.The main issue is that attribute gidNumber is compulsory of posixGroup class, but usually is not required when using graphical LDAP client like phpLDAPadmin since they fill automatically this field like an auto-integer.
Here the question: gidNumber is an auto integer attribute for default, or just using client like the quoted above? Must i specify it during the posixGroup entry creation?
def ldap_cn_entry(self):
import ldap.modlist as modlist
dn = u"cn=myGroupName,ou=plant,dc=ldap,dc=dem2m,dc=it"
# A dict to help build the "body" of the object
attrs = {}
attrs['objectclass'] = ['posixGroup']
attrs['cn'] = 'myGroupName'
attrs['gidNumber'] = '508'
# Convert our dict to nice syntax for the add-function using modlist-module
ldif = modlist.addModlist(attrs)
# Do the actual synchronous add-operation to the ldapserver
self.connector.add_s(dn, ldif)
connector is first instanced in the constructor of the class where this method is built. The constructor provides also to the LDAP initialization and binding. Than, the connection will be closed by the destructor.
to use the method i begin instancing the class it belongs, so it also connects to LDAP server. Than i use the method and finally i destroy the object i instanced before to close the connection. All works, indeed, if use this procedure to create a different entry, or if i specify the gidNumber manually.
The fact is i CAN'T specify the gidNumber any time i want to create a group to goal my purpose. I should leave it filling automatically (if that's possible) or think about another way to complete it.
I'm not posting more code about the class i made to not throng the page.
I'll provide more information if needed. Thank you.

The LDAP protocol has no method for auto-integer.
You need to specify the value when creating the entry.
You can do some tricks to help.
We often put the last used value on an OU (We add an AUX class with custom Attribute to the OU) in LDAP and then read, increment and then use the value when using the gidNumber.
Found this described.
-jim

Following #jeemster suggestion, i found the way to manage gidNumber.
Fist of all: i created a new entry on my LDAP called "gidNumber" and i added the optional attribute description to contain the last gidNumber i used (class: organizationalUnit, ou: gidNumber, description: 500).
Then i created the following functions:
def ldap_gid_finder(self):
# Locates the suport-entry with a simple query
self.baseDN = "ou=impianti,dc=ldap,dc=dem2m,dc=it"
self.searchScope = ldap.SCOPE_SUBTREE
self.retrieveAttributes = None
self.searchFilter = "ou=*gidNumber*"
# Results are putted in a dictionary
self.ldap_result = self.connector.search(
self.baseDN, self.searchScope, self.searchFilter, self.retrieveAttributes)
result_set = []
while 1:
result_type, result_data = self.connector.result(self.ldap_result, 0)
if (result_data == []):
break
else:
if result_type == ldap.RES_SEARCH_ENTRY:
result_set.append(result_data)
# The attribute containing gidNumber is passed to an instanced variable
self.actual_gid_number = int(result_set[0][0][1]['description'][0])
# Provides to gidNumber incrementation
def ldap_gid_increment(self):
dn = "ou=gidNumber,ou=impianti,dc=ldap,dc=dem2m,dc=it"
old = {'description': str(self.actual_gid_number)}
new = {'description': str(self.actual_gid_number + 1)}
ldif = modlist.modifyModlist(old,new)
self.connector.modify_s(dn, ldif)
As i sad above, these methods are defined in a class of which i overrided constructor and destructor, in order to bind/unbind automatically to LDAP server when i instance or delete the instance.
Then, i used a query on LDAP to find the object called gidNumber (the ou i created before), and i filled a dictionary with resulting information. In the dictionary i found the variable representing the gidNumber and i used integer casting to manipulate it for incrementing. And that's all.
This procedure i really efficent because i server reboots you don't lose gidNumber information! Thank you again, jeemster.

Python: Assign an object's variable from a function (OpenERP)

I'm working on a OpenERP environment, but maybe my issue can be answered from a pure python perspective. What I'm trying to do is define a class whose "_columns" variable can be set from a function that returns the respective dictionary. So basically:
class repos_report(osv.osv):
_name = "repos.report"
_description = "Reposition"
_auto = False
def _get_dyna_cols(self):
ret = {}
cr = self.cr
cr.execute('Select ... From ...')
pass #<- Fill dictionary
return ret
_columns = _get_dyna_cols()
def init(self, cr):
pass #Other stuff here too, but I need to set my _columns before as per openerp
repos_report()
I have tried many ways, but these code reflects my basic need. When I execute my module for installation I get the following error.
TypeError: _get_dyna_cols() takes exactly 1 argument (0 given)
When defining the the _get_dyna_cols function I'm required to have self as first parameter (even before executing). Also, I need a reference to openerp's 'cr' cursor in order to query data to fill my _columns dictionary. So, how can I call this function so that it can be assigned to _columns? What parameter could I pass to this function?
From an OpenERP perspective, I guess I made my need quite clear. So any other approach suggested is also welcome.

From an OpenERP perspective, the right solution depends on what you're actually trying to do, and that's not quite clear from your description.
Usually the _columns definition of a model must be static, since it will be introspected by the ORM and (among other things) will result in the creation of corresponding database columns. You could set the _columns in the __init__ method (not init1) of your model, but that would not make much sense because the result must not change over time, (and it will only get called once when the model registry is initialized anyway).
Now there are a few exceptions to the "static columns" rules:
Function Fields
When you simply want to dynamically handle read/write operations on a virtual column, you can simply use a column of the fields.function type. It needs to emulate one of the other field types, but can do anything it wants with the data dynamically. Typical examples will store the data in other (real) columns after some pre-processing. There are hundreds of example in the official OpenERP modules.
Dynamic columns set
When you are developing a wizard model (a subclass of TransientModel, formerly osv_memory), you don't usually care about the database storage, and simply want to obtain some input from the user and take corresponding actions.
It is not uncommon in that case to need a completely dynamic set of columns, where the number and types of the columns may change every time the model is used. This can be achieved by overriding a few key API methods to simulate dynamic columns`:
fields_view_get is the API method that is called by the clients to obtain the definition of a view (form/tree/...) for the model.
fields_get is included in the result of fields_view_get but may be called separately, and returns a dict with the columns definition of the model.
search, read, write and create are called by the client in order to access and update record data, and should gracefully accept or return values for the columns that were defined in the result of fields_get
By overriding properly these methods, you can completely implement dynamic columns, but you will need to preserve the API behavior, and handle the persistence of the data (if any) yourself, in real static columns or in other models.
There are a few examples of such dynamic columns sets in the official addons, for example in the survey module that needs to simulate survey forms based on the definition of the survey campaign.
1 The init() method is only called when the model's module is installed or updated, in order to setup/update the database backend for this model. It relies on the _columns to do this.

When you write _columns = _get_dyna_cols() in the class body, that function call is made right there, in the class body, as Python is still parsing the class itself. At that point, your _get_dyn_cols method is just a function object in the local (class body) namespace - and it is called.
The error message you get is due to the missing self parameter, which is inserted only when you access your function as a method - but this error message is not what is wrong here: what is wrong is that you are making an imediate function call and expecting an special behavior, like late execution.
The way in Python to achieve what you want - i.e. to have the method called authomatically when the attribute colluns is accessed is to use the "property" built-in.
In this case, do just this: _columns = property(_get_dyna_cols) -
This will create a class attribute named "columns" which through a mechanism called "descriptor protocol" will call the desired method whenever the attribute is accessed from an instance.
To leran more about the property builtin, check the docs: http://docs.python.org/library/functions.html#property

Keeping an Audit Trail of any/all Python Database objects in GAE

I'm new to Python. I'm trying to figure out how to emulate an existing application I've coded using PHP and MS-SQL, and re-create the basic back-end functionality on the Google Apps Engine.
One of the things I'm trying to do is emulate the current activity on certain tables I have in MS-SQL, which is an Insert/Delete/Update trigger which inserts a copy of the current (pre-change) record into an audit table, and stamps it with a date and time. I'm then able to query this audit table at a later date to examine the history of changes that the record went through.
I've found the following code here on stackoverflow:
class HistoryEventFieldLevel(db.Model):
# parent, you don't have to define this
date = db.DateProperty()
model = db.StringProperty()
property = db.StringProperty() # Name of changed property
action = db.StringProperty( choices=(['insert', 'update', 'delete']) )
old = db.StringProperty() # Old value for field, empty on insert
new = db.StringProperty() # New value for field, empty on delete
However, I'm unsure how this code can be applied to all objects in my new database.
Should I create get() and put() functions for each of my objects, and then in the put() function I create a child object of this class, and set its particular properties?

This is certainly possible, albeit somewhat tricky. Here's a few tips to get you started:
Overriding the class's put() method isn't sufficient, since entities can also be stored by calling db.put(), which won't call any methods on the class being written.
You can get around this by monkeypatching the SDK to call pre/post call hooks, as documented in my blog post here.
Alternately, you can do this at a lower level by implementing RPC hooks, documented in another blog post here.
Storing the audit record as a child entity of the modified entity is a good idea, and means you can do it transactionally, though that would require further, more difficult changes.
You don't need a record per field. Entities have a natural serialization format, Protocol Buffers, and you can simply store the entity as an encoded Protocol Buffer in the audit record. If you're operating at the model level, use model_to_protobuf to convert a model into a Protocol Buffer.
All of the above are far more easily applied to storing the record after it's modified, rather than before it was changed. This shouldn't be an issue, though - if you need the record before it was modified, you can just go back one entry in the audit log.

I am bit out of touch of GAE and also no sdk with me to test it out, so here is some guidelines to given you a hint what you may do.
Create a metaclass AuditMeta which you set in any models you want audited
AuditMeta while creating a new model class should copy Class with new name with "_audit" appended and should also copy the attribute too, which becomes a bit tricky on GAE as attributes are itself descriptors
Add a put method to each such class and on put create a audit object for that class and save it, that way for each row in tableA you will have history in tableA_audit
e.g. a plain python example (without GAE)
import new
class AuditedModel(object):
def put(self):
print "saving",self,self.date
audit = self._audit_class()
audit.date = self.date
print "saving audit",audit,audit.date
class AuditMeta(type):
def __new__(self, name, baseclasses, _dict):
# create model class, dervied from AuditedModel
klass = type.__new__(self, name, (AuditedModel,)+baseclasses, _dict)
# create a audit class, copy of klass
# we need to copy attributes properly instead of just passing like this
auditKlass = new.classobj(name+"_audit", baseclasses, _dict)
klass._audit_class = auditKlass
return klass
class MyModel(object):
__metaclass__ = AuditMeta
date = "XXX"
# create object
a = MyModel()
a.put()
output:
saving <__main__.MyModel object at 0x957aaec> XXX
saving audit <__main__.MyModel_audit object at 0x957ab8c> XXX
Read audit trail code , only 200 lines, to see how they do it for django

Using Property Builtin with GAE Datastore's Model

I want to make attributes of GAE Model properties. The reason is for cases like to turn the value into uppercase before storing it. For a plain Python class, I would do something like:
Foo(db.Model):
def get_attr(self):
return self.something
def set_attr(self, value):
self.something = value.upper() if value != None else None
attr = property(get_attr, set_attr)
However, GAE Datastore have their own concept of Property class, I looked into the documentation and it seems that I could override get_value_for_datastore(model_instance) to achieve my goal. Nevertheless, I don't know what model_instance is and how to extract the corresponding field from it.
Is overriding GAE Property classes the right way to provides getter/setter-like functionality? If so, how to do it?
Added:
One potential issue of overriding get_value_for_datastore that I think of is it might not get called before the object was put into datastore. Hence getting the attribute before storing the object would yield an incorrect value.

Subclassing GAE's Property class is especially helpful if you want more than one "field" with similar behavior, in one or more models. Don't worry, get_value_for_datastore and make_value_from_datastore are going to get called, on any store and fetch respectively -- so if you need to do anything fancy (including but not limited to uppercasing a string, which isn't actually all that fancy;-), overriding these methods in your subclass is just fine.
Edit: let's see some example code (net of imports and main):
class MyStringProperty(db.StringProperty):
def get_value_for_datastore(self, model_instance):
vv = db.StringProperty.get_value_for_datastore(self, model_instance)
return vv.upper()
class MyModel(db.Model):
foo = MyStringProperty()
class MainHandler(webapp.RequestHandler):
def get(self):
my = MyModel(foo='Hello World')
k = my.put()
mm = MyModel.get(k)
s = mm.foo
self.response.out.write('The secret word is: %r' % s)
This shows you the string's been uppercased in the datastore -- but if you change the get call to a simple mm = my you'll see the in-memory instance wasn't affected.
But, a db.Property instance itself is a descriptor -- wrapping it into a built-in property (a completely different descriptor) will not work well with the datastore (for example, you can't write GQL queries based on field names that aren't really instances of db.Property but instances of property -- those fields are not in the datastore!).
So if you want to work with both the datastore and for instances of Model that have never actually been to the datastore and back, you'll have to choose two names for what's logically "the same" field -- one is the name of the attribute you'll use on in-memory model instances, and that one can be a built-in property; the other one is the name of the attribute that ends up in the datastore, and that one needs to be an instance of a db.Property subclass and it's this second name that you'll need to use in queries. Of course the methods underlying the first name need to read and write the second name, but you can't just "hide" the latter because that's the name that's going to be in the datastore, and so that's the name that will make sense to queries!

What you want is a DerivedProperty. The procedure for writing one is outlined in that post - it's similar to what Alex describes, but by overriding get instead of get_value_for_datastore, you avoid issues with needing to write to the datastore to update it. My aetycoon library has it and other useful properties included.