The documentation (https://cloud.google.com/appengine/docs/python/ndb/) states that
NDB uses Memcache as a cache service for "hot spots" in the data
I am now using memcache only as follows:
memcache.set(key=(id), value=params, time=0)
That expires (auto flushes) pretty often and so I would like to use NDB Datastore.
I thought I would have to always put the key-value in both NDB and Memcache, then check both.
Is this being done automatically by NDB?
Ie.
ancestor_key = ndb.Key("Book", guestbook_name or "*notitle*")
greetings = Greeting.query_book(ancestor_key).fetch(20)
Would that implicitly set Memcache ?
And when I read from NDB, would it implicitly try a memcache.get(key) first?
Thanks for your patience.
EDIT - What I tried:
As a test I tried something like this:
class Book(ndb.Model):
content = ndb.StringProperty()
class update(webapp2.RequestHandler):
def post(self):
p1='1'
p2='2'
p3='3'
p4='4'
p5='5'
id='test'
paramarray = (p1,p2,p3,p4,p5)
book = Book(name=id,value=paramarray)
# OR likes this - book = Book(ndb.Key(id),value=paramarray)
book.put()
Both versions error out.
Trying to get a key of the var id with the values of paramarray
EDIT 2 Daniel, Thank you for everything.
Have follow up formatting questions, will ask a new question.
Yes; see the full documentation on ndb caching. Basically, every write is cached both in a request-local in-context cache, and in the main memcached store; a get by key will look up in both caches first before falling back to the real datastore.
Edit I can't understand why you think your example would work. You defined a model with a content property, but then try to set name and value properties on it; naturally that will fail.
You should go through the ndb documentation, which gives a good introduction to using the model class.
Related
I have a datastore entity with several properties. Each property is updated using a separate method. However, every so often I find that either a method overwrites a property it is not modifying with an old value (Null).
For example.
class SomeModel(ndb.Model):
property1 = ndb.StringProperty()
property2 = ndb.StringProperty()
def method1(self, entity_key_urlsafe):
data1 = ndb.Key(urlsafe = entity_key_urlsafe).get()
data1.property1 = "1"
data1.put()
The data 1 entity now has property1 with value of "1"
def method2(self, entity_key_urlsafe):
data1 = ndb.Key(urlsafe = entity_key_urlsafe).get()
data1.property2 = "2"
data1.put()
The data 1 entity now has property2 with value of "2"
However, if these methods are run to closely in succession - method2 seems to overwrite property1 with its initial value (Null).
To get around this issue, I've been using the deferred library, however it's not reliable (deferred entities seem to disappear every now-and-then) or predictable (the _countdown time seems to be for guidance at best) enough.
My question is: Is there a way to only retrieve and modify one property of a datastore entity without overwriting the rest when you call data1.put()? I.e. In the case of method2 - could I only write to property2 without overwriting property1?
The way to prevent such overwrites, is to make sure your updates are done inside transactions. With NDB this is really easy - just attach the #ndb.transactional decorator to your methods:
#ndb.transactional
def method1(self, entity_key_urlsafe):
data1 = ndb.Key(urlsafe = entity_key_urlsafe).get()
data1.property1 = "1"
data1.put()
The documentation on transactions with NDB doesn't give as much background as the (older) DB version, so to familiarise yourself fully with the limitations and options, you should read both.
I say No
I have never seen a reference to that or a trick or a hack.
I also think that it would be quite difficult for such an operation to exist.
When you perform .put() on an entity the entity is serialised and then written.
An entity is an instance of the Class that you can save or retrieve from the Datastore.
Imagine if you had a date property that has auto_now? What would have to happen then? Which of the 2 saves should edit that property?
Though your problem seems to be different. One of your functions commits first and nullifies the other methods value because it retrieves an outdated copy, and not the expected one.
#Greg's Answer talks about transactions. You might want to take a look at them.
Transactions are used for concurrent requests and not that much for succession.
Imagine that 2 users pressing the save button to increase a counter at the same time. There transactions work.
#ndb.transactional
def increase_counter(entity_key_urlsafe):
entity = ndb.Key(urlsafe = entity_key_urlsafe).get()
entity.counter += 1
entity.put()
Transactions will ensure that the counter is correct.
The first that tries to commit the above transaction will succeed and the later will have to retry if retries are on (3 by default).
Though succession is something different. Said that, I and #Greg advise you to change your logic towards using transaction if the problem you want to solve is something like the counter example.
I'm having some trouble understanding how entities and keys work in Google App Engine NDB.
I have a post entity and a user entity. How do I set the user_key on post to user?
In the interactive console, I have this so far:
from google.appengine.ext import ndb
from app.lib.posts import Post
from app.lib.users import User
from random import shuffle
users = User.query()
posts = Post.query().fetch()
for post in posts:
post.user_key = shuffle(users)[0]
post.put()
I'm just trying to set up some seed data for development. I know this probably isn't the ideal way to set things, but my first question is:
How do I get a key from an entity (the reverse is described in the docs)
How do I set associations in ndb?
try:
post.user_key = shuffle(users)[0].key
Maybe this helps to understand the NDB. I had the same questions with you.
class Person(ndb.Expando):
pass
class Favourite(ndb.Expando):
pass
class Picture(ndb.Expando):
pass
person = Person()
person.put()
picture = Picture()
picture.put()
fav = Favourite(parent=person.key,
person=person.key,
picture=picture.key
)
fav.put()
Verify that shuffle works in this case, as User.query() returns an iter, not a list. (You can convert it to a list using shuffle( [ x for x in users ] ). Beware, this list could be looong.
NDB has some really wired behavior sometimes, so id recommend you dont store an NDB-Key, but its serialized string, which is also compatible to ext.db: post.user_key = shuffle( [ x for x in users ] ).key.urlsafe()
You could use KeyProperty for associations. If you need a more fine-graned control over your relations, you must implement them yourself.
See https://developers.google.com/appengine/docs/python/ndb/properties#structured
I believe this is trival but fairly new to Python.
I am trying to create a model using google app engine.
Basically from a E/R point of view
I have 2 objects with a join table (the join table captures the point in time of the join)
Something like this
Person | Idea | Person_Idea
-------------------------------
person.key idea.key person.key
idea.key
date_of_idea
my Python code would look like
class Person (db.Model):
#some properties here....
class Idea(db.Model):
#some properties here....
class IdeaCreated(db.Model):
person= db.ReferenceProperty(Person)
idea= db.ReferenceProperty(Idea)
created = db.DateTimeProperty(auto_now_add = True)
What I want to be able to do is have a convient way to get all ideas a person has (bypass idea created objects) -sometimes I will need the list of ideas directly.
The only way I can think to do this is to add the follow method on the User class
def allIdeas(self):
ideas = []
for ideacreated in self.ideacreated_set:
ideas.append(ideacreated.idea)
return ideas
Is this the only way to do this? I is there a nicer way that I am missing?
Also assuming I could have a GQL and bypass hydrating the ideaCreated instances (not sure the exact syntax) but putting a GQL query smells wrong to me.
you should use the person as an ancestor/parent for the idea.
idea = Idea(parent=some_person, other_field=field_value).put()
then you can query all ideas where some_person is the ancestor
persons_ideas = Idea.all().ancestor(some_person_key).fetch(1000)
the ancestor key will be included in the Idea entities key and you won't be able to change that the ancestor once the entity is created.
i highly suggest you to use ndb instead of db https://developers.google.com/appengine/docs/python/ndb/
with ndb you could even use StructuredProperty or LocalStructuredProperty
https://developers.google.com/appengine/docs/python/ndb/properties#structured
EDIT:
if you need a many to many relationship look in to ListProperties and store the Persons keys in that property. then you can query for all Ideas with that Key in that property.
class Idea(db.Model):
person = db.StringListProperty()
idea = Idea(person = [str(person.key())], ....).put()
add another person to the idea
idea.person.append(str(another_person.key())).put()
ideas = Idea.filter(person=str(person.key())).fetch(1000)
look into https://developers.google.com/appengine/docs/python/datastore/typesandpropertyclasses#ListProperty
I'm trying to do:
MyModel({'text': db.Text('text longer than 500 byets')})
But get:
BadValueError: Indexed value fb_education must be at most 500 bytes
I'm thinking this is just a carry over from this issue with the old db api.
https://groups.google.com/forum/?fromgroups#!topic/google-appengine/wLAwrjtsuks
First create entity dynamically :
kindOfEntity = "MyTable"
class DynamicEntity(ndb.Expando):
#classmethod
def _get_kind(cls):
return kindOfEntity
then after to assign Text Properties run time/dynamically as shown below
dbObject = DynamicEntity()
key = "studentName"
value = "Vijay Kumbhani"
textProperties = ndb.TextProperty(key)
dbObject._properties[key] = {}
dbObject._values[key] = {}
dbObject._properties[key] = textProperties
dbObject._values[key] = value
dbObject.put()
then after key properties assign with Text properties
You're trying to use a db.Text, part of the old API, with NDB, which isn't going to work.
To the best of my knowledge, there's no good way to set unindexed properties in an Expando in NDB, currently. You can set _default_indexed = False on your expando subclass, as (briefly) documented here, but that will make the default for all expando properties unindexed.
A better solution would be to avoid the use of Expando alltogether; there are relatively few compelling uses for it where you wouldn't be better served by defining a model (or even defining one dynamically).
Yeah, I know question is old. But I also googled for same solutions and not found any result.
So here receipt that works for me (I expand User() with "permissions" property):
prop = ndb.GenericProperty("permissions", indexed=False)
prop._code_name = "permissions"
user._properties["permissions"] = prop
prop._set_value(user, permissions)
The previous answer was VERY use to me... Thanks!!! I just wanted to add that it appears you can also create a specific property type using this technique (if you know the datatype you want to create). When the entity is later retrieved, the dynamic property is set to the specific type instead of GenericProperty. This can be handy for ndb.PickleProperty and ndb.JsonProperty values in particular (to get the in/out conversions).
prop = ndb.TextProperty("permissions", indexed=False)
prop._code_name = "permissions"
user._properties["permissions"] = prop
prop._set_value(user, permissions)
I was trying to just change one property of an entity to Text. But when you don't map your properties explicitly, Expando/Model seem to change all properties of an entity to GenericProperty (after get).
When you put those entities again (to change the desired property), it affects other existing TextProperties, changing then to regular strings.
Only the low-level datastore api seems to work:
https://gist.github.com/feroult/75b9ab32b463fe7f9e8a
You can call this from the remote_api_shell.py:
from yawp import *
yawp(kind).migrate(20, 'change_property_to_text', 'property_name')
I defined a simple class in GAE for keeping user profiles data like this:
class User(db.Model):
email = db.EmailProperty()
role = db.StringProperty(default=roles.USER)
first_name = db.StringProperty()
last_name = db.StringProperty()
...
I use memcache to keep session information. memcache data looks like this { 'key': 'agpjYW5kaXJhdGVzcgoLEgRVc2VyGCMM'}. I get session_id value from the cookie. When I try to get user info linked to that cookie like this:
session_id = request['session_id']
data = memcache.get(session_id)
user = User.get(data['key'])
I get KindError exception:
KindError: Kind 'User' is not a subclass of kind 'User'
I know this user exists, memcache exists. User class is defined only once in my project. Why this error occurs and how can I make it work?
UPDATE: I tried to use db.get() instead of User.get() and it worked. So, what's the problem there can be?
Model.get() does check whether the supplied key is of the correct kind, as defined in the documentation. If not of the correct kind it will throw a KindError.
db.get() does not do any type checking and therefore will succeed with the supplied value if it exists in the data store, but will not necessarily return a User entity.
So you need to check whether the key in your memcache is actually of the User kind. Are you sure it's not overwritten with the key of a different model at some point?
The App Engine framework defines a class called 'User' as part of the Users API. In addition, you have your own class by the same name. When the exception occurs, you're trying to use one, but getting the other.
To avoid this, rename your model. You should also be careful how you import modules in Python. Instead of:
from google.appengine.api.users import User
or worse:
from google.appengine.api.users import *
you should use:
from google.appengine.api import users
And then refer to users.User, which is unambiguous.
The problem, it seems to me, is more subtle than that. I was getting the error with this call to Model.get() (I'm retrieving a top-level singleton object, always there):
datastore = GDSDatastore.get(gds.Key.from_path(*path))
so I investigated with this code:
datastore = gds.get(gds.Key.from_path(*path))
if not(datastore is None or isinstance(datastore, GDSDatastore)):
logger.error("KindError isinstance(GDSDatastore)=%s class=%s" % (isinstance(datastore, GDSDatastore), datastore.__class__.__name__))
raise gds.KindError('Kind %r is not a GDSDatastore instance' %
(datastore.kind()))
The vast majority of the time I get no error, but today I got this interesting log:
KindError isinstance(GDSDatastore)=False class=GDSDatastore
Now, that strikes me as rather peculiar.
(Note: GDSDatastore is defined locally: class GDSDatastore(gds.Model))