I'm trying to implement the code that Guido posted as an answer to the question Avoiding Memcache 1M limit of values. Everything seems(?) to be working when I first load the page and add values to memcache, but when I re-load it and retrieve values from the memcache I get a strange error:
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 194, in Handle
for chunk in result:
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1036, in appstats_wsgi_wrapper
result = app(environ, appstats_start_response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/appengineurlhere/mypythoncode.py", line 87, in get
entitylist = retrieve("entitylist")
File "/appengineurlhere/mypythoncode.py", line 53, in retrieve
return pickle.loads(serialized)
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
KeyError: ':'
Here's the store/retrieve code:
def store(key, value, chunksize=750000):
serialized = pickle.dumps(value, 2)
values = {}
for i in xrange(0, len(serialized), chunksize):
values['%s.%s' % (key, i//chunksize)] = serialized[i : i+chunksize]
memcache.set_multi(values)
def retrieve(key):
result = memcache.get_multi(['%s.%s' % (key, i) for i in xrange(32)])
serialized = ''.join([v for v in result.values() if v is not None])
return pickle.loads(serialized)
And this is how I'm using it:
try:
entitylist = retrieve("entitylist")
except EOFError:
entitylist = MyModel.all().fetch(None)
store("entitylist", entitylist)
Other oddities:
I don't see this on my development server; just the production App Engine site. (Though the data on my development server is slightly different; I believe it all fits into the standard 1MB memcache size, where the production data is larger.)
When I search for "entitylist" on both my development and production admin panel, App Engine tells me "no such key". Yet based on the total size of the memcache shown (this is the only place I've implemented memcache), it absolutely appears that something is getting cached.
Can anyone please point me in the direction of something I should look at or fix?
Is it possible that you have some old instances of MyModel that don't match with the current definition of MyModel? You could get errors while pickling if there are unrecognized or missing attributes.
Related
I am new to app engine and trying to deploy a python app. The app works correctly when I preview it on my local host but when I deploy it I get the following error: "Internal Server Error
The server has either erred or is incapable of performing the requested operation."
And traceback:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~ecd-signup/20160620t104611.393644814392119591/hpvball.py", line 78, in get
entries_c = qry_c.fetch(100)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py", line 160, in positional_wrapper
return wrapped(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1203, in fetch
return self.fetch_async(limit, **q_options).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 980, in _run_to_list
batch = yield rpc
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
result = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2921, in __query_result_hook
yaml_index=yaml, xml_index=xml)
NeedIndexError: no matching index found. recommended index is:
- kind: Entry
ancestor: yes
properties:
- name: committed
- name: date
The suggested index for this query is:
- kind: Entry
ancestor: yes
properties:
- name: committed
- name: date
If this is a new/recent deployment of the app it's possible for the newly added/modified indexes to be still building, the condition is transitory and will eventually go away. See this answer: https://stackoverflow.com/a/29807764/4495081.
It's possible in some cases that the suggested index is not automatically generated by the local development server. Check your local index file for the existance of the suggested index. If missing add it manually.
It's also possible in some cases (some multi-module apps, for example) that the app deployment doesn't automatically upload the local app index file. If the indexes displayed in the developer console don't match those you see in the local index file you should deploy them explicitly, using the update_indexes appcfg.py option.
I'm trying to insert an instance of a model into an ndb database.
It keeps on giving me the "Model is not immutable" error.
I have tried with different model names, but still the same error.
class User(ndb.Model):
username = ndb.StringProperty()
email = ndb.StringProperty()
lwr_username = ndb.ComputedProperty(lambda self: self.username.lower())
lwr_email = ndb.ComputedProperty(lambda self: self.email.lower())
Here's my insert code:
entity = User()
entity.email = ""
entity.username = "bob"
logging.info(entity)
#Commit data asynchronously
entities = [entity]
futures = ndb.put_multi_async(entities)
#Build Response whilst database is committing
response = {
}
#Wait for commits to finish
ndb.Future.wait_all(entities)
This is the full stack trace
Model is not immutable
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~myapp-api/1.377037445874907069/v1/handler/userHandler.py", line 11, in post
responseCode, response = UserService.create(locale, json.loads(self.request.body), **kwargs)
File "/base/data/home/apps/s~myapp-api/1.377037445874907069/v1/service/userService.py", line 37, in create
ndb.Future.wait_all(entities)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 345, in wait_all
waiting_on = set(futures)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3017, in __hash__
raise TypeError('Model is not immutable')
TypeError: Model is not immutable
You need to use the futures list, not entities to wait for the async process to complete:
ndb.Future.wait_all(futures)
The wait_all() function stores these in a set() object, and sets require the contents to be hashable. Since mutable objects cannot be stored in sets or dictionaries, Google engineers added an explicit TypeError to the Model.__hash__ method, which is what you see raised here.
My application experiences over quota issues and I would like to handle such cases properly in my code. The limit is reached just for Datastore Read Operations, but I get TypeError: ConjunctionNode() requires at least one node exception when try to read the data from the memcache - entries = memcache.get('mykey').
Why this exception happens?
Exception details:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~myapp/1.373233284460557570/myapp.py", line 595, in get
entries = memcache.get('mykey')
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 559, in get
results = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 624, in __get_hook
self._do_unpickle)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 271, in _decode_value
return do_unpickle(value)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 401, in _do_unpickle
return unpickler.load()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 550, in __new__
raise TypeError('ConjunctionNode() requires at least one node.')
TypeError: ConjunctionNode() requires at least one node.
Upd. here is what I save to memcache:
entries = MyModel.query()
entries = entries.fetch(keys_only=True)
entries = random.sample(entries, 10)
entries = [list_key.get() for list_key in entries]
memcache.set('mykey', entries, 60*60*24)
Upd2. Online memcache viewer shows the following value stored (just first part is shown below):
Type: Object
..cgoogle.appengine.ext.ndb.query.Query.q.).q.}q.(U._Query__projectionq.NU._Query__filtersq.cgoogle.appengine.ext.ndb.query.ConjunctionNode.q.).q.}q.U._ConjunctionNode__nodesq.]q.(cgoogle.appengine.ext.ndb.query.FilterNode.q.).q.}q.(U._Filt
I believe that this happens simply because you are trying to put a pickled Query object into memcache, but Query objects are not pickle-able.
What I do instead in this case is to convert the Query to a list prior to putting it into the cache. If your code doesn't care about the distinction (that is, if it's not calling .filter() or something like that) then it'll probably work fine for you too.
I have a simple Python program that uses GAE, and I am using GAE's memcache module to make sure that I do not screw up when updating the cache:
from google.appengine.api import memcache
class NewPost(BlogHandler):
def get(self):
self.render("newpost.html")
def post(self):
#update cache for the front page
val, unique = memcache.gets(FRONT_PAGE_KEY)
for p in val:
logging.warning(p)
Now this code should run without problem, but instead when I use the method post, it blows up:
Traceback (most recent call last):
File "/home/pedro/google_appengine/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/home/pedro/google_appengine/lib/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/home/pedro/google_appengine/hw6/blog.py", line 172, in post
val, unique = memcache.gets(FRONT_PAGE_KEY)
AttributeError: 'module' object has no attribute 'gets'
INFO 2014-05-08 13:36:49,525 module.py:639] default: "POST /blog/newpost HTTP/1.1" 500 -
This makes no sense at all, specially because I know that memcache has a method called gets(key):
https://developers.google.com/appengine/docs/python/memcache/
Based on my research on stackoverflow I found this discussion:
App Engine Python: AttributeError: 'module' object has no attribute 'Stock'
And so I have flushed my cache and deleted all my local DB's content, but I am still getting the error.
What am I doing wrong?
You're mistaken: the memcache module does not have a gets function. See the documentation.
gets is a method of the memcache Client object: again, see the docs.
Using app engine's search API, I tried to search the queries which contain ,, =, (, etc.
It returns the following error:
Failed to parse query "engines (Modular)"
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~generatestock/12.362076640167792770/search.py", line 1641, in get
result = find_search_document(search_item)
File "/base/data/home/apps/s~generatestock/12.362076640167792770/search.py", line 177, in find_search_document
return search.Index(name=_INDEX_NAME).search(query)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 2715, in search
query = Query(query_string=query)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 2286, in __init__
_CheckQuery(self._query_string)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 1964, in _CheckQuery
raise QueryError('Failed to parse query "%s"' % query)
QueryError: Failed to parse query "Engines (Modular)"
Why? Does the search API support these characters?
To parse a phrase, use quotes around the phrase:
query = '"Engines (Modular)"'
search.Index(name=_INDEX_NAME).search(query)
You can simply put quotes around an existing query:
query = '"{0}"'.format(query)
search.Index(name=_INDEX_NAME).search(query)
but you probably have to remove existing quotes in the query to make this work. Google's documentation is silent on how to include quotes in your search queries. So the complete, fail-safe method would be:
query = '"{0}"'.format(query.replace('"', ''))
search.Index(name=_INDEX_NAME).search(query)