My application experiences over quota issues and I would like to handle such cases properly in my code. The limit is reached just for Datastore Read Operations, but I get TypeError: ConjunctionNode() requires at least one node exception when try to read the data from the memcache - entries = memcache.get('mykey').
Why this exception happens?
Exception details:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
result = handler(dict(self._environ), self._StartResponse)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~myapp/1.373233284460557570/myapp.py", line 595, in get
entries = memcache.get('mykey')
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 559, in get
results = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 624, in __get_hook
self._do_unpickle)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 271, in _decode_value
return do_unpickle(value)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 401, in _do_unpickle
return unpickler.load()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 550, in __new__
raise TypeError('ConjunctionNode() requires at least one node.')
TypeError: ConjunctionNode() requires at least one node.
Upd. here is what I save to memcache:
entries = MyModel.query()
entries = entries.fetch(keys_only=True)
entries = random.sample(entries, 10)
entries = [list_key.get() for list_key in entries]
memcache.set('mykey', entries, 60*60*24)
Upd2. Online memcache viewer shows the following value stored (just first part is shown below):
Type: Object
..cgoogle.appengine.ext.ndb.query.Query.q.).q.}q.(U._Query__projectionq.NU._Query__filtersq.cgoogle.appengine.ext.ndb.query.ConjunctionNode.q.).q.}q.U._ConjunctionNode__nodesq.]q.(cgoogle.appengine.ext.ndb.query.FilterNode.q.).q.}q.(U._Filt
I believe that this happens simply because you are trying to put a pickled Query object into memcache, but Query objects are not pickle-able.
What I do instead in this case is to convert the Query to a list prior to putting it into the cache. If your code doesn't care about the distinction (that is, if it's not calling .filter() or something like that) then it'll probably work fine for you too.
Related
I am new to app engine and trying to deploy a python app. The app works correctly when I preview it on my local host but when I deploy it I get the following error: "Internal Server Error
The server has either erred or is incapable of performing the requested operation."
And traceback:
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~ecd-signup/20160620t104611.393644814392119591/hpvball.py", line 78, in get
entries_c = qry_c.fetch(100)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py", line 160, in positional_wrapper
return wrapped(*args, **kwds)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1203, in fetch
return self.fetch_async(limit, **q_options).get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 980, in _run_to_list
batch = yield rpc
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
result = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2921, in __query_result_hook
yaml_index=yaml, xml_index=xml)
NeedIndexError: no matching index found. recommended index is:
- kind: Entry
ancestor: yes
properties:
- name: committed
- name: date
The suggested index for this query is:
- kind: Entry
ancestor: yes
properties:
- name: committed
- name: date
If this is a new/recent deployment of the app it's possible for the newly added/modified indexes to be still building, the condition is transitory and will eventually go away. See this answer: https://stackoverflow.com/a/29807764/4495081.
It's possible in some cases that the suggested index is not automatically generated by the local development server. Check your local index file for the existance of the suggested index. If missing add it manually.
It's also possible in some cases (some multi-module apps, for example) that the app deployment doesn't automatically upload the local app index file. If the indexes displayed in the developer console don't match those you see in the local index file you should deploy them explicitly, using the update_indexes appcfg.py option.
I'm trying to insert an instance of a model into an ndb database.
It keeps on giving me the "Model is not immutable" error.
I have tried with different model names, but still the same error.
class User(ndb.Model):
username = ndb.StringProperty()
email = ndb.StringProperty()
lwr_username = ndb.ComputedProperty(lambda self: self.username.lower())
lwr_email = ndb.ComputedProperty(lambda self: self.email.lower())
Here's my insert code:
entity = User()
entity.email = ""
entity.username = "bob"
logging.info(entity)
#Commit data asynchronously
entities = [entity]
futures = ndb.put_multi_async(entities)
#Build Response whilst database is committing
response = {
}
#Wait for commits to finish
ndb.Future.wait_all(entities)
This is the full stack trace
Model is not immutable
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~myapp-api/1.377037445874907069/v1/handler/userHandler.py", line 11, in post
responseCode, response = UserService.create(locale, json.loads(self.request.body), **kwargs)
File "/base/data/home/apps/s~myapp-api/1.377037445874907069/v1/service/userService.py", line 37, in create
ndb.Future.wait_all(entities)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 345, in wait_all
waiting_on = set(futures)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3017, in __hash__
raise TypeError('Model is not immutable')
TypeError: Model is not immutable
You need to use the futures list, not entities to wait for the async process to complete:
ndb.Future.wait_all(futures)
The wait_all() function stores these in a set() object, and sets require the contents to be hashable. Since mutable objects cannot be stored in sets or dictionaries, Google engineers added an explicit TypeError to the Model.__hash__ method, which is what you see raised here.
Using app engine's search API, I tried to search the queries which contain ,, =, (, etc.
It returns the following error:
Failed to parse query "engines (Modular)"
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~generatestock/12.362076640167792770/search.py", line 1641, in get
result = find_search_document(search_item)
File "/base/data/home/apps/s~generatestock/12.362076640167792770/search.py", line 177, in find_search_document
return search.Index(name=_INDEX_NAME).search(query)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 2715, in search
query = Query(query_string=query)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 2286, in __init__
_CheckQuery(self._query_string)
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/api/search/search.py", line 1964, in _CheckQuery
raise QueryError('Failed to parse query "%s"' % query)
QueryError: Failed to parse query "Engines (Modular)"
Why? Does the search API support these characters?
To parse a phrase, use quotes around the phrase:
query = '"Engines (Modular)"'
search.Index(name=_INDEX_NAME).search(query)
You can simply put quotes around an existing query:
query = '"{0}"'.format(query)
search.Index(name=_INDEX_NAME).search(query)
but you probably have to remove existing quotes in the query to make this work. Google's documentation is silent on how to include quotes in your search queries. So the complete, fail-safe method would be:
query = '"{0}"'.format(query.replace('"', ''))
search.Index(name=_INDEX_NAME).search(query)
I'm trying to implement the code that Guido posted as an answer to the question Avoiding Memcache 1M limit of values. Everything seems(?) to be working when I first load the page and add values to memcache, but when I re-load it and retrieve values from the memcache I get a strange error:
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 194, in Handle
for chunk in result:
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1036, in appstats_wsgi_wrapper
result = app(environ, appstats_start_response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1519, in __call__
response = self._internal_error(e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/appengineurlhere/mypythoncode.py", line 87, in get
entitylist = retrieve("entitylist")
File "/appengineurlhere/mypythoncode.py", line 53, in retrieve
return pickle.loads(serialized)
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 1382, in loads
return Unpickler(file).load()
File "/base/python27_runtime/python27_dist/lib/python2.7/pickle.py", line 858, in load
dispatch[key](self)
KeyError: ':'
Here's the store/retrieve code:
def store(key, value, chunksize=750000):
serialized = pickle.dumps(value, 2)
values = {}
for i in xrange(0, len(serialized), chunksize):
values['%s.%s' % (key, i//chunksize)] = serialized[i : i+chunksize]
memcache.set_multi(values)
def retrieve(key):
result = memcache.get_multi(['%s.%s' % (key, i) for i in xrange(32)])
serialized = ''.join([v for v in result.values() if v is not None])
return pickle.loads(serialized)
And this is how I'm using it:
try:
entitylist = retrieve("entitylist")
except EOFError:
entitylist = MyModel.all().fetch(None)
store("entitylist", entitylist)
Other oddities:
I don't see this on my development server; just the production App Engine site. (Though the data on my development server is slightly different; I believe it all fits into the standard 1MB memcache size, where the production data is larger.)
When I search for "entitylist" on both my development and production admin panel, App Engine tells me "no such key". Yet based on the total size of the memcache shown (this is the only place I've implemented memcache), it absolutely appears that something is getting cached.
Can anyone please point me in the direction of something I should look at or fix?
Is it possible that you have some old instances of MyModel that don't match with the current definition of MyModel? You could get errors while pickling if there are unrecognized or missing attributes.
I'm new to Python (as well as GAE), I'm trying to include gaesessions in my application, this is exactly what I did:
I copied the gaesessions folder to my src folder (the folder includes __init__.py file)
and added this to my working code:
from gaesessions import get_current_session
session = get_current_session()
I received the following error:
Traceback (most recent call last): File "C:\Program
Files\Google\google_appengine\google\appengine\runtime\wsgi.py", line
174, in Handle
result = handler(self._environ, self._StartResponse)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1519, in
__call__
response = self._internal_error(e)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1511, in
__call__
rv = self.handle_exception(request, response, e)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1505, in
__call__
rv = self.router.dispatch(request, response)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1253, in
default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1077, in
__call__
return handler.dispatch()
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 547, in
dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program
Files\Google\google_appengine\lib\webapp2\webapp2.py", line 545, in
dispatch
return method(*args, **kwargs)
File "D:....\src\helloworld.py", line 13, in get
session = get_current_session()
File "D:....\src\gaesessions\__init__.py", line 36, in
get_current_session
return _tls.current_session
File "C:\Python27\lib\_threading_local.py", line 193, in
__getattribute__
return object.__getattribute__(self, name)
You've omitted the actual exception from your stack trace, but I'm guessing it's this:
AttributeError: 'local' object has no attribute 'current_session'
If so, you've most likely skipped the middleware configuration step. Create an appengine_config.py per the instructions and restart the dev server; this should resolve your error.