I want my deferred task to only try once more after failing.
After reading this related question: Specifying retry limit for tasks queued using GAE deferred library it was obvious that I needed to follow the accepted answer, so I modified my code to look like this:
from google.appengine.ext import deferred
deferred.defer(MyFunction, DATA, _retry_options={'task_retry_limit': 1})
Now I get this error:
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/ext/deferred/deferred.py", line 269, in defer
return task.add(queue, transactional=transactional)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1143, in add
return self.add_async(queue_name, transactional).get_result()
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1139, in add_async
return Queue(queue_name).add_async(self, transactional, rpc)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 1889, in add_async
rpc)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2008, in __AddTasks
fill_request(task, request.add_add_request(), transactional)
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2093, in __FillAddPushTasksRequest
task.retry_options, task_request.mutable_retry_parameters())
File "/usr/local/google-cloud-sdk/platform/google_appengine/google/appengine/api/taskqueue/taskqueue.py", line 2033, in __FillTaskQueueRetryParameters
if retry_options.min_backoff_seconds is not None:
AttributeError: 'dict' object has no attribute 'min_backoff_seconds'
Obviously im making a silly mistake, I just can't figure out what it is.
You need to pass a TaskRetryOptions instance, not just a dict:
from google.appengine.ext import deferred
from google.appengine.api.taskqueue import TaskRetryOptions
options = TaskRetryOptions(task_retry_limit=1)
deferred.defer(MyFunction, DATA, _retry_options=options)
Related
I am getting error like this below in the 500 internal server error page.
File "/usr/local/lib/python3.5/dist-packages/pyDatalog/pyParser.py", line 388, in __call__
literal = Literal.make(self._pyD_name, tuple(args), kwargs)
File "/usr/local/lib/python3.5/dist-packages/pyDatalog/pyParser.py", line 510, in make
return precalculations & Query(predicate_name, terms, kwargs, prearity, aggregate)
File "/usr/local/lib/python3.5/dist-packages/pyDatalog/pyParser.py", line 574, in __init__
Literal.__init__(self, predicate_name, terms, kwargs, prearity, aggregate)
File "/usr/local/lib/python3.5/dist-packages/pyDatalog/pyParser.py", line 500, in __init__
self.lua = pyEngine.Literal(self.predicate_name, tbl, self.prearity, aggregate)
File "pyDatalog\pyEngine.py", line 402, in pyDatalog.pyEngine.Literal.__init__ (pyDatalog/pyEngine.c:15254)
File "pyDatalog\pyEngine.py", line 333, in pyDatalog.pyEngine.Pred.__new__ (pyDatalog/pyEngine.c:13917)
File "pyDatalog\pyEngine.py", line 334, in pyDatalog.pyEngine.Pred.__new__ (pyDatalog/pyEngine.c:13431)
AttributeError: '_thread._local' object has no attribute 'logic'
How can access the PyDataLog variables
I have found the answer here in the Thread safety and multi-models section. If it can help anyone else facing what I faced.
A Python program may start several threads. Each thread should have these statements to initialize pyDatalog :
from pyDatalog import pyDatalog, Logic
Logic() # initializes the pyDatalog engine
I am using the orientdb and gremlin server in python, Gremlin server is started successfully, but when I am trying to add one vertex to the orientdb through gremlin code it's giving me an error.
query = """graph.addVertex(label, "Test", "title", "abc", "title", "abc")"""
following is the Traceback
/usr/bin/python3.6 /home/admin-12/Documents/bitbucket/ecodrone/ecodrone/test/test1.py
Traceback (most recent call last):
File "/home/admin-12/Documents/bitbucket/ecodrone/ecodrone/test/test1.py", line 27, in <module>
result = execute_query("""graph.addVertex(label, "Test", "title", "abc", "title", "abc")""")
File "/home/admin-12/Documents/bitbucket/ecodrone/ecodrone/GremlinConnector.py", line 21, in execute_query
results = future_results.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/admin-12/.local/lib/python3.6/site-packages/gremlin_python/driver/resultset.py", line 81, in cb
f.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/admin-12/.local/lib/python3.6/site-packages/gremlin_python/driver/connection.py", line 77, in _receive
self._protocol.data_received(data, self._results)
File "/home/admin-12/.local/lib/python3.6/site-packages/gremlin_python/driver/protocol.py", line 106, in data_received
"{0}: {1}".format(status_code, data["status"]["message"]))
gremlin_python.driver.protocol.GremlinServerError: 599: Error during serialization: Infinite recursion (StackOverflowError) (through reference chain: com.orientechnologies.orient.core.id.ORecordId["record"]->com.orientechnologies.orient.core.record.impl.ODocument["schemaClass"]->com.orientechnologies.orient.core.metadata.schema.OClassImpl["document"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"]->com.orientechnologies.orient.core.record.impl.ODocument["owners"])
Process finished with exit code 1
First of all, I very much recommend that you do not use the Graph API to make mutation. Prefer the Traversal API for that and do:
g.addV('Test').
property('title1', 'abc').
property('title2', 'abc')
Second, I think that your error is occurring because you are returning a Vertex which contains an ORecordId which is the vertex identifier and Gremlin Server doesn't know how to handle that. I don't know if OrientDB has serializers built to handle that, but if they do then you would want to add them to Gremlin Server configurations which is described in a bit more detail here - basically, you would want to know if OrientDB exposes an TinkerPop IORegistry for all their custom classes that might be sent back over the wire.
If they do not, then you would want to avoid returning those or convert them yourself. TinkerPop already recommends that you not return full Vertex objects and only return data that you need. So, rather than g.V() you would want to convert that Vertex into a Map with g.V().valueMap('title') or something similar (perhaps use project() step). If you definitely need the vertex identifier then you would need to convert that to something TinkerPop serializers understand. That might mean something as simple as:
g.V().has("title1","abc").id().next().toString()
Over the last 48 hours or so my small python GAE app has started getting AssertionErrors from ndb.get_multi calls.
Full traceback is appended, and the errors are being generated on the production server in _BaseValue's __init__ on line 734 of /base/data/.../ndb/model.py, and the failing assertion is b_val is not None with message "Cannot wrap None"
The error doesn't appear to be related to a particular entity or entities, but I've only seen it with one entity type so far (yet to test others).
The get_multi call is for only up to a dozen keys, and the error is intermittent so that repeating it will sometimes succeed. Or not...
I'm not seeing this error via remote shell, but I note that my local install is 1.9.23 while the log entry says the production server is 1.9.25 (GoogleAppEngineLauncher says my local install is up to date)
I'm adding a workaround to catch the exception and iterate through the keys to get them individually but I'm still seeing an upstream warning about a "suspended generator get" on line 744 of context.py.
The warning appears on the first get of this entity type from the list, for at least 2 different lists of keys (as well as preceding the AssertionError).
I don't want to have to wrap all get_multi calls in this way.
What's going on?
TRACEBACK:
Cannot wrap None
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~thegapnetball/115.386356111937586421/handlers/assess.py", line 50, in get
rs = ndb.get_multi(t.players)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3905, in get_multi
for future in get_multi_async(keys, **ctx_options)]
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 326, in get_result
self.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 372, in _help_tasklet_along
value = gen.send(val)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 751, in get
pbs = entity._to_pb(set_key=False).SerializePartialToString()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3147, in _to_pb
prop._serialize(self, pb, projection=self._projection)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 2379, in _serialize
projection=projection)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1405, in _serialize
values = self._get_base_value_unwrapped_as_list(entity)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1175, in _get_base_value_unwrapped_as_list
wrapped = self._get_base_value(entity)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1163, in _get_base_value
return self._apply_to_values(entity, self._opt_call_to_base_type)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1335, in _apply_to_values
value[:] = map(function, value)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 1217, in _opt_call_to_base_type
value = _BaseValue(self._call_to_base_type(value))
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 734, in \__init__
assert b_val is not None, "Cannot wrap None"
AssertionError: Cannot wrap None
Tim Hoffman and Patrick Costello put me on the right track to solve this.
I incremented version to protect some changes but took longer to finish than I expected.
One change added a repeated StructuredProperty to a model derived from ndb.Model, and I put several entities with the extra property (about 30 out of 1100 total).
The previous version without the extra property was still the default and was being lightly used, so the entities became just inconsistent enough to produce the intermittent AssertionError.
The main lesson is to take note of the recommendations in Google's schema update article, particularly changing the underlying parent to Expando and/or disabling datastore edits until any migration is complete.
https://cloud.google.com/appengine/articles/update_schema
The fix was to add the property to the previous version, get all the entities and then put them.
Thanks Tim and Patrick for the pointer!
I'm just getting started with python so please bear with me ;)
While following a basic tutorial I ran into a problem while opening a file, here's the traceback:
File "/home/nick/Dropbox/workspace/pytest/schlange.py", line 55, in <module>
f=open("file.csv","r")
File "/usr/lib/python2.6/aifc.py", line 922, in open
return Aifc_read(f)
File "/usr/lib/python2.6/aifc.py", line 335, in __init__
self.initfp(f)
File "/usr/lib/python2.6/aifc.py", line 288, in initfp
raise Error, 'file does not start with FORM id'
aifc.Error: file does not start with FORM id
Does that mean that 'aifc.py' in my python installation is broken or did I miss something important here?
You've decided to import * from aifc, and its open() has shadowed the built-in open(). This is why we don't import *. Import the module itself instead and use the reference to get to its names when required, e.g. aifc.open().
I recently started writing a simple client using the Blogger API to do some basic posting I implemented the client in Python and used the example code verbatim from the Blogger Developer's Guide to login, get the blog id, and make a new post. I ran the script and everything went fine until I got to this line:
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
I got the error message:
Traceback (most recent call last):
File "cs1121post.py", line 38, in <module>
cs1121post()
File "cs1121post.py", line 33, in cs1121post
return blogger_service.Post(entry, '/feeds/%s/posts/default' % blog_id)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1236, in Post
media_source=media_source, converter=converter)
File "/usr/local/lib/python2.7/dist-packages/gdata/service.py", line 1322, in PostOrPut
headers=extra_headers, url_params=url_params)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 93, in optional_warn_function
return f(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 176, in request
content_length = CalculateDataLength(data)
File "/usr/local/lib/python2.7/dist-packages/atom/service.py", line 736, in CalculateDataLength
return len(str(data))
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 377, in __str__
return self.ToString()
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 374, in ToString
return ElementTree.tostring(self._ToElementTree(), encoding=string_encoding)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 369, in _ToElementTree
self._AddMembersToElementTree(new_tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 331, in _AddMembersToElementTree
member._BecomeChildElement(tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 357, in _BecomeChildElement
self._AddMembersToElementTree(new_child)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 342, in _AddMembersToElementTree
ExtensionContainer._AddMembersToElementTree(self, tree)
File "/usr/local/lib/python2.7/dist-packages/atom/__init__.py", line 224, in _AddMembersToElementTree
tree.text = self.text.decode(MEMBER_STRING_ENCODING)
AttributeError: 'list' object has no attribute 'decode'
By which I'm taking it that ElementTree is at fault here. I installed ElementTree via
sudo python setup.py install
in case it matters. Is there some known incompatibility between ElementTree and Python v2.7.1? Has this happened to anybody else and how did you get it working? If you need any additional information, please reply to the thread. All the source code that is relevant is basically just the example code from the Developers Guide mentioned above. I haven't modified that at all (not even the variable names). Any input is greatly appreciated.
The stacktrace is actually pretty clear about this: You're calling decode() on a list instead of a tree element. Try getting the first element from the list and calling decode() on that:
firsttext = self.text[0].decode(MEMBER_STRING_ENCODING)