I'm using Nick Johnson's Bulk Update library on google appengine (http://blog.notdot.net/2010/03/Announcing-a-robust-datastore-bulk-update-utility-for-App-Engine). It works wonderfully for other tasks, but for some reason with the following code:
from google.appengine.ext import db
from myapp.main.models import Story, Comment
import bulkupdate
class Migrate(bulkupdate.BulkUpdater):
DELETE_COMPLETED_JOBS_DELAY = 0
DELETE_FAILED_JOBS = False
PUT_BATCH_SIZE = 1
DELETE_BATCH_SIZE = 1
MAX_EXECUTION_TIME = 10
def get_query(self):
return Story.all().filter("hidden", False).filter("visible", True)
def handle_entity(self, entity):
comments = entity.comment_set
for comment in comments:
s = Story()
s.parent_story = comment.story
s.user = comment.user
s.text = comment.text
s.submitted = comment.submitted
self.put(s)
job = Migrate()
job.start()
I get the following error in my logs:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 258, in post
run(self.request.body)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 122, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: 'module' object has no attribute 'Migrate'
It seems quite bizarre to me. Clearly that class is right above the job, they're in the same file and clearly the job.start is being called. Why can't it see my Migrate class?
EDIT: I added this update job in a newer version of the code, which isn't the default. I invoke the job with the correct URL (http://version.myapp.appspot.com/migrate). Is it possible this is related to the fact that it isn't the 'default' version served by App Engine?
It seems likely that your declaration of the 'Migrate' class is in the handler script (Eg, the one directly invoked by app.yaml). A limitation of deferred is that you can't use it to call functions defined in the handler module.
Incidentally, my bulk update library is deprecated in favor of App Engine's mapreduce support; you should probably use that instead.
Related
I am trying to get started with Python + gRCP and so I checked out their repository as mentioned in the gRPC guide (https://grpc.io/docs/quickstart/python/).
Now I could execute the Hello World-Script (Client + Server), and so I tried to modify it. To ensure I did not missconfigure anything I just extended the Hello World-function (that use to work out before). I added the following lines:
import time
def SayHello(self, request, context):
currentTime = time.clock_gettime(time.CLOCK_REALTIME)
return helloworld_pb2.HelloReply(message='Time is, %s!' % currentTime)
Now what I inmagined it would do is to simply pass the currentTime-object back in this message I am returning upon that function is called - yet, what happens is the following error:
ERROR:grpc._server:Exception calling application: 'module' object has
no attribute 'clock_gettime' Traceback (most recent call last): File
"/home/user/.local/lib/python2.7/site-packages/grpc/_server.py", line
435, in _call_behavior
response_or_iterator = behavior(argument, context) File "greeter_server.py", line 29, in SayHello
currentTime = time.clock_gettime(time.CLOCK_REALTIME) AttributeError: 'module' object has no attribute 'clock_gettime'
I tried to Google around and I found that this might occur if you have a file named time in the same directory (so Python confuses the file in the current directory with the time-file. Yet there is no such file and he seems to find the correct time-file (since I can see the documentation when I hover the import and the function). What did I do wrong here?
The "full" Server Code (up to the serve() function):
from concurrent import futures
import logging
import time
import grpc
import helloworld_pb2
import helloworld_pb2_grpc
class Greeter(helloworld_pb2_grpc.GreeterServicer):
def SayHello(self, request, context):
currentTime = time.clock_gettime(time.CLOCK_REALTIME)
return helloworld_pb2.HelloReply(message='Time is, %s!' % currentTime)
Edit: I am using Ubuntu if that is important.
time.clock_gettime is a 3.3+ API and you are using 2.7.
I have a celery task that's declared on my Django project that I'm trying to call from the same module it's declared in. Right now, it looks like the following:
# myapp.admin.py
from myproject.celery import app as celery_app
#celery_app.task(name='myapp.admin.add')
def add(x, y):
time.sleep(10000)
return x + y
def my_custom_admin_action(modeladmin, request, queryset):
add.delay(2, 4)
# action later declared in the ModelAdmin
Knowing that celery sometimes is complicated with relative imports, I've specified the name. I even added the following to my settings.py:
CELERY_IMPORTS = ('myapp.admin', )
But when I try to use the admin action, I get the following message in my manage.py celeryd output:
[2014-09-18 14:58:25,413: ERROR/MainProcess] Received unregistered task of type 'myapp.admin.add'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.
Traceback (most recent call last):
File "/Users/JJ/.virtualenvs/TCJ/lib/python2.7/site-packages/celery/worker/consumer.py", line 455, in on_task_received
strategies[name](message, body,
KeyError: 'myapp.admin.add'
What am I doing wrong here? I even tried importing within the action as from . import add, but that didn't seem to help.
Celery is not picking your add task. One alternate way to solve this is to modify the instance of your Celery.
In myproject/celery.py
change instance of celery
app = Celery('name', backend='your_backend', broker='your_broker')
to
app = Celery('name', backend='your_backend', broker='your_broker',
include['myapp.admin',])
I have a Flask project on GAE and I'd like to start adding try/except blocks around database writes in case the datastore has problems, which will definitely fire when there's a real error, but I'd like to mimic that error in a unittest so I can have confidence of what will really happen during an outage.
For example, my User model:
class User(ndb.Model):
guser = ndb.UserProperty()
user_handle = ndb.StringProperty()
and in other view/controller code:
def do_something():
try:
User(guser=users.get_current_user(), user_handle='barney').put()
except CapabilityDisabledError:
flash('Oops, database is down, try again later', 'danger')
return redirect(url_for('registration_done'))
Here's a gist of my test code: https://gist.github.com/iandouglas/10441406
In a nutshell, GAE allows us to use capabilities to temporarily disable the stubs for memcache, datastore_v3, etc., and in the main test method:
def test_stuff(self):
# this test ALWAYS passes, making me believe the datastore is temporarily down
self.assertFalse(capabilities.CapabilitySet('datastore_v3').is_enabled())
# but this write to the datastore always SUCCEEDS, so the exception never gets
# thrown, therefore this "assertRaises" always fails
self.assertRaises(CapabilityDisabledError,
lambda: User(guser=self.guser, pilot_handle='foo').put())
I read some other post recommending calling the User.put() as a lambda which results in this traceback:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 28, in test_stuff
self.assertRaises(CapabilityDisabledError, lambda: User(
AssertionError: CapabilityDisabledError not raised
If I remove the lambda: portion, I get this traceback instead:
Traceback (most recent call last):
File "/home/id/src/project/tests/integration/views/test_datastore_offline.py", line 31, in test_stuff
pilot_handle_lower='foo'
File "/usr/lib/python2.7/unittest/case.py", line 475, in assertRaises
callableObj(*args, **kwargs)
TypeError: 'Key' object is not callable
Google's tutorials show you how to turn these capabilities on and off for unit testing, and in other tutorials they show you which exceptions could get thrown if their services are offline or experiencing intermittent issues, but they have no tutorials showing how they might work together in a unit test.
Thanks for any ideas.
The datastore stub does not support returning a CapabilityDisabledError, so enabled the error in the capabilities stub will not affect calls to datastore.
As a separate note, if you are using the High Replication Datastore, you'll never experience the CapabilityDisabledError because it does not have scheduled downtime.
I'm running into a few problems with adding gae-sessions to a relatively mature GAE app. I followed the readme carefully and also looked at the demo.
First, just adding the gaesesions directory to my app causes the following error when running tests with nose and nose-gae:
Exception ImportError: 'No module named threading' in <bound method local.__del__ of <_threading_local.local object at 0x103e10628>> ignored
All the tests run fine so not a big problem but suggests that something isn't right.
Next, if I add the following two lines of code:
from gaesessions import get_current_session
session = get_current_session()
And run my tests, then I get the following error:
Traceback (most recent call last):
File "/Users/.../unit_tests.py", line 1421, in testParseFBRequest
data = tasks.parse_fb_request(sr)
File "/Users/.../tasks.py", line 220, in parse_fb_request
session = get_current_session()
File "/Users/.../gaesessions/__init__.py", line 36, in get_current_session
return _tls.current_session
File "/Library/.../python2.7/_threading_local.py", line 193, in __getattribute__
return object.__getattribute__(self, name)
AttributeError: 'local' object has no attribute 'current_session'
This error does not happen on the dev server.
Any suggestions on fixing the above would be greatly appreciated.
I ran into the same problem. The problem seems to be that the gae testbed behaves differently than the development server. I don't know the specifics but ended up solving it by adding
def setUp(self):
testbed.Testbed().activate()
# after activating the testbed:
from gaesessions import Session, set_current_session
set_current_session(Session())
Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.
Been referencing following presentation:
https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle
Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery:
Client code:
from stackless import run, schedule, tasklet
from sPickle import SPickleTools
def test_method():
print "hello from test method"
tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)
pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)
Server code:
pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)
Results in:
[2012-03-09 14:24:59,104: ERROR/MainProcess] Task
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'
Any suggestions on what I am doing incorrect or if this is even possible?
Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). I have experimented with doing:
py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)
but the dynamically loaded module does not seem to persist through different calls to celeryd, i.e. different remote calls.
You must define test_method within its own module. Currently sPickle detects whether test_method is defined in a module that can be imported. An alternative way is to set the __module__ attribute of the function to None.
def test_method():
pass
test_method.__module__ = None