That's a small story...
I had this error:
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
When changed tasks.py, like Diederik said at Celery with RabbitMQ: AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
app = Celery('tasks', backend='rpc://', broker='amqp://guest#localhost//')
ran it
>>> from tasks import add
>>> result = add.delay(4,50)
>>> result.ready()
got DisabledBackend again ... hmm what was that..
put code to file run.py and it returned True...
from tasks import add
try:
result = add.delay(1,4)
print (result.ready())
except:
print "exept"
I see that if I call >>> from tasks import add after tasks.py changed, it doesn't get the updates... That behaviour is the same for ipython, so because of I can't understand the reason, I advice people to DEBUG from scripts like ~runthis.py
Will be glad for answer which will smash my idea...
If using the interpreter, you need to
reload(tasks)
this will force reimport tasks module
Related
I am trying to save and restore GroupResult object after running task in Celery. The group result ID is returned and saved as expected. When I am trying to restore GroupResult with it:
from project.celery import app
print(smstask.celery_result_id)
print(app.GroupResult.restore(smstask.celery_result_id))
I get (as example):
4780fc17-44d8-478f-a41a-e4333aaa03d4
None
For the Celery backend I am using Djcelery. Can it be the cause of the problem?
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'
What else can you recommend to try? Thank you!
You should save() GroupResult if you have plan to use it later.
Then you can restore it by providing GroupResult.id to restore() method:
from celery import group, result
from app.tasks import foo, bar
group_result = group([foo.s(), bar.s()])()
group_result.save()
restored_group_result = result.GroupResult.restore(group_result.id)
print(restored_group_result)
I'm trying to solve a problem in celery:
I have one task that queries an API for ids, and then starts a sub-task for each of these.
I do not know, ahead of time, what the ids are, or how many there are.
For each id, I go through a big calculation that then dumps some data into a database.
After all the sub-tasks are complete, I want to run a summary function (export DB results to an Excel format).
Ideally, I do not want to block my main worker querying the status of the sub-tasks (Celery gets angry if you try this.)
This question looks very similar (if not identical?): Celery: Callback after task hierarchy
So using the "solution" (which is a link to this discussion, I tried the following test script:
# test.py
from celery import Celery, chord
from celery.utils.log import get_task_logger
app = Celery('test', backend='redis://localhost:45000/10?new_join=1', broker='redis://localhost:45000/11')
app.conf.CELERY_ALWAYS_EAGER = False
logger = get_task_logger(__name__)
#app.task(bind=True)
def get_one(self):
print('hello world')
self.replace(get_two.s())
return 1
#app.task
def get_two():
print('Returning two')
return 2
#app.task
def sum_all(data):
print('Logging data')
logger.error(data)
return sum(data)
if __name__ == '__main__':
print('Running test')
x = chord(get_one.s() for i in range(3))
body = sum_all.s()
result = x(body)
print(result.get())
print('Finished w/ test')
It doesn't work for me. I get an error:
AttributeError: 'get_one' object has no attribute 'replace'
Note that I do have new_join=1 in my backend URL, though not the broker. If I put it there, I get an error:
TypeError: _init_params() got an unexpected keyword argument 'new_join'
What am I doing wrong? I'm using the Python 3.4.3 and the following packages:
amqp==1.4.6
anyjson==0.3.3
billiard==3.3.0.20
celery==3.1.18
kombu==3.0.26
pytz==2015.4
redis==2.10.3
The Task.replace method will be added in Celery 3.2: http://celery.readthedocs.org/en/master/whatsnew-3.2.html#task-replace (that changelog entry is misleading, because it suggests that Task.replace existed before and has been changed.)
I have a celery task that's declared on my Django project that I'm trying to call from the same module it's declared in. Right now, it looks like the following:
# myapp.admin.py
from myproject.celery import app as celery_app
#celery_app.task(name='myapp.admin.add')
def add(x, y):
time.sleep(10000)
return x + y
def my_custom_admin_action(modeladmin, request, queryset):
add.delay(2, 4)
# action later declared in the ModelAdmin
Knowing that celery sometimes is complicated with relative imports, I've specified the name. I even added the following to my settings.py:
CELERY_IMPORTS = ('myapp.admin', )
But when I try to use the admin action, I get the following message in my manage.py celeryd output:
[2014-09-18 14:58:25,413: ERROR/MainProcess] Received unregistered task of type 'myapp.admin.add'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.
Traceback (most recent call last):
File "/Users/JJ/.virtualenvs/TCJ/lib/python2.7/site-packages/celery/worker/consumer.py", line 455, in on_task_received
strategies[name](message, body,
KeyError: 'myapp.admin.add'
What am I doing wrong here? I even tried importing within the action as from . import add, but that didn't seem to help.
Celery is not picking your add task. One alternate way to solve this is to modify the instance of your Celery.
In myproject/celery.py
change instance of celery
app = Celery('name', backend='your_backend', broker='your_broker')
to
app = Celery('name', backend='your_backend', broker='your_broker',
include['myapp.admin',])
Trying to use stackless python (2.7.2) with SPickle to send a test method over celery for execution on a different machine. I would like the test method (code) to be included with the pickle and not forced to exist on the executing machines python path.
Been referencing following presentation:
https://ep2012.europython.eu/conference/talks/advanced-pickling-with-stackless-python-and-spickle
Trying to use the technique shown in the checkpointing slide 11. The RPC example doesn't seem right given that we are using celery:
Client code:
from stackless import run, schedule, tasklet
from sPickle import SPickleTools
def test_method():
print "hello from test method"
tasks = []
test_tasklet = tasklet(test_method)()
tasks.append(test_tasklet)
pt = SPickleTools(serializeableModules=['__test_method__'])
pickled_task = pt.dumps(tasks)
Server code:
pt = sPickle.SPickleTools()
unpickledTasks = pt.loads(pickled_task)
Results in:
[2012-03-09 14:24:59,104: ERROR/MainProcess] Task
celery_tasks.test_exec_method[8f462bd6-7952-4aa1-9adc-d84ee4a51ea6] raised exception:
AttributeError("'module'
object has no attribute 'test_method'",)
Traceback (most recent call last):
File "c:\Python27\lib\site-packages\celery\execute\trace.py", line 153, in trace_task
R = retval = task(*args, **kwargs)
File "c:\Python27\celery_tasks.py", line 16, in test_exec_method
unpickledTasks = pt.loads(pickled_task)
File "c:\Python27\lib\site-packages\sPickle\_sPickle.py", line 946, in loads
return unpickler.load()
AttributeError: 'module' object has no attribute 'test_method'
Any suggestions on what I am doing incorrect or if this is even possible?
Alternative suggestions for doing dynamic module loading in a celeryd would also be good (as an alternative for using sPickle). I have experimented with doing:
py_mod = imp.load_source(module_name,'some script path')
sys.modules.setdefault(module_name,py_mod)
but the dynamically loaded module does not seem to persist through different calls to celeryd, i.e. different remote calls.
You must define test_method within its own module. Currently sPickle detects whether test_method is defined in a module that can be imported. An alternative way is to set the __module__ attribute of the function to None.
def test_method():
pass
test_method.__module__ = None
I'm using Nick Johnson's Bulk Update library on google appengine (http://blog.notdot.net/2010/03/Announcing-a-robust-datastore-bulk-update-utility-for-App-Engine). It works wonderfully for other tasks, but for some reason with the following code:
from google.appengine.ext import db
from myapp.main.models import Story, Comment
import bulkupdate
class Migrate(bulkupdate.BulkUpdater):
DELETE_COMPLETED_JOBS_DELAY = 0
DELETE_FAILED_JOBS = False
PUT_BATCH_SIZE = 1
DELETE_BATCH_SIZE = 1
MAX_EXECUTION_TIME = 10
def get_query(self):
return Story.all().filter("hidden", False).filter("visible", True)
def handle_entity(self, entity):
comments = entity.comment_set
for comment in comments:
s = Story()
s.parent_story = comment.story
s.user = comment.user
s.text = comment.text
s.submitted = comment.submitted
self.put(s)
job = Migrate()
job.start()
I get the following error in my logs:
Permanent failure attempting to execute task
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 258, in post
run(self.request.body)
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 122, in run
raise PermanentTaskFailure(e)
PermanentTaskFailure: 'module' object has no attribute 'Migrate'
It seems quite bizarre to me. Clearly that class is right above the job, they're in the same file and clearly the job.start is being called. Why can't it see my Migrate class?
EDIT: I added this update job in a newer version of the code, which isn't the default. I invoke the job with the correct URL (http://version.myapp.appspot.com/migrate). Is it possible this is related to the fact that it isn't the 'default' version served by App Engine?
It seems likely that your declaration of the 'Migrate' class is in the handler script (Eg, the one directly invoked by app.yaml). A limitation of deferred is that you can't use it to call functions defined in the handler module.
Incidentally, my bulk update library is deprecated in favor of App Engine's mapreduce support; you should probably use that instead.