I've created a task where the program uploads a file and the task acts as expected i.e it successfully uploads the file but even after the task does its work, the task does not "terminate", the state still remains PENDING. How can I fix this?
From the celery docs:
PENDING: Task is waiting for execution or unknown. Any task id that is
not know is implied to be in the pending state.
Are you sure the task finishes what it does cleanly? Best to post code.
Related
I have three Celery workers as follows, each running on a different ECS node:
Producer: Keeps generating & sending tasks to the consumer worker. Each task is expected to take several minutes to compute and has a database record.
Consumer: Receives computation tasks and immediately starts execution.
Watchdog: Periodically inspects database records, finds out computation tasks that are executing, and then does celery inspect active to verify whether there is actually a worker carrying out the computation.
We ensured that when the Consumer node is being terminated, the Celery worker on it will begin graceful shutdown, so that the ongoing computation can finish normally. Because Celery will unregister a gracefully stopping worker, the consumer will become invisible to the Watchdog, who will mistakenly think a computation task has mysteriously lost... even though the Consumer is still working on the task.
Is it possible to let a Celery worker broadcast an "I am dying" message upon receiving a warm shutdown signal? Or even better, can we somehow let the Watchdog worker still see shutting workers?
Yes, it is possible. Nodes in Celery cluster I am responsible for are doing something similar. Here is a snippet:
#worker_shutdown.connect
def handle_worker_shutdown(**kwargs):
_handle_worker_shutdown(app, _LOGGER, **kwargs)
#worker_ready.connect
def handle_worker_ready(**kwargs):
_handle_worker_ready(app, _LOGGER, **kwargs)
There are few other, very useful signals that you should have a look, but these two are essential. Maybe the worker_shutting_down is more suitable for your use-case...
After reading docs; what I understand is that you cannot rerun celery tasks outside of application contexts.
Initially; what i thought was; terminated task would resume running once the worker has been restarted; however it didn't. I am currently using
celery.control.terminate(task_id)
That terminates my celery task id; I then tried running a worker with the same name hoping my revoked task would resume and finish; it didn't. After doing bit of research; I saw that a task can be reran with the same arguments; I thought MAYBE it would resume if I reran the same task again, it didn't. How can I revoke a task - then be able to re run it.
I'm using .apply_async() to intiate my task.
Use revoke instead of terminate e.g:
celery_app.control.revoke(task_id)
you can refer this solution as well.
Cancel an already executing task with Celery?
Does celery detect the changes of task code even if task already is prefetched as past task code?
No, you must reload the workers.
I'm using celery 3.X and RabbitMQ backend. From time to time it needs to restart celery (to push a new source code update to the server). But there is a task with big loop and try/catch inside of the loop; it can takes a few hours to accomplish the task. Nothing critical will happen if I will stop it and will restart it later.
QUESTION: The problem is every time after I stopped the workers (via sudo service celeryd stop) I have to KILL the task manually (via kill -9); the task ignores SIGTERM from worker. I've read throw Celery docs & Stackoverflow but I can't find working solution. Any ideas how to fix the problem?
Sending the QUIT signal will stop workers immediately: sudo service celeryd stop -QUIT
If the CELERY_ACKS_LATE setting is set to True, tasks that were running when the worker stopped will run again when the worker starts back up.
Celery is not intended to run long tasks cause it blocks the worker for your task only. I recommend re-arranging your logic, making the task invoke itself instead of making the loop. Once shutdown is in progress, your current task will complete and will resume right at the same point where it stopped before celery shutdown.
Also, having task split into chunks, you will be able to divert the task to another worker/host which is probably what you would like to do in the future.
I've launched a lot of tasks, but some of then hasn't finished (763 tasks), are in PENDING state, but the system isn't processing anything...
It's possible to retry this tasks giving celery the task_id?
You can't.
You can retry a task only from inside itself, you can't do it from outside.
The best thing to do in this case is to run again the task type with the same args, in this way you will do the same JOB but with a new PID that identify your process/task.
Remember also that the celery PENDING state not means only that the task is waiting for execution, but maybe that is unknown.
http://celeryq.org/docs/userguide/tasks.html#pending
I hope this could help
This works now after setting celery.conf.update(result_extended=True) which persists the arguments passed to the task:
def retry_task(task_id):
meta=celery.backend.get_task_meta(task_id)
task = celery.tasks[meta['name']]
task.apply_async(args=meta['args'], kwargs=meta['kwargs']) #specify any other parameters you might be passing