There are altogether eight tasks running in celery in different periods. All of them are event-driven tasks. After a certain event, they got fired. And the particular task works continuously until certain conditions were satisfied.
I have registered a task which checks for certain conditions for almost two minutes. This task works fine most of the time. But sometimes the expected behavior of the task is not attained.
The signature of the task is as below:
tasks.py
import time
from celery import shared_task
#shared_task()
def some_celery_task(a, b):
main_time_end = time.time() + 120
while time.time() < main_time_end:
...
# some db operations here with given function arguments 'a' and 'b'
# this part of the task get execute most of the time
if time.time() > main_time_end:
...
# some db operations here.
# this part is the part of the task that doesn't get executed sometimes
views.py
# the other part of the view not mentioned here
# only the task invoked part
some_celery_task.apply_async(args=(5, 9), countdown=0)
I am confused about the celery task timeout scenarios. Does that mean the task will stop from where it timeouts or will retry automatically?
It will be a great help if any clear idea about timeout and retries you guys got.
What could be the reason behind the explained scenarios above? Any help on this question will be highly appreciated. Thank you.
Check Celery documentation on Tasks - basics are documented very well.
If task fails or was terminated - task will have states.FAILURE status. It will not be re-tried unless specifically coded. If logging is correctly configured - you might see exception messages in logs in case of timeouts or other code exceptions.
When Celery Task TIME_LIMIT is exceeded - task is terminated right away:
The worker processing the task will be killed and replaced with a new one.
Also, TimeLimitExceeded exception will be raised with message like Task handler raised error: "TimeLimitExceeded(2700)"
If Celery SOFT_TIME_LIMIT is set and is smaller than TIME_LIMIT and is exceeded - than SoftTimeLimitExceeded exception will be raised allowing it to be catched in the task and perform clean-up actions.
When worker consumes message (task) from the broker queue - broker needs to know that the message was consumed successfully. To confirm successful consumpion of message worker acknowledges (ACK) to broker. Until message is not acknowledged it is not deleted from broker but also not available for consumption ("invisible"). In not acknowledged - message will be re-delivered back to broker queue available again for consumption.
Redelivering un-acknowledged messages logic depends on broker:
AMQP (RabbitMQ) broker - tracks connection status with worker, and if connection is lost - returns message back to queue.
Redis or SQS broker has its own timeout after which message will be re-delivered to broker queue if not ACKed.
By default celery worker acknowledges message right at the start of the task.
If ACKS_LATE is set - worker acknowledges to broker only after successfully executing task.
One can RETRY task, by catching exception in the task and sending same task back to the broker for re-execution - then this same task with same id will be queued at broker. Countdown option allows to specify delay before the task will be retried.
Celery Task Execution and other Options can be set globally in settings.py or per task as arguments.
Recommended way it to design tasks / logic with consideration of such events to be totally legit and see them normal (but not actually expected) to happen sometime and be ready:
tasks may fail (next same task may do work for both or checks that specific work was not done and re-fire task)
same task may run again (idempotency)
similar tasks can be run simultaneously (locking)
Related
I am using Celery to parallelize the execution of a Python function calling a third-party API.
This API imposes to wait for at least 3 seconds between each call.
Is there a way to specify a Message Broker (RabbitMQ or Redis) to respect this delay between each worker call ?
In Celery, you can use the countdown method. See https://docs.celeryproject.org/en/stable/userguide/calling.html#eta-and-countdown
RabbitMQ has a few different options for supporting delayed messages including the delayed message exchange plugin and dead lettering. Unfortunately, Celery doesn't support either of these methods. What it does instead is delay execution of the message until the countdown time is reached. The message itself is immediately sent to the worker.
Would check out https://docs.celeryproject.org/en/stable/userguide/tasks.html#retrying
Specify the retry delay to be 3 seconds and you can set a limit for the number of retries to be X as part of the task definition as seen in the docs.
I have three Celery workers as follows, each running on a different ECS node:
Producer: Keeps generating & sending tasks to the consumer worker. Each task is expected to take several minutes to compute and has a database record.
Consumer: Receives computation tasks and immediately starts execution.
Watchdog: Periodically inspects database records, finds out computation tasks that are executing, and then does celery inspect active to verify whether there is actually a worker carrying out the computation.
We ensured that when the Consumer node is being terminated, the Celery worker on it will begin graceful shutdown, so that the ongoing computation can finish normally. Because Celery will unregister a gracefully stopping worker, the consumer will become invisible to the Watchdog, who will mistakenly think a computation task has mysteriously lost... even though the Consumer is still working on the task.
Is it possible to let a Celery worker broadcast an "I am dying" message upon receiving a warm shutdown signal? Or even better, can we somehow let the Watchdog worker still see shutting workers?
Yes, it is possible. Nodes in Celery cluster I am responsible for are doing something similar. Here is a snippet:
#worker_shutdown.connect
def handle_worker_shutdown(**kwargs):
_handle_worker_shutdown(app, _LOGGER, **kwargs)
#worker_ready.connect
def handle_worker_ready(**kwargs):
_handle_worker_ready(app, _LOGGER, **kwargs)
There are few other, very useful signals that you should have a look, but these two are essential. Maybe the worker_shutting_down is more suitable for your use-case...
I am currently experimenting with future tasks in celery using the ETA feature and a redis broker. One of the known issues with using a redis broker has to do with the visibility timeout:
If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.
This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.
Some tasks that I can envision will have an ETA on the timescale of weeks/months. Setting the visibility timeout large enough to encompass these tasks is probably unwise.
Are there any paths forward for processing these tasks with a redis broker? I am aware of this question. Is changing brokers the only option?
I am doing this with redis in the following way:
We have customers that can schedule a release of some of their content. We store the release in our database with the time it should be executed at.
Then we use celery beat to perform a periodic task (hourly or what suits you) that checks our releases table for releases that are scheduled within the next period (again hour or what suits you). if any are found we then schedule a task for them with celery. This allows us to have a short ETA.
I have no experience with Celery so I'm looking into it if my use case is solvable in Celery.
The client will submit a job to Celery, this job will be executed in CeleryTask. Then the client has to send keepalive every 30 seconds to keep this job active. Once the job is not refreshed by keepalive message, the job will be cancelled.
I can think of two solutions:
Each job-task created will have hard time limit of 30s. When client sends keepalive the router will send message to relevant worker to reset the hard time limit.
Each job-task will have no time limit. For each job-task there will be another special watchdog task launched. The watchdog task will be launched with delay of 30s. If new keepalive arrives from the client the watchdog task is cancelled and recreated. Again with delay of 30 seconds. If the watchdog is executed, it will kill the job-task, thus eliminating it from system.
The 1. is much simpler, but I'm not sure how to reset the task timelimit. The solution 2. seems more correct, but I'm afraid there will be various race conditions. The watchdog task should be probably running in separate queue reserved for watchdogs only.
How is something like this possible? Even one of my solution or some other.
In my understanding, you just want to have a middleware(one to receive keepalive, on to control task), then this link will help youcelery.app.control.Control.terminate.
Get task_id of task when apply_async (destination or all workers)
listen the keepalive from client
if time limit arrived and no keepalive, terminate task app.control.terminate(task_id, reply=True)
I'm using Celery 3.0.12.
I have two queues: Q1, Q2.
In general I put the main task in Q1, which call then subtasks that go in Q2.
I don't want to store any results for the subtasks. So my subtasks have the decorator #celery.task(ignore_results=True).
My main task should now wait until the subtask has finished. Because I write no results. I can't use: AsyncResult. Is there a way to wait in the main task to wait until the subtask finishes without storing the states to the backend. All my attempts with AsyncResults are not successfuel, (it relies on the backend). It seems also get() relies on the backend.
The whole story in code:
#celery.task(ignore_result=True)
def subtask():
#Do something
#celery.task
def maintask():
# Do something
# Call subtask on Q2:
res = subtask(options={'queue':'Q2'}).delay()
# Need to wait till subtask finishes
# NOT WORKING (DOES NEVER RETURN)
res.get()
I'm monitoring the whole application with Celery Flower and I can see that subtask is successfuelly finishing. How can Celery detect that state? I browsed their code but couldn't find out how they do the detection.
My main task should now wait until the subtask has finished.
You should never wait for a subtask as this may lead to resource starvation and deadlock (all
tasks waiting for another task, but no more workers to process them).
Instead you should use a callback to do additional actions after the subtask completes
(see the Canvas guide in the Celery user guide).
I'm monitoring the whole application with Celery Flower and I can see that subtask is
successfuelly finishing. How can Celery detect that state? I browsed their code but couldn't
find out how they do the detection.
Flower and other monitors does not use results (task state), instead they use what we call events.
Event messages are emitted when certain actions occur in the worker, and this becomes a transient stream of messages. Processes can subscribe to certain events (or all of them) to monitor the cluster.
The events are separate from task states because,
Events are not persistent (transient)
Missing an event is not regarded as a critical failure.
Complex fields are not serialized
Events are for diagnostic and informational purposes, and should not be used
to introspect task return values or exceptions
for example, as only the repr() of these is stored to make sure monitors
can be written in other languages, and big fields may be truncated to ensure
faster transmission.