I want to implement a custom logic for aborting tasks that run a loop. The task is the following:
#celery_app.task(bind=True, acks_late=True, base=AbortableTask,name="task for groups")
def parallel_task(self, i, total):
for idx, item in enumerate(range(i)):
if self.is_aborted():
print("task is aborted")
self.update_state(state=REVOKED)
raise Ignore()
self.update_state(state="PROGRESS", meta={"current": idx + 1, "total": total})
print(f"Doing parallel job for items {item}")
time.sleep(random.randint(1, 4))
return random.randint(1, 100)
#celery_app.task(bind=True, name="callback")
def callback_task(self, results):
print("Excecuting callback")
print(f"Results: {results}")
time.sleep(random.randint(1, 4))
def revoke_or_abort(res: Union[AsyncResult, AbortableAsyncResult]):
res.revoke()
parent = res.parent
while parent:
parent.revoke()
if isinstance(parent, GroupResult):
for child in parent.children:
if isinstance(child, AbortableAsyncResult):
child.abort()
parent = parent.parent
revoke_or_abort revokes tasks that haven't started yet, and/or tries to abort them if already started.
Workflow is created and run like this:
>>> group_tasks = group(parallel_task.s(30, 100) for i in range(2))
>>> workflow = chord(header=group_tasks, body=callback_task.s())
>>> res = workflow.apply_async()
>>> revoke_or_abort(res)
celery -A my_module.celery worker -l info --concurrency=2 --prefetch-multiplier=1
Althoug tasks get aborted (and revoked) and also the chord's callback isn't executed I have the following side-effects:
Tasks are not updated to state REVOKED but they remain ABORTED. Also rerruning the workflow shows the tasks as if they were not acknoledged.
# running again
res = workflow.apply_async()
I see in celery:
[2020-06-14 10:49:03,968: INFO/MainProcess] Received task: task for groups[ff5c43a2-75a6-43c5-b700-66e8700573ec]
[2020-06-14 10:49:03,969: INFO/MainProcess] Received task: task for groups[6be5b2d3-0b1b-4087-b946-b54b2dcb812f]
[2020-06-14 10:49:03,974: WARNING/ForkPoolWorker-1] task is aborted
[2020-06-14 10:49:03,974: WARNING/ForkPoolWorker-2] task is aborted
[2020-06-14 10:49:03,976: INFO/ForkPoolWorker-2] Task task for groups[ff5c43a2-75a6-43c5-b700-66e8700573ec] ignored
[2020-06-14 10:49:03,976: INFO/ForkPoolWorker-1] Task task for groups[6be5b2d3-0b1b-4087-b946-b54b2dcb812f] ignored
which are the same task-ids as the first time I ran workflow. Why is that?. Can I trigger some how the revoke signal when a task is aborted, so I can "simulate" revoke functionality?
Related
I want to send a chain task at startup o worker like in this https://stackoverflow.com/a/14589445/3922534 question, but task run out of order.
Logs from worker
[2022-07-12 20:51:47,369: INFO/MainProcess] Task task.add_newspapers[5de1f446-65af-472a-a4b6-d9752142b588] received
[2022-07-12 20:51:47,372: WARNING/MainProcess] Now Runing Newspaper Function
[2022-07-12 20:51:47,408: INFO/MainProcess] Task task.check_tasks_are_created[33d6b9d1-660b-4a80-a726-6f167e246480] received
[2022-07-12 20:51:47,412: WARNING/MainProcess] Now Runing Podcast Function
[2022-07-12 20:51:47,427: INFO/MainProcess] Task task.add_newspapers[5de1f446-65af-472a-a4b6-d9752142b588] succeeded in 0.0470000000204891s: 'Now Runing Podcast Function'
[2022-07-12 20:51:47,432: INFO/MainProcess] Task task.add_yt_channels[26179491-2632-46bd-95c1-9e9dbb9e8130] received
[2022-07-12 20:51:47,433: WARNING/MainProcess] None
[2022-07-12 20:51:47,457: INFO/MainProcess] Task task.check_tasks_are_created[33d6b9d1-660b-4a80-a726-6f167e246480] succeeded in 0.0470000000204891s: None
[2022-07-12 20:51:47,463: INFO/MainProcess] Task task.add_podcasts[ad94a119-c6b2-475a-807b-b1a73bef589e] received
[2022-07-12 20:51:47,468: WARNING/MainProcess] Now Runing Check Tasks are Created Function
[2022-07-12 20:51:47,501: INFO/MainProcess] Task task.add_yt_channels[26179491-2632-46bd-95c1-9e9dbb9e8130] succeeded in 0.06299999984912574s: 'Now Runing Check Tasks are Created Function'
[2022-07-12 20:51:47,504: INFO/MainProcess] Task task.add_podcasts[ad94a119-c6b2-475a-807b-b1a73bef589e] succeeded in 0.030999999959021807s: 'Now Runing Yotube Channels Function'
Code How i send the task:
#worker_ready.connect
def at_start(sender, **k):
with sender.app.connection() as conn:
#sender.app.send_task(name='task.print_word', args=["I Send Task On Startup"],connection=conn,)
#ch = [add_newspapers.s(),add_podcasts.s(),add_yt_channels.s(),check_tasks_are_created.s()]
ch = [
signature("task.add_podcasts"),
signature("task.add_yt_channels"),
signature("task.check_tasks_are_created"),
]
sender.app.send_task(name='task.add_newspapers',chain=ch,connection=conn,)
Then I try it to run chain task like normally run apply_async(), but it runs at every worker. I want to run just once at one worker
#worker_ready.connect
def at_start(sender, **k):
chain(add_newspapers.s(),add_podcasts.s(),add_yt_channels.s(),check_tasks_are_created.s()).apply_async()
Then I try it to recognize the worker then apply .apply_async(), but it does not catch the if statment.
Documentation https://docs.celeryq.dev/en/latest/userguide/signals.html#celeryd-init
celery -A celery_app.celery worker --loglevel=INFO -P gevent --concurrency=40 -n celeryworker1
#worker_ready.connect
def at_start(sender, **k):
print("This is host name ", sender.hostname)
if sender == "celery#celeryworker1":
with sender.app.connection() as conn:
chain(add_newspapers.s(),add_podcasts.s(),add_yt_channels.s(),check_tasks_are_created.s()).apply_async()
Am I doing something wrong or is it just a bug?
Since a task doesn't need the return value of the previous task you can run it as:
chain(add_newspapers.si(),add_podcasts.si(),add_yt_channels.si(),check_tasks_are_created.si()).apply_async()
(change call from s() to si()
You can read about immutability here.
#worker_ready.connect handler will run on every worker. So, if you have 10 workers, you will send the same task 10 times, when they broadcast the "worker_ready" signal. Is this intentional?
Tasker class setups the initial job when instantiated. Basically what I want is put a job in the 'main_queue', decide if job is running or if there is already same job that is queued in the 'process_queue', return from the the current 'main_queue' job. Else queue a job in the 'process_queue'. When that process queue finishes, put a job in the 'main_queue'.
however, the 'process_queue' has the same job with id for that duration, despite it should have been finished looking at the outputs. So a new job is never put in to process. Is there a deadlock happening that I am unable to see?
main_queue worker
$ rq worker main_queue --with-scheduler
22:44:19 Worker rq:worker:7fe23a24ae404135a10e301f7509eb7e: started, version 1.9.0
22:44:19 Subscribing to channel rq:pubsub:7fe23a24ae404135a10e301f7509eb7e
22:44:19 *** Listening on main_queue...
22:44:19 Trying to acquire locks for main_queue
22:44:19 Scheduler for main_queue started with PID 3747
22:44:19 Cleaning registries for queue: main_queue
22:44:33 main_queue: tasks.redis_test_job() (e90e0dff-bbcc-48ab-afed-6d1ba8b020a8)
None
Job is enqueued to process_queue!
22:44:33 main_queue: Job OK (e90e0dff-bbcc-48ab-afed-6d1ba8b020a8)
22:44:33 Result is kept for 500 seconds
22:44:47 main_queue: tasks.redis_test_job() (1a7f91d0-73f4-466e-92f4-9f918a9dd1e9)
<Job test_job: tasks.print_job()>
!!Scheduler added job to main but same job is already queued in process_queue!!
22:44:47 main_queue: Job OK (1a7f91d0-73f4-466e-92f4-9f918a9dd1e9)
22:44:47 Result is kept for 500 seconds
process_queue worker
$ rq worker process_queue
22:44:24 Worker rq:worker:d70daf20ff324c18bc17f0ea9576df52: started, version 1.9.0
22:44:24 Subscribing to channel rq:pubsub:d70daf20ff324c18bc17f0ea9576df52
22:44:24 *** Listening on process_queue...
22:44:24 Cleaning registries for queue: process_queue
22:44:33 process_queue: tasks.print_job() (test_job)
The process job executed.
22:44:42 process_queue: Job OK (test_job)
22:44:42 Result is kept for 500 seconds
tasker.py
class Tasker():
def __init__(self):
self.tasker_conn = RedisClient().conn
self.process_queue = Queue(name='process_queue', connection=Redis(),
default_timeout=-1)
self.main_queue = Queue(name='main_queue', connection=Redis(),
default_timeout=-1)
self.__setup_tasks()
def __setup_tasks(self):
self.main_queue.enqueue_in(timedelta(seconds=3), tasks.redis_test_job)
tasks.py
import tasks
def redis_test_job():
q = Queue('process_queue', connection=Redis(), default_timeout=-1)
queued = q.fetch_job('test_job')
print(queued)
if queued:
print("!!Scheduler added job to main but same job is already queued in process_queue!!")
return False
else:
q.enqueue(tasks.print_job, job_id='test_job')
print("Job is enqueued to process_queue!")
return True
def print_job():
sleep(8)
print("The process job executed.")
q = Queue('main_queue', connection=Redis(), default_timeout=-1)
q.enqueue_in(timedelta(seconds=5), tasks.redis_test_job)
From the docs, enqueued jobs have a result_ttl that defaults to 500 seconds if you don't define it.
If you want to change it to e.g. make the job and result live for only 1 second, enqueue your job like this:
q.enqueue(tasks.print_job, job_id='test_job', result_ttl=1)
The celery worker is reporting:
[2015-09-29 16:13:50,411: INFO/MainProcess] Task xxx.tasks.tasks.process_files[e91db27d-9d16-487f-acae-d6966205ba16] succeeded in 52.9951906s: None
From my client, I can access some task info:
from celery.result import AsyncResult
task_id = 'e91db27d-9d16-487f-acae-d6966205ba16'
res = AsyncResult(task_id)
print(res.state)
print(res.info)
I would also like to know:
When was the task triggered?
How long did it take to complete?
Can I access this information from the AsyncResult? I see nothing in the documentation.
I defined some tasks with a time limit of 1200:
#celery.task(time_limit=1200)
def create_ne_list(text):
c = Client()
return c.create_ne_list(text)
I'm also using the worker_process_init signal to do some initialization, each time a new process starts:
#worker_process_init.connect
def init(sender=None, conf=None, **kwargs):
init_system(celery.conf)
init_pdf(celery.conf)
This initialization function takes several seconds to execute.
Besides that, I'm using the following configuration:
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']
BROKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_TIMEZONE = 'Europe/Berlin'
CELERY_ENABLE_UTC = True
and start my worker with the following command:
celery -A isc worker -l info --concurrency=3
As expected, starting the worker results in the initialization function being called three times. Now, I can send tasks and they are being executed and everything seems to run smoothly.
BUT: As soon as a tasks exceeds its time limit, the worker gets caught in an infinite loop of spawning and being killed off again because of exceeding the time limit.
[2014-06-13 09:46:18,978: ERROR/MainProcess] Timed out waiting for UP message from <Worker(Worker-20381, started daemon)>
[2014-06-13 09:46:20,000: ERROR/MainProcess] Process 'Worker-20381' pid:18953 exited with 'signal 9 (SIGKILL)'
// new worker 20382 getting started, initialization getting triggerd and soon after that -->
[2014-06-13 09:46:18,978: ERROR/MainProcess] Timed out waiting for UP message from <Worker(Worker-20382, started daemon)>
[2014-06-13 09:46:20,000: ERROR/MainProcess] Process 'Worker-20382' pid:18954 exited with 'signal 9 (SIGKILL)'
// and so on....
Does anyone has an idea why this is happening?
The answer seems to be that the signal worker_process_init requires the handler to not be blocking for more than 4 seconds.
http://celery.readthedocs.org/en/latest/userguide/signals.html#worker-process-init
Because my init function takes longer to execute, the worker will be terminated automatically. After that it naturally restarts and triggers the init function again, which then results in the worker being terminated again and so on.
I tried using this code to try to dynamically add / remove scheduled tasks.
My tasks.py file looks like this:
from celery.decorators import task
import logging
log = logging.getLogger(__name__)
#task
def mytask():
log.debug("Executing task")
return
The problem is that the tasks do not actually execute (i.e there is no log output), but I get the following messages in my celery log file, exactly on schedule:
[2013-05-10 04:53:00,005: INFO/MainProcess] Got task from broker: cron.tasks.mytask[dfcf397b-e30b-45bd-9f5f-11a17a51b6c4]
[2013-05-10 04:54:00,007: INFO/MainProcess] Got task from broker: cron.tasks.mytask[f013b3cd-6a0f-4060-8bcc-3bb51ffaf092]
[2013-05-10 04:55:00,007: INFO/MainProcess] Got task from broker: cron.tasks.mytask[dfc0d563-ff4b-4132-955a-4293dd3a9ac7]
[2013-05-10 04:56:00,012: INFO/MainProcess] Got task from broker: cron.tasks.mytask[ba093535-0d70-4dc5-89e4-441b72cfb61f]
I can definitely confirm that the logger is configured correctly and working fine. If I were to try and call result = mytask.delay() in the interactive shell, result.state will indefinitely contain the state PENDING.
EDIT: See also Django Celery Periodic Tasks Run But RabbitMQ Queues Aren't Consumed