Tasker class setups the initial job when instantiated. Basically what I want is put a job in the 'main_queue', decide if job is running or if there is already same job that is queued in the 'process_queue', return from the the current 'main_queue' job. Else queue a job in the 'process_queue'. When that process queue finishes, put a job in the 'main_queue'.
however, the 'process_queue' has the same job with id for that duration, despite it should have been finished looking at the outputs. So a new job is never put in to process. Is there a deadlock happening that I am unable to see?
main_queue worker
$ rq worker main_queue --with-scheduler
22:44:19 Worker rq:worker:7fe23a24ae404135a10e301f7509eb7e: started, version 1.9.0
22:44:19 Subscribing to channel rq:pubsub:7fe23a24ae404135a10e301f7509eb7e
22:44:19 *** Listening on main_queue...
22:44:19 Trying to acquire locks for main_queue
22:44:19 Scheduler for main_queue started with PID 3747
22:44:19 Cleaning registries for queue: main_queue
22:44:33 main_queue: tasks.redis_test_job() (e90e0dff-bbcc-48ab-afed-6d1ba8b020a8)
None
Job is enqueued to process_queue!
22:44:33 main_queue: Job OK (e90e0dff-bbcc-48ab-afed-6d1ba8b020a8)
22:44:33 Result is kept for 500 seconds
22:44:47 main_queue: tasks.redis_test_job() (1a7f91d0-73f4-466e-92f4-9f918a9dd1e9)
<Job test_job: tasks.print_job()>
!!Scheduler added job to main but same job is already queued in process_queue!!
22:44:47 main_queue: Job OK (1a7f91d0-73f4-466e-92f4-9f918a9dd1e9)
22:44:47 Result is kept for 500 seconds
process_queue worker
$ rq worker process_queue
22:44:24 Worker rq:worker:d70daf20ff324c18bc17f0ea9576df52: started, version 1.9.0
22:44:24 Subscribing to channel rq:pubsub:d70daf20ff324c18bc17f0ea9576df52
22:44:24 *** Listening on process_queue...
22:44:24 Cleaning registries for queue: process_queue
22:44:33 process_queue: tasks.print_job() (test_job)
The process job executed.
22:44:42 process_queue: Job OK (test_job)
22:44:42 Result is kept for 500 seconds
tasker.py
class Tasker():
def __init__(self):
self.tasker_conn = RedisClient().conn
self.process_queue = Queue(name='process_queue', connection=Redis(),
default_timeout=-1)
self.main_queue = Queue(name='main_queue', connection=Redis(),
default_timeout=-1)
self.__setup_tasks()
def __setup_tasks(self):
self.main_queue.enqueue_in(timedelta(seconds=3), tasks.redis_test_job)
tasks.py
import tasks
def redis_test_job():
q = Queue('process_queue', connection=Redis(), default_timeout=-1)
queued = q.fetch_job('test_job')
print(queued)
if queued:
print("!!Scheduler added job to main but same job is already queued in process_queue!!")
return False
else:
q.enqueue(tasks.print_job, job_id='test_job')
print("Job is enqueued to process_queue!")
return True
def print_job():
sleep(8)
print("The process job executed.")
q = Queue('main_queue', connection=Redis(), default_timeout=-1)
q.enqueue_in(timedelta(seconds=5), tasks.redis_test_job)
From the docs, enqueued jobs have a result_ttl that defaults to 500 seconds if you don't define it.
If you want to change it to e.g. make the job and result live for only 1 second, enqueue your job like this:
q.enqueue(tasks.print_job, job_id='test_job', result_ttl=1)
Related
I want to implement a custom logic for aborting tasks that run a loop. The task is the following:
#celery_app.task(bind=True, acks_late=True, base=AbortableTask,name="task for groups")
def parallel_task(self, i, total):
for idx, item in enumerate(range(i)):
if self.is_aborted():
print("task is aborted")
self.update_state(state=REVOKED)
raise Ignore()
self.update_state(state="PROGRESS", meta={"current": idx + 1, "total": total})
print(f"Doing parallel job for items {item}")
time.sleep(random.randint(1, 4))
return random.randint(1, 100)
#celery_app.task(bind=True, name="callback")
def callback_task(self, results):
print("Excecuting callback")
print(f"Results: {results}")
time.sleep(random.randint(1, 4))
def revoke_or_abort(res: Union[AsyncResult, AbortableAsyncResult]):
res.revoke()
parent = res.parent
while parent:
parent.revoke()
if isinstance(parent, GroupResult):
for child in parent.children:
if isinstance(child, AbortableAsyncResult):
child.abort()
parent = parent.parent
revoke_or_abort revokes tasks that haven't started yet, and/or tries to abort them if already started.
Workflow is created and run like this:
>>> group_tasks = group(parallel_task.s(30, 100) for i in range(2))
>>> workflow = chord(header=group_tasks, body=callback_task.s())
>>> res = workflow.apply_async()
>>> revoke_or_abort(res)
celery -A my_module.celery worker -l info --concurrency=2 --prefetch-multiplier=1
Althoug tasks get aborted (and revoked) and also the chord's callback isn't executed I have the following side-effects:
Tasks are not updated to state REVOKED but they remain ABORTED. Also rerruning the workflow shows the tasks as if they were not acknoledged.
# running again
res = workflow.apply_async()
I see in celery:
[2020-06-14 10:49:03,968: INFO/MainProcess] Received task: task for groups[ff5c43a2-75a6-43c5-b700-66e8700573ec]
[2020-06-14 10:49:03,969: INFO/MainProcess] Received task: task for groups[6be5b2d3-0b1b-4087-b946-b54b2dcb812f]
[2020-06-14 10:49:03,974: WARNING/ForkPoolWorker-1] task is aborted
[2020-06-14 10:49:03,974: WARNING/ForkPoolWorker-2] task is aborted
[2020-06-14 10:49:03,976: INFO/ForkPoolWorker-2] Task task for groups[ff5c43a2-75a6-43c5-b700-66e8700573ec] ignored
[2020-06-14 10:49:03,976: INFO/ForkPoolWorker-1] Task task for groups[6be5b2d3-0b1b-4087-b946-b54b2dcb812f] ignored
which are the same task-ids as the first time I ran workflow. Why is that?. Can I trigger some how the revoke signal when a task is aborted, so I can "simulate" revoke functionality?
I have 2 apps on 2 separate servers, let's call them A and B. Both apps have a Celery worker active, listening to separate queues (QueueA and QueueB).
Server B pushes a task to QueueB, using apply_async.
Here is server B's tasks:
#app.task(bind=True, queue="QueueB", name="name_on_server_A")
def taskForServerB():
# nothing is executed here
#app.task(bind=True)
def success(result):
print('Task succeeded')
#app.task(bind=True):
def failure(...):
print('task failed')
taskForServerB.s().apply_async(link=success.s(), link_error=failure.s())
On Server A, the task name_on_server_A receives the tasks and executes it. If it completes successfully, the task success is execute properly on ServerB, but it name_on_server_A fails, the task failure is not executed. Instead, Server A throws a NotRegisteredError for a task with name failure.
Is there something I am missing? How can I get the failure task to be executed on ServerB, where the first task is called from?
There are two issues here:
The route of task to the correct queue which you defined for name_on_server_A (with the queue assignment) - which is by the way something that is new for me (I'm using ROUTER in the celery config and route each task by it's name to the right queue.
when you define your celery app you might forgot to include the task failure so it unregister:
app = Celery(broker='amqp://', backend='...', include=['file1.py', 'file2.py', ..])
I need to call the following 2 apply_async tasks:
escalate.apply_async((e.id), countdown=3)
escalate.apply_async((e.id), countdown=3)
My tasks implementation looks like:
#app.task
def escalate(id, group):
escalation_email, created = EscalationEmail.objects.get_or_create()
escalation_email.send()
return 'sup email sent'
I run the work with the following command:
celery -A proj worker -l info --concurrency=10
The problem is that when I look at the worker, only 1 tasks is received and then only 1 succeeds. Also, only 1 email sends.
It seems that most of the time the second escalate task runs.
How can I ensure that these tasks both fire 100% of the time with reliability?
The problem was that I did not choose a queue to associate the task with.
The celery worker is reporting:
[2015-09-29 16:13:50,411: INFO/MainProcess] Task xxx.tasks.tasks.process_files[e91db27d-9d16-487f-acae-d6966205ba16] succeeded in 52.9951906s: None
From my client, I can access some task info:
from celery.result import AsyncResult
task_id = 'e91db27d-9d16-487f-acae-d6966205ba16'
res = AsyncResult(task_id)
print(res.state)
print(res.info)
I would also like to know:
When was the task triggered?
How long did it take to complete?
Can I access this information from the AsyncResult? I see nothing in the documentation.
I'm posting a data to a web-service in Celery. Sometimes, the data is not posted to web-service because of the internet is down, and the task is retried infinite times until it is posted. The retrying of the task is un-necessary because the net was down and hence its not required to re-try it again.
I thought of a better solution, ie if a task fails thrice (retrying a min of 3 times), then it is shifted to another queue. This queue contains list of all failed tasks.
Now when the internet is up and the data is posted over the net , ie the task has been completed from the normal queue, it then starts processing the tasks from the queue having failed tasks.
This will not waste the CPU memory of retrying the task again and again.
Here's my code :- As of right now, I'm just retrying the task again, But I doubt whether that'll be the right way of doing it.
#shared_task(default_retry_delay = 1 * 60, max_retries = 10)
def post_data_to_web_service(data,url):
try :
client = SoapClient(
location = url,
action = 'http://tempuri.org/IService_1_0/',
namespace = "http://tempuri.org/",
soap_ns='soap', ns = False
)
response= client.UpdateShipment(
Weight = Decimal(data['Weight']),
Length = Decimal(data['Length']),
Height = Decimal(data['Height']),
Width = Decimal(data['Width']) ,
)
except Exception, exc:
raise post_data_to_web_service.retry(exc=exc)
How do I maintain 2 queues simultaneous and trying to execute tasks from both the queues.
Settings.py
BROKER_URL = 'redis://localhost:6379/0'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
By default celery adds all tasks to queue named celery. So you can run your task here and when an exception occurs, it retries, once it reaches maximum retries, you can shift them to a new queue say foo
from celery.exceptions import MaxRetriesExceededError
#shared_task(default_retry_delay = 1 * 60, max_retries = 10)
def post_data_to_web_service(data,url):
try:
#do something with given args
except MaxRetriesExceededError:
post_data_to_web_service([data, url], queue='foo')
except Exception, exc:
raise post_data_to_web_service.retry(exc=exc)
When you start your worker, this task will try to do something with given data. If it fails it will retry 10 times with a dealy of 60 seconds. Then when it encounters MaxRetriesExceededError it posts the same task to new queue foo.
To consume these tasks you have to start a new worker
celery worker -l info -A my_app -Q foo
or you can also consume this task from the default worker if you start it with
celery worker -l info -A my_app -Q celery,foo