I need to call the following 2 apply_async tasks:
escalate.apply_async((e.id), countdown=3)
escalate.apply_async((e.id), countdown=3)
My tasks implementation looks like:
#app.task
def escalate(id, group):
escalation_email, created = EscalationEmail.objects.get_or_create()
escalation_email.send()
return 'sup email sent'
I run the work with the following command:
celery -A proj worker -l info --concurrency=10
The problem is that when I look at the worker, only 1 tasks is received and then only 1 succeeds. Also, only 1 email sends.
It seems that most of the time the second escalate task runs.
How can I ensure that these tasks both fire 100% of the time with reliability?
The problem was that I did not choose a queue to associate the task with.
Related
My problem
I'm trying to write some functional tests for a Django app that uses Celery. Those tests call celery tasks that must be asynchronous and concurrent.
In order to be more flexible and run those tests on gitlab pipelines (some times we want to have app.conf.task_always_eager = True in our tests to make them synchronous) each test class will in it's setUpClass method start it's own celery worker with
from celery.contrib.testing.worker import start_worker
# ...
cls.celery_worker = start_worker(app, log="info", perform_ping_check=False, concurrency=2)
cls.celery_worker.__enter__() # and __exit__ in tearDownClass() of course
However, it seems that the concurrency=2 param is useless, as it doesn't allow for any kind of concurrency.
To reproduce
Install the the latest celery version (5.2.3).
In celery_tasks.py:
import time
from celery import Celery, shared_task
app = Celery("main_app")
app.config_from_object("celery_config")
#shared_task
def sleep_task(secs):
print(f"Sleeping for {secs} seconds...")
for i in range(secs):
time.sleep(i)
print(f"\t{i + 1}")
return "DONE"
In celery_config.py:
broker_url = "redis://127.0.0.1:6379/0"
result_backend = "redis://127.0.0.1:6379/0"
Normal way to start a task (what is expected):
in a terminal , use the command python -m celery -A celery_tasks worker
in main.py, write and execute in another terminal:
from celery_tasks import sleep_task
if __name__ == '__main__':
sleep_task.delay(3)
sleep_task.delay(3)
Now, in the terminal which runs celery, you can see logs
[2022-03-21 11:42:21,866: WARNING/ForkPoolWorker-7] Sleeping for 3 seconds...
[2022-03-21 11:42:21,866: WARNING/ForkPoolWorker-1] Sleeping for 3 seconds...
[2022-03-21 11:42:21,866: WARNING/ForkPoolWorker-7] 1
[2022-03-21 11:42:21,867: WARNING/ForkPoolWorker-1] 1
[2022-03-21 11:42:22,868: WARNING/ForkPoolWorker-1] 2
[2022-03-21 11:42:22,868: WARNING/ForkPoolWorker-7] 2
[2022-03-21 11:42:24,869: WARNING/ForkPoolWorker-1] 3
[2022-03-21 11:42:24,870: WARNING/ForkPoolWorker-7] 3
We clearly see that those tasks are asynchronous and concurrent
We start our own worker using start_worker:
Do not start a celery worker in another terminal (the script will handle it)
in main.py, write and execute:
from time import sleep
from celery_tasks import sleep_task, app
from celery.contrib.testing.worker import start_worker
if __name__ == '__main__':
app.conf.task_always_eager = False # will sometimes be True, but in that case the concurrence question is irrelevant
celery_worker = start_worker(app, perform_ping_check=False, concurrency=2)
with celery_worker: # equivalent to celery_worker.__enter__() and automatic __exit__()
print("queuing first sleep_task()")
sleep_task.delay(3)
print("queuing second sleep_task()")
sleep_task.delay(3)
sleep(10) # this sleep is here to see full output and allow both tasks to execute
Logs produced, in the same terminal where main.py was executed:
queuing first sleep_task()
queuing second sleep_task()
Sleeping for 3 seconds...
1
2
3
Sleeping for 3 seconds...
1
2
3
As you can see, even when the call were asynchronous, they are not concurrent, with both tasks executing one after the other, even with the concurrency=2 param in start_worker
Is this a bug, or is this intended ? If this is intended, what's your suggestion to make it work. To be clear, I want those test classes to run on gitlab pipelines, have some of them behave synchronously with app.conf.task_always_eager = True
I tried to create a task that should run every minute in celery along with redis server
To execute redis I ran "redis-server"
To execute celery I ran "celery -A tasks worker --loglevel=info"
This is my tasks.py file
from celery import Celery
from celery.schedules import crontab
from celery.task import periodic_task
app = Celery('tasks', backend='redis://localhost', broker='redis://localhost')
#app.task
def add(x, y):
return x + y
#periodic_task(run_every=(crontab(minute='1')),name="run_every_minute",ignore_result=True)
def run_every_minute():
print("hehe")
return "ok"
When I ran in python console
from tasks.py import run_every_minute
z=run_every_minute.delay()
I got output at celery running terminal as
[2019-06-05 01:35:02,591: INFO/MainProcess] Received task: run_every_minute[06498b4b-1d13-45af-b91c-fb10476e0aa3]
[2019-06-05 01:35:02,595: WARNING/Worker-2] hehe
[2019-06-05 01:35:02,599: INFO/MainProcess] Task run_every_minute[06498b4b-1d13-45af-b91c-fb10476e0aa3] succeeded in
0.004713802001788281s: 'ok'
But this should execute every minute since its a periodic task. How this can happen.
Also, how can we execute a celery task at some specific time say 5:30 GMT(for example).
Ok, based on the commentary
First periodic_task needs the scheduler/beat be started (Periodic Tasks), with this the scheduler will send the task depending in the run_every parameter
celery -A tasks beat
Next, if you need to send the beat every minute, you need the crontab be like this
#periodic_task(run_every=(crontab(minute='*')),name="run_every_minute",ignore_result=True)
def run_every_minute():
print("hehe")
return "ok"
With minute='*', it will send the task every minute. minute=1 will send the task at every hour in the minute one
Answering your last comment:
run_every=(crontab(minute='1'))
You have specified 'minute of hour' = 1, so celery beat runs your periodic task every hour at minute '1', e.g. 00:01, 01:01 and so on.
You should set hour attribute for your crontab, propably as a range
I have 2 apps on 2 separate servers, let's call them A and B. Both apps have a Celery worker active, listening to separate queues (QueueA and QueueB).
Server B pushes a task to QueueB, using apply_async.
Here is server B's tasks:
#app.task(bind=True, queue="QueueB", name="name_on_server_A")
def taskForServerB():
# nothing is executed here
#app.task(bind=True)
def success(result):
print('Task succeeded')
#app.task(bind=True):
def failure(...):
print('task failed')
taskForServerB.s().apply_async(link=success.s(), link_error=failure.s())
On Server A, the task name_on_server_A receives the tasks and executes it. If it completes successfully, the task success is execute properly on ServerB, but it name_on_server_A fails, the task failure is not executed. Instead, Server A throws a NotRegisteredError for a task with name failure.
Is there something I am missing? How can I get the failure task to be executed on ServerB, where the first task is called from?
There are two issues here:
The route of task to the correct queue which you defined for name_on_server_A (with the queue assignment) - which is by the way something that is new for me (I'm using ROUTER in the celery config and route each task by it's name to the right queue.
when you define your celery app you might forgot to include the task failure so it unregister:
app = Celery(broker='amqp://', backend='...', include=['file1.py', 'file2.py', ..])
Consider the code:
from celery import Celery, group
from time import time
app = Celery('tasks', broker='redis:///0', backend='redis:///1', task_ignore_result=False)
#app.task
def test_task(i):
print('hi')
return i
x = test_task.delay(3)
print(x.get())
I run it by calling python script.py, but I'm getting no results. Why?
You don't get any results because you've asked your celery app to execute a task without starting a worker process to do the work executing it. The process you did start is blocked on the call to get().
First things first, when using celery it is critical that you do not have tasks get executed when a module is imported, so let's put your task execution inside of a main() function, and put it in a file called celery_test.py.
from celery import Celery, group
from time import time
app = Celery('tasks', broker='redis:///0', backend='redis:///1', task_ignore_result=False)
#app.task
def test_task(i):
print('hi')
return i
def main():
x = test_task.delay(3)
print(x.get())
if __name__ == '__main__':
main()
Now let's start a pool of celery workers to execute tasks for this app. You can do this by opening a new terminal and executing the following.
celery worker -A celery_test --loglevel=INFO
The -A flag refers to the module where celery will find an application to add workers to. You should see some output in the terminal to indicate that the the celery worker is running and ready for tasks to process.
Now, try executing your script again with python celery_test.py. You should see hi show up in the worker's log output, but the the value 3 returned in the script that called get().
Be warned, if you've been playing with celery without running a worker, it probably has lots of tasks waiting in your broker to execute. The first time you start up the worker pool, you'll see them all execute in parallel until the broker runs out of tasks.
I'm working with Celery http://celery.readthedocs.org/en/latest/index.html
I need to run a periodic tasks at a specific moment. But I only want to start my task after starting the celery worker.
For that I'm trying to create my own "PeriodicTask". But I'm dealing with a problem.
When I'm starting the worker and executing the run_tasks.py in another terminal, it seems that my periodic tasks is executed only one time.
How could I do to have my periodic task running every 3 seconds.
Here is a part of the code.
Start celery :
celery worker --app=worker_manager.celery --loglevel=info
file tasks.py
class MyPeriodicTask(PeriodicTask):
name = "periodic-task"
run_every = timedelta(seconds=3)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
logger.info("Running periodic task!")
file run_tasks.py
tasks.register(MyPeriodicTask)
wmi_collector_task = worker_app.tasks[MyPeriodicTask.name]
Thanks in advance.
To run periodic tasks you need to start celery beat. You can do this by passing -B argument when starting workers:
celery worker -B --app=worker_manager.celery --loglevel=info