I'm trying to execute a periodic task using celery to delete users who didn't activate their account in time. The screenshot bellow shows that the task is correctly discovered and executed, but when i check the database no changes are done.
The celery task :
#tasks.py
from celery.task.schedules import crontab
from celery.decorators import periodic_task
from celery.utils.log import get_task_logger
from .utils import unconfirmed_users_delete
logger = get_task_logger(__name__)
# A periodic task that will run every minute (the symbol "*" means every)
#periodic_task(run_every=(crontab(hour="*", minute="*", day_of_week="*")))
def delete_unconfirmed_users():
return unconfirmed_users_delete()
The queryset to execute (checked in django shell and correctly working) :
#utils.py
from django.contrib.auth.models import User
from django.utils import timezone
def unconfirmed_users_delete():
return User.objects.filter(is_active=False).filter(profile__key_expires__lt=timezone.now()).delete()
The task is correctly called every minute :
What could be wrong ?
As #schillingt mentioned most of the time, we forget to (re)start worker process for the periodic task.
This happens because we have a beat scheduler which schedules the task and worker which executes the task.
celery -A my_task beat # schedule tasks
celery worker -A my_task -l info # consume tasks
A much better solution is to have a worker which schedules task & executes. You can do that using
celery worker -A my_task -l info --beat # schedule & consume tasks
This schedules the periodic task and consumes it.
Related
I tried to create a task that should run every minute in celery along with redis server
To execute redis I ran "redis-server"
To execute celery I ran "celery -A tasks worker --loglevel=info"
This is my tasks.py file
from celery import Celery
from celery.schedules import crontab
from celery.task import periodic_task
app = Celery('tasks', backend='redis://localhost', broker='redis://localhost')
#app.task
def add(x, y):
return x + y
#periodic_task(run_every=(crontab(minute='1')),name="run_every_minute",ignore_result=True)
def run_every_minute():
print("hehe")
return "ok"
When I ran in python console
from tasks.py import run_every_minute
z=run_every_minute.delay()
I got output at celery running terminal as
[2019-06-05 01:35:02,591: INFO/MainProcess] Received task: run_every_minute[06498b4b-1d13-45af-b91c-fb10476e0aa3]
[2019-06-05 01:35:02,595: WARNING/Worker-2] hehe
[2019-06-05 01:35:02,599: INFO/MainProcess] Task run_every_minute[06498b4b-1d13-45af-b91c-fb10476e0aa3] succeeded in
0.004713802001788281s: 'ok'
But this should execute every minute since its a periodic task. How this can happen.
Also, how can we execute a celery task at some specific time say 5:30 GMT(for example).
Ok, based on the commentary
First periodic_task needs the scheduler/beat be started (Periodic Tasks), with this the scheduler will send the task depending in the run_every parameter
celery -A tasks beat
Next, if you need to send the beat every minute, you need the crontab be like this
#periodic_task(run_every=(crontab(minute='*')),name="run_every_minute",ignore_result=True)
def run_every_minute():
print("hehe")
return "ok"
With minute='*', it will send the task every minute. minute=1 will send the task at every hour in the minute one
Answering your last comment:
run_every=(crontab(minute='1'))
You have specified 'minute of hour' = 1, so celery beat runs your periodic task every hour at minute '1', e.g. 00:01, 01:01 and so on.
You should set hour attribute for your crontab, propably as a range
I am trying to establish a periodic task using Celery (4.2.0) and RabbitMQ (3.7.14) running with Python 3.7.2 on an Azure VM using Ubuntu 16.04. I am able to start the beat and worker and see the message get kicked off from beat to the worker but at this point I'm met with an error like so
[2019-03-29 21:35:00,081: ERROR/MainProcess] Received
unregistered task of type 'facebook-call.facebook_api'.
The message has been ignored and discarded.
Did you remember to import the module containing this task?
Or maybe you're using relative imports?
My code is as follows:
from celery import Celery
from celery.schedules import crontab
app = Celery('facebook-call', broker='amqp://localhost//')
#app.task
def facebook_api():
{function here}
app.conf.beat.schedule = {
'task': 'facebook-call.facebook_api',
'schedule': crontab(hour=0, minute =0, day='0-6'),
}
I am starting the beat and worker processes by using the name of the python file which contains all of the code
celery -A FacebookAPICall beat --loglevel=info
celery -A FacebookAPICall worker --loglevel=info
Again, the beat process starts and I can see the message being successfully passed to the worker but cannot figure out how to "register" the task so that it is processed by the worker.
I was able to resolve the issue by renaming the app from facebook-call to coincide with the name of the file FacebookAPICall
Before:
app = Celery('facebook-call', broker='amqp://localhost//'
After:
app = Celery('FacebookAPICall', broker='amqp://localhost//'
From reading the Celery documentation, I don't totally understand why the name of the app must also be the name of the .py file but that seems to do the trick.
Consider the code:
from celery import Celery, group
from time import time
app = Celery('tasks', broker='redis:///0', backend='redis:///1', task_ignore_result=False)
#app.task
def test_task(i):
print('hi')
return i
x = test_task.delay(3)
print(x.get())
I run it by calling python script.py, but I'm getting no results. Why?
You don't get any results because you've asked your celery app to execute a task without starting a worker process to do the work executing it. The process you did start is blocked on the call to get().
First things first, when using celery it is critical that you do not have tasks get executed when a module is imported, so let's put your task execution inside of a main() function, and put it in a file called celery_test.py.
from celery import Celery, group
from time import time
app = Celery('tasks', broker='redis:///0', backend='redis:///1', task_ignore_result=False)
#app.task
def test_task(i):
print('hi')
return i
def main():
x = test_task.delay(3)
print(x.get())
if __name__ == '__main__':
main()
Now let's start a pool of celery workers to execute tasks for this app. You can do this by opening a new terminal and executing the following.
celery worker -A celery_test --loglevel=INFO
The -A flag refers to the module where celery will find an application to add workers to. You should see some output in the terminal to indicate that the the celery worker is running and ready for tasks to process.
Now, try executing your script again with python celery_test.py. You should see hi show up in the worker's log output, but the the value 3 returned in the script that called get().
Be warned, if you've been playing with celery without running a worker, it probably has lots of tasks waiting in your broker to execute. The first time you start up the worker pool, you'll see them all execute in parallel until the broker runs out of tasks.
I would like to launch a periodic task every second but only if the previous task ended (db polling to send task to celery).
In the Celery documentation they are using the Django cache to make a lock.
I tried to use the example:
from __future__ import absolute_import
import datetime
import time
from celery import shared_task
from django.core.cache import cache
LOCK_EXPIRE = 60 * 5
#shared_task
def periodic():
acquire_lock = lambda: cache.add('lock_id', 'true', LOCK_EXPIRE)
release_lock = lambda: cache.delete('lock_id')
a = acquire_lock()
if a:
try:
time.sleep(10)
print a, 'Hello ', datetime.datetime.now()
finally:
release_lock()
else:
print 'Ignore'
with the following configuration:
app.conf.update(
CELERY_IGNORE_RESULT=True,
CELERY_ACCEPT_CONTENT=['json'],
CELERY_TASK_SERIALIZER='json',
CELERY_RESULT_SERIALIZER='json',
CELERYBEAT_SCHEDULE={
'periodic_task': {
'task': 'app_task_management.tasks.periodic',
'schedule': timedelta(seconds=1),
},
},
)
But in the console, I never see the Ignore message and I have Hello every second. It seems that the lock is not working fine.
I launch the periodic task with:
celeryd -B -A my_app
and the worker with:
celery worker -A my_app -l info
Could you please correct my misunderstanding?
From the Django Cache Framework documentation about local-memory cache:
Note that each process will have its own private cache instance, which
means no cross-process caching is possible.
So basically your workers are each dealing with their own cache. If you need a low resource cost cache backend I would recommend File Based Cache or Database Cache, both allow cross-process.
I'm working with Celery http://celery.readthedocs.org/en/latest/index.html
I need to run a periodic tasks at a specific moment. But I only want to start my task after starting the celery worker.
For that I'm trying to create my own "PeriodicTask". But I'm dealing with a problem.
When I'm starting the worker and executing the run_tasks.py in another terminal, it seems that my periodic tasks is executed only one time.
How could I do to have my periodic task running every 3 seconds.
Here is a part of the code.
Start celery :
celery worker --app=worker_manager.celery --loglevel=info
file tasks.py
class MyPeriodicTask(PeriodicTask):
name = "periodic-task"
run_every = timedelta(seconds=3)
def run(self, **kwargs):
logger = self.get_logger(**kwargs)
logger.info("Running periodic task!")
file run_tasks.py
tasks.register(MyPeriodicTask)
wmi_collector_task = worker_app.tasks[MyPeriodicTask.name]
Thanks in advance.
To run periodic tasks you need to start celery beat. You can do this by passing -B argument when starting workers:
celery worker -B --app=worker_manager.celery --loglevel=info