I have configured Django + Celery: all works, i can execute tasks, that called from views.py i.e. mul.apply_async((2, 5), queue='celery', countdown=5)
I need to shedule periodic task that will chain simple tasks with argument that passed from users.
I read docs http://docs.celeryproject.org/en/latest/userguide/canvas.html and know how to chain, i know how to make periodic task without parameters #periodic_task(run_every=(crontab(hour="*", minute="*", day_of_week="*")))
But how to combine this?
What i want workflow:
User creates project with parameters. 5 tasks executed, using that parameters.
Then i need shedule to repeat all 5 tasks every 24 hours. So here i dont know how to pass parameters (they save to db).
In other answer i saw this syntax:
CELERYBEAT_SCHEDULE = {
# crontab(hour=0, minute=0, day_of_week='saturday')
'schedule-name': { # example: 'file-backup'
'task': 'some_django_app.tasks....', # example: 'files.tasks.cleanup'
'schedule': crontab(...)
'args': (2, 3)
},
}
But the problem here is that it located in settings.py of Django but not in tasks.py and i cannot dynamically pass args.
The celerybeat task you register could be a wrapper and perform the project/task logic inside of it, firing off other tasks as appropriate. You could fetch the project tasks inside of your celery beat job.
CELERYBEAT_SCHEDULE.task -> 'some_django_app.project_beat_task'
Then project beat task could retrieve the correct projects and all tasks associated with them, perhaps spawning a chain of tasks for each project
Related
I am in the process of writing my own task app using Django and would like a few specific functions to be executed every day at a certain time (updating tasks, checking due dates, etc.). Is there a way to have Django run functions on a regular basis or how do I go about this in general?
Does it make sense to write an extra program with an infinite loop for this or are there better ways?
Celery is a good option here:
First steps with Django
Periodic Tasks
app.conf.beat_schedule = {
'add-every-30-seconds': {
'task': 'tasks.add',
'schedule': 30.0,
'args': (16, 16)
},
}
app.conf.timezone = 'UTC'
With celery you can define periodic tasks at any given interval. Celery workers will then pick up those tasks when needed. You will need to run something like RabbitMQ or Redis to support the celery workers.
The alternative, simpler, way is to add an entry to your urls.py that catches any url you don't otherwise use, and use that as a prompt to check your database as to whether another task is due. This leverages the fact that your website will be hit by a lot of bot traffic. The timing isn't entirely reliable but it doesn't require any extra set up.
you can use django-cronjobs or maybe Schedule your job with schedule library.
There is a specific periodic task that needs to be removed from message queue. I am using the configuration of Redis and celery here.
tasks.py
#periodic_task(run_every=crontab(minute='*/6'))
def task_abcd():
"""
some operations here
"""
There are other periodic tasks also in the project but I need to stop this specific task to stop from now on.
As explained in this answer, the following code will work?
#periodic_task(run_every=crontab(minute='*/6'))
def task_abcd():
pass
In this example periodic task schedule is defined directly in code, meaning it is hard-coded and cannot be altered dynamically without code change and app re-deploy.
The provided code with task logic deleted or with simple return at the beginning - will work, but will not be the answer to the question - task will still run, there just is no code that will run with it.
Also, it is recommended NOT to use #periodic_task:
"""Deprecated decorator, please use :setting:beat_schedule."""
so it is not recommended to use it.
First, change method from being #periodic_task to just regular celery #task, and because you are using Django - it is better to go straightforward for #shared_task:
from celery import shared_task
#shared_task
def task_abcd():
...
Now this is just one of celery tasks, which needs to be called explicitly. Or it can be run periodically if added to celery beat schedule.
For production and if using multiple workers it is not recommended to run celery worker with embedded beat (-B) - run separate instance of celery beat scheduler.
Schedule can specified in celery.py or in django project settings (settings.py).
It is still not very dynamic, as to re-read settings app needs to be reloaded.
Then, use Database Scheduler which will allow dynamically creating schedules - which tasks need to be run and when and with what arguments. It even provides nice django admin web views for administration!
That code will work but I'd go for something that doesn't force you to update your code every time you need to disable/enable the task.
What you could do is to use a configurable variable whose value could come from an admin panel, a configuration file, or whatever you want, and use that to return before your code runs if the task is in disabled mode.
For instance:
#periodic_task(run_every=crontab(minute='*/6'))
def task_abcd():
config = load_config_for_task_abcd()
if not config.is_enabled:
return
# some operations here
In this way, even if your task is scheduled, its operations won't be executed.
If you simply want to remove the periodic task, have you tried to remove the function and then restart your celery service. You can restart your Redis service as well as your Django server for safe measure.
Make sure that the function you removed is not referenced anywhere else.
Using Celery ver.3.1.23, I am trying to dynamically add a scheduled task to celery beat. I have one celery worker and one celery beat instance running.
Triggering a standard celery task y running task.delay() works ok. When I define a scheduled periodic task as a setting in configuration, celery beat runs it.
However what I need is to be able to add a task that runs at specified crontab at runtime. After adding a task to persistent scheduler, celery beat doesn't seem to detect the newly added new task. I can see that the celery-schedule file does have an entry with new task.
Code:
scheduler = PersistentScheduler(app=current_app, schedule_filename='celerybeat-schedule')
scheduler.add(name="adder",
task="app.tasks.add",
schedule=crontab(minute='*/1'),
args=(1,2))
scheduler.close()
When I run:
print(scheduler.schedule)
I get:
{'celery.backend_cleanup': <Entry: celery.backend_cleanup celery.backend_cleanup() <crontab: 0 4 * * * (m/h/d/dM/MY)>,
'adder': <Entry: adder app.tasks.add(1, 2) <crontab: */1 * * * * (m/h/d/dM/MY)>}
Note that app.tasks.add has the #celery.task decorator.
Instead of trying to find a good workaround, I suggest you switch to the Celery Redbeat.
You may solve your problem by enabling autoreloading.
However I'm not 100% sure it will work for your config file but it should if is in the CELERY_IMPORTS paths.
Hoverer note that this feature is experimental and to don't be used in production.
If you really want to have dynamic celerybeat scheduling you can always use another scheduler like the django-celery one to manage periodic tasks on db via a django admin.
I'm having a similar problem and a solution I thought about is to pre-define some generic periodic tasks (every 1s, every 5mins, etc) and then have them getting, from DB, a list of function to be executed.
Every time you want to add a new task you just add an entry in your DB.
Celery beat stores all the periodically scheduled tasks in the model PeriodicTask . As a beat task can be scheduled in different ways including crontab, interval or solar. All these fields are a foreign key in the PeriodicTask model.
In order to dynamically add a scheduled task, just populate the relevant models in celery beat, the scheduler will detect changes. The changes are detected when either the count of tuple changes or save() function is called.
from django_celery_beat.models import PeriodicTask, CrontabSchedule
# -- Inside the function you want to add task dynamically
schedule = CrontabSchedule.objects.create(minute='*/1')
task = PeriodicTask.objects.create(name='adder',
task='apps.task.add', crontab=schedule)
task.save()
I wanted to develop an asynchronous task in my python project in which the task runs once everyday at a particular time.
I have researched a lot on the various ways of getting over with this function but I am very confused between celery beat and crontabs and their functionalities.
I would be glad if anyone helped me understanding the difference between the two (if any), including their performance considerations.
as you can see in this code:
CELERYBEAT_SCHEDULE = {
'add-every-30-seconds': {
'task': 'tasks.add',
'schedule': crontab(hour=7, minute=30, day_of_week=1),
'args': (16, 16)
},
}
the celery beat is actually the process itself,think of it like a function with settings.
the function is the task and the settings are the arguments,name and schedule, here comes the crontab, its a period of a time that tells celery when to cycle through it.
you can also see here a list of crontabs types.
I posted a question :How to execute a command at exact time once a day in Django?
I got my answer that the Celery is the easiest option to do it, but now i have another question regarding the celery:
from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
# Executes every Monday morning at 7:30 A.M
'every-monday-morning': {
'task': 'tasks.add',
'schedule': crontab(hour=7, minute=30, day_of_week=1),
'args': (16, 16),
},
}
I have three question in regarding the above code i.e:
I have to execute execute_command(User, command) method. I want that this method will execute at the given scheduled time.
What if i want to change the schedule at 7:30 AM but every weekdays?.
What about the args. In my case should i pass the value of User and command from args. Or i can simply pass from the task key ?
I just read the docs of celery, but didn't got my answer. Would you please help me?
Did you find the Celery periodic tasks documentation?
You'll have to use the identifier of your method to register a scheduler entry. If your execute_command task lives in a module named foobar, the task value of the CELERYBEAT_SCHEDULE structure should be foobar.execute_command.
Celery will import the task for you, provided that import foobar.execute_command would work.
Check the celery.schedule.crontab API; the following should execute on weekdays at 07:30 am:
crontab(minute=30, hour=7, day_of_week=’mon-fri’)
Remember that this task is going to be performed asynchronously. You cannot query for a database object when you schedule this task and expect it to be there still when the task is called.
Thus, you should only pass in python values that remain constant, and have your task connect to the database and look things up based on the arguments you pass in.
If this task is only ever going to execute tasks for one specific user, then by all means pass in the identifier for that user (user id, email, whatever you can use to look up the user from the database).