Develop Reference

how to start a task at a specific time with django & celery

I'm using Celery, and it works for async, but I need to set up an operation to a specific datetime.
For example "on the 30th of August, 2019, at 11:35, do this."
My celery.py is very simple now, but it works:
import time
from datetime import datetime, timedelta
from datetime import date
from celery import shared_task,current_task, task
from celery import Celery
app = Celery()
#app.task
def test():
print ('1')
todaynow = datetime.now()
print todaynow
I call it from view and run print on rabbit
Any idea for how to program a task?
ty
EDIT:
I try in view to call "test"
test.apply_async(eta=datetime(2019, 7, 31, 6, 28))
in flower it receive the task but not execute it, why?

To run a task at a specific time you can pass the eta parameter to apply_async
test.apply_async(eta=datetime.datetime(2019, 8, 30, 11, 35))

Celery component responsible for scheduling tasks to run at specific time, or repeatedly after some time is called the Celery Beat (scheduler). Celery documentation has a complete section describing it, with details how to run it, and how to configure it. If you are familiar with crontab you will easily create your own scheduled task-runs.

You can create a single executed periodic scheduler for "the 30 of august 2019 at 11 and 35 min do this" by using celery such as:
import time
from datetime import datetime, timedelta
from datetime import date
from celery import Celery, shared_task,current_task, task
from celery.schedules import crontab
app = Celery()
#app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(
crontab(hour=11, minute=35, day_of_month=30, month_of_year=8),
test.s(),
expires=1
)
#app.task
def test():
print ('1')
todaynow = datetime.now()
print todaynow

To schedule task you need to use celery beat .
from celery.task import periodic_task
#periodic_task(run_every=crontab(minute="*")) # It will run your task every minute
def schedule_task():
print('1')
todaynow = datetime.now()
print(todaynow)
You can schedule your task at any specific time using periodic task .
To know more use this link
https://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html .
Don't forget to restart your celery beat after creation of task .

Django - run a function every x seconds

I'm working on a Django app. I have an API endpoint, which if requested, must carry out a function that must be repeated a few times (until a certain condition is true). How I'm dealing with it right now is -
def shut_down(request):
# Do some stuff
while True:
result = some_fn()
if result:
break
time.sleep(2)
return True
While I know that this is a terrible approach and that I shouldn't be blocking for 2 seconds, I can't figure out how to get around it.
This works, after say a wait of 4 seconds. But I'd like something that keeps the loop running in the background, and stop once some_fn returns True. (Also, it is certain that some_fn will return True)
EDIT -
Reading Oz123's response gave me an idea which seems to work. Here's what I did -
def shut_down(params):
# Do some stuff
# Offload the blocking job to a new thread
t = threading.Thread(target=some_fn, args=(id, ), kwargs={})
t.setDaemon(True)
t.start()
return True
def some_fn(id):
while True:
# Do the job, get result in res
# If the job is done, return. Or sleep the thread for 2 seconds before trying again.
if res:
return
else:
time.sleep(2)
This does the job for me. It's simple but I don't know how efficient multithreading is in conjunction with Django.
If anyone can point out pitfalls of this, criticism is appreciated.

For many small projects celery is overkill. For those projects you can use schedule, it's very easy to use.
With this library you can make any function execute a task periodically:
import schedule
import time
def job():
print("I'm working...")
schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)
while True:
schedule.run_pending()
time.sleep(1)
The example runs in a blocking manner, but if you look in the FAQ, you will find that you can also run tasks in a parallel thread, such that you are not blocking, and remove the task once not needed anymore:
import threading
import time
from schedule import Scheduler
def run_continuously(self, interval=1):
"""Continuously run, while executing pending jobs at each elapsed
time interval.
#return cease_continuous_run: threading.Event which can be set to
cease continuous run.
Please note that it is *intended behavior that run_continuously()
does not run missed jobs*. For example, if you've registered a job
that should run every minute and you set a continuous run interval
of one hour then your job won't be run 60 times at each interval but
only once.
"""
cease_continuous_run = threading.Event()
class ScheduleThread(threading.Thread):
#classmethod
def run(cls):
while not cease_continuous_run.is_set():
self.run_pending()
time.sleep(interval)
continuous_thread = ScheduleThread()
continuous_thread.setDaemon(True)
continuous_thread.start()
return cease_continuous_run
Scheduler.run_continuously = run_continuously
Here is an example for usage in a class method:
def foo(self):
...
if some_condition():
return schedule.CancelJob # a job can dequeue it
# can be put in __enter__ or __init__
self._job_stop = self.scheduler.run_continuously()
logger.debug("doing foo"...)
self.foo() # call foo
self.scheduler.every(5).seconds.do(
self.foo) # schedule foo for running every 5 seconds
...
# later on foo is not needed any more:
self._job_stop.set()
...
def __exit__(self, exec_type, exc_value, traceback):
# if the jobs are not stop, you can stop them
self._job_stop.set()

This answer expands on Oz123's answer a little bit.
In order to get things working, I created a file called mainapp/jobs.py to contain my scheduled jobs. Then, in my apps.py module, I put from . import jobs in the ready method. Here's my entire apps.py file:
from django.apps import AppConfig
import os
class MainappConfig(AppConfig):
name = 'mainapp'
def ready(self):
from . import jobs
if os.environ.get('RUN_MAIN', None) != 'true':
jobs.start_scheduler()
(The RUN_MAIN check is because python manage.py runserver runs the ready method twice—once in each of two processes—but we only want to run it once.)
Now, here's what I put in my jobs.py file. First, the imports. You'll need to import Scheduler, threading and time as below. The F and UserHolding imports are just for what my job does; you won't import these.
from django.db.models import F
from schedule import Scheduler
import threading
import time
from .models import UserHolding
Next, write the function you want to schedule. The following is purely an example; your function won't look anything like this.
def give_admin_gold():
admin_gold_holding = (UserHolding.objects
.filter(inventory__user__username='admin', commodity__name='gold'))
admin_gold_holding.update(amount=F('amount') + 1)
Next, monkey-patch the schedule module by adding a run_continuously method to its Scheduler class. Do this by using the below code, which is copied verbatim from Oz123's answer.
def run_continuously(self, interval=1):
"""Continuously run, while executing pending jobs at each elapsed
time interval.
#return cease_continuous_run: threading.Event which can be set to
cease continuous run.
Please note that it is *intended behavior that run_continuously()
does not run missed jobs*. For example, if you've registered a job
that should run every minute and you set a continuous run interval
of one hour then your job won't be run 60 times at each interval but
only once.
"""
cease_continuous_run = threading.Event()
class ScheduleThread(threading.Thread):
#classmethod
def run(cls):
while not cease_continuous_run.is_set():
self.run_pending()
time.sleep(interval)
continuous_thread = ScheduleThread()
continuous_thread.setDaemon(True)
continuous_thread.start()
return cease_continuous_run
Scheduler.run_continuously = run_continuously
Finally, define a function to create a Scheduler object, wire up your job, and call the scheduler's run_continuously method.
def start_scheduler():
scheduler = Scheduler()
scheduler.every().second.do(give_admin_gold)
scheduler.run_continuously()

I recommend you use Celery's task management. You can refer this to set up this app (package if you're from javaScript background).
Once set, you can alter the code to:
#app.task
def check_shut_down():
if not some_fun():
# add task that'll run again after 2 secs
check_shut_down.delay((), countdown=3)
else:
# task completed; do something to notify yourself
return True

I can't comment on oz123's (https://stackoverflow.com/a/44897678/1108505) and Tanner Swett's (https://stackoverflow.com/a/60244694/5378866) excellent post, but as a final note I wanted to add that if you use Gunicorn and you have X number of workers, the section:
from django.apps import AppConfig
import os
class MainappConfig(AppConfig):
name = 'mainapp'
def ready(self):
from . import jobs
if os.environ.get('RUN_MAIN', None) != 'true':
jobs.start_scheduler()
will be executed that same number of times, launching X schedulers at the same time.
If we only want it to run only one instance (for example if you're going to create objects in the database), we would have to add in our gunicorn.conf.py file something like this:
def on_starting(server):
from app_project import jobs
jobs.start_scheduler()
And finally in the gunicorn call add the argument --preload

Here is my solution, with sources noted. This function will allow you to create a scheduler that you can start with your app, then add and subtract jobs at will. The check_interval variable allows you to trade-off between system resources and job execution timing.
from schedule import Scheduler
import threading
import warnings
import time
class RepeatTimer(threading.Timer):
"""Add repeated run of target to timer functionality. Source: https://stackoverflow.com/a/48741004/16466191"""
running: bool = False
def __init__(self, *args, **kwargs):
threading.Timer.__init__(self, *args, **kwargs)
def start(self) -> None:
"""Protect from running start method multiple times"""
if not self.running:
super(RepeatTimer, self).start()
self.running = True
else:
warnings.warn('Timer is already running, cannot be started again.')
def cancel(self) -> None:
"""Protect from running stop method multiple times"""
if self.running:
super(RepeatTimer, self).cancel()
self.running = False
else:
warnings.warn('Timer is already canceled, cannot be canceled again.')
def run(self):
"""Replace run method of timer to run continuously"""
while not self.finished.wait(self.interval):
self.function(*self.args, **self.kwargs)
class ThreadedScheduler(Scheduler, RepeatTimer):
"""Non-blocking scheduler. Advice taken from: https://stackoverflow.com/a/50465583/16466191"""
def __init__(
self,
run_pending_interval: float,
):
"""Initialize parent classes"""
Scheduler.__init__(self)
super(RepeatTimer, self).__init__(
interval=run_pending_interval,
function=self.run_pending,
)
def print_work(what_to_say: str):
print(what_to_say)
if __name__ == '__main__':
my_schedule = ThreadedScheduler(run_pending_interval=1)
job1 = my_schedule.every(1).seconds.do(print_work, what_to_say='Did_job1')
job2 = my_schedule.every(2).seconds.do(print_work, what_to_say='Did_job2')
my_schedule.cancel()
my_schedule.start()
time.sleep(7)
my_schedule.cancel_job(job1)
my_schedule.start()
time.sleep(7)
my_schedule.cancel()

User defined celery task class : init is getting called during import

I am trying to use celery task as a class and looking at following behavior. I guess that I missed something. Let me first tell you what I am trying to achieve :
1. Create a class with its init function which would be called only once by celery. This will setup required params for my class. I am gonna create a threadpool here.
2. Create instance of this celery task object in producer and put jobs in it.
To achieve the same I tried naive example mentioned on celery site and created a sample class. I am creating task using :
celery -c 1 -A proj worker --loglevel=debug
it seems to be working at first but then I observed that init of task is getting called at import in tester.py, I could stop this init in object usage by passing flag but init during import is a real concern here.
Can you please point me to correct usage of this example. I do not want init of task class to be called more than what I invoked using celery command. In real life scenario it would create unnecessary threads.
Also if possible, point me to right an example which is closest to my requirement mentioned above.
celery.py
from __future__ import absolute_import
from celery import Celery
app = Celery('proj',
broker='amqp://',
backend='amqp://',
include=['proj.tasks'])
# Optional configuration, see the application user guide.
app.conf.update(
CELERY_TASK_RESULT_EXPIRES=3600,
)
if __name__ == '__main__':
app.start()
tasks.py
from __future__ import absolute_import
from proj.celery import app
class NaiveAuthenticateServer(app.Task):
def __init__(self, celeryInst = 1):
if celeryInst == 1:
print "Hi, I am celery instant"
else :
print "Did you invoke me from command"
self.users = {'george': 'password'}
def run(self, username, password):
try:
return self.users[username] == password
except KeyError:
return False
tester.py
from proj import tasks
obj = tasks.NaiveAuthenticateServer(0)
res = obj.delay('hi', 'hello')
print res.get()
o/p of tester.py
Hi, I am celery instant
Did you invoke me from command
False

You should not create an instance of the task class yourself, but rather let celery do that for you automatically when the process starts up.
Therefore, you need to define a task function that uses the base class:
#app.task(base=NaiveAuthenticateServer)
def my_task(arg1, arg2):
print arg1, arg2
And then submit the task like this:
from proj import tasks
tasks.my_task.delay('hi', 'hello')

gevent + flask seems to block

I want to run a background worker in the same script as flask is running and flask seems to be blocking which I guess is understandable. Pretty much I want a script to check key system metrics every second so I don't want to use something like celery or a big queueing system to do it.
Simple code example
#!/usr/bin/env python
import gevent
from flask import Flask
class Monitor:
def __init__(self, opts):
self.opts = opts
def run(self):
print "do something: %i" % self.opts
gevent.sleep(1)
self.run()
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
threads = []
for mon in [1,2]:
monitor = Monitor(mon)
threads.append(gevent.spawn(monitor.run))
threads.append(gevent.spawn(app.run))
gevent.joinall(threads)
My output looks like
$ ./so.py
do something: 1
do something: 2
* Running on http://127.0.0.1:5000/
If I remove the theads.append for the app.run it runs fine. Is this possible to do or am I barking up the wrong tree?
Thanks

add in your script next two lines:
from gevent import monkey
monkey.patch_all()
before the line:
from flask import Flask
and all be OK

This is how I ended up handling the issue using apscheduler v2
#!/usr/bin/env python
import gevent
import time
from flask import Flask
from apscheduler.scheduler import Scheduler
sched = Scheduler()
sched.start()
class Monitor:
def __init__(self, opts):
self.opts = opts
def run(self):
#sched.interval_schedule(seconds=1)
def handle_run():
print "do something: %i" % self.opts
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World!'
if __name__ == '__main__':
for mon in [1,2]:
monitor = Monitor(mon)
monitor.run()
app.run(threaded=True)

try using the below:
class Monitor:
def init(self, opts):
self.opts = opts
def run(self):
While True:
print "do something: %i" % self.opts
gevent.sleep(1)
and then maybe don't joinall, since it doesn't seem like you really want to wait for them to complete before doing something else.
You may also need to put a try/except statement inside the while loop, and respawn if there is an error that kills the greenlet.

Celery not running with flask application

I'm using celery in my flask application but celery(3.1.8).This is my configuration with the flask application
celery.py
from __future__ import absolute_import
from celery import Celery
from cuewords.settings import CELERY_BROKER_URL,CELERY_RESULT_BACKEND
app = Celery('proj',
broker=CELERY_BROKER_URL,
backend=CELERY_RESULT_BACKEND)
app.conf.update(CELERY_TASK_RESULT_EXPIRES=3600)
if __name__ == '__main__':
app.start()
setting.py
CELERY_BROKER_URL='redis://localhost:6379/0'
CELERY_RESULT_BACKEND='redis://localhost:6379/0'
BROKER_TRANSPORT = 'redis'
api.py
class Webcontent(Resource):
def post(self,session=session):
args = self.parser.parse_args()
site_url = args["url"]
url_present=Websitecontent.site_url_present(session,site_url)
if site_url.strip() != "" and not url_present:
try:
#add data and commit
session.commit()
websitecontent=Websitecontent(params*)
websitecontent.update_url(id,session)
except:
session.rollback()
raise
finally:
session.close()
else:
return "No data created / data already present"
And in my model i'm adding a method to task
model.py
from cuewords.celery import app
class Websitecontent(Base):
#app.task(name='update_url')
def update_url(self,id,session):
...code goes here..
And this how i run the celery from command prompt
celery -A cuewords.celery worker
And i also using flower to monitor the task i can see a worker running but i couldn't see any task its empty .Any idea what im missing or doing wrong ..
Thanks

The problem is that your tasks never get imported into the Python runtime when running the worker(s). The celery command is your entry point. And you're telling Celery to import your cuewords.celery module because thats where you're app instance resides. However, this is where the chain of events ends and no further Python code is imported.
Now, the most common mistake is to import the tasks into the same module as the Celery app instance. Unfortunately this will result in two modules trying to import things from each other and will result in a circular import error. This is no good.
To get around this one could import the task functions into the Celery app module and register them without using the decorator style. For example:
from celery import Celery
from models import my_task
app = Celery()
app.task(name='my_task')(my_task)
This would remove the need to import the app instance in your model module.
However, you're using method tasks. Method tasks need to be treated differently than function tasks as noted here: http://docs.celeryproject.org/en/latest/reference/celery.contrib.methods.html. Method tasks are different from function tasks, because they are associated with an instance of an object. In other words, the function is a class function. So to use the previous style of registering tasks, you'd need an instance of the class first. To get around this you should consider making your tasks functions instead of methods.