Do you think it is possible to use asyncio to run a task every n seconds in django, so that the main process wouldn't be blocked?
Something for example that would print every 5 min in console, like:
import asyncio
from random import randint
async def do_stuff(something, howmany):
for i in range(howmany):
print('We are doing {}'.format(something))
await asyncio.sleep(randint(0, 5))
if __name__ == '__main__':
loop = asyncio.get_event_loop()
work = [
asyncio.ensure_future(do_stuff('something', 5)),
]
loop.run_until_complete(asyncio.gather(*work))
It seems that django will stop working while the loop is running. Even if this can be made to work in development, how would it behave when the site goes live on something like apache or gunicorn?
While it would be possible to achieve this with a lot of hard work. Much simpler is to use the time honoured practice of using a cron job. See this: Using django for CLI tool
Nowadays a more popular approach (among django devs at any rate) is to use celery. Specifically celery beat
celery beat is a scheduler; It kicks off tasks at regular intervals,
that are then executed by available worker nodes in the cluster.
Related
I am using Celery in python. I have the following task:
#app.task
def data():
while 1:
response = requests.get(url, timeout=300).json()
db.colloectionName.insert_many(response)
sleep(10000)
This task gets data from a web server and saves it on MongoDB in a loop.
I have called it by the following code:
data.delay()
it works fine. But, I want to kill it by programming. I tried by data.AsyncResult(task_id).revoke()
but it does not work.
How can I kill a running task in celery?
Why are you doing data.AsyncResult? You can try doing something like this.
from celery.task.control import revoke
revoke(task_id, terminate=True)
Also, further details can be found here:
http://docs.celeryproject.org/en/latest/userguide/workers.html#revoke-revoking-tasks
You can try:
res = data.delay()
res.revoke()
but I don't understand the point of using Celery in your scenario. How does Celery help you if you're doing a while 1 loop?
Consider breaking it down to a small task that performs a single HTTP request and add Celery beat to call it every 10 seconds.
You can stop beats whenever you like.
You can use following code:
revoke(task_Id, terminate=True)
I'm building a django app where I use a camera to capture images, analyze them, store metadata and results of the analysis in a database, and finally present the data to users.
I'm considering using Celery to handle to background process of capturing images and then processing them:
app = Celery('myapp')
#app.task
def capture_and_process_images(camera):
while True:
image = camera.get_image()
process_image(image)
sleep(5000)
#app.task
def process_image(image):
# do some calculations
# django orm calls
# etc...
The first task will run perpetually, while the second should take ~20 seconds, so there will be multiple images being processed at once.
I haven't found any examples online of using Celery in this way, so I'm not sure if this is bad practice or not.
Can/should Celery be used to handle perpetually running tasks?
Thank you.
Running perpetual tasks in Celery is a done in practise. Take a look at daemonization, which essentially runs a permanent task without user interaction, so I wouldn't say there is anything wrong with running it permanently in your case.
Having celery task running infinitely is not seems like a good idea to me.
If you are going to capture images at some intervals I would suggest you to use some cron-like script getting an image every 5 seconds and launching celery task to process it.
Note also that it is a best practice to avoid synchronous subtasks in celery, see docs for more details.
I want to do a job every half an hour. My application is based on Flask, and running on Windows.
Now I create a task for the job, using Windows scheduler service.
I want to know if there is another way I can do cyclic tasks using Flask’s built-in functions...
Sorry for my poor English.
I want to know if there is another way I can do [periodic] tasks using Flask’s built-in functions.
Being somewhat a minimalist microframework, I don't think Flask has or ever will have a built-in feature to schedule periodic tasks.
The customary way is what you have already done, you write some Flask code that can be called as an HTTP endpoint or a script, then use a OS scheduling tool to execute it (e.g. Windows Task Scheduler or cron on UNIX/Linux)
Otherwise, Flask works well with other specialized libraries that take care of this, like Celery (periodic tasks) that takes care of those details and adds some features that may not be available otherwise.
from datetime import timedelta
CELERYBEAT_SCHEDULE = {
'every-half-hour': {
'task': 'tasks.your_task',
'schedule': timedelta(minutes=30),
'args': ('task_arg1', 'task_arg2')
},
}
CELERY_TIMEZONE = 'UTC'
I m not sure if this help, but I've been testing the schedule module and it's easy to use and it works well:
$pip install schedule
and this is a sample from the official documentation:
import schedule
import time
def job():
print("I'm working...")
schedule.every(30).minutes.do(job)
while True:
schedule.run_pending()
time.sleep(1)
Hope this help =)
I need explanation for scheduled task
I need to run task in every end of the day automatically like cron
I tried schedule app in my project
import schedule
import time
def job():
pprint.pprint("I'm working...")
schedule.every(10).minutes.do(job)
while True:
schedule.run_pending()
time.sleep(1)
when i add above code in project site was loading continuously
Question: Need to create task run automatically in background without user knowledge and without any command? It is possible?
I am new for python and django
Please suggest any idea for this task
If it is not an overkill - I recommend Celery.
It has "Celerybeat" which is like "cron"
Actually I think this is exactly what you need.
Usually you create a management command (https://docs.djangoproject.com/en/dev/howto/custom-management-commands/) and run it from a cron job.
I'm building a website which provides some information to the visitors. This information is aggregated in the background by polling a couple external APIs every 5 seconds. The way I have it working now is that I use APScheduler jobs. I initially preferred APScheduler because it makes the whole system more easy to port (since I don't need to set cron jobs on the new machine). I start the polling functions as follows:
from apscheduler.scheduler import Scheduler
#app.before_first_request
def initialize():
apsched = Scheduler()
apsched.start()
apsched.add_interval_job(checkFirstAPI, seconds=5)
apsched.add_interval_job(checkSecondAPI, seconds=5)
apsched.add_interval_job(checkThirdAPI, seconds=5)
This kinda works, but there's some trouble with it:
For starters, this means that the interval-jobs are running outside of the Flask context. So far this hasn't been much of a problem, but when calling an endpoint fails I want the system to send me an email (saying "hey calling API X failed"). Because it doesn't run within the Flask context however, it complaints that flask-mail cannot be executed (RuntimeError('working outside of application context')).
Secondly, I wonder how this is going to behave when I don't use the Flask built-in debug server anymore, but a production server with lets say 4 workers. Will it start every job four times then?
All in all I feel that there should be a better way of running these recurring tasks, but I'm unsure how. Does anybody out there have an interesting solution to this problem? All tips are welcome!
[EDIT]
I've just been reading about Celery with its schedules. Although I don't really see how Celery is different from APScheduler and whether it could thus solve my two points, I wonder if anyone reading this thinks that I should investigate more in Celery?
[CONCLUSION]
About two years later I'm reading this, and I thought I could let you guys know what I ended up with. I figured that #BluePeppers was right in saying that I shouldn't be tied so closely to the Flask ecosystem. So I opted for regular cron-jobs running every minute which are set using Ansible. Although this makes it a bit more complex (I needed to learn Ansible and convert some code so that running it every minute would be enough) I think this is more robust.
I'm currently using the awesome pythonr-rq for queueing a-sync jobs (checking APIs and sending emails). I just found out about rq-scheduler. I haven't tested it yet, but it seems to do precisely what I needed in the first place. So maybe this is a tip for future readers of this question.
For the rest, I just wish all of you a beautiful day!
(1)
You can use the app.app_context() context manager to set the application context. I imagine usage would go something like this:
from apscheduler.scheduler import Scheduler
def checkSecondApi():
with app.app_context():
# Do whatever you were doing to check the second API
#app.before_first_request
def initialize():
apsched = Scheduler()
apsched.start()
apsched.add_interval_job(checkFirstAPI, seconds=5)
apsched.add_interval_job(checkSecondAPI, seconds=5)
apsched.add_interval_job(checkThirdAPI, seconds=5)
Alternatively, you could use a decorator
def with_application_context(app):
def inner(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
with app.app_context():
return func(*args, **kwargs)
return wrapper
return inner
#with_application_context(app)
def checkFirstAPI():
# Check the first API as before
(2)
Yes it will still work. The sole (significant) difference is that your application will not be communicating directly with the world; it will be going through a reverse proxy or something via fastcgi/uwsgi/whatever. The only concern is that if you have multiple instances of the app starting, then multiple schedulers will be created. To manage this, I would suggest you move your backend tasks out of the Flask application, and use a tool designed for running tasks regularly (i.e. Celery). The downside to this is that you won't be able to use things like Flask-Mail, but imo, it's not too good to be so closely tied to the Flask ecosystem; what are you gaining by using Flask-Mail over a standard, non Flask, mail library?
Also, breaking up your application makes it much easier to scale up individual components as the capacity is required, compared to having one monolithic web application.