I want to run a particular script every day in the morning for my rest api deployed on heroku the script itself it's only a database filler. how can that be done? I forgot to mention that i'm using django rest_framework
There are quite a few options. Here's what I do on an app I recently built that regularly scrapes a number of news websites.
In the root of my project I created a clock.py and then used apscheduler to create a scheduled job. Here's a simple version:
from apscheduler.schedulers.blocking import Blocking Scheduler
#IMPORTANT! Have to make sure things are working before this starts
from django import setup
setup()
sched = BlockingScheduler()
#sched.scheduled_job('interval', days=1):
def timed_job():
#do something here
...
I deployed my project on Heroku, so I then went through the process of spinning up a machine to run my clock.py file. Once it was up and running the process ran as intended.
There are a number of options besides interval you can use to specify when you want the process to run. You can use chron style scheduling if you want it to run at specific times and dates. You can read more about that here
Related
I'm hosting a Flask application in Heroku(free) which acts as an API and reads from an SQLite database file.
The way the project ran on my computer, I had scheduled Python scripts which would run every night and append new data to my SQLite database, which would then in turn be able to be used by my Flask Application.
However, hosted on Heroku, I don't think I will be able to run my Flask application and a python script 24/7. I know there is an alternate solution APScheduler on Flask, which would carry out tasks in Python functions in the Flask application. However, according to Heroku's free use guidelines, if there is no traffic to my page in 30 minutes, the application will "sleep." I'm assuming that means scheduled tasks will no longer work once the application is asleep, which defeats the purpose of using APScheduler.
Are there any alternatives I could use to go about this?
I would like to know how to automatically execute a code in Django within a defined period of time.
I'm building a web crawler that collects information and stores it in a JSON file and a function that reads the file and stores the file information in a SQLite database. This information is rendered and visible on my website. At the moment I have to run the crawler and the function that saves the data in a database with a click on a button, but that is very ineffective. It would be better if the database information were updated automatically, about every 6 hours (of course, only if the server is running).
If you want to keep the code on your webserver and you have the permissions then a CRON job (Linux) or Scheduled Task (Windows) will do what you want. Set your cron job to run every six hours and to call your Django script.
You can run the script as a Django command line, eg manage.py mycommand or run as a stand-alone PY file where you include the libs and run django.setup()
Or if you want the crawler to be run from within the context of your webserver then your cron job can initiate a GET request using HTTPIE. That gives you a nice API approach.
When I run this I have a flag in a database (a model called Job) that records what is running just in case my scheduled task is already running when the cron gets called, I have some very long-running asynchronous tasks and I tend to use cron rather than Celery.
If you are running your site on AWS or GCP then both have a cron equivalent. Eg you could create a GCP cloud scheduler link that fires off a GET to your webserver that triggers the code to run the crawler.
I am hoping to gain a basic understanding of scheduled task processes and why things like Celery are recommended for Flask.
My situation is a web-based tool which generates spreadsheets based on user input. I save those spreadsheets to a temp directory, and when the user clicks the "download" button, I use Flask's "send_from_directory" function to serve the file as an attachment. I need a background service to run every 15 minutes or so to clear the temp directory of all files older than 15 minutes.
My initial plan was a basic python script running in a while(True) loop, but I did some research to find what people normally do, and everything recommends Celery or other task managers. I looked into Celery and found that I also need to learn about redis, and I need to apparently host redis in a unix environment. This is a lot of trouble for a script that just deletes files every 15 minutes.
I'm developing my Flask app locally in Windows with the built-in development server and deploying to a virtual machine on company intranet with IIS. I'm learning as I go, so please explain why this much machinery is needed to regularly call a script that simply deletes things. It seems like a vast overcomplication, but as I said, I'm trying to learn as I go so I want to do/learn it correctly.
Thanks!
You wouldn't use Celery or redis for this. A cron job would be perfectly appropriate.
Celery is for jobs that need to be run asynchronously but in response to events in the main server processes. For example, if a sign up form requires sending an email notification, that would be scheduled and run via Celery so as not to block the main web response.
I have a web app running on heroku using flask and SQLAlchemy. I am now wondering how i can start a schedule task that runs daily and does some database related tasks (deleting some row if you need to know:)
The documentation on heroku recommends to use APScheduler but i would like to do it with Heroku-Scheduler. Dispite this decision i would like to know how i connect to my postgres database in this scheduler task. I could not find any example or hint for that.
thanks for your time
Torsten
Heroku scheduler will run any command you throw at it. The typical way would be to create a Python script/command as part of your flask app. You can do something similar to http://flask-script.readthedocs.org/en/latest/. Then within the scheduler you would schedule it similar to:
python manage.py mytask
Is it possible to run a script each time the dev server starts? Also at each deploy to google?
I want the application to fill the database based on what some methods returns.
Is there any way to do this?
..fredrik
I use appengine python with the django helper. As far as I know you cannot hook anything on the deploy, but you could put a call to check if you need to do your setup in the main function of main.py. This is how the helper initializes itself on the first request. I haven't looked at webapp in a while, but I assume main.py acts in a similar fashion for that framework.
Be aware that main is run on the first request, not when you first deploy. It will also happen if appengine starts up a new instance to handle load, or if all instances were stopped because of inactivity. So make sure you check to see if you need to do your initialization and then only do it if needed.
You can do this by writing a script in your favorite scripting language that performs the actions that you desire and then runs the dev server or runs appcfg.py update.
Try to make wrapper around the server runner and script that run deployment. So you will be able to run custom code when you need.
Warmup Requests in combination with min_idle_instances will probably work in your deploy usecase.