I am creating a fairly straightforward little app, and am trying to use apscheduler to send a text to a user at a certain time. It works wonderfully locally, but the clock process, as specified in the Procfile, crashes every time. The only information I get is "Process exited with status code 0. State changed from up to crashed." when I run heroku logs -t -p clock.
My Procfile looks as follows:
web: uvicorn remind_me.main:app --host=0.0.0.0 --port=${PORT:-8000} clock: python remind_me/views/register.py
I also ran the heroku ps:scale clock=1 command.
The code for my "view" is below:
from apscheduler.schedulers.background import BackgroundScheduler
from dateutil.parser import parse`
sched = BackgroundScheduler({'apscheduler.timezone': 'EST'})
def scheduled_job(msg, number, carrier):
send(msg, number, carrier)
#router.post('/')
#template()
async def home(request: Request):
vm = HomeViewModel(request)
await vm.load()
if vm.error:
return vm.to_dict()
msg, number, carrier, run_date = vm.task, vm.number, vm.carrier, vm.date_and_time
run_date = parse(run_date)
sched.add_job(scheduled_job, args=(msg,number,carrier), trigger='date', run_date=run_date)
try:
sched.start()
except:
pass
db_storage.store_events(vm.name, vm.number, vm.carrier, vm.task, vm.date_and_time)
return fastapi.responses.RedirectResponse(url="/", status_code=status.HTTP_302_FOUND)
I obviously left out some imports related to FastAPI and some stuff I was doing with the database, but I left those out for brevity. Everything works great locally, and all the pages work perfectly on Heroku, it's just that the clock process crashes every time.
I have also tried this with the blocking scheduler, but have had no luck, as that doesn't return and obviously I need the page to return/redirect.
I am wondering if it has something to do with the async aspect of the view itself, and if I might need to be APScheduler's async method (I can't remember its name offhand.). Any advice that you folks could give would be much appreciated. Thanks!
I have this application that works how I want to but now I want it to grab live data from some virtual machines every 5 minutes. In the code below I have it set to renew every ten seconds just to see if it works but nothing is happening. I am using time.sleep. What else am I missing?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
#app.route("/") # this sets the route to this page
def home():
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
Edit (this is an update with the suggestions, it's still not working as i'd like):
Edit 2 more info: I have one file with a function, grab_cpu, that does an api call to a vm and returns the percentage of usage. I have another file called test doc.html which just displays the html. From these responses i'm guessing I need to use some javascript and something with sockets. Can someone please drop a link to point me in the right direction?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
#app.route("/") # this sets the route to this page
def home():
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
app.run(use_reloader = True, debug = True)
Here is the page:
Thank you.
I recommend against using Flask to handle the scheduling. The intended flow for Flask requests is:
Receive an HTTP request from the browser
Generate a response as quickly as possible (ideally in a few milliseconds, but almost always less than a few seconds)
Return the response to the browser
The intention of the code above seems to be to use Flask to push updates down to the browser, but Flask can only respond to incoming requests, not force the browser to change.
For the use case you're describing, a simpler solution would be to handle the refresh logic in the browser.
For a very rudimentary example, put this into your testdoc.html template:
<script type="text/javascript">
setTimeout(function () {
location.reload();
}, 10 * 1000);
</script>
That will reload the page every 10 seconds, which will generate a new request to your Flask server and display updated information in your browser.
If you want to get fancier and avoid reloading the entire page, you can use JavaScript XmlHttpRequests or the more modern Fetch API to update specific elements of the page asynchronously.
Also, here's a suggested simplification of the Python code:
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
build_machines = list(map(lambda i: 'build%02d' % i, range(5, 16)))
win_build_machines = list(map(lambda i: 'winbuild%02d' % i, range(10, 19)))
machines = build_machines + win_build_machines
# Handle HTTP GET requests to / route
#app.route("/", methods=['GET'])
def home():
cpu_percentage = list(map(lambda b: grab_cpu(b), machines))
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
Your code needs some work, Flask views are not meant to be declared inside a loop. I would suggest the following:
Remove the Flask view from inside the while loop.
Declare the server outside the loop too.
Write your code inside a function.
Run the while loop just calling the function to grab the information from your sources.
Based on what I think is just an example made on the run, I will also make some assumptions about your requirement:
This is not exactly your code.
Your code works perfectly, but this implementation is lacking.
Your project has some level of complexity.
You need to show this data somewhere else.
Base on these (and perhaps additional conditions), I would say you have two ways to achieve in an effective way what you need:
(Not recommended) Create a Cron Job and run everything in a dedicated script.
(My favorite) Encapsulate the logic of your script inside a Flask API call, a method, or a function, and declare it as a Celery task, scheduled to run every 5 seconds, updating your database, and use a view with some JS reactivity to show the data in realtime.
Setup :
Language i am using is Python
I am running a flask app with threaded =True and inside the flask app when an endpoint is hit, it starts a thread and returns a thread started successfully with status code 200.
Inside the thread, there is re.Search happening, which takes 30-40 seconds(worst case) and once the thread is completed it hits a callback URL completing one request.
Ideally, the Flask app is able to handle concurrent requests
Issue:
When the re.search is happening inside the thread, the flask app is not accepting concurrent requests. I am assuming some kind of thread locking is happening and unable to figure out.
Question :
is it ok to do threading(using multi processing) inside flask app which has "threaded = True"?
When regex is happening does it do any thread locking?
Code snippet: hit_callback does a post request to another api which is not needed for this issue.
import threading
from flask import *
import re
app = Flask(__name__)
#app.route(/text)
def temp():
text = request.json["text"]
def extract_company(text)
attrib_list = ["llc","ltd"] # realone is 700 in length
entity_attrib = r"((\s|^)" + r"(.)?(\s|$)|(\s|^)".join(attrib_list) + r"(\s|$))"
raw_client = re.search("(.*(?:" + entity_attrib + "))", text, re.I)
hit_callback(raw_client)
extract_thread = threading.Thread(target=extract_company, )
extract_thread.start()
return jsonify({"Response": True}), 200
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=4557, threaded=True)
Please read up on the GIL - basically, Python can only execute ONE piece of code at the same time - even if in threads. So your re-search runs and blocks all other threads until run to completion.
Solutions: Use Gunicorn etc. pp to run multiple flask processes - do not attempt to do everything in one flask process.
To add something: Also your design is problematic - 40 seconds or so for a http answer is way to long. A better design would have at least two services: a web server and a search service. The first would register a search and give back an id. Your search service would communicate asynchronously with the web service and return resullt +id when the result is ready. Your clients could pull the web service until they get a result.
I have a web endpoint for users to upload file.
When the endpoint receives the request, I want to run a background job to process the file.
Since the job would take time to complete, I wish to return the job_id to the user to track the status of the request while the job is running in background.
I am wondering if asyncio would help in this case.
import asyncio
#asyncio.coroutine
def process_file(job_id, file_obj):
<process the file and dump results in db>
#app.route('/file-upload', methods=['POST'])
def upload_file():
job_id = uuid()
process_file(job_id, requests.files['file']) . # I want this call to be asyc without any await
return jsonify('message' : 'Request received. Track the status using: " + `job_id`)
With the above code, process_file method is never called. Not able to understand why.
I am not sure if this is the right way to do it though, please help if I am missing something.
Flask doesn't support async calls yet.
To create and execute heavy tasks in background you can use https://flask.palletsprojects.com/en/1.1.x/patterns/celery/ Celery library.
You can use this for reference:
Making an asynchronous task in Flask
Official documentation:
http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#installing-celery
Even though you wrote #asyncio.coroutine() around a function it is never awaited which tells a function to return result.
Asyncio is not good for such kind of tasks, because they are blocking I/O. It is usually used to make function calls and return results fast.
As #py_dude mentioned, Flask does not support async calls. If you are looking for a library that functions and feels similar to Flask but is asynchronous, I recommend checking out Sanic. Here is some sample code:
from sanic import Sanic
from sanic.response import json
app = Sanic()
#app.route("/")
async def test(request):
return json({"hello": "world"})
if __name__ == "__main__":
app.run(host="0.0.0.0", port=8000)
Updating your database asynchronously shouldn't be an issue; refer to here to find asyncio-supported database drivers. For processing your file, check out aiohttp. You can run your server extremely fast on a single thread without any hickup if you do so asynchronously.
Flask is a single thread web server. But I want to make it won't block when handle some time consuming request.
For example:
from flask import Flask
import time
import sys
app = Flask(__name__)
#app.route("/")
def hello():
print "request"
sys.stdout.flush()
for _ in range(10000000):
for j in range(10000000):
i = 1
return "Hello World!"
if __name__ == "__main__":
app.run(debug=True)
I want when every client request to server, it always output "request" on console immediately. I have try gunicorn and run with gunicorn -k gevent -w 4 a:app but it still appears synchronous.
This snippet is a good starting point.
You also should look into Celery or RQ, they're the right thing to use for larger projects, more importantly they're not Flask-specific.
They also have Flask integration each, Flask-Celery and Flask-RQ.
I believe you are asking about something called "streaming". For Flask this can be accomplished using generator functions and the yield keyword.
Streaming is covered in more detail in the official Flask documentation, have a look here.