I have this application that works how I want to but now I want it to grab live data from some virtual machines every 5 minutes. In the code below I have it set to renew every ten seconds just to see if it works but nothing is happening. I am using time.sleep. What else am I missing?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
#app.route("/") # this sets the route to this page
def home():
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
Edit (this is an update with the suggestions, it's still not working as i'd like):
Edit 2 more info: I have one file with a function, grab_cpu, that does an api call to a vm and returns the percentage of usage. I have another file called test doc.html which just displays the html. From these responses i'm guessing I need to use some javascript and something with sockets. Can someone please drop a link to point me in the right direction?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
#app.route("/") # this sets the route to this page
def home():
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
app.run(use_reloader = True, debug = True)
Here is the page:
Thank you.
I recommend against using Flask to handle the scheduling. The intended flow for Flask requests is:
Receive an HTTP request from the browser
Generate a response as quickly as possible (ideally in a few milliseconds, but almost always less than a few seconds)
Return the response to the browser
The intention of the code above seems to be to use Flask to push updates down to the browser, but Flask can only respond to incoming requests, not force the browser to change.
For the use case you're describing, a simpler solution would be to handle the refresh logic in the browser.
For a very rudimentary example, put this into your testdoc.html template:
<script type="text/javascript">
setTimeout(function () {
location.reload();
}, 10 * 1000);
</script>
That will reload the page every 10 seconds, which will generate a new request to your Flask server and display updated information in your browser.
If you want to get fancier and avoid reloading the entire page, you can use JavaScript XmlHttpRequests or the more modern Fetch API to update specific elements of the page asynchronously.
Also, here's a suggested simplification of the Python code:
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
build_machines = list(map(lambda i: 'build%02d' % i, range(5, 16)))
win_build_machines = list(map(lambda i: 'winbuild%02d' % i, range(10, 19)))
machines = build_machines + win_build_machines
# Handle HTTP GET requests to / route
#app.route("/", methods=['GET'])
def home():
cpu_percentage = list(map(lambda b: grab_cpu(b), machines))
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
Your code needs some work, Flask views are not meant to be declared inside a loop. I would suggest the following:
Remove the Flask view from inside the while loop.
Declare the server outside the loop too.
Write your code inside a function.
Run the while loop just calling the function to grab the information from your sources.
Based on what I think is just an example made on the run, I will also make some assumptions about your requirement:
This is not exactly your code.
Your code works perfectly, but this implementation is lacking.
Your project has some level of complexity.
You need to show this data somewhere else.
Base on these (and perhaps additional conditions), I would say you have two ways to achieve in an effective way what you need:
(Not recommended) Create a Cron Job and run everything in a dedicated script.
(My favorite) Encapsulate the logic of your script inside a Flask API call, a method, or a function, and declare it as a Celery task, scheduled to run every 5 seconds, updating your database, and use a view with some JS reactivity to show the data in realtime.
Related
I want to test how much http request an easy ping-pong service can handle in a certain amount of time. I have already implemented it in Java and Go and it works fine. But when testing it with python one single ping-pong cycle needs a bit more than 2s on my machine and that's enormously long. In the other languages I have nearly 1 per each milliseconds. More than 2000 times what I get with python... how can that be? I think the http server needs to be build up each time again with python, but is it really that and how could I fix it?
And of course the code to my little python scripts is down below:
import requests
add = 'aaaaaaaaaa'
sum = ''
def getTime(i):
site_request = requests.post("http://localhost:8082/ping", data=i)
site_response = str(site_request.content)
return site_response
def printToFile():
return
for x in range(5):
print(getTime(sum))
sum += add
from flask import Flask
from flask import request
import requests
import time
app = Flask(__name__)
#app.route("/ping", methods=['POST'])
def region():
data = request.get_data()
start = time.time()
site_request = requests.post("http://localhost:8080/pong", data=data)
site_response = str(site_request.content)
end = time.time()
cl = int(site_request.headers['content-length'])
if cl != len(data):
return end - start + ", " + cl + "," + 500
return str(end - start) + "," + str(cl) + "," + str(200)
app.run(host='127.0.0.1', port= 8082)
from flask import Flask
from flask import request
app = Flask(__name__)
#app.route("/pong", methods=['POST'])
def region():
data = request.get_data()
return data
app.run(host='127.0.0.1', port= 8080)
The problem you are facing here is that the Python requests library is a synchronous library.
While your Flask app (if set up correctly) will be capable of handling many requests at once, your code used to send the requests will only send one at a time and will block until each one is finished in sequence.
A better test of the speed of your server will be to use Python's somewhat newer async features utilizing a library like asyncio
Give that a try as the method for firing off your test requests and see if it is still slow. If so, at least you will have ruled out one potential issue!
As an example of an excellent library you could try that utilizes asyncio, look at AIOHTTP.
Setup :
Language i am using is Python
I am running a flask app with threaded =True and inside the flask app when an endpoint is hit, it starts a thread and returns a thread started successfully with status code 200.
Inside the thread, there is re.Search happening, which takes 30-40 seconds(worst case) and once the thread is completed it hits a callback URL completing one request.
Ideally, the Flask app is able to handle concurrent requests
Issue:
When the re.search is happening inside the thread, the flask app is not accepting concurrent requests. I am assuming some kind of thread locking is happening and unable to figure out.
Question :
is it ok to do threading(using multi processing) inside flask app which has "threaded = True"?
When regex is happening does it do any thread locking?
Code snippet: hit_callback does a post request to another api which is not needed for this issue.
import threading
from flask import *
import re
app = Flask(__name__)
#app.route(/text)
def temp():
text = request.json["text"]
def extract_company(text)
attrib_list = ["llc","ltd"] # realone is 700 in length
entity_attrib = r"((\s|^)" + r"(.)?(\s|$)|(\s|^)".join(attrib_list) + r"(\s|$))"
raw_client = re.search("(.*(?:" + entity_attrib + "))", text, re.I)
hit_callback(raw_client)
extract_thread = threading.Thread(target=extract_company, )
extract_thread.start()
return jsonify({"Response": True}), 200
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=4557, threaded=True)
Please read up on the GIL - basically, Python can only execute ONE piece of code at the same time - even if in threads. So your re-search runs and blocks all other threads until run to completion.
Solutions: Use Gunicorn etc. pp to run multiple flask processes - do not attempt to do everything in one flask process.
To add something: Also your design is problematic - 40 seconds or so for a http answer is way to long. A better design would have at least two services: a web server and a search service. The first would register a search and give back an id. Your search service would communicate asynchronously with the web service and return resullt +id when the result is ready. Your clients could pull the web service until they get a result.
In the process of using Eventbrite's API to power a website with their events, however, I am stuck on how to implement project.
Currently, I am wanting to filter events via dates and show them on the front page, however, Eventbrite API limits to only 50 items per page. The API request pushes out 86 pages with 50 items each.
How can I go about presenting those events at once while lowering latency?
What I currently have takes 104 seconds to respond and render results to page. I'm scratching the surface on Big O Notation theory, however, I think I'm using O(n²).
here's the code:
routes.py:
from flask import Flask, render_template
from eventbrite import Eventbrite
from datetime import datetime
import requests, json
app = Flask(__name__)
# TODO: Remove outh key in file and register in sys
oKey = ‘<oauth-key>’
o_token_url = '&token={0}'.format(oKey)
#app.route("/")
def index():
base_url = 'https://www.eventbriteapi.com/v3/'
regions_url = base_url + 'system/regions/?token={0}'.format(oKey)
regions_result = requests.get(regions_url)
regions_obj = regions_result.json()
e_url = base_url + 'events/search/?token={0}&location.address=<city>&start_date.range_start=2018-07-02T12%3A00%3A00Z&page=2'.format(oKey)
e_result = requests.get(e_url)
e_obj = e_result.json()
list_events = []
for pg in range(1, e_obj['pagination']['page_count']):
print(pg)
startTime = datetime.now()
e_url = base_url + 'events/search/?token={0}&location.address=<city>&start_date.range_start=2018-07-02T12%3A00%3A00Z&page={1}'.format(oKey, pg)
e_result = requests.get(e_url)
e_obj = e_result.json()
for i in e_obj['events']:
an_event = {}
an_event['name'] = i['name']['text']
list_events.append(htx_event)
print(datetime.now() - startTime)
return render_template('index.html', events=list_events)
if __name__ == "__main__":
app.run(host='0.0.0.0')
index.html:
{% for event in events %}
{{event['name']}}<br>
{% endfor %}
I am aware of EB API batched requests endpoint however, I can't wrap my head around on how to send the list of url dictionaries with requests.
Any tips on how to optimize the code above would be appreciated. I figured I can limit the amount of events that are shown at any given time, however, interested in how others would approach this issue.
It takes long time not because it's a CPU intensive operation (which could be solved with algorithm optimization), but because each HTTP request takes time. To mitigate that, you can make requests concurrent (i.e. simultaneous). To further reduce latency, you can tune re-use of TCP connections so you wouldn't waste time to create a TCP connection for each HTTP request.
I have an odd problem at which when I run my code below in PyCharm or through the console (python script.py) the flask server takes an extremely long time to boot meaning when trying to access it it shows no content for a good few minutes.
import threading
from flask import render_template, request, logging, Flask, redirect
def setupFlask():
appn = Flask(__name__)
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
#appn.route('/')
def page():
return render_template('index.html')
#appn.route('/submit', methods=['POST'])
def submit():
token = request.form['ID']
ID = token
return redirect('/')
appn.run()
a = threading.Thread(target=setupFlask)
a.daemon = True
a.start()
while True:
pass
The odd thing is that when I run the same code above in the PyCharm debugger, the flask server takes about 5 seconds to boot, massively quicker than the few minutes it takes when run in the console. I would love that kind of speed when running the script normally and cant find a solution due to the problem fixing itself when in debugger!
This code snippet is part of a larger application, however I have adapted it to be run on its own and the same problem occurs.
I am not running in a virtualenv.
All help appreciated.
EDIT: The index.html document is very basic and only contains a few scripts and elements therefore I could not see it taking a long time to load.
The problem was with your Flask installation, but there's another one. You should not wait for your thread with a while loop. The better way is to join your thread, like this:
a = threading.Thread(target=setupFlask)
a.daemon = True
a.start()
a.join()
For a while now I am trying to learn how to use Ajax with Flask. On the official website of flask there is an example:
from flask import Flask, jsonify, render_template, request
app = Flask(__name__)
#app.route('/_add_numbers')
def add_numbers():
a = request.args.get('a', 0, type=int)
b = request.args.get('b', 0, type=int)
return jsonify(result=a + b)
#app.route('/')
def index():
return render_template('index.html')
It works for me well. But I am looking for the following program:
a jQuery code sends an initial number to the python app
the python app stores the number and responds 'received: [the number]'
while true: the python app waits for requests 'increase' for which it adds 1 to the number and returns it
The jQuery part doesn't matter, I can do that but I am not sure how to implement the python part:
#app.route('/_inc_number')
def inc_number():
n = request.args.get('n', 0, type=int)
while true:
req = request.args.get('req', 0, type=string)
if req == 'increase':
n = n + 1
return n #It exits the function. how respond without quit?
Please explain me how I can data back? I am new to both Ajax and Flask too, and I suspect that it is not "real" ajax...Is that right? How would you implement a simple function with the same functionality in flask?
I think what you are missing is that each time the client requests a number increase there is an independent request. Your inc_number handler would be coded as follows:
#app.route('/_inc_number')
def inc_number():
n = request.args.get('n', 0, type=int)
n = n + 1
return n
Then from the jQuery side you have to invoke an independent Ajax request each time you want to increase the number.
Note that with this type of solution the jQuery app is the one that keeps track of the current value of the counter, and it has to send it to Flask with each request. Another possibility would be to have the Flask side remember the number in a user session. For that type of solution your Flask app would have two view functions:
from flask import session
# you need to set a SECRET_KEY in configuration, used to sign the user session
#app.route('/set_number')
def set_number():
# store n in the user session
session['n'] = request.args.get('n', 0, type=int)
#app.route('/inc_number')
def inc_number():
session['n'] = session['n'] + 1
return session['n']
With this solution now jQuery can set the number and not have to send it every time when it invokes inc_number.
I hope this helps.
HTTP requests don't have a memory, they are independent of each other. It means when your Python app gets a request, it does something, sends a response immediately and quits. That's the nature of HTTP.
If you want something persistent (like your number) that lives through more requests, you need:
Persistent storage on the server. It can be a file, a database, or in case of Flask simply an object (variable) in memory.
Identify the user between separate requests. That's what session handling and cookies are for.
A very primitive method (shouldn't be used on production system):
persistent storage: create a global dict (called num) in main()
in index():
create a random session identifier (si)
set num[si] = 0
send si as a cookie
in inc_number() use si from cookie (sent back by the browser) to increase the appropriate num[si]