I have this application that works how I want to but now I want it to grab live data from some virtual machines every 5 minutes. In the code below I have it set to renew every ten seconds just to see if it works but nothing is happening. I am using time.sleep. What else am I missing?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
#app.route("/") # this sets the route to this page
def home():
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
Edit (this is an update with the suggestions, it's still not working as i'd like):
Edit 2 more info: I have one file with a function, grab_cpu, that does an api call to a vm and returns the percentage of usage. I have another file called test doc.html which just displays the html. From these responses i'm guessing I need to use some javascript and something with sockets. Can someone please drop a link to point me in the right direction?
import time
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
#app.route("/") # this sets the route to this page
def home():
starttime = time.time()
while True:
machines =["build05", "build06", "build07","build08", "build09", "build10", "build11", "build12","build14","build15", "winbuild10","winbuild11", "winbuild12", "winbuild13", "wbuild14", "wbuild15", "winbuild16", "winbuild17", "winbuild18"]
cpu_percentage =[grab_cpu("build05"), grab_cpu("build06"),grab_cpu("build07"),
grab_cpu("build08"), grab_cpu("build09"), grab_cpu("build10"), grab_cpu("build11"), grab_cpu("build12"), grab_cpu("build13"), grab_cpu("build14"), grab_cpu("build15"), grab_cpu("winbuild10"), grab_cpu("winbuild11"), grab_cpu("winbuild12"), grab_cpu("winbuild14"), grab_cpu("winbuild15"), grab_cpu("winbuild16"), grab_cpu("winbuild17"), grab_cpu("winbuild18")]
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
time.sleep(10.0 - ((time.time() - starttime) % 10.0))
app.run(use_reloader = True, debug = True)
Here is the page:
Thank you.
I recommend against using Flask to handle the scheduling. The intended flow for Flask requests is:
Receive an HTTP request from the browser
Generate a response as quickly as possible (ideally in a few milliseconds, but almost always less than a few seconds)
Return the response to the browser
The intention of the code above seems to be to use Flask to push updates down to the browser, but Flask can only respond to incoming requests, not force the browser to change.
For the use case you're describing, a simpler solution would be to handle the refresh logic in the browser.
For a very rudimentary example, put this into your testdoc.html template:
<script type="text/javascript">
setTimeout(function () {
location.reload();
}, 10 * 1000);
</script>
That will reload the page every 10 seconds, which will generate a new request to your Flask server and display updated information in your browser.
If you want to get fancier and avoid reloading the entire page, you can use JavaScript XmlHttpRequests or the more modern Fetch API to update specific elements of the page asynchronously.
Also, here's a suggested simplification of the Python code:
from flask import Flask, render_template
from testapi import grab_cpu
app = Flask(__name__)
build_machines = list(map(lambda i: 'build%02d' % i, range(5, 16)))
win_build_machines = list(map(lambda i: 'winbuild%02d' % i, range(10, 19)))
machines = build_machines + win_build_machines
# Handle HTTP GET requests to / route
#app.route("/", methods=['GET'])
def home():
cpu_percentage = list(map(lambda b: grab_cpu(b), machines))
return render_template('testdoc.html', len = len(machines), machines = machines, cpu_percentage = cpu_percentage)
app.run(use_reloader = True, debug = True)
Your code needs some work, Flask views are not meant to be declared inside a loop. I would suggest the following:
Remove the Flask view from inside the while loop.
Declare the server outside the loop too.
Write your code inside a function.
Run the while loop just calling the function to grab the information from your sources.
Based on what I think is just an example made on the run, I will also make some assumptions about your requirement:
This is not exactly your code.
Your code works perfectly, but this implementation is lacking.
Your project has some level of complexity.
You need to show this data somewhere else.
Base on these (and perhaps additional conditions), I would say you have two ways to achieve in an effective way what you need:
(Not recommended) Create a Cron Job and run everything in a dedicated script.
(My favorite) Encapsulate the logic of your script inside a Flask API call, a method, or a function, and declare it as a Celery task, scheduled to run every 5 seconds, updating your database, and use a view with some JS reactivity to show the data in realtime.
I want to create web server, and listen now.
Server must have different functions for each endpoint (and method).
I want to get (e.g. to variable) parameters (and data if POST)
Respond to get (and POST if its possible)
Respond in JSON
Someone can help me with this?
PS: I will be run it on Heroku, and send requests to it via Roblox's HttpService
Below see examples of each of your requirements using the Flask lightweight web framework.
After that is a link to a short description of how to deploy to Heroku.
# app.py
from flask import Flask
from flask import request, render_template
app = Flask(__name__)
#app.route('/test-get-request-parameters')
def test_get_request_parameters():
# 1. different function per endpoint
# 2. GET parameter to variable
# 3. respond to GET
var = request.args.get('some_request_variable')
return render_template('hello_world.html')
#app.route('/test-post-method',methods=['POST'])
def test_post_method():
# 2. receive POST data
# 3. respond to POST
print(request.get_json())
return 'hello, world!'
#app.route('/test-get-or-post', methods=['GET','POST'])
def test_get_or_post():
# 4. respond in JSON
if request.method == 'POST':
d = {'hello':'world'}
return d # this will be JSON response
return render_template('test.html')
To deploy to Heroku you need a Procfile with something like this in it:
web: gunicorn app:app
And you can follow these instructions: https://devcenter.heroku.com/articles/getting-started-with-python
I'm looking for some advice, or a relevant tutorial regarding the following:
My task is to set up a flask route that POSTs to API endpoint X, receives a new endpoint Y in X's response, then GETs from endpoint Y repeatedly until it receives a certain status message in the body of Y's response, and then returns Y's response.
The code below (irrelevant data redacted) accomplishes that goal in, I think, a very stupid way. It returns the appropriate data occasionally, but not reliably. (It times out 60% of the time.) When I console log very thoroughly, it seems as though I have bogged down my server with multiple while loops running constantly, interfering with each other.
I'll also receive this error occasionally:
SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /book
import sys, requests, time, json
from flask import Flask, request
# create the Flask app
app = Flask(__name__)
# main booking route
#app.route('/book', methods=['POST']) #GET requests will be blocked
def book():
# defining the api-endpoints
PRICING_ENDPOINT = ...
# data to be sent to api
data = {...}
# sending post request and saving response as response object
try:
r_pricing = requests.post(url = PRICING_ENDPOINT, data = data)
except requests.exceptions.RequestException as e:
return e
sys.exit(1)
# extracting response text
POLL_ENDPOINT = r_pricing.headers['location']
# setting data for poll
data_for_poll = {...}
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
# poll loop, looking for 'UpdatesComplete'
j = 1
poll_json = r_poll.json()
update_status = poll_json['Status']
while update_status == 'UpdatesPending':
time.sleep(2)
j = float(j) + float(1)
r_poll = requests.get(POLL_ENDPOINT, data = data_for_poll)
poll_json = r_poll.json()
update_status = poll_json['Status']
return r_poll.text
This is more of an architectural issue more than a Flask issue. Long-running tasks in Flask views are always a poor design choice. In this case, the route's response is dependent on two endpoints of another server. In effect, apart from carrying the responsibility of your app, you are also carrying the responsibility of another server.
Since the application's design seems to be a proxy for another service, I would recommend creating the proxy in the right way. Just like book() offers the proxy for PRICING_ENDPOINT POST request, create another route for POLL_ENDPOINT GET request and move the polling logic to the client code (JS).
Update:
If you cannot for some reason trust the client (browser -> JS) with the POLL_ENDPOINT information in a hidden proxy like situation, then maybe move the polling to a task runner like Celery or Python RQ. Although, it will introduce additional components to your application, it would be the right way to go.
Probably you get that error because of the HTTP connection time out with your API server that is looping. There are some standards for HTTP time connection and loop took more time that is allowed for the connection. The first (straight) solution is to "play" with Apache configs and increase the HTTP connection time for your wsgi. You can also make a socket connection and in it check the update status and close it while the goal was achieved. Or you can move your logic to the client side.
I am using Flask kvsession to avoid replay attacks, as the client side cookie based session used by Flask-login are prone to it.
Eg: If on /index page your cookie in the header is set for your app header like
myapp_session : 'value1'
and if you navigate to /important page you will get a new header like
myapp_session : 'value2' so if a hacker gets 'value1' he can perform replay attacks and misuse it, as it is never invalidated.
To solve this I am using flask-kvsession which stores the session cookie header value in a cache or some backend. SO basically only one myapp_session is generated and invalidated when you logout. But the problem is :-
__init__.py
from simplekv.memory.redisstore import RedisStore
import redis
store = RedisStore(redis.StrictRedis())
#store = memcache.Client(['127.0.0.1:11211'], debug =0)
store.ttl_support = True
app = create_app(__name__)
current_kvsession = KVSessionExtension(store, app)
If you look at the cleanup_session part of the code for kv-session
http://pythonhosted.org/Flask-KVSession/#flask_kvsession.KVSessionExtension.cleanup_sessions
It only deletes the expired sessions. But If I want to explicitly delete the value for the current myapp_session for a particular user on logout, how do I do that?
#app.before_request
def redirect_if_logout():
if request.path == url_for('logout'):
for key in app.kvsession_store.keys():
logger.debug(key)
m = current_kvsession.key_regex.match(key)
logger.debug('found %s', m)
app.kvsession_store.delete(key)
But this deletes all the keys as I don`t know what the unique key for the current session is.
Q2. Also, how to use memcache instead of redis as it doesn`t have the app.kvsession_store.keys() function and gives i/o error.
I think I just figured the 1st part of your question on how you can delete the specific key on logout.
As mentioned in the docs:
Internally, Flask-KVSession stores session ids that are serialized as
KEY_CREATED, where KEY is a random number (the sessions “true” id) and
CREATED a UNIX-timestamp of when the session was created.
Sample cookie value that gets created on client side (you can check with that cookie manager extenion for firefox):
c823af88aedaf496_571b3fd5.4kv9X8UvyQqtCtNV87jTxy3Zcqc
and session id stored in redis as key:
c823af88aedaf496_571b3fd5
So on logout handler, you just need to read the cookie value, split it and use the first part of the string:
Sample Code which worked for me:
import redis
from flask import Flask
from flask_kvsession import KVSessionExtension
from simplekv.memory.redisstore import RedisStore
store = RedisStore(redis.StrictRedis())
app = Flask(__name__)
KVSessionExtension(store, app)
#Logout Handler
#app.route('/logout', methods=['GET'])
def logout():
#here you are reading the cookie
cookie_val = request.cookies.get('session').split(".")[0]
store.delete(cookie_val)
and since you have added ttl_support:
store.ttl_support = True
It will match the TTL(seconds) value from permanent_session_lifetime, if you have set that in config file or in the beginning of your app.py file.
For example, in my application I have set in the beginning of app.py file as:
session.permanent = True
app.permanent_session_lifetime = timedelta(minutes=5)
now, when I logout, it deletes the key in redis but it will not be removed until TTL for that turns to 0 from 300 (5 Min as mentioned in permanent_session_lifetime value ).
If you want to remove it from redis immediately, for that you can manually change the app.permanent_session_lifetime to 1 second, which will in turn change TTL for redis.
import redis
import os
from flask import Flask
from flask_kvsession import KVSessionExtension
from simplekv.memory.redisstore import RedisStore
store = RedisStore(redis.StrictRedis())
app = Flask(__name__)
KVSessionExtension(store, app)
#Logout Handler
#app.route('/logout', methods=['GET'])
def logout():
cookie_val = request.cookies.get('session').split(".")[0]
app.permanent_session_lifetime = timedelta(seconds=1)
store.delete(cookie_val)
Using the above code, I was able to thwart session replay attacks.
and solution to your 2nd question:
3 possible mistakes that I can see are:
1: In the beginning of your code you have created:
store = RedisStore(redis.StrictRedis())
but in the loop you are using it as kvsession_store instead of just store:
app.kvsession_store.keys()
To use it without any errors/exceptions you can use it as store.keys() instead of app.store.keys():
from flask_kvsession import KVSessionExtension
from simplekv.memory.redisstore import RedisStore
store = RedisStore(redis.StrictRedis())
for key in store.keys():
print key
store.delete(key) is not deleting the all keys, you are running it inside the loop which is one by one deleting all keys.
I'm new to python and wondering if it is possible using BaseHTTPServer to store a global var that is accessible to all requests? Basically I have an async process that requires a POST back into the server as a separate request from the original request, that I would like to write back through to the original request, via a token I'd generate.
No.
To maintain state, you'll need a web framework that supports global variables across requests, or some kind of session management.
Flask is super easy to get up and running and has sessions available.
import flask
app = flask.Flask(__name__)
#app.route('/')
def index():
flask.session['post_token'] = MakeToken()
return '...Content...'
#app.route('/poster', methods=['POST'])
def poster():
if flask.session['post_token']:
DO STUFF HERE
# set the secret key. keep this really secret:
app.secret_key = 'A0Zr98j/3yX R~XHH!jxxxRT'