I'm trying to serve a Flask app and would like to reload a pickle file at a specific time window (e.g. 9AM every day). I've tried to put a while loop into the end of my flask app with a time counter, but this ends up hanging my application. Currently the set up is...
# main.wsgi
from main import app as application
# main.py
data = pickle.load("/path/to/pickle.file")
#app.route("/")
def func():
return render_template("base.html", data_to_serve = data)
# Can I write something here to reload the data at specific time points?
I am assuming the goal here is to do what I call a "poor man's cache". Ideally you'd opt to use something like pymemcache and Flask's cache utils, but the snippet below will accomplish what you want. You can refactor this if you want to reload the pickle each time; kind of would be defeating the purpose I think.
Additionally, note that I have used a range of time to return the pickle data; 9 AM to 12 PM. You can also do something like if now.time() == time(hour=9) to accomplish what you want.
import pickle
from datetime import datetime, time
cached_data = pickle.load("/path/to/pickle.file")
START_TIME = time(hour=9)
END_TIME = time(hour=12) # Can also use something like timedelta
def in_range():
now = datetime.now()
if START_TIME <= now.time() <= END_TIME:
return True
return False
app.route("/")
def func():
if in_range():
return render_template("base.html", data_to_serve = cached_data)
# else do normal business
data = 'compute new data...'
return render_template("base.html", data_to_serve = data)
Happy coding!
You want to reload the data at specific point of time then you have 2 options:
Do it from the client size using javascript and ajax requests using some timer.
Use web sockets. There is a library for flask called flask-socketio. There is an example on how to use it.
Related
I have a script that continually runs, processing data that it gets from an external device. The core logic follows something like:
from external_module import process_data, get_data, load_interesting_things
class MyService:
def __init__(self):
self.interesting_items = load_interesting_things()
self.run()
def run(self):
try:
while True:
data = get_data()
for item in self.interesting_items:
item.add_datapoint(process_data(data, item))
except KeyboardInterrupt:
pass
I would like to add the ability to request information for the various interesting things via a RESTful API.
Is there a way in which I can add something like a Flask web service to the program such that the web service can get a stat from the interesting_items list to return? For example something along the lines of:
#app.route("/item/<idx>/average")
def average(idx: int):
avg = interesting_items[idx].getAverage()
return jsonify({"average":avg})
Assuming there is the necessary idx bounds checking and any appropriate locking implemented.
It does not have to be Flask, but it should be light weight. I want to avoid using a database. I would prefer to use a webservice, but if it is not possible without completely restructuring the code base I can use a socket instead, but this is less preferable.
The server would be running on a local network only and usually only handling a single user, sometimes it may have a few.
I needed to move the run() method out of the __init__() method, so that I could have a global reference to the service, and start the run method in a separate thread. Something along the lines of:
service = MyService()
service_thread = threading.Thread(target=service.run, daemon=True)
service_thread.start()
app = flask.Flask("appname")
...
#app.route("/item/<idx>/average")
def average(idx: int):
avg = service.interesting_items[idx].getAverage()
return jsonify({"average":avg})
...
app.run()
I have this following python code for a Flask server. I am trying to have this part of the code list all my vehicles that match the horsepower that I put in through my browser. I want it to return all the car names that match the horsepower, but what I have doesn't seem to be working? It returns nothing. I know the issue is somewhere in the "for" statement, but I don't know how to fix it.
This is my first time doing something like this and I've been trying multiple things for hours. I can't figure it out. Could you please help?
from flask import Flask
from flask import request
import os, json
app = Flask(__name__, static_folder='flask')
#app.route('/HORSEPOWER')
def horsepower():
horsepower = request.args.get('horsepower')
message = "<h3>HORSEPOWER "+str(horsepower)+"</h3>"
path = os.getcwd() + "/data/vehicles.json"
with open(path) as f:
data = json.load(f)
for record in data:
horsepower=int(record["Horsepower"])
if horsepower == record:
car=record["Car"]
return message
The following example should meet your expectations.
from flask import Flask
from flask import request
import os, json
app = Flask(__name__)
#app.route('/horsepower')
def horsepower():
# The type of the URL parameters are automatically converted to integer.
horsepower = request.args.get('horsepower', type=int)
# Read the file which is located in the data folder relative to the
# application root directory.
path = os.path.join(app.root_path, 'data', 'vehicles.json')
with open(path) as f:
data = json.load(f)
# A list of names of the data sets is created,
# the performance of which corresponds to the parameter passed.
cars = [record['Car'] for record in data if horsepower == int(record["Horsepower"])]
# The result is then output separated by commas.
return f'''
<h3>HORSEPOWER {horsepower}</h3>
<p>{','.join(cars)}<p>
'''
There are many different ways of writing the loop. I used a short variant in the example. In more detail, you can use these as well.
cars = []
for record in data:
if horsepower == int(record['Horsepower']):
cars.append(record['Car'])
As a tip:
Pay attention to when you overwrite the value of a variable by using the same name.
My Flask app, will get data from an url only from certain time. If it is outside the range of time, it will used the last query data from the url that save in Cache. Outside the range of time, the url will return no data. Thus, I want to reuse the last data in cache
from flask_app import app
from flask import jsonify,abort,make_response,request
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.cache import Cache
from datetime import datetime, time
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)
#app.route('/top', methods=['GET'])
#app.cache.cached(timeout=60)
def top():
now = datetime.now()
now_time = now.time()
if now_time >= time(10,30) and now_time <= time(16,30):
print "within time, use data save in cache"
# function that use last data query from url, save in cache
else:
page = requests.get('http://www.abvfc.com')
data = re.findall(r'items:(.*)',page.content)
return jsonify(data)
The problem is I can't get the last Cache data. If there is no access to the api /top in the last 60 seconds, there will be no data.
Cache the data one minutes before the url return no data at 16.30
User can use the cache data outside range of time
I am not familiar with cache, so may be my current idea is not the best way.
i am not a flask user but perhaps this is your wanted decorator
def timed_cache(cache_time:int, nullable:bool=False):
result = ''
timeout = 0
def decorator(function):
def wrapper(*args, **kwargs):
nonlocal result
nonlocal timeout
if timeout <= time.time() or not (nullable or result):
result = function(*args, **kwargs)
timeout = time.time() + cache_time
return result
return wrapper
return decorator
Assuming that you only want your cache to be working between 10:30h and 16:30h, I'd change a bit the approach you are using with the cache.
The problem I see with your current implementation is, apart from the undesirable cache expiration during the critical time range, that you'll also need to disable it at the moments you want to actually return an updated response.
That said, I'll use a different strategy: saving to cache every time I compute an updated response and retrieving the response from the cache in the critical time period.
Regarding the Flask-Cache documentation and the information in this tutorial, I'd modify your code as follows:
from flask_app import app
from flask import jsonify,abort,make_response,request
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.cache import Cache
from datetime import datetime, time
app.config['CACHE_TYPE'] = 'simple'
app.cache = Cache(app)
#app.route('/top', methods=['GET'])
def top():
now = datetime.now()
now_time = now.time()
if now_time >= time(10,30) and now_time <= time(16,30):
print "within time, use data save in cache"
return app.cache.get('last_top_response')
page = requests.get('http://www.abvfc.com')
data = re.findall(r'items:(.*)',page.content)
response = jsonify(data)
app.cache.set('last_top_response', response)
return response
I hope this to suit your needs
I have python file called testing_file.py:
from datetime import datetime
import MySQLdb
# Open database connection
class DB():
def __init__(self, server, user, password, db_name):
db = MySQLdb.connect(server, user, password, db_name )
self.cur = db.cursor()
def time_statistic(self, start_date, end_date):
time_list = {}
sql = "SELECT activity_log.datetime, activity_log.user_id FROM activity_log"
self.cur.execute(sql)
self.date_data = self.cur.fetchall()
for content in self.date_data:
timestamp = str(content[0])
datetime_object = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S')
timestamps = datetime.strftime(datetime_object, "%Y-%m-%d")
if start_dt <= timestamps and timestamps <= end_dt:
if timestamps not in time_list:
time_list[timestamps]=1
else:
time_list[timestamps]+=1
return json.dumps(time_list)
start_date = datetime.strptime(str('2017-4-7'), '%Y-%m-%d')
start_dt = datetime.strftime(start_date, "%Y-%m-%d")
end_date = datetime.strptime(str('2017-5-4'), '%Y-%m-%d')
end_dt = datetime.strftime(end_date, "%Y-%m-%d")
db = DB("host","user_db","pass_db","db_name")
db.time_statistic(start_date, end_date)
I want to access the result (time_list) thru API using Flask. This is what i've wrote so far, doesn't work and also I've tried another way:
from flask import Flask
from testing_api import *
app = Flask(__name__)
#app.route("/")
def get():
db = DB("host","user_db","pass_db","db_name")
d = db.time_statistic()
return d
if __name__ == "__main__":
app.run(debug=True)
Question: This is my first time work with API and Flask. Can anyone please help me thru this. Any hints are appreciated. Thank you
I've got empty list as result {}
There are many things wrong with what you are doing.
1.> def get(self, DB) why self? This function does not belong to a class. It is not an instance function. self is a reference of the class instance when an instance method is called. Here not only it is not needed, it is plain and simple wrong.
2.> If you look into flask's routing declaration a little bit, you will see how you should declare a route with parameter. This is the link. In essence you should something like this
#app.route("/path/<variable>")
def route_func(variable):
return variable
3.> Finally, one more thing I would like to mention, Please do not call a regular python file test_<filename>.py unless you plan to use it as a unit testing file. This is very confusing.
Oh, and you have imported DB from your module already no need to pass it as a parameter to a function. It should be anyway available inside it.
There are quite a few things that are wrong (ranging from "useless and unclear" to "plain wrong") in your code.
wrt/ the TypeError: as the error message says, your get() function expects two arguments (self and DB) which won't be passed by Flask - and are actually not used in the function anyway. Remove both arguments and you'll get rid of this error - just to find out you now have a NameError on the first line of the get() function (obviously since you didn't import time_statistic nor defined start_date and end_date).
I need to compare the performances of a Flask application that uses jinja2.Template.render against jinja2.Template.stream to be sure that there is no loss of performances using the streaming of templates.
My (very simple) idea was to print the timestamp before and after the rendering of the template, but since the webpage is returned by a function, I'm not quite sure how to implement this...
My function is:
def index():
"""main function"""
env = Environment(loader=FileSystemLoader(basedir+'templates'))
#I modify the global variables
env.globals['foo'] = 'bar'
env.globals['build_response']=build_response
get_url = request.args.get('to_print', None)
message = "the input was \"%s\"" % (get_url,)
template = env.get_template('body.html')
return Response(template.stream(message=message))
#return template.render(message=message)
And what I need to measure is the time spent by
return template.render(message=message)
vs
return Response(template.stream(message=message))
thanks!
You need to look at the system clock before and after the call.
It's easier to do this if you don't immediately return the response from template.render or template.stream.
IE:
from time import time
def index():
# THE REST OF YOUR CODE GOES HERE
start = time.time()
resp = template.render(message=message)
end = time.time()
processing_time = end - start
# LOG PROCESSING TIME SOMEPLACE
return resp