I try to save serialized BaseQuery object to the flask.session['t'] object but every time the get request is sent to this endpoint the session.get('t') is None.
q_transactions is non-empty list.
Could you help me to understand why it behaves that way? Did I miss or missunderstand something?
# app.py
#bp.route('/testing')
def testing():
import sys
from flask import session
from sqlalchemy.ext.serializer import loads, dumps
form = FiltersForm(request.args)
if session.get('t') is not None:
print('|'*80)
print(loads(session.get('t')), file=sys.stdout)
print('|'*80)
filters = form.data
q_transactions = current_user.transactions()
q_transactions = Transaction.apply_filters(q_transactions, filters)
session['t'] = dumps(q_transactions)
return render_template('test_edit_transaction.html', form=form)
EDIT:
The issue is probably caused by too large data.
I'm trying to serve a Flask app and would like to reload a pickle file at a specific time window (e.g. 9AM every day). I've tried to put a while loop into the end of my flask app with a time counter, but this ends up hanging my application. Currently the set up is...
# main.wsgi
from main import app as application
# main.py
data = pickle.load("/path/to/pickle.file")
#app.route("/")
def func():
return render_template("base.html", data_to_serve = data)
# Can I write something here to reload the data at specific time points?
I am assuming the goal here is to do what I call a "poor man's cache". Ideally you'd opt to use something like pymemcache and Flask's cache utils, but the snippet below will accomplish what you want. You can refactor this if you want to reload the pickle each time; kind of would be defeating the purpose I think.
Additionally, note that I have used a range of time to return the pickle data; 9 AM to 12 PM. You can also do something like if now.time() == time(hour=9) to accomplish what you want.
import pickle
from datetime import datetime, time
cached_data = pickle.load("/path/to/pickle.file")
START_TIME = time(hour=9)
END_TIME = time(hour=12) # Can also use something like timedelta
def in_range():
now = datetime.now()
if START_TIME <= now.time() <= END_TIME:
return True
return False
app.route("/")
def func():
if in_range():
return render_template("base.html", data_to_serve = cached_data)
# else do normal business
data = 'compute new data...'
return render_template("base.html", data_to_serve = data)
Happy coding!
You want to reload the data at specific point of time then you have 2 options:
Do it from the client size using javascript and ajax requests using some timer.
Use web sockets. There is a library for flask called flask-socketio. There is an example on how to use it.
I have python file called testing_file.py:
from datetime import datetime
import MySQLdb
# Open database connection
class DB():
def __init__(self, server, user, password, db_name):
db = MySQLdb.connect(server, user, password, db_name )
self.cur = db.cursor()
def time_statistic(self, start_date, end_date):
time_list = {}
sql = "SELECT activity_log.datetime, activity_log.user_id FROM activity_log"
self.cur.execute(sql)
self.date_data = self.cur.fetchall()
for content in self.date_data:
timestamp = str(content[0])
datetime_object = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S')
timestamps = datetime.strftime(datetime_object, "%Y-%m-%d")
if start_dt <= timestamps and timestamps <= end_dt:
if timestamps not in time_list:
time_list[timestamps]=1
else:
time_list[timestamps]+=1
return json.dumps(time_list)
start_date = datetime.strptime(str('2017-4-7'), '%Y-%m-%d')
start_dt = datetime.strftime(start_date, "%Y-%m-%d")
end_date = datetime.strptime(str('2017-5-4'), '%Y-%m-%d')
end_dt = datetime.strftime(end_date, "%Y-%m-%d")
db = DB("host","user_db","pass_db","db_name")
db.time_statistic(start_date, end_date)
I want to access the result (time_list) thru API using Flask. This is what i've wrote so far, doesn't work and also I've tried another way:
from flask import Flask
from testing_api import *
app = Flask(__name__)
#app.route("/")
def get():
db = DB("host","user_db","pass_db","db_name")
d = db.time_statistic()
return d
if __name__ == "__main__":
app.run(debug=True)
Question: This is my first time work with API and Flask. Can anyone please help me thru this. Any hints are appreciated. Thank you
I've got empty list as result {}
There are many things wrong with what you are doing.
1.> def get(self, DB) why self? This function does not belong to a class. It is not an instance function. self is a reference of the class instance when an instance method is called. Here not only it is not needed, it is plain and simple wrong.
2.> If you look into flask's routing declaration a little bit, you will see how you should declare a route with parameter. This is the link. In essence you should something like this
#app.route("/path/<variable>")
def route_func(variable):
return variable
3.> Finally, one more thing I would like to mention, Please do not call a regular python file test_<filename>.py unless you plan to use it as a unit testing file. This is very confusing.
Oh, and you have imported DB from your module already no need to pass it as a parameter to a function. It should be anyway available inside it.
There are quite a few things that are wrong (ranging from "useless and unclear" to "plain wrong") in your code.
wrt/ the TypeError: as the error message says, your get() function expects two arguments (self and DB) which won't be passed by Flask - and are actually not used in the function anyway. Remove both arguments and you'll get rid of this error - just to find out you now have a NameError on the first line of the get() function (obviously since you didn't import time_statistic nor defined start_date and end_date).
I'm trying to restrict the API calls in my code. I already found a nice python library ratelimiter==1.0.2.post0
https://pypi.python.org/pypi/ratelimiter
However, this library can only limit the rate in local scope. i.e) in function and loops
# Decorator
#RateLimiter(max_calls=10, period=1)
def do_something():
pass
# Context Manager
rate_limiter = RateLimiter(max_calls=10, period=1)
for i in range(100):
with rate_limiter:
do_something()
Because I have several functions, which make API calls, in different places, I want to limit the API calls in global scope.
For example, suppose I want to limit the APIs call to one time per second. And, suppose I have functions x and y in which two API calls are made.
#rate(...)
def x():
...
#rate(...)
def y():
...
By decorating the functions with the limiter, I'm able to limit the rate against the two functions.
However, if I execute the above two functions sequentially, it looses track of the number of API calls in global scope because they are unaware of each other. So, y will be called right after the execution of x without waiting another second. And, this will violate the one time per second restriction.
Is there any way or library that I can use to limit the rate globally in python?
I had the same problem, I had a bunch of different functions that calls the same API and I wanted to make rate limiting work globally. What I ended up doing was to create an empty function with rate limiting enabled.
PS: I use a different rate limiting library found here: https://pypi.org/project/ratelimit/
from ratelimit import limits, sleep_and_retry
# 30 calls per minute
CALLS = 30
RATE_LIMIT = 60
#sleep_and_retry
#limits(calls=CALLS, period=RATE_LIMIT)
def check_limit():
''' Empty function just to check for calls to API '''
return
Then I just call that function at the beginning of every function that calls the API:
def get_something_from_api(http_session, url):
check_limit()
response = http_session.get(url)
return response
If the limit is reached, the program will sleep until the (in my case) 60 seconds have passed, and then resume normally.
After all, I implemented my own Throttler class. By proxying every API request to the request method, we can keep track of all API requests. Taking advantage of passing function as the request method parameter, it also caches the result in order to reduce API calls.
class TooManyRequestsError(Exception):
def __str__(self):
return "More than 30 requests have been made in the last five seconds."
class Throttler(object):
cache = {}
def __init__(self, max_rate, window, throttle_stop=False, cache_age=1800):
# Dict of max number of requests of the API rate limit for each source
self.max_rate = max_rate
# Dict of duration of the API rate limit for each source
self.window = window
# Whether to throw an error (when True) if the limit is reached, or wait until another request
self.throttle_stop = throttle_stop
# The time, in seconds, for which to cache a response
self.cache_age = cache_age
# Initialization
self.next_reset_at = dict()
self.num_requests = dict()
now = datetime.datetime.now()
for source in self.max_rate:
self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))
self.num_requests[source] = 0
def request(self, source, method, do_cache=False):
now = datetime.datetime.now()
# if cache exists, no need to make api call
key = source + method.func_name
if do_cache and key in self.cache:
timestamp, data = self.cache.get(key)
logging.info('{} exists in cached # {}'.format(key, timestamp))
if (now - timestamp).seconds < self.cache_age:
logging.info('retrieved cache for {}'.format(key))
return data
# <--- MAKE API CALLS ---> #
# reset the count if the period passed
if now > self.next_reset_at.get(source):
self.num_requests[source] = 0
self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))
# throttle request
def halt(wait_time):
if self.throttle_stop:
raise TooManyRequestsError()
else:
# Wait the required time, plus a bit of extra padding time.
time.sleep(wait_time + 0.1)
# if exceed max rate, need to wait
if self.num_requests.get(source) >= self.max_rate.get(source):
logging.info('back off: {} until {}'.format(source, self.next_reset_at.get(source)))
halt((self.next_reset_at.get(source) - now).seconds)
self.num_requests[source] += 1
response = method() # potential exception raise
# cache the response
if do_cache:
self.cache[key] = (now, response)
logging.info('cached instance for {}, {}'.format(source, method))
return response
Many API providers constrain developers from making too many API calls.
Python ratelimit packages introduces a function decorator preventing a function from being called more often than that allowed by the API provider.
from ratelimit import limits
import requests
TIME_PERIOD = 900 # time period in seconds
#limits(calls=15, period=TIME_PERIOD)
def call_api(url):
response = requests.get(url)
if response.status_code != 200:
raise Exception('API response: {}'.format(response.status_code))
return response
Note: This function will not be able to make more then 15 API call within a 15 minute time period.
Adding to Sunil answer, you need to add #sleep_and_retry decorator, otherwise your code will break when reach the rate limit:
#sleep_and_retry
#limits(calls=0.05, period=1)
def api_call(url, api_key):
r = requests.get(
url,
headers={'X-Riot-Token': api_key}
)
if r.status_code != 200:
raise Exception('API Response: {}'.format(r.status_code))
return r
There are lots of fancy libraries that will provide nice decorators, and special safety features, but the below should work with django.core.cache or any other cache with a get and set method:
def hit_rate_limit(key, max_hits, max_hits_interval):
'''Implement a basic rate throttler. Prevent more than max_hits occurring
within max_hits_interval time period (seconds).'''
# Use the django cache, but can be any object with get/set
from django.core.cache import cache
hit_count = cache.get(key) or 0
logging.info("Rate Limit: %s --> %s", key, hit_count)
if hit_count > max_hits:
return True
cache.set(key, hit_count + 1, max_hits_interval)
return False
Using the Python standard library:
import threading
from time import time, sleep
b = threading.Barrier(2)
def belay(s=1):
"""Block the main thread for `s` seconds."""
while True:
b.wait()
sleep(s)
def request_something():
b.wait()
print(f'something at {time()}')
def request_other():
b.wait()
print(f'or other at {time()}')
if __name__ == '__main__':
thread = threading.Thread(target=belay)
thread.daemon = True
thread.start()
# request a lot of things
i = 0
while (i := i+1) < 5:
request_something()
request_other()
There's about s seconds between each timestamp printed. Because the main thread waits rather than sleeps, time it spends responding to requests is unrelated to the (minimum) time between requests.
I need to compare the performances of a Flask application that uses jinja2.Template.render against jinja2.Template.stream to be sure that there is no loss of performances using the streaming of templates.
My (very simple) idea was to print the timestamp before and after the rendering of the template, but since the webpage is returned by a function, I'm not quite sure how to implement this...
My function is:
def index():
"""main function"""
env = Environment(loader=FileSystemLoader(basedir+'templates'))
#I modify the global variables
env.globals['foo'] = 'bar'
env.globals['build_response']=build_response
get_url = request.args.get('to_print', None)
message = "the input was \"%s\"" % (get_url,)
template = env.get_template('body.html')
return Response(template.stream(message=message))
#return template.render(message=message)
And what I need to measure is the time spent by
return template.render(message=message)
vs
return Response(template.stream(message=message))
thanks!
You need to look at the system clock before and after the call.
It's easier to do this if you don't immediately return the response from template.render or template.stream.
IE:
from time import time
def index():
# THE REST OF YOUR CODE GOES HERE
start = time.time()
resp = template.render(message=message)
end = time.time()
processing_time = end - start
# LOG PROCESSING TIME SOMEPLACE
return resp