I'm new to both flask and python. I've got an application I'm working on to hold weather data. I'm allowing for both get and post commands to come into my flask application. unfortunately, the automated calls for my API are not always coming back with the proper results. I'm currently storing my data in a global variable when a post command is called, the new data is appended to my existing data. Unfortunately sometimes when the get is called, it is not receiving the most up to date version of my global data variable. I believe that the issue is that the change is not being passed up from the post function to the global variable before the get is called because I can run the get and the proper result comes back.
weatherData = [filed with data read from csv on initialization]
class FullHistory(Resource):
def get(self):
ret = [];
for row in weatherData:
val = row['DATE']
ret.append({"DATE":str(val)})
return ret
def post(self):
global weatherData
newWeatherData = weatherData
args = parser.parse_args()
newVal = int(args['DATE'])
newWeatherData.append({'DATE':int(args['DATE']),'TMAX':float(args['TMAX']),'TMIN':float(args['TMIN'])})
weatherData = newWeatherData
#time.sleep(5)
return {"DATE":str(newVal)},201
class SelectHistory(Resource):
def get(self, date_id):
val = int(date_id)
bVal = False
#time.sleep(5)
global weatherData
for row in weatherData:
if(row['DATE'] == val):
wd = row
bVal = True
break
if bVal:
return {"DATE":str(wd['DATE']),"TMAX":float(wd['TMAX']),"TMIN":float(wd['TMIN'])}
else:
return "HTTP Error code 404",404
def delete(self, date_id):
val = int(date_id)
wdIter = None
for row in weatherData:
if(row['DATE'] == val):
wdIter = row
break
if wdIter != None:
weatherData.remove(wdIter)
return {"DATE":str(val)},204
else:
return "HTTP Error code 404",404
Is there any way I can assure that my global variable is up to date or make my API wait to return until I'm sure that the update has been passed along? This was supposed to be a simple application. I would really rather not have to learn how to use threads in python just yet. I've made sure that my calls get request is not starting until after the post has given a response. I know that one workaround was to use sleep to delay my responses, I would rather understand why my update isn't occurring immediately in the first place.
I believe your problem is the application context. As stated here:
The application context is created and destroyed as necessary. It
never moves between threads and it will not be shared between
requests. As such it is the perfect place to store database connection
information and other things. The internal stack object is called
flask._app_ctx_stack. Extensions are free to store additional
information on the topmost level, assuming they pick a sufficiently
unique name and should put their information there, instead of on the
flask.g object which is reserved for user code.
Though it says you can store data at the "topmost level," it's not reliable, and if you extrapolate your project to use worker processes with uWSGI, for instance, you'll need persistence to share data between threads regardless. You should be using a database, redis, or at very least updating your .csv file each time you mutate your data.
Related
I'm trying to access variables that are being passed from the client (iOS; Swift) to the server on a Flask-SocketIO connection on the connect action. Let me explain. When you want to do a random action you have something like this on the server which includes a callback (see data in the code below):
#socketio.on('custom action', namespace = '/mynamespace')
def handle_custom_action(data):
print data
There are some preset actions (like connect) and apparently connect does not have any callback when it's called so the client cannot send any data on the connect action:
#socketio.on('connect', namespace = '/mynamespace')
def handle_connection(data):
print data # nothing gets printed
I looked into the code a bit deeper and found this. The definition of the on function is:
def on(self, message, namespace=None):
And then within that function (I'm omitting a bit of code to get to the point):
if message == 'connect':
ret = handler()
else:
ret = handler(*args)
I could be wrong but it appears that code explicitly does not return anything back on connect and I'm not sure why? I've found some evidence that this is possible in node.js (I will update this with proper links when I find them) so I'm wondering why this isn't possible in the Flask-SocketIO library or whether I'm just misunderstanding what I'm looking at (and if so, how to get those parameters).
Thanks!
Update:
I did find a way to access connection parameters but it doesn't seem like the 'right' way. I'm using the global request and splitting the GET parameters / query string that come through on the request:
data = dict(item.split("=") for item in request.event["args"][0]["QUERY_STRING"].split("&"))
OR as two lines:
data = request.event["args"][0]["QUERY_STRING"].split("&"))
data = dict(item.split("=") for item in data.split("&"))
Flask-SocketIO adds event which connects a dictionary with keys of message and args and within args is the QUERY_STRING which I then split add turn into a dictionary. This works fine but it doesn't necessarily answer the original question as to why there is no callback?
Here is an example of the iOS connection params being passed:
let connectParams = SocketIOClientOption.connectParams(["user_id" : Int(user.userId)!, "connection_id" : self.socketConnectionId])
self.socket = SocketIOClient(socketURL: URL(string: "http://www.myurl.com")!, config: [.nsp("/namespace"), .forceWebsockets(true), .forceNew(true), connectParams])
My flask program (simulation in the view) runs in the following order (detailed code is also attached):
1> read my variable 'tx_list' from session. tx_list = session.get('tx_list', None)
2> for t in tx_list: do someting with t.
3> store tx_list in session: session['tx_list'] = tx_list
The reason I use session is because I want to change 'tx_list' every time I invoke this 'simulation' function.
The problem now is that if I print (console.log(tx_list)) in the front-end, it only updates itself a few times. But in the same time, when I print the values in the simulation function, it always updates. So I suspect the problem is because of the session???
I've tried to add another 'time_now' variable in the simulation function, which is independent of session. Then in the front-end (html) always updates 'time_now'. So the problem must be because of the usage of session??? How can update my 'tx_list' if session is not the best way to do it?
-------------------code is below----------------------------
My view is like below: In my view, I simply read my var 'tx_list' from session, do something with it, then store it back to the session.
#app.route('/simulation/<param>')
def simulation(param):
tx_list = session.get('tx_list', None)
today = date.today()
if t0 == '0':
time_now = today.strftime("%Y-%m-%d %H")
else:
time_now = (today + relativedelta(hours=int(param))).strftime("%Y-%m-%d %H")
return_val = jsonify({'time':time_now, 'tx_list':tx_list_0})
for t in tx_list:
###########I have my code here to change t.
print(t)
session['tx_list'] = tx_list
return return_val
problem solved once I installed Flask-Session and initilize it.
I feel puzzled why it updates OK for only a few times without installing the module.
I would like to have a computational simulation running on a background process (started with redis rq) where I can query its current state, as well as change parameters using Django.
For the sake of simplicity: let's say I want to run the following code for a long time (which I would set up through a python worker):
def simulation(a=1):
value = 0
while a != None:
value += a
time.sleep(5)
Then, by visiting a URL, it would tell me the current value of value. I could also POST to a URL to change the value of a i.e. a=None to stop the simulation or a=-10 to change the behavior.
What is the best way to do this?
This best way I've found to do this is using cache
from django.core.cache import cache
def simulation(a=1):
value = 0
while a != None:
value += a
cache.set('value', value, 3600)
time.sleep(5)
a = cache.get('a', None)
This does work, but it's quite slow for my needs. Perhaps there's a method using sockets, but I wasn't abe to get it to work. The socket is blocked in the background process.
I'm writing an application in python using the twisted.web framework to stream video using html 5.
The videos are being server via static.File('pathtovideo').render_GET()
The problem is that only one video can be streamed at a time as it ties up the entire process.
Is there anyway to make the streaming async or non-block, whichever term would be appropriate here.
I tried using deferToThread but that still tied up the process.
This is the class Im currently using, where Movie is an ORM table and mid is just an id to an arbitrary row.
class MovieStream(Resource):
isLeaf=True
def __init__(self, mid):
Resource.__init__(self)
self.mid = mid
def render_GET(self, request):
movie = Movie.get(Movie.id == self.mid)
if movie:
defered = deferToThread(self._start_stream, path=movie.source), request=request)
defered.addCallback(self._finish_stream, request)
return NOT_DONE_YET
else:
return NoResource()
`
def _start_stream(self, path, request):
stream = File(path)
return stream.render_GET(request)
def _finish_stream(self, ret, request):
request.finish()
The part of this code that looks like it blocks is actually the Movie.get call.
It is incorrect to call _start_stream with deferToThread because _start_stream uses Twisted APIs (File and whatever File.render_GET uses) and it is illegal to use Twisted APIs except in the reactor thread (in other words, it is illegal to use them in a function you call with deferToThread).
Fortunately you can just delete the use of deferToThread to fix that bug.
To fix the problem that Movie.get blocks you'll need to find a way to access your database asynchronously. Perhaps using deferToThread(Movie.get, Movie.id == self.mid) - if the database library that implements Movie.get is thread-safe, that is.
For what it's worth, you can also avoid the render_GET hijinx by moving your database lookup logic earlier in the resource traversal hierarchy.
For example, I imagine your URLs look something like /foo/bar/<movie id>. In this case, the resource at /foo/bar gets asked for <movie id> children. If you implement that lookup like this:
from twisted.web.resource import Resource
from twisted.web.util import DeferredResource
class MovieContainer(Resource):
def getChild(self, movieIdentifier):
condition = (Movie.id == movieIdentifier)
getting = deferToThread(Movie.get, condition)
return DeferredResource(getting)
(assuming here that Movie.get is thread-safe) then you'll essentially be done.
Resource traversal will conclude with the object constructed by DeferredResource(getting) and when that object is rendered it will take care of waiting for getting to have a result (for the Deferred to "fire", in the lingo) and of calling the right method on it, eg render_GET, to produce a response for the request.
This question already has answers here:
Are global variables thread-safe in Flask? How do I share data between requests?
(4 answers)
Closed 2 years ago.
I am trying to save a cache dictionary in my flask application.
As far as I understand it, the Application Context, in particular the flask.g object should be used for this.
Setup:
import flask as f
app = f.Flask(__name__)
Now if I do:
with app.app_context():
f.g.foo = "bar"
print f.g.foo
It prints bar.
Continuing with the following:
with app.app_context():
print f.g.foo
AttributeError: '_AppCtxGlobals' object has no attribute 'foo'
I don’t understand it and the docs are not helping at all. If I read them correctly the state should have been preserved.
Another idea I had was to simply use module-wide variables:
cache = {}
def some_function():
cache['foo'] = "bar"
But it seems like these get reset with every request.
How to do this correctly?
Edit: Flask 10.1
Based on your question, I think you're confused about the definition of "global".
In a stock Flask setup, you have a Flask server with multiple threads and potentially multiple processes handling requests. Suppose you had a stock global variable like "itemlist = []", and you wanted to keep adding to it in every request - say, every time someone made a POST request to an endpoint. This is totally possible in theory and practice. It's also a really bad idea.
The problem is that you can't easily control which threads and processes "win" - the list could up in a really wonky order, or get corrupted entirely. So now you need to talk about locks, mutexs, and other primitives. This is hard and annoying.
You should keep the webserver itself as stateless as possible. Each request should be totally independent and not share any state in the server. Instead, use a database or caching layer which will handle the state for you. This seems more complicated but is actually simpler in practice. Check out SQLite for example ; it's pretty simple.
To address the 'flask.g' object, that is a global object on a per request basis.
http://flask.pocoo.org/docs/api/#flask.g
It's "wiped clean" between requests and cannot be used to share state between them.
I've done something similar to your "module-wide variables" idea that I use in a flask server that I use to integrate two pieces of software where I know I will only ever have one simultaneous "user" (being the sender software).
My app.py looks like this:
from flask import Flask
from flask.json import jsonify
app = Flask(__name__)
cache = {}
#app.route("/create")
def create():
cache['foo'] = 0
return jsonify(cache['foo'])
#app.route("/increment")
def increment():
cache['foo'] = cache['foo'] + 1
return jsonify(cache['foo'])
#app.route("/read")
def read():
return jsonify(cache['foo'])
if __name__ == '__main__':
app.run()
You can test it like this:
import requests
print(requests.get('http://127.0.0.1:5000/create').json())
print(requests.get('http://127.0.0.1:5000/increment').json())
print(requests.get('http://127.0.0.1:5000/increment').json())
print(requests.get('http://127.0.0.1:5000/read').json())
print(requests.get('http://127.0.0.1:5000/increment').json())
print(requests.get('http://127.0.0.1:5000/create').json())
print(requests.get('http://127.0.0.1:5000/read').json())
Outputs:
0
1
2
2
3
0
0
Use with caution as I expect this to not behave in a proper multi user web server environment.
This line
with app.app_context():
f.g.foo = "bar"
Since you are using the "with" keyword, once this loop is executed, it calls the __exit__ method of the AppContext class. See this. So the 'foo' is popped out once done. Thats why you don't have it available again. You can instead try:
ctx = app.app_context()
f.g.foo = 'bar'
ctx.push()
Until you call the following, g.foo should be available
ctx.pop()
I am howver not sure if you want to use this for the purpose of caching.