I'm sending the stream of data from flask server via yield. I'm able to see this stream if I go directly to api url. However I don't know how to receive it on frontend. I would appreciate the help. Here is how my backend looks like:
STREAM
`
def redis_stream():
global lock
channel = r.pubsub()
channel.subscribe('CellGridMapClose')
for msg in channel.listen():
if msg['type'] == 'message':
obj = tm.CellGridMapping()
obj.ParseFromString(msg['data'])
objects = obj.objects
movement = []
for vehicle in objects:
x, y = vehicle.pos.x, vehicle.pos.y
movement.append({'posx': x/50, 'posy': y/40})
yield bytes(json.dumps(movement), 'utf-8')
`
ROUTE
`
#app.route('/redis-stream')
def redis_data():
return Response(redis_stream(), mimetype='application/json')
`
This is how my frontend looks. I've tried many variants. This is the last one, however it's not working
FRONTEND
`
const response = await axios.get("/redis-stream", {responseType: 'arraybuffer'});;
console.log(response.data);
`
I'm attempting to send consumers.py information to display on the client end outside of consumers.py.
I've referenced Send message using Django Channels from outside Consumer class this previous question, but the sub process .group_send or .group_add don't seem to exist, so I feel it's possible I'm missing something very easy.
Consumers.py
from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync
class WSConsumer(WebsocketConsumer):
def connect(self):
async_to_sync(self.channel_layer.group_add)("appDel", self.channel_name)
self.accept()
self.render()
appAlarm.py
def appFunc(csvUpload):
#csvUpload=pd.read_csv(request.FILES['filename'])
csvFile = pd.DataFrame(csvUpload)
colUsernames = csvFile.usernames
print(colUsernames)
channel_layer = get_channel_layer()
for user in colUsernames:
req = r.get('https://reqres.in/api/users/2')
print(req)
t = req.json()
data = t['data']['email']
print(user + " " + data)
message = user + " " + data
async_to_sync(channel_layer.group_send)(
'appDel',
{'type': 'render', 'message': message}
)
It's throwing this error:
async_to_sync(channel_layer.group_send)(
AttributeError: 'NoneType' object has no attribute 'group_send'
and will throw the same error for group_add when stripping it back more to figure out what's going on, but per the documentation HERE I feel like this should be working.
To anyone looking at this in the future, I was not able to use redis or even memurai in Windows OS due to cost. I ended up using server side events (SSE), specifically django-eventstream, and so far it's worked great as I didn't need the client to interact with the server, for a chat application this would not work.
Eventstream creates an endpoint at /events/ the client can connect to and receive a streaming http response.
Sending data from externalFunc.py:
send_event('test', 'message', {'text': 'Hello World'})
Event listener in HTML page:
var es = new ReconnectingEventSource('/events/');
es.addEventListener('message', function (e) {
console.log(e.data);
var source = new EventSource("/events/")
var para = document.createElement("P");
const obj = JSON.parse(event.data)
para.innerText = obj.text;
document.body.appendChild(para)
}, false);
es.addEventListener('stream-reset', function (e) {
}, false);
Using Tornado, I have a POST request that takes a long time as it makes many requests to another API service and processes the data. This can take minutes to fully complete. I don't want this to block the entire web server from responding to other requests, which it currently does.
I looked at multiple threads here on SO, but they are often 8 years old and the code does not work anylonger as tornado removed the "engine" component from tornado.gen.
Is there an easy way to kick off this long get call and not have it block the entire web server in the process? Is there anything I can put in the code to say.. "submit the POST response and work on this one function without blocking any concurrent server requests from getting an immediate response"?
Example:
main.py
def make_app():
return tornado.web.Application([
(r"/v1", MainHandler),
(r"/v1/addfile", AddHandler, dict(folderpaths = folderpaths)),
(r"/v1/getfiles", GetHandler, dict(folderpaths = folderpaths)),
(r"/v1/getfile", GetFileHandler, dict(folderpaths = folderpaths)),
])
if __name__ == "__main__":
app = make_app()
sockets = tornado.netutil.bind_sockets(8888)
tornado.process.fork_processes(0)
tornado.process.task_id()
server = tornado.httpserver.HTTPServer(app)
server.add_sockets(sockets)
tornado.ioloop.IOLoop.current().start()
addHandler.py
class AddHandler(tornado.web.RequestHandler):
def initialize(self, folderpaths):
self.folderpaths = folderpaths
def blockingFunction(self):
time.sleep(320)
post("AWAKE")
def post(self):
user = self.get_argument('user')
folderpath = self.get_argument('inpath')
outpath = self.get_argument('outpath')
workflow_value = self.get_argument('workflow')
status_code, status_text = validateInFolder(folderpath)
if (status_code == 200):
logging.info("Status Code 200")
result = self.folderpaths.add_file(user, folderpath, outpath, workflow_value)
self.write(result)
self.finish()
#At this point the path is validated.
#POST response should be send out. Internal process should continue, new
#requests should not be blocked
self.blockingFunction()
Idea is that if input-parameters are validated the POST response should be sent out.
Then internal process (blockingFunction()) should be started, that should not block the Tornado Server from processing another API POST request.
I tried defining the (blockingFunction()) as async, which allows me to process multiple concurrent user requests - however there was a warning about missing "await" with async method.
Any help welcome. Thank you
class AddHandler(tornado.web.RequestHandler):
def initialize(self, folderpaths):
self.folderpaths = folderpaths
def blockingFunction(self):
time.sleep(320)
post("AWAKE")
async def post(self):
user = self.get_argument('user')
folderpath = self.get_argument('inpath')
outpath = self.get_argument('outpath')
workflow_value = self.get_argument('workflow')
status_code, status_text = validateInFolder(folderpath)
if (status_code == 200):
logging.info("Status Code 200")
result = self.folderpaths.add_file(user, folderpath, outpath, workflow_value)
self.write(result)
self.finish()
#At this point the path is validated.
#POST response should be send out. Internal process should continue, new
#requests should not be blocked
await loop.run_in_executor(None, self.blockingFunction)
#if this had multiple parameters it would be
#await loop.run_in_executor(None, self.blockingFunction, param1, param2)
Thank you #xyres
Further read: https://www.tornadoweb.org/en/stable/faq.html
I am trying to build a small site with the server push functionality on Flask micro-web framework, but I did not know if there is a framework to work with directly.
I used Juggernaut, but it seems to be not working with redis-py in current version, and Juggernaut has been deprecated recently.
Does anyone has a suggestion with my case?
Have a look at Server-Sent Events. Server-Sent Events is a
browser API that lets you keep open a socket to your server, subscribing to a
stream of updates. For more Information read Alex MacCaw (Author of
Juggernaut) post on why he kills juggernaut and why the simpler
Server-Sent Events are in manny cases the better tool for the job than
Websockets.
The protocol is really easy. Just add the mimetype text/event-stream to your
response. The browser will keep the connection open and listen for updates. An Event
sent from the server is a line of text starting with data: and a following newline.
data: this is a simple message
<blank line>
If you want to exchange structured data, just dump your data as json and send the json over the wire.
An advantage is that you can use SSE in Flask without the need for an extra
Server. There is a simple chat application example on github which
uses redis as a pub/sub backend.
def event_stream():
pubsub = red.pubsub()
pubsub.subscribe('chat')
for message in pubsub.listen():
print message
yield 'data: %s\n\n' % message['data']
#app.route('/post', methods=['POST'])
def post():
message = flask.request.form['message']
user = flask.session.get('user', 'anonymous')
now = datetime.datetime.now().replace(microsecond=0).time()
red.publish('chat', u'[%s] %s: %s' % (now.isoformat(), user, message))
#app.route('/stream')
def stream():
return flask.Response(event_stream(),
mimetype="text/event-stream")
You do not need to use gunicron to run the
example app. Just make sure to use threading when running the app, because
otherwise the SSE connection will block your development server:
if __name__ == '__main__':
app.debug = True
app.run(threaded=True)
On the client side you just need a Javascript handler function which will be called when a new
message is pushed from the server.
var source = new EventSource('/stream');
source.onmessage = function (event) {
alert(event.data);
};
Server-Sent Events are supported by recent Firefox, Chrome and Safari browsers.
Internet Explorer does not yet support Server-Sent Events, but is expected to support them in
Version 10. There are two recommended Polyfills to support older browsers
EventSource.js
jquery.eventsource
Redis is overkill: use Server-Sent Events (SSE)
Late to the party (as usual), but IMHO using Redis may be overkill.
As long as you're working in Python+Flask, consider using generator functions as described in this excellent article by Panisuan Joe Chasinga. The gist of it is:
In your client index.html
var targetContainer = document.getElementById("target_div");
var eventSource = new EventSource("/stream")
eventSource.onmessage = function(e) {
targetContainer.innerHTML = e.data;
};
...
<div id="target_div">Watch this space...</div>
In your Flask server:
def get_message():
'''this could be any function that blocks until data is ready'''
time.sleep(1.0)
s = time.ctime(time.time())
return s
#app.route('/')
def root():
return render_template('index.html')
#app.route('/stream')
def stream():
def eventStream():
while True:
# wait for source data to be available, then push it
yield 'data: {}\n\n'.format(get_message())
return Response(eventStream(), mimetype="text/event-stream")
As a follow-up to #peter-hoffmann's answer, I've written a Flask extension specifically to handle server-sent events. It's called Flask-SSE, and it's available on PyPI. To install it, run:
$ pip install flask-sse
You can use it like this:
from flask import Flask
from flask_sse import sse
app = Flask(__name__)
app.config["REDIS_URL"] = "redis://localhost"
app.register_blueprint(sse, url_prefix='/stream')
#app.route('/send')
def send_message():
sse.publish({"message": "Hello!"}, type='greeting')
return "Message sent!"
And to connect to the event stream from Javascript, it works like this:
var source = new EventSource("{{ url_for('sse.stream') }}");
source.addEventListener('greeting', function(event) {
var data = JSON.parse(event.data);
// do what you want with this data
}, false);
Documentation is available on ReadTheDocs. Note that you'll need a running Redis server to handle pub/sub.
As a committer of https://github.com/WolfgangFahl/pyFlaskBootstrap4 i ran into the same need and created a flask blueprint for Server Sent Events that has no dependency to redis.
This solutions builds on the other answers that have been given here in the past.
https://github.com/WolfgangFahl/pyFlaskBootstrap4/blob/main/fb4/sse_bp.py has the source code (see also sse_bp.py below).
There are unit tests at https://github.com/WolfgangFahl/pyFlaskBootstrap4/blob/main/tests/test_sse.py
The idea is that you can use different modes to create your SSE stream:
by providing a function
by providing a generator
by using a PubSub helper class
by using the PubSub helper class and use pydispatch at the same time.
As of 2021-02-12 this is alpha code which i want to share nevertheless. Please comment here or as issues in the project.
There is a demo at http://fb4demo.bitplan.com/events and a description of the example use e.g. for a progress bar or time display at: http://wiki.bitplan.com/index.php/PyFlaskBootstrap4#Server_Sent_Events
example client javascript/html code
<div id="event_div">Watch this space...</div>
<script>
function fillContainerFromSSE(id,url) {
var targetContainer = document.getElementById(id);
var eventSource = new EventSource(url)
eventSource.onmessage = function(e) {
targetContainer.innerHTML = e.data;
};
};
fillContainerFromSSE("event_div","/eventfeed");
</script>
example server side code
def getTimeEvent(self):
'''
get the next time stamp
'''
time.sleep(1.0)
s=datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
return s
def eventFeed(self):
'''
create a Server Sent Event Feed
'''
sse=self.sseBluePrint
# stream from the given function
return sse.streamFunc(self.getTimeEvent)
sse_bp.py
'''
Created on 2021-02-06
#author: wf
'''
from flask import Blueprint, Response, request, abort,stream_with_context
from queue import Queue
from pydispatch import dispatcher
import logging
class SSE_BluePrint(object):
'''
a blueprint for server side events
'''
def __init__(self,app,name:str,template_folder:str=None,debug=False,withContext=False):
'''
Constructor
'''
self.name=name
self.debug=debug
self.withContext=False
if template_folder is not None:
self.template_folder=template_folder
else:
self.template_folder='templates'
self.blueprint=Blueprint(name,__name__,template_folder=self.template_folder)
self.app=app
app.register_blueprint(self.blueprint)
#self.app.route('/sse/<channel>')
def subscribe(channel):
def events():
PubSub.subscribe(channel)
self.stream(events)
def streamSSE(self,ssegenerator):
'''
stream the Server Sent Events for the given SSE generator
'''
response=None
if self.withContext:
if request.headers.get('accept') == 'text/event-stream':
response=Response(stream_with_context(ssegenerator), content_type='text/event-stream')
else:
response=abort(404)
else:
response= Response(ssegenerator, content_type='text/event-stream')
return response
def streamGen(self,gen):
'''
stream the results of the given generator
'''
ssegen=self.generateSSE(gen)
return self.streamSSE(ssegen)
def streamFunc(self,func,limit=-1):
'''
stream a generator based on the given function
Args:
func: the function to convert to a generator
limit (int): optional limit of how often the generator should be applied - 1 for endless
Returns:
an SSE Response stream
'''
gen=self.generate(func,limit)
return self.streamGen(gen)
def generate(self,func,limit=-1):
'''
create a SSE generator from a given function
Args:
func: the function to convert to a generator
limit (int): optional limit of how often the generator should be applied - 1 for endless
Returns:
a generator for the function
'''
count=0
while limit==-1 or count<limit:
# wait for source data to be available, then push it
count+=1
result=func()
yield result
def generateSSE(self,gen):
for result in gen:
yield 'data: {}\n\n'.format(result)
def enableDebug(self,debug:bool):
'''
set my debugging
Args:
debug(bool): True if debugging should be switched on
'''
self.debug=debug
if self.debug:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s.%(msecs)03d %(levelname)s:\t%(message)s', datefmt='%Y-%m-%d %H:%M:%S')
def publish(self, message:str, channel:str='sse', debug=False):
"""
Publish data as a server-sent event.
Args:
message(str): the message to send
channel(str): If you want to direct different events to different
clients, you may specify a channel for this event to go to.
Only clients listening to the same channel will receive this event.
Defaults to "sse".
debug(bool): if True enable debugging
"""
return PubSub.publish(channel=channel, message=message,debug=debug)
def subscribe(self,channel,limit=-1,debug=False):
def stream():
for message in PubSub.subscribe(channel,limit,debug=debug):
yield str(message)
return self.streamGen(stream)
class PubSub:
'''
redis pubsub duck replacement
'''
pubSubByChannel={}
def __init__(self,channel:str='sse',maxsize:int=15, debug=False,dispatch=False):
'''
Args:
channel(string): the channel name
maxsize(int): the maximum size of the queue
debug(bool): whether debugging should be switched on
dispatch(bool): if true use the pydispatch library - otherwise only a queue
'''
self.channel=channel
self.queue=Queue(maxsize=maxsize)
self.debug=debug
self.receiveCount=0
self.dispatch=False
if dispatch:
dispatcher.connect(self.receive,signal=channel,sender=dispatcher.Any)
#staticmethod
def reinit():
'''
reinitialize the pubSubByChannel dict
'''
PubSub.pubSubByChannel={}
#staticmethod
def forChannel(channel):
'''
return a PubSub for the given channel
Args:
channel(str): the id of the channel
Returns:
PubSub: the PubSub for the given channel
'''
if channel in PubSub.pubSubByChannel:
pubsub=PubSub.pubSubByChannel[channel]
else:
pubsub=PubSub(channel)
PubSub.pubSubByChannel[channel]=pubsub
return pubsub
#staticmethod
def publish(channel:str,message:str,debug=False):
'''
publish a message via the given channel
Args:
channel(str): the id of the channel to use
message(str): the message to publish/send
Returns:
PubSub: the pub sub for the channel
'''
pubsub=PubSub.forChannel(channel)
pubsub.debug=debug
pubsub.send(message)
return pubsub
#staticmethod
def subscribe(channel,limit=-1,debug=False):
'''
subscribe to the given channel
Args:
channel(str): the id of the channel to use
limit(int): limit the maximum amount of messages to be received
debug(bool): if True debugging info is printed
'''
pubsub=PubSub.forChannel(channel)
pubsub.debug=debug
return pubsub.listen(limit)
def send(self,message):
'''
send the given message
'''
sender=object();
if self.dispatch:
dispatcher.send(signal=self.channel,sender=sender,msg=message)
else:
self.receive(sender,message)
def receive(self,sender,message):
'''
receive a message
'''
if sender is not None:
self.receiveCount+=1;
if self.debug:
logging.debug("received %d:%s" % (self.receiveCount,message))
self.queue.put(message)
def listen(self,limit=-1):
'''
listen to my channel
this is a generator for the queue content of received messages
Args:
limit(int): limit the maximum amount of messages to be received
Return:
generator: received messages to be yielded
'''
if limit>0 and self.receiveCount>limit:
return
yield self.queue.get()
def unsubscribe(self):
'''
unsubscribe me
'''
if self.dispatch:
dispatcher.disconnect(self.receive, signal=self.channel)
pass
I'm writing a chat feature (like the Faceboook.com one) for a Google App Engine site. I need a way to keep track of what users have new messages. I'm currently trying to use Memcache:
class Message():
def __init__(self, from_user_key, message_text)
self.from_user_key = from_user_key
self.message_text = message_text
class NewMessages():
def __init__(self):
self.messages = []
def add_message(self, message):
self.messages.append(message)
def get_messages(self):
return self.messages
def messages_sent(self):
self.messages = [] #Clear all messages
def ChatUserManager():
def load(user_key):
manager = memcache.get("chat_user_%s" % user_key)
if manager is not None:
return manager
else:
manager = ChatUserManager(user_key)
memcache.set("chat_user_%s" % user_key, manager)
return manager
def save(self):
memcache.set("chat_user_%s" % user_key, self)
def __init__(self, user_key):
self.online = True
self.new_messages = NewMessages()
self.new_data = False
self.user_key = user_key
def recieve_message(self, message):
self.new_data = True
self.new_messages.add_message(Message(from_user_key, message_text))
def send_message(self, message):
to_manager = ChatUserManager.load(message.from_user_key)
to_manager.recieve_message(message)
def client_receive_success(self):
self.new_data = False
self.new_messages.messages_sent()
This chat is user to user, like Facebook or an IM session, not group chat.
Each user will poll a url with ajax to get new messages addressed to them every x seconds. The chat manager will be loaded on that page (ChatUserManager.load(user_key)) and new messages will be checked for. When they are sent the manager will be told that the messages have been sent (manager.client_receive_success()), and then saved back to memcache (manager.save()).
When a user sends a message in the javascript client, it will send an ajax request to a url. The url will load the client's UserChatManager and call .send_message(Message(to_user_key, message_string))).
I'm concerned about the practicality of this model. If everything is in memcache how will it be synchronized across different pages?
Is there a better way to do this?
I do admit that I'm not a python pro yet so the code might not be very pythonic, are there any best practices I'm missing?
The problem isn't so much about how to share data between "pages" but how will the usability of the service will be impacted by using memcache.
There are no guarantees associated with data persistence in memcache: one moment its there, the other it might not.