Develop Reference

[Python][Tornado] Is it possible to have multi websocket with different port & message in one python program?

I am a newbie in Python. I am now having a project that using a raspberry pi to connect to local web, a web server & a mobile apps via websocket. Both of them are interactive. I can now communicate with each of them separately with 3 different program. But I face some problems when I want to integrate them into 1.
I read a number of post in here & find that all answers suggest to use a list to store each websocket and all of them sending the same message. Is it possible to send different message? Maybe something this
import tornado.ioloop
import tornado.web
import tornado.websocket
class WebSocketHandler_web(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
class WebSocketHandler_mobile(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
class WebSocketHandler_server(tornado.websocket.WebSocketHandler):
def __init__(self, application, **kwargs):
pass
def open(self):
# do something
def on_message(self, msg):
# do something
def on_close(self):
# do something
app_web = tornado.web.Application([(r'/ws/', WebSocketHandler_web),])
app_mobile = tornado.web.Application([(r'/ws/', WebSocketHandler_mobile),])
app_server = tornado.web.Application([(r'/ws/', WebSocketHandler_server),])
def main_task():
# do something
if(mode == 1):
webSocket_web.write_message("Mode 1")
elif(mode == 2):
webSocket_mobile.write_message("Mode 2")
elif(mode == 3):
webSocket_server.write_message("Mode 3")
# do something
if __name__ == "__main__":
app_web.listen(7777)
app_mobile.listen(8888)
app_server.listen(9999)
webSocket_web = WebSocketHandler_web(app_web)
webSocket_mobile = WebSocketHandler_mobile(app_mobile)
webSocket_server = WebSocketHandler_server(app_server)
tornado.ioloop.PeriodicCallback(main_task,1000).start()
tornado.ioloop.IOLoop.instance.start()
But seems that websocket.init needs a parameter request. What is that?

Is it possible to send different message Just iterate over a list of stored websockets and send different message, simple as that.
init needs a parameter request Tornado on each incoming request creates an instance of a request handler, and request is passed to init to bind a handler to request. When you override an init method in your handler you should follow parent's method interface. Don't worry about value of request argument, tornado will pass it for you.

How to integrate channels and DRF together

I'm currently trying to create a backend server to communicate with some clients with a websocket. The clients makes some request to the backend and the backend responds directly to the client through a consumer.
In addition, I've got an API that needs to send some requests to the client. It has to go through the opened socket of the consumer. I'm using Django Rest Framework for the API. So I've got 2 apps for now. One for the consumer and one for the API. I want to know if it's the correct way or not.
This is actually the code I'm thinking about right now:
# mybackendapp/consumers.py
class MyConsumer(AsyncWebsocketConsumer):
async def connect(self):
self.client_id = self.scope['url_route']['kwargs']['client_id']
# This line I don't get it very well. It comes from:
# [channels doc: single channels][1]
# I don't know if I should create the Clients myself or if it's
# created automatically
Clients.objects.create(channel_name=self.channel_name,
self.client_id)
self.accept()
async def disconnect(self):
Clients.objects.filter(channel_name=self.channel_name).delete()
async def receive(self, text_data):
self.recv_data = json.loads(text_data)
if self.recv_data[0] == CLIENT_REQUEST:
self.handler = ClientRequestHandler(self.client_id,
self.recv_data)
await self.handler.run()
self.sent_data = self.handler.response
self.send(self.sent_data)
elif self.recv_data[0] == CLIENT_RESPONSE:
self.handler = ClientResponseHandler(self.client_id,
self.recv_data)
channel_layer = get_channel_layer()
# Here I'm not sure but I may have several API requests so
# several row with the same client_id.
# I welcome info on how to deal with that.
api_channel_name = self.another_handler.ext_channel_name
channel_layer.send(api_channel_name, {
"text_data": self.handler.response,
})
async def message_from_api(self, event):
self.api_channel_name = event['channel_name_answer']
# this line is for hiding the fact that I'm only manipulating data
# to send it to a correct format to the socket
self.another_handler = ExternalDataHandler(event['json_data'])
query_to_client = another_handler.get_formatted_query()
self.send(query_to_client)
In receive, this consumer handles differently the messages from the client depending if it's initiated by the client or the rest API. You can see that with CLIENT_REQUEST and CLIENT_RESPONSE constants.
Now from the API view:
# myapi/views.py
from channels.layers import get_channel_layer
def my_api_view(request, client_id):
channel_layer = get_channel_layer()
if request.method == 'POST':
ext_request_data_json = request.data
client_channel_name = Clients.objects.get(
client_id=client_id).channel_name
# I don't know what type is listen_channel_name. I assume it's str
listen_channel_name = async_to_sync(channels_layers.new_channel)()
async_to_sync(channel_layer.send)(
client_channel_name, {
'type': 'message.from.api',
'json_data': ext_request_data_json,
'channel_name_answer': listen_channel_name
}
)
received_msg = channel_layer.receive(listen_channel_name)
I believe that this code should work. I want to know if it's the correct way to do it.

See djangochannelsrestframework as a possible alternative solution.
Django Channels Rest Framework provides a DRF like interface for building channels-v3 websocket consumers.

How to implement server push in Flask framework?

I am trying to build a small site with the server push functionality on Flask micro-web framework, but I did not know if there is a framework to work with directly.
I used Juggernaut, but it seems to be not working with redis-py in current version, and Juggernaut has been deprecated recently.
Does anyone has a suggestion with my case?

Have a look at Server-Sent Events. Server-Sent Events is a
browser API that lets you keep open a socket to your server, subscribing to a
stream of updates. For more Information read Alex MacCaw (Author of
Juggernaut) post on why he kills juggernaut and why the simpler
Server-Sent Events are in manny cases the better tool for the job than
Websockets.
The protocol is really easy. Just add the mimetype text/event-stream to your
response. The browser will keep the connection open and listen for updates. An Event
sent from the server is a line of text starting with data: and a following newline.
data: this is a simple message
<blank line>
If you want to exchange structured data, just dump your data as json and send the json over the wire.
An advantage is that you can use SSE in Flask without the need for an extra
Server. There is a simple chat application example on github which
uses redis as a pub/sub backend.
def event_stream():
pubsub = red.pubsub()
pubsub.subscribe('chat')
for message in pubsub.listen():
print message
yield 'data: %s\n\n' % message['data']
#app.route('/post', methods=['POST'])
def post():
message = flask.request.form['message']
user = flask.session.get('user', 'anonymous')
now = datetime.datetime.now().replace(microsecond=0).time()
red.publish('chat', u'[%s] %s: %s' % (now.isoformat(), user, message))
#app.route('/stream')
def stream():
return flask.Response(event_stream(),
mimetype="text/event-stream")
You do not need to use gunicron to run the
example app. Just make sure to use threading when running the app, because
otherwise the SSE connection will block your development server:
if __name__ == '__main__':
app.debug = True
app.run(threaded=True)
On the client side you just need a Javascript handler function which will be called when a new
message is pushed from the server.
var source = new EventSource('/stream');
source.onmessage = function (event) {
alert(event.data);
};
Server-Sent Events are supported by recent Firefox, Chrome and Safari browsers.
Internet Explorer does not yet support Server-Sent Events, but is expected to support them in
Version 10. There are two recommended Polyfills to support older browsers
EventSource.js
jquery.eventsource

Redis is overkill: use Server-Sent Events (SSE)
Late to the party (as usual), but IMHO using Redis may be overkill.
As long as you're working in Python+Flask, consider using generator functions as described in this excellent article by Panisuan Joe Chasinga. The gist of it is:
In your client index.html
var targetContainer = document.getElementById("target_div");
var eventSource = new EventSource("/stream")
eventSource.onmessage = function(e) {
targetContainer.innerHTML = e.data;
};
...
<div id="target_div">Watch this space...</div>
In your Flask server:
def get_message():
'''this could be any function that blocks until data is ready'''
time.sleep(1.0)
s = time.ctime(time.time())
return s
#app.route('/')
def root():
return render_template('index.html')
#app.route('/stream')
def stream():
def eventStream():
while True:
# wait for source data to be available, then push it
yield 'data: {}\n\n'.format(get_message())
return Response(eventStream(), mimetype="text/event-stream")

As a follow-up to #peter-hoffmann's answer, I've written a Flask extension specifically to handle server-sent events. It's called Flask-SSE, and it's available on PyPI. To install it, run:
$ pip install flask-sse
You can use it like this:
from flask import Flask
from flask_sse import sse
app = Flask(__name__)
app.config["REDIS_URL"] = "redis://localhost"
app.register_blueprint(sse, url_prefix='/stream')
#app.route('/send')
def send_message():
sse.publish({"message": "Hello!"}, type='greeting')
return "Message sent!"
And to connect to the event stream from Javascript, it works like this:
var source = new EventSource("{{ url_for('sse.stream') }}");
source.addEventListener('greeting', function(event) {
var data = JSON.parse(event.data);
// do what you want with this data
}, false);
Documentation is available on ReadTheDocs. Note that you'll need a running Redis server to handle pub/sub.

As a committer of https://github.com/WolfgangFahl/pyFlaskBootstrap4 i ran into the same need and created a flask blueprint for Server Sent Events that has no dependency to redis.
This solutions builds on the other answers that have been given here in the past.
https://github.com/WolfgangFahl/pyFlaskBootstrap4/blob/main/fb4/sse_bp.py has the source code (see also sse_bp.py below).
There are unit tests at https://github.com/WolfgangFahl/pyFlaskBootstrap4/blob/main/tests/test_sse.py
The idea is that you can use different modes to create your SSE stream:
by providing a function
by providing a generator
by using a PubSub helper class
by using the PubSub helper class and use pydispatch at the same time.
As of 2021-02-12 this is alpha code which i want to share nevertheless. Please comment here or as issues in the project.
There is a demo at http://fb4demo.bitplan.com/events and a description of the example use e.g. for a progress bar or time display at: http://wiki.bitplan.com/index.php/PyFlaskBootstrap4#Server_Sent_Events
example client javascript/html code
<div id="event_div">Watch this space...</div>
<script>
function fillContainerFromSSE(id,url) {
var targetContainer = document.getElementById(id);
var eventSource = new EventSource(url)
eventSource.onmessage = function(e) {
targetContainer.innerHTML = e.data;
};
};
fillContainerFromSSE("event_div","/eventfeed");
</script>
example server side code
def getTimeEvent(self):
'''
get the next time stamp
'''
time.sleep(1.0)
s=datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S')
return s
def eventFeed(self):
'''
create a Server Sent Event Feed
'''
sse=self.sseBluePrint
# stream from the given function
return sse.streamFunc(self.getTimeEvent)
sse_bp.py
'''
Created on 2021-02-06
#author: wf
'''
from flask import Blueprint, Response, request, abort,stream_with_context
from queue import Queue
from pydispatch import dispatcher
import logging
class SSE_BluePrint(object):
'''
a blueprint for server side events
'''
def __init__(self,app,name:str,template_folder:str=None,debug=False,withContext=False):
'''
Constructor
'''
self.name=name
self.debug=debug
self.withContext=False
if template_folder is not None:
self.template_folder=template_folder
else:
self.template_folder='templates'
self.blueprint=Blueprint(name,__name__,template_folder=self.template_folder)
self.app=app
app.register_blueprint(self.blueprint)
#self.app.route('/sse/<channel>')
def subscribe(channel):
def events():
PubSub.subscribe(channel)
self.stream(events)
def streamSSE(self,ssegenerator):
'''
stream the Server Sent Events for the given SSE generator
'''
response=None
if self.withContext:
if request.headers.get('accept') == 'text/event-stream':
response=Response(stream_with_context(ssegenerator), content_type='text/event-stream')
else:
response=abort(404)
else:
response= Response(ssegenerator, content_type='text/event-stream')
return response
def streamGen(self,gen):
'''
stream the results of the given generator
'''
ssegen=self.generateSSE(gen)
return self.streamSSE(ssegen)
def streamFunc(self,func,limit=-1):
'''
stream a generator based on the given function
Args:
func: the function to convert to a generator
limit (int): optional limit of how often the generator should be applied - 1 for endless
Returns:
an SSE Response stream
'''
gen=self.generate(func,limit)
return self.streamGen(gen)
def generate(self,func,limit=-1):
'''
create a SSE generator from a given function
Args:
func: the function to convert to a generator
limit (int): optional limit of how often the generator should be applied - 1 for endless
Returns:
a generator for the function
'''
count=0
while limit==-1 or count<limit:
# wait for source data to be available, then push it
count+=1
result=func()
yield result
def generateSSE(self,gen):
for result in gen:
yield 'data: {}\n\n'.format(result)
def enableDebug(self,debug:bool):
'''
set my debugging
Args:
debug(bool): True if debugging should be switched on
'''
self.debug=debug
if self.debug:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s.%(msecs)03d %(levelname)s:\t%(message)s', datefmt='%Y-%m-%d %H:%M:%S')
def publish(self, message:str, channel:str='sse', debug=False):
"""
Publish data as a server-sent event.
Args:
message(str): the message to send
channel(str): If you want to direct different events to different
clients, you may specify a channel for this event to go to.
Only clients listening to the same channel will receive this event.
Defaults to "sse".
debug(bool): if True enable debugging
"""
return PubSub.publish(channel=channel, message=message,debug=debug)
def subscribe(self,channel,limit=-1,debug=False):
def stream():
for message in PubSub.subscribe(channel,limit,debug=debug):
yield str(message)
return self.streamGen(stream)
class PubSub:
'''
redis pubsub duck replacement
'''
pubSubByChannel={}
def __init__(self,channel:str='sse',maxsize:int=15, debug=False,dispatch=False):
'''
Args:
channel(string): the channel name
maxsize(int): the maximum size of the queue
debug(bool): whether debugging should be switched on
dispatch(bool): if true use the pydispatch library - otherwise only a queue
'''
self.channel=channel
self.queue=Queue(maxsize=maxsize)
self.debug=debug
self.receiveCount=0
self.dispatch=False
if dispatch:
dispatcher.connect(self.receive,signal=channel,sender=dispatcher.Any)
#staticmethod
def reinit():
'''
reinitialize the pubSubByChannel dict
'''
PubSub.pubSubByChannel={}
#staticmethod
def forChannel(channel):
'''
return a PubSub for the given channel
Args:
channel(str): the id of the channel
Returns:
PubSub: the PubSub for the given channel
'''
if channel in PubSub.pubSubByChannel:
pubsub=PubSub.pubSubByChannel[channel]
else:
pubsub=PubSub(channel)
PubSub.pubSubByChannel[channel]=pubsub
return pubsub
#staticmethod
def publish(channel:str,message:str,debug=False):
'''
publish a message via the given channel
Args:
channel(str): the id of the channel to use
message(str): the message to publish/send
Returns:
PubSub: the pub sub for the channel
'''
pubsub=PubSub.forChannel(channel)
pubsub.debug=debug
pubsub.send(message)
return pubsub
#staticmethod
def subscribe(channel,limit=-1,debug=False):
'''
subscribe to the given channel
Args:
channel(str): the id of the channel to use
limit(int): limit the maximum amount of messages to be received
debug(bool): if True debugging info is printed
'''
pubsub=PubSub.forChannel(channel)
pubsub.debug=debug
return pubsub.listen(limit)
def send(self,message):
'''
send the given message
'''
sender=object();
if self.dispatch:
dispatcher.send(signal=self.channel,sender=sender,msg=message)
else:
self.receive(sender,message)
def receive(self,sender,message):
'''
receive a message
'''
if sender is not None:
self.receiveCount+=1;
if self.debug:
logging.debug("received %d:%s" % (self.receiveCount,message))
self.queue.put(message)
def listen(self,limit=-1):
'''
listen to my channel
this is a generator for the queue content of received messages
Args:
limit(int): limit the maximum amount of messages to be received
Return:
generator: received messages to be yielded
'''
if limit>0 and self.receiveCount>limit:
return
yield self.queue.get()
def unsubscribe(self):
'''
unsubscribe me
'''
if self.dispatch:
dispatcher.disconnect(self.receive, signal=self.channel)
pass

sleekxmpp threaded authentication

so... I have a simple chat client like so:
class ChatClient(sleekxmpp.ClientXMPP):
def __init__(self, jid, password, server):
sleekxmpp.ClientXMPP.__init__(self, jid, password, ssl=True)
self.add_event_handler("session_start", self.start)
self.register_plugin('xep_0030')
self.register_plugin('xep_0004')
self.register_plugin('xep_0060')
self.register_plugin('xep_0199')
self.ssl_version = ssl.PROTOCOL_SSLv3
self.connected = self.connect()
if self.connected:
self.process(threaded=True)
def start(self, event):
self.send_presence(priority = "-9001")
self.get_roster(blocking = True, timeout = 3)
def message(self, targets, msg):
for target in targets:
self.send_message(target, msg)
and I have an "verify" function to make sure you input your username/pass right:
def authenticate(username, password, server):
xmppuser = username + '#' + server
passTester = ChatClient(xmppuser, password)
passTester.disconnect(wait = True)
authenticated = passTester.authenticated
return authenticated
Now, the problem comes in where I have the chat client as threaded, I run into the situation where I try to check ChatClient.authenticated before the server had a chance to actually connect. As you can see, I tried to "wait" on the disconnect but there's nothing in the send queue so it disconnects right away.
An alternate I tried is this:
def authenticate(username, password, server):
xmppuser = username + '#' + server
passTester = ChatClient(xmppuser, password)
passTester.message('bogusName', 'ladfhkjdglkhjdfg')
passTester.disconnect(wait = True)
authenticated = passTester.authenticated
return authenticated
Now that I sent a bogus message the disconnect call has something to wait for. when I input a correct username/pass with this code, the disconnect waits for a message to get sent (which involves waiting for a real connection), sends nothing and ChatClient.authenticated is set to True!
Unfortunately, when I input a wrong username/pass the message never gets sent and the disconnect(wait=True) never disconnects as the message never gets sent.
Does anyone else have a more proper way to "authenticate"?

This would be a good reason for changing the .authenticated and related fields to be threading.Event objects so that you could use wait() for situations like this, but I'm not sure how much that would break existing user code.
But short of modifying SleekXMPP, what you will need to do is wait for certain events to fire. For example, if your client successfully authenticated, then there will be a session_start event (I may add an auth_success or similar event later). Likewise, if authentication failed for a single mechanism, there will be a failed_auth event. If no authentication methods at all succeeded (which is what you'd be interested in), there will be a no_auth event.
So you can add handlers for these events, and have those handlers place a token in a queue, and then wait for the desired token to arrive.
class ChatClient(ClientXMPP):
def __init__(self, ...):
...
self.auth_queue = queue.Queue()
self.add_event_handler('no_auth', self.failed)
def start(self, event):
self.auth_queue.put('success')
...
def failed(self, event):
self.auth_queue.put('failed')
def authenticate(username, password, server):
xmppuser = username + '#' + server
passTester = ChatClient(xmppuser, password)
try:
result = passTester.auth_queue.get(timeout=10)
except queue.Empty:
result = 'failed'
passTester.disconnect()
return result == 'success'
Don't forget that we also have the chat room at sleek#conference.jabber.org.
-- Lance

How to consolidate two Python classes containing methods with identical signatures?

I'm working on a REST client library, and recently started working on adding support for sending batch messages.
However, I don't like the current design. It requires you to maintain large Client and RequestMessage classes with identical method signatures.
I am looking for a way to consolidate the two classes.
The original Client class's methods contained all the code needed to prepare and send the request:
class Client(object):
def __init__(self, config):
self.request = Request(config, "application/json")
def create_vertex(self, data):
path = "vertices"
params = remove_null_values(data)
self.request.post(path, params)
To add support for batch messages, I pulled out the guts of the code inside the each Client method, and put it into a separate RequestMessage class so that you can add the messages to the batch without sending it, until you're ready:
class Client(object):
def __init__(self, config):
self.request = Request(config, "application/json")
self.message = RequestMessage(config)
def create_vertex(self, data):
message = self.message.create_vertex(data)
return self.request.send(message)
# ...more REST client methods...
def batch(self, messages):
path = "batch"
params = messages
return self.request.post(path, params)
class RequestMessage(object):
def __init__(self, config):
self.config = config
def create_vertex(self, data):
path = "vertices"
params = remove_null_values(data)
return POST, path, params
# ...more REST client methods...
But I don't like the design because now you have to maintain the Client and RequestMessage class -- two large classes with identical signatures.
Here's what the Batch class looks like:
class Batch(object):
def __init__(self, client):
self.client = client
self.messages = []
def add(self, message):
self.messages.append(message)
def send(self):
return self.client.batch(self.messages)
Here's example usage for creating a vertex on the server:
>>> client = Client(config)
>>> vertex = client.create_vertex({'name':'James'})
Here's example usage for creating a batch of vertices on the server:
>>> message1 = client.message.create_vertex({'name':'James'})
>>> message2 = client.message.create_vertex({'name':'Julie'})
>>> batch = Batch(client)
>>> batch.add(message1)
>>> batch.add(message2)
>>> batch.send()
Batch is used less frequently than Client so I want to make the normal Client interface easiest to use.
Here's one idea, but I'm not quite sure how to achieve it or if something else would be better:
>>> vertex = client.create_vertex(data)
>>> message = client.create_vertex(data).message()

My personal preference is an API like:
client.create_vertex({'name':'James'}) # single
client.create_vertex([{'name':'James'}, {'name':'Julie'}]) # batch
That way you use the exact same function, you're just giving it more data. Typical usage would probably look more like:
batch = []
batch.append({'name':'James'})
batch.append({'name':'Julie'})
client.create_vertex(batch)

I would agree with #Karl with some modifications
client.create_vertex({'name':'James'}) # single
client.create_vertex({'name':'James'}, {'name':'Julie'}) # batch
batch = []
batch.append({'name':'James'})
batch.append({'name':'Julie'})
client.create_vertex(*batch)
That way you don't have to check the type of your input. Easier to write, easier to use.