I've got a hobby project of building an autonomous boat. I now built a GUI using a Vuejs frontend and a Django backend. In this GUI I can see the boat on a map, and send commands to it. Those commands are sent over ZeroMQ sockets which works great.
I'm using Django channels to send the commands from the frontend over a websocket to the backend and from there I send it on over the ZeroMQ socket. My consumer (which works great) looks as follows:
import zmq
from channels.generic.websocket import WebsocketConsumer
from .tools import get_vehicle_ip_address, get_vehicle_steer_socket
context = zmq.Context()
class SteerConsumer(WebsocketConsumer):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.forward_steer_socket = get_vehicle_steer_socket(context, get_vehicle_ip_address())
def connect(self):
self.accept()
def receive(self, text_data):
print("Passing on the commands from the frontend:", text_data, "to the boat")
self.forward_steer_socket.send_string(text_data)
Next to this I also receive location information from the boat over a ZeroMQ socket which I save to the database. I'm running this in a separate script and the frontend simply polls the backend every 2 seconds for updates. Here's the script receiving the boat info:
import os
import django
import zmq
os.environ['DJANGO_SETTINGS_MODULE'] = 'server.settings'
django.setup()
# Socket to receive the boat location
context = zmq.Context()
location_socket = context.socket(zmq.SUB)
location_socket.setsockopt(zmq.CONFLATE, True)
location_socket.bind('tcp://*:6001')
location_socket.setsockopt_string(zmq.SUBSCRIBE, '')
while True:
boat_location = location_socket.recv_json()
print(boat_location)
# HERE I STORE THE BOAT LOCATION in the DB
I would now like to add this location_socket to the Consumer so that the Consumer can also receive the boat location on the ZeroMQ socket and send it to the frontend over the websocket.
I can of course simply add the location_socket to the Consumer its __init__() method as follows:
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.forward_steer_socket = get_vehicle_steer_socket(context, get_vehicle_ip_address())
self.location_socket = context.socket(zmq.SUB)
self.location_socket.setsockopt(zmq.CONFLATE, True)
self.location_socket.bind('tcp://*:6001')
self.location_socket.setsockopt_string(zmq.SUBSCRIBE, '')
But I obviously cannot include the while True loop in the Consumer. So from here I'm not sure what to do. I actually don't know whether this is even possible, since Django Channels seems to specifically been made for websockets. I guess I could start using multithreading or multiprocessing libraries, but that is uncharted territory for me.
Does anybody know whether and how it is possible to make a ZeroMQ listener in a Django Channel?
It is possible to send message to your consumer directly from your separated script via: https://channels.readthedocs.io/en/latest/topics/channel_layers.html#using-outside-of-consumers
When the new client connects to your consumer inside SteerConsumer you have self.channel_name which is unique for that client. To send message to that consumer you just have to execute (in your example from separated script):
from channels.layers import get_channel_layer
channel_layer = get_channel_layer()
# "channel_name" should be replaced for the proper name of course
channel_layer.send("channel_name", {
"type": "chat.message",
"text": "Hello there!",
})
and add inside your SteerConsumer method to handle this message:
def chat_message(self, event):
# Handles the "chat.message" event when it's sent to us.
self.send(text_data=event["text"])
Related
I'm working on an application for use in a reception/office environment that allows for the sending of notices to specific offices/computers or in my case for testing at the moment globally to all connected clients. The issue I have faced is that the server claims to send and write to the transport the notification but the client never receives it, all the initial connection data is processed and printed but nothing further unless apart of the main reconnection functions and other methods built-in to the twisted override methods.
Is there perhaps something in the server end that is trying to write to transport but just isn't actually being done? I've split the code in a few files which are all over here: https://github.com/FlopsiBunny/ReceptionDemo but the main culprit is:
# ReceptionDemo/Console/lib/offices.py
def submit(self, *args):
# Submit Notification to Server
title = self.titleEntry.get("1.0", END)
message = self.messageEntry.get("1.0", END)
# Notification Setup
notif = Notification(title, message)
# Send Notification
if self.office == None:
reactor.callFromThread(self.parent.factory.send_global_notice, notif)
# Destroy Window
self.destroy()
# ReceptionDemo/Console/lib/server.py (Function inside the Client Protocol class)
def sendNotice(self, notification):
# Send Notification
token = self.token
notification.update_token(token)
notifData = jsonpickle.encode(notification)
response = {
"token": token,
"payload": notifData
}
responseData = jsonpickle.encode(response)
print("Sending...")
reactor.callFromThread(self.transport.write, responseData.encode("utf-8"))
self.transport.write(str("Tada").encode("utf-8"))
self.console.log("Sending Notice to " + str(self._peer.host + ":" + str(self._peer.port)))
print(responseData)
print("Sent")
If anyone could provide a solution that would be great, I have a feeling it's a threading issue but I am not the most adept at threading.
Found the problem, bug with Tksupport of Twisted and child windows of the root window, one example is dialogs will freeze the Twisted reactor whereas a Toplevel window will deadlock the reactor.
I need to send system logs to the browser and so I have a tornado-based websocket server running like so.
class WSHandler(tornado.websocket.WebSocketHandler):
def check_origin(self, origin):
return True
def get(self, *args, **kwargs):
self.ip = self.get_argument('ip', None)
self.action = self.get_argument('action', None)
super(WSHandler, self).get(self, *args, **kwargs)
def open(self, *args, **kwargs):
clients.append(self)
def on_message(self, message):
ll = eval(message)
for cl in clients:
if cl.ip and cl.ip != ll.user:
continue
if cl.action and cl.action != ll.action:
continue
message = '%s %s' % (ll.action, ll.url)
cl.write_message(message)
def on_close(self):
try:
clients.remove(self)
except ValueError:
pass
The examples I've encountered so far revolve around Tornado-based servers and js-based clients.
What I need, however, is an easy way to connect to this websocket from a Python client, preferably powered by Tornado. The client does not need to receive messages - only send them. I thought I had my answer with this SO post,
How to run functions outside websocket loop in python (tornado)
...but I need to send a message whenever a log event occurs, and preferably from my code that's parsing the events. The examples I've encountered so far revolve around Tornado-based servers and js-based clients. Is there a short & sweet tornado-based client that only sends messages, that can be called from a for-loop?
Also, I developed a complete Tornado WebSocket Client/Server example.
https://github.com/ilkerkesen/tornado-websocket-client-example
If you want WebSocket Authentication/Authorization, look at my other projects trebol and sugar.
There is tornad-websocket-client project. Pay attention on it.
Also there is simple websocket-client to just send messages.
Tornado includes a websocket client: http://www.tornadoweb.org/en/stable/websocket.html#client-side-support
I am trying to understand the examples given here: https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/wamp/basic/pubsub/basic
I built this script which is supposed to handle multiple pub/sub websocket connections and also open a tcp port ( 8123 ) for incoming control messages. When a message comes on the 8123 port, the application should broadcast to all the connected subscribers the message received on port 8123. How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message.
Another thing that i do not understand is the url. The client javascript connects to the url http://:8080/ws . Where does the "ws" come from ?
Also can someone explain the purpose of RouterFactory, RouterSessionFactory and this bit:
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
my code is below:
import sys, time
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory
from twisted.internet.defer import inlineCallbacks
from autobahn.twisted.wamp import ApplicationSession
from autobahn.twisted.util import sleep
class NotificationProtocol(Protocol):
def __init__(self, factory):
self.factory = factory
def dataReceived(self, data):
print "received new data"
class NotificationFactory(Factory):
protocol = NotificationProtocol
class WsNotificationComponent(ApplicationSession):
#inlineCallbacks
def onJoin(self, details):
counter = 0
while True:
self.publish("com.myapp.topic1", "test %d" % counter )
counter += 1
yield sleep(1)
## we use an Autobahn utility to install the "best" available Twisted reactor
##
from autobahn.twisted.choosereactor import install_reactor
reactor = install_reactor()
## create a WAMP router factory
##
from autobahn.wamp.router import RouterFactory
router_factory = RouterFactory()
## create a WAMP router session factory
##
from autobahn.twisted.wamp import RouterSessionFactory
session_factory = RouterSessionFactory(router_factory)
from autobahn.wamp import types
session_factory.add( WsNotificationComponent(types.ComponentConfig(realm = "realm1" )))
from autobahn.twisted.websocket import WampWebSocketServerFactory
transport_factory = WampWebSocketServerFactory(session_factory)
transport_factory.setProtocolOptions(failByDrop = False)
from twisted.internet.endpoints import serverFromString
## start the server from an endpoint
##
server = serverFromString(reactor, "tcp:8080")
server.listen(transport_factory)
notificationFactory = NotificationFactory()
reactor.listenTCP(8123, notificationFactory)
reactor.run()
"How do i make NotificationProtocol or NotificationFactory talk to the websocket and make the websocket server broadcast a message":
Check out one of my other answers on SO: Persistent connection in twisted. Jump down to the example code and model your websocket logic like the "IO" logic and you'll have a good fit (You might also want to see the follow-on answer about the newer endpoint calls from one of the twisted core-team too)
"Where does the "ws" come from ?"
Websockets are implemented by retasking http connections, which by their nature have to have a specific path on the request. That "ws" path typically would map to a special http handler that autobahn is building for you to process websockets (or at least that's what your javascript is expecting...). Assuming thing are setup right you can actually point your web-browswer at that url and it should print back an error about the websocket handshake (Expected WebSocket Headers in my case, but I'm using cyclones websockets not autobahn).
P.S. one of the cool side-effects from "websockets must have a specific path" is that you can actually mix websockets and normal http content on the same handler/listen/port, this gets really handy when your trying to run them all on the same SSL port because your trying to avoid the requirement of a proxy front-ending your code.
I'm working in a project with Python, Twisted and Redis. So the team decided to use txredisapi for the communication between the Python modules and Redis. This project does a lot of different things and we need to subscribe to several channels for listen the messages sent by Redis without the other functionalities stops (asynchronously).
Can one execution handle all the work and listen the messages sent by Redis at the same time or must we separate and execute the code in differents flows?
We use the following code for listen the messages:
import txredisapi as redis
class RedisListenerProtocol(redis.SubscriberProtocol):
def connectionMade(self):
self.subscribe("channelName")
def messageReceived(self, pattern, channel, message):
print "pattern=%s, channel=%s message=%s" %(pattern, channel, message)
def connectionLost(self, reason):
print "lost connection:", reason
class RedisListenerFactory(redis.SubscriberFactory):
maxDelay = 120
continueTrying = True
protocol = RedisListenerProtocol
We try to listen the messages with:
self.connRedisChannels = yield redis.ConnectionPool()
I'm interested to know how can I specify that the Connection must use the "RedisListenerFactory", then I guess that the function "messageReceived" will be fired when a message arrives.
Any suggestions, example or correction will be apreciated.
Thanks!
The following code solves the problem:
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(HOST, PORT)
Thanks to Philippe T. for the help.
If you want to use directly the redis.Connection() may be you can do this before:
redis.SubscriberFactory.protocol = RedisListenerProtocol
the package make internal call to is factory for connection.
other way is to rewrite *Connection class and make*Connection factory to use your factory.
to make the connection on other part of your code you can do something like this :
from twisted.internet.protocol import ClientCreator
from twisted.internet import reactor
# some where :
defer = ClientCreator(reactor, RedisListenerProtocol).connectTCP(__HOST__, __PORT__)
# the defer will have your client when the connection is done
I am building a realtime web application. I want to be able to send broadcast messages from the server-side implementation of my python application.
Here is the setup:
socketio.js on the client-side
TornadIO2 server as Socket.IO server
python on the server-side (Django framework)
I can succesfully send socket.io messages from the client to the server. The server handles these and can send a response. In the following i will describe how i did that.
Current Setup and Code
First, we need to define a Connection which handles socket.io events:
class BaseConnection(tornadio2.SocketConnection):
def on_message(self, message):
pass
# will be run if client uses socket.emit('connect', username)
#event
def connect(self, username):
# send answer to client which will be handled by socket.on('log', function)
self.emit('log', 'hello ' + username)
Starting the server is done by a Django management custom method:
class Command(BaseCommand):
args = ''
help = 'Starts the TornadIO2 server for handling socket.io connections'
def handle(self, *args, **kwargs):
autoreload.main(self.run, args, kwargs)
def run(self, *args, **kwargs):
port = settings.SOCKETIO_PORT
router = tornadio2.TornadioRouter(BaseConnection)
application = tornado.web.Application(
router.urls,
socket_io_port = port
)
print 'Starting socket.io server on port %s' % port
server = SocketServer(application)
Very well, the server runs now. Let's add the client code:
<script type="text/javascript">
var sio = io.connect('localhost:9000');
sio.on('connect', function(data) {
console.log('connected');
sio.emit('connect', '{{ user.username }}');
});
sio.on('log', function(data) {
console.log("log: " + data);
});
</script>
Obviously, {{ user.username }} will be replaced by the username of the currently logged in user, in this example the username is "alp".
Now, every time the page gets refreshed, the console output is:
connected
log: hello alp
Therefore, invoking messages and sending responses works. But now comes the tricky part.
Problems
The response "hello alp" is sent only to the invoker of the socket.io message. I want to broadcast a message to all connected clients, so that they can be informed in realtime if a new user joins the party (for example in a chat application).
So, here are my questions:
How can i send a broadcast message to all connected clients?
How can i send a broadcast message to multiple connected clients that are subscribed on a specific channel?
How can i send a broadcast message anywhere in my python code (outside of the BaseConnection class)? Would this require some sort of Socket.IO client for python or is this builtin with TornadIO2?
All these broadcasts should be done in a reliable way, so i guess websockets are the best choice. But i am open to all good solutions.
I've recently written a very similar application on a similar setup, so I do have several insights.
The proper way of doing what you need is to have a pub-sub backend. There's only so much you can do with simple ConnectionHandlers. Eventually, handling class-level sets of connections starts to get ugly (not to mention buggy).
Ideally, you'd want to use something like Redis, with async bindings to tornado (check out brukva). That way you don't have to mess with registering clients to specific channels - Redis has all that out of the box.
Essentially, you have something like this:
class ConnectionHandler(SockJSConnection):
def __init__(self, *args, **kwargs):
super(ConnectionHandler, self).__init__(*args, **kwargs)
self.client = brukva.Client()
self.client.connect()
self.client.subscribe('some_channel')
def on_open(self, info):
self.client.listen(self.on_chan_message)
def on_message(self, msg):
# this is a message broadcast from the client
# handle it as necessary (this implementation ignores them)
pass
def on_chan_message(self, msg):
# this is a message received from redis
# send it to the client
self.send(msg.body)
def on_close(self):
self.client.unsubscribe('text_stream')
self.client.disconnect()
Note that I used sockjs-tornado which I found to be much more stable than socket.io.
Anyway, once you have this sort of setup, sending messages from any other client (such as Django, in your case) is as easy as opening a Redis connection (redis-py is a safe bet) and publishing a message:
import redis
r = redis.Redis()
r.publish('text_channel', 'oh hai!')
This answer turned out pretty long, so I went the extra mile and made a blog post out of it: http://blog.y3xz.com/blog/2012/06/08/a-modern-python-stack-for-a-real-time-web-application/
I write here, because it's difficult write in comments section. You can view examples for tornadoio2 in examples directory where you can find implementation of chat, and:
class ChatConnection(tornadio2.conn.SocketConnection):
# Class level variable
participants = set()
def on_open(self, info):
self.send("Welcome from the server.")
self.participants.add(self)
def on_message(self, message):
# Pong message back
for p in self.participants:
p.send(message)
As you can see they implemented participants as set ))
If you're already using django, why not have a look at gevent-socketio.