Develop Reference

Python - Async callback/receiver for WebSocket

I am trying to implement WebSocket connection to a server (Python app <=> Django app)
Whole system runs in big Asyncio loop with many tasks. Code snippet is just very small testing part.
I am able to send any data to a server at any moment and many of them will be type request something and wait for response. But I would like to have some "always running" handler for all incoming messages. (When something in Django database will change I want to send changes to python app).
How can Include always running receiver/ or add callback to websocket? I am not able to find any solution for this.
My code snippet:
import asyncio, json, websockets, logging
class UpdateConnection:
async def connect(self,botName):
self.sock = await websockets.connect('ws://localhost:8000/updates/bot/'+botName)
async def send(self,data):
try:
await self.sock.send(json.dumps(data))
except:
logging.info("Websocket connection lost!")
# Find a way how to reconenct... or make socket reconnect automatically
if __name__ == '__main__':
async def DebugLoop(socketCon):
await socketCon.connect("dev")
print("Running..")
while True:
data = {"type": "debug"}
await socketCon.send(data)
await asyncio.sleep(1)
uSocket = UpdateConnection()
loop = asyncio.get_event_loop()
loop.create_task(DebugLoop(uSocket))
loop.run_forever()
My debug server after connection will start sending random messages to the client in random intervals and I would like to somehow handle them in async way.
Thanks for any help :)

You don't have to do it so complicated. First of all I suggest you use the context patterns offered by websockets module.
From the documentation:
connect() can be used as an infinite asynchronous iterator to reconnect automatically on errors:
async for websocket in websockets.connect(...):
try:
...
except websockets.ConnectionClosed:
continue
Additionally, you simply keep the connection alive by awaiting incoming messages:
my_websocket = None
async for websocket in websockets.connect('ws://localhost:8000/updates/bot/' + botName):
try:
my_websocket = websocket
async for message in websocket:
pass # here you could also process incoming messages
except websockets.ConnectionClosed:
my_websocket = None
continue
As you can see we have a nested loop here:
The outer loop constantly reconnects to the server
The inner loop processes one incoming message at a time
If you are connected, and no messages are coming in from the server, this will just sleep.
The other thing that happens here is that my_websocket is set to the active connection, and unset again when the connection is lost.
In other parts of your script you can use my_websocket to send data. Note that you will need to check if it is currently set wherever you use it:
async def send(data):
if my_websocket:
await my_websocket.send(json.dumps(data))
This is just an illustration, you can also keep the websocket object as an object member, or pass it to another component through a setter function, etc.

python websockets - how to do a simple synchronous send command?

I'm new to websockets. I've been using the examples on the Getting Started page of the websockets docs, mainly the Synchronization Example.
In this use case, I have a sqlite3 database on localhost. I edit that database from a python GUI program on localhost which just imports the database code layer directly. The client then tells the websocket server to send out some extracted data to all clients.
(Eventually this will be on a LAN, with the server machine running a Flask API.)
This is working, with the code below, but it's not clear if I'm doing it correctly. Basically I want to send websockets messages when certain database activity takes place, and I'm confused about how to do a 'simple' non-async send, when invoked from code, ultimately in response to a GUI interaction, as opposed to doing a send in response to an incoming websocket message. In pseudo-code:
def send(ws,msg):
ws.send(msg)
send(ws,'OK!')
The way I'm accomplishing that is wrapping the async def that does the sending in a non-async 'vanilla' def.
The websocket server code:
# modified from https://websockets.readthedocs.io/en/stable/intro.html#synchronization-example
import asyncio
import websockets
USERS = set()
async def register(websocket):
print("register: "+str(websocket))
USERS.add(websocket)
async def unregister(websocket):
print("unregister: "+str(websocket))
USERS.remove(websocket)
# each new connection calls trackerHandler, resulting in a new USERS entry
async def trackerHandler(websocket, path):
await register(websocket)
try:
async for message in websocket:
await asyncio.wait([user.send(message) for user in USERS])
finally:
await unregister(websocket)
start_server = websockets.serve(trackerHandler, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
in the database interface code (on localhost, this file is just imported directly to the GUI app; but on the LAN server, this is the file specified in the WSGI call in Flask):
import asyncio
import websockets
# uri = "ws://localhost:8765"
# wrap the asynchronous send function inside a synchronous function
def wsSend(uri,msg):
async def send():
async with websockets.connect(uri) as websocket:
# await websocket.send(str.encode(str(msg)))
await websocket.send(json.dumps({"msg":msg}))
# print(f"> {msg}")
# greeting = await websocket.recv()
# print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(send())
...
...
def tdbPushTables(uri,teamsViewList=None,assignmentsViewList=None,teamsCountText="---",assignmentsCountText="---"):
# uri = "ws://localhost:8765"
if not teamsViewList:
teamsViewList=tdbGetTeamsView()
if not assignmentsViewList:
assignmentsViewList=tdbGetAssignmentsView()
if uri=='pusher':
pusher_client.trigger('my-channel','teamsViewUpdate',teamsViewList)
pusher_client.trigger('my-channel','assignmentsViewUpdate',teamsViewList)
else:
wsSend(uri,json.dumps({
"teamsView":teamsViewList,
"assignmentsView":assignmentsViewList,
"teamsCount":teamsCountText,
"assignmentsCount":assignmentsCountText}))
it's actually the client that initiates the call to tdbPushTables:
def buildLists(self):
self.teamsList=tdbGetTeamsView()
self.assignmentsList=tdbGetAssignmentsView()
self.updateCounts()
tdbPushTables('ws://localhost:8765',self.teamsList,self.assignmentsList,self.teamsCountText,self.assignmentsCountText)
Feels spooky. Is it spooky or is this actually the right way to do it? Should I be using the websockets module for the server, but a different module to do the 'simple'/synchronous sending of the websocket message to the server?
Two known side effects of this solution: 1) it opens and closes the websocket connection on every call - probably not really a problem...?, and 2) it results in non-fatal handled messages like this in the server transcript:
register: <websockets.server.WebSocketServerProtocol object at 0x041C46F0>
Task exception was never retrieved
future: <Task finished coro=<WebSocketCommonProtocol.send() done, defined at C:\Users\caver\AppData\Roaming\Python\Python37\site-packages\websockets\protocol.py:521> exception=ConnectionClosedOK('code = 1000 (OK), no reason')>
Traceback (most recent call last):
File "C:\Users\caver\AppData\Roaming\Python\Python37\site-packages\websockets\protocol.py", line 555, in send
await self.ensure_open()
File "C:\Users\caver\AppData\Roaming\Python\Python37\site-packages\websockets\protocol.py", line 812, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedOK: code = 1000 (OK), no reason
unregister: <websockets.server.WebSocketServerProtocol object at 0x041C46F0>
EDIT: looks like the websocket (singular) module has a synchronous interface, and the websockets (plural) docs explain that if you want to go synchronous you should use a different module; so, this works:
(instead of importing asyncio and websockets)
from websocket import create_connection
def wsSend(uri,msg):
ws=create_connection(uri)
ws.send(json.dumps({"msg":msg}))
ws.close()
It does still result in the same handled traceback showing up in the server transcript each time wsSend is called; there's probably a way to silence that traceback output, but, regardless, it still doesn't seem to affect anything.

Your code feels spooky, because you are mixing async code with synchronous code.
Based on personal experience, the code is simpler to follow if you keep most of the code asynchronous.
The structure will become something like:
import asyncio
import websockets
async def main():
# Create websocket connection
async with websockets.connect(uri) as websocket:
await your_function_that_does_some_processing(websocket)
asyncio.get_event_loop().run_until_complete(main())
Have in mind that big sections of blocking code can generate trouble.

Sending and receiving frames over the same websocket connection without blocking

Sorry for the long post but I've been poking at this for over a week so I've tried a lot of different stuff. I know Python well enough but I don't have any experience with asyncio or non-blocking functions in Python.
I'm writing an API library/module/package/whatever for a web service that requires a websocket connection. There are many incoming messages to act on, and some control-related (web app level, not websocket control messages) that I need to send on occasion. I can easily receive messages over the connection and act on them. I can send messages, but only in response to received messages because the receive loop is always blocking waiting for messages. I don't want to wait for an incoming messages to process an outgoing one so the script doesn't have to hang on input until a new messages is received. In my struggles to get two-way communication working as desired I discovered I need to use something like Twisted, Tornado, or asyncio but so far every implementation I've tried has failed. Note that the sending has to happen over the same connection. Opening a short-lived connection outside of the receive loop will not work. Here's what I've done so far:
The first iteration of the websocket code was using the websocket-client package. It was very close to the example from the docs:
import websocket
try:
import thread
except ImportError:
import _thread as thread
import time
def on_message(ws, message):
# Send message frames to respective functions
# for sorting, objectification, and processing
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
def run(*args):
# Send initial frames required for server to send the desired frames
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp(buildWebsocketURL()),
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
This blocks any further execution outside of the loop. I tried learning up on the _thread module but I couldn't find any indication that I could "communicate" with the websocket thread from outside. I tried setting up a pub/sub listener function that would forward data to ws.send() from another sender function but it didn't work. No errors or anything, just no indication of any sent messages.
Next I tried the Websockets module. This one seems to be built from the ground up to utilize asyncio. Again, I got a client build that would send initial messages and act on received messages but the progress stopped there:
async def wsconnection():
async with websockets.connect(getWebsocketURL()) as websocket:
while True:
message = await websocket.recv()
if message == '{"type":"broadcaster.ready"}':
subscriptions = getSubscriptions() # Get subscriptions from ident data
logging.info('Sending bookmarks to server as subscription keys')
subscriptionupdate = '{{"type": "subscribe","subscription_keys": ["{0}"],"subscription_scope": "update"}}'.format(
'","'.join(subscriptions))
subscriptioncontent = '{{"subscription_keys": ["{0}"],"subscription_scope": "content","type": "subscribe"}}'.format(
'","'.join(subscriptions))
logging.debug(subscriptioncontent)
await websocket.send(subscriptionupdate)
await websocket.send(subscriptioncontent)
await websocket.send(
'{"type":"message_lobby.read","lobby_id":"1","message_id:"16256829"}')
sortframe(message)
asyncio.get_event_loop().run_until_complete(wsconnection())
I tried the aforementioned pub/sub listener applied here to no avail. Upon reading the docs for this module more thoroughly I tried getting the websocket protocol object (that contains the send() and recv() methods) outside of the loop then creating two coroutines(?), one listening for incoming messages and one listening for and sending outgoing messages. So far I've been completely unable to get the websocket protocol object without running the async with websockets.connect(getWebsocketURL()) as websocket: line within the scope of the wsconnection() function. I tried using websocket = websockets.client.connect() which according to the docs I thought would set the protocol object I need but it doesn't. All of the examples I can find don't seem to reveal any apparent way to structure the websockets sender and receiver in the way I require without extensive knowledge of asyncio.
I also poked around with autobahn with similar code structures as above using both asyncio and Twisted but I came up with all the same problems as above.
So far the closest I've gotten was with the Websockets package above. The docs have an example snippet for a send/recv connection but I can't really read what's going on there as it's all very specific to asyncio. I'm really having trouble wrapping my head around asyncio in general and I think a big problem is it seems to have very rapidly evolved recently so there is a ton of very version-specific information floating around that conflicts. Not good for learning, unfortunately. ~~~~This is what I tried using that example and it connects, receives initial messages, then the connection is lost/closed:
async def producer(message):
print('Sending message')
async def consumer_handler(websocket, path):
while True:
message = await websocket.recv()
await print(message)
await pub.sendMessage('sender', message)
async def producer_handler(websocket, path):
while True:
message = await producer()
await websocket.send(message)
async def wsconnect():
async with websockets.connect(getWebsocketURL()) as websocket:
path = "443"
async def handler(websocket, path):
consumer_task = asyncio.ensure_future(
consumer_handler(websocket, path))
producer_task = asyncio.ensure_future(
producer_handler(websocket, path))
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
pub.subscribe(producer, 'sender')
asyncio.get_event_loop().run_until_complete(wsconnect())
So how do I structure this code to get sending and receiving over the same websocket connection? I also have various API calls to make in the same script while the websocket connection is open which further complicates things.
I'm using Python 3.6.6 and this script is intended to be imported as a module into other scripts so the websocket functionality will need to be wrapped up in a function or class for external calls.

I am in the exact same situation as u. I know that this is a very inelegant solution
because it still isn't full-duplex but i can't seem to find any example on the internet or stackoverflow involving asyncio and the websockets module which i used.
I don't think i completely understand your websockets example (is that server-side or client-side code?) but i'm going to explain my situation and "solution" and maybe that would be usable for you too.
So i have a server main function that has a websocket listening for messages in a loop with recv(). When i send "start" it will start a function that will send data every second to the javascript client in the browser. But while the function is sending data i sometimes want to pause or stop the stream of data from my client be sending a stop message. The problem is that when i use recv() while the data sending has already begun the server stops sending data and only waits for a message. I tried threads,multiprocessing and some other stuff but eventually i came to the hopefully temporarily solution of sending a "pong" message to the server immediately after the client receives a piece of data so that the server continues sending data at the next loop iteration or stop sending data if the "pong" message is "stop" instead for example but yeah this is not real duplex just fast half-duplex...
code on my python "server"
async def start_server(self,websocket,webserver_path):
self.websocket = websocket
self.webserver_path = webserver_path
while True:
command = await self.websocket.recv()
print("received command")
if command == "start":
await self.analyze()
asyncio.sleep(1)
in my analyze function:
for i,row in enumerate(data)
await self.websocket.send(json.dumps(row))
msg = await self.websocket.recv()
if msg == "stop":
self.stopFlag = True
return
await asyncio.sleep(1)
main
start_server = websockets.serve(t.start_server, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
code on the javascript client
var ws = new WebSocket("ws://127.0.0.1:5678/");
ws.onmessage = function (event) {
var datapoint = JSON.parse(event.data);
console.log(counter);
counter++;
data.push(datapoint);
if (data.length > 40){
var element = data.shift();
render(data);
}
ws.send("pong");//sending dummy message to let server continue
};
I know it is not THE solution and i hope somebody else provides a better one but since i have the same or very similar problem and there are no other answers i decided to post and i hope it helps.

How can I refactor to accept multiple clients?

I don't understand why server.py Version 1 allows a client to be keyboard-interrupted and restarted, while server.py Version 2 doesn't:
server.py Version1:
import asyncio
async def handle_client(reader, writer):
while True:
request = (await reader.read(128)).decode()
writer.write('Received ok.'.encode())
await writer.drain()
async def main():
loop.create_task(asyncio.start_server(handle_client, 'localhost', 15555))
loop = asyncio.new_event_loop()
loop.create_task(main())
loop.run_forever()
server.py Version 2:
import asyncio
async def handle_client(reader, writer):
while True:
request = (await reader.read(128)).decode()
if request == "hello":
writer.write('Received ok.'.encode())
await writer.drain()
async def main():
loop.create_task(asyncio.start_server(handle_client, 'localhost', 15555))
loop = asyncio.new_event_loop()
loop.create_task(main())
loop.run_forever()
client.py:
import asyncio
async def make_connections():
reader, writer = await asyncio.open_connection('localhost', 15555, loop=loop)
loop.create_task(connect(reader, writer))
async def connect(reader, writer):
writer.write("hello".encode())
await writer.drain()
result = await reader.read(128)
print(result.decode())
loop = asyncio.new_event_loop()
loop.create_task(make_connections())
loop.run_forever()
Version 2 works fine for a single client, but if I send a keyboard interrupt to the client I can no longer connect after I restart the client. It's annoying to ssh in and kill/restart the server every time I alter code in the client. I don't see why the second version doesn't accept the client the second time it attempts to connect.

I don't understand why server.py Version 1 allows a client to be keyboard-interrupted and restarted, while server.py Version 2 doesn't
Both versions have a bug that they don't correctly check for the end-of-file condition. When you interrupt the client, the socket gets closed and reading from it returns EOF, while writing to it raises an exception. Awaiting writer.drain() in version 1 delivers the exception and interrupts the coroutine. (This exception is probably displayed on the server's standard error.)
Version 2 has a problem, though: the if request == "hello" test is false at EOF because reader.read() keeps returning an empty byte string to mark the EOF condition. This prevents await writer.drain() from executing and delivering the exception, so the coroutine remains stuck in an infinite loop. A simple fix is to add something like if not request: break after the read.
Why version 2 gets stuck
But the above doesn't fully explain why in Version 2 the whole server is broken and new clients unable to connect. Surely one would expect await to either return a result or yield control to other coroutines. But the observed behavior is that, despite containing an await in the while loop, the coroutine doesn't allow other coroutines to run!
The problem is that await doesn't mean "pass control to the event loop", as it is often understood. It means "request value from the provided awaitable object, yielding control to the event loop if the object indicates that it does not have a value ready." The part after the if is crucial: if the object does have a value ready, this value will be used immediately without ever deferring to the event loop. In other words, await doesn't guarantee that the event loop will get a chance to run.
A stream at EOF always has data to return - the empty string that marks the EOF. As a result, it never gets suspended and the loop ends up completely blocking the event loop. To guarantee that other tasks get a chance to run, you can add await asyncio.sleep(0) in a loop - but this should not be necessary in correctly written code, where requesting IO data will soon result in a wait, at which point the event loop will kick in. Once the EOF handling bug is corrected, the server will function correctly.

How to detect write failure in asyncio?

As a simple example, consider the network equivalent of /dev/zero, below. (Or more realistically, just a web server sending a large file.)
If a client disconnects early, you get a barrage of log messages:
WARNING:asyncio:socket.send() raised exception.
But I'm not finding any way to catch said exception. The hypothetical server continues reading gigabytes from disk and sending them to a dead socket, with no effort on the client's part, and you've got yourself a DoS attack.
The only thing I've found from the docs is to yield from a read, with an empty string indicating closure. But that's no good here because a normal client isn't going to send anything, blocking the write loop.
What's the right way to detect failed writes, or be notified that the TCP connection has been closed, with the streams API or otherwise?
Code:
from asyncio import *
import logging
#coroutine
def client_handler(reader, writer):
while True:
writer.write(bytes(1))
yield from writer.drain()
logging.basicConfig(level=logging.INFO)
loop = get_event_loop()
coro = start_server(client_handler, '', 12345)
server = loop.run_until_complete(coro)
loop.run_forever()

I did some digging into the asyncio source to expand on dano's answer on why the exceptions aren't being raised without explicitly passing control to the event loop. Here's what I've found.
Calling yield from wirter.drain() gives the control over to the StreamWriter.drain coroutine. This coroutine checks for and raises any exceptions that that the StreamReaderProtocol set on the StreamReader. But since we passed control over to drain, the protocol hasn't had the chance to set the exception yet. drain then gives control over to the FlowControlMixin._drain_helper coroutine. This coroutine the returns immediately because some more flags haven't been set yet, and the control ends up back with the coroutine that called yield from wirter.drain().
And so we have gone full circle without giving control to the event loop to allow it handle other coroutines and bubble up the exceptions to writer.drain().
yielding before a drain() gives the transport/protocol a chance to set the appropriate flags and exceptions.
Here's a mock up of what's going on, with all the nested calls collapsed:
import asyncio as aio
def set_exception(ctx, exc):
ctx["exc"] = exc
#aio.coroutine
def drain(ctx):
if ctx["exc"] is not None:
raise ctx["exc"]
return
#aio.coroutine
def client_handler(ctx):
i = 0
while True:
i += 1
print("write", i)
# yield # Uncommenting this allows the loop.call_later call to be scheduled.
yield from drain(ctx)
CTX = {"exc": None}
loop = aio.get_event_loop()
# Set the exception in 5 seconds
loop.call_later(5, set_exception, CTX, Exception("connection lost"))
loop.run_until_complete(client_handler(CTX))
loop.close()
This should probably fixed upstream in the Streams API by the asyncio developers.

This is a little bit strange, but you can actually allow an exception to reach the client_handler coroutine by forcing it to yield control to the event loop for one iteration:
import asyncio
import logging
#asyncio.coroutine
def client_handler(reader, writer):
while True:
writer.write(bytes(1))
yield # Yield to the event loop
yield from writer.drain()
logging.basicConfig(level=logging.INFO)
loop = asyncio.get_event_loop()
coro = asyncio.start_server(client_handler, '', 12345)
server = loop.run_until_complete(coro)
loop.run_forever()
If I do that, I get this output when I kill the client connection:
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<client_handler() done, defined at aio.py:4> exception=ConnectionResetError(104, 'Connection reset by peer')>
Traceback (most recent call last):
File "/usr/lib/python3.4/asyncio/tasks.py", line 238, in _step
result = next(coro)
File "aio.py", line 9, in client_handler
yield from writer.drain()
File "/usr/lib/python3.4/asyncio/streams.py", line 301, in drain
raise exc
File "/usr/lib/python3.4/asyncio/selector_events.py", line 700, in write
n = self._sock.send(data)
ConnectionResetError: [Errno 104] Connection reset by peer
I'm really not quite sure why you need to explicitly let the event loop get control for the exception to get through - don't have time at the moment to dig into it. I assume some bit needs to get flipped to indicate the connection dropped, and calling yield from writer.drain() (which can short-circuit going through the event loop) in a loop is preventing that from happening, but I'm really not sure. If I get a chance to investigate, I'll update the answer with that info.

The stream based API doesn't have a callback you can specify for when the connection is closed. But the Protocol API does, so use it instead: https://docs.python.org/3/library/asyncio-protocol.html#connection-callbacks