Develop Reference

FastAPI websockets not working when using Redis pubsub functionality

currently I'm using websockets to pass through data that I receive from a Redis queue (pub/sub). But for some reason the websocket doesn't send messages when using this redis queue.
What my code looks like
My code works as folllow:
I accept the socket connection
I connect to the redis queue
For each message that I receive from the subscription, i sent a message through the socket. (at the moment only text for testing)
#check_route.websocket_route("/check")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
for result in subscribe.listen():
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e
The issue with the code
When I'm using this code it's just not sending the message through the socket. But when I accept the websocket within the subscribe.listen() loop it does work but it reconnects every time (see code below).
#check_route.websocket_route("/check")
async def websocket_endpoint(websocket: WebSocket):
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
for result in subscribe.listen():
await websocket.accept()
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e
I think that the subscribe.listen() causes some problems that make the websocket do nothing when websocket.accept() is outside the for loop.
I hope someone knows whats wrong with this.

I'm not sure if this will work, but you could try this:
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
results = await subscribe.listen()
for result in results:
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e

After a few days more research I found a solution for this issue. I solved it by using aioredis. This solution is based on the following GitHub Gist.
import json
import aioredis
from fastapi import APIRouter, WebSocket
from app.service.config_service import load_config
check_route = APIRouter()
#check_route.websocket("/check")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
# ---------------------------- REDIS REQUIREMENTS ---------------------------- #
config = load_config()
redis_uri: str = f"redis://{config.redis.host}:{config.redis.port}"
redis_channel = config.redis.redis_socket_queue.channel
redis = await aioredis.create_redis_pool(redis_uri)
# ------------------ SEND SUBSCRIBE RESULT THROUGH WEBSOCKET ----------------- #
(channel,) = await redis.subscribe(redis_channel)
assert isinstance(channel, aioredis.Channel)
try:
while True:
response_raw = await channel.get()
response_str = response_raw.decode("utf-8")
response = json.loads(response_str)
if response:
await websocket.send_json({
"event": 'NEW_CHECK_RESULT',
"data": response
})
except Exception as e:
raise e

Python asyncio loop.run_forever()

I try to do a trading bot and the most things work fine. But everytime the internet connection is gone for a short time, the code fails. I use asyncio run_forever() function and I think that the code should run forever until it gets stopped, but it does not work.
Here is my code:
import json
async def listen():
url = "wss://phemex.com/ws"
async with websockets.connect(url) as ws:
sub_msg = json.dumps({
"id": sequence,
"method": "kline.subscribe",
"params": [symbol, kline_interval],
})
await ws.send(sub_msg)
while True:
msg = await ws.recv()
msg = json.loads(msg)["kline"]
And I call the loop like this:
loop = asyncio.get_event_loop()
try:
loop.create_task(listen())
asyncio.ensure_future(listen())
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print("Closing Loop")
loop.close()
As soon as the connection is lost, there is the following error:
Task exception was never retrieved
future: <Task finished coro=<listen() done, defined at c:\Users\danis\Desktop\AWS\Dateien\Crypto_Bot_Telegram_PSAR_PHEMEX_inkl_websocket_reconnect.py:351> exception=ConnectionClosedError('code = 1006 (connection closed abnormally [internal]), no reason',)>
Traceback (most recent call last):
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 750, in transfer_data
message = await self.read_message()
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 819, in read_message
frame = await self.read_data_frame(max_size=self.max_size)
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 895, in read_data_frame
frame = await self.read_frame(max_size)
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\protocol.py", line 975, in read_frame
extensions=self.extensions,
File "C:\Users\danis\.conda\envs\python36\lib\site-packages\websockets\legacy\framing.py", line 55, in read
data = await reader(2)
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\streams.py", line 674, in readexactly
yield from self._wait_for_data('readexactly')
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\streams.py", line 464, in _wait_for_data
yield from self._waiter
File "C:\Users\danis\.conda\envs\python36\lib\asyncio\selector_events.py", line 714, in _read_ready
data = self._sock.recv(self.max_size)
ConnectionResetError: [WinError 10054] Eine vorhandene Verbindung wurde vom Remotehost geschlossen
How can I run this code forever?

In asyncio the event loop is responsible of run asynchronous tasks, but it doesn't handle errors they can throw. There are different ways of run the loop, you can run it for execute specific tasks or run it forever, so it keeps running awaiting for new tasks.
In your case the task listen is throwing an uncaught exception (ConnectionResetError), the event loop noitifies you about that with the traceback but it keeps running, maybe with other tasks (that means run forever). You are just notified, and like an error happened in your coroutine, the task stops running.
Solution:
Handle the error of the traceback in your courutine, you must make sure it will run forever, the event loop doesn't do that.
async def listen():
url = "wss://phemex.com/ws"
while True:
try:
async with websockets.connect(url) as ws:
sub_msg = "{\"id\":" + str(
sequence) + ", \"method\": \"kline.subscribe\", \"params\":[\"" + symbol + "\","
+ str(kline_interval) + "]}"
await ws.send(sub_msg)
while True:
msg = await ws.recv()
msg = json.loads(msg)["kline"]
except ConnectionResetError:
print("ConnectionResetError, reconnecting...")

Your loop run forever, but it can't stop the program to crash on unhandled exception. In your case you try to recv data from client that probably lost connection. You can catch this exception by yourself!
while True:
async with websockets.connect(url) as ws:
sub_msg = "{\"id\":" + str(
sequence) + ", \"method\": \"kline.subscribe\", \"params\":[\"" + symbol + "\","
+ str(kline_interval) + "]}"
await ws.send(sub_msg)
while True:
try:
msg = await ws.recv()
except ConnectionResetError:
break
msg = json.loads(msg)["kline"]

'task() takes 0 positional arguments but 1 was given' for python asyncio function

Within my python code, I am trying to design a piece of client code that connects to a WebSockets Server every second and then prints the timestamp and the obtained value from the server in a .csv file. This is given below:
import asyncio
import websockets
import logging
import datetime
import time
starttime = time.time() # start value for timed data acquisition
logger = logging.getLogger("websockets")
logger.setLevel(logging.INFO) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
class Timer: # class for asynchronous (non-blocking) counter
def __init__(self, interval, first_immediately, callback):
self._interval = interval
self._first_immediately = first_immediately
self._callback = callback
self._is_first_call = True
self._ok = True
self._task = asyncio.ensure_future(self._job())
print("init timer done")
async def _job(self):
try:
while self._ok:
if not self._is_first_call or not self._first_immediately:
await asyncio.sleep(self._interval)
await self._callback(self)
self._is_first_call = False
except Exception as ex:
print(ex)
def cancel(self):
self._ok = False
self._task.cancel()
async def test():
async with websockets.connect(
"ws://198.162.1.177:80/", ping_interval=None
) as websocket:
await websocket.send(
str(1.001)
) # send a message to the websocket server
response = (
await websocket.recv()
) # wait to get a response from the server
print(response)
dataline_pv1 = (
datetime.datetime.today().isoformat()
+ ","
+ str(response)
+ ","
+ str(0)
+ "\n"
) # format and assemble data line
file_name_pv1 = (
"{:%Y%m%d}".format(datetime.datetime.today()) + "_flow.csv"
) # generate file name
with open(
file_name_pv1, "a"
) as etherm_file1: # append dataline to file
etherm_file1.write(dataline_pv1)
# asyncio.get_event_loop().run_forever(test()) # run until test() is finished while True:
timer = Timer(interval=1, first_immediately=True, callback=test)
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(test())
loop.run_forever()
except KeyboardInterrupt:
timer.cancel()
pass
finally:
print("Closing Loop")
loop.close()
When this runs, I obtain the following message of my terminal (however the code does not crash):
test() takes 0 positional arguments but 1 was given
I have seen from this question (TypeError: takes 0 positional arguments but 1 was given) that this error occurs when a Class object is not defined properly, but my error seems to be occurring outside of a class framework. In addition, the desired .csv file is produced, however only one line is printed to the file, and does not repeat every second as desired.
What am I missing here? (also I am a complete novice with asyncio programming)
UPDATE: After changing the definition of test() to async def test(timer=None), my code now runs as expected and outputs the values to a .csv file every second (roughly), but still throws up an error. Specifically:
Task exception was never retrieved
future: <Task finished coro=<test() done, defined at flowmeterclient_v2.py:36> exception=ConnectionRefusedError(111, "Connect call failed ('198.162.1.177', 80)")>
Traceback (most recent call last):
File "flowmeterclient_v2.py", line 37, in test
async with websockets.connect("ws://198.162.1.177:80/", ping_interval=None) as websocket:
File "/usr/lib64/python3.6/site-packages/websockets/legacy/client.py", line 604, in __aenter__
return await self
File "/usr/lib64/python3.6/site-packages/websockets/legacy/client.py", line 622, in __await_impl__
transport, protocol = await self._create_connection()
File "/usr/lib64/python3.6/asyncio/base_events.py", line 798, in create_connection
raise exceptions[0]
File "/usr/lib64/python3.6/asyncio/base_events.py", line 785, in create_connection
yield from self.sock_connect(sock, address)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 439, in sock_connect
return (yield from fut)
File "/usr/lib64/python3.6/asyncio/selector_events.py", line 469, in _sock_connect_cb
raise OSError(err, 'Connect call failed %s' % (address,))
ConnectionRefusedError: [Errno 111] Connect call failed ('198.162.1.177', 80)

Your Timer code passes the timer itself as the first argument in
await self._callback(self)
You may wish to change the signature of test to
async def test(timer=None):
so you can call it with or without the timer.

I think you don't really need the Timer here at all. Simply have an asyncio task that loops forever and has an asyncio.sleep() internally.
This also doesn't reconnect to the websocket server for each request, like your previous code did.
import asyncio
import websockets
import logging
import datetime
logger = logging.getLogger("websockets")
logger.setLevel(logging.INFO) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
async def test():
async with websockets.connect(
"ws://198.162.1.177:80/",
ping_interval=None,
) as websocket:
while True:
await websocket.send(str(1.001))
response = await websocket.recv()
print(response)
now = datetime.datetime.now()
dataline_pv1 = f"{now.isoformat()},{response},0\n"
file_name_pv1 = f"{now:%Y%m%d}_flow.csv"
with open(file_name_pv1, "a") as etherm_file1:
etherm_file1.write(dataline_pv1)
await asyncio.sleep(1)
asyncio.run(test())
Following up on comments, if you actually do need to reconnect for each request, you can refactor this like so:
import asyncio
import websockets
import logging
import datetime
logger = logging.getLogger("websockets")
logger.setLevel(
logging.INFO
) # Switch to DEBUG for full error information
logger.addHandler(logging.StreamHandler())
async def get_data():
async with websockets.connect(
"ws://198.162.1.177:80/",
ping_interval=None,
) as websocket:
await websocket.send(str(1.001))
response = await websocket.recv()
return response
def save_data(response):
now = datetime.datetime.now()
dataline_pv1 = f"{now.isoformat()},{response},0\n"
file_name_pv1 = f"{now:%Y%m%d}_flow.csv"
with open(file_name_pv1, "a") as etherm_file1:
etherm_file1.write(dataline_pv1)
async def test():
while True:
response = await get_data()
save_data(response)
await asyncio.sleep(1)
asyncio.run(test())

Send message using websockets with another thread without RuntimeWarning: coroutine 'WebSocketCommonProtocol.send' was never awaited

Am trying to send motion sensor data in json string using websocket client in another thread to avoid execution blocking for the rest of the code down by an infinite loop in MotionSensor class. but apparently the ws.send() needs await keyword. And if i add it throught i get an error
RuntimeWarning: coroutine 'MotionSensors.run' was never awaited
self.run()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
and it doesn't send anything to the server
# motionSensor.py
import threading
import time
from client.ClientRequest import Request
class MotionSensors(threading.Thread):
def __init__(self, ws):
threading.Thread.__init__(self)
self.ws = ws
self.open = True
async def run(self):
await self.SendData()
async def SendData(self):
while self.open:
print("Sending motion state....")
state = 1 # Motion state demo value
request = Request()
request.push("mcu/sensors/motion")
request.addBody({
"state_type": "single",
"devices": {"state": state, "device_no": "DVC-876435"}
})
await self.ws.send(request.getAsJsonString())
print("sleeping now for 2 seconds....")
time.sleep(2)
here is my main code
client.py
# client.py
import settings
import asyncio
import websockets
from client.ClientHandler import Devices
from client.Rounte import Route
from ClientRequest import Request
from client.dbHandler import mcuConfig
from client.devices.motionSensor import MotionSensors
def ResponseMSG(request):
print(request)
route = Route()
route.addRoute("/response", ResponseMSG)
def onMessage(request):
route.fireRequest(request)
async def WsClient():
uri = settings.WS_URL
async with websockets.connect(uri) as websocket:
#####################################
###INITIALIZE DEVICES
motion = MotionSensors(websocket)
motion.start()
while True:
print("waiting to recieve......")
message = await websocket.recv()
onMessage(message)
loop = asyncio.get_event_loop()
loop.run_until_complete(WsClient())
loop.run_forever()
Guys i need your help to send data in another thread with a while loop without blocking the execution of code down and without errors. Thank you so much in advance

Actually i changed the code of motionSensor.py
i created a new event loop and i set it to the new thread
and it worked for the case of even those using python 3.7 and below. it works. All thanks to #user4815162342
# motionSensor.py
import threading
import time
import asyncio
from client.ClientRequest import Request
class MotionSensor(threading.Thread):
def __init__(self, ws):
threading.Thread.__init__(self)
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
self.ws = ws
self.open = True
def run(self):
self.loop.run_until_complete(self.SendData())
# #asyncio.coroutine
async def SendData(self):
while True:
print("Sending motion state....")
state = 0
request = Request()
request.push("mcu/sensors/motion")
request.addBody({
"state_type": "single",
"devices": {"state": state, "device_no": "DVC-876435"}
})
await self.ws.send(request.getAsJsonString())
print("sleeping now for 5 seconds....")
time.sleep(5)

Python Tornado KeyError When removing client from clients set

I have a Python Tornado Websocket server that stores clients in a shared set() so that I know how many clients are connected.
The challenge is that calling on_close after WebSocketClosedError raises a KeyError and the client-instance is not removed from the set of connected clients. This error has caused my server to accumulate over 1000 clients even when the active clients are only around 5.
My Code:
import tornado.iostream
import tornado.websocket
import asyncio
class SocketHandler(tornado.websocket.WebSocketHandler):
socket_active_message = {"status": "Socket Connection Active"}
waiters = set()
def initialize(self):
self.client_name = "newly_connected"
def open(self):
print('connection opened')
# https://kite.com/python/docs/tornado.websocket.WebSocketHandler.set_nodelay
self.set_nodelay(True)
SocketHandler.waiters.add(self)
def on_close(self):
print("CLOSED!", self.client_name)
SocketHandler.waiters.remove(self)
def check_origin(self, origin):
# Override the origin check if needed
return True
async def send_updates(self, message):
print('starting socket service loop')
loop_counter = 0
while True:
try:
await self.write_message({'status': 82317581})
except tornado.websocket.WebSocketClosedError:
self.on_close()
except tornado.iostream.StreamClosedError:
self.on_close()
except Exception as e:
self.on_close()
print('Exception e:', self.client_name)
await asyncio.sleep(0.05)
async def on_message(self, message):
print("RECEIVED :", message)
self.client_name = message
await self.send_updates(message)
def run_server():
# Create tornado application and supply URL routes
webApp = tornado.web.Application(
[
(
r"/",
SocketHandler,
{},
),
]
)
application = tornado.httpserver.HTTPServer(webApp)
webApp.listen(3433)
# Start IO/Event loop
tornado.ioloop.IOLoop.instance().start()
run_server()
The Stack-trace:
Traceback (most recent call last):
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 278, in get
await self.ws_connection.accept_connection(self)
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 881, in accept_connection
await self._accept_connection(handler)
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 964, in _accept_connection
await self._receive_frame_loop()
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 1118, in _receive_frame_loop
await self._receive_frame()
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 1209, in _receive_frame
await handled_future
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback
ret = callback()
File "/mnt/c/Users/EE/projects/new/venv/lib/python3.8/site-packages/tornado/websocket.py", line 658, in <lambda>
self.stream.io_loop.add_future(result, lambda f: f.result())
File "ask_So.py", line 50, in on_message
await self.send_updates(message)
File "ask_So.py", line 39, in send_updates
self.on_close()
File "ask_So.py", line 26, in on_close
SocketHandler.waiters.remove(self)
KeyError: <__main__.SocketHandler object at 0x7ffef9f25520>
I have tried moving the waiters set outside the class but it still produces the same behaviour.
To simulate WebSocketClosedError: open many browser tabs as clients and close one browser tab at a time.

It seems like self.on_close() is being called twice. Once you're calling it manually from inside send_updates() and then later, when a connection is actually closed, Tornado is also calling self.on_close(). Since the self object was already removed from the set the first time, it raises a KeyError the second time.
If you want to close the connection, just call self.close(). The self.on_close() method will be called by Tornado automatically.
Also, you can handle the exception in a try...except block inside on_close.
Update
The previous part of this answer should fix the KeyError related problem. This update is regarding why the clients are not being removed from waiters set.
So, I tested your code and found a major problem with it here:
async def on_message(self, message):
print("RECEIVED :", message)
self.client_name = message
await self.send_updates(message) # <- This is problematic
Whenever a client sends a message, it will run self.send_updates method. So even if there's only one client that sends a message, let's say, 10 times, send_updates will also be called 10 times and, as a result, you will have 10 while loops running simultaneously!
As the number of loops increase, it ultimately blocks the server. That means Tornado has no time to run other code as it's busy juggling so many while loops. Hence, the clients from the waiters are never removed.
Solution
Instead of calling send_updates everytime a message arrives, you can call it just one time. Just have a single while loop to send updates to all clients.
I'd update the code like this:
class SocketHandler(...):
# Make it a classmethod so that it can be
# called without an instance
#classmethod
async def send_updates(cls):
print('starting socket service loop')
loop_counter = 0
while True:
for waiter in cls.waiters:
# use `waiter` instead of `self`
try:
await waiter.write_message({'status': 82317581})
...
await asyncio.sleep(0.05)
Instead of calling send_updates from on_message, you'll have to tel IOLoop to call it once:
def run_server():
...
# schedule SocketHandler.send_updates to be run
tornado.ioloop.IOLoop.current().add_callback(SocketHandler.send_updates)
tornado.ioloop.IOLoop.current().start()
This will have only one while loop running for all clients.