I am looking for a solution for using FastAPI with Redis pubsub without using threads. I have looked around for solutions using aioredis, redis-py and web sockets, but I can't seem to make them work.
Would love to have a simple example of a unicorn server running, while subscribed to a Redis channel, printing the message when receiving it. I am using Python3.6.12 so python3.6 solutions are preferred.
Thanks!
Below is the most recent try on aioredis without FASTapi, can't get the print message when publishing on the Redis in another terminal.
import asyncio
import aioredis
import async_timeout
STOPWORD = "STOP"
async def pubsub():
redis = aioredis.Redis.from_url(
"redis://:password#localhost:6379", max_connections=10, decode_responses=True
)
await redis.publish("channel:1", "abc")
psub = redis.pubsub()
async def reader(channel: aioredis.client.PubSub):
while True:
try:
async with async_timeout.timeout(1):
# print("trying to get message")
message = await channel.get_message(ignore_subscribe_messages=True)
if message is not None:
print(f"(Reader) Message Received: {message}")
if message["data"] == STOPWORD:
print("(Reader) STOP")
break
await asyncio.sleep(0.01)
except asyncio.TimeoutError:
pass
async with psub as p:
await p.subscribe("channel:1")
await reader(p) # wait for reader to complete
await p.unsubscribe("channel:1")
# closing all open connections
await psub.close()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
result = loop.run_until_complete(pubsub())
print('finished')
Related
Im trying to send and receive at the same time from websockets. But the while-loop is blocking my received websockets. Im using python 3.6 so I cannot use asyncio.run but have to stick to asyncio.get_event_loop()
websocket.recv()
receives the message only once. After that it is blocked by the while loop
Here is my code.
import asyncio
import websockets
import json
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
count = 2
while True:
count = count + 1
counting = json.dumps(count)
await websocket.send(counting)
await asyncio.sleep(0.1)
start_server = websockets.serve(hello, "192.168.1.104", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
How can I separate the processes and run them in parallell with asyncio?
currently I'm using websockets to pass through data that I receive from a Redis queue (pub/sub). But for some reason the websocket doesn't send messages when using this redis queue.
What my code looks like
My code works as folllow:
I accept the socket connection
I connect to the redis queue
For each message that I receive from the subscription, i sent a message through the socket. (at the moment only text for testing)
#check_route.websocket_route("/check")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
for result in subscribe.listen():
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e
The issue with the code
When I'm using this code it's just not sending the message through the socket. But when I accept the websocket within the subscribe.listen() loop it does work but it reconnects every time (see code below).
#check_route.websocket_route("/check")
async def websocket_endpoint(websocket: WebSocket):
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
for result in subscribe.listen():
await websocket.accept()
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e
I think that the subscribe.listen() causes some problems that make the websocket do nothing when websocket.accept() is outside the for loop.
I hope someone knows whats wrong with this.
I'm not sure if this will work, but you could try this:
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
redis = Redis(host='::1', port=6379, db=1)
subscribe = redis.pubsub()
subscribe.subscribe('websocket_queue')
try:
results = await subscribe.listen()
for result in results:
await websocket.send_text('test')
print('test send')
except Exception as e:
await websocket.close()
raise e
After a few days more research I found a solution for this issue. I solved it by using aioredis. This solution is based on the following GitHub Gist.
import json
import aioredis
from fastapi import APIRouter, WebSocket
from app.service.config_service import load_config
check_route = APIRouter()
#check_route.websocket("/check")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
# ---------------------------- REDIS REQUIREMENTS ---------------------------- #
config = load_config()
redis_uri: str = f"redis://{config.redis.host}:{config.redis.port}"
redis_channel = config.redis.redis_socket_queue.channel
redis = await aioredis.create_redis_pool(redis_uri)
# ------------------ SEND SUBSCRIBE RESULT THROUGH WEBSOCKET ----------------- #
(channel,) = await redis.subscribe(redis_channel)
assert isinstance(channel, aioredis.Channel)
try:
while True:
response_raw = await channel.get()
response_str = response_raw.decode("utf-8")
response = json.loads(response_str)
if response:
await websocket.send_json({
"event": 'NEW_CHECK_RESULT',
"data": response
})
except Exception as e:
raise e
I am learning how to use the websockets package for python 3.6 with asyncio.
Using the Websockets Getting Started example, here are my server and client code (both running in two separate console using python <script>)
wsserver.py
import asyncio
import websockets
msg_queue = asyncio.Queue()
async def consumer_handler(websocket):
global msg_queue
while True:
message = await websocket.recv()
print("Received message {}".format(message))
await msg_queue.put("Hello {}".format(message))
print("Message queued")
async def producer_handler(websocket):
global msg_queue
while True:
print("Waiting for message in queue")
message = await msg_queue.get()
print("Poped message {}".format(message))
websocket.send(message)
print("Message '{}' sent".format(message))
async def handler(websocket, path):
print("Got a new connection...")
consumer_task = asyncio.ensure_future(consumer_handler(websocket))
producer_task = asyncio.ensure_future(producer_handler(websocket))
done, pending = await asyncio.wait([consumer_task, producer_task]
, return_when=asyncio.FIRST_COMPLETED)
print("Connection closed, canceling pending tasks")
for task in pending:
task.cancel()
start_server = websockets.serve(handler, 'localhost', 5555)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
wsclient.py
import asyncio
import websockets
async def repl():
async with websockets.connect('ws://localhost:5555') as websocket:
while True:
name = input("\nWhat's your name? ")
await websocket.send(name)
print("Message sent! Waiting for server answer")
greeting = await websocket.recv()
# never goes here
print("> {}".format(greeting))
asyncio.get_event_loop().run_until_complete(repl())
During the execution, the server is doing what is expected of him :
Wait for a client message
Queue 'Hello $message'
Dequeue it
Send the dequeued message back to the sender
The client does work up to the waiting of the server response :
Wait for a user input
Send it to the server
Wait answer from the server <-- Holds on indefinitely
Print it & loop
Here are the console outputs of the execution :
Server
Got a new connection...
Waiting for message in queue
Received message TestName
Message queued
Poped message Hello TestName
Message 'Hello TestName' sent
Waiting for message in queue
Client
What's your name? TestName
Message sent! Waiting for server answer
_
What am I missing?
Server-side, you're missing an await on the websocket.send(message) line.
To find those kind of bugs, start your program with the PYTHONASYNCIODEBUG environment variable, like: PYTHONASYNCIODEBUG=1 python3 wsserver.py which prints:
<CoroWrapper WebSocketCommonProtocol.send() running at […]/site-packages/websockets/protocol.py:301, created at wsserver.py:23> was never yielded from
I need to write some async code which runs a subprocess as part of its tasks. Even though I am using asyncio.subprocess my code is still blocking. My server looks like this:
import asyncio
import asyncio.subprocess
import websockets
async def handler(websocket, path):
while True:
data = await websocket.recv()
print('I received a message')
player = await asyncio.create_subprocess_exec(
'sleep', '5',
stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL)
await player.wait()
print('Finished waiting')
server = websockets.serve(handler, '0.0.0.0', '8000')
asyncio.get_event_loop().run_until_complete(server)
asyncio.get_event_loop().run_forever()
And a very basic client:
import asyncio
import websockets
async def client():
async with websockets.connect('ws://localhost:8000') as websocket:
for i in range(5):
await websocket.send('message')
await asyncio.sleep(0.5)
asyncio.get_event_loop().run_until_complete(client())
I would expect the output to look like this:
I received a message
I received a message
I received a message
I received a message
I received a message
Finished waiting
Finished waiting
Finished waiting
Finished waiting
Finished waiting
But instead I get this:
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
I received a message
Finished waiting
With a 5 second wait after each "I received a message" line.
The line await player.wait() does not block other async operations, but waits for 5 seconds!
If you don't want to wait for the response, try using ensure_future() instead:
# add:
async def wait_for_player(player, path):
print("Waiting...", path)
await player.wait()
print("Done", path)
# and replace await player.wait() with:
asyncio.ensure_future(wait_for_player(player, path))
You can actually also move create_subprocess_exec() to wait_for_player().
To see your code is not blocking see try these:
Client:
import asyncio
import websockets
async def client(n):
async with websockets.connect('ws://localhost:8000/{}/'.format(n)) as websocket:
print(n, "start")
for i in range(5):
print(n, i)
await websocket.send('message')
await asyncio.sleep(0.5)
print(n, "done")
tasks = [client(i) for i in range(5)]
asyncio.get_event_loop().run_until_complete(asyncio.wait(tasks))
Server:
import asyncio
import asyncio.subprocess
import random
import websockets
async def handler(websocket, path):
try:
while True:
data = await websocket.recv()
pause = random.randint(1, 5)
print('I received a message', path, "Pausing:", pause)
player = await asyncio.create_subprocess_exec(
'sleep', str(pause),
stdin=asyncio.subprocess.DEVNULL,
stdout=asyncio.subprocess.DEVNULL,
stderr=asyncio.subprocess.DEVNULL)
await player.wait()
print('Finished waiting', path)
except websockets.ConnectionClosed:
print("Connection closed!", path)
server = websockets.serve(handler, '0.0.0.0', '8000')
asyncio.get_event_loop().run_until_complete(server)
asyncio.get_event_loop().run_forever()
Your ws server seems ok. Actually it is your client that is blocking. If you want to test the async behavior of your server, You need to make asynchronous requests. The for loop in your client blocks the thread. So remove it and instead, use asyncio.gather to run your client() method 5 times asynchronously
import asyncio
import websockets
async def client():
async with websockets.connect('ws://localhost:8000') as websocket:
await websocket.send('message')
await asyncio.sleep(0.5)
tasks = asyncio.gather(*[client() for i in range(5)])
asyncio.get_event_loop().run_until_complete(tasks)
I am trying to implement a basic websocket client using asyncio and websockets with Python 3.5.2.
Basically, I want connect_to_dealer to be a blocking call, but wait for the websocket message on a different thread.
After reading some docs (I have very little exp with Python), I concluded that asyncio.ensure_future() passing a coroutine (listen_for_message) was the way to go.
Now, I get to run listen_for_message on a different thread, but from within the coroutine I can't seem to use await or any other mechanism to make the calls synchronous. If I do it, the execution waits forever (it hangs) even for a simple sleep.
I'd like to know what I'm doing wrong.
async def listen_for_message(self, future, websocket):
while (True):
try:
await asyncio.sleep(1) # It hangs here
print('Listening for a message...')
message = await websocket.recv() # If I remove the sleep, hangs here
print("< {}".format(message))
future.set_result(message)
future.done()
except websockets.ConnectionClosed as cc:
print('Connection closed')
except Exception as e:
print('Something happened')
def handle_connect_message(self, future):
# We must first remove the websocket-specific payload because we're only interested in the connect protocol msg
print(future.result)
async def connect_to_dealer(self):
print('connect to dealer')
websocket = await websockets.connect('wss://mywebsocket'))
hello_message = await websocket.recv()
print("< {}".format(hello_message))
# We need to parse the connection ID out of the message
connection_id = hello_message['connectionId']
print('Got connection id {}'.format(connection_id))
sub_response = requests.put('https://subscribetotraffic{user_id}?connection={connection_id}'.format(user_id='username', connection_id=connection_id), headers=headers)
if sub_response.status_code == 200:
print('Now we\'re observing traffic')
else:
print('Oops request failed with {code}'.format(code=sub_response.status_code))
# Now we need to handle messages but continue with the regular execution
try:
future = asyncio.get_event_loop().create_future()
future.add_done_callback(self.handle_connect_message)
asyncio.ensure_future(self.listen_for_message(future, websocket))
except Exception as e:
print(e)
Is there a specific reason you need to work with explicit futures?
With asyncio you can use a combination of coroutines and Tasks to achieve most purposes. Tasks are essentially wrapped coroutines that go about cranking themselves over in the background, independently of other async code, so you don't have to explicitly manage their flow or juggle them with other bits of code.
I am not entirely sure of your end goal, but perhaps the approach elaborated below gives you something to work with:
import asyncio
async def listen_for_message(websocket):
while True:
await asyncio.sleep(0)
try:
print('Listening for a message...')
message = await websocket.recv()
print("< {}".format(message))
except websockets.ConnectionClosed as cc:
print('Connection closed')
except Exception as e:
print('Something happened')
async def connect_to_dealer():
print('connect to dealer')
websocket = await websockets.connect('wss://mywebsocket')
hello_message = await websocket.recv()
print("< {}".format(hello_message))
# We need to parse the connection ID out of the message
connection_id = hello_message['connectionId']
print('Got connection id {}'.format(connection_id))
sub_response = requests.put('https://subscribetotraffic{user_id}?connection={connection_id}'.format(
user_id='username', connection_id=connection_id), headers=headers)
if sub_response.status_code == 200:
print('Now we\'re observing traffic')
else:
print('Oops request failed with {code}'.format(code=sub_response.status_code))
async def my_app():
# this will block until connect_to_dealer() returns
websocket = await connect_to_dealer()
# start listen_for_message() in its own task wrapper, so doing it continues in the background
asyncio.ensure_future(listen_for_message(websocket))
# you can continue with other code here that can now coexist with listen_for_message()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(my_app())
loop.run_forever()