Develop Reference

FastAPI with Redis pubsub

I am looking for a solution for using FastAPI with Redis pubsub without using threads. I have looked around for solutions using aioredis, redis-py and web sockets, but I can't seem to make them work.
Would love to have a simple example of a unicorn server running, while subscribed to a Redis channel, printing the message when receiving it. I am using Python3.6.12 so python3.6 solutions are preferred.
Thanks!
Below is the most recent try on aioredis without FASTapi, can't get the print message when publishing on the Redis in another terminal.
import asyncio
import aioredis
import async_timeout
STOPWORD = "STOP"
async def pubsub():
redis = aioredis.Redis.from_url(
"redis://:password#localhost:6379", max_connections=10, decode_responses=True
)
await redis.publish("channel:1", "abc")
psub = redis.pubsub()
async def reader(channel: aioredis.client.PubSub):
while True:
try:
async with async_timeout.timeout(1):
# print("trying to get message")
message = await channel.get_message(ignore_subscribe_messages=True)
if message is not None:
print(f"(Reader) Message Received: {message}")
if message["data"] == STOPWORD:
print("(Reader) STOP")
break
await asyncio.sleep(0.01)
except asyncio.TimeoutError:
pass
async with psub as p:
await p.subscribe("channel:1")
await reader(p) # wait for reader to complete
await p.unsubscribe("channel:1")
# closing all open connections
await psub.close()
if __name__ == '__main__':
loop = asyncio.get_event_loop()
result = loop.run_until_complete(pubsub())
print('finished')

How do I make my async coroutine function not block?

I'm trying to write some concurrent Python.
In my app below, I'm trying to subscribe to a websocket feed, but then continue through the app, so this code should just print hello. Instead, it blocks on feed.__init__.
I'm hoping I can build a websocket client to some update some application state.
How do I do feed.__init__ and continue running my program, while the websocket runs in the background?
def main():
myFeed = feed('XBTUSD')
while True:
"hello"
import websockets
import asyncio
import json
class feed:
def __init__(self, symbol):
self.symbol = symbol
self.uri = "wss://www.bitmex.com/realtime?subscribe=instrument,quote:{}".format(symbol)
asyncio.get_event_loop().run_until_complete(self.socket())
# asyncio.get_event_loop().run_forever()
async def socket(self):
async with websockets.connect(self.uri) as websocket:
while True:
msg = await websocket.recv()
self.process_msg(json.loads(msg))
def process_msg(self, msg):
try:
data = msg['data'][0]
if data['symbol'] == self.symbol:
self.last_message = msg
except:
print("Unhandled")

How can I use unique query for every client connected to websocket

I'm trying to have only one database connection in my websocket and return this information to each client connected. Is it possible to do that?
There is my current code:
import asyncio
import aiopg
import websockets
import logging
import sys
import configparser
config = configparser.ConfigParser()
config.read('config/config.ini')
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logger = logging.getLogger('websockets.server')
logger2 = logging.getLogger('asyncio')
logger2.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())
logger2.addHandler(logging.StreamHandler())
async def listen(websocket, path):
async with aiopg.create_pool(config.get('default', 'connexion_bd'), maxsize=1, pool_recycle=0) as pool:
async with pool.acquire() as conn1:
async with conn1.cursor() as cur:
await cur.execute(config.get('default', 'pg_listen'))
while True:
msg = await conn1.notifies.get()
if msg.payload == 'finish':
return
else:
await websocket.send(msg.payload)
start_server = websockets.serve(listen, 'localhost', config.getint('default', 'port_websocket_server'))
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Currently, every time a client listen on my websocket, I got a new connection in my database. Every client will get same content so I don't need them to connect to database, only my websocket.
I try to split my code (connect on another websocket who is connecting on this one) but I got the same problem.
Any hints will be appreciated.
Thanks

Got it
Need to put the websocket.serve after the query.
import asyncio
import functools
import aiopg
import websockets
import logging
import sys
import configparser
config = configparser.ConfigParser()
config.read('config/config.ini')
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logger = logging.getLogger('websockets.server')
logger2 = logging.getLogger('asyncio')
logger2.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler())
logger2.addHandler(logging.StreamHandler())
# Pour starter
# env/bin/python3 websocket_server_aiopg.py
USERS = set()
async def listen(websocket, path, conn1):
USERS.add(websocket)
while True:
msg = await conn1.notifies.get()
if msg.payload == 'finish':
return
else:
await asyncio.wait(([user.send(msg.payload) for user in USERS]))
async def run_server():
async with aiopg.create_pool(config.get('default', 'connexion_bd')) as pool:
async with pool.acquire() as conn1:
async with conn1.cursor() as cur:
await cur.execute(config.get('default', 'pg_listen'))
async with websockets.serve(functools.partial(listen, conn1=conn1), 'localhost', config.getint('default', 'port_websocket_server')) as ws:
await asyncio.sleep(1_000_000_000) # consider waiting on an exit condition instead
asyncio.get_event_loop().run_until_complete(run_server())

chat app using websocket python+django+angular

I want to make one real-time chat application using websockets and the frontend is angular5.
So, I create websocket in purepython and backend is Django and frontend is angular5.
Myquestion is when i create websocket service in python. So, do i have to make websockets services in angular too?
this is my python websocket service
async def consumer_handler(websocket):
global glob_message
while True:
message = await websocket.recv()
await glob_message.put(message)
print("this went in glob_message: {}".format(message))
async def producer_handler(websocket):
global glob_message
while True:
message = await glob_message.get()
await websocket.send(message)
async def handler(websocket, path):
producer_task = asyncio.ensure_future(producer_handler(websocket))
consumer_task = asyncio.ensure_future(consumer_handler(websocket))
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
if __name__ == '__main__':
glob_message = asyncio.Queue()
start_server = websockets.serve(
handler,
'127.0.0.1', 8788)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
So, i want to create two user can send and receive messages.so must be design in angular ..
so i am asking that i already create one service in python so do i have to create websocket service(Observable subscribers) in angular too?

In your angular side, you should open a connection indeed.
This is how you can use Observables to communicate with your server :
Client side
// open the socket connection
const ws = new WebSocket('ws://127.0.0.1:8788')
// when opened, print all messages
ws.onopen = open => {
Observable.fromEvent(ws, 'message')
.subscribe(message => console.log(message))
}
To send a message, simply use :
ws.send('test')
Server Side
You can use the demo in https://pypi.org/project/websocket-client/ to build your websocket server.
to import WebSocket :
pip install websocket-client
Then :
import websocket
try:
import thread
except ImportError:
import _thread as thread
import time
def on_message(ws, message):
print(message)
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
def run(*args):
for i in range(3):
time.sleep(1)
ws.send("Hello %d" % i)
time.sleep(1)
ws.close()
print("thread terminating...")
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp("ws://127.0.0.1:8788",
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()

Asyncio + aiohttp - redis Pub/Sub and websocket read/write in single handler

I'm currently playing with aiohttp to see how it will perform as a server application for mobile app with websocket connection.
Here is simple "Hello world" example (as gist here):
import asyncio
import aiohttp
from aiohttp import web
class WebsocketEchoHandler:
#asyncio.coroutine
def __call__(self, request):
ws = web.WebSocketResponse()
ws.start(request)
print('Connection opened')
try:
while True:
msg = yield from ws.receive()
ws.send_str(msg.data + '/answer')
except:
pass
finally:
print('Connection closed')
return ws
if __name__ == "__main__":
app = aiohttp.web.Application()
app.router.add_route('GET', '/ws', WebsocketEchoHandler())
loop = asyncio.get_event_loop()
handler = app.make_handler()
f = loop.create_server(
handler,
'127.0.0.1',
8080,
)
srv = loop.run_until_complete(f)
print("Server started at {sock[0]}:{sock[1]}".format(
sock=srv.sockets[0].getsockname()
))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
loop.run_until_complete(handler.finish_connections(1.0))
srv.close()
loop.run_until_complete(srv.wait_closed())
loop.run_until_complete(app.finish())
loop.close()
The problem
Now I would like to use structure described below (node server = python aiohttp). To be more specific, use Redis Pub/Sub mechanism with asyncio-redis to read and write both to websocket connection and Redis in my WebsocketEchoHandler.
WebsocketEchoHandler is a dead simple loop so I'm not sure how should this be done. Using Tornado and brükva I would just use callbacks.
Extra (offtopic perhaps) question
Since I'm using Redis already, which of two approaches should I take:
Like in "classic" web app, have a controller/view for everything, use Redis just for messaging etc.
Web app should be just a layer between client and Redis used also as task queue (simplest Python RQ). Every request should be delegated to workers.
EDIT
Image from http://goldfirestudios.com/blog/136/Horizontally-Scaling-Node.js-and-WebSockets-with-Redis
EDIT 2
It seems that I need to clarify.
Websocket-only handler is shown above
Redis Pub/Sub handler might look like that:
class WebsocketEchoHandler:
#asyncio.coroutine
def __call__(self, request):
ws = web.WebSocketResponse()
ws.start(request)
connection = yield from asyncio_redis.Connection.create(host='127.0.0.1', port=6379)
subscriber = yield from connection.start_subscribe()
yield from subscriber.subscribe(['ch1', 'ch2'])
print('Connection opened')
try:
while True:
msg = yield from subscriber.next_published()
ws.send_str(msg.value + '/answer')
except:
pass
finally:
print('Connection closed')
return ws
This handler just subscribes to Redis channel ch1 and ch2 and sends every received message from those channels to websocket.
I want to have this handler:
class WebsocketEchoHandler:
#asyncio.coroutine
def __call__(self, request):
ws = web.WebSocketResponse()
ws.start(request)
connection = yield from asyncio_redis.Connection.create(host='127.0.0.1', port=6379)
subscriber = yield from connection.start_subscribe()
yield from subscriber.subscribe(['ch1', 'ch2'])
print('Connection opened')
try:
while True:
# If message recived from redis OR from websocket
msg_ws = yield from ws.receive()
msg_redis = yield from subscriber.next_published()
if msg_ws:
# push to redis / do something else
self.on_msg_from_ws(msg_ws)
if msg_redis:
self.on_msg_from_redis(msg_redis)
except:
pass
finally:
print('Connection closed')
return ws
But following code is always called sequentially so reading from websocket blocks reading from Redis:
msg_ws = yield from ws.receive()
msg_redis = yield from subscriber.next_published()
I want reading to be done on event where event is message received from one of two sources.

You should use two while loops - one that handles messages from the websocket, and one that handles messages from redis. Your main handler can just kick off two coroutines, one handling each loop, and then wait on both of them:
class WebsocketEchoHandler:
#asyncio.coroutine
def __call__(self, request):
ws = web.WebSocketResponse()
ws.start(request)
connection = yield from asyncio_redis.Connection.create(host='127.0.0.1', port=6379)
subscriber = yield from connection.start_subscribe()
yield from subscriber.subscribe(['ch1', 'ch2'])
print('Connection opened')
try:
# Kick off both coroutines in parallel, and then block
# until both are completed.
yield from asyncio.gather(self.handle_ws(ws), self.handle_redis(subscriber))
except Exception as e: # Don't do except: pass
import traceback
traceback.print_exc()
finally:
print('Connection closed')
return ws
#asyncio.coroutine
def handle_ws(self, ws):
while True:
msg_ws = yield from ws.receive()
if msg_ws:
self.on_msg_from_ws(msg_ws)
#asyncio.coroutine
def handle_redis(self, subscriber):
while True:
msg_redis = yield from subscriber.next_published()
if msg_redis:
self.on_msg_from_redis(msg_redis)
This way you can read from any of the two potential sources without having to care about the other.

recent we can use async await in python 3.5 and above..
async def task1(ws):
async for msg in ws:
if msg.type == WSMsgType.TEXT:
data = msg.data
print(data)
if data:
await ws.send_str('pong')
## ch is a redis channel
async def task2(ch):
async for msg in ch1.iter(encoding="utf-8", decoder=json.loads):
print("receving", msg)
user_token = msg['token']
if user_token in r_cons.keys():
_ws = r_cons[user_token]
await _ws.send_json(msg)
coroutines = list()
coroutines.append(task1(ws))
coroutines.append(task2(ch1))
await asyncio.gather(*coroutines)
this is what I do.when the websockets need to wait message from mutli source.
main point here is using asyncio.gather to run two corotine together like
#dano mentioned.