Develop Reference

How to properly use asyncio in this context?

I have an idea of using an event-loop in a websocket-based server (based on websockets library), but I'm struggling to do so, since I really don't get the idea of how to use asyncio and write your own coroutines.
# let us define a dictionary of callbacks awaiting to be triggered
CALLBACKS = dict()
# let us have an infinite loop, which reads new messages from a websocket connection
async def websocket_connection(ws, path):
async for msg in ws:
# we received a message
msg = json.loads(msg)
msg_id = msg['id']
msg_data = msg['data']
# we check, if there is a callback waiting to be triggered with such id
if msg_id in CALLBACKS:
# if such id exists, we get the callback function
callback = CALLBACKS[msg_id]
# we delete the callback from the dictionary
del CALLBACKS[msg_id]
# we call it passing the data from the message in there
callback(msg_data)
else:
# we don't have such id, throw some error
raise Exception("bad id " + msg_id)
# this is the function which submits a message
async def submit(ws, data):
# it generates a random request id
from uuid import uuid4
request_id = str(uuid4())
# registers a callback, which will simply return the received data
CALLBACKS[request_id] = lambda data: # <- this is the part I don't know what it should really do to be able to return the value from the await
# after it encodes the message as json
msg = json.dumps({'id': request_id, 'data': data})
# and sends via the websocket
await ws.send(msg)
# and let us have the actual function, which will be the actual script
async def websocket_script(ws):
# this is what I would like to be able to do in the end.
# pass in commands to the websocket and await their completion
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))
websocket_connection and websocket_script would need to be going side by side for it to work. I suppose gather or some other async function would work. I would really like to get rid of callbacks, since this is the initial purpose of using asyncio in the first place.
How could this be done? It seems like a job for asyncio.

I think what you search is asyncio.Future if I understand you correctly.
FUTURES = dict()
async def websocket_connection(ws, path):
# ...
fut = FUTURES[msg_id]
fut.set_result(msg_data) # will "unblock" await fut below
del FUTURES[msg_id]
# ...
async def submit(ws, data):
from uuid import uuid4
request_id = str(uuid4())
fut = asyncio.Future()
FUTURES[request_id] = fut
msg = json.dumps({'id': request_id, 'data': data})
await ws.send(msg)
# ...
return (await fut)
async def websocket_script(ws):
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))

Asynchronous, Multiple HTTP requests in a While Loop

The code below is intended to send multiple HTTP requests asynchronously in a while loop, and depending on the response from each request(request "X" always returns "XXX", "Y" always returns "YYY" and so on), do something and sleep for interval seconds specified for each request.
However, it throws an error...
RuntimeError: cannot reuse already awaited coroutine
Could anyone help me how I could fix the code to realise the intended behaviour?
class Client:
def __init__(self):
pass
async def run_forever(self, coro, interval):
while True:
res = await coro
await self._onresponse(res, interval)
async def _onresponse(self, res, interval):
if res == "XXX":
# ... do something with the resonse ...
await asyncio.sleep(interval)
if res == "YYY":
# ... do something with the resonse ...
await asyncio.sleep(interval)
if res == "ZZZ":
# ... do something with the resonse ...
await asyncio.sleep(interval)
async def request(something):
# ... HTTP request using aiohttp library ...
return response
async def main():
c = Client()
await c.run_forever(request("X"), interval=1)
await c.run_forever(request("Y"), interval=2)
await c.run_forever(request("Z"), interval=3)
# ... and more

As the error says, you can't await a coroutine more than once. Instead of passing a coroutine into run_forever and then awaiting it in a loop, passing the coroutine's argument(s) instead and await a new coroutine each iteration of the loop.
class Client:
async def run_forever(self, value, interval):
while True:
res = await rqequest(value)
await self._response(response, interval)
You also need to change how you await run_forever. await is blocking, so when you await something with an infinite loop, you'll never reach the next line. Instead, you want to gather multiple coroutines as once.
async def main():
c = Client()
await asyncio.gather(
c.run_forever("X", interval=1),
c.run_forever("Y", interval=2),
c.run_forever("Z", interval=3),
)

How do you extract data from multiple websocket streams to use in a formula?

The following code is attempting to obtain stream messages from "connections" set. I'm trying to figure out how to extract two float values--one from each stream--to calculate a quotient. The aggregated stream messages appear unordered and non-sequential, and my routine stalls if I declare global variables and attempt to populate them iteratively. I'd like to be able to connect to multiple streams and produce live calculated values/results using the message data. Please help if you can. I'm new to python and asyncio.
Here's the code:
import asyncio
import websockets
import json
connections = set()
connections.add("mystream1")
connections.add("mystream2")
async def handle_socket1(url):
async with websockets.connect(url) as websocket1:
async for message in websocket1:
json_msg = json.loads(message)
if json_msg["X"] == "A":
value1 = float(json_msg["b"])
if json_msg["Y"] == "B":
value2 = float(json_msg["a"])
print(value2/value1)
async def handler1():
await asyncio.wait([handle_socket1(url) for url in connections])

You can funnel both streams into a queue, and have a separate function that pulls either value out of the queue and recalculates the quotient:
async def handle_stream(url, identifier, queue):
async with websockets.connect(url) as websocket1:
async for message in websocket1:
json_msg = json.loads(message)
await queue.put((identifier, json_msg["value"]))
async def calculate(queue):
value_a = value_b = None
while True:
identifier, value = await queue.get()
if identifier == 'A':
value_a = value
else:
value_b = value
if value_a is not None and value_b is not None:
print(value_a / value_b)
async def main():
queue = asyncio.Queue()
await asyncio.gather(
handle_stream("mystream1", "A", queue),
handle_stream("mystream2", "B", queue),
calculate(queue),
)

How to take a "peek" to see whether a read coroutine would complete instantaneously?

I'm trying to write an async bridge between two instant messengers, let's say IRC and Telegram. Here's my code so far that doesn't work:
import asyncio
import logging
import random
async def read_from_irc(sock):
n = random.randint(1, 5)
await asyncio.sleep(n)
return "read_from_irc" + str(n)
async def read_from_telegram(sock):
n = random.randint(1, 5)
await asyncio.sleep(n)
return "read_from_telegram" + str(n)
async def main():
sock1 = await connect_to_irc()
sock2 = await connect_to_telegram()
while True:
w1 = read_from_irc(sock1)
w2 = read_from_telegram(sock2)
waits = {w1, w2}
done, pending = await asyncio.wait(waits)
print(done)
print(pending)
if w1 in done:
msg = await w1
await write_to_telegram(msg)
if w2 in done:
await write_to_irc(msg)
if __name__ == '__main__':
logging.basicConfig(level='DEBUG')
asyncio.run(main())
My problem is that in reality, read_from_irc() and read_from_telegram() are running read() functions that can block if there's no data waiting. Sometimes I want to send a message from IRC to Telegram, sometimes the other way round so I think I need to know when it's a good moment to read from IRC and when there's some data pending on Telegram. How to determine that in this example? This is the error I'm trying to avoid:
RuntimeError: readuntil() called while another coroutine is already waiting for incoming data

Get data out of Redis Subscription not possible?

I am trying to obtain data from a redis channel by using a subscription on my client application. I am using python with asyncio and aioredis for this purpose.
I would like to use my subscription to have a variable of my main application updated when this one changes on the server, but I cannot manage to pass the data received from the subscription to my main thread.
According to aioredis website, I implemented my Subscription with:
sub = await aioredis.create_redis(
'redis://localhost')
ch1 = await sub.subscribe('channel:1')
assert isinstance(ch1, aioredis.Channel)
async def async_reader(channel, globarVar):
while await channel.wait_message():
msg = await channel.get(encoding='utf-8')
# ... process message ...
globarVar = float(msg)
print("message in {}: {}".format(channel.name, msg))
tsk1 = asyncio.ensure_future(async_reader(ch1, upToDateValue))
But I cannot get to update the global variable, I guess python pass just the current value as argument (which I expected to, but wanted to be sure).
Is there any viable option to get data out of a subscription? or to pass a reference to a shared variable or queue I could use?

You should redesign your code so you don't need a global variable. All of your processing should occur when receiving the message. However to modify a global variable you need to declare it in the function with the global keyword. You don't pass global variables around - you just use them.
Sub:
import aioredis
import asyncio
import json
gvar = 2
# Do everything you need here or call another function
# based on the message. Don't use a global variable.
async def process_message(msg):
global gvar
gvar = msg
async def async_reader(channel):
while await channel.wait_message():
j = await channel.get(encoding='utf-8')
msg = json.loads(j)
if msg == "stop":
break
print(gvar)
await process_message(msg)
print(gvar)
async def run(loop):
sub = await aioredis.create_redis('redis://localhost')
res = await sub.subscribe('channel:1')
ch1 = res[0]
assert isinstance(ch1, aioredis.Channel)
await async_reader(ch1)
await sub.unsubscribe('channel:1')
sub.close()
loop = asyncio.get_event_loop()
loop.run_until_complete( run(loop) )
loop.close()
publisher:
import asyncio
import aioredis
async def main():
pub = await aioredis.create_redis('redis://localhost')
res = await pub.publish_json('channel:1', ["Hello", "world"])
await asyncio.sleep(1)
res = await pub.publish_json('channel:1', "stop")
pub.close()
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())