I have an idea of using an event-loop in a websocket-based server (based on websockets library), but I'm struggling to do so, since I really don't get the idea of how to use asyncio and write your own coroutines.
# let us define a dictionary of callbacks awaiting to be triggered
CALLBACKS = dict()
# let us have an infinite loop, which reads new messages from a websocket connection
async def websocket_connection(ws, path):
async for msg in ws:
# we received a message
msg = json.loads(msg)
msg_id = msg['id']
msg_data = msg['data']
# we check, if there is a callback waiting to be triggered with such id
if msg_id in CALLBACKS:
# if such id exists, we get the callback function
callback = CALLBACKS[msg_id]
# we delete the callback from the dictionary
del CALLBACKS[msg_id]
# we call it passing the data from the message in there
callback(msg_data)
else:
# we don't have such id, throw some error
raise Exception("bad id " + msg_id)
# this is the function which submits a message
async def submit(ws, data):
# it generates a random request id
from uuid import uuid4
request_id = str(uuid4())
# registers a callback, which will simply return the received data
CALLBACKS[request_id] = lambda data: # <- this is the part I don't know what it should really do to be able to return the value from the await
# after it encodes the message as json
msg = json.dumps({'id': request_id, 'data': data})
# and sends via the websocket
await ws.send(msg)
# and let us have the actual function, which will be the actual script
async def websocket_script(ws):
# this is what I would like to be able to do in the end.
# pass in commands to the websocket and await their completion
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))
websocket_connection and websocket_script would need to be going side by side for it to work. I suppose gather or some other async function would work. I would really like to get rid of callbacks, since this is the initial purpose of using asyncio in the first place.
How could this be done? It seems like a job for asyncio.
I think what you search is asyncio.Future if I understand you correctly.
FUTURES = dict()
async def websocket_connection(ws, path):
# ...
fut = FUTURES[msg_id]
fut.set_result(msg_data) # will "unblock" await fut below
del FUTURES[msg_id]
# ...
async def submit(ws, data):
from uuid import uuid4
request_id = str(uuid4())
fut = asyncio.Future()
FUTURES[request_id] = fut
msg = json.dumps({'id': request_id, 'data': data})
await ws.send(msg)
# ...
return (await fut)
async def websocket_script(ws):
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))
Related
I am trying to play with asyncio and websockets those days and I am having troubles to reach a specific behaviour.
Context:
I am subscribing to two streams that return timestamps:
The stream "block_timestamp" return timestamps regularly, around each 2 sec.
The stream "mint_start" will only return one timestamp at random time with a timestamp in the future.
Goal:
When the "mint_start" timestamp is superior or equal to the "block_timstamp" timestamp, I would like to execute an other function.
Below is the event listener that works correctly:
async def event_listener(id, params, queue):
async with websockets.connect(url) as ws:
await ws.send(json.dumps({"id": 1, "method": "eth_subscribe", "params": params}))
subscription_response = await ws.recv()
subscription_response = json.loads(subscription_response)
if subscription_response.get("error"):
return subscription_response["error"]["code"], subscription_response["error"]["message"]
else:
print(f"Subscribed with success to stream id: {subscription_response['result']}")
while True:
message = await ws.recv()
await queue.put((id, message))
Here is the function that manage the timestamp returned by the two stream.
async def on_message(queue):
while True:
id, value = await queue.get()
if id == "block_timestamp":
block_timestamp = (json.loads(value)['params']['result']["timestamp"])
else:
start_mint = (decode_single('(uint112,uint112)', bytearray.fromhex(value['params']['result']['data'][2:])))
if start_mint >= block_timestamp:
# execute function
And finally the main function that run the two streams and the message handler concurently:
async def main():
params1 = ["newHeads"]
params2 = ["logs", {"address": ["0xf4003f4efbe8691b60249e6afbd307abe7758adb"],
"topics": [Web3.keccak(text="Sync(uint112,uint112)").hex()]}]
queue = asyncio.Queue()
await asyncio.gather(
event_listener("block_timestamp", params1, queue),
event_listener("mint_start", params2, queue),
on_message(queue)
)
def run():
asyncio.run(main())
When a specific condition is meet at on_message function level: start_mint >= block_timestamp, I would like to execute an other function.
I would like to properly close the websocket and loop event that is running.
However so far I have no clue how to manage this correctly.
Thanks for the help
I am making a discord bot that will grab a json using requests from time to time, and then send the relevant information to a specific channel.
I have the following classes:
Helper, which is the discord bot itself, that runs async from the start, inside an asyncio.gather;
tasker that controls the interval which calls the class that will do the requests. It runs in a different thread so it doesn't stop the async Helper while it waits
getInfo that does the requests, store the info and should talk with Helper
I am having 2 problems right now:
While the tasker is on a different thread, every time I try to talk with Helper via getInfo it gives me the errors RuntimeError: no running event loop and RuntimeWarning: coroutine 'getInfo.discordmsg' was never awaited
If I dont run it on a different thread, however, it does work on the TestStatus: 1 but it makes Helper get stuck and stop running with TestStatus: 2
Anyway, here is the code
import requests
import asyncio
import discord
from discord.ext import commands, tasks
from datetime import datetime, timedelta
import threading
class Helper(discord.Client):
async def on_ready(self):
global discordbot, taskervar
servername = 'ServerName'
discordbot = self
self.servidores = dict()
self.canais = dict()
for i in range(len(self.guilds)):
self.servidores[self.guilds[i].name] = {}
self.servidores[self.guilds[i].name]['guild']=self.guilds[i]
servidor = self.guilds[i]
for k in range(len(servidor.channels)):
canal = servidor.channels[k]
self.canais[str(canal.name)] = canal
if 'bottalk' not in self.canais.keys():
newchan = await self.servidores[self.guilds[i].name]['guild'].create_text_channel('bottalk')
self.canais[str(newchan.name)] = newchan
self.servidores[self.guilds[i].name]['canais'] = self.canais
self.bottalk = self.get_channel(self.servidores[servername]['canais']['bottalk'].id)
await self.msg("Bot online: " + converteHora(datetime.now(),True))
print(f'{self.user} has connected to Discord!')
taskervar.startprocess()
async def msg(self, msg):
await self.bottalk.send(msg)
async def on_message(self, message):
if message.author == self.user:
return
else:
print(message)
class tasker:
def __init__(self):
global discordbot, taskervar
print('Tasker start')
taskervar = self
self.waiter = threading.Event()
self.lastupdate = datetime.now()
self.nextupdate = datetime.now()
self.thread = threading.Thread(target=self.requests)
def startprocess(self):
if not self.thread.is_alive():
self.waiter = threading.Event()
self.interval = 60*5
self.thread = threading.Thread(target=self.requests)
self.thread.start()
def requests(self):
while not self.waiter.is_set():
getInfo()
self.lastupdate = datetime.now()
self.nextupdate = datetime.now()+timedelta(seconds=self.interval)
self.waiter.wait(self.interval)
def stopprocess(self):
self.waiter.set()
class getInfo:
def __init__(self):
global discordbot, taskervar
self.requests()
async def discordmsg(self,msg):
await discordbot.msg(msg)
def requests(self):
jsondata = {"TestStatus": 1}
if jsondata['TestStatus'] == 1:
print('here')
asyncio.create_task(self.discordmsg("SOMETHING WENT WRONG"))
taskervar.stopprocess()
return
elif jsondata['TestStatus'] == 2:
print('test')
hora = converteHora(datetime.now(),True)
asyncio.create_task(self.discordmsg(str("Everything is fine but not now: " + hora )))
print('test2')
def converteHora(dateUTC, current=False):
if current:
response = (dateUTC.strftime("%d/%m/%Y, %H:%M:%S"))
else:
response = (dateutil.parser.isoparse(dateUTC)-timedelta(hours=3)).strftime("%d/%m/%Y, %H:%M:%S")
return response
async def main():
TOKEN = 'TOKEN GOES HERE'
tasker()
await asyncio.gather(
await Helper().start(TOKEN)
)
if __name__ == '__main__':
asyncio.run(main())
Your primary problem is you don't give your secondary thread access to the asyncio event loop. You can't just await and/or create_task a coroutine on a global object (One of many reasons to avoid using global objects in the first place). Here is how you could modify your code to accomplish that:
class tasker:
def __init__(self):
# ...
self.loop = asyncio.get_running_loop()
# ...
class getInfo:
#...
def requests(self):
# replace the create_tasks calls with this.
asyncio.run_coroutine_threadsafe(self.discordmsg, taskervar.loop)
This uses your global variables because I don't want to rewrite your entire program, but I still strongly recommend avoiding them and considering a re-write yourself.
All that being said, I suspect you will still have this bug:
If I dont run it on a different thread, however, it does work on the TestStatus: 1 but it makes Helper get stuck and stop running with TestStatus: 2
I can't tell what would cause this issue and I'm running into trouble reproducing this on my machine. Your code is pretty hard to read and is missing some details for reproducibility. I would imagine that is part of the reason why you didn't get an answer in the first place. I'm sure you're aware of this article but might be worth a re-visit for better practices in sharing code. https://stackoverflow.com/help/minimal-reproducible-example
The code below is intended to send multiple HTTP requests asynchronously in a while loop, and depending on the response from each request(request "X" always returns "XXX", "Y" always returns "YYY" and so on), do something and sleep for interval seconds specified for each request.
However, it throws an error...
RuntimeError: cannot reuse already awaited coroutine
Could anyone help me how I could fix the code to realise the intended behaviour?
class Client:
def __init__(self):
pass
async def run_forever(self, coro, interval):
while True:
res = await coro
await self._onresponse(res, interval)
async def _onresponse(self, res, interval):
if res == "XXX":
# ... do something with the resonse ...
await asyncio.sleep(interval)
if res == "YYY":
# ... do something with the resonse ...
await asyncio.sleep(interval)
if res == "ZZZ":
# ... do something with the resonse ...
await asyncio.sleep(interval)
async def request(something):
# ... HTTP request using aiohttp library ...
return response
async def main():
c = Client()
await c.run_forever(request("X"), interval=1)
await c.run_forever(request("Y"), interval=2)
await c.run_forever(request("Z"), interval=3)
# ... and more
As the error says, you can't await a coroutine more than once. Instead of passing a coroutine into run_forever and then awaiting it in a loop, passing the coroutine's argument(s) instead and await a new coroutine each iteration of the loop.
class Client:
async def run_forever(self, value, interval):
while True:
res = await rqequest(value)
await self._response(response, interval)
You also need to change how you await run_forever. await is blocking, so when you await something with an infinite loop, you'll never reach the next line. Instead, you want to gather multiple coroutines as once.
async def main():
c = Client()
await asyncio.gather(
c.run_forever("X", interval=1),
c.run_forever("Y", interval=2),
c.run_forever("Z", interval=3),
)
The following code is attempting to obtain stream messages from "connections" set. I'm trying to figure out how to extract two float values--one from each stream--to calculate a quotient. The aggregated stream messages appear unordered and non-sequential, and my routine stalls if I declare global variables and attempt to populate them iteratively. I'd like to be able to connect to multiple streams and produce live calculated values/results using the message data. Please help if you can. I'm new to python and asyncio.
Here's the code:
import asyncio
import websockets
import json
connections = set()
connections.add("mystream1")
connections.add("mystream2")
async def handle_socket1(url):
async with websockets.connect(url) as websocket1:
async for message in websocket1:
json_msg = json.loads(message)
if json_msg["X"] == "A":
value1 = float(json_msg["b"])
if json_msg["Y"] == "B":
value2 = float(json_msg["a"])
print(value2/value1)
async def handler1():
await asyncio.wait([handle_socket1(url) for url in connections])
You can funnel both streams into a queue, and have a separate function that pulls either value out of the queue and recalculates the quotient:
async def handle_stream(url, identifier, queue):
async with websockets.connect(url) as websocket1:
async for message in websocket1:
json_msg = json.loads(message)
await queue.put((identifier, json_msg["value"]))
async def calculate(queue):
value_a = value_b = None
while True:
identifier, value = await queue.get()
if identifier == 'A':
value_a = value
else:
value_b = value
if value_a is not None and value_b is not None:
print(value_a / value_b)
async def main():
queue = asyncio.Queue()
await asyncio.gather(
handle_stream("mystream1", "A", queue),
handle_stream("mystream2", "B", queue),
calculate(queue),
)
I am trying to obtain data from a redis channel by using a subscription on my client application. I am using python with asyncio and aioredis for this purpose.
I would like to use my subscription to have a variable of my main application updated when this one changes on the server, but I cannot manage to pass the data received from the subscription to my main thread.
According to aioredis website, I implemented my Subscription with:
sub = await aioredis.create_redis(
'redis://localhost')
ch1 = await sub.subscribe('channel:1')
assert isinstance(ch1, aioredis.Channel)
async def async_reader(channel, globarVar):
while await channel.wait_message():
msg = await channel.get(encoding='utf-8')
# ... process message ...
globarVar = float(msg)
print("message in {}: {}".format(channel.name, msg))
tsk1 = asyncio.ensure_future(async_reader(ch1, upToDateValue))
But I cannot get to update the global variable, I guess python pass just the current value as argument (which I expected to, but wanted to be sure).
Is there any viable option to get data out of a subscription? or to pass a reference to a shared variable or queue I could use?
You should redesign your code so you don't need a global variable. All of your processing should occur when receiving the message. However to modify a global variable you need to declare it in the function with the global keyword. You don't pass global variables around - you just use them.
Sub:
import aioredis
import asyncio
import json
gvar = 2
# Do everything you need here or call another function
# based on the message. Don't use a global variable.
async def process_message(msg):
global gvar
gvar = msg
async def async_reader(channel):
while await channel.wait_message():
j = await channel.get(encoding='utf-8')
msg = json.loads(j)
if msg == "stop":
break
print(gvar)
await process_message(msg)
print(gvar)
async def run(loop):
sub = await aioredis.create_redis('redis://localhost')
res = await sub.subscribe('channel:1')
ch1 = res[0]
assert isinstance(ch1, aioredis.Channel)
await async_reader(ch1)
await sub.unsubscribe('channel:1')
sub.close()
loop = asyncio.get_event_loop()
loop.run_until_complete( run(loop) )
loop.close()
publisher:
import asyncio
import aioredis
async def main():
pub = await aioredis.create_redis('redis://localhost')
res = await pub.publish_json('channel:1', ["Hello", "world"])
await asyncio.sleep(1)
res = await pub.publish_json('channel:1', "stop")
pub.close()
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())