Develop Reference

How to make the body of a loop awaited?

I have two functions that work, let's independently. One generates data inside a loop and and saves them on a db. The other is pinging the db every few seconds and if new data have been saved it plots them. In pseudocode it looks like:
def dashborard():
ping_db(time_interval)
if new_values:
plot(new_values)
def generate_data(d):
## do something with d
## and return some values
return values
for i in range(10):
generate_data(i)
I would like to run the dashboard and the data-generating loop asynchronously but I havent managed to do it unless I put an await asyncio.sleep() call inside the function the runs the loop. Again, in pseudocode this is what I have done:
import asyncio
def run_async(self):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main())
return result
async def main():
asyncio.create_task(dashboard())
out = await self.run_loop()
return out
async def run_loop(self):
for i in range(10):
values = await generate_data(i)
await asyncio.sleep(some_seconds) ## <---- How can I remove this?
return values
The await asycion,sleep() argument is at least the number of seconds each loop takes. In this case, the dashboard opens and works fine. If I remove it however, then the dashboard doesnt open until we reach the very end of the loop.
How can I have the loop and the dashboard run asynchronously please? Any help is greatly appreciated.

The (self) pieces in your code suggest use of a class??? Can't figure this. However assuming these are in the same event loop (otherwise why use asyncio???) the following illustrates an approach using asyncio Queues.
import asyncio
async def dashboard(queue):
while True:
new_value = await queue.get()
if new_value == '__END__':
return
print(new_value)
async def generate_data(i):
"""
Presumably your database access awaits the database update.
Unless you await something here the co-routines cannot co-operate.
----------- ----------
"""
await asyncio.sleep(1.0)
return f"something from {i}"
async def run_loop(queue: asyncio.Queue):
values = []
for i in range(5):
value = await generate_data(i)
values.append(value)
queue.put_nowait(value)
# await asyncio.sleep(some_seconds) # <---- How can I remove this?
queue.put_nowait('__END__')
return values
async def main():
queue = asyncio.Queue()
last = asyncio.create_task(dashboard(queue))
out = await run_loop(queue)
print(out)
await asyncio.gather(last)
if __name__ == "__main__":
asyncio.run(main())
The output is:
something from 0
something from 1
something from 2
something from 3
['something from 0', 'something from 1', 'something from 2', 'something from 3', 'something from 4']
something from 4

How to properly use asyncio in this context?

I have an idea of using an event-loop in a websocket-based server (based on websockets library), but I'm struggling to do so, since I really don't get the idea of how to use asyncio and write your own coroutines.
# let us define a dictionary of callbacks awaiting to be triggered
CALLBACKS = dict()
# let us have an infinite loop, which reads new messages from a websocket connection
async def websocket_connection(ws, path):
async for msg in ws:
# we received a message
msg = json.loads(msg)
msg_id = msg['id']
msg_data = msg['data']
# we check, if there is a callback waiting to be triggered with such id
if msg_id in CALLBACKS:
# if such id exists, we get the callback function
callback = CALLBACKS[msg_id]
# we delete the callback from the dictionary
del CALLBACKS[msg_id]
# we call it passing the data from the message in there
callback(msg_data)
else:
# we don't have such id, throw some error
raise Exception("bad id " + msg_id)
# this is the function which submits a message
async def submit(ws, data):
# it generates a random request id
from uuid import uuid4
request_id = str(uuid4())
# registers a callback, which will simply return the received data
CALLBACKS[request_id] = lambda data: # <- this is the part I don't know what it should really do to be able to return the value from the await
# after it encodes the message as json
msg = json.dumps({'id': request_id, 'data': data})
# and sends via the websocket
await ws.send(msg)
# and let us have the actual function, which will be the actual script
async def websocket_script(ws):
# this is what I would like to be able to do in the end.
# pass in commands to the websocket and await their completion
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))
websocket_connection and websocket_script would need to be going side by side for it to work. I suppose gather or some other async function would work. I would really like to get rid of callbacks, since this is the initial purpose of using asyncio in the first place.
How could this be done? It seems like a job for asyncio.

I think what you search is asyncio.Future if I understand you correctly.
FUTURES = dict()
async def websocket_connection(ws, path):
# ...
fut = FUTURES[msg_id]
fut.set_result(msg_data) # will "unblock" await fut below
del FUTURES[msg_id]
# ...
async def submit(ws, data):
from uuid import uuid4
request_id = str(uuid4())
fut = asyncio.Future()
FUTURES[request_id] = fut
msg = json.dumps({'id': request_id, 'data': data})
await ws.send(msg)
# ...
return (await fut)
async def websocket_script(ws):
sum = int(await submit(ws, {"eval": "2+2"}))
division = float(await submit(ws, {"eval": "3/2"}))

understanding async/await with an example

I want to understand why async await didnt work while looping around a range, below is the code. Is it that the function I called is not asynchronous. Not able to understand it. Problem is same thing worked when I used non async process instead of for loop. Is there something i'm missing. Thanks
import asyncio
import time
async def test():
response = {}
s = time.time()
tasks=[]
answers=[]
for k in range(4):
print(k)
tasks.append(asyncio.create_task(take_time()))
answers = await asyncio.gather(*tasks)
response['ans'] = answers
response["Time Taken"] = round((time.time() - s),2)
print(response)
return response
async def take_time():
# # time.sleep(4)
# await asyncio.sleep(4)
for i in range(100000000):
o=i
return str(o)
if __name__=='__main__':
asyncio.run(test())

Cannot trigger an async function from another threaded function in Python

I am making a discord bot that will grab a json using requests from time to time, and then send the relevant information to a specific channel.
I have the following classes:
Helper, which is the discord bot itself, that runs async from the start, inside an asyncio.gather;
tasker that controls the interval which calls the class that will do the requests. It runs in a different thread so it doesn't stop the async Helper while it waits
getInfo that does the requests, store the info and should talk with Helper
I am having 2 problems right now:
While the tasker is on a different thread, every time I try to talk with Helper via getInfo it gives me the errors RuntimeError: no running event loop and RuntimeWarning: coroutine 'getInfo.discordmsg' was never awaited
If I dont run it on a different thread, however, it does work on the TestStatus: 1 but it makes Helper get stuck and stop running with TestStatus: 2
Anyway, here is the code
import requests
import asyncio
import discord
from discord.ext import commands, tasks
from datetime import datetime, timedelta
import threading
class Helper(discord.Client):
async def on_ready(self):
global discordbot, taskervar
servername = 'ServerName'
discordbot = self
self.servidores = dict()
self.canais = dict()
for i in range(len(self.guilds)):
self.servidores[self.guilds[i].name] = {}
self.servidores[self.guilds[i].name]['guild']=self.guilds[i]
servidor = self.guilds[i]
for k in range(len(servidor.channels)):
canal = servidor.channels[k]
self.canais[str(canal.name)] = canal
if 'bottalk' not in self.canais.keys():
newchan = await self.servidores[self.guilds[i].name]['guild'].create_text_channel('bottalk')
self.canais[str(newchan.name)] = newchan
self.servidores[self.guilds[i].name]['canais'] = self.canais
self.bottalk = self.get_channel(self.servidores[servername]['canais']['bottalk'].id)
await self.msg("Bot online: " + converteHora(datetime.now(),True))
print(f'{self.user} has connected to Discord!')
taskervar.startprocess()
async def msg(self, msg):
await self.bottalk.send(msg)
async def on_message(self, message):
if message.author == self.user:
return
else:
print(message)
class tasker:
def __init__(self):
global discordbot, taskervar
print('Tasker start')
taskervar = self
self.waiter = threading.Event()
self.lastupdate = datetime.now()
self.nextupdate = datetime.now()
self.thread = threading.Thread(target=self.requests)
def startprocess(self):
if not self.thread.is_alive():
self.waiter = threading.Event()
self.interval = 60*5
self.thread = threading.Thread(target=self.requests)
self.thread.start()
def requests(self):
while not self.waiter.is_set():
getInfo()
self.lastupdate = datetime.now()
self.nextupdate = datetime.now()+timedelta(seconds=self.interval)
self.waiter.wait(self.interval)
def stopprocess(self):
self.waiter.set()
class getInfo:
def __init__(self):
global discordbot, taskervar
self.requests()
async def discordmsg(self,msg):
await discordbot.msg(msg)
def requests(self):
jsondata = {"TestStatus": 1}
if jsondata['TestStatus'] == 1:
print('here')
asyncio.create_task(self.discordmsg("SOMETHING WENT WRONG"))
taskervar.stopprocess()
return
elif jsondata['TestStatus'] == 2:
print('test')
hora = converteHora(datetime.now(),True)
asyncio.create_task(self.discordmsg(str("Everything is fine but not now: " + hora )))
print('test2')
def converteHora(dateUTC, current=False):
if current:
response = (dateUTC.strftime("%d/%m/%Y, %H:%M:%S"))
else:
response = (dateutil.parser.isoparse(dateUTC)-timedelta(hours=3)).strftime("%d/%m/%Y, %H:%M:%S")
return response
async def main():
TOKEN = 'TOKEN GOES HERE'
tasker()
await asyncio.gather(
await Helper().start(TOKEN)
)
if __name__ == '__main__':
asyncio.run(main())

Your primary problem is you don't give your secondary thread access to the asyncio event loop. You can't just await and/or create_task a coroutine on a global object (One of many reasons to avoid using global objects in the first place). Here is how you could modify your code to accomplish that:
class tasker:
def __init__(self):
# ...
self.loop = asyncio.get_running_loop()
# ...
class getInfo:
#...
def requests(self):
# replace the create_tasks calls with this.
asyncio.run_coroutine_threadsafe(self.discordmsg, taskervar.loop)
This uses your global variables because I don't want to rewrite your entire program, but I still strongly recommend avoiding them and considering a re-write yourself.
All that being said, I suspect you will still have this bug:
If I dont run it on a different thread, however, it does work on the TestStatus: 1 but it makes Helper get stuck and stop running with TestStatus: 2
I can't tell what would cause this issue and I'm running into trouble reproducing this on my machine. Your code is pretty hard to read and is missing some details for reproducibility. I would imagine that is part of the reason why you didn't get an answer in the first place. I'm sure you're aware of this article but might be worth a re-visit for better practices in sharing code. https://stackoverflow.com/help/minimal-reproducible-example

How to use an asyncio loop inside another asyncio loop

I have been trying all kinds of things to be able to use an asyncio loop inside another asyncio loop. Most of the time my test just end in errors, such as:
RuntimeError: This event loop is already running
My example code below is just the base test I started with, so you can see the basics of what I am trying to do. I tried so many things after this test, it was just too confusing, so I figured I should keep it simple when asking for help. If anyone can point me in the right direction, that would be great. Thank you for your time!
import asyncio
async def fetch(data):
message = 'Hey {}!'.format(data)
other_data = ['image_a.com', 'image_b.com', 'image_c.com']
images = sub_run(other_data)
return {'message' : message, 'images' : images}
async def bound(sem, data):
async with sem:
r = await fetch(data)
return r
async def build(dataset):
tasks = []
sem = asyncio.Semaphore(400)
for data in dataset:
task = asyncio.ensure_future(bound(sem, data))
tasks.append(task)
r = await asyncio.gather(*tasks)
return r
def run(dataset):
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(build(dataset))
responses = loop.run_until_complete(future)
loop.close()
return responses
async def sub_fetch(data):
image = 'https://{}'.format(data)
return image
async def sub_bound(sem, data):
async with sem:
r = await sub_fetch(data)
return r
async def sub_build(dataset):
tasks = []
sem = asyncio.Semaphore(400)
for data in dataset:
task = asyncio.ensure_future(sub_bound(sem, data))
tasks.append(task)
r = await asyncio.gather(*tasks)
return r
def sub_run(dataset):
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(sub_build(dataset))
responses = loop.run_until_complete(future)
loop.close()
return responses
if __name__ == '__main__':
dataset = ['Joe', 'Bob', 'Zoe', 'Howard']
responses = run(dataset)
print (responses)

Running loop.run_until_compete inside a running event loop would block the outer loop, thus defeating the purpose of using asyncio. Because of that, asyncio event loops aren't recursive, and one shouldn't need to run them recursively. Instead of creating an inner event loop, await a task on the existing one.
In your case, remove sub_run and simply replace its usage:
images = sub_run(other_data)
with:
images = await sub_build(other_data)
And it will work just fine, running the sub-coroutines and not continuing with the outer coroutine until the inner one is complete, as you likely intended from the sync code.