I have been trying all kinds of things to be able to use an asyncio loop inside another asyncio loop. Most of the time my test just end in errors, such as:
RuntimeError: This event loop is already running
My example code below is just the base test I started with, so you can see the basics of what I am trying to do. I tried so many things after this test, it was just too confusing, so I figured I should keep it simple when asking for help. If anyone can point me in the right direction, that would be great. Thank you for your time!
import asyncio
async def fetch(data):
message = 'Hey {}!'.format(data)
other_data = ['image_a.com', 'image_b.com', 'image_c.com']
images = sub_run(other_data)
return {'message' : message, 'images' : images}
async def bound(sem, data):
async with sem:
r = await fetch(data)
return r
async def build(dataset):
tasks = []
sem = asyncio.Semaphore(400)
for data in dataset:
task = asyncio.ensure_future(bound(sem, data))
tasks.append(task)
r = await asyncio.gather(*tasks)
return r
def run(dataset):
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(build(dataset))
responses = loop.run_until_complete(future)
loop.close()
return responses
async def sub_fetch(data):
image = 'https://{}'.format(data)
return image
async def sub_bound(sem, data):
async with sem:
r = await sub_fetch(data)
return r
async def sub_build(dataset):
tasks = []
sem = asyncio.Semaphore(400)
for data in dataset:
task = asyncio.ensure_future(sub_bound(sem, data))
tasks.append(task)
r = await asyncio.gather(*tasks)
return r
def sub_run(dataset):
loop = asyncio.get_event_loop()
future = asyncio.ensure_future(sub_build(dataset))
responses = loop.run_until_complete(future)
loop.close()
return responses
if __name__ == '__main__':
dataset = ['Joe', 'Bob', 'Zoe', 'Howard']
responses = run(dataset)
print (responses)
Running loop.run_until_compete inside a running event loop would block the outer loop, thus defeating the purpose of using asyncio. Because of that, asyncio event loops aren't recursive, and one shouldn't need to run them recursively. Instead of creating an inner event loop, await a task on the existing one.
In your case, remove sub_run and simply replace its usage:
images = sub_run(other_data)
with:
images = await sub_build(other_data)
And it will work just fine, running the sub-coroutines and not continuing with the outer coroutine until the inner one is complete, as you likely intended from the sync code.
Related
I have two functions that work, let's independently. One generates data inside a loop and and saves them on a db. The other is pinging the db every few seconds and if new data have been saved it plots them. In pseudocode it looks like:
def dashborard():
ping_db(time_interval)
if new_values:
plot(new_values)
def generate_data(d):
## do something with d
## and return some values
return values
for i in range(10):
generate_data(i)
I would like to run the dashboard and the data-generating loop asynchronously but I havent managed to do it unless I put an await asyncio.sleep() call inside the function the runs the loop. Again, in pseudocode this is what I have done:
import asyncio
def run_async(self):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main())
return result
async def main():
asyncio.create_task(dashboard())
out = await self.run_loop()
return out
async def run_loop(self):
for i in range(10):
values = await generate_data(i)
await asyncio.sleep(some_seconds) ## <---- How can I remove this?
return values
The await asycion,sleep() argument is at least the number of seconds each loop takes. In this case, the dashboard opens and works fine. If I remove it however, then the dashboard doesnt open until we reach the very end of the loop.
How can I have the loop and the dashboard run asynchronously please? Any help is greatly appreciated.
The (self) pieces in your code suggest use of a class??? Can't figure this. However assuming these are in the same event loop (otherwise why use asyncio???) the following illustrates an approach using asyncio Queues.
import asyncio
async def dashboard(queue):
while True:
new_value = await queue.get()
if new_value == '__END__':
return
print(new_value)
async def generate_data(i):
"""
Presumably your database access awaits the database update.
Unless you await something here the co-routines cannot co-operate.
----------- ----------
"""
await asyncio.sleep(1.0)
return f"something from {i}"
async def run_loop(queue: asyncio.Queue):
values = []
for i in range(5):
value = await generate_data(i)
values.append(value)
queue.put_nowait(value)
# await asyncio.sleep(some_seconds) # <---- How can I remove this?
queue.put_nowait('__END__')
return values
async def main():
queue = asyncio.Queue()
last = asyncio.create_task(dashboard(queue))
out = await run_loop(queue)
print(out)
await asyncio.gather(last)
if __name__ == "__main__":
asyncio.run(main())
The output is:
something from 0
something from 1
something from 2
something from 3
['something from 0', 'something from 1', 'something from 2', 'something from 3', 'something from 4']
something from 4
I can't figure this out at all, I looked all the questions, videos, and documents. So I keep getting back was never awaited for async def renk_calc() -> bool: So I'm trying to get this function to preform its calculation and to return. I can't get it to work, no matter what method I tried, I ran into a different problem.
From reading trying to understand I have to run this function on separate thread or a separate processor, but I kept getting errors back its not coroutine is not callable, and coroutine is none, and coroutine cannot be a pickle. It's probably because I'm very new to asyncio, and I was reading the documentation to implement a solution.
but in essence this function gets input from async def renko_append(): as this appends the dictionary then that feeds the DataFrame, and I know the DataFrame is causing the issue because it's not a awaitable object or curatable object. But I don't know how to fix it.
then the dictionary gets cleared. So it loops back and refills it again and that's the marry go round. The idea is then to grab the returned value in the function and utilize it in the next block.
nest_asyncio.apply()
list = []
renk = {"DATE":[],"OPEN":[],"HIGH":[],"LOW":[],"CLOSE":[]}
async def renko_append():
current_time = datetime.now()
global list
# print(f"this is the {list}")
renk["DATE"].append(current_time.strftime("%Y-%m-%d %H:%M:%S"))
renk["OPEN"].append(list[0])
renk["HIGH"].append(max(list))
renk["LOW"].append(min(list))
renk["CLOSE"].append(list[-1])
print(renk)
list.clear()
async def renk_calc() -> bool:
df = pd.DataFrame.from_dict(renk)
df.columns = [i.lower() for i in df.columns]
renko_ind = indicators.Renko(df)
renko_ind.brick_size = 0.0006
renko_ind.chart_type = indicators.Renko.PERIOD_CLOSE
data = renko_ind.get_ohlc_data()
result = data["uptrend"]
return result
async def clr_dict():
renk["DATE"] = []
renk["OPEN"] = []
renk["HIGH"] = []
renk["LOW"] = []
renk["CLOSE"] =[]
print(renk)
async def main():
async def deal_msg(msg):
if msg['topic'] == '/contractMarket/ticker:ADAUSDTM':
ns = msg["data"]["ts"]
time = datetime.fromtimestamp(ns // 1000000000)
my_time = time.strftime("%H:%M:%S")
price = msg["data"]["price"]
# print(f'Get ADAUSDTM Ticker:price: {price} side: {msg["data"]["side"]} time: {my_time}')
list.append(price)
# client = WsToken()
client = WsToken(key='', secret='', passphrase='', is_sandbox=False, url='')
ws_client = await KucoinFuturesWsClient.create(loop, client, deal_msg, private=False)
await ws_client.subscribe('/contractMarket/ticker:ADAUSDTM')
while True:
await asyncio.sleep(10, loop=loop)
await asyncio.create_task(renko_append())
await asyncio.create_task(renk_calc())
await asyncio.create_task(clr_dict())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
I edited to add the error: I figured out the "await" problem, but now I'm stuck on this error.
TypeError: 'coroutine' object is not callable
async def renk_calc() this is whats not callable
I want to understand why async await didnt work while looping around a range, below is the code. Is it that the function I called is not asynchronous. Not able to understand it. Problem is same thing worked when I used non async process instead of for loop. Is there something i'm missing. Thanks
import asyncio
import time
async def test():
response = {}
s = time.time()
tasks=[]
answers=[]
for k in range(4):
print(k)
tasks.append(asyncio.create_task(take_time()))
answers = await asyncio.gather(*tasks)
response['ans'] = answers
response["Time Taken"] = round((time.time() - s),2)
print(response)
return response
async def take_time():
# # time.sleep(4)
# await asyncio.sleep(4)
for i in range(100000000):
o=i
return str(o)
if __name__=='__main__':
asyncio.run(test())
I am trying to use asyncio with a method I have which takes a while to run. I want to call the same method many times with slightly different inputs and I don't care which call completes first, I just want to gather up all the results when they are all finished (prime candidate for asyncio loop right?) Here is some code I have been using when I know how many times I want to call my method:
async def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, myhttpreq, 'US')
future2 = loop.run_in_executor(None, myhttpreq, 'RU')
future3 = loop.run_in_executor(None, myhttpreq, 'CN')
response1 = await future1
response2 = await future2
response3 = await future3
print(response1)
print(response2)
print(response3)
As you can tell by the params being passed with myhttpreq the reason this method takes so long is because it calls endpints from all over the world. My question is this:
Say the list of countries I want to call is pretty variable, how can I generate, and await on a variable amount of futures?
I ended up doing this:
async def main2(countries):
loop = asyncio.get_event_loop()
futures = []
for country in countries:
futures.append(loop.run_in_executor(None, myhttpreq, country))
for future in futures:
print(await future)
Does this seem dumb though?
I have 2 functions: The first one, def_a, is an asynchronous function and the second one is def_b which is a regular function and called with the result of def_a as a callback with the add_done_callback function.
My code looks like this:
import asyncio
def def_b(result):
next_number = result.result()
# some work on the next_number
print(next_number + 1)
async def def_a(number):
await some_async_work(number)
return number + 1
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(def_a(1))
task.add_done_callback(def_b)
response = loop.run_until_complete(task)
loop.close()
And it's work perfectly.
The problem began when also the second function, def_b, became asynchronous. Now it looks like this:
async def def_b(result):
next_number = result.result()
# some asynchronous work on the next_number
print(next_number + 1)
But now I can not provide it to the add_done_callback function, because it's not a regular function.
My question is- Is it possible and how can I provide def_b to the add_done_callback function if def_b is asynchronous?
add_done_callback is considered a "low level" interface. When working with coroutines, you can chain them in many ways, for example:
import asyncio
async def my_callback(result):
print("my_callback got:", result)
return "My return value is ignored"
async def coro(number):
await asyncio.sleep(number)
return number + 1
async def add_success_callback(fut, callback):
result = await fut
await callback(result)
return result
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(coro(1))
task = add_success_callback(task, my_callback)
response = loop.run_until_complete(task)
print("response:", response)
loop.close()
Keep in mind add_done_callback will still call the callback if your future raises an exception (but calling result.result() will raise it).
This only works for one future job, if you have multiple async jobs, they will blocks each other, a better way is using asyncio.as_completed() to iterate future list:
import asyncio
async def __after_done_callback(future_result):
# await for something...
pass
async def __future_job(number):
await some_async_work(number)
return number + 1
loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(__future_job(x)) for x in range(100)] # create 100 future jobs
for f in asyncio.as_completed(tasks):
result = await f
await __after_done_callback(result)
loop.close()
You can try the aiodag library. It's a very lightweight wrapper around asyncio that abstracts away some of the async plumbing that you usually have to think about. From this example you won't be able to tell that things are running asynchronously since it's just 1 task that depends on another, but it is all running async.
import asyncio
from aiodag import task
#task
async def def_b(result):
# some asynchronous work on the next_number
print(result + 1)
#task
async def def_a(number):
await asyncio.sleep(number)
return number + 1
async def main():
a = def_a(1)
b = def_b(a) # this makes task b depend on task a
return await b
loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
response = loop.run_until_complete(main())