Develop Reference

async_generator' object is not iterable

I need to return a value in async function.
I tried to use synchronous form of return:
import asyncio
async def main():
for i in range(10):
return i
await asyncio.sleep(1)
print(asyncio.run(main()))
output:
0 [Finished in 204ms]
But it just return value of the first loop, which is not expexted. So changed the code as below:
import asyncio
async def main():
for i in range(10):
yield i
await asyncio.sleep(1)
for _ in main():
print(_)
output:
TypeError: 'async_generator' object is not iterable
by using async generator I am facing with this error. How can I return a value for every loop of async function?
Thanks

You need to use an async for which itself needs to be inside an async function:
async def get_result():
async for i in main():
print(i)
asyncio.run(get_result())

Beginner async/await question for api requests

I want speed up some API requests... for that I try to figure out how to do and copy some code which run but when I try my own code its no longer asynchrone. Maybe someone find the fail?
Copy Code (guess from stackoverflow):
#!/usr/bin/env python3
import asyncio
#asyncio.coroutine
def func_normal():
print('A')
yield from asyncio.sleep(5)
print('B')
return 'saad'
#asyncio.coroutine
def func_infinite():
for i in range(10):
print("--%d" % i)
return 'saad2'
loop = asyncio.get_event_loop()
tasks = func_normal(), func_infinite()
a, b = loop.run_until_complete(asyncio.gather(*tasks))
print("func_normal()={a}, func_infinite()={b}".format(**vars()))
loop.close()
My "own" code (I need at the end a list returned and merge the results of all functions):
import asyncio
import time
#asyncio.coroutine
def say_after(start,count,say,yep=True):
retl = []
if yep:
time.sleep(5)
for x in range(start,count):
retl.append(x)
print(say)
return retl
def main():
print(f"started at {time.strftime('%X')}")
loop = asyncio.get_event_loop()
tasks = say_after(10,20,"a"), say_after(20,30,"b",False)
a, b = loop.run_until_complete(asyncio.gather(*tasks))
print("func_normal()={a}, func_infinite()={b}".format(**vars()))
loop.close()
c = a + b
#print(c)
print(f"finished at {time.strftime('%X')}")
main()
Or I m completly wrong and should solve that with multithreading? What would be the best way for API requests that returns a list that I need to merge?

Added comment for each section that needs improvement. Removed some to simply code.
In fact, I didn't find any performance uplift with using range() wrapped in coroutine and using async def, might worth with heavier operations.
import asyncio
import time
# #asyncio.coroutine IS DEPRECATED since python 3.8
#asyncio.coroutine
def say_after(wait=True):
result = []
if wait:
print("I'm sleeping!")
time.sleep(5)
print("'morning!")
# This BLOCKs thread, but release GIL so other thread can run.
# But asyncio runs in ONE thread, so this still harms simultaneity.
# normal for is BLOCKING operation.
for i in range(5):
result.append(i)
print(i, end='')
print()
return result
def main():
start = time.time()
# Loop argument will be DEPRECATED from python 3.10
# Make main() as coroutine, then use asyncio.run(main()).
# It will be in asyncio Event loop, without explicitly passing Loop.
loop = asyncio.get_event_loop()
tasks = say_after(), say_after(False)
# As we will use asyncio.run(main()) from now on, this should be await-ed.
a, b = loop.run_until_complete(asyncio.gather(*tasks))
print(f"Took {time.time() - start:5f}")
loop.close()
main()
Better way:
import asyncio
import time
async def say_after(wait=True):
result = []
if wait:
print("I'm sleeping!")
await asyncio.sleep(2) # 'await' a coroutine version of it instead.
print("'morning!")
# wrap iterator in generator - or coroutine
async def asynchronous_range(end):
for _i in range(end):
yield _i
# use it with async for
async for i in asynchronous_range(5):
result.append(i)
print(i, end='')
print()
return result
async def main():
start = time.time()
tasks = say_after(), say_after(False)
a, b = await asyncio.gather(*tasks)
print(f"Took {time.time() - start:5f}")
asyncio.run(main())
Result
Your code:
DeprecationWarning: "#coroutine" decorator is deprecated since Python 3.8, use "async def" instead
def say_after(wait=True):
I'm sleeping!
'morning!
01234
01234
Took 5.003802
Better async code:
I'm sleeping!
01234
'morning!
01234
Took 2.013863
Note that fixed code now finish it's job while other task is sleeping.

Async iterator ticking at a regular interval

I am implementing an asynchronous iterator to be used with async for which should return a new value at a (mostly) regular interval.
We can illustrate such iterator with a simple clock that will increment a counter every ~n seconds:
import asyncio
class Clock(object):
def __init__(self, interval=1):
self.counter = 0
self.interval = interval
self.tick = asyncio.Event()
asyncio.ensure_future(self.tick_tock())
async def tick_tock(self):
while True:
self.tick.clear()
await asyncio.sleep(self.interval)
self.counter = self.__next__()
self.tick.set()
def __next__(self):
self.counter += 1
return self.counter
def __aiter__(self):
return self
async def __anext__(self):
await self.tick.wait()
return self.counter
Is there a better or cleaner approach than using asyncio.Event? More than one coroutine will async for on this iterator.

In my opinion, your approach is fine. Note that since python 3.6, you can also use asynchronous generators:
async def clock(start=0, step=1, interval=1.):
for i in count(start, step):
yield i
await asyncio.sleep(interval)
However, you won't be able to share them between multiple coroutines. You would have to run the clock in a task and make the data available through an asynchronous iteration interface, which is essentially what you did in your code. Here's a possible implementation.

If you're using Python 3.6+ you can use asynchronous generators which are more readable.
async def Clock(interval=1):
counter = 0
while True:
await asyncio.sleep(interval)
counter += 1
yield counter
async def main():
async for i in Clock(1):
print(i)
if i == 4:
break
if __name__ == '__main__':
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()

Builtin way to transform asynchronous iterable to synchronous iterable list

Python3.6 now asynchronous iterables. Is there builtin way to transform a asynchronous iterable to a synchronous iterable.
I currently have this helper function, but it feels very un-pythonic. Is there a better way to do this?
async def aiter_to_list(aiter):
l = []
async for i in aiter:
l.append(i)
return l

From Python 3.6 you can use Asynchronous Comprehensions
async def async_iter():
for i in range(0,5):
yield i
# async comprehension
sync_list = [gen async for gen in async_iter()]
print(sync_list) # [0, 1, 2, 3, 4]

You can use aiostream.stream.list:
from aiostream import stream
async def agen():
yield 1
yield 2
yield 3
async def main():
lst = await stream.list(agen())
print(lst) # prints [1, 2, 3]
More operators and examples in the documentation.

Your "asynchronous to synchronous" helper is itself asynchronous; not a big change at all. In general: no, you cannot make something asynchronous synchronous. An asynchronous value will be supplied "sometime later"; you cannot make that into "now" because the value doesn't exist "now" and you will have to wait for it, asynchronously.

These functions allow you to convert from / to iterable <==> async iterable, not just simple lists.
Basic imports
import asyncio
import threading
import time
DONE = object()
TIMEOUT = 0.001
The function to_sync_iterable will convert any async iterable to a sync iterable:
def to_sync_iterable(async_iterable, maxsize = 0):
def sync_iterable():
queue = asyncio.Queue(maxsize=maxsize)
loop = asyncio.get_event_loop()
t = threading.Thread(target=_run_coroutine, args=(loop, async_iterable, queue))
t.daemon = True
t.start()
while True:
if not queue.empty():
x = queue.get_nowait()
if x is DONE:
break
else:
yield x
else:
time.sleep(utils.TIMEOUT)
t.join()
return sync_iterable()
def _run_coroutine(loop, async_iterable, queue):
loop.run_until_complete(_consume_async_iterable(async_iterable, queue))
async def _consume_async_iterable(async_iterable, queue):
async for x in async_iterable:
await queue.put(x)
await queue.put(DONE)
You can use it like this:
async def slow_async_generator():
yield 0
await asyncio.sleep(1)
yield 1
await asyncio.sleep(1)
yield 2
await asyncio.sleep(1)
yield 3
for x in to_sync_iterable(slow_async_generator()):
print(x)
The function to_async_iterable will convert any sync iterable to an async iterable:
def to_async_iterable(iterable, maxsize = 0):
async def async_iterable():
queue = asyncio.Queue(maxsize=maxsize)
loop = asyncio.get_event_loop()
task = loop.run_in_executor(None, lambda: _consume_iterable(loop, iterable, queue))
while True:
x = await queue.get()
if x is DONE:
break
else:
yield x
await task
return async_iterable()
def _consume_iterable(loop, iterable, queue):
for x in iterable:
while True:
if not queue.full():
loop.call_soon_threadsafe(queue.put_nowait, x)
break
else:
time.sleep(TIMEOUT)
while True:
if not queue.full():
loop.call_soon_threadsafe(queue.put_nowait, DONE)
break
else:
time.sleep(TIMEOUT)
This one is specially useful for asyncio programs because it won't block the event loop even if the the sync iterable blocks. You can use it like this:
def slow_sync_generator():
yield 0
time.sleep(1)
yield 1
time.sleep(1)
yield 2
time.sleep(1)
yield 3
async def async_task():
async for x in to_async_iterable(slow_sync_generator()):
print(x)
asyncio.get_event_loop().run_until_complete(async_task())

Lazy iterators (generators) with asyncio

I have a blocking, non-async code like this:
def f():
def inner():
while True:
yield read()
return inner()
With this code the caller can choose when to stop the function to generate data. How to change this to async? This solution doesn't work:
async def f():
async def inner():
while True:
yield await coroutine_read()
return inner()
... because yield can't be used in async def functions. If i remove the async from the inner() signature, I can't use await anymore.

Upd:
Starting with Python 3.6 we have asynchronous generators and able to use yield directly inside coroutines.
As noted above, you can't use yield inside async funcs. If you want to create coroutine-generator you have to do it manually, using __aiter__ and __anext__ magic methods:
import asyncio
# `coroutine_read()` generates some data:
i = 0
async def coroutine_read():
global i
i += 1
await asyncio.sleep(i)
return i
# `f()` is asynchronous iterator.
# Since we don't raise `StopAsyncIteration`
# it works "like" `while True`, until we manually break.
class f:
async def __aiter__(self):
return self
async def __anext__(self):
return await coroutine_read()
# Use f() as asynchronous iterator with `async for`:
async def main():
async for i in f():
print(i)
if i >= 3:
break
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Output:
1
2
3
[Finished in 6.2s]
You may also like to see other post, where StopAsyncIteration uses.