Goal: run main() consisting of a bunch of asyncio functions between start_time and end_time
import datetime as dt
start_time, end_time= dt.time(9, 29), dt.time(16, 20)
current_time() just keeps adding the current time to work space. This time is used at several different points in the script not shown here
async def current_time():
while True:
globals()['now'] = dt.datetime.now().replace(microsecond=0)
await asyncio.sleep(.001)
Another function that does something:
async def balance_check():
while True:
#do something here
await asyncio.sleep(.001)
main() awaits the previous coroutines:
async def main():
while True:
#Issue:This is my issue. I am trying to make these
#coroutines only run between start_time and end_time.
#Outside this interval, I want the loop to
#shut down and for the script to stop.
if start_time < now.time() < end_time :
await asyncio.wait([
current_time(),
balance_check(),
])
else:
print('loop stopped since {} is outside {} and {}'.format(now, start_time, end_time))
loop.stop()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(main())
finally:
loop.close()
Issue: this keeps working even after end_time
The problem is incorrect use of await asyncio.wait([current_time(), balance_check(),]).
Awaiting asyncio.wait() waits for the specified awaitables to complete, i.e. either return a value or raise an exception. Since neither current_time nor balance_check ever return out of their infinite loops, the execution of main() never gets passed await asyncio.wait(...), as this expression waits for both to finish. In turn, the while loop in main() never gets to its second iteration, and loop.stop() has no chance to run.
If the code's intention was to use asyncio.wait() to give coroutines a chance to run, asyncio.wait is not the tool for that. Instead, one can just start the two tasks by calling asyncio.create_task(), and then do nothing. As long as the event loop can run (i.e. it's not blocked with a call to time.sleep() or similar), asyncio will automatically switch between the coroutines, in this case current_time and balanced_check at their ~1-millisecond pace. Of course, you will want to regain control by the end_time deadline, so "doing nothing" is best expressed as a single call to asyncio.sleep():
async def main():
t1 = asyncio.create_task(current_time())
t2 = asyncio.create_task(balance_check())
end = dt.datetime.combine(dt.date.today(), end_time)
now = dt.datetime.now()
await asyncio.sleep((end - now).total_seconds())
print('loop stopped since {} is outside {} and {}'.format(now, start_time, end_time))
t1.cancel()
t2.cancel()
Note that an explicit loop.stop() is not even necessary, because run_until_complete() will automatically stop the loop once the given coroutine completes. Calling cancel() on the tasks has no practical effect because the loop stops pretty much immediately; it is included to make those tasks complete, so that the garbage collector doesn't warn about destroying tasks that are pending.
As noted in the comments, this code doesn't wait for start_time, but that functionality is easily accomplished by adding another sleep before spawning the tasks.
You could use synchronizing primitives' to do this, specifically events.
import datetime as dt
import asyncio
async def function(start_event, end_event):
# Waits until we get the start event command.
await start_event.wait()
# Run our function until we get the end_event being set to true.
index = 0
while not end_event.is_set():
print(f"Function running; index {index}.")
await asyncio.sleep(2)
index += 1
async def controller(start_event, end_event):
# Will usually be this. Commented out for testing purposes. Can be passed by main.
# Subtraction/addition does not work on dt.datetime(). Inequality does (> or <)
# now, start, end = dt.datetime.now().time(), dt.time(17, 24), dt.time(18, 0, 30)
now = dt.datetime.now()
start, end = now + dt.timedelta(seconds=-10), now + dt.timedelta(seconds=10)
# Starts our function.
print(f"Starting at {start}. Ending in {end - now}. ")
start_event.set()
# Keeps checking until we pass the scheduled time.
while start < now < end:
print(f"Ending in {end - now}.")
await asyncio.sleep(2)
now = dt.datetime.now()
# Ends our function.
end_event.set()
print(f"Function ended at {now}.")
async def main():
# Creates two indepedent events.
# start_event is flagged when controller is ready for function to be ran (prevents misfires).
# end_event is flagged when controller see's start < time.now < end (line 30).
start_event = asyncio.Event()
end_event = asyncio.Event()
# Starts both functions at the same time.
await asyncio.gather(
controller(start_event, end_event), function(start_event, end_event)
)
if __name__ == "__main__":
# Starting our process.
asyncio.run(main())
This method will require both function and controller to take up space in the asyncio loop. However, function will be locked for most of the time, and it will instead be the controller hogging loop resources with its while loop, so keep that in the back of your mind.
Related
From the last post, the duplicate post cannot answer my question.
Right now, I have a function f1() which contains CPU intensive part and async IO intensive part. Therefore f1() itself is an async function. How can I run the whole f1() with given timeout? I found the method provided in the post cannot solve my situation. For the following part, it shows RuntimeWarning: coroutine 'f1' was never awaited handle = None # Needed to break cycles when an exception occurs.
import asyncio
import time
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(1)
async def f1():
print("start sleep")
time.sleep(3) # simulate CPU intensive part
print("end sleep")
print("start asyncio.sleep")
await asyncio.sleep(3) # simulate IO intensive part
print("end asyncio.sleep")
async def process():
print("enter process")
loop = asyncio.get_running_loop()
await loop.run_in_executor(executor, f1)
async def main():
print("-----f1-----")
t1 = time.time()
try:
await asyncio.wait_for(process(), timeout=2)
except:
pass
t2 = time.time()
print(f"f1 cost {(t2 - t1)} s")
if __name__ == '__main__':
asyncio.run(main())
From previous post, loop.run_in_executor can only work for normal function not async function.
one way to do it is to make process not an async function, so it can run in another thread, and have it start an asyncio loop in the other thread to run f1.
note that starting another loops means you cannot share coroutines and futures between the two loops.
import asyncio
import time
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(1)
async def f1():
print("start sleep")
time.sleep(3) # simulate CPU intensive part
print("end sleep")
print("start asyncio.sleep")
await asyncio.sleep(3) # simulate IO intensive part
print("end asyncio.sleep")
def process():
print("enter process")
asyncio.run(asyncio.wait_for(f1(),2))
async def main():
print("-----f1-----")
t1 = time.time()
try:
loop = asyncio.get_running_loop()
await loop.run_in_executor(executor, process)
except:
pass
t2 = time.time()
print(f"f1 cost {(t2 - t1)} s")
if __name__ == '__main__':
asyncio.run(main())
-----f1-----
enter process
start sleep
end sleep
start asyncio.sleep
f1 cost 3.0047199726104736 s
keep in mind that you must wait for any IO to return f1 to the eventloop so the future can be cancelled, you cannot cancel the CPU-intensive part of the code unless it does something like await asyncio.sleep(0) which returns to the event-loop momentarily, which is why time.sleep cannot be cancelled.
I have explained the cause of the issue. You should remove or replace the time.sleep at f1 as it blocks the thread, and asyncio.wait_for cannot handle the timeout.
Regarding to the RuntimeWarning
RuntimeWarning: coroutine 'f1' was never awaited handle = None # Needed to break cycles when an exception occurs.
It occurs because the loop.run_in_executor expects a non-async function as a second argument.
I have an async coroutine that I want to terminate using a timer/thread. The coroutine is based of this example from aiortc.
args = parse_args()
client = Client(connection, media, args.role)
# run event loop
loop = asyncio.get_event_loop()
try:
timer = None
if args.timeout:
print("Timer started")
timer = threading.Timer(args.timeout, loop.run_until_complete, args=(client.close(),))
timer.start()
loop.run_until_complete(client.run())
if timer:
timer.join()
except KeyboardInterrupt:
pass
finally:
# cleanup
loop.run_until_complete(client.close())
This does not work and raises RuntimeError('This event loop is already running')
Why does this raise the error? My guess is that it's because the loop is running on a different thread. However, creating a new loop doesn't work as it gets a future attached to a different loop.
def timer_callback():
new_loop = asyncio.new_event_loop()
new_loop.run_until_complete(client.close())
Following that, how can I use a timer to end the script?
Following that, how can I use a timer to end the script?
You can call asyncio.run_coroutine_threadsafe() to submit a coroutine to an event loop running in another thread:
if args.timeout:
print("Timer started")
timer = threading.Timer(
args.timeout,
asyncio.run_coroutine_threadsafe,
args=(client.close(), loop),
)
timer.start()
Note, however, that since you're working with asyncio, you don't need a dedicated thread for the timer, you could just create a coroutine and tell it to wait before doing something:
if args.timeout:
print("Timer started")
async def close_after_timeout():
await asyncio.sleep(args.timeout)
await client.close()
loop.create_task(close_after_timeout())
This isn't the general solution that I was looking for, I added a timeout variable to the client constructor, and within client.run() I added asyncio.sleep(timeout) which would exit the loop. This is enough for my purposes.
I have 3 tasks. -
def task_a():
while True:
file.write()
asyncio.sleep(10)
def task_b():
while True:
file.write()
asyncio.sleep(10)
def task_c():
# do something
main.py -
try:
loop = asyncio.get_event_loop()
A = loop.create_task(task_a)
B = loop.create_task(task_b)
C = loop.create_task(task_c)
awaitable_pending_tasks = asyncio.all_tasks()
execution_group = asyncio.gather(*awaitable_pending_tasks, return_exceptions=True)
fi_execution = loop.run_until_complete(execution_group)
finally:
loop.run_forever()
I want to make sure that the loop is exited when the task_c is completed.
Tried with loop.close() in finally but since it's async, it closes in between.
task_a and task_b write to a file and there is another process running that checks the time the file was modified. If it's greater than a minute it will result in an error(which I don't want) hence I've put the while loop in it and once its written I added a sleep()
Once task_c is complete, I need the loop to stop.
Other answers on StackOverflow looked complicated to understand.
Any way we can do this?
You could call loop.run_until_complete or asyncio.run (but not run_forever) to run a function that prepares the tasks you need and then only awaits the one you want to terminate the loop (untested):
async def main():
asyncio.create_task(task_a)
asyncio.create_task(task_b)
await task_c
tasks = set(asyncio.all_tasks()) - set([asyncio.current_task()])
for t in tasks:
t.cancel()
await asyncio.gather(*tasks, return_exceptions=True)
asyncio.run(main())
# or asyncio.get_event_loop().run_until_complete(main())
I am writing a Python program that run tasks taken from a queue concurrently, to learn asyncio.
Items will be put onto a queue by interacting with a main thread (within REPL).
Whenever a task is put onto the queue, it should be consumed and executed immediately.
My approach is to kick off a separate thread and pass a queue to the event loop within that thread.
The tasks are running but only sequentially and I am not clear on how to run the tasks concurrently. My attempt is as follows:
import asyncio
import time
import queue
import threading
def do_it(task_queue):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
status = f'{clock()} {name}_{total_time}:'
print(status, 'START')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(status, 'processing...')
await asyncio.sleep(1)
current_time = time.time()
print(status, 'DONE.')
async def main():
while True:
item = task_queue.get()
if item == _sentinel:
break
await asyncio.create_task(process(*item))
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
tasks = queue.Queue()
th = threading.Thread(target=do_it, args=(tasks,))
th.start()
tasks.put(('abc', 5))
tasks.put(('def', 3))
Any advice pointing me in the direction of running these tasks concurrently would be greatly appreciated!
Thanks
UPDATE
Thank you Frank Yellin and cynthi8! I have reformed main() according to your advice:
removed await before asyncio.create_task - fixed concurrency
added wait while loop so that main would not return prematurely
used non-blocking mode of Queue.get()
The program now works as expected 👍
UPDATE 2
user4815162342 has offered further improvements, I have annotated his suggestions below.
'''
Starts auxiliary thread which establishes a queue and consumes tasks within a
queue.
Allow enqueueing of tasks from within __main__ and termination of aux thread
'''
import asyncio
import time
import threading
import functools
def do_it(started):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
print(f'{clock()} {name}_{total_time}:', 'Started.')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(f'{clock()} {name}_{total_time}:', 'Processing...')
await asyncio.sleep(1)
current_time = time.time()
print(f'{clock()} {name}_{total_time}:', 'Done.')
async def main():
# get_running_loop() get the running event loop in the current OS thread
# out to __main__ thread
started.loop = asyncio.get_running_loop()
started.queue = task_queue = asyncio.Queue()
started.set()
while True:
item = await task_queue.get()
if item == _sentinel:
# task_done is used to tell join when the work in the queue is
# actually finished. A queue length of zero does not mean work
# is complete.
task_queue.task_done()
break
task = asyncio.create_task(process(*item))
# Add a callback to be run when the Task is done.
# Indicate that a formerly enqueued task is complete. Used by queue
# consumer threads. For each get() used to fetch a task, a
# subsequent call to task_done() tells the queue that the processing
# on the task is complete.
task.add_done_callback(lambda _: task_queue.task_done())
# keep loop going until all the work has completed
# When the count of unfinished tasks drops to zero, join() unblocks.
await task_queue.join()
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
# started Event is used for communication with thread th
started = threading.Event()
th = threading.Thread(target=do_it, args=(started,))
th.start()
# started.wait() blocks until started.set(), ensuring that the tasks and
# loop variables are available from the event loop thread
started.wait()
tasks, loop = started.queue, started.loop
# call_soon schedules the callback callback to be called with args arguments
# at the next iteration of the event loop.
# call_soon_threadsafe is required to schedule callbacks from another thread
# put_nowait enqueues items in non-blocking fashion, == put(block=False)
loop.call_soon_threadsafe(tasks.put_nowait, ('abc', 5))
loop.call_soon_threadsafe(tasks.put_nowait, ('def', 3))
loop.call_soon_threadsafe(tasks.put_nowait, 'STOP')
As others pointed out, the problem with your code is that it uses a blocking queue which halts the event loop while waiting for the next item. The problem with the proposed solution, however, is that it introduces latency because it must occasionally sleep to allow other tasks to run. In addition to introducing latency, it prevents the program from ever going to sleep, even when there are no items in the queue.
An alternative is to switch to asyncio queue which is designed for use with asyncio. This queue must be created inside the running loop, so you can't pass it to do_it, you must retrieve it. Also, since it's an asyncio primitive, its put method must be invoked through call_soon_threadsafe to ensure that the event loop notices it.
One final issue is that your main() function uses another busy loop to wait for all the tasks to complete. This can be avoided by using Queue.join, which is explicitly designed for this use case.
Here is your code adapted to incorporate all of the above suggestions, with the process function remaining unchanged from your original:
import asyncio
import time
import threading
def do_it(started):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
status = f'{clock()} {name}_{total_time}:'
print(status, 'START')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(status, 'processing...')
await asyncio.sleep(1)
current_time = time.time()
print(status, 'DONE.')
async def main():
started.loop = asyncio.get_running_loop()
started.queue = task_queue = asyncio.Queue()
started.set()
while True:
item = await task_queue.get()
if item == _sentinel:
task_queue.task_done()
break
task = asyncio.create_task(process(*item))
task.add_done_callback(lambda _: task_queue.task_done())
await task_queue.join()
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
started = threading.Event()
th = threading.Thread(target=do_it, args=(started,))
th.start()
started.wait()
tasks, loop = started.queue, started.loop
loop.call_soon_threadsafe(tasks.put_nowait, ('abc', 5))
loop.call_soon_threadsafe(tasks.put_nowait, ('def', 3))
loop.call_soon_threadsafe(tasks.put_nowait, 'STOP')
Note: an unrelated issue with your code was that it awaited the result of create_task(), which nullified the usefulness of create_task() because it wasn't allowed to run in the background. (It would be equivalent to immediately joining a thread you've just started - you can do it, but it doesn't make much sense.) This issue is fixed both in the above code and in your edit to the question.
There are two problems with your code.
First, you should not have the await before the asyncio.create_task. This is possibly what is causing your code to run synchronously.
Then, once you've made your code run asynchronously, you need something after the while loop in main so that the code doesn't return immediately, but instead waits for all the jobs to finish. Another stackoverflow answer recommends:
while len(asyncio.Task.all_tasks()) > 1: # Any task besides main() itself?
await asyncio.sleep(0.2)
Alternatively there are versions of Queue that can keep track of running tasks.
As an additional problem:
If a queue.Queue is empty, get() blocks by default and does not return a sentinel string. https://docs.python.org/3/library/queue.html
I am new to python and struggling to understand why my coroutine is not working.
In the current code, the only one job is running and another is always stays idle. Why?
class Worker:
def job1_sync(self):
count = 0
while True:
print('JOB A:', count)
count = count + 1
def job2_sync(self):
count = 0
while True:
print('JOB B:', count)
count = count + 1
async def job1(self):
await self.job1_sync()
async def job2(self):
await self.job2_sync()
worker = Worker()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(asyncio.gather(worker.job1(), worker.job2()))
Asyncio does not do multi-tasking or multithreading. What it does is it schedules tasks within one thread, using a cooperative model.
That is, the event loop runs again when the current task awaits something that will "block", and only then it schedules another task. Under the hood, async functions are coroutines, and calls to await make the corouting yield to the event loop, which resumes it a later point, when awaited condition arises.
Here you never await anything, so job1 never relinquishes control, so the event loop never has a chance to distribute computing power to other tasks.
Now if your job was to actually relinquish control, say by triggering a delay, then your code would work:
async def job1_sync(self): # note the async : only async functions can await
count = 0
while True:
print('JOB A:', count)
count = count + 1
await asyncio.sleep(1) # main even loop gets control
TLDR: asyncio is useful for what it says: doing stuff asynchronously, allowing other tasks to make progress while current task waits for something. Nothing runs in parallel.