I have an asyncio event loop to which I add a signal handler, using loop.add_signal_handler(). The signals I want to catch are SIGINT, SIGHUP and SIGTERM to gracefully shutdown my event loop.
From this event loop I want to fork processes, using multiprocessing.Process(). This process p I want to be able to terminate from the event loop using p.terminate(). However, the signal handler will catch the SIGTERM signal issued by p.terminate(), prompting the execution of my shutdown code, leaving p running.
I have not found any solutions to this. Most posts say you should refrain from using termination signals and seek to use multiprocessing.Queue() by passing e.g. None and handle this in the child-process. While I see the usefulness and cleanness of this approach, in my case using a multiprocessing.Queue() will not be feasible. Am I trying something impossible or have I missed something?
I have created a minimum example:
import asyncio
import multiprocessing as mp
import signal
import time
class Test():
def __init__(self):
self.event_loop = asyncio.get_event_loop()
def create_mp(self):
# Wrapper for creating process
self.p = mp.Process(target=worker)
self.p.start()
async def shutdown_mp(self):
# Shutdown after 5 sec
await asyncio.sleep(5)
self.p.terminate()
async def async_print(self):
# Simple coroutine printing
while True:
await asyncio.sleep(1)
print("Async_test")
async def shutdown_event_loop(self, signal):
# Graceful shutdown, inspiration from roguelynn (Lynn Root)
print("Received exit signal {}".format(signal.name))
tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
# Cancel all tasks
[task.cancel() for task in tasks]
print("Cancelling {} outstanding tasks".format(len(tasks)))
await asyncio.gather(*tasks, return_exceptions=True)
self.event_loop.stop()
def run(self):
# Add signals
signals = (signal.SIGINT, signal.SIGHUP, signal.SIGTERM)
for s in signals:
self.event_loop.add_signal_handler(
s, lambda s=s: self.event_loop.create_task(self.shutdown_event_loop(s)))
# Schedule async task
self.event_loop.create_task(self.async_print())
# Start processes
self.create_mp()
# Schedule process to be terminated
self.event_loop.create_task(self.shutdown_mp())
self.event_loop.run_forever()
def worker():
# Simple process
while True:
print("Test")
time.sleep(1)
if __name__ == "__main__":
test = Test()
test.run()
I have not been able to simply KeyboardInterrupt cancel the processes. Instead I use pgrep python3 and kill -9 PID.
Related
I want to start a new Process (Pricefeed) from my Executor class and then have the Executor class keep running in its own event loop (the shoot method). In my current attempt, the asyncio loop gets blocked on the line p.join(). However, without that line, my code just exits. How do I do this properly?
Note: fh.run() blocks as well.
import asyncio
from multiprocessing import Process, Queue
from cryptofeed import FeedHandler
from cryptofeed.defines import L2_BOOK
from cryptofeed.exchanges.ftx import FTX
class Pricefeed(Process):
def __init__(self, queue: Queue):
super().__init__()
self.coin_symbol = 'SOL-USD'
self.fut_symbol = 'SOL-USD-PERP'
self.queue = queue
async def _book_update(self, feed, symbol, book, timestamp, receipt_timestamp):
self.queue.put(book)
def run(self):
fh = FeedHandler()
fh.add_feed(FTX(symbols=[self.fut_symbol, self.coin_symbol], channels=[L2_BOOK],
callbacks={L2_BOOK: self._book_update}))
fh.run()
class Executor:
def __init__(self):
self.q = Queue()
async def shoot(self):
print('in shoot')
for i in range(5):
msg = self.q.get()
print(msg)
await asyncio.sleep(1) # do some stuff
async def run(self):
asyncio.create_task(self.shoot())
p = Pricefeed(self.q)
p.start()
p.join()
async def main():
g = Executor()
await g.run()
if __name__ == '__main__':
asyncio.run(main())
Since you're using a queue to communicate this is a somewhat tricky problem. To answer your first question as to why removing join makes the program work, join blocks until the process finishes. In asyncio you can't do anything blocking in a function marked async or it will freeze the event loop. To do this properly you'll need to run your process with the asyncio event loop's run_in_executor method which will run things in a process pool and return an awaitable that is compatible with the asyncio event loop.
Secondly, you'll need to use a multiprocessing Manager which creates shared state that can be used by multiple processes to properly share your queue. Managers directly support creation of a shared queue. Using these two bits of knowledge you can adapt your code to something like the following which works:
import asyncio
import functools
import time
from multiprocessing import Manager
from concurrent.futures import ProcessPoolExecutor
def run_pricefeed(queue):
i = 0
while True: #simulate putting an item on the queue every 250ms
queue.put(f'test-{i}')
i += 1
time.sleep(.25)
class Executor:
async def shoot(self, queue):
print('in shoot')
for i in range(5):
while not queue.empty():
msg = queue.get(block=False)
print(msg)
await asyncio.sleep(1) # do some stuff
async def run(self):
with ProcessPoolExecutor() as pool:
with Manager() as manager:
queue = manager.Queue()
asyncio.create_task(self.shoot(queue))
await asyncio.get_running_loop().run_in_executor(pool, functools.partial(run_pricefeed, queue))
async def main():
g = Executor()
await g.run()
if __name__ == '__main__':
asyncio.run(main())
This code has a drawback in that you need to empty the queue in a non-blocking fashing from your asyncio process and wait for a while for new items to come in before emptying it again, effectively implementing a polling mechanism. If you don't wait after emptying, you'll wind up with blocking code and you will freeze the event loop again. This isn't as good as just waiting for the queue to have an item in it by blocking, but may suit your needs. If possible, I would avoid asyncio here and use multiprocessing entirely, for example, by implementing queue processing as a separate process.
I am writing a Python program that run tasks taken from a queue concurrently, to learn asyncio.
Items will be put onto a queue by interacting with a main thread (within REPL).
Whenever a task is put onto the queue, it should be consumed and executed immediately.
My approach is to kick off a separate thread and pass a queue to the event loop within that thread.
The tasks are running but only sequentially and I am not clear on how to run the tasks concurrently. My attempt is as follows:
import asyncio
import time
import queue
import threading
def do_it(task_queue):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
status = f'{clock()} {name}_{total_time}:'
print(status, 'START')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(status, 'processing...')
await asyncio.sleep(1)
current_time = time.time()
print(status, 'DONE.')
async def main():
while True:
item = task_queue.get()
if item == _sentinel:
break
await asyncio.create_task(process(*item))
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
tasks = queue.Queue()
th = threading.Thread(target=do_it, args=(tasks,))
th.start()
tasks.put(('abc', 5))
tasks.put(('def', 3))
Any advice pointing me in the direction of running these tasks concurrently would be greatly appreciated!
Thanks
UPDATE
Thank you Frank Yellin and cynthi8! I have reformed main() according to your advice:
removed await before asyncio.create_task - fixed concurrency
added wait while loop so that main would not return prematurely
used non-blocking mode of Queue.get()
The program now works as expected 👍
UPDATE 2
user4815162342 has offered further improvements, I have annotated his suggestions below.
'''
Starts auxiliary thread which establishes a queue and consumes tasks within a
queue.
Allow enqueueing of tasks from within __main__ and termination of aux thread
'''
import asyncio
import time
import threading
import functools
def do_it(started):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
print(f'{clock()} {name}_{total_time}:', 'Started.')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(f'{clock()} {name}_{total_time}:', 'Processing...')
await asyncio.sleep(1)
current_time = time.time()
print(f'{clock()} {name}_{total_time}:', 'Done.')
async def main():
# get_running_loop() get the running event loop in the current OS thread
# out to __main__ thread
started.loop = asyncio.get_running_loop()
started.queue = task_queue = asyncio.Queue()
started.set()
while True:
item = await task_queue.get()
if item == _sentinel:
# task_done is used to tell join when the work in the queue is
# actually finished. A queue length of zero does not mean work
# is complete.
task_queue.task_done()
break
task = asyncio.create_task(process(*item))
# Add a callback to be run when the Task is done.
# Indicate that a formerly enqueued task is complete. Used by queue
# consumer threads. For each get() used to fetch a task, a
# subsequent call to task_done() tells the queue that the processing
# on the task is complete.
task.add_done_callback(lambda _: task_queue.task_done())
# keep loop going until all the work has completed
# When the count of unfinished tasks drops to zero, join() unblocks.
await task_queue.join()
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
# started Event is used for communication with thread th
started = threading.Event()
th = threading.Thread(target=do_it, args=(started,))
th.start()
# started.wait() blocks until started.set(), ensuring that the tasks and
# loop variables are available from the event loop thread
started.wait()
tasks, loop = started.queue, started.loop
# call_soon schedules the callback callback to be called with args arguments
# at the next iteration of the event loop.
# call_soon_threadsafe is required to schedule callbacks from another thread
# put_nowait enqueues items in non-blocking fashion, == put(block=False)
loop.call_soon_threadsafe(tasks.put_nowait, ('abc', 5))
loop.call_soon_threadsafe(tasks.put_nowait, ('def', 3))
loop.call_soon_threadsafe(tasks.put_nowait, 'STOP')
As others pointed out, the problem with your code is that it uses a blocking queue which halts the event loop while waiting for the next item. The problem with the proposed solution, however, is that it introduces latency because it must occasionally sleep to allow other tasks to run. In addition to introducing latency, it prevents the program from ever going to sleep, even when there are no items in the queue.
An alternative is to switch to asyncio queue which is designed for use with asyncio. This queue must be created inside the running loop, so you can't pass it to do_it, you must retrieve it. Also, since it's an asyncio primitive, its put method must be invoked through call_soon_threadsafe to ensure that the event loop notices it.
One final issue is that your main() function uses another busy loop to wait for all the tasks to complete. This can be avoided by using Queue.join, which is explicitly designed for this use case.
Here is your code adapted to incorporate all of the above suggestions, with the process function remaining unchanged from your original:
import asyncio
import time
import threading
def do_it(started):
'''Process tasks in the queue until the sentinel value is received'''
_sentinel = 'STOP'
def clock():
return time.strftime("%X")
async def process(name, total_time):
status = f'{clock()} {name}_{total_time}:'
print(status, 'START')
current_time = time.time()
end_time = current_time + total_time
while current_time < end_time:
print(status, 'processing...')
await asyncio.sleep(1)
current_time = time.time()
print(status, 'DONE.')
async def main():
started.loop = asyncio.get_running_loop()
started.queue = task_queue = asyncio.Queue()
started.set()
while True:
item = await task_queue.get()
if item == _sentinel:
task_queue.task_done()
break
task = asyncio.create_task(process(*item))
task.add_done_callback(lambda _: task_queue.task_done())
await task_queue.join()
print('event loop start')
asyncio.run(main())
print('event loop end')
if __name__ == '__main__':
started = threading.Event()
th = threading.Thread(target=do_it, args=(started,))
th.start()
started.wait()
tasks, loop = started.queue, started.loop
loop.call_soon_threadsafe(tasks.put_nowait, ('abc', 5))
loop.call_soon_threadsafe(tasks.put_nowait, ('def', 3))
loop.call_soon_threadsafe(tasks.put_nowait, 'STOP')
Note: an unrelated issue with your code was that it awaited the result of create_task(), which nullified the usefulness of create_task() because it wasn't allowed to run in the background. (It would be equivalent to immediately joining a thread you've just started - you can do it, but it doesn't make much sense.) This issue is fixed both in the above code and in your edit to the question.
There are two problems with your code.
First, you should not have the await before the asyncio.create_task. This is possibly what is causing your code to run synchronously.
Then, once you've made your code run asynchronously, you need something after the while loop in main so that the code doesn't return immediately, but instead waits for all the jobs to finish. Another stackoverflow answer recommends:
while len(asyncio.Task.all_tasks()) > 1: # Any task besides main() itself?
await asyncio.sleep(0.2)
Alternatively there are versions of Queue that can keep track of running tasks.
As an additional problem:
If a queue.Queue is empty, get() blocks by default and does not return a sentinel string. https://docs.python.org/3/library/queue.html
Goal: To cancel pending tasks safely across all threads. Then safely end all the asyncio loops across all threads.
Code explanation:
I opened two threads, one for the server to run and the other for background processing. Each thread has there own separate asyncio loops.
Desired Function:
When I receive a message called onClose from the client, I want to immediately shutdown all processes safely across all the threads. Desired Function is in the func_websocket_connection() function after print('Running On Close Function')
Techniques Tried:
Of course I tried os._exit(0) to abruptly stop everything. It accomplishes what I want but I also know it is not safe and can corrupt processing data. I also tried
print('Running On Close Function')
loop = asyncio.get_event_loop()
loop = loop.stop()
which also works but I get Task was destroyed but it is pending!
Non-Server Code:
import asyncio
import websockets
from threading import Thread
from time import sleep
#=============================================
# Open Websocket Server:
def func_run_server():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
asyncio.ensure_future(func_websocket_connection())
loop.run_forever()
# Background Processing Function 1:
async def func_websocket_connection():
while i in range(100):
await asyncio.sleep(0.5)
print('Run Step Server 0')
#Some If Statement
if i == 10:
print('Running On Close Function')
#=============================================
# Open Background Processing:
def func_run_background():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
future_run_step0 = asyncio.ensure_future(func_run_step0())
future_run_step1 = asyncio.ensure_future(func_run_step1())
loop.run_until_complete(future_run_step0)
loop.run_until_complete(future_run_step1)
# Background Processing Function 1:
async def func_run_step0():
await asyncio.sleep(5.0)
print('Run Step 0')
# Background Processing Function 2:
async def func_run_step1():
await asyncio.sleep(5.0)
print('Run Step 1')
#================================================================================
#Running two separate threads
Thread(target=func_run_server).start()
Thread(target=func_run_background).start()
I have a program with one main thread where I spawn a second thread that uses asyncio. Are there any tools provided to synchronize these two threads? If everything was asyncio, I could do it with its synchronization primitives, eg:
import asyncio
async def taskA(lst, evt):
print(f'Appending 1')
lst.append(1)
evt.set()
async def taskB(lst, evt):
await evt.wait()
print('Retrieved:', lst.pop())
lst = []
evt = asyncio.Event()
asyncio.get_event_loop().run_until_complete(asyncio.gather(
taskA(lst, evt),
taskB(lst, evt),
))
However, this does not work with multiple threads. If I just use a threading.Event then it will block the asyncio thread. I figured out I could defer the wait to an executor:
import asyncio
import threading
def taskA(lst, evt):
print(f'Appending 1')
lst.append(1)
evt.set()
async def taskB(lst, evt):
asyncio.get_event_loop().run_in_executor(None, evt.wait)
print('Retrieved:', lst.pop())
def targetA(lst, evt):
taskA(lst, evt)
def targetB(lst, evt):
asyncio.set_event_loop(asyncio.new_event_loop())
asyncio.get_event_loop().run_until_complete(taskB(lst, evt))
lst = []
evt = threading.Event()
threadA = threading.Thread(target=targetA, args=(lst, evt))
threadB = threading.Thread(target=targetB, args=(lst, evt))
threadA.start()
threadB.start()
threadA.join()
threadB.join()
However, having an executor thread only to wait for a mutex seems unnatural. Is this the way this is supposed to be done? Or is there any other way to wait for synchronization between OS threads asynchronously?
A simple way to synchronize an asyncio coroutine with an event coming from another thread is to await an asyncio.Event in taskB, and set it from taskA using loop.call_soon_threadsafe.
To be able to pass values and exceptions between the two, you can use futures; however then you are inventing much of run_in_executor. If the only job of taskA is to take tasks off a queue, you might as well make a single-worker "pool" and use it as your worker thread. Then you can use run_in_executor as intended:
worker = concurrent.futures.ThreadPoolExecutor(max_workers=1)
async def taskB(lst):
loop = asyncio.get_event_loop()
# or result = await ..., if taskA has a useful return value
# This will also propagate exceptions raised by taskA
await loop.run_in_executor(worker, taskA, lst)
print('Retrieved:', lst.pop())
The semantics are the same as in your version with an explicit queue - the queue is still there, it's just inside the ThreadPoolExecutor.
Consider the following program.
import asyncio
import multiprocessing
from multiprocessing import Queue
from concurrent.futures.thread import ThreadPoolExecutor
import sys
def main():
executor = ThreadPoolExecutor()
loop = asyncio.get_event_loop()
# comment the following line and the shutdown will work smoothly
asyncio.ensure_future(print_some(executor))
try:
loop.run_forever()
except KeyboardInterrupt:
print("shutting down")
executor.shutdown()
loop.stop()
loop.close()
sys.exit()
async def print_some(executor):
print("Waiting...Hit CTRL+C to abort")
queue = Queue()
loop = asyncio.get_event_loop()
some = await loop.run_in_executor(executor, queue.get)
print(some)
if __name__ == '__main__':
main()
All I want is a graceful shutdown when I hit "CTRL+C". However, the executor thread seems to prevent that (even though I do call shutdown)
You need to send a poison pill to make the workers stop listening on the queue.get call. Worker threads in the ThreadPoolExecutor pool will block Python from exiting if they have active work. There's a comment in the source code that describes the reasoning for this behavior:
# Workers are created as daemon threads. This is done to allow the interpreter
# to exit when there are still idle threads in a ThreadPoolExecutor's thread
# pool (i.e. shutdown() was not called). However, allowing workers to die with
# the interpreter has two undesirable properties:
# - The workers would still be running during interpreter shutdown,
# meaning that they would fail in unpredictable ways.
# - The workers could be killed while evaluating a work item, which could
# be bad if the callable being evaluated has external side-effects e.g.
# writing to a file.
#
# To work around this problem, an exit handler is installed which tells the
# workers to exit when their work queues are empty and then waits until the
# threads finish.
Here's a complete example that exits cleanly:
import asyncio
import multiprocessing
from multiprocessing import Queue
from concurrent.futures.thread import ThreadPoolExecutor
import sys
def main():
executor = ThreadPoolExecutor()
loop = asyncio.get_event_loop()
# comment the following line and the shutdown will work smoothly
fut = asyncio.ensure_future(print_some(executor))
try:
loop.run_forever()
except KeyboardInterrupt:
print("shutting down")
queue.put(None) # Poison pill
loop.run_until_complete(fut)
executor.shutdown()
loop.stop()
loop.close()
async def print_some(executor):
print("Waiting...Hit CTRL+C to abort")
loop = asyncio.get_event_loop()
some = await loop.run_in_executor(executor, queue.get)
print(some)
queue = None
if __name__ == '__main__':
queue = Queue()
main()
The run_until_complete(fut) call is needed to avoid a warning about a pending task hanging around when the asyncio eventloop exits. If you don't care about that, you can leave that call out.