Consider the following program.
import asyncio
import multiprocessing
from multiprocessing import Queue
from concurrent.futures.thread import ThreadPoolExecutor
import sys
def main():
executor = ThreadPoolExecutor()
loop = asyncio.get_event_loop()
# comment the following line and the shutdown will work smoothly
asyncio.ensure_future(print_some(executor))
try:
loop.run_forever()
except KeyboardInterrupt:
print("shutting down")
executor.shutdown()
loop.stop()
loop.close()
sys.exit()
async def print_some(executor):
print("Waiting...Hit CTRL+C to abort")
queue = Queue()
loop = asyncio.get_event_loop()
some = await loop.run_in_executor(executor, queue.get)
print(some)
if __name__ == '__main__':
main()
All I want is a graceful shutdown when I hit "CTRL+C". However, the executor thread seems to prevent that (even though I do call shutdown)
You need to send a poison pill to make the workers stop listening on the queue.get call. Worker threads in the ThreadPoolExecutor pool will block Python from exiting if they have active work. There's a comment in the source code that describes the reasoning for this behavior:
# Workers are created as daemon threads. This is done to allow the interpreter
# to exit when there are still idle threads in a ThreadPoolExecutor's thread
# pool (i.e. shutdown() was not called). However, allowing workers to die with
# the interpreter has two undesirable properties:
# - The workers would still be running during interpreter shutdown,
# meaning that they would fail in unpredictable ways.
# - The workers could be killed while evaluating a work item, which could
# be bad if the callable being evaluated has external side-effects e.g.
# writing to a file.
#
# To work around this problem, an exit handler is installed which tells the
# workers to exit when their work queues are empty and then waits until the
# threads finish.
Here's a complete example that exits cleanly:
import asyncio
import multiprocessing
from multiprocessing import Queue
from concurrent.futures.thread import ThreadPoolExecutor
import sys
def main():
executor = ThreadPoolExecutor()
loop = asyncio.get_event_loop()
# comment the following line and the shutdown will work smoothly
fut = asyncio.ensure_future(print_some(executor))
try:
loop.run_forever()
except KeyboardInterrupt:
print("shutting down")
queue.put(None) # Poison pill
loop.run_until_complete(fut)
executor.shutdown()
loop.stop()
loop.close()
async def print_some(executor):
print("Waiting...Hit CTRL+C to abort")
loop = asyncio.get_event_loop()
some = await loop.run_in_executor(executor, queue.get)
print(some)
queue = None
if __name__ == '__main__':
queue = Queue()
main()
The run_until_complete(fut) call is needed to avoid a warning about a pending task hanging around when the asyncio eventloop exits. If you don't care about that, you can leave that call out.
Related
The following code requires 3 presses of CTRL-C to end, how can I make it end with one only? (So it works nicely in Docker)
import asyncio
import time
def sleep_blocking():
print("Sleep blocking")
time.sleep(1000)
async def main():
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, sleep_blocking)
try:
asyncio.run(main())
except KeyboardInterrupt:
print("Nicely shutting down ...")
I've read many asyncio related questions and answers but can't figure this one out yet. The 1st CTRL-C does nothing, the 2nd prints "Nicely shutting down ..." and then hangs. The 3rd CTRL-C prints an ugly error.
I'm on Python 3.9.10 and Linux.
(edit: updated code per comment #mkrieger1)
The way to exit immediately and unconditionally from a Python program is by calling os._exit(). If your background threads are in the middle of doing something important this may not be wise. However the following program does what you asked (python 3.10, Windows10):
import asyncio
import time
import os
def sleep_blocking():
print("Sleep blocking")
time.sleep(1000)
async def main():
loop = asyncio.get_event_loop()
loop.run_until_complete(loop.run_in_executor(None, sleep_blocking))
try:
asyncio.run(main())
except KeyboardInterrupt:
print("Nicely shutting down ...")
os._exit(42)
From here we know that it's effectively impossible to kill a task running in a thread executor. If I replace the default thread executor with a ProcessPoolExecutor, I get the behavior you're looking for. Here's the code:
import concurrent.futures
import asyncio
import time
def sleep_blocking():
print("Sleep blocking")
time.sleep(1000)
async def main():
loop = asyncio.get_event_loop()
x = concurrent.futures.ProcessPoolExecutor()
await loop.run_in_executor(x, sleep_blocking)
try:
asyncio.run(main())
except KeyboardInterrupt:
print("Nicely shutting down ...")
And the result is:
$ python asynctest.py
Sleep blocking
^CNicely shutting down ...
Assume we have the following script:
def sync_sleep(x):
print("sync_sleep going to sleep")
time.sleep(x)
print("sync_sleep awake")
async def main():
loop = asyncio.get_running_loop()
loop.run_in_executor(None, sync_sleep, 4)
print("main going to sleep")
await asyncio.sleep(1)
print("main awake ")
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print("main finished")
Note: I have omitted loop.close intentionally.
Now when I execute the above script:
python script.py
I have the following output:
sync_sleep going to sleep
main going to sleep
main awake
main finished
sync_sleep awake
I expected after running the script, the process to exit after printing "main finished", but it did not until the job sync_sleep has finished. I mean, if I wanted to wait for that job, I would have added the line:
loop.run_until_complete(loop.shutdown_default_executor())
Is my expectation wrong?
The default ThreadPoolExecutor waits at exit for daemon threads, meaning the process waits for the thread to finish, and only then stops.
Apart from implementing an Executor class yourself , there's no option other than using the threading module to create a new thread with daemon=True. If you wish to wait for that thread in asyncio, you can always run an executor to await the thread.join().
I am prototyping a FastAPI app with an endpoint that will launch long-running process using subprocess module. The obvious solution is to use concurrent.futures and ProcessPoolExecutor, however I am unable to get the behavior I want. Code sample:
import asyncio
from concurrent.futures import ProcessPoolExecutor
import subprocess as sb
import time
import random
pool = ProcessPoolExecutor(5)
def long_task(s):
print("started")
time.sleep(random.randrange(5, 15))
sb.check_output(["touch", str(s)])
print("done")
async def async_task():
loop = asyncio.get_event_loop()
print("started")
tasks = [loop.run_in_executor(pool, long_task, i) for i in range(10)]
while True:
print("in async task")
done, _ = await asyncio.wait(tasks, timeout=1)
for task in done:
await task
await asyncio.sleep(1)
def main():
loop = asyncio.get_event_loop()
loop.run_until_complete(async_task())
if __name__ == "__main__":
main()
This sample works fine, on the surface, but spawned processes do not get stopped after execution completes - I see all of python processes in ps aux | grep python. Shouldn't awaiting completed task stop it? In the end I do not care much about the result of the execution, it just should happen in the background and exit cleanly - without any hanging processes.
You must close the ProcessPool when you are done using it, either by explicitly calling its shutdown() method, or using it in a ContextManager. I used the ContextManager approach.
I don't know what subprocess.check_output does, so I commented it out.
I also replaced your infinite loop with a single call to asyncio.gather, which will yield until the Executor is finished.
I'm on Windows, so to observe the creation/deletion of Processes I watched the Windows Task Manager. The program creates 5 subprocesses and closes them again when the ProcessPool context manager exits.
import asyncio
from concurrent.futures import ProcessPoolExecutor
# import subprocess as sb
import time
import random
def long_task(s):
print("started")
time.sleep(random.randrange(5, 15))
# sb.check_output(["touch", str(s)])
print("done", s)
async def async_task():
loop = asyncio.get_event_loop()
print("started")
with ProcessPoolExecutor(5) as pool:
tasks = [loop.run_in_executor(pool, long_task, i) for i in range(10)]
await asyncio.gather(*tasks)
print("Completely done")
def main():
asyncio.run(async_task())
if __name__ == "__main__":
main()
I have an asyncio event loop to which I add a signal handler, using loop.add_signal_handler(). The signals I want to catch are SIGINT, SIGHUP and SIGTERM to gracefully shutdown my event loop.
From this event loop I want to fork processes, using multiprocessing.Process(). This process p I want to be able to terminate from the event loop using p.terminate(). However, the signal handler will catch the SIGTERM signal issued by p.terminate(), prompting the execution of my shutdown code, leaving p running.
I have not found any solutions to this. Most posts say you should refrain from using termination signals and seek to use multiprocessing.Queue() by passing e.g. None and handle this in the child-process. While I see the usefulness and cleanness of this approach, in my case using a multiprocessing.Queue() will not be feasible. Am I trying something impossible or have I missed something?
I have created a minimum example:
import asyncio
import multiprocessing as mp
import signal
import time
class Test():
def __init__(self):
self.event_loop = asyncio.get_event_loop()
def create_mp(self):
# Wrapper for creating process
self.p = mp.Process(target=worker)
self.p.start()
async def shutdown_mp(self):
# Shutdown after 5 sec
await asyncio.sleep(5)
self.p.terminate()
async def async_print(self):
# Simple coroutine printing
while True:
await asyncio.sleep(1)
print("Async_test")
async def shutdown_event_loop(self, signal):
# Graceful shutdown, inspiration from roguelynn (Lynn Root)
print("Received exit signal {}".format(signal.name))
tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]
# Cancel all tasks
[task.cancel() for task in tasks]
print("Cancelling {} outstanding tasks".format(len(tasks)))
await asyncio.gather(*tasks, return_exceptions=True)
self.event_loop.stop()
def run(self):
# Add signals
signals = (signal.SIGINT, signal.SIGHUP, signal.SIGTERM)
for s in signals:
self.event_loop.add_signal_handler(
s, lambda s=s: self.event_loop.create_task(self.shutdown_event_loop(s)))
# Schedule async task
self.event_loop.create_task(self.async_print())
# Start processes
self.create_mp()
# Schedule process to be terminated
self.event_loop.create_task(self.shutdown_mp())
self.event_loop.run_forever()
def worker():
# Simple process
while True:
print("Test")
time.sleep(1)
if __name__ == "__main__":
test = Test()
test.run()
I have not been able to simply KeyboardInterrupt cancel the processes. Instead I use pgrep python3 and kill -9 PID.
I'm using Python 3. I have the following code that tries to catch CTRL+C when running a pool of async workers. Each worker just runs in an infinite loop waiting for messages to show up in a queue for processing. What I don't understand is why the log.info inside the except block and print message are not printed when I press CTRL+C. It just drops me back into the xterm. Is the with block doing something to prevent this?
def worker(q):
"""Worker to retrieve item from queue and process it.
Args:
q -- Queue
"""
# Run in an infinite loop. Get an item from the queue to process it. We MUST call q.task_done() to indicate
# that item is processed to prevent deadlock.
while True:
try:
# item = q.get()
q.get()
# TODO: We'll do work here.
log.info('Processed message')
finally:
q.task_done()
def some_func():
...
# Run in an infinite loop unless killed by user.
try:
log.info('Create pool with worker=%d to process messages', args.workers)
with mp.Pool(processes=4) as pool:
p = pool.apply_async(worker, (queue,))
p.get()
except KeyboardInterrupt:
log.info('Received CTRL-C ... exiting')
pass
print('got here')
return 0
Use asyncio, not multiprocessing
Depending on the nature of work to be done (whether CPU intensive or IO intensive) you might try asyncio, which has a simple pattern for graceful shutdown:
def main():
queue = asyncio.Queue()
loop = asyncio.get_event_loop()
try:
loop.create_task(publish(queue))
loop.create_task(consume(queue))
loop.run_forever()
except KeyboardInterrupt:
logging.info("Process interrupted")
finally:
loop.close()
^ from https://www.roguelynn.com/words/asyncio-graceful-shutdowns/
If you must use multiprocessing
This question has been answered here: Python multiprocessing : Killing a process gracefully