I have some threads running, and one of those threads contains an object that will be spawning subprocesses. I want one such subprocess to be able to kill the entire application. The aforementioned object will need to save some state when it receives this signal. Unfortunately I can't get the signal to be handled in the thread that causes the kill.
Here is some example code that attempts to replicate the situation.
parent.py: starts a thread. that thread runs some subprocesses, one of which will try to kill the parent process.
#!/usr/local/bin/python3
import subprocess, time, threading, random
def killer_func():
possible_cmds = [['echo', 'hello'],
['echo', 'world'],
['/work/turbulencetoo/tmp/killer.py']
]
random.shuffle(possible_cmds)
for cmd in possible_cmds:
try:
time.sleep(2)
subprocess.check_call(cmd)
time.sleep(2)
except KeyboardInterrupt:
print("Kill -2 caught properly!!")
print("Here I could properly save my state")
break
except Exception as e:
print("Unhandled Exception: {}".format(e))
else:
print("No Exception")
killer_thread = threading.Thread(target=killer_func)
killer_thread.start()
try:
while True:
killer_thread.join(4)
if not killer_thread.is_alive():
print("The killer thread has died")
break
else:
print("Killer thread still alive, try to join again.")
except KeyboardInterrupt:
print("Caught the kill -2 in the main thread :(")
print("Main program shutting down")
killer.py, a simple program that tries to kill its parent process with SIGINT:
#!/usr/local/bin/python3
import time, os, subprocess, sys
ppid = os.getppid()
# -2 specifies SIGINT, python handles this as a KeyboardInterrupt exception
cmd = ["kill", "-2", "{}".format(ppid)]
subprocess.check_call(cmd)
time.sleep(3)
sys.exit(0)
Here is some sample output from running the parent program:
$ ./parent.py
hello
Killer thread still alive, try to join again.
No Exception
Killer thread still alive, try to join again.
Caught the kill -2 in the main thread :(
Main program shutting down
No Exception
world
No Exception
I've tried using signal.signal() inside killer_func, but it doesn't work in a sub thread.
Is there a way to force the signal or exception to be handled by the function without the main thread being aware?
The main thread of your program will always be the one that receives the signal. The signal module documentation states this:
Some care must be taken if both signals and threads are used in the
same program. The fundamental thing to remember in using signals and
threads simultaneously is: always perform signal() operations in the
main thread of execution. Any thread can perform an alarm(),
getsignal(), pause(), setitimer() or getitimer(); only the main thread
can set a new signal handler, and the main thread will be the only one
to receive signals (this is enforced by the Python signal module, even
if the underlying thread implementation supports sending signals to
individual threads). This means that signals can’t be used as a means
of inter-thread communication. Use locks instead.
You'll need to refactor your program such that the main thread receiving the signal doesn't prevent you from saving state. The easiest way is use something like threading.Event() to tell the background thread that the program has been aborted, and let it clean up when it sees the event has been set:
import subprocess
import threading
import random
def killer_func(event):
possible_cmds = [['echo', 'hello'],
['echo', 'world'],
['/home/cycdev/killer.py']
]
random.shuffle(possible_cmds)
for cmd in possible_cmds:
subprocess.check_call(cmd)
event.wait(4)
if event.is_set():
print("Main thread got a signal. Time to clean up")
# save state here.
return
event = threading.Event()
killer_thread = threading.Thread(target=killer_func, args=(event,))
killer_thread.start()
try:
killer_thread.join()
except KeyboardInterrupt:
print("Caught the kill -2 in the main thread :)")
event.set()
killer_thread.join()
print("Main program shutting down")
Signals are always handled in the main thread. When you receive a signal, you don't know where it comes from. You can't say "handle it in the thread that spawned the signal-sending-process" because you don't know what signal-sending-process is.
The way to solve this is to use Condition Variables to notify all threads that a signal was received and that they have to shut down.
import threading
got_interrupt = False # global variable
def killer_func(cv):
...
with cv:
cv.wait(2)
interupted = got_interrupt # Read got_interrupt while holding the lock
if interrupted:
cleanup()
...
lock = threading.Lock()
notifier_cv = threading.Condition(lock)
killer_thread = threading.Thread(target=killer_func, args=(notifier_cv,))
killer_thread.start()
try:
...
except KeyboardInterrupt:
with cv:
got_interrupt = True
cv.notify_all()
Related
The only mechanism I can find for handling a keyboard interrupt is to poll. Without the while loop below, the signal processing never happens and the process hangs forever.
Is there any graceful mechanism for allowing a keyboard interrupt to function when given a concurrent future object?
Putting polling loops all over my code base seems to defeat the purpose of using futures at all.
More info:
Waiting on the future in the main thread in Windows blocks all signal handling, even if it's fully cancellable and even if it has not "started" yet. The word "exiting" doesn't even print. So 'cancellability' is only part (the easy part) of the issue.
In my real code, I obtain futures via executors (run coro threadsafe, in this case), this was just a simplified example
import concurrent.futures
import signal
import time
import sys
fut = concurrent.futures.Future()
def handler(signum, frame):
print("exiting")
fut.cancel()
signal.signal(signal.SIGINT, orig)
sys.exit()
orig = signal.signal(signal.SIGINT, handler)
# a time sleep is fully interruptible with a signal... but a future isnt
# time.sleep(100)
while True:
try:
fut.result(.03)
except concurrent.futures.TimeoutError:
pass
Defeating the purpose or not, it is how futures currently work in Python.
First of all, directly instantiating a Future() should only be done for testing purposes, normally you would obtain an instance by submitting work to an executor.
Furthermore, you cannot really cancel() a future cleanly that is executing in a thread; attempting to do so will make cancel() return False.
Indeed, in the following test I get could cancel: False printed out:
import concurrent.futures
import signal
import time
import sys
def task(delay):
time.sleep(delay)
return delay
def handler(signum, frame):
print("exiting")
print("could cancel:", fut.cancel())
raise RuntimeError("if in doubt, use brute force")
signal.signal(signal.SIGINT, handler)
with concurrent.futures.ThreadPoolExecutor() as executor:
fut = executor.submit(task, 240)
try:
print(fut.result())
except Exception as ex:
print(f"fut.result() ==> {type(ex).__name__}: {ex}")
If I also raise an exception in my signal handler, that exception is caught when trying to fetch the result, and I'm also seeing fut.result() ==> RuntimeError: if in doubt, use brute force printed out. However, that does not exit the executor loop immediately either, because the task is still running there.
Interestingly, clicking Ctrl-C a couple more times would eventually break even the cleanup loop, and the program would exit, but it's probably not what you're after. You might also be able to kill off futures more freely by employing a ProcessPoolExecutor, but .cancel() would still return False for running futures.
In that light, I think your approach to poll result() is not an unreasonable one. If possible, you could also move your program to asyncio where you would be able to cancel tasks at the yield points or I/O, or somehow make your task itself react to user input by exiting earlier, potentially based on information from a signal.
For instance, here I'm setting a global variable from my interrupt handler, which is then polled from my task:
import concurrent.futures
import signal
import time
import sys
interrupted = False
def task(delay):
slept = 0
for _ in range(int(delay)):
time.sleep(1)
slept += 1
if interrupted:
print("interrupted, wrapping up work prematurely")
break
return slept
def handler(signum, frame):
global interrupted
print("exiting")
print("could cancel:", fut.cancel())
interrupted = True
signal.signal(signal.SIGINT, handler)
with concurrent.futures.ThreadPoolExecutor() as executor:
fut = executor.submit(task, 40)
try:
print(fut.result())
except Exception as ex:
print(f"fut.result() ==> {type(ex).__name__}: {ex}")
Now I'm able to interrupt my work in a more fine grained fashion:
^Cexiting
could cancel: False
interrupted, wrapping up work prematurely
5
In addition, you might also be able to split your work into many smaller tasks, then you could cancel any futures that aren't running yet, also improving responsiveness to SIGINT or other types of user input.
OK, I wrote a solution to this based on digging in cypython source and some bug reports - but it's not pretty.
If you want to be able to interrupt a future, especially on Windows, the following seems to work:
#contextlib.contextmanager
def interrupt_futures(futures): # pragma: no cover
"""Allows a list of futures to be interrupted.
If an interrupt happens, they will all have their exceptions set to KeyboardInterrupt
"""
# this has to be manually tested for now, because the tests interfere with the test runner
def do_interr(*_):
for ent in futures:
try:
ent.set_exception(KeyboardInterrupt)
except:
# if the future is already resolved or cancelled, ignore it
pass
return 1
if sys.platform == "win32":
from ctypes import wintypes # pylint: disable=import-outside-toplevel
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
CTRL_C_EVENT = 0
CTRL_BREAK_EVENT = 1
HANDLER_ROUTINE = ctypes.WINFUNCTYPE(wintypes.BOOL, wintypes.DWORD)
#HANDLER_ROUTINE
def handler(ctrl):
if ctrl == CTRL_C_EVENT:
handled = do_interr()
elif ctrl == CTRL_BREAK_EVENT:
handled = do_interr()
else:
handled = False
# If not handled, call the next handler.
return handled
if not kernel32.SetConsoleCtrlHandler(handler, True):
raise ctypes.WinError(ctypes.get_last_error())
was = signal.signal(signal.SIGINT, do_interr)
yield
signal.signal(signal.SIGINT, was)
# restore default handler
kernel32.SetConsoleCtrlHandler(handler, False)
else:
was = signal.signal(signal.SIGINT, do_interr)
yield
signal.signal(signal.SIGINT, was)
This allows you to do this:
with interrupt_futures([fut]):
fut.result()
For the duration of that call, interrupt signals will be intercepted and will result in the future raising a KeyboardInterrupt to the caller requesting the result - instead of simply ignoring all interrupts.
I stumble upon pice of code that is a bit werid, I'm expecting when the program is signaled, the exit() should raise SystemExit once and cause the program to exit, however in this case, when the main thread is blocking on th.join(), the exit() statement needs be called twice for the program to exit.
It is not a practical exercise, but I want to know what is going on under the hood.
import threading
import time
import signal
def task():
while True:
time.sleep(1)
def sig_handler(self, *_):
# raise ValueError()
exit()
def main():
signal.signal(signal.SIGINT, sig_handler)
th = threading.Thread(target=task)
th.start()
th.join()
if __name__ == "__main__":
main()
Usually, our main program implicitly waits until all other threads have completed their work. Using daemon threads is useful for services where there may not be an easy way to interrupt the thread or where letting the thread die in the middle of its work without losing or corrupting data. to set a thread as a daemon that runs without blocking the main program from exiting. use setDaemon() method. Your main function will be:
def main():
signal.signal(signal.SIGINT, sig_handler)
th = threading.Thread(target=task)
th.setDaemon(True)
th.start()
th.join()
I have a main python(testmain.py) script that executes another python script(test.py) using subprocess.Popen command. When I press Ctrl-C , I want the child to exit with exit code 2 and then the parent to display that exit code and then terminate .
I have signal handlers in both parent and child scripts.
testmain.py
def signal_handler(signal, frame):
print "outer signal handler"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
def execute()
proc=subprocess.Popen("python test.py",shell=True)
streamdata=proc.communicate()[0]
rc=proc.returncode
print "return code:",rc
execute()
test.py
def signal_handler(signal, frame):
print "exiting: inner function"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
I checked Delegate signal handling to a child process in python that is kind of similar to my question but in that case, the parent is continuing it's execution, which I don't want.
I want to: 1.exit test.py with exit(2) 2.print that exit code in testmain.py 3.exit test.py with exit(2)
could someone please provide suggestions to do this?
Thanks.
UPDATE : Handling the signal only in the child (test.py) and checking the return code in parent(testmain.py) will do what I want .
if rc==2:
print "child was terminated"
exit(2)
but I was wondering if there is a clean way to do this using signal handling.
Your child process shouldn't care what the parent does i.e., if you want the child to exit with specific status on Ctrl+C then just do that:
import sys
try:
main()
except KeyboardInterrupt: # use default SIGINT handler
sys.exit(2)
Or you could define the signal handler explicitly:
import os
import signal
def signal_handler(signal, frame):
os.write(1, b"outer signal handler\n")
os._exit(2)
signal.signal(signal.SIGINT, signal_handler)
main()
There might be a difference in behavior if there are atexit handlers and/or multiple threads.
Unrelated: depending on what your main() function does, there could be a significant delay before a signal is handled in Python. Some blocking methods on Python 2 may ignore the signal completely: use Python 3 or apply a custom workaround for a specific case e.g., using a timeout parameter for some calls.
You could handle SIGINT in a similar way in the parent:
for cmd in commands:
process = Popen(cmd)
try:
process.wait()
except KeyboardInterrupt:
# child process may still be alive here
for _ in range(5): # wait a while
if process.poll() is not None:
break # the process is dead
time.sleep(.1)
else: # no break, kill the process explicitly
try:
process.kill()
except OSError:
pass
sys.exit("Child exited with %d" % process.wait())
Python 2 doesn't restore signals for child processes e.g., if you SIG_IGN the SIGINT signal in the parent, you could reset the necessary hanlders explicitly using preexec_fn parameter.
Duplicate of How to prevent a block of code from being interrupted by KeyboardInterrupt in Python?
Purpose:
When program receives a signal, it exits only when it is ready.
Problem:
Desired behavior:
Signal arrives during target block. Program exits after target block finishes.
Signal arrives outside target block. Program exits.
Current behavior:
Signal arrives during target block. Program hangs.
Signal arrives outside target block. Program exits.
Generic implementation of my design:
import threading, signal
def huphandler(signum, frame):
with sig_lock:
os._exit(0)
def loop():
print 'in lock'
time.sleep(5)
def idle_loop():
while True:
with sig_lock:
t = multiprocessing.Process(target=loop)
t.start()
t.join()
print 'out lock'
time.sleep(5)
sig_lock = threading.Lock()
signal.signal(signal.SIGHUP, huphandler)
idle_loop()
If you kill with the lock acquired then you put your program in a deadlock state.
Because you just killed the thread or subprocess that holds the lock, it can no longer release it. Then your main thread, or any other thread, will try to acquire the lock, and block until the lock is available. Thus, ending in a deadlock.
1.1. Problem:
The signal handler will interrupt the main thread, causing deadlock.
1.2. Solution:
Use a boolean instead of a lock.
2.1. Problem:
When a thread is interrupted, it may cease mid-operation (i.e., time.sleep) which is unstable.
2.2. Solution:
For greater stability, wrap the target code in a subprocess.
Generic implementation which provides desired results:
import threading, signal
def huphandler(signum, frame):
print idle_loop.sig_lock
if idle_loop.sig_lock:
idle_loop.continuee = False
else:
print 'dont b here'
os._exit(0)
def loop():
print 'in lock'
time.sleep(5)
def idle_loop():
while True:
idle_loop.sig_lock = True
t = multiprocessing.Process(target=loop)
t.start()
t.join()
print 'out lock'
if idle_loop.continuee:
idle_loop.sig_lock = False
time.sleep(5)
else: return
idle_loop.sig_lock = False
idle_loop.continuee = True
signal.signal(signal.SIGHUP, huphandler)
idle_loop()
I'm running this simple code:
import threading, time
class reqthread(threading.Thread):
def run(self):
for i in range(0, 10):
time.sleep(1)
print('.')
try:
thread = reqthread()
thread.start()
except (KeyboardInterrupt, SystemExit):
print('\n! Received keyboard interrupt, quitting threads.\n')
But when I run it, it prints
$ python prova.py
.
.
^C.
.
.
.
.
.
.
.
Exception KeyboardInterrupt in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
In fact python thread ignore my Ctrl+C keyboard interrupt and doesn't print Received Keyboard Interrupt. Why? What is wrong with this code?
Try
try:
thread=reqthread()
thread.daemon=True
thread.start()
while True: time.sleep(100)
except (KeyboardInterrupt, SystemExit):
print '\n! Received keyboard interrupt, quitting threads.\n'
Without the call to time.sleep, the main process is jumping out of the try...except block too early, so the KeyboardInterrupt is not caught. My first thought was to use thread.join, but that seems to block the main process (ignoring KeyboardInterrupt) until the thread is finished.
thread.daemon=True causes the thread to terminate when the main process ends.
To summarize the changes recommended in the comments, the following works well for me:
try:
thread = reqthread()
thread.start()
while thread.isAlive():
thread.join(1) # not sure if there is an appreciable cost to this.
except (KeyboardInterrupt, SystemExit):
print '\n! Received keyboard interrupt, quitting threads.\n'
sys.exit()
Slight modification of ubuntu's solution.
Removing tread.daemon = True as suggested by Eric and replacing the sleeping loop by signal.pause():
import signal
try:
thread=reqthread()
thread.start()
signal.pause() # instead of: while True: time.sleep(100)
except (KeyboardInterrupt, SystemExit):
print '\n! Received keyboard interrupt, quitting threads.\n'
My (hacky) solution is to monkey-patch Thread.join() like this:
def initThreadJoinHack():
import threading, thread
mainThread = threading.currentThread()
assert isinstance(mainThread, threading._MainThread)
mainThreadId = thread.get_ident()
join_orig = threading.Thread.join
def join_hacked(threadObj, timeout=None):
"""
:type threadObj: threading.Thread
:type timeout: float|None
"""
if timeout is None and thread.get_ident() == mainThreadId:
# This is a HACK for Thread.join() if we are in the main thread.
# In that case, a Thread.join(timeout=None) would hang and even not respond to signals
# because signals will get delivered to other threads and Python would forward
# them for delayed handling to the main thread which hangs.
# See CPython signalmodule.c.
# Currently the best solution I can think of:
while threadObj.isAlive():
join_orig(threadObj, timeout=0.1)
else:
# In all other cases, we can use the original.
join_orig(threadObj, timeout=timeout)
threading.Thread.join = join_hacked
Putting the try ... except in each thread and also a signal.pause() in true main() works for me.
Watch out for import lock though. I am guessing this is why Python doesn't solve ctrl-C by default.