Develop Reference

Python threads and linux ioctl wait

I have the following toy example for Python threading module
from __future__ import print_function
import threading
import time
import signal
import sys
import os
import time
class ThreadShutdown(Exception):
# Custom exception to allow clean thread exit
pass
def thread_shutdown(signum, frame):
print(" o Signal {} caught and raising ThreadShutdown exception".format(signum))
raise ThreadShutdown
def main():
"""
Register the signal handlers needed to stop
cleanly the child accessing thread
"""
signal.signal(signal.SIGTERM, thread_shutdown)
signal.signal(signal.SIGINT, thread_shutdown)
test_run_seconds = 120
try:
thread = ChildThread()
thread.start()
time.sleep(1)
while test_run_seconds > 0:
test_run_seconds -= 1
print(" o [{}] remaining time is {} seconds".format(time.asctime( time.localtime(time.time()) ), test_run_seconds))
time.sleep(1)
except ThreadShutdown:
thread.shutdown_flag.set()
thread.join()
print(" o ThreadShutdown procedure complete")
return
proc.terminate()
thread.shutdown_flag.set()
thread.join()
print(" o Test terminated")
class ChildThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.shutdown_flag = threading.Event()
def run(self):
while not self.shutdown_flag.is_set():
print(" o [{}] is the current time in child, sleep for 10s".format(time.asctime( time.localtime(time.time()))))
time.sleep(10)
return
if __name__ == "__main__":
sys.exit(main())
which behaves as expected (the main thread counts every second while the spawned thread prints only every 10 seconds.
I was trying to understand the behaviour of the same code snippet in presence of blocking waits in kernel mode in the spawned thread. For example, assume that the spawned thread now goes into a killable wait in an ioctl with a timeout of 10s, I would still expect to have the main thread counting every second. For some reason, it instead counts every 10s, as if it was blocked as well in the wait of the spawned thread. What is the reason?

delegate SIGINT signal to child process and then cleanup and terminate the parent

I have a main python(testmain.py) script that executes another python script(test.py) using subprocess.Popen command. When I press Ctrl-C , I want the child to exit with exit code 2 and then the parent to display that exit code and then terminate .
I have signal handlers in both parent and child scripts.
testmain.py
def signal_handler(signal, frame):
print "outer signal handler"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
def execute()
proc=subprocess.Popen("python test.py",shell=True)
streamdata=proc.communicate()[0]
rc=proc.returncode
print "return code:",rc
execute()
test.py
def signal_handler(signal, frame):
print "exiting: inner function"
exit(2)
signal.signal(signal.SIGINT, signal_handler)
I checked Delegate signal handling to a child process in python that is kind of similar to my question but in that case, the parent is continuing it's execution, which I don't want.
I want to: 1.exit test.py with exit(2) 2.print that exit code in testmain.py 3.exit test.py with exit(2)
could someone please provide suggestions to do this?
Thanks.
UPDATE : Handling the signal only in the child (test.py) and checking the return code in parent(testmain.py) will do what I want .
if rc==2:
print "child was terminated"
exit(2)
but I was wondering if there is a clean way to do this using signal handling.

Your child process shouldn't care what the parent does i.e., if you want the child to exit with specific status on Ctrl+C then just do that:
import sys
try:
main()
except KeyboardInterrupt: # use default SIGINT handler
sys.exit(2)
Or you could define the signal handler explicitly:
import os
import signal
def signal_handler(signal, frame):
os.write(1, b"outer signal handler\n")
os._exit(2)
signal.signal(signal.SIGINT, signal_handler)
main()
There might be a difference in behavior if there are atexit handlers and/or multiple threads.
Unrelated: depending on what your main() function does, there could be a significant delay before a signal is handled in Python. Some blocking methods on Python 2 may ignore the signal completely: use Python 3 or apply a custom workaround for a specific case e.g., using a timeout parameter for some calls.
You could handle SIGINT in a similar way in the parent:
for cmd in commands:
process = Popen(cmd)
try:
process.wait()
except KeyboardInterrupt:
# child process may still be alive here
for _ in range(5): # wait a while
if process.poll() is not None:
break # the process is dead
time.sleep(.1)
else: # no break, kill the process explicitly
try:
process.kill()
except OSError:
pass
sys.exit("Child exited with %d" % process.wait())
Python 2 doesn't restore signals for child processes e.g., if you SIG_IGN the SIGINT signal in the parent, you could reset the necessary hanlders explicitly using preexec_fn parameter.

How to process SIGTERM signal gracefully?

Let's assume we have such a trivial daemon written in python:
def mainloop():
while True:
# 1. do
# 2. some
# 3. important
# 4. job
# 5. sleep
mainloop()
and we daemonize it using start-stop-daemon which by default sends SIGTERM (TERM) signal on --stop.
Let's suppose the current step performed is #2. And at this very moment we're sending TERM signal.
What happens is that the execution terminates immediately.
I've found that I can handle the signal event using signal.signal(signal.SIGTERM, handler) but the thing is that it still interrupts the current execution and passes the control to handler.
So, my question is - is it possible to not interrupt the current execution but handle the TERM signal in a separated thread (?) so that I was able to set shutdown_flag = True so that mainloop() had a chance to stop gracefully?

A class based clean to use solution:
import signal
import time
class GracefulKiller:
kill_now = False
def __init__(self):
signal.signal(signal.SIGINT, self.exit_gracefully)
signal.signal(signal.SIGTERM, self.exit_gracefully)
def exit_gracefully(self, *args):
self.kill_now = True
if __name__ == '__main__':
killer = GracefulKiller()
while not killer.kill_now:
time.sleep(1)
print("doing something in a loop ...")
print("End of the program. I was killed gracefully :)")

First, I'm not certain that you need a second thread to set the shutdown_flag.
Why not set it directly in the SIGTERM handler?
An alternative is to raise an exception from the SIGTERM handler, which will be propagated up the stack. Assuming you've got proper exception handling (e.g. with with/contextmanager and try: ... finally: blocks) this should be a fairly graceful shutdown, similar to if you were to Ctrl+C your program.
Example program signals-test.py:
#!/usr/bin/python
from time import sleep
import signal
import sys
def sigterm_handler(_signo, _stack_frame):
# Raises SystemExit(0):
sys.exit(0)
if sys.argv[1] == "handle_signal":
signal.signal(signal.SIGTERM, sigterm_handler)
try:
print "Hello"
i = 0
while True:
i += 1
print "Iteration #%i" % i
sleep(1)
finally:
print "Goodbye"
Now see the Ctrl+C behaviour:
$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
^CGoodbye
Traceback (most recent call last):
File "./signals-test.py", line 21, in <module>
sleep(1)
KeyboardInterrupt
$ echo $?
1
This time I send it SIGTERM after 4 iterations with kill $(ps aux | grep signals-test | awk '/python/ {print $2}'):
$ ./signals-test.py default
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Terminated
$ echo $?
143
This time I enable my custom SIGTERM handler and send it SIGTERM:
$ ./signals-test.py handle_signal
Hello
Iteration #1
Iteration #2
Iteration #3
Iteration #4
Goodbye
$ echo $?
0

Here is a simple example without threads or classes.
import signal
run = True
def handler_stop_signals(signum, frame):
global run
run = False
signal.signal(signal.SIGINT, handler_stop_signals)
signal.signal(signal.SIGTERM, handler_stop_signals)
while run:
pass # do stuff including other IO stuff

I think you are near to a possible solution.
Execute mainloop in a separate thread and extend it with the property shutdown_flag. The signal can be caught with signal.signal(signal.SIGTERM, handler) in the main thread (not in a separate thread). The signal handler should set shutdown_flag to True and wait for the thread to end with thread.join()

Based on the previous answers, I have created a context manager which protects from sigint and sigterm.
import logging
import signal
import sys
class TerminateProtected:
""" Protect a piece of code from being killed by SIGINT or SIGTERM.
It can still be killed by a force kill.
Example:
with TerminateProtected():
run_func_1()
run_func_2()
Both functions will be executed even if a sigterm or sigkill has been received.
"""
killed = False
def _handler(self, signum, frame):
logging.error("Received SIGINT or SIGTERM! Finishing this block, then exiting.")
self.killed = True
def __enter__(self):
self.old_sigint = signal.signal(signal.SIGINT, self._handler)
self.old_sigterm = signal.signal(signal.SIGTERM, self._handler)
def __exit__(self, type, value, traceback):
if self.killed:
sys.exit(0)
signal.signal(signal.SIGINT, self.old_sigint)
signal.signal(signal.SIGTERM, self.old_sigterm)
if __name__ == '__main__':
print("Try pressing ctrl+c while the sleep is running!")
from time import sleep
with TerminateProtected():
sleep(10)
print("Finished anyway!")
print("This only prints if there was no sigint or sigterm")

Found easiest way for me.
Here an example with fork for clarity that this way is useful for flow control.
import signal
import time
import sys
import os
def handle_exit(sig, frame):
raise(SystemExit)
def main():
time.sleep(120)
signal.signal(signal.SIGTERM, handle_exit)
p = os.fork()
if p == 0:
main()
os._exit()
try:
os.waitpid(p, 0)
except (KeyboardInterrupt, SystemExit):
print('exit handled')
os.kill(p, signal.SIGTERM)
os.waitpid(p, 0)

The simplest solution I have found, taking inspiration by responses above is
class SignalHandler:
def __init__(self):
# register signal handlers
signal.signal(signal.SIGINT, self.exit_gracefully)
signal.signal(signal.SIGTERM, self.exit_gracefully)
self.logger = Logger(level=ERROR)
def exit_gracefully(self, signum, frame):
self.logger.info('captured signal %d' % signum)
traceback.print_stack(frame)
###### do your resources clean up here! ####
raise(SystemExit)

Sample of my code how I use signal:
#! /usr/bin/env python
import signal
def ctrl_handler(signum, frm):
print "You can't cannot kill me"
print "Installing signal handler..."
signal.signal(signal.SIGINT, ctrl_handler)
print "done"
while True:
# do something
pass

You can set a threading.Event when catching the signal.
threading.Event is threadsafe to use and pass around, can be waited on, and the same event can be set and cleared from other places.
import signal, threading
quit_event = threading.Event()
signal.signal(signal.SIGTERM, lambda *_args: quit_event.set())
while not quit_event.is_set():
print("Working...")

Python: Catch Ctrl-C command. Prompt "really want to quit (y/n)", resume execution if no

I have a program that may have a lengthy execution. In the main module I have the following:
import signal
def run_program()
...time consuming execution...
def Exit_gracefully(signal, frame):
... log exiting information ...
... close any open files ...
sys.exit(0)
if __name__ == '__main__':
signal.signal(signal.SIGINT, Exit_gracefully)
run_program()
This works fine, but I'd like the possibility to pause execution upon catching SIGINT, prompting the user if they would really like to quit, and resuming where I left off in run_program() if they decide they don't want to quit.
The only way I can think of doing this is running the program in a separate thread, keeping the main thread waiting on it and ready to catch SIGINT. If the user wants to quit the main thread can do cleanup and kill the child thread.
Is there a simpler way?

The python signal handlers do not seem to be real signal handlers; that is they happen after the fact, in the normal flow and after the C handler has already returned. Thus you'd try to put your quit logic within the signal handler. As the signal handler runs in the main thread, it will block execution there too.
Something like this seems to work nicely.
import signal
import time
import sys
def run_program():
while True:
time.sleep(1)
print("a")
def exit_gracefully(signum, frame):
# restore the original signal handler as otherwise evil things will happen
# in raw_input when CTRL+C is pressed, and our signal handler is not re-entrant
signal.signal(signal.SIGINT, original_sigint)
try:
if raw_input("\nReally quit? (y/n)> ").lower().startswith('y'):
sys.exit(1)
except KeyboardInterrupt:
print("Ok ok, quitting")
sys.exit(1)
# restore the exit gracefully handler here
signal.signal(signal.SIGINT, exit_gracefully)
if __name__ == '__main__':
# store the original SIGINT handler
original_sigint = signal.getsignal(signal.SIGINT)
signal.signal(signal.SIGINT, exit_gracefully)
run_program()
The code restores the original signal handler for the duration of raw_input; raw_input itself is not re-entrable, and re-entering it
will lead to RuntimeError: can't re-enter readline being raised from time.sleep which is something we don't want as it is harder to catch than KeyboardInterrupt. Rather, we let 2 consecutive Ctrl-C's to raise KeyboardInterrupt.

from https://gist.github.com/rtfpessoa/e3b1fe0bbfcd8ac853bf
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
# your code here
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
Bye!

when procedure end then do something
suppose you just want to the procedure will do something after the task end
import time
class TestTask:
def __init__(self, msg: str):
self.msg = msg
def __enter__(self):
print(f'Task Start!:{self.msg}')
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print('Task End!')
#staticmethod
def do_something():
try:
time.sleep(5)
except:
pass
with TestTask('Hello World') as task:
task.do_something()
when the process leaves with that will run __exit__ even with KeyboardInterrupt happen that are same.
if you don't like to see the error, add try ... except ...
#staticmethod
def do_something():
try:
time.sleep(5)
except:
pass
pause, continue, reset, and etc.
I don't have a perfect solution, but it may be useful to you.
It's means divided your process to many subprocesses and save it that finished.it will not be executed again since you find it already done.
import time
from enum import Enum
class Action(Enum):
EXIT = 0
CONTINUE = 1
RESET = 2
class TestTask:
def __init__(self, msg: str):
self.msg = msg
def __enter__(self):
print(f'Task Start!:{self.msg}')
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print('Task End!')
def do_something(self):
tuple_job = (self._foo, self._bar) # implement by yourself
list_job_state = [0] * len(tuple_job)
dict_keep = {} # If there is a need to communicate between jobs, and you don’t want to use class members, you can use this method.
while 1:
try:
for idx, cur_process in enumerate(tuple_job):
if not list_job_state[idx]:
cur_process(dict_keep)
list_job_state[idx] = True
if all(list_job_state):
print('100%')
break
except KeyboardInterrupt:
print('KeyboardInterrupt. input action:')
msg = '\n\t'.join([f"{action + ':':<10}{str(act_number)}" for act_number, action in
enumerate([name for name in vars(Action) if not name.startswith('_')])
])
case = Action(int(input(f'\t{msg}\n:')))
if case == Action.EXIT:
break
if case == Action.RESET:
list_job_state = [0] * len(tuple_job)
#staticmethod
def _foo(keep_dict: dict) -> bool: # implement by yourself
time.sleep(2)
print('1%')
print('2%')
print('...')
print('60%')
keep_dict['status_1'] = 'status_1'
return True
#staticmethod
def _bar(keep_dict: dict) -> bool: # implement by yourself
time.sleep(2)
print('61%')
print(keep_dict.get('status_1'))
print('...')
print('99%')
return True
with TestTask('Hello World') as task:
task.do_something()
console
input action number:2
Task Start!:Hello World
1%
2%
...
60%
KeyboardInterrupt. input action:
EXIT: 0
CONTINUE: 1
RESET: 2
:1
61%
status_1
...
99%
100%
Task End!

How do I capture SIGINT in Python?

I'm working on a python script that starts several processes and database connections. Every now and then I want to kill the script with a Ctrl+C signal, and I'd like to do some cleanup.
In Perl I'd do this:
$SIG{'INT'} = 'exit_gracefully';
sub exit_gracefully {
print "Caught ^C \n";
exit (0);
}
How do I do the analogue of this in Python?

Register your handler with signal.signal like this:
#!/usr/bin/env python
import signal
import sys
def signal_handler(sig, frame):
print('You pressed Ctrl+C!')
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
print('Press Ctrl+C')
signal.pause()
Code adapted from here.
More documentation on signal can be found here.
 

You can treat it like an exception (KeyboardInterrupt), like any other. Make a new file and run it from your shell with the following contents to see what I mean:
import time, sys
x = 1
while True:
try:
print x
time.sleep(.3)
x += 1
except KeyboardInterrupt:
print "Bye"
sys.exit()

And as a context manager:
import signal
class GracefulInterruptHandler(object):
def __init__(self, sig=signal.SIGINT):
self.sig = sig
def __enter__(self):
self.interrupted = False
self.released = False
self.original_handler = signal.getsignal(self.sig)
def handler(signum, frame):
self.release()
self.interrupted = True
signal.signal(self.sig, handler)
return self
def __exit__(self, type, value, tb):
self.release()
def release(self):
if self.released:
return False
signal.signal(self.sig, self.original_handler)
self.released = True
return True
To use:
with GracefulInterruptHandler() as h:
for i in xrange(1000):
print "..."
time.sleep(1)
if h.interrupted:
print "interrupted!"
time.sleep(2)
break
Nested handlers:
with GracefulInterruptHandler() as h1:
while True:
print "(1)..."
time.sleep(1)
with GracefulInterruptHandler() as h2:
while True:
print "\t(2)..."
time.sleep(1)
if h2.interrupted:
print "\t(2) interrupted!"
time.sleep(2)
break
if h1.interrupted:
print "(1) interrupted!"
time.sleep(2)
break
From here: https://gist.github.com/2907502

You can handle CTRL+C by catching the KeyboardInterrupt exception. You can implement any clean-up code in the exception handler.

From Python's documentation:
import signal
import time
def handler(signum, frame):
print 'Here you go'
signal.signal(signal.SIGINT, handler)
time.sleep(10) # Press Ctrl+c here

Yet Another Snippet
Referred main as the main function and exit_gracefully as the Ctrl+C handler
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
pass
finally:
exit_gracefully()

I adapted the code from #udi to support multiple signals (nothing fancy) :
class GracefulInterruptHandler(object):
def __init__(self, signals=(signal.SIGINT, signal.SIGTERM)):
self.signals = signals
self.original_handlers = {}
def __enter__(self):
self.interrupted = False
self.released = False
for sig in self.signals:
self.original_handlers[sig] = signal.getsignal(sig)
signal.signal(sig, self.handler)
return self
def handler(self, signum, frame):
self.release()
self.interrupted = True
def __exit__(self, type, value, tb):
self.release()
def release(self):
if self.released:
return False
for sig in self.signals:
signal.signal(sig, self.original_handlers[sig])
self.released = True
return True
This code support the keyboard interrupt call (SIGINT) and the SIGTERM (kill <process>)

In contrast to Matt J his answer, I use a simple object. This gives me the possibily to parse this handler to all the threads that needs to be stopped securlery.
class SIGINT_handler():
def __init__(self):
self.SIGINT = False
def signal_handler(self, signal, frame):
print('You pressed Ctrl+C!')
self.SIGINT = True
handler = SIGINT_handler()
signal.signal(signal.SIGINT, handler.signal_handler)
Elsewhere
while True:
# task
if handler.SIGINT:
break

If you want to ensure that your cleanup process finishes I would add on to Matt J's answer by using a SIG_IGN so that further SIGINT are ignored which will prevent your cleanup from being interrupted.
import signal
import sys
def signal_handler(signum, frame):
signal.signal(signum, signal.SIG_IGN) # ignore additional signals
cleanup() # give your process a chance to clean up
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler) # register the signal with the signal handler first
do_stuff()

You can use the functions in Python's built-in signal module to set up signal handlers in python. Specifically the signal.signal(signalnum, handler) function is used to register the handler function for signal signalnum.

thanks for existing answers, but added signal.getsignal()
import signal
# store default handler of signal.SIGINT
default_handler = signal.getsignal(signal.SIGINT)
catch_count = 0
def handler(signum, frame):
global default_handler, catch_count
catch_count += 1
print ('wait:', catch_count)
if catch_count > 3:
# recover handler for signal.SIGINT
signal.signal(signal.SIGINT, default_handler)
print('expecting KeyboardInterrupt')
signal.signal(signal.SIGINT, handler)
print('Press Ctrl+c here')
while True:
pass

Personally, I couldn't use try/except KeyboardInterrupt because I was using standard socket (IPC) mode which is blocking. So the SIGINT was cueued, but came only after receiving data on the socket.
Setting a signal handler behaves the same.
On the other hand, this only works for an actual terminal. Other starting environments might not accept Ctrl+C, or pre-handle the signal.
Also, there are "Exceptions" and "BaseExceptions" in Python, which differ in the sense that interpreter needs to exit cleanly itself, so some exceptions have a higher priority than others (Exceptions is derived from BaseException)