By registering the signal handler, I can put my clean up code in signal_handler
signal.signal(signal.SIGINT, signal_handler)
But the problem is when user presses ctrl+c multiple times, the signal handler run multiple times and the clean up goes crazy.
My problem is that, how can I make sure that the clean up handler is being ran once only before exit.
Here is my trial using lock, but deadlock is occured
def cleanup_handler():
lock.acquire()
if not done:
try:
cleanup()
done = True
finally:
print "release lock"
lock.release()
The problem is that finally block is never ran (that is I can't see the "release lock" being printed).
Remark
After doing a little experiment, it seems that the signaler handler is not being ran at the same time. When new Ctrl+C is received, the old handler is killed and a new handler is ran. Am I correct? If yes, the problem becomes more complicated as I don't want my handler ends in the middle of the....
You could use a decorator that prevents your function from being run more than once:
from functools import wraps
def run_once(function):
#wraps(function)
def wrapper(*args):
if not wrapper.has_run:
wrapper.has_run = True
return function()
wrapper.has_run = False
return wrapper
Like so:
#run_once
def signal_handler():
print 'Exit'
Related
I want to stop my program when the user presses ctrl-C.
The following answer suggests catching the KeyboardInterrupt exception.
python: how to terminate a thread when main program ends
Sometimes it works. But in the following example, it stops working after I increase the number of threads from 25 to 30.
import threading, sys, signal, os
stderr_lock = threading.Lock()
def Log(module, msg):
with stderr_lock:
sys.stderr.write("%s: %s\n" % (module, msg))
class My_Thread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
Log("Init", "Initing.")
self.start()
def run(self):
try:
while True:
Log("Run", "Running.")
except KeyboardInterrupt:
os._exit(0)
for i in range(30):
My_Thread()
# trap ctrl-C in main thread
try:
while True:
pass
except KeyboardInterrupt:
os._exit(0)
This has a very suspiciously similar feel to the following question:
Thread-Safe Signal API in Python 2.7
In that case, I was unable to catch signals after increasing the number of threads beyond 87.
There are actually two different issues with your code that gives this behavior. The first is that your threads should be made into daemon threads, so that they automatically stops when the main thread exits, the second is that your try block does not encapsulate the thread creation and start-up.
When you create a number of threads, the thread creation won't be finished for quite a while (since it is continuously interrupted by the created threads and the GIL prevents them to run in parallel). Therefore, you send your KeyboardInterrupt before being set up to be handled. However, the KeyboardInterrupt will still kill the main thread (with a Traceback), but not the child threads.
Thus, your code works if you modify it as:
import threading, sys, signal, os
stderr_lock = threading.Lock()
def Log(module, msg):
with stderr_lock:
sys.stderr.write("%s: %s\n" % (module, msg))
class My_Thread(threading.Thread):
def __init__(self, value):
threading.Thread.__init__(self)
self.value = value
Log("Init", "Initing %d." % self.value)
self.daemon = True
self.start()
def run(self):
while True:
Log("Run", "Running %d." % self.value)
# trap ctrl-C in main thread
try:
for i in range(1000):
My_Thread(i)
while True:
pass
except KeyboardInterrupt:
os._exit(0)
Note, that making the threads into daemons is not strictly necessary in the current example, but I would consider that to be good practice for threads that are supposed to end when the main program ends.
You may want to read https://stackoverflow.com/a/35430500/1656850, namely:
There are 3 exit functions, in addition to raising SystemExit.
The underlying one is os._exit, which requires 1 int argument, and
exits immediately with no cleanup. It's unlikely you'll ever want to
touch this one, but it is there.
sys.exit is defined in sysmodule.c and just runs
PyErr_SetObject(PyExc_SystemExit, exit_code);, which is effectively
the same as directly raising SystemExit. In fine detail, raising
SystemExit is probably faster, since sys.exit requires an LOAD_ATTR
and CALL_FUNCTION vs RAISE_VARARGS opcalls. Also, raise SystemExit
produces slightly smaller bytecode (4bytes less), (1 byte extra if you
use from sys import exit since sys.exit is expected to return None,
so includes an extra POP_TOP).
The last exit function is defined in site.py, and aliased to exit or
quit in the REPL. It's actually an instance of the Quitter class (so
it can have a custom repr, so is probably the slowest running.
Also, it closes sys.stdin prior to raising SystemExit, so it's
recommended for use only in the REPL.
As for how SystemExit is handled, it eventually causes the VM to call
os._exit, but before that, it does some cleanup. It also runs
atexit._run_exitfuncs() which runs any callbacks registered via the
atexit module. Calling os._exit directly bypasses the atexit step.
so, raise SystemExit may be the preferable way to exit when the exception is caught.
I've got a script that runs an infinite loop and adds things to a database and does things that I can't just stop halfway through, so I can't just press Ctrl+C and stop it.
I want to be able to somehow stop a while loop, but let it finish it's last iteration before it stops.
Let me clarify:
My code looks something like this:
while True:
do something
do more things
do more things
I want to be able to interrupt the while loop at the end, or the beginning, but not between doing things because that would be bad.
And I don't want it to ask me after every iteration if I want to continue.
Thanks for the great answers, I'm super grateful but my implementation doesn't seem to be working:
def signal_handler(signal, frame):
global interrupted
interrupted = True
class Crawler():
def __init__(self):
# not relevant
def crawl(self):
interrupted = False
signal.signal(signal.SIGINT, signal_handler)
while True:
doing things
more things
if interrupted:
print("Exiting..")
break
When I press Ctrl+C the program just keeps going ignoring me.
What you need to do is catch the interrupt, set a flag saying you were interrupted but then continue working until it's time to check the flag (at the end of each loop). Because python's try-except construct will abandon the current run of the loop, you need to set up a proper signal handler; it'll handle the interrupt but then let python continue where it left off. Here's how:
import signal
import time # For the demo only
def signal_handler(signal, frame):
global interrupted
interrupted = True
signal.signal(signal.SIGINT, signal_handler)
interrupted = False
while True:
print("Working hard...")
time.sleep(3)
print("All done!")
if interrupted:
print("Gotta go")
break
Notes:
Use this from the command line. In the IDLE console, it'll trample on IDLE's own interrupt handling.
A better solution would be to "block" KeyboardInterrupt for the duration of the loop, and unblock it when it's time to poll for interrupts. This is a feature of some Unix flavors but not all, hence python does not support it (see the third "General rule")
The OP wants to do this inside a class. But the interrupt function is invoked by the signal handling system, with two arguments: The signal number and a pointer to the stack frame-- no place for a self argument giving access to the class object. Hence the simplest way to set a flag is to use a global variable. You can rig a pointer to the local context by using closures (i.e., define the signal handler dynamically in __init__(), but frankly I wouldn't bother unless a global is out of the question due to multi-threading or whatever.
Caveat: If your process is in the middle of a system call, handling an signal may interrupt the system call. So this may not be safe for all applications. Safer alternatives would be (a) Instead of relying on signals, use a non-blocking read at the end of each loop iteration (and type input instead of hitting ^C); (b) use threads or interprocess communication to isolate the worker from the signal handling; or (c) do the work of implementing real signal blocking, if you are on an OS that has it. All of them are OS-dependent to some extent, so I'll leave it at that.
the below logic will help you do this,
import signal
import sys
import time
run = True
def signal_handler(signal, frame):
global run
print("exiting")
run = False
signal.signal(signal.SIGINT, signal_handler)
while run:
print("hi")
time.sleep(1)
# do anything
print("bye")
while running this, try pressing CTRL + C
To clarify #praba230890's solution: The interrupted variable was not defined in the correct scope. It was defined in the crawl function and the handler could not reach it as a global variable, according to the definition of the handler at the root of the program.
Here is edited example of the principle above. It is the infinitive python loop in a separate thread with the safe signal ending. Also has thread-blocking sleep step - up to you to keep it, replace for asyncio implementation or remove.
This function could be imported to any place in an application, runs without blocking other code (e.g. good for REDIS pusub subscription). After the SIGINT catch the thread job ends peacefully.
from typing import Callable
import time
import threading
import signal
end_job = False
def run_in_loop(job: Callable, interval_sec: int = 0.5):
def interrupt_signal_handler(signal, frame):
global end_job
end_job = True
signal.signal(signal.SIGINT, interrupt_signal_handler)
def do_job():
while True:
job()
time.sleep(interval_sec)
if end_job:
print("Parallel job ending...")
break
th = threading.Thread(target=do_job)
th.start()
You forgot to add global statement in crawl function.
So result will be
import signal
def signal_handler(signal, frame):
global interrupted
interrupted = True
class Crawler():
def __init__(self):
... # or pass if you don't want this to do anything. ... Is for unfinished code
def crawl(self):
global interrupted
interrupted = False
signal.signal(signal.SIGINT, signal_handler)
while True:
# doing things
# more things
if interrupted:
print("Exiting..")
break
I hope below code would help you:
#!/bin/python
import sys
import time
import signal
def cb_sigint_handler(signum, stack):
global is_interrupted
print("SIGINT received")
is_interrupted = True
if __name__ == "__main__":
is_interrupted = False
signal.signal(signal.SIGINT, cb_sigint_handler)
while True:
# do stuff here
print("processing...")
time.sleep(3)
if is_interrupted:
print("Exiting..")
# do clean up
sys.exit(0)
I have a function I'm calling every 5 seconds like such:
def check_buzz(super_buzz_words):
print 'Checking buzz'
t = Timer(5.0, check_buzz, args=(super_buzz_words,))
t.dameon = True
t.start()
buzz_word = get_buzz_word()
if buzz_word is not 'fail':
super_buzz_words.put(buzz_word)
main()
check_buzz()
I'm exiting the script by either catching a KeyboardInterrupt or by catching a System exit and calling this:
sys.exit('\nShutting Down\n')
I'm also restarting the program every so often by calling:
execv(sys.executable, [sys.executable] + sys.argv)
My question is, how do I get that timer thread to shut off? If I keyboard interrupt, the timer keeps going.
I think you just spelled daemon wrong, it should have been:
t.daemon = True
Then sys.exit() should work
Expanding on the answer from notorious.no, and the comment asking:
How can I call t.cancel() if I have no access to t oustide the
function?
Give the Timer thread a distinct name when you first create it:
import threading
def check_buzz(super_buzz_words):
print 'Checking buzz'
t = Timer(5.0, check_buzz, args=(super_buzz_words,))
t.daemon = True
t.name = "check_buzz_daemon"
t.start()
Although the local variable t soon goes out of scope, the Timer thread that t pointed to still exists and still retains the name assigned to it.
Your atexit-registered method can then identify this thread by its name and cancel it:
from atexit import register
def all_done():
for thr in threading._enumerate():
if thr.name == "check_buzz_daemon":
if thr.is_alive():
thr.cancel()
thr.join()
register(all_done)
Calling join() after calling cancel()is based on a StackOverflow answer by Cédric Julien.
HOWEVER, your thread is set to be a Daemon. According to this StackOverflow post, daemon threads do not need to be explicitly terminated.
from atexit import register
def all_done():
if t.is_alive():
# do something that will close your thread gracefully
register(all_done)
Basically when your code is about to exit, it will fire one last function and this is where you will check if your thread is still running. If it is, do something that will either cancel the transaction or otherwise exit gracefully. In general, it's best to let threads finish by themselves, but if it's not doing anything important (please note the emphasis) than you can just do t.cancel(). Design your code so that threads will finish on their own if possible.
Another way would be to use the Queue() module to send and recieve info from a thread using the .put() outside the thread and the .get() inside the thread.
What you can also do is create a txt file and make program write to it when you exit And put an if statement in the thread function to check it after each iteration (this is not a really good solution but it also works)
I would have put a code exemple but i am writing from mobile sorry
I got simple program as below:
import threading
import time
import signal
WITH_DEADLOCK = 0
lock = threading.Lock()
def interruptHandler(signo, frame):
print str(frame), 'received', signo
lock.acquire()
try:
time.sleep(3)
finally:
if WITH_DEADLOCK:
print str(frame), 'release'
lock.release()
signal.signal(signal.SIGINT, interruptHandler)
for x in xrange(60):
print time.strftime("%H:%M:%S"), 'main thread is working'
time.sleep(1)
So, if you start that program and even Ctrl+C is pressed twice within 3 seconds, there is no deadlock. Each time you press Ctrl + C proper line is displayed.
If you change WITH_DEADLOCK=1 and you would press Ctrl+C twice (withing 3 seconds) then program will be hung.
Does anybody may explain why print operation make such difference?
(My python version is 2.6.5)
To be honest I think J.F. Sebastian's comment is the most appropriate answer here - you need to make your signal handler reentrant, which it currently isn't, and it is mostly just surprising that it works anyway without the print statement.
I'm writing a multithreaded Python app on Windows.
I used to terminate the app using ctrl-c, but once I added threading.Timer instances ctrl-c stopped working (or sometimes takes a very long time).
How could this be?
What's the relation between having Timer threads and ctrl-c?
UPDATE:
I found the following in Python's thread documentation:
Threads interact strangely with
interrupts: the KeyboardInterrupt
exception will be received by an
arbitrary thread. (When the signal
module is available, interrupts always
go to the main thread.)
The way threading.Thread (and thus threading.Timer) works is that each thread registers itself with the threading module, and upon interpreter exit the interpreter will wait for all registered threads to exit before terminating the interpreter proper. This is done so threads actually finish execution, instead of having the interpreter brutally removed from under them. So when you hit ^C, the main thread receives the signal, decides to terminate and waits for the timers to finish.
You can set threads daemonic (with the setDaemon method) to make the threading module not wait for these threads, but if they happen to be executing Python code while the interpreter exits, you get confusing errors during exit. Even if you cancel the threading.Timer (and set it daemonic) it can still wake up while the interpreter is being destroyed -- because threading.Timer's cancel method just tells the threading.Timer not to execute anything when it wakes up, but it has to actually execute Python code to make that determination.
There is no graceful way to terminate threads (other than the current one), and no reliable way to interrupt a thread that's blocked. A more manageable approach to timers is usually an event loop, like the ones GUIs and other event-driven systems offer you. What to use depends entirely on what else your program will be doing.
There is a presentation by David Beazley that sheds some light on the topic. The PDF is available here. Look around pages 22--25 ("Interlude: Signals" to "Frozen Signals").
This is a possible workaround: using time.sleep() instead of Timer means a "graceful shutdown" mechanism can be implemented ... for Python3 where, it appears, KeyboardInterrupt is only raised in user code for the main thread. Otherwise, it appears, the exception is "ignored" as per here: in fact it results in the thread where it occurs dying immediately, but not any ancestor threads, where problematically it can't be caught.
Let's say you want Ctrl-C responsiveness to be 0.5 seconds, but you only want to repeat some actual work every 5 seconds (work is of random duration as below):
import threading, sys, time, random
blip_counter = 0
work_threads=[]
def repeat_every_5():
global blip_counter
print( f'counter: {blip_counter}')
def real_work():
real_work_duration_s = random.randrange(10)
print( f'do some real work every 5 seconds, lasting {real_work_duration_s} s: starting...')
# in a real world situation stop_event.is_set() can be tested anywhere in the code
for interval_500ms in range( real_work_duration_s * 2 ):
if threading.current_thread().stop_event.is_set():
print( f'stop_event SET!')
return
time.sleep(0.5)
print( f'...real work ends')
# clean up work_threads as appropriate
for work_thread in work_threads:
if not work_thread.is_alive():
print(f'work thread {work_thread} dead, removing from list' )
work_threads.remove( work_thread )
new_work_thread = threading.Thread(target=real_work)
# stop event for graceful shutdown
new_work_thread.stop_event = threading.Event()
work_threads.append(new_work_thread)
# in fact, because a graceful shutdown is now implemented, new_work_thread doesn't have to be daemon
# new_work_thread.daemon = True
new_work_thread.start()
blip_counter += 1
time.sleep( 5 )
timer_thread = threading.Thread(target=repeat_every_5)
timer_thread.daemon = True
timer_thread.start()
repeat_every_5()
while True:
try:
time.sleep( 0.5 )
except KeyboardInterrupt:
print( f'shutting down due to Ctrl-C..., work threads left: {len(work_threads)}')
# trigger stop event for graceful shutdown
for work_thread in work_threads:
if work_thread.is_alive():
print( f'work_thread {work_thread}: setting STOP event')
work_thread.stop_event.set()
print( f'work_thread {work_thread}: joining to main...')
work_thread.join()
print( f'work_thread {work_thread}: ...joined to main')
else:
print( f'work_thread {work_thread} has died' )
sys.exit(1)
This while True: mechanism looks a bit clunky. But I think, as I say, that currently (Python 3.8.x) KeyboardInterrupt can only be caught on the main thread.
PS according to my experiments, handling child processes may be easier, in the sense that Ctrl-C will, it seems, in a simple case at least, cause a KeyboardInterrupt to occur simultaneously in all running processes.
Wrap your main while loop in a try except:
from threading import Timer
import time
def randomfn():
print ("Heartbeat sent!")
class RepeatingTimer(Timer):
def run(self):
while not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.wait(self.interval)
t = RepeatingTimer(10.0, function=randomfn)
print ("Starting...")
t.start()
while (True):
try:
print ("Hello")
time.sleep(1)
except:
print ("Cancelled timer...")
t.cancel()
print ("Cancelled loop...")
break
print ("End")
Results:
Heartbeat sent!
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Cancelled timer...
Cancelled loop...
End