What is the most efficient way (in terms of polling overhead) to request a Python program to stop (in a controlled way) from a Bash script. On python side I want a function (which executes as fast as possible) which returns true when a stop is requested or false if not. If true we save our work, release resources and exit.
For some simple tools I implemented the following:
In bash I do a touch /tmp/stop
My Python program polls on a frequent basis /tmp/stop does exist. If it exists if quits in a controlled way.
My bash script waits (loop - sleep - ps) until the related process is stopped.
This solution works, but polling for this file is most likely not the most efficient way.
Are there other options with less overhead (in terms of Python polling time)?
You could send an interrupt signal (SIGINT) to the python process. That's the same signal your shell would send when you hit Ctrl+C:
Looks like this in Bash:
python my_script.py & # start the script in background
pyscript_pid=$! # store the python interpreter' PID
sleep(5) # pause 5 seconds
kill -s SIGINT $pyscript_pid # send the SIGINT signal to the process
And in Python you simply catch the KeyboardInterrupt exception that gets thrown when the interpreter receives the SIGINT signal:
try:
print ("I'm still running...")
# do something useful, but it must be interruptible at any time!
except KeyboardInterrupt:
print ("I'm going to quit now.")
# tidy up...
# ... and exit
You should not do stuff that would break anything when interrupted half way inside the try block though, only perform stuff that can be interrupted or reset to the last valid state in the tidying up code. Alternatively you might use try: ... finally: ... to ensure code in the finally block will be started always, even if the code in the try gets interrupted while it's running.
You may also look at How do I capture SIGINT in Python? or #Robᵩ's answer to find out how to capture all possible signals and not only SIGINT and how to register event handler for them instead of using try-catch(-finally), but this here would be the simplest approach.
The UNIX signal mechanism would be an excellent choice. You don't need any temporary files, and the polling overhead is essentially zero.
You may shutdown the following python program gracefully like so: kill -USR1 $pid.
import signal
import time
import sys
please_stop = False
def setup_signal():
def handler(x,y):
global please_stop
please_stop = True
signal.signal(signal.SIGUSR1, handler)
def main_task():
for i in range(10):
print "Working hard on iteration #%d"%i
time.sleep(1)
if please_stop:
print "Stopping now"
sys.exit(0)
setup_signal()
main_task()
Related
Is there a way in python to interrupt a thread when it's sleeping?
(As we can do in java)
I am looking for something like that.
import threading
from time import sleep
def f():
print('started')
try:
sleep(100)
print('finished')
except SleepInterruptedException:
print('interrupted')
t = threading.Thread(target=f)
t.start()
if input() == 'stop':
t.interrupt()
The thread is sleeping for 100 seconds and if I type 'stop', it interrupts
The correct approach is to use threading.Event. For example:
import threading
e = threading.Event()
e.wait(timeout=100) # instead of time.sleep(100)
In the other thread, you need to have access to e. You can interrupt the sleep by issuing:
e.set()
This will immediately interrupt the sleep. You can check the return value of e.wait to determine whether it's timed out or interrupted. For more information refer to the documentation: https://docs.python.org/3/library/threading.html#event-objects .
How about using condition objects: https://docs.python.org/2/library/threading.html#condition-objects
Instead of sleep() you use wait(timeout). To "interrupt" you call notify().
If you, for whatever reason, needed to use the time.sleep function and happened to expect the time.sleep function to throw an exception and you simply wanted to test what happened with large sleep values without having to wait for the whole timeout...
Firstly, sleeping threads are lightweight and there's no problem just letting them run in daemon mode with threading.Thread(target=f, daemon=True) (so that they exit when the program does). You can check the result of the thread without waiting for the whole execution with t.join(0.5).
But if you absolutely need to halt the execution of the function, you could use multiprocessing.Process, and call .terminate() on the spawned process. This does not give the process time to clean up (e.g. except and finally blocks aren't run), so use it with care.
Is there a way in python to interrupt a thread when it's sleeping?
(As we can do in java)
I am looking for something like that.
import threading
from time import sleep
def f():
print('started')
try:
sleep(100)
print('finished')
except SleepInterruptedException:
print('interrupted')
t = threading.Thread(target=f)
t.start()
if input() == 'stop':
t.interrupt()
The thread is sleeping for 100 seconds and if I type 'stop', it interrupts
The correct approach is to use threading.Event. For example:
import threading
e = threading.Event()
e.wait(timeout=100) # instead of time.sleep(100)
In the other thread, you need to have access to e. You can interrupt the sleep by issuing:
e.set()
This will immediately interrupt the sleep. You can check the return value of e.wait to determine whether it's timed out or interrupted. For more information refer to the documentation: https://docs.python.org/3/library/threading.html#event-objects .
How about using condition objects: https://docs.python.org/2/library/threading.html#condition-objects
Instead of sleep() you use wait(timeout). To "interrupt" you call notify().
If you, for whatever reason, needed to use the time.sleep function and happened to expect the time.sleep function to throw an exception and you simply wanted to test what happened with large sleep values without having to wait for the whole timeout...
Firstly, sleeping threads are lightweight and there's no problem just letting them run in daemon mode with threading.Thread(target=f, daemon=True) (so that they exit when the program does). You can check the result of the thread without waiting for the whole execution with t.join(0.5).
But if you absolutely need to halt the execution of the function, you could use multiprocessing.Process, and call .terminate() on the spawned process. This does not give the process time to clean up (e.g. except and finally blocks aren't run), so use it with care.
This question already has answers here:
Best way to implement a non-blocking wait?
(5 answers)
Closed 1 year ago.
I need to run my Python program forever in an infinite loop..
Currently I am running it like this -
#!/usr/bin/python
import time
# some python code that I want
# to keep on running
# Is this the right way to run the python program forever?
# And do I even need this time.sleep call?
while True:
time.sleep(5)
Is there any better way of doing it? Or do I even need time.sleep call?
Any thoughts?
Yes, you can use a while True: loop that never breaks to run Python code continually.
However, you will need to put the code you want to run continually inside the loop:
#!/usr/bin/python
while True:
# some python code that I want
# to keep on running
Also, time.sleep is used to suspend the operation of a script for a period of time. So, since you want yours to run continually, I don't see why you would use it.
How about this one?
import signal
signal.pause()
This will let your program sleep until it receives a signal from some other process (or itself, in another thread), letting it know it is time to do something.
I know this is too old thread but why no one mentioned this
#!/usr/bin/python3
import asyncio
loop = asyncio.get_event_loop()
try:
loop.run_forever()
finally:
loop.close()
sleep is a good way to avoid overload on the cpu
not sure if it's really clever, but I usually use
while(not sleep(5)):
#code to execute
sleep method always returns None.
Here is the complete syntax,
#!/usr/bin/python3
import time
def your_function():
print("Hello, World")
while True:
your_function()
time.sleep(10) #make function to sleep for 10 seconds
for OS's that support select:
import select
# your code
select.select([], [], [])
I have a small script interruptableloop.py that runs the code at an interval (default 1sec), it pumps out a message to the screen while it's running, and traps an interrupt signal that you can send with CTL-C:
#!/usr/bin/python3
from interruptableLoop import InterruptableLoop
loop=InterruptableLoop(intervalSecs=1) # redundant argument
while loop.ShouldContinue():
# some python code that I want
# to keep on running
pass
When you run the script and then interrupt it you see this output, (the periods pump out on every pass of the loop):
[py36]$ ./interruptexample.py
CTL-C to stop (or $kill -s SIGINT pid)
......^C
Exiting at 2018-07-28 14:58:40.359331
interruptableLoop.py:
"""
Use to create a permanent loop that can be stopped ...
... from same terminal where process was started and is running in foreground:
CTL-C
... from same user account but through a different terminal
$ kill -2 <pid>
or $ kill -s SIGINT <pid>
"""
import signal
import time
from datetime import datetime as dtt
__all__=["InterruptableLoop",]
class InterruptableLoop:
def __init__(self,intervalSecs=1,printStatus=True):
self.intervalSecs=intervalSecs
self.shouldContinue=True
self.printStatus=printStatus
self.interrupted=False
if self.printStatus:
print ("CTL-C to stop\t(or $kill -s SIGINT pid)")
signal.signal(signal.SIGINT, self._StopRunning)
signal.signal(signal.SIGQUIT, self._Abort)
signal.signal(signal.SIGTERM, self._Abort)
def _StopRunning(self, signal, frame):
self.shouldContinue = False
def _Abort(self, signal, frame):
raise
def ShouldContinue(self):
time.sleep(self.intervalSecs)
if self.shouldContinue and self.printStatus:
print( ".",end="",flush=True)
elif not self.shouldContinue and self.printStatus:
print ("Exiting at ",dtt.now())
return self.shouldContinue
If you mean run as service then you can use any rest framework
from flask import Flask
class A:
def one(port):
app = Flask(__name__)
app.run(port = port)
call it:
one(port=1001)
it will always keep listening on 1001
* Running on http://127.0.0.1:1001/ (Press CTRL+C to quit)
I'm writing a multithreaded Python app on Windows.
I used to terminate the app using ctrl-c, but once I added threading.Timer instances ctrl-c stopped working (or sometimes takes a very long time).
How could this be?
What's the relation between having Timer threads and ctrl-c?
UPDATE:
I found the following in Python's thread documentation:
Threads interact strangely with
interrupts: the KeyboardInterrupt
exception will be received by an
arbitrary thread. (When the signal
module is available, interrupts always
go to the main thread.)
The way threading.Thread (and thus threading.Timer) works is that each thread registers itself with the threading module, and upon interpreter exit the interpreter will wait for all registered threads to exit before terminating the interpreter proper. This is done so threads actually finish execution, instead of having the interpreter brutally removed from under them. So when you hit ^C, the main thread receives the signal, decides to terminate and waits for the timers to finish.
You can set threads daemonic (with the setDaemon method) to make the threading module not wait for these threads, but if they happen to be executing Python code while the interpreter exits, you get confusing errors during exit. Even if you cancel the threading.Timer (and set it daemonic) it can still wake up while the interpreter is being destroyed -- because threading.Timer's cancel method just tells the threading.Timer not to execute anything when it wakes up, but it has to actually execute Python code to make that determination.
There is no graceful way to terminate threads (other than the current one), and no reliable way to interrupt a thread that's blocked. A more manageable approach to timers is usually an event loop, like the ones GUIs and other event-driven systems offer you. What to use depends entirely on what else your program will be doing.
There is a presentation by David Beazley that sheds some light on the topic. The PDF is available here. Look around pages 22--25 ("Interlude: Signals" to "Frozen Signals").
This is a possible workaround: using time.sleep() instead of Timer means a "graceful shutdown" mechanism can be implemented ... for Python3 where, it appears, KeyboardInterrupt is only raised in user code for the main thread. Otherwise, it appears, the exception is "ignored" as per here: in fact it results in the thread where it occurs dying immediately, but not any ancestor threads, where problematically it can't be caught.
Let's say you want Ctrl-C responsiveness to be 0.5 seconds, but you only want to repeat some actual work every 5 seconds (work is of random duration as below):
import threading, sys, time, random
blip_counter = 0
work_threads=[]
def repeat_every_5():
global blip_counter
print( f'counter: {blip_counter}')
def real_work():
real_work_duration_s = random.randrange(10)
print( f'do some real work every 5 seconds, lasting {real_work_duration_s} s: starting...')
# in a real world situation stop_event.is_set() can be tested anywhere in the code
for interval_500ms in range( real_work_duration_s * 2 ):
if threading.current_thread().stop_event.is_set():
print( f'stop_event SET!')
return
time.sleep(0.5)
print( f'...real work ends')
# clean up work_threads as appropriate
for work_thread in work_threads:
if not work_thread.is_alive():
print(f'work thread {work_thread} dead, removing from list' )
work_threads.remove( work_thread )
new_work_thread = threading.Thread(target=real_work)
# stop event for graceful shutdown
new_work_thread.stop_event = threading.Event()
work_threads.append(new_work_thread)
# in fact, because a graceful shutdown is now implemented, new_work_thread doesn't have to be daemon
# new_work_thread.daemon = True
new_work_thread.start()
blip_counter += 1
time.sleep( 5 )
timer_thread = threading.Thread(target=repeat_every_5)
timer_thread.daemon = True
timer_thread.start()
repeat_every_5()
while True:
try:
time.sleep( 0.5 )
except KeyboardInterrupt:
print( f'shutting down due to Ctrl-C..., work threads left: {len(work_threads)}')
# trigger stop event for graceful shutdown
for work_thread in work_threads:
if work_thread.is_alive():
print( f'work_thread {work_thread}: setting STOP event')
work_thread.stop_event.set()
print( f'work_thread {work_thread}: joining to main...')
work_thread.join()
print( f'work_thread {work_thread}: ...joined to main')
else:
print( f'work_thread {work_thread} has died' )
sys.exit(1)
This while True: mechanism looks a bit clunky. But I think, as I say, that currently (Python 3.8.x) KeyboardInterrupt can only be caught on the main thread.
PS according to my experiments, handling child processes may be easier, in the sense that Ctrl-C will, it seems, in a simple case at least, cause a KeyboardInterrupt to occur simultaneously in all running processes.
Wrap your main while loop in a try except:
from threading import Timer
import time
def randomfn():
print ("Heartbeat sent!")
class RepeatingTimer(Timer):
def run(self):
while not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.wait(self.interval)
t = RepeatingTimer(10.0, function=randomfn)
print ("Starting...")
t.start()
while (True):
try:
print ("Hello")
time.sleep(1)
except:
print ("Cancelled timer...")
t.cancel()
print ("Cancelled loop...")
break
print ("End")
Results:
Heartbeat sent!
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Cancelled timer...
Cancelled loop...
End
How can I exit my entire Python application from one of its threads? sys.exit() only terminates the thread in which it is called, so that is no help.
I would not like to use an os.kill() solution, as this isn't very clean.
Short answer: use os._exit.
Long answer with example:
I yanked and slightly modified a simple threading example from a tutorial on DevShed:
import threading, sys, os
theVar = 1
class MyThread ( threading.Thread ):
def run ( self ):
global theVar
print 'This is thread ' + str ( theVar ) + ' speaking.'
print 'Hello and good bye.'
theVar = theVar + 1
if theVar == 4:
#sys.exit(1)
os._exit(1)
print '(done)'
for x in xrange ( 7 ):
MyThread().start()
If you keep sys.exit(1) commented out, the script will die after the third thread prints out. If you use sys.exit(1) and comment out os._exit(1), the third thread does not print (done), and the program runs through all seven threads.
os._exit "should normally only be used in the child process after a fork()" -- and a separate thread is close enough to that for your purpose. Also note that there are several enumerated values listed right after os._exit in that manual page, and you should prefer those as arguments to os._exit instead of simple numbers like I used in the example above.
If all your threads except the main ones are daemons, the best approach is generally thread.interrupt_main() -- any thread can use it to raise a KeyboardInterrupt in the main thread, which can normally lead to reasonably clean exit from the main thread (including finalizers in the main thread getting called, etc).
Of course, if this results in some non-daemon thread keeping the whole process alive, you need to followup with os._exit as Mark recommends -- but I'd see that as the last resort (kind of like a kill -9;-) because it terminates things quite brusquely (finalizers not run, including try/finally blocks, with blocks, atexit functions, etc).
Using thread.interrupt_main() may not help in some situation. KeyboardInterrupts are often used in command line applications to exit the current command or to clean the input line.
In addition, os._exit will kill the process immediately without running any finally blocks in your code, which may be dangerous (files and connections will not be closed for example).
The solution I've found is to register a signal handler in the main thread that raises a custom exception. Use the background thread to fire the signal.
import signal
import os
import threading
import time
class ExitCommand(Exception):
pass
def signal_handler(signal, frame):
raise ExitCommand()
def thread_job():
time.sleep(5)
os.kill(os.getpid(), signal.SIGUSR1)
signal.signal(signal.SIGUSR1, signal_handler)
threading.Thread(target=thread_job).start() # thread will fire in 5 seconds
try:
while True:
user_input = raw_input('Blocked by raw_input loop ')
# do something with 'user_input'
except ExitCommand:
pass
finally:
print('finally will still run')
Related questions:
Why does sys.exit() not exit when called inside a thread in Python?
Python: How to quit CLI when stuck in blocking raw_input?
The easiest way to exit the whole program is, we should terminate the program by using the process id (pid).
import os
import psutil
current_system_pid = os.getpid()
ThisSystem = psutil.Process(current_system_pid)
ThisSystem.terminate()
To install psutl:- "pip install psutil"
For Linux you can use the kill() command and pass the current process' ID and the SIGINT signal to start the steps to exit the app.
import signal
os.kill(os.getpid(), signal.SIGINT)