I have a code which may run into long loops. These are not infinite loops but certain inputs may cause the loop to continue for extended periods. I wish to use an interrupt in case a loop goes too long. With this interrupt, the loop will re-evaluate its inputs, so a keyboard interrupt is not what I am looking for. I am using python 2.7 on Windows.
One possible approach would be polling using time.time(), yet it would be extremely inefficient. The loop itself is not long, however even in normal runs it is iterated 100k times, so I can't poll the time after each iteration, it would decrease efficiency a lot. Alternatively, I could introduce on more variable,
i=0
while i<some_very_large_number:
function(foo, foo1, foo2)
i=i+1
but this again, would be a mediocre solution.
In short, what I am looking for is some python equivalent to the hardware interrupt of microprocessors. I have no experience in multi-threading, so if the answer lies in multi-threading, please elaborate a little bit.
I have checked here and here, yet I do not think they answer my question. The second link could actually help yet apparently signal package is not available in Windows.
The code is long but straightforward. It basically has such a structure.
def function(foo, foo1, foo2, N):
for i in range(N):
performance = performance_evaluator(foo, foo1, foo2)
if performance_takes_too_long:
interrupt ##this if clause is the main objective of this question##
record performance
new_foo, new_foo1, new_foo2 = evolve(foo, foo1, foo2)
One possible approach is to modify your script so that it takes its inputs via commandline arguments, then use subprocess module to run it with a timeout:
# manager.py
import subprocess
try:
code = subprocess.call('python work.py 5', timeout=2)
print('Ended with code:', code)
except subprocess.TimeoutExpired:
print('Ended with timeout')
# work.py
import sys
from time import sleep
try:
wait = int(sys.argv[1])
except:
wait = 10
sleep(wait)
print(f'Waited for {wait} seconds')
output:
Ended with timeout
You can also execute a long-running code in a separate worker process and try to terminate it when a timeout is exceeded and the worker has not been finished yet:
import time
from multiprocessing import Process
def process_data():
while True:
print("processing...")
time.sleep(1)
def main():
worker = Process(target=process_data)
worker.start()
timeout = 5
time.sleep(timeout)
if worker.is_alive():
print("exceeded timeout", timeout, "sec")
print("terminate worker", worker)
worker.terminate()
worker.join()
print("is worker", worker, "alive:", worker.is_alive())
if __name__ == "__main__":
main()
Here is the output:
processing...
processing...
processing...
processing...
processing...
exceeded timeout 5 sec
terminate worker <Process(Process-1, started)>
is worker <Process(Process-1, stopped[SIGTERM])> alive: False
Related
Is there a way in python to interrupt a thread when it's sleeping?
(As we can do in java)
I am looking for something like that.
import threading
from time import sleep
def f():
print('started')
try:
sleep(100)
print('finished')
except SleepInterruptedException:
print('interrupted')
t = threading.Thread(target=f)
t.start()
if input() == 'stop':
t.interrupt()
The thread is sleeping for 100 seconds and if I type 'stop', it interrupts
The correct approach is to use threading.Event. For example:
import threading
e = threading.Event()
e.wait(timeout=100) # instead of time.sleep(100)
In the other thread, you need to have access to e. You can interrupt the sleep by issuing:
e.set()
This will immediately interrupt the sleep. You can check the return value of e.wait to determine whether it's timed out or interrupted. For more information refer to the documentation: https://docs.python.org/3/library/threading.html#event-objects .
How about using condition objects: https://docs.python.org/2/library/threading.html#condition-objects
Instead of sleep() you use wait(timeout). To "interrupt" you call notify().
If you, for whatever reason, needed to use the time.sleep function and happened to expect the time.sleep function to throw an exception and you simply wanted to test what happened with large sleep values without having to wait for the whole timeout...
Firstly, sleeping threads are lightweight and there's no problem just letting them run in daemon mode with threading.Thread(target=f, daemon=True) (so that they exit when the program does). You can check the result of the thread without waiting for the whole execution with t.join(0.5).
But if you absolutely need to halt the execution of the function, you could use multiprocessing.Process, and call .terminate() on the spawned process. This does not give the process time to clean up (e.g. except and finally blocks aren't run), so use it with care.
I've got the following code which uses a concurrent.futures.ThreadPoolExecutor to launch processes of another program in a metered way (no more than 30 at a time). I additionally want the ability to stop all work if I ctrl-C the python process. This code works with one caveat: I have to ctrl-C twice. The first time I send the SIGINT, nothing happens; the second time, I see the "sending SIGKILL to processes", the processes die, and it works. What is happening to my first SIGINT?
execution_list = [['prog', 'arg1'], ['prog', 'arg2']] ... etc
processes = []
def launch_instance(args):
process = subprocess.Popen(args)
processes.append(process)
process.wait()
try:
with concurrent.futures.ThreadPoolExecutor(max_workers=30) as executor:
results = list(executor.map(launch_instance, execution_list))
except KeyboardInterrupt:
print('sending SIGKILL to processes')
for p in processes:
if p.poll() is None: #If process is still alive
p.send_signal(signal.SIGKILL)
I stumbled upon your question while trying to solve something similar. Not 100% sure that it will solve your use case (I'm not using subprocesses), but I think it will.
Your code will stay within the context manager of the executor as long as the jobs are still running. My educated guess is that the first KeyboardInterrupt will be caught by the ThreadPoolExecutor, whose default behaviour would be to not start any new jobs, wait until the current ones are finished, and then clean up (and probably reraise the KeyboardInterrupt). But the processes are probably long running, so you wouldn't notice. The second KeyboardInterrupt then interrupts this error handling.
How I solved my problem (inifinite background processes in separate threads) is with the following code:
from concurrent.futures import ThreadPoolExecutor
import signal
import threading
from time import sleep
def loop_worker(exiting):
while not exiting.is_set():
try:
print("started work")
sleep(10)
print("finished work")
except KeyboardInterrupt:
print("caught keyboardinterrupt") # never caught here. just for demonstration purposes
def loop_in_worker():
exiting = threading.Event()
def signal_handler(signum, frame):
print("Setting exiting event")
exiting.set()
signal.signal(signal.SIGTERM, signal_handler)
with ThreadPoolExecutor(max_workers=1) as executor:
executor.submit(loop_worker, exiting)
try:
while not exiting.is_set():
sleep(1)
print('waiting')
except KeyboardInterrupt:
print('Caught keyboardinterrupt')
exiting.set()
print("Main thread finished (and thus all others)")
if __name__ == '__main__':
loop_in_worker()
It uses an Event to signal to the threads that they should stop what they are doing. In the main loop, there is a loop just to keep busy and check for any exceptions. Note that this loop is within the context of the ThreadPoolExecutor.
As a bonus it also handles the SIGTERM signal by using the same exiting Event.
If you add a loop in between processes.append(process) and process.wait() that checks for a signal, then it will probably solve your use case as well. It depends on what you want to do with the running processes what actions you should take there.
If you run my script from the command line and press ctrl-C you should see something like:
started work
waiting
waiting
^CCaught keyboardinterrupt
# some time passes here
finished work
Main thread finished (and thus all others)
Inspiration for my solution came from this blog post
What's the best way to kill a function (that is still running) after a given amount of time in Python? These are two approaches I have found so far:
Say this is our base function:
import time
def foo():
a_long_time = 10000000
time.sleep(a_long_time)
TIMEOUT = 5 # seconds
1. Multiprocessing Approach
import multiprocessing
if __name__ == '__main__':
p = multiprocessing.Process(target=foo, name="Foo")
p.start()
p.join(TIMEOUT)
if p.is_alive()
print('function terminated')
p.terminate()
p.join()
2. Signal Approach
import signal
class TimeoutException(Exception):
pass
def timeout_handler(signum, frame):
raise TimeoutException
signal.signal(signal.SIGALRM, timeout_handler)
signal.alarm(TIMEOUT)
try:
foo()
except TimeoutException:
print('function terminated')
What are the advantages and disadvantages in terms of scope, safety and usability of these two methods? Are there any better approaches?
Well, as always, it depends.
As you probably have already verified, both these methods work. I would say it depends on your application and correct implementation (your signalling method is a bit wrong...)
Both methods can be considered "safe" if implemented correctly. It depends if your main program outside the foo function needs to do something, or can it just sit and wait for foo to either complete or timeout. The signalling method does not allow any parallel processing, as your main program will be in foo() until it either completes or times out. BUT you need then to defuse the signal. If your foo completes in one second, your main program leaves the try/except structure, and four seconds later ... kaboom ... an exception is raised and probably uncaught. Not good.
try:
foo()
signal.alarm(0)
except TimeoutException:
print ("function terminated")
solves the problem.
I would personally prefer the multiprocessing approach. It is simpler and does not require signals and exception handling that in theory can go wrong if your program execution is not where you expect it to be when a signal is raised. If it is ok for your program to wait in join(), then you are done. However, if you want to do something in the main process while you wait, you can enter a loop, track time in a variable, check if over timeout and if so, terminate the process. You would just use join with a tiny timeout to "peek" if the process is still running.
Another method, depending on your foo(), is to use threads with a class or a global variable. If your foo keeps processing commands instead of possibly waiting for a long time for a command to finish, you can add an if clause there:
def foo():
global please_exit_now
while True:
do_stuff
do_more_stuff
if foo_is_ready:
break
if please_exit_now is True:
please_exit_now = False
return
finalise_foo
return
If do_stuff and do_more_stuff complete in a reasonable amount of time, you could then process things in your main program and just set global please_exit_now as True, and your thread would eventually notice that and exit.
I would probably just go for your multiprocessing and join, though.
Hannu
Is there a way in python to interrupt a thread when it's sleeping?
(As we can do in java)
I am looking for something like that.
import threading
from time import sleep
def f():
print('started')
try:
sleep(100)
print('finished')
except SleepInterruptedException:
print('interrupted')
t = threading.Thread(target=f)
t.start()
if input() == 'stop':
t.interrupt()
The thread is sleeping for 100 seconds and if I type 'stop', it interrupts
The correct approach is to use threading.Event. For example:
import threading
e = threading.Event()
e.wait(timeout=100) # instead of time.sleep(100)
In the other thread, you need to have access to e. You can interrupt the sleep by issuing:
e.set()
This will immediately interrupt the sleep. You can check the return value of e.wait to determine whether it's timed out or interrupted. For more information refer to the documentation: https://docs.python.org/3/library/threading.html#event-objects .
How about using condition objects: https://docs.python.org/2/library/threading.html#condition-objects
Instead of sleep() you use wait(timeout). To "interrupt" you call notify().
If you, for whatever reason, needed to use the time.sleep function and happened to expect the time.sleep function to throw an exception and you simply wanted to test what happened with large sleep values without having to wait for the whole timeout...
Firstly, sleeping threads are lightweight and there's no problem just letting them run in daemon mode with threading.Thread(target=f, daemon=True) (so that they exit when the program does). You can check the result of the thread without waiting for the whole execution with t.join(0.5).
But if you absolutely need to halt the execution of the function, you could use multiprocessing.Process, and call .terminate() on the spawned process. This does not give the process time to clean up (e.g. except and finally blocks aren't run), so use it with care.
I am writing an queue processing application which uses threads for waiting on and responding to queue messages to be delivered to the app. For the main part of the application, it just needs to stay active. For a code example like:
while True:
pass
or
while True:
time.sleep(1)
Which one will have the least impact on a system? What is the preferred way to do nothing, but keep a python app running?
I would imagine time.sleep() will have less overhead on the system. Using pass will cause the loop to immediately re-evaluate and peg the CPU, whereas using time.sleep will allow the execution to be temporarily suspended.
EDIT: just to prove the point, if you launch the python interpreter and run this:
>>> while True:
... pass
...
You can watch Python start eating up 90-100% CPU instantly, versus:
>>> import time
>>> while True:
... time.sleep(1)
...
Which barely even registers on the Activity Monitor (using OS X here but it should be the same for every platform).
Why sleep? You don't want to sleep, you want to wait for the threads to finish.
So
# store the threads you start in a your_threads list, then
for a_thread in your_threads:
a_thread.join()
See: thread.join
If you are looking for a short, zero-cpu way to loop forever until a KeyboardInterrupt, you can use:
from threading import Event
Event().wait()
Note: Due to a bug, this only works on Python 3.2+. In addition, it appears to not work on Windows. For this reason, while True: sleep(1) might be the better option.
For some background, Event objects are normally used for waiting for long running background tasks to complete:
def do_task():
sleep(10)
print('Task complete.')
event.set()
event = Event()
Thread(do_task).start()
event.wait()
print('Continuing...')
Which prints:
Task complete.
Continuing...
signal.pause() is another solution, see https://docs.python.org/3/library/signal.html#signal.pause
Cause the process to sleep until a signal is received; the appropriate handler will then be called. Returns nothing. Not on Windows. (See the Unix man page signal(2).)
I've always seen/heard that using sleep is the better way to do it. Using sleep will keep your Python interpreter's CPU usage from going wild.
You don't give much context to what you are really doing, but maybe Queue could be used instead of an explicit busy-wait loop? If not, I would assume sleep would be preferable, as I believe it will consume less CPU (as others have already noted).
[Edited according to additional information in comment below.]
Maybe this is obvious, but anyway, what you could do in a case where you are reading information from blocking sockets is to have one thread read from the socket and post suitably formatted messages into a Queue, and then have the rest of your "worker" threads reading from that queue; the workers will then block on reading from the queue without the need for neither pass, nor sleep.
Running a method as a background thread with sleep in Python:
import threading
import time
class ThreadingExample(object):
""" Threading example class
The run() method will be started and it will run in the background
until the application exits.
"""
def __init__(self, interval=1):
""" Constructor
:type interval: int
:param interval: Check interval, in seconds
"""
self.interval = interval
thread = threading.Thread(target=self.run, args=())
thread.daemon = True # Daemonize thread
thread.start() # Start the execution
def run(self):
""" Method that runs forever """
while True:
# Do something
print('Doing something imporant in the background')
time.sleep(self.interval)
example = ThreadingExample()
time.sleep(3)
print('Checkpoint')
time.sleep(2)
print('Bye')