This is my Job class:
class QueryJob(workerpool.Job):
"Job for downloading a given URL."
def __init__(self, query):
self.query = query # The query we'll need to download when the job runs
def run(self):
try:
// Query something...
except (Exception, KeyboardInterrupt, SystemExit):
# TODO: The KeyboardInterrupt does not seem to work...
print '*** shutting down ***'
pool.shutdown()
pool.wait()
This is how I start it:
# Initialize a pool, 12 threads in this case
pool = workerpool.WorkerPool(size=12)
# Loop over input file and create a job to download the URL on each line
for query in open(options.file):
job = QueryJob(query)
pool.put(job)
If I'd like it to stop before it's finished, I hit Ctrl-C, but nothing happens. I then try Ctrl-C repeatedly also to no avail. Finally, I'll do Ctrl-Z and then find the process id and do a kill -9 to stop all the threads.
Is this the only want to do it? Is there no way to actually catch the KeyboardInterrupt like I'm trying to do above?
Note, I've tried other things in the except like sys.exit() and raise. But it seems like it's not even reaching that point and Ctrl-C has no affect at all once the threads are executing.
Is there something trivial that I'm missing?
Thanks.
I found this:
http://code.activestate.com/recipes/577187-python-thread-pool/
It seems to function just as workerpool does, but actually will listen to the KeyboardInterrupt and halt the script.
This works for me, so I'm answering my own question with it. I'm still up for finding a way to use workerpool, but for anybody else with this same situation - in the meantime I recommend using the Python thread module as is done in the above recipe.
Related
What is the most efficient way (in terms of polling overhead) to request a Python program to stop (in a controlled way) from a Bash script. On python side I want a function (which executes as fast as possible) which returns true when a stop is requested or false if not. If true we save our work, release resources and exit.
For some simple tools I implemented the following:
In bash I do a touch /tmp/stop
My Python program polls on a frequent basis /tmp/stop does exist. If it exists if quits in a controlled way.
My bash script waits (loop - sleep - ps) until the related process is stopped.
This solution works, but polling for this file is most likely not the most efficient way.
Are there other options with less overhead (in terms of Python polling time)?
You could send an interrupt signal (SIGINT) to the python process. That's the same signal your shell would send when you hit Ctrl+C:
Looks like this in Bash:
python my_script.py & # start the script in background
pyscript_pid=$! # store the python interpreter' PID
sleep(5) # pause 5 seconds
kill -s SIGINT $pyscript_pid # send the SIGINT signal to the process
And in Python you simply catch the KeyboardInterrupt exception that gets thrown when the interpreter receives the SIGINT signal:
try:
print ("I'm still running...")
# do something useful, but it must be interruptible at any time!
except KeyboardInterrupt:
print ("I'm going to quit now.")
# tidy up...
# ... and exit
You should not do stuff that would break anything when interrupted half way inside the try block though, only perform stuff that can be interrupted or reset to the last valid state in the tidying up code. Alternatively you might use try: ... finally: ... to ensure code in the finally block will be started always, even if the code in the try gets interrupted while it's running.
You may also look at How do I capture SIGINT in Python? or #Robᵩ's answer to find out how to capture all possible signals and not only SIGINT and how to register event handler for them instead of using try-catch(-finally), but this here would be the simplest approach.
The UNIX signal mechanism would be an excellent choice. You don't need any temporary files, and the polling overhead is essentially zero.
You may shutdown the following python program gracefully like so: kill -USR1 $pid.
import signal
import time
import sys
please_stop = False
def setup_signal():
def handler(x,y):
global please_stop
please_stop = True
signal.signal(signal.SIGUSR1, handler)
def main_task():
for i in range(10):
print "Working hard on iteration #%d"%i
time.sleep(1)
if please_stop:
print "Stopping now"
sys.exit(0)
setup_signal()
main_task()
I can't get my Python app to exit. After a call to sys.exit(), python.exe stays running and I have to kill it with task manager.
I've spent the past 4 hours looking into this, and I'm stumped.
This is Python 3.4.4 on Windows 10 x86.
First, I do have a multithreaded application. However I have verified that all threads are exiting with only the main thread running before I call sys.exit(). (I did this by calling threading.enumerate() in a while loop and waiting until there's only the main thread remaining, printing the list of running threads and watching it get smaller on each loop until only the main thread remains.)
Also, I've confirmed that I don't have anything wrapped in a try: block that would be swallowing the SystemExit exception. If I print sys.exc_info() I get (None, None, None), and if I call raise then it also confirms there are no exceptions pending.
What's interesting is that I've narrowed this down to the offending thread by commenting out different parts of my app to disable each thread one-by-one. (I have 4 threads total, each doing different things.)
If I comment out the thread in question, I can quit my app no problem. But again, even when I have that thread running, that thread does successfully exit, there's just something in there that's preventing the main Python exe from exiting.
I've tried setting the daemon flag, but that doesn't do anything either way. The offending thread's purpose is to wait at a PriorityQueue() with a 1 second timeout, and then when that times out it checks a threading.Event() flag to exit itself gracefully. Again, that works fine. I can see in my while() loop while the program is exiting that that thread is running, then stops.
The only other information is this application is launched via a console_scripts entry. I've looked at the script file that setuptools creates and see that just wraps the call to my entry point in a sys.exit(), but even hacking that file, I just cannot get this thing to exit.
I've tried calling sys.exit, raising SystemExit, and simply returning to let the console_script call sys.exit. None of those work.
I've also tried more brute force efforts, like os._exit(), but that also doesn't work.
What's really weird is that if I create a recursive loop (a simple one-line method that just calls itself), and I put that in my stop method before I set my threading Event which stops the threads, then Python will exit as it should. (I did that by mistake and first and was dumbfounded that that works. But if I move that loop call down a few lines to just before I call sys.exit, then the recursive loop doesn't kill python.exe. So even though my problem thread exits properly, something about it trying to exit is causing Python.exe to hang.
So, my question, does anyone have any other ideas or things to try about why Python won't exit? Specifically why my problem thread stops and only the main thread remains, yet sys.exit or os._exit() do nothing? I'm completely stumped.
My app consumes about 90MB of memory, and in task manager, I can see the GC doing its job as when my app is "hung" after the sys.exit() call, I see the memory usage drop from 90MB to 0.1MB over the course of about 30 seconds. But even after leaving it, python.exe doesn't stop.
Update: Here's some code that demonstrates what things look like:
From the module and function that's registered as the console_script:
def run_from_command_line(args=None):
path = os.path.abspath(os.path.curdir)
CommandLineUtility(path).execute()
From the CommandLineUtility() which starts my app. This is the last line:
def __init__(...):
... skipping a bunch of setup stuff
MpfMc(options=vars(args), config=mpf_config,
machine_path=machine_path).run() # this is not a threading run, just the name of the method for my app
From MpfMc():
def __init__(...):
...
self.thread_stopper = threading.Event()
...
self.asset_manager = AssetManager(self)
From AssetManager():
self.loader_thread = AssetLoader(loader_queue=self.loader_queue,
loaded_queue=self.loaded_queue,
exception_queue=self.machine.crash_queue,
thread_stopper=self.machine.thread_stopper)
self.loader_thread.daemon = True
self.loader_thread.start()
From AssetLoader:
def run(self):
"""Run loop for the loader thread."""
while True:
try:
asset = self.loader_queue.get(block=True, timeout=1)
except Empty:
asset = None
if self.thread_stopper.is_set():
return
if asset:
if not asset.loaded:
with asset.lock:
asset.do_load()
self.loaded_queue.put(asset)
From the MpfMc.stop() method that stops the app:
def stop(self):
self.log.info("Stopping ...")
self.thread_stopper.set()
while [x for x in self.threads if x.is_alive()]:
# self.threads is a list of threads I created, not the main thread.
print("Waiting for threads to stop")
print([x for x in self.threads if x.is_alive()])
print(threading.enumerate())
time.sleep(0.5)
for thread in self.threads:
# verify none of the sub threads are alive
print("THREAD", thread, thread.is_alive())
sys.exit() # here's where I also tried raise SystemExit, os._exit(), etc
Thanks!
I wish to create a non-blocking thread/Greenlit in gevent. The thread is meant to run until some kind of signal is sent to stop it, at which point I wish to perform an action (save some data).
From the gevent kill docs I would expect the following code to print Done upon being killed:
import gevent
def myloop():
try:
while True:
pass
except gevent.greenlet.GreenletExit:
print "Done"
thread = gevent.spawn(myloop)
thread.kill()
However, that doesn't happen. Any ideas as to why? Am I doing something horribly wrong? How can I achieve the specified behaviour?
As your greenlet is never actually "blocking", but consuming CPU forever, it's never killed. From gevent's point of view, it's going to be killed, but this will never happen because the greenlet will never leave its while-loop.
I'm pretty sure that if you add sleep(0) in your while loop, sleep will raise GreenletExit and the loop will break.
In fact, you can deduce this by looking at your code :
try:
while True:
pass
except ...:
...
There is syntaxically nothing that can raise an exception here. We need to do something that can raise, and that's sleep(0).
I looked online and found some SO discussing and ActiveState recipes for running some code with a timeout. It looks there are some common approaches:
Use thread that run the code, and join it with timeout. If timeout elapsed - kill the thread. This is not directly supported in Python (used private _Thread__stop function) so it is bad practice
Use signal.SIGALRM - but this approach not working on Windows!
Use subprocess with timeout - but this is too heavy - what if I want to start interruptible task often, I don't want fire process for each!
So, what is the right way? I'm not asking about workarounds (eg use Twisted and async IO), but actual way to solve actual problem - I have some function and I want to run it only with some timeout. If timeout elapsed, I want control back. And I want it to work on Linux and Windows.
A completely general solution to this really, honestly does not exist. You have to use the right solution for a given domain.
If you want timeouts for code you fully control, you have to write it to cooperate. Such code has to be able to break up into little chunks in some way, as in an event-driven system. You can also do this by threading if you can ensure nothing will hold a lock too long, but handling locks right is actually pretty hard.
If you want timeouts because you're afraid code is out of control (for example, if you're afraid the user will ask your calculator to compute 9**(9**9)), you need to run it in another process. This is the only easy way to sufficiently isolate it. Running it in your event system or even a different thread will not be enough. It is also possible to break things up into little chunks similar to the other solution, but requires very careful handling and usually isn't worth it; in any event, that doesn't allow you to do the same exact thing as just running the Python code.
What you might be looking for is the multiprocessing module. If subprocess is too heavy, then this may not suit your needs either.
import time
import multiprocessing
def do_this_other_thing_that_may_take_too_long(duration):
time.sleep(duration)
return 'done after sleeping {0} seconds.'.format(duration)
pool = multiprocessing.Pool(1)
print 'starting....'
res = pool.apply_async(do_this_other_thing_that_may_take_too_long, [8])
for timeout in range(1, 10):
try:
print '{0}: {1}'.format(duration, res.get(timeout))
except multiprocessing.TimeoutError:
print '{0}: timed out'.format(duration)
print 'end'
If it's network related you could try:
import socket
socket.setdefaulttimeout(number)
I found this with eventlet library:
http://eventlet.net/doc/modules/timeout.html
from eventlet.timeout import Timeout
timeout = Timeout(seconds, exception)
try:
... # execution here is limited by timeout
finally:
timeout.cancel()
For "normal" Python code, that doesn't linger prolongued times in C extensions or I/O waits, you can achieve your goal by setting a trace function with sys.settrace() that aborts the running code when the timeout is reached.
Whether that is sufficient or not depends on how co-operating or malicious the code you run is. If it's well-behaved, a tracing function is sufficient.
An other way is to use faulthandler:
import time
import faulthandler
faulthandler.enable()
try:
faulthandler.dump_tracebacks_later(3)
time.sleep(10)
finally:
faulthandler.cancel_dump_tracebacks_later()
N.B: The faulthandler module is part of stdlib in python3.3.
If you're running code that you expect to die after a set time, then you should write it properly so that there aren't any negative effects on shutdown, no matter if its a thread or a subprocess. A command pattern with undo would be useful here.
So, it really depends on what the thread is doing when you kill it. If its just crunching numbers who cares if you kill it. If its interacting with the filesystem and you kill it , then maybe you should really rethink your strategy.
What is supported in Python when it comes to threads? Daemon threads and joins. Why does python let the main thread exit if you've joined a daemon while its still active? Because its understood that someone using daemon threads will (hopefully) write the code in a way that it wont matter when that thread dies. Giving a timeout to a join and then letting main die, and thus taking any daemon threads with it, is perfectly acceptable in this context.
I've solved that in that way:
For me is worked great (in windows and not heavy at all) I'am hope it was useful for someone)
import threading
import time
class LongFunctionInside(object):
lock_state = threading.Lock()
working = False
def long_function(self, timeout):
self.working = True
timeout_work = threading.Thread(name="thread_name", target=self.work_time, args=(timeout,))
timeout_work.setDaemon(True)
timeout_work.start()
while True: # endless/long work
time.sleep(0.1) # in this rate the CPU is almost not used
if not self.working: # if state is working == true still working
break
self.set_state(True)
def work_time(self, sleep_time): # thread function that just sleeping specified time,
# in wake up it asking if function still working if it does set the secured variable work to false
time.sleep(sleep_time)
if self.working:
self.set_state(False)
def set_state(self, state): # secured state change
while True:
self.lock_state.acquire()
try:
self.working = state
break
finally:
self.lock_state.release()
lw = LongFunctionInside()
lw.long_function(10)
The main idea is to create a thread that will just sleep in parallel to "long work" and in wake up (after timeout) change the secured variable state, the long function checking the secured variable during its work.
I'm pretty new in Python programming, so if that solution has a fundamental errors, like resources, timing, deadlocks problems , please response)).
solving with the 'with' construct and merging solution from -
Timeout function if it takes too long to finish
this thread which work better.
import threading, time
class Exception_TIMEOUT(Exception):
pass
class linwintimeout:
def __init__(self, f, seconds=1.0, error_message='Timeout'):
self.seconds = seconds
self.thread = threading.Thread(target=f)
self.thread.daemon = True
self.error_message = error_message
def handle_timeout(self):
raise Exception_TIMEOUT(self.error_message)
def __enter__(self):
try:
self.thread.start()
self.thread.join(self.seconds)
except Exception, te:
raise te
def __exit__(self, type, value, traceback):
if self.thread.is_alive():
return self.handle_timeout()
def function():
while True:
print "keep printing ...", time.sleep(1)
try:
with linwintimeout(function, seconds=5.0, error_message='exceeded timeout of %s seconds' % 5.0):
pass
except Exception_TIMEOUT, e:
print " attention !! execeeded timeout, giving up ... %s " % e
I have some threads fishing into a queue for jobs, something like this:
class Worker(Thread):
[...]
def run(self):
while not self.terminated:
job = myQueue.get_nowait()
job.dosomething()
sleep(0.5)
Now, self.terminated is just a bool value I use to exit the loop but, this is the problem, several times in a day they stop working without my intervention. All of them but one: the application starts with, lets say, 5 working threads and at random time I check them and one only is working. All the others have both _Thread__initialized and _Thread__stopped fields true. Threads and jobs does not interact with each other. What I should look for?
PS: I understand it's really hard to try to figure out the issue without the actual code, but it's huge.
UPDATE: actually Queue.Empty is the only exception trapped - guess I believed to let all the jobs' internal errors to propagate without kill the threads eheh - so I'm going to block all the exceptions and see...
If that is the actual code it's pretty obvious: myQueue.get_nowait() raises an Exception (Empty) when the queue is empty!
As example, an exception inside the loop will stop the thread.
Why do you use get_nowait() and not get()? What if the Queue is empty?
stackoverflow? :)
I have two suggestions.
1) get_nowait() will raise a Queue.Empty exception if no items are available. Make sure exceptions aren't killing your threads.
2) Use get() instead. Put a None in your queue to signal the thread to exit instead of the boolean flag. Then you don't need a half second sleep and you'll process items faster.
def run(self):
while True:
job = queue.get()
if job:
try:
job.do_something()
except Exception as e:
print e
else: # exit thread when job is None
break
GIL? at one time only an interpreter is doing the job if you want true parallelization you must use multiprocessing see http://docs.python.org/library/multiprocessing.html