Please forgive me if my question's terminology is not quite correct; I am not a professional programmer.
This is the function my Daemon thread is calling:
def _pause_console(self, pause_time):
print("pausing console")
self.javaserver.server_muted = True
print("muted")
time.sleep(pause_time)
print("unmuted")
self.javaserver.server_muted = False
It is called by this code (all in the same class):
pausetime = 30
print("starting pause...")
cm = threading.Thread(target=self._pause_console(pausetime), args=())
cm.daemon = True
cm.start()
print("pause daemon thread started.")
The console output prints my print statements (30 second sleep being specified) in this order:
starting pause
pausing console
muted
unmuted
pause daemon thread started
I was expecting that "pause daemon.." would at least show up before 'unmuted'... I.e., that the main loop execution would continue after the thread was started. However, it appears (to me) that the 'cm.start()' must exit before the 'print("pause daemon thread started.")' statement is executed?
What am I doing wrong or where is my conceptual error here?
(hard to believe no one has encountered this before, but I can't seem to find similar questions on here)
Because your code immediately calls target function while constructing Thread object. You should replace it with following:
cm = threading.Thread(target=self._pause_console, args=(pausetime,))
Related
This seems like a particularly confusing question based on the other similar answers I found on SO. I have code similar to the following:
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
threading.Thread(target = anotherFunction2, args=(var1, var2)).start()
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
threading.Timer(10, childFunction, args=(var1,var2)).start()
print("PRINT #2: Running in child function") # Prints every 10 seconds
# Other code
if (someConditionIsMet):
print("PRINT #3: Exiting")
end_process_and_exit_here()
So basically, when I ran the parentFunction(), I would go into a neverending loop where ever 10 seconds, my console would print "PRINT #2: Running in child function". When the someConditionIsMet was true, my console would print "PRINT #3: Exiting" but then it wouldn't exit. Hence, my loop would carry on forever. I am not sure if it's relevant, but parts of the code has a Threading.Lock as well.
Where I have written end_process_and_exit_here() above, I tried using several methods to kill a thread such as
Raising exceptions and setting flags - These assume that I have started my thread outside of my loop so it's not comparable.
Even this qn about looping threads assumes the thread isnt being looped
Killing using join or stop - stop() was not an option I could access. join() was available but it didn't work i.e. after it was called, the next thread (PRINT #2) continued printing.
Other answers suggesting the use of signals (1) (2), also didn't work.
Using sys.exit() or break in different parts of my code also did not result in the threads stopping.
Is there any method for me to easily exit from such a looping thread?
Note: I need to use threading and not multiprocessing.
You could use python-worker, simply add #worker above you function
pip install python-worker
from worker import worker
#worker
def anotherFunction2(var1,var2):
# your code here
pass
#worker
def parentFunction():
# Other code
while True:
var1, var2 = anotherFunction1() # Getting client details after listening on open port
function2Worker = anotherFunction2(var1,var2) # this will automatically run as thread since you put #worker above your function
childFunction(var1,var2)
print("PRINT #1: Running in Parent Function") # This only prints once for some reason
def childFunction(var1, var2):
parentWorker = parentFunction(var1, var2)
# Other code
if (someConditionIsMet):
parentWorker.abort()
So as an update, I have managed to resolve this issue. The problem with the other answer stated by me (shown below) is that just .cancel() by itself only seemed to work for one timer thread. But as can be seen in the problem, childFunction() itself calls childFunction() and can also be called by the parentFunction, meaning that there may be multiple timer threads.
What worked for my specific case was naming my threads as below:
t1 = threading.Timer(10, childFunction, args=(var1,var2,number))
t1.name = t1.name + "_timer" + str(number)
t1.start()
Thereafter, I could cancel all timer threads that were created from this process by:
for timerthread in threading.enumerate():
if timerthread.name.endswith('timer' + str(number)):
timerthread.cancel()
Below is the ORIGINAL METHOD I USED WHICH CAUSED MANY ISSUES:
I'm not certain if this is a bad practice (in fact I feel it may be based on the answers linked in the question saying that we should never 'kill a thread'). I'm sure there are reasons why this is not good and I'd appreciate anyone telling me why. However, the solution that ultimately worked for me was to use .cancel().
So first change would be to assign your thread Timer to a variable instead of calling it directly. So instead of threading.Timer(10, childFunction, args=(var1,var2)).start(), it should be
t = threading.Timer(10, childFunction, args=(var1,var2))
t.start()
Following that, instead of end_process_and_exit_here(), you should use t.cancel(). This seems to work and stops all threads mid-process. However, the bad thing is that it doesn't seem to carry on with other parts of the program.
I can't get my Python app to exit. After a call to sys.exit(), python.exe stays running and I have to kill it with task manager.
I've spent the past 4 hours looking into this, and I'm stumped.
This is Python 3.4.4 on Windows 10 x86.
First, I do have a multithreaded application. However I have verified that all threads are exiting with only the main thread running before I call sys.exit(). (I did this by calling threading.enumerate() in a while loop and waiting until there's only the main thread remaining, printing the list of running threads and watching it get smaller on each loop until only the main thread remains.)
Also, I've confirmed that I don't have anything wrapped in a try: block that would be swallowing the SystemExit exception. If I print sys.exc_info() I get (None, None, None), and if I call raise then it also confirms there are no exceptions pending.
What's interesting is that I've narrowed this down to the offending thread by commenting out different parts of my app to disable each thread one-by-one. (I have 4 threads total, each doing different things.)
If I comment out the thread in question, I can quit my app no problem. But again, even when I have that thread running, that thread does successfully exit, there's just something in there that's preventing the main Python exe from exiting.
I've tried setting the daemon flag, but that doesn't do anything either way. The offending thread's purpose is to wait at a PriorityQueue() with a 1 second timeout, and then when that times out it checks a threading.Event() flag to exit itself gracefully. Again, that works fine. I can see in my while() loop while the program is exiting that that thread is running, then stops.
The only other information is this application is launched via a console_scripts entry. I've looked at the script file that setuptools creates and see that just wraps the call to my entry point in a sys.exit(), but even hacking that file, I just cannot get this thing to exit.
I've tried calling sys.exit, raising SystemExit, and simply returning to let the console_script call sys.exit. None of those work.
I've also tried more brute force efforts, like os._exit(), but that also doesn't work.
What's really weird is that if I create a recursive loop (a simple one-line method that just calls itself), and I put that in my stop method before I set my threading Event which stops the threads, then Python will exit as it should. (I did that by mistake and first and was dumbfounded that that works. But if I move that loop call down a few lines to just before I call sys.exit, then the recursive loop doesn't kill python.exe. So even though my problem thread exits properly, something about it trying to exit is causing Python.exe to hang.
So, my question, does anyone have any other ideas or things to try about why Python won't exit? Specifically why my problem thread stops and only the main thread remains, yet sys.exit or os._exit() do nothing? I'm completely stumped.
My app consumes about 90MB of memory, and in task manager, I can see the GC doing its job as when my app is "hung" after the sys.exit() call, I see the memory usage drop from 90MB to 0.1MB over the course of about 30 seconds. But even after leaving it, python.exe doesn't stop.
Update: Here's some code that demonstrates what things look like:
From the module and function that's registered as the console_script:
def run_from_command_line(args=None):
path = os.path.abspath(os.path.curdir)
CommandLineUtility(path).execute()
From the CommandLineUtility() which starts my app. This is the last line:
def __init__(...):
... skipping a bunch of setup stuff
MpfMc(options=vars(args), config=mpf_config,
machine_path=machine_path).run() # this is not a threading run, just the name of the method for my app
From MpfMc():
def __init__(...):
...
self.thread_stopper = threading.Event()
...
self.asset_manager = AssetManager(self)
From AssetManager():
self.loader_thread = AssetLoader(loader_queue=self.loader_queue,
loaded_queue=self.loaded_queue,
exception_queue=self.machine.crash_queue,
thread_stopper=self.machine.thread_stopper)
self.loader_thread.daemon = True
self.loader_thread.start()
From AssetLoader:
def run(self):
"""Run loop for the loader thread."""
while True:
try:
asset = self.loader_queue.get(block=True, timeout=1)
except Empty:
asset = None
if self.thread_stopper.is_set():
return
if asset:
if not asset.loaded:
with asset.lock:
asset.do_load()
self.loaded_queue.put(asset)
From the MpfMc.stop() method that stops the app:
def stop(self):
self.log.info("Stopping ...")
self.thread_stopper.set()
while [x for x in self.threads if x.is_alive()]:
# self.threads is a list of threads I created, not the main thread.
print("Waiting for threads to stop")
print([x for x in self.threads if x.is_alive()])
print(threading.enumerate())
time.sleep(0.5)
for thread in self.threads:
# verify none of the sub threads are alive
print("THREAD", thread, thread.is_alive())
sys.exit() # here's where I also tried raise SystemExit, os._exit(), etc
Thanks!
I have a function I'm calling every 5 seconds like such:
def check_buzz(super_buzz_words):
print 'Checking buzz'
t = Timer(5.0, check_buzz, args=(super_buzz_words,))
t.dameon = True
t.start()
buzz_word = get_buzz_word()
if buzz_word is not 'fail':
super_buzz_words.put(buzz_word)
main()
check_buzz()
I'm exiting the script by either catching a KeyboardInterrupt or by catching a System exit and calling this:
sys.exit('\nShutting Down\n')
I'm also restarting the program every so often by calling:
execv(sys.executable, [sys.executable] + sys.argv)
My question is, how do I get that timer thread to shut off? If I keyboard interrupt, the timer keeps going.
I think you just spelled daemon wrong, it should have been:
t.daemon = True
Then sys.exit() should work
Expanding on the answer from notorious.no, and the comment asking:
How can I call t.cancel() if I have no access to t oustide the
function?
Give the Timer thread a distinct name when you first create it:
import threading
def check_buzz(super_buzz_words):
print 'Checking buzz'
t = Timer(5.0, check_buzz, args=(super_buzz_words,))
t.daemon = True
t.name = "check_buzz_daemon"
t.start()
Although the local variable t soon goes out of scope, the Timer thread that t pointed to still exists and still retains the name assigned to it.
Your atexit-registered method can then identify this thread by its name and cancel it:
from atexit import register
def all_done():
for thr in threading._enumerate():
if thr.name == "check_buzz_daemon":
if thr.is_alive():
thr.cancel()
thr.join()
register(all_done)
Calling join() after calling cancel()is based on a StackOverflow answer by Cédric Julien.
HOWEVER, your thread is set to be a Daemon. According to this StackOverflow post, daemon threads do not need to be explicitly terminated.
from atexit import register
def all_done():
if t.is_alive():
# do something that will close your thread gracefully
register(all_done)
Basically when your code is about to exit, it will fire one last function and this is where you will check if your thread is still running. If it is, do something that will either cancel the transaction or otherwise exit gracefully. In general, it's best to let threads finish by themselves, but if it's not doing anything important (please note the emphasis) than you can just do t.cancel(). Design your code so that threads will finish on their own if possible.
Another way would be to use the Queue() module to send and recieve info from a thread using the .put() outside the thread and the .get() inside the thread.
What you can also do is create a txt file and make program write to it when you exit And put an if statement in the thread function to check it after each iteration (this is not a really good solution but it also works)
I would have put a code exemple but i am writing from mobile sorry
I'm writing a multithreaded Python app on Windows.
I used to terminate the app using ctrl-c, but once I added threading.Timer instances ctrl-c stopped working (or sometimes takes a very long time).
How could this be?
What's the relation between having Timer threads and ctrl-c?
UPDATE:
I found the following in Python's thread documentation:
Threads interact strangely with
interrupts: the KeyboardInterrupt
exception will be received by an
arbitrary thread. (When the signal
module is available, interrupts always
go to the main thread.)
The way threading.Thread (and thus threading.Timer) works is that each thread registers itself with the threading module, and upon interpreter exit the interpreter will wait for all registered threads to exit before terminating the interpreter proper. This is done so threads actually finish execution, instead of having the interpreter brutally removed from under them. So when you hit ^C, the main thread receives the signal, decides to terminate and waits for the timers to finish.
You can set threads daemonic (with the setDaemon method) to make the threading module not wait for these threads, but if they happen to be executing Python code while the interpreter exits, you get confusing errors during exit. Even if you cancel the threading.Timer (and set it daemonic) it can still wake up while the interpreter is being destroyed -- because threading.Timer's cancel method just tells the threading.Timer not to execute anything when it wakes up, but it has to actually execute Python code to make that determination.
There is no graceful way to terminate threads (other than the current one), and no reliable way to interrupt a thread that's blocked. A more manageable approach to timers is usually an event loop, like the ones GUIs and other event-driven systems offer you. What to use depends entirely on what else your program will be doing.
There is a presentation by David Beazley that sheds some light on the topic. The PDF is available here. Look around pages 22--25 ("Interlude: Signals" to "Frozen Signals").
This is a possible workaround: using time.sleep() instead of Timer means a "graceful shutdown" mechanism can be implemented ... for Python3 where, it appears, KeyboardInterrupt is only raised in user code for the main thread. Otherwise, it appears, the exception is "ignored" as per here: in fact it results in the thread where it occurs dying immediately, but not any ancestor threads, where problematically it can't be caught.
Let's say you want Ctrl-C responsiveness to be 0.5 seconds, but you only want to repeat some actual work every 5 seconds (work is of random duration as below):
import threading, sys, time, random
blip_counter = 0
work_threads=[]
def repeat_every_5():
global blip_counter
print( f'counter: {blip_counter}')
def real_work():
real_work_duration_s = random.randrange(10)
print( f'do some real work every 5 seconds, lasting {real_work_duration_s} s: starting...')
# in a real world situation stop_event.is_set() can be tested anywhere in the code
for interval_500ms in range( real_work_duration_s * 2 ):
if threading.current_thread().stop_event.is_set():
print( f'stop_event SET!')
return
time.sleep(0.5)
print( f'...real work ends')
# clean up work_threads as appropriate
for work_thread in work_threads:
if not work_thread.is_alive():
print(f'work thread {work_thread} dead, removing from list' )
work_threads.remove( work_thread )
new_work_thread = threading.Thread(target=real_work)
# stop event for graceful shutdown
new_work_thread.stop_event = threading.Event()
work_threads.append(new_work_thread)
# in fact, because a graceful shutdown is now implemented, new_work_thread doesn't have to be daemon
# new_work_thread.daemon = True
new_work_thread.start()
blip_counter += 1
time.sleep( 5 )
timer_thread = threading.Thread(target=repeat_every_5)
timer_thread.daemon = True
timer_thread.start()
repeat_every_5()
while True:
try:
time.sleep( 0.5 )
except KeyboardInterrupt:
print( f'shutting down due to Ctrl-C..., work threads left: {len(work_threads)}')
# trigger stop event for graceful shutdown
for work_thread in work_threads:
if work_thread.is_alive():
print( f'work_thread {work_thread}: setting STOP event')
work_thread.stop_event.set()
print( f'work_thread {work_thread}: joining to main...')
work_thread.join()
print( f'work_thread {work_thread}: ...joined to main')
else:
print( f'work_thread {work_thread} has died' )
sys.exit(1)
This while True: mechanism looks a bit clunky. But I think, as I say, that currently (Python 3.8.x) KeyboardInterrupt can only be caught on the main thread.
PS according to my experiments, handling child processes may be easier, in the sense that Ctrl-C will, it seems, in a simple case at least, cause a KeyboardInterrupt to occur simultaneously in all running processes.
Wrap your main while loop in a try except:
from threading import Timer
import time
def randomfn():
print ("Heartbeat sent!")
class RepeatingTimer(Timer):
def run(self):
while not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.wait(self.interval)
t = RepeatingTimer(10.0, function=randomfn)
print ("Starting...")
t.start()
while (True):
try:
print ("Hello")
time.sleep(1)
except:
print ("Cancelled timer...")
t.cancel()
print ("Cancelled loop...")
break
print ("End")
Results:
Heartbeat sent!
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Hello
Cancelled timer...
Cancelled loop...
End
I'm trying to run the following code (it i simplified a bit):
def RunTests(self):
from threading import Thread
import signal
global keep_running
keep_running = True
signal.signal( signal.SIGINT, stop_running )
for i in range(0, NumThreads):
thread = Thread(target = foo)
self._threads.append(thread)
thread.start()
# wait for all threads to finish
for t in self._threads:
t.join()
def stop_running(signl, frme):
global keep_testing
keep_testing = False
print "Interrupted by the Master. Good by!"
return 0
def foo(self):
global keep_testing
while keep_testing:
DO_SOME_WORK();
I expect that the user presses Ctrl+C the program will print the good by message and interrupt. However it doesn't work. Where is the problem?
Thanks
Unlike regular processes, Python doesn't appear to handle signals in a truly asynchronous manner. The 'join()' call is somehow blocking the main thread in a manner that prevents it from responding to the signal. I'm a bit surprised by this since I don't see anything in the documentation indicating that this can/should happen. The solution, however, is simple. In your main thread, add the following loop prior to calling 'join()' on the threads:
while keep_testing:
signal.pause()
From the threading docs:
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property.
You could try setting thread.daemon = True before calling start() and see if that solves your problem.