I have a python application where thread1 calls an api to see 'what reports are ready to download' and sends that report_id to thread2 which 'downloads/processes those reports. these threads iterate over a dictionary and then wait 5 minutes. Even while code is 'doing nothing' it is pegging on CPU.
I am not sure if the CPU is being pegged in a thread or in main. In main I have some handlers to check for a stop signal so I posted most of that code. I have a few threads that do similar tasks with similar ways they wait at end of loop. Areas that I suspect that code relate to pegging of cpu a)In main - the while run: pass. b)In each of the threads is_killed = self._kill.wait(600) and that being a theading.Event()
Any idea what is pegging cpu.
if __name__ == '__main__':
t2 = ProcessReport()
t2.start()
t1 = RequestReport(t2)
t1.start()
t3 = report_test()
t3.start()
t4 = run_flask()
t4.start()
run = True
signal.signal(signal.SIGINT, handler_stop_signals)
signal.signal(signal.SIGTERM, handler_stop_signals)
signal.signal(signal.SIGHUP, handler_stop_signals)
while run:
pass # Stay here until kill
print("About to kill all threads in clean order")
t3.kill()
t3.join()
t1.kill()
t1.join()
t2.kill()
t2.join()
print("Clean Exit")
sys.exit()
Signal Handler
def handler_stop_signals(signum, frame):
global run
run = False
One of the threads
class report_test(Thread):
def __init__(self):
Thread.__init__(self)
self._kill = threading.Event()
def kill(self):
self._kill.set()
def run(self):
while True:
cursor.execute("SELECT * FROM tbl_rpt_log where cron='1'")
reports_to_run = cursor.fetchall()
for row in reports_to_run:
report_on_row(row)
is_killed = self._kill.wait(600)
if is_killed:
print("Killing - ReportCheckTable")
break
The problem (the main one, at least) is:
while run:
pass # Stay here until kill
This is because the only operation here is evaluating loop's condition and the CPU "can't catch a break".
I didn't look through the whole code, to understand it high level, but the quickest way to work around it is:
import time
# ...
while run:
time.sleep(0.1) # Stay here until kill
Related
I have multiple threads that uses an event object to wait with a timeout. If I wanted to call set() on the event, this would unblock all of the threads. What would be a good way to unblock a specific thread, and leave the other threads in a waiting state?
I've thought about instead of waiting, each thread would have a while loop using a global variable as a condition to signal when the thread should return, however I'm not sure how this could keep the timeout I want for each thread, without checking for timestamps.
import threading
import time
t1 = threading.Thread(target=startTimeout)
t2 = threading.Thread(target=startTimeout)
timeoutEvent = threading.Event()
time.sleep(0.3)
timeoutEvent.set()
# How to have indivial timeoutEvents for specific threads?
def startTimeout():
check = timeoutEvent.wait(1)
if (check):
# set was called
else:
# Timeout
you can create an event for each thread, or a group of threads, just pass it as argument to them or store it somewhere.
import threading
import time
def startTimeout(event):
check = event.wait(1)
if (check):
print('pass')
else:
print("didn't pass")
timeoutEvent1 = threading.Event()
timeoutEvent2 = threading.Event()
t1 = threading.Thread(target=startTimeout,args=(timeoutEvent1,))
t2 = threading.Thread(target=startTimeout,args=(timeoutEvent2,))
t1.start()
t2.start()
time.sleep(0.3)
timeoutEvent1.set()
t1.join()
t2.join()
pass
didn't pass
I'm writing a timer in python. When the timer reaches 0, I want the thread I made to automatically exit.
class Rollgame:
timer = 0
def timerf(self, timer):
self.timer = timer
while self.timer > 0:
time.sleep(0.1)
self.timer -= 0.1
sys.exit(0)
Is this a valid way to exit a thread? It seems to be working in the context of the program im building, however I'm not sure if it's a good way to do it.
If I ever choose to implement this in something like a flask/django app, will this still be valid?
Sorry if the question seems stupid or too simple, I've never worked with threading in python before.
In general, killing threads abruptly is considered a bad programming practice. Killing a thread abruptly might leave a critical resource that must be closed properly, open. But you might want to kill a thread once some specific time period has passed or some interrupt has been generated. There are the various methods by which you can kill a thread in python.
Set/Reset stop flag :
In order to kill a threads, we can declare a stop flag and this flag will be check occasionally by the thread. For Example:
# Python program showing
# how to kill threads
# using set/reset stop
# flag
import threading
import time
def run():
while True:
print('thread running')
global stop_threads
if stop_threads:
break
stop_threads = False
t1 = threading.Thread(target = run)
t1.start()
time.sleep(1)
stop_threads = True
t1.join()
print('thread killed')
In the above code, as soon as the global variable stop_threads is set, the target function run() ends and the thread t1 can be killed by using t1.join(). But one may refrain from using global variable due to certain reasons. For those situations, function objects can be passed to provide a similar functionality as shown below:
# Python program killing
# threads using stop
# flag
import threading
import time
def run(stop):
while True:
print('thread running')
if stop():
break
def main():
stop_threads = False
t1 = threading.Thread(target = run, args =(lambda : stop_threads, ))
t1.start()
time.sleep(1)
stop_threads = True
t1.join()
print('thread killed')
main()
Using traces to kill threads :
This methods works by installing traces in each thread. Each trace terminates itself on the detection of some stimulus or flag, thus instantly killing the associated thread. For Example:
# Python program using
# traces to kill threads
import sys
import trace
import threading
import time
class thread_with_trace(threading.Thread):
def __init__(self, *args, **keywords):
threading.Thread.__init__(self, *args, **keywords)
self.killed = False
def start(self):
self.__run_backup = self.run
self.run = self.__run
threading.Thread.start(self)
def __run(self):
sys.settrace(self.globaltrace)
self.__run_backup()
self.run = self.__run_backup
def globaltrace(self, frame, event, arg):
if event == 'call':
return self.localtrace
else:
return None
def localtrace(self, frame, event, arg):
if self.killed:
if event == 'line':
raise SystemExit()
return self.localtrace
def kill(self):
self.killed = True
def func():
while True:
print('thread running')
t1 = thread_with_trace(target = func)
t1.start()
time.sleep(2)
t1.kill()
t1.join()
if not t1.isAlive():
print('thread killed')
In this code, start() is slightly modified to set the system trace function using settrace(). The local trace function is defined such that, whenever the kill flag (killed) of the respective thread is set, a SystemExit exception is raised upon the excution of the next line of code, which end the execution of the target function func. Now the thread can be killed with join().
Finally, Using the multiprocessing module to kill threads :
The multiprocessing module of Python allows you to spawn processes in the similar way you spawn threads using the threading module. The interface of the multithreading module is similar to that of the threading module. For Example, in a given code we created three threads(processes) which count from 1 to 9. Now, suppose we wanted to terminate all of the threads. You could use multiprocessing to do that.
# Python program killing
# a thread using multiprocessing
# module
import multiprocessing
import time
def func(number):
for i in range(1, 10):
time.sleep(0.01)
print('Processing ' + str(number) + ': prints ' + str(number*i))
# list of all processes, so that they can be killed afterwards
all_processes = []
for i in range(0, 3):
process = multiprocessing.Process(target=func, args=(i,))
process.start()
all_processes.append(process)
# kill all processes after 0.03s
time.sleep(0.03)
for process in all_processes:
process.terminate()
To sum it up, there are many ways to terminate threads, but I peronally wouldn't use sys.exit().
I'm trying to stop the code from running if the user presses ctrl+shift+c. I use the code below. Unfortunately sys.exit() stops only "wait_for_ctrl_shift_c" function, but not "main_func". What should I use to stop them both?
Thanks.
def wait_for_ctrl_shift_c():
print ('wait_for_ctrl_shift_c is working')
keyboard.wait('ctrl+shift+c')
print('wait_for_ctrl_shift_c was pressed!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
sys.exit()
def main_func():
a=0
while True:
print ('Working2 ',a)
a=a+1
sleep(1)
if __name__ == '__main__':
Thread(target = wait_for_ctrl_shift_c).start()
Thread(target = main_func).start()
There are multiple ways to do it. First of all you have 3 threads, one main thread and the other 2 (infinite loop & keyboard one) you create.
You can register signals and handle it, also you can call interrupt_main to interrupt main thread (not the while loop thread). Interrupt will go to main exception handler. Also instead of True i changed the second thread to have an attribute to check if it should run for clean exit.
import os
import threading
import time
import sys
import _thread
def wait_for_ctrl_shift_c():
print ('wait_for_ctrl_shift_c is working')
keyboard.wait('ctrl+shift+c')
print ('exiting thread')
_thread.interrupt_main()
sys.exit()
def main_func():
a=0
t = threading.currentThread()
while getattr(t, "run", True):
print ('Working2 ',a)
a=a+1
time.sleep(1)
print ('exiting main_func')
if __name__ == '__main__':
try:
t1 = threading.Thread(target = wait_for_ctrl_shift_c)
t2 = threading.Thread(target = main_func)
t1.start()
t2.start()
t1.join()
t2.join()
except:
print ('main exiting')
t2.run = False
sys.exit()
Open shell in another window, type ps to list running processes, and kill the Python one (via kill 3145, if 3145 is its PID) to stop them both. This way we kill the process within which these threads run.
I want to kill a thread in python. This thread can run in a blocking operation and join can't terminate it.
Simular to this:
from threading import Thread
import time
def block():
while True:
print("running")
time.sleep(1)
if __name__ == "__main__":
thread = Thread(target = block)
thread.start()
#kill thread
#do other stuff
My problem is that the real blocking operation is in another module that is not from me so there is no place where I can break with a running variable.
The thread will be killed when exiting the main process if you set it up as a daemon:
from threading import Thread
import time
def block():
while True:
print("running")
time.sleep(1)
if __name__ == "__main__":
thread = Thread(target = block, daemon = True)
thread.start()
sys.exit(0)
Otherwise just set a flag, I'm using a bad example (you should use some synchronization not just a plain variable):
from threading import Thread
import time
RUNNING = True
def block():
global RUNNING
while RUNNING:
print("running")
time.sleep(1)
if __name__ == "__main__":
thread = Thread(target = block, daemon = True)
thread.start()
RUNNING = False # thread will stop, not killed until next loop iteration
.... continue your stuff here
Use a running variable:
from threading import Thread
import time
running = True
def block():
global running
while running:
print("running")
time.sleep(1)
if __name__ == "__main__":
thread = Thread(target = block)
thread.start()
running = False
# do other stuff
I would prefer to wrap it all in a class, but this should work (untested though).
EDIT
There is a way to asynchronously raise an exception in a separate thread which could be caught by a try: except: block, but it's a dirty dirty hack: https://gist.github.com/liuw/2407154
Original post
"I want to kill a thread in python." you can't. Threads are only killed when they're daemons when there are no more non-daemonic threads running from the parent process. Any thread can be asked nicely to terminate itself using standard inter-thread communication methods, but you state that you don't have any chance to interrupt the function you want to kill. This leaves processes.
Processes have more overhead, and are more difficult to pass data to and from, but they do support being killed by sending SIGTERM or SIGKILL.
from multiprocessing import Process, Queue
from time import sleep
def workfunction(*args, **kwargs): #any arguments you send to a child process must be picklable by python's pickle module
sleep(args[0]) #really long computation you might want to kill
return 'results' #anything you want to get back from a child process must be picklable by python's pickle module
class daemon_worker(Process):
def __init__(self, target_func, *args, **kwargs):
self.return_queue = Queue()
self.target_func = target_func
self.args = args
self.kwargs = kwargs
super().__init__(daemon=True)
self.start()
def run(self): #called by self.start()
self.return_queue.put(self.target_func(*self.args, **self.kwargs))
def get_result(self): #raises queue.Empty if no result is ready
return self.return_queue.get()
if __name__=='__main__':
#start some work that takes 1 sec:
worker1 = daemon_worker(workfunction, 1)
worker1.join(3) #wait up to 3 sec for the worker to complete
if not worker1.is_alive(): #if we didn't hit 3 sec timeout
print('worker1 got: {}'.format(worker1.get_result()))
else:
print('worker1 still running')
worker1.terminate()
print('killing worker1')
sleep(.1) #calling worker.is_alive() immediately might incur a race condition where it may or may not have shut down yet.
print('worker1 is alive: {}'.format(worker1.is_alive()))
#start some work that takes 100 sec:
worker2 = daemon_worker(workfunction, 100)
worker2.join(3) #wait up to 3 sec for the worker to complete
if not worker2.is_alive(): #if we didn't hit 3 sec timeout
print('worker2 got: {}'.format(worker2.get_result()))
else:
print('worker2 still running')
worker2.terminate()
print('killing worker2')
sleep(.1) #calling worker.is_alive() immediately might incur a race condition where it may or may not have shut down yet.
print('worker2 is alive: {}'.format(worker2.is_alive())
I'm looking to terminate some threads after a certain amount of time. These threads will be running an infinite while loop and during this time they can stall for a random, large amount of time. The thread cannot last longer than time set by the duration variable.
How can I make it so after the length set by duration, the threads stop.
def main():
t1 = threading.Thread(target=thread1, args=1)
t2 = threading.Thread(target=thread2, args=2)
time.sleep(duration)
#the threads must be terminated after this sleep
This will work if you are not blocking.
If you are planing on doing sleeps, its absolutely imperative that you use the event to do the sleep. If you leverage the event to sleep, if someone tells you to stop while "sleeping" it will wake up. If you use time.sleep() your thread will only stop after it wakes up.
import threading
import time
duration = 2
def main():
t1_stop = threading.Event()
t1 = threading.Thread(target=thread1, args=(1, t1_stop))
t2_stop = threading.Event()
t2 = threading.Thread(target=thread2, args=(2, t2_stop))
time.sleep(duration)
# stops thread t2
t2_stop.set()
def thread1(arg1, stop_event):
while not stop_event.is_set():
stop_event.wait(timeout=5)
def thread2(arg1, stop_event):
while not stop_event.is_set():
stop_event.wait(timeout=5)
If you want the threads to stop when your program exits (as implied by your example), then make them daemon threads.
If you want your threads to die on command, then you have to do it by hand. There are various methods, but all involve doing a check in your thread's loop to see if it's time to exit (see Nix's example).
If you want to use a class:
from datetime import datetime,timedelta
class MyThread():
def __init__(self, name, timeLimit):
self.name = name
self.timeLimit = timeLimit
def run(self):
# get the start time
startTime = datetime.now()
while True:
# stop if the time limit is reached :
if((datetime.now()-startTime)>self.timeLimit):
break
print('A')
mt = MyThread('aThread',timedelta(microseconds=20000))
mt.run()
An alternative is to use signal.pthread_kill to send a stop signal. While it's not as robust as #Nix's answer (and I don't think it will work on Windows), it works in cases where Events don't (e.g., stopping a Flask server).
test.py
from signal import pthread_kill, SIGTSTP
from threading import Thread
import time
DURATION = 5
def thread1(arg):
while True:
print(f"processing {arg} from thread1...")
time.sleep(1)
def thread2(arg):
while True:
print(f"processing {arg} from thread2...")
time.sleep(1)
if __name__ == "__main__":
t1 = Thread(target=thread1, args=(1,))
t2 = Thread(target=thread2, args=(2,))
t1.start()
t2.start()
time.sleep(DURATION)
# stops all threads
pthread_kill(t2.ident, SIGTSTP)
result
$ python test.py
processing 1 from thread1...
processing 2 from thread2...
processing 1 from thread1...
processing 2 from thread2...
processing 1 from thread1...
processing 2 from thread2...
processing 1 from thread1...
processing 2 from thread2...
processing 1 from thread1...
processing 2 from thread2...
[19]+ Stopped python test.py