I want to have a main program that works like a console from where I can call other processes (infinite loops) and kill them selectively whenever certain commands are entered.
For that I created this class:
class RunInThread(threading.Thread):
def __init__(self, function):
self.function = function
self.kill_pill = threading.Event()
threading.Thread.__init__(self)
def start(self): # This is controversial.
self.__init__(self.function)
threading.Thread.start(self)
def stop(self):
self.kill_pill.set()
def run(self):
while not self.kill_pill.is_set():
self.function()
The documentation for thread.Thread says that only the __init__() and run() methods should be overridden.
Is there any clear issue with my code? It works the way I intended but since it's going to be running for long periods of time I need to make sure I'm not creating any memory problems.
EDIT:
What about this solution?:
class StoppableThread(threading.Thread):
# threading.Thread class but can be stopped with the stop() method.
def __init__(self, function):
threading.Thread.__init__(self)
self.function = function
self.kill_pill = threading.Event()
def stop(self):
self.kill_pill.set()
def run(self):
while not self.kill_pill.is_set():
self.function()
class RunInThread():
def __init__(self, function, prnt=False):
self.function = function
self.running = False
self.prnt = prnt
def start(self):
if not self.running:
self.thread = StoppableThread(self.function)
self.thread.start()
self.running = True
else:
if self.prnt:
print('Thread already running.')
def stop(self):
self.thread.stop()
self.running = False
If you want to find out what things that could break, I'd suggest looking into the implementation of Thread class.
Among other things, Thread.__init__() initialises an Event() object to detect thread startup and shutdown, manages cleanup hooks/callbacks, some internal lock objects, and registers the thread to a list so you can introspect running threads. By calling Thread.__init__(), these variables gets reinitialised and screws up the internal mechanisms of many of these functionalities.
What could go wrong? I didn't test any of these, but from skimming through threading.py, these are likely some of the things that I expect could go wrong:
your python process now will be running a OS threads that doesn't show up in enumerate_thread()
multiple OS thread will now return the same Thread object when it calls current_thread(), which will likely also break threadlocal and anything that depends on threadlocal
Thread.join() depends on some internal locks, which likely would now become thread unsafe to call
Unhandled reception can go to the wrong exception hook handler
register_at_fork and shutdown handler likely will get confused
In other words, don't try to be sneaky. Create a new Thread object for each thread you want to start.
There's a good reason that the Thread class spent efforts trying to prevent you from accidentally calling start() twice. Don't try to subvert this.
I'm trying to create a thread, that does stuff in the background. I need to be able to effectively 'pause' it when I need to and 'resume' it again later. Also, if the thread is in the middle of doing something when I 'pause' it, it should make the calling thread wait until it finishes what it's doing.
I'm pretty new to Multithreading in Python, so I haven't gotten all that far.
What I have pretty much does everything except make the calling thread wait if pause is called while my thread is doing something.
Here's the outline of what I'm trying to achieve in code:
import threading, time
class Me(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
#flag to pause thread
self.paused = False
def run(self):
while True:
if not self.paused:
#thread should do the thing if
#not paused
print 'do the thing'
time.sleep(5)
def pause(self):
self.paused = True
#this is should make the calling thread wait if pause() is
#called while the thread is 'doing the thing', until it is
#finished 'doing the thing'
#should just resume the thread
def resume(self):
self.paused = False
I think I basically need a locking mechanism, but within the same thread?
Conditions can be used for this.
Here's an example filling in your skeleton:
class Me(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
#flag to pause thread
self.paused = False
# Explicitly using Lock over RLock since the use of self.paused
# break reentrancy anyway, and I believe using Lock could allow
# one thread to pause the worker, while another resumes; haven't
# checked if Condition imposes additional limitations that would
# prevent that. In Python 2, use of Lock instead of RLock also
# boosts performance.
self.pause_cond = threading.Condition(threading.Lock())
def run(self):
while True:
with self.pause_cond:
while self.paused:
self.pause_cond.wait()
#thread should do the thing if
#not paused
print 'do the thing'
time.sleep(5)
def pause(self):
self.paused = True
# If in sleep, we acquire immediately, otherwise we wait for thread
# to release condition. In race, worker will still see self.paused
# and begin waiting until it's set back to False
self.pause_cond.acquire()
#should just resume the thread
def resume(self):
self.paused = False
# Notify so thread will wake after lock released
self.pause_cond.notify()
# Now release the lock
self.pause_cond.release()
Hope that helps.
Use threading.Event instead of a boolean variable, and add another event for busy state:
def __init__(self):
...
self.can_run = threading.Event()
self.thing_done = threading.Event()
self.thing_done.set()
self.can_run.set()
def run(self):
while True:
self.can_run.wait()
try:
self.thing_done.clear()
print 'do the thing'
finally:
self.thing_done.set()
def pause(self):
self.can_run.clear()
self.thing_done.wait()
def resume(self):
self.can_run.set()
edit: previous answer was wrong, I fixed it and changed variable names to be clear
I want to make a simple server-like program, which can run in loop and read and process messages sent to it. And when I start it like Server().start it obviously runs in loop forever. Is there a way to run it in background and feed it with data, which will be proceeded?
class Server:
def __init__(self):
self.messages = []
self.running = False
def start(self):
self.running = True
self.work()
def send_mess(self, message):
self.messages.append(message)
def handle_mess(self):
for mess in self.messages:
self.do_calculations(mess)
def work(self):
while self.running:
self.handle_mess(self)
self.do_some_important_stuff()
def do_some_important_stuff():
pass
def do_calculations():
pass
Seems like you could use Thread class from the threading module.
It works by inheriting it and redefine run method. Then you issue obj.start() and you'll make start method run in parallel.
Roughly, your class can be define like this (I made some corrections to some methods in order to run)
import threading
class Server(threading.Thread):
def __init__(self):
super(Server, self).__init__()
self.messages = []
self.running = False
def run(self): # changed name start for run
self.running = True
self.work()
def send_mess(self, message):
self.messages.append(message)
def handle_mess(self):
for mess in self.messages:
self.do_calculations(mess)
def work(self):
while self.running:
self.handle_mess()
self.do_some_important_stuff()
def do_some_important_stuff(self):
pass
def do_calculations(self):
pass
s = Server()
s.start() # now is in another another thread running
s.join() # wait for it to finnish
IMPORTANT: Copying #Alfe comment which I found extremely useful:
One MUST point out that by entering the world of concurrency (by threading) one opens a nasty can of worms. OP, you really really should read a little more about concurrency problems which occur in parallel environments. Otherwise you are bound to end with a serious problem sooner or later which you have no clue how to solve. See that you understand Queue.Queue (Queue.queue in Python3) and the things in threading like Events, Locks, Semaphores and what they are good for.
Hope this helps!
An easy way would be:
def start(self):
self.running = True
thread = Thread(target = self.work, args = ())
thread.start()
To start just one background thread (another way is to extend the threading.Thread class).
Or:
def work(self):
while self.running:
message = self.handle_mess(self) # gets a message
def threaded_part(m):
self.do_some_important_stuff(m)
self.do_other_important_stuff(m)
thread = Thread(target = threaded_part, args = (message))
thread.start()
To start a thread for each message you receive. Anyway, with a thread pool it would probably be better.
What's the proper way to tell a looping thread to stop looping?
I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits.
I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up.
Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets)
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
def run(self):
config = controller.getConfig()
fmt = config['timefmt']
start_time = datetime.now().strftime(fmt)
try:
if onlinecheck.check_status(self.asset):
status = "online"
else:
status = "offline"
except socket.gaierror:
status = "an invalid asset tag."
msg =("{}: {} is {}. \n".format(start_time, self.asset, status))
wx.CallAfter(self.window.Logger, msg)
And in my wxPyhton Frame I have this function called from a Start button:
def CheckAsset(self, asset):
self.count += 1
thread = PingAssets(self.count, asset, self)
self.threads.append(thread)
thread.start()
Threaded stoppable function
Instead of subclassing threading.Thread, one can modify the function to allow
stopping by a flag.
We need an object, accessible to running function, to which we set the flag to stop running.
We can use threading.currentThread() object.
import threading
import time
def doit(arg):
t = threading.currentThread()
while getattr(t, "do_run", True):
print ("working on %s" % arg)
time.sleep(1)
print("Stopping as you wish.")
def main():
t = threading.Thread(target=doit, args=("task",))
t.start()
time.sleep(5)
t.do_run = False
if __name__ == "__main__":
main()
The trick is, that the running thread can have attached additional properties. The solution builds
on assumptions:
the thread has a property "do_run" with default value True
driving parent process can assign to started thread the property "do_run" to False.
Running the code, we get following output:
$ python stopthread.py
working on task
working on task
working on task
working on task
working on task
Stopping as you wish.
Pill to kill - using Event
Other alternative is to use threading.Event as function argument. It is by
default False, but external process can "set it" (to True) and function can
learn about it using wait(timeout) function.
We can wait with zero timeout, but we can also use it as the sleeping timer (used below).
def doit(stop_event, arg):
while not stop_event.wait(1):
print ("working on %s" % arg)
print("Stopping as you wish.")
def main():
pill2kill = threading.Event()
t = threading.Thread(target=doit, args=(pill2kill, "task"))
t.start()
time.sleep(5)
pill2kill.set()
t.join()
Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead.
Stopping multiple threads with one pill
Advantage of pill to kill is better seen, if we have to stop multiple threads
at once, as one pill will work for all.
The doit will not change at all, only the main handles the threads a bit differently.
def main():
pill2kill = threading.Event()
tasks = ["task ONE", "task TWO", "task THREE"]
def thread_gen(pill2kill, tasks):
for task in tasks:
t = threading.Thread(target=doit, args=(pill2kill, task))
yield t
threads = list(thread_gen(pill2kill, tasks))
for thread in threads:
thread.start()
time.sleep(5)
pill2kill.set()
for thread in threads:
thread.join()
This has been asked before on Stack. See the following links:
Is there any way to kill a Thread in Python?
Stopping a thread after a certain amount of time
Basically you just need to set up the thread with a stop function that sets a sentinel value that the thread will check. In your case, you'll have the something in your loop check the sentinel value to see if it's changed and if it has, the loop can break and the thread can die.
I read the other questions on Stack but I was still a little confused on communicating across classes. Here is how I approached it:
I use a list to hold all my threads in the __init__ method of my wxFrame class: self.threads = []
As recommended in How to stop a looping thread in Python? I use a signal in my thread class which is set to True when initializing the threading class.
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
self.signal = True
def run(self):
while self.signal:
do_stuff()
sleep()
and I can stop these threads by iterating over my threads:
def OnStop(self, e):
for t in self.threads:
t.signal = False
I had a different approach. I've sub-classed a Thread class and in the constructor I've created an Event object. Then I've written custom join() method, which first sets this event and then calls a parent's version of itself.
Here is my class, I'm using for serial port communication in wxPython app:
import wx, threading, serial, Events, Queue
class PumpThread(threading.Thread):
def __init__ (self, port, queue, parent):
super(PumpThread, self).__init__()
self.port = port
self.queue = queue
self.parent = parent
self.serial = serial.Serial()
self.serial.port = self.port
self.serial.timeout = 0.5
self.serial.baudrate = 9600
self.serial.parity = 'N'
self.stopRequest = threading.Event()
def run (self):
try:
self.serial.open()
except Exception, ex:
print ("[ERROR]\tUnable to open port {}".format(self.port))
print ("[ERROR]\t{}\n\n{}".format(ex.message, ex.traceback))
self.stopRequest.set()
else:
print ("[INFO]\tListening port {}".format(self.port))
self.serial.write("FLOW?\r")
while not self.stopRequest.isSet():
msg = ''
if not self.queue.empty():
try:
command = self.queue.get()
self.serial.write(command)
except Queue.Empty:
continue
while self.serial.inWaiting():
char = self.serial.read(1)
if '\r' in char and len(msg) > 1:
char = ''
#~ print('[DATA]\t{}'.format(msg))
event = Events.PumpDataEvent(Events.SERIALRX, wx.ID_ANY, msg)
wx.PostEvent(self.parent, event)
msg = ''
break
msg += char
self.serial.close()
def join (self, timeout=None):
self.stopRequest.set()
super(PumpThread, self).join(timeout)
def SetPort (self, serial):
self.serial = serial
def Write (self, msg):
if self.serial.is_open:
self.queue.put(msg)
else:
print("[ERROR]\tPort {} is not open!".format(self.port))
def Stop(self):
if self.isAlive():
self.join()
The Queue is used for sending messages to the port and main loop takes responses back. I've used no serial.readline() method, because of different end-line char, and I have found the usage of io classes to be too much fuss.
Depends on what you run in that thread.
If that's your code, then you can implement a stop condition (see other answers).
However, if what you want is to run someone else's code, then you should fork and start a process. Like this:
import multiprocessing
proc = multiprocessing.Process(target=your_proc_function, args=())
proc.start()
now, whenever you want to stop that process, send it a SIGTERM like this:
proc.terminate()
proc.join()
And it's not slow: fractions of a second.
Enjoy :)
My solution is:
import threading, time
def a():
t = threading.currentThread()
while getattr(t, "do_run", True):
print('Do something')
time.sleep(1)
def getThreadByName(name):
threads = threading.enumerate() #Threads list
for thread in threads:
if thread.name == name:
return thread
threading.Thread(target=a, name='228').start() #Init thread
t = getThreadByName('228') #Get thread by name
time.sleep(5)
t.do_run = False #Signal to stop thread
t.join()
I find it useful to have a class, derived from threading.Thread, to encapsulate my thread functionality. You simply provide your own main loop in an overridden version of run() in this class. Calling start() arranges for the object’s run() method to be invoked in a separate thread.
Inside the main loop, periodically check whether a threading.Event has been set. Such an event is thread-safe.
Inside this class, you have your own join() method that sets the stop event object before calling the join() method of the base class. It can optionally take a time value to pass to the base class's join() method to ensure your thread is terminated in a short amount of time.
import threading
import time
class MyThread(threading.Thread):
def __init__(self, sleep_time=0.1):
self._stop_event = threading.Event()
self._sleep_time = sleep_time
"""call base class constructor"""
super().__init__()
def run(self):
"""main control loop"""
while not self._stop_event.isSet():
#do work
print("hi")
self._stop_event.wait(self._sleep_time)
def join(self, timeout=None):
"""set stop event and join within a given time period"""
self._stop_event.set()
super().join(timeout)
if __name__ == "__main__":
t = MyThread()
t.start()
time.sleep(5)
t.join(1) #wait 1s max
Having a small sleep inside the main loop before checking the threading.Event is less CPU intensive than looping continuously. You can have a default sleep time (e.g. 0.1s), but you can also pass the value in the constructor.
Sometimes you don't have control over the running target. In those cases you can use signal.pthread_kill to send a stop signal.
from signal import pthread_kill, SIGTSTP
from threading import Thread
from itertools import count
from time import sleep
def target():
for num in count():
print(num)
sleep(1)
thread = Thread(target=target)
thread.start()
sleep(5)
pthread_kill(thread.ident, SIGTSTP)
result
0
1
2
3
4
[14]+ Stopped
I want to force threads termination in python: I don't want to set an event and wait until the thread checks it and exits. I'm looking for a simple solution like kill -9. Is this possible to do that without dirty hacks like operating with private methods etc.?
If you do not mind your code running about ten times slower, you can use the Thread2 class implemented below. An example follows that shows how calling the new stop method should kill the thread on the next bytecode instruction.
import threading
import sys
class StopThread(StopIteration): pass
threading.SystemExit = SystemExit, StopThread
class Thread2(threading.Thread):
def stop(self):
self.__stop = True
def _bootstrap(self):
if threading._trace_hook is not None:
raise ValueError('Cannot run thread with tracing!')
self.__stop = False
sys.settrace(self.__trace)
super()._bootstrap()
def __trace(self, frame, event, arg):
if self.__stop:
raise StopThread()
return self.__trace
class Thread3(threading.Thread):
def _bootstrap(self, stop_thread=False):
def stop():
nonlocal stop_thread
stop_thread = True
self.stop = stop
def tracer(*_):
if stop_thread:
raise StopThread()
return tracer
sys.settrace(tracer)
super()._bootstrap()
################################################################################
import time
def main():
test = Thread2(target=printer)
test.start()
time.sleep(1)
test.stop()
test.join()
def printer():
while True:
print(time.time() % 1)
time.sleep(0.1)
if __name__ == '__main__':
main()
The Thread3 class appears to run code approximately 33% faster than the Thread2 class.
Threads end when they do.
You can signal a thread that you want it to terminate ASAP, but that assumes collaboration of the code running in a thread, and it offers no upper bound guarantee for when that happens.
A classic way is to use a variable like exit_immediately = False and have threads' main routines periodically check it and terminate if the value is True. To have the threads exit, you set exit_immediately = True and call .join() on all threads. Obviously, this works only when threads are able to check in periodically.
If what you want is to just be able to let the program terminate at its end without caring about what happens to some threads, what you want is daemon threads.
From the docs:
The entire Python program exits when no alive non-daemon threads are
left.
Example usage program:
import threading
import time
def test():
while True:
print "hey"
time.sleep(1)
t = threading.Thread(target=test)
t.daemon = True # <-- set to False and the program will not terminate
t.start()
time.sleep(10)
Trivia: daemon threads are referred to as background threads in .Net.