i have a question about Timer object in Python:
https://docs.python.org/2/library/threading.html#timer-objects
This object is basically a Thread that calls a specific function after an amount of user defined seconds. My question is: after the function call, the Thread object is automatically eliminated of the program, or this object keeps running, consuming memory?
This snippet of code shows what i'm talking about:
from threading import Timer
from random import randint
def call_function(x):
print "Timer number " + str(x) + " hit call function!"
for x in range(200000):
#memory used: 700.000 K
print "Creating timer number: " + str(x)
Timer(randint(1, 10), call_function, [x]).start()
After 200.000 threads Timer created and being called in a space of 10 seconds (after this call they were supposed to be dead, releasing space, right?), the program end it's execution with 700.000 K of memory consumption, almost 1 GB.
Thanks,
Timer is just a Thread subclass (created by factory function named Timer) with simple run method:
class _Timer(Thread):
def __init__(self, interval, function, args=[], kwargs={}):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args
self.kwargs = kwargs
self.finished = Event()
def cancel(self):
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
thread terminates just after the run() method terminates:
Once the thread’s activity is started, the thread is considered
‘alive’. It stops being alive when its run() method terminates –
either normally, or by raising an unhandled exception. The is_alive()
method tests whether the thread is alive.
timer class object itself should be garbage collected automatically if it will not be used later.
And you cant start 200000 threads, its too much (_Timer object will be created but thread will not start). Check this question: How many threads is too many?
Related
I want to have a main program that works like a console from where I can call other processes (infinite loops) and kill them selectively whenever certain commands are entered.
For that I created this class:
class RunInThread(threading.Thread):
def __init__(self, function):
self.function = function
self.kill_pill = threading.Event()
threading.Thread.__init__(self)
def start(self): # This is controversial.
self.__init__(self.function)
threading.Thread.start(self)
def stop(self):
self.kill_pill.set()
def run(self):
while not self.kill_pill.is_set():
self.function()
The documentation for thread.Thread says that only the __init__() and run() methods should be overridden.
Is there any clear issue with my code? It works the way I intended but since it's going to be running for long periods of time I need to make sure I'm not creating any memory problems.
EDIT:
What about this solution?:
class StoppableThread(threading.Thread):
# threading.Thread class but can be stopped with the stop() method.
def __init__(self, function):
threading.Thread.__init__(self)
self.function = function
self.kill_pill = threading.Event()
def stop(self):
self.kill_pill.set()
def run(self):
while not self.kill_pill.is_set():
self.function()
class RunInThread():
def __init__(self, function, prnt=False):
self.function = function
self.running = False
self.prnt = prnt
def start(self):
if not self.running:
self.thread = StoppableThread(self.function)
self.thread.start()
self.running = True
else:
if self.prnt:
print('Thread already running.')
def stop(self):
self.thread.stop()
self.running = False
If you want to find out what things that could break, I'd suggest looking into the implementation of Thread class.
Among other things, Thread.__init__() initialises an Event() object to detect thread startup and shutdown, manages cleanup hooks/callbacks, some internal lock objects, and registers the thread to a list so you can introspect running threads. By calling Thread.__init__(), these variables gets reinitialised and screws up the internal mechanisms of many of these functionalities.
What could go wrong? I didn't test any of these, but from skimming through threading.py, these are likely some of the things that I expect could go wrong:
your python process now will be running a OS threads that doesn't show up in enumerate_thread()
multiple OS thread will now return the same Thread object when it calls current_thread(), which will likely also break threadlocal and anything that depends on threadlocal
Thread.join() depends on some internal locks, which likely would now become thread unsafe to call
Unhandled reception can go to the wrong exception hook handler
register_at_fork and shutdown handler likely will get confused
In other words, don't try to be sneaky. Create a new Thread object for each thread you want to start.
There's a good reason that the Thread class spent efforts trying to prevent you from accidentally calling start() twice. Don't try to subvert this.
I'm using the threading.Timer package to execute a method after x seconds. However, in some cases I want to execute this method earlier and cancel the timer (so it isn't called twice). How do I unit test this?
I want to know if the timer has stopped so that the method is not called anymore. I am now using the following code, unfortunately the is_alive still returns True
from threading import Timer
Class X():
def __init__(self, timeout):
self.timer = Timer(timeout, self.some_method)
self.timer.start()
def some_method(self):
# Do something
def other_method(self):
self.timer.cancel()
self.some_method()
import unittest
Class TestX(unittest.TestCase):
def test_cancel_timer(self):
x = X(1000)
x.other_method()
self.assertFalse(x.timer.is_alive())
Form the documentation the is_alive method returns True during the run operation;
Return whether the thread is alive.
This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads.
The documentation on the cancel method says the following;
Stop the timer, and cancel the execution of the timer’s action. This will only work if the timer is still in its waiting stage.
Does this mean that the cancel method does not stop the run action? Or is is still in the grey area after the run method and returns True for that reason?
With timer.is_alive() you are just checking if the timer-thread itself is alive, so if you want to "check if timer.cancel() was called", you're testing for the wrong thing.
Does this mean that the cancel method does not stop the run action?
It does not stop the run()-function, right. timer.cancel() just sets a flag in an Event-object which gets checked by run. You can test if the flag is set with:
self.assertTrue(x.timer.finished.is_set())
Unfortunately, checking for cancellation is not enough to prevent repeated execution, since run can have already crossed the check like you can see in the source code:
# threading.py (Python 3.7.1):
class Timer(Thread):
"""Call a function after a specified number of seconds:
t = Timer(30.0, f, args=None, kwargs=None)
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
def __init__(self, interval, function, args=None, kwargs=None):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args if args is not None else []
self.kwargs = kwargs if kwargs is not None else {}
self.finished = Event()
def cancel(self):
"""Stop the timer if it hasn't finished yet."""
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
Some more effort is needed to ensure unique execution. I've written up a possible solution to this in my answer here.
I want to know how python threading.Timer works.
In more detail, When i run a couple of threading.Timer, does it run separate thread for counting a time and running the handler ?
Or one thread manages and counts a couple of timer together ?
I am asking because my application need to schedule many event, But
If threading.Timer runs separate each thread for counting a timer, and i run many timers, it may affect performance so much.
So i am worry that if i have to implement a scheduler running only one thread if it has big effect in performance.
threading.Timer class is a subclass of threading.Thread and basically it just runs a separate thread in which it sleeps for the specified amount of time and runs the corresponding function.
It is definitely not an efficient way to schedule events. Better way is to do the scheduling in a single thread by using Queue.PriorityQueue in which you would put your events where "priority" actually means "next fire date". Similar to how cron works.
Or even better: use something that already exists, do not reinvent the wheel: Cron, Celery, whatever...
A very simplified example of making a scheduler via Queue.PriorityQueue:
import time
from Queue import PriorityQueue
class Task(object):
def __init__(self, fn, crontab):
# TODO: it should be possible to pass args, kwargs
# so that fn can be called with fn(*args, **kwargs)
self.fn = fn
self.crontab = crontab
def get_next_fire_date(self):
# TODO: evaluate next fire date based on self.crontab
pass
class Scheduler(object):
def __init__(self):
self.event_queue = PriorityQueue()
self.new_task = False
def schedule_task(self, fn, crontab):
# TODO: add scheduling language, crontab or something
task = Task(fn, crontab)
next_fire = task.get_next_fire_date()
if next_fire:
self.new_task = True
self.event_queue.put((next_fire, task))
def run(self):
self.new_task = False
# TODO: do we really want an infinite loop?
while True:
# TODO: actually we want .get() with timeout and to handle
# the case when the queue is empty
next_fire, task = self.event_queue.get()
# incremental sleep so that we can check
# if new tasks arrived in the meantime
sleep_for = int(next_fire - time.time())
for _ in xrange(sleep_for):
time.sleep(1)
if self.new_task:
self.new_task = False
self.event_queue.put((next_fire, task))
continue
# TODO: run in separate thread?
task.fn()
time.sleep(1)
next_fire = task.get_next_fire_date()
if next_fire:
event_queue.put((next_fire, task))
def test():
return 'hello world'
sch = Scheduler()
sch.schedule_task(test, '5 * * * *')
sch.schedule_task(test, '0 22 * * 1-5')
sch.schedule_task(test, '1 1 * * *')
sch.run()
It's just an idea. You would have to properly implement both Task and Scheduler classes, i.e. get_next_fire_date method plus some kind of scheduling language (crontab?) and error handling. I still strongly suggest to use one of the existing libraries.
From the CPython 2.7 source:
def Timer(*args, **kwargs):
"""Factory function to create a Timer object.
Timers call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
return _Timer(*args, **kwargs)
class _Timer(Thread):
"""Call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
def __init__(self, interval, function, args=[], kwargs={}):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args
self.kwargs = kwargs
self.finished = Event()
def cancel(self):
"""Stop the timer if it hasn't finished yet"""
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
As said in another answer, it is a separate thread (since it subclasses Thread). The callback function when the timer runs out is called from the new thread.
What's the proper way to tell a looping thread to stop looping?
I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits.
I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up.
Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets)
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
def run(self):
config = controller.getConfig()
fmt = config['timefmt']
start_time = datetime.now().strftime(fmt)
try:
if onlinecheck.check_status(self.asset):
status = "online"
else:
status = "offline"
except socket.gaierror:
status = "an invalid asset tag."
msg =("{}: {} is {}. \n".format(start_time, self.asset, status))
wx.CallAfter(self.window.Logger, msg)
And in my wxPyhton Frame I have this function called from a Start button:
def CheckAsset(self, asset):
self.count += 1
thread = PingAssets(self.count, asset, self)
self.threads.append(thread)
thread.start()
Threaded stoppable function
Instead of subclassing threading.Thread, one can modify the function to allow
stopping by a flag.
We need an object, accessible to running function, to which we set the flag to stop running.
We can use threading.currentThread() object.
import threading
import time
def doit(arg):
t = threading.currentThread()
while getattr(t, "do_run", True):
print ("working on %s" % arg)
time.sleep(1)
print("Stopping as you wish.")
def main():
t = threading.Thread(target=doit, args=("task",))
t.start()
time.sleep(5)
t.do_run = False
if __name__ == "__main__":
main()
The trick is, that the running thread can have attached additional properties. The solution builds
on assumptions:
the thread has a property "do_run" with default value True
driving parent process can assign to started thread the property "do_run" to False.
Running the code, we get following output:
$ python stopthread.py
working on task
working on task
working on task
working on task
working on task
Stopping as you wish.
Pill to kill - using Event
Other alternative is to use threading.Event as function argument. It is by
default False, but external process can "set it" (to True) and function can
learn about it using wait(timeout) function.
We can wait with zero timeout, but we can also use it as the sleeping timer (used below).
def doit(stop_event, arg):
while not stop_event.wait(1):
print ("working on %s" % arg)
print("Stopping as you wish.")
def main():
pill2kill = threading.Event()
t = threading.Thread(target=doit, args=(pill2kill, "task"))
t.start()
time.sleep(5)
pill2kill.set()
t.join()
Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead.
Stopping multiple threads with one pill
Advantage of pill to kill is better seen, if we have to stop multiple threads
at once, as one pill will work for all.
The doit will not change at all, only the main handles the threads a bit differently.
def main():
pill2kill = threading.Event()
tasks = ["task ONE", "task TWO", "task THREE"]
def thread_gen(pill2kill, tasks):
for task in tasks:
t = threading.Thread(target=doit, args=(pill2kill, task))
yield t
threads = list(thread_gen(pill2kill, tasks))
for thread in threads:
thread.start()
time.sleep(5)
pill2kill.set()
for thread in threads:
thread.join()
This has been asked before on Stack. See the following links:
Is there any way to kill a Thread in Python?
Stopping a thread after a certain amount of time
Basically you just need to set up the thread with a stop function that sets a sentinel value that the thread will check. In your case, you'll have the something in your loop check the sentinel value to see if it's changed and if it has, the loop can break and the thread can die.
I read the other questions on Stack but I was still a little confused on communicating across classes. Here is how I approached it:
I use a list to hold all my threads in the __init__ method of my wxFrame class: self.threads = []
As recommended in How to stop a looping thread in Python? I use a signal in my thread class which is set to True when initializing the threading class.
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
self.signal = True
def run(self):
while self.signal:
do_stuff()
sleep()
and I can stop these threads by iterating over my threads:
def OnStop(self, e):
for t in self.threads:
t.signal = False
I had a different approach. I've sub-classed a Thread class and in the constructor I've created an Event object. Then I've written custom join() method, which first sets this event and then calls a parent's version of itself.
Here is my class, I'm using for serial port communication in wxPython app:
import wx, threading, serial, Events, Queue
class PumpThread(threading.Thread):
def __init__ (self, port, queue, parent):
super(PumpThread, self).__init__()
self.port = port
self.queue = queue
self.parent = parent
self.serial = serial.Serial()
self.serial.port = self.port
self.serial.timeout = 0.5
self.serial.baudrate = 9600
self.serial.parity = 'N'
self.stopRequest = threading.Event()
def run (self):
try:
self.serial.open()
except Exception, ex:
print ("[ERROR]\tUnable to open port {}".format(self.port))
print ("[ERROR]\t{}\n\n{}".format(ex.message, ex.traceback))
self.stopRequest.set()
else:
print ("[INFO]\tListening port {}".format(self.port))
self.serial.write("FLOW?\r")
while not self.stopRequest.isSet():
msg = ''
if not self.queue.empty():
try:
command = self.queue.get()
self.serial.write(command)
except Queue.Empty:
continue
while self.serial.inWaiting():
char = self.serial.read(1)
if '\r' in char and len(msg) > 1:
char = ''
#~ print('[DATA]\t{}'.format(msg))
event = Events.PumpDataEvent(Events.SERIALRX, wx.ID_ANY, msg)
wx.PostEvent(self.parent, event)
msg = ''
break
msg += char
self.serial.close()
def join (self, timeout=None):
self.stopRequest.set()
super(PumpThread, self).join(timeout)
def SetPort (self, serial):
self.serial = serial
def Write (self, msg):
if self.serial.is_open:
self.queue.put(msg)
else:
print("[ERROR]\tPort {} is not open!".format(self.port))
def Stop(self):
if self.isAlive():
self.join()
The Queue is used for sending messages to the port and main loop takes responses back. I've used no serial.readline() method, because of different end-line char, and I have found the usage of io classes to be too much fuss.
Depends on what you run in that thread.
If that's your code, then you can implement a stop condition (see other answers).
However, if what you want is to run someone else's code, then you should fork and start a process. Like this:
import multiprocessing
proc = multiprocessing.Process(target=your_proc_function, args=())
proc.start()
now, whenever you want to stop that process, send it a SIGTERM like this:
proc.terminate()
proc.join()
And it's not slow: fractions of a second.
Enjoy :)
My solution is:
import threading, time
def a():
t = threading.currentThread()
while getattr(t, "do_run", True):
print('Do something')
time.sleep(1)
def getThreadByName(name):
threads = threading.enumerate() #Threads list
for thread in threads:
if thread.name == name:
return thread
threading.Thread(target=a, name='228').start() #Init thread
t = getThreadByName('228') #Get thread by name
time.sleep(5)
t.do_run = False #Signal to stop thread
t.join()
I find it useful to have a class, derived from threading.Thread, to encapsulate my thread functionality. You simply provide your own main loop in an overridden version of run() in this class. Calling start() arranges for the object’s run() method to be invoked in a separate thread.
Inside the main loop, periodically check whether a threading.Event has been set. Such an event is thread-safe.
Inside this class, you have your own join() method that sets the stop event object before calling the join() method of the base class. It can optionally take a time value to pass to the base class's join() method to ensure your thread is terminated in a short amount of time.
import threading
import time
class MyThread(threading.Thread):
def __init__(self, sleep_time=0.1):
self._stop_event = threading.Event()
self._sleep_time = sleep_time
"""call base class constructor"""
super().__init__()
def run(self):
"""main control loop"""
while not self._stop_event.isSet():
#do work
print("hi")
self._stop_event.wait(self._sleep_time)
def join(self, timeout=None):
"""set stop event and join within a given time period"""
self._stop_event.set()
super().join(timeout)
if __name__ == "__main__":
t = MyThread()
t.start()
time.sleep(5)
t.join(1) #wait 1s max
Having a small sleep inside the main loop before checking the threading.Event is less CPU intensive than looping continuously. You can have a default sleep time (e.g. 0.1s), but you can also pass the value in the constructor.
Sometimes you don't have control over the running target. In those cases you can use signal.pthread_kill to send a stop signal.
from signal import pthread_kill, SIGTSTP
from threading import Thread
from itertools import count
from time import sleep
def target():
for num in count():
print(num)
sleep(1)
thread = Thread(target=target)
thread.start()
sleep(5)
pthread_kill(thread.ident, SIGTSTP)
result
0
1
2
3
4
[14]+ Stopped
Using the Timer class and restarting the timer in the callable, is the standard method of running a periodic timer in the background in python.
This has two major drawbacks:
it is not really periodic: setup of the timer, ...
it creates a new thread for each period
Is there an alternative to the Timer class? I have taken a look at the sched class, but running in the MainThread will block it, and it is not recommended to run it in a multithreaded environment.
How can I have a high frequency periodic timer (100 ms period) in python, for example to regularly empty a document queue when collecting bulk data to send to a database?
I have come up with the following alternative:
import threading
import time
class PeriodicThread(StoppableThread):
'''Similar to a Timer(), but uses only one thread, stops cleanly and exits when the main thread exits'''
def __init__ (self, period, callable, *args, **kwargs):
super(PeriodicThread, self).__init__()
self.period = period
self.args = args
self.callable = callable
self.kwargs = kwargs
self.daemon = True
def run(self):
while not self.stopped():
self.callable(*self.args, **self.kwargs)
time.sleep(self.period)