Using the Timer class and restarting the timer in the callable, is the standard method of running a periodic timer in the background in python.
This has two major drawbacks:
it is not really periodic: setup of the timer, ...
it creates a new thread for each period
Is there an alternative to the Timer class? I have taken a look at the sched class, but running in the MainThread will block it, and it is not recommended to run it in a multithreaded environment.
How can I have a high frequency periodic timer (100 ms period) in python, for example to regularly empty a document queue when collecting bulk data to send to a database?
I have come up with the following alternative:
import threading
import time
class PeriodicThread(StoppableThread):
'''Similar to a Timer(), but uses only one thread, stops cleanly and exits when the main thread exits'''
def __init__ (self, period, callable, *args, **kwargs):
super(PeriodicThread, self).__init__()
self.period = period
self.args = args
self.callable = callable
self.kwargs = kwargs
self.daemon = True
def run(self):
while not self.stopped():
self.callable(*self.args, **self.kwargs)
time.sleep(self.period)
Related
I'm using the threading.Timer package to execute a method after x seconds. However, in some cases I want to execute this method earlier and cancel the timer (so it isn't called twice). How do I unit test this?
I want to know if the timer has stopped so that the method is not called anymore. I am now using the following code, unfortunately the is_alive still returns True
from threading import Timer
Class X():
def __init__(self, timeout):
self.timer = Timer(timeout, self.some_method)
self.timer.start()
def some_method(self):
# Do something
def other_method(self):
self.timer.cancel()
self.some_method()
import unittest
Class TestX(unittest.TestCase):
def test_cancel_timer(self):
x = X(1000)
x.other_method()
self.assertFalse(x.timer.is_alive())
Form the documentation the is_alive method returns True during the run operation;
Return whether the thread is alive.
This method returns True just before the run() method starts until just after the run() method terminates. The module function enumerate() returns a list of all alive threads.
The documentation on the cancel method says the following;
Stop the timer, and cancel the execution of the timer’s action. This will only work if the timer is still in its waiting stage.
Does this mean that the cancel method does not stop the run action? Or is is still in the grey area after the run method and returns True for that reason?
With timer.is_alive() you are just checking if the timer-thread itself is alive, so if you want to "check if timer.cancel() was called", you're testing for the wrong thing.
Does this mean that the cancel method does not stop the run action?
It does not stop the run()-function, right. timer.cancel() just sets a flag in an Event-object which gets checked by run. You can test if the flag is set with:
self.assertTrue(x.timer.finished.is_set())
Unfortunately, checking for cancellation is not enough to prevent repeated execution, since run can have already crossed the check like you can see in the source code:
# threading.py (Python 3.7.1):
class Timer(Thread):
"""Call a function after a specified number of seconds:
t = Timer(30.0, f, args=None, kwargs=None)
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
def __init__(self, interval, function, args=None, kwargs=None):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args if args is not None else []
self.kwargs = kwargs if kwargs is not None else {}
self.finished = Event()
def cancel(self):
"""Stop the timer if it hasn't finished yet."""
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
Some more effort is needed to ensure unique execution. I've written up a possible solution to this in my answer here.
when I call a function with threading.Timer like below:
threading.Timer(LOOP_TIME,self.broadCast).start()
does broadCast get run in a separate thread? Or is it just in the same thread?
I'm using threading.Timer so I can have a function called every so much time interval. I do not want the broadCast function to be called outside of the main thread.
Yes. You can look into Python threading.py source code:
def Timer(*args, **kwargs):
"""Factory function to create a Timer object.
Timers call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
return _Timer(*args, **kwargs)
class _Timer(Thread):
"""Call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
def __init__(self, interval, function, args=[], kwargs={}):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args
self.kwargs = kwargs
self.finished = Event()
Source code available in Python source code repository.
If you want timers and your main thread is not doing co-operative multitasking, I suggest you refactor your code so that you can use it from other threads.
I want to know how python threading.Timer works.
In more detail, When i run a couple of threading.Timer, does it run separate thread for counting a time and running the handler ?
Or one thread manages and counts a couple of timer together ?
I am asking because my application need to schedule many event, But
If threading.Timer runs separate each thread for counting a timer, and i run many timers, it may affect performance so much.
So i am worry that if i have to implement a scheduler running only one thread if it has big effect in performance.
threading.Timer class is a subclass of threading.Thread and basically it just runs a separate thread in which it sleeps for the specified amount of time and runs the corresponding function.
It is definitely not an efficient way to schedule events. Better way is to do the scheduling in a single thread by using Queue.PriorityQueue in which you would put your events where "priority" actually means "next fire date". Similar to how cron works.
Or even better: use something that already exists, do not reinvent the wheel: Cron, Celery, whatever...
A very simplified example of making a scheduler via Queue.PriorityQueue:
import time
from Queue import PriorityQueue
class Task(object):
def __init__(self, fn, crontab):
# TODO: it should be possible to pass args, kwargs
# so that fn can be called with fn(*args, **kwargs)
self.fn = fn
self.crontab = crontab
def get_next_fire_date(self):
# TODO: evaluate next fire date based on self.crontab
pass
class Scheduler(object):
def __init__(self):
self.event_queue = PriorityQueue()
self.new_task = False
def schedule_task(self, fn, crontab):
# TODO: add scheduling language, crontab or something
task = Task(fn, crontab)
next_fire = task.get_next_fire_date()
if next_fire:
self.new_task = True
self.event_queue.put((next_fire, task))
def run(self):
self.new_task = False
# TODO: do we really want an infinite loop?
while True:
# TODO: actually we want .get() with timeout and to handle
# the case when the queue is empty
next_fire, task = self.event_queue.get()
# incremental sleep so that we can check
# if new tasks arrived in the meantime
sleep_for = int(next_fire - time.time())
for _ in xrange(sleep_for):
time.sleep(1)
if self.new_task:
self.new_task = False
self.event_queue.put((next_fire, task))
continue
# TODO: run in separate thread?
task.fn()
time.sleep(1)
next_fire = task.get_next_fire_date()
if next_fire:
event_queue.put((next_fire, task))
def test():
return 'hello world'
sch = Scheduler()
sch.schedule_task(test, '5 * * * *')
sch.schedule_task(test, '0 22 * * 1-5')
sch.schedule_task(test, '1 1 * * *')
sch.run()
It's just an idea. You would have to properly implement both Task and Scheduler classes, i.e. get_next_fire_date method plus some kind of scheduling language (crontab?) and error handling. I still strongly suggest to use one of the existing libraries.
From the CPython 2.7 source:
def Timer(*args, **kwargs):
"""Factory function to create a Timer object.
Timers call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
return _Timer(*args, **kwargs)
class _Timer(Thread):
"""Call a function after a specified number of seconds:
t = Timer(30.0, f, args=[], kwargs={})
t.start()
t.cancel() # stop the timer's action if it's still waiting
"""
def __init__(self, interval, function, args=[], kwargs={}):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args
self.kwargs = kwargs
self.finished = Event()
def cancel(self):
"""Stop the timer if it hasn't finished yet"""
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
As said in another answer, it is a separate thread (since it subclasses Thread). The callback function when the timer runs out is called from the new thread.
i have a question about Timer object in Python:
https://docs.python.org/2/library/threading.html#timer-objects
This object is basically a Thread that calls a specific function after an amount of user defined seconds. My question is: after the function call, the Thread object is automatically eliminated of the program, or this object keeps running, consuming memory?
This snippet of code shows what i'm talking about:
from threading import Timer
from random import randint
def call_function(x):
print "Timer number " + str(x) + " hit call function!"
for x in range(200000):
#memory used: 700.000 K
print "Creating timer number: " + str(x)
Timer(randint(1, 10), call_function, [x]).start()
After 200.000 threads Timer created and being called in a space of 10 seconds (after this call they were supposed to be dead, releasing space, right?), the program end it's execution with 700.000 K of memory consumption, almost 1 GB.
Thanks,
Timer is just a Thread subclass (created by factory function named Timer) with simple run method:
class _Timer(Thread):
def __init__(self, interval, function, args=[], kwargs={}):
Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args
self.kwargs = kwargs
self.finished = Event()
def cancel(self):
self.finished.set()
def run(self):
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
thread terminates just after the run() method terminates:
Once the thread’s activity is started, the thread is considered
‘alive’. It stops being alive when its run() method terminates –
either normally, or by raising an unhandled exception. The is_alive()
method tests whether the thread is alive.
timer class object itself should be garbage collected automatically if it will not be used later.
And you cant start 200000 threads, its too much (_Timer object will be created but thread will not start). Check this question: How many threads is too many?
Quick question on the use of QThread in PyQt4 and Python 2.7. I am creating a process inherited from QObject, and assigning this to a Qthread I have created in a separate class (also inherited from QObject).
Is it safe to pass the QThread object to the process object, so that I can call thread.msleep(mseconds) from within the process itself?
I want to be able to make the thread wait or sleep, but I have read that time.sleep(seconds) is dodgy when used with PyQt multi-threading.
I did try to send a signal from the process object to a slot in the main thread (attached to thread.msleep(mseconds) for that process object), but I found that this failed to work; the process object continued executing until complete, with the slot only being executed after this time. Even after adjusting priorities, this continued to happen. This is unacceptable since I want the process loop to run continuously.
Any other recommendations?
I eventually managed to alter my code to achieve the functionality that I required in my question: namely the ability to make a thread wait or sleep for a specified amount of time.
Firstly, my research seems to show that one of the main reasons subclassing QThread became ill-advised in Qt was that a thread should not be able to manage itself. Though there is no official documentation on my question, I can only surmise that passing the thread object to the process object running on it would also be ill-advised, because the thread would again be able to control itself directly.
The solution I have found is to dispense with msleep() altogether. Qt documentation on QThread recommends that sleep() and wait() functions are avoided because they do not fit well with the event driven nature of Qt. They recommend that QTimer() is used to call a function via a signal after it times out, in place of msleep(). By default QTimer() is used to send a repeating signal every time interval, but can also send a signal once using QTimer.singleShot(). It is also stated in the documentation that it is safe to call QSleep() from within a thread.
I only use a repeating QTimer to call a single slot foo() multiple times, but to add a delay within foo(), QTimer.singleShot() could be used to call a second function moo() after a set number of milliseconds.
EDIT: I have decided to include my threading code, which subclasses QObject and QThread to perform a task on a thread in a continual loop every given time interval. It is, as far as I can tell, fully functional, though could do with a little work.
# -*- coding: utf-8 -*-
import sys
from PyQt4 import QtCore, QtGui
# Class to be assigned to a thread.
# This should be subclassed to provide new functionality.
class GenericLoop(QtCore.QObject):
def __init__(self):
super(GenericLoop, self).__init__()
# We use this signal to tell the main thread
# when this thread is finished.
finished_Sig = QtCore.pyqtSignal()
# Default timeout is 0, i.e. do work on thread after
# other events have been dealt with
__timeout = 0
__processTimer = None
__args = None
__kwargs = None
# We use this function to set the arguments used by run(),
# if we want to change them mid-execution
#QtCore.pyqtSlot(tuple, dict)
def changeArgs(self, args, kwargs):
self.__args = args
self.__kwargs = kwargs
# We can change the timeout used to make the thread run
# at given intervals. Note that the timing is not exact,
# since this is impossible with a real time operating system
#QtCore.pyqtSlot(int)
def setTimeout(self, mseconds):
self.__timeout = int(mseconds)
# Call either a singleShot QTimer (one execution),
# or a normal QTimer (repeated), to start the loop
#QtCore.pyqtSlot(bool, tuple, dict)
def startTimer(self, singleShot, args, kwargs):
self.__processTimer = QtCore.QTimer()
# We can't pass args and kwargs directly because QTimer.timeout
# emits a signal with allowing no contained variables
# so we copy args and kwargs to local variables instead
self.changeArgs(args, kwargs)
if singleShot:
self.__processTimer.singleShot(self.__timeout, self.callRun)
else:
self.__processTimer.timeout.connect(self.callRun)
self.__processTimer.start(self.__timeout)
# Call finish from within subclass using self.finish(), or
# from another thread using signals. finish() will stop the
# QTimer causing execution of the loop. The loop can be started again
# by calling startTimer() or stopTimer() from another thread
#QtCore.pyqtSlot()
def stopTimer(self):
if self.__processTimer.isActive():
self.__processTimer.stop()
else:
print "ERROR: stopTimer() has been called but no timer is running!"
# We call this to delete the thread.
#QtCore.pyqtSlot()
def deleteThread(self):
self.finished_Sig.emit()
# This calls run(), in order to enable the passing of
# command line arguments to the loop
#QtCore.pyqtSlot()
def callRun(self):
self.run(self.__args, self.__kwargs)
# run() can be called directly from another thread if required
#QtCore.pyqtSlot(tuple, dict)
def run(self, args, kwargs):
print "ERROR: run() has not been defined! Stopping thread..."
self.stopTimer()
# Class for creating threads
class GenericThread(QtCore.QObject):
# Private variables include the thread.
__sendArguments_Sig = QtCore.pyqtSignal(tuple, dict)
__startTimer_Sig = QtCore.pyqtSignal(int, tuple, dict)
__setTimeout_Sig = QtCore.pyqtSignal(int)
__obj = None
__finished_Sig = None
__thread = QtCore.QThread()
# Object to be threaded must be specified when
# creating a GenericThread object
def __init__(self, obj):
super(GenericThread, self).__init__()
self.__obj = obj
self.moreInit()
# Set up object on thread
def moreInit(self):
self.__thread = QtCore.QThread()
self.__obj.moveToThread(self.__thread)
# Allows thread to delete itself when done
self.__obj.finished_Sig.connect(self.__thread.deleteLater)
self.__sendArguments_Sig.connect(self.__obj.changeArgs)
self.__startTimer_Sig.connect(self.__obj.startTimer)
self.__setTimeout_Sig.connect(self.__obj.setTimeout)
self.__thread.start()
# Sets the QTimer timeout and does some checking
# to make sure that types are as they should be
def setTimeout(self, mseconds):
if mseconds >= 0 and type(mseconds) is type(int()):
self.__setTimeout_Sig.emit(mseconds)
elif mseconds < 0 and type(mseconds) is type(int()):
print "Error: timeout of below 0 ms specified."
else:
print "Error: timeout period is specified with a type other than int."
# Starts a function in the thread via signals, and can pass
# it arguments if required. Function executes until QTimer is stopped
def startLoop(self, *args, **kwargs):
if (self.__thread == None):
print "ERROR: Thread has been deleted!"
else:
self.__startTimer_Sig.emit(False, args, kwargs)
# Starts a function in the thread via signals, once
def startOnce(self, *args, **kwargs):
if (self.__thread == None):
print "ERROR: Thread has been deleted!"
else:
self.__startTimer_Sig.emit(True, args, kwargs)
# Calls a very simple GUI just to show that the program is responsive
class GUIBox(QtGui.QWidget):
def __init__(self):
super(GUIBox, self).__init__()
self.initUI()
def initUI(self):
self.resize(250, 150)
self.setWindowTitle('Threading!')
self.show()
# Subclass GenericLoop to reimplement run and such.
class SubClassedLoop(GenericLoop):
def __init__(self):
super(SubClassedLoop, self).__init__()
__i = 0
#QtCore.pyqtSlot(tuple, dict)
def run(self, args, kwargs):
if self.__i>=50:
self.stopTimer()
return
print self.__i, args
self.__i += 1
app = QtGui.QApplication(sys.argv)
ex = GUIBox()
# Create 3 worker objects to do the actual calculation
worker1 = SubClassedLoop()
worker2 = SubClassedLoop()
worker3 = SubClassedLoop()
# Create 3 thread managing objects to do the thread control
thread1 = GenericThread(worker1)
thread2 = GenericThread(worker2)
thread3 = GenericThread(worker3)
# Set the threads to execute as soon as there is no work to do
thread1.setTimeout(125)
thread2.setTimeout(125)
thread3.setTimeout(125)
# Start threads
thread1.startLoop(1)
thread2.startLoop(2)
thread3.startLoop(3)
# Quit the program when the GUI window is closed
sys.exit( app.exec_() )