Develop Reference

Retrieving remaining time from python timer

Looking for simple approach to obtaining remaining and elapsed time from python timer. Currently have (based on github source for threading.Timer and previous post):
import threading
import time
class CountdownTimer(threading.Thread):
def __init__(self, interval, function, args=None, kwargs=None):
threading.Thread.__init__(self)
self.interval = interval
self.function = function
self.args = args if args is not None else []
self.kwargs = kwargs if kwargs is not None else {}
self.finished = Event()
self.started_at = None
def cancel(self):
self.finished.set()
def elapsed(self):
return time.time() - self.started_at
def remaining(self):
return self.interval - self.elapsed()
def run(self):
self.started_at = time.time()
self.finished.wait(self.interval)
if not self.finished.is_set():
self.function(*self.args, **self.kwargs)
self.finished.set()
Does this look reasonably effective (do not need accuracy in excess of what threading.Timer currently provides)?

perf_counter()
import time
start = time.perf_counter()
time.sleep(2)
finish = time.perf_counter()
print(f'Finished in {round(finish-start, 2)} second(s)')
Advantages of perf_counter() :
perf_counter() will give you more precise value than time.clock() function .
From Python3.8 time.clock() function will be deleted and perf_counter will be used.
We can calculate float and integer both values of time in seconds and nanoseconds.

Making a timer: timeout inaccuracy of threading.Event.wait - Python 3.6

First of all, I am new to Python and not familiar with its functionalities. I've been mainly using MATLAB.
PC brief spec.: Windows 10, Intel i7
I am trying to make a timer class for periodic execution of a function such as MATLAB has, which is obviously borrowed from Java timer. The MATLAB timer has an about 1 ms resolution and I've never seen it exceeds 2 ms in any situation. In fact, it is accurate enough for my project.
Recently, I planned to move to Python because of the poor parallel computing and web access features of MATLAB. However, unfortunately, the standard packages of Python offer somewhat low-level of timer (threading.Timer) compared to MATLAB that I had to make my own timer class. First, I referred to the QnA Executing periodic actions in Python [duplicate]. The solution suggested by Michael Anderson gives a simple idea of drift correction. He used time.sleep() to keep the period. The approach is highly accurate and sometimes showed better accuracy over the MATLAB timer. approx. 0.5 ms resolution. However, the timer cannot be interrupted (pause or resume) during being captured in time.sleep(). But I sometimes have to stop immediately regardless of whether it is in sleep() or not.
A solution to the problem I found is to utilize the Event class in threading package. Refer to Python threading.timer - repeat function every 'n' seconds
. Using the timeout feature of Event.wait(), I could make a time gap between executions and it is used to keep the period. That is, the event is usually cleared so that wait(timeout) can act like time.sleep(interval) and I could exit from wait() immediately, when needed, by setting event.
Everything seemed fine then but there is a critical problem in Event.wait(). The time delay varies too largely from 1 ~ 15 ms. I think it comes from the overhead of Event.wait().
I made an example code that shows accuracy comparison between time.sleep() and Event.wait(). This sums total of 1000 iterations of 1 ms sleep() and wait() to see the accumulated time error. The expected result is about 1.000.
import time
from threading import Event
time.sleep(3) # to relax
# time.sleep()
tspan = 1
N = 1000
t1 = time.perf_counter()
for _ in range(N):
time.sleep(tspan/N)
t2 = time.perf_counter()
print(t2-t1)
time.sleep(3) # to relax
# Event.wait()
tspan = 1
event = Event()
t1 = time.perf_counter()
for _ in range(N):
event.wait(tspan/N)
t2 = time.perf_counter()
print(t2-t1)
Result:
1.1379848184879964
15.614547161211096
The result shows that time.sleep() is much better in accuracy. But I cannot purely rely on time.sleep() as previously mentioned.
In summary,
time.sleep(): accurate but not interruptible
threading.Event.wait(): inaccurate but interruptible
I am currently thinking of a compromise: just as in the example, make a loop of tiny time.sleep() (of 0.5 ms interval) and exit the loop using if-statement and break when needed. As far as I know, the method is used in Python 2.x Python time.sleep() vs event.wait().
It was a verbose introduction, but my question can be summarized as follows.
Can I force thread process to break from time.sleep() by an external signal or event? (This seems to be most efficient.???)
To make Event.wait() more accurate or to reduce overhead time.
Are there any better approaches aside of sleep() and Event.wait() approach to improve timing precision.
Thank you very much.

I ran into the same timing issue with Event.wait(). The solution I came up with was to create a class which mimics threading.Event. Internally, it uses a combination of a time.sleep() loop and a busy loop for greatly increased precision. The sleep loop runs in a separate thread so that the blocking wait() call in the main thread can still be immediately interrupted. When the set() method is called, the sleep thread should terminate shortly afterwards. Also, in order to minimize CPU utilization, I made sure that the busy loop will never run for more than 3 milliseconds.
Here is my custom Event class along with a timing demo at the end (the printed execution times from the demo will be in nanoseconds):
import time
import _thread
import datetime
class Event:
__slots__ = (
"_flag", "_lock", "_nl",
"_pc", "_waiters"
)
_lock_type = _thread.LockType
_timedelta = datetime.timedelta
_perf_counter = time.perf_counter
_new_lock = _thread.allocate_lock
class _switch:
__slots__ = ("_on",)
def __call__(self, on: bool = None):
if on is None:
return self._on
self._on = on
def __bool__(self):
return self._on
def __init__(self):
self._on = False
def clear(self):
with self._lock:
self._flag(False)
def is_set(self) -> bool:
return self._flag()
def set(self):
with self._lock:
self._flag(True)
waiters = self._waiters
for waiter in waiters:
waiter.release()
waiters.clear()
def wait(
self,
timeout: float = None
) -> bool:
with self._lock:
return self._wait(self._pc(), timeout)
def _new_waiter(self) -> _lock_type:
waiter = self._nl()
waiter.acquire()
self._waiters.append(waiter)
return waiter
def _wait(
self,
start: float,
timeout: float,
td=_timedelta,
pc=_perf_counter,
end: _timedelta = None,
waiter: _lock_type = None,
new_thread=_thread.start_new_thread,
thread_delay=_timedelta(milliseconds=3)
) -> bool:
flag = self._flag
if flag:
return True
elif timeout is None:
waiter = self._new_waiter()
elif timeout <= 0:
return False
else:
delay = td(seconds=timeout)
end = td(seconds=start) + delay
if delay > thread_delay:
mark = end - thread_delay
waiter = self._new_waiter()
new_thread(
self._wait_thread,
(flag, mark, waiter)
)
lock = self._lock
lock.release()
try:
if waiter:
waiter.acquire()
if end:
while (
not flag and
td(seconds=pc()) < end
):
pass
finally:
lock.acquire()
if waiter and not flag:
self._waiters.remove(waiter)
return flag()
#staticmethod
def _wait_thread(
flag: _switch,
mark: _timedelta,
waiter: _lock_type,
td=_timedelta,
pc=_perf_counter,
sleep=time.sleep
):
while not flag and td(seconds=pc()) < mark:
sleep(0.001)
if waiter.locked():
waiter.release()
def __new__(cls):
_new_lock = cls._new_lock
_self = object.__new__(cls)
_self._waiters = []
_self._nl = _new_lock
_self._lock = _new_lock()
_self._flag = cls._switch()
_self._pc = cls._perf_counter
return _self
if __name__ == "__main__":
def test_wait_time():
wait_time = datetime.timedelta(microseconds=1)
wait_time = wait_time.total_seconds()
def test(
event=Event(),
delay=wait_time,
pc=time.perf_counter
):
pc1 = pc()
event.wait(delay)
pc2 = pc()
pc1, pc2 = [
int(nbr * 1000000000)
for nbr in (pc1, pc2)
]
return pc2 - pc1
lst = [
f"{i}.\t\t{test()}"
for i in range(1, 11)
]
print("\n".join(lst))
test_wait_time()
del test_wait_time

Chris D's custom Event class works impressively well! For practical purposes, I have included it into an installable package (https://github.com/ovinc/oclock, install with pip install oclock) that also includes other timing tools. From version 1.3.0 of oclock and onwards, one can use the custom Event class discussed in Chris D's answer, e.g.
from oclock import Event
event = Event()
event.wait(1)
with the usual set(), clear(), is_set(), wait() methods of the Event class.
The timing accuracy is much better than with threading.Event, in Windows in particular. For example on a Windows machine with 1000 repeated loops, I get a standard deviation in the duration of the loop of 7ms for threading.Event and less than 0.01 ms for oclock.Event. Props to Chris D!
Note: The oclock package is under the GPLv3 license for compatibility with StackOverflow's CC BY-SA 4.0.

Thank you for this topic and all answers. I also experienced some troubles with inaccurate timing (Windows 10 + Python 3.9 + Threading).
The solution is to use oclock package and also change (temporarily) resolution of Windows system timer by wres package. This package utilizes undocumented Windows API function NtSetTimerResolution (warning: resolution is changed system-wide).
Application of oclock package only does not solve the problem.
With both python packages applied, the code below schedules periodic event correctly and precisely enough. If terminated, original timer resolution is restored.
import threading
import datetime
import time
import oclock
import wres
class Job(threading.Thread):
def __init__(self, interval, *args, **kwargs):
threading.Thread.__init__(self)
# use oclock.Event() instead of threading.Event()
self.stopped = oclock.Event()
self.interval = interval.total_seconds()
self.args = args
self.kwargs = kwargs
def stop(self):
self.stopped.set()
self.join()
def run(self):
prevTime = time.time()
while not self.stopped.wait(self.interval):
now = time.time()
print(now - prevTime)
prevTime = now
# Set system timer resolution to 1 ms
# Automatically restore previous resolution when exit with statement
with wres.set_resolution(10000):
# Create thread with periodic task called every 10 ms
job = Job(interval=datetime.timedelta(seconds=0.010))
job.start()
try:
while True:
time.sleep(1)
# Hit Ctrl+C to terminate main loop and spawned thread
except KeyboardInterrupt:
job.stop()

Class Decorators Singleton?

So for example, I'm making an async decorator and wanted to limit the number of concurrent threads:
from multiprocessing import cpu_count
from threading import Thread
class async:
def __init__(self, function):
self.func = function
self.max_threads = cpu_count()
self.current_threads = []
def __call__(self, *args, **kwargs):
func_thread = Thread(target = self.func, args = args, kwargs = kwargs)
func_thread.start()
self.current_threads.append(func_thread)
while len(self.current_threads) > self.max_threads:
self.current_threads = [t for t in self.current_threads if t.isAlive()]
from time import sleep
#async
def printA():
sleep(1)
print "A"
#async
def printB():
sleep(1)
print "B"
Is this going to limit the total concurrent threads? IE. If I had 8 cores, would the current code end up having 16+ threads due to two separate async objects existing?
If so, how would I fix that?
Thanks!

Start python Process with output and timeout

I'm trying to find the way to start a new Process and get its output if it takes less than X seconds. If the process takes more time I would like to ignore the Process result, kill the Process and carry on.
I need to basically add the timer to the code below. Now sure if there's a better way to do it, I'm open to a different and better solution.
from multiprocessing import Process, Queue
def f(q):
# Ugly work
q.put(['hello', 'world'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print q.get()
p.join()
Thanks!

You may find the following module useful in your case:
Module
#! /usr/bin/env python3
"""Allow functions to be wrapped in a timeout API.
Since code can take a long time to run and may need to terminate before
finishing, this module provides a set_timeout decorator to wrap functions."""
__author__ = 'Stephen "Zero" Chappell ' \
'<stephen.paul.chappell#atlantis-zero.net>'
__date__ = '18 December 2017'
__version__ = 1, 0, 1
__all__ = [
'set_timeout',
'run_with_timeout'
]
import multiprocessing
import sys
import time
DEFAULT_TIMEOUT = 60
def set_timeout(limit=None):
"""Return a wrapper that provides a timeout API for callers."""
if limit is None:
limit = DEFAULT_TIMEOUT
_Timeout.validate_limit(limit)
def wrapper(entry_point):
return _Timeout(entry_point, limit)
return wrapper
def run_with_timeout(limit, polling_interval, entry_point, *args, **kwargs):
"""Execute a callable object and automatically poll for results."""
engine = set_timeout(limit)(entry_point)
engine(*args, **kwargs)
while engine.ready is False:
time.sleep(polling_interval)
return engine.value
def _target(queue, entry_point, *args, **kwargs):
"""Help with multiprocessing calls by being a top-level module function."""
# noinspection PyPep8,PyBroadException
try:
queue.put((True, entry_point(*args, **kwargs)))
except:
queue.put((False, sys.exc_info()[1]))
class _Timeout:
"""_Timeout(entry_point, limit) -> _Timeout instance"""
def __init__(self, entry_point, limit):
"""Initialize the _Timeout instance will all needed attributes."""
self.__entry_point = entry_point
self.__limit = limit
self.__queue = multiprocessing.Queue()
self.__process = multiprocessing.Process()
self.__timeout = time.monotonic()
def __call__(self, *args, **kwargs):
"""Begin execution of the entry point in a separate process."""
self.cancel()
self.__queue = multiprocessing.Queue(1)
self.__process = multiprocessing.Process(
target=_target,
args=(self.__queue, self.__entry_point) + args,
kwargs=kwargs
)
self.__process.daemon = True
self.__process.start()
self.__timeout = time.monotonic() + self.__limit
def cancel(self):
"""Terminate execution if possible."""
if self.__process.is_alive():
self.__process.terminate()
#property
def ready(self):
"""Property letting callers know if a returned value is available."""
if self.__queue.full():
return True
elif not self.__queue.empty():
return True
elif self.__timeout < time.monotonic():
self.cancel()
else:
return False
#property
def value(self):
"""Property that retrieves a returned value if available."""
if self.ready is True:
valid, value = self.__queue.get()
if valid:
return value
raise value
raise TimeoutError('execution timed out before terminating')
#property
def limit(self):
"""Property controlling what the timeout period is in seconds."""
return self.__limit
#limit.setter
def limit(self, value):
self.validate_limit(value)
self.__limit = value
#staticmethod
def validate_limit(value):
"""Verify that the limit's value is not too low."""
if value <= 0:
raise ValueError('limit must be greater than zero')
To use, see the following example that demonstrates its usage:
Example
from time import sleep
def main():
timeout_after_four_seconds = timeout(4)
# create copies of a function that have a timeout
a = timeout_after_four_seconds(do_something)
b = timeout_after_four_seconds(do_something)
c = timeout_after_four_seconds(do_something)
# execute the functions in separate processes
a('Hello', 1)
b('World', 5)
c('Jacob', 3)
# poll the functions to find out what they returned
results = [a, b, c]
polling = set(results)
while polling:
for process, name in zip(results, 'abc'):
if process in polling:
ready = process.ready
if ready is True: # if the function returned
print(name, 'returned', process.value)
polling.remove(process)
elif ready is None: # if the function took too long
print(name, 'reached timeout')
polling.remove(process)
else: # if the function is running
assert ready is False, 'ready must be True, False, or None'
sleep(0.1)
print('Done.')
def do_something(data, work):
sleep(work)
print(data)
return work
if __name__ == '__main__':
main()

Does the process you are running involve a loop?
If so you can get the timestamp prior to starting the loop and include an if statement within the loop with an sys.exit(); command terminating the script if the current timestamp differs from the recorded start time stamp by more than x seconds.

All you need to adapt the queue example from the docs to your case is to pass the timeout to the q.get() call and terminate the process on timeout:
from Queue import Empty
...
try:
print q.get(timeout=timeout)
except Empty: # no value, timeout occured
p.terminate()
q = None # the queue might be corrupted after the `terminate()` call
p.join()
Using a Pipe might be more lightweight otherwise the code is the same (you could use .poll(timeout), to find out whether there is a data to receive).

Calling thread.timer() more than once

The code:
from threading import Timer
import time
def hello():
print "hello"
a=Timer(3,hello,())
a.start()
time.sleep(4)
a.start()
After running this script I get error: RuntimeError: threads can only be started once
so how do I deal with this error. I want to start the timer more than once.

threading.Timer inherits threading.Thread. Thread object is not reusable. You can create Timer instance for each call.
from threading import Timer
import time
class RepeatableTimer(object):
def __init__(self, interval, function, args=[], kwargs={}):
self._interval = interval
self._function = function
self._args = args
self._kwargs = kwargs
def start(self):
t = Timer(self._interval, self._function, *self._args, **self._kwargs)
t.start()
def hello():
print "hello"
a=RepeatableTimer(3,hello,())
a.start()
time.sleep(4)
a.start()

Since I'm used to start my oven timer each time I bake a cookie, I was surprised to see that python's timers are one-shot only.
That said I share a small timer class which btw offers some more options on the start method:
returns itself to allow a one line timer creation and start
optional parameter to restart a new timer or not if timer is still alive
Implementation:
from threading import Timer, Lock
class TimerEx(object):
"""
A reusable thread safe timer implementation
"""
def __init__(self, interval_sec, function, *args, **kwargs):
"""
Create a timer object which can be restarted
:param interval_sec: The timer interval in seconds
:param function: The user function timer should call once elapsed
:param args: The user function arguments array (optional)
:param kwargs: The user function named arguments (optional)
"""
self._interval_sec = interval_sec
self._function = function
self._args = args
self._kwargs = kwargs
# Locking is needed since the '_timer' object might be replaced in a different thread
self._timer_lock = Lock()
self._timer = None
def start(self, restart_if_alive=True):
"""
Starts the timer and returns this object [e.g. my_timer = TimerEx(10, my_func).start()]
:param restart_if_alive: 'True' to start a new timer if current one is still alive
:return: This timer object (i.e. self)
"""
with self._timer_lock:
# Current timer still running
if self._timer is not None:
if not restart_if_alive:
# Keep the current timer
return self
# Cancel the current timer
self._timer.cancel()
# Create new timer
self._timer = Timer(self._interval_sec, self.__internal_call)
self._timer.start()
# Return this object to allow single line timer start
return self
def cancel(self):
"""
Cancels the current timer if alive
"""
with self._timer_lock:
if self._timer is not None:
self._timer.cancel()
self._timer = None
def is_alive(self):
"""
:return: True if current timer is alive (i.e not elapsed yet)
"""
with self._timer_lock:
if self._timer is not None:
return self._timer.is_alive()
return False
def __internal_call(self):
# Release timer object
with self._timer_lock:
self._timer = None
# Call the user defined function
self._function(*self._args, **self._kwargs)
Here an example:
from time import sleep
def my_func(msg):
print(msg)
my_timer = TimerEx(interval_sec=5, function=my_func, msg="Here is my message").start()
sleep(10)
my_timer.start()
sleep(10)
Note: I'm using python 3.7, so I'm not 100% sure this works on Python 2