The sort of application to have in mind is an oscilloscope or high speed data logger. I have a function which retrieves the required information, I just need to work out how to call it over and over again, very quickly and with high precision.
There are limitations to time.sleep(), I don't think that is the way to go.
I have looked into the built in event scheduler, but I don't think it's precise enough and doesn't quite fill my needs.
The requirements for this are:
High speed sampling. 10ms is the most that will be asked of it.
High accuracy intervals. At 10ms, a 10% error is acceptable (±1ms).
Fairly low CPU usage, some load is acceptable at 10ms, but it should be less than ~5% for 100ms intervals and beyond. I know this is subjective, I guess what I'm saying is that hogging the CPU is unacceptable.
Ideally, the timer will be initialised with an interval time, and then started when required. The required function should then be called at the correct interval over and over again until the timer is stopped.
It will (not must) only ever run on a Windows machine.
Are there any existing libraries that fulfil these requirements? I don't want to re-invent the wheel, but if I have to I will probably use the Windows multimedia timer (winmm.dll). Any comments/suggestions with that?
I know I'm late to the game answering my own question, but hopefully it will help someone.
I wrote a wrapper to the Windows Multimedia Timer purely as a test. It seems to work well, but the code isn't fully tested and hasn't been optimized.
mmtimer.py:
from ctypes import *
from ctypes.wintypes import UINT
from ctypes.wintypes import DWORD
timeproc = WINFUNCTYPE(None, c_uint, c_uint, DWORD, DWORD, DWORD)
timeSetEvent = windll.winmm.timeSetEvent
timeKillEvent = windll.winmm.timeKillEvent
class mmtimer:
def Tick(self):
self.tickFunc()
if not self.periodic:
self.stop()
def CallBack(self, uID, uMsg, dwUser, dw1, dw2):
if self.running:
self.Tick()
def __init__(self, interval, tickFunc, stopFunc=None, resolution=0, periodic=True):
self.interval = UINT(interval)
self.resolution = UINT(resolution)
self.tickFunc = tickFunc
self.stopFunc = stopFunc
self.periodic = periodic
self.id = None
self.running = False
self.calbckfn = timeproc(self.CallBack)
def start(self, instant=False):
if not self.running:
self.running = True
if instant:
self.Tick()
self.id = timeSetEvent(self.interval, self.resolution,
self.calbckfn, c_ulong(0),
c_uint(self.periodic))
def stop(self):
if self.running:
timeKillEvent(self.id)
self.running = False
if self.stopFunc:
self.stopFunc()
Periodic test code:
from mmtimer import mmtimer
import time
def tick():
print("{0:.2f}".format(time.clock() * 1000))
t1 = mmtimer(10, tick)
time.clock()
t1.start(True)
time.sleep(0.1)
t1.stop()
Output in milliseconds:
0.00
10.40
20.15
29.91
39.68
50.43
60.19
69.96
79.72
90.46
100.23
One-shot test code:
from mmtimer import mmtimer
import time
def tick():
print("{0:.2f}".format(time.clock() * 1000))
t1 = mmtimer(150, tick, periodic=False)
time.clock()
t1.start()
Output in milliseconds:
150.17
As you can see from the results, it's pretty accurate. However, this is only using time.clock() so take them with a pinch of salt.
During a prolonged test with a 10ms periodic timer, CPU usage is around 3% or less on my old dual code 3GHz machine. The machine also seems to use that when it's idle though, so I'd say additional CPU usage is minimal.
Edit: After writing the stuff below, I'd be inclined to implement a similar test for the python event scheduler. I don't see why you think it would be insufficiently accurate.
Something like the following seems to work pretty well under Linux with me (and I have no reason to think it won't work with Windows). Every 10ms, on_timer_event() is called which prints out the time since the last call based on the real-time clock. This shows the approximate accuracy of the timers. Finally, the total time is printed out to show there is no drift.
There seems to be one issue with the code below with events occasionally appearing at spurious (and short intervals). I've no idea why this is, but no doubt with some playing you can make it reliable. I think this sort of approach is the way to go.
import pygame
import time
pygame.init()
TIMER_EVENT = pygame.USEREVENT+1
pygame.time.set_timer(TIMER_EVENT, 10)
timer_count = 0
MAX_TIMER_COUNT = 1000
def on_timer_event():
global last_time
global timer_count
new_time = time.time()
print new_time - last_time
last_time = new_time
timer_count += 1
if timer_count > MAX_TIMER_COUNT:
print last_time - initial_time
pygame.event.post(pygame.event.Event(pygame.QUIT, {}))
initial_time = time.time()
last_time = initial_time
while True:
event = pygame.event.wait()
if event.type == TIMER_EVENT:
on_timer_event()
elif event.type == pygame.QUIT:
break
timed-count was designed for exactly this. It doesn't suffer from temporal drift, so it can be used to repeatedly capture data streams and synchronise them afterwards.
There's a relevant high speed example here.
Related
I'm hoping to use an asyncio.loop to set callbacks at specific times. My problem is that I need to schedule these based on datetime.datetime objects (UTC) but asyncio.loop.call_at() uses an internal reference time.
A quick test on python 3.7.3 running on Ubuntu shows that asyncio.loop.time() is reporting the system uptime. For conversion my first thought is to naively store a reference time and use it later:
from asyncio import new_event_loop
from datetime import datetime, timedelta
_loop = new_event_loop()
_loop_base_time = datetime.utcnow() - timedelta(seconds=_loop.time())
def schedule_at(when, callback, *args):
_loop.call_at((when - _loop_base_time).total_seconds(), callback, *args)
However it's not clear whether or not this offset (datetime.utcnow() - timedelta(seconds=loop.time())) is stable. I have no idea whether system up-time drifts in comparison to UTC even where the system clock is modified (eg: through NTP updates).
Bearing in mind this is for monitoring software which will potentially be running for months at a time, small drifts might be very significant. I should note that I've seen systems lose minutes per day without an NTP daemon and one off NTP updates can shift times by many minutes in a short space of time. Since I don't know if the two are kept in sync, it's unclear how much I need to be concerned.
Note: I am aware of python's issue with scheduling events more than 24 hours in the future. I will get round this by storing distant future events in a list and polling for up-coming events every 12 hours, scheduling them only when they are < 24 hours in the future.
Is it possible to reliably convert from datetime.datetime to asyncio.loop times? or are the two time systems incomparable?. If they are comparable, is there anything special I need to do to ensure my calculations are correct.
You could compute the difference in seconds using the same time framework as the one you're using for scheduling, then use asyncio.call_later with the computed delay:
def schedule_at(when, callback, *args):
delay = (when - datetime.utcnow()).total_seconds()
_loop.call_later(delay, callback, *args)
This would work around the question of whether the difference between the loop's time and utcnow is stable; it only needs to be stable between the time of scheduling the task and the time of its execution (which, according to your notes, should be less than 12 hours).
For example: if the event loop's internal clock drifts 1 second apart from utcnow every hour (a deliberately extreme example), you would drift at most 12 seconds per task, but you would not accumulate this error over months of runtime. Compared with the approach of using a fixed reference, this approach gives a better guarantee.
Alternative approach would be not to rely on a loop internal clock at all. You can run a task in background and periodically check if callback should be executed.
This method's inaccuracy corresponds to a time you wait before next check, but I don't think it's critical considering any other possible inaccuracies (like Python GC's stop-the-world, for example).
On a good side is that you aren't limited by 24 hours.
This code shows main idea:
import asyncio
import datetime
class Timer:
def __init__(self):
self._callbacks = set()
self._task = None
def schedule_at(self, when, callback):
self._callbacks.add((when, callback,))
if self._task is None:
self._task = asyncio.create_task(self._checker())
async def _checker(self):
while True:
await asyncio.sleep(0.01)
self._exec_callbacks()
def _exec_callbacks(self):
ready_to_exec = self._get_ready_to_exec()
self._callbacks -= ready_to_exec
for _, callback in ready_to_exec:
callback()
def _get_ready_to_exec(self):
now = datetime.datetime.utcnow()
return {
(when, callback,)
for (when, callback,)
in self._callbacks
if when <= now
}
timer = Timer()
async def main():
now = datetime.datetime.utcnow()
s1_after = now + datetime.timedelta(seconds=1)
s3_after = now + datetime.timedelta(seconds=3)
s5_after = now + datetime.timedelta(seconds=5)
timer = Timer()
timer.schedule_at(s1_after, lambda: print('Hey!'))
timer.schedule_at(s3_after, lambda: print('Hey!'))
timer.schedule_at(s5_after, lambda: print('Hey!'))
await asyncio.sleep(6)
if __name__ == '__main__':
asyncio.run(main())
I would like to run a function asynchronously in Python, calling the function repeatedly at a fixed time interval. This java class has functionality similar to what I want. I was hoping for something in python like:
pool = multiprocessing.Pool()
pool.schedule(func, args, period)
# other code to do while that runs in the background
pool.close()
pool.join()
Are there any packages which provide similar functionality? I would prefer something simple and lightweight.
How could I implement this functionality in python?
This post is similar, but asks for an in process solution. I want a multiprocess async solution.
Here is one possible solution. One caveat is that func needs to return faster than rate, else it wont be called as frequently as rate and if it ever gets quicker it will be scheduled faster than rate while it catches up. This approach seems like a lot of work, but then again parallel programming is often tough. I would appreciate a second look at the code to make sure I don't have a deadlock waiting somewhere.
import multiprocessing, time, math
def func():
print('hello its now {}'.format(time.time()))
def wrapper(f, period, event):
last = time.time() - period
while True:
now = time.time()
# returns True if event is set, otherwise False after timeout
if event.wait(timeout=(last + period - now)):
break
else:
f()
last += period
def main():
period = 2
# event is the poison pill, setting it breaks the infinite loop in wrapper
event = multiprocessing.Event()
process = multiprocessing.Process(target=wrapper, args=(func, period, event))
process.start()
# burn some cpu cycles, takes about 20 seconds on my machine
x = 7
for i in range(50000000):
x = math.sqrt(x**2)
event.set()
process.join()
print('x is {} by the way'.format(x))
if __name__ == '__main__':
main()
I am getting a timestamp every time a key is pressed like this:
init_timestamp = time.time()
while (True):
c = getch()
offset = time.time() - init_timestamp
print("%s,%s" % (c,offset), file=f)
(getch from this answer).
I am verifying the timestamps against an audio recording of me actually typing the keys. After lining the first timestamp up with the waveform, subsequent timestamps drift slighty but consistently. By this I mean that the saved timestamps are later than the keypress waveforms and get later and later as time goes on.
I am reasonably sure the waveform timing is correct (i.e. the recording is not fast or slow), because in the recording I also included the ticking of a very accurate clock which lines up perfectly with the second markers.
I am aware that there are unavoidable limits to the accuracy of time.time(), but this does not seem to account for what I'm seeing - if it was equally wrong on both sides that would be acceptable, but I do not want it to gradually diverge more and more from the truth.
Why would I be seeing this drifting behaviour and what can I do to avoid it?
Just solved this by using time.monotonic() instead of time.time(). time.time() seems to use gettimeofday (at least here it does) which is apparently really bad for measuring walltime differences because of NTP syncing issues:
gettimeofday() and time() should only be used to get the current time if the current wall-clock time is actually what you want. They should never be used to measure time or schedule an event X time into the future.
You usually aren't running NTP on your wristwatch, so it probably won't jump a second or two (or 15 minutes) in a random direction because it happened to sync up against a proper clock at that point. Good NTP implementations try to not make the time jump like this. They instead make the clock go faster or slower so that it will drift to the correct time. But while it's drifting you either have a clock that's going too fast or too slow. It's not measuring the passage of time properly.
(link). So basically measuring differences between time.time() calls is a bad idea.
Depending on which OS you are using you will either need to use time.time() or time.clock().
For windows OS's you will need to use time.clock this give you will clock seconds as a float. time.time() on windows if I remember correctly time.time() is only accurate within 16ms.
For posix systems (linux, osx) you should be using time.time() this is a float which returns the number of seconds since the epoch.
In your code add the following to make your application a little more cross system compatible.
import os
if os.name == 'posix':
from time import time as get_time
else:
from time import clock as get_time
# now use get_time() to return the timestamp
init_timestamp = get_time()
while (True):
c = getch()
offset = get_time() - init_timestamp
print("%s,%s" % (c,offset), file=f)
...
I need to send repeating messages from a list quickly and precisely. One list needs to send the messages every 100ms, with a +/- 10ms window. I tried using the code below, but the problem is that the timer waits the 100ms, and then all the computation needs to be done, making the timer fall out of the acceptable window.
Simply decreasing the wait is a messy, and unreliable hack. The there is a Lock around the message loop in the event the list gets edited during the loop.
Thoughts on how to get python to send messages consistently around 100ms? Thanks
from threading import Timer
from threading import Lock
class RepeatingTimer(object):
def __init__(self,interval, function, *args, **kwargs):
super(RepeatingTimer, self).__init__()
self.args = args
self.kwargs = kwargs
self.function = function
self.interval = interval
self.start()
def start(self):
self.callback()
def stop(self):
self.interval = False
def callback(self):
if self.interval:
self.function(*self.args, **self.kwargs)
Timer(self.interval, self.callback, ).start()
def loop(messageList):
listLock.acquire()
for m in messageList:
writeFunction(m)
listLock.release()
MESSAGE_LIST = [] #Imagine this is populated with the messages
listLock = Lock()
rt = RepeatingTimer(0.1,loop,MESSAGE_LIST)
#Do other stuff after this
I do understand that the writeFunction will cause some delay, but not more than the 10ms allowed. I essentially need to call the function every 100ms for each message. The messagelist is small, usually less than elements.
The next challenge is to have this work with every 10ms, +/-1ms :P
Yes, the simple waiting is messy and there are better alternatives.
First off, you need a high-precision timer in Python. There are a few alternatives and depending on your OS, you might want to choose the most accurate one.
Second, you must be aware of the basics preemptive multitasking and understand that there is no high-precision sleep function, and that its actual resolution will differ from OS to OS too. For example, if we're talking Windows, the minimal sleep interval might be around 10-13 ms.
And third, remember that it's always possible to wait for a very accurate interval of time (assuming you have a high-resolution timer), but with a trade-off of high CPU load. The technique is called busy waiting:
while(True):
if time.clock() == something:
break
So, the actual solution is to create a hybrid timer. It will use the regular sleep function to wait the main bulk of the interval, and then it'll start probing the high-precision timer in the loop, while doing the sleep(0) trick. Sleep(0) will (depending on the platform) wait the least possible amount of time, releasing the rest of the remaining time slice to other processes and switching the CPU context. Here is a relevant discussion.
The idea is thoroughly described in the Ryan Geiss's Timing in Win32 article. It's in C and for Windows API, but the basic principles apply here as well.
Store the start time. Send the message. Get the end time. Calculate timeTaken=end-start. Convert to FP seconds. Sleep(0.1-timeTaken). Loop back.
try this:
#!/usr/bin/python
import time; # This is required to include time module.
from threading import Timer
def hello(start, interval, count):
ticks = time.time()
t = Timer(interval - (ticks-start-count*interval), hello, [start, interval, count+1])
t.start()
print "Number of ticks since 12:00am, January 1, 1970:", ticks, " #", count
dt = 1.25 # interval in sec
t = Timer(dt, hello, [round(time.time()), dt, 0]) # start over at full second, round only for testing here
t.start()
I need to measure the time certain parts of my program take (not for debugging but as a feature in the output). Accuracy is important because the total time will be a fraction of a second.
I was going to use the time module when I came across timeit, which claims to avoid a number of common traps for measuring execution times. Unfortunately it has an awful interface, taking a string as input which it then eval's.
So, do I need to use this module to measure time accurately, or will time suffice? And what are the pitfalls it refers to?
Thanks
According to the Python documentation, it has to do with the accuracy of the time function in different operating systems:
The default timer function is platform
dependent. On Windows, time.clock()
has microsecond granularity but
time.time()‘s granularity is 1/60th of
a second; on Unix, time.clock() has
1/100th of a second granularity and
time.time() is much more precise. On
either platform, the default timer
functions measure wall clock time, not
the CPU time. This means that other
processes running on the same computer
may interfere with the timing ... On Unix, you can
use time.clock() to measure CPU time.
To pull directly from timeit.py's code:
if sys.platform == "win32":
# On Windows, the best timer is time.clock()
default_timer = time.clock
else:
# On most other platforms the best timer is time.time()
default_timer = time.time
In addition, it deals directly with setting up the runtime code for you. If you use time you have to do it yourself. This, of course saves you time
Timeit's setup:
def inner(_it, _timer):
#Your setup code
%(setup)s
_t0 = _timer()
for _i in _it:
#The code you want to time
%(stmt)s
_t1 = _timer()
return _t1 - _t0
Python 3:
Since Python 3.3 you can use time.perf_counter() (system-wide timing) or time.process_time() (process-wide timing), just the way you used to use time.clock():
from time import process_time
t = process_time()
#do some stuff
elapsed_time = process_time() - t
The new function process_time will not include time elapsed during sleep.
Python 3.7+:
Since Python 3.7 you can also use process_time_ns() which is similar to process_time()but returns time in nanoseconds.
You could build a timing context (see PEP 343) to measure blocks of code pretty easily.
from __future__ import with_statement
import time
class Timer(object):
def __enter__(self):
self.__start = time.time()
def __exit__(self, type, value, traceback):
# Error handling here
self.__finish = time.time()
def duration_in_seconds(self):
return self.__finish - self.__start
timer = Timer()
with timer:
# Whatever you want to measure goes here
time.sleep(2)
print timer.duration_in_seconds()
The timeit module looks like it's designed for doing performance testing of algorithms, rather than as simple monitoring of an application. Your best option is probably to use the time module, call time.time() at the beginning and end of the segment you're interested in, and subtract the two numbers. Be aware that the number you get may have many more decimal places than the actual resolution of the system timer.
I was annoyed too by the awful interface of timeit so i made a library for this, check it out its trivial to use
from pythonbenchmark import compare, measure
import time
a,b,c,d,e = 10,10,10,10,10
something = [a,b,c,d,e]
def myFunction(something):
time.sleep(0.4)
def myOptimizedFunction(something):
time.sleep(0.2)
# comparing test
compare(myFunction, myOptimizedFunction, 10, input)
# without input
compare(myFunction, myOptimizedFunction, 100)
https://github.com/Karlheinzniebuhr/pythonbenchmark
Have you reviewed the functionality provided profile or cProfile?
http://docs.python.org/library/profile.html
This provides much more detailed information than just printing the time before and after a function call. Maybe worth a look...
The documentation also mentions that time.clock() and time.time() have different resolution depending on platform. On Unix, time.clock() measures CPU time as opposed to wall clock time.
timeit also disables garbage collection when running the tests, which is probably not what you want for production code.
I find that time.time() suffices for most purposes.
From Python 2.6 on timeit is not limited to input string anymore. Citing the documentation:
Changed in version 2.6: The stmt and setup parameters can now also take objects that are callable without arguments. This will embed calls to them in a timer function that will then be executed by timeit(). Note that the timing overhead is a little larger in this case because of the extra function calls.