This question already has answers here:
How can I time a code segment for testing performance with Pythons timeit?
(9 answers)
Closed 6 years ago.
I'm trying to measure the time it takes to run a block of instructions in Python, but I don't want to write things like:
start = time.clock()
...
<lines of code>
...
time_elapsed = time.clock() - start
Instead, I want to know if there is a way I can send the block of instructions as a parameter to a function that returns the elapsed time, like
time_elapsed = time_it_takes(<lines of code>)
The implementation of this method could be something like
def time_it_takes(<lines of code>):
start = time.clock()
result = <lines of code>
return (result, time.clock() - start)
Does anybody know if there is some way I can do this? Thanks in advance.
This would be a good use of a decorator. You could write a decorator that does that like this
import time
def timer(func):
def wrapper(*args, **kwargs):
start = time.time()
func(*args, **kwargs)
print('The function ran for', time.time() - start)
return wrapper
#timer
def just_sleep():
time.sleep(5)
just_sleep()
Output
The function ran for 5.0050904750823975
and then you can decorate any function you want to time with #timer and you can also do some other fancy things inside the decorator. Like if the function ran for more than 15 seconds do something...else do another thing
Note: This is not the most accurate way to measure execution time of a function in python
You can build your own context manager to time relatively long bits of code.
import time
class MyTimer(object):
def __enter__(self):
self.start = time.clock()
return self
def __exit__(self, typ, value, traceback):
self.duration = time.clock() - self.start
with MyTimer() as timer:
time.sleep(3)
print(timer.duration)
But be careful about what you are measuring. On Linux time.clock is cpu run time, but on Windows (where cpu run time is not easily available) it is a wall-clock.
If you use IPython, and it's a good thing to do, and you can construct your code to be a single line, i.e. a function call:
%timeit your-code
It's been handy for me. Hope it helps.
use python -m cProfile myscript.py It provides a full log about time consumption of methods.
Related
Specification of the problem:
I'm searching through really great amount of lines of a log file and I'm distributing those lines to groups in order to regular expressions(RegExses) I have stored using the re.match() function. Unfortunately some of my RegExses are too complicated and Python sometimes gets himself to backtracking hell. Due to this I need to protect it with some kind of timeout.
Problems:
re.match, I'm using, is Python's function and as I found out somewhere here on StackOverflow (I'm really sorry, I can not find the link now :-( ). It is very difficult to interrupt thread with running Python's library. For this reason threads are out of the game.
Because evaluating of re.match function takes relatively short time and I want to analyse with this function great amount of lines, I need some timeout function that wont't take too long to execute (this makes threads even less suitable, it takes really long time to initialise new thread) and can be set to less than one second.
For those reasons, answers here - Timeout on a function call
and here - Timeout function if it takes too long to finish with decorator (alarm - 1sec and more) are off the table.
I've spent this morning searching for solution to this question but I did not find any satisfactory answer.
Solution:
I've just modified a script posted here: Timeout function if it takes too long to finish.
And here is the code:
from functools import wraps
import errno
import os
import signal
class TimeoutError(Exception):
pass
def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
def decorator(func):
def _handle_timeout(signum, frame):
raise TimeoutError(error_message)
def wrapper(*args, **kwargs):
signal.signal(signal.SIGALRM, _handle_timeout)
signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
try:
result = func(*args, **kwargs)
finally:
signal.alarm(0)
return result
return wraps(func)(wrapper)
return decorator
And then you can use it like this:
from timeout import timeout
from time import time
#timeout(0.01)
def loop():
while True:
pass
try:
begin = time.time()
loop()
except TimeoutError, e:
print "Time elapsed: {:.3f}s".format(time.time() - begin)
Which prints
Time elapsed: 0.010s
Specification of the problem:
I'm searching through really great amount of lines of a log file and I'm distributing those lines to groups in order to regular expressions(RegExses) I have stored using the re.match() function. Unfortunately some of my RegExses are too complicated and Python sometimes gets himself to backtracking hell. Due to this I need to protect it with some kind of timeout.
Problems:
re.match, I'm using, is Python's function and as I found out somewhere here on StackOverflow (I'm really sorry, I can not find the link now :-( ). It is very difficult to interrupt thread with running Python's library. For this reason threads are out of the game.
Because evaluating of re.match function takes relatively short time and I want to analyse with this function great amount of lines, I need some timeout function that wont't take too long to execute (this makes threads even less suitable, it takes really long time to initialise new thread) and can be set to less than one second.
For those reasons, answers here - Timeout on a function call
and here - Timeout function if it takes too long to finish with decorator (alarm - 1sec and more) are off the table.
I've spent this morning searching for solution to this question but I did not find any satisfactory answer.
Solution:
I've just modified a script posted here: Timeout function if it takes too long to finish.
And here is the code:
from functools import wraps
import errno
import os
import signal
class TimeoutError(Exception):
pass
def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
def decorator(func):
def _handle_timeout(signum, frame):
raise TimeoutError(error_message)
def wrapper(*args, **kwargs):
signal.signal(signal.SIGALRM, _handle_timeout)
signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
try:
result = func(*args, **kwargs)
finally:
signal.alarm(0)
return result
return wraps(func)(wrapper)
return decorator
And then you can use it like this:
from timeout import timeout
from time import time
#timeout(0.01)
def loop():
while True:
pass
try:
begin = time.time()
loop()
except TimeoutError, e:
print "Time elapsed: {:.3f}s".format(time.time() - begin)
Which prints
Time elapsed: 0.010s
So, I am interested in timing some of the code I am setting up. Borrowing a timer function from the 4th edition of Learning Python, I tried:
import time
reps = 100
repslist = range(reps)
def timer(func):
start = time.clock()
for i in repslist:
ret = func()
elasped = time.clock()-start
return elapsed
Then, I paste in whatever I want to time, and put:
print(timer(func)) #replace func with the function you want to time
When I run it on my code, I do get an answer, but it's nonsense. Suspecting something was wrong, I put a time.sleep(0.1) call in my code, and got a result of 0.8231
Does anybody know why this might be the case or how to fix it? I suspect that the time.clock() call might be at fault.
According to the help docs for clock:
Return the CPU time or real time since the start of the process or since the first call to clock(). This has as much precision as the system records.
The second call to clock already returns the elapsed time between it and the first clock call. You don't need to manually subtract start.
Change
elasped = time.clock()-start
to
elasped = time.clock()
If you want to timer a function perhaps give decorators a try(documentation here):
import time
def timeit(f):
def timed(*args, **kw):
ts = time.time()
result = f(*args, **kw)
te = time.time()
print 'func:%r args:[%r, %r] took: %2.4f sec' % \
(f.__name__, args, kw, te-ts)
return result
return timed
Then when you write a function you just use the decorator, here:
#timeit
def my_example_function():
for i in range(10000):
print "x"
This will print out the time the function took to execute:
func:'my_example_function' args:[(), {}] took: 0.4220 sec
After fixing the typo in the first intended use of elapsed, your code works fine with either time.clock or time.time (or Py3's time.monotonic for that matter) on my Linux system.
The difference would be in the (OS specific) behavior for clock; on most UNIX-like OSes it will return the processor time used by the program since it launched (so time spent blocked, on I/O, locks, page faults, etc. wouldn't count), while on Windows it's a wall clock timer (so time spent blocked would count) that counts seconds since first call.
The UNIX-like version of time.clock is also fairly unreliable if used in a long running program when clock_t is only 32 bits; the value it returns will wrap roughly every 72 minutes of processor time.
Of course, time.time isn't perfect either; it follows the system clock, so an NTP time update (or any other change to the system clock) occurring between calls will give erroneous results (on Python 3.3+, you'd use time.monotonic to avoid this problem). It's also not guaranteed to have granularity finer than 1 second, so if your function doesn't take an awfully long time to run, on a system with low res time.time you won't get particularly useful results.
Really, you should be looking at the Python batteries designed for this (that also handle issues like garbage collection overhead and the like). The timeit module already has a function that does what you want, but handles all the edge cases and issues I mentioned. For example, to time some global function named foo for 100 reps, you'd just do:
import timeit
def foo():
...
print(timeit.timeit('foo()', 'from __main__ import foo', number=100))
It fixes most of the issues I mention by selecting the best timing function for the OS you're on (and also fixes other sources of jitter, e.g. cyclic garbage collection, which is disabled during the test and reenabled at the end).
Even if you don't want to use that for some reason, if you're using Python 3.3 or higher, take a look at the replacements for time.clock, e.g. time.perf_counter (includes time spent sleeping) or time.process_time (includes only CPU time), both of which are portable, reliable, fast, and high resolution for better accuracy.
The time.sleep() will terminate for any signal. read about it here ...
http://www.tutorialspoint.com/python/time_sleep.htm
When I run my Python script, there is some function that takes up to a few minutes to complete, so I want to display on the shell some kind of timer that informs the user on the time elapsed.
Is there any such thing already ready to use in Python?
One simplistic way is to include a clock in your sys.ps1 prompt (the thing that normally defines the >>> prompt)
From the documentation for sys.ps1:
If a non-string object is assigned to either variable, its str() is re-evaluated each time the interpreter prepares to read a new interactive command; this can be used to implement a dynamic prompt.
In ~/.local/usercustomize.py (or more accurately, in whatever folder python -c 'import site; print site.USER_BASE' displays), you can add:
import sys
import datetime
class ClockPS1(object):
def __repr__(self):
now = datetime.datetime.now()
return str(now.strftime("%H:%M:%S >>> "))
sys.ps1 = ClockPS1()
Then your prompt will look like this:
16:26:24 >>> import time
16:26:27 >>> time.sleep(10)
16:26:40 >>>
It's not perfect, as the last time will be when the prompt appeared, not when the line was executed, but it might be of help. You could easily make this display the time in seconds between __repr__ invokations, and show that in the prompt.
If you are on a Linux or BSD system, try the pv command (http://www.ivarch.com/programs/pv.shtml).
$ python -c 'import time;time.sleep(5)' | pv
0B 0:00:05 [ 0B/s ] [<=> ]
It will give you a timer and depending on how you code the output of your app, some other stats as well.
The simplest way to do it would be to calculate the elapsed time in the function that takes a few minutes to complete and simply print that time to the shell. However depending on your function this probably is not the best solution.
The second way to do it would to use multi-threading. So have the function that takes awhile run in a thread, while your program then sits in a loop and prints out the elapsed time every so often and looks for the thread to be completed.
Something like:
import threading
import time
arg1=0
arg2=1
etc=2
# your function that takes a while.
# Note: If your function returns something or if you want to pass variables in/out,
# you have to use Queues
def yourFunction(arg1,arg2,etc):
time.sleep(10) #your code would replace this
# Setup the thread
processthread=threading.Thread(target=yourFunction,args=(arg1,arg1,etc)) #set the target function and any arguments to pass
processthread.daemon=True
processthread.start() # start the thread
#loop to check thread and display elapsed time
while processthread.isAlive():
print time.clock()
time.sleep(1) # you probably want to only print every so often (i.e. every second)
print 'Done'
You can then get fancier by overwriting the time in the shell or even better, using a gui to display a progress bar!
Are you talking about measuring the time it takes a function to complete and then print the HH:MM:SS.MS?
You can do:
import datetime, time
time_begin = datetime.datetime.fromtimestamp(time.time())
# call your function here
time_end = datetime.datetime.fromtimestamp(time.time())
print("Time elapsed: ", str(time_end - time_begin))
you can use datetime
for example,
import datetime
import time
class Profiler(object):
def __init__(self):
self.start = 0
self.duration = 0
def start(self):
self.start = datetime.datetime.now()
def end(self):
self.duration = datetime.datetime.now() - self.start
class SomeClass(Profiler):
def __init__(self):
Profiler.__init__(self)
def do_someting(self):
self.start()
time.sleep(10)
self.end()
if __name__ == "__main__":
foo = SomeClass()
foo.do_something()
print 'Time :: ', foo.duration
I need to have a base class which I will use to inherit other classes which I would like to measure execution time of its functions.
So intead of having something like this:
class Worker():
def doSomething(self):
start = time.time()
... do something
elapsed = (time.time() - start)
print "doSomething() took ", elapsed, " time to finish"
#outputs: doSomething() took XX time to finish
I would like to have something like this:
class Worker(BaseClass):
def doSomething(self):
... do something
#outputs the same: doSomething() took XX time to finish
So the BaseClass needs to dealing with measuring time
One way to do this would be with a decorator (PEP for decorators) (first of a series of tutorial articles on decorators). Here's an example that does what you want.
from functools import wraps
from time import time
def timed(f):
#wraps(f)
def wrapper(*args, **kwds):
start = time()
result = f(*args, **kwds)
elapsed = time() - start
print "%s took %d time to finish" % (f.__name__, elapsed)
return result
return wrapper
This is an example of its use
#timed
def somefunction(countto):
for i in xrange(countto):
pass
return "Done"
To show how it works I called the function from the python prompt:
>>> timedec.somefunction(10000000)
somefunction took 0 time to finish
'Done'
>>> timedec.somefunction(100000000)
somefunction took 2 time to finish
'Done'
>>> timedec.somefunction(1000000000)
somefunction took 22 time to finish
'Done'
Have you checked the "profile" module?
I.e. are you sure you need to implement your own custom framework instead of using the default profiling mechanism for the language?
You could also google for "python hotshot" for a similar solution.
There is also timeit, which is part of the standard library, and is really easy to use. Remember: don't reinvent the wheel!