I need to have a base class which I will use to inherit other classes which I would like to measure execution time of its functions.
So intead of having something like this:
class Worker():
def doSomething(self):
start = time.time()
... do something
elapsed = (time.time() - start)
print "doSomething() took ", elapsed, " time to finish"
#outputs: doSomething() took XX time to finish
I would like to have something like this:
class Worker(BaseClass):
def doSomething(self):
... do something
#outputs the same: doSomething() took XX time to finish
So the BaseClass needs to dealing with measuring time
One way to do this would be with a decorator (PEP for decorators) (first of a series of tutorial articles on decorators). Here's an example that does what you want.
from functools import wraps
from time import time
def timed(f):
#wraps(f)
def wrapper(*args, **kwds):
start = time()
result = f(*args, **kwds)
elapsed = time() - start
print "%s took %d time to finish" % (f.__name__, elapsed)
return result
return wrapper
This is an example of its use
#timed
def somefunction(countto):
for i in xrange(countto):
pass
return "Done"
To show how it works I called the function from the python prompt:
>>> timedec.somefunction(10000000)
somefunction took 0 time to finish
'Done'
>>> timedec.somefunction(100000000)
somefunction took 2 time to finish
'Done'
>>> timedec.somefunction(1000000000)
somefunction took 22 time to finish
'Done'
Have you checked the "profile" module?
I.e. are you sure you need to implement your own custom framework instead of using the default profiling mechanism for the language?
You could also google for "python hotshot" for a similar solution.
There is also timeit, which is part of the standard library, and is really easy to use. Remember: don't reinvent the wheel!
Related
I have a threaded code in python i need to find out the time between execution of a function a and again re execution of the same function. How to to do so ??
I have tried to use timeit module but getting wrong results
minimal code is
def abc1 ():
process
def abc2():
process
def abc3:
process
As above there are many functions which are threaded I want to find if function abc1 is executed at time 0 then after how much time abc1 will be executed again
If you are comfortable with the concept of decorators and thread-safety, here is a solution:
import time
import threading
# Let us define a list that stores the execution start times
exec_history = []
# Define a lock which allows safe access to this list
list_lock = threading.Lock()
# Define a decorator, which will "log" the execution of its function to the list
def log_func_call(fn):
def wrapper(*args, **kwargs):
with list_lock:
# Note down which function and when it was called.
# Alternatively, you can also log to a file or terminal, etc.
exec_history.append((fn, time.time()))
return fn(*args, **kwargs)
return wrapper
# Decorate whichever method you want to track
#log_func_call
def abc1():
pass
#log_func_call
def abc2():
pass
The exec_history will get updated as and when any function is called, along with its start time. Is this what you are looking for?
Specification of the problem:
I'm searching through really great amount of lines of a log file and I'm distributing those lines to groups in order to regular expressions(RegExses) I have stored using the re.match() function. Unfortunately some of my RegExses are too complicated and Python sometimes gets himself to backtracking hell. Due to this I need to protect it with some kind of timeout.
Problems:
re.match, I'm using, is Python's function and as I found out somewhere here on StackOverflow (I'm really sorry, I can not find the link now :-( ). It is very difficult to interrupt thread with running Python's library. For this reason threads are out of the game.
Because evaluating of re.match function takes relatively short time and I want to analyse with this function great amount of lines, I need some timeout function that wont't take too long to execute (this makes threads even less suitable, it takes really long time to initialise new thread) and can be set to less than one second.
For those reasons, answers here - Timeout on a function call
and here - Timeout function if it takes too long to finish with decorator (alarm - 1sec and more) are off the table.
I've spent this morning searching for solution to this question but I did not find any satisfactory answer.
Solution:
I've just modified a script posted here: Timeout function if it takes too long to finish.
And here is the code:
from functools import wraps
import errno
import os
import signal
class TimeoutError(Exception):
pass
def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
def decorator(func):
def _handle_timeout(signum, frame):
raise TimeoutError(error_message)
def wrapper(*args, **kwargs):
signal.signal(signal.SIGALRM, _handle_timeout)
signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
try:
result = func(*args, **kwargs)
finally:
signal.alarm(0)
return result
return wraps(func)(wrapper)
return decorator
And then you can use it like this:
from timeout import timeout
from time import time
#timeout(0.01)
def loop():
while True:
pass
try:
begin = time.time()
loop()
except TimeoutError, e:
print "Time elapsed: {:.3f}s".format(time.time() - begin)
Which prints
Time elapsed: 0.010s
This question already has answers here:
How can I time a code segment for testing performance with Pythons timeit?
(9 answers)
Closed 6 years ago.
I'm trying to measure the time it takes to run a block of instructions in Python, but I don't want to write things like:
start = time.clock()
...
<lines of code>
...
time_elapsed = time.clock() - start
Instead, I want to know if there is a way I can send the block of instructions as a parameter to a function that returns the elapsed time, like
time_elapsed = time_it_takes(<lines of code>)
The implementation of this method could be something like
def time_it_takes(<lines of code>):
start = time.clock()
result = <lines of code>
return (result, time.clock() - start)
Does anybody know if there is some way I can do this? Thanks in advance.
This would be a good use of a decorator. You could write a decorator that does that like this
import time
def timer(func):
def wrapper(*args, **kwargs):
start = time.time()
func(*args, **kwargs)
print('The function ran for', time.time() - start)
return wrapper
#timer
def just_sleep():
time.sleep(5)
just_sleep()
Output
The function ran for 5.0050904750823975
and then you can decorate any function you want to time with #timer and you can also do some other fancy things inside the decorator. Like if the function ran for more than 15 seconds do something...else do another thing
Note: This is not the most accurate way to measure execution time of a function in python
You can build your own context manager to time relatively long bits of code.
import time
class MyTimer(object):
def __enter__(self):
self.start = time.clock()
return self
def __exit__(self, typ, value, traceback):
self.duration = time.clock() - self.start
with MyTimer() as timer:
time.sleep(3)
print(timer.duration)
But be careful about what you are measuring. On Linux time.clock is cpu run time, but on Windows (where cpu run time is not easily available) it is a wall-clock.
If you use IPython, and it's a good thing to do, and you can construct your code to be a single line, i.e. a function call:
%timeit your-code
It's been handy for me. Hope it helps.
use python -m cProfile myscript.py It provides a full log about time consumption of methods.
Specification of the problem:
I'm searching through really great amount of lines of a log file and I'm distributing those lines to groups in order to regular expressions(RegExses) I have stored using the re.match() function. Unfortunately some of my RegExses are too complicated and Python sometimes gets himself to backtracking hell. Due to this I need to protect it with some kind of timeout.
Problems:
re.match, I'm using, is Python's function and as I found out somewhere here on StackOverflow (I'm really sorry, I can not find the link now :-( ). It is very difficult to interrupt thread with running Python's library. For this reason threads are out of the game.
Because evaluating of re.match function takes relatively short time and I want to analyse with this function great amount of lines, I need some timeout function that wont't take too long to execute (this makes threads even less suitable, it takes really long time to initialise new thread) and can be set to less than one second.
For those reasons, answers here - Timeout on a function call
and here - Timeout function if it takes too long to finish with decorator (alarm - 1sec and more) are off the table.
I've spent this morning searching for solution to this question but I did not find any satisfactory answer.
Solution:
I've just modified a script posted here: Timeout function if it takes too long to finish.
And here is the code:
from functools import wraps
import errno
import os
import signal
class TimeoutError(Exception):
pass
def timeout(seconds=10, error_message=os.strerror(errno.ETIME)):
def decorator(func):
def _handle_timeout(signum, frame):
raise TimeoutError(error_message)
def wrapper(*args, **kwargs):
signal.signal(signal.SIGALRM, _handle_timeout)
signal.setitimer(signal.ITIMER_REAL,seconds) #used timer instead of alarm
try:
result = func(*args, **kwargs)
finally:
signal.alarm(0)
return result
return wraps(func)(wrapper)
return decorator
And then you can use it like this:
from timeout import timeout
from time import time
#timeout(0.01)
def loop():
while True:
pass
try:
begin = time.time()
loop()
except TimeoutError, e:
print "Time elapsed: {:.3f}s".format(time.time() - begin)
Which prints
Time elapsed: 0.010s
When I run my Python script, there is some function that takes up to a few minutes to complete, so I want to display on the shell some kind of timer that informs the user on the time elapsed.
Is there any such thing already ready to use in Python?
One simplistic way is to include a clock in your sys.ps1 prompt (the thing that normally defines the >>> prompt)
From the documentation for sys.ps1:
If a non-string object is assigned to either variable, its str() is re-evaluated each time the interpreter prepares to read a new interactive command; this can be used to implement a dynamic prompt.
In ~/.local/usercustomize.py (or more accurately, in whatever folder python -c 'import site; print site.USER_BASE' displays), you can add:
import sys
import datetime
class ClockPS1(object):
def __repr__(self):
now = datetime.datetime.now()
return str(now.strftime("%H:%M:%S >>> "))
sys.ps1 = ClockPS1()
Then your prompt will look like this:
16:26:24 >>> import time
16:26:27 >>> time.sleep(10)
16:26:40 >>>
It's not perfect, as the last time will be when the prompt appeared, not when the line was executed, but it might be of help. You could easily make this display the time in seconds between __repr__ invokations, and show that in the prompt.
If you are on a Linux or BSD system, try the pv command (http://www.ivarch.com/programs/pv.shtml).
$ python -c 'import time;time.sleep(5)' | pv
0B 0:00:05 [ 0B/s ] [<=> ]
It will give you a timer and depending on how you code the output of your app, some other stats as well.
The simplest way to do it would be to calculate the elapsed time in the function that takes a few minutes to complete and simply print that time to the shell. However depending on your function this probably is not the best solution.
The second way to do it would to use multi-threading. So have the function that takes awhile run in a thread, while your program then sits in a loop and prints out the elapsed time every so often and looks for the thread to be completed.
Something like:
import threading
import time
arg1=0
arg2=1
etc=2
# your function that takes a while.
# Note: If your function returns something or if you want to pass variables in/out,
# you have to use Queues
def yourFunction(arg1,arg2,etc):
time.sleep(10) #your code would replace this
# Setup the thread
processthread=threading.Thread(target=yourFunction,args=(arg1,arg1,etc)) #set the target function and any arguments to pass
processthread.daemon=True
processthread.start() # start the thread
#loop to check thread and display elapsed time
while processthread.isAlive():
print time.clock()
time.sleep(1) # you probably want to only print every so often (i.e. every second)
print 'Done'
You can then get fancier by overwriting the time in the shell or even better, using a gui to display a progress bar!
Are you talking about measuring the time it takes a function to complete and then print the HH:MM:SS.MS?
You can do:
import datetime, time
time_begin = datetime.datetime.fromtimestamp(time.time())
# call your function here
time_end = datetime.datetime.fromtimestamp(time.time())
print("Time elapsed: ", str(time_end - time_begin))
you can use datetime
for example,
import datetime
import time
class Profiler(object):
def __init__(self):
self.start = 0
self.duration = 0
def start(self):
self.start = datetime.datetime.now()
def end(self):
self.duration = datetime.datetime.now() - self.start
class SomeClass(Profiler):
def __init__(self):
Profiler.__init__(self)
def do_someting(self):
self.start()
time.sleep(10)
self.end()
if __name__ == "__main__":
foo = SomeClass()
foo.do_something()
print 'Time :: ', foo.duration