I have a wrapper to time the execution of certain functions in a list. Most of these functions have one and the same parameter: era. I run the functions like displayed below. However, some functions require an extra parameter, e.g. the function dummy_function(). I've been looking for a way to be able to add this parameter in a Pythonic way. I found some solutions but they are very ugly and not quite scalable. Any help or suggestions would be tremendously appreciated!
def dummy_function(self, period, letter='A'):
""" Debugging purposes only """
print(f'This function prints the letter {letter}.')
from time import sleep
sleep(3)
def timed_execution(callbacks, era):
for callback in callbacks:
start_time = time.time()
callback(era)
end_time = time.time()
print(f'{callback.__name__} took {end_time-start_time:.3f}s')
def calculate_insights(era):
timed_execution([
dummy_function,
another_function,
yet_another_function,
], era)
calculate_insights(era)
Perhaps the best way is to actually pass the arguments for their respective function or just try to use a wrapper to calculate the time of a function.
Code taken from another question
from functools import wraps
from time import time
def timing(f):
#wraps(f)
def wrap(*args, **kw):
ts = time()
result = f(*args, **kw)
te = time()
print 'func:%r args:[%r, %r] took: %2.4f sec' % \
(f.__name__, args, kw, te-ts)
return result
return wrap
Then you can do something along the lines of
#timming
def dummy_function(self, period, letter='A'):
""" Debugging purposes only """
print(f'This function prints the letter {letter}.')
from time import sleep
sleep(3)
def calculate_insights():
dummy_function(era)
or you could just a dict with all the parameters passed into each callback but that doesn't sounds to pythonic for me.
Related
How could one write a debounce decorator in python which debounces not only on function called but also on the function arguments/combination of function arguments used?
Debouncing means to supress the call to a function within a given timeframe, say you call a function 100 times within 1 second but you only want to allow the function to run once every 10 seconds a debounce decorated function would run the function once 10 seconds after the last function call if no new function calls were made. Here I'm asking how one could debounce a function call with specific function arguments.
An example could be to debounce an expensive update of a person object like:
#debounce(seconds=10)
def update_person(person_id):
# time consuming, expensive op
print('>>Updated person {}'.format(person_id))
Then debouncing on the function - including function arguments:
update_person(person_id=144)
update_person(person_id=144)
update_person(person_id=144)
>>Updated person 144
update_person(person_id=144)
update_person(person_id=355)
>>Updated person 144
>>Updated person 355
So calling the function update_person with the same person_id would be supressed (debounced) until the 10 seconds debounce interval has passed without a new call to the function with that same person_id.
There's a few debounce decorators but none includes the function arguments, example: https://gist.github.com/walkermatt/2871026
I've done a similar throttle decorator by function and arguments:
def throttle(s, keep=60):
def decorate(f):
caller = {}
def wrapped(*args, **kwargs):
nonlocal caller
called_args = '{}'.format(*args)
t_ = time.time()
if caller.get(called_args, None) is None or t_ - caller.get(called_args, 0) >= s:
result = f(*args, **kwargs)
caller = {key: val for key, val in caller.items() if t_ - val > keep}
caller[called_args] = t_
return result
# Keep only calls > keep
caller = {key: val for key, val in caller.items() if t_ - val > keep}
caller[called_args] = t_
return wrapped
return decorate
The main takaway is that it keeps the function arguments in caller[called_args]
See also the difference between throttle and debounce: http://demo.nimius.net/debounce_throttle/
Update:
After some tinkering with the above throttle decorator and the threading.Timer example in the gist, I actually think this should work:
from threading import Timer
from inspect import signature
import time
def debounce(wait):
def decorator(fn):
sig = signature(fn)
caller = {}
def debounced(*args, **kwargs):
nonlocal caller
try:
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
called_args = fn.__name__ + str(dict(bound_args.arguments))
except:
called_args = ''
t_ = time.time()
def call_it(key):
try:
# always remove on call
caller.pop(key)
except:
pass
fn(*args, **kwargs)
try:
# Always try to cancel timer
caller[called_args].cancel()
except:
pass
caller[called_args] = Timer(wait, call_it, [called_args])
caller[called_args].start()
return debounced
return decorator
I've had the same need to build a debounce annotation for a personal project, after stumbling upon the same gist / discussion you have, I ended up with the following solution:
import threading
def debounce(wait_time):
"""
Decorator that will debounce a function so that it is called after wait_time seconds
If it is called multiple times, will wait for the last call to be debounced and run only this one.
"""
def decorator(function):
def debounced(*args, **kwargs):
def call_function():
debounced._timer = None
return function(*args, **kwargs)
# if we already have a call to the function currently waiting to be executed, reset the timer
if debounced._timer is not None:
debounced._timer.cancel()
# after wait_time, call the function provided to the decorator with its arguments
debounced._timer = threading.Timer(wait_time, call_function)
debounced._timer.start()
debounced._timer = None
return debounced
return decorator
I've created an open-source project to provide functions such as debounce, throttle, filter ... as decorators, contributions are more than welcome to improve on the solution I have for these decorators / add other useful decorators: decorator-operations repository
Basically I want to do something like this:
How can I hook a function in a python module?
but I want to call the old function after my own code.
like
import whatever
oldfunc = whatever.this_is_a_function
def this_is_a_function(parameter):
#my own code here
# and call original function back
oldfunc(parameter)
whatever.this_is_a_function = this_is_a_function
Is this possible?
I tried copy.copy, copy.deepcopy original function but it didn't work.
Something like this? It avoids using globals, which is generally a good thing.
import whatever
import functools
def prefix_function(function, prefunction):
#functools.wraps(function)
def run(*args, **kwargs):
prefunction(*args, **kwargs)
return function(*args, **kwargs)
return run
def this_is_a_function(parameter):
pass # Your own code here that will be run before
whatever.this_is_a_function = prefix_function(
whatever.this_is_a_function, this_is_a_function)
prefix_function is a function that takes two functions: function and prefunction. It returns a function that takes any parameters, and calls prefunction followed by function with the same parameters. The prefix_function function works for any callable, so you only need to program the prefixing code once for any other hooking you might need to do.
#functools.wraps makes it so that the docstring and name of the returned wrapper function is the same.
If you need this_is_a_function to call the old whatever.this_is_a_function with arguments different than what was passed to it, you could do something like this:
import whatever
import functools
def wrap_function(oldfunction, newfunction):
#functools.wraps(function)
def run(*args, **kwargs):
return newfunction(oldfunction, *args, **kwargs)
return run
def this_is_a_function(oldfunc, parameter):
# Do some processing or something to customize the parameters to pass
newparams = parameter * 2 # Example of a change to newparams
return oldfunc(newparams)
whatever.this_is_a_function = wrap_function(
whatever.this_is_a_function, this_is_a_function)
There is a problem that if whatever is a pure C module, it's typically impossible (or very difficult) to change its internals in the first place.
So, here's an example of monkey-patching the time function from the time module.
import time
old_time = time.time
def time():
print('It is today... but more specifically the time is:')
return old_time()
time.time = time
print time.time()
# Output:
# It is today... but more specifically the time is:
# 1456954003.2
However, if you are trying to do this to C code, you will most likely get an error like cannot overwrite attribute. In that case, you probably want to subclass the C module.
You may want to take a look at this question.
This is the perfect time to tout my super-simplistic Hooker
def hook(hookfunc, oldfunc):
def foo(*args, **kwargs):
hookfunc(*args, **kwargs)
return oldfunc(*args, **kwargs)
return foo
Incredibly simple. It will return a function that first runs the desired hook function (with the same parameters, mind you) and will then run the original function that you are hooking and return that original value. This also works to overwrite a class method. Say we have static method in a class.
class Foo:
#staticmethod
def bar(data):
for datum in data:
print(datum, end="") # assuming python3 for this
print()
But we want to print the length of the data before we print out its elements
def myNewFunction(data):
print("The length is {}.".format(len(data)))
And now we simple hook the function
Foo.bar(["a", "b", "c"])
# => a b c
Foo.bar = hook(Foo.bar, myNewFunction)
Foo.bar(["x", "y", "z"])
# => The length is 3.
# => x y z
Actually, you can replace the target function's func_code. The example below
# a normal function
def old_func():
print "i am old"
# a class method
class A(object):
def old_method(self):
print "i am old_method"
# a closure function
def make_closure(freevar1, freevar2):
def wrapper():
print "i am old_clofunc, freevars:", freevar1, freevar2
return wrapper
old_clofunc = make_closure('fv1', 'fv2')
# ===============================================
# the new function
def new_func(*args):
print "i am new, args:", args
# the new closure function
def make_closure2(freevar1, freevar2):
def wrapper():
print "i am new_clofunc, freevars:", freevar1, freevar2
return wrapper
new_clofunc = make_closure2('fv1', 'fv2')
# ===============================================
# hook normal function
old_func.func_code = new_func.func_code
# hook class method
A.old_method.im_func.func_code = new_func.func_code
# hook closure function
# Note: the closure function's `co_freevars` count should be equal
old_clofunc.func_code = new_clofunc.func_code
# ===============================================
# call the old
old_func()
A().old_method()
old_clofunc()
output:
i am new, args: ()
i am new, args: (<__main__.A object at 0x0000000004A5AC50>,)
i am new_clofunc, freevars: fv1 fv2
I have a lot of necessary time.sleep() in my script. I want to clean up the code and possibly append the wait/pause to the previous line instead of creating a new line. Example:
call(['networksetup', '-setv4off', 'direct IP'])
time.sleep(5)
Is there any way of achieving this? (Python 3.4)
Thanks in advance.
If you mean combine these two lines in one line, try:
call(['networksetup', '-setv4off', 'direct IP']); time.sleep(5)
If you unconditionally want to be able to sleep after a particular function (or group of functions), the easiest way of doing this that I'm aware of is to use a decorator.
from functools import wraps
import time
def sleep_decorator(f, duration = 5.0):
#wraps(f)
def wrapper(*args, **kwargs):
result = f(*args, **kwargs)
time.sleep( duration )
return result
return wrapper
#sleep_decorator
def call(*args, **kwargs):
return None
call(['networksetup', '-setv4off', 'direct IP'])
The issue with that solution is that the duration of the sleep can't be altered, and you can't enable and disable the sleep call on a case-by-case basis.
To enhance the solution you can add arguments to your decorator.
from functools import wraps
import time
def sleep_decorator(duration = 5.0, active = True):
def decorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
result = f(*args, **kwargs)
sleep_duration = kwargs.get('duration', duration)
sleep_active = kwargs.get('active', active)
if sleep_active:
time.sleep( sleep_duration )
return result
return wrapper
return decorator
Which means you could now write you code as follows:
#sleep_decorator(active = False, duration = 0.1)
def call(*args, **kwargs):
return None
call(['networksetup', '-setv4off', 'direct IP'])
call(['networksetup', '-setv4off', 'direct IP'], active = True)
call(['networksetup', '-setv4off', 'direct IP'], active = True, duration = 1.0)
One other alternative that can let you inject code is to rewrite the Abstract Syntax Tree of your code at runtime. This would allow you to inject arbitrary function calls. Libraries like numba and pony use tricks like this to transform Python code into something domain specific.
Is there any way to get the total amount of time that "unittest.TextTestRunner().run()" has taken to run a specific unit test.
I'm using a for loop to test modules against certain scenarios (some having to be used and some not, so they run a few times), and I would like to print the total time it has taken to run all the tests.
Any help would be greatly appreciated.
UPDATED, thanks to #Centralniak's comment.
How about simple
from datetime import datetime
tick = datetime.now()
# run the tests here
tock = datetime.now()
diff = tock - tick # the result is a datetime.timedelta object
print(diff.total_seconds())
You could record start time in the setup function and then print elapsed time in cleanup.
Following Eric's one-line answer I have a little snippet I work with here:
from datetime import datetime
class SomeTests(unittest.TestCase):
"""
... write the rest yourself! ...
"""
def setUp(self):
self.tick = datetime.now()
def tearDown(self):
self.tock = datetime.now()
diff = self.tock - self.tick
print (diff.microseconds / 1000), "ms"
# all the other tests below
This works fine enough for me, for now, but I want to fix some minor formatting issues. The result ok is now on the next line, and FAIL has priority. This is ugly.
I do this exactly as Eric postulated -- here's a decorator I use for tests (often more functional-test-y than strict unit tests)...
# -*- coding: utf-8 -*-
from __future__ import print_function
from functools import wraps
from pprint import pprint
WIDTH = 60
print_separator = lambda fill='-', width=WIDTH: print(fill * width)
def timedtest(function):
"""
Functions so decorated will print the time they took to execute.
Usage:
import unittest
class MyTests(unittest.TestCase):
#timedtest
def test_something(self):
assert something is something_else
# … etc
# An optional return value is pretty-printed,
# along with the timing values:
return another_thing
"""
#wraps(function)
def wrapper(*args, **kwargs):
print()
print("TESTING: %s(…)" % getattr(function, "__name__", "<unnamed>"))
print_separator()
print()
t1 = time.time()
out = function(*args, **kwargs)
t2 = time.time()
dt = str((t2 - t1) * 1.00)
dtout = dt[:(dt.find(".") + 4)]
print_separator()
if out is not None:
print('RESULTS:')
pprint(out, indent=4)
print('Test finished in %s seconds' % dtout)
print_separator('=')
return out
return wrapper
That's the core of it -- from there, if you want, you can stash the times in a database for analysis, or draw graphs, et cetera. A decorator like this (using #wraps(…) from the functools module) won't interfere with any of the dark magic that unit-test frameworks occasionally resort to.
Besides using datetime, you could also use time
from time import time
t0 = time()
# do your stuff here
print(time() - t0) # it will show in seconds
I originally made a custom function for timing functions that looks like this:
def timefunc(function, *args):
start = time.time()
data = function(*args)
end = time.time()
time_taken = end - start
print "Function: "+function.__name__
print "Time taken:",time_taken
return data
Now, having learned about the timeit module, I want to implement the same thing using that. I just can't figure out how to do it while sending it the function and *args arguments. I have already figured out that I have to do this in the setup arg:
"from __main__ import function"
But I can't figure out what to do about *args, since both the 'stmt' and 'setup' arguments are strings, how can I pass the variables?
Starting with Python 2.6 the timeit module besides a string also accepts a callable as stmt. With that in mind you can do this:
import timeit
def timefunc(function, *args):
def wrap():
function(*args)
t = timeit.Timer(wrap)
return t.timeit(100)
def test(arg):
foo = arg
print(timefunc(test, 'bar'))