My goal is to create a function that I can use to measure the execution and resource use of another function. Using a tutorial, I've create the below using Python's ThreadPoolExecutor:
from resource import *
from time import sleep
from concurrent.futures import ThreadPoolExecutor
class MemoryMonitor:
def __init__(self):
self.keep_measuring = True
def measure_usage(self):
max_usage = 0
u_run_time = 0
s_run_time = 0
while self.keep_measuring:
max_usage = max(max_usage, getrusage(RUSAGE_SELF).ru_maxrss)
u_run_time = max(u_run_time, getrusage(RUSAGE_SELF).ru_utime)
s_run_time = max(s_run_time, getrusage(RUSAGE_SELF).ru_stime)
sleep(0.1) # run this loop every 0.1 seconds
return [max_usage, u_run_time, s_run_time]
def execute(function):
with ThreadPoolExecutor() as executor:
monitor = MemoryMonitor()
stats_thread = executor.submit(monitor.measure_usage)
try:
fn_thread = executor.submit(function)
result = fn_thread.result()
print("print result")
print(result)
print("print result type")
print(type(result))
finally:
monitor.keep_measuring = False
stats = stats_thread.result()
print(stats)
return result
def foo():
i = 0
while i < 3:
print("foo")
i+=1
return 1
def bar(x):
while x < 3:
print("foobar")
x+=1
return 1
var = execute(foo)
print("Var = " + str(var))
var = execute(bar(0))
print("Var = " + str(var))
If I pass the function foo as an argument to the function execute, it prints the correct results and returns the value returned by foo.
If I pass the function bar in the same way, but with bar itself requiring an argument, the function runs (prints 3 times) and then I get the following error:
result = self.fn(*self.args, **self.kwargs)
TypeError: 'int' object is not callable
After some testing, the part where I'm stuck appears to be passing a function as an argument, if that function itself requires an argument. As I understand the ThreadPoolExecutor, the fn_thread object encapsulates the execution of the function submitted. The result object should simply hold the result of that execution - what am I missing that this cannot handle being passed a function with an argument?
You are submitting
bar(0)
instead of
bar, 0
To clarify, look at the submit's signature:
submit(fn, *args, **kwargs)
the result of
bar(0)
is an integer, and the executor cannot call an integer, since it is not 'callable', as the error message suggests.
Related
How could one write a debounce decorator in python which debounces not only on function called but also on the function arguments/combination of function arguments used?
Debouncing means to supress the call to a function within a given timeframe, say you call a function 100 times within 1 second but you only want to allow the function to run once every 10 seconds a debounce decorated function would run the function once 10 seconds after the last function call if no new function calls were made. Here I'm asking how one could debounce a function call with specific function arguments.
An example could be to debounce an expensive update of a person object like:
#debounce(seconds=10)
def update_person(person_id):
# time consuming, expensive op
print('>>Updated person {}'.format(person_id))
Then debouncing on the function - including function arguments:
update_person(person_id=144)
update_person(person_id=144)
update_person(person_id=144)
>>Updated person 144
update_person(person_id=144)
update_person(person_id=355)
>>Updated person 144
>>Updated person 355
So calling the function update_person with the same person_id would be supressed (debounced) until the 10 seconds debounce interval has passed without a new call to the function with that same person_id.
There's a few debounce decorators but none includes the function arguments, example: https://gist.github.com/walkermatt/2871026
I've done a similar throttle decorator by function and arguments:
def throttle(s, keep=60):
def decorate(f):
caller = {}
def wrapped(*args, **kwargs):
nonlocal caller
called_args = '{}'.format(*args)
t_ = time.time()
if caller.get(called_args, None) is None or t_ - caller.get(called_args, 0) >= s:
result = f(*args, **kwargs)
caller = {key: val for key, val in caller.items() if t_ - val > keep}
caller[called_args] = t_
return result
# Keep only calls > keep
caller = {key: val for key, val in caller.items() if t_ - val > keep}
caller[called_args] = t_
return wrapped
return decorate
The main takaway is that it keeps the function arguments in caller[called_args]
See also the difference between throttle and debounce: http://demo.nimius.net/debounce_throttle/
Update:
After some tinkering with the above throttle decorator and the threading.Timer example in the gist, I actually think this should work:
from threading import Timer
from inspect import signature
import time
def debounce(wait):
def decorator(fn):
sig = signature(fn)
caller = {}
def debounced(*args, **kwargs):
nonlocal caller
try:
bound_args = sig.bind(*args, **kwargs)
bound_args.apply_defaults()
called_args = fn.__name__ + str(dict(bound_args.arguments))
except:
called_args = ''
t_ = time.time()
def call_it(key):
try:
# always remove on call
caller.pop(key)
except:
pass
fn(*args, **kwargs)
try:
# Always try to cancel timer
caller[called_args].cancel()
except:
pass
caller[called_args] = Timer(wait, call_it, [called_args])
caller[called_args].start()
return debounced
return decorator
I've had the same need to build a debounce annotation for a personal project, after stumbling upon the same gist / discussion you have, I ended up with the following solution:
import threading
def debounce(wait_time):
"""
Decorator that will debounce a function so that it is called after wait_time seconds
If it is called multiple times, will wait for the last call to be debounced and run only this one.
"""
def decorator(function):
def debounced(*args, **kwargs):
def call_function():
debounced._timer = None
return function(*args, **kwargs)
# if we already have a call to the function currently waiting to be executed, reset the timer
if debounced._timer is not None:
debounced._timer.cancel()
# after wait_time, call the function provided to the decorator with its arguments
debounced._timer = threading.Timer(wait_time, call_function)
debounced._timer.start()
debounced._timer = None
return debounced
return decorator
I've created an open-source project to provide functions such as debounce, throttle, filter ... as decorators, contributions are more than welcome to improve on the solution I have for these decorators / add other useful decorators: decorator-operations repository
I would like to execute set of tasks in parallel. I have defined a function in a class which takes the parameter and executes the operation based on parameter. Class structure is like below.
from threading import Thread
from concurrent.futures import *
class Test(object):
def process_dataframe(self,id:int):
print(id*id)
def run_task(self):
thd = []
for i in range(1,10):
thd.append( "self.process_dataframe({0})".format(i))
self.run_functions_in_parallel(thd)
def run_functions_in_parallel(self,fns)->bool:
def wrap_function(self,fnToCall):
try:
eval(fnToCall)
return ("0")
except Exception as e:
return "{0}".format(e)
thd = []
isError = False
executor = ThreadPoolExecutor(max_workers=len(fns))
errorMessage = ""
for fn in fns:
t = executor.submit(wrap_function,self,fn)
thd.append(t)
for td in thd:
ret = td.result()
if ret != "0":
isError = True
errorMessage = errorMessage + """
""" + ret
if isError == True:
print (errorMessage)
raise Exception (errorMessage)
else:
return True
d=Test()
d.run_task()
I have managed to make it work and tasks are executing properly. I am wondering whether there is better/simpler way to accomplish the same. I would like to keep run_functions_in_parallel method generic so that it can be used as common method in a module.
You don't need to use a wrapper, since ThreadPoolExecutor catches errors in a better way. A function, that always returns True or raises an error, don't need a return value, but if you have functions with return values, you want to call in parallel, you should return their results.
It is a bad idea to use a magic string as indicator for errors. format(e) of a KeyError: 0 also leads to "0". Better use a unique value, like None in our case.
Don't use eval if you don't have to. In your case, you can use partial.
Don't use a to large value for max_workers.
from functools import partial
from concurrent.futures import ThreadPoolExecutor
class Test(object):
def process_dataframe(self, id):
print(id*id)
def run_task(self):
functions = []
for i in range(1,10):
functions.append(partial(self.process_dataframe, i))
self.run_functions_in_parallel(functions)
def run_functions_in_parallel(self, functions, max_workers=8):
executor = ThreadPoolExecutor(max_workers=max_workers)
futures = [
executor.submit(function)
for function in functions
]
errors = []
results = []
for future in futures:
try:
result = future.result()
except Exception as e:
errors.append(e)
else:
results.append(result)
if errors:
raise Exception(errors)
return results
d = Test()
d.run_task()
I have a simple decorator to track the runtime of a function call:
def timed(f):
def caller(*args):
start = time.time()
res = f(*args)
end = time.time()
return res, end - start
return caller
This can be used as follows, and returns a tuple of the function result and the execution time.
#timed
def test(n):
for _ in range(n):
pass
return 0
print(test(900)) # prints (0, 2.69e-05)
Simple enough. But now I want to apply this to recursive functions. Applying the above wrapper to a recursive function results in nested tuples with the times of each recursive call, as is expected.
#timed
def rec(n):
if n:
return rec(n - 1)
else:
return 0
print(rec(3)) # Prints ((((0, 1.90e-06), 8.10e-06), 1.28e-05), 1.90e-05)
What's an elegant way to write the decorator so that it handles recursion properly? Obviously, you could wrap the call if a timed function:
#timed
def wrapper():
return rec(3)
This will give a tuple of the result and the time, but I want all of it to be handled by the decorator so that the caller does not need to worry about defining a new function for every call. Ideas?
The problem here isn't really the decorator. The problem is that rec needs rec to be a function that behaves one way, but you want rec to be a function that behaves differently. There's no clean way to reconcile that with a single rec function.
The cleanest option is to stop requiring rec to be two things at once. Instead of using decorator notation, assign timed(rec) to a different name:
def rec(n):
...
timed_rec = timed(rec)
If you don't want two names, then rec needs to be written to understand the actual value that the decorated rec will return. For example,
#timed
def rec(n):
if n:
val, runtime = rec(n-1)
return val
else:
return 0
I prefer the other answers so far (particularly user2357112's answer), but you can also make a class-based decorator that detects whether the function has been activated, and if so, bypasses the timing:
import time
class fancy_timed(object):
def __init__(self, f):
self.f = f
self.active = False
def __call__(self, *args):
if self.active:
return self.f(*args)
start = time.time()
self.active = True
res = self.f(*args)
end = time.time()
self.active = False
return res, end - start
#fancy_timed
def rec(n):
if n:
time.sleep(0.01)
return rec(n - 1)
else:
return 0
print(rec(3))
(class written with (object) so that this is compatible with py2k and py3k).
Note that to really work properly, the outermost call should use try and finally. Here's the fancied up fancy version of __call__:
def __call__(self, *args):
if self.active:
return self.f(*args)
try:
start = time.time()
self.active = True
res = self.f(*args)
end = time.time()
return res, end - start
finally:
self.active = False
You could structure your timer in a different way by *ahem* abusing the contextmanager and function attribute a little...
from contextlib import contextmanager
import time
#contextmanager
def timed(func):
timed.start = time.time()
try:
yield func
finally:
timed.duration = time.time() - timed.start
def test(n):
for _ in range(n):
pass
return n
def rec(n):
if n:
time.sleep(0.05) # extra delay to notice the difference
return rec(n - 1)
else:
return n
with timed(rec) as r:
print(t(10))
print(t(20))
print(timed.duration)
with timed(test) as t:
print(t(555555))
print(t(666666))
print(timed.duration)
Results:
# recursive
0
0
1.5130000114440918
# non-recursive
555555
666666
0.053999900817871094
If this is deemed a bad hack I'll gladly accept your criticism.
Although it is not an overall solution to the problem of integrating recursion with decorators, for the problem of timing only, I have verified that the last element of the tuple of the times is the overall run time, as this is the time from the upper-most recursive call. Thus if you had
#timed
def rec():
...
to get the overall runtime given the original function definitions you could simply do
rec()[1]
Getting the result of the call, on the other hand, would then require recusing through the nested tuple:
def get(tup):
if isinstance(tup, tuple):
return get(tup[0])
else:
return tup
This might be too complicated to simply get the result of your function.
I encountered the same issue when trying to profile a simple quicksort implementation.
The main issue is that decorators are executed on each function call and we need something that can keep a state, so we can sum all calls at the end. Decorators are not the right tool the job
However, one idea is to abuse the fact that functions are objects and can have atributes. This is explored below with a simple decorator. Something that must be understood is that, by using decorator's sintax sugar (#), the function will always be accumulating its timings.
from typing import Any, Callable
from time import perf_counter
class timeit:
def __init__(self, func: Callable) -> None:
self.func = func
self.timed = []
def __call__(self, *args: Any, **kwds: Any) -> Any:
start = perf_counter()
res = self.func(*args, **kwds)
end = perf_counter()
self.timed.append(end - start)
return res
# usage
#timeit
def rec(n):
...
if __name__ == "__main__":
result = rec(4) # rec result
print(f"Took {rec.timed:.2f} seconds")
# Out: Took 3.39 seconds
result = rec(4) # rec result
# timings between calls are accumulated
# Out: Took 6.78 seconds
Which brings us to a solution inspired by #r.ook, below is a simple context manager that stores each run timing and prints its sum at the end (__exit__). Notice that, because for each timing we require a with statement, this will not accumulate different runs.
from typing import Any, Callable
from time import perf_counter
class timeit:
def __init__(self, func: Callable) -> None:
self.func = func
self.timed = []
def __call__(self, *args: Any, **kwds: Any) -> Any:
start = perf_counter()
res = self.func(*args, **kwds)
end = perf_counter()
self.timed.append(end - start)
return res
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, exc_traceback):
# TODO: report `exc_*` if an exception get raised
print(f"Took {sum(self.timed):.2f} seconds")
return
# usage
def rec(n):
...
if __name__ == "__main__":
with timeit(rec) as f:
result = f(a) # rec result
# Out: Took 3.39 seconds
I would like to get the return data from a CAPL function called from python.
Please help me with this.
Currently I can only call the function with parameter in the example .
from win32com import client
import pythoncom
import time
function1 = None
canoe_app = None
is_running = False
class EventHandler:
def OnInit(self):
global canoe_app
global function1
function1 = canoe_app.CAPL.GetFunction('Test1')
def OnStart(self):
global is_running
is_running = True
canoe_app = client.Dispatch('CANoe.Application')
measurement = canoe_app.Measurement
measurement_events = client.WithEvents(measurement, EventHandler)
measurement.Start()
# The following loop takes care of any pending events and, once, the Measurement
# starts, it will call the CAPL function "function1" 10 times and then exit!
count = 0
while count < 10:
if (is_running):
ret = []
function1.Call(count)
function1.Call(count+1)
print(ret)
count += 1
pythoncom.PumpWaitingMessages()
time.sleep(1)
measurement.Stop()
You just have to assign the calling statement to a variable.
var1 = (int)function1.Call(count)
Note that the return variable type should only be int.
I have created a class that can take a function with a set of arguments. I would like to run the passed function every time the event handler signals.
I am attaching my code below which runs when I pass a fun2 which has no arguments but not with fun1. Any suggestions that I can make to the code below work with fun1 and fun2? If I omit the return statement from fun1, I get an error that 'str' object is not callable.
>>> TimerTest.main()
function 1. this function does task1
my function from init from function1
my function in start of runTimerTraceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Program Files (x86)\IronPython 2.7\TimerTest.py", line 57, in main
File "C:\Program Files (x86)\IronPython 2.7\TimerTest.py", line 25, in runTime
r
TypeError: str is not callable
import System
from System.Timers import (Timer, ElapsedEventArgs)
class timerTest:
def __init__ (self, interval,autoreset, fun):
self.Timer = Timer()
self.Timer.Interval= interval
self.Timer.AutoReset = autoreset
self.Timer.Enabled = True
self.myfunction = fun
def runTimer(self):
print 'my function in start of runTimer', self.myfunction ()
self.Timer.Start()
def OnTimedEvent (s, e):
print "The Elapsed event was raised at " , e.SignalTime
print 'printing myfunction...', self.myfunction()
self.myfunction()
self.Timer.Elapsed += OnTimedEvent
def stopTimer(self):
self.Timer.Stop()
self.Timer.Dispose= True
def fun1(a,b):
print 'function 1. this function does task1'
return 'from function1'
def fun2():
print 'Function 2. This function does something'
print 'Test 1...2...3...'
return 'From function 2'
def main():
a = timerTest(1000, True, fun1(10,20))
a.runTimer()
b= timerTest(3000,True,fun2)
b.runTimer()
if __name__ == '__main__':
main()
I am learning Python and I apologize if my questions are basic.
To change the interval, I stop the timer using a stopTimer method I added to the timerTest class:
def stopTimer(self):
self.Timer.Stop()
I take the new user input to call the runTimer method which I have revised per Paolo Moretti's suggestions:
def runTimer(self, interval,autoreset,fun,arg1, arg2, etc.):
self.Timer.Interval= interval
self.Timer.AutoReset = autoreset
myfunction = fun
my_args = args
self.Timer.Start()
def OnTimedEvent (s, e):
print "The Elapsed event was raised at " , e.SignalTime
myfunction(*my_args)
self.Timer.Elapsed += OnTimedEvent
Whenever a command button is pressed, the following method is called:
requestTimer.runTimer((self.intervalnumericUpDown.Value* 1000),True, function, *args)
I do not understand why stopping the timer and sending the request causes the runTimer method to be executed multiple times and it seems dependent on how many times I change the interval. I have tried a couple of methods: Close and Dispose with no success.
A second question on slightly different subject.
I have been looking at other .NET classes with Timer classes. A second question is on how I would translate the following VB sample code into Python. Is "callback As TimerCallback" equivalent to myfunction(*my_args)?
Public Sub New ( _
callback As TimerCallback, _
state As Object, _
dueTime As Integer, _
period As Integer _
)
per .NET documentation:
callback
Type: System.Threading.TimerCallback
A TimerCallback delegate representing a method to be executed.
I can partially get the timer event to fire if I define a function with no arguments such as:
def fun2(stateinfo):
# function code
which works with:
self.Timer = Timer(fun2, self.autoEvent, self.dueTime,self.period)
The function call fails if I replace fun2 with a more generic function call myfunction(*my_args)
You can also use * syntax for calling a function with an arbitrary argument list:
class TimerTest:
def __init__(self, interval, autoreset, fun, *args):
# ...
self.my_function = fun
self.my_args = args
# ...
def run_timer(self):
# ...
def on_timed_event(s, e):
# ...
self.my_function(*self.my_args)
# ...
Usage:
>>> t1 = TimerTest(1000, True, fun1, 10, 20)
>>> t2 = TimerTest(1000, True, fun2)
And check out the PEP8 style guide as well. Python's preferred coding conventions are different than many other common languages.
Question 1
Every time you use the addition assignment operator (+=) you are attaching a new event handler to the event. For example this code:
timer = Timer()
def on_timed_event(s, e):
print "Hello form my event handler"
timer.Elapsed += on_timed_event
timer.Elapsed += on_timed_event
timer.Start()
will print the "Hello form my event handler"phrase twice.
For more information you can check out the MSDN documentation, in particular Subscribe to and Unsubscribe from Events .
So, you should probably move the event subscription to the __init__ method, and only start the timer in your run_timer method:
def run_timer(self):
self.Timer.Start()
You could also add a new method (or use a property) for changing the interval:
def set_interval(self, interval):
self.Timer.Interval = interval
Question 2
You are right about TimerCallback: it's a delegate representing a method to be executed.
For example, this Timer constructor:
public Timer(
TimerCallback callback
)
is expecting a void function with a single parameter of type Object.
public delegate void TimerCallback(
Object state
)
When you are invoking a function using the * syntax you are doing something completely different. It's probably easier if I'll show you an example:
def foo(a, b, *args):
print a
print b
print args
>>> foo(1, 2, 3, 4, 5)
1
2
(3, 4, 5)
>>> args = (1, 2, 3)
>>> foo(1, 2, *args)
1
2
(1, 2, 3)
Basically in the second case you are invoking a function with additional arguments unpacked from a tuple.
So If you want to pass a function with a different signature to a constructor which accepts a TimerCallback delegate you have to create a new function, like #Lasse is suggesting.
def my_func(state, a, b):
pass
You can do this either using the lambda keyword:
t1 = Timer(lambda state: my_func(state, 1, 2))
or by declaring a new function:
def time_proc(state):
my_func(state, 1, 2)
t2 = Timer(time_proc)
If the function takes no parameters, simply pass it without calling it:
b = timerTest(3000, True, fun2)
If it takes parameters, you need to convert it to a function that doesn't take parameters. What you're doing is calling it, and then you pass the result, which in this case is a string. Instead do this:
a = timerTest(1000, True, lambda: fun1(10, 20))