I need to keep track of the number of times each function in a collection has been called. If a function is called more than x times within n seconds, my program needs to pause, after which the count for that function is reset.
My functions calls might look something like this:
a(1)
b(1,param2=2, param3=3)
c(1,param3=3)
My best idea is to have a wrapper function keep track of all of the limits. Something like
def wrapper(function, function_params,x,n):
if not hasattr(wrapper, "function_dict"):
wrapper.function_dict = {}
if function not in wrapper.function_dict.keys():
wrapper.function_dict[function] = {
remaining = x, expires = time.time() + n
}
remaining = wrapper.function_dict[function]['remaining']
expires = wrapper.function_dict[function]['expires']
if remaining == 0:
time.sleep(expires - time.time())
wrapper.function_dict[function] = {
remaining = x, expires = time.time() + n
}
results = ????? # call function, this is what I don't know how to do
wrapper.function_dict[function]['remaining'] -= 1
My question is, how to I handle the parameters for the functions? I'm not sure how exactly to account for the fact that there might be a variable number of parameters, and that some might be named. For example, the function definition for c might be:
def c(param1,param2=2, param3=3):
return param1 + param2 + param3
But I might need to call it with only param1 and param3.
Do I have the right general approach? This feels like something I could accomplish with the ** operator, but I'm stuck on how exactly to proceed.
Write a decorator, and use a splat operator to handle arbitrary arguments.
Example:
def pause_wrapper(x, n):
def decorator(f):
config = [x, time.time()+n]
def wrapped(*args, **kwargs):
if config[0] == 0:
time.sleep(config[1] - time.time())
config = [x, time.time() + n]
return f(*args, **kwargs)
return wrapped
return decorator
and usage:
#pause_wrapper(x, n)
def function(a, b, c):
...
The *args and **kwargs are informally called "splat" arguments. A function that takes *args, **kwargs receives all positional parameters in the tuple args and all keyword arguments in the dictionary kwargs. (You can have other arguments besides the splats, in which case the splats soak up all arguments not sent to named arguments).
Passing *args and **kwargs has the opposite effect, passing the contents of args as extra positional parameters, and kwargs as keyword parameters.
Using both allows you to handle any set of arguments, in or out, letting you do transparent wrapping (like this example).
this is basically what decorators were made for
from collections import defaultdict
class counted:
calls = defaultdict(int)
def __init__(self,x,n):
self.x = x
self.n = n
def __call__(self,fn,*args,**kwargs):
results = fn(*args,**kwargs)
calls[fn.__name__] += 1
#do something with the count ...
#counted(3,9)
def functionWhatever(arg1,arg2,arg3,**kwargs):
return "55"
functionWhatever(1,2,3,something=5)
Related
Mock version of the problem
For a function
def f(a,b,c):
return a+b+c
The function
def fix(func, **kwargs):
fa = kwargs.get('a')
fb = kwargs.get('b')
if fa is not None and fb is not None:
def f(*args):
return func(a=fa, b=fb, c=args[0])
elif fa is not None:
def f(*args):
return func(a=fa, b=args[0], c=args[1])
elif fb is not None:
def f(*args):
return func(a=args[0],b=fb, c=args[1])
else:
def f(*args):
return func(args)
return f
allows to obtain a new function by fixing some of the parameters of func.
For example: fix(g, b=3) would give us a function like
def fixed_b_in_g(a,c):
return g(a,3,c)
Question: I would like to see if there is some trick to use fix in such a way that produces a function like
def fix_a_equal_b_in_g(a,c):
return g(a,a,c)
Concrete problem
The function scipy.stats.rv_continuous.fit allows to fit parameters of a distribution to an input sample. It allows to input some keyword arguments (like fix above does) to tell it to keep some of the parameters fixed to values that the user inputs. Internally scipy.stats.rv_continuous.fit has a function, scipy.stats.rv_continuous._reduce_func, that does more or less what dix does (better implemented than my fix for example).
In my case, rather than fixing some parameters to values, I would like to fit to keep two parameters (say a and b) equal to each other, but still free during the fitting.
We can use this function to copy a keyword argument whose name is base_kwarg_name to added_kwarg_name:
def with_copied_kwargs(func, added_kwarg_names_by_base):
def fixed_func(*args, **base_kwargs):
added_kwargs = {
added_kwarg_name: base_kwargs[base_kwarg_name]
for base_kwarg_name, added_kwarg_name in added_kwarg_names_by_base.items()
}
return func(*args, **base_kwargs, **added_kwargs)
return fixed_func
Given:
def add(*, a, b, c):
return a + b + c
then modified_add = with_copied_kwargs(add, {"b": "c"}) is equivalent to:
def modified_add(*, a, b):
return add(a=a, b=b, c=b)
with_copied_kwargs can then be used along with functools.partial to both both copy keyword arguments and provide values incrementally. modified_add = functools.partial(with_copied_kwargs(add, {"b": "c"}), a=1) is equivalent to:
def modified_add(*, b):
return add(a=1, b=b, c=b)
Note that I add * (see PEP 3102) before all parameters in functions I then apply with_copied_kwargs to because the minute people start using positional arguments, things would get messy. So better to restrict it to keyword-only arguments.
As per manual, functools partial() is 'used for partial function application which “freezes” some portion of a function’s arguments and/or keywords resulting in a new object with a simplified signature.'
What's the best way to specify the positions of the arguments that one wishes to evaluate?
EDIT
Note as per comments, the function to be partially evaluated may contain named and unnamed arguments (these functions should be completely arbitrary and may be preexisting)
END EDIT
For example, consider:
def f(x,y,z):
return x + 2*y + 3*z
Then, using
from functools import partial
both
partial(f,4)(5,6)
and
partial(f,4,5)(6)
give 32.
But what if one wants to evaluate, say the third argument z or the first and third arguments x, and z?
Is there a convenient way to pass the position information to partial, using a decorator or a dict whose keys are the desired arg positions and the respective values are the arg values? eg to pass the x and z positions something like like this:
partial_dict(f,{0:4,2:6})(5)
No, partial is not designed to freeze positional arguments at non-sequential positions.
To achieve the desired behavior outlined in your question, you would have to come up with a wrapper function of your own like this:
def partial_positionals(func, positionals, **keywords):
def wrapper(*args, **kwargs):
arg = iter(args)
return func(*(positionals[i] if i in positionals else next(arg)
for i in range(len(args) + len(positionals))), **{**keywords, **kwargs})
return wrapper
so that:
def f(x, y, z):
return x + 2 * y + 3 * z
print(partial_positionals(f, {0: 4, 2: 6})(5))
outputs:
32
Simply use keyword arguments. Using your definition of f above,
>>> g = partial(f, z=10)
>>> g(2, 4)
40
>>> h = partial(f, y=4, z=10)
>>> h(2)
40
Note that once you use a keyword argument for a given parameter, you must use keyword arguments for all remaining arguments. For example, the following would not be valid:
>>> j = partial(f, x=2, z=10)
>>> j(4)
TypeError: f() got multiple values for argument 'x'
But continuing to use keyword arguments is:
>>> j = partial(f, x=2, z=10)
>>> j(y=4)
40
When you use functools.partial, you store the values of *args and **kwargs for later interpolation. When you later call the "partially applied" function, the implementation of functools.partial effectively adds the previously provided *args and **kwargs to the argument list at the front and end, respectively, as though you had inserted these argument-unpackings yourself. I.e., calling
h = partial(1, z=10)
f(4)
is roughly equivalent to writing
args = [1]
kwargs = {'z': 10}
f(*args, 4, **kwargs)
As such, the semantics of how you provide arguments to functools.partial is the same as how you would need to store arguments in the args and kwargs variables above such that the final call to f is sensible. For more information, take a look at the pseduo-implementation of functools.partial given in the functools module documentation
For easier usage, you can create a new object specifically to specify a positional argument that is to be skipped when sequentially listing values for positional arguments to be frozen with partial:
SKIP = object()
def partial_positionals(func, *positionals, **keywords):
def wrapper(*args, **kwargs):
arg = iter(args)
return func(*(*(next(arg) if i is SKIP else i for i in positionals), *arg),
**{**keywords, **kwargs})
return wrapper
so that:
def f(x, y, z):
return x + 2 * y + 3 * z
print(partial_positionals(f, 4, SKIP, 6)(5))
outputs:
32
I have a dictionary of functions, all of which use 1 or 2 optional arguments. I want to iterate through this dictionary and pass both arguments to each iterated function, and have the functions that only need 1 argument to ignore the second. In these cases, however, I get an unexpected keyword argument error.
def getNumFrames(self, **kwargs):
return len(self.x)
def getConcentration(self, **kwargs):
if ((not gradient) or (not maxX)):
return 0
return (gradient / maxX) * self.x[0]
fields = {'numFrames': getNumFrames, 'concentration': getConcentration}
for field, fieldFunction in fields.items():
for track in tracks:
fieldFunction(object, maxX = 10, gradient = 2)
In this example, getConcentration would work, but getFrames would say maxX is an unexpected keyword.
*I edited my post to include my actual minimalist code, as was suggested.
A much better way to avoid all of this trouble is to use the following paradigm:
def func(obj, **kwargs):
return obj + kwargs.get(a, 0) + kwargs.get(b,0)
This makes use of the fact that kwargs is a dictionary consisting of the passed arguments and their values and get() performs lookup and returns a default value if the key does not exist.
Always access your named arguments through the dictionary kwargs. This makes your life much simpler.
I would go about this by creating a function like the one below with the second parameter optional.
func (obj, a = 0, b = 0):
return obj + a + b
I am trying to create a set of functions in python that will all do a similar operation on a set of inputs. All of the functions have one input parameter fixed and half of them also need a second parameter. For the sake of simplicity, below is a toy example with only two functions.
Now, I want, in my script, to run the appropriate function, depending on what the user input as a number. Here, the user is the random function (so the minimum example works). What I want to do is something like this:
def function_1(*args):
return args[0]
def function_2(*args):
return args[0] * args[1]
x = 10
y = 20
i = random.randint(1,2)
f = function_1 if i==1 else function_2
return_value = f(x,y)
And it works, but it seems messy to me. I would rather have function_1 defined as
def function_1(x):
return x
Another way would be to define
def function_1(x,y):
return x
But that leaves me with a dangling y parameter.
but that will not work as easily. Is my way the "proper" way of solving my problem or does there exist a better way?
There are couple of approaches here, all of them adding more boiler-plate code.
There is also this PEP which may be interesting to you.
But 'pythonic' way of doing it is not as elegant as usual function overloading due to the fact that functions are just class attributes.
So you can either go with function like that:
def foo(*args):
and then count how many args you've got which will be very broad but very flexible as well.
another approach is the default arguments:
def foo(first, second=None, third=None)
less flexible but easier to predict, and then lastly you can also use:
def foo(anything)
and detect the type of anything in your function acting accordingly.
Your monkey-patching example can work too, but it becomes more complex if you use it with class methods, and does make introspection tricky.
EDIT: Also, for your case you may want to keep the functions separate and write single 'dispatcher' function that will call appropriate function for you depending on the arguments, which is probably best solution considering above.
EDIT2: base on your comments I believe that following approach may work for you
def weigh_dispatcher(*args, **kwargs):
#decide which function to call base on args
if 'somethingspecial' in kwargs:
return weight2(*args, **kwargs)
def weight_prep(arg):
#common part here
def weight1(arg1, arg2):
weitht_prep(arg1)
#rest of the func
def weight2(arg1, arg2, arg3):
weitht_prep(arg1)
#rest of the func
alternatively you can move the common part into the dispatcher
You may also have a function with optional second argument:
def function_1(x, y = None):
if y != None:
return x + y
else:
return x
Here's the sample run:
>>> function_1(3)
3
>>> function_1(3, 4)
7
Or even optional multiple arguments! Check this out:
def function_2(x, *args):
return x + sum(args)
And the sample run:
>>> function_2(3)
3
>>> function_2(3, 4)
7
>>> function_2(3, 4, 5, 6, 7)
25
You may here refer to args as to list:
def function_3(x, *args):
if len(args) < 1:
return x
else:
return x + sum(args)
And the sample run:
>>> function_3(1,2,3,4,5)
15
If I have to wrap an existing method, let us say wrapee() from a new method, say wrapper(), and the wrapee() provides default values for some arguments, how do I preserve its semantics without introducing unnecessary dependencies and maintenance? Let us say, the goal is to be able to use wrapper() in place of wrapee() without having to change the client code. E.g., if wrapee() is defined as:
def wrapee(param1, param2="Some Value"):
# Do something
Then, one way to define wrapper() is:
def wrapper(param1, param2="Some Value"):
# Do something
wrapee(param1, param2)
# Do something else.
However, wrapper() has to make assumptions on the default value for param2 which I don't like. If I have the control on wrapee(), I would define it like this:
def wrapee(param1, param2=None):
param2 = param2 or "Some Value"
# Do something
Then, wrapper() would change to:
def wrapper(param1, param2=None):
# Do something
wrapee(param1, param2)
# Do something else.
If I don't have control on how wrapee() is defined, how best to define wrapper()? One option that comes into mind is to use to create a dict with non-None arguments and pass it as dictionary arguments, but it seems unnecessarily tedious.
Update:
The solution is to use both the list and dictionary arguments like this:
def wrapper(param1, *args, **argv):
# Do something
wrapee(param1, *args, **argv)
# Do something else.
All the following calls are then valid:
wrapper('test1')
wrapper('test1', 'test2')
wrapper('test1', param2='test2')
wrapper(param2='test2', param1='test1')
Check out argument lists in the Python docs.
>>> def wrapper(param1, *stuff, **kargs):
... print(param1)
... print(stuff)
... print(args)
...
>>> wrapper(3, 4, 5, foo=2)
3
(4, 5)
{'foo': 2}
Then to pass the args along:
wrapee(param1, *stuff, **kargs)
The *stuff is a variable number of non-named arguments, and the **kargs is a variable number of named arguments.
I'd hardly say that it isn't tedious, but the only approach that I can think of is to introspect the function that you are wrapping to determine if any of its parameters have default values. You can get the list of parameters and then determine which one is the first that has default values:
from inspect import getargspec
method_signature = getargspec(method)
param_names = method_signature[0]
default_values = method_signature[3]
params = []
# If any of method's parameters has default values, we need
# to know the index of the first one that does.
param_with_default_loc = -1
if default_values is not None and len(default_values) > 0:
param_slice_index = len(default_values) * -1
param_with_default = param_names[param_slice_index:][0]
param_with_default_loc = param_names.index(param_with_default)
At that point, you can iterate over param_names, copying into the dict that is passed to wrappee. Once your index >= param_with_default_loc, you can obtain the default values by looking in the default_values list with an index of your index - param_with_default_loc.
Does that make any sesne?
Of course, to make this generic, you would to define it as a wrapper function, adding yet another layer of wrapping.
def wrapper(param1, param2=None):
if param2:
wrapee(param1, param2)
else:
wrapee(param1)
is this what you want?
#!/usr/bin/python
from functools import wraps
def my_decorator(f):
#wraps(f)
def wrapper(*args, **kwds):
print 'Calling decorated function'
return f(*args, **kwds)
return wrapper
def f1(x, y):
print x, y
def f2(x, y="ok"):
print x, y
my_decorator(f1)(1,2)
my_decorator(f2)(1,2)
my_decorator(f2)(1)
adapted from http://koala/doc/python2.6-doc/html/library/functools.html#module-functools