Related
There's a function which takes optional arguments.
def alpha(p1="foo", p2="bar"):
print('{0},{1}'.format(p1, p2))
Let me iterate over what happens when we use that function in different ways:
>>> alpha()
foo,bar
>>> alpha("FOO")
FOO,bar
>>> alpha(p2="BAR")
foo,BAR
>>> alpha(p1="FOO", p2=None)
FOO,None
Now consider the case where I want to call it like alpha("FOO", myp2) and myp2 will either contain a value to be passed, or be None. But even though the function handles p2=None, I want it to use its default value "bar" instead.
Maybe that's worded confusingly, so let me reword that:
If myp2 is None, call alpha("FOO"). Else, call alpha("FOO", myp2).
The distinction is relevant because alpha("FOO", None) has a different result than alpha("FOO").
How can I concisely (but readably) make this distinction?
One possibility would usually be to check for None within alpha, which would be encouraged because that would make the code safer. But assume that alpha is used in other places where it is actually supposed to handle None as it does.
I'd like to handle that on the caller-side.
One possibility is to do a case distinction:
if myp2 is None:
alpha("FOO")
else:
alpha("FOO", myp2)
But that can quickly become much code when there are multiple such arguments. (exponentially, 2^n)
Another possibility is to simply do alpha("FOO", myp2 or "bar"), but that requires us to know the default value. Usually, I'd probably go with this approach, but I might later change the default values for alpha and this call would then need to be updated manually in order to still call it with the (new) default value.
I am using python 3.4 but it would be best if your answers can provide a good way that works in any python version.
The question is technically finished here, but I reword some requirement again, since the first answer did gloss over that:
I want the behaviour of alpha with its default values "foo", "bar" preserved in general, so it is (probably) not an option to change alpha itself.
In yet again other words, assume that alpha is being used somewhere else as alpha("FOO", None) where the output FOO,None is expected behaviour.
Pass the arguments as kwargs from a dictionary, from which you filter out the None values:
kwargs = dict(p1='FOO', p2=None)
alpha(**{k: v for k, v in kwargs.items() if v is not None})
But assume that alpha is used in other places where it is actually supposed to handle None as it does.
To respond to this concern, I have been known to have a None-like value which isn't actually None for this exact purpose.
_novalue = object()
def alpha(p1=_novalue, p2=_novalue):
if p1 is _novalue:
p1 = "foo"
if p2 is _novalue:
p2 = "bar"
print('{0},{1}'.format(p1, p2))
Now the arguments are still optional, so you can neglect to pass either of them. And the function handles None correctly. If you ever want to explicitly not pass an argument, you can pass _novalue.
>>> alpha(p1="FOO", p2=None)
FOO,None
>>> alpha(p1="FOO")
FOO,bar
>>> alpha(p1="FOO", p2=_novalue)
FOO,bar
and since _novalue is a special made-up value created for this express purpose, anyone who passes _novalue is certainly intending the "default argument" behavior, as opposed to someone who passes None who might intend that the value be interpreted as literal None.
although ** is definitely a language feature, it's surely not created for solving this particular problem. Your suggestion works, so does mine. Which one works better depends on the rest of the OP's code. However, there is still no way to write f(x or dont_pass_it_at_all)
- blue_note
Thanks to your great answers, I thought I'd try to do just that:
# gen.py
def callWithNonNoneArgs(f, *args, **kwargs):
kwargsNotNone = {k: v for k, v in kwargs.items() if v is not None}
return f(*args, **kwargsNotNone)
# python interpreter
>>> import gen
>>> def alpha(p1="foo", p2="bar"):
... print('{0},{1}'.format(p1,p2))
...
>>> gen.callWithNonNoneArgs(alpha, p1="FOO", p2=None)
FOO,bar
>>> def beta(ree, p1="foo", p2="bar"):
... print('{0},{1},{2}'.format(ree,p1,p2))
...
>>> beta('hello', p2="world")
hello,foo,world
>>> beta('hello', p2=None)
hello,foo,None
>>> gen.callWithNonNoneArgs(beta, 'hello', p2=None)
hello,foo,bar
This is probably not perfect, but it seems to work: It's a function that you can call with another function and it's arguments, and it applies deceze's answer to filter out the arguments that are None.
You could inspect the default values via alpha.__defaults__ and then use them instead of None. That way you circumvent the hard-coding of default values:
>>> args = [None]
>>> alpha('FOO', *[x if x is not None else y for x, y in zip(args, alpha.__defaults__[1:])])
I had the same problem when calling some Swagger generated client code, which I couldn't modify, where None could end up in the query string if I didn't clean up the arguments before calling the generated methods. I ended up creating a simple helper function:
def defined_kwargs(**kwargs):
return {k: v for k, v in kwargs.items() if v is not None}
>>> alpha(**defined_kwargs(p1="FOO", p2=None))
FOO,bar
It keeps things quite readable for more complex invocations:
def beta(a, b, p1="foo", p2="bar"):
print('{0},{1},{2},{3}'.format(a, b, p1, p2,))
p1_value = "foo"
p2_value = None
>>> beta("hello",
"world",
**defined_kwargs(
p1=p1_value,
p2=p2_value))
hello,world,FOO,bar
I'm surprised nobody brought this up
def f(p1="foo", p2=None):
p2 = "bar" if p2 is None else p2
print(p1+p2)
You assign None to p2 as standart (or don't, but this way you have the true standart at one point in your code) and use an inline if. Imo the most pythonic answer. Another thing that comes to mind is using a wrapper, but that would be way less readable.
EDIT:
What I'd probably do is use a dummy as standart value and check for that. So something like this:
class dummy():
pass
def alpha(p1="foo", p2=dummy()):
if isinstance(p2, dummy):
p2 = "bar"
print("{0},{1}".format(p1, p2))
alpha()
alpha("a","b")
alpha(p2=None)
produces:
foo,bar
a,b
foo,None
Unfortunately, there's no way to do what you want. Even widely adopted python libraries/frameworks use your first approach. It's an extra line of code, but it is quite readable.
Do not use the alpha("FOO", myp2 or "bar") approach, because, as you mention yourself, it creates a terrible kind of coupling, since it requires the caller to know details about the function.
Regarding work-arounds: you could make a decorator for you function (using the inspect module), which checks the arguments passed to it. If one of them is None, it replaces the value with its own default value.
Not a direct answer, but I think this is worth considering:
See if you can break your function into several functions, neither of which has any default arguments. Factor any shared functionality out to a function you designate as internal.
def alpha():
_omega('foo', 'bar')
def beta(p1):
_omega(p1, 'bar')
def _omega(p1, p2):
print('{0},{1}'.format(p1, p2))
This works well when the extra arguments trigger "extra" functionality, as it may allow you to give the functions more descriptive names.
Functions with boolean arguments with True and/or False defaults frequently benefit from this type of approach.
Another possibility is to simply do alpha("FOO", myp2 or "bar"), but that requires us to know the default value. Usually, I'd probably go with this approach, but I might later change the default values for alpha and this call would then need to be updated manually in order to still call it with the (new) default value.
Just create a constant:
P2_DEFAULT = "bar"
def alpha(p1="foo", p2=P2_DEFAULT):
print('{0},{1}'.format(p1, p2))
and call the function:
alpha("FOO", myp2 or P2_DEFAULT)
If default values for alpha will be changed, we have to change only one constant.
Be careful with logical or for some cases, see https://stackoverflow.com/a/4978745/3605259
One more (better) use case
For example, we have some config (dictionary). But some values are not present:
config = {'name': 'Johnny', 'age': '33'}
work_type = config.get('work_type', P2_DEFAULT)
alpha("FOO", work_type)
So we use method get(key, default_value) of dict, which will return default_value if our config (dict) does not contain such key.
As I cannot comment on answers yet, I'd like to add that the first solution (unpacking the kwargs) would fit nicely in a decorator as follows:
def remove_none_from_kwargs(func):
#wraps(func)
def wrapper(self, *args, **kwargs):
func(self,*args, **{k: v for k, v in kwargs.items() if v is not None})
return wrapper
On Codewars.com I encountered the following task:
Create a function add that adds numbers together when called in succession. So add(1) should return 1, add(1)(2) should return 1+2, ...
While I'm familiar with the basics of Python, I've never encountered a function that is able to be called in such succession, i.e. a function f(x) that can be called as f(x)(y)(z).... Thus far, I'm not even sure how to interpret this notation.
As a mathematician, I'd suspect that f(x)(y) is a function that assigns to every x a function g_{x} and then returns g_{x}(y) and likewise for f(x)(y)(z).
Should this interpretation be correct, Python would allow me to dynamically create functions which seems very interesting to me. I've searched the web for the past hour, but wasn't able to find a lead in the right direction. Since I don't know how this programming concept is called, however, this may not be too surprising.
How do you call this concept and where can I read more about it?
I don't know whether this is function chaining as much as it's callable chaining, but, since functions are callables I guess there's no harm done. Either way, there's two ways I can think of doing this:
Sub-classing int and defining __call__:
The first way would be with a custom int subclass that defines __call__ which returns a new instance of itself with the updated value:
class CustomInt(int):
def __call__(self, v):
return CustomInt(self + v)
Function add can now be defined to return a CustomInt instance, which, as a callable that returns an updated value of itself, can be called in succession:
>>> def add(v):
... return CustomInt(v)
>>> add(1)
1
>>> add(1)(2)
3
>>> add(1)(2)(3)(44) # and so on..
50
In addition, as an int subclass, the returned value retains the __repr__ and __str__ behavior of ints. For more complex operations though, you should define other dunders appropriately.
As #Caridorc noted in a comment, add could also be simply written as:
add = CustomInt
Renaming the class to add instead of CustomInt also works similarly.
Define a closure, requires extra call to yield value:
The only other way I can think of involves a nested function that requires an extra empty argument call in order to return the result. I'm not using nonlocal and opt for attaching attributes to the function objects to make it portable between Pythons:
def add(v):
def _inner_adder(val=None):
"""
if val is None we return _inner_adder.v
else we increment and return ourselves
"""
if val is None:
return _inner_adder.v
_inner_adder.v += val
return _inner_adder
_inner_adder.v = v # save value
return _inner_adder
This continuously returns itself (_inner_adder) which, if a val is supplied, increments it (_inner_adder += val) and if not, returns the value as it is. Like I mentioned, it requires an extra () call in order to return the incremented value:
>>> add(1)(2)()
3
>>> add(1)(2)(3)() # and so on..
6
You can hate me, but here is a one-liner :)
add = lambda v: type("", (int,), {"__call__": lambda self, v: self.__class__(self + v)})(v)
Edit: Ok, how this works? The code is identical to answer of #Jim, but everything happens on a single line.
type can be used to construct new types: type(name, bases, dict) -> a new type. For name we provide empty string, as name is not really needed in this case. For bases (tuple) we provide an (int,), which is identical to inheriting int. dict are the class attributes, where we attach the __call__ lambda.
self.__class__(self + v) is identical to return CustomInt(self + v)
The new type is constructed and returned within the outer lambda.
If you want to define a function to be called multiple times, first you need to return a callable object each time (for example a function) otherwise you have to create your own object by defining a __call__ attribute, in order for it to be callable.
The next point is that you need to preserve all the arguments, which in this case means you might want to use Coroutines or a recursive function. But note that Coroutines are much more optimized/flexible than recursive functions, specially for such tasks.
Here is a sample function using Coroutines, that preserves the latest state of itself. Note that it can't be called multiple times since the return value is an integer which is not callable, but you might think about turning this into your expected object ;-).
def add():
current = yield
while True:
value = yield current
current = value + current
it = add()
next(it)
print(it.send(10))
print(it.send(2))
print(it.send(4))
10
12
16
Simply:
class add(int):
def __call__(self, n):
return add(self + n)
If you are willing to accept an additional () in order to retrieve the result you can use functools.partial:
from functools import partial
def add(*args, result=0):
return partial(add, result=sum(args)+result) if args else result
For example:
>>> add(1)
functools.partial(<function add at 0x7ffbcf3ff430>, result=1)
>>> add(1)(2)
functools.partial(<function add at 0x7ffbcf3ff430>, result=3)
>>> add(1)(2)()
3
This also allows specifying multiple numbers at once:
>>> add(1, 2, 3)(4, 5)(6)()
21
If you want to restrict it to a single number you can do the following:
def add(x=None, *, result=0):
return partial(add, result=x+result) if x is not None else result
If you want add(x)(y)(z) to readily return the result and be further callable then sub-classing int is the way to go.
The pythonic way to do this would be to use dynamic arguments:
def add(*args):
return sum(args)
This is not the answer you're looking for, and you may know this, but I thought I would give it anyway because if someone was wondering about doing this not out of curiosity but for work. They should probably have the "right thing to do" answer.
I am a c++ guy, learning the lambda function in python and wanna know it inside out. did some seraches before posting here. anyway, this piece of code came up to me.
<1> i dont quite understand the purpose of lambda function here. r we trying to get a function template? If so, why dont we just set up 2 parameters in the function input?
<2> also, make_incrementor(42), at this moment is equivalent to return x+42, and x is the 0,1 in f(0) and f(1)?
<3> for f(0), does it not have the same effect as >>>f = make_incrementor(42)? for f(0), what are the values for x and n respectively?
any commments are welcome! thanks.
>>> def make_incrementor(n):
... return lambda x: x + n
...
>>> f = make_incrementor(42)
>>> f(0)
42
>>> f(1)
43
Yes, this is similar to a C++ int template. However, instead of at compile time (yes, Python (at least for CPython) is "compiled"), the function is created at run time. Why the lambda is used in this specific case is unclear, probably only for demonstration that functions can be returned from other functions rather than practical use. Sometimes, however, statements like this may be necessary if you need a function taking a specified number of arguments (e.g. for map, the function must take the same number of arguments as the number of iterables given to map) but the behaviour of the function should depend on other arguments.
make_incrementor returns a function that adds n (here, 42) to any x passed to that function. In your case the x values you tried are 0 and `1``
f = make_incrementor(42) sets f to a function that returns x + 42. f(0), however, returns 0 + 42, which is 42 - the returned types and values are both different, so the different expressions don't have the same effect.
The purpose is to show a toy lambda return. It lets you create a function with data baked in. I have used this less trivial example of a similar use.
def startsWithFunc(testString):
return lambda x: x.find(testString) == 0
Then when I am parsing, I create some functions:
startsDesctription = startsWithFunc("!Sample_description")
startMatrix = startsWithFunc("!series_matrix_table_begin")
Then in code I use:
while line:
#.... other stuff
if startsDesctription(line):
#do description work
if startMatrix(line):
#do matrix start work
#other stuff ... increment line ... etc
Still perhaps trival, but it shows creating general funcitons with data baked it.
Excuse my poor wording in the title, but here's a longer explanation:
I have a function which as arguments takes some functions which are used to determine which data to retrieve from a database, as such:
def customer_data(customer_name, *args):
# initialize dictionary with ids
codata = dict([(data.__name__, []) for data in args])
codata['customer_observer_id'] = _customer_observer_ids(customer_name)
# add values to dictionary using function name as key
for data in args:
for coid in codata['customer_observer_id']:
codata[data.__name__].append(data(coid))
return codata
Which makes the call to the function looking something like this:
customer_data('customername', target_parts, source_group, ...)
One of these functions is defined with an extra parameter:
def polarization_value(customer_observer_id, timespan='day')
What I would like is a way to change the timespan variable in a clever way. One obvious way is to include a keyword argument in customer_observer and add an exception when the function name being called is 'polarization_value', but I have a feeling there is a better way to do this.
You can use functools.partial and pass polarization_value as :
functools.partial(polarization_value, timespan='day')
Example:
>>> import functools
def func(x, y=1):
print x, y
...
>>> new_func = functools.partial(func, y=20)
>>> new_func(100)
100 20
You may also find this helpful: Python: Why is functools.partial necessary?
Background
I have a function that takes a number of parameters and returns an error measure which I then want to minimize (using scipy.optimize.leastsq, but that is beside the point right now).
As a toy example, let's assume my function to optimize take the four parameters a,b,c,d:
def f(a,b,c,d):
err = a*b - c*d
return err
The optimizer then want a function with the signature func(x, *args) where x is the parameter vector.
That is, my function is currently written like:
def f_opt(x, *args):
a,b,c,d = x
err = a*b - c*d
return err
But, now I want to do a number of experiments where I fix some parameters while keeping some parameters free in the optimization step.
I could of course do something like:
def f_ad_free(x, b, c):
a, d = x
return f(a,b,c,d)
But this will be cumbersome since I have over 10 parameters which means the combinations of different numbers of free-vs-fixed parameters will potentially be quite large.
First approach using dicts
One solution I had was to write my inner function f with keyword args instead of positional args and then wrap the solution like this:
def generate(func, all_param, fixed_param):
param_dict = {k : None for k in all_param}
free_param = [param for param in all_param if param not in fixed_param]
def wrapped(x, *args):
param_dict.update({k : v for k, v in zip(fixed_param, args)})
param_dict.update({k : v for k, v in zip(free_param, x)})
return func(**param_dict)
return wrapped
Creating a function that fixes 'b' and 'c' then turns into the following:
all_params = ['a','b','c']
f_bc_fixed = generate(f_inner, all_params, ['b', 'c'])
a = 1
b = 2
c = 3
d = 4
f_bc_fixed((a,d), b, c)
Question time!
My question is whether anyone can think of a neater way solve this. Since the final function is going to be run in an optimization step I can't accept too much overhead for each function call.
The time it takes to generate the optimization function is irrelevant.
I can think of several ways to avoid using a closure as you do above, though after doing some testing, I'm not sure either of these will be faster. One approach might be to skip the wrapper and just write a function that accepts
A vector
A list of free names
A dictionary mapping names to values.
Then do something very like what you do above, but in the function itself:
def f(free_vals, free_names, params):
params.update(zip(free_names, free_vals))
err = params['a'] * params['b'] - params['c'] * params['d']
return err
For code that uses variable names multiple times, make vars local up front, e.g.
a = params['a']
b = params['b']
and so on. This might seem cumbersome, but it has the advantage of making everything explicit, avoiding the kinds of namespace searches that could make closures slow.
Then pass a list of free names and a dictionary of fixed params via the args parameter to optimize.leastsq. (Note that the params dictionary is mutable, which means that there could be side effects in theory; but in this case it shouldn't matter because only the free params are being overwritten by update, so I omitted the copy step for the sake of speed.)
The main downsides of this approach are that it shifts some complexity into the call to optimize.leastsq, and it makes your code less reusable. A second approach avoids those problems though it might not be quite as fast: using a callable class.
class OptWrapper(object):
def __init__(self, func, free_names, **fixed_params):
self.func = func
self.free_names = free_names
self.params = fixed_params
def __call__(self, x, *args):
self.params.update(zip(self.free_names, x))
return self.func(**self.params)
You can see that I simplified the parameter structure for __init__; the fixed params are passed here as keyword arguments, and the user must ensure that free_names and fixed_params don't have overlapping names. I think the simplicity is worth the tradeoff but you can easily enforce the separation between the two just as you did in your wrapper code.
I like this second approach best; it has the flexibility of your closure-based approach, but I find it more readable. All the names are in (or can be accessed through) the local namespace, which I thought that would speed things up -- but after some testing I think there's reason to believe that the closure approach will still be faster than this; accessing the __call__ method seems to add about 100 ns per call of overhead. I would strongly recommend testing if performance is a real issue.
Your generate function is basically the same as functools.partial, which is what I would use here.