I have been working with Python and I set up the following code situation:
import timeit
setting = """
import functools
def f(a,b,c):
pass
g = functools.partial(f,c=3)
h = functools.partial(f,b=5,c=3)
i = functools.partial(f,a=4,b=5,c=3)
"""
print timeit.timeit('f(4,5,3)', setup = setting, number=100000)
print timeit.timeit('g(4,5)', setup = setting, number=100000)
print timeit.timeit('h(4)', setup = setting, number=100000)
print timeit.timeit('i()', setup = setting, number=100000)
I get the following as a result:
f: 0.181384086609
g: 0.39066195488
h: 0.425783157349
i: 0.391901016235
Why do the calls to the partial functions take longer? Is the partial function just forwarding the parameters to the original function or is it mapping the static arguments throughout? And also, is there a function in Python to return the body of a function filled in given that all the parameters are predefined, like with function i?
Why do the calls to the partial functions take longer?
The code with partial takes about two times longer because of the additional function call. Function calls are expensive:
Function call overhead in Python is relatively high, especially compared with the execution speed of a builtin function.
-
Is the partial function just forwarding the parameters to the original function or is it mapping the static arguments throughout?
As far as i know - yes, it just forwards the arguments to the original function.
-
And also, is there a function in Python to return the body of a function filled in given that all the parameters are predefined, like with function i?
No, i am not aware of such built-in function in Python. But i think it's possible to do what you want, as functions are objects which can be copied and modified.
Here is a prototype:
import timeit
import types
# http://stackoverflow.com/questions/6527633/how-can-i-make-a-deepcopy-of-a-function-in-python
def copy_func(f, name=None):
return types.FunctionType(f.func_code, f.func_globals, name or f.func_name,
f.func_defaults, f.func_closure)
def f(a, b, c):
return a + b + c
i = copy_func(f, 'i')
i.func_defaults = (4, 5, 3)
print timeit.timeit('f(4,5,3)', setup = 'from __main__ import f', number=100000)
print timeit.timeit('i()', setup = 'from __main__ import i', number=100000)
which gives:
0.0257439613342
0.0221881866455
Calls to a function with partially applied arguments are more expensive because you double the number of function calls. The effect of functools.partial() is similar to this example:
def apply_one_of_two(f, a):
def g(b):
return f(a, b)
return g
That means that apply_one_of_two() returns a function and when it's called than this results in the additional call of the original funciton f.
Since Python usually doesn't optimize this away it directly translates into additional runtime efforts.
But this isn't the only factor to consider in your microbenchmark. You also switch from positional to keyword arguments in your partial invocations, which introduces additional overhead.
When you reverse the argument ordering in your original function you don't need keyword arguments in the partial calls and then the runtime difference somewhat decreases, e.g.:
import timeit
setting = """
import functools
def f(a,b,c):
pass
g = functools.partial(f, 4)
h = functools.partial(f, 4, 5)
i = functools.partial(f, 4, 5, 3)
"""
print(timeit.timeit('f(4, 5, 3)', setup = setting, number=100000))
print(timeit.timeit('g(5, 3)', setup = setting, number=100000))
print(timeit.timeit('h(3)', setup = setting, number=100000))
print(timeit.timeit('i()', setup = setting, number=100000))
Output (on an Intel Skylake i7 under Fedora 27/Python 3.6):
0.010069019044749439
0.01681053702486679
0.018060395028442144
0.011366961000021547
Related
I only just started learning Python and found out that I can pass a function as the parameter of another function. Now if I call foo(bar()) it will not pass as a function pointer but the return value of the used function. Calling foo(bar) will pass the function, but this way I am not able to pass any additional arguments. What if I want to pass a function pointer that calls bar(42)?
I want the ability to repeat a function regardless of what arguments I have passed to it.
def repeat(function, times):
for calls in range(times):
function()
def foo(s):
print s
repeat(foo("test"), 4)
In this case the function foo("test") is supposed to be called 4 times in a row.
Is there a way to accomplish this without having to pass "test" to repeat instead of foo?
You can either use a lambda:
repeat(lambda: bar(42))
Or functools.partial:
from functools import partial
repeat(partial(bar, 42))
Or pass the arguments separately:
def repeat(times, f, *args):
for _ in range(times):
f(*args)
This final style is quite common in the standard library and major Python tools. *args denotes a variable number of arguments, so you can use this function as
repeat(4, foo, "test")
or
def inquisition(weapon1, weapon2, weapon3):
print("Our weapons are {}, {} and {}".format(weapon1, weapon2, weapon3))
repeat(10, inquisition, "surprise", "fear", "ruthless efficiency")
Note that I put the number of repetitions up front for convenience. It can't be the last argument if you want to use the *args construct.
(For completeness, you could add keyword arguments as well with **kwargs.)
You will need to pass the parameters for foo, to the repeat function:
#! /usr/bin/python3.2
def repeat (function, params, times):
for calls in range (times):
function (*params)
def foo (a, b):
print ('{} are {}'.format (a, b) )
repeat (foo, ['roses', 'red'], 4)
repeat (foo, ['violets', 'blue'], 4)
While many of the answers here are good, this one might be helpful because it doesn't introduce any unnecessary repetition and the reason for callbacks in the first place is often to synchronize with other work outside of the main UI thread.
Enjoy!
import time, threading
def callMethodWithParamsAfterDelay(method=None, params=[], seconds=0.0):
return threading.Timer(seconds, method, params).start()
def cancelDelayedCall(timer):
timer.cancel()
# Example
def foo (a, b):
print ('{} are {}'.format (a, b) )
callMethodWithParametersAfterDelay(foo, ['roses', 'red'], 0)
I'm doing memoized decorator exercise with fibonacci function. The memoized function should be much faster when the input gets large, because it returns the result from a dictionary instead of computing the result again.
I'm using timeit.timeit() to measure the execution time of the function with memoized on vs off.
The result I get are exactly the opposite of what I expect. The execution without the decorator runs much faster.
# memorized decorator for fibonacci series
def mem_fib(f):
def wrapper(n):
wrapper.d = {} # create the attr member of THIS wrapper
if n in wrapper.d:
return wrapper.d[n]
wrapper.d[n] = f(n) # save f() return in a dict
return wrapper.d[n]
return wrapper
#mem_fib
def fibonacci(n):
assert n >= 0
if n < 2:
return n
return fibonacci(n-1) + fibonacci(n-2)
I'm running commands on PyCharm python's console.
#with decorator
>>> print(timeit.timeit('decorators.fibonacci(7)', setup='import decorators'))
19.6940833939
>>> print(timeit.timeit('decorators.fibonacci(10)', setup='import decorators'))
85.7157191166
without the decorator
>>> print(timeit.timeit('decorators.fibonacci(7)', setup='import decorators'))
5.10131571594
>>> print(timeit.timeit('decorators.fibonacci(10)', setup='import decorators'))
21.9784012801
I ran the timeit multiple times, I just put one output to sum it up. What am I missing?
Thanks
Update: Thanks to Daniel's answer, I found my mistake. I moved the dictionary creation outside of wrapper, and the results were much better.
>>> print(timeit.timeit('decorators.fibonacci(10)', setup='import decorators'))
0.248986574759
You create a new dictionary every time the function is called, because your wrapper function always does wrapper.d = {}. Therefore the cache will never be populated, and your code has the additional overhead of creating the dictionary each time.
That line should go outside that function, before you return it from mem_fib.
Is it possible to assign a function to a variable with modified default arguments?
To make it more concrete, I'll give an example.
The following obviously doesn't work in the current form and is only meant to show what I need:
def power(a, pow=2):
ret = 1
for _ in range(pow):
ret *= a
return ret
cube = power(pow=3)
And the result of cube(5) should be 125.
functools.partial to the rescue:
Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords. If more arguments are supplied to the call, they are appended to args. If additional keyword arguments are supplied, they extend and override keywords.
from functools import partial
cube = partial(power, pow=3)
Demo:
>>> from functools import partial
>>>
>>> def power(a, pow=2):
... ret = 1
... for _ in range(pow):
... ret *= a
... return ret
...
>>> cube = partial(power, pow=3)
>>>
>>> cube(5)
125
The answer using partial is good, using the standard library, but I think it's worth mentioning that the following approach is equivalent:
def cube(a):
return power(a, pow=3)
Even though this doesn't seem like assignment because there isn't a =, it is doing much the same thing (binding a name to a function object). I think this is often more legible.
In specific there's a special function for exponents:
>>> 2**3
8
But I also solved it with a lambda function, which is a nicer version of a function pointer.
# cube = power(pow=3) # original
cube = lambda x: power(x,3)
For partial function application, I know there are several ways to do that in Python. However, they seems not to preserve the original function's docstring.
Take functools.partial as example:
from functools import partial
def foo(a, b, c=1):
"""Return (a+b)*c."""
return (a+b)*c
bar10_p = partial(foo, b=10)
print bar10_p.__doc__
partial(func, *args, **keywords) - new function with partial application
of the given arguments and keywords.
Let's try fn.py:
from fn import F
def foo(a, b, c=1):
"""Return (a+b)*c."""
return (a+b)*c
bar10_F = F(foo, b=10)
print bar10_F.__doc__
Provide simple syntax for functions composition
(through << and >> operators) and partial function
application (through simple tuple syntax).
Usage example:
>>> func = F() << (_ + 10) << (_ + 5)
>>> print(func(10))
25
>>> func = F() >> (filter, _ < 6) >> sum
>>> print(func(range(10)))
15
Is there any Python package/module providing partial application with preserved docstring?
UPDATE
As #Kevin and #Martijn Pieters mentioned, the function signature has changed such that it is not suggested to stick to the original function's docstring. I realized that I'm looking for an updated docstring with something like foo() with a default b value of 10 (Thanks for Kevin's simple but direct example.).
__doc__ is writable, on partial objects as well as on functions; simply copy it over:
bar10_p = partial(foo, b=10)
bar10_p.__doc__ = func.__doc__
or use the functools.update_wrapper() function to do the copying for you; it'll copy a few other pieces of metadata for you too:
from functools import update_wrapper
bar10_p = partial(foo, b=10)
update_wrapper(bar10_p, foo)
Just write a new __doc__.
bar10_p = partial(foo, b=10)
bar10_p.__doc__ = """foo() with a default b value of 10.
See foo().
"""
Your function has a different interface from the original, so it should not copy the docstring exactly.
Partial has access to func method which is the original function. So through original function, you have access to original function docstring.
Try this:
from math import cos
from functools import partial
cos_partial = partial(cos, 0.5)
print(cos_partial.func.__doc__)
With makefun you can do it:
from makefun import partial
def foo(a, b, c=1):
"""Return (a+b)*c."""
return (a + b) * c
bar10_p = partial(foo, b=10)
assert bar10_p(0) == 10
assert bar10_p(0, c=2) == 20
help(bar10_p)
It yields:
Help on function foo in module makefun.tests.test_so:
foo(a, c=1)
<This function is equivalent to 'foo(a, c=1, b=10)', see original 'foo' doc below.>
Return (a+b)*c.
Note that if you have any comment on how the docstring should be updated, do not hesitate to propose an issue on the git repo !
(I'm the author by the way)
I am trying to run this snippet from http://www.ibm.com/developerworks/linux/library/l-prog3.html on a python 2.6 runtime.
from functional import *
taxcalc = lambda income,rate,deduct: (income-(deduct))*rate
taxCurry = curry(taxcalc)
taxCurry = taxCurry(50000)
taxCurry = taxCurry(0.30)
taxCurry = taxCurry(10000)
print "Curried taxes due =",taxCurry
print "Curried expression taxes due =", \
curry(taxcalc)(50000)(0.30)(10000)
Ok, so I understand from http://www.python.org/dev/peps/pep-0309/ that functional is renamed to functools and curry to partial but just doing the renames doesn't help. I get the error:
taxCurry = taxCurry(50000)
TypeError: <lambda>() takes exactly 3 arguments (1 given)
The following does work but do I really have to change it so much?
from functools import partial
taxcalc = lambda income,rate,deduct: (income-(deduct))*rate
taxCurry = partial(taxcalc)
taxCurry = partial(taxCurry, 50000)
taxCurry = partial(taxCurry, 0.30)
taxCurry = partial(taxCurry, 10000)
print "Curried taxes due =", taxCurry()
print "Curried expression taxes due =", \
taxcalc(50000, 0.30, 10000)
Is there a better way of preserving the mechanics of the original example? Lastly was the original example truly currying or just partial application? (as per http://www.uncarved.com/blog/not_currying.mrk)
Thanks for your time
I guess the reason why they changed it is because Python is dynamically typed. This means that it would be really hard to debug the original curry code if anything goes wrong - much harder than in a language like Haskell where you would directly get a nice type error. So I think it was a reasonable decision to replace it with the more explicit partial version (looks more pythonic to me).
Your example is also a bit strange, since you just reassigning the partially applied functions to the same name. Normally the partially applied function would be given to another function. At least that is the only reasonable use case in Python I can think of.
The implementation of curry in the toolz project should be a drop-in replacement.
$ pip install toolz
>>> from toolz import curry
I wrote an implementation of a curry decorator that works well:
def curry(func):
"""
Decorator to curry a function, typical usage:
>>> #curry
... def foo(a, b, c):
... return a + b + c
The function still work normally:
>>> foo(1, 2, 3)
6
And in various curried forms:
>>> foo(1)(2, 3)
6
>>> foo(1)(2)(3)
6
This also work with named arguments:
>>> foo(a=1)(b=2)(c=3)
6
>>> foo(b=1)(c=2)(a=3)
6
>>> foo(a=1, b=2)(c=3)
6
>>> foo(a=1)(b=2, c=3)
6
And you may also change your mind on named arguments,
But I don't know why you may want to do that:
>>> foo(a=1, b=0)(b=2, c=3)
6
Finally, if you give more parameters than expected, the exception
is the expected one, not some garbage produced by the currying
mechanism:
>>> foo(1, 2)(3, 4)
Traceback (most recent call last):
...
TypeError: foo() takes exactly 3 arguments (4 given)
"""
def curried(*args, **kwargs):
if len(args) + len(kwargs) >= func.__code__.co_argcount:
return func(*args, **kwargs)
return (lambda *args2, **kwargs2:
curried(*(args + args2), **dict(kwargs, **kwargs2)))
return curried
if __name__ == "__main__":
import doctest
doctest.testmod()