Related
I need to define functions that get called in two stages.
The first stage only sets a subset of parameters and the second stage runs the function with some additional parameters.
I didn't know how to name the title to this question.
Currently I'm doing this by defining three functions.
The first function returns a new function that has the parameters set.
After that, calling the returned function will actually call the final function.
The current solution looks like this:
def my_function(stage1_param1, stage1_param2):
return lambda stage2_param1, stage2_param2: my_function_op(stage1_param1, stage1_param2, stage2_param1, stage2_param2)
def my_function_op(stage1_param1, stage1_param2, stage2_param1, stage2_param2):
# do stuff
I would like to reduce boilerplate here and have a more compact version.
Is there a better / shorter solution?
I think you are just looking for partial function application; your my_function is in some sense a specialized implementation of functools.partial.
from functools import partial
def my_function_op(p1, p2, p3, p4):
# do stuff
f = partial(my_function_op, a1, a2)
x = f(a3, a4) # x = my_function_op(a1, a2, a3, a4)
You can just define a function within a function:
def my_function(stage1_param1, stage1_param2):
def stage2(stage2_param1, stage2_param2):
... # do stuff
return stage2
You can then use my_function like this:
>>> f = my_function("foo", "bar")
>>> f("ham", "spam")
Inside the inner function, all the arguments are bound to the correct values.
This can be nested as far as needed, just remember to return the defined functions.
Excuse my poor wording in the title, but here's a longer explanation:
I have a function which as arguments takes some functions which are used to determine which data to retrieve from a database, as such:
def customer_data(customer_name, *args):
# initialize dictionary with ids
codata = dict([(data.__name__, []) for data in args])
codata['customer_observer_id'] = _customer_observer_ids(customer_name)
# add values to dictionary using function name as key
for data in args:
for coid in codata['customer_observer_id']:
codata[data.__name__].append(data(coid))
return codata
Which makes the call to the function looking something like this:
customer_data('customername', target_parts, source_group, ...)
One of these functions is defined with an extra parameter:
def polarization_value(customer_observer_id, timespan='day')
What I would like is a way to change the timespan variable in a clever way. One obvious way is to include a keyword argument in customer_observer and add an exception when the function name being called is 'polarization_value', but I have a feeling there is a better way to do this.
You can use functools.partial and pass polarization_value as :
functools.partial(polarization_value, timespan='day')
Example:
>>> import functools
def func(x, y=1):
print x, y
...
>>> new_func = functools.partial(func, y=20)
>>> new_func(100)
100 20
You may also find this helpful: Python: Why is functools.partial necessary?
I've found various detailed explanations on how to pass long lists of arguments into a function, but I still kinda doubt if that's proper way to do it.
In other words, I suspect that I'm doing it wrong, but I can't see how to do it right.
The problem: I have (not very long) recurrent function, which uses quite a number of variables and needs to modify some content in at least some of them.
What I end up with is sth like this:
def myFunction(alpha, beta, gamma, zeta, alphaList, betaList, gammaList, zetaList):
<some operations>
myFunction(alpha, beta, modGamma, zeta, modAlphaList, betaList, gammaList, modZetaList)
...and I want to see the changes I did on original variables (in C I would just pass a reference, but I hear that in Python it's always a copy?).
Sorry if noob, I don't know how to phrase this question so I can find relevant answers.
You could wrap up all your parameters in a class, like this:
class FooParameters:
alpha = 1.0
beta = 1.0
gamma = 1.0
zeta = 1.0
alphaList = []
betaList = []
gammaList = []
zetaList = []
and then your function takes a single parameter instance:
def myFunction(params):
omega = params.alpha * params.beta + exp(params.gamma)
# more magic...
calling like:
testParams = FooParameters()
testParams.gamma = 2.3
myFunction(testParams)
print params.zetaList
Because the params instance is passed by reference, changes in the function are preserved.
This is commonly used in matplotlib, for example. They pass the long list of arguments using * or **, like:
def function(*args, **kwargs):
do something
Calling function:
function(1,2,3,4,5, a=1, b=2, b=3)
Here 1,2,3,4,5 will go to args and a=1, b=2, c=3 will go to kwargs, as a dictionary. So that they arrive at your function like:
args = [1,2,3,4,5]
kwargs = {a:1, b:2, c:3}
And you can treat them in the way you want.
I don't know where you got the idea that Python copies values when passing into a function. That is not at all true.
On the contrary: each parameter in a function is an additional name referring to the original object. If you change the value of that object in some way - for example, if it's a list and you change one of its members - then the original will also see that change. But if you rebind the name to something else - say by doing alpha = my_completely_new_value - then the original remains unchanged.
You may be tempted to something akin to this:
def myFunction(*args):
var_names = ['alpha','beta','gamma','zeta']
locals().update(zip(var_names,args))
myFunction(alpha,beta,gamma,zeta)
However, this 'often' won't work. I suggest introducing another namespace:
from collections import OrderedDict
def myFunction(*args):
var_names = ['alpha','beta','gamma','zeta']
vars = OrderedDict(zip(var_names,args))
#get them all via vars[var_name]
myFunction(*vars.values()) #since we used an orderedDict we can simply do *.values()
you can capture the non-modfied values in a closure:
def myFunction(alpha, beta, gamma, zeta, alphaList, betaList, gammaList, zetaList):
def myInner(g=gamma, al, zl):
<some operations>
myInner(modGamma, modAlphaList, modZetaList)
myInner(al=alphaList, zl=zetaList)
(BTW, this is about the only way to write a truly recursive function in Python.)
You could pass in a dictionary and return a new dictionary. Or put your method in a class and have alpha, beta etc. be attributes.
You should put myFunction in a class. Set up the class with the appropriate attributes and call the appropriate functions. The state is then well contained in the class.
I am not able to get my head on how the partial works in functools.
I have the following code from here:
>>> sum = lambda x, y : x + y
>>> sum(1, 2)
3
>>> incr = lambda y : sum(1, y)
>>> incr(2)
3
>>> def sum2(x, y):
return x + y
>>> incr2 = functools.partial(sum2, 1)
>>> incr2(4)
5
Now in the line
incr = lambda y : sum(1, y)
I get that whatever argument I pass to incr it will be passed as y to lambda which will return sum(1, y) i.e 1 + y.
I understand that. But I didn't understand this incr2(4).
How does the 4 gets passed as x in partial function? To me, 4 should replace the sum2. What is the relation between x and 4?
Roughly, partial does something like this (apart from keyword args support etc):
def partial(func, *part_args):
def wrapper(*extra_args):
args = list(part_args)
args.extend(extra_args)
return func(*args)
return wrapper
So, by calling partial(sum2, 4) you create a new function (a callable, to be precise) that behaves like sum2, but has one positional argument less. That missing argument is always substituted by 4, so that partial(sum2, 4)(2) == sum2(4, 2)
As for why it's needed, there's a variety of cases. Just for one, suppose you have to pass a function somewhere where it's expected to have 2 arguments:
class EventNotifier(object):
def __init__(self):
self._listeners = []
def add_listener(self, callback):
''' callback should accept two positional arguments, event and params '''
self._listeners.append(callback)
# ...
def notify(self, event, *params):
for f in self._listeners:
f(event, params)
But a function you already have needs access to some third context object to do its job:
def log_event(context, event, params):
context.log_event("Something happened %s, %s", event, params)
So, there are several solutions:
A custom object:
class Listener(object):
def __init__(self, context):
self._context = context
def __call__(self, event, params):
self._context.log_event("Something happened %s, %s", event, params)
notifier.add_listener(Listener(context))
Lambda:
log_listener = lambda event, params: log_event(context, event, params)
notifier.add_listener(log_listener)
With partials:
context = get_context() # whatever
notifier.add_listener(partial(log_event, context))
Of those three, partial is the shortest and the fastest.
(For a more complex case you might want a custom object though).
partials are incredibly useful.
For instance, in a 'pipe-lined' sequence of function calls (in which the returned value from one function is the argument passed to the next).
Sometimes a function in such a pipeline requires a single argument, but the function immediately upstream from it returns two values.
In this scenario, functools.partial might allow you to keep this function pipeline intact.
Here's a specific, isolated example: suppose you want to sort some data by each data point's distance from some target:
# create some data
import random as RND
fnx = lambda: RND.randint(0, 10)
data = [ (fnx(), fnx()) for c in range(10) ]
target = (2, 4)
import math
def euclid_dist(v1, v2):
x1, y1 = v1
x2, y2 = v2
return math.sqrt((x2 - x1)**2 + (y2 - y1)**2)
To sort this data by distance from the target, what you would like to do of course is this:
data.sort(key=euclid_dist)
but you can't--the sort method's key parameter only accepts functions that take a single argument.
so re-write euclid_dist as a function taking a single parameter:
from functools import partial
p_euclid_dist = partial(euclid_dist, target)
p_euclid_dist now accepts a single argument,
>>> p_euclid_dist((3, 3))
1.4142135623730951
so now you can sort your data by passing in the partial function for the sort method's key argument:
data.sort(key=p_euclid_dist)
# verify that it works:
for p in data:
print(round(p_euclid_dist(p), 3))
1.0
2.236
2.236
3.606
4.243
5.0
5.831
6.325
7.071
8.602
Or for instance, one of the function's arguments changes in an outer loop but is fixed during iteration in the inner loop. By using a partial, you don't have to pass in the additional parameter during iteration of the inner loop, because the modified (partial) function doesn't require it.
>>> from functools import partial
>>> def fnx(a, b, c):
return a + b + c
>>> fnx(3, 4, 5)
12
create a partial function (using keyword arg)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(b=4, c=5)
21
you can also create a partial function with a positional argument
>>> pfnx = partial(fnx, 12)
>>> pfnx(4, 5)
21
but this will throw (e.g., creating partial with keyword argument then calling using positional arguments)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(4, 5)
Traceback (most recent call last):
File "<pyshell#80>", line 1, in <module>
pfnx(4, 5)
TypeError: fnx() got multiple values for keyword argument 'a'
another use case: writing distributed code using python's multiprocessing library. A pool of processes is created using the Pool method:
>>> import multiprocessing as MP
>>> # create a process pool:
>>> ppool = MP.Pool()
Pool has a map method, but it only takes a single iterable, so if you need to pass in a function with a longer parameter list, re-define the function as a partial, to fix all but one:
>>> ppool.map(pfnx, [4, 6, 7, 8])
short answer, partial gives default values to the parameters of a function that would otherwise not have default values.
from functools import partial
def foo(a,b):
return a+b
bar = partial(foo, a=1) # equivalent to: foo(a=1, b)
bar(b=10)
#11 = 1+10
bar(a=101, b=10)
#111=101+10
Partials can be used to make new derived functions that have some input parameters pre-assigned
To see some real world usage of partials, refer to this really good blog post here
A simple but neat beginner's example from the blog, covers how one might use partial on re.search to make code more readable. re.search method's signature is:
search(pattern, string, flags=0)
By applying partial we can create multiple versions of the regular expression search to suit our requirements, so for example:
is_spaced_apart = partial(re.search, '[a-zA-Z]\s\=')
is_grouped_together = partial(re.search, '[a-zA-Z]\=')
Now is_spaced_apart and is_grouped_together are two new functions derived from re.search that have the pattern argument applied(since pattern is the first argument in the re.search method's signature).
The signature of these two new functions(callable) is:
is_spaced_apart(string, flags=0) # pattern '[a-zA-Z]\s\=' applied
is_grouped_together(string, flags=0) # pattern '[a-zA-Z]\=' applied
This is how you could then use these partial functions on some text:
for text in lines:
if is_grouped_together(text):
some_action(text)
elif is_spaced_apart(text):
some_other_action(text)
else:
some_default_action()
You can refer the link above to get a more in depth understanding of the subject, as it covers this specific example and much more..
In my opinion, it's a way to implement currying in python.
from functools import partial
def add(a,b):
return a + b
def add2number(x,y,z):
return x + y + z
if __name__ == "__main__":
add2 = partial(add,2)
print("result of add2 ",add2(1))
add3 = partial(partial(add2number,1),2)
print("result of add3",add3(1))
The result is 3 and 4.
This answer is more of an example code. All the above answers give good explanations regarding why one should use partial. I will give my observations and use cases about partial.
from functools import partial
def adder(a,b,c):
print('a:{},b:{},c:{}'.format(a,b,c))
ans = a+b+c
print(ans)
partial_adder = partial(adder,1,2)
partial_adder(3) ## now partial_adder is a callable that can take only one argument
Output of the above code should be:
a:1,b:2,c:3
6
Notice that in the above example a new callable was returned that will take parameter (c) as it's argument. Note that it is also the last argument to the function.
args = [1,2]
partial_adder = partial(adder,*args)
partial_adder(3)
Output of the above code is also:
a:1,b:2,c:3
6
Notice that * was used to unpack the non-keyword arguments and the callable returned in terms of which argument it can take is same as above.
Another observation is:
Below example demonstrates that partial returns a callable which will take the
undeclared parameter (a) as an argument.
def adder(a,b=1,c=2,d=3,e=4):
print('a:{},b:{},c:{},d:{},e:{}'.format(a,b,c,d,e))
ans = a+b+c+d+e
print(ans)
partial_adder = partial(adder,b=10,c=2)
partial_adder(20)
Output of the above code should be:
a:20,b:10,c:2,d:3,e:4
39
Similarly,
kwargs = {'b':10,'c':2}
partial_adder = partial(adder,**kwargs)
partial_adder(20)
Above code prints
a:20,b:10,c:2,d:3,e:4
39
I had to use it when I was using Pool.map_async method from multiprocessing module. You can pass only one argument to the worker function so I had to use partial to make my worker function look like a callable with only one input argument but in reality my worker function had multiple input arguments.
Also worth to mention, that when partial function passed another function where we want to "hard code" some parameters, that should be rightmost parameter
def func(a,b):
return a*b
prt = partial(func, b=7)
print(prt(4))
#return 28
but if we do the same, but changing a parameter instead
def func(a,b):
return a*b
prt = partial(func, a=7)
print(prt(4))
it will throw error,
"TypeError: func() got multiple values for argument 'a'"
Adding couple of case from machine learning where the functional programming currying with functools.partial can be quite useful:
Build multiple models on the same dataset
the following example shows how linear regression, support vector machine and random forest regression models can be fitted on the same diabetes dataset, to predict the target and compute the score.
The (partial) function classify_diabetes() is created from the function classify_data() by currying (using functools.partial()). The later function does not require the data to be passed anymore and we can straightaway pass only the instances of the classes for the models.
from functools import partial
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import load_diabetes
def classify_data(data, model):
reg = model.fit(data['data'], data['target'])
return model.score(data['data'], data['target'])
diabetes = load_diabetes()
classify_diabetes = partial(classify_data, diabetes) # curry
for model in [LinearRegression(), SVR(), RandomForestRegressor()]:
print(f'model {type(model).__name__}: score = {classify_diabetes(model)}')
# model LinearRegression: score = 0.5177494254132934
# model SVR: score = 0.2071794500005485
# model RandomForestRegressor: score = 0.9216794155402649
Setting up the machine learning pipeline
Here the function pipeline() is created with currying which already uses StandardScaler() to preprocess (scale / normalize) the data prior to fitting the model on it, as shown in the next example:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipeline = partial(make_pipeline, StandardScaler()) # curry
for model in [LinearRegression(), SVR(), RandomForestRegressor()]:
print(f"model {type(model).__name__}: " \
f"score = {pipeline(model).fit(diabetes['data'], diabetes['target'])\
.score(diabetes['data'], diabetes['target'])}")
# model LinearRegression: score = 0.5177494254132934
# model SVR: score = 0.2071794500005446
# model RandomForestRegressor: score = 0.9180227193805106
I need to run several functions in a module as follws:
mylist = open('filing2.txt').read()
noTables = remove_tables(mylist)
newPassage = clean_text_passage(noTables)
replacement = replace(newPassage)
ncount = count_words(replacement)
riskcount = risk_count(ncount)
Is there any way that I can run all the functions at once? Should I make all the functions into a big function and run that big function?
Thanks.
You should make a new function in the module which executes the common sequence being used. This will require you to figure out what input arguments are required and what results to return. So given the code you posted, the new function might look something like this -- I just guessed as to what final results you might be interested in. Also note that I opened the file within a with statement to ensure that it gets closed after reading it.
def do_combination(file_name):
with open(file_name) as input:
mylist = input.read()
noTables = remove_tables(mylist)
newPassage = clean_text_passage(noTables)
replacement = replace(newPassage)
ncount = count_words(replacement)
riskcount = risk_count(ncount)
return replacement, riskcount
Example of usage:
replacement, riskcount = do_combination('filing2.txt')
If you simply store these lines in a Python (.py) file you can simply execute them.
Or am I missing something here?
Creating a function is also easy to call them though:
def main():
mylist = open('filing2.txt').read()
noTables = remove_tables(mylist)
newPassage = clean_text_passage(noTables)
replacement = replace(newPassage)
ncount = count_words(replacement)
riskcount = risk_count(ncount)
main()
As far as I understood, use need function composition. There is no special function for this in Python stdlib, but you can do this with reduce function:
funcs = [remove_tables, clean_text_passage, replace, count_words, risk_count]
do_all = lambda args: reduce(lambda prev, f: f(prev), funcs, args)
Using as
with open('filing2.txt') as f:
riskcount = do_all(f.read())
Here's another approach.
You could write a general function somewhat like that shown in the First-class composition section of the Wikipedia article on Function composition. Note that unlike in the article the functions are applied in the the order they are listed in the call to compose().
try:
from functools import reduce # Python 3 compatibility
except:
pass
def compose(*funcs, **kwargs):
"""Compose a series of functions (...(f3(f2(f1(*args, **kwargs))))) into
a single composite function which passes the result of each
function as the argument to the next, from the first to last
given.
"""
return reduce(lambda f, g:
lambda *args, **kwargs: f(g(*args, **kwargs)),
reversed(funcs))
Here's a trivial example illustrating what it does:
f = lambda x: 'f({!r})'.format(x)
g = lambda x: 'g({})'.format(x)
h = lambda x: 'h({})'.format(x)
my_composition = compose(f, g, h)
print my_composition('X')
Output:
h(g(f('X')))
Here's how it could be applied to the series of functions in your module:
my_composition = compose(remove_tables, clean_text_passage, replace,
count_words, risk_count)
with open('filing2.txt') as input:
riskcount = my_composition(input.read())