I am using a 3rd party library function which has a large number of positional and named arguments. The function is called from numerous points in my code with identical arguments/values.
For ease of maintenance I don't want to hard-code the dozens of identical arguments multiple times throughout my code. I was hoping there was a way of storing them once in a data structure so I just need to pass the data structure. Along the lines of the following:
Assume the signature of the function I'm calling looks like:
def lib_function(arg1, arg2, arg3=None, arg4=None):
Assume that throughout my code I want to call it with values of
a for arg1,
b for arg2
d for arg4
(and I'm not using arg3).
I've tried defining a data structure as follows:
arguments = ('a', 'b', {'arg4':'d'})
and using it like this:
res = lib_function(arguments)
but this is obviously not correct - the tuple and dict are not unpacked during the call but are handled as a single first argument.
Is there a way of doing this in Python?
The obvious alternative is to proxy lib_function in my code, with the arguments hard-coded in the proxy which is then called with no arguments. Something like:
def proxy_lib_function():
return lib_function('a', 'b', arg4='d')
res = proxy_lib_function()
However I wanted to check there isn't a more Pythonic way of doing this.
Separate positional and named arguments and use asterisk unpacking:
def lib_function(arg1, arg2, arg3=None, arg4=None):
print(locals())
args = ("a", "b")
kwargs = {"arg4": "d"}
res = lib_function(*args, **kwargs)
Your function is:
def lib_function(arg1, arg2, arg3=None, arg4=None):
Define another function:
def yourFunction(x):
return lib_function(x[0], x[1], x[2], x[3])
The "data structure" would be:
data = [yourData1, yourData2, yourData3, yourData4]
Then you can call your new function with this:
yourFunction(data)
How can I wrap print() so that I can add arbitrary strings to the beginning and end of the things that are passed as arguments to get printed?
def xprint(*args):
print("XXX", *args, "XXX")
xprint("hi", "yo", 4)
doesn't work.
Basically, I want my custom function xprint() to work like print() but add 'XXX' to the beginning and end of every output.
Will work for python 2 and 3 when there are no keyword arguments
def xprint(*args):
print( "XXX"+" ".join(map(str,args))+"XXX")
In [5]: xprint("hi", "yo", 4)
XXXhi yo 4XXX
For the python 3 print() function (or when using print_function from __future__ in python 2), keyword arguments may be present as well. To ensure these are passed use the form
def xprint(*args, **kwargs):
print( "XXX"+" ".join(map(str,args))+"XXX", **kwargs)
You can do the same thing without changing print function name. Just add below code in your script.
xprint = print
def print(*args, **kwargs):
# do whatever you want to do
xprint('statement before print')
xprint(*args, **kwargs)
print(f'hello')
How about this:
def xprint(*args):
args = ("XXX",)+args+("XXX",)
print(*args)
xprint("hi", "yo", 4)
output is XXX hi yo 4 XXX
def xprint(*args, **kwargs):
# put a generator on all elements in args that will
# use string.format to prepend / append XXX to all args
args = ( 'XXX' '{}' 'XXX'.format(i) for i in args)
# pass it on to print()
print(*args, **kwargs)
>>> xprint("hi", "yo", 4)
XXXhiXXX XXXyoXXX XXX4XXX
So what are keyword arguments?
Well just in case you wanna add more fancy decoration to the text modding the standard separator you can do that like this:
>>> xprint("hi", "yo", 4, Sep="=- -=")
XXXhiXXX=- -=XXXyoXXX=- -=XXX4XXX
By passing on **kwargs you'll enable xprint to retain that functionality.
Why I mention this? Seems that most authors of answers here don't know or care about.
Optional you may add the line
kwargs["file"]=sys.stderr
into xprint to make it output text to stderr instead of stdout.
In most cases I wrap print() for that purpose.
... and another inconspicuous neat thing that's worth mentioning here is
generator comprehension. These generators are a functional programming thing. Instead of immediately apply changes to the data/memory a generator just 'hooks on' the iterator for that data. To be more precise a generator expression just return a new iterator. That iterator is used to later to fetch the actual data in a for .. in .. loop. Only when the actual data is read the generator gets applied.
That may significant reduce the amount of data copy around in memory can so can lead to big savings in memory usage.
Changing the line to:
args = [ 'XXX' '{}' 'XXX'.format(i) for i in args]
Note: just the brackets change from () to [].
It's will turn the whole thing into a List comprehension expression. That will do work the more 'conservative' way. Return a new list with the changes and overwrite the old args list.
You Can Use this code as python user define module. save this file as custom_print.py
from termcolor import colored
xprint = print
def print(*args, **kwargs):
# print args
if len(args) > 0:
xprint("|> {}".format(*args))
# print Kwargs
count_ = 0
for key, value in kwargs.items():
# for black and white
# xprint("In [{}]:: {}\t {}".format(count_, key, value))
# For colored output
xprint(colored("In [{}]::".format(count_), "red"), colored("
{}\t".format(key), "green"), colored(":{}".format(value), "blue"))
count_ += 1
if __name__ == "__main__":
# demo Testing
# demo 1
print("hello World")
# Demo 2
print(100, 500,
name="Nitish Sharma",
age=25,
city="Motihari",
email= "example#gmail.com"
)
Now Create a main.py and import custom_print Like
from custom_print import *
print("Hello World",
name="Nitish Kumar",
age=25,
city="Motihari")
:: OUTPUT
|> Hello World
In [0]:: name :Nitish Kumar
In [1]:: age :25
In [2]:: city :Motihari
Output
I am not able to get my head on how the partial works in functools.
I have the following code from here:
>>> sum = lambda x, y : x + y
>>> sum(1, 2)
3
>>> incr = lambda y : sum(1, y)
>>> incr(2)
3
>>> def sum2(x, y):
return x + y
>>> incr2 = functools.partial(sum2, 1)
>>> incr2(4)
5
Now in the line
incr = lambda y : sum(1, y)
I get that whatever argument I pass to incr it will be passed as y to lambda which will return sum(1, y) i.e 1 + y.
I understand that. But I didn't understand this incr2(4).
How does the 4 gets passed as x in partial function? To me, 4 should replace the sum2. What is the relation between x and 4?
Roughly, partial does something like this (apart from keyword args support etc):
def partial(func, *part_args):
def wrapper(*extra_args):
args = list(part_args)
args.extend(extra_args)
return func(*args)
return wrapper
So, by calling partial(sum2, 4) you create a new function (a callable, to be precise) that behaves like sum2, but has one positional argument less. That missing argument is always substituted by 4, so that partial(sum2, 4)(2) == sum2(4, 2)
As for why it's needed, there's a variety of cases. Just for one, suppose you have to pass a function somewhere where it's expected to have 2 arguments:
class EventNotifier(object):
def __init__(self):
self._listeners = []
def add_listener(self, callback):
''' callback should accept two positional arguments, event and params '''
self._listeners.append(callback)
# ...
def notify(self, event, *params):
for f in self._listeners:
f(event, params)
But a function you already have needs access to some third context object to do its job:
def log_event(context, event, params):
context.log_event("Something happened %s, %s", event, params)
So, there are several solutions:
A custom object:
class Listener(object):
def __init__(self, context):
self._context = context
def __call__(self, event, params):
self._context.log_event("Something happened %s, %s", event, params)
notifier.add_listener(Listener(context))
Lambda:
log_listener = lambda event, params: log_event(context, event, params)
notifier.add_listener(log_listener)
With partials:
context = get_context() # whatever
notifier.add_listener(partial(log_event, context))
Of those three, partial is the shortest and the fastest.
(For a more complex case you might want a custom object though).
partials are incredibly useful.
For instance, in a 'pipe-lined' sequence of function calls (in which the returned value from one function is the argument passed to the next).
Sometimes a function in such a pipeline requires a single argument, but the function immediately upstream from it returns two values.
In this scenario, functools.partial might allow you to keep this function pipeline intact.
Here's a specific, isolated example: suppose you want to sort some data by each data point's distance from some target:
# create some data
import random as RND
fnx = lambda: RND.randint(0, 10)
data = [ (fnx(), fnx()) for c in range(10) ]
target = (2, 4)
import math
def euclid_dist(v1, v2):
x1, y1 = v1
x2, y2 = v2
return math.sqrt((x2 - x1)**2 + (y2 - y1)**2)
To sort this data by distance from the target, what you would like to do of course is this:
data.sort(key=euclid_dist)
but you can't--the sort method's key parameter only accepts functions that take a single argument.
so re-write euclid_dist as a function taking a single parameter:
from functools import partial
p_euclid_dist = partial(euclid_dist, target)
p_euclid_dist now accepts a single argument,
>>> p_euclid_dist((3, 3))
1.4142135623730951
so now you can sort your data by passing in the partial function for the sort method's key argument:
data.sort(key=p_euclid_dist)
# verify that it works:
for p in data:
print(round(p_euclid_dist(p), 3))
1.0
2.236
2.236
3.606
4.243
5.0
5.831
6.325
7.071
8.602
Or for instance, one of the function's arguments changes in an outer loop but is fixed during iteration in the inner loop. By using a partial, you don't have to pass in the additional parameter during iteration of the inner loop, because the modified (partial) function doesn't require it.
>>> from functools import partial
>>> def fnx(a, b, c):
return a + b + c
>>> fnx(3, 4, 5)
12
create a partial function (using keyword arg)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(b=4, c=5)
21
you can also create a partial function with a positional argument
>>> pfnx = partial(fnx, 12)
>>> pfnx(4, 5)
21
but this will throw (e.g., creating partial with keyword argument then calling using positional arguments)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(4, 5)
Traceback (most recent call last):
File "<pyshell#80>", line 1, in <module>
pfnx(4, 5)
TypeError: fnx() got multiple values for keyword argument 'a'
another use case: writing distributed code using python's multiprocessing library. A pool of processes is created using the Pool method:
>>> import multiprocessing as MP
>>> # create a process pool:
>>> ppool = MP.Pool()
Pool has a map method, but it only takes a single iterable, so if you need to pass in a function with a longer parameter list, re-define the function as a partial, to fix all but one:
>>> ppool.map(pfnx, [4, 6, 7, 8])
short answer, partial gives default values to the parameters of a function that would otherwise not have default values.
from functools import partial
def foo(a,b):
return a+b
bar = partial(foo, a=1) # equivalent to: foo(a=1, b)
bar(b=10)
#11 = 1+10
bar(a=101, b=10)
#111=101+10
Partials can be used to make new derived functions that have some input parameters pre-assigned
To see some real world usage of partials, refer to this really good blog post here
A simple but neat beginner's example from the blog, covers how one might use partial on re.search to make code more readable. re.search method's signature is:
search(pattern, string, flags=0)
By applying partial we can create multiple versions of the regular expression search to suit our requirements, so for example:
is_spaced_apart = partial(re.search, '[a-zA-Z]\s\=')
is_grouped_together = partial(re.search, '[a-zA-Z]\=')
Now is_spaced_apart and is_grouped_together are two new functions derived from re.search that have the pattern argument applied(since pattern is the first argument in the re.search method's signature).
The signature of these two new functions(callable) is:
is_spaced_apart(string, flags=0) # pattern '[a-zA-Z]\s\=' applied
is_grouped_together(string, flags=0) # pattern '[a-zA-Z]\=' applied
This is how you could then use these partial functions on some text:
for text in lines:
if is_grouped_together(text):
some_action(text)
elif is_spaced_apart(text):
some_other_action(text)
else:
some_default_action()
You can refer the link above to get a more in depth understanding of the subject, as it covers this specific example and much more..
In my opinion, it's a way to implement currying in python.
from functools import partial
def add(a,b):
return a + b
def add2number(x,y,z):
return x + y + z
if __name__ == "__main__":
add2 = partial(add,2)
print("result of add2 ",add2(1))
add3 = partial(partial(add2number,1),2)
print("result of add3",add3(1))
The result is 3 and 4.
This answer is more of an example code. All the above answers give good explanations regarding why one should use partial. I will give my observations and use cases about partial.
from functools import partial
def adder(a,b,c):
print('a:{},b:{},c:{}'.format(a,b,c))
ans = a+b+c
print(ans)
partial_adder = partial(adder,1,2)
partial_adder(3) ## now partial_adder is a callable that can take only one argument
Output of the above code should be:
a:1,b:2,c:3
6
Notice that in the above example a new callable was returned that will take parameter (c) as it's argument. Note that it is also the last argument to the function.
args = [1,2]
partial_adder = partial(adder,*args)
partial_adder(3)
Output of the above code is also:
a:1,b:2,c:3
6
Notice that * was used to unpack the non-keyword arguments and the callable returned in terms of which argument it can take is same as above.
Another observation is:
Below example demonstrates that partial returns a callable which will take the
undeclared parameter (a) as an argument.
def adder(a,b=1,c=2,d=3,e=4):
print('a:{},b:{},c:{},d:{},e:{}'.format(a,b,c,d,e))
ans = a+b+c+d+e
print(ans)
partial_adder = partial(adder,b=10,c=2)
partial_adder(20)
Output of the above code should be:
a:20,b:10,c:2,d:3,e:4
39
Similarly,
kwargs = {'b':10,'c':2}
partial_adder = partial(adder,**kwargs)
partial_adder(20)
Above code prints
a:20,b:10,c:2,d:3,e:4
39
I had to use it when I was using Pool.map_async method from multiprocessing module. You can pass only one argument to the worker function so I had to use partial to make my worker function look like a callable with only one input argument but in reality my worker function had multiple input arguments.
Also worth to mention, that when partial function passed another function where we want to "hard code" some parameters, that should be rightmost parameter
def func(a,b):
return a*b
prt = partial(func, b=7)
print(prt(4))
#return 28
but if we do the same, but changing a parameter instead
def func(a,b):
return a*b
prt = partial(func, a=7)
print(prt(4))
it will throw error,
"TypeError: func() got multiple values for argument 'a'"
Adding couple of case from machine learning where the functional programming currying with functools.partial can be quite useful:
Build multiple models on the same dataset
the following example shows how linear regression, support vector machine and random forest regression models can be fitted on the same diabetes dataset, to predict the target and compute the score.
The (partial) function classify_diabetes() is created from the function classify_data() by currying (using functools.partial()). The later function does not require the data to be passed anymore and we can straightaway pass only the instances of the classes for the models.
from functools import partial
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import load_diabetes
def classify_data(data, model):
reg = model.fit(data['data'], data['target'])
return model.score(data['data'], data['target'])
diabetes = load_diabetes()
classify_diabetes = partial(classify_data, diabetes) # curry
for model in [LinearRegression(), SVR(), RandomForestRegressor()]:
print(f'model {type(model).__name__}: score = {classify_diabetes(model)}')
# model LinearRegression: score = 0.5177494254132934
# model SVR: score = 0.2071794500005485
# model RandomForestRegressor: score = 0.9216794155402649
Setting up the machine learning pipeline
Here the function pipeline() is created with currying which already uses StandardScaler() to preprocess (scale / normalize) the data prior to fitting the model on it, as shown in the next example:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
pipeline = partial(make_pipeline, StandardScaler()) # curry
for model in [LinearRegression(), SVR(), RandomForestRegressor()]:
print(f"model {type(model).__name__}: " \
f"score = {pipeline(model).fit(diabetes['data'], diabetes['target'])\
.score(diabetes['data'], diabetes['target'])}")
# model LinearRegression: score = 0.5177494254132934
# model SVR: score = 0.2071794500005446
# model RandomForestRegressor: score = 0.9180227193805106
I have created a class MyClassthat contains a lot of simulation data. The class groups simulation results for different simulations that have a similar structure. The results can be retreived with a MyClass.get(foo) method. It returns a dictionary with simulationID/array pairs, array being the value of foo for each simulation.
Now I want to implement a method in my class to apply any function to all the arrays for foo. It should return a dictionary with simulationID/function(foo) pairs.
For a function that does not need additional arguments, I found the following solution very satisfying (comments always welcome :-) ):
def apply(self, function, variable):
result={}
for k,v in self.get(variable).items():
result[k] = function(v)
return result
However, for a function requiring additional arguments I don't see how to do it in an elegant way. A typical operation would be the integration of foo with bar as x-values like np.trapz(foo, x=bar), where both foo and bar can be retreived with MyClass.get(...)
I was thinking in this direction:
def apply(self, function_call):
"""
function_call should be a string with the complete expression to evaluate
eg: MyClass.apply('np.trapz(QHeat, time)')
"""
result={}
for SID in self.simulations:
result[SID] = eval(function_call, locals=...)
return result
The problem is that I don't know how to pass the locals mapping object. Or maybe I'm looking in a wrong direction. Thanks on beforehand for your help.
Roel
You have two ways. The first is to use functools.partial:
foo = self.get('foo')
bar = self.get('bar')
callable = functools.partial(func, foo, x=bar)
self.apply(callable, variable)
while the second approach is to use the same technique used by partial, you can define a function that accept arbitrary argument list:
def apply(self, function, variable, *args, **kwds):
result={}
for k,v in self.get(variable).items():
result[k] = function(v, *args, **kwds)
return result
Note that in both case the function signature remains unchanged. I don't know which one I'll choose, maybe the first case but I don't know the context on you are working on.
I tried to recreate (the relevant part of) the class structure the way I am guessing it is set up on your side (it's always handy if you can provide a simplified code example for people to play/test).
What I think you are trying to do is translate variable names to variables that are obtained from within the class and then use those variables in a function that was passed in as well. In addition to that since each variable is actually a dictionary of values with a key (SID), you want the result to be a dictionary of results with the function applied to each of the arguments.
class test:
def get(self, name):
if name == "valA":
return {"1":"valA1", "2":"valA2", "3":"valA3"}
elif name == "valB":
return {"1":"valB1", "2":"valB2", "3":"valB3"}
def apply(self, function, **kwargs):
arg_dict = {fun_arg: self.get(sim_args) for fun_arg, sim_args in kwargs.items()}
result = {}
for SID in arg_dict[kwargs.keys()[0]]:
fun_kwargs = {fun_arg: sim_dict[SID] for fun_arg, sim_dict in arg_dict.items()}
result[SID] = function(**fun_kwargs)
return result
def joinstrings(string_a, string_b):
return string_a+string_b
my_test = test()
result = my_test.apply(joinstrings, string_a="valA", string_b="valB")
print result
So the apply method gets an argument dictionary, gets the class specific data for each of the arguments and creates a new argument dictionary with those (arg_dict).
The SID keys are obtained from this arg_dict and for each of those, a function result is calculated and added to the result dictionary.
The result is:
{'1': 'valA1valB1', '3': 'valA3valB3', '2': 'valA2valB2'}
The code can be altered in many ways, but I thought this would be the most readable. It is of course possible to join the dictionaries instead of using the SID's from the first element etc.