Develop Reference

Decorator for partial application /and/or/ currying: How to determine if mandatory arguments are supplied?

I've been tinkering with decorators lately and (as an academic exercise) tried to implement a decorator that allows for partial application and/or currying of the decorated function. Furthermore this decorator should be optionally parameterizable and take a kwarg asap which determines if the decorated function should return as soon as all mandatory args/kwargs are aquired (default: asap=True) or if the decoratored function should keep caching args/kwargs until the function is called without arguments (asap=False).
Here is the decorator I came up with:
def partialcurry(_f=None, *, asap: bool=True):
""" Decorator; optionally parameterizable; Allows partial application /and/or/ currying of the decorated function F. Decorated F fires as soon as all mandatory args and kwargs are supplied, or, if ASAP=False, collects args and kwargs and fires only if F is called without args/kwargs. """
def _decor(f, *args, **kwargs):
_all_args, _all_kwargs = list(args), kwargs
#functools.wraps(f)
def _wrapper(*more_args, **more_kwargs):
nonlocal _all_args, _all_kwargs # needed for resetting, not mutating
_all_args.extend(more_args)
_all_kwargs.update(more_kwargs)
if asap:
try:
result = f(*_all_args, **_all_kwargs)
# reset closured args/kwargs caches
_all_args, _all_kwargs = list(), dict()
except TypeError:
result = _wrapper
return result
elif not asap:
if more_args or more_kwargs:
return _wrapper
else:
result = f(*_all_args, **_all_kwargs)
# again, reset closured args/kwargs caches
_all_args, _all_kwargs = list(), dict()
return result
return _wrapper
if _f is None:
return _decor
return _decor(_f)
### examples
#partialcurry
def fun(x, y, z=3):
return x, y, z
print(fun(1)) # preloaded function object
print(fun(1, 2)) # all mandatory args supplied; (1,1,2); reset
print(fun(1)(2)) # all mandatory args supplied; (1,2,3); reset
print()
#partialcurry(asap=False)
def fun2(x, y, z=3):
return x, y, z
print(fun2(1)(2, 3)) # all mandatory args supplied; preloaded function object
print(fun2()) # fire + reset
print(fun2(1)(2)) # all mandatory args supplied; preloaded function object
print(fun2(4)()) # load one more and fire + reset
I am sure that this can be generally improved (implementing this as a class would be a good idea for example) and any suggestions are much appreciated, my main question however is how to determine if all mandatory args/kwargs are supplied, because I feel like to check for a TypeError is too generic and could catch all kinds of TypeErrors. One idea would be to define a helper function that calculates the number of mandatory arguments, maybe something like this:
def _required_args_cnt(f):
""" Auxiliary function: Calculate the number of /required/ args of a function F. """
all_args_cnt = f.__code__.co_argcount + f.__code__.co_kwonlyargcount
def_args_cnt = len(f.__defaults__) if f.__defaults__ else 0
return all_args_cnt - def_args_cnt
Obviously unsatisfactory..
Any suggestions are much appreciated!

How to pass arguments to a function that itself is passed as a kwarg

In the following example I'm trying to pass arguments to a function that itself has been passed as a kwarg. I have not been successful in passing arguments to the function 'func' from within the class 'TestClass' in the following example:
import sys, threading; from threading import Thread
def func(kwargs):
print('IN:', sys._getframe(0).f_code.co_name)
for key, value in kwargs.items() :
print ('KEY:', key, ', VAL:', value, sep='')
class TestClass(Thread):
def __init__(self, name = sys._getframe(0).f_code.co_name, kwargs = None):
Thread.__init__(self)
self.name = name
self.kwargs = kwargs
print('IN:', self.name)
def run(self):
func = self.kwargs['func']
func_kwargs_inner = {'arg_1': 'INNER-1', 'arg_2': 'INNER-2'}
func() # how to pass func_kwargs_inner to func?
def main():
func_kwargs = {'arg_1': 'OUTER-1', 'arg_2': 'OUTER-2'} # these get passed
# func_kwargs = {} # func_kwargs never gets populated
kwargs = {'func': (lambda: func(func_kwargs))}
test = TestClass(name='my-test', kwargs=kwargs)
test.start()
print('PROGRAM END')
if __name__ == '__main__':
main()
If I try to pass 'func_kwargs_inner' to 'func()', I get syntax errors; if I leave the argument list empty - as in the example - the result is:
IN: my-test
IN: func
KEY:arg_1, VAL:OUTER-1
KEY:arg_2, VAL:OUTER-2
PROGRAM END
whereas the required output once I find a way to pass the arguments correctly is:
IN: my-test
IN: func
KEY:arg_1, VAL:INNER-1
KEY:arg_2, VAL:INNER-2
PROGRAM END
How do I pass 'func_kwargs_inner' to 'func()'?

It seems that if you do the obvious thing, then it will work, and that your code at present explicitly avoids passing the arguments that you want. Specifically, in your TestClass.run you are not passing any arguments to func but instead relies on function arguments that are hard-coded into the lambda expression. So change your line:
func() # how to pass func_kwargs_inner to func?
to pass the arguments:
func(func_kwargs_inner)
Then in main, instead of that lambda expression:
kwargs = {'func': (lambda: func(func_kwargs))}
simply pass the function object itself:
kwargs = {'func': func}
Then you get the expected output:
IN: my-test
IN: func
PROGRAM END
KEY:arg_1, VAL:INNER-1
KEY:arg_2, VAL:INNER-2

What does this to in python super().__init__(**kwargs)

I know that super is used to call the init method in the superclass, I'm having trouble understanding what kwargs does I know it takes key arguments
what does it do here?
class LoginScreen(GridLayout):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.cols = 2
self.add_widget(Label(text = "Username: "))
self.username = TextInput(multiline = False)
self.add_widget(self.username)
self.add_widget(Label(text="Password: "))
self.username = TextInput(multiline=False, password=True)
self.add_widget(self.username)
self.add_widget(Label(text="Two Factor Auth: "))
self.tffa = TextInput(multiline=False, password=True)
self.add_widget(self.tffa)

def __init__(self, **kwargs): packs all of the keyword arguments used in any given call to __init__ into a dict
super().__init__(**kwargs): expands them into keyword arguments again.
It's like a wildcard for function parameters. It can be a lazy way to give the subclass the same parameter signature as the parent without bothering to type all of the possible keyword parameters in again.
Just grab them as a blob and shovel them off to the parent to figure out.

For example you put this :
x = LoginScreen(size=1, blahblah=2, something=3)
We have now:
>>> print(kwargs)
{'size': 1, 'blahblah': 2, 'something': 3}
So if it reaches the following line: super().__init__(**kwargs)
it will be equal to that : super().__init__(size=1, blahblah=2, something=3)
Becarefully: if you didn't put double asterisks "**" it will be equal to one argument as key:
`super().__init__({'size': 1, 'blahblah': 2, 'something': 3})`
And you can also use it like that:
options = [1,2,3]
def func(first=None, second=None, last=None):
print("first arg:", first)
print("second arg:", second)
print("third arg:", last)
func(*options)
Here we use one asterisk since this is a list not a dict so that means expand the options list for each argument, the output will be:
first arg: 1
second arg: 2
third arg: 3
But if we call options without the asterisk.
func(options)
See what's gonna happen:
first arg: [1, 2, 3]
second arg: None
third arg: None
The samething with kwargs except they go like that:
fuction(key1=value1, key2=value2, key3=value3, ...)

Passing uncertain amount of inputs to one function's sub-functions

Sorry if I did not explain myself clearly.
I would like to create a wrapper to call pre-defined functions with different number of inputs. Of course, I can create an individual wrapper for each function, but I am wondering if there is a way to create a generic wrapper for all cases.
The functions that should be called are named 'fun1' and 'fun2' with different number of inputs. I need to create a wrapper 'fun_wrap(func_name, uncertain amount of inputs)', which only needs the function name to be called and its associated amount of inputs.
One more thing, I need to change the input names by adding '_in' and make them global variables first. Below is my broken code. Thanks for any suggestions!
def fun1(a,b):
return a+b
def fun2(a,b,c):
return a*b/c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func_name, uncertain amount of inputs):
ffunc_name(set_globals(uncertain amount of inputs))
In this way, if I can call final_fun with arguments like:
fun_wrap(fun1,a,b)
fun_wrap(fun2,a,b)
UPDATE
I tried to use *arg, but failed...
def fun1(a,b):
return a+b
def fun2(a,b,c):
return a*b/c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func_name, *arg):
func_name(set_globals(*arg))
fun_wrap(fun2,a=1,b=2,c=3)
got error:
Traceback (most recent call last):
File "D:\Dropbox\AppPest\rice\try.py", line 19, in <module>
fun_wrap(fun2,a=1,b=2,c=3)
TypeError: fun_wrap() got an unexpected keyword argument 'a'

def fun1(a,b):
return a + b
def fun2(a,b,c):
return a * b / c
def set_globals(**kwargs):
for argname in kwargs:
globals()['%s_in' % argname] = kwargs[argname]
def fun_wrap(func, **kwargs):
set_globals(**kwargs) # made the call to set_globals before calling your function
return func(**kwargs) # return the value returned by the functions called

argparse argument order

I have a little problem.
I use argparse to parse my arguments, and it's working very well.
To have the args, I do :
p_args = parser.parse_args(argv)
args = dict(p_args._get_kwargs())
But the problem with p_args is that I don't know how to get these arguments ordered by their position in the command line, because it's a dict.
So is there any possibility to have the arguments in a tuple/list/ordered dict by their order in the command line?

To keep arguments ordered, I use a custom action like this:
import argparse
class CustomAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if not 'ordered_args' in namespace:
setattr(namespace, 'ordered_args', [])
previous = namespace.ordered_args
previous.append((self.dest, values))
setattr(namespace, 'ordered_args', previous)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', action=CustomAction)
parser.add_argument('--test2', action=CustomAction)
To use it, for example:
>>> parser.parse_args(['--test2', '2', '--test1', '1'])
Namespace(ordered_args=[('test2', '2'), ('test1', '1')], test1=None, test2=None)

If you need to know the order in which the arguments appear in your parser, you can set up the parser like this:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('--optional1')
parser.add_argument('positionals', nargs='+')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
Here's a quick example of running this code:
$ python s.py --optional1 X --optional2 Y 1 2 3 4 5
['1', '2', '3', '4', '5']
Note that args.positionals is a list with the positional arguments in order. See the argparse documentation for more information.

This is a bit fragile since it relies on understanding the internals of argparse.ArgumentParser, but in lieu of rewriting argparse.ArgumentParser.parse_known_args, here's what I use:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_arg_order"] = []
self.__dict__["_arg_order_first_time_through"] = True
argparse.Namespace.__init__(self, **kwargs)
def __setattr__(self, name, value):
#print("Setting %s -> %s" % (name, value))
self.__dict__[name] = value
if name in self._arg_order and hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
self.__dict__["_arg_order"].append(name)
def _finalize(self):
if hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
def _latest_of(self, k1, k2):
try:
print self._arg_order
if self._arg_order.index(k1) > self._arg_order.index(k2):
return k1
except ValueError:
if k1 in self._arg_order:
return k1
return k2
This works through the knowledge that argparse.ArgumentParser.parse_known_args runs through the entire option list once setting default values for each argument. Meaning that user specified arguments begin the first time __setattr__ hits an argument that it's seen before.
Usage:
options, extra_args = parser.parse_known_args(sys.argv, namespace=OrderedNamespace())
You can check options._arg_order for the order of user specified command line args, or use options._latest_of("arg1", "arg2") to see which of --arg1 or --arg2 was specified later on the command line (which, for my purposes was what I needed: seeing which of two options would be the overriding one).
UPDATE: had to add _finalize method to handle pathological case of sys.argv() not containing any arguments in the list)

There is module especially made to handle this :
https://github.com/claylabs/ordered-keyword-args
without using orderedkwargs module
def multiple_kwarguments(first , **lotsofothers):
print first
for i,other in lotsofothers:
print other
return True
multiple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
output:
first
second
fifth
fourth
third
On using orderedkwargs module
from orderedkwargs import ordered kwargs
#orderedkwargs
def mutliple_kwarguments(first , *lotsofothers):
print first
for i, other in lotsofothers:
print other
return True
mutliple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
Output:
first
second
third
fourth
fifth
Note: Single asterik is required while using this module with decorator above the function.

I needed this because, for logging purposes, I liked to print the arguments after they were parsed. The problem was that the arguments are not printed in order, which was really annoying.
The custom action class just flat out did not work for me. I had other arguments which used a different action such as 'store_true' and default arguments also don't work since the custom action class is not called if the argument is not given in the command line. What worked for me was creating a wrapper class like this:
import collections
from argparse import ArgumentParser
class SortedArgumentParser():
def __init__(self, *args, **kwargs):
self.ap = ArgumentParser(*args, **kwargs)
self.args_dict = collections.OrderedDict()
def add_argument(self, *args, **kwargs):
self.ap.add_argument(*args, **kwargs)
# Also store dest kwarg
self.args_dict[kwargs['dest']] = None
def parse_args(self):
# Returns a sorted dictionary
unsorted_dict = self.ap.parse_args().__dict__
for unsorted_entry in unsorted_dict:
self.args_dict[unsorted_entry] = unsorted_dict[unsorted_entry]
return self.args_dict
The pros are that the add_argument method should have the exact same functionality as the original ArgumentParser. The cons are that if you want other methods you will have to write wrapped for all of them. Luckily for me all I ever used was add_argument and parse_args, so this served my purposes pretty well. You would also need to do more work if you wanted to use parent ArgumentParsers.

This is my simple solution based on the existing ones:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_order"] = [None]
super().__init__(**kwargs)
def __setattr__(self, attr, value):
super().__setattr__(attr, value)
if attr in self._order:
self.__dict__["_order"].clear()
self.__dict__["_order"].append(attr)
def ordered(self):
if self._order and self._order[0] is None:
self._order.clear()
return ((attr, getattr(self, attr)) for attr in self._order)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', default=1)
parser.add_argument('--test2')
parser.add_argument('-s', '--slong', action='store_false')
parser.add_argument('--test3', default=3)
args = parser.parse_args(['--test2', '2', '--test1', '1', '-s'], namespace=OrderedNamespace())
print(args)
print(args.test1)
for a, v in args.ordered():
print(a, v)
Output:
OrderedNamespace(_order=['test2', 'test1', 'slong'], slong=False, test1='1', test2='2', test3=3)
1
test2 2
test1 1
slong False
It allows actions in add_argument(), which is harder for customized action class solution.