I have a little problem.
I use argparse to parse my arguments, and it's working very well.
To have the args, I do :
p_args = parser.parse_args(argv)
args = dict(p_args._get_kwargs())
But the problem with p_args is that I don't know how to get these arguments ordered by their position in the command line, because it's a dict.
So is there any possibility to have the arguments in a tuple/list/ordered dict by their order in the command line?
To keep arguments ordered, I use a custom action like this:
import argparse
class CustomAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if not 'ordered_args' in namespace:
setattr(namespace, 'ordered_args', [])
previous = namespace.ordered_args
previous.append((self.dest, values))
setattr(namespace, 'ordered_args', previous)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', action=CustomAction)
parser.add_argument('--test2', action=CustomAction)
To use it, for example:
>>> parser.parse_args(['--test2', '2', '--test1', '1'])
Namespace(ordered_args=[('test2', '2'), ('test1', '1')], test1=None, test2=None)
If you need to know the order in which the arguments appear in your parser, you can set up the parser like this:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('--optional1')
parser.add_argument('positionals', nargs='+')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
Here's a quick example of running this code:
$ python s.py --optional1 X --optional2 Y 1 2 3 4 5
['1', '2', '3', '4', '5']
Note that args.positionals is a list with the positional arguments in order. See the argparse documentation for more information.
This is a bit fragile since it relies on understanding the internals of argparse.ArgumentParser, but in lieu of rewriting argparse.ArgumentParser.parse_known_args, here's what I use:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_arg_order"] = []
self.__dict__["_arg_order_first_time_through"] = True
argparse.Namespace.__init__(self, **kwargs)
def __setattr__(self, name, value):
#print("Setting %s -> %s" % (name, value))
self.__dict__[name] = value
if name in self._arg_order and hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
self.__dict__["_arg_order"].append(name)
def _finalize(self):
if hasattr(self, "_arg_order_first_time_through"):
self.__dict__["_arg_order"] = []
delattr(self, "_arg_order_first_time_through")
def _latest_of(self, k1, k2):
try:
print self._arg_order
if self._arg_order.index(k1) > self._arg_order.index(k2):
return k1
except ValueError:
if k1 in self._arg_order:
return k1
return k2
This works through the knowledge that argparse.ArgumentParser.parse_known_args runs through the entire option list once setting default values for each argument. Meaning that user specified arguments begin the first time __setattr__ hits an argument that it's seen before.
Usage:
options, extra_args = parser.parse_known_args(sys.argv, namespace=OrderedNamespace())
You can check options._arg_order for the order of user specified command line args, or use options._latest_of("arg1", "arg2") to see which of --arg1 or --arg2 was specified later on the command line (which, for my purposes was what I needed: seeing which of two options would be the overriding one).
UPDATE: had to add _finalize method to handle pathological case of sys.argv() not containing any arguments in the list)
There is module especially made to handle this :
https://github.com/claylabs/ordered-keyword-args
without using orderedkwargs module
def multiple_kwarguments(first , **lotsofothers):
print first
for i,other in lotsofothers:
print other
return True
multiple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
output:
first
second
fifth
fourth
third
On using orderedkwargs module
from orderedkwargs import ordered kwargs
#orderedkwargs
def mutliple_kwarguments(first , *lotsofothers):
print first
for i, other in lotsofothers:
print other
return True
mutliple_kwarguments("first", second="second", third="third" ,fourth="fourth" ,fifth="fifth")
Output:
first
second
third
fourth
fifth
Note: Single asterik is required while using this module with decorator above the function.
I needed this because, for logging purposes, I liked to print the arguments after they were parsed. The problem was that the arguments are not printed in order, which was really annoying.
The custom action class just flat out did not work for me. I had other arguments which used a different action such as 'store_true' and default arguments also don't work since the custom action class is not called if the argument is not given in the command line. What worked for me was creating a wrapper class like this:
import collections
from argparse import ArgumentParser
class SortedArgumentParser():
def __init__(self, *args, **kwargs):
self.ap = ArgumentParser(*args, **kwargs)
self.args_dict = collections.OrderedDict()
def add_argument(self, *args, **kwargs):
self.ap.add_argument(*args, **kwargs)
# Also store dest kwarg
self.args_dict[kwargs['dest']] = None
def parse_args(self):
# Returns a sorted dictionary
unsorted_dict = self.ap.parse_args().__dict__
for unsorted_entry in unsorted_dict:
self.args_dict[unsorted_entry] = unsorted_dict[unsorted_entry]
return self.args_dict
The pros are that the add_argument method should have the exact same functionality as the original ArgumentParser. The cons are that if you want other methods you will have to write wrapped for all of them. Luckily for me all I ever used was add_argument and parse_args, so this served my purposes pretty well. You would also need to do more work if you wanted to use parent ArgumentParsers.
This is my simple solution based on the existing ones:
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_order"] = [None]
super().__init__(**kwargs)
def __setattr__(self, attr, value):
super().__setattr__(attr, value)
if attr in self._order:
self.__dict__["_order"].clear()
self.__dict__["_order"].append(attr)
def ordered(self):
if self._order and self._order[0] is None:
self._order.clear()
return ((attr, getattr(self, attr)) for attr in self._order)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', default=1)
parser.add_argument('--test2')
parser.add_argument('-s', '--slong', action='store_false')
parser.add_argument('--test3', default=3)
args = parser.parse_args(['--test2', '2', '--test1', '1', '-s'], namespace=OrderedNamespace())
print(args)
print(args.test1)
for a, v in args.ordered():
print(a, v)
Output:
OrderedNamespace(_order=['test2', 'test1', 'slong'], slong=False, test1='1', test2='2', test3=3)
1
test2 2
test1 1
slong False
It allows actions in add_argument(), which is harder for customized action class solution.
Related
I want to call a module function with getattr (or maybe something else?) from a string like this:
import bar
funcStr = "myFunc(\"strParam\", 123, bar.myenum.val1, kwarg1=\"someString\", kwarg2=456, kwarg3=bar.myenum.val2)"
[function, args, kwargs] = someParsingFunction(funcStr)
# call module function
getattr(bar, function)(*args, **kwargs)
How can I extract the args and kwargs from the string, so I can pass them to getattr?
I tried a literal_eval approach with pythons ast module. But ast is not able to evaluate the enums of the module bar. And all other examples on SO pass a kwargs map with only strings in it. And they especially never parse the arguments from a string. Or is there another way to directly call the function from the string?
EDIT: A python script reads the function string from file. So using eval here is not advised.
EDIT2: Using python 3.6.3
EDIT3: Thanks to the first two answers I came up with two ideas. After parsing the args and kwargs out of the input string there are two possibilities for getting the right type of the arguments.
We could use ast.literal_eval(<value of the argument>). For arguments with standard type like in kwarg2 it will return the needed value. If this excepts, which will happen for the enums, then we we will use getattr on the bar module and get the enums. If this excepts as well, then the string is not valid.
We could use the inspect module and iterate through the parameters of myFunc. Then for every arg and kwarg we will check if the value is an instance of a myFunc parameter (type). If so, we will cast the arg/kwarg to the myFunc parameter type. Otherwise we raise an exception because the given arg/kwarg is not an instance of a myFunc parameter. This solution is more flexible than the first one.
Both solutions feel more like a workaround. First tests seem to work. I will post my results later here.
Does this help?
funcStr = r"""myFunc(\"strParam\", 123, bar.myenum.val1, kwarg1=\"someString\", kwarg2=456, kwarg3=bar.myenum.val2)"""
def someParsingFunction(s):
func, s1 = s.split('(', 1)
l = s1.replace('\\','').strip(')').split(', ')
arg_ = [x.strip('"') for x in l if '=' not in x]
kwarg_ = {x.split('=')[0]:x.split('=')[-1] for x in l if '=' in x}
return func, arg_, kwarg_
class bar:
def myFunc(self, *args, **kwargs):
print(*args)
print(kwargs)
[function, args, kwargs] = someParsingFunction(funcStr)
getattr(bar, function)(*args, **kwargs)
# 123 bar.myenum.val1
# {'kwarg1': '"someString"', 'kwarg2': '456', 'kwarg3': 'bar.myenum.val2'}
Alternatively
funcStr = r"""myFunc(\"strParam\", 123, bar.val1, kwarg1=\"someString\", kwarg2=456, kwarg3=bar.val2)"""
def someParsingFunction(s):
func, s1 = s.split('(', 1)
l = s1.replace('\\','').strip(')').split(', ')
arg_ = [x.strip('"') for x in l if '=' not in x]
kwarg_ = {x.split('=')[0]:x.split('=')[-1] for x in l if '=' in x}
return func, arg_, kwarg_
class Bar:
def __init__(self):
self.val1 = '111'
[function, args, kwargs] = someParsingFunction(funcStr)
bar = Bar()
obj_name = 'bar' + '.'
args = [bar.__getattribute__(x.split(obj_name)[-1]) if x.startswith(obj_name) else x for x in args]
print(args)
def get_bar_args(arg_str):
"""
example:
arg_str='bar.abc.def'
assumess 'bar' module is imported
"""
from functools import reduce
reduce(getattr, arg_str.split('.')[1:], bar)
def parseFuncString(func_str):
'''
example: func_str = "myFunc(\"strParam\", 123, bar.myenum.val1, kwarg1=\"someString\", kwarg2=456, kwarg3=bar.myenum.val2)"
'''
import re
all_args_str = re.search("(.*)\((.*)\)", func_str)
all_args = all_args_str.group(2).split(',')
all_args = [x.strip() for x in all_args]
kwargs = {kw.group(1): kw.group(2) for x in all_args if (kw:=re.search('(^\w+)=(.*)$', x))}
pargs = [x for x in all_args if not re.search('(^\w+)=(.*)$', x)]
pargs = [get_bar_args(x) if x.startswith('bar.') else x for x in pargs]
kwargs = {k: get_bar_args(v) if v.startswith('bar.') else v for k, v in kwargs.items()}
print(f'{all_args=}\n{kwargs=}\n{pargs=}')
func_name = func_str.split("(")[0]
return func_name, pargs, kwargs
Basically imagine that I have argparser that has multiple arguments.
I have a particular function definition that looks like this:
def add_to_parser(self, parser):
group = parser.add_argument_group('')
group.add_argument( '--deprecateThis', action='throw exception', help='Stop using this. this is deprecated')
Whether I can try and create that action to throw an exception and stop the code or if I can wrap it to check for the deprecateThis flag and then throw an exception, I'd like to know how to do it and which is best! Thanks.
Here's what I came up with:
You can register custom actions for your arguments, I registered one to print out a deprecation warning and remove the item from the resulting namespace:
class DeprecateAction(argparse.Action):
def __init__(self, *args, **kwargs):
self.call_count = 0
if 'help' in kwargs:
kwargs['help'] = f'[DEPRECATED] {kwargs["help"]}'
super().__init__(*args, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
if self.call_count == 0:
sys.stderr.write(f"The option `{option_string}` is deprecated. It will be ignored.\n")
sys.stderr.write(self.help + '\n')
delattr(namespace, self.dest)
self.call_count += 1
if __name__ == "__main__":
my_parser = ArgumentParser('this is the description')
my_parser.register('action', 'ignore', DeprecateAction)
my_parser.add_argument(
'-f', '--foo',
help="This argument is deprecated",
action='ignore')
args = my_parser.parse_args()
# print(args.foo) # <- would throw an exception
I have the following test-code
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--verbose", default = 0, type=int)
subparsers = parser.add_subparsers(dest = "parser_name")
parser_lan = subparsers.add_parser('car')
parser_lan.add_argument("--boo")
parser_lan.add_argument("--foo")
parser_serial = subparsers.add_parser('bus')
parser_serial.add_argument("--fun")
print parser.parse_args()
which defines two sub-parsers, having a different set of arguments. When I call the testcode as
tester.py --verbose 3 car --boo 1 --foo 2
I get the expected result
Namespace(boo='1', foo='2', parser_name='car', verbose=3)
What I want to have instead is the values from each subparser in a separate namespace or dict, something like
Namespace(subparseargs={boo:'1', foo:'2'}, parser_name='car', verbose=3)
so that the arguments from each subparser are logical separated from the arguments from the main parser (as verbose in this example).
How can I achieve this, with the arguments for each subparser in the same namespace (subparseargs in the example).
You need to go into the bowels of argparse a bit but changing your script to the following should do the trick:
import argparse
from argparse import _HelpAction, _SubParsersAction
class MyArgumentParser(argparse.ArgumentParser):
def parse_args(self, *args, **kw):
res = argparse.ArgumentParser.parse_args(self, *args, **kw)
from argparse import _HelpAction, _SubParsersAction
for x in parser._subparsers._actions:
if not isinstance(x, _SubParsersAction):
continue
v = x.choices[res.parser_name] # select the subparser name
subparseargs = {}
for x1 in v._optionals._actions: # loop over the actions
if isinstance(x1, _HelpAction): # skip help
continue
n = x1.dest
if hasattr(res, n): # pop the argument
subparseargs[n] = getattr(res, n)
delattr(res, n)
res.subparseargs = subparseargs
return res
parser = MyArgumentParser()
parser.add_argument("--verbose", default = 0, type=int)
subparsers = parser.add_subparsers(dest = "parser_name")
parser_lan = subparsers.add_parser('car')
parser_lan.add_argument("--boo")
parser_lan.add_argument("--foo")
parser_serial = subparsers.add_parser('bus')
parser_serial.add_argument("--fun")
print parser.parse_args()
I have started to develop a different approach (but similar to the suggestion by Anthon) and come up with a much shorter code. However, I am not sure my approach is a general solution for the problem.
To similar what Anthon is proposing, I define a new method which creates a list of 'top-level' arguments which are kept in args, while all the other arguments are returned as an additional dictionary:
class MyArgumentParser(argparse.ArgumentParser):
def parse_subargs(self, *args, **kw):
# parse as usual
args = argparse.ArgumentParser.parse_args(self, *args, **kw)
# extract the destination names for top-level arguments
topdest = [action.dest for action in parser._actions]
# loop over all arguments given in args
subargs = {}
for key, value in args.__dict__.items():
# if sub-parser argument found ...
if key not in topdest:
# ... remove from args and add to dictionary
delattr(args,key)
subargs[key] = value
return args, subargs
Comments on this approach welcome, especially any loopholes I overlooked.
Or manually, you could parse the args and create a dict with details:
# parse args
args = parser.parse_args()
args_dict = {}
for group in parser._action_groups:
# split into groups based on title
args_dict[group.title] = {}
for arg in group._group_actions:
if hasattr(args, arg.dest):
args_dict[group.title][arg.dest] = getattr(args, arg.dest)
# or args_dict[arg.dest] = getattr(args, arg.dest)
delattr(args, arg.dest)
# add remaining items into subparser options
args_dict["subparser"] |= vars(args)
return args_dict
I'm a python noob and I'm trying to solve my problems the 'pythonic' way. I have a class, who's __init__ method takes 6 parameters. I need to validate each param and throw/raise an Exception if any fails to validate.
Is this the right way?
class DefinitionRunner:
def __init__(self, canvasSize, flightId, domain, definitionPath, harPath):
self.canvasSize = canvasSize
self.flightId = flightId
self.domain = domain
self.harPath = harPath
self.definitionPath = definitionPath
... bunch of validation checks...
... if fails, raise ValueError ...
If you want the variables to be settable independently of __init__, you could use properties to implement validations in separate methods.
They work only for new style classes though, so you need to define the class as class DefinitionRunner(object)
So for example,
#property
def canvasSize(self):
return self._canvasSize
#canvasSize.setter
def canvasSize(self, value):
# some validation here
self._canvasSize = value
Broadly speaking, that looks like the way you'd do it. Though strictly speaking, you might as well do validation before rather than after assignment, especially if assignment could potentially be time or resource intensive. Also, style convention says not to align assignment blocks like you are.
I would do it like you did it. Except the validating stuff. I would validate in a setter method and use it to set the attributes.
You could do something like this. Make a validator for each type of input. Make a helper function to run validation:
def validate_and_assign(obj, items_d, validators):
#validate all entries
for key, validator in validators.items():
if not validator[key](items_d[key]):
raise ValueError("Validation for %s failed" % (key,))
#set all entries
for key, val in items_d.items():
setattr(obj, key, val)
Which you'd use like this:
class DefinitionRunner:
validators = {
'canvasSize': canvasSize_validator,
'flightId': flightId_validator,
'domain': domain_validator,
'definitionPath': definitionPath_validator,
'harPath': harPath_validator,
}
def __init__(self, canvasSize, flightId, domain, definitionPath, harPath):
validate_and_assign(self, {
'canvasSize': canvasSize,
'flightId': flightId,
'domain': domain,
'definitionPath': definitionPath,
'harPath': harPath,
}, DefinitionRunner.validators)
The validators might be the same function, of course, if the data type is the same.
I'm not sure if this is exactly "Pythonic", but I've defined a function decorator called require_type. (To be honest, I think I found it somewhere online.)
def require_type(my_arg, *valid_types):
'''
A simple decorator that performs type checking.
#param my_arg: string indicating argument name
#param valid_types: list of valid types
'''
def make_wrapper(func):
if hasattr(func, 'wrapped_args'):
wrapped = getattr(func, 'wrapped_args')
else:
body = func.func_code
wrapped = list(body.co_varnames[:body.co_argcount])
try:
idx = wrapped.index(my_arg)
except ValueError:
raise(NameError, my_arg)
def wrapper(*args, **kwargs):
def fail():
all_types = ', '.join(str(typ) for typ in valid_types)
raise(TypeError, '\'%s\' was type %s, expected to be in following list: %s' % (my_arg, all_types, type(arg)))
if len(args) > idx:
arg = args[idx]
if not isinstance(arg, valid_types):
fail()
else:
if my_arg in kwargs:
arg = kwargs[my_arg]
if not isinstance(arg, valid_types):
fail()
return func(*args, **kwargs)
wrapper.wrapped_args = wrapped
return wrapper
return make_wrapper
Then, to use it:
class SomeObject(object):
#require_type("prop1", str)
#require_type("prop2", numpy.complex128)
def __init__(self, prop1, prop2):
pass
I would like to know how to get order of optional argument passed from commandline to argparse
I have image processing class which is able to apply different actions to image - like rotate, crop, resize...
And order in which these actions are applied is often essential (for example: you want to crop image before you resize it)
I have this code:
parser = argparse.ArgumentParser(description='Image processing arguments')
parser.add_argument('source_file', help='source file')
parser.add_argument('target_file', help='target file')
parser.add_argument('-resize', nargs=2, help='resize image', metavar=('WIDTH', 'HEIGHT'))
parser.add_argument('-rotate', nargs=1, help='rotate image', metavar='ANGLE')
parser.add_argument('-crop', nargs=4, help='crop image', metavar=('START_X','START_Y','WIDTH','HEIGHT'))
ar = parser.parse_args()
print ar
But - no matter in which order I pass parameters to script:
cmd.py test.jpg test2.jpg -crop 10 10 200 200 -resize 450 300
cmd.py test.jpg test2.jpg -resize 450 300 -crop 10 10 200 200
in Namespace items are always in same order (alphabetical I suppose):
Namespace(crop=['10', '10', '200', '200'], resize=['450', '300'], rotate=None, source_file='test.jpg', target_file='test
2.jpg')
Is there way to order them by position in command line string or to get their index?
You could always peek at sys.argv which is a list (and thus ordered) and simply iterate over it checking which argument comes first or use the list.index() to see the respective positions of your keywords in the list...
sys.argv contains a list of the words entered in the command line (the delimiter of such "word" is a space unless a string was surrounded by quotation marks). This means that if the user entered something like ./my_proggie -resize 500 then sys.argv would contain a list like this: ['./my_proggie', '-resize', '500'].
The Namespace is a simple object whose str() lists its attributes according to the order of the keys in its __dict__. Attributes are set with setattr(namespace, dest, value).
One solution is to define a custom Namespace class. For example:
class OrderNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__['order'] = []
super(OrderNamespace, self).__init__(**kwargs)
def __setattr__(self,attr,value):
self.__dict__['order'].append(attr)
super(OrderNamespace, self).__setattr__(attr, value)
and use
args = parser.parse_args(None, OrderNamespace())
producing for your two examples
OrderNamespace(crop=..., order=[..., 'crop', 'resize'], resize=...)
OrderNamespace(crop=..., order=[..., 'resize', 'crop'], resize=...)
The order attribute gives the order in which the other attributes are set. The initial items are for defaults and the file positionals. Adding default=argparse.SUPPRESS to the arguments will suppress some of these items. This custom class could be more elaborate, using for example an OrderedDictionary, only noting the order for selected arguments, or using order to control the display of the attributes.
Another option is to use a custom Action class that creates this order attribute, e.g.
class OrderAction(argparse._StoreAction):
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values)
order = getattr(namespace, 'order') if hasattr(namespace, 'order') else []
order.append(self.dest)
setattr(namespace, 'order', order)
I adapted the approach from hpaulj:
class OrderNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__['order'] = []
super(OrderNamespace, self).__init__(**kwargs)
def __setattr__(self,attr,value):
if value:
self.__dict__['order'].append(attr)
super(OrderNamespace, self).__setattr__(attr, value)
parser.add_argument('-g',action='append',default=argparse.SUPPRESS,help='some action')
By adding the "if value:" ... you only get every used argument the correct number of times.
There is a problem with #Martin 's solution: it does not work with cases like this:
parser.add_argument('-s', '--slong', action='store_false')
Here is my solution:
import argparse
class OrderedNamespace(argparse.Namespace):
def __init__(self, **kwargs):
self.__dict__["_order"] = []
super().__init__(**kwargs)
def __setattr__(self, attr, value):
super().__setattr__(attr, value)
if attr in self._order:
self.__dict__["_order"].clear()
self.__dict__["_order"].append(attr)
def ordered(self):
return ((attr, getattr(self, attr)) for attr in self._order)
parser = argparse.ArgumentParser()
parser.add_argument('--test1', default=1)
parser.add_argument('--test2')
parser.add_argument('-s', '--slong', action='store_false')
parser.add_argument('--test3', default=3)
args = parser.parse_args(['--test2', '2', '--test1', '1', '-s'], namespace=OrderedNamespace())
print(args)
print(args.test1)
for a, v in args.ordered():
print(a, v)
Output is:
OrderedNamespace(_order=['test2', 'test1', 'slong'], slong=False, test1='1', test2='2', test3=3)
1
test2 2
test1 1
slong False