Develop Reference

How to grab all defaults from `argparse`?

I have an argument parser like so:
def args_parser():
parser = argparse.ArgumentParser(description='myparser')
parser.add_argument('--k', type=int, default=100, help = 'numbers')
return parser
Now, all I want to do is just get the default mappings, like so:
args = args_parser()
And then I want args['k'] to return 100, since that is the default. How do I make this happen?

This should do that for you. I think you should be able to find this in the Python docs on ArgumentParser as well:
vars(args.parse_args())['k']

You need to call the .parse_args() function on your parser object in order to actually return the command-line arguments parsed and assigned to attributes as defined in your .add_argument() statement. So something like:
parser = args_parser()
args = parser.parse_args()
k_value = args.k
The 'args' variable returned by .parse_args() will have attributes assigned corresponding to the names you assign to it via the .add_argument() call, although if there is no default and the user does not provide a given argument its value will be 'None'.

def args_parser():
parser = argparse.ArgumentParser(description='myparser')
parser.add_argument('--k', type=int, default=100, help = 'numbers')
return parser
parser = args_parser():
args = parser.parse_args([])
print(args) # see all values
print(args.k) # see just the k attribute
parse_args() parses the sys.argv as provided in the commandline. parse_args([]) parses an empty list, same as if you had called the script without arguments. I regularly test a parser setup with calls like this.
With your argument definition the [] works fine and shows the default k value. With other defintions, [] might produce an error. For example if you included
parser.add_argument('infile', help='positional file name')
now a string is required. For a simple positional like that a default is not needed nor does it make sense. But other parameter combinations can make use of a default.
I could show you how to access the default as set in the add_argument command, but that involves some undocumented features. In addition there are several ways of defining defaults, so the topic can get complicated.
Anyways simply running the parse_args with an empty list is often enough.
You can also make it display the default in the help line with a little addition:
parser.add_argument('--k', type=int, default=100, help = 'numbers, %(default)s')
For example, in an interactive ipython session:
In [73]: parser=argparse.ArgumentParser()
In [74]: parser.add_argument('--k', type=int, default=100, help = 'numbers, %(default)s')
Out[74]: _StoreAction(option_strings=['--k'], dest='k', nargs=None,
const=None, default=100, type=<class 'int'>, choices=None,
help='numbers, %(default)s', metavar=None)
In [75]: parser.print_help()
usage: ipython3 [-h] [--k K]
optional arguments:
-h, --help show this help message and exit
--k K numbers, 100
In [76]: parser.parse_args([]) # displays args
Out[76]: Namespace(k=100)
In [78]: parser.parse_args(['--k','200'])
Out[78]: Namespace(k=200)

How to stop processing arguments when using argparse's CustomAction

I've implemented a CustomAction in argparse for my python project. The CustomAction is used to be able to specify any number of name=value pair style arguments on the command line ie nargs='*'.
class NameValueAction(argparse.Action):
""" CustomAction for argparse to be able to process name,value \
pairs specified as command line arguments. Specified as
$ python runner.py --env=target_env --props name1=value1 name2=value2 module/
"""
def __call__(self, parser, namespace, values, option_string=None):
for value in values:
n, v = value.split('=')
setattr(namespace, n, v)
The trouble is that there is no way to stop __call__ from processing the module/ argument that is on the command line. How can the __call__ method be appropriately ended without consuming the module/ argument and allow for it to be processed by the runner.py?
PS: I've tried exiting on the last argument that is not name=value but this does not work, since module has already been consumed and I don't know how to put it back on the stack.

I'll try your latest custom Action:
In [34]: parser=argparse.ArgumentParser()
In [35]: parser.add_argument('--env')
In [36]: parser.add_argument('--props',nargs='*',action=NameValueAction)
Out[36]: NameValueAction(option_strings=['--props'], dest='props', nargs='*', const=None, default=None, type=None, choices=None, help=None, metavar=None)
I get an unrecognized arguments error with parse_args. Your action correctly defined as unknown:
In [37]: args=parser.parse_args('--env=target_env --props name1=value1 name2=value2 module/'.split())
usage: ipython2.7 [-h] [--env ENV] [--props [PROPS [PROPS ...]]]
ipython2.7: error: unrecognized arguments: module/
...
With parse_known_args I can see the args and extras without the error message:
In [38]: parser.parse_known_args('--env=target_env --props name1=value1 name2=value2 module/'.split())
Out[38]:
(Namespace(env='target_env', name1='value1', name2='value2', props=None),
['module/'])
So all the strings after --props were passed as values to that Action. It assigned the values to the Namespace, and returned. parse_known_args took the unrecognized values out of the Namespace and put them in that extras list.
Now I will add a positional in hopes that it will take the module/ string:
In [39]: parser.add_argument('foo')
In [40]: parser.parse_known_args('--env=target_env --props name1=value1 name2=value2 module/'.split())
usage: ipython2.7 [-h] [--env ENV] [--props [PROPS [PROPS ...]]] foo
ipython2.7: error: too few arguments
...
Oops, a different error, even with parse_known_args. The problem is that 'module/' is still being given to --props, leaving nothing for foo. --props has a * nargs, which means it gets everything that qualifies as an argument (no -) that follows. Putting 'module/' in the namespace as unknown did not help. The parser does not reevaluate strings in this list.
I can use '--' to indicate that all strings that follow are positionals. Now --props does not receive or handle 'module\'. Instead it is consumed by foo the next time positionals are handled.
In [41]: parser.parse_known_args('--env=target_env --props name1=value1 name2=value2 -- module/'.split())
Out[41]:
(Namespace(env='target_env', foo='module/', name1='value1', name2='value2', props=None),
[])
Another optional, such as '--env' could be used to mark the end of the '--props' arguments:
In [42]: parser.parse_known_args('--props name1=value1 name2=value2 --env=target_env module/'.split())
Out[42]:
(Namespace(env='target_env', foo='module/', name1='value1', name2='value2', props=None),
[])
Note that progs=None appears in the Namespace. This is because the parser loads all Action defaults into the Namespace at the start of parsing. You could use default=argparse.SUPPRESS to prevent that.
See this bugs/issue for explanation on how arguments are allocated to a '*' optional, and what can be done to reserve some for following positionals:
http://bugs.python.org/issue9338
argparse optionals with nargs='?', '*' or '+' can't be followed by positionals
https://stackoverflow.com/a/33405960/901925 is another recent SO question that involves a regular positional followed by two '?' positionals.
As I noted in a comment, argparse is different from optparse. I believe in optparse each Action (or equivalent) consumes as many strings as it wants, and leaves the rest for following Actions. In argparse individual Actions don't have access to the master list (arg_strings). It's the parser that decides how many strings an Action gets.
More details from the argparse.py file. It's a summary of the relevant parts of parse_args.
_parse_known_args(self, arg_strings, namespace):
# arg_strings - master list of strings from sys.argv
start_index = 0
while start_index<amax:
# step through arg_strings processing postionals and optionals
consume_positionals()
start_index = next_option_string_index
start_index = consume_optional(start_index)
consume_optional(start_index): # function local to _parse_known_args
...
start = start_index + 1
arg_count = <fn of available arguments and nargs>
stop = start + arg_count
args = arg_strings[start:stop]
<action = CustomAction.__call__>
take_action(action, args, option_string)
return stop
take_action(action, argument_strings, ...): # another local function
# argument_strings is a slice of arg_strings
argument_values = self._get_values(action, argument_strings)
# _get_values passes strings through the action.type function
action(self, namespace, argument_values, option_string)
# no return
The net effect is that your CustomAction.__call__ gets a list of values that were derived from a slice of the master arg_strings list. It has no access to arg_strings, and no access to the start and stop of that slice. So it can't alter the allocation of strings to itself or any subsequent Actions.
Another idea is to put the values that you can't parse into self.dest.
class NameValueAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
extras = []
for value in values:
try:
n, v = value.split('=')
setattr(namespace, n, v)
except ValueError:
extras.append(value)
if len(extras):
setattr(namespace, self.dest, extras)
Then parsing (without the foo positional) would produce:
In [56]: parser.parse_args('--props name1=value1 p1 name2=value2 module/'.split())
Out[56]: Namespace(env=None, name1='value1', name2='value2', props=['p1', 'module/'])
args.props now contains ['p1','module/'], strings that --props got, but could not parse as n=v pairs. These can be realocated after parsing as needed.

There's no way* to prevent 'module/' from being consumed since it has no name or flags associated to indicate that it's a separate argument and not to be consumed by --props.
I assume you've setup --props as:
parser.add_argument('--props', nargs='*', action=NameValueAction)
so that will consume as many args as possible. You'd need to give a -m or --module option to get argparse to store 'module/' separately.
Otherwise, you could put module as a positional arg parser.add_argument('module') and specify it before --props on the command line:
parser.add_argument('--env')
parser.add_argument('--props', nargs='*', action=NameValueAction)
parser.add_argument('module')
""" Usage:
$ python runner.py --env=target_env module/ --props name1=value1 name2=value2
or
$ python runner.py module/ --env=target_env --props name1=value1 name2=value2
"""
That processes as:
>>> parser.parse_args('--env=target_env module/ --props name1=value1 name2=value2'.split())
Namespace(env='target_env', module='module/', name1='value1', name2='value2', props=None)
Btw, using your existing code and without the change suggested above, you could just specify module=module at the command line and it will process just like the name=value pairs:
>>> parser.parse_args('--env=target_env --props name1=value1 name2=value2 module=module/'.split())
Namespace(env='target_env', module='module/', name1='value1', name2='value2', props=None)
* If you really cannot put it as a separate arg, then you'll have to handle it within NameValueAction. I modified the __call__ in yours as:
def __call__(self, parser, namespace, values, option_string=None):
for value in values:
try:
n, v = value.split('=')
setattr(namespace, n, v) # better to put this in the else clause actually
except ValueError: # "need more than 1 value to unpack"
# raised when there's no '=' sign
setattr(namespace, 'module', value)
>>> parser.parse_args('--env=target_env --props name1=value1 name2=value2 MOARmodules/'.split())
Namespace(env='target_env', module='MOARmodules/', name1='value1', name2='value2', props=None)
Of course, the downside of that is how complex the remaining actions are. The one implemented above behaves like action=store and will only apply it to 'module'.
You could also try experimenting with appending values to sys.argv but considering that's being consumed while you're doing that, may have unexpected side-affects, similar to why you shouldn't insert/delete from a list while iterating over it.

After #aneroid's clue to look into handling within NameValueAction I read through the argparse module to find a possible way. Actions do the command line parsing in argparse. An Action under argparse is triggered on a part of the command line to the program. argparse maintains a list of default Actions (eg: store, store_true, const etc) and CustomAction objects that are defined by the user. These are then looped over and processed sequentially against a portion of the command line to find matches and build the Namespace corresponding to each Action. In each iteration argparse.Action may find that part of the command line does not match anything handled by the Action and returns them (in the field _UNRECOGNIZED_ARGS_ATTR which is identified by the attribute '_unrecognized_args' of Namespace) back to the caller
From argparse.py#parse_known_args(..):
try:
namespace, args = self._parse_known_args(args, namespace)
if hasattr(namespace, _UNRECOGNIZED_ARGS_ATTR):
args.extend(getattr(namespace, _UNRECOGNIZED_ARGS_ATTR))
delattr(namespace, _UNRECOGNIZED_ARGS_ATTR)
return namespace, args
except ArgumentError:
err = _sys.exc_info()[1]
self.error(str(err))
As seen above if any unrecognized arguments are found they are returned back to the caller in args. The NameValueAction class can utilise this to leave them for processing by any other Actions that follow or the project's (runner.py) module. The class changes thus:
class NameValueAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
for value in values:
try:
n, v = value.split('=')
setattr(namespace, n, v)
except ValueError:
# when input has ended without an option, probably at module name
setattr(namespace, '_unrecognized_args', values[values.index(value):])
So the cmd line works as follows:
$ python runner.py --env=target_env --props name1=value1 name2=value2 module/
In case additional options are specified after --props, argparse will stop processing the current Action and iterate forward. So the following will also work
$ python runner.py --env=target_env --props name1=value1 name2=value2 --timeout=300 module/

(Answering only because I also needed to "eat" a number of arguments unknown beforehand from the list and the solution below is reasonably generic.)
As #hpaulj mentioned above, using positional arguments won't work without subclassing ArgumentParser as the parser simply passes everything to the Action, however if you just want options parsed and get non-option arguments back as a list (ie. passing them to a different parser), the following works (on Python 3.4 at least):
#!/usr/bin/env python3
import argparse
import itertools
class EatUnknown(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, *args, **kwargs):
nargs = argparse.REMAINDER
super().__init__(option_strings, dest, nargs, *args, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
def all_opt_strings(parser):
nested = (x.option_strings for x in parser._actions
if x.option_strings)
return itertools.chain.from_iterable(nested)
all_opts = list(all_opt_strings(parser))
eaten = []
while len(values) > 0:
if values[0] in all_opts:
break
eaten.append(values.pop(0))
setattr(namespace, self.dest, eaten)
_, extras = parser._parse_known_args(values, namespace)
try:
getattr(namespace, argparse._UNRECOGNIZED_ARGS_ATTR).extend(extras)
except AttributeError:
setattr(namespace, argparse._UNRECOGNIZED_ARGS_ATTR, extras)
parser = argparse.ArgumentParser()
parser.add_argument("--foo", action="append")
parser.add_argument('--eatme', action=EatUnknown)
parser.add_argument('--eater', action=EatUnknown)
print(parser.parse_known_args())
Produces
$ ./argparse_eater.py --foo 1 AAA --eater 2 --unk-opt 3 --foo 4 BBB --eatme 5 --another-unk --foo 6 CCC
(Namespace(eater=['2', '--unk-opt', '3'], eatme=['5', '--another-unk'], foo=['1', '4', '6']), ['AAA', 'CCC', 'BBB'])
This example "eats up" any non-options as well as unknown option arguments (where nargs='*' cannot be used, justifying the example), though is not allow_abbrev compatible.
The idea is to use a simple recursion, which apparently works as the code is reentrant. Probably not the best idea to rely upon, but using _unrecognized_args is not much better.
Given the OP, this would work for multiple occurences of --props.

How do I get back the option string using argparse?

parser = argparse.ArgumentParser()
parser.add_argument("-p", "--pattern", help="Pattern file")
args = parser.parse_args()
Now is it possible to get back the string "--pattern" from args?
I need the string so that I can construct a cmd list to pass to Popen like Popen(['some_other_program', args.pattern.option_string, args.pattern], ...) without repeating it (and having to maintain it in two places) (Popen(['some_other_prog', '--pattern', args.pattern], ...)).
I need to create a wrapper for another program. Some of the args need to be passed to the wrapped program (via Popen) and some are required by the wrapper.
Is there a better method than the following example?
pass_1 = '--to-be-passed'
parser = argparse.ArgumentParser()
parser.add_argument("-p", pass_1, help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
...
process = Popen(['wrapped_program', pass_1, args.pass_1], ...)
...
This method of keeping the args in variables is not very good as:
Maintaining short options along with long options becomes difficult.
Popen if called in another function requires passing these variables(or a dict of them) to the function. This seems redundant as args passed to it should be sufficient.

Add a dest to your add_argument call.
parser.add_argmument("p", "--pattern", dest="pattern", help="your help text")
args = parser.parse_args()
args = vars(args)
The you can reference the pattern with args["pattern"] .

There doesn't seem to be an easy way to get the original option strings from the result of a parser.parse_args(), but you can get them from the parser object. You just need to peek into its __dict__, in order to retrieve the parser settings after it's created. In your case you want the _option_string_actions field. Unfortunately this doesn't seem officially supported, as I couldn't find a ArgumentParser method dedicated to this, so YMMV. On Python 3:
Demo:
parser = argparse.ArgumentParser()
parser.add_argument('--foo', '-f', type=int, default=1000, help='intensity of bar')
parser.add_argument('--bar', '-b', type=str, default='bla', help='whatever')
store_action_dict=vars(parser)['_option_string_actions']
print(store_action_dict.keys()) # dict_keys(['--help', '-b', '-f', '-h', '--foo', '--bar'])

The deleted answers and comments indicate there is some confusion as to what you want. So I'll add to that confusion.
Normally the parser does not record the option string. However it is provided to the Action __call__ method. So a custom Action class could save it. The FooAction custom class example in the argparse docs illustrates this.
If I define this action subclass:
In [324]: class PassThru(argparse._StoreAction):
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, [values, option_string])
In [324]: p.add_argument('-o','--other',action=PassThru)
The option string is recorded along with the value ('-o' or '--other'):
In [322]: p.parse_args('-p test -o teseting'.split())
Out[322]: Namespace(other=['teseting', '-o'], pass_me_on='test')
In [323]: p.parse_args('-p test --other teseting'.split())
Out[323]: Namespace(other=['teseting', '--other'], pass_me_on='test')
Obviously the option_string and value could be recorded in a different order, in a dictionary, as seperate attributes in the Namespace, etc.
There are other ways of passing options to another program, particularly if the wrapping parser does not need to handle them itself.
argparse gets the arguments from sys.argv[1:], and does not change it. So even if your parser uses some of the arguments, you could pass that list on to popen (all or in part).
The argparse docs has an example, under nargs=REMAINDER, of parsing some arguments for itself, and collecting the rest to pass to another program. This is their example:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')
So you could call popen with something like
plist = ['wrapped_program']
plist.extend(args.args)
popen(plist, ...)
Using parse.parse_known_args can also be used to collect unparsed words into an 'extras' list. That section of the docs talks about passing those strings on to another program (just as you are doing). In contrast with the REMAINDER case, the extra stuff does not have to be last.
These work, of course, only if this parser doesn't need --pattern for itself. If it parses it, then it won't appear appear in the REMAINDER or extras. In that case you will have to add it back to the list that you give popen
I would tweak your parser thus:
pass_1 = 'passed' # without the -- or internal -
dpass_` = '--'+pass_
parser = argparse.ArgumentParser()
parser.add_argument("-p", dpass_1, help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
process = Popen(['wrapped_program', dpass_1, getattr(args, pass_1)], ...)
another option:
parser = argparse.ArgumentParser()
pass_action = parser.add_argument("-p", '--pass-me-on', help="Pass me on")
parser.add_argument("-k", "--arg-for-wrapper")
args = parser.parse_args()
If you print pass_action (in a shell) you'll get something like:
_StoreAction(option_strings=['-p', '--pass-me-on'], dest='pass_me_on', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
So you could pull the --name and dest from that object, thus:
process = Popen(['wrapped_program', pass_action.option_strings[-1], getattr(args, pass_action.dest), ...], ...)
You have to look in sys.argv to see which option_string was used (the long, short or other). The parser does not record that anywhere.
Note '--pass-me-on' produced dest='pass_me_on'. The conversion of - to _ can complicate deriving one string from the other.
If you have a dest string, you have to use getattr to pull it from the args namespace, or use vars(args)[dest] (dictionary access).
Another issue. If --patten has nargs='+', its value will be a list, as opposed to a string. You'd have to careful when merging that into thepopen` argument list.

Using argparse to create custom command line formats

I realize that the question is rather general but I didn't know exactly how to ask it for what I am doing, but here goes.
I want to create a tool that allows option in the following format which also uses custom actions:
tool.py {start|stop|restart|configure}
Each of the above commands are mutually exclusive and some can have separate unique options. All will call a custom action (subclassed argparse.Action).
tool.py start
The above will do nothing because no arguments (via "add_argument()") was defined.
I though about making a subparser, but doing so doesn't work initially unless you set default arguments, via "set_defaults()". However, doing this and setting:
class CustomAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
print('Args: %r %r %r' % (namespace, values, option_string))
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser(help="Basic daemon.")
subparsers = parser.add_subparsers()
start_parser = subparsers.add_parser("start")
start_parser.set_defaults(start=True, action=CustomAction)
doesn't seem to kick off the custom action as expected. Below is the output I get:
$ custom_parser.py start
Namespace(action=<class '__main__.BasicAction'>, start=True)
I can see that the values are being assigned, but NOT called.
I basically want to have exclusive parent options that can be specified without child argument but still allow exclusive sub-arguments like so, if desired:
tool.py configure {interval|recipients}
Any ideas?

You can use subparsers coupled with default functions
def start_something():
do_starting_actions()
def stop_something():
do_terminal_actions()
def parse_args():
parser = ArgumentParser()
subparsers = parser.add_subparsers()
start = subparsers.add_parser("start")
start.set_defaults(func=start_something)
stop = subparsers.add_parser("stop")
stop.set_defaults(func=stop_something)
# ...
return parser.parse_args()
def main():
args = parse_args()
args.func()
Then you can call the parser from the command line
mymodule.py start
If you wanted to extend the subparser you could do it like:
start = subparsers.add_parser("start")
start.add_argument("--foo")

argparse call a function after parsing other options

Currently, I am using argparse to parse arguments and store flags as boolean options. I then check to see which flag is set to true and execute that function. Argparse parses an input file, which is opened and passed to the called function as an argument.
So:
parser.add_argument('input_data', action='store', help='some help')
parser.add_argument('outputname', action='store',default=None, help='some help')
parser.add_argument('--flag','-f', action='store_true', dest='flag', default=False, help='help!')
I have to open the input_data to read some information from it before the flag function is called. This is currently implemented as:
if args.flag == True:
array_out = flag(array_read_from_input)
if args.outputname == None:
name = 'Flag.tif'
It is possible to subclass argparse to have the action keyword call a function.
Is it possible to parse the input_data option, perform some processing, and then call the flag function without having nested if loops for each argument, eg., by subclassing argparse's action parameter?

Is it possible to parse the input_data option, perform some
processing, and then call the flag function without having nested if
loops for each argument, eg., by subclassing argparse's action
parameter?
As per your question;
class FooAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
<< some processing of values >>
array_out = flag(values)
setattr(namespace, self.dest, array_out)
parser = argparse.ArgumentParser()
parser.add_argument('input_data', action=FooAction, help='some help')