I have this very simple ArgumentParser instance, with an optional positional argument and an option, which writes a constant to the same destination:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', action='store_const', dest='path', const='<all>')
parser.add_argument('path', nargs='?')
# Prints None instead of '<all>'
print(parser.parse_args(['-a']).path)
But no matter what, parsing the command line ['-a'] does not yield a Namespace instance with path set to that constant. Instead, the default from the positional argument is used.
What am I doing wrong?
My use case is that the user should be able to specify a path (actually a list of paths). This list of paths defaults to the current working directory. But instead of using that default, -a can be passed, which should result in some configured root directory to be used. The full code for this part of the argument parser is this:
all_sentinel = object()
parser = argparse.ArgumentParser()
paths_group = parser.add_mutually_exclusive_group()
paths_group.add_argument('-a', action='store_const', dest='paths', const=all_sentinel)
paths_group.add_argument('paths', nargs='*', default=['.'])
A positional with nargs='?' has some special handling of its default (here None).
Normally defaults are assigned to the Namespace at the start of parsing, and overwritten by the actions, such as the optional.
Because an empty list of values satisfies the nargs, that positional is always 'seen'. But rather than assign [] or some other 'blank' to it, the parser assigns the default. So the positional's default overwrites the value set by '-a'.
nargs='*' gets the same kind of special handling.
I suspect that if you had another positional argument before the '-a', that you wouldn't see this effect. The '?*' positional would be processed before the '-a', and not overwrite its value.
Optionals are only processed if the flag occurs. Positionals are always processed, regardless of the nargs. The 'optional' positionals are processed, but with some extra handling of the defaults. But when they are processed relative to flagged arguments can vary.
That's some tricky behavior that I'm aware of simply because I've studied the code in detail, and answered a lot questions here and on the Python bug/issues.
Sharing the dest often does work, but that's more by default than design. It's the result of other design choices. argparse makes no promises in that regard. So if it isn't reliable, don't use it.
Related
I would like to have a program with subparsers that handles specific arguments while also keep some positional and optional arguments to the previous parsers (In fact what I really want is only one option, I mean, a valid subparser OR a valid local argument).
Example of something I wish to have: Program [{sectionName [{a,b}]}] [{c,d}]. Being c/d incompatible if sectionName was provided and viceversa.
However, the best I could achieve is this test.py [-h] {sectionName} ... [{c,d}]. This means, argparse don't allow me to use the positional arguments c or d without specifying a valid sectionName.
Here is the code:
import argparse
mainparser = argparse.ArgumentParser()
# Subparser
subparser = mainparser.add_subparsers(title="section", required=False)
subparser_parser = subparser.add_parser("sectionName")
subparser_parser.add_argument("attribute", choices=['a', 'b'], nargs='?')
# Main parser positional and optional attributes
mainparser.add_argument("attribute", choices=['c', 'd'], nargs='?')
mainparser.parse_args()
I'm getting crazy with this. Any help would be much appreciated!
Edit: I'm using Python 3.8
The subparser object is actually a positional Action, one that takes choices - in this case {'sectionName'}. positinal arguments are filled in the order that they are defined, using the nargs pattern to allocate strings.
Once the main parser gets the 'sectionName' it passes the parsing to subparser_parser. Than handles the rest of the input, such as the {'a','b'} positional. Anything it can't handle is put on the 'unrecognized' list, and control returns main for final processing. main does not do any further argument processing. Thus your attribute argument is ignored.
You could put define a attribute positional before the add_subparsers, but I wouldn't try to make it nargs='?'.
So it's best to define all main arguments before the subparsers, and to use optionals. This will give the cleanest and most reliable parsing.
We're trying to build a wrapper script over a command line tool we're using. We would like to set some tool arguments based on options in our wrapper scripts. We would also like to have the possibility to pass native arguments to the command line tool directly as they are written on the command line.
Here is what we came up with:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-f', '--foo', action='store_true')
parser.add_argument('-b', '--bar', action='store_true')
parser.add_argument('native_arg', nargs='*')
args = parser.parse_args()
print (args)
positional is mandatory. Based on the options -f and -b we would add some extra options to our tool call. Anything that is left afterwards (if anything) should be treated as a native tool argument and given to the tool directly. Calling our script with -h produces the following usage:
usage: test.py [-h] [-f] [-b] positional [native_arg [native_arg ...]]
The trick is that these native arguments are themselves options for the tool and contain leading dashes, for example -native0 and -native1. We already know about the trick with the double dash to stop argparse from looking for more options. The following call:
./test.py pos -- -native0 -native1
produces the expected parsed arguments:
Namespace(bar=False, foo=False, native_arg=['-native0', '-native1'], positional='pos')
Trying to add an option after the first positional argument doesn't work, though. More specifically, the following call:
./test.py pos --foo -- -native0 -native1
produces the following output:
usage: [...shortened...]
test.py: error: unrecognized arguments: -- -native0 -native1
Putting the optional arguments before the positionals:
./test.py --foo pos -- -native0 -native1
seems to work, as the following is printed:
Namespace(bar=False, foo=True, native_arg=['-native0', '-native1'], positional='pos')
Even stranger, changing the value of nargs for native_arg to '+' works in all the above situations (with the caveat, of course, that at least one native_arg is expected).
Are we doing something wrong in our Python code or is this some kind of argparse bug?
argparse does have a hard time when you mix non-required positional arguments with optional arguments (see https://stackoverflow.com/a/47208725/1399279 for details into the bug report). Rather than suggesting a way to solve this issue, I am going to present an alternative approach.
You should check out the parse_known_args method, which was created for the situation you describe (i.e. passing options to a wrapped tool).
In [1]: import argparse
In [2]: parser = argparse.ArgumentParser()
In [3]: parser.add_argument('positional')
In [4]: parser.add_argument('-f', '--foo', action='store_true')
In [5]: parser.add_argument('-b', '--bar', action='store_true')
In [6]: parser.parse_known_args(['pos', '--foo', '-native0', '-native1'])
Out[6]: (Namespace(bar=False, foo=True, positional='pos'), ['-native0', '-native1'])
Unlike parse_args, the output of parse_known_args is a two-element tuple. The first element is the Namespace instance you would expect to get from parse_args, and it contains all the attributes defined by calls to add_argument. The second element is a list of all the arguments not known to the parser.
I personally prefer this method because the user does not need to remember any tricks about how to call your program, or which option order does not result in errors.
This is a known issue (https://bugs.python.org/issue15112, argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional)
The parsing alternates handling positionals and optionals. When dealing with positionals it tries to handle as many as the input strings require. But an ? or * positional is satisfied with [], an empty list of strings. + on the other hand requires at least one string
./test.py pos --foo -- -native0 -native1
The parser gives 'pos' to positional, and [] to native-arg. Then it gives '--foo' to its optional. There aren't anymore positionals left to hand the remaining strings, so it raises the error.
The allocation of input strings is done with a stylized form of regex string matching. Imagine matching a pattern that looks like AA?.
To correct this, parser would have to look ahead, and delay handling native-arg. We've suggested patches but they aren't in production.
#SethMMorton's suggestion of using parse_known_args is a good one.
Earlier parsers (e.g. Optparse) handle all the flagged arguments, but return the rest, the positionals, as a undifferentiated list. It's up to the user to split that list. argparse has added the ability to name and parse positionals, but the algorithm works best with fixed nargs, and gets flaky with too many variable nargs.
I have the following snippet:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--p', required=True)
parser.add_argument('arg', action=MyAction)
parser.parse_args()
, where MyAction is a simple custom action class.
As you see, I want to enforce the presence of the p argument. However, my action is performed even if the argument is not present, and then an error message is shown that indicates the fact the the argument is missing.
Obviously, I could check in my action class for the presence of the p argument, but this defies the purpose of having a required parameter in the first place. Why is my action being run if the argument is not present?
Parsing is driven by the commandline strings, and tries to be order agnostic. Within those rules, it alternates between parsing an optional and a positional.
For example, with myprog --p one two:
'--p' - pass the 'one' string to the p action (e.g. setattr(namespace, 'p', 'one')
'two' - matches the nargs for 'arg'. Calls your MyAction.__call__ with values='one'.
at the end of parsing it checks if all required actions have been 'seen'. With your setup both '--p' and 'arg' are required.
With myprog two --p one it does the same, except arg is processed first. The namespace may have a default value for p.
With myprog two, arg is processed, and the required test will raise an error. error: the following arguments are required: --p
Since you have written a custom Action, you can easily explore how the namespace contents vary depending on the commandline arguments and their order.
So the --p and the arg will be processed independently, and in either order, depending on the commandline strings. required testing is performed at the end, using a list of seen_actions. And default values are set at the start of parsing. It is difficult to implement reliable inter-action tests within custom Actions. Usually it is better to perform such tests after parsing.
The defined Actions only change the args namespace. So parsing does not change anything beyond what it returns. Unless there's an error and it forces an sys.exit. A custom MyAction class can change that, but at your own risk.
argparse applies type conversion to arguments of options, and to the default values of these if the default values are strings. However, it seems it doesn't do so for positional arguments:
import argparse as ap
p = ap.ArgumentParser()
p.add_argument('file', nargs='*',
metavar='FILE',
default='-',
type=ap.FileType(),
help='Input files. - is stdin. Default: %(default)s')
print(p.parse_args([]))
# Namespace(file='-')
print(p.parse_args(['-']))
# Namespace(file=[<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>])
(Ideone)
If I change file to --file so that these are option arguments, the default value is converted as expected.
Do I have any options better than explicitly passing argparse.FileType()('-') as the default?
The values put into sys.argv are always str objects, because the underlying operating system construct is an array is C character pointers. You need type to convert them to some other Python type.
The value of default, however, is under no such restriction. You can use a value of any type you like for that keyword argument.
p.add_argument('file',
nargs='*',
metavar='FILE',
default=[sys.stdin],
type=ap.FileType(),
help='Input files. - is stdin. Default: standard input')
I modified the help; getting the value of repr(sys.stdin) isn't particular useful, and the user should not be confused by describing the default value rather than specifying an exact Python object.
I have the following command line tool:
import argparse
parser = argparse.ArgumentParser(description = "A cool application.")
parser.add_argument('positional')
parser.add_argument('--optional1')
parser.add_argument('--optional2')
args = parser.parse_args()
print args.positionals
The output of python args.py is:
usage: args.py [-h] [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
positional
however I would like to have:
usage: args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]
How could I have that reordering?
You would either have to provide your own help formatter, or specify an explicit usage string:
parser = argparse.ArgumentParser(
description="A cool application.",
usage="args.py [-h] positional [--optional1 OPTIONAL1] [--optional2 OPTIONAL2]")
The order in the help message, though, does not affect the order in which you can specify the arguments. argparse processes any defined options left-to-right, then assigns any remaining arguments to the positional parameters from left to right. Options and positional arguments can, for the most part, be mixed.
With respect to each other the order of positionals is fixed - that's why they are called that. But optionals (the flagged arguments) can occur in any order, and usually can be interspersed with the postionals (there are some practical constrains when allowing variable length nargs.)
For the usage line, argparse moves the positionals to the end of the list, but that just a display convention.
There have been SO questions about changing that display order, but I think that is usually not needed. If you must change the display order, using a custom usage parameter is the simplest option. The programming way requires subclassing the help formatter and modifying a key method.