I'm trying to build a process with some parsing option, some mandatory and others optinal.
I have a problem with the following code:
bandlist=[1,2,3,4,5,6,7,8]
process=['rad', 'ref', 'radref']
sensors=['LC', 'LO', 'LE', 'LT']
series=['4', '5', '7', '8']
usage = "usage: %prog [options] "
parser = argparse.ArgumentParser(usage=usage)
parser.add_argument('-d', '--directory', dest='directory', action='store', type=str, \
help='complete path of landsat product folder: mydir/filename/')
parser.add_argument('-p', '--process', dest='operation', action='store', choices = process, \
help='process requested: radiance, reflectance, both', default='rad')
parser.add_argument('-l', '--series', dest='satellite', action='store', choices = series , \
help='Landsat series:4, 5, 7, 8')
parser.add_argument('-s', '--sensor', dest='sensor', action='store', choices = sensors, \
help='sensor acronymous, for example LO for Landsat OLI, or LE for Landsat ETM+, etc..', default=None)
parser.add_argument('-o', '--output', dest='output', type=str, \
help='Directory of output raster. \n \
Unless specified, output directory will be workDirectory/results/datafolder/. \n \
If specified, the output directory wil be mydirectory/results/filename/rad (and/or ref)/', default=None)
parser.add_argument('-x', action='store_true', dest='bool', help='activate iterative radiance and/or reflectance computation for all the bands', default=False)
parser.add_argument('-b', '--bands', dest='bands', choices = bandlist, type=int, \
help='bands to process', nargs='*', default=None)
(options, args) = parser.parse_args()
and there is the following error:
Traceback (most recent call last):
File "C:\Users\lbernardi\Desktop\extract\LandsatTMroutine_RadiometricCorrection_1.0.py", line 1210, in <module>
main()
File "C:\Users\lbernardi\Desktop\extract\LandsatTMroutine_RadiometricCorrection_1.0.py", line 784, in main
(options, args) = parser.parse_args()
TypeError: 'Namespace' object is not iterable
I don't understand what the error is about.
Thank for your help
parse_args doesn't return two items, it returns one.
args = parser.parse_args()
The error was issued by the interpreter while performing the (options, args) = ... assignment. The parser returned one object, a argparse.Namespace. But the assignment tries to split it into two items, e.g.
(options, args) = (args[0], args[1])
But the Namespace class definition does not implement a list or tuple like iteration. A custom Namespace class could, in theory, do so.
That's the technical detail behind the error. The practical issue is that argparse differs from optparse.
From the end of the argparse docs:
Replace (options, args) = parser.parse_args() with args = parser.parse_args() and add additional ArgumentParser.add_argument() calls for the positional arguments. Keep in mind that what was previously called options, now in argparse context is called args.
optparse processes all the flagged strings, and puts their values in an options object (I forget its exact nature). Strings that it can't process are returned as a list as the 2nd return value.
argparse expects you to define all arguments, both the flagged ones (called 'optionals') and unflagged ones (called 'positionals'). So values that in optparse would appear in the args list, appear under their own 'name/dest' in the argparse namespace.
There is another way of calling the argparse parser, parser.parse_known_args that behaves more like optparse. Strings it can't handle are returned in an extras list.
You mention that some of your arguments are required and some are not. At a first glance, your code makes everything 'optional'. That is, if you don't include the relevant flag in the commandline, that argument will get its default value.
One of your arguments uses nargs='*'. If that isn't specified the default will be None or [] (I forget the details).
You can also specify required=True parameter which makes an 'optional' required. Sorry about the confusing terminology. In that case, the parser will raise an error if you don't supply the flag in the commandline.
I didn't look much at your previous optparse question to see whether you expected an values in the args variable. In argparse usage those are 'positionals' and are required (unless their nargs makes them 'optional').
So while the simple fix is just use args = parser.parse_args(), I suspect there's more under the surface.
A style point:
parser.add_argument('-d', '--directory', dest='directory', action='store', type=str, \
help='complete path of t product folder: mydir/filename/')
can be simplified to
parser.add_argument('-d', '--directory', \
help='complete path of landsat product folder: mydir/filename/')
If not given dest is infered from the first long flag string. store is the default action value, and str is the default type.
In argparse, type is a function, one that converts the input string to something else. int and float are the most common alternatives. str works because it does nothing - returns the same string it was given. argparse actually uses an identify lambda function as the default type (e.g. lambda x:x).
Related
I have a script that consumes command line arguments and I would like to implement two argument-passing schemes, namely:
Typing the arguments out at the command line.
Storing the argument list in a file, and passing the name of this file to the program via the command line.
To that end I am passing the argument fromfile_prefix_chars to the ArgumentParser constructor.
script.py
from argparse import ArgumentParser
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
args.txt
--foo
2
--bar
2
Sample use cases
$ python script.py --foo 3
Namespace(bar=1, filename=None, foo='3')
$ python script.py #args.txt --foo 3
Namespace(bar='2', filename=None, foo='3')
I was expecting that args.filename would retain the name of the file, but surprinsingly enough it has the value None instead. I am aware that I could get the file name from sys.argv through a bit of processing. Is there a cleaner way (ideally an argparse-based approach) to elicit the name of the arguments file?
Your script.py, plus the file. I have added the file name to the file itself.
args.txt
args.txt
--foo
2
--bar
2
testing:
1803:~/mypy$ python3 stack56811067.py --foo 3
Namespace(bar=1, filename=None, foo='3')
1553:~/mypy$ python3 stack56811067.py #args.txt --foo 3
Namespace(bar='2', filename='args.txt', foo='3')
From my testing, using fromfile_prefix_chars means that argparse will not actually pass the argument to your program. Instead, argparse sees the #args.txt, intercepts it, reads from it, and passes the arguments without #args.txt to your program. This is presumably because most people don't really need the filename, just need the arguments within, so argparse saves you the trouble of creating another argument to store something you don't need.
Unfortunately, all of the arguments are stored as local variables in argparse.py, so we cannot access them. I suppose that you could override some of argparse's functions. Keep in mind that this is a horrible, disgusting, hacky solution and I feel that parsing sys.argv is 100% better.
from argparse import ArgumentParser
# Most of the following is copied from argparse.py
def customReadArgs(self, arg_strings):
# expand arguments referencing files
new_arg_strings = []
for arg_string in arg_strings:
# for regular arguments, just add them back into the list
if not arg_string or arg_string[0] not in self.fromfile_prefix_chars:
new_arg_strings.append(arg_string)
# replace arguments referencing files with the file content
else:
try:
fn = arg_string[1:]
with open(fn) as args_file:
# What was changed: before was []
arg_strings = [fn]
for arg_line in args_file.read().splitlines():
for arg in self.convert_arg_line_to_args(arg_line):
arg_strings.append(arg)
arg_strings = self._read_args_from_files(arg_strings)
new_arg_strings.extend(arg_strings)
except OSError:
err = _sys.exc_info()[1]
self.error(str(err))
# return the modified argument list
return new_arg_strings
ArgumentParser._read_args_from_files = customReadArgs
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('filename', nargs='?')
parser.add_argument('--foo', nargs='?', default=1)
parser.add_argument('--bar', nargs='?', default=1)
args = parser.parse_args()
print(args)
Just for the record, here's a quick and dirty solution I came up with. It basically consists in creating a copy of parser and set its from_file_prefix_chars attribute to None:
import copy
parser_dupe = copy.copy(parser)
parser_dupe.fromfile_prefix_chars = None
args_raw = parser_dupe.parse_args()
if args_raw.filename:
args.filename = args_raw.filename[1:]
I'm trying to parse command-line arguments such that the three possibilities below are possible:
script
script file1 file2 file3 …
script -p pattern
Thus, the list of files is optional. If a -p pattern option is specified, then nothing else can be on the command line. Said in a "usage" format, it would probably look like this:
script [-p pattern | file [file …]]
I thought the way to do this with Python's argparse module would be like this:
parser = argparse.ArgumentParser(prog=base)
group = parser.add_mutually_exclusive_group()
group.add_argument('-p', '--pattern', help="Operate on files that match the glob pattern")
group.add_argument('files', nargs="*", help="files to operate on")
args = parser.parse_args()
But Python complains that my positional argument needs to be optional:
Traceback (most recent call last):
File "script", line 92, in <module>
group.add_argument('files', nargs="*", help="files to operate on")
…
ValueError: mutually exclusive arguments must be optional
But the argparse documentation says that the "*" argument to nargs meant that it is optional.
I haven't been able to find any other value for nargs that does the trick either. The closest I've come is using nargs="?", but that only grabs one file, not an optional list of any number.
Is it possible to compose this kind of argument syntax using argparse?
short answer
Add a default to the * positional
long
The code that is raising the error is,
if action.required:
msg = _('mutually exclusive arguments must be optional')
raise ValueError(msg)
If I add a * to the parser, I see that the required attribute is set:
In [396]: a=p.add_argument('bar',nargs='*')
In [397]: a
Out[397]: _StoreAction(option_strings=[], dest='bar', nargs='*', const=None, default=None, type=None, choices=None, help=None, metavar=None)
In [398]: a.required
Out[398]: True
while for a ? it would be False. I'll have dig a bit further in the code to see why the difference. It could be a bug or overlooked 'feature', or there might a good reason. A tricky thing with 'optional' positionals is that no-answer is an answer, that is, an empty list of values is valid.
In [399]: args=p.parse_args([])
In [400]: args
Out[400]: Namespace(bar=[], ....)
So the mutually_exclusive has to have some way to distinguish between a default [] and real [].
For now I'd suggest using --files, a flagged argument rather than a positional one if you expect argparse to perform the mutually exclusive testing.
The code that sets the required attribute of a positional is:
# mark positional arguments as required if at least one is
# always required
if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
kwargs['required'] = True
if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
kwargs['required'] = True
So the solution is to specify a default for the *
In [401]: p=argparse.ArgumentParser()
In [402]: g=p.add_mutually_exclusive_group()
In [403]: g.add_argument('--foo')
Out[403]: _StoreAction(option_strings=['--foo'], dest='foo', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None)
In [404]: g.add_argument('files',nargs='*',default=None)
Out[404]: _StoreAction(option_strings=[], dest='files', nargs='*', const=None, default=None, type=None, choices=None, help=None, metavar=None)
In [405]: p.parse_args([])
Out[405]: Namespace(files=[], foo=None)
The default could even be []. The parser is able to distinguish between the default you provide and the one it uses if none is given.
oops - default=None was wrong. It passes the add_argument and required test, but produces the mutually_exclusive error. Details lie in how the code distinguishes between user defined defaults and the automatic ones. So use anything but None.
I don't see anything in the documentation about this. I'll have to check the bug/issues to see it the topic has been discussed. It's probably come up on SO before as well.
You are trying to use the 'files' argument to catch a number of files but you are not supplying it on the cmdline examples. I think the library gets confused that you're not using the dash prefix. I would suggest the following:
import argparse
parser = argparse.ArgumentParser(prog="base")
group = parser.add_mutually_exclusive_group()
group.add_argument('-p', '--pattern', action="store", help="Operate on files that match the glob pattern")
group.add_argument('-f', '--files', nargs="*", action="store", help="files to operate on")
args = parser.parse_args()
print args.pattern
print args.files
import argparse
parse = argparse.ArgumentParser()
parse.add_argument("-p",'--pattern',help="Operates on File")
parse.add_argument("files",nargs = "*",help="Files to operate on")
arglist = parse.parse_args(["-p","pattern"])
print arglist
arglist = parse.parse_args()
print arglist
arglist = parse.parse_args(["file1","file2","file3"])
print arglist
I am now starting exploring Python, and was testing how arguments can be passed to a script with "argparse".
The way I wrote a sample script was as following, where arguments passed through flags -i and -o are compulsory and flag -u is optional:
#!/usr/bin/python
import sys
import argparse
## set usage options and define arguments
usage = "usage: %prog [options]"
parser = argparse.ArgumentParser(usage)
parser.add_argument("-i", action="store", dest="input", help="input file")
parser.add_argument("-o", action="store", dest="output", help="output file")
parser.add_argument("-u", action="store_true", dest="isunfolded", default=False, help="optional flag")
args = parser.parse_args()
print len(sys.argv)
if len(sys.argv) < 2:
# parser.print_help()
print 'Incorrect number of params'
exit()
else:
print "Correct number of params: ", len(sys.argv)
Running this script:
> ./test-args.py -i a -o b
prints:
5
Correct number of params: 5
I understand the printing statement in the if conditional (5 is higher than 2), however, after reading the argparse documentation (https://docs.python.org/3/library/argparse.html) I still don't quite understand why -i and -o flags are counted as arguments. This behaviour seems to be quite different from e.g. perl Getopt::Std, which I'm more used to.
So, the question is what is the best way of parsing arguments in Python and to evaluate the presence of mandatory arguments (without using required=True)
It gives you 5 because sys.argv contains the raw input passed to python as arguments (the script name and 4 arguments).
You can see argparse as an abstraction for this, so once you use it, you can forget about sys.argv. In most cases it is better not to mix these two methods.
argparse its a nice way to handle arguments, I don't quite get why you don't want to use the required option when that's exactly the way to go. Another alternative is to parse the sys.argv yourself (regex maybe?) and drop argparse altogether.
There's a Python getopt which probably is similar to the Perl one (assuming both are modelled after the C/Unix version).
https://docs.python.org/2/library/getopt.html
In your code, sys.argv is a list of strings from the command line (as interpreted by the shell and interpreter). It is the raw input for any of the parsers ('getopt', 'optparse', 'argparse'). And it is available for your parsing as well. When learning it is a good idea to include a
print sys.argv
line. parser.parse_args() uses this list. sys.argv[0] is used as prog attribute (in the default usage), while sys.argv[1:] (the rest) is parsed according to the rules you define with add_argument. For testing I often like to use parse_args with a custom list of strings, e.g.
print parser.parse_args(['-i', 'input', '-o', 'output', '-u'])
With your definition I'd expect to see something like:
Namespace(input='input', output='output', isunfolded=True)
The parser returns an object (type argparse.Namespace), which has attributes defined by your arguments. Values are usually accessed with expressions like args.input, args.isunfolded. The docs also show how you easily express this as a dictionary.
By long standing UNIX conventions, arguments flagged by strings like '-i' are options, that is they are optional. argparse generalizes this concept by letting you specify a required=True parameter.
Other arguments are positionals. They are interpreted according to their order. And as such they are normally required. What argparse adds is the ability to define those positionals, such as type, nargs, etc. With nargs='?' they are optional. Many of the nargs values are similar to regular expression characters (e.g. +?*). In fact argparse uses a form a regular expression parsing to allocate strings among arguments.
I'd refine your arguments thus (taking advantage of various defaults)
a1 = parser.add_argument("-i", "--input", help="input file") # 'store' is the default
a2 = parser.add_argument("-o", "--output",help="output file") # use the --output as dest
a3 = parser.add_argument("-u", "--isunfolded", action="store_true", help="optional flag")
If input and output were required, I could change them to:
parser.add_argument("input", help="input file") # 'store' is the default
parser.add_argument("output",help="output file") # use the --output as dest
parser.add_argument("-u", "--isunfolded", action="store_true", help="optional flag")
Now input and output are positional arguments, as in test.py -u inputfile outputfile
By using a1 = parser... I can look at the object produced by this statement.
print a1
produces
_StoreAction(option_strings=['-i', '--input'], dest='input', nargs=None, const=None,
default=None, type=None, choices=None, help='input file', metavar=None)
This tells me that a1 is a _StoreAction object (a subclass of argparse.Action). It also displays a number (not all) of its attributes, ones that define its action. A positional, on the other hand, has values like these:
a2 = p.add_argument("output", help="output file")
_StoreAction(option_strings=[], dest='output', nargs=None, const=None,
default=None, type=None, choices=None, help='output file', metavar=None)
It may also be instructive to look at a1.required and a2.required, which are respectively False and True. required is an Action attribute that is not routinely displayed, but is, never the less accessible.
I've pulled all these test values from a parser defined in an interactive shell (Ipython). It's a great way to explore Python and modules like argparse.
After reading other related posts it seems that the best way to do this is as was suggested by #Rufflewind and inspect the args itself:
if not args.input or not args.output:
print 'Incorrect number of params'
exit()
else:
print "Correct number of params"
I am trying to make a Python program that uses the argparse module to parse command-line options.
I want to make an optional argument that can either be named or positional. For example, I want myScript --username=batman to do the same thing as myScript batman. I also want myScript without a username to be valid. Is this possible? If so, how can it be done?
I tried various things similar to the code below without any success.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin")
group.add_argument("user-name", default="admin")
args = parser.parse_args()
EDIT: The above code throws an exception saying ValueError: mutually exclusive arguments must be optional.
I am using Python 2.7.2 on OS X 10.8.4.
EDIT: I tried Gabriel Jacobsohn's suggestion but I couldn't get it working correctly in all cases.
I tried this:
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument("-u", "--user-name", default="admin", nargs="?")
group.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)
and running myScript batman would print Namespace(user_name='batman'), but myScript -u batman and myScript --user-name=batman would print Namespace(user_name='admin').
I tried changing the name user-name to user_name in the 1st add_argument line and that sometimes resulted in both batman and admin in the namespace or an error, depending on how I ran the program.
I tried changing the name user_name to user-name in the 2nd add_argument line but that would print either Namespace(user-name='batman', user_name='admin') or Namespace(user-name='admin', user_name='batman'), depending on how I ran the program.
The way the ArgumentParser works, it always checks for any trailing positional arguments after it has parsed the optional arguments. So if you have a positional argument with the same name as an optional argument, and it doesn't appear anywhere on the command line, it's guaranteed to override the optional argument (either with its default value or None).
Frankly this seems like a bug to me, at least when used in a mutually exclusive group, since if you had specified the parameter explicitly it would have been an error.
That said, my suggested solution, is to give the positional argument a different name.
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument('-u','--username')
group.add_argument('static_username',nargs='?',default='admin')
Then when parsing, you use the optional username if present, otherwise fallback to the positional static_username.
results = parser.parse_args()
username = results.username or results.static_username
I realise this isn't a particularly neat solution, but I don't think any of the answers will be.
Here is a solution that I think does everything you want:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
# Gather all extra args into a list named "args.extra"
parser.add_argument("extra", nargs='*')
args = parser.parse_args()
# Set args.user_name to first extra arg if it is not given and len(args.extra) > 0
if args.user_name == parser.get_default("user_name") and args.extra:
args.user_name = args.extra.pop(0)
print args
If you run myScript -u batman or myScript --user-name=batman, args.user_name is set to 'batman'. If you do myScript batman, args.user_name is again set to 'batman'. And finally, if you just do myScript, args.user_name is set to 'admin'.
Also, as an added bonus, you now have all of the extra arguments that were passed to the script stored in args.extra. Hope this helps.
Try to use the "nargs" parameter of the add_argument method.
This way it works for me.
Now you can add the username twice,
so you have to check it and raise an error, if you want.
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("-u", "--user-name", default="admin")
parser.add_argument("user_name", default="admin", nargs="?")
args = parser.parse_args()
print(args)
I have the following code at the minute:
parser = argparse.ArgumentParser(prog='Tempus')
ex_group = parser.add_mutually_exclusive_group(required=True)
## Miscellaneous but needed args
parser.add_argument('--mode', type=str, choices=['xml', 'text', 'term'],
dest='mode', required=True, help='export mode')
parser.add_argument('-v', '--verbose', action='store_true',
dest='verbose', help='enable verbose/debug mode')
# Input methods
ex_group.add_argument('--i', action='store_true',
dest='interactive', help='enter interactive mode')
ex_group.add_argument('--p', metavar='I', type=float,
dest='integer', help='percentage to use')
args = parser.parse_args()
However when I pass the arguments that are needed, in any order, I get the error:
Tempus: error: argument --mode is required
And this happens, even when I pass that argument. Any way to sort this?
Thank you!
EDIT: Thanks everyone, I got it working, turns out executing it without first specifying the python exe before it doesn't pass arguments.
Just another quick question: is it possible to create an argument similar to --mode in my code, but let an additional argument be passed to one of the choices?
For example, have a command such as --input which can take two arguments 'integer' and 'interactive', but I can also pass a number to the --input integer command? So the command would read: --input integer 23 for example? Is this possible?
Try
1. to print sys.argv before parsing or
2. to explicitely add arguments inside the script as in http://docs.python.org/dev/library/argparse.html#parsing-arguments
So you can exclude any confusion about which arguments argparse actually sees.