I have the following code for reading the arguments from a file and process them using argparse, but I am getting an error, why is this the case?
import argparse
from ConfigParser import ConfigParser
import shlex
parser = argparse.ArgumentParser(description='Short sample app',
fromfile_prefix_chars='#')
parser.add_argument('--abool', action="store_true", default=False)
parser.add_argument('--bunit', action="store", dest="bunit",type=int)
parser.add_argument('--cpath', action="store", dest="c", type=str)
print parser.parse_args(['#argparse_fromfile_prefix_chars.txt']) #name of the file is argparse_fromfile_prefix_chars.txt
Error:
usage: -c [-h] [--abool] [--bunit BUNIT] [--cpath C]
-c: error: unrecognized arguments: --bunit 289 --cpath /path/to/file.txt
To exit: use 'exit', 'quit', or Ctrl-D.
Contents of the file argparse_fromfile_prefix_chars.txt
--abool
--bunit 289
--cpath /path/to/file.txt
argparse expects arguments from files to be one per line. Meaning the whole line is one quoted argument. So your current args file is interpreted as
python a.py '--abool' '--bunit 289' '--cpath /path/to/file.txt'
which causes the error. Instead, your args file should look like this
--abool
--bunit
289
--cpath
/path/to/file.txt
The documentation for fromfile_prefix_chars states:
Arguments read from a file must by default be one per line (but see
also convert_arg_line_to_args()) and are treated as if they were in
the same place as the original file referencing argument on the
command line.
Note that one argument does not mean one option followed by all its arguments. It means a command line argument. Currently the whole lines are interpreted as if they were a single argument.
In other words your file should look like:
--abool
--bunit
289
--cpath
/path/to/file.txt
Alternatively you can override the convert_arg_line_to_args() method to parse the file in an other way. The documentation already provides an implementation that parses white-space separated arguments instead of line-separated arguments:
def convert_arg_line_to_args(self, arg_line):
# consider using shlex.split() instead of arg_line.split()
for arg in arg_line.split():
if not arg.strip():
continue
yield arg
I believe you can either subclass ArgumentParser and reimplement this method, or, probably, even setting the attribute on an ArgumentParser instance should work.
For some reason the default implementation of convert_arg_line_to_args doesn't work properly:
$echo '--abool
--bunit
289
--cpath
/here/is/a/path
' > file.txt
$cat test_argparse.py
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('--abool', action='store_true')
parser.add_argument('--bunit', type=int)
parser.add_argument('--cpath')
print(parser.parse_args(['#file.txt']))
$python test_argparse.py
usage: test_argparse.py [-h] [--abool] [--bunit BUNIT] [--cpath CPATH]
test_argparse.py: error: unrecognized arguments:
However if you use the implementation above it works:
$cat test_argparse.py
import argparse
def convert_arg_line_to_args(arg_line):
for arg in arg_line.split():
if not arg.strip():
continue
yield arg.strip()
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('--abool', action='store_true')
parser.add_argument('--bunit', type=int)
parser.add_argument('--cpath')
parser.convert_arg_line_to_args = convert_arg_line_to_args
print(parser.parse_args(['#file.txt']))
$python test_argparse.py
Namespace(abool=True, bunit=289, cpath='/here/is/a/path')
An other workaround is to use the --option=argument syntax:
--abool
--bunit=289
--cpath=/the/path/to/file.txt
However this will not work when an option has more than one argument. In such a case you have to use a different implementation of convert_arg_line_to_args.
Trying to debug, it seems like the convert_line_arg_to_args gets called with an empty string which gets added to the arguments, and the empty string is considered an argument (which isn't defined).
The problem is that there are two newlines at the end of the file.
In fact if you create the file without this double newline at the end, it works:
$echo -n '--abool
--bunit
289
--cpath
/here/is/a/path
' > file.txt
$python test_argparse.py
Namespace(abool=True, bunit=289, cpath='/here/is/a/path')
(echo -n doesn't add a newline at the end of the output).
Related
python test.py --arg -foo -bar
test.py: error: argument --arg: expected at least one argument
python test.py --arg -8
['-8']
How do I allow -non_number to work with argparse?
Is there a way to disable short arguments?
Call it like this:
python test.py --arg='-foo'
To allow specifying multiple:
parser.add_argument('--arg', action='append')
# call like python test.py --arg=-foo --arg=-bar
I think you're looking for the nargs parameter to argparser.
parser.add_argument('--arg', nargs='?')
At the moment, --arg is interpreting the value '-8' as the input, whereas it thinks '-f' (with parameters 'oo') is a new argument.
Alternatively, you could use action='store_true', which will represent the presence or absence of the argument with a boolean.
parser.add_argument('--arg', action='store_true')
https://docs.python.org/3/library/argparse.html#nargs
I am writing a program which, among other things, allows the user to specify through an argument a module to load (and then use to perform actions). I am trying to set up a way to easily pass arguments through to this inner module, and I was attempting to use ArgParse's action='append' to have it build a list of arguments that I would then pass through.
Here is a basic layout of the arguments that I am using
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
required=True)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
However - if I then try to run this program with python my_program -M module:class -A "-f filename" (where I would like to pass through the -f filename to my module) it seems to be parsing the -f as its own argument (and I get the error my_program: error: argument -A/--module_args: expected one argument
Any ideas?
With:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
import sys
print(sys.argv)
print(parser.parse_args())
I get:
1028:~/mypy$ python stack45146728.py -M module:class -A "-f filename"
['stack45146728.py', '-M', 'module:class', '-A', '-f filename']
Namespace(module='module:class', module_args=['-f filename'])
This is using a linux shell. The quoted string remains one string, as seen in the sys.argv, and is properly interpreted as an argument to -A.
Without the quotes the -f is separate and interpreted as a flag.
1028:~/mypy$ python stack45146728.py -M module:class -A -f filename
['stack45146728.py', '-M', 'module:class', '-A', '-f', 'filename']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
Are you using windows or some other OS/shell that doesn't handle quotes the same way?
In Argparse `append` not working as expected
you asked about a slightly different command line:
1032:~/mypy$ python stack45146728.py -A "-k filepath" -A "-t"
['stack45146728.py', '-A', '-k filepath', '-A', '-t']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
As I already noted -k filepath is passed through as one string. Because of the space, argparse does not interpret that as a flag. But it does interpret the bare '-t' as a flag.
There was a bug/issue about the possibility of interpreting undefined '-xxx' strings as arguments instead of flags. I'd have to look that up to see whether anything made it into to production.
Details of how strings are categorized as flag or argument can be found in argparse.ArgumentParser._parse_optional method. It contains a comment:
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
http://bugs.python.org/issue9334 argparse does not accept options taking arguments beginning with dash (regression from optparse) is an old and long bug/issue on the topic.
The solution is to accept arbitrary arguments - there's an example in argparse's doc here:
argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')
I would like to parse a required, positional argument containing a comma-separated list of integers. If the first integer contains a leading minus ('-') sign, argparse complains:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional')
parser.add_argument('-t', '--test', action='store_true')
opts = parser.parse_args()
print opts
$ python example.py --test 1,2,3,4
Namespace(positional='1,2,3,4', test=True)
$ python example.py --test -1,2,3,4
usage: example.py [-h] [-t] positional
example.py: error: too few arguments
$ python example.py --test "-1,2,3,4"
usage: example.py [-h] [-t] positional
example.py: error: too few arguments
I've seen people suggest using some other character besides - as the flag character, but I'd rather not do that. Is there another way to configure argparse to allow both --test and -1,2,3,4 as valid arguments?
You need to insert a -- into your command-line arguments:
$ python example.py --test -- -1,2,3,4
Namespace(positional='-1,2,3,4', test=True)
The double-dash stops argparse looking for any more optional switches; it's the defacto standard way of handling exactly this use case for command-line tools.
From the documentation:
The parse_args() method attempts to give errors whenever the user has
clearly made a mistake, but some situations are inherently ambiguous.
For example, the command-line argument -1 could either be an attempt
to specify an option or an attempt to provide a positional argument.
The parse_args() method is cautious here: positional arguments may
only begin with - if they look like negative numbers and there are no
options in the parser that look like negative numbers:
Since -1,2,3,4 does not look like a negative number you must "escape" it with the -- as in most *nix systems.
An other solution would be to use nargs for the positional and pass the numbers as space separated:
#test.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('positional', nargs='*') #'+' for one or more numbers
print parser.parse_args()
Output:
$ python test.py -1 2 3 -4 5 6
Namespace(positional=['-1', '2', '3', '-4', '5', '6'])
A third way to obtain what you want is to use parse_known_args instead of parse_args.
You do not add the positional argument to the parser and parse it manually instead:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--test', action='store_true')
parsed, args = parser.parse_known_args()
print parsed
print args
Result:
$ python test.py --test -1,2,3,4
Namespace(test=True)
['-1,2,3,4']
This has the disadvantage that the help text will be less informative.
I am trying to specify an optional argument that takes stdin. This will be mainly used for piping data in my program, so someprog that outputs | python my_prog.
I followed the argparse documentation and I read a lot of questions/answers on this on Stackoverflow but none of them seem to work for me.
Here's what I originally have:
parser = argparse.ArgumentParser(description='Upgrade Instance.')
parser.add_argument('--app', '-a', dest='app', action='store', required=True)
parser.add_argument('--version', '-v', dest='version', action='store', default='', required=False)
parser.add_argument('--config', '-c', dest='config', action='store', default = '', required=False)
args = parser.parse_args()
Now what I want to do is allow the user to pass in version using a pipe, instead of passing it in.
I added parser.add_argument('infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin) to the top but that makes it a positional argument. How is that possible? I thought nargs=? makes it optional.
I need it to be an optional argument. So I changed it to:
parser.add_argument('--infile', nargs='?', type=argparse.FileType('r'), default=sys.stdin)
This makes it an optional argument, but the program hangs waiting for stdin as thats default, if no pipe is passed. Removing the default=sys.stdin and piping something into my program I get:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
when running it. When I print args, I get: Namespace(app='app', config='', g=False, hosts='03.app', infile=None, version='').
It seems what I am doing is very simple, common and many people asked about it. But it doesn't seem to be working with me.
Any suggestions on how I can get it working?
This does it... without specifying arguments. If you pass pipe input to the program it goes, it you don't, it still goes. raw_input() will work as well.
import sys
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
print stdin
sys.stdin = open('/dev/tty')
else:
print "No stdin"
test_raw = raw_input()
print test_raw
Demo -
rypeck$ echo "stdin input" | python test_argparse.py -a test
['stdin input\n']
raw_input working!
raw_input working!
rypeck$ python test_argparse.py -a test
No stdin
raw_input working!
raw_input working!
I was poking at this issue myself and found a small improvement on the options here--hopefully it'll help future travelers. It's not 100% clear if you were hoping to only read from stdin when you provide a flag, but it seems that may have been your goal; my answer is predicated on this assumption. I'm working in 3.4, in case that becomes an issue...
I can declare a single optional argument like:
parser.add_argument("-v", "--version", nargs='?', const=sys.stdin, action=StreamType)
I'm not specifying a default because it is used only if the option is completely absent, while the const is used only if the flag is present but has no arguments. StreamType is a tiny argparse.Action subclass I wrote which just tries to read a line from the stdin stream and just saves the raw value if that doesn't work:
class StreamType(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
try:
setattr(namespace, self.dest, values.readline().strip())
except AttributeError:
setattr(namespace, self.dest, values)
This should produce something like:
$ blah --version v1.1.1
Namespace(version='v1.1.1')
$ echo "v1.0.3" | blah --version
Namespace(version='v1.0.3')
What do you do with args.infile? since you get a Namespace, argparse is not the part that is hanging or giving the error.
p = argparse.ArgumentParser()
p.add_argument('--infile', type=argparse.FileType('r'),default='-')
# p.add_argument('infile', nargs='?', type=argparse.FileType('r'),default='-') # positional version
args = p.parse_args()
print(args)
print args.infile.read()
-----------
$ cat myprog.py | python myprog.py --infile -
$ cat myprog.py | python myprog.py
$ python myprog.py myprog.py # with the positional version
$ python myprog.py - < myprog.py # with the positional version
echos the code nicely. The second call works with the 'optional positional' as well.
There is an unfortunate overlap in terminology, optional/positional and optional/required.
If a positional argument (yes, another use of 'positional') has a prefix character like - or -- it is called optional. By default its required parameter is False, but you may set it to True. But if the argument is 'infile' (no prefix), it is positional, even though with ? is is optional (not required).
By the way, default action is 'store', so you don't need to specify that. Also you don't need to specify required, unless it is True.
With a FileType, a handy way of specifying stdin is -.
Don't use '?' with --infile unless you really want a None
I'm not sure if I got your question correctly, but even if not, as I came here trying to solve my problem, this answer may help.
import argparse
import sys
def get_args():
parser = argparse.ArgumentParser(prog="my program")
parser.add_argument("--version", type=str, default = None)
args,unknown = parser.parse_known_args()
print(f"normal unknown: {unknown}")
if not sys.stdin.isatty():
stdin = sys.stdin.readlines()
stdin = ' '.join( (x.strip() for x in stdin ))
args, unknown2 = parser.parse_known_args(stdin.split(), namespace = args)
unknown.extend(unknown2)
print(f"stdin unknown: {unknown2}")
print(f"args: {args}")
get_args()
Now, I get
echo A --version=1.2.3 B | parse.py --C=some_value
normal unknown: ['--C=some_value']
stdin unknown: ['A', 'B']
args: Namespace(version='1.2.3')
My application is a specialized file comparison utility and obviously it does not make sense to compare only one file, so nargs='+' is not quite appropriate.
nargs=N only excepts a maximum of N arguments, but I need to accept an infinite number of arguments as long as there are at least two of them.
Short answer is you can't do that because nargs doesn't support something like '2+'.
Long answer is you can workaround that using something like this:
parser = argparse.ArgumentParser(usage='%(prog)s [-h] file file [file ...]')
parser.add_argument('file1', nargs=1, metavar='file')
parser.add_argument('file2', nargs='+', metavar='file', help=argparse.SUPPRESS)
namespace = parser.parse_args()
namespace.file = namespace.file1 + namespace.file2
The tricks that you need are:
Use usage to provide you own usage string to the parser
Use metavar to display an argument with a different name in the help string
Use SUPPRESS to avoid displaying help for one of the variables
Merge two different variables just adding a new attribute to the Namespace object that the parser returns
The example above produces the following help string:
usage: test.py [-h] file file [file ...]
positional arguments:
file
optional arguments:
-h, --help show this help message and exit
and will still fail when less than two arguments are passed:
$ python test.py arg
usage: test.py [-h] file file [file ...]
test.py: error: too few arguments
Couldn't you do something like this:
import argparse
parser = argparse.ArgumentParser(description = "Compare files")
parser.add_argument('first', help="the first file")
parser.add_argument('other', nargs='+', help="the other files")
args = parser.parse_args()
print args
When I run this with -h I get:
usage: script.py [-h] first other [other ...]
Compare files
positional arguments:
first the first file
other the other files
optional arguments:
-h, --help show this help message and exit
When I run it with only one argument, it won't work:
usage: script.py [-h] first other [other ...]
script.py: error: too few arguments
But two or more arguments is fine. With three arguments it prints:
Namespace(first='one', other=['two', 'three'])