Develop Reference

How can I get argparse to accept "--" as an argument to an option?

I have a command-line option that requires an argument. I would like to be able to supply "--" as the argument, but I can't figure out how to do it.
Sample code: (test-argparse.py)
#!/usr/bin/env python
from __future__ import print_function
import argparse
import sys
def main(argv):
ap = argparse.ArgumentParser()
ap.add_argument("-x", "--foo", metavar="VALUE", default="",
help="Test option.")
args = ap.parse_args(argv[1:])
print(args.foo)
if __name__ == "__main__":
sys.exit(main(sys.argv))
All my attempts to try to pass "--" as an argument fail:
$ test-argparse.py --foo --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: argument -x/--foo: expected one argument
$ test-argparse.py --foo -- --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: argument -x/--foo: expected one argument
$ test-argparse.py --foo=--
[]
$ test-argparse.py --foo=-- --
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: unrecognized arguments: --
$ test-argparse.py --foo="--"
[]
$ test-argparse.py '--foo --'
usage: test-argparse.py [-h] [-x VALUE]
test-argparse.py: error: unrecognized arguments: --foo --
$ test-argparse.py -x--
[]
$ test-argparse.py '-x --'
--
The last case is the closest, but it includes the space (and I can't just strip whitespace, because what if I want to allow " " as a value?). Is there any way that I can accomplish this?
That argparse forces argument permutation on clients (leading to unnecessary ambiguity) is very frustrating.
(I am using Python 2.7.12.)

Ideally --foo=-- should work, but the current parser deletes all '--', leaving an empty string in its place, hence the foo=[] result. I proposed a patch a couple of years ago that should have fixed that, but it's caught in the argparse backlog. http://bugs.python.org/issue13922, http://bugs.python.org/issue14364, http://bugs.python.org/issue9571
Python argparse with -- as the value suggests preprocessing sys.argv replacing one or more of the -- with something else.
If you are game for patching your argparse.py file (or subclass the ArgumentParser class), I could revisit my earlier work and suggest a fix. The trick is to accept that =-- but still use the first free -- as the 'rest-are-positionals' flag (and retain any following --). Unfortunately one method that needs to be patched is nested in a much larger one.

There is a specific reason that this doesn't work: -- means "Skip this token and consider the rest of the arguments to be positional, even if they start with a dash."
Many, many programs won't accept -- as an argument, but they will accept -. The single dash is even a standard way of specifying "Use standard input or output" in place of a filename.
So the best thing you can do for the users of your program is probably to not design it to require --, because that's not something that's usually done, and not something that most modern command-line parsing libraries are likely able to parse.
You could use -- as a positional option, so you could probably support this:
--foo -- --
If you make --foo have action='store_true' (i.e. it is an option taking no argument), plus one non-mandatory positional argument. That will probably work, because the first -- means "stop processing dashes as options" and the second is a positional argument.

Need to embed `-` character into arguments in python argparse

I am designing a tool to meet some spec. I have a scenario where I want the argument to contain - its string. Pay attention to arg-1 in the below line.
python test.py --arg-1 arg1Data
I am using the argparse library on python27. For some reason the argparse gets confused with the above trial.
My question is how to avoid this? How can I keep the - in my argument?
A sample program (containing the -, if this is removed everything works fine):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--arg-1", help="increase output verbosity")
args = parser.parse_args()
if args.args-1:
print "verbosity turned on"

Python argparse module replace dashes by underscores, thus:
if args.arg_1:
print "verbosity turned on"
Python doc (second paragraph of section 15.4.3.11. dest) states:
Any internal - characters will be converted to _ characters to make
sure the string is a valid attribute name.

import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--arg-1", help="increase output verbosity")
parser.add_argument("arg-2")
args = parser.parse_args()
print(args)
produces:
1750:~/mypy$ python stack34970533.py -h
usage: stack34970533.py [-h] [--arg-1 ARG_1] arg-2
positional arguments:
arg-2
optional arguments:
-h, --help show this help message and exit
--arg-1 ARG_1 increase output verbosity
and
1751:~/mypy$ python stack34970533.py --arg-1 xxx yyy
Namespace(arg-2='yyy', arg_1='xxx')
The first argument is an optional. You can use '--arg-1' in commandline, but the value is stored as args.arg_1. Python would interpret args.arg-1 as args.arg - 1. There's a long history of unix commandlines allowing flags with a -. It tries to balance both traditions.
It leaves you in full control of the positionals dest attribute, and does not change the - to _. If you want to access that you have to use the getattr approach. There is bug/issue discussing whether this behavior should be changed or not. But for now, if you want to make it hard on yourself, that's your business.
Internally, argparse accesses the namespace with getattr and setattr to minimize restrictions on the attribute names.

how to take into account '-' in an argument in argparse [duplicate]

Is there a way to make argparse recognize anything between two quotes as a single argument? It seems to keep seeing the dashes and assuming that it's the start of a new option
I have something like:
mainparser = argparse.ArgumentParser()
subparsers = mainparser.add_subparsers(dest='subcommand')
parser = subparsers.add_parser('queue')
parser.add_argument('-env', '--extraEnvVars', type=str,
help='String of extra arguments to be passed to model.')
...other arguments added to parser...
But when I run:
python Application.py queue -env "-s WHATEVER -e COOL STUFF"
It gives me:
Application.py queue: error: argument -env/--extraEnvVars: expected one argument
If I leave off the first dash, it works totally fine, but it's kind of crucial that I be able to pass in a string with dashes in it. I've tried escaping it with \ , which causes it to succeed but adds the \ to the argument string Does anyone know how to get around this? This happens whether or not -s is an argument in parser.
EDIT: I'm using Python 2.7.
EDIT2:
python Application.py -env " -env"
works perfectly fine, but
python Application.py -env "-env"
does not.
EDIT3: Looks like this is actually a bug that's being debated already: http://www.gossamer-threads.com/lists/python/bugs/89529, http://python.6.x6.nabble.com/issue9334-argparse-does-not-accept-options-taking-arguments-beginning-with-dash-regression-from-optp-td578790.html. It's only in 2.7 and not in optparse.
EDIT4: The current open bug report is: http://bugs.python.org/issue9334

Updated answer:
You can put an equals sign when you call it:
python Application.py -env="-env"
Original answer:
I too have had troubles doing what you are trying to do, but there is a workaround build into argparse, which is the parse_known_args method. This will let all arguments that you haven't defined pass through the parser with the assumption that you would use them for a subprocess. The drawbacks are that you won't get error reporting with bad arguments, and you will have to make sure that there is no collision between your options and your subprocess's options.
Another option could be to force the user's to use a plus instead of a minus:
python Application.py -e "+s WHATEVER +e COOL STUFF"
and then you change the '+' to '-' in post processing before passing to your subprocess.

This issue is discussed in depth in http://bugs.python.org/issue9334. Most of the activity was in 2011. I added a patch last year, but there's quite a backlog of argparse patches.
At issue is the potential ambiguity in a string like '--env', or "-s WHATEVER -e COOL STUFF" when it follows an option that takes an argument.
optparse does a simple left to right parse. The first --env is an option flag that takes one argument, so it consumes the next, regardless of what it looks like. argparse, on the other hand, loops through the strings twice. First it categorizes them as 'O' or 'A' (option flag or argument). On the second loop it consumes them, using a re like pattern matching to handle variable nargs values. In this case it looks like we have OO, two flags and no arguments.
The solution when using argparse is to make sure an argument string will not be confused for an option flag. Possibilities that have been shown here (and in the bug issue) include:
--env="--env" # clearly defines the argument.
--env " --env" # other non - character
--env "--env " # space after
--env "--env one two" # but not '--env "-env one two"'
By itself '--env' looks like a flag (even when quoted, see sys.argv), but when followed by other strings it does not. But "-env one two" has problems because it can be parsed as ['-e','nv one two'], a `'-e' flag followed by a string (or even more options).
-- and nargs=argparse.PARSER can also be used to force argparse to view all following strings as arguments. But they only work at the end of argument lists.
There is a proposed patch in issue9334 to add a args_default_to_positional=True mode. In this mode, the parser only classifies strings as option flags if it can clearly match them with defined arguments. Thus '--one' in '--env --one' would be classed as as an argument. But the second '--env' in '--env --env' would still be classed as an option flag.
Expanding on the related case in
Using argparse with argument values that begin with a dash ("-")
parser = argparse.ArgumentParser(prog="PROG")
parser.add_argument("-f", "--force", default=False, action="store_true")
parser.add_argument("-e", "--extra")
args = parser.parse_args()
print(args)
produces
1513:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra='--foo one', force=False)
1513:~/mypy$ python3 stack16174992.py --extra "-foo one"
usage: PROG [-h] [-f] [-e EXTRA]
PROG: error: argument -e/--extra: expected one argument
1513:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra='-bar one', force=False)
1514:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra='one', force=True)
The "-foo one" case fails because the -foo is interpreted as the -f flag plus unspecified extras. This is the same action that allows -fe to be interpreted as ['-f','-e'].
If I change the nargs to REMAINDER (not PARSER), everything after -e is interpreted as arguments for that flag:
parser.add_argument("-e", "--extra", nargs=argparse.REMAINDER)
All cases work. Note the value is a list. And quotes are not needed:
1518:~/mypy$ python3 stack16174992.py --extra "--foo one"
Namespace(extra=['--foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-foo one"
Namespace(extra=['-foo one'], force=False)
1519:~/mypy$ python3 stack16174992.py --extra "-bar one"
Namespace(extra=['-bar one'], force=False)
1519:~/mypy$ python3 stack16174992.py -fe one
Namespace(extra=['one'], force=True)
1520:~/mypy$ python3 stack16174992.py --extra --foo one
Namespace(extra=['--foo', 'one'], force=False)
1521:~/mypy$ python3 stack16174992.py --extra -foo one
Namespace(extra=['-foo', 'one'], force=False)
argparse.REMAINDER is like '*', except it takes everything that follows, whether it looks like a flag or not. argparse.PARSER is more like '+', in that it expects a positional like argument first. It's the nargs that subparsers uses.
This uses of REMAINDER is documented, https://docs.python.org/3/library/argparse.html#nargs

You can start the argument with a space python tst.py -e ' -e blah' as a very simple workaround. Simply lstrip() the option to put it back to normal, if you like.
Or, if the first "sub-argument" is not also a valid argument to the original function then you shouldn't need to do anything at all. That is, the only reason that python tst.py -e '-s hi -e blah' doesn't work is because -s is a valid option to tst.py.
Also, the optparse module, now deprecated, works without any issue.

I have ported a script from optparse to argparse, where certain arguments took values that could start with a negative number. I ran into this problem because the script is used in many places without using the '=' sign to join negative values to the flag. After reading the discussion here and in http://bugs.python.org/issue9334, I know the arguments only take one value and there was no risk in accepting a succeeding argument (ie, a missing value) as the value. FWIW, my solution was to preprocess the arguments and join the problematic ones with '=' before passing to parse_args():
def preprocess_negative_args(argv, flags=None):
if flags is None:
flags = ['--time', '--mtime']
result = []
i = 0
while i < len(argv):
arg = argv[i]
if arg in flags and i+1 < len(argv) and argv[i+1].startswith('-'):
arg = arg + "=" + argv[i+1]
i += 1
result.append(arg)
i += 1
return result
This approach at least does not require any user changes, and it only modifies the arguments which explicitly need to allow negative values.
>>> import argparse
>>> parser = argparse.ArgumentParser("prog")
>>> parser.add_argument("--time")
>>> parser.parse_args(preprocess_negative_args("--time -1d,2".split()))
Namespace(time='-1d,2')
It would be more convenient to tell argparse which arguments should explicitly allow values with a leading dash, but this approach seems like a reasonable compromise.

Similar problem. And I solve this by replace space by "\ ". For example:
replace
python Application.py "cmd -option"
by
python Application.py "cmd\ -option".
Not sure for your problem.

paser.add_argument("--argument_name", default=None, nargs=argparse.REMAINDER)
python_file.py --argument_name "--abc=10 -a=1 -b=2 cdef"
Note: the argument values have to be passed only within double quotes and this doesn't work with single quotes

To bypass having to deal with argparse even looking at a '-' for something that isn't a flag you want, you can edit sys.argv before argparse reads it. Just save the argument that you don't want seen, put a filler argument in it's place, and then replace the filler with the original after argparse process sys.argv. I just had to do this for my own code. It's not pretty, but it works and it's easy. You could also use a for loop to iterate through sys.argv if your flags aren't always in the same order.
parser.add_argument('-n', '--input', nargs='*')
spot_saver = ''
if sys.argv[1] == '-n': #'-n' can be any flag you use
if sys.argv[2][0] == '-': #This checks the first character of the element
spot_saver = sys.argv[2]
sys.argv[2] = "fillerText"
args = parser.parse_args()
if args.input[0] == 'fillerText':
args.input[0] = spot_saver

Parsing command-line arguments similar to archlinux pacman

I'm creating a python script with usage in the same style as pacman in Arch Linux, summarized by:
prog <operation> [options] [targets]
operations are of the form -X (hyphen, uppercase letter), and one is required when calling the script.
options are of the form -x (hyphen, lowercase letter), and can mean different things for different operations.
For example:
pacman -Syu means perform the sync operation with refresh and sysupgrade options, upgrading the entire system with fresh packages.
pacman -Qu means perform the query operation with the upgrades option, listing all outdated packages.
pacman -Ss <arg> means perform the sync operation with the search option, which expects another argument as the pattern to search for in the sync packages.
The punchline:
I've been looking into the argparse library for python, trying to figure out how to implement this. I've run into some problems/design issues so far:
argparse only accepts hyphen-prefixed arguments as optional arguments. All my "operations" would show up as optional arguments, when one is definitely required.
I could make my script have one "positional"/required argument, which would be the operation (I would have to switch operations to words, like upgrade or add), followed by optional arguments. This, however, still wouldn't solve the same-option-symbol-working-differently issue, and also wouldn't let me easily list all the supported operations in the --help text.
What's the smoothest way to handle this argument parsing? I'm not against changing my command's usage, but as I said above, it doesn't seem to help my situation as far as I can tell.
Thanks

One option would be to make -S and -Q part of a mutually exclusive option group with the required keyword argument set to True. This wouldn't enforce the requirement to make those the first arguments given, nor would it restrict which other options could be used with each. You'd have to enforce the latter after calling parse_args.
Another option I thought of was to make -S and -Q subcommands. Calling add_parser with a first argument starting with '-' seems to be legal, but the errors you get when you actually try to call your script makes me think that either the support is buggy/unintended, or that the error reporting is buggy.

Yet another option would be to use getopt: http://docs.python.org/library/getopt.html

So I found this support for sub-commands buried in the argparse help. It's exactly what I need, with the only caveat being I am not using -X as the format for operations; I am just using words like add and search instead.
For completeness here's an example of using sub-parsers from the link above:
>>> # create the top-level parser
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo', action='store_true', help='foo help')
>>> subparsers = parser.add_subparsers(help='sub-command help')
>>>
>>> # create the parser for the "a" command
>>> parser_a = subparsers.add_parser('a', help='a help')
>>> parser_a.add_argument('bar', type=int, help='bar help')
>>>
>>> # create the parser for the "b" command
>>> parser_b = subparsers.add_parser('b', help='b help')
>>> parser_b.add_argument('--baz', choices='XYZ', help='baz help')
>>>
>>> # parse some argument lists
>>> parser.parse_args(['a', '12'])
Namespace(bar=12, foo=False)
>>> parser.parse_args(['--foo', 'b', '--baz', 'Z'])
Namespace(baz='Z', foo=True)

With Python's optparse module, how do you create an option that takes a variable number of arguments?

With Perl's Getopt::Long you can easily define command-line options that take a variable number of arguments:
foo.pl --files a.txt --verbose
foo.pl --files a.txt b.txt c.txt --verbose
Is there a way to do this directly with Python's optparse module? As far as I can tell, the nargs option attribute can be used to specify a fixed number of option arguments, and I have not seen other alternatives in the documentation.

This took me a little while to figure out, but you can use the callback action to your options to get this done. Checkout how I grab an arbitrary number of args to the "--file" flag in this example.
from optparse import OptionParser,
def cb(option, opt_str, value, parser):
args=[]
for arg in parser.rargs:
if arg[0] != "-":
args.append(arg)
else:
del parser.rargs[:len(args)]
break
if getattr(parser.values, option.dest):
args.extend(getattr(parser.values, option.dest))
setattr(parser.values, option.dest, args)
parser=OptionParser()
parser.add_option("-q", "--quiet",
action="store_false", dest="verbose",
help="be vewwy quiet (I'm hunting wabbits)")
parser.add_option("-f", "--filename",
action="callback", callback=cb, dest="file")
(options, args) = parser.parse_args()
print options.file
print args
Only side effect is that you get your args in a list instead of tuple. But that could be easily fixed, for my particular use case a list is desirable.

My mistake: just found this Callback Example 6.

I believe optparse does not support what you require (not directly -- as you noticed, you can do it if you're willing to do all the extra work of a callback!-). You could also do it most simply with the third-party extension argparse, which does support variable numbers of arguments (and also adds several other handy bits of functionality).
This URL documents argparse's add_argument -- passing nargs='*' lets the option take zero or more arguments, '+' lets it take one or more arguments, etc.

Wouldn't you be better off with this?
foo.pl --files a.txt,b.txt,c.txt --verbose

I recently has this issue myself: I was on Python 2.6 and needed an option to take a variable number of arguments. I tried to use Dave's solution but found that it wouldn't work without also explicitly setting nargs to 0.
def arg_list(option, opt_str, value, parser):
args = set()
for arg in parser.rargs:
if arg[0] == '-':
break
args.add(arg)
parser.rargs.pop(0)
setattr(parser.values, option.dest, args)
parser=OptionParser()
parser.disable_interspersed_args()
parser.add_option("-f", "--filename", action="callback", callback=arg_list,
dest="file", nargs=0)
(options, args) = parser.parse_args()
The problem was that, by default, a new option added by add_options is assumed to have nargs = 1 and when nargs > 0 OptionParser will pop items off rargs and assign them to value before any callbacks are called. Thus, for options that do not specify nargs, rargs will always be off by one by the time your callback is called.
This callback is can be used for any number of options, just have callback_args be the function to be called instead of setattr.

Here's one way: Take the fileLst generating string in as a string and then use http://docs.python.org/2/library/glob.html to do the expansion ugh this might not work without escaping the *
Actually, got a better way:
python myprog.py -V -l 1000 /home/dominic/radar/*.json <- If this is your command line
parser, opts = parse_args()
inFileLst = parser.largs