Develop Reference

Using only one argument in argparse for multiple cases

$ script.py status
$ script.py terminate
$ script.py tail /tmp/some.log
As you can see the script can perform 3 tasks. The last task requires an additional argument (path of the file).
I want to use add_argument only once like below.
parser.add_argument("command")
And then check what command was requested by user and create conditionals based upon the same. If the command is tail I need to access the next argument (file path)

You might have to create a sub-parser for each command. This way it is extendable if those other commands also need arguments at some point. Something like this:
import argparse
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('--foo', action='store_true', help='global optional argument')
subparsers = parser.add_subparsers(dest="command", help='sub-command help')
# create the parser for the "status" command
parser_status = subparsers.add_parser('status', help='status help')
# create the parser for the "tail" command
parser_tail = subparsers.add_parser('tail', help='tail help')
parser_tail.add_argument('path', help='path to log')
print parser.parse_args()
The dest keyword of the add_subparsers ensures that you can still get the command name afterwards, as explained here and in the documentation.
Example usage:
$ python script.py status
Namespace(command='status', foo=False)
$ python script.py tail /tmp/some.log
Namespace(command='tail', foo=False, path='/tmp/some.log')
Note that any global argument needs to come before the command:
$ python script.py tail /tmp/some.log --foo
usage: PROG [-h] [--foo] {status,tail} ...
PROG: error: unrecognized arguments: --foo
$ python script.py --foo tail /tmp/some.log
Namespace(command='tail', foo=True, path='/tmp/some.log')

Configure argparse to accept quoted arguments

I am writing a program which, among other things, allows the user to specify through an argument a module to load (and then use to perform actions). I am trying to set up a way to easily pass arguments through to this inner module, and I was attempting to use ArgParse's action='append' to have it build a list of arguments that I would then pass through.
Here is a basic layout of the arguments that I am using
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
required=True)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
However - if I then try to run this program with python my_program -M module:class -A "-f filename" (where I would like to pass through the -f filename to my module) it seems to be parsing the -f as its own argument (and I get the error my_program: error: argument -A/--module_args: expected one argument
Any ideas?

With:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-M', '--module',
help="Module to run on changed files - should be in format MODULE:CLASS\n\
Specified class must have function with the signature run(src, dest)\
and return 0 upon success",
)
parser.add_argument('-A', '--module_args',
help="Arg to be passed through to the specified module",
action='append',
default=[])
import sys
print(sys.argv)
print(parser.parse_args())
I get:
1028:~/mypy$ python stack45146728.py -M module:class -A "-f filename"
['stack45146728.py', '-M', 'module:class', '-A', '-f filename']
Namespace(module='module:class', module_args=['-f filename'])
This is using a linux shell. The quoted string remains one string, as seen in the sys.argv, and is properly interpreted as an argument to -A.
Without the quotes the -f is separate and interpreted as a flag.
1028:~/mypy$ python stack45146728.py -M module:class -A -f filename
['stack45146728.py', '-M', 'module:class', '-A', '-f', 'filename']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
Are you using windows or some other OS/shell that doesn't handle quotes the same way?
In Argparse `append` not working as expected
you asked about a slightly different command line:
1032:~/mypy$ python stack45146728.py -A "-k filepath" -A "-t"
['stack45146728.py', '-A', '-k filepath', '-A', '-t']
usage: stack45146728.py [-h] [-M MODULE] [-A MODULE_ARGS]
stack45146728.py: error: argument -A/--module_args: expected one argument
As I already noted -k filepath is passed through as one string. Because of the space, argparse does not interpret that as a flag. But it does interpret the bare '-t' as a flag.
There was a bug/issue about the possibility of interpreting undefined '-xxx' strings as arguments instead of flags. I'd have to look that up to see whether anything made it into to production.
Details of how strings are categorized as flag or argument can be found in argparse.ArgumentParser._parse_optional method. It contains a comment:
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
http://bugs.python.org/issue9334 argparse does not accept options taking arguments beginning with dash (regression from optparse) is an old and long bug/issue on the topic.

The solution is to accept arbitrary arguments - there's an example in argparse's doc here:
argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('--foo')
>>> parser.add_argument('command')
>>> parser.add_argument('args', nargs=argparse.REMAINDER)
>>> print(parser.parse_args('--foo B cmd --arg1 XX ZZ'.split()))
Namespace(args=['--arg1', 'XX', 'ZZ'], command='cmd', foo='B')

accessing multiple files using argparse

I'm trying to read number of files using argparse:
parser.add_argument(
'-f',
'--text-file',
metavar='IN FILE',
type=argparse.FileType('r'),
nargs='*')
...
...
args = parser.parse_args()
print args
when more than one files are passed as command line arguments, only last file appears into args:
python example.py -o x.xml -s sss -c ccc -t "hello world" --report_failure -f ex.1 -f ex.2
Namespace(outputfile=<open file 'x.xml', mode 'w' at 0x028AD4F0>, report_failure=True, test_case='ccc', test_suite='sss', text='hello world', text_file=[<open file 'ex.2', mode 'r' at 0x028AD5A0>])
What I did wrong and how to access all files I passed from the command line?
Note: I'm using python 2.7.6 on Windows.

The complication occurs because you're passing multiple arguments to the same parameter -f and each argument replaces the argument before it. What would work in this case is:
python example.py -o x.xml -s sss -c ccc -t "hello world"
--report_failure -f ex.1 ex.2
This will collect ex.1 and ex.2 into a list, which is what I assume you want to do.
as a reference here is the docs on nargs:
'*'. All command-line arguments present are gathered into a list. Note
that it generally doesn’t make much sense to have more than one
positional argument with nargs='', but multiple optional arguments
with nargs='' is possible.
https://docs.python.org/3/library/argparse.html#nargs

argparse, two arguments depend on each other

I would like to make the parser like cmd [-a xxx -b xxx] -c xxx -d xxx
When -a is used, I want -b to be used too. likewise, if -b is used, -a must be used too. It's ok both -a and -b are not used.
How do I do that? I have tried custom actions, but it does not go well.

A better design would be to have a single option that takes two arguments:
parser.add_argument('-a', nargs=2)
Then you either specify the option with 2 arguments, or you don't specify it at all.
$ script -a 1 2
or
$ script
A custom action (or postprocessing) can split the tuple args.a into two separate values args.a and args.b.

Argparse doesn't natively support this type of use.
The most effective thing to do is check and see if those types of conditions are met after parsing:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-b')
parser.add_argument('-c')
args = parser.parse_args()
required_together = ('b','c')
# args.b will be None if b is not provided
if not all([getattr(args,x) for x in required_together]):
raise RuntimeError("Cannot supply -c without -b")

Python subprocess with complex arguments

I'm looking for the safest and most convenient way to call a shell command from python(3). Here a ps to pdf conversion:
gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile="${pdf_file}" "${ps_file}"
I use subprocess, shlex and avoid shell=True.
But I find the resulting command inconsistent:
cmd = ['gs', '-dBATCH', '-dNOPAUSE', '-sDEVICE=pdfwrite', '-sOutputFile={0}'.format(pdf_filename), ps_filename]
What do I miss?! subprocess.call() syntax looks so clean with space separated arguments, and looks such a mess everywhere else.
What's the difference when calling subprocess.call(cmd) (at python level, ie. escaping, injection protection, quoting, etc.) between:
cmd = ['do', '--something', arg]
cmd = ['do', '--someting {0}'.format(arg)]
If none, is this, also, a good way to do it ?
cmd = ['gs', '-dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile={0} {1}'.format(pdf_filename, ps_filename)]
Another example of inconsistency:
hg merge -r 3 would be cmd = ['hg', 'merge', '-r', revision_id]
hg merge --rev=3 would be cmd = ['hg', 'merge', '--rev={0}'.format(revision_id)]
despite the fact, it is two ways to send the same arguments.

The difference is that the command may have a --something option which accepts an argument, but it doesn't have a --something foo option -- which is what you would be telling it. When you run a command in your shell, like wc -l myfile.txt, your shell splits up that commandline where it finds spaces -- so the command that gets run is ['wc', '-l', 'myfile.txt'].
The subprocess module does not perform such splitting. You have to do it yourself (unless you use the 'shell' option, but that's generally less secure, so avoid it if you can.).
Some anti-examples...
Try to run a command named "wc -l myfile.txt". Of course, there is no "wc -l myfile.txt" command installed, only a "wc" command, so this will fail:
['wc -l myfile.txt']
Try to run a command "wc" with an option "-l myfile.txt". There is an "-l" option, but no "-l myfile.txt" option. This will fail:
['wc', '-l myfile.txt']
and a correct example:
['wc', '-l', 'myfile.txt']
This calls wc with the -l option (print only the line count) and myfile.txt as the only filename.
Something you may have found confusing is fragments like this:
'-sOutputFile={0}'
This is an 'inline' style of giving the argument of an option. If this is supported, the help for the program usually says so explicitly. Python does not split these -- the program receiving them does.
There are three main styles of 'inline' arguments. I'll use grep options to demo the first two:
--context=3
-C3
(the above two lines are equivalent)
The third type is only found in imagemagick and a few other programs that tend to have reams of commandline arguments, such as gs:
-sOutputFile=foo
This is just a minor variation on the GNU standard --long-option=VALUE form shown above.
The GNU libc manual's "argument syntax" section gives a full explanation of these option passing conventions.
In regards to escaping: No escaping is done, and no escaping is normally needed. The string values are passed exactly as you specify to the command. Naturally, no quoting is done nor is it needed, since you already took care of that in your Python code.
In regards to injection: this is not possible unless you use the 'shell' option. Don't use the 'shell' option :).

Difference between what you asked.. easy to check:
arg = 'foo'
cmd = ['do', '--something', arg]
print cmd
cmd = ['do', '--someting {0}'.format(arg)]
print cmd
>>>
['do', '--something', 'foo']
['do', '--someting foo']
As you can see they are not the same.
In order to call your subprocess correctly, you should do this:
cmd = ['gs', '-dBATCH', '-dNOPAUSE', '-sDEVICE=pdfwrite', '-sOutputFile={0}'.format(pdf_filename), ps_filename]
subprocess.Popen(cmd, ...)
OR:
cmd = 'gs -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile={0} {1}'.format(pdf_filename, ps_filename)
subprocess.Popen(cmd, shell=True, ...)
The difference between using a list of arguments or a string:
When you use a list of arguments, you are passing those as the arguments to the shell (or executable if you specify)
And when you send a string with shell=True you let the shell parse the string and make its own arguments...
So ['do', '--something', 'foo'] is 3 arguments, while ['do', '--someting foo'] is only 2 arguments.