Develop Reference

Python: help menu doesn't show all options when using parse_known_args()

I went through a "little" issue using python argument parser, I've created a parser like shown below:
parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument('-l', '--log-level', help="log level")
parser.add_argument('-o', '--out-dir', help="output directory")
args, remaining = parser.parse_known_args()
outpu_dir = args.out_dir
parser.add_argument('-f', '--log-file', help = "log file directory")
args = parser.parse_args()
and the problem is that when calling the program with --help option, arguments added after parse_known_args() are not listed in the menu, I checked other topics related to this, but my case is a bit different! could any one give a help here please ?

Any code which is below the call to parse_known_args will not execute at all.
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument('-a')
parser.parse_known_args()
1/0
When running with --help this generates
usage: main.py [-h] [-a A]
optional arguments:
-h, --help show this help message and exit
-a A
Process finished with exit code 0
and not a ZeroDivisionError exception.

You can work around it by adding the help option after all the other args have been added & you've parsed the known args.
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter, SUPPRESS
# don't add help right now
parser = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter, add_help=False)
parser.add_argument('-l', '--log-level', help="log level")
parser.add_argument('-o', '--out-dir', help="output directory")
args, remaining = parser.parse_known_args()
outpu_dir = args.out_dir
parser.add_argument('-f', '--log-file', help="log file directory")
# add help later, when all other args have been added
parser.add_argument('-h', '--help', action='help', default=SUPPRESS,
help='Show this help message and exit.')
args = parser.parse_args()
Which results in:
(venv) ➜ python main.py --help
usage: main.py [-l LOG_LEVEL] [-o OUT_DIR] [-f LOG_FILE] [-h]
optional arguments:
-l LOG_LEVEL, --log-level LOG_LEVEL
log level (default: None)
-o OUT_DIR, --out-dir OUT_DIR
output directory (default: None)
-f LOG_FILE, --log-file LOG_FILE
log file directory (default: None)
-h, --help Show this help message and exit.
But it's much better if you can restructure your code to add all the argusments first before doing any parsing (i.e. avoid the call to parse_known_aergs before any other add_argument calls)

deferring exiting on --help when using parse_known_args()

I have a code that needs to build its command line arguments dynamically, based on configuration file. Schematically, what I end up doing is
parser = argparse.ArgumentParser()
parser.add_argument("-c", "--config", type=str, default="",
help="Path to config file.")
args, unknown = parser.parse_known_args()
#here do stuff with args.config to extend parser list of arguments, and then:
parser.parse_args()
The argument management seems to work perfectly well but the trouble I have is that --help will exit at the first call parse_known_args, instead of the second parser.parse_args() which would have shown all the dynamically added arguments.... Is there a way to solve this?

I had the same problem some time ago. It can be solved by using two parsers.
A first pre-parser, with add_help=False to determine the configuration file;
The second full parser with help.
# test.py
import argparse
# First pre-parser to determine the configuration file
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument("-c", "--config", default="", help="Path to config file.")
args = parser.parse_known_args()
# Do something with args.config
# Full and real parser
parser = argparse.ArgumentParser()
parser.add_argument("-c", "--config", default="", help="Path to config file.")
parser.add_argument("-o", "--option", help="Option with default value from config file")
parser.parse_args()
This result in:
$ python3 test.py --help
usage: test.py [-h] [-c CONFIG] [-o OPTION]
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
Path to config file.
-o OPTION, --option OPTION
Option with default value from config file

Python 2.7. Parse input parameters

Good time of the day! First of all, let me say that I'm a newbie in Python world. I've problems with parsing input parameters. At this moment I'm using Python 2.7 and module which is called argparse. I'm trying to design simple application which will be able to parse simple input parameters. Here is a short example:
my_app.py sync --force
Second example:
my_app.py patch --branch
I see that for this I can use add_argument which can work with positional and optional arguments. Like in my case I want to have few positional (but optional at the same time) and few optional arguments.
To do that I've design small script
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='My App')
parser.add_argument('sync', type=bool, const=True, nargs='?')
parser.add_argument('-f', '--force', dest='sync_force', type=bool, const=True, nargs='?')
parser.add_argument('-b', '--branch', type=str, const=True, nargs='?')
parser.add_argument('-u', '--url', type=str, const=True, nargs='?')
parser.add_argument('patch', type=bool, const=True, nargs='?')
parser.add_argument('revert', type=bool, const=True, nargs='?')
parser.add_argument('verify', type=bool, const=True, nargs='?')
values = parser.parse_args()
if values.revert:
handler.revert()
else:
parser.print_help()
I see that I can use nargs='?' to specify the positional parameter as optional, but each time when I'm calling my script it shows like I got 'sync' as input parameter, even if I specified 'patch'. So, I think that it shows just first added element.
Could you tell me what's wrong and where is a problem?
Update:
I'm trying to achieve a situation when I will be able to have only one positional argument at the same time(and at least one, but with any additional optional parameters). For example
my_app.py sync
my_app.py path
my_app.py verify --force
my_app.pyrevert --branch

With:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='My App')
parser.add_argument('cmd', choices=['sync','patch', 'revert','verify'])
parser.add_argument('-f', '--force', action='store_true')
parser.add_argument('-b', '--branch')
parser.add_argument('-u', '--url')
args = parser.parse_args()
print(args)
if args.cmd in ['revert']:
print('handler.revert()')
else:
parser.print_help()
testing
1243:~/mypy$ python stack60327766.py
usage: stack60327766.py [-h] [-f] [-b BRANCH] [-u URL]
{sync,patch,revert,verify}
stack60327766.py: error: too few arguments
1244:~/mypy$ python stack60327766.py revert
Namespace(branch=None, cmd='revert', force=False, url=None)
handler.revert()
1244:~/mypy$ python stack60327766.py revert -f -b foobar -u aurl
Namespace(branch='foobar', cmd='revert', force=True, url='aurl')
handler.revert()
1244:~/mypy$ python stack60327766.py verify-f -b foobar -u aurl
usage: stack60327766.py [-h] [-f] [-b BRANCH] [-u URL]
{sync,patch,revert,verify}
stack60327766.py: error: argument cmd: invalid choice: 'verify-f' (choose from 'sync', 'patch', 'revert', 'verify')
1245:~/mypy$ python stack60327766.py verify -f -b foobar -u aurl
Namespace(branch='foobar', cmd='verify', force=True, url='aurl')
usage: stack60327766.py [-h] [-f] [-b BRANCH] [-u URL]
{sync,patch,revert,verify}
My App
positional arguments:
{sync,patch,revert,verify}
optional arguments:
-h, --help show this help message and exit
-f, --force
-b BRANCH, --branch BRANCH
-u URL, --url URL

argparse has native support for subcommands.
Adapting from the above-linked documentation,
parser = argparse.ArgumentParser()
parser.add_argument('--foo')
subparsers = parser.add_subparsers()
sync_parser = subparsers.add_parser('sync')
patch_parser = subparsers.add_parser('patch')
revert_parser = subparsers.add_parser('revert')
verify_parser = subparsers.add_parser('verify')
etc. is probably what you're looking for.
Beyond that, I can only recommend the Click library for a more fluent interface for CLIs.

How to create subparser with argparse from existing program in Python 3?

Original post:
If one has an executable mini_program.py that uses argparse with the following structure:
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
opts = parser.parse_args()
if __name__ == "__main__":
main()
How can one create a controller program parent_program.py that uses argparse (I think with subparser?) to have a similar usage to below:
python parent_program.py --help
blah-blah list of programs that can be used
then using the subprogram:
python parent_program.py mini_program --help
-X description
-y description
etc...
How could all of the parameters propagate up from mini_program.py to the parent_program.py?
EDIT (More specific with error message):
The program
import argparse
def main():
parser = argparse.ArgumentParser()
# Subprograms
subprograms = parser.add_subparsers(title="subprograms")
# ============
# mini-program
# ============
parser_miniprogram = subprograms.add_parser("miniprogram")
# Input
parser_miniprogram.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
parser_miniprogram.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
opts = parser.parse_args()
opts_miniprogram = parser_miniprogram.parse_args()
print(opts_miniprogram.__dict__)
if __name__ == "__main__":
main()
Checking to make sure the docs work
# parent program
python parent_program.py --help
usage: parent_program.py [-h] {miniprogram} ...
optional arguments:
-h, --help show this help message and exit
subprograms:
{miniprogram}
# miniprogram
python parent_program.py miniprogram --help
usage: parent_program.py miniprogram [-h] [-X ATTRIBUTE_MATRIX]
[-y TARGET_VECTOR]
optional arguments:
-h, --help show this help message and exit
-X ATTRIBUTE_MATRIX, --attribute_matrix ATTRIBUTE_MATRIX
Input: Path/to/Tab-separated-value.tsv
-y TARGET_VECTOR, --target_vector TARGET_VECTOR
Input: Path/to/Tab-separated-value.tsv
Trying to run it:
python parent_program.py miniprogram -X ../../Data/X_iris.noise_100.tsv.gz -y ../../Data/y_iris.tsv
usage: parent_program.py miniprogram [-h] [-X ATTRIBUTE_MATRIX]
[-y TARGET_VECTOR]
parent_program.py miniprogram: error: unrecognized arguments: miniprogram

The parent program could have code like
import mini_program
import sys
<do its own parsing>
if 'use_mini':
<modify sys.argv>
mini_program.main()
As written, importing mini_program doesn't run its parser. But calling its main will, but using the list it finds in sys.argv.
The parent parser should be written in a way that it accepts arguments that it needs, and doesn't choke on inputs the mini wants, '-X' and '-y'. It would then puts those 'extra' values in a modified sys.argv, which the mini parser can handle.
parse_known_args is one way of accepting unknown arguments,
https://docs.python.org/3/library/argparse.html#partial-parsing
nargs=argparse.REMAINDER, https://docs.python.org/3/library/argparse.html#nargs, is another way of collecting remaining arguments for passing on.
If mini main was written as:
def main(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
opts = parser.parse_args(argv)
it could be called with
mini_program.main(['-X', 'astring','-y','another'])
that is, with an explicit argv list, instead of working through sys.argv.
Keeping the main parser from responding to a '-h' help could be tricky. subparsers is probably the cleanest way of doing that.
You could combine subparsers with the invocation of a the mini main. I won't try to work out those details now.
Another way to define the main is:
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-X', '--attribute_matrix', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
parser.add_argument('-y', '--target_vector', type=str, help = 'Input: Path/to/Tab-separated-value.tsv')
return parser
And use it as
opts = main().parse_args()
opts = mini_program.main().parse_args()
in other words, use main to define the parser, but delay the parsing.

My actual solution was an adaptation to the above:
# Controller
def main(argv=None):
parser = argparse.ArgumentParser(prog="parent_program", add_help=True)
parser.add_argument("subprogram")
opts = parser.parse_args(argv)
return opts.subprogram
# Initialize
if __name__ == "__main__":
# Get the subprogram
subprogram = main([sys.argv[1]])
module = importlib.import_module(subprogram)
module.main(sys.argv[2:])

Python argparse: name parameters

I'm writing a program that use argparse, for parsing some arguments that I need.
for now I have this:
parser.add_argument('--rename', type=str, nargs=2, help='some help')
when I run this script I see this:
optional arguments:
-h, --help show this help message and exit
--rename RENAME RENAME
some help
How can I change my code in that way that the help "page" will show me:
--rename OLDFILE NEWFILE
Can I then use OLDFILE and NEWFILE value in this way?
args.rename.oldfile
args.rename.newfile

If you set metavar=('OLDFILE', 'NEWFILE'):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--rename', type=str, nargs=2, help='some help',
metavar=('OLDFILE', 'NEWFILE'))
args = parser.parse_args()
print(args)
Then test.py -h yields
usage: test.py [-h] [--rename OLDFILE NEWFILE]
optional arguments:
-h, --help show this help message and exit
--rename OLDFILE NEWFILE
some help
You can then access the arguments with
oldfile, newfile = args.rename
If you really want to access the oldfile with args.rename.oldfile
you could set up a custom action:
import argparse
class RenameAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest,
argparse.Namespace(
**dict(zip(('oldfile', 'newfile'),
values))))
parser = argparse.ArgumentParser()
parser.add_argument('--rename', type=str, nargs=2, help='some help',
metavar=('OLDFILE', 'NEWFILE'),
action=RenameAction)
args = parser.parse_args()
print(args.rename.oldfile)
but it extra code does not really seem worth it to me.

Read the argparse documentation (http://docs.python.org/2.7/library/argparse.html#metavar):
Different values of nargs may cause the metavar to be used multiple times. Providing a tuple to metavar specifies a different display for each of the arguments:
>>> parser = argparse.ArgumentParser(prog='PROG')
>>> parser.add_argument('-x', nargs=2)
>>> parser.add_argument('--foo', nargs=2, metavar=('bar', 'baz'))
>>> parser.print_help()
usage: PROG [-h] [-x X X] [--foo bar baz]
optional arguments:
-h, --help show this help message and exit
-x X X
--foo bar baz