Develop Reference

Argparse, displaying custom help text without any of the boilerplate argparse text

After looking at about a dozen questions, I can't seem to find an answer.
I have a python CLI i've written using argparse. I have a main command that does nothing but regurgitate help text and then 4 subcommands. My boss wants a very specific output for the help text. He has me write it out as a text file and then we use that text file to display the help text.
However, in some circumstances, it STILL outputs parts of the argparse help text.
For example, if I run my program with no subcommands, it just outputs our help text from the file. But if I use "-h" or "--help" it will output our help text, followed by the list of positional and optional arguments and other argparse stuff. We don't want that.
I could use "add_help=False" but we want the user to be able to type -h and still get our help text. If we set add help to false, it will display our help text followed by the error "-h not recognized".
Also doing this does nothing for when the user uses -h after a subcommand. I set help=None and usage is set to my custom help text for each subcommand, but it still shows the boilerplate argparse info at the end.
This is what I want to happen: user types in the main command with no subcommands prints my custom help text and nothing else. The user types the main command, no subcommand, followed by -h/--help and it prints my custom help text and nothing else. User types in the main command, one of the subcommands, followed by -h/--help and it outputs my help text and nothing else. User types the main command, a subcommand, and then wrong arguments or too many/ too few arguments displays my help text and nothing else. Basically I only ever want it to print nothing, or print just my help text.
how do I do that? here is my main function where the parsers and subparsers are all configured:
def main():
# Import help text from file
p = Path(__file__).with_name("help.txt")
with p.open() as file:
help_text = file.read()
# Configure the top level Parser
parser = argparse.ArgumentParser(prog='myprog', description='My program', usage=help_text)
subparsers = parser.add_subparsers()
# Create Subparsers to give subcommands
parser_one = subparsers.add_parser('subcommandone', prog='subcommandone', usage=help_text, help=None)
parser_one.add_argument('arg1', type=str)
parser_one.add_argument('-o', '--option1', default='mydefault', type=str)
parser_two= subparsers.add_parser('subcommandtwo', usage=help_text, help=None, prog='subcommandtwo')
parser_three= subparsers.add_parser('subcommandthree', usage=help_text, help=None, prog='subcommandthree')
parser_four= subparsers.add_parser('subcommandfour', usage=help_text, help=None, prog='subcommandfour')
# Assign subparsers to their respective functions
parser_one.set_defaults(func=functionone)
parser_two.set_defaults(func=functiontwo)
parser_three.set_defaults(func=functionthree)
parser_four.set_defaults(func=functionfour)
parser.set_defaults(func=base_case)
# Parse the arguments and call appropriate functions
args = parser.parse_args()
if len(sys.argv) == 1:
args.func(args, parser)
else:
args.func(args)
Any thoughts?

You can use sys.exit() after the help text has been displayed, and before the parsing has begun, to avoid problems with "-h not recognized".
So anywhere before the line
# Parse the arguments and call appropriate functions
add
if len(sys.argv) == 1 or '-h' in sys.argv or '--help' in sys.argv:
print(help_text)
sys.exit(1)
In situations where that is not good enough you can subclass argparse.HelpFormatter like so
usage_help_str = 'myscript command [options]'
epilog_str = "More info can be found at https://..."
class Formatter(argparse.HelpFormatter):
# override methods and stuff
def formatter(prog):
return Formatter(prog)
parser = argparse.ArgumentParser(formatter_class=formatter, epilog=epilog_str, usage=usage_help_str, add_help=False)
I tried looking around for documentation on subclassing the helpFormatter, but I couldn't find anything. It looks like people are just looking at the source code to figure out how to subclass it.

Best architecture for a Python command-line tool with multiple subcommands

I am developing a command-line toolset for a project. The final tool shall support many subcommands, like so
foo command1 [--option1 [value]?]*
So there can be subcommands like
foo create --option1 value --
foo make file1 --option2 --option3
The tool uses the argparse library for handling command-line arguments and help functionality etc.
A few additional requirements and constraints:
Some options and functionality is identical for all subcommands (e.g. parsing a YAML configuration file etc.)
Some subcommands are quick and simple to code, because they e.g. just call an external bash script.
Some subcommands will be complex and hence long code.
Help for the basic tool should be available as well as for an individual subcommand:
foo help
Available commands are: make, create, add, xyz
foo help make
Details for the make subcommand
error codes should be uniform across the subcommands (like the same error code for "file not found")
For debugging purposes and for making progress with self-contained functionality for minimal viable versions, I would like to develop some subcommands as self-containted scripts and modules, like
make.py
that can be imported into the main foo.py script and later on invoked as both
make.py --option1 value etc.
and
foo.py make --option1 value
Now, my problem is: What is the best way to modularize such a complex CLI tool with minimal redundancy (e.g. the arguments definition and parsing should only be coded in one component)?
Option 1: Put everything into one big script, but that will become difficult to manage.
Option 2: Develop the functionality for a subcommand in individual modules / files (like make.py, add.py); but such must remain invocable (via if __name__ == '__main__' ...).
The functions from the subcommand modules could then be imported into the main script, and the parser and arguments from the subcommand added as a subparser.
Option 3: The main script could simply reformat the call to a subcommand to subprocess, like so
subprocess.run('./make.py {arguments}', shell=True, check=True, text=True)

I'm more used to answering questions about the details of numpy and argparse, but here's how I envisage a large package.
In a main.py:
import submod1
# ....
sublist = [submod1, ...]
def make_parser(sublist):
parser = argparse.ArgumentParser()
# parser.add_argument('-f','--foo') # main specific
# I'd avoid positionals
sp = parser.add_subparsers(dest='cmd', etc)
splist=[]
for md in sublist:
sp1 = sp.add_parser(help='', parents=[md.parser])
sp1.set_default(func=md.func) # subparser func as shown in docs
splist.append(sp1)
return parser
if name == 'main':
parser = make_parser(sublist)
args = parser.parse_args()
# print(args) # debugging display
args.func(args) # again the subparser func
In submod1.py
import argparse
def make_parser():
parser = argparse.ArgumentParser(add_help=False) # check docs?
parser.add_argument(...) # could add a common parents here
return parser
parser.make_parser()
def func(args):
# module specific 'main'
I'm sure this is incomplete in many ways, since I've written this on the fly without testing. It's a basic subparser definition as documented, but using parents to import subparsers as defined in the submodules. parents could also be used to define common arguments for subparsers; but utility functions would work just as well. I think parents is most useful when using a parser that you can't otherwise access; ie. an imported one.
parents essentially copies Actions from one parser to the new one - copy by reference (not by value or as a copy). It is not a highly developed tool, and there have been a number of SO where people ran into problems. So don't try to over extend it.

Consider using the Command Pattern along with the Factory Method Pattern.
In short, create an abstract class called Command and make each command it's own class inheriting from Command.
Example:
class Command():
def execute(self):
raise NotImplementedError()
class Command1(Command):
def __init__(self, *args):
pass
def execute(self):
pass
class Command2(Command):
def __init__(self, *args):
pass
def execute(self):
pass
This will handle execution of commands. For building, make a command factory.
class CommandFactory():
#staticmethod
def create(command, *args):
if command == 'command1':
return Command1(args)
elif command == 'command2':
return Command2(args)
Then you'd be able to execute a command with one line:
CommandFactory.create(command, args).execute()

Thanks for all of your suggestions!
I think the most elegant approach is using Typer and following this recipe:
https://typer.tiangolo.com/tutorial/subcommands/add-typer/

Argparse: options for subparsers override main if both share parent

I'm using argparse with several subparsers. I want my program to take options for verbosity anywhere in the args, including the subparser.
from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('--verbose', '-v', action='count')
sub = p.add_subparsers()
a = sub.add_parser('a')
print(p.parse_args())
By default, options for the main parser will throw an error if used for subparsers:
$ python tmp.py -v a
Namespace(verbose=1)
$ python tmp.py a -v
usage: tmp.py [-h] [--verbose] {a} ...
tmp.py: error: unrecognized arguments: -v
I looked into parent parsers, from this answer.
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main = ArgumentParser(parents=[parent])
sub = main.add_subparsers()
a = sub.add_parser('a', parents=[parent])
print(main.parse_args())
For some reason though, none of the shared flags work on the main parser.
$ python tmp2.py a -vvv
Namespace(verbose=3)
$ python tmp2.py -vvv a
Namespace(verbose=None)
Note that the main parser definitely has the appropriate arguments, because when I change it to main = ArgumentParser() I get error: unrecognized arguments: -v. What am I missing here?

First, a couple of general comments.
The main parser handles the input upto the subparser invocation, then the subparser is called and given the remaining argv. When it is done, it's namespace is merged back into the the main namespace.
The parents mechanism copies Actions from the parent by reference. So your main and subparsers share the same verbose Action object. That's been a problem when the subparser tries to set a different default or help. It may not be an issue here, but just keep it in mind.
Even without the parents mechanism, sharing a dest or options flag between main and subparser can be tricky. Should the default of the subparser Action be used? What if both are used? Does the subparser overwrite the main parser's actions?
Originally the main namespace was passed to the subparser, which it modified and returned. This was changed a while back (I can find the bug/issue if needed). Now the subparser starts with a default empty namespace, fills it. And these values are then merged into the main.
So in your linked SO question, be wary of older answers. argparse may have changed since then.
I think what's happening in your case is that the main and subparser verbose are counting separately. And when you get None it's the subparser's default that you see.
The __call__ for _Count_Action is:
def __call__(self, parser, namespace, values, option_string=None):
new_count = _ensure_value(namespace, self.dest, 0) + 1
setattr(namespace, self.dest, new_count)
I suspect that in older argparse when the namespace was shared, the count would have been cumulative, but I can't test it without recreating an older style subparser action class.
https://bugs.python.org/issue15327 - here the original developer suggests giving the two arguments different dest. That records the inputs from both main and sub. Your own code can then merge the results if needed.
https://bugs.python.org/issue27859 argparse - subparsers does not retain namespace. Here I suggest a way of recreating the older style.
https://bugs.python.org/issue9351 argparse set_defaults on subcommands should override top level set_defaults - this is the issue in 2014 that changed the namespace use.

My workaround for this behavior, which is very well described in #hpaulj's answer is to create a second parser that does not have subparsers but only the positional arguments that were first found.
The first parse_args, used with the first parser, will validate the positional arguments and flags, show an error message if needed or show the proper help.
The second parse_args, for the second parser, will correctly fill in the namespace.
Building on your example:
from argparse import ArgumentParser
parent = ArgumentParser(add_help=False)
parent.add_argument('--verbose', '-v', action='count')
main1 = ArgumentParser(parents=[parent])
sub = main1.add_subparsers()
# eg: tmp.py -vv a -v
a = sub.add_parser('a', parents=[parent])
a.set_defaults(which='a')
# eg: tmp.py -vv v -v --output toto
b = sub.add_parser('b', parents=[parent])
b.add_argument('--output', type=str)
b.set_defaults(which='b')
args = main1.parse_args()
print(args)
# parse a second time with another parser
main2 = ArgumentParser(parents=[parent])
if args.which == 'a':
main2.add_argument('a')
elif args.which == 'b':
main2.add_argument('b')
main2.add_argument('--output', type=str)
print(main2.parse_args())
Which gives:
$ ./tmp.py -vv a -v
Namespace(verbose=1, which='a')
Namespace(a='a', verbose=3)
$ ./tmp.py -vv b -v --output toto
Namespace(output='toto', verbose=1, which='b')
Namespace(b='b', output='toto', verbose=3)
$ ./tmp.py -vv a --output
usage: tmp.py [-h] [--verbose] {a,b} ...
tmp.py: error: unrecognized arguments: --output
I use this technique with multiple nested subparsers.

gem/git-style command line arguments in Python

Is there a Python module for doing gem/git-style command line arguments? What I mean by gem/git style is:
$ ./MyApp.py
The most commonly used MyApp commands are:
add Add file contents to the index
bisect Find by binary search the change that introduced a bug
branch List, create, or delete branches
checkout Checkout a branch or paths to the working tree
...
$ ./MyApp.py branch
* current-branch
master
With no arguments, the output tells you how you can proceed. And there is a special "help" command:
$ ./MyApp.py help branch
Which gets you deeper tips about the "branch" command.
Edit:
And by doing I mean it does the usage printing for you, exits with invalid input, runs your functions according to your CLI specification. Sort of a "URL mapper" for the command line.

Yes, argparse with add_subparsers().
It's all well explained in the Sub-commands section.
Copying one of the examples from there:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> checkout = subparsers.add_parser('checkout', aliases=['co'])
>>> checkout.add_argument('foo')
>>> parser.parse_args(['checkout', 'bar'])
Namespace(foo='bar')
Edit: Unfortunately there's no self generated special help command, but you can get the verbose help message (that you seem to want) with -h or --help like one normally would after the command:
$ ./MyApp.py branch --help
By verbose I don't mean that is like a man page, it's like every other --help kind of help: listing all the arguments, etc...
Example:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(description='Sub description')
>>> checkout = subparsers.add_parser('checkout', description='Checkout description')
>>> checkout.add_argument('foo', help='This is the foo help')
>>> parser.parse_args(['checkout', '--help'])
usage: checkout [-h] foo
Checkout description
positional arguments:
foo This is the foo help
optional arguments:
-h, --help show this help message and exit
If you need to, it should be easy to implement an help command that redirects to --help.

A reasonable hack to get the gem/git style "help" behavior (I just wrote this for what I'm working on anyway):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='sub_commands')
parser_branch = subparsers.add_parser('branch', description='list of branches')
parser_help = subparsers.add_parser('help')
parser_help.add_argument('command', nargs="?", default=None)
# I can't find a legitimate way to set a default subparser in the docs
# If you know of one, please let me know!
if len(sys.argv) < 2:
sys.argv.append('--help')
parsed = parser.parse_args()
if parsed.sub_commands == "help":
if not parsed.command:
parser.parse_args(['--help'])
else:
parser.parse_args([parsed.command, '--help'])
argparse is definitely a step up from optparse and other python solutions I've come across. But IMO the gem/git style of handling args is just a more logical and safer way to do things so it's annoying that it's not supported.

I wanted to do something similar to git commands, where I would load a second script based off of one of the command line options, and have that script populate more command line options, and also have the help work.
I was able to do this by disabling the help option, parse known args, add more arguments, re-enable the help option, and then parse the rest of the arguments.
This is what I came up with.
import argparse
#Note add_help=False
arg_parser = argparse.ArgumentParser(description='Add more arguments after parsing.',add_help=False)
arg_parser.add_argument('MODE', default='default',type=str, help='What commands to use')
args = arg_parser.parse_known_args()[0]
if args.MODE == 'branch':
arg_parser.add_argument('-d', '--delete', default='Delete a branch')
arg_parser.add_argument('-m', '--move', default='move a branch')
elif args.MODE == 'clone' :
arg_parser.add_argument('--local', '-l')
arg_parser.add_argument('--shared')
#Finally re-enable the help option, and reparse the arguments
arg_parser.add_argument(
'-h', '--help',
action='help', default=argparse.SUPPRESS,
help=argparse._('show this help message and exit'))
args = arg_parser.parse_args()

How to make `help` act the same as `--help` in argparse

I want the help option act the same as --help.
sidenote: I have created a program with the same command line behavior as svn or hg. I managed to do this with subparsers. However I want to make things consistent. That's why I want help to work.

You can do the following to create an alias and make program help act the same as program --help
import argparse
def help(args):
args.parser.print_help()
parser = argparse.ArgumentParser(description='my program')
subparsers = parser.add_subparsers()
p_help = subparsers.add_parser('help')
p_help.set_defaults(func=help, parser=parser)
args = parser.parse_args()
args.func(args)

It seems to me that you want to define help as another subparser. I would naively say that you could then link it to a print_help() function that would copy the output of your standard --help, but I wonder if there is a way to call the native help() function.