Develop Reference

Argparse, displaying custom help text without any of the boilerplate argparse text

After looking at about a dozen questions, I can't seem to find an answer.
I have a python CLI i've written using argparse. I have a main command that does nothing but regurgitate help text and then 4 subcommands. My boss wants a very specific output for the help text. He has me write it out as a text file and then we use that text file to display the help text.
However, in some circumstances, it STILL outputs parts of the argparse help text.
For example, if I run my program with no subcommands, it just outputs our help text from the file. But if I use "-h" or "--help" it will output our help text, followed by the list of positional and optional arguments and other argparse stuff. We don't want that.
I could use "add_help=False" but we want the user to be able to type -h and still get our help text. If we set add help to false, it will display our help text followed by the error "-h not recognized".
Also doing this does nothing for when the user uses -h after a subcommand. I set help=None and usage is set to my custom help text for each subcommand, but it still shows the boilerplate argparse info at the end.
This is what I want to happen: user types in the main command with no subcommands prints my custom help text and nothing else. The user types the main command, no subcommand, followed by -h/--help and it prints my custom help text and nothing else. User types in the main command, one of the subcommands, followed by -h/--help and it outputs my help text and nothing else. User types the main command, a subcommand, and then wrong arguments or too many/ too few arguments displays my help text and nothing else. Basically I only ever want it to print nothing, or print just my help text.
how do I do that? here is my main function where the parsers and subparsers are all configured:
def main():
# Import help text from file
p = Path(__file__).with_name("help.txt")
with p.open() as file:
help_text = file.read()
# Configure the top level Parser
parser = argparse.ArgumentParser(prog='myprog', description='My program', usage=help_text)
subparsers = parser.add_subparsers()
# Create Subparsers to give subcommands
parser_one = subparsers.add_parser('subcommandone', prog='subcommandone', usage=help_text, help=None)
parser_one.add_argument('arg1', type=str)
parser_one.add_argument('-o', '--option1', default='mydefault', type=str)
parser_two= subparsers.add_parser('subcommandtwo', usage=help_text, help=None, prog='subcommandtwo')
parser_three= subparsers.add_parser('subcommandthree', usage=help_text, help=None, prog='subcommandthree')
parser_four= subparsers.add_parser('subcommandfour', usage=help_text, help=None, prog='subcommandfour')
# Assign subparsers to their respective functions
parser_one.set_defaults(func=functionone)
parser_two.set_defaults(func=functiontwo)
parser_three.set_defaults(func=functionthree)
parser_four.set_defaults(func=functionfour)
parser.set_defaults(func=base_case)
# Parse the arguments and call appropriate functions
args = parser.parse_args()
if len(sys.argv) == 1:
args.func(args, parser)
else:
args.func(args)
Any thoughts?

You can use sys.exit() after the help text has been displayed, and before the parsing has begun, to avoid problems with "-h not recognized".
So anywhere before the line
# Parse the arguments and call appropriate functions
add
if len(sys.argv) == 1 or '-h' in sys.argv or '--help' in sys.argv:
print(help_text)
sys.exit(1)
In situations where that is not good enough you can subclass argparse.HelpFormatter like so
usage_help_str = 'myscript command [options]'
epilog_str = "More info can be found at https://..."
class Formatter(argparse.HelpFormatter):
# override methods and stuff
def formatter(prog):
return Formatter(prog)
parser = argparse.ArgumentParser(formatter_class=formatter, epilog=epilog_str, usage=usage_help_str, add_help=False)
I tried looking around for documentation on subclassing the helpFormatter, but I couldn't find anything. It looks like people are just looking at the source code to figure out how to subclass it.

Argparse and mutually exclusive command line arguments

I have created a pastebin terminal client in python. It can take some command line arguments, like -o to open a file, -n to set a paste name etc. It also has option -l which lists the pastes and allows you to delete or view pastes. The problem is that I don't know how to do it in a nice way (using argparse) - it should not allow to use -l with any other options.
I added a simple logic:
if args.name:
if args.list:
print('The -l should be used alone. Check "pb -h" for help.')
sys.exit()
Can it be done using just argparse?
I know about mutually exclusive groups, I even have one (to set paste privacy) but I haven't figured this one yet.
Full code is available here: https://github.com/lkorba/pbstc/blob/master/pb

I don't think you can use argparse to achieve your goal in "a nice way" as you say.
I see 2 options here:
1) The simpler solution as I get it would be to just check your arguments after parsing them. Nothing fancy just:
args = parser.parse_args()
if args.list is not None:
if not (args.name is None and args.open is None and
args.public is None and args.format is None and args.exp is None):
parser.error('Cannot use list with name, open, public, format or exp argument')
2) On the other hand you could revise a bit your program and use
subparsers like:
subparsers = parser.add_subparsers(title="commands", dest="command")
parser_a = subparsers.add_parser('list', help='list help')
parser_b = subparsers.add_parser('action', help='Any action here')
parser_b.add_argument('-f', action="store", help='Choose paste format/syntax: text=None, '
'mysql=MYSQL, perl=Perl, python=Python etc...')
parser_b.add_argument('-n', '--name', action="store")
parser_b.add_argument('-o', '--open', action="store", help='Open file')
...
args = parser.parse_args()
if args.command == 'list':
...
elif args.command == 'action':
...
So, for example if you pass list -n='Name' as arguments in the latter case you will get an error:
usage: subparser_example.py [-h] {list,action} ...
subparser_example.py: error: unrecognized arguments: -n='Name'
Of course you also get (as overhead) one extra parameter action here...

argparse mutually_exclusive_group with sub groups

I am trying to make a script with usage as follows:
my_script [-p parg -l larg] | [-s sarg]
i.e the script either takes -p and -l argument OR -s argument. It is an error if both -p and -s are specified. I tried the following but doesn't seem to work
import argparse
parser = argparse.ArgumentParser(description='Some Desc')
gp = parser.add_mutually_exclusive_group()
num_gp = gp.add_argument_group()
num_gp.add_argument('-p')
num_gp.add_argument('-l')
gp.add_argument('-s')
In [18]: parser.parse_args(['-p blahp', '-l blahl', '-s blahs'])
Out[18]: Namespace(l=' blahl', p=' blahp', s=' blahs') #ERROR Should have failed as I specify both `-p` and `-s` which belong to a mutually_exclusive_group

I think you're misusing the mutually exclusive groups. Groups are not mutually exclusive with other groups, members of a group are mutually exclusive with each other. Also, as far as I can tell from the docs, standard groups do not affect parsing logic, they only help to organize the help message generated by the parser.
Here is how you could make two options, -p and -s for example, mutually exclusive:
import argparse
parser = argparse.ArgumentParser(description='Some Desc')
group1 = parser.add_mutually_exclusive_group()
group1.add_argument("-p")
group1.add_argument("-s")
# This works
args = parser.parse_args(["-p", "argForP"])
# This will throw an error
args = parser.parse_args(["-p", "argForP", "-s", "argForS"])
But I'm not sure if this functionality will allow you to implement your use case because I'm not sure that an argument can belong to two mutually exclusive groups. However you can always do error checking yourself, and use parser.error. That would look something like this:
message = "invalid options"
# Tells us p is used without l or vise versa
if (args.p is None) != (args.l is None):
parser.error(message)
# Tells is if p-l is used with s
if (args.p is None) == (args.s is None):
# Either both are used or neigher
parser.error(message)

Detecting if any command-line options were specified more than once with optparse or argparse

Python optparse normally allows the user to specify an option more than once and silently ignores all occurrences of the option but the last one. For example, if the action of option --foo is store and the action of option --flag is store_const, store_true or store_false, the following commands will be equivalent:
my-command --foo=bar --foo=another --flag --foo=last --flag
my-command --flag --foo=last
(Update: argparse does just the same thing by default.)
Now, I have a lot of options, and specifying any of them more than once doesn't make sense. If a user specifies the same option more than once I'd like to warn them about the possible error.
What is the most elegant way to detect options that were specified multiple times? Note that the same option can have a short form, a long form and abbreviated long forms (so that -f, --foobar, --foob and --foo are all the same option). It would be even better if it was possible to detect the case when multiple options that have the same destination were specified simultaneously, so that a warning can be given if a user specifies both --quiet and --verbose while both options store a value into the same destination and effectively override each other.
Update: To be more user-friendly, the warning should refer to the exact option names as used on the command line. Using append actions instead of store is possible, but when we detect a conflict, we cannot say which options caused it (was it -q and --verbose or --quiet --quiet?).
Unfortunately I'm stuck with optparse and cannot use argparse because I have to support Python 2.6.
P. S. If you know of a solution that works only with argparse, please post it, too. While I try to minimize the number of external dependencies, using argparse under Python 2.6 is still an option.

I think the correct way would be to "define your action" in some way.
For example, you could use the action callback and implement a function that implement your desired behaviour.
You could write a function that first checks if the destination was already filled, if it is filled then it stores the overlapping options into a list.
When the parsing is finished you should check if these lists are empty, and if they are not raise the appropriate exception.
Another way of doing this could be to define your own action. You can have a look here
A small example that uses the callback:
import sys
import functools
from optparse import OptionParser
bad_option = 'BAD OPTION'
def store(option, opt, value, parser, dest, val):
"""Set option's destination *dest* to *val* if there are no conflicting options."""
list_name = dest + '_options_list'
try:
# if this option is a conflict, save its name and set the value to bad_option
getattr(parser.values, list_name).append(opt)
setattr(parser.values, dest, bad_option)
except AttributeError:
# no conflicts, set the option value and add the options list
setattr(parser.values, dest, val)
setattr(parser.values, list_name, [opt])
store_true = functools.partial(store, val=True)
store_false = functools.partial(store, val=False)
parser = OptionParser()
parser.add_option('-v', '--verbose',
action='callback', callback=store_true,
help='Increase output verbosity',
callback_kwargs={'dest': 'verbose'})
parser.add_option('-q', '--quiet',
action='callback', callback=store_false,
help='Decrease output verbosity',
callback_kwargs={'dest': 'verbose'})
opts, args = parser.parse_args()
# detects all conflicting options for all destinations
found = False
for dest in ('verbose',):
if getattr(opts, dest) == bad_option:
conflicting_opts = ', '.join(getattr(opts, dest + '_options_list'))
print('Conflicting options %s for destination %s'
% (conflicting_opts, dest))
found = True
if found:
parser.print_usage()
sys.exit(2)
And the output:
$ python testing_optparse.py -v -q
Conflicting options -v, -q for destination verbose
Usage: prova_optparse.py [options]
Probably it would be better to raise an OptionValueError when detecting conflicts, even though this would allow to get only couple of conflicting options. If you want to get all conflicting options you have to parse the remaining arguments( in parser.rargs).

You can use action="append" (optparse) and then check the number of appended elements. See http://docs.python.org/library/optparse.html#other-actions

gem/git-style command line arguments in Python

Is there a Python module for doing gem/git-style command line arguments? What I mean by gem/git style is:
$ ./MyApp.py
The most commonly used MyApp commands are:
add Add file contents to the index
bisect Find by binary search the change that introduced a bug
branch List, create, or delete branches
checkout Checkout a branch or paths to the working tree
...
$ ./MyApp.py branch
* current-branch
master
With no arguments, the output tells you how you can proceed. And there is a special "help" command:
$ ./MyApp.py help branch
Which gets you deeper tips about the "branch" command.
Edit:
And by doing I mean it does the usage printing for you, exits with invalid input, runs your functions according to your CLI specification. Sort of a "URL mapper" for the command line.

Yes, argparse with add_subparsers().
It's all well explained in the Sub-commands section.
Copying one of the examples from there:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers()
>>> checkout = subparsers.add_parser('checkout', aliases=['co'])
>>> checkout.add_argument('foo')
>>> parser.parse_args(['checkout', 'bar'])
Namespace(foo='bar')
Edit: Unfortunately there's no self generated special help command, but you can get the verbose help message (that you seem to want) with -h or --help like one normally would after the command:
$ ./MyApp.py branch --help
By verbose I don't mean that is like a man page, it's like every other --help kind of help: listing all the arguments, etc...
Example:
>>> parser = argparse.ArgumentParser()
>>> subparsers = parser.add_subparsers(description='Sub description')
>>> checkout = subparsers.add_parser('checkout', description='Checkout description')
>>> checkout.add_argument('foo', help='This is the foo help')
>>> parser.parse_args(['checkout', '--help'])
usage: checkout [-h] foo
Checkout description
positional arguments:
foo This is the foo help
optional arguments:
-h, --help show this help message and exit
If you need to, it should be easy to implement an help command that redirects to --help.

A reasonable hack to get the gem/git style "help" behavior (I just wrote this for what I'm working on anyway):
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='sub_commands')
parser_branch = subparsers.add_parser('branch', description='list of branches')
parser_help = subparsers.add_parser('help')
parser_help.add_argument('command', nargs="?", default=None)
# I can't find a legitimate way to set a default subparser in the docs
# If you know of one, please let me know!
if len(sys.argv) < 2:
sys.argv.append('--help')
parsed = parser.parse_args()
if parsed.sub_commands == "help":
if not parsed.command:
parser.parse_args(['--help'])
else:
parser.parse_args([parsed.command, '--help'])
argparse is definitely a step up from optparse and other python solutions I've come across. But IMO the gem/git style of handling args is just a more logical and safer way to do things so it's annoying that it's not supported.

I wanted to do something similar to git commands, where I would load a second script based off of one of the command line options, and have that script populate more command line options, and also have the help work.
I was able to do this by disabling the help option, parse known args, add more arguments, re-enable the help option, and then parse the rest of the arguments.
This is what I came up with.
import argparse
#Note add_help=False
arg_parser = argparse.ArgumentParser(description='Add more arguments after parsing.',add_help=False)
arg_parser.add_argument('MODE', default='default',type=str, help='What commands to use')
args = arg_parser.parse_known_args()[0]
if args.MODE == 'branch':
arg_parser.add_argument('-d', '--delete', default='Delete a branch')
arg_parser.add_argument('-m', '--move', default='move a branch')
elif args.MODE == 'clone' :
arg_parser.add_argument('--local', '-l')
arg_parser.add_argument('--shared')
#Finally re-enable the help option, and reparse the arguments
arg_parser.add_argument(
'-h', '--help',
action='help', default=argparse.SUPPRESS,
help=argparse._('show this help message and exit'))
args = arg_parser.parse_args()