Develop Reference

How to create multiple CLI options identified my package name in a python?

I want to build a cli interface for my application which has nested functionality. Example:
├── ...
├── setup.py
└── package-name
├──__init__.py
├──command1.py
└──command2.py
package-name command1 --arg .. ..
package-name command2 --arg ..
AND
python -m package-name command1 --arg ..
The thing to note here is that command1 and command2 are independent modules that accept different command-line args. So linking them together in a __main__.py might also be another challenge.
I came across similar questions that use entry_points in setup.py to create similar cli functionality but its not exactly what I'm looking for. I found this similar question.
How to create a CLI in Python that can be installed with PIP?

If you want access multiple sub-cli's in one entry command, you can implement a sub-command manager at __main__.py, which can parse sub-command from sys.argv and then dispatch to target module.
1️⃣First, i recommend the google fire, which can satify you in most scenarios without extra code.
Here is a example, you can replace the add/multiply function to your sub-command function by using from command1 import xx，and use entry points to expose the main function.
import fire
def add(x, y):
return x + y
def multiply(x, y):
return x * y
def main():
fire.Fire({
'add': add,
'multiply': multiply,
})
if __name__ == '__main__':
main()
We can debug this in the same way as below:
$ python example.py add 10 20
30
$ python example.py multiply 10 20
200
2️⃣Second, if you need implements by your self for some purpose, such as using argparse to define options for each command. A typical practice is the Django command, the official demo: Writing custom django-admin commands
The core steps is :
define a BaseCommand
implements BaseCommand in sub commands.py, and named it as Command.
__main__.py implements find command and call
# base_command.py
class BaseCommand(object):
def create_parser(self, prog_name, subcommand, **kwargs):
"""
Create and return the ``ArgumentParser`` which will be used to
parse the arguments to this command.
"""
# TODO: create your ArgumentParser
return CommandParser(**kwargs)
def add_arguments(self, parser):
"""
Entry point for subclassed commands to add custom arguments.
"""
pass
def run_from_argv(self, argv):
"""
Entry point for commands to be run from the command line.
"""
parser = self.create_parser(argv[0], argv[1])
options = parser.parse_args(argv[2:])
cmd_options = vars(options)
args = cmd_options.pop('args', ())
self.handle(*args, **cmd_options)
def handle(self, *args, **options):
"""
The actual logic of the command. Subclasses must implement
this method.
"""
raise NotImplementedError('subclasses of BaseCommand must provide a handle() method')
# command1.py
class Command(BaseCommand):
def handle(self, *args, **options):
print("Hello, it is command1!")
# command2.py
class Command(BaseCommand):
def handle(self, *args, **options):
print("Hello, it is command2!")
# __main__.py
def main():
# sub command name is the second argument
command_name = sys.argv[1]
# build sub command module name, and import
# package-name is the module name you mentioned
cmd_package_name = f'package-name.{command_name}'
instance = importlib.import_module(cmd_package_name)
# create instance of sub command, the Command must exist
command = instance.Command()
command.run_from_argv(sys.argv)
if __name__ == "__main__":
main()

How can command list display be categorised within a Click chained group?

I'm starting a CLI pipe-type application project which will eventually have a rather large collection of commands (which will be further extensible with plug-in). As a result, I would like to categorise them in the --help text:
Here is how it looks now:
Usage: my_pipe [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Options:
--help Show this message and exit.
Commands:
another_filter help about that filter
another_generator help about that generator
another_sink help about that sink
some_filter help about this filter
some_generator help about this generator
some_sink help about this sink
This is more or less how I would like it to look:
Usage: my_pipe [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
Options:
--help Show this message and exit.
Commands:
Generators:
some_generator help about this generator
another_generator help about that generator
Filters:
some_filter help about this filter
another_filter help about that filter
Sinks:
some_sink help about this sink
another_sink help about that sink
How can this be achieved? Note that apart from the look of --help, I'm happy with the flat logical command organisation. Also, sub-groups are not an option as they are not allowed inside a chain=True group.

If you inherit from click.Group you can add a bit of code to group the commands and then show those groups in the help.
Custom Class
class GroupedGroup(click.Group):
def command(self, *args, **kwargs):
"""Gather the command help groups"""
help_group = kwargs.pop('group', None)
decorator = super(GroupedGroup, self).command(*args, **kwargs)
def wrapper(f):
cmd = decorator(f)
cmd.help_group = help_group
return cmd
return wrapper
def format_commands(self, ctx, formatter):
# Modified fom the base class method
commands = []
for subcommand in self.list_commands(ctx):
cmd = self.get_command(ctx, subcommand)
if not (cmd is None or cmd.hidden):
commands.append((subcommand, cmd))
if commands:
longest = max(len(cmd[0]) for cmd in commands)
# allow for 3 times the default spacing
limit = formatter.width - 6 - longest
groups = {}
for subcommand, cmd in commands:
help_str = cmd.get_short_help_str(limit)
subcommand += ' ' * (longest - len(subcommand))
groups.setdefault(
cmd.help_group, []).append((subcommand, help_str))
with formatter.section('Commands'):
for group_name, rows in groups.items():
with formatter.section(group_name):
formatter.write_dl(rows)
Using the Custom Class
To make use of the custom class, use the cls parameter to pass the class to the click.group() decorator.
#click.group(cls=GroupedGroup)
def cli():
"""My awesome cli"""
Then for each command mark the help group for the command to be included in like:
#cli.command(group='A Help Group')
def command():
"""This is a command"""
How does this work?
This works because click is a well designed OO framework. The #click.group() decorator usually instantiates a click.Group object but allows this behavior to be overridden with the cls parameter. So it is a relatively easy matter to inherit from click.Group in our own class and override desired methods.
In this case we override the click.Group.command() decorator to gather up the desired help group for each command. Then we override the click.Group.format_commands() method to use those groups when constructing the help.
Test Code
import click
#click.group(cls=GroupedGroup)
def cli():
"""My awesome cli"""
#cli.command(group='Generators')
def some_generator():
"""This is Some Generator"""
#cli.command(group='Generators')
def another_generator():
"""This is Another Generator"""
#cli.command(group='Filters')
def some_filter():
"""This is Some Filter"""
#cli.command(group='Filters')
def another_filter():
"""This is Another Filter"""
cli()
Results
Usage: test.py [OPTIONS] COMMAND [ARGS]...
My awesome cli
Options:
--help Show this message and exit.
Commands:
Filters:
another-filter This is Another Filter
some-filter This is Some Filter
Generators:
another-generator This is Another Generator
some-generator This is Some Generator

Feature flag for Python click commands

In my python-click-CLI script, I'm building some commands for features that should not be visible for users (to not confuse them), but visible for e.g. developers.
Is it possible to use feature flags for Python-click commands?
I would like to be able to configure (via a config file, etc) if a command is available or not. If a command-feature is disabled, the command should not be callable and the help should not show it.
Like this:
FLAG_ENABLED = False
# This command should not be shown and not be callable as long as the flag is disabled
#cli.command(name='specialfeature', active=FLAG_ENABLED)
def special_feature_command()
....
Obviously, I could change the body of my function:
#cli.command(name='specialfeature', active=FLAG_ENABLED)
def special_feature_command()
if FLAG_ENABLED:
...
else:
...
But then my command would still show up in help, which I would like to avoid.

You can add a feature flag functionality with a custom class like:
Custom Class
This class over rides the click.Group.command() method which is used to decorate command functions. It adds the ability to pass an active flag, which when False will skip adding the command to the group.
import click
class FeatureFlagCommand(click.Group):
def command(self, *args, active=True, **kwargs):
"""Behaves the same as `click.Group.command()` except added an
`active` flag which can be used to disable to command.
"""
if active:
return super(FeatureFlagCommand, self).command(*args, **kwargs)
else:
return lambda f: f
Using the Custom Class
By passing the cls parameter to the click.group() decorator, any commands added to the group via the the group.command() will be gated with the active flag.
#click.group(cls=FeatureFlagCommand)
def cli():
...
#cli.command(name='specialfeature', active=FLAG_ENABLED)
def special_feature_command()
...
How does this work?
This works because click is a well designed OO framework. It is easy to inherit from click.Group and build a new command() decorator. In the new command() decorator if the active flag is False we return the undecorated function instead of adding the function to the group.
Test Code:
#click.group(cls=FeatureFlagCommand)
def cli():
"""My Awesome Click Program"""
#cli.command(active=False)
def command1():
click.echo('Command 1')
#cli.command(active=True)
def command2():
click.echo('Command 2')
#cli.command()
def command3():
click.echo('Command 3')
if __name__ == "__main__":
commands = (
'command1',
'command2',
'command3',
'--help',
'',
)
import sys, time
time.sleep(1)
print('Click Version: {}'.format(click.__version__))
print('Python Version: {}'.format(sys.version))
for cmd in commands:
try:
time.sleep(0.1)
print('-----------')
print('> ' + cmd)
time.sleep(0.1)
cli(cmd.split())
except BaseException as exc:
if str(exc) != '0' and \
not isinstance(exc, (click.ClickException, SystemExit)):
raise
Results:
Click Version: 6.7
Python Version: 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
-----------
> command1
Usage: test.py [OPTIONS] COMMAND [ARGS]...
Error: No such command "command1".
-----------
> command2
Command 2
-----------
> command3
Command 3
-----------
> --help
Usage: test.py [OPTIONS] COMMAND [ARGS]...
My Awesome Click Program
Options:
--help Show this message and exit.
Commands:
command2
command3
-----------
>
Usage: test.py [OPTIONS] COMMAND [ARGS]...
My Awesome Click Program
Options:
--help Show this message and exit.
Commands:
command2
command3

Creating a shell command line application with Python and Click

I'm using click (http://click.pocoo.org/3/) to create a command line application, but I don't know how to create a shell for this application.
Suppose I'm writing a program called test and I have commands called subtest1 and subtest2
I was able to make it work from terminal like:
$ test subtest1
$ test subtest2
But what I was thinking about is a shell, so I could do:
$ test
>> subtest1
>> subtest2
Is this possible with click?

This is not impossible with click, but there's no built-in support for that either. The first you would have to do is making your group callback invokable without a subcommand by passing invoke_without_command=True into the group decorator (as described here). Then your group callback would have to implement a REPL. Python has the cmd framework for doing this in the standard library. Making the click subcommands available there involves overriding cmd.Cmd.default, like in the code snippet below. Getting all the details right, like help, should be doable in a few lines.
import click
import cmd
class REPL(cmd.Cmd):
def __init__(self, ctx):
cmd.Cmd.__init__(self)
self.ctx = ctx
def default(self, line):
subcommand = cli.commands.get(line)
if subcommand:
self.ctx.invoke(subcommand)
else:
return cmd.Cmd.default(self, line)
#click.group(invoke_without_command=True)
#click.pass_context
def cli(ctx):
if ctx.invoked_subcommand is None:
repl = REPL(ctx)
repl.cmdloop()
#cli.command()
def a():
"""The `a` command prints an 'a'."""
print "a"
#cli.command()
def b():
"""The `b` command prints a 'b'."""
print "b"
if __name__ == "__main__":
cli()

There is now a library called click_repl that does most of the work for you. Thought I'd share my efforts in getting this to work.
The one difficulty is that you have to make a specific command the repl command, but we can repurpose #fpbhb's approach to allow calling that command by default if another one isn't provided.
This is a fully working example that supports all click options, with command history, as well as being able to call commands directly without entering the REPL:
import click
import click_repl
import os
from prompt_toolkit.history import FileHistory
#click.group(invoke_without_command=True)
#click.pass_context
def cli(ctx):
"""Pleasantries CLI"""
if ctx.invoked_subcommand is None:
ctx.invoke(repl)
#cli.command()
#click.option('--name', default='world')
def hello(name):
"""Say hello"""
click.echo('Hello, {}!'.format(name))
#cli.command()
#click.option('--name', default='moon')
def goodnight(name):
"""Say goodnight"""
click.echo('Goodnight, {}.'.format(name))
#cli.command()
def repl():
"""Start an interactive session"""
prompt_kwargs = {
'history': FileHistory(os.path.expanduser('~/.repl_history'))
}
click_repl.repl(click.get_current_context(), prompt_kwargs=prompt_kwargs)
if __name__ == '__main__':
cli(obj={})
Here's what it looks like to use the REPL:
$ python pleasantries.py
> hello
Hello, world!
> goodnight --name fpbhb
Goodnight, fpbhb.
And to use the command line subcommands directly:
$ python pleasntries.py goodnight
Goodnight, moon.

I know this is super old, but I've been working on fpbhb's solution to support options as well. I'm sure this could use some more work, but here is a basic example of how it could be done:
import click
import cmd
import sys
from click import BaseCommand, UsageError
class REPL(cmd.Cmd):
def __init__(self, ctx):
cmd.Cmd.__init__(self)
self.ctx = ctx
def default(self, line):
subcommand = line.split()[0]
args = line.split()[1:]
subcommand = cli.commands.get(subcommand)
if subcommand:
try:
subcommand.parse_args(self.ctx, args)
self.ctx.forward(subcommand)
except UsageError as e:
print(e.format_message())
else:
return cmd.Cmd.default(self, line)
#click.group(invoke_without_command=True)
#click.pass_context
def cli(ctx):
if ctx.invoked_subcommand is None:
repl = REPL(ctx)
repl.cmdloop()
#cli.command()
#click.option('--foo', required=True)
def a(foo):
print("a")
print(foo)
return 'banana'
#cli.command()
#click.option('--foo', required=True)
def b(foo):
print("b")
print(foo)
if __name__ == "__main__":
cli()

I was trying to do something similar to the OP, but with additional options / nested sub-sub-commands. The first answer using the builtin cmd module did not work in my case; maybe with some more fiddling.. But I did just run across click-shell. Haven't had a chance to test it extensively, but so far, it seems to work exactly as expected.

how to pass arguments to a python fabric custom task

I cannot figure out how to actually pass arguments to a fabric custom task.
I have a bunch of tasks that all need to do the same setup, so I was hoping to subclass the task and have the base class do the setup and then run the specific subtasks. Both the setup code and the subtasks need access to some arguments that are passed in from the command-line to the task. I also need to be able to set default values for the arguments.
Original Attempt
My original attempt shows what I am trying to do without any sub classes.
This code works correctly.
The code below is in file tmp1.py:
from fabric.api import task
def do_setup(myarg):
''' common setup for all tasks '''
print "in do_setup(myarg=%s)" % myarg
# do setup using myarg for something important
#task
def actual_task1(myarg='default_value', alias='at'):
print "In actual_task1(myarg=%s)" % myarg
do_setup(myarg)
# do rest of work ...
#task
def actual_task2(myarg='default_value', alias='at'):
print "In actual_task2(myarg=%s)" % myarg
do_setup(myarg)
# do rest of work ...
I run it from the command-line without any args and correctly see the default for myarg of 'default_value'
fab -f ./tmp1.py actual_task1
Prints:
In actual_task1(myarg=default_value)
in do_setup(myarg=default_value)
Done.
Then I call it with myarg='hello' and see that 'hello' gets passed through correctly
fab -f ./tmp1.py actual_task1:myarg='hello'
It outputs:
In actual_task1(myarg=hello)
in do_setup(myarg=hello)
Done.
Attempt with a custom task
My next attempt is to make a common task to encapsulate the setup part.
This is copied from http://docs.fabfile.org/en/1.5/usage/tasks.html
The code below is in the file tmp2.py:
from fabric.api import task
from fabric.tasks import Task
def do_setup(myarg):
''' common setup for all tasks '''
print "in do_setup(myarg=%s)" % myarg
# do setup using myarg for something important
'''
Attempt to make a common task to encapsulate the setup part
copied from http://docs.fabfile.org/en/1.5/usage/tasks.html
'''
class CustomTask(Task):
def init(self, func, myarg, args, *kwargs):
super(CustomTask, self).init(args, *kwargs)
print("=> init(myarg=%s, args=%s, kwargs=%s" % (myarg, args, kwargs))
self.func = func
self.myarg = myarg
print "in init: self.func=",self.func,"self.myarg=",self.myarg
def run(self, *args, **kwargs):
return self.func(self.myarg, *args, **kwargs)
#task(task_class=CustomTask, myarg='default_value', alias='at')
def actual_task1():
print "In actual_task1(myarg=%s)" % myarg
# do rest of work ...
When run, there are 2 problems:
__init__ gets "default_value" instead of "Hello"
It complains that actual_task1() expects 0 arguments
I run it this way:
fab -f ./tmp2.py actual_task1:myarg="Hello"
Prints:
=> init(myarg=default_value, args=(), kwargs={'alias': 'at'}
in init: self.func= self.myarg= default_value
Traceback (most recent call last):
File "/home/xxx/Documents/pyenvs/xxx/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main args, *kwargs
File "/home/xxx/Documents/pyenvs/xxx/local/lib/python2.7/site-packages/fabric/tasks.py", line 405, in execute results[''] = task.run(args, *new_kwargs)
File "/home/xxx/test_fab/tmp2.py", line 21, in run
return self.func(self.myarg, args, *kwargs)
TypeError: actual_task1() takes no arguments (1 given)
I spent quite a bit of time trying to make this work but I cannot seem to solve the default_value issue. I must be missing something?
I would appreciate some help figuring out how to make this sample program run. The second version with the custom task needs to behave just like the original version I showed.
Thank you for any help with this issue.

Fixed example with setup:
from fabric.api import task
from fabric.tasks import Task
def do_setup(foo, verbose):
''' common setup for all tasks '''
print "IN do_setup(foo=%s, verbose=%s)" % (foo, verbose)
# do setup using foo and verbose...
class CustomTask(Task):
def __init__(self, func, *args, **kwargs):
'''
The special args like hosts and roles do not show up in
args, and kwargs, they are stripped already.
args and kwargs may contain task specific special arguments
(e.g. aliases, alias, default, and name) to customize the
task. They are set in the #task decorator and cannot be passed
on the command-line. Note also that these special task
arguments are not passed to the run method.
Non-special arguments (there are none in this example) are
set in the task decorator. These other arguments are not
passed to the run method and cannot be overridden from the
command-line.
Note that if you pass any "task specific special arguments" or
"non-special arguments declared in the task decorator" from the
command-line, they are treated as different arguments and the
command-line values are passed to the run method but not to
this method.
'''
super(CustomTask, self).__init__(*args, **kwargs)
print "IN __init__(args=%s, kwargs=%s)" % (args, kwargs)
self.func = func
def run(self, foo='foo_default_val', verbose='verbose_default_val',
*args, **kwargs):
'''
The arguments to this method will be:
1) arguments from the actual task (e.g. foo and verbose). This method
is where you set a default value for the arguments from the
actual_task, not on the actual_task.
2) task specific arguments from the command-line
(e.g. actual_task:bar='xxx'). This example is not expecting any,
so it strips them and does not pass them to the
actual_function one (e.g. it calls self.func with only foo
and verbose and does not pass args and kwargs)
'''
print "IN run(foo=%s, verbose=%s, args=%s, kwargs=%s)" % \
(foo, verbose, args, kwargs)
do_setup(foo, verbose)
return self.func(foo, verbose)
#task(task_class=CustomTask, alias="RUNME")
def actual_task(foo, verbose):
print 'IN task actual_task(foo=%s, verbose=%s)' % (foo, verbose)
Run with only host specified on the command-line:
fab -f ./example_with_setup.py actual_task:host='hhh'
IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=foo_default_val, verbose=verbose_default_val, args=(), kwargs={})
IN do_setup(foo=foo_default_val, verbose=verbose_default_val)
IN task actual_task(foo=foo_default_val, verbose=verbose_default_val)
Run specifying foo on the commandline:
fab -f ./example_with_setup.py actual_task:host='hhh',foo='bar'
IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=bar, verbose=verbose_default_val, args=(), kwargs={})
IN do_setup(foo=bar, verbose=verbose_default_val)
IN task actual_task(foo=bar, verbose=verbose_default_val)
Run specifying both foo and verbose on the command-line:
fab -f ./example_with_setup.py actual_task:host='hhh',foo='bar',verbose=True
IN __init__(args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(foo=bar, verbose=True, args=(), kwargs={})
IN do_setup(foo=bar, verbose=True)
IN task actual_task(foo=bar, verbose=True)

In the custom class section, the function actual_task1 doesn't actually take arguments, so the only valid way to invoke your fabric file is:
fab -f ./tmp2.py actual_task1
Furthermore, I don't think you're actually calling do_setup in either CustomTask or actual_task1

This is the fixed example.
# fixed the example from http://docs.fabfile.org/en/1.8/usage/tasks.html
from fabric.api import task
from fabric.tasks import Task
class CustomTask(Task):
def __init__(self, func, myarg1, *args, **kwargs):
'''
The special args like hosts and roles do not show up in
args, and kwargs, they are stripped already.
args and kwargs may contain task specific special arguments
(e.g. aliases, alias, default, and name) to customize the
task. They are set in the #task decorator and cannot be passed
on the command-line. Note also that these special task
arguments are not passed to the run method.
Non-special arguments (in this example myarg1) are set in the task
decorator. These other arguments are not passed to the run
method and cannot be overridden from the command-line.
Note that if you pass any "task specific special arguments" or
"non-special arguments declared in the task decorator" from the
command-line, they are treated as different arguments and the
command-line values are passed to the run method but not to
this method.
'''
super(CustomTask, self).__init__(*args, **kwargs)
print "IN __init__(myarg1=%s, args=%s, kwargs=%s)" % \
(myarg1, args, kwargs)
self.func = func
self.myarg1 = myarg1
def run(self, myarg2='default_value2', *args, **kwargs):
'''
The arguments to this method will be:
1) arguments from the actual task (e.g. myarg2). This method
is where you set a default value for the arguments from the
actual_task, not on the actual_task.
2) task specific arguments from the command-line
(e.g. actual_host:foo='foo'). This example is not expecting
any, so it strips them and does not pass them to the
actual_function (e.g. it calls self.func with only myarg2 and
does not pass args and kwargs)
'''
print "IN run(myarg2=%s, args=%s, kwargs=%s)" % \
(myarg2, args, kwargs)
return self.func(myarg2)
#task(task_class=CustomTask, myarg1='special_value', alias='RUNME')
def actual_task(myarg2):
print "IN actual_task(myarg2=%s)" % myarg2
Run with only hosts specified on the command-line:
fab -f ./fixed_example actual_task:hosts="hhh"
IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=default_value2, args=(), kwargs={})
IN actual_task(myarg2=default_value2)
Run specifying myarg2 on the command-line:
fab -f ./fixed_example actual_task:hosts="hhh",myarg2="good_value"
IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=good_value, args=(), kwargs={})
IN actual_task(myarg2=good_value)
Bad run specifying myarg1 and alias on the command-line. Notice that init gets the values specified in the task decorator and not the values from the command-line. Notice that run gets myarg1 and alias as arguments now.
fab -f ./fixed_example actual_task:hosts="hhh",myarg1="myarg1_from_commandline",alias="alias_from_commandline"
IN __init__(myarg1=special_value, args=(), kwargs={'alias': 'RUNME'})
[hhh] Executing task 'actual_task'
IN run(myarg2=default_value2, args=(), kwargs={'alias': 'alias_from_commandline', 'myarg1': 'myarg1_from_commandline'})
IN actual_task(myarg2=default_value2)