Click is a popular Python library for developing CLI applications with. Sphinx is a popular library for documenting Python packages with. One problem that some have faced is integrating these two tools so that they can generate Sphinx documentation for their click-based commands.
I ran into this problem recently. I decorated some of my functions with click.command and click.group, added docstrings to them and then generated HTML documentation for them using Sphinx's autodoc extension. What I found is that it omitted all documentation and argument descriptions for these commands because they had been converted into Command objects by the time autodoc got to them.
How can I modify my code to make the documentation for my commands available to both the end user when they run --help on the CLI, and also to people browsing the Sphinx-generated documentation?
You can use a sphinx extension sphinx-click for this now. It can generate docs for nested commands with options and arguments description. The output will be like when you run --help.
Usage
Install the extension
pip install sphinx-click
Enable the plugin in your Sphinx conf.py file:
extensions = ['sphinx_click.ext']
Use plugin wherever necessary in the documentation
.. click:: module:parser
:prog: hello-world
:show-nested:
Example
There is simple click application, which is defined in the hello_world module:
import click
#click.group()
def greet():
"""A sample command group."""
pass
#greet.command()
#click.argument('user', envvar='USER')
def hello(user):
"""Greet a user."""
click.echo('Hello %s' % user)
#greet.command()
def world():
"""Greet the world."""
click.echo('Hello world!')
For documenting all subcommands we will use code below with the :show-nested: option
.. click:: hello_world:greet
:prog: hello-world
:show-nested:
Before building docs make sure that your module and any additional dependencies are available in sys.path either by installing package with setuptools or by manually including it.
After building we will get this:
generated docs
More detailed information on various options available is provided in documentation of the extension
Decorating command containers
One possible solution to this problem that I've recently discovered and seems to work would be to start off defining a decorator that can be applied to classes. The idea is that the programmer would define commands as private members of a class, and the decorator creates a public function member of the class that's based on the command's callback. For example, a class Foo containing a command _bar would gain a new function bar (assuming Foo.bar does not already exist).
This operation leaves the original commands as they are, so it shouldn't break existing code. Because these commands are private, they should be omitted in generated documentation. The functions based on them, however, should show up in documentation on account of being public.
def ensure_cli_documentation(cls):
"""
Modify a class that may contain instances of :py:class:`click.BaseCommand`
to ensure that it can be properly documented (e.g. using tools such as Sphinx).
This function will only process commands that have private callbacks i.e. are
prefixed with underscores. It will associate a new function with the class based on
this callback but without the leading underscores. This should mean that generated
documentation ignores the command instances but includes documentation for the functions
based on them.
This function should be invoked on a class when it is imported in order to do its job. This
can be done by applying it as a decorator on the class.
:param cls: the class to operate on
:return: `cls`, after performing relevant modifications
"""
for attr_name, attr_value in dict(cls.__dict__).items():
if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
cmd = attr_value
try:
# noinspection PyUnresolvedReferences
new_function = copy.deepcopy(cmd.callback)
except AttributeError:
continue
else:
new_function_name = attr_name.lstrip('_')
assert not hasattr(cls, new_function_name)
setattr(cls, new_function_name, new_function)
return cls
Avoiding issues with commands in classes
The reason that this solution assumes commands are inside classes is because that's how most of my commands are defined in the project I'm currently working on - I load most of my commands as plugins contained within subclasses of yapsy.IPlugin.IPlugin. If you want to define the callbacks for commands as class instance methods, you may run into a problem where click doesn't supply the self parameter to your command callbacks when you try to run your CLI. This can be solved by currying your callbacks, like below:
class Foo:
def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
if isinstance(cmd, click.Group):
commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
cmd.commands = {}
for subcommand in commands:
cmd.add_command(subcommand)
try:
if cmd.callback:
cmd.callback = partial(cmd.callback, self)
if cmd.result_callback:
cmd.result_callback = partial(cmd.result_callback, self)
except AttributeError:
pass
return cmd
Example
Putting this all together:
from functools import partial
import click
from click.testing import CliRunner
from doc_inherit import class_doc_inherit
def ensure_cli_documentation(cls):
"""
Modify a class that may contain instances of :py:class:`click.BaseCommand`
to ensure that it can be properly documented (e.g. using tools such as Sphinx).
This function will only process commands that have private callbacks i.e. are
prefixed with underscores. It will associate a new function with the class based on
this callback but without the leading underscores. This should mean that generated
documentation ignores the command instances but includes documentation for the functions
based on them.
This function should be invoked on a class when it is imported in order to do its job. This
can be done by applying it as a decorator on the class.
:param cls: the class to operate on
:return: `cls`, after performing relevant modifications
"""
for attr_name, attr_value in dict(cls.__dict__).items():
if isinstance(attr_value, click.BaseCommand) and attr_name.startswith('_'):
cmd = attr_value
try:
# noinspection PyUnresolvedReferences
new_function = cmd.callback
except AttributeError:
continue
else:
new_function_name = attr_name.lstrip('_')
assert not hasattr(cls, new_function_name)
setattr(cls, new_function_name, new_function)
return cls
#ensure_cli_documentation
#class_doc_inherit
class FooCommands(click.MultiCommand):
"""
Provides Foo commands.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._commands = [self._curry_instance_command_callbacks(self._calc)]
def list_commands(self, ctx):
return [c.name for c in self._commands]
def get_command(self, ctx, cmd_name):
try:
return next(c for c in self._commands if c.name == cmd_name)
except StopIteration:
raise click.UsageError('Undefined command: {}'.format(cmd_name))
#click.group('calc', help='mathematical calculation commands')
def _calc(self):
"""
Perform mathematical calculations.
"""
pass
#_calc.command('add', help='adds two numbers')
#click.argument('x', type=click.INT)
#click.argument('y', type=click.INT)
def _add(self, x, y):
"""
Print the sum of x and y.
:param x: the first operand
:param y: the second operand
"""
print('{} + {} = {}'.format(x, y, x + y))
#_calc.command('subtract', help='subtracts two numbers')
#click.argument('x', type=click.INT)
#click.argument('y', type=click.INT)
def _subtract(self, x, y):
"""
Print the difference of x and y.
:param x: the first operand
:param y: the second operand
"""
print('{} - {} = {}'.format(x, y, x - y))
def _curry_instance_command_callbacks(self, cmd: click.BaseCommand):
if isinstance(cmd, click.Group):
commands = [self._curry_instance_command_callbacks(c) for c in cmd.commands.values()]
cmd.commands = {}
for subcommand in commands:
cmd.add_command(subcommand)
if cmd.callback:
cmd.callback = partial(cmd.callback, self)
return cmd
#click.command(cls=FooCommands)
def cli():
pass
def main():
print('Example: Adding two numbers')
runner = CliRunner()
result = runner.invoke(cli, 'calc add 1 2'.split())
print(result.output)
print('Example: Printing usage')
result = runner.invoke(cli, 'calc add --help'.split())
print(result.output)
if __name__ == '__main__':
main()
Running main(), I get this output:
Example: Adding two numbers
1 + 2 = 3
Example: Printing usage
Usage: cli calc add [OPTIONS] X Y
adds two numbers
Options:
--help Show this message and exit.
Process finished with exit code 0
Running this through Sphinx, I can view the documentation for this in my browser:
Related
I want to build a cli interface for my application which has nested functionality. Example:
├── ...
├── setup.py
└── package-name
├──__init__.py
├──command1.py
└──command2.py
package-name command1 --arg .. ..
package-name command2 --arg ..
AND
python -m package-name command1 --arg ..
The thing to note here is that command1 and command2 are independent modules that accept different command-line args. So linking them together in a __main__.py might also be another challenge.
I came across similar questions that use entry_points in setup.py to create similar cli functionality but its not exactly what I'm looking for. I found this similar question.
How to create a CLI in Python that can be installed with PIP?
If you want access multiple sub-cli's in one entry command, you can implement a sub-command manager at __main__.py, which can parse sub-command from sys.argv and then dispatch to target module.
1️⃣First, i recommend the google fire, which can satify you in most scenarios without extra code.
Here is a example, you can replace the add/multiply function to your sub-command function by using from command1 import xx，and use entry points to expose the main function.
import fire
def add(x, y):
return x + y
def multiply(x, y):
return x * y
def main():
fire.Fire({
'add': add,
'multiply': multiply,
})
if __name__ == '__main__':
main()
We can debug this in the same way as below:
$ python example.py add 10 20
30
$ python example.py multiply 10 20
200
2️⃣Second, if you need implements by your self for some purpose, such as using argparse to define options for each command. A typical practice is the Django command, the official demo: Writing custom django-admin commands
The core steps is :
define a BaseCommand
implements BaseCommand in sub commands.py, and named it as Command.
__main__.py implements find command and call
# base_command.py
class BaseCommand(object):
def create_parser(self, prog_name, subcommand, **kwargs):
"""
Create and return the ``ArgumentParser`` which will be used to
parse the arguments to this command.
"""
# TODO: create your ArgumentParser
return CommandParser(**kwargs)
def add_arguments(self, parser):
"""
Entry point for subclassed commands to add custom arguments.
"""
pass
def run_from_argv(self, argv):
"""
Entry point for commands to be run from the command line.
"""
parser = self.create_parser(argv[0], argv[1])
options = parser.parse_args(argv[2:])
cmd_options = vars(options)
args = cmd_options.pop('args', ())
self.handle(*args, **cmd_options)
def handle(self, *args, **options):
"""
The actual logic of the command. Subclasses must implement
this method.
"""
raise NotImplementedError('subclasses of BaseCommand must provide a handle() method')
# command1.py
class Command(BaseCommand):
def handle(self, *args, **options):
print("Hello, it is command1!")
# command2.py
class Command(BaseCommand):
def handle(self, *args, **options):
print("Hello, it is command2!")
# __main__.py
def main():
# sub command name is the second argument
command_name = sys.argv[1]
# build sub command module name, and import
# package-name is the module name you mentioned
cmd_package_name = f'package-name.{command_name}'
instance = importlib.import_module(cmd_package_name)
# create instance of sub command, the Command must exist
command = instance.Command()
command.run_from_argv(sys.argv)
if __name__ == "__main__":
main()
I am writing Python code (based on PyQt5 signals and slots). I need to make my code "Scriptable". By scriptable I mean the user can utilize the internal objects himself in a user-defined python script to develop/automate some functions,..etc. But I have no idea of how to implement it clearly.
I have tried using (exec) function in python in the following way:
user-def.py
def script_entry(main_object_in_my_code):
# Connecting signal from main_object_in_my_code (inherited from QObject) to other functions in this
# file. example:
main_object_in_my_code.event_1.connect(function_1)
#QtCore.pyqtSlot(str)
def function_1 (args):
#do user-defined logic using those args.
then in my script when user want to execute it, he inputs (as example)
source user-def.py
the main script reads the script and uses exec as the following:
with open(script_path) as f:
script = f.read()
exec(script, globals())
the problem is that events are triggered but function function_1 is not executed.
I am sure this is not the right way to do this. So, How can I implement my code to be (scriptable) using user defined scripts?
I would recomend to create a class and extend from it, let the 'user' call the functions when s/he needs.
If you are not in touch with class inheritance check this tutorial
source_def.py
class Main:
def __init__(self):
super(Main, self).__init__()
def script_entry(self, main_object_in_my_code):
main_object_in_my_code.event_1.connect( function_1 )
#QtCore.pyqtSlot(str)
def function_1( self, args ):
#this checks if the function is set
invert_op = getattr(self, "user_function", None)
if callable(user_function):
eval('self.user_function( args )')
user_def.py
from source_def import Main
class UserClass( Main ):
def __init__(self):
super(UserClass, self).__init__()
def user_function(self , args ):
print( args )
Try this
I would like to have some CLI options to be re-used between two programs that share a common library file. Currently, each file has a similar structure to the following:
from .shared import G
#click.command()
#click.option("--thing1", …)
#click.optoin("--thing2", …)
…
def main(**kwargs):
g = G()
g.do_your_thing()
What I'd like to do is move all that to the __init__ of G or a subclass of G.
class G:
#click.command()
#click.option("--thing1")
def __init__(**kwargs):
pass
class G6(G):
#click.command()
#click.option("--specific-flag")
def __init__(**kwargs):
super().__init__(**kwargs)
if kwargs.get("--specific-flag"):
do_something()
self.seven = 7
However, when I try this, I get the following error.
Usage: something.py [OPTIONS]
Try 'something.py --help' for help.
Error: no such option: --thing1
My primary goal with this refactoring is to eliminate lots of duplicative code decorating the main function of each program by moving the shared decorators to the library.
If this is possible, is it also possible to use an option decorator of the subclass to supersede the same decorator on the base class? I assume the answer is yes.
Edit
Based on the answer below, my program now works with a structure similar to the following snippet:
_common_options = [
click.option(…), click.option(…), …
]
def common_options(func):
for option in _common_options:
func = option(func)
return func
class G:
def __init__(**kwargs):
print(kwargs)
#click.command()
#common_options
#click.option("--specific-option", …)
def main(**kwargs):
f = G(**kwargs)
The problem with my original attempt was putting all the #click.whatever stuff as a decorator of __init__ instead of decorating main and passing **kwargs to __init__.
What I've done previously was create a list of options to be applied to a command or group. For example:
_global_options = [
click.option('-v', '--verbose', count=True, default=0, help='Verbose output.'),
click.option('-n', '--dry-run', is_flag=True, default=False, help='Dry-run mode.')
]
def common_options(func):
for option in reversed(_global_options):
func = option(func)
return func
These would then be applied as such:
click.group(name='my_group')
#common_options
def my_group(*args, **kwargs):
pass
Decorators seem to work really well with Click.
Does this help?
I struggled to think of a good title so I'll just explain it here. I'm using Python in Maya, which has some event callback options, so you can do something like on save: run function. I have a user interface class, which I'd like it to update when certain events are triggered, which I can do, but I'm looking for a cleaner way of doing it.
Here is a basic example similar to what I have:
class test(object):
def __init__(self, x=0):
self.x = x
def run_this(self):
print self.x
def display(self):
print 'load user interface'
#Here's the main stuff that used to be just 'test().display()'
try:
callbacks = [callback1, callback2, ...]
except NameError:
pass
else:
for i in callbacks:
try:
OpenMaya.MEventMessage.removeCallback(i)
except RuntimeError:
pass
ui = test(5)
callback1 = OpenMaya.MEventMessage.addEventCallback('SomeEvent', ui.run_this)
callback2 = OpenMaya.MEventMessage.addEventCallback('SomeOtherEvent', ui.run_this)
callback3 = ......
ui.display()
The callback persists until Maya is restarted, but you can remove it using removeCallback if you pass it the value that is returned from addEventCallback. The way I have currently is just check if the variable is set before you set it, which is a lot more messy than the previous one line of test().display()
Would there be a way that I can neatly do it in the function? Something where it'd delete the old one if I ran the test class again or something similar?
There are two ways you might want to try this.
You can an have a persistent object which represents your callback manager, and allow it to hook and unhook itself.
import maya.api.OpenMaya as om
import maya.cmds as cmds
om.MEventMessage.getEventNames()
class CallbackHandler(object):
def __init__(self, cb, fn):
self.callback = cb
self.function = fn
self.id = None
def install(self):
if self.id:
print "callback is currently installed"
return False
self.id = om.MEventMessage.addEventCallback(self.callback, self.function)
return True
def uninstall(self):
if self.id:
om.MEventMessage.removeCallback(self.id)
self.id = None
return True
else:
print "callback not currently installed"
return False
def __del__(self):
self.uninstall()
def test_fn(arg):
print "callback fired 2", arg
cb = CallbackHandler('NameChanged', test_fn)
cb.install()
# callback is active
cb.uninstall()
# callback not active
cb.install()
# callback on again
del(cb) # or cb = None
# callback gone again
In this version you'd store the CallbackHandlers you create for as long as you want the callback to persist and then manually uninstall them or let them fall out of scope when you don't need them any more.
Another option would be to create your own object to represent the callbacks and then add or remove any functions you want it to trigger in your own code. This keeps the management entirely on your side instead of relying on the api, which could be good or bad depending on your needs. You'd have an Event() class which was callable (using __call__() and it would have a list of functions to fire then its' __call__() was invoked by Maya. There's an example of the kind of event handler object you'd want here
I use a simple command to run my tests with nose:
#manager.command
def test():
"""Run unit tests."""
import nose
nose.main(argv=[''])
However, nose supports many useful arguments that now I could not pass.
Is there a way to run nose with manager command (similar to the call above) and still be able to pass arguments to nose? For example:
python manage.py test --nose-arg1 --nose-arg2
Right now I'd get an error from Manager that --nose-arg1 --nose-arg2 are not recognised as valid arguments. I want to pass those args as nose.main(argv= <<< args comming after python manage.py test ... >>>)
Flask-Script has an undocumented capture_all_flags flag, which will pass remaining args to the Command.run method. This is demonstrated in the tests.
#manager.add_command
class NoseCommand(Command):
name = 'test'
capture_all_args = True
def run(self, remaining):
nose.main(argv=remaining)
python manage.py test --nose-arg1 --nose-arg2
# will call nose.main(argv=['--nose-arg1', '--nose-arg2'])
In the sources of flask_script you can see that the "too many arguments" error is prevented when the executed Command has the attribute capture_all_args set to True which isn't documented anywhere.
You can set that attribute on the class just before you run the manager
if __name__ == "__main__":
from flask.ext.script import Command
Command.capture_all_args = True
manager.run()
Like this additional arguments to the manager are always accepted.
The downside of this quick fix is that you cannot register options or arguments to the commands the normal way anymore.
If you still need that feature you could subclass the Manager and override the command decorator like this
class MyManager(Manager):
def command(self, capture_all=False):
def decorator(func):
command = Command(func)
command.capture_all_args = capture_all
self.add_command(func.__name__, command)
return func
return decorator
Then you can use the command decorator like this
#manager.command(True) # capture all arguments
def use_all(*args):
print("args:", args[0])
#manager.command() # normal way of registering arguments
def normal(name):
print("name", name)
Note that for some reason flask_script requires use_all to accept a varargs but will store the list of arguments in args[0] which is a bit strange. def use_all(args): does not work and fails with TypeError "got multiple values for argument 'args'"
Ran into an issue with davidism's soln where only some of the args were being received
Looking through the docs a bit more it is documented that nose.main automatically picks up the stdin
http://nose.readthedocs.io/en/latest/api/core.html
So we are now just using:
#manager.add_command
class NoseCommand(Command):
name = 'nose'
capture_all_args = True
def run(self, remaining):
nose.main()