I have written some scripts that I'm trying to integrate with click. All the scripts are written in python OOP's.
The issue is that i am trying to build command section in oop's way but couldn't do it properly.
let me show you, what i am trying to do and please note that i am sharing here dummy code it is very similar to the real code.
First thing the directory structure:
-customcmd <dir>
|
|->commands <dir>
| -> abc-command.py
| -> __init__.py
|
|->__init__.py
|->main.py
|->setup.py
1) I have created one file called main.py, which contains following code:
import click
import os
plugin_folder = os.path.join(os.path.dirname(__file__), 'commands')
class MyCLI(click.MultiCommand):
def list_commands(self, ctx):
rv = []
for filename in os.listdir(plugin_folder):
if filename.startswith('__'):
continue
if filename.endswith('.py'):
rv.append(filename[:-3])
rv.sort()
return rv
def get_command(self, ctx, name):
ns = {}
fn = os.path.join(plugin_folder, name + '.py')
with open(fn) as f:
code = compile(f.read(), fn, 'exec')
eval(code, ns, ns)
return ns['cli']
cli = MyCLI()#help='This tool\'s subcommands are loaded from a ''plugin folder dynamically.'
if __name__ == '__main__':
cli()
2) abc-command.py
#click.command()
#click.option("--file-loc", '-fl', type=open, required=True, default=None, help="Path to the file")
def cli(file_loc):
"""
This is test command
"""
print("Path to file {}".format(file_loc))
Output of above code when you call main.py:
(automation) K:\Pythonenv\automation\customcmd>python main.py
Usage: main.py [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
policyresult This is test command
Output of above code when you call sub command:
(automation) K:\Pythonenv\automation\customcmd>python main.py policyresult --help
Usage: main.py policyresult [OPTIONS]
This is test command
Options:
-fl, --file-loc OPEN Path to the file [required]
--help Show this message and exit.
3) This is how I converted the procedural code of abc-command.py code:
class policyresult():
def __init__(self):
pass
#click.command()
#click.option("--file-loc", '-fl', type=open, required=True, default=None, help="Path to the file")
def cli(self,file_loc):
"""
This is test command
"""
print("Path to file {}".format(file_loc))
obj = policyresult()
obj.cli()
Output of above code doesn't match with the previous output when the code was procedural in abc-command.py:
Here i am calling the main.py
(automation) K:\Pythonenv\automation\customcmd>python main.py
Usage: main.py [OPTIONS]
Try "main.py --help" for help.
Error: Missing option "--file-loc" / "-fl".
In the above output you can see it is directly going into the sub-command options things and giving error as well.
As far as i understand list_commands() which is in main.py can't list out the commands, this part i can't understand why it is not working properly.
I tried various things but couldn't find the proper way to implement OOP's in abc-command.py because of that my ouput doesn't match.
I am new to this click framework, please suggest any new changes in my approach if needed.
please look into this, sorry for this weird way to explaining this.
abc-command.py is evaluated before click parses command options because of this line in the file invoking the cli method:
obj.cli()
Also, in the get_command method implemented for the multi-command, commands are supposed to expose a 'cli' name in their namespace.
To fix this error, in abc-command.py update the line invoking the cli command with:
cli = obj.cli
so that a cli name is exposed in abc-command.py module
Related
I am trying to run a click cli through a bash with a different command groups run through a single command collection.
src/preprocessing_extract_data/scripts/main.py
import click
#click.group()
def run_preprocessing_extract_data():
pass
#run_preprocessing_extract_data.command()
#click.option(
"--start_date",
type=click.DateTime(formats=["%Y-%m-%d"]),
required=True,
help="Start date for the pipeline",
)
#click.option(
"--end_date",
type=click.DateTime(formats=["%Y-%m-%d"]),
required=True,
help="End date for the pipeline",
)
def main(start_date, end_date):
...
if __name__ == "__main__":
main()
src/scripts/main.py
from click import CommandCollection
from src.preprocessing_extract_data.scripts.main import run_preprocessing_extract_data
if __name__ == "__main__":
cmds = [
run_preprocessing_extract_data,
# a few more similar command groups
]
cli = CommandCollection(sources=cmds)
cli()
scripts/entrypoint.sh
#!/bin/sh
start_date="$1"
end_date="$2"
python src/scripts/main.py run_preprocessing_extract_data --start_date=$start_date --end_date=$end_date
I run it using ./scripts/entrypoint.sh --start_date="2020-11-01" --end_date="2021-12-01" --today="2021-12-10" but it keeps failing and throws the following error:
Usage: main.py [OPTIONS] COMMAND [ARGS]...
Try 'main.py --help' for help.
Error: No such command 'run_preprocessing_extract_data'.
From the docs:
The default implementation for such a merging system is the CommandCollection class. It accepts a list of other multi commands and makes the commands available on the same level.
Hence, your script now has a command main; you can check this by running your script with --help (or no arguments at all): python src/scripts/main.py --help.
Hence you can do the following:
python src/scripts/main.py main --start_date="$start_date" --end_date="$end_date"
By the way, invoking your shell script should be done without the --start_date: ./scripts/entrypoint.sh "2020-11-01" "2021-12-01".
For a Python script that uses argparse and has a very long argument list, is it possible to make argparse page what it prints to the terminal when calling the script with the -h option?
I could not find a quick answer, so I wrote a little something:
# hello.py
import argparse
import os
import shlex
import stat
import subprocess as sb
import tempfile
def get_pager():
"""
Get path to your pager of choice, or less, or more
"""
pagers = (os.getenv('PAGER'), 'less', 'more',)
for path in (os.getenv('PATH') or '').split(os.path.pathsep):
for pager in pagers:
if pager is None:
continue
pager = iter(pager.split(' ', 1))
prog = os.path.join(path, next(pager))
args = next(pager, None) or ''
try:
md = os.stat(prog).st_mode
if md & (stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH):
return '{p} {a}'.format(p=prog, a=args)
except OSError:
continue
class CustomArgParser(argparse.ArgumentParser):
"""
A custom ArgumentParser class that prints help messages
using either your pager, or less or more, if available.
Otherwise, it does what ArgumentParser would do.
Use the PAGER environment variable to force it to use your pager
of choice.
"""
def print_help(self, file=None):
text = self.format_help()
pager = get_pager()
if pager is None:
return super().print_help(file)
fd, fname = tempfile.mkstemp(prefix='simeon_help_', suffix='.txt')
with open(fd, 'w') as fh:
super().print_help(fh)
cmd = shlex.split('{p} {f}'.format(p=pager, f=fname))
with sb.Popen(cmd) as proc:
rc = proc.wait()
if rc != 0:
super().print_help(file)
try:
os.unlink(fname)
except:
pass
if __name__ == '__main__':
parser = CustomArgParser(description='Some little program')
parser.add_argument('--message', '-m', help='Your message', default='hello world')
args = parser.parse_args()
print(args.message)
This snippet does main things. First, it defines a function to get the absolute path to a pager. If you set the environment variable PAGER, it will try and use it to display the help messages. Second, it defines a custom class that inherits pretty much everything from argparse.ArgumentParser. The only method that gets overridden here is print_help. It implements print_help by defaulting to super().print_help() whenever a valid pager is not found. If a valid is found, then it writes the help message to a temporary file and then opens a child process that invokes the pager with the path to the temporary file. When the pager returns, the temporary file is deleted. That's pretty much it.
You are more than welcome to update get_pager to add as many pager programs as you see fit.
Call the script:
python3 hello.py --help ## Uses less
PAGER='nano --view' python3 hello.py --help ## Uses nano
PAGER=more python3 hello.py --help ## Uses more
I am trying to make a kind of recursive call on my first Click CLI app.
The main point is to have sub-commands associated to the first and, so, I was trying to separate it all in different files/modules to improve it's maintainability.
I have the current directory:
root
|-commands
|-project
|---__init__
|---command1
|---command2
|-database
|---__init__
|---command1
|---command2
This is my main file:
import click
from commands.project import project
from commands.database import database
#click.group(help="Main command")
def main():
pass
main.add_command(project)
main.add_command(database)
My projects __init__ file:
from commands.project.command1 import *
from commands.project.command2 import *
import click
#click.group(help="Projects")
def project():
pass
project.add_command(command1)
project.add_command(command2)
My commands.project.command1 file:
import click
#click.command()
def command1():
"""
Execute all the steps required to update the project.
"""
pass
The main point here is that, every time I want to add a new subcommand, I need to:
Add .py file with all code to the command, in respective subcommand/submodule folder (obviously!)
Add it's import statement on it's __init__ file
Relate this new command to it's parent (project/database, in this case)
Is there any way to do a circular/dynamic load to avoid step no.2 and 3?
EDIT
After tried Stephen Rauch way, it successfully includes all provided files, but none of the commands works with - only with function name (eg: update-project -> update_project).
root
|-commands
|-project
|---update
|---install_project
|-database
|---command_one
|---command_two
main.py
# main command ----------------------------------------------------------- ###
#click.group(help="CLI tool!", context_settings=dict(max_content_width=120))
def main():
pass
# PROJECT command group -------------------------------------------------------- ###
#main.group(cls=group_from_folder("commands/project"),
short_help="Project installation and upgrade utils.",
help="Project installation and upgrade.")
def project():
pass
commands/project/install_project.py
import click
#click.command(name="install-project",
help="This options allows you to easily install project",
short_help="Install a brand new project")
#click.pass_context
def install_project(ctx):
CLI result main project --help (note the install_project instead install-project sub command)
Usage: main project [OPTIONS] COMMAND [ARGS]...
Project installation and upgrade.
Options:
--help Show this message and exit.
Commands:
install_project Install a brand new project one
Modifying the example from here, you can eliminate steps two and three. I suggest creating a custom class for each folder via a closure. This completely eliminates the need for the __init__.py in the commands folder. Additionally there is no need to import the folder (module) or the commands in the folder.
Custom Group Class Creator:
import click
import os
def group_from_folder(group_folder_name):
folder = os.path.join(os.path.dirname(__file__), group_folder_name)
class FolderCommands(click.MultiCommand):
def list_commands(self, ctx):
return sorted(
f[:-3] for f in os.listdir(folder) if f.endswith('.py'))
def get_command(self, ctx, name):
namespace = {}
command_file = os.path.join(folder, name + '.py')
with open(command_file) as f:
code = compile(f.read(), command_file, 'exec')
eval(code, namespace, namespace)
return namespace[name.replace('-', '_').lower()]
return FolderCommands
Using the Custom Class:
To use the custom class, first place the commands (as structured in the question) into a folder. Then decorate the group command using the cls parameter, and pass a custom class which was initialized pointing to the folder containing the commands.
#cli.group(cls=group_from_folder('project'))
def group():
"command for grouping"
Test Code:
#click.group()
def cli():
"My awesome script"
#cli.group(cls=group_from_folder('group'))
def group():
"command for grouping"
if __name__ == "__main__":
commands = (
'group command-test',
'group',
'group --help',
'',
)
import sys, time
time.sleep(1)
print('Click Version: {}'.format(click.__version__))
print('Python Version: {}'.format(sys.version))
for cmd in commands:
try:
time.sleep(0.1)
print('-----------')
print('> ' + cmd)
time.sleep(0.1)
cli(cmd.split())
except BaseException as exc:
if str(exc) != '0' and \
not isinstance(exc, (click.ClickException, SystemExit)):
raise
File group/command-test.py
import click
#click.command('command-test')
def command_test():
"""
Execute all the steps required to update the project.
"""
click.echo('Command Test')
Results:
Click Version: 6.7
Python Version: 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)]
-----------
> group command-test
Command Test
-----------
> group
Usage: test.py group [OPTIONS] COMMAND [ARGS]...
command for grouping
Options:
--help Show this message and exit.
Commands:
command-test Execute all the steps required to update the...
-----------
> group --help
Usage: test.py group [OPTIONS] COMMAND [ARGS]...
command for grouping
Options:
--help Show this message and exit.
Commands:
command-test Execute all the steps required to update the...
-----------
>
Usage: test.py [OPTIONS] COMMAND [ARGS]...
My awesome script
Options:
--help Show this message and exit.
Commands:
group command for grouping
I suggest you just read your commands from specific Python package and then add to you entry group.
Suppose we have such structure:
|--app
|--commands
|--__init__.py
|--group1
|--__init__.py
|--command1.py
|--group2
|--__init__.py
|--command2.py
|--__init__.py
|--cli.py
Then your commands files need to contain one click.Command with a specified name and a function with a name 'command':
import click
#click.command(name="your-first-command")
def command():
pass
Init files in each of your group need to contain doc string to have proper 'help' value for your click.Group.
And most interesting cli.py:
import click
import importlib
import pkgutil
import os.path
def get_commands_from_pkg(pkg) -> dict:
pkg_obj = importlib.import_module(pkg)
pkg_path = os.path.dirname(pkg_obj.__file__)
commands = {}
for module in pkgutil.iter_modules([pkg_path]):
module_obj = importlib.import_module(f"{pkg}.{module.name}")
if not module.ispkg:
commands[module_obj.command.name] = module_obj.command
else:
commands[module.name.replace('_', '-')] = click.Group(
context_settings={'help_option_names': ['-h', '--help']},
help=module_obj.__doc__,
commands=get_commands_from_pkg(f"{pkg}.{module.name}")
)
return commands
#click.group(context_settings={'help_option_names': ['-h', '--help']}, help="Your CLI",
commands=get_commands_from_pkg('app.commands'))
def cli():
pass
As you can see we recursively create click groups and add the click command to the specific group.
I have a simple python (v2.7) script (test.py)
#!/usr/bin/python
import sys
from optparse import OptionParser
def main():
parser = OptionParser()
parser.add_option("--files", dest="files",
metavar="FILES", default=None,
help="A file pattern matching ottcall logs.")
(options, args) = parser.parse_args()
print "FILES_PATTERN %s" % options.files
if not options.files:
parser.error("Files_pattern option is mandatory - Abort execution.")
return 0
if __name__ == "__main__":
sys.exit(main())
User must provide a file pattern or a filename
Run script in command line if option is missing returns error:
python test.py
FILES_PATTERN None
Usage: test.py [options]
test.py: error: Files_pattern option is mandatory - Abort execution.
If option files is missing some letters (--fil instead of --files):
python test.py --fil "a_pattern_for_files"
FILES_PATTERN a_pattern_for_files
I think I should have an error like the following
python test.py --fl "a_pattern_for_files"
Usage: test.py [options]
test.py: error: no such option: --fl
Why don't I get an error from OptionParser when I use --fil instead of the correct argument --files ?
Not only I do not get an error but variable files stores the value: a_pattern_for_files (which is printed).
I am expecting argument files to have value: None (default) unless in command line --files exists
optparse allows abbreviated forms of long options. --fil is a prefix of --files and not a prefix of any other long options the program supports, so --fil is treated as equivalent to --files.
This is barely mentioned in the docs, and there is no option to turn it off. argparse has an option to turn it off, but only in Python 3.5+.
Running the following code results in this error:
TypeError: init() got an unexpected keyword argument 'help'
Code:
import click
#click.command()
#click.argument('command', required=1, help="start|stop|restart")
#click.option('--debug/--no-debug', default=False, help="Run in foreground")
def main(command, debug):
print (command)
print (debug)
if __name__ == '__main__':
main()
Full error output:
$ python3 foo.py start
Traceback (most recent call last):
File "foo.py", line 5, in <module>
#click.option('--debug/--no-debug', default=False, help="Run in foreground")
File "/home/cbetti/python/lib/python3/dist-packages/click-4.0-py3.4.egg/click/decorators.py", line 148, in decorator
_param_memo(f, ArgumentClass(param_decls, **attrs))
File "/home/cbetti/python/lib/python3/dist-packages/click-4.0-py3.4.egg/click/core.py", line 1618, in __init__
Parameter.__init__(self, param_decls, required=required, **attrs)
TypeError: __init__() got an unexpected keyword argument 'help'
Why does this error occur?
I run into this again and again because the trace output does not correspond to the parameter causing the error. Notice ArgumentClass in the trace, that's your hint.
'help' is an acceptable parameter to #click.option. The click library prefers, however, that you document your own arguments. The #click.argument help parameter is causing this exception.
This code works: (notice the lack of , help="start|stop|restart" in #click.argument)
import click
#click.command()
#click.argument('command', required=1)
#click.option('--debug/--no-debug', default=False, help="Run in foreground")
def main(command, debug):
""" COMMAND: start|stop|restart """
print (command)
print (debug)
if __name__ == '__main__':
main()
Output:
$ python3 foo.py start
start
False
Help Output:
Usage: test.py [OPTIONS] COMMAND
COMMAND: start|stop|restart
Options:
--debug / --no-debug Run in foreground
--help Show this message and exit.
You are defining commands as arguments. Note that click has a better way to define commands then what you are trying here.
#click.group()
def main():
pass
#main.command()
def start():
"""documentation for the start command"""
print("running command `start`")
#main.command()
def stop():
"""documentation for the stop command"""
print("running command `stop`")
if __name__ == '__main__':
main()
will result in the following default help text:
Usage: test_cli.py [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
start documentation for the start command
stop documentation for the stop command
Having said that, should you really need arguments, you cannot use the help parameter. The click documentation indeed states that you should document your own arguments. However, I have no idea how to do that. Any hints?
EDIT
As mentioned in the comments: this is to align with the Unix standard to document arguments in the main help text.
click library does not allow -help parameter inside click.arguments (including the current version 6.7 when this comment has been written).
But, you can use the -help parameter inside click.options.
You can check current click documentation at http://click.pocoo.org/6/documentation/ or previous at http://click.pocoo.org/5/documentation/ this behaviour has not change recently.
Then, it is a WAD. It is not a BUG.
For me it was because my variable looked like DnsCryptExractDir and I have to chnage it too this dnscryptexractdir because *args could not find it.
For same error I got it because my argument name was url_Hook (camelCase). After I changed it to url_hook it got resolved.