So I wrote a Python 3 library, which serves as an application 'backend'. Now I can sit down with the interpreter, import the source file(s), and hack around using the lib - I know how to do this.
But I would also like to build a command line 'frontent' application using the library. My library defines a few objects which have high-level commands, which should be visible by the application. Such commands may return some data structures and the high-level commands would print them nicely. In other words, the command line app would be a thin wrapper around the lib, passing her input to the library commands, and presenting results to the user.
The best example of what I'm trying to achieve would probably be Mercurial SCM, as it is written in Python and the 'hg' command does what I'm looking for - for instance, 'hg commit -m message' will find the code responsible for the 'commit' command implementation, pass the arguments from the user and do its work. On the way back, it might get some results and print them out nicely.
Is there a general way of doing it in Python? Like exposing classes/methods/functions as 'high level' commands with an annotation? Does anybody know of any tutorials?
You can do this with argparse. For example here is the start of my deploy script.
def main(argv):
"""
Entry point for the deploy script.
Arguments:
argv: All command line arguments save the name of the script.
"""
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('-v', '--verbose', action='store_true',
help='also report if files are the same')
parser.add_argument('-V', '--version', action='version',
version=__version__)
parser.add_argument('command', choices=['check', 'diff', 'install'])
fname = '.'.join(['filelist', pwd.getpwuid(os.getuid())[0]])
args = parser.parse_args(argv)
It uses an argument with choices to pick a function. You could define a dictionary mapping choices to functions;
cmds = {'check': do_check, 'diff': do_diff, 'install': do_install}
fn = cmds[args.command]
If you make sure that all the dict keys are in the command choices, you don't need to catch KeyError.
Related
I am developing a command-line toolset for a project. The final tool shall support many subcommands, like so
foo command1 [--option1 [value]?]*
So there can be subcommands like
foo create --option1 value --
foo make file1 --option2 --option3
The tool uses the argparse library for handling command-line arguments and help functionality etc.
A few additional requirements and constraints:
Some options and functionality is identical for all subcommands (e.g. parsing a YAML configuration file etc.)
Some subcommands are quick and simple to code, because they e.g. just call an external bash script.
Some subcommands will be complex and hence long code.
Help for the basic tool should be available as well as for an individual subcommand:
foo help
Available commands are: make, create, add, xyz
foo help make
Details for the make subcommand
error codes should be uniform across the subcommands (like the same error code for "file not found")
For debugging purposes and for making progress with self-contained functionality for minimal viable versions, I would like to develop some subcommands as self-containted scripts and modules, like
make.py
that can be imported into the main foo.py script and later on invoked as both
make.py --option1 value etc.
and
foo.py make --option1 value
Now, my problem is: What is the best way to modularize such a complex CLI tool with minimal redundancy (e.g. the arguments definition and parsing should only be coded in one component)?
Option 1: Put everything into one big script, but that will become difficult to manage.
Option 2: Develop the functionality for a subcommand in individual modules / files (like make.py, add.py); but such must remain invocable (via if __name__ == '__main__' ...).
The functions from the subcommand modules could then be imported into the main script, and the parser and arguments from the subcommand added as a subparser.
Option 3: The main script could simply reformat the call to a subcommand to subprocess, like so
subprocess.run('./make.py {arguments}', shell=True, check=True, text=True)
I'm more used to answering questions about the details of numpy and argparse, but here's how I envisage a large package.
In a main.py:
import submod1
# ....
sublist = [submod1, ...]
def make_parser(sublist):
parser = argparse.ArgumentParser()
# parser.add_argument('-f','--foo') # main specific
# I'd avoid positionals
sp = parser.add_subparsers(dest='cmd', etc)
splist=[]
for md in sublist:
sp1 = sp.add_parser(help='', parents=[md.parser])
sp1.set_default(func=md.func) # subparser func as shown in docs
splist.append(sp1)
return parser
if name == 'main':
parser = make_parser(sublist)
args = parser.parse_args()
# print(args) # debugging display
args.func(args) # again the subparser func
In submod1.py
import argparse
def make_parser():
parser = argparse.ArgumentParser(add_help=False) # check docs?
parser.add_argument(...) # could add a common parents here
return parser
parser.make_parser()
def func(args):
# module specific 'main'
I'm sure this is incomplete in many ways, since I've written this on the fly without testing. It's a basic subparser definition as documented, but using parents to import subparsers as defined in the submodules. parents could also be used to define common arguments for subparsers; but utility functions would work just as well. I think parents is most useful when using a parser that you can't otherwise access; ie. an imported one.
parents essentially copies Actions from one parser to the new one - copy by reference (not by value or as a copy). It is not a highly developed tool, and there have been a number of SO where people ran into problems. So don't try to over extend it.
Consider using the Command Pattern along with the Factory Method Pattern.
In short, create an abstract class called Command and make each command it's own class inheriting from Command.
Example:
class Command():
def execute(self):
raise NotImplementedError()
class Command1(Command):
def __init__(self, *args):
pass
def execute(self):
pass
class Command2(Command):
def __init__(self, *args):
pass
def execute(self):
pass
This will handle execution of commands. For building, make a command factory.
class CommandFactory():
#staticmethod
def create(command, *args):
if command == 'command1':
return Command1(args)
elif command == 'command2':
return Command2(args)
Then you'd be able to execute a command with one line:
CommandFactory.create(command, args).execute()
Thanks for all of your suggestions!
I think the most elegant approach is using Typer and following this recipe:
https://typer.tiangolo.com/tutorial/subcommands/add-typer/
I am new to argparser in python . I am trying to create argparser for file which contains two functions download and upload file on/from box. It will do only one functionality at once according according to that i am trying to create parser for that file as follows but it's not working for me:
parser = argparse.ArgumentParser(description='Download or Upload file on box.')
parser.add_argument('-df', '--download', required=True,
help='download file box')
parser.add_argument('-uf', '--upload', nargs='+', required=True,
help='upload file of box')
parser.add_argument('-fp', '--filepath', required=True,
help='file path to upload(which file to upload) or download(where to download file)')
parser.add_argument('-fn', '--filename', required=True,
help='what should be the name of file on box')
parser.add_argument('-fi', '--fileid', required=True,
help='file id of file to download from box')
args = vars(parser.parse_args())
NOTE :- every time only -df or -uf options will be there, -fp is mandatory for both and if it is -df then -fi is only option and it is mandatory and if -uf then -fn is only option and it's mandatory.
How do I achieve this, following are example how will i pass argument to file
pyhton abc.py -df -fp 'Download/boxfile/xyz.txt' -fi 123
python abc.py -uf -fp 'Download/boxfile/xyz.txt' -fn 'qwe.txt'
As written all 5 of the arguments are required - you made that explicit. If that's what you really want, all the rest of the question is irrelevant. You'll have to provide all 5 regardless.
But the comments indicate that you want to use either -df or -uf, but probably not both (though that bit's unclear).
While there is a mutually_exclusive_mechanism in argparse, there isn't a inclusive equivalent - something that says if -f is present, then -g must also be given.
But subparsers mechanism can be used that way. You could define an download subparser, with a required -fi argument. And an upload with its own argument.
Another option is to set -df to take 2 arguments (nargs=2), both its box and its file.
If -df and -uf are mutually exclusive, why not use the same argument for the name of the file? That is, replace -fn and -fi with one file argument.
Another option is to make all (or most) of the arguments not-required, and check for the correct combinations after parsing. It's easier to implement complicated logic in your own code than to force argparse to do it for you.
e.g.
if args.download is not None: # not default
<check for `args.filename`> etc
It can also be a good idea to provide defaults for optional arguments. That way the code can run even if the user doesn't provide all items.
On a matter of style. Short option flags, with on - are usually a single character. -d, -u, etc. The ability to combine several into one string only works in that case, e.g. -df foo. In this case it probably doesn't matter since none of your arguments are store_true.
I'm not incredibly familiar with argparse, however from reading the documentation I don't believe there's a way to use mutually_exclusive_group and required to force this. It seems conditional statements or equivalent are necessary to confirm that valid parameter combinations were passed.
See: Python Argparse conditionally required arguments
I'd suggest that you get rid of most of these arguments, maybe all, and think about standard Unix ways of doing things.
Consider scp, which has a syntax of: scp source destination
For instance: scp Download/boxfile/xyz.txt qwe.txt
If you don't supply a destination, it infers that you want the file to be called the same thing, and land right here, so these two things are equivalent:
scp Download/boxfile/xyz.txt xyz.txt
scp Download/boxfile/xyz.txt
Of course, scp can talk to machines across the internet, so there is a format for that:
scp hostname:Download/boxfile/xyz.txt xyz.txt
And if you are uploading, you simple switch the order:
scp xyz.txt hostname:Download/boxfile/xyz.txt
I have a script that finds test names and is widely used in our company. It operates on the command line like so:
find_test.py --type <type> --name <testname>
Inside the script is the equivalent of:
import argparse
parser = argparse.ArgumentParser(description='Get Test Path')
parser.add_argument('--type', dest='TYPE', type=str, default=None, help="TYPE [REQUIRED]")
parser.add_argument('--name', dest='test_name', type=str, default=None, help="Test Name (Slow)")
parser.add_argument('--id', dest='test_id', type=str, default=None, help="Test ID (Fast)")
parser.add_argument('--normalise', dest='normalise', action="store_true", default=False, help="Replace '/' with '.' in Test Name")
args = parser.parse_args()
(Not sure what all these arguments do, I personally only use the first two). These lines are then proceeded by the code that uses these arguments.
I want to refactor this script so I can import it as a module, but also preserve its command line functionality - since lots of people use this script and it is also called in some of our csh scripts.
I have refactored it so far, like so:
def _main():
<all the former code that was in find_test.py>
if __name__ == "__main__":
_main()
And this still runs fine from the command line. But I don't know how then in my parent script I pass arguments with the relevant switches into this.
How do I refactor this further and then call it in parent script?
Is this possible?
I'd also rather not use docopts which i've read is the new argparse unless necessary - i.e. can't be done with argparse, since it's not installed company wide and this can be an arduous procedure.
You shouldn't just move all the code directly into a function; that doesn't help at all.
What you should do is move the code that needs to run whatever happens into a function. (And since it is the external interface, it should not begin with _.) The code that only needs to run from the command line - ie the parser stuff - should stay in the __name__ == '__main__' block, and it should pass its results to main().
So:
def main(TYPE, test_name, test_id=None, normalise=False):
# ... body of script...
if __name__ == "__main__":
parser = ...
...
args = parser.parse_args()
main(**vars(args))
(And docopt isn't the new anything; it's an external library which some people like.)
Using argparse, is there a simple way to specify arguments which are mutually exclusive so that the application asks for one of these arguments have to be provided but only one of them?
Example of fictive use-case:
> myapp.py foo --bar
"Foo(bar) - DONE"
> myapp.py read truc.txt
"Read: truc.txt - DONE"
>myapp.py foo read
Error: use "myapp.py foo [options]" or "myapp.py read [options]" (or something similar).
> myapp.py foo truc.txt
Error: "foo" action don't need additional info.
> myapp.py read --bar
Error: "read" action don't have a "--bar" option.
My goal is to have a "driver" application(1) that would internally apply one action depending on the first command line argument and have arguments depending on the action.
So far I see no obvious ways to do this with argparse without manually processing the arguments myself, but maybe I missed something Pythonic? (I'm not a Python3 expert...yet)
I call it "driver" because it might be implemented by calling another application, like gcc does with different compilers.
What you're trying to do is actually supported quite well in Python.
See Mutual Exclusion
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('foo', dest='foo', nargs=1)
group.add_argument('read', dest='read', nargs=1)
args = parser.parse_args()
return args
I would like to provide database for my program that uses elixir for ORM. Right now the database file (I am using SQLite) must be hardcoded in metadata, but I would like to be able to pass this in argv. Is there any way to do this nice?
The only thing I thought of is to:
from sys import argv
metadata.bind = argv[1]
Can I set this in the main script and it would be used in all modules, that define any Entities?
I have some code that does this in a slightly nicer fashion than just using argv
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-u", "--user", dest="user",
help="Database username")
parser.add_option("-p", "--password", dest="password",
help="Database password")
parser.add_option("-D", "--database", dest="database", default="myDatabase",
help="Database name")
parser.add_option("-e", "--engine", dest="engine", default="mysql",
help="Database engine")
parser.add_option("-H", "--host", dest="host", default="localhost",
help="Database host")
(options, args) = parser.parse_args()
def opt_hash(name):
global options
return getattr(options, name)
options.__getitem__ = opt_hash
metadata.bind = '%(engine)s://%(user)s:%(password)s#%(host)s/%(database)s' % options
Note that the part using opt_hash is a bit of a hack. I use it because OptionParser doesn't return a normal hash, which is what is really needed for the niceness of the bind string I use in the last line.
Your question seems to be more related to general argument parsing in python than with elixir.
Anyway, I had a similar problem, and I have solved it by using different configuration files and parsing them with the configparse module in python.
For example, I have two config files, and each of them describes the db url, username, password, etc.. of one database. When I want to switch to another configuration, I pass an option like --configfile guest to the script (I use argparse for the command line interface), then the script looks for a config file called guest.txt, and reads all the information there.
This is a lot safer, because if you pass a metadata string as a command line argument you can have some security issues, and moreover it is a lot longer to type.
By the way, you can also find useful to write a Makefile to store the most common options.
e.g. cat >Makefile
debug_db:
ipython connect_db.py -config guest -i
connect_root:
ipython connect_db.py -config db1_root -i
connect_db1:
ipython connect_db.py -config db1 -i
and on the command line, you only have to type 'make debug_db' or 'make connect_db1' to execute a rule.