Develop Reference

How can I use click to parse a string for arguments?

Say I have a list of strings containing arguments and options, with argparse, I’m able to parse this list using the parse_args function into an object, as follows:
import argparse
extra_params = [‘—sum’, ‘7’, ‘-1’, ‘42’]
parser=argparse.ArgumentParser(description=“argparse docs example”)
parser.add_argument(‘integers’, metavar=‘N’, type=int, nargs=‘+’,
help=‘an integer for the accumulator’)
parser.add_argument(‘—sum’, dest=‘accumulate’, action=‘store_const’,
const=sum, default=max,
help=‘sum the integers (default: find the max)’)
parsed_object=parser.parse_args(extra_params)
Here, argparse has parsed a provided iterable of strings. Can one use click to also parse a provided iterable of strings?
I’ve searched through the API documentation for click and it appears that there’s a parse_args function within the *Command set of classes but don’t see anything in the docs around how I can do this. I’ve tried instantiating BaseCommand as well as Command but not sure how to get parse_args working without a correct context.
For broader context, this question is a result of having built a launcher application that end users use as a scaffold to launch their own applications. Here, the launcher consumes a number of arguments for which click decorators work perfectly. Unknown arguments can be handled as shown in the documentation here. This launcher then calls an end-user provided callable with these unparsed parameters. Click leaves unparsed parameters as a tuple of strings. How would the end-user, in this situation, be able to use Click to parse the argument's they're interested in? Here's a snippet to illustrate the issue:
import click
from typing import Tuple
#click.command(name="TestLauncher", context_settings={
"ignore_unknown_options": True
})
#click.option('--uri', '-u',
help="URI for the server")
#click.argument('unprocessed_args', nargs=-1,
type=click.UNPROCESSED)
def main(uri: str, unprocessed_args: Tuple[str, ...]) -> None:
print(f"Was passed a URI of {uri}")
print(f"Additional args are {unprocessed_args}")
child_function(unprocessed_args)
def child_function(unprocessed_args: Tuple[str, ...]) -> None:
# How do I get Click to parse the provided args for me?
pass
if __name__ == "__main__":
# pylint: disable=no-value-for-parameter, unexpected-keyword-arg
main()
Running this from the command line:
python3 so_test.py --uri test.com --prog-arg 10
Was passed a URI of test.com
Additional args are ('--prog-arg', '10')

For the calling function not knowing anything about parameters for the child function, you can try this:
#click.command(name="TestLauncher", context_settings={
"ignore_unknown_options": True
})
#click.option('--uri', '-u',
help="URI for the server")
#click.argument('unprocessed_args', nargs=-1,
type=click.UNPROCESSED)
def main(uri: str, unprocessed_args: Tuple[str, ...]) -> None:
print(f"Was passed a URI of {uri}")
print(f"Additional args are {unprocessed_args}")
unprocessed_args = dict([(unprocessed_args[i].replace('--', '').replace('-', '_'), unprocessed_args[i+1]) for i in range(0, len(unprocessed_args), 2)])
click.get_current_context().invoke(child_function, **unprocessed_args)
#click.command(context_settings={"ignore_unknown_options": True})
#click.option('-p', '--prog-arg')
def child_function(prog_arg: str, **kwargs) -> None:
# How do I get Click to parse the provided args for me?
print(f"Child function passed: {prog_arg}")
# all remaining unknown options are in **kwargs
if __name__ == "__main__":
# pylint: disable=no-value-for-parameter, unexpected-keyword-arg
main()
However, note that:
unprocessed_args = dict([(unprocessed_args[i].replace('--', '').replace('-', '_'), unprocessed_args[i+1]) for i in range(0, len(unprocessed_args), 2)])
This assumes you can only have one value per option. The alternative is to call your script by passing in options like below, splitting the string on = and doing whatever pre-formatting you deem necessary.
--prog-arg=<Your-desired-values>

Try something like this:
import click
#click.command()
#click.option('--count', default=1, help='number of greetings')
#click.option('--test', default='test_was_not_provided', help='test option')
#click.argument('name')
def hello(*args, **kwargs):
click.echo(f"Hello World! {kwargs['name']} {kwargs['count']}")
if __name__ == '__main__':
hello()
run with something like: python main.py haha --test this_is_a_test --count=40

Reviewing the comments and my ensuing edit, made me think that simply applying the click decorators to the child function may work. Indeed it seems to but I don't entirely know why.
import click
from typing import Tuple
#click.command(name="TestLauncher", context_settings={
"ignore_unknown_options": True
})
#click.option('--uri', '-u',
help="URI for the server")
#click.argument('unprocessed_args', nargs=-1,
type=click.UNPROCESSED)
def main(uri: str, unprocessed_args: Tuple[str, ...]) -> None:
print(f"Was passed a URI of {uri}")
print(f"Additional args are {unprocessed_args}")
child_function(unprocessed_args)
#click.command()
#click.option('--prog-arg')
def child_function(prog_arg: str) -> None:
# How do I get Click to parse the provided args for me?
print(f"Child function passed: {prog_arg}")
if __name__ == "__main__":
# pylint: disable=no-value-for-parameter, unexpected-keyword-arg
main()
python3 so_test.py --uri test.com --prog-arg 10
Was passed a URI of test.com
Additional args are ('--prog-arg', '10')
Child function passed: 10

How can I collect unspecified command line options when using Python's `argh`?

With the Python CLI library argh I want to write a wrapper tool. This wrapper tool is suppose to read the two options -a and -b and to pass all other options to a function (which then calls the wrapped UNIX tool with the left-over options via subprocess).
I have experimented with dispatch's parameter skip_unknown_args:
def wrapper(a=True, b=False):
print("Enter wrapper")
# 1. process a and b
# 2. call_unix_tool(left-over-args)
if __name__ == '__main__':
parser = argh.ArghParser()
argh.set_default_command(parser, wrapper)
argh.dispatch(parser, skip_unknown_args=True)
However the program still does exit when it encounters unknown options and it does not enter the function wrapper as needed. Additionally I don't know where the unknown/skipped arguments are stored, so that I can pass them to the UNIX tool.
How can I tell argh to go into wrapper with the skipped arguments?

I believe this is a bug.
when skip_unknown_args=True, here namespace_obj is a tuple, with a namespace object and remaining args:
(Pdb) p namespace_obj
(ArghNamespace(_functions_stack=[<function wrapper at 0x105cb5e18>], a=False, b=True), ['-c'])
underlying _get_function_from_namespace_obj expects an unary one:
154 function = _get_function_from_namespace_obj(namespace_obj)
...
191 if isinstance(namespace_obj, ArghNamespace):
I checked its coressponding issue and unittest, no idea what is the legitmate behivour the author expects, have dropped a comment there as well.
why not use argparse directly?

You cannot do this with skip_unknown_args=True, because as #georgexsh pointed out the argh library doesn't seem to behave sensibly with that option. However you can provide your own parser class which injects the unknown arguments into the normal namespace:
class ArghParserWithUnknownArgs(argh.ArghParser):
def parse_args(self, args=None, namespace=None):
namespace = namespace or ArghNamespace()
(namespace_obj, unknown_args) = super(ArghParserWithUnknownArgs, self).parse_known_args(args=args, namespace=namespace)
namespace_obj.__dict__['unknown_args'] = unknown_args
return namespace_obj
Note that this class' parse_args method calls ArgParser's parse_known_args method!
With this class defined you can write the wrapper code following way:
def wrapper(a=True, b=False, unknown_args={}):
print("a = %s, b = %s" % (a,b))
print("unknown_args = %s" % unknown_args)
if __name__ == '__main__':
parser = ArghParserWithUnknownArgs()
argh.set_default_command(parser, wrapper)
argh.dispatch(parser)
In your main function wrapper you can access all unknown arguments via the parameter unknown_args and pass this on to your subprocess command
ps: In order to keep the help message tidy decorate wrapper with
#argh.arg('--unknown_args', help=argparse.SUPPRESS)
Addendum: I created an enhanced version of the parser and compiled it into a ready-to-use module. Find it on Github.

how to pass module/method name as command line argument python 2.7.10

I am trying to find a way to add method name as an argument in command line. I am using argparse module in python 2.7.10.
def create_parser():
parser = argparse.ArgumentParser(description='add method name')
parser.add_argument('--method_name', help='pass method name to be used')
return parser
def foo():
return "I am here."
def bar():
return "I am not here."
def where_are_you(method_name):
return method_name()
def main():
return where_are_you(args.method_name)
if __name__ == '__main__':
main()
I am trying to call one of the methods thru command line and getting this error:
"TypeError: 'str' object is not callable." Please guide me here I am new to python.

I've added a dictionary that maps from names to function objects. This dictionary can also be used to limit the choices that the parser accepts.
import argparse
def create_parser():
parser = argparse.ArgumentParser(description='add method name')
parser.add_argument('--method_name', help='pass method name to be used',
choices=adict)
return parser
def foo():
return "I am here."
def bar():
return "I am not here."
adict = {'foo': foo, 'bar': bar} # map names to functions
def where_are_you(method_name):
fn = adict[method_name]
# could also use getattr
return fn()
def main():
args = create_parser().parse_args() # get the string
print(args.method_name) # debugging
return where_are_you(args.method_name)
if __name__ == '__main__':
print(main()) # show the return
testing:
0001:~/mypy$ python stack45667979.py -h
usage: stack45667979.py [-h] [--method_name {foo,bar}]
add method name
optional arguments:
-h, --help show this help message and exit
--method_name {foo,bar}
pass method name to be used
0001:~/mypy$ python stack45667979.py --method_name foo
foo
I am here.
0002:~/mypy$ python stack45667979.py --method_name baz
usage: stack45667979.py [-h] [--method_name {foo,bar}]
stack45667979.py: error: argument --method_name: invalid choice: 'baz' (choose from 'foo', 'bar')
The previous duplicate
Calling a function of a module from a string with the function's name
uses getattr to perform a general search of the module namespace to match a string with a function. That could be used here, but may be too advanced for a beginner, and maybe too open ended even for advanced programmer. It is generally better to limit the choices you give your user to the set of known valid ones.
If this dictionary mapping is confusing I'd suggest a more obvious mapping
if args.method == 'foo':
foo()
elif args.method == 'bar':
bar()
else:
parser.error('bad method choice')

Can two Python argparse objects be combined?

I have an object A which contains parserA - an argparse.ArgumentParser object
There is also object B which contains parserB - another argparse.ArgumentParser
Object A contains an instance of object B, however object B's arguments now need to be parsed by the parser in object A (since A is the one being called from the command line with the arguments, not B)
Is there a way to write in Python object A: parserA += B.parserB?

argparse was developed around objects. Other than a few constants and utility functions it is all class definitions. The documentation focuses on use rather than that class structure. But it may help to understand a bit of that.
parser = argparse.ArgumentParser(...)
creates a parser object.
arg1 = parser.add_argument(...)
creates an argparse.Action (subclass actually) object and adds it to several parser attributes (lists). Normally we ignore the fact that the method returns this Action object, but occasionally I find it helpful. And when I build a parser in an interactive shell I see a this action.
args = parser.parse_args()
runs another method, and returns an namespace object (class argparse.Namespace).
The group methods and subparsers methods also create and return objects (groups, actions and/or parsers).
The ArgumentParser method takes a parents parameter, where the value is a list of parser objects.
With
parsera = argparse.ArgumentParser(parents=[parserb])
during the creation of parsera, the actions and groups in parserb are copied to parsera. That way, parsera will recognize all the arguments that parserb does. I encourage you to test it.
But there are a few qualifications. The copy is by reference. That is, parsera gets a pointer to each Action defined in parserb. Occasionally that creates problems (I won't get into that now). And one or the other has to have add_help=False. Normally a help action is added to a parser at creation. But if parserb also has a help there will be conflict (a duplication) that has to be resolved.
But parents can't be used if parsera has been created independently of parserb. There's no existing mechanism for adding Actions from parserb. It might possible to make a new parser, with both as parents
parserc = argparse.ArgumentParser(parents=[parsera, parserb])
I could probably write a function that would add arguments from parserb to parsera, borrowing ideas from the method that implements parents. But I'd have to know how conflicts are to be resolved.
Look at the argparse._ActionsContainer._add_container_actions to see how arguments (Actions) are copies from a parent to a parser. Something that may be confusing is that each Action is part of a group (user defined or one of the 2 default groups (seen in the help)) in addition to being in a parser.
Another possibility is to use
[argsA, extrasA] = parserA.parse_known_args()
[argsB, extrasB] = parserB.parse_known_args() # uses the same sys.argv
# or
args = parserB.parse_args(extrasA, namespace=argsA)
With this each parser handles the arguments it knows about, and returns the rest in the extras list.
Unless the parsers are designed for this kind of integration, there will be rough edges with this kind of integration. It may be easier to deal with those conficts with Arnial's approach, which is to put the shared argument definitions in your own methods. Others like to put the argument parameters in some sort of database (list, dictionary, etc), and build the parser from that. You can wrap parser creation in as many layers of boilerplate as you find convenient.

You can't use one ArgumentParser inside another. But there is a way around. You need to extract to method code that add arguments to parser.
Then you will be able to use them to merge arguments in parser.
Also it will be easer to group arguments (related to their parsers). But you must be shore that sets of arguments names do not intersect.
Example:
foo.py:
def add_foo_params( group ):
group.add_argument('--foo', help='foo help')
if __name__ = "__main__":
parser = argparse.ArgumentParser(prog='Foo')
boo.py
def add_boo_params( group ):
group.add_argument('--boo', help='boo help')
if __name__ = "__main__":
parser = argparse.ArgumentParser(prog='Boo')
fooboo.py
from foo import add_foo_params
from boo import add_boo_params
if __name__ = "__main__":
parser = argparse.ArgumentParser(prog='FooBoo')
foo_group = parser.add_argument_group(title="foo params")
boo_group = parser.add_argument_group(title="boo params")
add_foo_params( foo_group )
add_boo_params( boo_group )

For your use case, if you can, you could try simply sharing the same argparse object between classes via a dedicated method.
Below is based on what it seems like your situation is.
import argparse
class B(object):
def __init__(self, parserB=argparse.ArgumentParser()):
super(B, self).__init__()
self.parserB = parserB
def addArguments(self):
self.parserB.add_argument("-tb", "--test-b", help="Test B", type=str, metavar="")
#Add more arguments specific to B
def parseArgs(self):
return self.parserB.parse_args()
class A(object):
def __init__(self, parserA=argparse.ArgumentParser(), b=B()):
super(A, self).__init__()
self.parserA = parserA
self.b = b
def addArguments(self):
self.parserA.add_argument("-ta", "--test-a", help="Test A", type=str, metavar="")
#Add more arguments specific to A
def parseArgs(self):
return self.parserA.parse_args()
def mergeArgs(self):
self.b.parserB = self.parserA
self.b.addArguments()
self.addArguments()
Code Explanation:
As stated, in the question, object A and object B contain their own parser objects. Object A also contains an instance of object B.
The code simply separates the anticipated flow into separate methods so that it is possible to keep adding arguments to a single parser before attempting to parse it.
Test Individual
a = A()
a.addArguments()
print(vars(a.parseArgs()))
# CLI Command
python test.py -ta "Testing A"
# CLI Result
{'test_a': 'Testing A'}
Combined Test
aCombined = A()
aCombined.mergeArgs()
print(vars(aCombined.parseArgs()))
# CLI Command
testing -ta "Testing A" -tb "Testing B"
# CLI Result
{'test_b': 'Testing B', 'test_a': 'Testing A'}
Additional
You can also make a general method that takes variable args, and would iterate over and keep adding the args of various classes. I created class C and D for sample below with a general "parser" attribute name.
Multi Test
# Add method to Class A
def mergeMultiArgs(self, *objects):
parser = self.parserA
for object in objects:
object.parser = parser
object.addArguments()
self.addArguments()
aCombined = A()
aCombined.mergeMultiArgs(C(), D())
print(vars(aCombined.parseArgs()))
# CLI Command
testing -ta "Testing A" -tc "Testing C" -td "Testing D"
# CLI Result
{'test_d': 'Testing D', 'test_c': 'Testing C', 'test_a': 'Testing A'}

Yes they can be combined, do this:
Here is a function that merges two args:
def merge_args_safe(args1: Namespace, args2: Namespace) -> Namespace:
"""
Merges two namespaces but throws an error if there are keys that collide.
ref: https://stackoverflow.com/questions/56136549/how-can-i-merge-two-argparse-namespaces-in-python-2-x
:param args1:
:param args2:
:return:
"""
# - the merged args
# The vars() function returns the __dict__ attribute to values of the given object e.g {field:value}.
args = Namespace(**vars(args1), **vars(args2))
return args
test
def merge_args_test():
args1 = Namespace(foo="foo", collided_key='from_args1')
args2 = Namespace(bar="bar", collided_key='from_args2')
args = merge_args(args1, args2)
print('-- merged args')
print(f'{args=}')
output:
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1202, in <module>
merge_args_test()
File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1192, in merge_args_test
args = merge_args(args1, args2)
File "/Users/brando/ultimate-utils/ultimate-utils-proj-src/uutils/__init__.py", line 1116, in merge_args
args = Namespace(**vars(args1), **vars(args2))
TypeError: argparse.Namespace() got multiple values for keyword argument 'collided_key'
python-BaseException
you can find it in this library: https://github.com/brando90/ultimate-utils
If you want to have collisions resolved do this:
def merge_two_dicts(starting_dict: dict, updater_dict: dict) -> dict:
"""
Starts from base starting dict and then adds the remaining key values from updater replacing the values from
the first starting/base dict with the second updater dict.
For later: how does d = {**d1, **d2} replace collision?
:param starting_dict:
:param updater_dict:
:return:
"""
new_dict: dict = starting_dict.copy() # start with keys and values of starting_dict
new_dict.update(updater_dict) # modifies starting_dict with keys and values of updater_dict
return new_dict
def merge_args(args1: Namespace, args2: Namespace) -> Namespace:
"""
ref: https://stackoverflow.com/questions/56136549/how-can-i-merge-two-argparse-namespaces-in-python-2-x
:param args1:
:param args2:
:return:
"""
# - the merged args
# The vars() function returns the __dict__ attribute to values of the given object e.g {field:value}.
merged_key_values_for_namespace: dict = merge_two_dicts(vars(args1), vars(args2))
args = Namespace(**merged_key_values_for_namespace)
return args
test:
def merge_args_test():
args1 = Namespace(foo="foo", collided_key='from_args1')
args2 = Namespace(bar="bar", collided_key='from_args2')
args = merge_args(args1, args2)
print('-- merged args')
print(f'{args=}')
assert args.collided_key == 'from_args2', 'Error in merge dict, expected the second argument to be the one used' \
'to resolve collision'

Python cmd module - parsing values from line

I am working on a quick python script using the cmd module that will allow the user to enter text commands followed by parameters in basic url query string format. The prompts will be answered with something like
commandname foo=bar&baz=brack
Using cmd, I can't seem to find which method to override to affect the way the argument line is handed off to all the do_* methods. I want to run urlparse.parse_qs on these values, and calling this upon line in every do_* method seems clumsy.
The precmd method gets the whole line, before the commandname is split off and interpreted, so this will not work for my purposes. I'm also not terribly familiar with how to place a decorator inside a class like this and haven't been able to pull it off without breaking the scope.
Basically, the python docs for cmd say the following
Repeatedly issue a prompt, accept input, parse an initial prefix off
the received input, and dispatch to action methods, passing them the
remainder of the line as argument.
I want to make a method that will do additional processing to that "remainder of the line" and hand that generated dictionary off to the member functions as the line argument, rather than interpreting them in every function.
Thanks!

You could potentially override the onecmd() method, as the following quick example shows. The onecmd() method there is basically a copy of the one from the original cmd.py, but adds a call to urlparse.parse_qs() before passing the arguments to a function.
import cmd
import urlparse
class myCmd(cmd.Cmd):
def onecmd(self, line):
"""Mostly ripped from Python's cmd.py"""
cmd, arg, line = self.parseline(line)
arg = urlparse.parse_qs(arg) # <- added line
if not line:
return self.emptyline()
if cmd is None:
return self.default(line)
self.lastcmd = line
if cmd == '':
return self.default(line)
else:
try:
func = getattr(self, 'do_' + cmd)
except AttributeError:
return self.default(line)
return func(arg)
def do_foo(self, arg)
print arg
my_cmd = myCmd()
my_cmd.cmdloop()
Sample output:
(Cmd) foo
{}
(Cmd) foo a b c
{}
(Cmd) foo a=b&c=d
{'a': ['b'], 'c': ['d']}
Is this what you are trying to achieve?
Here's another potential solution that uses a class decorator to modify a
cmd.Cmd subclass and basically apply a decorator function to all do_*
methods of that class:
import cmd
import urlparse
import types
# function decorator to add parse_qs to individual functions
def parse_qs_f(f):
def f2(self, arg):
return f(self, urlparse.parse_qs(arg))
return f2
# class decorator to iterate over all attributes of a class and apply
# the parse_qs_f decorator to all do_* methods
def parse_qs(cls):
for attr_name in dir(cls):
attr = getattr(cls, attr_name)
if attr_name.startswith('do_') and type(attr) == types.MethodType:
setattr(cls, attr_name, parse_qs_f(attr))
return cls
#parse_qs
class myCmd(cmd.Cmd):
def do_foo(self, args):
print args
my_cmd = myCmd()
my_cmd.cmdloop()
I quickly cobbled this together and it appears to work as intended, however, I'm
open to suggestions on any pitfalls or how this solution could be improved.