Develop Reference

Python convert dictionary to argparse

Right now, I have a script that can accept command line arguments using argparse. For example, like this:
#foo.py
def function_with_args(optional_args=None):
parser = argparse.ArgumentParser()
# add some arguments
args = parser.parse_args(optional_args)
# do something with args
However, I'd like to be able to use this function with a dictionary instead, for example with something like this:
def function_using_dict(**kwargs):
# define parser and add some arguments
args = parser.parse_dict_args(kwargs)
# everything else is the same
Note that I have a lot of arguments with default values in argparse which I'd like to use, so the following wouldn't work:
def function_no_default_args(**kwargs):
args = kwargs # not using default values that we add to the parser!

argparse.Namespace is a relatively simple object subclass, with most of its code devoted to displaying the attributes (as print(args) shows). Internally parse_args uses get_attr and set_attr to access the namespace, minimizing the assumptions about attributes names.
When using subparsers, the subparser starts with a 'blank' namespace, and uses the following code to copy its values to the main namespace.
# In case this subparser defines new defaults, we parse them
# in a new namespace object and then update the original
# namespace for the relevant parts.
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)
Originally the main namespace was passed to the subparser, eg. parser.parse_known_args(arg_strings, namespace), but the current version lets the subparser defaults take priority.
Handling defaults is a bit complicated. If you don't have any required arguments then
args = parser.parse_args([])
will set all the defaults. Or you could look at the start of parse.parse_known_args to see how defaults are inserted into the namespace at the start of parsing. Just beware that there's an added step at the end of parsing that runs remaining defaults through their respective type functions.

If you are trying to convert the result of parse_args into a dict, you can probably just do this:
kwargs = vars(args)
After your comment, I thought about it. Going to go with your existing function.
#foo.py
def function_with_args_and_default_kwargs(optional_args=None, **kwargs):
parser = argparse.ArgumentParser()
# add some arguments
# add the other arguments
for k, v in kwargs.items():
parser.add_argument('--' + k, default=v)
args = parser.parse_args(optional_args)
# do something with args

Parameter 'from' in python3: formal parameter name expected [duplicate]

I am using the Click library but I can't seem to find a behavior similar to dest from argparse.
For example, I have
#click.option('--format', type=click.Choice(['t', 'j']))
def plug(format):
pass
Notice that I am using a flag with --format that gets translated into a built-in Python construct format which is not ideal.
Is there a way to change the argument passed into the click function for options?

While Click doesn't have dest-equivalent of argparse, it has certain argument-naming behavior which can be exploited. Specifically, for parameters with multiple possible names, it will prefer non-dashed to dashed names, and as secondary preference will prioritize longer names over shorter names.
URL: http://click.pocoo.org/dev/parameters/#parameter-names
So if you declare your option as...
#click.option('--format', 'not-format', type=click.Choice(['t', 'j']))
...then Click will prioritize non-dashed variant ('not-format') and call your function with not_format=... argument.
Of course it also means that this alternative spelling can also be used in command line. If that is not desired, then I guess you could add a decorator to rename keyword arguments:
import functools
def rename_kwargs(**replacements):
def actual_decorator(func):
#functools.wraps(func)
def decorated_func(*args, **kwargs):
for internal_arg, external_arg in replacements.iteritems():
if external_arg in kwargs:
kwargs[internal_arg] = kwargs.pop(external_arg)
return func(*args, **kwargs)
return decorated_func
return actual_decorator
Testing code:
if __name__ == '__main__':
#rename_kwargs(different_arg='format')
def tester(different_arg):
print different_arg
tester(format='test value')
Test output:
$ python test_decor.py
test value
In your case, it would look like:
#click.option('--format', type=click.Choice(['t', 'j']))
#replace_kwargs(not_format='format')
def plug(not_format):
pass

Renaming an option to a differently named function argument is possible by decorating the function with
#click.option('--format', '-f', 'format_arg_name')
def plug(format_arg_name):
print(format_arg_name)
then it will remap the option named format and make it available as the format_arg_name parameter.
format_arg_name will not be available as a command line option, but --format and -f are.

Python: Typehints for argparse.Namespace objects

Is there a way to have Python static analyzers (e.g. in PyCharm, other IDEs) pick up on Typehints on argparse.Namespace objects? Example:
parser = argparse.ArgumentParser()
parser.add_argument('--somearg')
parsed = parser.parse_args(['--somearg','someval']) # type: argparse.Namespace
the_arg = parsed.somearg # <- Pycharm complains that parsed object has no attribute 'somearg'
If I remove the type declaration in the inline comment, PyCharm doesn't complain, but it also doesn't pick up on invalid attributes. For example:
parser = argparse.ArgumentParser()
parser.add_argument('--somearg')
parsed = parser.parse_args(['--somearg','someval']) # no typehint
the_arg = parsed.somaerg # <- typo in attribute, but no complaint in PyCharm. Raises AttributeError when executed.
Any ideas?
Update
Inspired by Austin's answer below, the simplest solution I could find is one using namedtuples:
from collections import namedtuple
ArgNamespace = namedtuple('ArgNamespace', ['some_arg', 'another_arg'])
parser = argparse.ArgumentParser()
parser.add_argument('--some-arg')
parser.add_argument('--another-arg')
parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: ArgNamespace
x = parsed.some_arg # good...
y = parsed.another_arg # still good...
z = parsed.aint_no_arg # Flagged by PyCharm!
While this is satisfactory, I still don't like having to repeat the argument names. If the argument list grows considerably, it will be tedious updating both locations. What would be ideal is somehow extracting the arguments from the parser object like the following:
parser = argparse.ArgumentParser()
parser.add_argument('--some-arg')
parser.add_argument('--another-arg')
MagicNamespace = parser.magically_extract_namespace()
parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: MagicNamespace
I haven't been able to find anything in the argparse module that could make this possible, and I'm still unsure if any static analysis tool could be clever enough to get those values and not bring the IDE to a grinding halt.
Still searching...
Update 2
Per hpaulj's comment, the closest thing I could find to the method described above that would "magically" extract the attributes of the parsed object is something that would extract the dest attribute from each of the parser's _actions.:
parser = argparse.ArgumentParser()
parser.add_argument('--some-arg')
parser.add_argument('--another-arg')
MagicNamespace = namedtuple('MagicNamespace', [act.dest for act in parser._actions])
parsed = parser.parse_args(['--some-arg', 'val1', '--another-arg', 'val2']) # type: MagicNamespace
But this still does not cause attribute errors to get flagged in static analysis. This is true also true if I pass namespace=MagicNamespace in the parser.parse_args call.

Typed argument parser was made for exactly this purpose. It wraps argparse. Your example is implemented as:
from tap import Tap
class ArgumentParser(Tap):
somearg: str
parsed = ArgumentParser().parse_args(['--somearg', 'someval'])
the_arg = parsed.somearg
Here's a picture of it in action.
It's on PyPI and can be installed with: pip install typed-argument-parser
Full disclosure: I'm one of the creators of this library.

Consider defining an extension class to argparse.Namespace that provides the type hints you want:
class MyProgramArgs(argparse.Namespace):
def __init__():
self.somearg = 'defaultval' # type: str
Then use namespace= to pass that to parse_args:
def process_argv():
parser = argparse.ArgumentParser()
parser.add_argument('--somearg')
nsp = MyProgramArgs()
parsed = parser.parse_args(['--somearg','someval'], namespace=nsp) # type: MyProgramArgs
the_arg = parsed.somearg # <- Pycharm should not complain

Most of these answers involve using another package to handle the typing. This would be a good idea only if there wasn't such a simple solution as the one I am about to propose.
Step 1. Type Declarations
First, define the types of each argument in a dataclass like so:
from dataclasses import dataclass
#dataclass
class MyProgramArgs:
first_var: str
second_var: int
Step 2. Argument Declarations
Then you can set up your parser however you like with matching arguments. For example:
import argparse
parser = argparse.ArgumentParser("This CLI program uses type hints!")
parser.add_argument("-a", "--first-var")
parser.add_argument("-b", "--another-var", type=int, dest="second_var")
Step 3. Parsing the Arguments
And finally, we parse the arguments in a way that the static type checker will know about the type of each argument:
my_args = MyProgramArgs(**vars(parser.parse_args())
Now the type checker knows that my_args is of type MyProgramArgs so it knows exactly which fields are available and what their type is.

I don't know anything about how PyCharm handles these typehints, but understand the Namespace code.
argparse.Namespace is a simple class; essentially an object with a few methods that make it easier to view the attributes. And for ease of unittesting it has a __eq__ method. You can read the definition in the argparse.py file.
The parser interacts with the namespace in the most general way possible - with getattr, setattr, hasattr. So you can use almost any dest string, even ones you can't access with the .dest syntax.
Make sure you don't confuse the add_argument type= parameter; that's a function.
Using your own namespace class (from scratch or subclassed) as suggested in the other answer may be the best option. This is described briefly in the documentation. Namespace Object. I haven't seen this done much, though I've suggested it a few times to handle special storage needs. So you'll have to experiment.
If using subparsers, using a custom Namespace class may break, http://bugs.python.org/issue27859
Pay attention to handling of defaults. The default default for most argparse actions is None. It is handy to use this after parsing to do something special if the user did not provide this option.
if args.foo is None:
# user did not use this optional
args.foo = 'some post parsing default'
else:
# user provided value
pass
That could get in the way type hints. Whatever solution you try, pay attention to the defaults.
A namedtuple won't work as a Namespace.
First, the proper use of a custom Namespace class is:
nm = MyClass(<default values>)
args = parser.parse_args(namespace=nm)
That is, you initial an instance of that class, and pass it as the parameter. The returned args will be the same instance, with new attributes set by parsing.
Second, a namedtuple can only created, it can't be changed.
In [72]: MagicSpace=namedtuple('MagicSpace',['foo','bar'])
In [73]: nm = MagicSpace(1,2)
In [74]: nm
Out[74]: MagicSpace(foo=1, bar=2)
In [75]: nm.foo='one'
...
AttributeError: can't set attribute
In [76]: getattr(nm, 'foo')
Out[76]: 1
In [77]: setattr(nm, 'foo', 'one') # not even with setattr
...
AttributeError: can't set attribute
A namespace has to work with getattr and setattr.
Another problem with namedtuple is that it doesn't set any kind of type information. It just defines field/attribute names. So there's nothing for the static typing to check.
While it is easy to get expected attribute names from the parser, you can't get any expected types.
For a simple parser:
In [82]: parser.print_usage()
usage: ipython3 [-h] [-foo FOO] bar
In [83]: [a.dest for a in parser._actions[1:]]
Out[83]: ['foo', 'bar']
In [84]: [a.type for a in parser._actions[1:]]
Out[84]: [None, None]
The Actions dest is the normal attribute name. But type is not the expected static type of that attribute. It is a function that may or may not convert the input string. Here None means the input string is saved as is.
Because static typing and argparse require different information, there isn't an easy way to generate one from the other.
I think the best you can do is create your own database of parameters, probably in a dictionary, and create both the Namespace class and the parsesr from that, with your own utility function(s).
Let's say dd is dictionary with the necessary keys. Then we can create an argument with:
parser.add_argument(dd['short'],dd['long'], dest=dd['dest'], type=dd['typefun'], default=dd['default'], help=dd['help'])
You or someone else will have to come up with a Namespace class definition that sets the default (easy), and static type (hard?) from such a dictionary.

If you are in a situation where you can start from scratch there are interesting solutions like
the already mentioned typed argument parser (TAP)
typer
typed-args
However, in my case they weren't an ideal solution because:
I have many existing CLIs based on argparse, and I cannot afford to re-write them all using such args-inferred-from-types approaches.
When inferring args from types it can be tricky to support all advanced CLI features that plain argparse supports.
Re-using common arg definitions in multiple CLIs is often easier in plain imperative argparse compared to alternatives.
Therefore I worked on a tiny library typed_argparse that allows to introduce typed args without much refactoring. The idea is to add a type derived from a special TypedArg class, which then simply wraps the plain argparse.Namespace object:
# Step 1: Add an argument type.
class MyArgs(TypedArgs):
foo: str
num: Optional[int]
files: List[str]
def parse_args(args: List[str] = sys.argv[1:]) -> MyArgs:
parser = argparse.ArgumentParser()
parser.add_argument("--foo", type=str, required=True)
parser.add_argument("--num", type=int)
parser.add_argument("--files", type=str, nargs="*")
# Step 2: Wrap the plain argparser result with your type.
return MyArgs(parser.parse_args(args))
def main() -> None:
args = parse_args(["--foo", "foo", "--num", "42", "--files", "a", "b", "c"])
# Step 3: Done, enjoy IDE auto-completion and strong type safety
assert args.foo == "foo"
assert args.num == 42
assert args.files == ["a", "b", "c"]
This approach slightly violates the single-source-of-truth principle, but the library performs a full runtime validation to ensure that the type annotations match the argparse types, and it is just a very simple option to migrate towards typed CLIs.

Another way to do it which could be ideal if you have few arguments is as follows.
First make a function that sets up the parser and returns the namespace. For example:
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("-a")
parser.add_argument("-b", type=int)
return parser.parse_args()
Then you define a main function which takes the args you declared above individually; like so.
def main(a: str, b: int):
print("hello world", a, b)
And when you call your main, you do it like this:
if __name__ == "__main__":
main(**vars(parse_args())
From your main onwards, you'll have your variables a and b properly recognised by your static type checker, although you won't have an object any more containing all your arguments, which may be a good or bad thing depending on your use case.

a super solution to just type hint the NameSpace return value of parse_args method.
import argparse
from typing import Type
class NameSpace(argparse.Namespace, Type):
name: str
class CustomParser(argparse.ArgumentParser):
def parse_args(self) -> NameSpace:
return super().parse_args()
parser = CustomParser()
parser.add_argument("--name")
if __name__ == "__main__":
args = parser.parse_args()
print(args.name)

Triggering callback on default value in optparse

I'm using Python's optparse to do what it does best, but I can't figure out how to make the option callback trigger on the default argument value if no other is specified via command-line; is this even possible? This would make my code much cleaner.
I can't use argparse unfortunately, as the platform I'm running on has an outdated Python version.
Edit:
To provide more detail, I'm adding an option with a callback and a default value
parser.add_option(
"-f",
"--format",
type = "string",
action = "callback",
callback = format_callback,
default = "a,b,c,d")
The callback function is defined as follows:
def format_callback(option, opt, value, parser):
# some processing
parser.values.V = processed_value
Basically I'm processing the "--format" value and putting the result into the parser. This works fine, when "--format" is specified directly via command-line, but I'd like the callback to be triggered on the default "a,b,c,d" value as well.

It is simply not possible.
The optparse.OptionParser implementation of parse_args starts with:
def parse_args(self, args=None, values=None):
"""
parse_args(args : [string] = sys.argv[1:],
values : Values = None)
-> (values : Values, args : [string])
Parse the command-line options found in 'args' (default:
sys.argv[1:]). Any errors result in a call to 'error()', which
by default prints the usage message to stderr and calls
sys.exit() with an error message. On success returns a pair
(values, args) where 'values' is an Values instance (with all
your option values) and 'args' is the list of arguments left
over after parsing options.
"""
rargs = self._get_args(args)
if values is None:
values = self.get_default_values()
Default values are set before processing any arguments. Actual values then overwrite defaults as options are parsed; the option callbacks are called when a corresponding argument is found.
So callbacks simply cannot be invoked for defaults. The design of the optparse module makes this very hard to change.

You can inject the default when calling parse_args
options, args = parser.parse_args(args=["--option=default"] + sys.argv[1:])
Since flags passed later in the argument list override those passed earlier, this should work. It's possible you may need to modify your callback function to expect this depending on what it is doing.

Passing optional arguments from optparse

I'm trying to figure out how to pass optional arguments from optparse. The problem I'm having is if an optparse option is not specified, it defaults to a None type, but if I pass the None type into a function, it yells at me instead of using the default (Which is understandable and valid).
conn = psycopg2.connect(database=options.db, hostname=options.hostname, port=options.port)
The question is, how do I use the function's defaults for optional arguments but still pass in user inputs if there is an input without having a huge number of if statements.

Define a function remove_none_values that filters a dictionary for none-valued arguments.
def remove_none_values(d):
return dict((k,v) for (k,v) in d.iteritems() if not v is None)
kwargs = {
'database': options.db,
'hostname': options.hostname,
...
}
conn = psycopg2.connect(**remove_none_values(kwargs))
Or, define a function wrapper that removes none values before passing the data on to the original function.
def ignore_none_valued_kwargs(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
newkwargs = dict((k,v) for (k,v) in d.iteritems() if not v is None)
return f(*args, **kwargs)
return wrapper
my_connect = ignore_none_valued_kwargs(psycopg2)
conn = my_connect(database=options.db, hostname=options.hostname, port=options.port)

The opo module of my thebops package (pip install thebops, https://bitbucket.org/therp/thebops) contains an add_optval_option function.
This uses an additional keyword argument empty which specifies the value to use if the option is used without a value. If one of the option strings is found in the commandline, this value is injected into the argument list.
This is still hackish, but at least it is made a simple-to-use function ...
It works well under the following circumstances:
The argument vector does already exist when the option is created. This is usually true.
All programs I found which sport arguments with optional values require the given value to be attached as --option=value or -ovalue rather than --option value or -o value.
Maybe I'll tweak thebops.optparse to support the empty argument as well; but I'd like to have a test suite first to prevent regressions, preferably the original Optik / optparse tests.
This is the code:
from sys import argv
def add_optval_option(pog, *args, **kwargs):
"""
Add an option which can be specified without a value;
in this case, the value (if given) must be contained
in the same argument as seen by the shell,
i.e.:
--option=VALUE, --option will work;
--option VALUE will *not* work
Arguments:
pog -- parser or group
empty -- the value to use when used without a value
Note:
If you specify a short option string as well, the syntax given by the
help will be wrong; -oVALUE will be supported, -o VALUE will not!
Thus it might be wise to create a separate option for the short
option strings (in a "hidden" group which isn't added to the parser after
being populated) and just mention it in the help string.
"""
if 'empty' in kwargs:
empty_val = kwargs.pop('empty')
# in this case it's a good idea to have a <default> value; this can be
# given by another option with the same <dest>, though
for i in range(1, len(argv)):
a = argv[i]
if a == '--':
break
if a in args:
argv.insert(i+1, empty_val)
break
pog.add_option(*args, **kwargs)