I'm using Python's optparse to do what it does best, but I can't figure out how to make the option callback trigger on the default argument value if no other is specified via command-line; is this even possible? This would make my code much cleaner.
I can't use argparse unfortunately, as the platform I'm running on has an outdated Python version.
Edit:
To provide more detail, I'm adding an option with a callback and a default value
parser.add_option(
"-f",
"--format",
type = "string",
action = "callback",
callback = format_callback,
default = "a,b,c,d")
The callback function is defined as follows:
def format_callback(option, opt, value, parser):
# some processing
parser.values.V = processed_value
Basically I'm processing the "--format" value and putting the result into the parser. This works fine, when "--format" is specified directly via command-line, but I'd like the callback to be triggered on the default "a,b,c,d" value as well.
It is simply not possible.
The optparse.OptionParser implementation of parse_args starts with:
def parse_args(self, args=None, values=None):
"""
parse_args(args : [string] = sys.argv[1:],
values : Values = None)
-> (values : Values, args : [string])
Parse the command-line options found in 'args' (default:
sys.argv[1:]). Any errors result in a call to 'error()', which
by default prints the usage message to stderr and calls
sys.exit() with an error message. On success returns a pair
(values, args) where 'values' is an Values instance (with all
your option values) and 'args' is the list of arguments left
over after parsing options.
"""
rargs = self._get_args(args)
if values is None:
values = self.get_default_values()
Default values are set before processing any arguments. Actual values then overwrite defaults as options are parsed; the option callbacks are called when a corresponding argument is found.
So callbacks simply cannot be invoked for defaults. The design of the optparse module makes this very hard to change.
You can inject the default when calling parse_args
options, args = parser.parse_args(args=["--option=default"] + sys.argv[1:])
Since flags passed later in the argument list override those passed earlier, this should work. It's possible you may need to modify your callback function to expect this depending on what it is doing.
Related
Right now, I have a script that can accept command line arguments using argparse. For example, like this:
#foo.py
def function_with_args(optional_args=None):
parser = argparse.ArgumentParser()
# add some arguments
args = parser.parse_args(optional_args)
# do something with args
However, I'd like to be able to use this function with a dictionary instead, for example with something like this:
def function_using_dict(**kwargs):
# define parser and add some arguments
args = parser.parse_dict_args(kwargs)
# everything else is the same
Note that I have a lot of arguments with default values in argparse which I'd like to use, so the following wouldn't work:
def function_no_default_args(**kwargs):
args = kwargs # not using default values that we add to the parser!
argparse.Namespace is a relatively simple object subclass, with most of its code devoted to displaying the attributes (as print(args) shows). Internally parse_args uses get_attr and set_attr to access the namespace, minimizing the assumptions about attributes names.
When using subparsers, the subparser starts with a 'blank' namespace, and uses the following code to copy its values to the main namespace.
# In case this subparser defines new defaults, we parse them
# in a new namespace object and then update the original
# namespace for the relevant parts.
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)
Originally the main namespace was passed to the subparser, eg. parser.parse_known_args(arg_strings, namespace), but the current version lets the subparser defaults take priority.
Handling defaults is a bit complicated. If you don't have any required arguments then
args = parser.parse_args([])
will set all the defaults. Or you could look at the start of parse.parse_known_args to see how defaults are inserted into the namespace at the start of parsing. Just beware that there's an added step at the end of parsing that runs remaining defaults through their respective type functions.
If you are trying to convert the result of parse_args into a dict, you can probably just do this:
kwargs = vars(args)
After your comment, I thought about it. Going to go with your existing function.
#foo.py
def function_with_args_and_default_kwargs(optional_args=None, **kwargs):
parser = argparse.ArgumentParser()
# add some arguments
# add the other arguments
for k, v in kwargs.items():
parser.add_argument('--' + k, default=v)
args = parser.parse_args(optional_args)
# do something with args
I have the following snippet:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--p', required=True)
parser.add_argument('arg', action=MyAction)
parser.parse_args()
, where MyAction is a simple custom action class.
As you see, I want to enforce the presence of the p argument. However, my action is performed even if the argument is not present, and then an error message is shown that indicates the fact the the argument is missing.
Obviously, I could check in my action class for the presence of the p argument, but this defies the purpose of having a required parameter in the first place. Why is my action being run if the argument is not present?
Parsing is driven by the commandline strings, and tries to be order agnostic. Within those rules, it alternates between parsing an optional and a positional.
For example, with myprog --p one two:
'--p' - pass the 'one' string to the p action (e.g. setattr(namespace, 'p', 'one')
'two' - matches the nargs for 'arg'. Calls your MyAction.__call__ with values='one'.
at the end of parsing it checks if all required actions have been 'seen'. With your setup both '--p' and 'arg' are required.
With myprog two --p one it does the same, except arg is processed first. The namespace may have a default value for p.
With myprog two, arg is processed, and the required test will raise an error. error: the following arguments are required: --p
Since you have written a custom Action, you can easily explore how the namespace contents vary depending on the commandline arguments and their order.
So the --p and the arg will be processed independently, and in either order, depending on the commandline strings. required testing is performed at the end, using a list of seen_actions. And default values are set at the start of parsing. It is difficult to implement reliable inter-action tests within custom Actions. Usually it is better to perform such tests after parsing.
The defined Actions only change the args namespace. So parsing does not change anything beyond what it returns. Unless there's an error and it forces an sys.exit. A custom MyAction class can change that, but at your own risk.
I'm trying to use the default value in Python argparse instead of the user specified argument. For example, here's an argument:
parser.add_argument('--fin', default='file.txt')
If a user calls --fin dog, that's obviously not a file, so I want to be able to use the default instead. How would I do that? Does argparse have a function that uses the default instead of the input argument? Also sorry for any typing mistakes, I'm doing this on a phone.
There's no way to access default arguments using the return value from parser.parse_args(). More importantly, how is "dog" obviously not a file? That's a perfectly valid filename on any modern operating system; file extensions are common but they are by no means required.
The only way to determine if something is or is not a file is by trying to open it. If it fails, either it wasn't a file or you don't have access to it.
In this case, the best solution may be something like:
DEFAULT_FIN = 'file.txt'
parser.add_argument('--fin', default=DEFAULT_FIN)
And later on:
try:
fd = open(args.fin)
except IOError:
fd = open(DEFAULT_FIN)
Although I would argue that if the user specifies a filename on the command line and you are unable to open it, then you should print an error and exit.
You can:
argument_file = parser.add_argument('--fin', default='file.txt')
args = parser.parse_args()
try:
fo = open(args.fin)
except:
fo = open(argument_file.default)
print fo.read()
But solution by larsks with constant seems better.
It is possible to find the default value of an argument if you hang onto the identifier of the argument when you define it. e.g.
In [119]: parser=argparse.ArgumentParser()
In [120]: arg1 = parser.add_argument('--fin', default='default.txt')
In [121]: arg1.default
Out[121]: 'default.txt'
arg1, the value returned by the add_argument method is the Action object that the parser uses. It has all the information that you provided to the method. arg1.default is just one of its attributes.
So you could use arg1.default in your post parse_args code that checks whether args.fin is a valid file or not.
larsks approach is just as good, since you, the user, are defining the default in the first place.
You could also write a custom Action class, or argument type that does this checking. There is a builtin argparse.FileType that tries to open the file, and raises an ArgumentType error if it can't. That could be modified to use the default instead. But this a more advanced solution, and isn't obviously superior to doing your own checking after parsing.
I have written a python module mymod.py that can be used also as a standalone program from command line.
In mymod.py I have defined a few functions (where default values are set using keywords)
and an if __name__=="__main__" block to use the module as a standalone program.
I want the possibility to override some of the default options, therefore in the main program I import argparse and use it to parse the options. I use a dictionary to store
the default values, so that if some day I need to change the default values I can easily
do it modifying its value in one place only.
It works, but I find that the code is not "clean" and thought that probably I am not doing it in the proper pythonic way.
This is a toy example to show what I do:
#!/usr/bin/env python
#mymod.py
__default_options__={
"f1_x":10,
"f2_x":10
}
def f1(x=__default_options__["f1_x"]):
return x**2
def f2(x=__default_options__["f2_x"]):
return x**4
# this function is the "core" function which uses f1 and f2
# to produce the main task of the program
def f(x=__default_options__["f1_x"],y=__default_options__["f2_x"]):
return f1(x)+f2(y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application")
parser.add_argument("--f1-x",help="the parameter passed to f1",
default=__default_options__["f1_x"], type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
default=__default_options__["f2_x"], type = float, dest = "y")
options= parser.parse_args()
print f(options.x,options.y)
Passing the default values like I do it is a bit cumbersome and probably against the spirit both of Python and argparse.
How can this code be improved to be more pythonic and use argparse at its best?
You can use the `ArgumentParser.set_defaults method, in the following way
default_options={
"x":10,
"y":10
}
def f1(**kwargs):
x=kwargs.get('x', defalut_options['x'])
return x**2
def f2(**kwargs):
y=kwargs.get('y', defalut_options['y'])
return x**4
def f(**kwargs):
x=kwargs.get('x', defalut_options['x'])
y=kwargs.get('y', defalut_options['y'])
return f1(x=x, y=y)
if __name__=="__main__":
import argparse
parser = argparse.ArgumentParser(description = "A toy application", formatter_class=argparse.ArgumentDefaultsHelpFormatter )
parser.add_argument("--f1-x",help="the parameter passed to f1",
type = float,dest = "x")
parser.add_argument("--f2-x",help="the parameter passed to f2",
type = float, dest = "y")
parser.set_defaults(**default_options)
options= parser.parse_args()
print f(options.x,options.y)
It took me a while to make it work, because I didn't notice that you are using dest in add_argument (I never use it). If this keyword is not provided, argparse set the default dest to the long name of the argument (in this case f1_x and f2_x, as it substitutes - with _). To go to the point: if you want to provide a dictionary of defaults, the keys needs to match dest if provided. Besides, take care that parser.set_defaults just add arguments to the parser, so if you have some entry not in your parser, it will be added to the Namespace.
--Edited to add generic kwargs to the functions--
As #Francesco wrote in a comment, your defaults dictionary won't work as you probably intended: The functions will retain the defaults they had while loading the module, regardless of later changes to the dictionary. Here's how to make them track the current value of the dictionary:
_default_options = {
"f1_x":10,
"f2_x":10
}
def f1(x=None):
if x == None:
x = _default_options["f1_x"]
...
You can then modify _default_options via ArgumentParser, or in any other way, and f1() will use it if called with no arguments.
This requires that None could never be a meaningful value for x; if that's not the case, choose a suitable impossible value.
I'm trying to figure out how to pass optional arguments from optparse. The problem I'm having is if an optparse option is not specified, it defaults to a None type, but if I pass the None type into a function, it yells at me instead of using the default (Which is understandable and valid).
conn = psycopg2.connect(database=options.db, hostname=options.hostname, port=options.port)
The question is, how do I use the function's defaults for optional arguments but still pass in user inputs if there is an input without having a huge number of if statements.
Define a function remove_none_values that filters a dictionary for none-valued arguments.
def remove_none_values(d):
return dict((k,v) for (k,v) in d.iteritems() if not v is None)
kwargs = {
'database': options.db,
'hostname': options.hostname,
...
}
conn = psycopg2.connect(**remove_none_values(kwargs))
Or, define a function wrapper that removes none values before passing the data on to the original function.
def ignore_none_valued_kwargs(f):
#functools.wraps(f)
def wrapper(*args, **kwargs):
newkwargs = dict((k,v) for (k,v) in d.iteritems() if not v is None)
return f(*args, **kwargs)
return wrapper
my_connect = ignore_none_valued_kwargs(psycopg2)
conn = my_connect(database=options.db, hostname=options.hostname, port=options.port)
The opo module of my thebops package (pip install thebops, https://bitbucket.org/therp/thebops) contains an add_optval_option function.
This uses an additional keyword argument empty which specifies the value to use if the option is used without a value. If one of the option strings is found in the commandline, this value is injected into the argument list.
This is still hackish, but at least it is made a simple-to-use function ...
It works well under the following circumstances:
The argument vector does already exist when the option is created. This is usually true.
All programs I found which sport arguments with optional values require the given value to be attached as --option=value or -ovalue rather than --option value or -o value.
Maybe I'll tweak thebops.optparse to support the empty argument as well; but I'd like to have a test suite first to prevent regressions, preferably the original Optik / optparse tests.
This is the code:
from sys import argv
def add_optval_option(pog, *args, **kwargs):
"""
Add an option which can be specified without a value;
in this case, the value (if given) must be contained
in the same argument as seen by the shell,
i.e.:
--option=VALUE, --option will work;
--option VALUE will *not* work
Arguments:
pog -- parser or group
empty -- the value to use when used without a value
Note:
If you specify a short option string as well, the syntax given by the
help will be wrong; -oVALUE will be supported, -o VALUE will not!
Thus it might be wise to create a separate option for the short
option strings (in a "hidden" group which isn't added to the parser after
being populated) and just mention it in the help string.
"""
if 'empty' in kwargs:
empty_val = kwargs.pop('empty')
# in this case it's a good idea to have a <default> value; this can be
# given by another option with the same <dest>, though
for i in range(1, len(argv)):
a = argv[i]
if a == '--':
break
if a in args:
argv.insert(i+1, empty_val)
break
pog.add_option(*args, **kwargs)