I have a Python script that reads a file containing a command line invocation of some other tool. I'd like to modify the options of this invocation before calling the tool. For example, I might transform:
my_util --input file1.txt --option1 red --option2 blue
...to this:
my_util --input file1_001.txt --option1 red --option3 green
(More accurately, I'd be working on the arguments as lists.)
I figured that using the argparse module would be the easiest way to do this: I could parse the args, change, add or remove the options as I need to, and then reconstruct the command line.
But how do I do the last step? Given the Namespace object returned by parse_args(), can I easily reconstruct a list of command line options, such as could be passed to subprocess.Popen()?
A Namespace object is just a simple object subclass, so you can get the values out as a dict with vars:
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> args = parser.parse_args(['--foo', 'BAR'])
>>> vars(args)
{'foo': 'BAR'}
Or you can assign to a class directly and get the arguments out as class variables:
>>> class C(object):
... pass
...
>>> c = C()
>>> parser = argparse.ArgumentParser()
>>> parser.add_argument('--foo')
>>> parser.parse_args(args=['--foo', 'BAR'], namespace=c)
>>> c.foo
'BAR'
It would be fairly easy to use either of these structures to test/replace arguments and pass the results to Popen.
I know this is an old question, but I've just encountered the same problem. I realized that all I need is a way to iterate over the Action objects. Unfortunately, the internal list is not exposed by ArgParser itself. However, these objects are returned by add_argument(), so I can construct my own list. Well, putting actions.append() around each call looked like too much typing to me, so I store all options in a tuple:
def add_argument(*args, **kwargs):
return (args, kwargs)
parser = argparse.ArgumentParser()
options = (
add_argument('--verbose', action='store_true'),
add_argument('--author'),
add_argument('--subject', required=True),
add_argument('--cache', nargs='?'),
add_argument('files', nargs=''),
)
actions = []
for (args, kwargs) in options:
actions.append(parser.add_argument(*args, **kwargs))
args = parser.parse_args()
At this point, the options are parsed in args, and all argparse.Action objects are stored in the actions list. I can then iterate over this list and reconstruct the options like this:
cmdline = []
for action in actions:
value = getattr(args, action.dest)
if action.required or value != action.default:
if action.option_strings:
cmdline.append(action.option_strings[0])
if action.nargs is None:
cmdline.append(value)
elif action.nargs == '?':
if value != action.const:
cmdline.append(value)
elif action.nargs != 0:
cmdline += value
In my specific case, I also wanted to remove some options from the command line. To do that I simply added them separately with a call to parser.add_argument() and not through the options tuple.
Related
I have a function that looks like this:
import argparse
import sys
def execute():
parser = argparse.ArgumentParser()
if (total_args := len(sys.argv)) == 1:
do_stuff()
if total_args == 2:
first = sys.argv[1]
do_stuff2()
if total_args == 3:
first, second = sys.argv[1:3]
do_stuff3()
if total_args > 3:
first, second = sys.argv[1:3]
del sys.argv[1:3]
add_args(parser)
parser.parse_args()
do_stuff4()
Which should have a test function test_execute that will try different given args, the question: is there a clean way to do it without manually modifying sys.argv using sys.argv.extend(some_test_args) and delete the args later?
Note: I can't use argparse optional positional arguments by setting nargs=? in parser.add_argument() because the first 2 arguments are optional and each case (1, 2, 3, > 3 arguments) executes different functions. To understand further, please check the example below ...
parser = argparse.ArgumentParser()
parser.add_argument('arg1', nargs='?')
parser.add_argument('arg2', nargs='?')
args = parser.parse_known_args()
print(args)
which if called like the following, will result to the wrong variable saved in the second position:
>>> python my_script.py --unknown-arg 999
Will print:
(Namespace(arg1='999', arg2=None), ['--unknown-arg'])
which is totally not what I need. I'm expecting arg1 to have a None value. The reason sometimes there will be unknown arguments is that argparse does not support parsing arguments by specifying a group. Let's say I have argument group A and argument group B and I need to parse only group A, I can't do parser.parse_group('A') I will have to create parser_a = argparse.ArgumentParser() and add group A arguments and parse them and repeat for parser_b.
Therefore the best solution I have so far is using sys.argv despite the fact this is inconvenient for testing. Also adding all options without grouping, will create another problem because group B arguments depend on values parsed from group A.
One workaround is to specify using --unknown-arg=999 but this will create inconsistencies in the documentation and usage of the script and is also not what I need.
Could you pass in sys.argv into execute()?
Something like this:
import argparse
import sys
def execute(argv):
parser = argparse.ArgumentParser()
if (total_args := len(argv)) == 1:
do_stuff()
if total_args == 2:
first = argv[1]
do_stuff2()
if total_args == 3:
first, second = argv[1:3]
do_stuff3()
if total_args > 3:
first, second = argv[1:3]
del argv[1:3]
add_args(parser)
parser.parse_args(argv)
do_stuff4()
if __name__ == "__main__":
execute(sys.argv)
In your tests you could then do something along the lines of:
def test_execute():
test_argv = ["some", "args", "list"]
execute(test_argv)
# assert something
Let's imaging I have a CLI with an nargs="*" argument called --list. I would like to filter out certain elements of the this list.
For instance, if the user passes in --list foo bar foo baz I would like to filter out all values == "foo" so that the final list becomes ["bar", "baz"]:
parser = argparse.ArgumentParser()
parser.add_argument(
"--list",
nargs="*",
)
parser.parse_args(["--list", "foo", "bar", "foo", "baz"])
I could obviously solve that in a post-processing step on the resulting args. However I was wondering if there is a built-in mechanism in argparse that allows to implement such a filter without post-postprocessing.
I tried to use type=lambda: ... but that seems to be called only on the values themselves instead of the final list, so I cannot use it for filtering values I suppose?
There's a parameter action for argparse.add_argument. From the docs:
ArgumentParser objects associate command-line arguments with actions. These actions can do just about anything with the command-line arguments associated with them, though most actions simply add an attribute to the object returned by parse_args().
So we can do the filtering with an action. The documentation states:
The recommended way to create a custom action is to extend Action, overriding the __call__ method and optionally the __init__ and format_usage methods.
A filtering argparse.Action could look like this:
import argparse
from typing import Optional, Callable
class FilterNargsAction(argparse.Action):
def __init__(
self,
option_strings,
dest,
nargs: Optional[str] = None,
condition: Optional[Callable] = None,
**kwargs
):
if nargs not in ("*", "+"):
# We could also allow integers
raise ValueError("nargs must be one of (*, +)")
if condition is None:
raise ValueError("condition needs to be set")
self.condition = condition
super().__init__(option_strings, dest, nargs=nargs, **kwargs)
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, [v for v in values if self.condition(v)])
parser = argparse.ArgumentParser()
parser.add_argument(
"--list",
nargs="*",
action=FilterNargsAction,
condition=lambda x: x != "foo",
)
args = parser.parse_args(["--list", "foo", "bar", "foo", "baz"])
print(args.list)
If you absolutely need to filter arguments when parsing, you can use a custom action.
I would like to get subset of parsed arguments and send them to another function in python. I found this argument_group idea but I couldn't find to reach argument groups. This is what I want to try to do:
import argparse
def some_function(args2):
x = args2.bar_this
print(x)
def main():
parser = argparse.ArgumentParser(description='Simple example')
parser.add_argument('--name', help='Who to greet', default='World')
# Create two argument groups
foo_group = parser.add_argument_group(title='Foo options')
bar_group = parser.add_argument_group(title='Bar options')
# Add arguments to those groups
foo_group.add_argument('--bar_this')
foo_group.add_argument('--bar_that')
bar_group.add_argument('--foo_this')
bar_group.add_argument('--foo_that')
args = parser.parse_args()
# How can I get the foo_group arguments for example only ?
args2 = args.foo_group
some_function(args2)
I don't know whether a simpler solution exists, but you can create a custom "namespace" object selecting only the keys arguments you need from the parsed arguments.
args2 = argparse.Namespace(**{k: v for k, v in args._get_kwargs()
if k.startswith("foo_")})
You can customize the if clause to your needs and possibly change the argument names k, e.g. removing the foo_ prefix.
I'm working on something where I need to use argparse.
Here's the code I got a problem with:
parser = argparse.ArgumentParser()
parser.add_argument('--create n file', dest='create',
nargs=2, default=(1, 'world1.json'),
help='Create a party of n player with mission parameters in file')
I'm trying to find a way to either set both n and file to another value, or set only one of them. n is an int and file a str.
Here's what I would like to get, using the following command:
Command
Expected result
python mission.py --create 2
create = [2, 'world1.json']
python mission.py --create world2.json
create = [1, 'world2.json']
python mission.py --create 3 world2.json
create = [3, 'world2.json']
When --create is used (with or without specifying n/file), I'll need to start a function using the list as arguments.
I've tried multiple things and read argparse documentation more than once but can't find a way to do it.
The code below returns the expected results for the listed usecases. I decided to use an extra function to handle the argument, as the program must accept either an int or a string for the first argument passed.
I use a "try" block to see whether the single argument can be parsed as an int before proceeding.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--create n file', dest='create', nargs='+', default=(1,'world1.json'),
help='Create a party of n player with mission parameters in file')
args = parser.parse_args()
def get_n_file(arg):
if len(arg)==1:
try:
i = int(arg[0])
result = int(arg[0]), 'world'+str(arg[0])+'.json'
except:
s = arg[0]
result = 1, s
return result
elif len(arg)==2:
return int(arg[0]), arg[1]
print(args.create)
n, f = get_n_file(args.create)
print(n, f)
Let's say I have an args namespace after parsing my command line with argparse. Now, I want to use this to create some objects like this:
foo = Foo(bar=args.bar)
Unfortunately, I have the restriction that if a keyword argument is set, it must not be None. Now, I need to check if args.bar is set and act accordingly:
if args.bar:
foo = Foo(bar=args.bar)
else:
foo = Foo()
This is unwieldy and doesn't scale for more arguments. What I'd like to have, is something like this:
foo = Foo(**args.__dict__)
but this still suffers from my initial problem and additionally doesn't work for keys that are not keyword arguments of the __init__ method. Is there a good way to achieve these things?
You could try something like this:
>>> defined_args = {k:v for k,v in args._get_kwargs() if v is not None}
>>> foo = Foo(**defined_args)
For example:
>>> import argparse
>>> args = argparse.Namespace(key1=None,key2='value')
>>> {k:v for k,v in args._get_kwargs() if v is not None}
{'key2': 'value'}
Note, however, that _get_kwargs() is not part of the public API so may or may not be available in future releases/versions.
I think you can use vars():
args = parser.parse_args()
Foo(**vars(args))
vars([object]) returns the namespace as a dictionary