Develop Reference

Best way to pass command line arguments via file in python

I have a lot of arguments to pass to my main.py. It's easier to store them in a txt file. So, i would like to know best way of using "config" files to pass CL args.
Shell script is not what i need, unfortunatly.

If you plan to use argparse, then fromfile_prefix_chars is designed to solve exactly this problem.
In your launching program, put all of the arguments, one per line, into a file. Pass #file.txt to your child program. In your child program, pass a fromfile_prefix_chars parameter to the ArgumentParser() constructor:
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
argparse takes care of the rest for you.
Here is an example:
from argparse import ArgumentParser
parser = ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('-f', '--foo')
parser.add_argument('--bar')
parser.add_argument('q', nargs='*')
ns = parser.parse_args()
print(ns)
The contents of foo.txt:
-f
1
--bar=2
q one
q two
The command line and the output:
$ python zz.py #foo.txt
Namespace(bar='2', foo='1', q=['q one', 'q two'])

Use configparser. It uses .ini files and it's really easy to use.
Config file:
[DEFAULT]
KeepAlive = 45
ForwardX11 = yes
Example Code:
>>> config = configparser.ConfigParser()
>>> config.sections()
[]
>>> config.read('example.ini')
>>> for key in config['bitbucket.org']: print(key)
...
keepalive
forwardx11
>>> default = config['default']
>>> default['keepalive']
'45'
>>> default['ForwardX11']
'yes'

Here is a simple function that converts any #foo argument into the contents of foo, one argument per line. After the conversion, you may use sys.argv in any of the normal ways.
import sys
def expand_arg_files(args):
for arg in args:
if arg.startswith('#'):
with open(arg[1:]) as f:
file_args = f.read().splitlines()
yield from expand_arg_files(file_args)
else:
yield arg
sys.argv[:] = expand_arg_files(sys.argv[:])
print(sys.argv)
Notes:
The generator delegation syntax requires Python3.3 or higher.
You may have # args inside the argument file. The expansion is recursive.

How to use python argparse with args other than sys.argv?

Is there a way to use argparse with any list of strings, instead of only with sys.argv?
Here's my problem: I have a program which looks something like this:
# This file is program1.py
import argparse
def main(argv):
parser = argparse.ArgumentParser()
# Do some argument parsing
if __name__ == '__main__':
main(sys.argv)
This works fine when this program is called straight from the command line. However, I have another python script which runs batch versions of this script with different commandline arguments, which I'm using like this:
import program1
arguments = ['arg1', 'arg2', 'arg3']
program1.main(arguments)
I still want to be able to parse the arguments, but argparse automatically defaults to using sys.argv instead of the arguments that I give it. Is there a way to pass in the argument list instead of using sys.argv?

You can pass a list of strings to parse_args:
parser.parse_args(['--foo', 'FOO'])

Just change the script to default to sys.argv[1:] and parse arguments omitting the first one (which is the name of the invoked command)
import argparse,sys
def main(argv=sys.argv[1:]):
parser = argparse.ArgumentParser()
parser.add_argument("--level", type=int)
args = parser.parse_args(argv)
if __name__ == '__main__':
main()
Or, if you cannot omit the first argument:
import argparse,sys
def main(args=None):
# if None passed, uses sys.argv[1:], else use custom args
parser = argparse.ArgumentParser()
parser.add_argument("--level", type=int)
args = parser.parse_args(args)
# Do some argument parsing
if __name__ == '__main__':
main()
Last one: if you cannot change the called program, you can still do something
Let's suppose the program you cannot change is called argtest.py (I added a call to print arguments)
Then just change the local argv value of the argtest.sys module:
import argtest
argtest.sys.argv=["dummy","foo","bar"]
argtest.main()
output:
['dummy', 'foo', 'bar']

Python argparse now has a parameter nargs for add_argument (https://docs.python/3/library/argparse.html).
It allows us to have as many arguments as we want for a named parameter (here, alist)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--alist", nargs="*")
args = parser.parse_args()
print(args.alist)
All command line values that follow --alist are added to a list.
Example:
$ python3 argparse-01.py --alist fred barney pebbles "bamm bamm"
['fred', 'barney', 'pebbles', 'bamm bamm']
As you see, it is allowed to quote the arguments, but not necessary unless you need to protect a space.

`argparse` multiple choice argument?

I am using argparse to parse the Python command line which is supposed to look like this:
python script_name.py --sdks=first, second
My script looks like this:
sdk_choises = ['aio','sw']
parser = argparse.ArgumentParser(description='Blah blah')
parser.add_argument('--sdks', action='append', nargs='+', required=True, help='specifies target SDK(s)')
args = parser.parse_args()
if 'aio' in args.sdks:
# do something with aio
if 'sw' in args.sdks:
# do something with sw
When I execute:
python script_name.py --sdks=aio, sw I get error:
"usage: script.py [-h] --sdks SDKS [SDKS ...]
build.py: error: unrecognized arguments: sw"
I'd like to be able to choose one or all choices:
python script_name.py --sdks=first
python script_name.py --sdks=second
python script_name.py --sdks=first, second
Where did I go wrong?

The following works nice:
import argparse
parser = argparse.ArgumentParser(description='Blah blah')
parser.add_argument('--sdks', nargs='+', required=True, help='specifies target SDK(s)')
args = parser.parse_args()
print(args.sdks)
You don't need the = when passing options, just use:
$ python test.py --sdks ai pw
['ai', 'pw']

If you prefer your original form of the comma-separated list, plus check if the argument is valid, then I recommend:
parser.add_argument('--sdks', nargs=1, type=lambda s: [sdk_choises[sdk_choises.index(f)] for f in s.split(',')], ...
Even cleaner way is to define it in a separate function similar to lambda above:
parser.add_argument('--sdks', nargs=1, type=my_parse_function, ...
argparse docs has examples for the parsing function with proper error reporting.
With nargs=1 you'd need to remove one extra list layer.

Call function based on argparse

I'm new to python and currently playing with it.
I have a script which does some API Calls to an appliance. I would like to extend the functionality and call different functions based on the arguments given when calling the script.
Currently I have the following:
parser = argparse.ArgumentParser()
parser.add_argument("--showtop20", help="list top 20 by app",
action="store_true")
parser.add_argument("--listapps", help="list all available apps",
action="store_true")
args = parser.parse_args()
I also have a
def showtop20():
.....
and
def listapps():
....
How can I call the function (and only this) based on the argument given?
I don't want to run
if args.showtop20:
#code here
if args.listapps:
#code here
as I want to move the different functions to a module later on keeping the main executable file clean and tidy.

Since it seems like you want to run one, and only one, function depending on the arguments given, I would suggest you use a mandatory positional argument ./prog command, instead of optional arguments (./prog --command1 or ./prog --command2).
so, something like this should do it:
FUNCTION_MAP = {'top20' : my_top20_func,
'listapps' : my_listapps_func }
parser.add_argument('command', choices=FUNCTION_MAP.keys())
args = parser.parse_args()
func = FUNCTION_MAP[args.command]
func()

At least from what you have described, --showtop20 and --listapps sound more like sub-commands than options. Assuming this is the case, we can use subparsers to achieve your desired result. Here is a proof of concept:
import argparse
import sys
def showtop20():
print('running showtop20')
def listapps():
print('running listapps')
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# Create a showtop20 subcommand
parser_showtop20 = subparsers.add_parser('showtop20', help='list top 20 by app')
parser_showtop20.set_defaults(func=showtop20)
# Create a listapps subcommand
parser_listapps = subparsers.add_parser('listapps', help='list all available apps')
parser_listapps.set_defaults(func=listapps)
# Print usage message if no args are supplied.
# NOTE: Python 2 will error 'too few arguments' if no subcommand is supplied.
# No such error occurs in Python 3, which makes it feasible to check
# whether a subcommand was provided (displaying a help message if not).
# argparse internals vary significantly over the major versions, so it's
# much easier to just override the args passed to it.
if len(sys.argv) <= 1:
sys.argv.append('--help')
options = parser.parse_args()
# Run the appropriate function (in this case showtop20 or listapps)
options.func()
# If you add command-line options, consider passing them to the function,
# e.g. `options.func(options)`

There are lots of ways of skinning this cat. Here's one using action='store_const' (inspired by the documented subparser example):
p=argparse.ArgumentParser()
p.add_argument('--cmd1', action='store_const', const=lambda:'cmd1', dest='cmd')
p.add_argument('--cmd2', action='store_const', const=lambda:'cmd2', dest='cmd')
args = p.parse_args(['--cmd1'])
# Out[21]: Namespace(cmd=<function <lambda> at 0x9abf994>)
p.parse_args(['--cmd2']).cmd()
# Out[19]: 'cmd2'
p.parse_args(['--cmd1']).cmd()
# Out[20]: 'cmd1'
With a shared dest, each action puts its function (const) in the same Namespace attribute. The function is invoked by args.cmd().
And as in the documented subparsers example, those functions could be written so as to use other values from Namespace.
args = parse_args()
args.cmd(args)
For sake of comparison, here's the equivalent subparsers case:
p = argparse.ArgumentParser()
sp = p.add_subparsers(dest='cmdstr')
sp1 = sp.add_parser('cmd1')
sp1.set_defaults(cmd=lambda:'cmd1')
sp2 = sp.add_parser('cmd2')
sp2.set_defaults(cmd=lambda:'cmd2')
p.parse_args(['cmd1']).cmd()
# Out[25]: 'cmd1'
As illustrated in the documentation, subparsers lets you define different parameter arguments for each of the commands.
And of course all of these add argument or parser statements could be created in a loop over some list or dictionary that pairs a key with a function.
Another important consideration - what kind of usage and help do you want? The different approaches generate very different help messages.

If your functions are "simple enough" take adventage of type parameter https://docs.python.org/2.7/library/argparse.html#type
type= can take any callable that takes a single string argument and
returns the converted value:
In your example (even if you don't need a converted value):
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
This simple script:
import argparse
def showtop20(dummy):
print "{0}\n".format(dummy) * 5
parser = argparse.ArgumentParser()
parser.add_argument("--listapps", help="list all available apps",
type=showtop20,
action="store")
args = parser.parse_args()
Will give:
# ./test.py --listapps test
test
test
test
test
test
test

Instead of using your code as your_script --showtop20, make it into a sub-command your_script showtop20 and use the click library instead of argparse. You define functions that are the name of your subcommand and use decorators to specify the arguments:
import click
#click.group()
#click.option('--debug/--no-debug', default=False)
def cli(debug):
print(f'Debug mode is {"on" if debug else "off"}')
#cli.command() # #cli, not #click!
def showtop20():
# ...
#cli.command()
def listapps():
# ...
See https://click.palletsprojects.com/en/master/commands/

# based on parser input to invoke either regression/classification plus other params
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--path", type=str)
parser.add_argument("--target", type=str)
parser.add_argument("--type", type=str)
parser.add_argument("--deviceType", type=str)
args = parser.parse_args()
df = pd.read_csv(args.path)
df = df.loc[:, ~df.columns.str.contains('^Unnamed')]
if args.type == "classification":
classify = AutoML(df, args.target, args.type, args.deviceType)
classify.class_dist()
classify.classification()
elif args.type == "regression":
reg = AutoML(df, args.target, args.type, args.deviceType)
reg.regression()
else:
ValueError("Invalid argument passed")
# Values passed as : python app.py --path C:\Users\Abhishek\Downloads\adult.csv --target income --type classification --deviceType GPU

You can evaluate using evalwhether your argument value is callable:
import argparse
def list_showtop20():
print("Calling from showtop20")
def list_apps():
print("Calling from listapps")
my_funcs = [x for x in dir() if x.startswith('list_')]
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--function", required=True,
choices=my_funcs,
help="function to call", metavar="")
args = parser.parse_args()
eval(args.function)()

Using a file to store optparse arguments

I've been using optparse for a while now, and would like to add the ability to load the arguments from a config file.
So far the best I can think of is a wrapper batch script with the arguments hardcoded... seems clunky.
What is the most elegant way to do this?

I agree with S.Lott's idea of using a config file, but I'd recommend using the built-in ConfigParser (configparser in 3.0) module to parse it, rather than a home-brewed solution.
Here's a brief script that illustrates ConfigParser and optparse in action.
import ConfigParser
from optparse import OptionParser
CONFIG_FILENAME = 'defaults.cfg'
def main():
config = ConfigParser.ConfigParser()
config.read(CONFIG_FILENAME)
parser = OptionParser()
parser.add_option("-l",
"--language",
dest="language",
help="The UI language",
default=config.get("Localization", "language"))
parser.add_option("-f",
"--flag",
dest="flag",
help="The country flag",
default=config.get("Localization", "flag"))
print parser.parse_args()
if __name__ == "__main__":
main()
Output:
(<Values at 0x2182c88: {'flag': 'japan.png', 'language': 'Japanese'}>, [])
Run with "parser.py --language=French":
(<Values at 0x2215c60: {'flag': 'japan.png', 'language': 'French'}>, [])
Help is built in.
Run with "parser.py --help":
Usage: parser.py [options]
Options:
-h, --help show this help message and exit
-l LANGUAGE, --language=LANGUAGE
The UI language
-f FLAG, --flag=FLAG The country flag
The config file:
[Localization]
language=Japanese
flag=japan.png

You can use argparse module for that:
>>> open('args.txt', 'w').write('-f\nbar')
>>> parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
>>> parser.add_argument('-f')
>>> parser.parse_args(['-f', 'foo', '#args.txt'])
Namespace(f='bar')
It might be included in stdlib, see pep 389.

I had a similar problem, but also wanted to specific the config file as an argument. Inspired by S. Lott's answer, I came up with the following code.
Example terminal session:
$ python defaultconf.py # use hard-coded defaults
False
$ python defaultconf.py --verbose # verbose on command line
True
$ python defaultconf.py --loadconfig blah # load config with 'verbose':True
True
$ python defaultconf.py --loadconfig blah --quiet # Override configured value
False
Code:
#!/usr/bin/env python2.6
import optparse
def getParser(defaults):
"""Create and return an OptionParser instance, with supplied defaults
"""
o = optparse.OptionParser()
o.set_defaults(**defaults)
o.add_option("--verbose", dest = "verbose", action="store_true")
o.add_option("--quiet", dest = "verbose", action="store_false")
o.add_option("--loadconfig", dest = "loadconfig")
return o
def main():
# Hard coded defaults (including non-command-line-argument options)
my_defaults = {'verbose': False, 'config_only_variable': 42}
# Initially parse arguments
opts, args = getParser(my_defaults).parse_args()
if opts.loadconfig is not None:
# Load config from disk, update the defaults dictionary, and reparse
# Could use ConfigParser, simplejson, yaml etc.
config_file_values = {'verbose': True} # the dict loaded from disk
my_defaults.update(config_file_values)
opts, args = getParser(my_defaults).parse_args()
print opts.verbose
if __name__ == '__main__':
main()
A practical implementation can be found on Github: The defaults dictionary, the argument parser and the main function

That's what the set_defaults function is for. http://docs.python.org/library/optparse.html#optparse.OptionParser.set_defaults
Create a file that's the dictionary of default values.
{ 'arg1': 'this',
'arg2': 'that'
}
Then read this file, eval it to convert the text to a dictionary, and provide this dictionary as the arguments to set_defaults.
If you're really worried about eval, then use JSON (or YAML) notation for this file. Or you could even make an .INI file out of it and use configparser to get your defaults.
Or you can use a simple list of assignment statements and exec.
Config File.
arg1 = 'this'
arg2 = 'that'
Reading the config file.
defaults= {}
with open('defaults.py','r') as config
exec config in {}, defaults

Read the arguments in the same commandline format from a file e.g. #commands, then use your original parser to parse them.
options, args = parser.parse_args()
if args[0][0] == '#': # script.py #optfile
with open(args[0][1:]) as f:
fa = [l.strip() for l in f]
fa = fa + args[1:] # put back any other positional arguments
# Use your original parser to parse the new options
options, args = parser.parse_args(args=fa, values=options)

I've built a lot of scripts with flags and options lately, and I've come up with the solution described here.
Basically I instance an optionparser with a special flag that tells to try and load options from a file, so you can use normally your script specifying options from command line or provide them (or a set of them) from a file.
Update: i have shared code on GitHub