I would like to create a python file that can be run from the terminal - this file will be in charge of running various other python files depending on the functionality required along with their required arguments, respectively. For example, this is the main file:
import sys
from midi_to_audio import arguments, run
files = ["midi_to_audio.py"]
def main(file, args):
if file == "midi_to_audio.py":
if len(args) != arguments:
print("Incorrect argument length")
else:
run("test","t")
if __name__ == '__main__':
sys.argv.pop(0)
file = sys.argv[0]
sys.argv.pop(0)
if file not in files:
print("File does not exist")
else:
main(file, sys.argv)
And this is the first file used in the example (midi_to_audio.py):
arguments = 2
def run(file, output_file):
print("Ran this method")
So depending on which file I've specified when running the cmd via the terminal, it will go into a different file and call its run method. If the arguments are not as required in each file, it will not run
For example: >python main.py midi_to_audio.py file_name_here output_name_here
My problem is that, as I add more files with their own "arguments" and "run" functions, I wonder if python is going to get confused with which arguments or which run function to execute. Is there a more safer/generic way of doing this?
Also, is there a way of getting the names of the python files depending on which files I've imported? Because for now I have to import the file and manually add their file name to the files list in main.py
Your runner could look like this, to load a module by name and check it has run, and check the arguments given on the command line, and finally dispatch to the module's run function.
import sys
import importlib
def main():
args = sys.argv[1:]
if len(args) < 1:
raise Exception("No module name given")
module_name = args.pop(0).removesuffix(".py") # grab the first argument and remove the .py suffix
module = importlib.import_module(module_name) # import a module by name
if not hasattr(module, 'run'): # check if the module has a run function
raise Exception(f"Module {module_name} does not have a run function")
arg_count = getattr(module, 'arguments', 0) # get the number of arguments the module needs
if len(args) != arg_count:
raise Exception(f"Module {module_name} requires {arg_count} arguments, got {len(args)}")
module.run(*args)
if __name__ == '__main__':
main()
This works with the midi_to_audio.py module in your post.
I have a use case where I have a main python script with many command line arguments, I need to break it's functionality into multiple smaller scripts, a few command-line arguments will be common to more than one smaller scripts. I want to reduce code duplicacy. I tried to use decorators to register each argument to one or more scripts, but am not able to get around an error. Another caveat I have is I want to set default values for shared argument according to which script is being run. This is what I have currently
argument_parser.py
import argparse
import functools
import itertools
from scripts import Scripts
from collections import defaultdict
_args_register = defaultdict(list)
def argument(scope):
"""
Decorator to add argument to argument registry
:param scope: The module name to register current argument function to can also be a list of modules
:return: The decorated function after after adding it to registry
"""
def register(func):
if isinstance(scope, Scripts):
_args_register[scope].append(func)
elif isinstance(scope, list) and Scripts.ALL in scope:
_args_register[Scripts.ALL].append(func)
else:
for module in scope:
_args_register[module].append(func)
return func
return register
class ArgumentHandler:
def __init__(self, script, parser=None):
self._parser = parser or argparse.ArgumentParser(description=__doc__)
assert script in Scripts
self._script = script
#argument(scope=Scripts.ALL)
def common_arg(self):
self._parser.add_arg("--common-arg",
default=self._script,
help="An arg common to all scripts")
#argument(scope=[Scripts.TRAIN, Scripts.TEST])
def train_test_arg(self):
self._parser.add_arg("--train-test-arg",
default=self._script,
help=f"An arg common to train-test scripts added in argument handler"
)
def parse_args(self):
for argument in itertools.chain(_args_register[Scripts.ALL],
_args_register[self._script]):
argument()
_args = self._parser.parse_args()
return _args
One of the smaller scripts train.py
"""
A Train script to abstract away training tasks
"""
import argparse
from argument_parser import ArgumentHandler
from scripts import Scripts
current = Scripts.TRAIN
parser = argparse.ArgumentParser(description=__doc__)
def get_args() -> argparse.Namespace:
parser.add_argument('--train-arg',
default='blah',
help='a train argumrnt set in the train script')
args_handler = ArgumentHandler(parser=parser, script=current)
return args_handler.parse_args()
if __name__ == '__main__':
print(get_args())
When I run train.py I get the following error
File "../argument_parser.py", line 68, in parse_args
argument()
TypeError: common_arg() missing 1 required positional argument: 'self'
Process finished with exit code 1
I think this is because decorators are run at import time, but am not sure, is there any work around this? or any other better way to reduce code duplicacy? Any help will be highly appreciated. Thanks!
I'm trying to access to the "resources" folder with the ArgumentParser.
This code and the "resources" folder are in the same folder...
Just to try to run the code, I've put a print function in the predict function. However this error occurs:
predict.py: error: the following arguments are required: resources_path
How can I fix it?
from argparse import ArgumentParser
def parse_args():
parser = ArgumentParser()
parser.add_argument("resources_path", help='/resources')
return parser.parse_args()
def predict(resources_path):
print(resources_path)
pass
if __name__ == '__main__':
args = parse_args()
predict(args.resources_path)
I am guessing from your error message that you are trying to call your program like this:
python predict.py
The argument parser by default gets the arguments from sys.argv, i.e. the command line. You'll have to pass it yourself like this:
python predict.py resources
It's possible that you want the resources argument to default to ./resources if you don't pass anything. (And I further assume you want ./resources, not /resources.) There's a keyword argument for that:
....
parser.add_argument('resources_path', default='./resources')
...
I have a Python module that uses the argparse library. How do I write tests for that section of the code base?
You should refactor your code and move the parsing to a function:
def parse_args(args):
parser = argparse.ArgumentParser(...)
parser.add_argument...
# ...Create your parser as you like...
return parser.parse_args(args)
Then in your main function you should just call it with:
parser = parse_args(sys.argv[1:])
(where the first element of sys.argv that represents the script name is removed to not send it as an additional switch during CLI operation.)
In your tests, you can then call the parser function with whatever list of arguments you want to test it with:
def test_parser(self):
parser = parse_args(['-l', '-m'])
self.assertTrue(parser.long)
# ...and so on.
This way you'll never have to execute the code of your application just to test the parser.
If you need to change and/or add options to your parser later in your application, then create a factory method:
def create_parser():
parser = argparse.ArgumentParser(...)
parser.add_argument...
# ...Create your parser as you like...
return parser
You can later manipulate it if you want, and a test could look like:
class ParserTest(unittest.TestCase):
def setUp(self):
self.parser = create_parser()
def test_something(self):
parsed = self.parser.parse_args(['--something', 'test'])
self.assertEqual(parsed.something, 'test')
"argparse portion" is a bit vague so this answer focuses on one part: the parse_args method. This is the method that interacts with your command line and gets all the passed values. Basically, you can mock what parse_args returns so that it doesn't need to actually get values from the command line. The mock package can be installed via pip for python versions 2.6-3.2. It's part of the standard library as unittest.mock from version 3.3 onwards.
import argparse
try:
from unittest import mock # python 3.3+
except ImportError:
import mock # python 2.6-3.2
#mock.patch('argparse.ArgumentParser.parse_args',
return_value=argparse.Namespace(kwarg1=value, kwarg2=value))
def test_command(mock_args):
pass
You have to include all your command method's args in Namespace even if they're not passed. Give those args a value of None. (see the docs) This style is useful for quickly doing testing for cases where different values are passed for each method argument. If you opt to mock Namespace itself for total argparse non-reliance in your tests, make sure it behaves similarly to the actual Namespace class.
Below is an example using the first snippet from the argparse library.
# test_mock_argparse.py
import argparse
try:
from unittest import mock # python 3.3+
except ImportError:
import mock # python 2.6-3.2
def main():
parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')
args = parser.parse_args()
print(args) # NOTE: this is how you would check what the kwargs are if you're unsure
return args.accumulate(args.integers)
#mock.patch('argparse.ArgumentParser.parse_args',
return_value=argparse.Namespace(accumulate=sum, integers=[1,2,3]))
def test_command(mock_args):
res = main()
assert res == 6, "1 + 2 + 3 = 6"
if __name__ == "__main__":
print(main())
Make your main() function take argv as an argument rather than letting it read from sys.argv as it will by default:
# mymodule.py
import argparse
import sys
def main(args):
parser = argparse.ArgumentParser()
parser.add_argument('-a')
process(**vars(parser.parse_args(args)))
return 0
def process(a=None):
pass
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Then you can test normally.
import mock
from mymodule import main
#mock.patch('mymodule.process')
def test_main(process):
main([])
process.assert_call_once_with(a=None)
#mock.patch('foo.process')
def test_main_a(process):
main(['-a', '1'])
process.assert_call_once_with(a='1')
I did not want to modify the original serving script so I just mocked out the sys.argv part in argparse.
from unittest.mock import patch
with patch('argparse._sys.argv', ['python', 'serve.py']):
... # your test code here
This breaks if argparse implementation changes but enough for a quick test script. Sensibility is much more important than specificity in test scripts anyways.
Populate your arg list by using sys.argv.append() and then call
parse(), check the results and repeat.
Call from a batch/bash file with your flags and a dump args flag.
Put all your argument parsing in a separate file and in the if __name__ == "__main__": call parse and dump/evaluate the results then test this from a batch/bash file.
parse_args throws a SystemExit and prints to stderr, you can catch both of these:
import contextlib
import io
import sys
#contextlib.contextmanager
def captured_output():
new_out, new_err = io.StringIO(), io.StringIO()
old_out, old_err = sys.stdout, sys.stderr
try:
sys.stdout, sys.stderr = new_out, new_err
yield sys.stdout, sys.stderr
finally:
sys.stdout, sys.stderr = old_out, old_err
def validate_args(args):
with captured_output() as (out, err):
try:
parser.parse_args(args)
return True
except SystemExit as e:
return False
You inspect stderr (using err.seek(0); err.read() but generally that granularity isn't required.
Now you can use assertTrue or whichever testing you like:
assertTrue(validate_args(["-l", "-m"]))
Alternatively you might like to catch and rethrow a different error (instead of SystemExit):
def validate_args(args):
with captured_output() as (out, err):
try:
return parser.parse_args(args)
except SystemExit as e:
err.seek(0)
raise argparse.ArgumentError(err.read())
A simple way of testing a parser is:
parser = ...
parser.add_argument('-a',type=int)
...
argv = '-a 1 foo'.split() # or ['-a','1','foo']
args = parser.parse_args(argv)
assert(args.a == 1)
...
Another way is to modify sys.argv, and call args = parser.parse_args()
There are lots of examples of testing argparse in lib/test/test_argparse.py
When passing results from argparse.ArgumentParser.parse_args to a function, I sometimes use a namedtuple to mock arguments for testing.
import unittest
from collections import namedtuple
from my_module import main
class TestMyModule(TestCase):
args_tuple = namedtuple('args', 'arg1 arg2 arg3 arg4')
def test_arg1(self):
args = TestMyModule.args_tuple("age > 85", None, None, None)
res = main(args)
assert res == ["55289-0524", "00591-3496"], 'arg1 failed'
def test_arg2(self):
args = TestMyModule.args_tuple(None, [42, 69], None, None)
res = main(args)
assert res == [], 'arg2 failed'
if __name__ == '__main__':
unittest.main()
For testing CLI (command line interface), and not command output I did something like this
import pytest
from argparse import ArgumentParser, _StoreAction
ap = ArgumentParser(prog="cli")
ap.add_argument("cmd", choices=("spam", "ham"))
ap.add_argument("-a", "--arg", type=str, nargs="?", default=None, const=None)
...
def test_parser():
assert isinstance(ap, ArgumentParser)
assert isinstance(ap, list)
args = {_.dest: _ for _ in ap._actions if isinstance(_, _StoreAction)}
assert args.keys() == {"cmd", "arg"}
assert args["cmd"] == ("spam", "ham")
assert args["arg"].type == str
assert args["arg"].nargs == "?"
...
I'm trying to create a complex mercurial commit hook in python. I want to also be allowed to pass parameters using OptionParser. Here is the gist of what I have so far:
.hg/hgrc config:
[hooks]
commit = python:/mydir/pythonFile.py:main
# using python:/mydir/pythonFile.py doesn't work for some reason either
pythonFile.py:
def main(ui, repo, **kwargs):
from optparse import OptionParser
parser = OptionParser()
parser.add_option('--test-dir', action='store', type="string",
dest='test_dir', default='otherdir/',
help='help info')
(options, args) = parser.parse_args()
# do some stuff here
someFunc(options.test_dir)
if __name__ == '__main__':
import sys
main(sys.argv[0], sys.argv[1], sys.argv[2:])
When I run hg commit -m 'message' I get an error: "Usage: hg [options] hg: error: no such option: -m". When I try hg commit --test-dir '/somedir' I get an error: "hg commit: option --test-dir not recognized".
Lastly I tried specifying commit = python:/mydir/pythonFile.py:main --test-dir '/somedir' in the hgrc config and I got this error: "AttributeError: 'module' object has no attribute 'main --test-dir '/somedir''"
Thank you for your help.
I think your problem may be in trying to import something that isn't part of the python packaged with mercurial.
If what you need is to pass additional information to the hook such that you can configure it differently for different repos/branches etc, you could use
param_value= ui.config('ini_section', 'param_key', default='', untrusted=False)
where ini_section is the bit in [] in the mercurial.ini / .hgrc file and param_key is the name of the entry
so something like
[my_hook_params]
test-dir=/somedir
then use
test_dir = ui.config('my_hook_params', 'test-dir', default='otherdir/', untrusted=False)