In a file (say parser.py) I have:
import argparse
def parse_cmdline(cmdline=None):
parser = argparse.ArgumentParser()
parser.add_argument('--first-param',help="Does foo.")
parser.add_argument('--second-param',help="Does bar.")
if cmdline is not None:
args = parser.parse_args(cmdline)
else:
args = parser.parse_args()
return vars(args)
if __name__=='__main__':
print parse_cmdline()
Sure enough, when called from the command line it works and give me pretty much what I expect:
$ ./parser.py --first-param 123 --second-param 456
{'first_param': '123', 'second_param': '456'}
But then I want to unittest it, thus I write a test_parser.py file:
import unittest
from parser import parse_cmdline
class TestParser(unittest.TestCase):
def test_parse_cmdline(self):
parsed = parse_cmdline("--first-param 123 --second-param 456")
self.assertEqual(parsed['first_param'],'123')
self.assertEqual(parsed['second_param'],'456')
if __name__ == '__main__':
unittest.main()
Then I get the following error:
usage: test_parser.py [-h] [--first-param FIRST_PARAM]
[--second-param SECOND_PARAM]
test_parser.py: error: unrecognized arguments: - - f i r s t - p a r a m 1 2 3 - - s e c o n d - p a r a m 4 5 6
E
======================================================================
ERROR: test_parse_cmdline (__main__.TestParser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./test_parser.py", line 8, in test_parse_cmdline
parsed = parse_cmdline("--first-param 123 --second-param 456")
File "/home/renan/test_argparse/parser.py", line 12, in parse_cmdline
args = parser.parse_args(cmdline)
File "/usr/lib/python2.7/argparse.py", line 1691, in parse_args
self.error(msg % ' '.join(argv))
File "/usr/lib/python2.7/argparse.py", line 2361, in error
self.exit(2, _('%s: error: %s\n') % (self.prog, message))
File "/usr/lib/python2.7/argparse.py", line 2349, in exit
_sys.exit(status)
SystemExit: 2
----------------------------------------------------------------------
Ran 1 test in 0.004s
FAILED (errors=1)
As can be seen, the command line I specified (--first-param 123 --second-param 456) became
- - f i r s t - p a r a m 1 2 3 - - s e c o n d - p a r a m 4 5 6 (each character is separated by a space).
I don't understand why: what am I doing wrong?
argparse wants an "argument vector"—that is, a list of separate arguments—not a "command line" string.
And just calling split won't solve things. For example, this command line from the shell:
python script.py --first-param '123 456' --second-param 789
… will correctly give you 123 456 and 789, but this line of code:
parse_cmdline("--first-param '123 456' --second-param 789")
will not; it will give you '123 and 789, with an extra 456' that it doesn't know how to deal with.
In fact, even this is wrong:
parse_cmdline("--first-param '123' --second-param 789")
… because you'll get '123' instead of 123.
There are two ways to deal with this.
First, if you know what you're trying to pass, you can just pass it as a list instead of a string in the first place, and you don't need to worry about fiddly quoting and splitting details:
parse_cmdline(["--first-param", "123 456", "--second-param", "789"])
Alternatively, if you don't know exactly what you're trying to pass, you just know what it looks like on the shell, you can use shlex:
if cmdline is not None:
args = parser.parse_args(shlex.split(cmdline))
… and now Python will split your command line the same way as a standard Unix shell, getting the 123 456 as a single parameter.
To answer myself (I realized my mistake a few minutes later):
I am supposed to
if cmdline is not None:
args = parser.parse_args(cmdline.split())
else:
args = parser.parse_args()
Now the test passes correctly!
Related
How can we print the values of arguments passed to the functions in the call stack when an error stack trace is printed? I would like the output to be exactly as in the example below.
Example:
Traceback (most recent call last):
File "./file.py", line 615, in func0 **(arg0) arg0 = 0 was passed**
result = func1(arg1, arg2)
File "./file.py", line 728, in func1 **(arg1, arg2) arg1 = 1 and arg2 = 2 was passed**
return int_value[25]
TypeError: 'int' object is not iterable
I'd like the information inside the ** **s above to also be printed in addition to the normal output in the stack trace. What I envision is that the debugger automatically prints the passed arguments as well. That would give a clear picture of the "functional pipeline" that the data was passed through and what happened to it in the pipeline and which function did not do what it was supposed to do. This would help debugging a lot.
I searched quite a bit and found these related questions:
How to print call stack with argument values?
How to print function arguments in sys.settrace?
but the answers to neither of them worked for me: The answer to the 1st one led to ModuleNotFoundError: No module named 'stackdump'. The answer to the 2nd one crashed my ipython interpreter with a very long stack trace.
I also looked up:
https://docs.python.org/3/library/sys.html#sys.settrace
https://docs.python.org/3/library/traceback.html
There seems to be a capture_locals variable for TracebackExceptions, but I didn't quite understand how to make it work.
Pure Python3
Talking about TracebackExceptions and it's capture_locals argument, we can use it as follows:
#!/usr/bin/env python3
import traceback
c = ['Help me!']
def bar(a = 3):
d = {1,2,3}
e = {}
foo(a)
def foo(a=4):
b = 4
if a != b:
raise Exception("a is not equal to 4")
try:
bar(3)
except Exception as ex:
tb = traceback.TracebackException.from_exception(ex, capture_locals=True)
print("".join(tb.format()))
Which prints all local variables for each frame:
$ ./test.py
Traceback (most recent call last):
File "./test.py", line 21, in <module>
bar(3)
__annotations__ = {}
__builtins__ = <module 'builtins' (built-in)>
__cached__ = None
__doc__ = None
__file__ = './test.py'
__loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f81073704c0>
__name__ = '__main__'
__package__ = None
__spec__ = None
bar = <function bar at 0x7f81073b11f0>
c = ['Help me!']
ex = Exception('a is not equal to 4')
foo = <function foo at 0x7f810728a160>
traceback = <module 'traceback' from '/usr/lib/python3.8/traceback.py'>
File "./test.py", line 11, in bar
foo(a)
a = 3
d = {1, 2, 3}
e = {}
File "./test.py", line 17, in foo
raise Exception("a is not equal to 4")
a = 3
b = 4
Exception: a is not equal to 4
Looks a bit too verbose, but sometimes this data could be vital in debugging some crash.
Loguru
There is also a package loguru that prints "Fully descriptive exceptions":
2018-07-17 01:38:43.975 | ERROR | __main__:nested:10 - What?!
Traceback (most recent call last):
File "test.py", line 12, in <module>
nested(0)
└ <function nested at 0x7f5c755322f0>
> File "test.py", line 8, in nested
func(5, c)
│ └ 0
└ <function func at 0x7f5c79fc2e18>
File "test.py", line 4, in func
return a / b
│ └ 0
└ 5
ZeroDivisionError: division by zero
Probably exist better alternatives, but you can use a decorator for this:
def print_stack_arguments(func):
def new_func(*original_args, **original_kwargs):
try:
return func(*original_args, **original_kwargs)
except Exception as e:
print('Function: ', func.__name__)
print('Args: ', original_args)
print('Kwargs: ', original_kwargs)
print(e)
raise
return new_func
#print_stack_arguments
def print_error(value):
a = []
print(a[1])
#print_stack_arguments
def print_noerror(value):
print('No exception raised')
print_noerror('testing no exception')
print_error('testing exception')
I suspect this is kind of a klugefest on my part, but I'm working with the Luigi and Sciluigi modules which set a number of critical parameters PRIOR to 'self' being returned by an init. ANd if I try to manhandle these parameters AFTER self is returned, the Luigi.Parameter object masks them in such a way that I cant do what I need to do.
The luigi and sciluigi classes (as one uses them) contain no __init__. And if I try to insert an __init__ or call super(ChildClass, self).__init__(*args, **kwargs) ...I get weird errors 'unexpected parameter' errors.
So a Sciluigi class looks like this...
class MyTask(sciluigi.task):
param1 = sciluigi.Parameter(default='Yes') #String only
param2 = sciluigi.Parameter(default='No') #String only
def out_target(self):
return sciluigi.TargetInfo(self, self.out)
def run(self):
with self.out_target().open('w') as foofile:
foofile.write('foo\n')
SO...I'm hoping I can dynamically set some parameters via setattr PRIOR to 'self' actually being returned. But setattr requires the object.
I was hoping I could do something like...
setattr(inspect.stack()[?][?], 'DynamicVar', sciluigi.Parameter(default='Yes') )
EDIT: #Charles Duffy
Well, I'm not sure what info would be most helpful.
First issue is; I can't add an init. The actual code is below, with an __init__method added. I've included the resulting error if I try to run it. Its the same error is if I tr to run the super call to __init__
class FileConverter(sciluigi.Task):
"""
"""
in_target = None # Mandatory
out = sciluigi.Parameter() # <file>
exepath = sciluigi.Parameter(default = "")
def __init__(self):
self.var = 'anything'
def out_target(self):
log.debug("In 'FileConverter.out_target'... ")
return sciluigi.TargetInfo(self, self.out)
def run(self):
result = None
command = ''.join([
self.exepath, _slash, "FileConverter ",
" -in ", self.in_target().path,
" -out ", self.out_target().path,
" -out_type ", self.file_type
])
log.info("RUNNING COMMAND: " + command)
result = self.ex(command)
log.info("result: " + result[1])
Error
2017-02-24 17:01:48 | WARNING | Will not run MyWorkflow(instance_name=sciluigi_workflow) or any dependencies due to error in deps() method:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/luigi/worker.py", line 697, in _add
deps = task.deps()
File "/Library/Python/2.7/site-packages/luigi/task.py", line 572, in deps
return flatten(self._requires())
File "/Library/Python/2.7/site-packages/luigi/task.py", line 544, in _requires
return flatten(self.requires()) # base impl
File "/Library/Python/2.7/site-packages/sciluigi/workflow.py", line 105, in requires
workflow_output = self.workflow()
File "/Users/mikes/Documents/Eclipseworkspace/Bioproximity/OpenMS-Python-Luigi/site-packages/Bioproximity/sciluigi_tasks/PipelineTest1.py", line 33, in workflow
exepath = "/Applications/OpenMS-2.1.0/TOPP"
File "/Library/Python/2.7/site-packages/sciluigi/workflow.py", line 145, in new_task
newtask = sciluigi.new_task(instance_name, cls, self, **kwargs)
File "/Library/Python/2.7/site-packages/sciluigi/task.py", line 37, in new_task
newtask = cls.from_str_params(kwargs)
File "/Library/Python/2.7/site-packages/luigi/task.py", line 412, in from_str_params
return cls(**kwargs)
File "/Library/Python/2.7/site-packages/luigi/task_register.py", line 99, in __call__
h[k] = instantiate()
File "/Library/Python/2.7/site-packages/luigi/task_register.py", line 80, in instantiate
return super(Register, cls).__call__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'instance_name'
The second issue is:
If I wait for self to return, I can no longer differentiate between (for example using the above code)...
in_target = None # Mandatory
out = sciluigi.Parameter() # <file>
If I do a type(out), type reports that the parameter is just a string (not a sciluigi.Parameter object) so if I try to use a `ìsinstance(out, sciluigi.Parameter)...it returns False.
The bottom line is:
I need to be able to set the sciluigi.Parameter objects dynamically (programatically) and subsequently be able to differentiate between a sciluigi.Parameter() object variable (like out) and a 'real' str() object (like in_target)
I hope this makes sense.
Accept and silently discard arguments in your constructor, like so:
class FileConverter(sciluigi.Task):
def __init__(self, *_args, **_kwargs):
self.var = 'anything'
Just for future reference, the answer to the isolated question, "How to...
setattr(<thisClassObject>, 'DynamicVar', sciluigi.Parameter(default='Yes') )
Is to use the locals() built in function. I.e.
locals()['DynamicVar'] = sciluigi.Parameter(default='Yes') #String only
This is a snippet of how I solved my particular kluge ;)
deleteme.py
import sciluigi
class MyFooWriter(sciluigi.Task):
locals()['outfile'] = sciluigi.Parameter(default='./foo.txt') #String only
locals()['normalvar'] = 'Normalstring'
print "pre-self-returned outfile type =", type(outfile)
print "pre-self-returned normalvar type =", type(normalvar)
# locals()['param1'] =
def out_foo(self):
# raw_input("Enter...")
return sciluigi.TargetInfo(self, self.outfile)
def run(self):
print "self.outfile type =", type(self.outfile)
print "self.normalvar type =", type(self.normalvar)
# raw_input("Enter...")
with self.out_foo().open('w') as foofile:
foofile.write('foo\n')
class MyWorkflow(sciluigi.WorkflowTask):
def workflow(self):
print 'Starting workflow...'
foowriter = self.new_task('foowriter', MyFooWriter, outfile = 'testfile.txt')
return foowriter
if __name__ == '__main__':
sciluigi.run_local(main_task_cls=MyWorkflow)
OUTPUT
pre-self-returned outfile type = <class 'sciluigi.parameter.Parameter'>
pre-self-returned normalvar type = <type 'str'>
Starting workflow...
2017-02-27 12:08:37 | INFO | --------------------------------------------------------------------------------
2017-02-27 12:08:37 | INFO | SciLuigi: MyWorkflow Workflow Started (logging to log/workflow_myworkflow_started_20170227_110837_278707.log)
2017-02-27 12:08:37 | INFO | --------------------------------------------------------------------------------
2017-02-27 12:08:37 | INFO | Task foowriter started
self.outfile type = <type 'str'>
self.normalvar type = <type 'str'>
2017-02-27 12:08:37 | INFO | Task foowriter finished after 0.001s
Starting workflow...
2017-02-27 12:08:37 | INFO | --------------------------------------------------------------------------------
2017-02-27 12:08:37 | INFO | SciLuigi: MyWorkflow Workflow Finished (workflow log at log/workflow_myworkflow_started_20170227_110837_278707.log)
2017-02-27 12:08:37 | INFO | --------------------------------------------------------------------------------
I have the following file (written for use with py.test):
# test_fuser.py
import subprocess
def fuser(filename):
return subprocess.check_output(['fuser', filename]).split()
def test_fuser_0(tmpdir):
test_file = tmpdir.join('test.txt')
test_file.write('')
assert len(fuser(test_file.strpath)) == 0
def test_fuser_1(tmpdir):
test_file = tmpdir.join('test.txt')
test_file.write('')
with open(test_file.strpath, 'w'):
assert len(fuser(test_file.strpath)) == 1
Running py.test from the command-line yields:
$ py.test test_fuser.py
================================= test session starts ==================================
platform darwin -- Python 2.7.8 -- py-1.4.26 -- pytest-2.6.4
collected 2 items
test_fuser.py .F
======================================= FAILURES =======================================
_____________________________________ test_fuser_1 _____________________________________
tmpdir = local('/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-131/test_fuser_10')
def test_fuser_1(tmpdir):
test_file = tmpdir.join('test.txt')
test_file.write('')
with open(test_file.strpath, 'w'):
> assert len(fuser(test_file.strpath)) == 1
E assert 2 == 1
E + where 2 = len(['71433', '71437'])
E + where ['71433', '71437'] = fuser('/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-131/test_fuser_10/test.txt')
E + where '/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-131/test_fuser_10/test.txt' = local('/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-131/test_fuser_10/test.txt').strpath
test_fuser.py:18: AssertionError
--------------------------------- Captured stderr call ---------------------------------
/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-131/test_fuser_10/test.txt:
========================= 1 failed, 1 passed in 10.32 seconds =========================
This is unusual because one would expect fuser() to return one PID, not two, since only the call to open() should have the file open.
To investigate this further, I set a breakpoint in test_fuser_1() right before the call to fuser(), within the context of open() and then reran the test.
def test_fuser_1(tmpdir):
test_file = tmpdir.join('test.txt')
test_file.write('')
with open(test_file.strpath, 'w'):
pytest.set_trace()
assert len(fuser(test_file.strpath)) == 1
At the break point I grabbed the name of the temporary file:
>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>
> /Users/caesarbautista/Desktop/test_fuser.py(21)test_fuser_1()
-> assert len(fuser(test_file.strpath)) == 1
(Pdb) test_file.strpath
'/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-132/test_fuser_10/test.txt'
Then, calling fuser from another terminal I see one PID as expected.
$ fuser /private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-132/test_fuser_10/test.txt
/private/var/folders/mx/h4mybl9x1p52wbj2hnvk1lgh0000gn/T/pytest-132/test_fuser_10/test.txt: 71516
This seems to suggest that subprocess is responsible for the second PID, but I'm not certain why that might be and have run out of ideas for how to investigate this further. Any idea where the second PID is coming from?
I though perhaps py.test might be connected to this, so I rewrote the test for use with nose:
def test_fuser_0():
with open('test.txt', 'w') as fp:
fp.write('')
pids = fuser('test.txt')
assert len(pids) == 0, pids
def test_fuser_1():
with open('test.txt', 'w') as fp:
fp.write('')
pids = fuser('test.txt')
assert len(pids) == 1, pids
But running them with nose resulted in the same failure:
$ nosetests --tests=test_fuser.py
test.txt:
.test.txt:
F
======================================================================
FAIL: test_fuser.test_fuser_1
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/boxen/homebrew/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/caesarbautista/Desktop/test_fuser.py", line 34, in test_fuser_1
assert len(pids) == 1, pids
AssertionError: ['71865', '71869']
----------------------------------------------------------------------
Ran 2 tests in 11.026s
FAILED (failures=1
Let's say I have the following multi-line string:
cmd = """
a = 1 + 1
b = [
2 + 2,
4 + 4,
]
bork bork bork
"""
and I want to execute it in a particular scope:
scope = {}
exec( cmd, scope )
print scope[ 'b' ]
There's a SyntaxError at line 6 of the command, and I want to be able to report that to the user. How do I get the line number? I've tried this:
try:
exec( cmd, scope ) # <-- let's say this is on line 123 of the source file
except Exception, err:
a, b, c = sys.exc_info()
line_number = c.tb_lineno # <-- this gets me 123, not 6
print "%s at line %d (%s)" % ( a, line_number, b.message )
...but I get the line number of the exec statement, not the line number within the multi-line command.
Update: it turns out the handling of the type of exception that I arbitrarily chose for this example, the SyntaxError, is different from the handling of any other type. To clarify, I'm looking a solution that copes with any kind of exception.
For syntax errors, the source line number is available as the lineno flag on the exception object itself, in your case stored in err. This is specific to syntax errors where the line number is an integral part of the error:
>>> cmd = """
... 1 \ +
... 2 * "
... """
>>> try:
... exec cmd
... except SyntaxError as err:
... print err.lineno
...
2
If you want to also handle other errors, add a new except block except Exception, err, and use the traceback module to compute the line number for the runtime error.
import sys
import traceback
class InterpreterError(Exception): pass
def my_exec(cmd, globals=None, locals=None, description='source string'):
try:
exec(cmd, globals, locals)
except SyntaxError as err:
error_class = err.__class__.__name__
detail = err.args[0]
line_number = err.lineno
except Exception as err:
error_class = err.__class__.__name__
detail = err.args[0]
cl, exc, tb = sys.exc_info()
line_number = traceback.extract_tb(tb)[-1][1]
else:
return
raise InterpreterError("%s at line %d of %s: %s" % (error_class, line_number, description, detail))
Examples:
>>> my_exec("1+1") # no exception
>>>
>>> my_exec("1+1\nbork")
...
InterpreterError: NameError at line 2 of source string: name 'bork' is not defined
>>>
>>> my_exec("1+1\nbork bork bork")
...
InterpreterError: SyntaxError at line 2 of source string: invalid syntax
>>>
>>> my_exec("1+1\n'''")
...
InterpreterError: SyntaxError at line 2 of source string: EOF while scanning triple-quoted string
Sorry if this question is stupid. I created an unittest class which needs to take given inputs and outputs from outside. Thus, I guess these values should be initiated. However, I met some errors in the following code:
CODE:
import unittest
from StringIO import StringIO
##########Inputs and outputs from outside#######
a=[1,2]
b=[2,3]
out=[3,4]
####################################
def func1(a,b):
return a+b
class MyTestCase(unittest.TestCase):
def __init__(self,a,b,out):
self.a=a
self.b=b
self.out=out
def testMsed(self):
for i in range(self.tot_iter):
print i
fun = func1(self.a[i],self.b[i])
value = self.out[i]
testFailureMessage = "Test of function name: %s iteration: %i expected: %i != calculated: %i" % ("func1",i,value,fun)
self.assertEqual(round(fun,3),round(value,3),testFailureMessage)
if __name__ == '__main__':
f = MyTestCase(a,b,out)
from pprint import pprint
stream = StringIO()
runner = unittest.TextTestRunner(stream=stream, verbosity=2)
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
print 'Tests run', result.testsRun
However, I got the following error
Traceback (most recent call last):
File "C:testing.py", line 33, in <module>
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
File "C:\Python27\lib\unittest\loader.py", line 310, in makeSuite
return _makeLoader(prefix, sortUsing, suiteClass).loadTestsFromTestCase(testCaseClass)
File "C:\Python27\lib\unittest\loader.py", line 50, in loadTestsFromTestCase
if issubclass(testCaseClass, suite.TestSuite):
TypeError: issubclass() arg 1 must be a class
Can anyone give me some suggestions? Thanks!
The root of the problem is this line,
result = runner.run(unittest.makeSuite(MyTestCase(a,b,out)))
unittest.makeSuite expects a class, not an instance of a class. So just MyTestCase, not MyTestCase(a, b, out). This means that you can't pass parameters to your test case in the manner you are attempting to. You should probably move the code from init to a setUp function. Either access a, b, and out as globals inside setUp or take a look at this link for information regarding passing parameters to a unit test.
By the way, here is the source file within python where the problem originated. Might be informative to read.