I'm trying to write examples of how NOT to write tests in pytest. For this, I'd like to be able to somehow mod test functions so that they fail if any assert passes. This is easy with one assert using a decorator:
def faulty_test(test_fn):
def new_test_fn():
try:
test_fn()
except AssertionError:
pass
else:
raise AssertionError
return new_test_fn
#faulty_test
def test_sth():
assert 1 == 2
but I'd like to do it for a function with any number of asserts. The test should fail if any of the asserts pass. It doesn't have to be a decorator
Perhaps you should think about it this way: At the point of the exception, execution leaves the try block and enters the except block. You cannot, at least in any simple way I can think of, re-enter the try block from within the except block.
And perhaps I'm misunderstanding the question, but couldn't you simply replace your assertions with the inverse of those assertions? If you want to assert that a thing is false, you can:
assert not (1 == 2)
If for some reason you're dealing with a situation where you assume that the given functions will, for some reason, always fail their asserts and you cannot change that, there does not seem to be any way to ensure all the asserts run, since the design of assertions is to intentionally abort immediately on failure.
I guess it doesn't have to be multiple asserts. You can write:
assert a == b or c == d or # so on
Any condition is True will cause the assertion to pass.
If you "number" the assertions, catching an assertion with a value greater than 0 implies the previous assertions passed.
def faulty_test(test_fn):
def new_test_fn():
try:
test_fn()
except AssertionError as exc:
if exc.args[0] > 0:
# Assertions 0 through exc.args[0]-1 all passed
raise AssertionError
else:
raise AssertionError
return new_test_fn
#faulty_test
def test_sth():
assert 1 == 2, 0
assert 2 == 3, 1
assert 3 == 4, 2
You should probably handle that in the function code by using a _debug or other (hidden) argument. Then you can use a decorator on that. What I would do:
def deco(inF):
def wrapper(*args, **kwargs):
kwargs['_debug'] = True
output = inF(*args, **kwargs)
#print(output)
for i, _ in enumerate(zip(*output)):
condition, msg = _
# this will raise the first passed assert. Otherwise loop again to print all "passed" assertions, then raise
if not condition:
raise AssertionError('Condition {} succeded: {} is False'.format(1 + i, msg))
return output
return wrapper
#deco
def f(i, j , k, _debug=False):
cond1, msg1 = i == 1, 'i is not 1'
cond2, msg2 = j == 2, 'j is not 2'
cond3, msg3 = k == 3, 'k is not 3'
assert cond1 or _debug, msg1
assert cond2 or _debug, msg2
assert cond3 or _debug, msg3
if _debug:
return (cond1, cond2, cond3), (msg1, msg2, msg3)
return i + j + k
f(1,1,5)
>>AssertionError: Condition 2 succeded: j is not 2 is False
Related
Below is the full test code where each assertion is executed. This is unintuitive for me for one reason. If the value of the variable k is None then the function t throws an exception, and thus the code after calling t should not be executed and the exception should be caught by the context manager. However, this does not happen and I do not know why. Not that it bothers me, it's even fantastic that it executes this way, but I'd like to know why.
from contextlib import nullcontext as does_not_raise
import pytest
def t(k):
if k:
return k
else:
raise ValueError("Value")
#pytest.mark.parametrize("k, cntxt", [(None, pytest.raises(ValueError)), ("Value", does_not_raise())])
def test_t(k, cntxt):
with cntxt as ex:
kk = t(k)
if k:
assert kk == k
assert ex is None
else:
assert kk is None
assert str(ex.value) == "Value"
If the value of the variable k is None then the function t throws an exception, and thus the code after calling t should not be executed and the exception should be caught by the context manager.
Unless I have misunderstood you, I believe this is exactly what is happening. In the case when k is None then these lines aren't executed:
assert kk is None
assert str(ex.value) == "Value"
To be sure that is the case, you can change the test function to:
#pytest.mark.parametrize("k, cntxt", [(None, pytest.raises(ValueError)), ("Value", does_not_raise())])
def test_t(k, cntxt):
with cntxt as ex:
kk = t(k)
if k is None:
assert False
and it won't fail.
You may want to move the assertions outside of the context manager like so
#pytest.mark.parametrize("k, cntxt", [(None, pytest.raises(ValueError)), ("Value", does_not_raise())])
def test_t(k, cntxt):
with cntxt as ex:
kk = t(k)
if k:
print("one")
assert kk == k
assert ex is None
else:
print("two")
# Can't do this as kk was never assigned to anything
# assert kk is None
assert str(ex.value) == "Value"
$ pytest -s myfile.py
...
two
.
one
.
...
When k=None in your example, the code after kk = t(k) is not executed.
However, because everything is inside a with block, which is essentially a convenience syntax for try-except-finally, the code execution continues after the block, assert statements are not executed and your test is marked as passed while it should have failed.
https://docs.python.org/3/reference/compound_stmts.html#the-with-statement
You should put the assert statements outside the with block
Can assertions compare against print statements in python like this:
def printname(name):
print(name)
#to not raise an error:
assert printname("Hello") == "Hello"
#to raise an error:
assert printname("Hello") == "notHello"
No, assertions can't do that. If you want to do that try this:
def printname(name):
print(name)
return name
#to not raise an error:
assert printname("Hello") == "Hello"
#to raise an error:
assert printname("Hello") == "notHello"
This technically doesn't compare it, but its a valid workaround. If you really want to compare them take a look into StringIO.
Try using mock + stringIO to capture stdout/err:
from mock import patch
from StringIO import StringIO
def test_foobar():
out, err = StringIO(), StringIO()
with patch.multiple(sys, stdout=out, stderr=err):
do_stuff()
assert out.getvalue() == 'foo'
assert err.getvalue() == 'bar'
Also if you are doing this within a test framework, you may be able to do a variety of things. In particular for pytest:
def test_myoutput(capsys): # or use "capfd" for fd-level
print("hello")
sys.stderr.write("world\n")
captured = capsys.readouterr()
assert captured.out == "hello\n"
assert captured.err == "world\n"
print("next")
captured = capsys.readouterr()
assert captured.out == "next\n"
No. The function printname doesn't have an explicit return statement, so it return the value . So the assert compares None against a string and of course it raise an assertion error.
Is there any way in Python2.7, we can capture and log the assert statements in general python scripting despite of assert is True or False
Suppose I assert following line in code:
assert len(lst) ==4
So is any way can log what statement is passed, at what line and it is true or false. I do not want to use a wrapper function, looking for something inbuilt in python .
Note: What I want to achieve is , suppose I have legacy code having 1000's of assert statements , without changing the code, I should be able to log which assert statement is executed and what is the output, Is its achievable in python 2.7.
try:
assert len(lst) == 4
print "True"
except AssertionError as e:
print "False"
Yes, you can define a custom excepthook to log some extra information:
import sys
import logging
def excepthook(*args):
logging.getLogger().error('Uncaught exception:', exc_info=args)
sys.excepthook = excepthook
assert 1==2
EDIT: Whoops I forgot you wanted to log even if it's true :) oh well I'll leave this for a little bit in case it informs you or someone else...
This is what I could achieve closet, as does not seem it is possible in Python2.7 . I created a wrapper function.
import inspect
def assertit(condition,message):
# The caller method
(frame, filename, line_number,function_name, lines, index) = inspect.getouterframes(inspect.currentframe())[1]
detailed_message=" => {message} [ {filename} : {line_number}]".format(message=message,line_number=line_number,
filename=filename)
if condition:
print "True %s"%detailed_message
return
raise AssertionError("False: %s"%detailed_message)
assertit(1==1,"Check if 1 equal 1")
assertit(1==2,"Check if 1 equal 2")
### HERE IS THE OUTPUT
True => Check if 1 equal 1 [ test_cases.py : 20]
Traceback (most recent call last):
File "test_cases.py", line 21, in <module>
assertit(1==2,"Check if 1 equal 2")
File "test_cases.py", line 19, in assertit
raise AssertionError("False: %s"%detailed_message)
AssertionError: False: => Check if 1 equal 2 [ test_cases.py : 21]
This is a proof of concept. So please downvote, or I will delete it...
The idea is to replace the assert statement with print statement at the execution.
import ast
import inspect
from your_module import main
def my_assert(test_result, msg):
assert test_result, msg
# if test_result is True, just print the msg
return "assert: {}, {}".format(test_result, msg)
class Transformer(ast.NodeTransformer):
def visit_Assert(self, node):
f = ast.Name(id='my_assert', ctx=ast.Load())
c = ast.Call(func=f, args=[node.test, node.msg], keywords=[])
p = ast.Print(values=[c], nl=True)
# set line#, column offset same as the original assert statement
f.lineno = c.lineno = p.lineno = node.lineno
f.col_offset =c.col_offset = p.col_offset = node.col_offset
return p
def replace_assert_with_print(func):
source = inspect.getsource(func)
ast_tree = ast.parse(source)
Transformer().visit(ast_tree)
return compile(ast_tree, filename="<ast>", mode="exec")
if __name__ == '__main__':
exec(replace_assert_with_print(main))
main(4)
And here is your_module.py
def main(x):
assert x == 4, "hey {} is not 4".format(x)
return x
I'm writing a very basic Python script based on a main function that sequentially calls other functions.
What I would like to do is to wrap all of the functions that are called from main with something like:
result = func(*some_args):
if (result != True):
return result
For example, for this code:
def func1(arg1 , arg2):
if (some_cond):
return False #Or some err_val
return True
def func2(arg1):
if (some_cond):
return False
return True
def main():
func1(val1 , val2)
func2(val3)
return True
if __name__ == "__main__":
import sys
result = main()
if (result == err_val1):
# Do something. Maybe print. Maybe call some function.
sys.exit(1)
I want that if one of the functions fails main would break and return its error. Can I do this using decorators?
I maintain that the best solution would be the use of exceptions. If that's absolutely not what you want, you can do some short-circuiting:
return func1() and func2()
To extend this to more functions without a ton of ands:
from functools import partial
def main():
funcs = (partial(func1, arg1, arg2),
partial(func2, arg1))
if any(!f() for f in funcs):
return False
Though this doesn't return "its error" (the failed function's error), it just returns False. If you want to differentiate more between different kinds of errors... well, you're back to exceptions.
This is precisely what exceptions are built for in Python.
# imports belong at the *top* of the file
import sys
class SomeDescriptiveError(Exception): pass
class SomeOtherSpecialError(Exception): pass
def func1(arg1 , arg2):
if (some_cond):
raise SomeDescriptiveError('Cannot frobnosticate the fizzbuzz')
return arg1 + arg2
# or skip the return statement altogether
def func2(arg1):
if (some_cond):
raise SomeOtherSpecialError('The frontobulator is no longer cromulent')
return ''.join(reversed(arg1))
def main():
print(func1(val1 , val2))
print(func2(val3))
if __name__ == "__main__":
try:
result = main()
except SomeDescriptiveError as e:
print('Oh dear')
sys.exit(e.args[0])
except SomeOtherSpecialError as e:
print('Oh no')
sys.exit(e.args[0])
else:
print('All systems are fully operational')
finally:
print('I really should clean up all these bits.')
Since you do actually want the program to die when one of these errors occurs you might as well raise SystemExit. Here's a way to do it with a decorator.
flag = 2
def die_on_not_True(func):
def wrapper(*args):
rc = func(*args)
if rc is not True:
fmt = 'Function {} failed with return value {!r}'
print(fmt.format(func.__name__, rc))
raise SystemExit(1)
return True
return wrapper
#die_on_not_True
def func1(arg1 , arg2):
if arg1 == flag:
return 'error 1'
return True
#die_on_not_True
def func2(arg1):
if arg1 == flag:
return 'error 2'
return True
def main():
val1, val2, val3 = 1, 2, 3
print(func1(val1, val2))
print('one')
print(func2(val3))
print('two')
if __name__ == '__main__':
main()
output
True
one
True
two
If we set flag = 1, the output becomes
Function func1 failed with return value 'error 1'
If we set flag = 3, the output becomes
True
one
Function func2 failed with return value 'error 2'
When flag equals 2, the exit status of 0 is returned to the shell, when flag equals 1 or 3, the exit status of 1 is returned.
If you want to do further processing after printing the error message, then raise a custom exception instead of SystemExit and catch it by wrapping your main call in a try...except.
I guess, what you really want is a universal exception catcher, that would catch and return the exception of any wrapped function. You can easily do it this way.
def return_exception(func):
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
return e
return wrapper
Example
In [3]: #return_exception
...: def div(a, b):
...: return a / b
...:
In [4]: div(1, 0)
Out[4]: ZeroDivisionError('division by zero')
So then you can process the return exception object the way you want, though it's pretty hard to say why you need that.
Update As others have noted it's generally good to only catch particular exceptions. You can modify the decorator slightly.
def return_exception(*exception_types):
def build_wrapper(func):
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except exception_types as e:
return e
return wrapper
return build_wrapper
Example:
In [6]: #return_exception(ZeroDivisionError)
...: def div(a, b):
...: return a / b
...:
In [7]: div(0, 1)
Out[7]: 0.0
In [8]: div(1, 0)
Out[8]: ZeroDivisionError('division by zero')
In [9]: div(1, "a")
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
...
TypeError: unsupported operand type(s) for /: 'int' and 'str'
In [10]: #return_exception(ZeroDivisionError, TypeError)
....: def div(a, b):
....: return a / b
....:
In [11]: div(1, 0)
Out[11]: ZeroDivisionError('division by zero')
In [12]: div(1, "a")
Out[12]: TypeError("unsupported operand type(s) for /: 'int' and 'str'")
As you see, you only catch the specified exceptions (you can still specify the universal Exception class, though).
I have a problem with my code in the try block.
To make it easy this is my code:
try:
code a
code b #if b fails, it should ignore, and go to c.
code c #if c fails, go to d
code d
except:
pass
Is something like this possible?
You'll have to make this separate try blocks:
try:
code a
except ExplicitException:
pass
try:
code b
except ExplicitException:
try:
code c
except ExplicitException:
try:
code d
except ExplicitException:
pass
This assumes you want to run code c only if code b failed.
If you need to run code c regardless, you need to put the try blocks one after the other:
try:
code a
except ExplicitException:
pass
try:
code b
except ExplicitException:
pass
try:
code c
except ExplicitException:
pass
try:
code d
except ExplicitException:
pass
I'm using except ExplicitException here because it is never a good practice to blindly ignore all exceptions. You'll be ignoring MemoryError, KeyboardInterrupt and SystemExit as well otherwise, which you normally do not want to ignore or intercept without some kind of re-raise or conscious reason for handling those.
You can use fuckit module.
Wrap your code in a function with #fuckit decorator:
#fuckit
def func():
code a
code b #if b fails, it should ignore, and go to c.
code c #if c fails, go to d
code d
Extract (refactor) your statements. And use the magic of and and or to decide when to short-circuit.
def a():
try: # a code
except: pass # or raise
else: return True
def b():
try: # b code
except: pass # or raise
else: return True
def c():
try: # c code
except: pass # or raise
else: return True
def d():
try: # d code
except: pass # or raise
else: return True
def main():
try:
a() and b() or c() or d()
except:
pass
If you don't want to chain (a huge number of) try-except clauses, you may try your codes in a loop and break upon 1st success.
Example with codes which can be put into functions:
for code in (
lambda: a / b,
lambda: a / (b + 1),
lambda: a / (b + 2),
):
try: print(code())
except Exception as ev: continue
break
else:
print("it failed: %s" % ev)
Example with arbitrary codes (statements) directly in the current scope:
for i in 2, 1, 0:
try:
if i == 2: print(a / b)
elif i == 1: print(a / (b + 1))
elif i == 0: print(a / (b + 2))
break
except Exception as ev:
if i:
continue
print("it failed: %s" % ev)
Lets say each code is a function and its already written then the following can be used to iter through your coding list and exit the for-loop when a function is executed without error using the "break".
def a(): code a
def b(): code b
def c(): code c
def d(): code d
for func in [a, b, c, d]: # change list order to change execution order.
try:
func()
break
except Exception as err:
print (err)
continue
I used "Exception " here so you can see any error printed. Turn-off the print if you know what to expect and you're not caring (e.g. in case the code returns two or three list items (i,j = msg.split('.')).
You could try a for loop
for func,args,kwargs in zip([a,b,c,d],
[args_a,args_b,args_c,args_d],
[kw_a,kw_b,kw_c,kw_d]):
try:
func(*args, **kwargs)
break
except:
pass
This way you can loop as many functions as you want without making the code look ugly
I use a different way, with a new variable:
continue_execution = True
try:
command1
continue_execution = False
except:
pass
if continue_execution:
try:
command2
except:
command3
to add more commands you just have to add more expressions like this:
try:
commandn
continue_execution = False
except:
pass
I ran into this problem, but then it was doing the things in a loop which turned it into a simple case of issueing the continue command if successful. I think one could reuse that technique if not in a loop, at least in some cases:
while True:
try:
code_a
break
except:
pass
try:
code_b
break
except:
pass
etc
raise NothingSuccessfulError
Like Elazar suggested:
"I think a decorator would fit here."
# decorator
def test(func):
def inner(*args, **kwargs):
try:
func(*args, **kwargs)
except: pass
return inner
# code blocks as functions
#test
def code_a(x):
print(1/x)
#test
def code_b(x):
print(1/x)
#test
def code_c(x):
print(1/x)
#test
def code_d(x):
print(1/x)
# call functions
code_a(0)
code_b(1)
code_c(0)
code_c(4)
output:
1.0
0.25