My program is producing natural language sentences.
I would like to test it properly by setting the random seed to a fix value and then:
producing expected results;
comparing generated sentences with expected results;
if they differ, asking the user if the generated sentences were actually the expected results, and in this case, updating the expected results.
I already met such systems in JS, so I am surprised for not finding it in Python. How do you deal with such situations?
There are many testing frameworks in Python, with two of the most popular being PyTest and Nose. PyTest tends to cover all the bases, but Nose has a lot of nice extras as well.
With nose, fixtures are covered early on in the docs. The example they give looks something like
def setup_func():
"set up test fixtures"
def teardown_func():
"tear down test fixtures"
#with_setup(setup_func, teardown_func)
def test():
"test ..."
In your case, with the manual review, you may need to build that logic directly into the test itself.
Edit with a more specific example
Building on the example from Nose, one way you could address this is by writing the test
from nose.tools import eq_
def setup_func():
"set your random seed"
def teardown_func():
"whatever teardown you need"
#with_setup(setup_func, teardown_func)
def test():
expected = "the correct answer"
actual = "make a prediction ..."
_eq(expected, actual, "prediction did not match!")
When you run your tests, if the model does not produce the correct output, the tests will fail with "prediction did not match!". In that case, you should go to your test file and update expected with the expected value. This procedure isn't as dynamic as typing it in at runtime, but it has the advantage of being easily versioned and controlled.
One drawback of asking the user to replace the expected answer is that the automated test can not be run automatically. Therefore, test frameworks do not allow reading from input.
I really wanted this feature, so my implementation looks like:
def compare_results(expected, results):
if not os.path.isfile(expected):
logging.warning("The expected file does not exist.")
elif filecmp.cmp(expected, results):
logging.debug("%s is accepted." % expected)
return
content = Path(results).read_text()
print("The test %s failed." % expected)
print("Should I accept the results?")
print(content)
while True:
try:
keep = input("[y/n]")
except OSError:
assert False, "The test failed. Run directly this file to accept the result"
if keep.lower() in ["y", "yes"]:
Path(expected).write_text(content)
break
elif keep.lower() in ["n", "no"]:
assert False, "The test failed and you did not accept the answer."
break
else:
print("Please answer by yes or no.")
def test_example_iot_root(setup):
...
compare_results(EXPECTED_DIR / "iot_root.test", tmp.name)
if __name__ == "__main__":
from inspect import getmembers, isfunction
def istest(o):
return isfunction(o[1]) and o[0].startswith("test")
[random.seed(1) and o[1](setup) for o in getmembers(sys.modules[__name__]) \
if istest(o)]
When I run directly this file, it asks me whether or not it should replace the expected results. When I run from pytest, input creates an OSError that allows to quit the loop. Definitely not perfect.
Related
I might be guilty of using pytest in a way I'm not supposed to, but let's say I want to generate and display some short message on successful test completion. Like I'm developing a compression algorithm, and instead of "PASSED" I'd like to see "SQUEEZED BY 65.43%" or something like that. Is this even possible? Where should I start with the customization, or maybe there's a plugin I might use?
I've stumbled upon pytest-custom-report, but it provides only static messages, that are set up before tests run. That's not what I need.
I might be guilty of using pytest in a way I'm not supposed to
Not at all - this is exactly the kind of use cases the pytest plugin system is supposed to solve.
To answer your actual question: it's not clear where the percentage value comes from. Assuming it is returned by a function squeeze(), I would first store the percentage in the test, for example using the record_property fixture:
from mylib import squeeze
def test_spam(record_property):
value = squeeze()
record_property('x', value)
...
To display the stored percentage value, add a custom pytest_report_teststatus hookimpl in a conftest.py in your project or tests root directory:
# conftest.py
def pytest_report_teststatus(report, config):
if report.when == 'call' and report.passed:
percentage = dict(report.user_properties).get('x', float("nan"))
short_outcome = f'{percentage * 100}%'
long_outcome = f'SQUEEZED BY {percentage * 100}%'
return report.outcome, short_outcome, long_outcome
Now running test_spam in default output mode yields
test_spam.py 10.0% [100%]
Running in the verbose mode
test_spam.py::test_spam SQUEEZED BY 10.0% [100%]
I am looking for a way to access the return value of a test function in order to include that value in a test report file (similar to http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures).
Code example that I would like to use:
# modified example code from http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures
import pytest
import os.path
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
# execute all other hooks to obtain the report object
outcome = yield
rep = outcome.get_result()
if rep.when == "call" and rep.passed:
mode = "a" if os.path.exists("return_values") else "w"
with open("return_values.txt", mode) as f:
# THE FOLLOWING LINE IS THE ONE I CANNOT FIGURE OUT
# HOW DO I ACCESS THE TEST FUNCTION RETURN VALUE?
return_value = item.return_value
f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")
I expect the return value to be written to the file "return_values.txt". Instead, I get an AttributeError.
Background (in case you can recommend a totally different approach):
I have a Python library for data analysis on a given problem. I have a standard set of test data which I routinely run my analysis to produce various "benchmark" metrics on the quality of the analysis algorithms on. For example, one such metric is the trace of a normalized confusion matrix produced by the analysis code (which I would like to be as close to 1 as possible). Another metric is the CPU time to produce an analysis result.
I am looking for a nice way to include these benchmark results into a CI framework (currently Jenkins), such that it becomes easy to see whether a commit improves or degrades the analysis performance. Since I am already running pytest in the CI sequence, and since I would like to use various features of pytest for my benchmarks (fixtures, marks, skipping, cleanup) I thought about simply adding a post-processing hook in pytest (see http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures) that collects test function run time and return values and reports them (or only those which are marked as benchmarks) into a file, which will be collected and archived as a test artifact by my CI framework.
I am open to other ways to solve this problem, but my google search conclusion is that pytest is the framework closest to already providing what I need.
Sharing the same problem, here is a different solution i came up with:
using the fixture record_property in the test:
def test_mytest(record_property):
record_property("key", 42)
and then in conftest.py we can use the pytest_runtest_teardown hook:
#conftest.py
def pytest_runtest_teardown(item, nextitem):
results = dict(item.user_properties)
if not results:
return
with open(f'{item.name}_return_values.txt','a') as f:
for key, value in results.items():
f.write(f'{key} = {value}\n')
and then the content of test_mytest_return_values.txt:
key = 42
Two important notes:
This code will be executed even if the test failed. I couldn't find a way to get the outcome of the test.
This can be combined with heofling's answer using results = dict(item.user_properties) to obtain the keys and values that were added in the test instead of adding a dict to config and then access it in the test.
pytest ignores test functions return value, as can be seen in the code:
#hookimpl(trylast=True)
def pytest_pyfunc_call(pyfuncitem):
testfunction = pyfuncitem.obj
...
testfunction(**testargs)
return True
You can, however, store anything you need in the test function; I usually use the config object for that. Example: put the following snippet in your conftest.py:
import pathlib
import pytest
def pytest_configure(config):
# create the dict to store custom data
config._test_results = dict()
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
# execute all other hooks to obtain the report object
outcome = yield
rep = outcome.get_result()
if rep.when == "call" and rep.passed:
# get the custom data
return_value = item.config._test_results.get(item.nodeid, None)
# write to file
report = pathlib.Path('return_values.txt')
with report.open('a') as f:
f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")
Now store the data in tests:
def test_fizz(request):
request.config._test_results[request.node.nodeid] = 'mydata'
In the following code,
1.If we assert the end result of a function then is it right to say that we have covered all lines of the code while testing or we have to test each line of the code explicitly if so how ?
2.Also Is it fine that we can have the positive ,negative and more assert statements in one single test function.If not please give examples
def get_wcp(book_id):
update_tracking_details(book_id)
c_id = get_new_supreme(book_id)
if book_id == c_id:
return True
return False
class BookingentitntyTest(TestCase):
def setUp(self):
self.booking_id = 123456789# 9 digit number poistive test case
def test_get_wcp(self):
retVal = get_wcp(self.book_id)
self.assertTrue( retVal )
self.booking_id = 1# 1 digit number Negative test case
retVal = get_wcp(self.book_id)
self.assertFalse( retVal )
No, just because you asserted the final result of the statement doesn't mean all paths of your code has been evaluated. You don't need to write "test each line" in the sense that you only need to go through all the code paths possible.
As a general guideline, try to keep the number of assert statements in a test as minimum as possible. When one assert statement fails, the further statements are not executed, and doesn't count as failures which is often not required.
Besides try to write your tests as elegantly as possible, even more so than the code they are testing. We wouldn't want to write tests to test our tests, now would we?
def test_get_wcp_returns_true_when_valid_booking_id(self):
self.assertTrue(get_wcp(self.book_id))
def test_get_wcp_returns_false_if_invalid_booking_id(self):
self.assertFalse(get_wcp(1))
Complete bug-free code is not possible.
"Testing shows the presence, not the absence of bugs" - Edsger Dijkstra.
I am writing some Python unit tests using the "unittest" framework and run them in PyCharm. Some of the tests compare a long generated string to a reference value read from a file. If this comparison fails, I would like to see the diff of the two compared strings using PyCharms diff viewer.
So the the code is like this:
actual = open("actual.csv").read()
expected = pkg_resources.resource_string('my_package', 'expected.csv').decode('utf8')
self.assertMultiLineEqual(actual, expected)
And PyCharm nicely identifies the test as a failure and provides a link in the results window to click which opens the diff viewer. However, due to how unittest shortens the results, I get results such as this in the diff viewer:
Left side:
'time[57 chars]ercent
0;1;1;1;1;1;1;1
0;2;1;3;4;2;3;1
0;3;[110 chars]32
'
Right side:
'time[57 chars]ercen
0;1;1;1;1;1;1;1
0;2;1;3;4;2;3;1
0;3;2[109 chars]32
'
Now, I would like to get rid of all the [X chars] parts and just see the whole file(s) and the actual diff fully visualized by PyCharm.
I tried to look into unittest code but could not find a configuration option to print full results. There are some variables such as maxDiff and _diffThreshold but they have no impact on this print.
Also, I tried to run this in py.test but there the support in PyCharm was even less (no links even to failed test).
Is there some trick using the difflib with unittest or maybe some other tricks with another Python test framework to do this?
The TestCase.maxDiff=None answers given in many places only make sure that the diff shown in the unittest output is of full length. In order to also get the full diff in the <Click to see difference> link you have to set MAX_LENGTH.
import unittest
# Show full diff in unittest
unittest.util._MAX_LENGTH=2000
Source: https://stackoverflow.com/a/23617918/1878199
Well, I managed to hack myself around this for my test purposes. Instead of using the assertEqual method from unittest, I wrote my own and use that inside the unittest test cases. On failure, it gives me the full texts and the PyCharm diff viewer also shows the full diff correctly.
My assert statement is in a module of its own (t_assert.py), and looks like this
def equal(expected, actual):
msg = "'"+actual+"' != '"+expected+"'"
assert expected == actual, msg
In my test I then call it like this
def test_example(self):
actual = open("actual.csv").read()
expected = pkg_resources.resource_string('my_package', 'expected.csv').decode('utf8')
t_assert.equal(expected, actual)
#self.assertEqual(expected, actual)
Seems to work so far..
A related problem here is that unittest.TestCase.assertMultiLineEqual is implemented with difflib.ndiff(). This generates really big diffs that contain all shared content along with the differences. If you monkey patch to use difflib.unified_diff() instead, you get much smaller diffs that are less often truncated. This often avoids the need to set maxDiff.
import unittest
from unittest.case import _common_shorten_repr
import difflib
def assertMultiLineEqual(self, first, second, msg=None):
"""Assert that two multi-line strings are equal."""
self.assertIsInstance(first, str, 'First argument is not a string')
self.assertIsInstance(second, str, 'Second argument is not a string')
if first != second:
firstlines = first.splitlines(keepends=True)
secondlines = second.splitlines(keepends=True)
if len(firstlines) == 1 and first.strip('\r\n') == first:
firstlines = [first + '\n']
secondlines = [second + '\n']
standardMsg = '%s != %s' % _common_shorten_repr(first, second)
diff = '\n' + ''.join(difflib.unified_diff(firstlines, secondlines))
standardMsg = self._truncateMessage(standardMsg, diff)
self.fail(self._formatMessage(msg, standardMsg))
unittest.TestCase.assertMultiLineEqual = assertMultiLineEqual
I'm mildly confused. I'm testing a django application with python's unittest library. All of a sudden, after running my tests with 100 % success for some minutes, suddenly an error appears. Ok I thought, I must have just added some stupid syntax error. I started looking at the test and then my code, I then tried to print out the results which are being compared with assertEqual before they are compared. Suddenly if I do that, the test runs!!! :o
Why is this? has anyone experienced this before. I swear, the only change I made was adding a print statement inside my test function. I'll post this function before and after
Before (Fails)
def test_swap_conditionals(self):
"""
Test conditional template keys
"""
testStr = "My email is: {?email}"
swapStr = self.t.swap(testStr)
# With email
self.assertEqual(swapStr, "My email is: john#baconfactory.com")
# Without email
self.t.template_values = {"phone" : "00458493"}
swapStr = self.t.swap(testStr)
self.assertEqual(swapStr, "My email is: ")
After (Success)
def test_swap_conditionals(self):
"""
Test conditional template keys
"""
testStr = "My email is: {?email}"
swapStr = self.t.swap(testStr)
print(swapStr) #diff here
# With email
self.assertEqual(swapStr, "My email is: john#baconfactory.com")
# Without email
self.t.template_values = {"phone" : "00458493"}
swapStr = self.t.swap(testStr)
self.assertEqual(swapStr, "My email is: ")
It looks like there is some external reason.
What you can check:
Rerun the test several times under the same conditions. Is it always failing or passing? Or is it a 'flipper' test? This might be caused by timing issues (although unlikely).
Put the test in its own class, so there are no side effects from other unit tests.
If the test in its own test class was passing the reason is a side effect by:
other unit tests
startup/teardown functionality
Ok well embarrassing, but this was completely my fault. The swap function was looking up every conditional template variable on the line and then iterating over that list one conditional template variable at a time, so either it missed keys it already had or it got lucky and it happened to hit that key.
Example
line: "This is my {?email}, and this is my {?phone}"
finds:
[{?email}, {?phone}]
iterates over [{?email}, {?phone}]
1. {?email}
key being compared = phone : '00549684'
It has phone as a key but it completely disregards it and does not swap it out because
it's just holding {?email} so it simply returns "".
I'm sincerely sorry to waste all your time here. Thanks for good answers. I am refactoring the code now for the better, and definitely taking a coffee break :D