I cannot figure out how to run single unit tests from within a python script and collect the results.
Scenario: I have a battery of tests that check various methods producing various statistical distributions of different objects. The tests sometimes fail, as they should given that I am basically checking for particular kinds of randomness. I would like to run the tests repeatedly from a script or even from the interpreter and collect results for further analysis.
Suppose I have a module myTest.py with:
class myTest(unittest.TestCase):
def setup(self):
...building objects, etc....
def testTest1(self):
..........
def testTest2(self):
..........
Basically I need to:
run the setup method
run testTest1 (say), 100 times
collect the failures
return the failures
The closest I got to was (using code from a similar question):
from unittest import TextTestRunner, TestSuite
runner = TextTestRunner(verbosity = 2)
tests = ['testTest1']
suite = unittest.TestSuite(map(myTest, tests))
runner.run(suite)
But this does not work, because:
runner.run(suite) does not run the setup method
and
I cannot catch the exception it throws when testTest1 fails
you simply need to add the test that you want to run multiple times to the suite.
Here is a complete code example. You can also see this code running in an interactive Python console to prove that it does actually work.
import unittest
import random
class NullWriter(object):
def write(*_, **__):
pass
def flush(*_, **__):
pass
SETUP_COUNTER = 0
class MyTestCase(unittest.TestCase):
def setUp(self):
global SETUP_COUNTER
SETUP_COUNTER += 1
def test_one(self):
self.assertTrue(random.random() > 0.3)
def test_two(self):
# We just want to make sure this isn't run
self.assertTrue(False, "This should not have been run")
def suite():
tests = []
for _ in range(100):
tests.append('test_one')
return unittest.TestSuite(map(MyTestCase, tests))
results = unittest.TextTestRunner(stream=NullWriter()).run(suite())
print dir(results)
print 'setUp was run', SETUP_COUNTER, 'times'
Related
Is there a way to write (or run) a set of Python unittest tests so that there is no output except when there is a test failure?
For example, if tests/mytests.py contains some unittest tests, then running python3 test/mytests.py will only output to stdout (or stderr) if there is a test failure.
Yes there is. I was able to combine the techniques in the answers to these two questions to get it to work:
Python, unittest: Can one make the TestRunner completely quiet?
argparse and unittest python
You can uncomment the test_should_fail() test to verify what happens when a test fails.
# mymodule.py
def myfunc():
return True
import unittest
import os
class TestMyFunc(unittest.TestCase):
def test_myfunc(self):
self.assertEqual(myfunc(), True)
# def test_should_fail(self):
# self.assertEqual(True, False)
if __name__ == '__main__':
alltests = unittest.TestLoader().loadTestsFromTestCase(TestMyFunc)
runner = unittest.TextTestRunner(stream=open(os.devnull, 'w'))
result = runner.run(alltests)
if len(result.failures) or len(result.errors):
print('There were failures or errors.')
I want to run a set of tests under different conditions and therefore share these tests between two different TestCase-derived classes.
One creates its own standalone session and the other attaches to an existing session and executes the same tests in there.
I guess I'm kind of abusing the unittest framework when testing an API with it but it doesn't feel like it's too far from its original purpose. Am I good so far?
I hacked a few things together and got it kind of running. But the way it's done, doesn't feel right and I'm afraid will cause problems sooner or later.
These are the problems I have with my solution:
When simply running the thing with PyCharm without limiting the tests, it attempts to run not only the intended StandaloneSessionTests and ExistingSessionTests but also GroupOfTests which is only the collection and has no session, i.e. execution context.
I can make it not run GroupOfTests by not deriving that one from TestCase but then PyCharm complains that it doesn't know about the assert...() functions. Rightly so, because GroupOfTest only gets indirect access to these functions at runtime when a derived class also inherits from TestCase. Coming from a C++ background, this feels like black magic and I don't think I should be doing this.
I tried passing the session creation classes to the constructor of GroupOfTests like this: __init__(self, session_class). But this causes problems when the unittest framework attempts to instantiate the tests: It doesn't know what to do with the additional __init__ parameter.
I learned about #classmethod, which seems to be a way to get around the "only one constructor" limitation of Python but I couldn't figure out a way to get it running.
I'm looking for a solution that lets me state something as straightforward as this:
suite = unittest.TestSuite()
suite.addTest(GroupOfTests(UseExistingSession))
suite.addTest(GroupOfTests(CreateStandaloneSession))
...
This is what I got so far:
#!/usr/bin/python3
import unittest
def existing_session():
return "existingsession"
def create_session():
return "123"
def close_session(session_id):
print("close session %s" % session_id)
return True
def do_thing(session_id):
return len(session_id)
class GroupOfTests(unittest.TestCase): # GroupOfTests gets executed, which makes no sense.
#class GroupOfTests: # use of assertGreaterThan() causes pycharm warning
session_id = None
def test_stuff(self):
the_thing = do_thing(self.session_id)
self.assertGreater(the_thing, 2)
# Original code contains many other tests, which must not be duplicated
class UseExistingSession(unittest.TestCase):
session_id = None
def setUp(self):
self.session_id = existing_session()
def tearDown(self):
pass # Nothing to do
class CreateStandaloneSession(unittest.TestCase):
session_id = None
def setUp(self):
self.session_id = create_session()
def tearDown(self):
close_session(self.session_id)
# unittest framework runs inherited test_stuff()
class StandaloneSessionTests(CreateStandaloneSession, GroupOfTests):
pass
# unittest framework runs inherited test_stuff()
class ExistingSessionTests(UseExistingSession, GroupOfTests):
pass
def main():
suite = unittest.TestSuite()
suite.addTest(StandaloneSessionTests)
suite.addTest(ExistingSessionTests)
runner = unittest.TextTestRunner()
runner.run(suite())
if __name__ == '__main__':
main()
I'm not sure if using pytest is an option for you but if so, here is an example which might do what you want.
import pytest
class Session:
def __init__(self, session_id=None):
self.id = session_id
existing_session = Session(999)
new_session = Session(111)
#pytest.fixture(params=[existing_session, new_session])
def session_fixture(request):
return request.param
class TestGroup:
def test_stuff(self, session_fixture):
print('(1) Test with session: {}'.format(session_fixture.id))
assert True
def test_more_stuff(self, session_fixture):
print('(2) Test with session: {}'.format(session_fixture.id))
assert True
Output:
$ pytest -v -s hmm.py
======================================================= test session starts ========================================================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /home/lettuce/Dropbox/Python/Python_3/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/lettuce/Dropbox/Python/Python_3, inifile:
collected 4 items
hmm.py::TestGroup::test_stuff[session_fixture0] (1) Test with session: 999
PASSED
hmm.py::TestGroup::test_stuff[session_fixture1] (1) Test with session: 111
PASSED
hmm.py::TestGroup::test_more_stuff[session_fixture0] (2) Test with session: 999
PASSED
hmm.py::TestGroup::test_more_stuff[session_fixture1] (2) Test with session: 111
PASSED
===================================================== 4 passed in 0.01 seconds =====================================================
If you are actually going to use pytest you will probably want follow the conventions for Python test discovery rather than using hmm.py as a filename though!
You can get pycharm to ignore these not existing function by creating abstract methods with raise NotImplementetError:
class GroupOfTests:
session_id = None
def test_stuff(self):
the_thing = do_thing(self.session_id)
self.assertGreater(the_thing, 2)
def assertGreater(self, a, b): # Pycharm treats these like abstract methods from the ABC module
raise NotImplementetError
This will let python believe this is an abstract class and will make pycharm raise errors if a subclass doesn't define these functions.
In a hobby project I intend to use nose for testing, I want to put all tests for specific classes into classes since these tests share setup and other functionality. But I can't seem to get nose to execute the setup methods inside the classes.
Here is an example class that is tested:
class mwe():
def __init__(self):
self.example = ""
def setExample(self, ex):
self.example = ex
The tests work, when I don't use classes:
from nose.tools import ok_
import mwe
exampleList = []
def setUp():
print("setup")
exampleList.append("1")
exampleList.append("2")
exampleList.append("3")
def test_Example():
print("test")
for ex in exampleList:
t = mwe.mwe()
t.setExample(ex)
yield check, t, ex
def check(e, ex):
ok_(e.example == ex)
The output is as expected:
setup
test
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s
OK
When a test class is used, the setup method is not executed and therefore no tests are executed.
from nose.tools import ok_
import mwe
class TestexampleClass(object):
def __init__(self):
print("__init__")
self.exampleList = []
def setup(self):
print("setup class")
self.exampleList.append("1")
self.exampleList.append("2")
self.exampleList.append("3")
def test_ExampleClass(self):
print("test class")
for ex in self.exampleList:
t = mwe.mwe()
t.setExample(ex)
yield self.check, t, ex
def check(self, we, ex):
print("check class")
ok_(we.example == ex)
I'm fairly new to python and a newbie with nose, my question is, why is setup not executed? Where is the error in my code?
__init__
test class
----------------------------------------------------------------------
Ran 0 tests in 0.002s
OK
I will be glad for any feedback.
When I use the code from this Question on SO the setup method is executed, as I would expect it.
SOLUTION: After a lot of desperation I found the following:
Nose executes the the class level setup method before the execution of the yielded function, not when the test_* methods are called, as I expected and as is the case for other test_* methods. This obviously goes against the nose documentation:
Setup and teardown functions may be used with test generators. However, please note that setup and teardown attributes attached to the generator function will execute only once. To execute fixtures for each yielded test, attach the setup and teardown attributes to the function that is yielded, or yield a callable object instance with setup and teardown attributes.
Looking at the bug reports, I found the bug report on github.
A possible workaround is to use class level fixtures:
#classmethod
def setup_class(cls):
#do stuff
pass
Your test class needs to extend TestCase and the setup method needs to be called setUp
from unittest import TestCase
class TestUtils(TestCase):
def setUp(self):
self.x = 1
def test_something(self):
self.assertEqual(1, self.x)
which outputs
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
I have two test cases (two different files) that I want to run together in a Test Suite. I can get the tests to run just by running python "normally" but when I select to run a python-unit test it says 0 tests run. Right now I'm just trying to get at least one test to run correectly.
import usertest
import configtest # first test
import unittest # second test
testSuite = unittest.TestSuite()
testResult = unittest.TestResult()
confTest = configtest.ConfigTestCase()
testSuite.addTest(configtest.suite())
test = testSuite.run(testResult)
print testResult.testsRun # prints 1 if run "normally"
Here's an example of my test case set up
class ConfigTestCase(unittest.TestCase):
def setUp(self):
##set up code
def runTest(self):
#runs test
def suite():
"""
Gather all the tests from this module in a test suite.
"""
test_suite = unittest.TestSuite()
test_suite.addTest(unittest.makeSuite(ConfigTestCase))
return test_suite
if __name__ == "__main__":
#So you can run tests from this module individually.
unittest.main()
What do I have to do to get this work correctly?
you want to use a testsuit. So you need not call unittest.main().
Use of testsuit should be like this:
#import usertest
#import configtest # first test
import unittest # second test
class ConfigTestCase(unittest.TestCase):
def setUp(self):
print 'stp'
##set up code
def runTest(self):
#runs test
print 'stp'
def suite():
"""
Gather all the tests from this module in a test suite.
"""
test_suite = unittest.TestSuite()
test_suite.addTest(unittest.makeSuite(ConfigTestCase))
return test_suite
mySuit=suite()
runner=unittest.TextTestRunner()
runner.run(mySuit)
All of the code to create a loader and suite is unnecessary. You should write your tests so that they are runnable via test discovery using your favorite test runner. That just means naming your methods in a standard way, putting them in an importable place (or passing a folder containing them to the runner), and inheriting from unittest.TestCase. After you've done that, you can use python -m unittest discover at the simplest, or a nicer third party runner to discover and then run your tests.
If you are trying to manually collect TestCases, this is useful: unittest.loader.findTestCases():
# Given a module, M, with tests:
mySuite = unittest.loader.findTestCases(M)
runner = unittest.TextTestRunner()
runner.run(mySuit)
I am assuming you are referring to running python-unit test against the module that consolidates the two test. It will work if you create test case for that module ie. subclassing unittest.TestCase and having a simple test that starts with the word 'test'.
e.g.
class testall(unittest.TestCase):
def test_all(self):
testSuite = unittest.TestSuite()
testResult = unittest.TestResult()
confTest = configtest.ConfigTestCase()
testSuite.addTest(configtest.suite())
test = testSuite.run(testResult)
print testResult.testsRun # prints 1 if run "normally"
if __name__ == "__main__":
unittest.main()
How do I run multiple Classes in a single test suite in Python using unit testing?
If you want to run all of the tests from a specific list of test classes, rather than all of the tests from all of the test classes in a module, you can use a TestLoader's loadTestsFromTestCase method to get a TestSuite of tests for each class, and then create a single combined TestSuite from a list containing all of those suites that you can use with run:
import unittest
# Some tests
class TestClassA(unittest.TestCase):
def testOne(self):
# test code
pass
class TestClassB(unittest.TestCase):
def testOne(self):
# test code
pass
class TestClassC(unittest.TestCase):
def testOne(self):
# test code
pass
def run_some_tests():
# Run only the tests in the specified classes
test_classes_to_run = [TestClassA, TestClassC]
loader = unittest.TestLoader()
suites_list = []
for test_class in test_classes_to_run:
suite = loader.loadTestsFromTestCase(test_class)
suites_list.append(suite)
big_suite = unittest.TestSuite(suites_list)
runner = unittest.TextTestRunner()
results = runner.run(big_suite)
# ...
if __name__ == '__main__':
run_some_tests()
I'm a bit unsure at what you're asking here, but if you want to know how to test multiple classes in the same suite, usually you just create multiple testclasses in the same python file and run them together:
import unittest
class TestSomeClass(unittest.TestCase):
def testStuff(self):
# your testcode here
pass
class TestSomeOtherClass(unittest.TestCase):
def testOtherStuff(self):
# testcode of second class here
pass
if __name__ == '__main__':
unittest.main()
And run with for example:
python mytestsuite.py
Better examples can be found in the official documention.
If on the other hand you want to run multiple test files, as detailed in "How to organize python test in a way that I can run all tests in a single command?", then the other answer is probably better.
The unittest.TestLoader.loadTestsFromModule() method will discover and load all classes in the specified module. So you can just do this:
import unittest
import sys
class T1(unittest.TestCase):
def test_A(self):
pass
def test_B(self):
pass
class T2(unittest.TestCase):
def test_A(self):
pass
def test_B(self):
pass
if __name__ == "__main__":
suite = unittest.TestLoader().loadTestsFromModule( sys.modules[__name__] )
unittest.TextTestRunner(verbosity=3).run( suite )
Normally you would do in the following way (which adds only one class per suite):
# Add tests.
alltests = unittest.TestSuite()
alltests.addTest(unittest.makeSuite(Test1))
alltests.addTest(unittest.makeSuite(Test2))
If you'd like to have multiple classes per suite, you can use add these tests in the following way:
for name in testnames:
suite.addTest(tc_class(name, cargs=args))
Here is same example to run all classes per separate suite you can define your own make_suite method:
# Credits: http://codereview.stackexchange.com/a/88662/15346
def make_suite(tc_class):
testloader = unittest.TestLoader()
testnames = testloader.getTestCaseNames(tc_class)
suite = unittest.TestSuite()
for name in testnames:
suite.addTest(tc_class(name, cargs=args))
return suite
# Add all tests.
alltests = unittest.TestSuite()
for name, obj in inspect.getmembers(sys.modules[__name__]):
if inspect.isclass(obj) and name.startswith("FooTest"):
alltests.addTest(make_suite(obj))
result = unittest.TextTestRunner(verbosity=2).run(alltests)
If above doesn't suite, you can convert above example into method which could accept multiple classes.
I've found nose to be a good tool for this. It discovers all unit tests in a directory structure and executes them.