I've written a BaseTestCase class (which is meant to be inherited by all test cases) and a number of TestCase<Number>(BaseTestCase) classes (each in its own .py file).
The project structure is:
BaseTestCase.py
TestCase1.py
TestCase2.py
Example:
class TestCase2(BaseTestCase):
# methods
def execute(self):
self.method1()
self.method2()
self.method3()
The execute method should be called on an instance of TestCase<Number> to run a specific test. What is the proper way to run all test cases, i.e. to orchestrate them all?
I'm assuming you are using Unittest framework and want to run tests programmatically and have the ability to select which tests you run. If that's the correct assumption, here is the answer.
Lets assume you have those tests:
class TestExample1(unittest.TestCase):
def test_something(self):
pass
def test_something_else(self):
pass
class TestExample2(unittest.TestCase):
def test_something(self):
pass
def test_something_else(self):
pass
If those are in the same file, you can simply run them like this
# assuming your tests are above in this same file
unittest.main()
However, if they are in different files, you will have to extract the tests from them like so
for example in [TestExample1, TestExample2]:
tests = unittest.TestLoader().loadTestsFromTestCase(example)
unittest.TextTestRunner().run(tests)
This will result in two different executions like this
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
If you are considering other frameworks for running your tests, I recommend to take a look at Test Junkie: https://www.test-junkie.com/get-started/ as its much easier to get started with and is very powerful. (disclaimer: I'm the author!).
Related
Background
In Python's unittest framework, it is a fairly common idiom to use inheritance on a base set of tests to apply an entire set of tests to a new problem, and occasionally to add additional tests. A trivial example would be:
from unittest import TestCase
class BaseTestCase(TestCase):
VAR = 3
def test_var_positive(self):
self.assertGreaterEqual(self.VAR, 0)
class SubTestCase(BaseTestCase):
VAR = 8
def test_var_even(self):
self.assertTrue(self.VAR % 2 == 0)
Which, when run, runs 3 tests:
$ python -m unittest -v
test_var_positive (test_unittest.BaseTestCase) ... ok
test_var_even (test_unittest.SubTestCase) ... ok
test_var_positive (test_unittest.SubTestCase) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
This is particularly useful if you are testing a class hierarchy, where each subclass is a subtype of the parent classes, and should thus be able to pass the parent class's test suite in addition to its own.
Problem
I would like to switch over to using pytest, but I have a lot of tests that are structured this way. From what I can tell, pytest intends to replace most of the functionality of TestCase classes with fixtures, but is there a pytest idiom that allows test inheritance, and if so what is it?
I am aware that pytest can be used to run unittest-style tests, but the support is limited, and I would like to use some of the "will never be supported" features of pytest in my tests.
Pytest allows you to group test cases in classes, so it naturally has support for test case inheritance.
When rewriting your unittest tests to pytest tests, remember to follow pytest's naming guidelines:
class names must begin with Test
function/method names must begin with test_
Failing to comply with this naming scheme will prevent your tests from being collected and executed.
Your tests rewritten for pytest would look like this:
class TestBase:
VAR = 3
def test_var_positive(self):
assert self.VAR >= 0
class TestSub(TestBase):
VAR = 8
def test_var_even(self):
assert self.VAR % 2 == 0
I'm getting stuck with some unittests.
Here's the simplest example I could come up with:
#testito.py
import unittest
class Prueba(unittest.TestCase):
def setUp(self):
pass
def printsTrue(self):
self.assertTrue(True)
if __name__=="__main__":
unittest.main()
Problem is, running this has no effect:
$ python testito.py
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
I'm scratching my head as I don't see any problem with the code above.
It happened with a couple of tests now and I don't really know what to do next.
Any idea?
By default, only functions whose name that start with test are run:
class Prueba(unittest.TestCase):
def setUp(self):
pass
def testPrintsTrue(self):
self.assertTrue(True)
From the unittest basic example:
A testcase is created by subclassing unittest.TestCase. The three individual tests are defined with methods whose names start with the letters test. This naming convention informs the test runner about which methods represent tests.
I'm working on a module using sockets with hundreds of test cases. Which is nice. Except now I need to test all of the cases with and without socket.setdefaulttimeout( 60 )... Please don't tell me cut and paste all the tests and set/remove a default timeout in setup/teardown.
Honestly, I get that having each test case laid out on it's own is good practice, but i also don't like to repeat myself. This is really just testing in a different context not different tests.
i see that unittest supports module level setup/teardown fixtures, but it isn't obvious to me how to convert my one test module into testing itself twice with two different setups.
any help would be much appreciated.
you could do something like this:
class TestCommon(unittest.TestCase):
def method_one(self):
# code for your first test
pass
def method_two(self):
# code for your second test
pass
class TestWithSetupA(TestCommon):
def SetUp(self):
# setup for context A
do_setup_a_stuff()
def test_method_one(self):
self.method_one()
def test_method_two(self):
self.method_two()
class TestWithSetupB(TestCommon):
def SetUp(self):
# setup for context B
do_setup_b_stuff()
def test_method_one(self):
self.method_one()
def test_method_two(self):
self.method_two()
The other answers on this question are valid in as much as they make it possible to actually perform the tests under multiple environments, but in playing around with the options I think I like a more self contained approach. I'm using suites and results to organize and display results of tests. In order to run one tests with two environments rather than two tests I took this approach - create a TestSuite subclass.
class FixtureSuite(unittest.TestSuite):
def run(self, result, debug=False):
socket.setdefaulttimeout(30)
super().run(result, debug)
socket.setdefaulttimeout(None)
...
suite1 = unittest.TestSuite(testCases)
suite2 = FixtureSuite(testCases)
fullSuite = unittest.TestSuite([suite1,suite2])
unittest.TextTestRunner(verbosity=2).run(fullSuite)
I would do it like this:
Make all of your tests derive from your own TestCase class, let's call it SynapticTestCase.
In SynapticTestCase.setUp(), examine an environment variable to determine whether to set the socket timeout or not.
Run your entire test suite twice, once with the environment variable set one way, then again with it set the other way.
Write a small shell script to invoke the test suite both ways.
If your code does not call socket.setdefaulttimeout then you can run tests the following way:
import socket
socket.setdeaulttimeout(60)
old_setdefaulttimeout, socket.setdefaulttimeout = socket.setdefaulttimeout, None
unittest.main()
socket.setdefaulttimeout = old_setdefaulttimeout
It is a hack, but it can work
You could also inherit and rerun the original suite, but overwrite the whole setUp or a part of it:
class TestOriginal(TestCommon):
def SetUp(self):
# common setUp here
self.current_setUp()
def current_setUp(self):
# your first setUp
pass
def test_one(self):
# your test
pass
def test_two(self):
# another test
pass
class TestWithNewSetup(TestOriginal):
def current_setUp(self):
# overwrite your first current_setUp
I'm having trouble figuring out the proper way of using pythons unittest framework
I currently have 3 different implementations for a data structure class and unittests to test various things in the class as follows:
import fooHorse
import fooSnake
import fooMoose
import unittest
foo = fooHorse.foo()
#foo = fooSnake.foo()
#foo = fooMoose.foo()
class testFoo(unittest.TestCase):
def testSomething(self):
foo.do_something()
...
def testSomethingelse(self):
...
# etc
if __name__ == "__main__":
unittest.main()
How do I refactor the code so that all the tests are run for fooSnake.foo, fooMoose.foo and fooHorse.foo?
Just factor all the testing into a function and call it from the three tests :
class testFoo(unittest.TestCase):
def _doTest(self, foo):
foo.do_something()
# ...
def testFoorHorse(self):
foo = fooHorse.foo()
self._doTest(foo)
# and so on.
I wouldn't try to do anything more clever so that the test logic stays simple enought to be obviously bug-free.
I think it is better to keep the testing code inside the classes under test (so, inside the fooHorse classes and such). I can't really tell from the code segment, so maybe you are already doing that. Then to combine multiple tests to be run at the same time, or selecting a few of them, you could create a separate testing class that creates a test suite. In there, the individual unittests can be assigned. See the python docs for details on this.
I have a test case:
class LoginTestCase(unittest.TestCase):
...
I'd like to use it in a different test case:
class EditProfileTestCase(unittest.TestCase):
def __init__(self):
self.t = LoginTestCase()
self.t.login()
This raises:
ValueError: no such test method in <class 'LoginTest: runTest`
I looked at the unittest code where the exception is being called, and it looks like the tests aren't supposed to be written this way. Is there a standard way to write something you'd like tested so that it can be reused by later tests? Or is there a workaround?
I've added an empty runTest method to LoginTest as a dubious workaround for now.
The confusion with "runTest" is mostly based on the fact that this works:
class MyTest(unittest.TestCase):
def test_001(self):
print "ok"
if __name__ == "__main__":
unittest.main()
So there is no "runTest" in that class and all of the test-functions are being called. However if you look at the base class "TestCase" (lib/python/unittest/case.py) then you will find that it has an argument "methodName" that defaults to "runTest" but it does NOT have a default implementation of "def runTest"
class TestCase:
def __init__(self, methodName='runTest'):
The reason that unittest.main works fine is based on the fact that it does not need "runTest" - you can mimic the behaviour by creating a TestCase-subclass instance for all methods that you have in your subclass - just provide the name as the first argument:
class MyTest(unittest.TestCase):
def test_001(self):
print "ok"
if __name__ == "__main__":
suite = unittest.TestSuite()
for method in dir(MyTest):
if method.startswith("test"):
suite.addTest(MyTest(method))
unittest.TextTestRunner().run(suite)
Here's some 'deep black magic':
suite = unittest.TestLoader().loadTestsFromTestCase(Test_MyTests)
unittest.TextTestRunner(verbosity=3).run(suite)
Very handy if you just want to test run your unit tests from a shell (i.e., IPython).
If you don't mind editing unit test module code directly, the simple fix is to add under case.py class TestCase a new method called runTest that does nothing.
The file to edit sits under pythoninstall\Lib\unittest\case.py
def runTest(self):
pass
This will stop you ever getting this error.
Guido's answer is almost there, however it doesn't explain the thing. I needed to look to unittest code to grasp the flow.
Say you have the following.
import unittest
class MyTestCase(unittest.TestCase):
def testA(self):
pass
def testB(self):
pass
When you use unittest.main(), it will try to discover test cases in current module. The important code is unittest.loader.TestLoader.loadTestsFromTestCase.
def loadTestsFromTestCase(self, testCaseClass):
# ...
# This will look in class' callable attributes that start
# with 'test', and return their names sorted.
testCaseNames = self.getTestCaseNames(testCaseClass)
# If there's no test to run, look if the case has the default method.
if not testCaseNames and hasattr(testCaseClass, 'runTest'):
testCaseNames = ['runTest']
# Create TestSuite instance having test case instance per test method.
loaded_suite = self.suiteClass(map(testCaseClass, testCaseNames))
return loaded_suite
What the latter does, is converting test case class into test suite, that holds the instances of the class per its test method. I.e. my example will be turned into unittest.suite.TestSuite([MyTestCase('testA'), MyTestCase('testB')]). So if you would like to create a test case manually, you need to do the same thing.
#dmvianna's answer got me very close to being able to run unittest in a jupyter (ipython) notebook, but I had to do a bit more. If I wrote just the following:
class TestStringMethods(unittest.TestCase):
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
suite = unittest.TestLoader().loadTestsFromModule (TestStringMethods)
unittest.TextTestRunner().run(suite)
I got
Ran 0 tests in 0.000s
OK
It's not broken, but it doesn't run any tests! If I instantiated the test class
suite = unittest.TestLoader().loadTestsFromModule (TestStringMethods())
(note the parens at the end of the line; that's the only change) I got
ValueError Traceback (most recent call last)
in ()
----> 1 suite = unittest.TestLoader().loadTestsFromModule (TestStringMethods())
/usr/lib/python2.7/unittest/case.pyc in init(self, methodName)
189 except AttributeError:
190 raise ValueError("no such test method in %s: %s" %
--> 191 (self.class, methodName))
192 self._testMethodDoc = testMethod.doc
193 self._cleanups = []
ValueError: no such test method in : runTest
The fix is now reasonably clear: add runTest to the test class:
class TestStringMethods(unittest.TestCase):
def runTest(self):
test_upper (self)
test_isupper (self)
test_split (self)
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
suite = unittest.TestLoader().loadTestsFromModule (TestStringMethods())
unittest.TextTestRunner().run(suite)
Ran 3 tests in 0.002s
OK
It also works correctly (and runs 3 tests) if my runTest just passes, as suggested by #Darren.
This is a little yucchy, requiring some manual labor on my part, but it's also more explicit, and that's a Python virtue, isn't it?
I could not get any of the techniques via calling unittest.main with explicit arguments from here or from this related question Unable to run unittest's main function in ipython/jupyter notebook to work inside a jupyter notebook, but I am back on the road with a full tank of gas.
unittest does deep black magic -- if you choose to use it to run your unit-tests (I do, since this way I can use a very powerful battery of test runners &c integrated into the build system at my workplace, but there are definitely worthwhile alternatives), you'd better play by its rules.
In this case, I'd simply have EditProfileTestCase derive from LoginTestCase (rather than directly from unittest.TestCase). If there are some parts of LoginTestCase that you do want to also test in the different environment of EditProfileTestCase, and others that you don't, it's a simple matter to refactor LoginTestCase into those two parts (possibly using multiple inheritance) and if some things need to happen slightly differently in the two cases, factor them out into auxiliary "hook methods" (in a "Template Method" design pattern) -- I use all of these approaches often to diminish boilerplate and increase reuse in the copious unit tests I always write (if I have unit-test coverage < 95%, I always feel truly uneasy -- below 90%, I start to feel physically sick;-).