Background
In Python's unittest framework, it is a fairly common idiom to use inheritance on a base set of tests to apply an entire set of tests to a new problem, and occasionally to add additional tests. A trivial example would be:
from unittest import TestCase
class BaseTestCase(TestCase):
VAR = 3
def test_var_positive(self):
self.assertGreaterEqual(self.VAR, 0)
class SubTestCase(BaseTestCase):
VAR = 8
def test_var_even(self):
self.assertTrue(self.VAR % 2 == 0)
Which, when run, runs 3 tests:
$ python -m unittest -v
test_var_positive (test_unittest.BaseTestCase) ... ok
test_var_even (test_unittest.SubTestCase) ... ok
test_var_positive (test_unittest.SubTestCase) ... ok
----------------------------------------------------------------------
Ran 3 tests in 0.000s
This is particularly useful if you are testing a class hierarchy, where each subclass is a subtype of the parent classes, and should thus be able to pass the parent class's test suite in addition to its own.
Problem
I would like to switch over to using pytest, but I have a lot of tests that are structured this way. From what I can tell, pytest intends to replace most of the functionality of TestCase classes with fixtures, but is there a pytest idiom that allows test inheritance, and if so what is it?
I am aware that pytest can be used to run unittest-style tests, but the support is limited, and I would like to use some of the "will never be supported" features of pytest in my tests.
Pytest allows you to group test cases in classes, so it naturally has support for test case inheritance.
When rewriting your unittest tests to pytest tests, remember to follow pytest's naming guidelines:
class names must begin with Test
function/method names must begin with test_
Failing to comply with this naming scheme will prevent your tests from being collected and executed.
Your tests rewritten for pytest would look like this:
class TestBase:
VAR = 3
def test_var_positive(self):
assert self.VAR >= 0
class TestSub(TestBase):
VAR = 8
def test_var_even(self):
assert self.VAR % 2 == 0
Related
I have a requirement to implement a test suite for multiple functions.
I am trying to figure out best practices to leverage existing pytest design pattern.
There are 2-3 common test cases for all the functions
Each function require different presetup condition
My current design :
/utils
logic.py
/tests
Test_Regression.py
Sedan/
Test_Sedan.py
SUV/
Test_SUV.py
Hatchback/
Test_Hatchback.py
/config
Configuration.py
Current folder structure
Regression.py : This class holds common testcases
Test_SUV.py : This class inherits Test_Regression class test cases and has SUV specific test cases
Utils : This folder stores the program logic
is this a good design practice for a test suite to have class inheritance
class Regression:
#pytest.parameterize(x, utils.logic_func())
#pytest.mark.testengine
def test_engine(x,self):
#validates logic
assert x == 0
#pytest.parameterize(y, utils.logic_func())
#pytest.mark.testheadlight
def test_headlight(y,self):
#validates logic
assert y == 0
class Test_SUV(Test_Regression):
def get_engine_values():
# calls program logic
return x
.
.
.
.
Or is there a better way to structure these test cases.
Preconditions function can be annotated with #pytest.fixture and that can be used as parameter to the test methods instead of utility functions. You can define the scope (function, class, module, package or session) of these fixture functions: More details about fixture: https://docs.pytest.org/en/6.2.x/fixture.html
Pytest over unittest - one reason is you can avoid implicit class requirement and keep your tests simple and less verbose. And if you are adding class for pytest, then we are losing this benefit. IMHO, you can keep your common tests in regression module and avoid Regression class and Test Class inheritance, because this is going to be hard to maintain in long run.
Your tests should be independent from each other. If there is a common functionality that you want to share or inherit, you can do that via fixtures and keep them in a module called conftest.py and all those functions in conftest.py will then be available to all modules in the package and sub-packages More details about conftest
I was studying unittest by following the examples here.
In the following code, def test_add() is supposed to be wrapped in class testClass(), but for my curiosity, I didn't encapsulate it.
# class testClass(unittest.TestCase):
def test_add(self):
result = cal_fun.add_fuc(5, 10)
self.assertEqual(result, 15)
if __name__ == '__main__':
unittest.main()
The result came out in VScode as:
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Why was no test run? Why must the def test_add() be wrapped in a class?
Here's an expansion on my initial comment.
The unittest module has a Test Discovery feature/function. When you run it on a directory, it looks for pre-defined structures, definitions, and patterns. Namely:
In order to be compatible with test discovery, all of the test files must be modules or packages (including namespace packages) importable from the top-level directory of the project (this means that their filenames must be valid identifiers).
The basic building blocks of unit testing are test cases — single scenarios that must be set up and checked for correctness. In unittest, test cases are represented by unittest.TestCase instances. To make your own test cases you must write subclasses of TestCase or use FunctionTestCase.
The relevant answer to your question is that test functions must be wrapped in a TestCase. Not simply wrapped in just some "class testClass()" as you said, but the class must specifically inherit from unittest.TestCase. In addition, that assertEqual method is only available as part of a TestCase, because it's a method of the TestCase class. If you somehow got that code of yours to run, self.assertEqual would result in an error. You would have to use plain assert's.
To go into more detail, you will have to read the section on unittest's load_tests Protocol and specifically the loadTestsFromModule(module, pattern=None) method:
Return a suite of all test cases contained in the given module. This method searches module for classes derived from TestCase and creates an instance of the class for each test method defined for the class.
Finally, it's not just about wrapping your test functions in a unittest.TestCase class, your tests must follow some pre-defined patterns:
By default, unittest looks for files matching the pattern test*.py. This is done by the discover(start_dir, pattern='test*.py', top_level_dir=None) method of unittest's TestLoader class, which is "used to create test suites from classes and modules".
By default, unittest looks for methods that start with test (ex. test_function). This is done by the testMethodPrefix attribute of unittest's TestLoader class, which is a "string giving the prefix of method names which will be interpreted as test methods. The default value is 'test'."
If you want some customized behavior, you'll have to override the load_tests protocol.
I've written a BaseTestCase class (which is meant to be inherited by all test cases) and a number of TestCase<Number>(BaseTestCase) classes (each in its own .py file).
The project structure is:
BaseTestCase.py
TestCase1.py
TestCase2.py
Example:
class TestCase2(BaseTestCase):
# methods
def execute(self):
self.method1()
self.method2()
self.method3()
The execute method should be called on an instance of TestCase<Number> to run a specific test. What is the proper way to run all test cases, i.e. to orchestrate them all?
I'm assuming you are using Unittest framework and want to run tests programmatically and have the ability to select which tests you run. If that's the correct assumption, here is the answer.
Lets assume you have those tests:
class TestExample1(unittest.TestCase):
def test_something(self):
pass
def test_something_else(self):
pass
class TestExample2(unittest.TestCase):
def test_something(self):
pass
def test_something_else(self):
pass
If those are in the same file, you can simply run them like this
# assuming your tests are above in this same file
unittest.main()
However, if they are in different files, you will have to extract the tests from them like so
for example in [TestExample1, TestExample2]:
tests = unittest.TestLoader().loadTestsFromTestCase(example)
unittest.TextTestRunner().run(tests)
This will result in two different executions like this
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK
If you are considering other frameworks for running your tests, I recommend to take a look at Test Junkie: https://www.test-junkie.com/get-started/ as its much easier to get started with and is very powerful. (disclaimer: I'm the author!).
I do have a set of unit tests written using the Python's unittest module. They are using the setUpModule() function to load a global variable with the shared "stuff" that is required to run the tests (including some HTTP sessions).
When running my tests with unittest, they are running fine. With py.test they are failing.
I patched it a little bit to make it run using the old pytest fixture functions (which happen not to have same names as unittest ones). It worked, but only when not executed on multiple threads, and that's a feature I do want to use.
The documentation examples are useless in my case because I do have like 20 classes (unittest.TestCase) with 10 tests inside each class. Obviously I do not want to add a new parameter to each test.
Until now I used the class setUp() method to load the shared dictionary inside self and use it from there inside each test.
#!/usr/bin/env python
# conftest.py
#pytest.fixture(scope="session")
def manager():
return { "a": "b"}
And now the tests:
#!/usr/bin/env python
# tests.py
class VersionTests(unittest.TestCase):
def setUp(self):
self.manager = manager
def test_create_version(self):
# do something with self.manager
pass
Please remember that I need a solution that would work with multiple threads, calling the fixture a single time.
pytest can run unittest tests for sure, as documented in Support for unittest.TestCase / Integration of fixtures. The tricky part is that using pytest funcargs fixtures directly is discouraged:
While pytest supports receiving fixtures via test function arguments for non-unittest test methods, unittest.TestCase methods cannot directly receive fixture function arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.
Suppose we have a tests module like this, using the standard unittest initialization facilities:
# test_unittest_tests.py (for the sake of clarity!)
import unittest
manager = None
def setUpModule():
global manager
manager = {1: 2}
class UnittestTests(unittest.TestCase):
def setUp(self):
self.manager = manager
def test_1_in_manager(self):
assert 1 in self.manager
def test_a_in_manager(self):
assert 'a' in self.manager
It yields the following output when ran with unittest:
$ python -m unittest -v test_unittest_tests
...
test_1_in_manager (test_unittest_tests.UnittestTests) ... ok
test_a_in_manager (test_unittest_tests.UnittestTests) ... FAIL
...
The test_a_in_manager fails as expected. There isn't any 'a' key in the manager directory.
We set up a conftest.py to provide scoped pytest fixtures for these tests. Like this for example, without breaking the standard unittest behavior and without the need to touch them tests at all using pytest autouse:
# conftest.py
import pytest
#pytest.fixture(scope='session', autouse=True)
def manager_session(request):
# create a session-scoped manager
request.session.manager = {'a': 'b'}
#pytest.fixture(scope='module', autouse=True)
def manager_module(request):
# set the sessions-scoped manager to the tests module at hand
request.module.manager = request.session.manager
Running the tests with pytest (using pytest-xdist) for parallelization, yields the following output:
$ py.test -v -n2
...
[gw1] PASSED test_unittest_tests.py:17: UnittestTests.test_a_in_manager
[gw0] FAILED test_unittest_tests.py:14: UnittestTests.test_1_in_manager
...
Now the test_1_in_manager fails instead; there isn't any 1 key in the pytest-provided manager dictionary.
I'm getting stuck with some unittests.
Here's the simplest example I could come up with:
#testito.py
import unittest
class Prueba(unittest.TestCase):
def setUp(self):
pass
def printsTrue(self):
self.assertTrue(True)
if __name__=="__main__":
unittest.main()
Problem is, running this has no effect:
$ python testito.py
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
I'm scratching my head as I don't see any problem with the code above.
It happened with a couple of tests now and I don't really know what to do next.
Any idea?
By default, only functions whose name that start with test are run:
class Prueba(unittest.TestCase):
def setUp(self):
pass
def testPrintsTrue(self):
self.assertTrue(True)
From the unittest basic example:
A testcase is created by subclassing unittest.TestCase. The three individual tests are defined with methods whose names start with the letters test. This naming convention informs the test runner about which methods represent tests.