I want to run a set of tests under different conditions and therefore share these tests between two different TestCase-derived classes.
One creates its own standalone session and the other attaches to an existing session and executes the same tests in there.
I guess I'm kind of abusing the unittest framework when testing an API with it but it doesn't feel like it's too far from its original purpose. Am I good so far?
I hacked a few things together and got it kind of running. But the way it's done, doesn't feel right and I'm afraid will cause problems sooner or later.
These are the problems I have with my solution:
When simply running the thing with PyCharm without limiting the tests, it attempts to run not only the intended StandaloneSessionTests and ExistingSessionTests but also GroupOfTests which is only the collection and has no session, i.e. execution context.
I can make it not run GroupOfTests by not deriving that one from TestCase but then PyCharm complains that it doesn't know about the assert...() functions. Rightly so, because GroupOfTest only gets indirect access to these functions at runtime when a derived class also inherits from TestCase. Coming from a C++ background, this feels like black magic and I don't think I should be doing this.
I tried passing the session creation classes to the constructor of GroupOfTests like this: __init__(self, session_class). But this causes problems when the unittest framework attempts to instantiate the tests: It doesn't know what to do with the additional __init__ parameter.
I learned about #classmethod, which seems to be a way to get around the "only one constructor" limitation of Python but I couldn't figure out a way to get it running.
I'm looking for a solution that lets me state something as straightforward as this:
suite = unittest.TestSuite()
suite.addTest(GroupOfTests(UseExistingSession))
suite.addTest(GroupOfTests(CreateStandaloneSession))
...
This is what I got so far:
#!/usr/bin/python3
import unittest
def existing_session():
return "existingsession"
def create_session():
return "123"
def close_session(session_id):
print("close session %s" % session_id)
return True
def do_thing(session_id):
return len(session_id)
class GroupOfTests(unittest.TestCase): # GroupOfTests gets executed, which makes no sense.
#class GroupOfTests: # use of assertGreaterThan() causes pycharm warning
session_id = None
def test_stuff(self):
the_thing = do_thing(self.session_id)
self.assertGreater(the_thing, 2)
# Original code contains many other tests, which must not be duplicated
class UseExistingSession(unittest.TestCase):
session_id = None
def setUp(self):
self.session_id = existing_session()
def tearDown(self):
pass # Nothing to do
class CreateStandaloneSession(unittest.TestCase):
session_id = None
def setUp(self):
self.session_id = create_session()
def tearDown(self):
close_session(self.session_id)
# unittest framework runs inherited test_stuff()
class StandaloneSessionTests(CreateStandaloneSession, GroupOfTests):
pass
# unittest framework runs inherited test_stuff()
class ExistingSessionTests(UseExistingSession, GroupOfTests):
pass
def main():
suite = unittest.TestSuite()
suite.addTest(StandaloneSessionTests)
suite.addTest(ExistingSessionTests)
runner = unittest.TextTestRunner()
runner.run(suite())
if __name__ == '__main__':
main()
I'm not sure if using pytest is an option for you but if so, here is an example which might do what you want.
import pytest
class Session:
def __init__(self, session_id=None):
self.id = session_id
existing_session = Session(999)
new_session = Session(111)
#pytest.fixture(params=[existing_session, new_session])
def session_fixture(request):
return request.param
class TestGroup:
def test_stuff(self, session_fixture):
print('(1) Test with session: {}'.format(session_fixture.id))
assert True
def test_more_stuff(self, session_fixture):
print('(2) Test with session: {}'.format(session_fixture.id))
assert True
Output:
$ pytest -v -s hmm.py
======================================================= test session starts ========================================================
platform linux -- Python 3.6.4, pytest-3.4.1, py-1.5.2, pluggy-0.6.0 -- /home/lettuce/Dropbox/Python/Python_3/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/lettuce/Dropbox/Python/Python_3, inifile:
collected 4 items
hmm.py::TestGroup::test_stuff[session_fixture0] (1) Test with session: 999
PASSED
hmm.py::TestGroup::test_stuff[session_fixture1] (1) Test with session: 111
PASSED
hmm.py::TestGroup::test_more_stuff[session_fixture0] (2) Test with session: 999
PASSED
hmm.py::TestGroup::test_more_stuff[session_fixture1] (2) Test with session: 111
PASSED
===================================================== 4 passed in 0.01 seconds =====================================================
If you are actually going to use pytest you will probably want follow the conventions for Python test discovery rather than using hmm.py as a filename though!
You can get pycharm to ignore these not existing function by creating abstract methods with raise NotImplementetError:
class GroupOfTests:
session_id = None
def test_stuff(self):
the_thing = do_thing(self.session_id)
self.assertGreater(the_thing, 2)
def assertGreater(self, a, b): # Pycharm treats these like abstract methods from the ABC module
raise NotImplementetError
This will let python believe this is an abstract class and will make pycharm raise errors if a subclass doesn't define these functions.
Related
I am using selenium for end to end testing and I can't get how to use setup_class and teardown_class methods.
I need to set up browser in setup_class method, then perform a bunch of tests defined as class methods and finally quit browser in teardown_class method.
But logically it seems like a bad solution, because in fact my tests will not work with class, but with object. I pass self param inside every test method, so I can access objects' vars:
class TestClass:
def setup_class(cls):
pass
def test_buttons(self, data):
# self.$attribute can be used, but not cls.$attribute?
pass
def test_buttons2(self, data):
# self.$attribute can be used, but not cls.$attribute?
pass
def teardown_class(cls):
pass
And it even seems not to be correct to create browser instance for class.. It should be created for every object separately, right?
So, I need to use __init__ and __del__ methods instead of setup_class and teardown_class?
According to Fixture finalization / executing teardown code, the current best practice for setup and teardown is to use yield instead of return:
import pytest
#pytest.fixture()
def resource():
print("setup")
yield "resource"
print("teardown")
class TestResource:
def test_that_depends_on_resource(self, resource):
print("testing {}".format(resource))
Running it results in
$ py.test --capture=no pytest_yield.py
=== test session starts ===
platform darwin -- Python 2.7.10, pytest-3.0.2, py-1.4.31, pluggy-0.3.1
collected 1 items
pytest_yield.py setup
testing resource
.teardown
=== 1 passed in 0.01 seconds ===
Another way to write teardown code is by accepting a request-context object into your fixture function and calling its request.addfinalizer method with a function that performs the teardown one or multiple times:
import pytest
#pytest.fixture()
def resource(request):
print("setup")
def teardown():
print("teardown")
request.addfinalizer(teardown)
return "resource"
class TestResource:
def test_that_depends_on_resource(self, resource):
print("testing {}".format(resource))
When you write "tests defined as class methods", do you really mean class methods (methods which receive its class as first parameter) or just regular methods (methods which receive an instance as first parameter)?
Since your example uses self for the test methods I'm assuming the latter, so you just need to use setup_method instead:
class Test:
def setup_method(self, test_method):
# configure self.attribute
def teardown_method(self, test_method):
# tear down self.attribute
def test_buttons(self):
# use self.attribute for test
The test method instance is passed to setup_method and teardown_method, but can be ignored if your setup/teardown code doesn't need to know the testing context. More information can be found here.
I also recommend that you familiarize yourself with py.test's fixtures, as they are a more powerful concept.
This might help http://docs.pytest.org/en/latest/xunit_setup.html
In my test suite, I group my test cases into classes. For the setup and teardown I need for all the test cases in that class, I use the setup_class(cls) and teardown_class(cls) classmethods.
And for the setup and teardown I need for each of the test case, I use the setup_method(method) and teardown_method(methods)
Example:
lh = <got log handler from logger module>
class TestClass:
#classmethod
def setup_class(cls):
lh.info("starting class: {} execution".format(cls.__name__))
#classmethod
def teardown_class(cls):
lh.info("starting class: {} execution".format(cls.__name__))
def setup_method(self, method):
lh.info("starting execution of tc: {}".format(method.__name__))
def teardown_method(self, method):
lh.info("starting execution of tc: {}".format(method.__name__))
def test_tc1(self):
<tc_content>
assert
def test_tc2(self):
<tc_content>
assert
Now when I run my tests, when the TestClass execution is starting, it logs the details for when it is beginning execution, when it is ending execution and same for the methods..
You can add up other setup and teardown steps you might have in the respective locations.
Hope it helps!
As #Bruno suggested, using pytest fixtures is another solution that is accessible for both test classes or even just simple test functions. Here's an example testing python2.7 functions:
import pytest
#pytest.fixture(scope='function')
def some_resource(request):
stuff_i_setup = ["I setup"]
def some_teardown():
stuff_i_setup[0] += " ... but now I'm torn down..."
print stuff_i_setup[0]
request.addfinalizer(some_teardown)
return stuff_i_setup[0]
def test_1_that_needs_resource(some_resource):
print some_resource + "... and now I'm testing things..."
So, running test_1... produces:
I setup... and now I'm testing things...
I setup ... but now I'm torn down...
Notice that stuff_i_setup is referenced in the fixture, allowing that object to be setup and torn down for the test it's interacting with. You can imagine this could be useful for a persistent object, such as a hypothetical database or some connection, that must be cleared before each test runs to keep them isolated.
Your code should work just as you expect it to if you add #classmethod decorators.
#classmethod
def setup_class(cls):
"Runs once per class"
#classmethod
def teardown_class(cls):
"Runs at end of class"
See http://pythontesting.net/framework/pytest/pytest-xunit-style-fixtures/
import pytest
class Test:
#pytest.fixture()
def setUp(self):
print("setup")
yield "resource"
print("teardown")
def test_that_depends_on_resource(self, setUp):
print("testing {}".format(setUp))
In order to run:
pytest nam_of_the_module.py -v
I'm not sure I got the specifics of using Selenium in your original questions, but in case you were simply asking about how to use a more classical setUp/tearDown style, Pytest supports most unittest features, so you could do something like:
import unittest
class TestHello(unittest.TestCase):
def setUp(self):
print('running setUp')
def test_one(self):
print('running test_one')
def test_two(self):
print('running test_two')
def tearDown(self):
print('running tearDown')
Which produces:
$ pytest -s -v
====================== test session starts =======================
platform linux -- Python 3.8.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /gnu/store/nckjv3ccwdi6096j478gvns43ssbls2p-python-wrapper-3.8.2/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/test/.hypothesis/examples')
rootdir: /tmp/test
plugins: hypothesis-5.4.1
collected 2 items
test_hw.py::TestHello::test_one running setUp
running test_one
running tearDown
PASSED
test_hw.py::TestHello::test_two running setUp
running test_two
running tearDown
PASSED
I am writing some python unittest cases and use setUpClass to start with, however, I like some user interface that can control the saving path for some intermediate result, something like following. It is not working though, but the idea is I have a report_path argument in setUpClass argument list, so in some way, when instantiate this class, the user could input his own path:
class MyTestCase(unittest.TestCase):
#classmethod
def setUpClass(cls, report_path=None):
cls.s = cls.analyserfunction(....)
if report_path is None:
cls.report_path = something I defined by default
else:
cls.report_path = report_path
#classmethod
def analyserfunction(cls, ...):
def test_func(self):
do something for self.s
And I use TestLoader() to run it, something like this
suite = unittest.TestLoader().loadTestsFromTestCase(
test01.MyTestCase)
testRunner = unittest.TextTestRunner(
stream=sys.stderr, descriptions=True, verbosity=2)
testRunner.run(suite)
It works perfectly fine if I use the code above, but if naively do this
suite = unittest.TestLoader().loadTestsFromTestCase(
test01.MyTestCase(report_path = 'some path'))
it will fail, it seems that the test suite can only accept the test case class without any argument, not like the traditional class. Is there any way that can get through this, i.e, load MyTestCase(report_path = 'some path')?
I am using selenium for end to end testing and I can't get how to use setup_class and teardown_class methods.
I need to set up browser in setup_class method, then perform a bunch of tests defined as class methods and finally quit browser in teardown_class method.
But logically it seems like a bad solution, because in fact my tests will not work with class, but with object. I pass self param inside every test method, so I can access objects' vars:
class TestClass:
def setup_class(cls):
pass
def test_buttons(self, data):
# self.$attribute can be used, but not cls.$attribute?
pass
def test_buttons2(self, data):
# self.$attribute can be used, but not cls.$attribute?
pass
def teardown_class(cls):
pass
And it even seems not to be correct to create browser instance for class.. It should be created for every object separately, right?
So, I need to use __init__ and __del__ methods instead of setup_class and teardown_class?
According to Fixture finalization / executing teardown code, the current best practice for setup and teardown is to use yield instead of return:
import pytest
#pytest.fixture()
def resource():
print("setup")
yield "resource"
print("teardown")
class TestResource:
def test_that_depends_on_resource(self, resource):
print("testing {}".format(resource))
Running it results in
$ py.test --capture=no pytest_yield.py
=== test session starts ===
platform darwin -- Python 2.7.10, pytest-3.0.2, py-1.4.31, pluggy-0.3.1
collected 1 items
pytest_yield.py setup
testing resource
.teardown
=== 1 passed in 0.01 seconds ===
Another way to write teardown code is by accepting a request-context object into your fixture function and calling its request.addfinalizer method with a function that performs the teardown one or multiple times:
import pytest
#pytest.fixture()
def resource(request):
print("setup")
def teardown():
print("teardown")
request.addfinalizer(teardown)
return "resource"
class TestResource:
def test_that_depends_on_resource(self, resource):
print("testing {}".format(resource))
When you write "tests defined as class methods", do you really mean class methods (methods which receive its class as first parameter) or just regular methods (methods which receive an instance as first parameter)?
Since your example uses self for the test methods I'm assuming the latter, so you just need to use setup_method instead:
class Test:
def setup_method(self, test_method):
# configure self.attribute
def teardown_method(self, test_method):
# tear down self.attribute
def test_buttons(self):
# use self.attribute for test
The test method instance is passed to setup_method and teardown_method, but can be ignored if your setup/teardown code doesn't need to know the testing context. More information can be found here.
I also recommend that you familiarize yourself with py.test's fixtures, as they are a more powerful concept.
This might help http://docs.pytest.org/en/latest/xunit_setup.html
In my test suite, I group my test cases into classes. For the setup and teardown I need for all the test cases in that class, I use the setup_class(cls) and teardown_class(cls) classmethods.
And for the setup and teardown I need for each of the test case, I use the setup_method(method) and teardown_method(methods)
Example:
lh = <got log handler from logger module>
class TestClass:
#classmethod
def setup_class(cls):
lh.info("starting class: {} execution".format(cls.__name__))
#classmethod
def teardown_class(cls):
lh.info("starting class: {} execution".format(cls.__name__))
def setup_method(self, method):
lh.info("starting execution of tc: {}".format(method.__name__))
def teardown_method(self, method):
lh.info("starting execution of tc: {}".format(method.__name__))
def test_tc1(self):
<tc_content>
assert
def test_tc2(self):
<tc_content>
assert
Now when I run my tests, when the TestClass execution is starting, it logs the details for when it is beginning execution, when it is ending execution and same for the methods..
You can add up other setup and teardown steps you might have in the respective locations.
Hope it helps!
As #Bruno suggested, using pytest fixtures is another solution that is accessible for both test classes or even just simple test functions. Here's an example testing python2.7 functions:
import pytest
#pytest.fixture(scope='function')
def some_resource(request):
stuff_i_setup = ["I setup"]
def some_teardown():
stuff_i_setup[0] += " ... but now I'm torn down..."
print stuff_i_setup[0]
request.addfinalizer(some_teardown)
return stuff_i_setup[0]
def test_1_that_needs_resource(some_resource):
print some_resource + "... and now I'm testing things..."
So, running test_1... produces:
I setup... and now I'm testing things...
I setup ... but now I'm torn down...
Notice that stuff_i_setup is referenced in the fixture, allowing that object to be setup and torn down for the test it's interacting with. You can imagine this could be useful for a persistent object, such as a hypothetical database or some connection, that must be cleared before each test runs to keep them isolated.
Your code should work just as you expect it to if you add #classmethod decorators.
#classmethod
def setup_class(cls):
"Runs once per class"
#classmethod
def teardown_class(cls):
"Runs at end of class"
See http://pythontesting.net/framework/pytest/pytest-xunit-style-fixtures/
import pytest
class Test:
#pytest.fixture()
def setUp(self):
print("setup")
yield "resource"
print("teardown")
def test_that_depends_on_resource(self, setUp):
print("testing {}".format(setUp))
In order to run:
pytest nam_of_the_module.py -v
I'm not sure I got the specifics of using Selenium in your original questions, but in case you were simply asking about how to use a more classical setUp/tearDown style, Pytest supports most unittest features, so you could do something like:
import unittest
class TestHello(unittest.TestCase):
def setUp(self):
print('running setUp')
def test_one(self):
print('running test_one')
def test_two(self):
print('running test_two')
def tearDown(self):
print('running tearDown')
Which produces:
$ pytest -s -v
====================== test session starts =======================
platform linux -- Python 3.8.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /gnu/store/nckjv3ccwdi6096j478gvns43ssbls2p-python-wrapper-3.8.2/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/tmp/test/.hypothesis/examples')
rootdir: /tmp/test
plugins: hypothesis-5.4.1
collected 2 items
test_hw.py::TestHello::test_one running setUp
running test_one
running tearDown
PASSED
test_hw.py::TestHello::test_two running setUp
running test_two
running tearDown
PASSED
I'm running into a strange problem that seems to come about from how python unit tests manage their imports and how this relates to the mock package. This is for a django project, using django-nose/nose for unit test running and mock for mocking.
I have a unit test using mock which works perfectly fine when run alone (python manage.py test tests/test_code.py)
inside test_code.py:
from my.app.models.bookstore import create_from_proxy
class MockTestCase( TestCase ):
def setUp( self ):
self.patcher = patch( 'my.app.models.bookstore.BookProxy', autospec=True )
self.mock_proxy = self.patcher.start()
self.proxy_instance = self.mock_proxy.return_value
self.proxy_instance.json = json.loads(BOOK_JSON)
def tearDown( self ):
self.patcher.stop()
def test_mock_works( self ):
book_id = 55
v = create_from_proxy( book_id )
self.assertTrue( self.mock_proxy.called )
... more tests ...
inside bookstore.py:
from my.app.proxies import BookProxy
def create_from_proxy( self, id ):
proxy = BookProxy(id)
...
However when I run this test as part of the entire suite of tests (python manage.py test) then the test fails because the bookstore.py code doesn't get the mock class injected and falls back to the actual code for BookProxy.
So there seems to be something stateful going on when all of the tests are run together, but I can't figure out what's causing the mock injection to fail. Other unit tests that use mock all seem to be cleaning up after theirselves (either using the decorator, the context, or the explicit patch object approach that I'm showing here).
Something like this ever been seen out there before?
In a hobby project I intend to use nose for testing, I want to put all tests for specific classes into classes since these tests share setup and other functionality. But I can't seem to get nose to execute the setup methods inside the classes.
Here is an example class that is tested:
class mwe():
def __init__(self):
self.example = ""
def setExample(self, ex):
self.example = ex
The tests work, when I don't use classes:
from nose.tools import ok_
import mwe
exampleList = []
def setUp():
print("setup")
exampleList.append("1")
exampleList.append("2")
exampleList.append("3")
def test_Example():
print("test")
for ex in exampleList:
t = mwe.mwe()
t.setExample(ex)
yield check, t, ex
def check(e, ex):
ok_(e.example == ex)
The output is as expected:
setup
test
...
----------------------------------------------------------------------
Ran 3 tests in 0.004s
OK
When a test class is used, the setup method is not executed and therefore no tests are executed.
from nose.tools import ok_
import mwe
class TestexampleClass(object):
def __init__(self):
print("__init__")
self.exampleList = []
def setup(self):
print("setup class")
self.exampleList.append("1")
self.exampleList.append("2")
self.exampleList.append("3")
def test_ExampleClass(self):
print("test class")
for ex in self.exampleList:
t = mwe.mwe()
t.setExample(ex)
yield self.check, t, ex
def check(self, we, ex):
print("check class")
ok_(we.example == ex)
I'm fairly new to python and a newbie with nose, my question is, why is setup not executed? Where is the error in my code?
__init__
test class
----------------------------------------------------------------------
Ran 0 tests in 0.002s
OK
I will be glad for any feedback.
When I use the code from this Question on SO the setup method is executed, as I would expect it.
SOLUTION: After a lot of desperation I found the following:
Nose executes the the class level setup method before the execution of the yielded function, not when the test_* methods are called, as I expected and as is the case for other test_* methods. This obviously goes against the nose documentation:
Setup and teardown functions may be used with test generators. However, please note that setup and teardown attributes attached to the generator function will execute only once. To execute fixtures for each yielded test, attach the setup and teardown attributes to the function that is yielded, or yield a callable object instance with setup and teardown attributes.
Looking at the bug reports, I found the bug report on github.
A possible workaround is to use class level fixtures:
#classmethod
def setup_class(cls):
#do stuff
pass
Your test class needs to extend TestCase and the setup method needs to be called setUp
from unittest import TestCase
class TestUtils(TestCase):
def setUp(self):
self.x = 1
def test_something(self):
self.assertEqual(1, self.x)
which outputs
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK