Suppose I have a fixture fixture1 and I want to be able to skip all tests that use this fixture from pytest commandline.
I found answer to this quetion, but I decided to do it different way.
I am marking all tests that uses fixture1 with using_fixture1 marker by defining following in my conftest.py
def pytest_collection_modifyitems(items):
for item in items:
try:
fixtures=item.fixturenames
if 'fixture1' in fixtures:
item.add_marker(pytest.mark.using_fixture1)
except:
pass
And then, when running pytest I use: py.test tests -m "not using_fixture1".
So the real question should be: Are there any better ways to do it and are there any drawbacks of my solution?
Related
I'm using the pytest framework to run tests that interface with a set of test instruments. I'd like to have a set of initial tests in a single file that will check the connection and configuration of these instruments. If any of these tests fail, I'd like to abort all future tests.
In the remaining tests and test files, I'd like pytest to continue if there are any test failures so that I get a complete report of all failed tests.
For example, the test run may look something like this:
pytest test_setup.py test_series1.py test_series2.py
In this example, I'd like pytest to exit if any tests in test_setup.py fail.
I would like to invoke the test session in a single call so that session based fixtures that I have only get called once. For example, I have a fixture that will connect to a power supply and configure it for the tests.
Is there a way to tell pytest to exit on any test only in a specific file? If I use the -x option it will not continue in subsequent tests.
Ideally, I'd prefer something like a decorator that tells pytest to exit if there is a failure. However, I have not seen anything like this.
Is this possible or am I thinking about this the wrong way?
Update
Based on the answer from pL3b, my final solution looks like this:
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
if 'critical' in [mark.name for mark in item.own_markers]:
result = outcome.get_result()
if result.when == "call" and result.failed:
print('FAILED')
pytest.exit('Exiting pytest due to critical test failure', 1)
I needed to inspect the failure code in order to check if the test failed or not. Otherwise this would exit on every call.
Additionally, I needed to register my custome marker. I chose to also put this in the conftest.py file like so:
def pytest_configure(config):
config.addinivalue_line(
"markers", "critical: make test as critical"
)
You may use following hook in your conftest.py to solve your problem:
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
yield
if 'critical' in [mark.name for mark in item.own_markers]:
pytest.exit('Exiting pytest')
Then just add #pytest.mark.critical decorator to desired tests/classes in test_setup.py.
Learn more about pytest hooks here so you can define desired output and so on.
I wish to configure pytest such that it excludes some tests by default; but it should be easy to include them again with some command line option. I only found -k, and I have the impression that that allows complex specifications, but am not sure how to go about my specific need...
The exclusion should be part of the source or a config file (it's permanent - think about very long-running tests which should only be included as conscious choice, certainly never in a build pipeline...).
Bonus question: if that is not possible, how would I use -k to exclude specific tests? Again, I saw hints in the documentation about a way to use not as a keyword, but that doesn't seem to work for me. I.e., -k "not longrunning" gives an error about not being able to find a file "notrunning", but does not exclude anything...
goal: by default skip tests marked as #pytest.mark.integration
conftest.py
import pytest
# function executed right after test items collected but before test run
def pytest_collection_modifyitems(config, items):
if not config.getoption('-m'):
skip_me = pytest.mark.skip(reason="use `-m integration` to run this test")
for item in items:
if "integration" in item.keywords:
item.add_marker(skip_me)
pytest.ini
[pytest]
markers =
integration: integration test that requires environment
now all tests marked as #pytest.mark.integration are skipped unless you use
pytest -m integration
You can use pytest to mark some tests and use -k arg to skip or include them.
For example consider following tests,
import pytest
def test_a():
assert True
#pytest.mark.never_run
def test_b():
assert True
def test_c():
assert True
#pytest.mark.never_run
def test_d():
assert True
you can run pytest like this to run all the tests
pytest
To skip the marked tests you can run pytest like this,
pytest -m "not never_run"
If you want to run the marked tests alone,
pytest -m "never_run"
What I have done in the past is create custom markers for these tests that way I can exclude them using the -m command line flag for running the tests. So for example in your pytest.ini file place the following content:
[pytest]
markers =
longrunning: marks a test as longrunning
Then we just have to mark our long running tests with this marker.
#pytest.mark.longrunning
def test_my_long_test():
time.sleep(100000)
Then when we run the tests we would do pytest -m "not longrunning" tests/ to exclude them and pytest tests to run everything as intended.
Assuming the following test suite:
# test_module.py
import unittest
class Tests(unittest.TestCase):
#unittest.skip
def test_1(self):
print("This should run only if explicitly asked to but not by default")
# assume many other test cases and methods with or without the skip marker
When invoking the unittest library via python -m unittest are there any arguments I can pass to it actually run and not skip Tests.test_1 without modifying the test code and running any other skipped tests?
python -m unittest test_module.Tests.test_1 correctly selects this as the only test to run, but it still skips it.
If there is no way to do it without modifying the test code, what is the most idiomatic change I can make to conditionally undo the #unittest.skip and run one specific test case test case?
In all cases, I still want python -m unittest discover (or any other invocation that doesn't explicitly turn on the test) to skip the test.
If you want to skip some expensive tests you can use a conditional skip together with a custom environment variable:
#skipIf(int(os.getenv('TEST_LEVEL', 0)) < 1)
def expensive_test(self):
...
Then you can include this test by specifying the corresponding environment variable:
TEST_LEVEL=1 python -m unittest discover
TEST_LEVEL=1 python -m unittest test_module.Tests.test_1
If you want to skip a test because you expect it to fail, you can use the dedicated expectedFailure decorator.
By the way, pytest has a dedicated decorator for marking slow tests.
I see from here that I can pick out tests based on their mark like so:
pytest -v -m webtest
Let's say I have a test decorated like so:
#pytest.mark.parametrize('platform,configuration', (
pytest.param('win', 'release')
pytest.param('win', 'debug')))
def test_Foo(self):
I'd like to do something like the following:
pytest -v -m parameterize.configuration.release
That way I run test_Foo with the parameter of 'release' but not the parameter of 'debug'. Is there a way to do this? I think I can do this by writing a wrapper test and then passing only the desired parameter down, but I want to avoid doing so because we already have a large number of tests parameterized as describe, and I want to avoid writing a large number of wrapper tests.
You can use -k for expression-based filtering:
$ pytest -k win-release
will run only tests containing win-release in their names. You can list all names without executing the tests by issuing
$ pytest --collect-only -q
Should an expression be not enough, you can always extend pytest by adding your custom filtering logic, for example by passing parameter name and value via command line args and selecting only tests that are parametrized accordingly:
# conftest.py
def pytest_addoption(parser):
parser.addoption('--param-name', action='store', help='parameter name')
parser.addoption('--param-value', action='store', help='parameter value')
def pytest_collection_modifyitems(session, config, items):
param_name = config.getoption('--param-name')
param_value = config.getoption('--param-value')
if param_name and param_value:
items[:] = [item for item in items
if hasattr(item, 'callspec')
and param_name in item.callspec.params
and item.callspec.params[param_name] == param_value]
Now you can e.g. call
$ pytest --param-name=platform --param-value=win
and only tests parametrized with platform=win will be executed.
An alternative to the official answer by hoefling is to create a special marker with pytest-pilot and to apply it:
conftest.py:
from pytest_pilot import EasyMarker
mymark = EasyMarker('mymark', has_arg=False, mode='hard_filter')
test_so.py:
import pytest
from .conftest import mymark
#pytest.mark.parametrize('platform,configuration', (
mymark.param('win', 'release'),
pytest.param('win', 'debug')
))
def test_foo(platform, configuration):
pass
You can now run pytest --mymark, it correctly runs only the test with the mark
test_so\test_so.py::test_foo[win-release] PASSED [ 50%]
test_so\test_so.py::test_foo[win-debug] SKIPPED [100%]
Of course it might not be relevant in all cases, as it requires code modification ; however for advanced filtering patterns, or if the filtering is here to stay and you wish to have some CLI shortcuts to perform it, it might be interesting. Note: I'm the author of this lib ;)
given that this is my test code:
# conftest.py
#pytest.fixture(scope='function')
def fixA(request)
pass
#pytest.fixture(scope='function')
def fixB(request)
pass
# test_my.py
pytestmark = pytest.mark.usefixtures("fixA")
def test_something():
pass
I want to be able to use fixB() insted of fixA(), using fixB() is easy enough, I could add pytest.ini like this:
[pytest]
usefixtures=fixB
but I could figure out how can I disable fixA, from the commandline of from the configuration.
is my use case is so far fetched ?
(the actual reason, I want to keep fixA working in our CI system, but for day to day work, I need people to be able to disable it on their desk)
I've found a way, (after fighting it in the debugger, py.test documentation isn't that clear regarding how fixtures are being selected)
Added this into the conftest.py:
# conftest.py
def pytest_runtest_setup(item):
for fixture in item.config.getini('ignorefixtures'):
del item._fixtureinfo.name2fixturedefs[fixture]
item._fixtureinfo.names_closure.remove(fixture)
def pytest_addoption(parser):
parser.addini('ignorefixtures', 'fixtures to ignore', 'linelist')
and then I could put this in pytest.ini:
[pytest]
usefixtures=fixB
ignorefixtures=fixA
I would be nice to those kind of things also in the command line...