Develop Reference

pytest exit on failure only for a specific file

I'm using the pytest framework to run tests that interface with a set of test instruments. I'd like to have a set of initial tests in a single file that will check the connection and configuration of these instruments. If any of these tests fail, I'd like to abort all future tests.
In the remaining tests and test files, I'd like pytest to continue if there are any test failures so that I get a complete report of all failed tests.
For example, the test run may look something like this:
pytest test_setup.py test_series1.py test_series2.py
In this example, I'd like pytest to exit if any tests in test_setup.py fail.
I would like to invoke the test session in a single call so that session based fixtures that I have only get called once. For example, I have a fixture that will connect to a power supply and configure it for the tests.
Is there a way to tell pytest to exit on any test only in a specific file? If I use the -x option it will not continue in subsequent tests.
Ideally, I'd prefer something like a decorator that tells pytest to exit if there is a failure. However, I have not seen anything like this.
Is this possible or am I thinking about this the wrong way?
Update
Based on the answer from pL3b, my final solution looks like this:
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
if 'critical' in [mark.name for mark in item.own_markers]:
result = outcome.get_result()
if result.when == "call" and result.failed:
print('FAILED')
pytest.exit('Exiting pytest due to critical test failure', 1)
I needed to inspect the failure code in order to check if the test failed or not. Otherwise this would exit on every call.
Additionally, I needed to register my custome marker. I chose to also put this in the conftest.py file like so:
def pytest_configure(config):
config.addinivalue_line(
"markers", "critical: make test as critical"
)

You may use following hook in your conftest.py to solve your problem:
#pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
yield
if 'critical' in [mark.name for mark in item.own_markers]:
pytest.exit('Exiting pytest')
Then just add #pytest.mark.critical decorator to desired tests/classes in test_setup.py.
Learn more about pytest hooks here so you can define desired output and so on.

Show full plan in Pytest

I'm looking for a way in Pytest to show the full test and fixture plan instead of just listing the test cases via --collect-only.
This is the best I can get now:
TestClass1
TestCase1
TestCase2
TestClass2
TestCase3
TestCase4
This is what I'm looking for (should match the execution order):
Fixture1_Setup_ModuleScope
Fixture2_Setup_ClassScope
TestClass1
Fixture3_Setup_FunctionScope
TestCase1
Fixture3_Teardown_FunctionScope
TestCase2
Fixture2_Teardown_ClassScope
TestClass2
TestCase3
TestCase4
Fixture1_Teardown_ModuleScope
I looked around for such Pytest plugin and none seems to provide this. Not even as parsing of the result, let alone something that could be generated without running the tests. I understand that it's not needed for Pytest testing, but it's something I've learned to like in one of our older internal test frameworks, if only for validating my intention with reality.
Am I missing some obvious solution here? How could I achieve this?

Have you tried pytest --setup-plan.
show what fixtures and tests would be executed but don't execute anything.
pytest --setup-plan
# ...
# assert_test.py
# assert_test.py::TestTest::test_test
# click_test.py
# click_test.py::test_echo_token
# fixture_test.py
# SETUP F env['dev']
# SETUP F folder['dev_data']
# fixture_test.py::test_are_folders_exist[dev-dev_data] (fixtures used: env, folder)
# TEARDOWN F folder['dev_data']
# TEARDOWN F env['dev']

Exclude some tests by default

I wish to configure pytest such that it excludes some tests by default; but it should be easy to include them again with some command line option. I only found -k, and I have the impression that that allows complex specifications, but am not sure how to go about my specific need...
The exclusion should be part of the source or a config file (it's permanent - think about very long-running tests which should only be included as conscious choice, certainly never in a build pipeline...).
Bonus question: if that is not possible, how would I use -k to exclude specific tests? Again, I saw hints in the documentation about a way to use not as a keyword, but that doesn't seem to work for me. I.e., -k "not longrunning" gives an error about not being able to find a file "notrunning", but does not exclude anything...

goal: by default skip tests marked as #pytest.mark.integration
conftest.py
import pytest
# function executed right after test items collected but before test run
def pytest_collection_modifyitems(config, items):
if not config.getoption('-m'):
skip_me = pytest.mark.skip(reason="use `-m integration` to run this test")
for item in items:
if "integration" in item.keywords:
item.add_marker(skip_me)
pytest.ini
[pytest]
markers =
integration: integration test that requires environment
now all tests marked as #pytest.mark.integration are skipped unless you use
pytest -m integration

You can use pytest to mark some tests and use -k arg to skip or include them.
For example consider following tests,
import pytest
def test_a():
assert True
#pytest.mark.never_run
def test_b():
assert True
def test_c():
assert True
#pytest.mark.never_run
def test_d():
assert True
you can run pytest like this to run all the tests
pytest
To skip the marked tests you can run pytest like this,
pytest -m "not never_run"
If you want to run the marked tests alone,
pytest -m "never_run"

What I have done in the past is create custom markers for these tests that way I can exclude them using the -m command line flag for running the tests. So for example in your pytest.ini file place the following content:
[pytest]
markers =
longrunning: marks a test as longrunning
Then we just have to mark our long running tests with this marker.
#pytest.mark.longrunning
def test_my_long_test():
time.sleep(100000)
Then when we run the tests we would do pytest -m "not longrunning" tests/ to exclude them and pytest tests to run everything as intended.

In pytest, how to skip all tests that use specified fixture?

Suppose I have a fixture fixture1 and I want to be able to skip all tests that use this fixture from pytest commandline.
I found answer to this quetion, but I decided to do it different way.
I am marking all tests that uses fixture1 with using_fixture1 marker by defining following in my conftest.py
def pytest_collection_modifyitems(items):
for item in items:
try:
fixtures=item.fixturenames
if 'fixture1' in fixtures:
item.add_marker(pytest.mark.using_fixture1)
except:
pass
And then, when running pytest I use: py.test tests -m "not using_fixture1".
So the real question should be: Are there any better ways to do it and are there any drawbacks of my solution?

Can tests with pytest fixtures be run interactively?

I have some tests written using pytest and fixtures, e.g.:
class TestThing:
#pytest.fixture()
def temp_dir(self, request):
my_temp_dir = tempfile.mkdtemp()
def fin():
shutil.rmtree(my_temp_dir)
request.addfinalizer(fin)
return my_temp_dir
def test_something(self, temp_dir)
with open(os.path.join(temp_dir, 'test.txt'), 'w') as f:
f.write('test')
This works fine when the tests are invoked from the shell, e.g.
$ py.test
but I don't know how to run them from within a python/ipython session; trying e.g.
tt = TestThing()
tt.test_something(tt.temp_dir())
fails because temp_dir requires a request object to be passed on. So, how does one invoke a fixture with a request object injected?

Yes. You don't have to manually assemble any test fixtures or anything like that. Everything runs just like calling pytest in the project directory.
Method1:
This is the best method because it gives you access to the debugger if your test fails
In ipython shell use:
**ipython**> run -m pytest prj/
This will run all your tests in the prj/tests directory.
This will give you access to the debugger, or allow you to set breakpoints if you have a
import ipdb; ipdb.set_trace() in your program (https://docs.pytest.org/en/latest/usage.html#setting-breakpoints).
Method2:
Use !pytest while in the test directory. This wont give you access to the debugger. However, if you use
**ipython**> !pytest --pdb
If you have a test failure, it will drop you into the debugger (subshell), so you can run your post-mortem analysis (https://docs.pytest.org/en/latest/usage.html#dropping-to-pdb-python-debugger-on-failures)
Using these methods you can even run individual modules/test_fuctions/TestClasses in ipython using (https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests)
**ipython**> run -m pytest prj/tests/test_module1.py::TestClass1::test_function1

You can bypass the pytest.fixture decorator and directly call the wrapped test function.
tmp = tt.temp_dir.__pytest_wrapped__.obj(request=...)
Accessing internals, it's bad, but when you need it...

The best method I have which is far from ideal is to just %run the test file, then manually assemble the fixtures, then simply call the tests. The problem with this is tracking down the modules where the default fixtures are defined, and then calling them in their order of dependencies.

you can use two cells for this:
first:
def test_something():
assert True
second:
from tempfile import mktemp
test_file = mktemp('.py', 'test_')
open(test_file, 'wb').write(_i) # write last cell input
!pytest $test_file
also u can do this on one cell like this but you won't have code highlighting:
from tempfile import mktemp
test_code = """
def test_something():
assert True
"""
test_file = mktemp('.py', 'test_')
open(test_file, 'wb').write(test_code)
!pytest $test_file

The simple answer is that you don't want to run py.test interactively from python. Most people set up some integration with their text editor or IDE to run py.test and parse it's output. But really it's a command line tool and that is how it should be used.
As a sidenode you may want to check out the built-in tmpdir fixture: http://pytest.org/latest/tmpdir.html Because it seems like you're re-inventing this.