I want to run pytest multiple times from within Python without restarting the script/interpreter.
The problem is that pytest is caching test contents/results. That is, if you modify a test file between two runs, pytest doesn't pick up the changes, showing the same results as before. (Unless you restart the script/exit the interpreter, which you'd naturally do when using pytest from the command line.)
Reproduction
test_foo.py:
def test_me():
assert False
In the Python shell:
>>> import pytest
>>> pytest.main(['test_foo.py'])
(...)
def test_me():
> assert False
E assert False
test_foo.py:2: AssertionError
Good so far. Now don't exit the interpreter but change the test to assert True and re-run pytest.
>>> pytest.main(['test_foo.py'])
(...)
def test_me():
> assert True
E assert False
test_foo.py:2: AssertionError
Expected result
Pytest should have picked up the change in the file and pass the rewritten test.
Solutions that don't work
Using importlib.reload(pytest) to reload pytest between the runs.
Running pytest with cleared caches: pytest.main(['--cache-clear', test_foo.py'])
(Running pytest as a subprocess isn't an option because I want have a reference to the pytest module from within my application.)
Any hints how to make pytest pick up these changes or how to properly reload the module?
Anyone landing here, another answer that may help, just install pytest-xdist plugin, and call pytest --looponfail
Related
I wish to configure pytest such that it excludes some tests by default; but it should be easy to include them again with some command line option. I only found -k, and I have the impression that that allows complex specifications, but am not sure how to go about my specific need...
The exclusion should be part of the source or a config file (it's permanent - think about very long-running tests which should only be included as conscious choice, certainly never in a build pipeline...).
Bonus question: if that is not possible, how would I use -k to exclude specific tests? Again, I saw hints in the documentation about a way to use not as a keyword, but that doesn't seem to work for me. I.e., -k "not longrunning" gives an error about not being able to find a file "notrunning", but does not exclude anything...
goal: by default skip tests marked as #pytest.mark.integration
conftest.py
import pytest
# function executed right after test items collected but before test run
def pytest_collection_modifyitems(config, items):
if not config.getoption('-m'):
skip_me = pytest.mark.skip(reason="use `-m integration` to run this test")
for item in items:
if "integration" in item.keywords:
item.add_marker(skip_me)
pytest.ini
[pytest]
markers =
integration: integration test that requires environment
now all tests marked as #pytest.mark.integration are skipped unless you use
pytest -m integration
You can use pytest to mark some tests and use -k arg to skip or include them.
For example consider following tests,
import pytest
def test_a():
assert True
#pytest.mark.never_run
def test_b():
assert True
def test_c():
assert True
#pytest.mark.never_run
def test_d():
assert True
you can run pytest like this to run all the tests
pytest
To skip the marked tests you can run pytest like this,
pytest -m "not never_run"
If you want to run the marked tests alone,
pytest -m "never_run"
What I have done in the past is create custom markers for these tests that way I can exclude them using the -m command line flag for running the tests. So for example in your pytest.ini file place the following content:
[pytest]
markers =
longrunning: marks a test as longrunning
Then we just have to mark our long running tests with this marker.
#pytest.mark.longrunning
def test_my_long_test():
time.sleep(100000)
Then when we run the tests we would do pytest -m "not longrunning" tests/ to exclude them and pytest tests to run everything as intended.
I have written a Test Suite.
myTestsuite.py
import unittest
from myTestCase2 import MyTestCase2
from prime_num_validation import Prime_Num_Validation
def my_test_suite():
suite = unittest.TestSuite()
suite.addTest(MyTestCase2('test_greaterCheck2'))
#To add only test case: test_greaterCheck2 from the MyTestCase2 class
suite.addTest(Prime_Num_Validation('test_prime_check'))
#To add only test case: test_prime_check from the MyTestCase2 class
return suite
if __name__ == '__main__':
runner = unittest.TextTestRunner()
runner.run(my_test_suite())
Now when I run this using command line with: python -m unittest -v myTestsuite, It runs all the test cases from the MyTestCase2 class, which actually has 3 TC's, but we added only one out of 3 in our suite.
How should we avoid invoking all test case and executing only those which are present in the suite.
When I run this using Pycharm editor,
it again executes all the test cases from MyTestCase2.
You can have the marker on top of your unit test, something call xfail that will skip the test.
for example,
Below example skip test_function3()
import sys
def test_function1():
def test_function2():
#pytest.mark.skipif(sys.version_info < (3,3),
reason="requires python3.3")
def test_function3():
For reference please go through this site, you will find more information py.test skipif
the link I provided above has example of using xfail marker also.
You can use the xfail marker or create your own custom marker also. xfail is to indicate that you expect a test to fail.
You can run xfail test using following command.
pytest --runxfail
As with skipif you can also mark your expectation of a failure on a particular
platform:
Example
import pytest
xfail = pytest.mark.xfail
#xfail
def test_func1():
assert 0
#xfail(run=False)
def test_func2():
assert 0
I also assumed that running python -m unittest -v myTestsuite would execute just the test cases in the defined test suites. Calling the myTestsuite.py test module directly (i.e., not passing module as a parameter to the unittest library module), however, should produce the result you are after. Try running the following:
python myTestsuite
NOTE: You will need to pass the "verbosity=1" argument to the TextTestRunner function instead of using "-v" on command line (or modify myTestsuite.py to take a "-v" parameter and pass that to TextTestRunner)
I have some tests written using pytest and fixtures, e.g.:
class TestThing:
#pytest.fixture()
def temp_dir(self, request):
my_temp_dir = tempfile.mkdtemp()
def fin():
shutil.rmtree(my_temp_dir)
request.addfinalizer(fin)
return my_temp_dir
def test_something(self, temp_dir)
with open(os.path.join(temp_dir, 'test.txt'), 'w') as f:
f.write('test')
This works fine when the tests are invoked from the shell, e.g.
$ py.test
but I don't know how to run them from within a python/ipython session; trying e.g.
tt = TestThing()
tt.test_something(tt.temp_dir())
fails because temp_dir requires a request object to be passed on. So, how does one invoke a fixture with a request object injected?
Yes. You don't have to manually assemble any test fixtures or anything like that. Everything runs just like calling pytest in the project directory.
Method1:
This is the best method because it gives you access to the debugger if your test fails
In ipython shell use:
**ipython**> run -m pytest prj/
This will run all your tests in the prj/tests directory.
This will give you access to the debugger, or allow you to set breakpoints if you have a
import ipdb; ipdb.set_trace() in your program (https://docs.pytest.org/en/latest/usage.html#setting-breakpoints).
Method2:
Use !pytest while in the test directory. This wont give you access to the debugger. However, if you use
**ipython**> !pytest --pdb
If you have a test failure, it will drop you into the debugger (subshell), so you can run your post-mortem analysis (https://docs.pytest.org/en/latest/usage.html#dropping-to-pdb-python-debugger-on-failures)
Using these methods you can even run individual modules/test_fuctions/TestClasses in ipython using (https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests)
**ipython**> run -m pytest prj/tests/test_module1.py::TestClass1::test_function1
You can bypass the pytest.fixture decorator and directly call the wrapped test function.
tmp = tt.temp_dir.__pytest_wrapped__.obj(request=...)
Accessing internals, it's bad, but when you need it...
The best method I have which is far from ideal is to just %run the test file, then manually assemble the fixtures, then simply call the tests. The problem with this is tracking down the modules where the default fixtures are defined, and then calling them in their order of dependencies.
you can use two cells for this:
first:
def test_something():
assert True
second:
from tempfile import mktemp
test_file = mktemp('.py', 'test_')
open(test_file, 'wb').write(_i) # write last cell input
!pytest $test_file
also u can do this on one cell like this but you won't have code highlighting:
from tempfile import mktemp
test_code = """
def test_something():
assert True
"""
test_file = mktemp('.py', 'test_')
open(test_file, 'wb').write(test_code)
!pytest $test_file
The simple answer is that you don't want to run py.test interactively from python. Most people set up some integration with their text editor or IDE to run py.test and parse it's output. But really it's a command line tool and that is how it should be used.
As a sidenode you may want to check out the built-in tmpdir fixture: http://pytest.org/latest/tmpdir.html Because it seems like you're re-inventing this.
How to just list all discovered tests?
I found this command:
python3.4 -m unittest discover -s .
But it's not exactly what I want, because the above command executes tests. I mean let's have a project with a lot of tests. Execution time is a few minutes. This force me to wait until tests are finished.
What I want is something like this (above command's output)
test_choice (test.TestSequenceFunctions) ... ok
test_sample (test.TestSequenceFunctions) ... ok
test_shuffle (test.TestSequenceFunctions) ... ok
or even better, something more like this (after editing above):
test.TestSequenceFunctions.test_choice
test.TestSequenceFunctions.test_sample
test.TestSequenceFunctions.test_shuffle
but without execution, only printing tests "paths" for copy&paste purpose.
Command line command discover is implemented using unittest.TestLoader. Here's the somewhat elegant solution
import unittest
def print_suite(suite):
if hasattr(suite, '__iter__'):
for x in suite:
print_suite(x)
else:
print(suite)
print_suite(unittest.defaultTestLoader.discover('.'))
Running example:
In [5]: print_suite(unittest.defaultTestLoader.discover('.'))
test_accounts (tests.TestAccounts)
test_counters (tests.TestAccounts)
# More of this ...
test_full (tests.TestImages)
This works because TestLoader.discover returns TestSuite objects, that implement __iter__ method and therefore are iterable.
You could do something like:
from your_tests import TestSequenceFunctions
print('\n'.join([f.__name__ for f in dir(TestSequenceFunctions) if f.__name__.startswith('test_')]))
I'm not sure if there is an exposed method for this via unittest.main.
nose discovers tests beginning with test_, as well as subclasses of unittest.TestCase.
If one wishes to run a single TestCase test, e.g.:
# file tests.py
class T(unittest.TestCase):
def test_something():
1/0
This can be done on the command line with:
nosetests tests:T.test_something
I sometimes prefer to write a simple function and skip all the unittest boilerplate:
def test_something_else():
assert False
In that case, the test will still be run by nose when it runs all my tests. But how can I tell nose to run only that test, using the (Unix) command line?
That would be:
nosetests tests:test_something_else
An additional tip is to use attributes
from nose.plugins.attrib import attr
#attr('now')
def test_something_else():
pass
To run all tests tagged with the attribute, execute:
nosetests -a now
Inversely, avoid running those tests:
nosetests -a !now