I'm working on a functional test suite using pytest with pytest-dependency. I 99% love these tools, but I can't figure out how to have a test in one file depend on a test in another file. Ideally, I'd like to have zero changes required to the dependee, and only change things in the depender. I'd like tests to be able to depend on test_one both like this:
# contents of test_one.py
#pytest.mark.dependency()
def test_one():
# do stuff
#pytest.mark.dependency(depends=["test_one"])
def test_point_one():
# do stuff
And like this:
# contents of test_two.py
#pytest.mark.dependency(depends=["test_one"])
def test_two():
# do stuff
When I run pytest test_one.py it correctly orders things (and skips test_point_one if test_one fails), but when I run pytest test_two.py, it skips test_two.
I've tried adding import test_one to test_two.py to no avail, and verified that the import is actually importing properly - it's not just getting passed over by pytest going "Oh hey, I've finished collecting tests, and there's nothing that I can't skip! Hooray for laziness!"
I know I could technically put test_two() in test_one.py and it would work, but I don't want to just dump every test in a single file (which is what this would ultimately devolve into). I'm trying to keep stuff tidy by putting everything on the right shelf, not just shoving it all into the closet.
Also, I realize the possibility of creating circular dependencies would exist if this is something I can do. I'm okay with this. If I shot myself in the foot like that, let's be honest, I'd deserve it.
Current status, 31-May-2018, pytest-dependency==0.3.2
At the moment, pytest-dependency does the dependency resolution on module level only. Although there is some rudimentary implementation for resolving session-scoped dependencies, the full support is not implemented at the moment of writing this. You can check that by slipping session scope instead of module scope:
# conftest.py
from pytest_dependency import DependencyManager
DependencyManager.ScopeCls['module'] = DependencyManager.ScopeCls['session']
Now test_two from your example will resolve the dependency to test_one. However, this is just a dirty hack for demonstration purposes that will easily corrupt the dependencies once you add another test named test_one so read further.
Solution proposal
There is a PR that adds the dependency resolution on session and class levels, but it's not accepted yet by the package maintainer It is now accepted.
You can use that instead:
$ pip uninstall -y pytest-dependency
$ pip install git+https://github.com/JoeSc/pytest-dependency.git#master
Now the dependency mark accepts an additional arg scope:
#pytest.mark.dependency(scope='session')
def test_one():
...
You will need to use the full test name (as printed by pytest -v) in order to depend on test_one in another module:
#pytest.mark.dependency(depends=['test_one.py::test_one'], scope='session')
def test_two():
...
Named dependencies are also supported:
#pytest.mark.dependency(name='spam', scope='session')
def test_one():
...
#pytest.mark.dependency(depends=['spam'], scope='session')
def test_two():
...
Related
I'm using pytest. I have a test which involves checking that an import is not made when something happens. This is easy enough to make, but when the test is run in pytest it gets run in the same process as many other tests, which may import that thing beforehand.
Is there some way to mark a test to be run in its own process? Ideally there'd be some kind of decorator like
#pytest.mark.run_in_isolation
def test_import_not_made():
....
But I haven't found anything like that.
I don't know of a pytest plugin that allows marking a test to run in its own process. The two I'd check are pytest-xdist and ptyest-xprocess (here's a list of pytest plugins), though they don't look like they'll do what you want.
I'd go with a different solution. I assume that the way you're checking whether a module is imported is whether it's in sys.modules. As such, I'd ensure sys.modules doesn't contain the module you're interested in before the test run.
Something like this will ensure sys.modules is in a clean state before your test run.
import sys
#pytest.fixture
def clean_sys_modules():
try:
del sys.modules['yourmodule']
except KeyError:
pass
assert 'yourmodule' not in sys.modules # Sanity check.
#pytest.mark.usefixtures('clean_sys_modules')
def test_foo():
# Do the thing you want NOT to do the import.
assert 'yourmodule' not in sys.modules
I wrote a python script to do all my tests automatically for me, and generate a HTML report. I discovered discover for unittests the other day which lets me run all the unittests in a given directory without explicitly naming them, and I'd really like to be able to do my doctests the same way, rather than having to import each module explicitly.
I found some info on how to do this at https://docs.python.org/2/library/doctest.html but didn't really get it. Could you please help me with using discover with my doctests?
Python test discovery with doctests, coverage and parallelism is related, but still doesn't answer my question.
coverage_module
import coverage
import doctest
import unittest
import os
# import test_module
import my_module
cov = coverage.Coverage()
cov.start()
# running doctest by explicity naming the module
doctest.testmod(my_module)
# running unittests by just specifying the folder to look into
testLoad = unittest.TestLoader()
testSuite = testLoad.discover(start_dir=os.getcwd())
runner = unittest.TextTestRunner()
runner.run(testSuite)
cov.stop()
cov.save()
cov.html_report()
print "tests completed"
test_module
import unittest
import doctest
from my_module import My_Class
class My_Class_Tests(unittest.TestCase):
def setUp(self):
# setup variables
def test_1(self):
# test code
# The bit that should load up the doctests? What's loader, tests, and ignore though?
# Is this in the right place?
def load_tests(loader, tests, ignore):
tests.addTests(doctest.DocTestSuite(module_with_doctests))
return tests
if __name__ == '__main__':
unittest.main()
Lets figure out what's happening there
1) unittest.discovery
It has no clue of doctests as doctests is a different framework.
So unittest isn't supposed to discover doctests out of the box.
That means you'll need to glue them together by hand
2) doctest
It's essentially a separate framework although it has some glueing classes to convert doctests into unittest-like TestCases.
https://docs.python.org/2.7/library/doctest.html#doctest.DocTestSuite
3) discover
Didn't get what discover you mean, I suppose it's
python -m unittest discover
If not and you're talking about https://pypi.python.org/pypi/discover then just forget about it - it's a backport for earlier versions of python
4) what to do
either scatter a lot of load_tests hooks across your code as described here https://docs.python.org/2.7/library/doctest.html#unittest-api or code a method to collect all the modules your have in one place and convert them into a DocTestSuite[s] https://docs.python.org/2.7/library/doctest.html#doctest.DocTestSuite
But honestly neither approach makes any sense nowadays as it boils down to:
$ py.test --doctest-modules
or
$ nosetests --with-doctest
Of course coverage and lots of bells & whistles are also supplied by these frameworks and you may keep sticking to unittest.TestCase, and you won't even need to create a coverage_module so I would dig into one of them rather then trying to come up with your own solution
I am writing functional tests using pytest for a software that can run locally and in the cloud. I want to create 2 modules, each with the same module/fixture names, and have pytest load one or the other depending if I'm running tests locally or in the cloud:
/fixtures
/fixtures/__init__.py
/fixtures/local_hybrids
/fixtures/local_hybrids/__init__.py
/fixtures/local_hybrids/foo.py
/fixtures/cloud_hybrids
/fixtures/cloud_hybrids/__init__.py
/fixtures/cloud_hybrids/foo.py
/test_hybrids/test_hybrids.py
foo.py (both of them):
import pytest
#pytest.fixture()
def my_fixture():
return True
/fixtures/__init__.py:
if True:
import local_hybrids as hybrids
else:
import cloud_hybrids as hybrids
/test_hybrids/test_hybrids.py:
from fixtures.hybrids.foo import my_fixture
def test_hybrid(my_fixture):
assert my_fixture
The last code block doesn't work of course, because import fixtures.hybrids is looking at the file system instead of __init__.py's "fake" namespace, which isn't like from fixtures import hybrids, which works (but then you cannot use the fixtures as the names would involve dot notation).
I realize that I could play with pytest_generate_test to alter the fixture dynamically (maybe?) but I'd really hate managing each fixture manually from within that function... I was hoping the dynamic import (if x, import this, else import that) was standard Python, unfortunately it clashes with the fixtures mechanism:
import fixtures
def test(fixtures.hybrids.my_fixture): # of course it doesn't work :)
...
I could also import each fixture function one after the other in init; more legwork, but still a viable option to fool pytest and get fixture names without dots.
Show me the black magic. :) Can it be done?
I think in your case it's better to define a fixture - environment or other nice name.
This fixture can be just a getter from os.environ['KEY'] or you can add custom command line argument like here
then use it like here
and the final use is here.
What im trying to tell is that you need to switch thinking into dependency injection: everything should be a fixture. In your case (and in my plugin as well), runtime environment should be a fixture, which is checked in all other fixtures which depend on the environment.
You might be missing something here: If you want to re-use those fixtures you need to say it explicitly:
from fixtures.hybrids.foo import my_fixture
#pytest.mark.usefixtures('my_fixture')
def test_hybrid(my_fixture):
assert my_fixture
In that case you could tweak pytest as following:
from local_hybrids import local_hybrids_fixture
from cloud_hybrids import cloud_hybrids_fixture
fixtures_to_test = {
"local":None,
"cloud":None
}
#pytest.mark.usefixtures("local_hybrids_fixture")
def test_add_local_fixture(local_hybrids_fixture):
fixtures_to_test["local"] = local_hybrids_fixture
#pytest.mark.usefixtures("cloud_hybrids_fixture")
def test_add_local_fixture(cloud_hybrids_fixture):
fixtures_to_test["cloud"] = cloud_hybrids_fixture
def test_on_fixtures():
if cloud_enabled:
fixture = fixtures_to_test["cloud"]
else:
fixture = fixtures_to_test["local"]
...
If there are better solutions around I am also interested ;)
I don't really think there is a "good way" of doing that in python, but still it is possible with a little amount of hacking. You can update sys.path for the subfolder with fixtures you would like to use and import fixtures directly. In dirty case it look's like that:
for your fixtures/__init__.py:
if True:
import local as hybrids
else:
import cloud as hybrids
def update_path(module):
from sys import path
from os.path import join, pardir, abspath
mod_dir = abspath(join(module.__file__, pardir))
path.insert(0, mod_dir)
update_path(hybrids)
and in the client code (test_hybrids/test_hybrids.py) :
import fixtures
from foo import spam
spam()
In other cases you can use much more complex actions to perform a fake-move of all modules/packages/functions etc from your cloud/local folder directly into the fixture's __init__.py. Still, I think - it does not worth a try.
One more thing - black magic is not the best thing to use, I would recommend you to use a dotted notation with "import X from Y" - this is much more stable solution.
Use the pytest plugins feature and put your fixtures in separate modules. Then at runtime select which plug-in you’ll be drawing from via a command line argument or an environment variable. It needs to be something global because you need to place different pytest_plugins list assignments based on the global value.
Take a look at the section Conditional Plugins from this repo https://github.com/jxramos/pytest_behavior/tree/main/conditional_plugins
I'd like to connect to a different database if my code is running under py.test. Is there a function to call or an environment variable that I can test that will tell me if I'm running under a py.test session? What's the best way to handle this?
A simpler solution I came to:
import sys
if "pytest" in sys.modules:
...
Pytest runner will always load the pytest module, making it available in sys.modules.
Of course, this solution only works if the code you're trying to test does not use pytest itself.
There's also another way documented in the manual:
https://docs.pytest.org/en/latest/example/simple.html#pytest-current-test-environment-variable
Pytest will set the following environment variable PYTEST_CURRENT_TEST.
Checking the existence of said variable should reliably allow one to detect if code is being executed from within the umbrella of pytest.
import os
if "PYTEST_CURRENT_TEST" in os.environ:
# We are running under pytest, act accordingly...
Note
This method works only when an actual test is being run.
This detection will not work when modules are imported during pytest collection.
A solution came from RTFM, although not in an obvious place. The manual also had an error in code, corrected below.
Detect if running from within a pytest run
Usually it is a bad idea to make application code behave differently
if called from a test. But if you absolutely must find out if your
application code is running from a test you can do something like
this:
# content of conftest.py
def pytest_configure(config):
import sys
sys._called_from_test = True
def pytest_unconfigure(config):
import sys # This was missing from the manual
del sys._called_from_test
and then check for the sys._called_from_test flag:
if hasattr(sys, '_called_from_test'):
# called from within a test run
else:
# called "normally"
accordingly in your application. It’s also a good idea to use your own
application module rather than sys for handling flag.
Working with pytest==4.3.1 the methods above failed, so I just went old school and checked with:
script_name = os.path.basename(sys.argv[0])
if script_name in ['pytest', 'py.test']:
print('Running with pytest!')
While the hack explained in the other answer (http://pytest.org/latest/example/simple.html#detect-if-running-from-within-a-pytest-run) does indeed work, you could probably design the code in such a way you would not need to do this.
If you design the code to take the database to connect to as an argument somehow, via a connection or something else, then you can simply inject a different argument when you're running the tests then when the application drives this. Your code will end up with less global state and more modulare and reusable. So to me it sounds like an example where testing drives you to design the code better.
This could be done by setting an environment variable inside the testing code. For example, given a project
conftest.py
mypkg/
__init__.py
app.py
tests/
test_app.py
In test_app.py you can add
import os
os.environ['PYTEST_RUNNING'] = 'true'
And then you can check inside app.py:
import os
if os.environ.get('PYTEST_RUNNING', '') == 'true':
print('pytest is running')
after watching a couple of presentations about django testing, I want to code my own TestRunner in order to skip django tests, and create better packages structures for my tests.
The problem is that we've changed the project structure and the test runner can't find the right path to do the tests discovery. This is how my project looks like:
project/
-src/
- project_name/
- apps/
- test/ # Not a good name, i know, will change it
- some_app/
- test_models.py
- manage.py
- development.db
Now, in order to test test_models.py I want to do this:
$ cd project/src/
$ python manage.py test some_app.test_models
The problem is that the test runner can't find that package (some_app) and module (test_models.py). It changes if I hardcode the name in the test runner, but i don't like to do it. Here's what I do to make it work.
test_labels = ["%s.%s" % ("project_name.test", l)
for l in test_labels
if not l.startswith("project_name.test")]
So, if you do
$ python manage.py test some_app.test_models
It will be rewritten to:
$ python manage.py test project_name.test.some_app.test_models
And that works fine.
I tried doing sys.path.append("(...)/project_name/test) but doesn't work neither.
This is the code of my TestRunner:
class DiscoveryDjangoTestSuiteRunner(DjangoTestSuiteRunner):
"""A test suite runner that uses unittest2 test discovery.
It's better than the default django test runner, becouse it
doesn't run Django tests and let you put your tests in different
packages, modules and classes.
To test everything in there:
$ ./manage.py test
To test a single package/module:
$ ./manage.py test package
$ ./manage.py test package.module
To test a single class:
$ ./manage.py test package.module.ClassName
"""
def build_suite(self, test_labels, extra_tests=None, **kwargs):
suite = None
discovery_root = settings.TEST_DISCOVERY_ROOT
if test_labels:
# This is where I append the path
suite = defaultTestLoader.loadTestsFromNames(test_labels)
# if single named module has no tests, do discovery within it
if not suite.countTestCases() and len(test_labels) == 1:
suite = None
discovery_root = import_module(test_labels[0]).__path__[0]
if suite is None:
suite = defaultTestLoader.discover(
discovery_root,
top_level_dir=settings.BASE_PATH,
)
if extra_tests:
for test in extra_tests:
suite.addTest(test)
return reorder_suite(suite, (TestCase,))
Your Python import hierarchy is rooted at project/src. Thus, the correct Python import path for your test_models module is project_name.test.some_app.test_models, so that's what I would expect to pass in as a test label.
But you don't like typing the project_name.test prefix every time you want to run a specific test module, since all your tests will be located there. That's fine: you're choosing to introduce some implicit non-obvious behavior in exchange for some convenience. You definitely should not add anything to sys.path in order to achieve this: the key to Python import sanity is having your import hierarchy for a given codebase rooted in one and exactly one place; overlapping sys.path entries will cause problems like doubled imports of the same module under different names.
Really all you want is a UI convenience, and it looks to me like the test-label-munging code you show is the obvious way to implement that convenience. You don't like having the project_name.test prefix hardcoded, but it's going to have to be hardcoded somewhere: there's no way the test runner is going to magically figure out that you want to prepend test labels with project_name.test. If you want your TestRunner to be more generic, you can pull it out into a setting like BASE_TEST_MODULE or some such and prepend the value of that setting to each test label.
Before you continue investing more time into your custom TestRunner, I would definitely recommend that you take a look at django-nose.
The custom test runner provided by django-nose implements nose's test runner which is extremely flexible and provides a lot of options for running your tests. It seamlessly overrides the default test management command and allows you to configure default test options in your project's settings module.
I'm really recommending it for several reasons:
The options for the test command are fully documented (take look at the output)
nose provides a lot of approaches for test discovery
Chances are your colleagues are already seasoned nose users
You didn't have to write the TestRunner class yourself