I'm creating some automatic tests for my app using pytest. Every test consists of some test actions and an assertion - nothing special. However some of these tests are intentionally a bit disruptive. I would like to gather some metrics from different resources of my app and from my test environment while the tests are running, and put these metrics into a log file - I'm not interested in failing the tests based on these metrics, I just want them to understand my system better.
I'm thinking about creating a script that gathers the information that I want and creating a fixture to run it in the background of my test by using subprocess.Popen(). I have also thought about creating a function to gather the data and run it in parallel to my test code by using multiprocessing. I don't know if there are other options.
I would like to know if there is a standard, simple way to do this. I want to avoid unnecessary complexity at all costs.
Thanks!
Related
I have a large test suite that contains thousands of tests of an information extraction engine. There are about five hundred inputs, and I have to extract 10-90 items of information from each input.
For maximal transparency, I test each item from each input separately, so that I can look at the pytest log and tell exactly which tests flip after a code change. But since it's inefficient to run extraction within each test dozens of times, I organized my tests so that the extraction is done only once for each input, a test-global variable is set and then each test simply checks one aspect of the result.
It seems that this set-up prevents any speedup gains when using python-xdist. Apparently, pytest will still initialize all 500 test files in one thread and then have individual workers execute the test methods - but the actual tests take no time, so I save nothing!
Is there a simple way to instruct pytest-xdist to distribute the test files rather than the test methods to the workers, so that I can parallelize the 500 calls to the engine?
According to official docs - https://pytest-xdist.readthedocs.io/en/latest/distribution.html, you can use one of the following:
custom groups (mark all related tests with #pytest.mark.xdist_group), and run cli with flag --dist loadgroup. It can help you run entire groups (in your cases - all methods of class) in isolated worker
if you use OOP style (Test classes with test methods or modules with tests) - use pytest CLI flag --dist loadscope , this approach will make groups automatically (modules, or classes) and guarantees that all tests in a group run in the same process.
if __name__ == '__main__':
if is_running_under_teamcity():
runner = TeamcityTestRunner()
else:
runner = HTMLTestRunner.HTMLTestRunner(
stream=outfile,
title='Test Report',
description='This is an example.'
)
unittest.main(testRunner=runner)
I am currently running some tests using unittest module in python this is my current code above. I am deploying this test setup on Teamcity, the first module allows me to convert the output into teamcity-messages and the second creates a html report of the results. Is there a way I can run both of these runners while only running one set of tests? The only option I can see at the minuete is to either try and combine both these modules into a hybrid or using another testing module that Teamcity supports. However I would like to keep the dependancies as low as possible
Any ideas would be great :)
Any ideas would be great :)
Looks like you'll have to handroll it, looking at the code TeamcityTestRunner is a pretty simple extension of the standard TextTestRunner, however HTMLTestRunner is a way more complex beast.
Sadly this is one area of the stdlib which is really badly architected: one could expect the test runner to be concerned solely with discovering and running tests, however it's also tasked with part of the test reporting rather than have an entirely separate test reporter (this test reporting is furthermore a split responsability with the test result, which shouldn't be part of that one's job description either).
Frankly if you don't have any further customisation I'd suggest just using pytest as your test runner instead of unittest with a custom runner:
it should be able to run unittest tests fine
IME it has better separation of concerns and pluggability so having multiple reporters / formatters should work out of the box
pytest-html certainly has no issue generating its reports without affecting the normal text output
according to the readme teamcity gets automatically enabled and used for pytest
so I'd assume generating html reports during your teamcity builds would work fine (to test)
and you can eventually migrate to using pytest tests (which are so much better it's not even funny)
Imagine that I have test/unit/... which are safe to run in parallel and test/functional/... which cannot be run in parallel yet.
Is there an easy way to convince pytest to run the functional ones sequentially? Consider that we are talking about a big number of tests so altering each test function/method would be very noisy.
At this moment we run tests with marker filters so mainly we run them separated. Still, I am looking for a solution for removing the need to run them separated.
You can implement your own scheduler that acts like load or loadscope, depending on what module the test is defined in. Example:
from xdist.scheduler.loadscope import LoadScopeScheduling
class MyScheduler(LoadScopeScheduling):
def _split_scope(self, nodeid):
if 'test/functional' in nodeid:
return 'functional-tests'
return nodeid
def pytest_xdist_make_scheduler(config, log):
return MyScheduler(config, log)
All tests under test/functional will be grouped under the same node functional-tests (and thus run in the same worker), the rest will be run in parallel as usual.
I have tests which have a huge variance in their runtime. Most will take much less than a second, some maybe a few seconds, some of them could take up to minutes.
Can I somehow specify that in my Nosetests?
In the end, I want to be able to run only a subset of my tests which take e.g. less than 1 second (via my specified expected runtime estimate).
Have a look at this write up about attribute plugin for nose tests, where you can manually tag tests as #attr('slow') and #attr('fast'). You can tun nosetests -a '!slow' afterward to run your tests quickly.
It would be great if you can do it automatically, but I'm afraid that you would have to write additional code to do it on the fly. If you are into rapid development, I would run the nose with xunit xml output enabled (which tracks the runtime of each test). Your test module can dynamically read in your xml output file from previous runs and set attribute settings for tests accordingly to filter out quick tests. This way you do not have to do it manually, alas with more work (and you have to run all tests at least once).
We are testing a Django applications with a black box (functional integration) testing approach, where a client performs tests with REST API calls to the Django application. The client is running on a different VM, so we can not use the typical coverage.py (I think).
Is there a way to compute the coverage of these black box tests? Can I somehow instruct Django to start and stop in test coverage mode and then report test coverage?
The coverage for functional integration tests are really a different layer of abstraction than unit test coverage which covers lines of code executed. You likely care more about coverage of use-cases in a true black-box test.
But if you are looking for code coverage anyways (and there are certainly reasons why you might want to), it looks like you should be able to use coverage.py if you have access to the server to set up test scenarios. You will need to implement a way to end the django process to allow coverage.py to write the coverage report.
From:
https://coverage.readthedocs.io/en/coverage-4.3.4/howitworks.html#execution
"At the end of execution, coverage.py writes the data it collected to
a data file"
This indicates that the python processes must come to completion naturally. Killing the process manually would also take out the coverage.py wrapper preventing the write.
Some ideas to end django: stop django command using sys.exit()
See:
https://docs.djangoproject.com/en/1.10/topics/testing/advanced/#integration-with-coverage-py