What I want to achieve is the following:
When running pytest, i want only to see a line per test, with nothing on stdout. If a test fails, then output of the failing test should be printed. This is the default behavior of pytest and it suits me well.
However, I also want the stdout and stderr of all tests to be written to a log file.
Preferably, written so that the output from different tests can be distinguished.
I couldn't find an option for doing so.
In
Pytest capture stdout of a certain test
a hook is added which does something similar (but for a subset of tests). I could adapt this, but I was wondering if there was not something already in pytest? Or a plugin?
Related
Is there a way to figure out which tests flipped their status since the last run? Status is either fail, pass or xfail.
You can output the results in JUnit XML format with the --junit-xml command line flag and compare them later in a reporting tool such as Allure. Or maybe one of the various reporting plugins can help you, such as the pytest-historic, which I haven't tried.
I am having troubles using the coverage.py module in my python script. The scenario is:
I have a script which calls upon test scripts and executes them as this was the only way to call multiple tests within my environment that works. I would like to get coverage reports generated for each of these tests that are called upon and executed within my script. For example:
Cov = coverage.Coverage()
Cov.start()
Subprocess.call("bazel run :test", shell=true) #or "bazel test ..."
Cov.stop()
Cov.html_report(directory=my/directory, include=test/name)
However, as you probably know, this implementation does not work and returns "no data to report". How can I get it so that I can start and stop coverage for each test? Or is this approach even possible?
If any of this is unclear, I can try to explain something better.
Thanks for your help,
I am relatively new to pytest at the moment and was curious if there is a way to store the pass/fail results of the test in a variable.
Essentially what I want to do is run my full suite of tests and after the tests are run, send the name of the tests run along with the pass/fail result to a server.
I understand that pytest provides options such as -r that will output the test run with pass or fail after execution, but is there a way to store those into variables or pass those results along?
is there a way to store those into variables or pass those results along?
Pytest can natively output JUnitXML files:
To create result files which can be read by Jenkins or other Continuous integration servers, use this invocation:
pytest --junitxml=path
to create an XML file at path.
There is an available schema for this format and there appear to be several Python libraries that can parse them with varying levels of support. This one looks like a good place to start.
There are also plugins that may be able to help. For example, pytest-json:
pytest-json is a plugin for py.test that generates JSON reports for test results
I've seen this question: py.test logging messages and test results/assertions into a single file
I've also read documentation here: https://docs.pytest.org/en/latest/logging.html
Neither comes close to a satisfactory solution.
I don't need assertion results together with logs, but it's OK if they are both in the logs.
I need all the logs produced during the test, but I don't need them for the tests themselves. I need them for analyzing the test results after the tests failed / succeeded.
I need logs for both succeeding and for failing tests.
I need stdout to only contain the summary (eg. test-name PASSED). It's OK if the summary also contains the stack trace for failing tests, but it's not essential.
Essentially, I need the test to produce 3 different outputs:
HTML / JUnit XML artifact for CI.
CLI minimal output for CI log.
Extended log for testers / automation team to analyze.
I tried pytest-logs plugin. As far as I can tell, it can override some default pytest behavior by displaying all logging while the test runs. This is slightly better than default behavior, but it's still very far from what I need. From the documentation I understand that pytest-catchlog will conflict with pytest, and I don't even want to explore that option.
Question
Is this achievable by configuring pytest or should I write a plugin, or, perhaps, even a plugin won't do it, and I will have to patch pytest?
You can use --junit-xml=xml-path switch to generate junit logs. If you want the report in html format, you can use pytest-html plugin. Similarly, you can use pytest-excel plugin to generate report in excel format.
You can use tee to pipe logs to two different processes. example: pytest --junit-xml=report.xml | tee log_for_testers.log It will generate logs in stdout for CI log, report.xml for CI artifact and log_for_testers.log for team analysis.
Earlier I was using python unittest in my project, and with it came unittest.TextTestRunner and unittest.defaultTestLoader.loadTestsFromTestCase. I used them for the following reasons,
Control the execution of unittest using a wrapper function which calls the unittests's run method. I did not want the command line approach.
Read the unittest's output from the result object and upload the results to a bug tracking system which allow us to generate some complex reports on code stability.
Recently there was a decision made to switch to py.test, how can I do the above using py.test ? I don't want to parse any CLI/HTML to get the output from py.test. I also don't want to write too much code on my unit test file to do this.
Can someone help me with this ?
You can use the pytest's hook to intercept the test result reporting:
conftest.py:
import pytest
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_logreport(report):
yield
# Define when you want to report:
# when=setup/call/teardown,
# fields: .failed/.passed/.skipped
if report.when == 'call' and report.failed:
# Add to the database or an issue tracker or wherever you want.
print(report.longreprtext)
print(report.sections)
print(report.capstdout)
print(report.capstderr)
Similarly, you can intercept one of these hooks to inject your code at the needed stage (in some cases, with the try-except around yield):
pytest_runtest_protocol(item, nextitem)
pytest_runtest_setup(item)
pytest_runtest_call(item)
pytest_runtest_teardown(item, nextitem)
pytest_runtest_makereport(item, call)
pytest_runtest_logreport(report)
Read more: Writing pytest plugins
All of this can be easily done either with a tiny plugin made as a simple installable library, or as a pseudo-plugin conftest.py which just lies around in one of the directories with the tests.
It looks like pytest lets you launch from Python code instead of using the command line. It looks like you just pass the same arguments to the function call that would be on the command line.
Pytest will create resultlog format files, but the feature is deprecated. The documentation suggests using the pytest-tap plugin that produces files in the Test Anything Protocol.