Develop Reference

pytest-cov options for code coverage from external libraries

Currently I'm trying to implement an automation testing tool for python projects. Is it possible to collect the code coverage from external libraries using pytest-cov module? As far as I know only the coverage module will report the code coverage from external libraries!
Example:
import random
def test_rand():
assert random.randint(0,10) == 5
Using the command coverage run -m --pylib pytest file.py::test_rand we can get the code coverage from external libraries (e.g. random module in our case).
Is it possible to do the same thing using pytest-cov instead?

By default pytest-cov will report coverage for all libraries, including external.
If you run pytest --cov against your code it will produce many lines of coverage including py, pytest, importlib, etc.
To limit the scope of the coverage, i.e. you only want to inspect coverage for random, just pass the module names to the cov option e.g. pytest --cov=random. The coverage report then only considers the named modules. You can also pass multiple modules by specifying multiple cov values, e.g. pytest --cov=random --cov=pytest
Here's an example running your test to produce coverage only against random
$ pytest --cov=random
====== test session starts ======
platform linux -- Python 3.6.12, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
plugins: cov-2.12.1
collected 1 item
test_something.py F
[100%]
=========== FAILURES ============
___________ test_rand ___________
def test_rand():
import random
> assert random.randint(0,10) == 5
E AssertionError: assert 0 == 5
E + where 0 = <bound method Random.randint of <random.Random object at ...>>(0, 10)
test_something.py:6: AssertionError
---------- coverage: platform linux, python 3.6.12-final-0 -----------
Name Stmts Miss Cover
/.../random.py 350 334 --
TOTAL 350 334 5%

Python coverage report covering only test file

I’m pretty new to contributing to open source projects and am trying to get some coverage reports so I can find out what needs more / better testing. However, I am having trouble getting the full coverage of a test. This is for pytorch
For example, lets say I want to get the coverage report of test_indexing_py.
I run the command:
pytest test_indexing.py --cov=../ --cov-report=html
Resulting in this:
================================================= test session starts =================================================
platform win32 -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
rootdir: C:\Projects\pytorch
plugins: hypothesis-5.4.1, arraydiff-0.3, cov-2.8.1, doctestplus-0.4.0, openfiles-0.4.0, remotedata-0.3.2
collected 62 items
test_indexing.py ............................s................................. [100%]
----------- coverage: platform win32, python 3.7.4-final-0 -----------
Coverage HTML written to dir htmlcov
=========================================== 61 passed, 1 skipped in 50.43s ============================================
Ok, looks like the tests ran. Now when I check the html coverage report, I only get the coverage for the test file and not for the classes tested (the tests are ordered by coverage percentage).
As you can see, I am getting coverage for only test_indexing.py. How do I get the full coverage report including the classes tested?
Any guidance will be greatly appreciated.

I think its because you are asking to check the coverage from the test running directory, ie where test_indexing.py is.
A better approach would be like running the test from the root directory itself, rather than test directory, it has several advantages like the configuration file reading and all.
And regarding your question, try running the test from the root directory and try
pytest path/to/test/ --cov --cov-report=html

Generating allure report using pytest

I am using py test allure adaptor and trying to generate input data required for allure report. But I am not able to generate any XML's. When I execute the py file using py.test sample.py, it did create pycache dir. Then I executed "allure generate -v 1.3.9 C:\allurereports" (This is the dir where I had the sample.py). It did create an allure html report but no of test cases was 0. No details were present.
The sample.py(it is same as given in the example)
import allure
#allure.feature('Feature1')
#allure.story('Story1')
def test_minor():
assert False
#allure.feature('Feature2')
#allure.story('Story2', 'Story3')
#allure.story('Story4')
class TestBar:
# will have 'Feature2 and Story2 and Story3 and Story4'
def test_bar(self):
pass
Here's the py.test command used:
py.test sample.py --allure_features=feature1,feature2
Can anybody help me how to generate an allure report from the file? What are the commands to execute?

Lavanya.
I'll try to explain the sequence you must to perform to generate allure report of autotest.
Install pip. Download get-pip.py and perform python get-pip.py.
Install pytest and pytest-allure-adaptor via pip. Perform python -m pip install pytest allure-pytest
Generate autotest allure xml report. Perform python -m pytest sample.py --alluredir <some directory>
In <some directory> appear xml autotest report, which contain results of sample.py tests. Let's make beauty html report via an allure-cli tool.
Install allure-cli. Download last version of allure-cli. allure-cli requires java. allure-cli doesn't require installation, just unpack and use it.
Generate html report. Find allure (allure.bat for Windows) in unpacked zip. Perform allure.bat generate -o <some directory> -v 1.4.0 <some directory>
Find index.html in <some directory> and open it via a browser.
*Note <some directory> the same for all steps

There is a very simple way to generate reports via allure:
first, install allure:
allure-pytest 2.6.0
allure-python-commons 2.6.0
Then, if you are unable to generate the reports, follow below steps:
(using pytest)
pytest test_xyz.py --alluredir=path_where_you_want_to_save_reports
allure serve report_path
If it is still showing allure is not recognized command (blah -blah), then install allure using npm plugin with below command:
npm install -g allure-commandline --save-dev
then follow step (2) again, then one server will start and you will be able to see allure reports.

This is something that I found working -
python3 -m pytest [your_test_class_name] --alluredir \Report
Then perform the below line where you saved your report -
python3 -m allure serve \Report
This will open up the allure report in your default browser.

You should specify directory with your test data (the directory which contains -testsuite.xml files), not a test directory.
You can use py.test --alluredir [path_to_report_dir] to specify it.
PS. Make sure you use right version of allure (latest pytest adapter supports only allure 1.4.*).
For more information see https://github.com/allure-framework/allure-python and https://github.com/allure-framework/allure-cli

now you must use allure-command-line instead of allure-cli to generate html-report, cause the second one is deprecated.

Why does nosetests say --with-coverage is not an option?

I am using nosetests to automatically discover and run my unittests. I would also like to have it generate coverage reports.
When I run nosetests with the following command everything works just fine
nosetests .
I looked up online that to generate the coverage, nosetests has a command line argument --with-coverage. I also double checked that this command exists using nosetests --help. However, whenever I run the following command I get the following output
nosetests --with-coverage .
Usage: nosetests [options]
nosetests: error: no such option: --with-coverage
I double checked that the coverage plugin is installed by running
nosetests --plugins
coverage shows up in the list along with a bunch of other plugins.
I also know I have coverage installed because I can manually run the coverage data collection using something along the lines of:
coverage run test.py
Am I misusing the --with-coverage option? Or is there something else I am missing?
Thanks in advance.

I never got the command line options working. I did what Janne Karila suggested and created a setup.cfg file in my projects main directory. Once I had that file I could just run nosetests with no arguments and everything would run.
One problem I had when trying to create my document was that I couldn't figure out what parameters were allowed in the config. But it turns out that any of the commands listed here https://nose.readthedocs.org/en/latest/usage.html#options can be used. Just leave off the double dashes before the command.
For reference my current config file is
[nosetests]
verbosity=1
detailed-errors=1
with-coverage=1
cover-erase=1
cover-package=application
cover-html=1
cover-html-dir=htmlcov
where=tests
This config file says to use coverage, to erase the previous run's coverage, to only report on the files in the application package, and to output an html report to the htmlcov directory.
Hopefully this will help someone else in the future.

Your syntax is correct. It maybe an issue with your environment, double check your python environment and where your have nose and coverage installed. As a sanity check, you can quickly setup a new virtualenv, install nose, and run the command with the coverage option.

As of nose 1.3.7 - the most recent version available on Pypy - that command doesn't exist:
https://github.com/nose-devs/nose/blob/release_1.3.7/nose/plugins/cover.py
It looks like the documentation is being generated from the master branch of the project that does have those options available:
https://github.com/nose-devs/nose/blob/master/nose/plugins/cover.py
What you can do is install nose from the master branch like this:
pip install git+https://github.com/nose-devs/nose#master --upgrade
It will say it just installed 1.3.7 but that's only because the version hasn't been bumped in the project setup.py file yet: https://github.com/nose-devs/nose/blob/master/setup.py#L4
Remember you have just installed an unreleased version of nose, there may be other bugs.

PyCharm, Django: zero code coverage

PyCharm has a "Run with Coverage" action for Django test targets. This runs the tests, but shows zero test coverage (0% files, not covered in the project pane, and all red in the editor). Checking or unchecking "Use bundled coverage.py" makes no difference.
Running the same tests from the CLI gives the expected results:
$ coverage --version
Coverage.py, version 3.5.1. http://nedbatchelder.com/code/coverage
$ coverage run ./manage.py test blackbox
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.002s
OK
Destroying test database for alias 'default'...
$ coverage report
Name Stmts Miss Cover
---------------------------------------------
__init__ 0 0 100%
blackbox/__init__ 0 0 100%
blackbox/models 5 0 100%
blackbox/rules/__init__ 1 0 100%
blackbox/rules/board 62 19 69%
blackbox/tests 49 6 88%
manage 11 4 64%
settings 24 0 100%
---------------------------------------------
TOTAL 152 29 81%
What could cause this?

If you access your project via any symlink in the path, coverage display will fail.
Try to open same project through real path, and you will get correct behavior.
https://youtrack.jetbrains.com/issue/PY-17616
PS: Refreshing old question, as bug still has not been fixed.

I had a similar issue using the PyCharm bundled coverage.py
The tests were running fine, but the coverage results were not loaded, "0%" or "not covered" everywhere.
There was an error logged in the PyCharm console though, following the output of the tests, that was coverage.py related:
/System/Library/Frameworks/Python.framework/Versions/2.6/bin/python
"/Applications/PyCharm 2.5 EAP.app/helpers/run_coverage.py"
run "--omit=/Applications/PyCharm 2.5 EAP.app/helpers" bin/test
Creating test database for alias 'default'...
................................
----------------------------------------------------------------------
Ran xx tests in xxs
OK
No source for code: '/path/file.py' (<- error)
Process finished with exit code 0
My solution was to run the bundled coverage.py with the option to ignore errors: "-i".
I've edit the bundled "run_coverage.py" file, you can see its location in the console output, and add the "-i" option in the last line:
main(["xml", "-o", coverage_file + ".xml"])
to:
main(["xml", "-i", "-o", coverage_file + ".xml"])
This worked for me, the error is ignored and all the coverage data are now loaded in the UI.
If using "-i" solve the issue on your side, a better solution would be to fix the errors, but until then, you'll see coverage results.

I've also been trying to solve this issue in Ubuntu.
At the moment I tried with both the apt-get Python and the Enthought Canopy stack, with no success. In Windows however it does work (using Canopy).
I've used the following code:
# in a.py
class A(object):
def p(self, a):
return a
# in test_a.py
from unittest import TestCase, main
from a import A
class TestA(TestCase):
def test_p(self):
inst = A()
val = inst.p("a")
self.assertEqual("a", val)
if _name_ == "__main__":
main()

I had a similar issue. I ended up generating xml data using nosetests --cover-xml, but you can also generate an xml from a previous coverage.py run with coverage xml
Then that report can be conveniently loaded in PyCharm/IDEA from the Analyze -> Show Coverage Data… -> + button and selecting the xml file.