Develop Reference

Cobertura coverage report not showing in MR's diff

I have recently added the cobertura coverage report to my repository, but it still does not show the coverage in an MR's diff.
Here is the job of my .gitlab-ci.yml that generates the coverage report:
coverage-report:
stage: coverage
script:
- tox -e coverage-report
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
name: "coverage"
paths:
- public/coverage
expire_in: 1 week
reports:
cobertura: public/coverage/coverage.xml
expose_as: "coverage"
And here is my tox.ini:
[tox]
envlist =
coverage-report
minversion = 3.4
[testenv:coverage-report]
basepython = python2.7-32
skip_install = True
deps =
coverage
commands =
coverage run -m pytest -s -vv -x --junitxml=public/test-report.xml tests/
coverage report
coverage html
coverage xml
I am pretty sure everything goes well with the report because not only does its XML exist under public/coverage (which I can see through the published artifacts), but the coverage % summary also shows up in the job and MR. But the coverage still does not show up in the MR's diff. I also tried opening the Network tab of my browser and look for the merge_requests/26/coverage_reports.json HTTP request, and that is coming up empty (more specifically, the response is {"files":{}}), which I do not think is supposed to be happening.
I am using Python 2.7-32 and Coverage.py to get the report. My GitLab is self-hosted with version 14.9.5-ee. Here is a link to download my coverage.xml. It is not the complete coverage, but it shows 2 files which show up in the MR's diff but have no coverage information.

Give coverage_report in this form.
artifacts:
expire_in: 1 week
paths:
- public/coverage
reports:
coverage_report:
coverage_format: cobertura
path: public/coverage/coverage.xml
Also at times it can take some time for the coverage to be reflected in the diff. Gitlab runs it a background process when it's completed it should show. Sometimes takes a while for me.

python test coverage cannot omit env/venv folder and throwing ERROR collecting env/Lib error

I have faced this issue when I am trying to test in a Django project from docker. Based on the instructions in the tutorial, I have implemented successfully black test and isort test successfully. However, when it comes t0 Pytest I am facing the aforementioned issue despite the fact that I have specified the exclusion of env files in my setup.cfg file using omit command of pytest-cov tool. Here are is my setup for setup.cfg file:
[flake8]
max-line-length = 119
exclude = .git,*/migrations/*,*env*,*venv*,__pycache__,*/staticfiles/*,*/mediafiles/*
[coverage:run]
source = .
omit=
*apps.py,
*settings.py,
*urls.py,
*wsgi.py,
*asgi.py,
manage.py,
conftest.py,
*base.py,
*development.py,
*production.py,
*__init__.py,
*/migrations/*,
*tests/*,
*/venv/*,
[coverage:report]
show_missing = True
The only variation between the youtube tutorial and my setup is that the Tutor create his virtual env using venv however I used env name. I have tested this but to no avail.
Furthermore, pytest.ini file was set up like this:
[pytest]
DJANGO_SETTINGS_MODULE = real_estate.settings.development
python_files = tests.py test_*.py *_tests.py
Also, project.toml is set up like this:
[tool.black]
extend-exclude = '''
/(
| env
)/
'''
However when I run the test using the command:
docker compose exec api pytest -p no:warnings --cov=.
I am facing the following errors:
ERROR collecting env/Lib/site-packages/social_core/tests/backends/test_twitch.py _
ERROR env/Lib/site-packages/social_core/tests/backends/test_lyft.py
ERROR env/Lib/site-packages/social_core/tests/backends/test_mailru.py
ERROR env/Lib/site-packages/social_core/tests/backends/test_mapmyfitness.py
ERROR env/Lib/site-packages/social_core/tests/backends/test_microsoft.py
Interrupted: 127 errors during collection 127 errors in 96.11s (0:01:36)

Pytest + Coverage generates wrong coverage

I have a source code structure that looks like this
src
main.py
test
default_case
test_default.py
Then if I run coverage against the test folder
coverage run --source test -m py.test
It gives me
$ poetry run coverage report
Name Stmts Miss Cover
--------------------------------------------------
test/default/test_default.py 35 1 97%
--------------------------------------------------
TOTAL 35 1 97%
If I run it against the source (src) folder
$ poetry run coverage report
Name Stmts Miss Cover
---------------------------------
src/main.py 105 105 0%
---------------------------------
TOTAL 105 105 0%
I get no coverage.
In my test, I run main.py as a child process.
Then I switched to run main() function inside main.py and the coverage result turned out to be the same.
Where am I wrong?

After a little digging, combining the Coverage official tutorial and this github thread, the problem was solved.
cd /path/to/project_root
poetry run coverage run --omit '*virtualenvs*' -m pytest
poetry run coverage report -m
Note that I use poetry to manage my dependencies, so you should remove poetry run if you don't use poetry.

Jenkins Build not failing though coverage is below 80%

Trying to fail Jenkins build even if one of the python test files has less than 80% coverage.
Towards it, in Jenkins I'm using nosetests to run test coverage on 2 python test files. It prints results as below. Though one of them has 78% coverage, the build passes. I would want the build to fail in this case
I have added the Cobertura plugin with post build options as, Fail builds if no reports, Fail unhealthy builds, Fail unstable builds. Also set threshold as 80,0,0 for Methods, Packages, Conditionals , Classes & Files.
I tried to run so that total goes below 80 but it still fails.
+ nosetests --with-xunit --with-coverage --cover-erase --cover-package=.
Name Stmts Miss **Cover**
test_sample_script.py 5 0 **100%**
test_sample_script1_80.py 9 2 **78%**
TOTAL 14 2 **86%**
Ran 2 tests in 0.110s
OK
+ python3 -m coverage xml
[Cobertura] Publishing Cobertura coverage report...
[Cobertura] Publishing Cobertura coverage results...
[Cobertura] Cobertura coverage report found.
Finished: SUCCESS

As mentioned here : https://github.com/jenkinsci/cobertura-plugin/issues/111#issuecomment-580886792
With lineCoverageTargets: '90.0, 80.1, 50':
Report health as 100% if line coverage > 90%
Report health as 0% if line coverage < 80.1%
Mark build as unstable if line coverage < 50%

coverage.py does not cover script if py.test executes it from another directory

I got a python script which takes command line arguments, working with some files.
I'm writing succeeding tests with py.test putting this script through its paces, executing it with subprocess.call.
Now I want to analyze code coverage with coverage.py.
Coverage, when used via the pytest-cov plugin (which has subprocess-handling built-in), does not see/cover my script when it is called from a temporary testing directory created with py.test's tmpdir fixture.
Coverage does see my script when it's called in the directory it resides in (and the filename argument points to a remote path).
In both situations, my tests pass! Coverage 3.6, pytest-2.3.5, pytest-cov 1.6, all from PyPi.
Question: How can I get coverage to recognize my script even if it's executed in another directory? Is this a bug in coverage, or something which is just not possible to do? Would be surprised if the latter, after all, tmpdir is a stock mechanism of py.test...
Minimal example:
I got a script my_script.py which just echoes the contents of a file arg_file.txt supplied via command-line argument. In two different tests, this is once called in a tmpdir, and once in the script's location. Both tests pass, but the in the tmpdir test, I get no coverage information!
Test run:
~/pytest_experiment$ py.test -s
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 2 items
tests/test_in_scriptdir.py
set_up: In directory /tmp/pytest-52/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
--Contents of arg_file.txt--
.
tests/test_in_tmpdir.py
set_up: In directory /tmp/pytest-52/test_11
Running in directory /tmp/pytest-52/test_11
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--
.
================================= 2 passed in 0.06 seconds =================================
Coverage:
~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_scriptdir.py .
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name Stmts Miss Cover
-------------------------------
my_script 3 0 100%
================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name Stmts Miss Cover
---------------------------
================================= 1 passed in 0.09 seconds =================================
The files are here: https://gist.github.com/bilderbuchi/6412754
Edit: Interstingly, when running the coverage tests with -s, too, there's more curious output - coverage warns that No data was collected, when obviously it was collected, and in the tmpdir test warns that Module my_script.py was never imported.??
~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_scriptdir.py
set_up: In directory /tmp/pytest-63/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-63/test_10/arg_file.txt
--Contents of arg_file.txt--
Coverage.py warning: No data was collected.
.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name Stmts Miss Cover
-------------------------------
my_script 3 0 100%
================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items
tests/test_in_tmpdir.py
set_up: In directory /tmp/pytest-64/test_10
Running in directory /tmp/pytest-64/test_10
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: No data was collected.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name Stmts Miss Cover
---------------------------
================================= 1 passed in 0.09 seconds =================================

I encountered the same issue when calling "py.test --cov ..." from tox. I found a hint on this page: http://blog.ionelmc.ro/2014/05/25/python-packaging/ even though it does not mention this explicitly. Using "--develop" for tox will make sure that coverage data gathering is called from the same directory as coverage analysis.
This section in tox.ini made it work for me to have a test environment for coverage:
[tox]
envlist = ...,py34,cov
[testenv:cov]
# necessary to make cov find the .coverage file
# see http://blog.ionelmc.ro/2014/05/25/python-packaging/
usedevelop = true
commands = py.test --cov=<MODULE_NAME>
deps = pytest pytest-cov

This turned out to be a problem of relative paths confusing coverage when the measured script is run from another directory. Coverage result files ended up in that directory, instead of the root directory of the project.
To solve this, I stopped using pytest-cov, and used pure coverage instead. I used full paths instead of relative paths wherever relevant.
So, e.g.
define the environment variable necessary to enable subprocess coverage via export COVERAGE_PROCESS_START=/full/path/to/.coveragerc.
In the .coveragerc, the coverage result file is specified via
[run]
data_file = /full/path/to/.coverage
and any --source and --include options should use full paths, too.
Then it was possible to get correct coverage measurement.

Another option with tox is to set the PYTHONPATH in tox.ini:
[testenv]
setenv =
PYTHONPATH = {toxinidir}
commands =
pytest --cov=<your package>
- codecov

according to this blog:
https://thomas-cokelaer.info/blog/2017/01/pytest-cov-collects-no-data-on-travis/
You should add all the __init__.py files in tests folder! This solution works for me.