Trying to fail Jenkins build even if one of the python test files has less than 80% coverage.
Towards it, in Jenkins I'm using nosetests to run test coverage on 2 python test files. It prints results as below. Though one of them has 78% coverage, the build passes. I would want the build to fail in this case
I have added the Cobertura plugin with post build options as, Fail builds if no reports, Fail unhealthy builds, Fail unstable builds. Also set threshold as 80,0,0 for Methods, Packages, Conditionals , Classes & Files.
I tried to run so that total goes below 80 but it still fails.
+ nosetests --with-xunit --with-coverage --cover-erase --cover-package=.
Name Stmts Miss **Cover**
test_sample_script.py 5 0 **100%**
test_sample_script1_80.py 9 2 **78%**
TOTAL 14 2 **86%**
Ran 2 tests in 0.110s
OK
+ python3 -m coverage xml
[Cobertura] Publishing Cobertura coverage report...
[Cobertura] Publishing Cobertura coverage results...
[Cobertura] Cobertura coverage report found.
Finished: SUCCESS
As mentioned here : https://github.com/jenkinsci/cobertura-plugin/issues/111#issuecomment-580886792
With lineCoverageTargets: '90.0, 80.1, 50':
Report health as 100% if line coverage > 90%
Report health as 0% if line coverage < 80.1%
Mark build as unstable if line coverage < 50%
Related
I have recently added the cobertura coverage report to my repository, but it still does not show the coverage in an MR's diff.
Here is the job of my .gitlab-ci.yml that generates the coverage report:
coverage-report:
stage: coverage
script:
- tox -e coverage-report
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
name: "coverage"
paths:
- public/coverage
expire_in: 1 week
reports:
cobertura: public/coverage/coverage.xml
expose_as: "coverage"
And here is my tox.ini:
[tox]
envlist =
coverage-report
minversion = 3.4
[testenv:coverage-report]
basepython = python2.7-32
skip_install = True
deps =
coverage
commands =
coverage run -m pytest -s -vv -x --junitxml=public/test-report.xml tests/
coverage report
coverage html
coverage xml
I am pretty sure everything goes well with the report because not only does its XML exist under public/coverage (which I can see through the published artifacts), but the coverage % summary also shows up in the job and MR. But the coverage still does not show up in the MR's diff. I also tried opening the Network tab of my browser and look for the merge_requests/26/coverage_reports.json HTTP request, and that is coming up empty (more specifically, the response is {"files":{}}), which I do not think is supposed to be happening.
I am using Python 2.7-32 and Coverage.py to get the report. My GitLab is self-hosted with version 14.9.5-ee. Here is a link to download my coverage.xml. It is not the complete coverage, but it shows 2 files which show up in the MR's diff but have no coverage information.
Give coverage_report in this form.
artifacts:
expire_in: 1 week
paths:
- public/coverage
reports:
coverage_report:
coverage_format: cobertura
path: public/coverage/coverage.xml
Also at times it can take some time for the coverage to be reflected in the diff. Gitlab runs it a background process when it's completed it should show. Sometimes takes a while for me.
I have a source code structure that looks like this
src
main.py
test
default_case
test_default.py
Then if I run coverage against the test folder
coverage run --source test -m py.test
It gives me
$ poetry run coverage report
Name Stmts Miss Cover
--------------------------------------------------
test/default/test_default.py 35 1 97%
--------------------------------------------------
TOTAL 35 1 97%
If I run it against the source (src) folder
$ poetry run coverage report
Name Stmts Miss Cover
---------------------------------
src/main.py 105 105 0%
---------------------------------
TOTAL 105 105 0%
I get no coverage.
In my test, I run main.py as a child process.
Then I switched to run main() function inside main.py and the coverage result turned out to be the same.
Where am I wrong?
After a little digging, combining the Coverage official tutorial and this github thread, the problem was solved.
cd /path/to/project_root
poetry run coverage run --omit '*virtualenvs*' -m pytest
poetry run coverage report -m
Note that I use poetry to manage my dependencies, so you should remove poetry run if you don't use poetry.
I have come up with this issue, where by following the instructions found in the coverage.py page I end up getting the report on the test files themselves instead of the application files. I am referring to a Flask application, using unittest as a testing framework, and coverage.py for coverage reports.
The project's folder tree can be represented as follows:
+ root
+
++++ Server
+
++++ server.py
++++ Models
+
++++ other_files
+
++++ Tests
+
++++ test_file.py
If I use the command:
coverage run -m unittest discover && coverage report
I get:
Name Stmts Miss Cover
---------------------------------------------
Models\__init__.py 0 0 100%
test_1.py 76 16 79%
test_2.py 67 3 96%
---------------------------------------------
TOTAL 143 19 87%
Where the output I get doesn't refer to the ApplicationFile(s).py.
If instead I use:
coverage run --source ./../ApplicationFile.py -m unittest discover && coverage report
I get:
Coverage.py warning: Module ./../ApplicationFile.py was never imported. (module-not-imported)
Coverage.py warning: No data was collected. (no-data-collected)
No data to report.
Does someone have an idea of where the issue could be?
I think --source is expecting a directory rather than a file. Try instead coverage run --source ./../ -m unittest discover && coverage report
I have a simple test:
class ModelTests(TestCase):
def test_method(self):
instance = Activity(title="Test")
self.assertEqual(instance.get_approved_member_count(), 0)
My problem is that coverage still shows get_approved_member_count line as NOT tested:
How do I satisfy the above for coverage?
To run the tests I'm using Django Nose with Coverage:
TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
NOSE_ARGS = [
'--with-coverage',
'--cover-html',
'--cover-package=apps.users,apps.activities',
]
Console:
python manage.py test
/Users/user/Documents/workspace/api/env/lib/python3.4/importlib/_bootstrap.py:321: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
return f(*args, **kwds)
/Users/user/Documents/workspace/api/env/lib/python3.4/importlib/_bootstrap.py:321: RemovedInDjango19Warning: The utilities in django.db.models.loading are deprecated in favor of the new application loading system.
return f(*args, **kwds)
nosetests --with-coverage --cover-html --cover-package=apps.users,apps.activities --verbosity=1
Name Stmts Miss Cover Missing
---------------------------------------------------------------------------------------
apps.activities 0 0 100%
apps.activities.admin 8 8 0% 1-14
activities.migrations 0 0 100%
activities.migrations.0001_initial 9 0 100%
apps.activities.urls 8 0 100%
etc etc etc
---------------------------------------------------------------------------------------
TOTAL 670 232 65%
----------------------------------------------------------------------
Ran 79 tests in 17.101s
The coverage report shows that the method is being called (line 80 is green). But it also shows that it was never defined (line 75 is red).
This is a classic problem of starting coverage too late. The simplest way to fix this is to use coverage to run your test runner, instead of using the test runner to run coverage:
$ coverage run -m nose --verbosity=1
UPDATED: to use with your original command:
$ coverage run manage.py test
but you'd want to uninstall the nose coverage plugin first.
I have my python app structured as follows:
proj
- comp1
- comp2
tests
- comp1
- comp2
other
- contains some python code
I am running nosetests as following:
nosetests --with-coverage --cover-package=proj --exclude-dir=other -v tests
However, in the coverage report that nosetests prints at the end I still entries from 'other'. How do I exclude 'other' from the coverage report?
Quite possible your tests or proj are referring to the other python code, and as result, it makes it into report. You could see debug coverage plugin printout with:
nosetests --with-coverage --cover-package=proj -vv tests -l nose.plugins.cover