Need help!
I have a job on Gitlab ci, that runs tests and reruns failed ones. If there are no failed tests, job fails with exit code 5, that means that there are no tests for running. I found out that there is plugin "pytest-custom_exit_code", but I don't know how to correctly use it.
I need just to add command 'pytest --suppress-no-test-exit-code' to my runner.sh?
It looks like this now:
#!/bin/sh
/usr/local/bin/pytest -m test
/usr/local/bin/pytest -m --last-failed --last-failed-no-failures none test
Assumption here is that plugin is installed first using
pip install pytest-custom_exit_code
command like option pytest --suppress-no-test-exit-code should work after that.
If configuration file like .pytest.ini is used , following lines should be added in it
[pytest]
addopts = --suppress-no-test-exit-code
Related
My file structure is below,
Logic
--packageA
----file1
----file2
--packageB
----file3
----file4
--tests
----testfile1
----testfile2
----testfile3
--.coveragerc
I'd like to include packageA only for tests, and run testfile1 and testfile2 to measure the coverage.
testfile3 is for packageB.
So I wrote my .coveragerc file,
[run]
branch = True
include = /packageA/*
omit = *tests*
and, when I run coverage with command coverage run --rcfile=../.coveragerc -m pytest in directory tests, it tries running testfile3 as well.
How can I run testfile1 and testfile2 with .coveragerc configuration?
You can use the [run] command_line option to set the command line to use when you run coverage run. But coverage isn't trying to be a general runner. You might want a shell script, a Makefile, or a tox.ini file instead.
Are you typing the coverage run --rcfile=../.coveragerc -m pytest command by hand?
I'm running my builds on my CI (bamboo) via tox on docker
my tox.ini look like this
[tox]
envlist = py27,py35
[testenv]
deps=-rrequirements.txt
commands=pytest
i'm running the tests like so
tox --recreate -vv -i $myindexserver
Testing the setup locally works (inside docker)
py27: commands succeeded
py35: commands succeeded
congratulations :)
But while running the same thing on the CI instance failes with
___________________________________ summary_________________________________
py27: commands succeeded
ERROR: py35: InterpreterNotFound: python3.5
inside the docker, running which python3 and which python3.5 succeeds
Has anyone faced similar issue?
Turns out that the docker container versions used by my local and the one used by the CI were different.
I'm keeping the answer here in the hopes that someone else finds this useful and possibly save the many hours of debugging that I had to waste.
do a docker images to find the tag that you're using locally, and check it against the version running inside your CI.
tl;dr:
I'm setting up CI for a project of mine, hosted on github, using tox and travis-ci. At the end of the build, I run converalls to push the coverage reports to coveralls.io. I would like to make this command 'conditional' - for execution only when the tests are run on travis; not when they are run on my local machine. Is there a way to make this happen?
The details:
The package I'm trying to test is a python package. I'm using / planning to use the following 'infrastructure' to set up the tests :
The tests themselves are of the py.test variety.
The CI scripting, so to speak, is from tox. This lets me run the tests locally, which is rather important to me. I don't want to have to push to github every time I need a test run. I also use numpy and matplotlib in my package, so running an inane number of test cycles on travis-ci seems overly wasteful to me. As such, ditching tox and simply using .travis.yml alone is not an option.
The CI server is travis-ci
The relevant test scripts look something like this :
.travis.yml
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
script:
- tox -e $TOX_ENV
tox.ini
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
coveralls
This file lets me run the tests locally. However, due to the final coveralls call, the test fails in principle, with :
py27 runtests: commands[1] | coveralls
You have to provide either repo_token in .coveralls.yml, or launch via Travis
ERROR: InvocationError: ...coveralls'
This is an expected error. The passenv bit sends along the necessary information from travis to be able to write to coveralls, and without travis there to provide this information, the command should fail. I don't want this to push the results to coveralls.io, either. I'd like to have coveralls run only if the test is occuring on travis-ci. Is there any way in which I can have this command run conditionally, or set up a build configuration which achieves the same effect?
I've already tried moving the coveralls portion into .travis.yml, but when that is executed coveralls seems to be unable to locate the appropriate .coverage file to send over. I made various attempts in this direction, none of which resulted in a successful submission to coveralls.io except the combination listed above. The following was what I would have hoped would work, given that when I run tox locally I do end up with a .coverage file where I'd expect it - in the root folder of my source tree.
No submission to coveralls.io
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
- pip install python-coveralls
script:
- tox -e $TOX_ENV
after_success:
- coveralls
An alternative solution would be to prefix the coveralls command with a dash (-) to tell tox to ignore its exit code as explained in the documentation. This way even failures from coveralls will be ignored and tox will consider the test execution as successful when executed locally.
Using the example configuration above, it would be as follows:
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
- coveralls
I have a similar setup with Travis, tox and coveralls. My idea was to only execute coveralls if the TRAVIS environment variable is set. However, it seems this is not so easy to do as tox has trouble parsing commands with quotes and ampersands. Additionally, this confused Travis me a lot.
Eventually I wrote a simple python script run_coveralls.py:
#!/bin/env/python
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
In tox.ini, replace your coveralls command with python {toxinidir}/run_coveralls.py.
I am using a environmental variable to run additional commands.
tox.ini
commands =
coverage run runtests.py
{env:POST_COMMAND:python --version}
.travis.yml
language: python
python:
- "3.6"
install: pip install tox-travis
script: tox
env:
- POST_COMMAND=codecov -e TOX_ENV
Now in my local setup, it print the python version. When run from Travis it runs codecov.
Alternative solution if you use a Makefile and dont want a new py file:
define COVERALL_PYSCRIPT
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
print("Not in Travis CI, skipping coveralls")
endef
export COVERALL_PYSCRIPT
coveralls: ## runs coveralls if TRAVIS in env
#python -c "$$COVERALL_PYSCRIPT"
In tox.ini add make coveralls to commands
Is it possible to configure pycharm / intellij idea to run tox tests? I want to test my code against different python versions in separated py environments. I was trying to configure it, but so far I only managed to configure single py.test support.
See this PyCharm issue: PY-9727
Credit goes to Andrey Vlasovskikh at PyCharm
As a workaround, you can create a Python file in your project that imports and launches Tox:
import tox
tox.cmdline()
and then run this file via your project interpreter using its context menu.
You'll get hyperlinks in the console output for stack traces, but nothing more.
I'm afraid it's not supported, PyCharm will use the configured interpreter to run the tests.
You are welcome to submit a feature request.
7 years passed and now you can run Tox inside PyCharm. If you use pytest for testing it will even show Test Result same way like with running test locally by PyCharm.
To run tests in pycharm I use config like this
[tox]
envlist = py3.{7,8},codestyle,flake8,lint
minversion = 3.7
[testenv]
usedevelop = true
deps =
pytest
pytest-cov
commands = pytest --cov=src
[testenv:codestyle]
deps = pycodestyle
commands = pycodestyle src tests
[testenv:flake8]
deps = flake8
commands = flake8 src tests
[testenv:lint]
deps = pylint
commands = pylint src tests --rcfile=.pylintrc
Here I described why it looks like this. Also, there is integration with GitHub CI to run it on every push.
When I run tests with ./manage.py test, whatever I send to the standard output through print doesn't show. When tests fail, I see an "stdout" block per failed test, so I guess Django traps it (but doesn't show it when tests pass).
Checked TEST_RUNNER in settings.py, it's using a project-specific runner that calls out to Nose. Nose has the -s option to stop it from capturing stdout, but if I run:
./manage.py test -s
manage.py captures it first and throws a "no such option" error. The help for manage.py doesn't mention this, but I found that if I run:
./manage.py test -- -s
it ignores the -s and lets me capture it on the custom runner's side, passing it to Nose without a problem.
Yeah, this issue is caused by NoseTestSuiteRunner. Adding -- -s is tricky and not the best solution.
Try to add the following lines in the settings.py:
NOSE_ARGS = ['--nocapture',
'--nologcapture',]
This solved my problems.
There are several levels of verbosity available which affects the amount of details we see:
You can try:
python manage.py test -v 2
Other levels available are:
-v 0 : Least amount of details
-v 1: Default
-v 2 : More details e.g. print statements included.
Using current versions of all the relevant packages (Django==1.11.2, django-nose==1.4.5 and nose==1.3.7) it is sufficient to add the --nocapture flag when running your tests. Thus a simple
./manage.py test --nocapture
will suffice.
Granted of course that you have
TEST_RUNNER = "django_nose.NoseTestSuiteRunner"
in your settings.py
You probably have some intermediate test runner, such as Nose, intercepting and storing stdout. Try either running the Django tests directly, or write to stderr instead.