Develop Reference

can I specify which test files to run in .coveragerc?

My file structure is below,
Logic
--packageA
----file1
----file2
--packageB
----file3
----file4
--tests
----testfile1
----testfile2
----testfile3
--.coveragerc
I'd like to include packageA only for tests, and run testfile1 and testfile2 to measure the coverage.
testfile3 is for packageB.
So I wrote my .coveragerc file,
[run]
branch = True
include = /packageA/*
omit = *tests*
and, when I run coverage with command coverage run --rcfile=../.coveragerc -m pytest in directory tests, it tries running testfile3 as well.
How can I run testfile1 and testfile2 with .coveragerc configuration?

You can use the [run] command_line option to set the command line to use when you run coverage run. But coverage isn't trying to be a general runner. You might want a shell script, a Makefile, or a tox.ini file instead.
Are you typing the coverage run --rcfile=../.coveragerc -m pytest command by hand?

How to minimize repetition in tox file

Goal: Successfully execute specific tox commands, and have it run for "just" that specific matched command.
Example: tox -e py35-integration
tox should run only for py35-integration and not including the default or standalone py35 definition.
I have tried two different approaches, which as far as I understand are the two ways to go about trying to do what I'm trying to do.
note the flake8 command is to easily isolate between different commands and indicate to me what is running. It is not an indication of the command I'm really trying to run.
Furthermore, ini files are only showing relevant parts.
First approach
[tox]
envlist = {py27,py35}, {py27,py35}-integration
[testenv]
commands =
py27: python -m testtools.run discover
py35: python -m testtools.run discover
py27-integration: flake8 {posargs}
py35-integration: flake8 {posargs}
With this approach, the understanding here is that I want to have tox -e py27-integration run without also running what is defined for the py27 command. This is not what is happening. Instead, it will run both py27 and py27-integration.
Second approach
[tox]
envlist = {py27,py35}, {py27,py35}-integration
[testenv]
commands =
python -m testtools.run discover
[testenv:integration]
commands =
flake8 {posargs}
Now, here I am explicitly isolating for a "sub" environment with its own command to run for "integration".
However, unfortunately, I'm met with the exact same behaviour of all matched patterns of "py27" being executed.
I'm trying to avoid repeating testenv structures as: [testenv:py27-integration] and [testenv:py35-integration], which contain the exact same definitions (goal is to minimize repetition).
I would love to know if there is a way I can achieve what I am trying to do.
I do not want to venture down doing something like p27-integration as an alternative naming scheme, since our CI pipelines have templates expecting certain name structures, and these names are also idiomatic to tox, in that py27 for example is understood to install a 2.7 virtual environment.

Updated
[tox]
minversion = 3.15
envlist = {py27,py35}, {py27,py35}-integration
[testenv]
commands =
python -m testtools.run discover
[testenv:py{27,35}-integration]
commands =
flake8 {posargs}

Conditional Commands in tox? (tox, travis-ci, and coveralls)

tl;dr:
I'm setting up CI for a project of mine, hosted on github, using tox and travis-ci. At the end of the build, I run converalls to push the coverage reports to coveralls.io. I would like to make this command 'conditional' - for execution only when the tests are run on travis; not when they are run on my local machine. Is there a way to make this happen?
The details:
The package I'm trying to test is a python package. I'm using / planning to use the following 'infrastructure' to set up the tests :
The tests themselves are of the py.test variety.
The CI scripting, so to speak, is from tox. This lets me run the tests locally, which is rather important to me. I don't want to have to push to github every time I need a test run. I also use numpy and matplotlib in my package, so running an inane number of test cycles on travis-ci seems overly wasteful to me. As such, ditching tox and simply using .travis.yml alone is not an option.
The CI server is travis-ci
The relevant test scripts look something like this :
.travis.yml
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
script:
- tox -e $TOX_ENV
tox.ini
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
coveralls
This file lets me run the tests locally. However, due to the final coveralls call, the test fails in principle, with :
py27 runtests: commands[1] | coveralls
You have to provide either repo_token in .coveralls.yml, or launch via Travis
ERROR: InvocationError: ...coveralls'
This is an expected error. The passenv bit sends along the necessary information from travis to be able to write to coveralls, and without travis there to provide this information, the command should fail. I don't want this to push the results to coveralls.io, either. I'd like to have coveralls run only if the test is occuring on travis-ci. Is there any way in which I can have this command run conditionally, or set up a build configuration which achieves the same effect?
I've already tried moving the coveralls portion into .travis.yml, but when that is executed coveralls seems to be unable to locate the appropriate .coverage file to send over. I made various attempts in this direction, none of which resulted in a successful submission to coveralls.io except the combination listed above. The following was what I would have hoped would work, given that when I run tox locally I do end up with a .coverage file where I'd expect it - in the root folder of my source tree.
No submission to coveralls.io
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
- pip install python-coveralls
script:
- tox -e $TOX_ENV
after_success:
- coveralls

An alternative solution would be to prefix the coveralls command with a dash (-) to tell tox to ignore its exit code as explained in the documentation. This way even failures from coveralls will be ignored and tox will consider the test execution as successful when executed locally.
Using the example configuration above, it would be as follows:
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
- coveralls

I have a similar setup with Travis, tox and coveralls. My idea was to only execute coveralls if the TRAVIS environment variable is set. However, it seems this is not so easy to do as tox has trouble parsing commands with quotes and ampersands. Additionally, this confused Travis me a lot.
Eventually I wrote a simple python script run_coveralls.py:
#!/bin/env/python
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
In tox.ini, replace your coveralls command with python {toxinidir}/run_coveralls.py.

I am using a environmental variable to run additional commands.
tox.ini
commands =
coverage run runtests.py
{env:POST_COMMAND:python --version}
.travis.yml
language: python
python:
- "3.6"
install: pip install tox-travis
script: tox
env:
- POST_COMMAND=codecov -e TOX_ENV
Now in my local setup, it print the python version. When run from Travis it runs codecov.

Alternative solution if you use a Makefile and dont want a new py file:
define COVERALL_PYSCRIPT
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
print("Not in Travis CI, skipping coveralls")
endef
export COVERALL_PYSCRIPT
coveralls: ## runs coveralls if TRAVIS in env
#python -c "$$COVERALL_PYSCRIPT"
In tox.ini add make coveralls to commands

How to convince python tox to run tests only for the available python interpreters?

I am using python tox to run python unittest for several versions of python, but these python interpreters are not all available on all machines or platforms where I'm running tox.
How can I configure tox so it will run tests only when python interpretors are available.
Example of tox.ini:
[tox]
envlist=py25,py27
[testenv]
...
[testenv:py25]
...
The big problem is that I do want to have a list of python environments which is auto-detected.

As of Tox version 1.7.2, you can pass the --skip-missing-interpreters flag to achieve this behavior. You can also set skip_missing_interpreters=true in your tox.ini file. More info here.
[tox]
envlist =
py24, py25, py26, py27, py30, py31, py32, py33, py34, jython, pypy, pypy3
skip_missing_interpreters =
true

First if you don't have tox : pip install tox.
Use this command : tox --skip-missing-interpreters , it skips for the compilers which are not available locally and just runs for the available versions of python

tox will display an Error if an interpreter cannot be found. Question is up if there should be a "SKIPPED" state and making tox return a "0" success result. This should probably be explicitely enabled via a command line option. If you agree, file an issue at http://bitbucket.org/hpk42/tox .

How do I tell tox not to collect tests from dependencies?

By default tox will collect the test from your dependencies too and I want it to collect only the ones from my package.
How can I do this?

Tox is a tool which creates a new virtualenv for each python version you have configured, installs the module your running and then runs a user-specified command to run the tests. It doesn't actually collect the tests to run. That's up to whichever testing tool you're using: py.test, nose, etc. To do that with tox, you'll edit/create a tox.ini to use the correct command that limits the collection of tests to whatever you want.
With nose:
[tox]
envlist = py26,py27
[testenv]
deps=nose
commands=nosetests test.module
With py.test:
[tox]
envlist = py26,py27
[testenv]
deps=pytest
commands=py.test test.module