Develop Reference

python tox: how to use a different python envlist in environment

Here is my tox.ini:
[tox]
envlist = py27,py35
[testenv]
deps =
Flask
connexion
pytest
coverage
pytest-cov
requests
six
commands=pytest --junitxml xunit-reports/xunit-result-XXX.xml --cov {envsitepackagesdir} --cov-report=xml
[testenv:local]
#HOW DO I SPECIFY A NEW LIST OF PYENV LIKE 31,36 IN HERE????
commands=
pytest --cov {envsitepackagesdir}/XXX --cov-report html
When I run tox it runs in py27 and py35. I want tox -e local to run in a different set of multiple python environments. I can't figure out how to do this. How do I do this? Right now it does not even respect the intial envlist and only runs on Python 2.7.

So what you want is to have two different sets of environments and run them independently.
What you have to understand first is that envlist is a list of all environments that would be run if you invoke tox without the -e option.
The next thing you have to understand is that there is only one of these lists per tox.ini and that one is in the global [tox] section.
The other thing you have to understand is that the pyXX factors (factors are the parts of environment names that are separated by the - sign) have a special meaning for tox, because they instruct it to build an environment with a specific interpreter. They are also called "default environments" (see basic usage). If you don't ask for that factor when calling tox, then the basepython interpreter will be used to build the virtualenv (the interpreter you are invoking tox from).
so if you invoke tox -e local with a tox.ini like yours, it will execute what is defined in [tox:local] with the basepython, because you are not defining which python should be used to create the virtualenv, so it uses the same interpreter that you invoked tox with.
If you want to be able to invoke your local factor with other interpreters, independent from those other environments, the following could get you started (described in the v2 config docs):
[tox]
envlist = {py27,py35}-remote,{py31,py36}-local
[testenv]
deps =
Flask
connexion
pytest
coverage
pytest-cov
requests
six
[testenv:remote]
commands=pytest --junitxml xunit-reports/xunit-result-XXX.xml --cov {envsitepackagesdir} --cov-report=xml
[testenv:local]
commands= pytest --cov {envsitepackagesdir}/XXX --cov-report html
Check which envs this creates with:
$ tox -a
py27-remote
py35-remote
py31-local
py36-local
What envlist with the curly braces notation does, is create environment names by combining all factors with their permutations (this can have more dimensions also).
if you say tox without -e they will all run and all use the correct interpreter.
If you want to run the local envs only you will have to call it with:
$ tox -e py31-local,py36
Then only those two will run. The thing to take away here is that if you want to run a subset of all environments you have to ask for them with their full names. There is no "sub generation" or extra envlist magic. Just list the full names of the envorenments in a comma separated list and you are golden.
UPDATE
Today I learned that you can also use the generation syntax from the command line. So you could type:
$ tox -e 'py{31,36}'-local
Thank you #phd for pointing it out.

A possible solution to what you are attempting to do is to use tox -l to list all the environments, filter the ones you want, and then feed them back into tox -e.
For example, to run all environments that have "local" in the name (using bash):
tox -e $(tox -l | grep local | paste -sd "," -)
Step-by-step explanation:
tox -l lists all the environments, one on each line
grep local filters only the lines of the input which contain the word "local"
paste -sd "," - joins the lines of the input with commas

[tox]envlist is only a default — a list of environments to run when tox is invoked without option -e and without TOXENV environment variable. Once you use tox -e [tox]envlist is ignored.
You can run local environment with different python versions, but I don't know any way to run it multiple times. You have to list all environments explicitly:
tox -e py33-local,py34-local
You can shorten the command line using tox' conventions:
tox -e 'py3{3,4}'-local

Use generative envlist and factor-conditional settings.
[tox]
envlist = {py27,py31,py35,py36}-{default,local}
[testenv]
deps =
Flask
connexion
pytest
coverage
pytest-cov
requests
six
commands =
{default,local}: python --version
default: pytest --junitxml xunit-reports/xunit-result-XXX.xml --cov {envsitepackagesdir} --cov-report=xml
local: pytest --cov {envsitepackagesdir}/XXX --cov-report html
List all possible combinations of python version and factors using: tox -l
For your "local" case you'd invoke tox in the one of the following ways:
tox -e py31-local
tox -e py36-local
tox -e 'py3{1,6}'-local
Answer heavily influenced by #oliver-bestwalter's answer, but I couldn't get that to work properly for some reason.

OK I greatly appreciate and upvoted the other two answers here but what I ended up doing was different. It seemed onerous just to achieve a separate
python version and command.
What I ended up doing was just making a seperate tox.ini and calling it like tox -c tox-local.ini

How to test single file under pytest

How do you test a single file in pytest? I could only find ignore options and no "test this file only" option in the docs.
Preferably this would work on the command line instead of setup.cfg, as I would like to run different file tests in the ide. The entire suite takes too long.

simply run pytest with the path to the file
something like
pytest tests/test_file.py
Use the :: syntax to run a specific test in the test file:
pytest test_mod.py::test_func
Here test_func can be a test method or a class (e.g.: pytest test_mod.py::TestClass).
For more ways and details, see "Specifying which tests to run" in the docs.

This is pretty simple:
$ pytest -v /path/to/test_file.py
The -v flag is to increase verbosity. If you want to run a specific test within that file:
$ pytest -v /path/to/test_file.py::test_name
If you want to run test which names follow a patter you can use:
$ pytest -v -k "pattern_one or pattern_two" /path/to/test_file.py
You also have the option of marking tests, so you can use the -m flag to run a subset of marked tests.
test_file.py
def test_number_one():
"""Docstring"""
assert 1 == 1
#pytest.mark.run_these_please
def test_number_two():
"""Docstring"""
assert [1] == [1]
To run test marked with run_these_please:
$ pytest -v -m run_these_please /path/to/test_file.py

This worked for me:
python -m pytest -k some_test_file.py
This works for individual test functions too:
python -m pytest -k test_about_something

How to use multiple cores with py.test?

Looking at the py.test documentation it would seem that passing -n NUM would specify the number of cores to be used.
My experience is the following:
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n
If I try to specify it in pytest.ini (which is my preference) I get the same result.
I'm adding addopts = -n4 to the ini file, which seems to be what the docs require.
I'm using py.test 2.8, and I've tried both python 2 and python 3.

You need to install xdist
pip install pytest-xdist

Conditional Commands in tox? (tox, travis-ci, and coveralls)

tl;dr:
I'm setting up CI for a project of mine, hosted on github, using tox and travis-ci. At the end of the build, I run converalls to push the coverage reports to coveralls.io. I would like to make this command 'conditional' - for execution only when the tests are run on travis; not when they are run on my local machine. Is there a way to make this happen?
The details:
The package I'm trying to test is a python package. I'm using / planning to use the following 'infrastructure' to set up the tests :
The tests themselves are of the py.test variety.
The CI scripting, so to speak, is from tox. This lets me run the tests locally, which is rather important to me. I don't want to have to push to github every time I need a test run. I also use numpy and matplotlib in my package, so running an inane number of test cycles on travis-ci seems overly wasteful to me. As such, ditching tox and simply using .travis.yml alone is not an option.
The CI server is travis-ci
The relevant test scripts look something like this :
.travis.yml
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
script:
- tox -e $TOX_ENV
tox.ini
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
coveralls
This file lets me run the tests locally. However, due to the final coveralls call, the test fails in principle, with :
py27 runtests: commands[1] | coveralls
You have to provide either repo_token in .coveralls.yml, or launch via Travis
ERROR: InvocationError: ...coveralls'
This is an expected error. The passenv bit sends along the necessary information from travis to be able to write to coveralls, and without travis there to provide this information, the command should fail. I don't want this to push the results to coveralls.io, either. I'd like to have coveralls run only if the test is occuring on travis-ci. Is there any way in which I can have this command run conditionally, or set up a build configuration which achieves the same effect?
I've already tried moving the coveralls portion into .travis.yml, but when that is executed coveralls seems to be unable to locate the appropriate .coverage file to send over. I made various attempts in this direction, none of which resulted in a successful submission to coveralls.io except the combination listed above. The following was what I would have hoped would work, given that when I run tox locally I do end up with a .coverage file where I'd expect it - in the root folder of my source tree.
No submission to coveralls.io
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
- pip install python-coveralls
script:
- tox -e $TOX_ENV
after_success:
- coveralls

An alternative solution would be to prefix the coveralls command with a dash (-) to tell tox to ignore its exit code as explained in the documentation. This way even failures from coveralls will be ignored and tox will consider the test execution as successful when executed locally.
Using the example configuration above, it would be as follows:
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
- coveralls

I have a similar setup with Travis, tox and coveralls. My idea was to only execute coveralls if the TRAVIS environment variable is set. However, it seems this is not so easy to do as tox has trouble parsing commands with quotes and ampersands. Additionally, this confused Travis me a lot.
Eventually I wrote a simple python script run_coveralls.py:
#!/bin/env/python
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
In tox.ini, replace your coveralls command with python {toxinidir}/run_coveralls.py.

I am using a environmental variable to run additional commands.
tox.ini
commands =
coverage run runtests.py
{env:POST_COMMAND:python --version}
.travis.yml
language: python
python:
- "3.6"
install: pip install tox-travis
script: tox
env:
- POST_COMMAND=codecov -e TOX_ENV
Now in my local setup, it print the python version. When run from Travis it runs codecov.

Alternative solution if you use a Makefile and dont want a new py file:
define COVERALL_PYSCRIPT
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
print("Not in Travis CI, skipping coveralls")
endef
export COVERALL_PYSCRIPT
coveralls: ## runs coveralls if TRAVIS in env
#python -c "$$COVERALL_PYSCRIPT"
In tox.ini add make coveralls to commands

How to use cProfile with nosetest --with-profile?

nosetest --with-profile --profile-stats-file output
The output can't read by runsnake, because nosetest uses hotshot, if I want to generate a file that can be read with runsnake, I need to convert it so:
st = hotshot.stats.load('output')
st.dump_stats('output_new')
Could I run the test with cProfile directly for read with runsnake?

Evolving on the answer of #squid, you can use a nose plugin called nose-cprof to replace the nose default profiler, hotshot, with cProfile.
To install it:
pip install nose-cprof
Then call nose like this:
nosetests --with-cprofile
It should generate a cProfile output file, that you can then analyze with tools like runsnakerun.

#cihanpesend's answer didn't quite work for me (cProfile couldn't find 'nosetests'), but I did have success on Linux using:
python -m cProfile -o profile.out `which nosetests` .
The resulting output works fine in runsnake.
(Presumably on Windows you could replace which nosetests with the hard-coded path to your nosetests top-level python script.)
I think you are right that the output from nosetests' hotshot profiler is not compatible with runsnake. Certainly the two don't play nice together out of the box for me either.

I don't have info about nosetest except it is python project. So;
python -m cProfile -o outputfile nosetest
Then,
runsnake outputfile
RunSnakeRun is extremly useful to visualize profiler.
Note: to run runsnake, you must install wx and numpy.
update: from omikron's comment; runsnakerun can not support python3 profile output. (i didn't try)

Or you can try nose-cprof plugin:　https://github.com/msherry/nose-cprof　
It replaces hotshot with cProfile

With pyprof2calltree:
$ pip install pyprof2calltree
$ nosetests --with-cprofile --profile-stats=profile.out tests/
$ pyprof2calltree -i profile.out -k
With xdot:
$ sudo apt install xdot
$ gprof2dot -f pstats profile.out | dot -Tpng -o profile.png