Develop Reference

How can I specify which positional argument to use in tox?

I have the following tox.ini:
[tox]
envlist = py37-{lint, test}
[testenv:py37-{lint, test}]
envdir = {toxworkdir}/lint_and_test_env
deps =
pylint
pytest
pytest-xdist
commands =
lint: pylint src {posargs}
test: pytest tests {posargs}
I want to run both environments in parallel and specify --jobs=4 for pylint and -n auto for pytest. Executing tox -p -- --jobs=4 -n auto fails because pylint does not recognize the -n argument and vice versa.
Is there a way to achieve my goal?

I see no way to do what you want.
That said, I would strongly recommend that you split your environments and use one for testing and one for linting.
[tox]
envlist = py37, lint
[testenv]
deps =
pytest
pytest-xdist
commands =
pytest tests {posargs}
[testenv:lint]
deps = pylint
commands = pylint src {posargs}
Then you can pass in different arguments to the commands, or run all envs via tox.
Also you could set defaults in case you do not specify {posargs}.
e.g.
[testenv:lint]
deps = pylint
commands = pylint src {--jobs=4:posargs}
If you want to run all tox envs in parallel you could run...
tox -p
P.S.: I am one of the tox maintainers, and I have contributed to hundreds of open source projects, and basically 99% use separate envs for testing and linting.
P.P.S.: If you want to run more than one linter, you should have a look at pre-commit.

Configute tox.ini to ignore library during tests with py27

I have the following tox.ini configuration file:
# Tox (http://tox.testrun.org/) is a tool for running tests
# in multiple virtualenvs. This configuration file will run the
# test suite on all supported python versions. To use it, "pip install tox"
# and then run "tox" from this directory.
[tox]
envlist = py27, py37, pycodestyle, pylint
[testenv]
commands =
pytest --junitxml=unit-tests.xml --cov=xivo --cov-report term --cov-report xml:coverage.xml xivo
deps =
-rrequirements.txt
-rtest-requirements.txt
pytest-cov
[testenv:pycodestyle]
# E501: line too long (80 chars)
commands =
-sh -c 'pycodestyle --ignore=E501 xivo > pycodestyle.txt'
deps =
pycodestyle
whitelist_externals =
sh
[testenv:pylint]
commands =
-sh -c 'pylint --rcfile=/usr/share/xivo-ci/pylintrc xivo > pylint.txt'
deps =
-rrequirements.txt
-rtest-requirements.txt
pylint
whitelist_externals =
sh
[testenv:black]
skip_install = true
deps = black
commands = black --skip-string-normalization .
[testenv:linters]
skip_install = true
deps =
flake8
flake8-colors
black
commands =
black --skip-string-normalization --check .
flake8
[testenv:integration]
basepython = python3.7
usedevelop = true
deps = -rintegration_tests/test-requirements.txt
changedir = integration_tests
passenv =
WAZO_TEST_DOCKER_OVERRIDE_EXTRA
commands =
make test-setup
pytest -v {posargs}
whitelist_externals =
make
sh
[flake8]
# E501: line too long (80 chars)
# W503: line break before binary operator
# E203: whitespace before ':' warnings
# NOTE(sileht):
# * xivo_config.py not python3 ready
exclude = .tox,.eggs,xivo/xivo_config.py
show-source = true
ignore = E501, E203, W503
max-line-length = 99
application-import-names = xivo
I have update my requirements.txt file in order to upgrade the version of Marshmallow from 3.0.0b14 to 3.10.0; like this:
https://github.com/wazo-platform/wazo-lib-rest-client/archive/master.zip
https://github.com/wazo-platform/wazo-auth-client/archive/master.zip
https://github.com/wazo-platform/xivo-bus/archive/master.zip
cheroot==6.5.4
flask==1.0.2
netifaces==0.10.4
psycopg2==2.7.7
pyyaml==3.13
python-consul==1.1.0
marshmallow==3.10.0
six==1.12.0
stevedore==1.29.0
Now my problem is that, when I run tox -e py37 everything works fine, but when I run this command tox -e py27, it fails. I get that the issue is that Marshmallow 3.10.0 is not supported on Python 2.7; so what I want to do is to change the tox.ini file in order to ignore Marshmallow library tests when it comes to running this command tox -e py27. I am new with tox, so I am not sure how to do this. Any help is welcome, thanks.

You need to mark those "Marshmellow" tests with a pytest marker.
https://docs.pytest.org/en/6.2.x/example/markers.html
e.g.
#pytest.mark.marshmellow
def test_xxx():
...
And then you need to run pytest -m "not marshmellow".
As you only want to do this for Python 2.7, you'd need to create a new testenv.
e.g.
[testenv:py27]
commands = pytest -m "not marshmellow" ...

The last version of Marshmallow that supports Python 2.7 is 2.21.0. You can mark your requirements.txt to install different versions of Marshmallow for different versions of Python:
marshmallow<3; python_version == '2.7'
marshmallow==3.10.0; python_version >= '3.6'
See https://www.python.org/dev/peps/pep-0496/

poetry dependencies not available when running script with tox

I have a python project that uses poetry and tox. It has source code, tests and scripts (juptext notebooks). I can't import the dev dependencies in the scripts, but I can in the tests.
When I came across this problem, I created the following minimal example. At first, it didn't work, then I fiddled around with it, and now it's working. So I stripped the project that has the actual problem down so it's indistinguishable other than the project name, location, virtual env, and .git directory, but that's still not working.
UPDATE deleting all build artifacts and the virtualenv for the minimal example makes it stop working again
UPDATE adding the line scripts: poetry install to the tox commands fixed only the minimal example
The source code, tests and scripts are in the following layout
foo
+--foo
| +--__init__.py
|
+--tests
| +--__init__.py
| +--test_foo.py
|
+--scripts
| +--foo_script.py
|
+--pyproject.toml
+--tox.ini
The files are either empty or as follows:
foo_script.py
import requests
test_foo.py
import requests
import pytest
def test():
assert True
pyproject.toml
[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]
[tool.poetry.dependencies]
python = "^3.7"
requests = "*"
[tool.poetry.dev-dependencies]
pytest = "^4.6"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
tox.ini
[tox]
envlist = test, scripts
isolated_build = true
skipsdist = true
[testenv]
basepython = python3.7
whitelist_externals =
pytest
bash
commands =
test: pytest
scripts: bash -c 'python3 scripts/*.py'
When I run tox, I get
test run-test-pre: PYTHONHASHSEED='4126239415'
test run-test: commands[0] | pytest
============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.2.1, py-1.8.0, pluggy-0.13.0
cachedir: .tox/test/.pytest_cache
rootdir: /home/#######/foo
collected 1 item
tests/test_foo.py . [100%]
============================== 1 passed in 0.09s ===============================
scripts run-test-pre: PYTHONHASHSEED='4126239415'
scripts run-test: commands[0] | bash -c 'python3 scripts/*.py'
Traceback (most recent call last):
File "scripts/foo_script.py", line 1, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
ERROR: InvocationError for command /bin/bash -c 'python3 scripts/*.py' (exited with code 1)
___________________________________ summary ____________________________________
test: commands succeeded
ERROR: scripts: commands failed

I believe something like the following should work:
pyproject.toml
[tool.poetry]
name = "foo"
version = "0.1.0"
description = ""
authors = ["foo maker"]
[tool.poetry.dependencies]
python = "^3.7"
requests = "*"
#
pytest = { version = "^4.6", optional = true }
[tool.poetry.extras]
test = ["pytest"]
# [tool.poetry.dev-dependencies]
# use 'test' extra instead
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
tox.ini
[tox]
envlist = test, scripts
isolated_build = true
[testenv]
basepython = python3.7
whitelist_externals =
pytest
bash
extras =
test
commands =
test: pytest
scripts: bash -c 'for f in scripts/*.py; do python "$f"; done'

Assuming you have installed poetry and tox and pytest are dependencies in your pyproject.yml (notice poetry run, see https://python-poetry.org/docs/cli/#run):
[tox]
envlist = py37
isolated_build = True
skipsdist = True
[testenv]
whitelist_externals = poetry
commands=
poetry run pytest
optionally you can make the install happen on running the tests by changing the last bit to (but then you will need to have tox installed outside of poetry, which could cause you issues down the line)
commands=
poetry install
poetry run pytest
Also depending on your root folder and where the tests are you can configure the path for tox to change directory to by adding
changedir = tests
In which case the whole file would look like this if you are in directory foo executing tox:
[tox]
envlist = py37
isolated_build = True
skipsdist = True
[testenv]
whitelist_externals = poetry
commands=
poetry run pytest
changedir = tests

This is an easy fix. First, run rm -rf .tox to delete the .tox directory, probably hasn't installed the files you wanted.
Here is an example of my tox.ini.
[tox]
isolated_build = true
skipsdist = true
envlist = py39
[testenv]
deps = -rrequirements-dev.txt
whitelist_externals =
poetry
skip_install = true
commands =
python -m pytest tests/
As you can see in the tox.ini file I have a separate requirements for dev. This can be generated from poetry via the commands below.
poetry export --format=requirements.txt --without-hashes --with dev --output=requirements.txt
poetry export --format=requirements.txt --without-hashes --output=requirements-dev.txt
Don't ask me why but the deps in [testenv] are concatenated out, as you can see with deps = -rrequirements-dev.txt, would also change your command to python -m pytest /tests
Ciao.

First I need to install the dependencies with poetry install. Then append poetry run to the beginning of the command to enable the dependencies. Also running python scripts like that will just run the first, passing the name of the others as args to the first program. Instead use for f in scripts/*.py; do python "$f"; done (see here)
All together
poetry install
poetry run bash -c 'for f in scripts/*.py; do python "$f"; done'

Run certain Tox commands only on TravisCI

I am trying to make it so that a code coverage command is only run in Tox for TravisCI. How would I do that?
The codecov library does not work if I just stick it into the .travis.yml file, and I want it to not be run during normal tox tests as it makes the console way too messy.

check for the presence of $TRAVIS environment variable, see here for others that Travis sets
maybe you need to add this line to your tox.ini :
passenv = TRAVIS
or
you can create a custom entry in your tox.ini
[testenv:travis]
deps= ...
commands= <your travis command>
if you need need to add a step
[testenv]
commands =
....
travis: <your travis command>
if you need only to change dependencies change your deps as
[testenv]
deps =
travis: <your travis dependencies>
in your travis.ini call tox -e travis
Note do not list travis in your envlist=
one more option if you need to add a step to a matrix
[testenv]
passenv = TRAVIS
commands =
- ...
- sh -c 'if [ '{env:TRAVIS:}' = 'true' ]; then ...... ; fi'

Conditional Commands in tox? (tox, travis-ci, and coveralls)

tl;dr:
I'm setting up CI for a project of mine, hosted on github, using tox and travis-ci. At the end of the build, I run converalls to push the coverage reports to coveralls.io. I would like to make this command 'conditional' - for execution only when the tests are run on travis; not when they are run on my local machine. Is there a way to make this happen?
The details:
The package I'm trying to test is a python package. I'm using / planning to use the following 'infrastructure' to set up the tests :
The tests themselves are of the py.test variety.
The CI scripting, so to speak, is from tox. This lets me run the tests locally, which is rather important to me. I don't want to have to push to github every time I need a test run. I also use numpy and matplotlib in my package, so running an inane number of test cycles on travis-ci seems overly wasteful to me. As such, ditching tox and simply using .travis.yml alone is not an option.
The CI server is travis-ci
The relevant test scripts look something like this :
.travis.yml
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
script:
- tox -e $TOX_ENV
tox.ini
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
coveralls
This file lets me run the tests locally. However, due to the final coveralls call, the test fails in principle, with :
py27 runtests: commands[1] | coveralls
You have to provide either repo_token in .coveralls.yml, or launch via Travis
ERROR: InvocationError: ...coveralls'
This is an expected error. The passenv bit sends along the necessary information from travis to be able to write to coveralls, and without travis there to provide this information, the command should fail. I don't want this to push the results to coveralls.io, either. I'd like to have coveralls run only if the test is occuring on travis-ci. Is there any way in which I can have this command run conditionally, or set up a build configuration which achieves the same effect?
I've already tried moving the coveralls portion into .travis.yml, but when that is executed coveralls seems to be unable to locate the appropriate .coverage file to send over. I made various attempts in this direction, none of which resulted in a successful submission to coveralls.io except the combination listed above. The following was what I would have hoped would work, given that when I run tox locally I do end up with a .coverage file where I'd expect it - in the root folder of my source tree.
No submission to coveralls.io
language: python
python: 2.7
env:
- TOX_ENV=py27
install:
- pip install tox
- pip install python-coveralls
script:
- tox -e $TOX_ENV
after_success:
- coveralls

An alternative solution would be to prefix the coveralls command with a dash (-) to tell tox to ignore its exit code as explained in the documentation. This way even failures from coveralls will be ignored and tox will consider the test execution as successful when executed locally.
Using the example configuration above, it would be as follows:
[tox]
envlist = py27
[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
pytest
coverage
pytest-cov
coveralls
commands =
py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
- coveralls

I have a similar setup with Travis, tox and coveralls. My idea was to only execute coveralls if the TRAVIS environment variable is set. However, it seems this is not so easy to do as tox has trouble parsing commands with quotes and ampersands. Additionally, this confused Travis me a lot.
Eventually I wrote a simple python script run_coveralls.py:
#!/bin/env/python
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
In tox.ini, replace your coveralls command with python {toxinidir}/run_coveralls.py.

I am using a environmental variable to run additional commands.
tox.ini
commands =
coverage run runtests.py
{env:POST_COMMAND:python --version}
.travis.yml
language: python
python:
- "3.6"
install: pip install tox-travis
script: tox
env:
- POST_COMMAND=codecov -e TOX_ENV
Now in my local setup, it print the python version. When run from Travis it runs codecov.

Alternative solution if you use a Makefile and dont want a new py file:
define COVERALL_PYSCRIPT
import os
from subprocess import call
if __name__ == '__main__':
if 'TRAVIS' in os.environ:
rc = call('coveralls')
raise SystemExit(rc)
print("Not in Travis CI, skipping coveralls")
endef
export COVERALL_PYSCRIPT
coveralls: ## runs coveralls if TRAVIS in env
#python -c "$$COVERALL_PYSCRIPT"
In tox.ini add make coveralls to commands