I am trying to switch a project from using setup.py to PEP518. I have written the following minimal pyproject.toml:
[build-system]
requires = ["cython", "setuptools", "wheel", "oldest-supported-numpy"]
build-backend = "setuptools.build_meta"
I need some custom installation logic relying on setup.py, so I cannot currently switch to a purely declarative setting.
Notably, my setup.py contains an import numpy which I use to add numpy.get_include() to the includes of an extension. I can build the sdist / wheel using python -m build, which works as intended (providing a build environment by installing the dependencies before calling into setup.py)
I also have a test suite which I run using tox. However, when I run tox in my project I see the following error:
GLOB sdist-make: /project/setup.py
ERROR: invocation failed (exit code 1), logfile: /project/.tox/log/GLOB-0.log
...
File "/project/setup.py", ...
ModuleNotFoundError: No module named 'numpy'
So, per default tox does not install the build dependencies before building the sdist to be used for testing later, causing everything to fail.
Therefore, as suggested in the tox example, I added
[tox]
isolated_build = True
[testenv]
commands = pytest
to the top of tox.ini, which should enable the isolated build. However, when I then execute tox now, all I get is
___ summary ___
congratulations :)
so nothing is actually built / tested (as opposed to a non-isolated build with numpy installed). Is this the expected behavior? How can I actually build and run tests in an isolated environment?
OK, so as it turns out, isolated builds require an envlist like this to work properly (as opposed to the ordinary one which defaults to using the current python environment):
[tox]
isolated_build = True
envlist = py310
Related
I am using nox in order to test my code against different Python versions. I have a dependent package that is stored on my local hard drive. I add this package via poetry with the following command:
poetry add ./path/to/package/local_package_py310.whl
My pyproject.toml file then contains the following line in the [tool.poetry.dependencies] section:
local_package = {path = "./path/to/package/local_package_py310.whl"}
This works fine for the regular Python version that I use (py 3.10). However, when using nox to test my package under Python 3.9, I need to install a different version of this package, namely ./path/to/package/local_package_py39.whl.
My noxfile.py looks like this and the tests for 3.10 do pass.
#nox.session(python=["3.10", "3.9"])
def tests(session) -> None:
"""Run the test suite."""
session.run("poetry", "install", external=True)
session.run("pytest")
However, the test are failing for 3.9 due to the fact that in this case my pyproject.toml is incorrect. It should read:
local_package = {path = "./path/to/package/local_package_py39.whl"}
Is it possible to modify the pyproject.toml according to the Python version that nox is using`?
Currently I have the following:
[gh-actions]
python =
3.7: py37
3.8: py38
3.9: py39
3.10: py310
pypy-3.7: pypy3
pypy-3.8: pypy3
[tox]
minversion = 1.9
envlist =
lint
py{37,38,39,py3}-django22-{sqlite,postgres}
py{37,38,39,310,py3}-django32-{sqlite,postgres}
py{38,39,310,py3}-django40-{sqlite,postgres}
py310-djangomain-{sqlite,postgres}
docs
examples
linkcheck
toxworkdir = {env:TOX_WORKDIR:.tox}
[testenv]
deps =
Pillow
SQLAlchemy
mongoengine
django22: Django>=2.2,<2.3
django32: Django>=3.2,<3.3
django40: Django>=4.0,<4.1
djangomain: https://github.com/django/django/archive/main.tar.gz
py{37,38,39,310}-django{22,32,40,main}-postgres: psycopg2-binary
py{py3}-django{22,32,40,main}-postgres: psycopg2cffi
I need to install a different psycopg2 depending on cpython vs pypy. I've tried all kinds of combinations, and nothing, it all ends in failure. I can't get any of the *-postgres envs to install.
What I'm doing wrong?
The issue is that you do not run the correct environments in your GitHub Actions.
For example. In your tox.ini you create an env with the name:
py37-django22-alchemy-mongoengine-postgres
Then you define the requirements as following:
py{37,38,39,310}-postgres: psycopg2-binary
Which means - install psycopg2-binary when the env name contains the factors py37 + postgres. This matches the above env! So far so good.
But in your gha your run:
- python-version: "3.7"
tox-environment: django22-postgres
... which does not contain the py37 factor - so no match - no installation.
The sqlite tests succeed as it sqlite comes along with Python.
I would suggest that you have a look at the django projects in the jazzband github organization. They all are heavy use of tox factors (the parts separated by dashes) and they also use gha - mostly via https://github.com/ymyzk/tox-gh-actions which I would recommend, too.
Basically you just run tox on gha and let the plugin do the heavy lifting of matching Python environments from tox to github.
Disclaimer:
I am one of the tox maintainers and you earn a prize for the most complex factor setup I have ever seen :-)
The issue was never tox or tox's configuration.
The issue was github actions, when you use tox-environment or python-version + tox-environment, tox-gh-actions won't parse it correctly. Causing it to never match.
This is what I removed.
This is what tox.ini looks like and what github actions looks like [and line 47]
Sorry this is a long question. See the sentence in bold at the bottom for the TL;DR version.
I've spent many hours trying to track down a problem where pylint sometimes doesn't report all the errors in a module. Note that it does find some errors (e.g. long lines), just not all of them (e.g. missing docstrings).
I'm running pylint 1.7.2 on Ubuntu 16.04. (The version available from apt was 1.5.2 but installing via pip gives 1.7.2.)
We typically run pylint from tox, with a tox.ini that looks something like this (this is a cut-down version):
[tox]
envlist = py35
[testenv]
setenv =
MODULE_NAME=our_module
ignore_errors = True
deps =
-r../requirements.txt
whitelist_externals = bash
commands =
pip install --editable=file:///{toxinidir}/../our_other_module
pip install -e .
bash -c \'set -o pipefail; pylint --rcfile=../linting/pylint.cfg our_module | tee pylint.log\'
Amongst other things, the ../requirements.txt file contains a line for pylint==1.7.2.
The behaviour is like this:
[wrong] When the line that imports our_other_module is present, pylint appears to complete successfully and not report any warnings, even though there are errors in the our_module code that it should pick up.
[correct] When that line is commented out, pylint generates the expected warnings.
As part of tracking this down I took two copies of the .tox folder with and without the module import, naming them .tox-no-errors-reported and .tox-with-errors-reported respectively.
So now, even without sourcing their respective tox virtualenvs, I can do the following:
$ .tox-no-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports no linting warnings
$ .tox-with-errors-reported/py35/bin/pylint --rcfile=../linting/pylint.cfg our_module -- reports the expected linting warnings
(where I just changed the pylint script's #! line in each case to reference the python3.5 inside that specific .tox directory instead of the unrenamed .tox)
By diffing .tox-no-errors-reported and .tox-with-errors-reported, I've found that they are very similar. But I can make the "no errors" version start to report errors by removing the path to our_other_module from .tox-no-errors-reported/py35/lib/python3.5/site-packages/easy-install.pth.
So my question is why is pylint using easy_install at runtime, and what is it picking up from our other component that is causing it to fail to report some errors.
As I understand it, pylint has dependencies on astroid and logilab-common, but including these in the requirements.txt doesn't make any difference.
One possible reason for the surprising pylint behavior is the --editable option.
it creates a special .egg-link file in the deployment directory, that links to your project’s source code. And, ..., it will also update the easy-install.pth file to include your project’s source code
The pth file will then affect the sys.path which affects the module import logic of astroid and it is deeply buried in the call stack of pylint.expand_files via pylint.utils.expand_modules. Also pylint identifies the module part and function names in the AST using astroid.modutils.get_module_part.
To test the theory, you can try calling some of the affected astroid functions manually:
import sys, astroid
print(sys.path)
print(astroid.modutils.get_module_part('your_package.sub_package.module'))
astroid.modutils.file_from_modpath(['your_package', 'sub_package', 'module'])
I'm trying to use Tox to test specific versions of Python and Django, but also include a general Pip requirements file of additional dependencies to use for all cases.
As the Tox docs explain, you do the first like:
deps =
django15: Django>=1.5,<1.6
django16: Django>=1.6,<1.7
py33-mysql: PyMySQL ; use if both py33 and mysql are in an env name
py26,py27: urllib3 ; use if any of py26 or py27 are in an env name
py{26,27}-sqlite: mock ; mocking sqlite in python 2.x
and you do the second like:
deps = -r{toxinidir}/pip-requirements.txt
-r{toxinidir}/pip-requirements-test.txt
but how do you combine these?
If I try to define multiple deps, Tox gives me the error "duplicate name 'deps'", but I don't see a way to combine the dictionary and list notatations for deps.
I also tried:
deps =
-r{toxinidir}/pip-requirements.txt
-r{toxinidir}/pip-requirements-test.txt
django15: Django>=1.5,<1.6
django16: Django>=1.6,<1.7
and although that doesn't give me any parsing error, when I go to run a test, I get the error:
ERROR: py27-django15: could not install deps [-r/usr/local/myproject/pip-requirements.txt, -r/usr/local/myproject/pip-requirements-test.txt, Django>=1.5,<1.6]; v = InvocationError('/usr/local/myproject/.tox/py27-django15/bin/pip install -r/usr/local/myproject/pip-requirements.txt -r/usr/local/myproject/pip-requirements-test.txt Django>=1.5,<1.6 (see /usr/local/myproject/.tox/py27-django15/log/py27-django15-1.log)', 1)
presumably because it's interpreting the requirements file as a literal Python package name.
I've been learning about how to do testing in tox for my python project.
I have (what should be) a fairly standard tox initialization file that looks like the following:
[tox]
envlist=py27,flake8
...
[testenv:flake8]
deps=flake8
commands=flake8 library # 'library' is temp. name of project
Everything looks normal, all the test works, and even the flake8 output comes through (output below). However, tox raises an InvocationError (it does the same for testing using pylint)
flake8 recreate: /Users/shostakovich/projects/project_templates/library/.tox/flake8
flake8 installdeps: flake8
flake8 inst: /Users/shostakovich/projects/project_templates/library/.tox/dist/library-0.1.0.zip
flake8 installed: flake8==2.4.1,library==0.1.0,mccabe==0.3,pep8==1.5.7,pyflakes==0.8.1,wheel==0.24.0
library/__main__.py:12:1: F401 'os' imported but unused
library/__main__.py:13:1: F401 're' imported but unused
...
ERROR: InvocationError: '/Users/shostakovich/projects/project_templates/library/.tox/flake8/bin/flake8 library'
I am running tox 2.0.2 on MaxOSX 10.9.5. The problem goes away if I just call flake8 or pylint directly (the version of flake8 is shown above).
tox doesn't fail, it works!
Your flake8 source code check has findings and therefore tox exits with failures, that's your test result. Fix the findings and your done!
You may configure the flake8 run to ignore specific codes with a section in your tox.ini. From the flake8 docs:
[flake8]
ignore = E226,E302,E41
There are more options you may be interested in, e.g. select = ... for whitelisting enabled code checks.
You can also tell flake8 to exit without failure, even if the test results are not perfect. This will suppress the misleading InvocationError. Just add to your command --exit-zero, so for example:
commands=flake8 library --exit-zero