Develop Reference

Pytest does not find pyYAML in Travis CI tests that have PyPI and conda package dependencies

I am trying to set up automatic testing for my Python package using Travis CI. My Python package depends on Iris as well as other packages such as PyYAML, numpy, etc. It also depends on a PyPI package (ScriptEngine). Now, I would like to set up a Travis CI environment using conda (to install Iris) and pip (to install the PyPI package as well as checking the requirements for PyYAML and numpy). I would then like to install my package using pip install ..
To test if this works, I have written one simple Pytest test that imports PyYAML.
I am currently trying to do this using this .travis.yml file:
language: python
python:
- "3.6"
- "3.7"
- "3.8"
# command to install dependencies
install:
- sudo apt-get update
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- source "$HOME/miniconda/etc/profile.d/conda.sh"
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
# Useful for debugging any issues with conda
- conda info -a
- conda env create -f tests/test-environment.yml python=$TRAVIS_PYTHON_VERSION
- conda activate test-environment
- conda install pip
- conda install -c conda-forge iris
- pip install -r requirements.txt
- pip install .
# command to run tests
script: pytest
Note: This is the first time for me to really work with Travis CI. This script is a mixture of examples from the Conda docs as well as the Travis CI docs.
Pytest then fails to import PyYAML (although it gets installed because of the requirements.txt as well as the Iris dependencies):
Here is the confirmation from the logs that it got installed:
Requirement already satisfied: pyYAML>=5.1 in /home/travis/miniconda/envs/test-environment/lib/python3.8/site-packages (from ece-4-monitoring==0.1.0) (5.3.1)
And this is the Error from Pytest:
$ pytest
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.3.1, py-1.7.0, pluggy-0.8.0
rootdir: /home/travis/build/valentinaschueller/ece-4-monitoring, inifile:
collected 1 item / 1 errors
==================================== ERRORS ====================================
_________________ ERROR collecting tests/test_file_handling.py _________________
ImportError while importing test module '/home/travis/build/valentinaschueller/sciptengine-tasks-ecearth/tests/test_file_handling.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_file_handling.py:3: in <module>
import helpers.file_handling as file_handling
helpers/file_handling.py:1: in <module>
import yaml
E ModuleNotFoundError: No module named 'yaml'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.12 seconds ============================
The command "pytest" exited with 2.
If I try this exact setup using a conda virtual environment locally on my computer, I do not get this problem. Why does this not work on the Travis CI virtual machine?

As cel suggested in their comment: I could fix the problem by explicitly requiring pytest in the requirements.txt.
It is not necessary to activate test-environment in the script part. However, a very helpful tip was to use echo $(which pytest) && pytest instead of just pytest to check if pytest is installed at /home/travis/miniconda/envs/test-environment/bin/pytest.

How to Install Dependencies from a Local Python Repository with Tox?

Introduction
I would like to install some dependencies (local-package1 and local-package2) into a virtualenv using Tox. Those dependencies are packages that only exist on my local Python Package Index (inside of an Artifactory instance).
Attempts
Here's my tox.ini:
[tox]
envlist = py27
indexserver =
LOCAL = https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple
[testenv]
deps =
:LOCAL:local-package1
mock
pykwalify
:LOCAL:local-package2
xlrd
XlsxWriter
xlwt
yamllint
commands =
make.sh
Here's the output from tox -v:
using tox.ini: repo/tox.ini
using tox-3.7.0 from /usr/local/lib/python2.7/site-packages/tox/__init__.pyc
GLOB sdist-make: ~/repo/setup.py
~/repo$ /usr/local/opt/python#2/bin/python2.7 ~/repo/setup.py sdist --formats=zip --dist-dir ~/repo/.tox/dist >~/repo/.tox/log/tox-0.log
package .tmp/package/1/example-0.0.0.zip links to dist/example-0.0.0.zip (~/repo/.tox)
py27 cannot reuse: no previous config ~/repo/.tox/py27/.tox-config1
py27 create: ~/repo/.tox/py27
~/repo/.tox$ /usr/local/opt/python#2/bin/python2.7 -m virtualenv --python /usr/local/opt/python#2/bin/python2.7 py27 >~/repo/.tox/py27/log/py27-0.log
py27 installdeps: :LOCAL:local-package1, mock, pykwalify, :LOCAL:local-package2, xlrd, XlsxWriter, xlwt, yamllint
~/repo$ ~/repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2 >~/repo/.tox/py27/log/py27-1.log
And doing a tail -f ~/repo/.tox/py27/log/py27-1.log yields the following:
actionid: py27
msg: getenv
cmdargs: 'repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2'
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Looking in indexes: https://me:****#artifactory.example.com/api/pypi/pypi-local/simple, https://artifactory.example.com/api/pypi/pypi-local/simple
Collecting local-package1
and it seems to hang indefinitely.
If I run the command:
repo/.tox/py27/bin/python -m pip install -i https://me:abc123#artifactory.example.com/api/pypi/pypi-local/simple local-package1 local-package2
outside of the virtualenv, I can install these packages without any issues.
Question
How can I properly install dependencies with Tox which includes authentication to a Python Package Index?

Capture just version from tox commands

I have following in my tox.ini :
[tox]
skipsdist = true
envlist = py27, py36, lint
[testenv]
whitelist_externals=flake8
commands =
version: python setup.py --version
When I run
tox -e version
I get the following output :
tox -e version 2>version
version runtests: PYTHONHASHSEED='3264081464'
version runtests: commands[0] | python setup.py --version
0.2.0
__________________________________________________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________________________________________________
version: commands succeeded
congratulations :)
I need to capture just the version (0.2.0) from tox -e version output, what the most elegant/pythonic way of doing it ? Also is there anyway I can have tox just output the output to the command and not the rest ?

Save the version into a file. In tox.ini:
[testenv]
whitelist_externals = /bin/sh
commands =
version: /bin/sh -c "python setup.py --version >version"
In shell:
tox -e version
cat version
rm version

Including -qq in the tox command should reduce the output to just 0.2.0.

Where did tox look when searching for "basepython"?

Currently, we have this in the Build Environment on cloudbees
curl -s -o use-python https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/python/use-python
chmod u+x use-python
# Get all Python distributions from CloudBees
supported=("2.7.14" "3.4.7" "3.5.4" "3.6.4")
for version in "${supported[#]}"; do
export PYTHON_VERSION=$version
. ./use-python
done
# Sanity checks.
python --version
python2 --version
python2.7 --version
python3 --version
python3.4 --version
python3.5 --version
python3.6 --version
which python
which python2
which python3
which python2.7
which python3.4
which python3.5
which python3.6
In the tox.ini, we've declared the basepython=3.4 at line https://github.com/nltk/nltk/blob/develop/tox.ini#L84
[testenv:py34-jenkins]
basepython = python3.4
commands = {toxinidir}/jenkins.sh
And our setup on the cloudbees CI server could find our Python3.4 interpreter, as we see with which python: https://nltk.ci.cloudbees.com/job/nltk/808/TOXENV=py34-jenkins,jdk=jdk8latestOnlineInstall/console
But when it ran tox, it threw and InterpreterNotFound:
+ which python3.4
/scratch/jenkins/python/python-3.4.7-x86_64/bin/python3.4
+ which python3.5
/scratch/jenkins/python/python-3.5.4-x86_64/bin/python3.5
+ which python3.6
/scratch/jenkins/python/python-3.6.4-x86_64/bin/python3.6
+ mkdir -p /home/jenkins/lib
+ '[' -f /home/jenkins/lib/libblas.so ']'
+ ln -sf /usr/lib64/libblas.so.3 /home/jenkins/lib/libblas.so
+ '[' -f /home/jenkins/lib/liblapack.so ']'
+ ln -sf /usr/lib64/liblapack.so.3 /home/jenkins/lib/liblapack.so
[EnvInject] - Script executed successfully.
[jdk8latestOnlineInstall] $ /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/bin/python -c "import pip; pip.main();" install --upgrade tox
Requirement already up-to-date: tox in /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/lib/python2.7/site-packages
Requirement already up-to-date: virtualenv>=1.11.2 in /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/lib/python2.7/site-packages (from tox)
Requirement already up-to-date: six in /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/lib/python2.7/site-packages (from tox)
Requirement already up-to-date: py>=1.4.17 in /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/lib/python2.7/site-packages (from tox)
Requirement already up-to-date: pluggy<1.0,>=0.3.0 in /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/lib/python2.7/site-packages (from tox)
You are using pip version 7.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[jdk8latestOnlineInstall] $ /scratch/jenkins/shiningpanda/jobs/280e15ca/tools/bin/python -c "import tox; tox.cmdline();" -c tox.ini
GLOB sdist-make: /scratch/jenkins/workspace/nltk/TOXENV/py34-jenkins/jdk/jdk8latestOnlineInstall/setup.py
py34-jenkins create: /scratch/jenkins/workspace/nltk/TOXENV/py34-jenkins/jdk/jdk8latestOnlineInstall/.tox/py34-jenkins
ERROR: InterpreterNotFound: python3.4
Couple of related questions:
Where did tox look when searching for basepython?
Is it trying to look for PYTHONPATH?
How can I setup the tox such that I append custom path(s) to where it should be looking for basepython?
python3.4 is inside the PATH environmental variable, why isn't tox looking there?

Pytest doesn't recognize -n option after pytest-xdist installation

I have installed pytest-xdist on top of a working pytest environment :
pip install pytest-xdist
and I have received this output
Downloading/unpacking pytest-xdist
Downloading pytest-xdist-1.10.tar.gz
Running setup.py egg_info for package pytest-xdist
no previously-included directories found matching '.hg'
Downloading/unpacking execnet>=1.1 (from pytest-xdist)
Downloading execnet-1.2.0.tar.gz (163kB): 163kB downloaded
Running setup.py egg_info for package execnet
warning: no files found matching 'conftest.py'
Requirement already satisfied (use --upgrade to upgrade): pytest>=2.4.2 in /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages (from pytest-xdist)
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.20 in /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages (from pytest>=2.4.2->pytest-xdist)
Installing collected packages: pytest-xdist, execnet
Running setup.py install for pytest-xdist
no previously-included directories found matching '.hg'
Running setup.py install for execnet
warning: no files found matching 'conftest.py'
Successfully installed pytest-xdist execnet
Cleaning up...
at this point I have tried to run my test suite in parallel
py.test -n 4
but I received this output instead
usage: py.test [options] [file_or_dir] [file_or_dir] [...]
py.test: error: unrecognized arguments: -n
Output of 'py.test --version is'
This is pytest version 2.6.2, imported from /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
pytest-capturelog-0.7 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_capturelog.pyc
pytest-contextfixture-0.1.1 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_contextfixture.pyc
pytest-cov-1.7.0 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_cov.pyc
pytest-django-2.6.2 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_django/plugin.pyc
pytest-pydev-0.1 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_pydev.pyc
pytest-runfailed-0.3 at /Users/sal/Documents/code/Python/VirtualEnv/Spring/lib/python2.7/site-packages/pytest_runfailed.pyc
and pytest-xdist is effectively missing.
What I was wrong? Thanks.

Like user2412166, I suffered the same issue. Unlike user2412166, the solution in my case was to relax the permissions on the xdist and pytest_xdist-1.14.dist-info system directories installed by pip3.
Some backstory: For security, I run a strict umask on my system prohibiting all access to other users and write access to group users by default:
$ umask
027
While this is usually a good thing, it also occasionally gets me into trouble. Installing python-xdist via pip3 under this umask:
$ sudo pip3 install pytest-xdist
...resulted in pip3 prohibiting read and execution access to non-superusers – which had better be only me:
$ ls -l /usr/lib64/python3.4/site-packages/xdist
drwxr-x--- 3 root root 4.0K 2016-04-10 01:19 xdist/
$ ls -l /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
drwxr-x--- 3 root root 4.0K 2016-04-10 01:19 xdist/
While pip3 was not wrong in doing so, py.test was (...arguably!) wrong in silently ignoring rather than explicitly reporting an obvious permissions issue during plugin detection.
This was trivially fixable by recursively granting other users both read and directory execution permissions for the afflicted system directories:
$ chmod -R o+rX /usr/lib64/python3.4/site-packages/xdist
$ chmod -R o+rX /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
The proof is the command-line pudding:
$ ls -l /usr/lib64/python3.4/site-packages/xdist
drwxr-xr-x 3 root root 4.0K 2016-04-10 01:19 xdist/
$ ls -l /usr/lib64/python3.4/site-packages/pytest_xdist-1.14.dist-info
drwxr-xr-x 3 root root 4.0K 2016-04-10 01:19 xdist/
$ py.test --version
This is pytest version 2.8.7, imported from /usr/lib64/python3.4/site-packages/pytest.py
setuptools registered plugins:
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/looponfail.py
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/plugin.py
pytest-xdist-1.14 at /usr/lib64/python3.4/site-packages/xdist/boxed.py
Thus was the unclear made clear, the buggy debugged, and the slow tests parallelized quickly.

I had the same problem. The problem is not with the version. Somehow py.test cannot see where xdist is. Here's what worked for me:
pip install pytest --user
pip install pytest-xdist --user
export PATH=$HOME/.local/bin:$PATH

Please try py.test --version and look at the output which explains where things are imported from, including plugins. Very likely are not running the py.test that you think you are running.

I ran into this problem and figured out it was because of a really old setuptools (the default version that ships on Centos6.7)
pip list | grep setuptools
setuptools (0.6rc11)
So first upgrade setuptools
sudo pip install --upgrade setuptools
Then reinstall pytest and pytest-xdist
sudo pip install --upgrade pytest pytest-xdist --force-reinstall
After this pytest was able to discover the xdist plugin.