I have a python project that makes use of libraries that needs to be built. Given that I use anaconda. I want to create a plan for travis that would let me test against multiple python versions and I am not able to do it. Here is what I have:
I want to test it against multiple python versions (e.g. 2.7, 3.5, 3.6)
I have requirements.yml file which looks like following:
channels:
- kne
dependencies:
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
My .travis.yml contains:
language: python
# sudo false implies containerized builds
sudo: false
python:
- 3.5
- 3.4
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
- conda env create -n testenv -f environment.yml python=$TRAVIS_PYTHON_VERSION
- source activate testenv
install:
- python setup.py install
script:
- python --version
- python -m pytest --cov=.
- codecov
If I put python version into environment.yml it works fine but I can't use multiple python versions. For me, it seems to if -f is provided, it ignores any additional packages listed for conda env create.
Also, adding - conda install -n testenv python=$TRAVIS_PYTHON_VERSION after env creation does not work.
UnsatisfiableError: The following specifications were found to be in conflict:
- functools32 -> python 2.7.*
- python 3.5*
Use "conda info <package>" to see the dependencies for each package.
What am I supposed to do, in order make it work?
// If you would like to see more details, it is available here: https://travis-ci.org/mbednarski/Chiron/jobs/220644726
You can use sed to modify the python dependency in environment.yml file before creating the conda environment.
Include python in the environment.yml:
channels:
- kne
dependencies:
- python=3.6
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
And then modify your .travis.yml:
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
# Edit the environment.yml file for the target Python version
- sed -i -E 's/(python=)(.*)/\1'$TRAVIS_PYTHON_VERSION'/' ./environment.yml
- conda env create -n testenv -f environment.yml
- source activate testenv
The sed regex will replace the text python=3.6 with the equivalent for the target python version.
BTW: I see in your repository you've worked around this by using multiple environment.yml files. This seems reasonable, even necessary for certain dependencies, but perhaps tedious to maintain for many Python versions.
Related
The following snapshot shows the file structure:
When I run on Gitlab CI, here is what I am seeing:
Why is this error occurring when Gitlab runs it but not when I run locally?
Here is my .gitlab-ci.yml file.
Note that this had been working before.
I recently made win_perf_counters a Git submodule instead of being an actual subdirectory. (Again, it works locally.)
test:
before_script:
- python -V
- pip install virtualenv
- virtualenv venv
- .\venv\Scripts\activate.ps1
- refreshenv
script:
- python -V
- echo "******* installing pip ***********"
- python -m pip install --upgrade pip
- echo "******* installing locust ********"
- python -m pip install locust
- locust -V
- python -m pip install multipledispatch
- python -m pip install pycryptodome
- python -m pip install pandas
- python -m pip install wmi
- python -m pip install pywin32
- python -m pip install influxdb_client
- set LOAD_TEST_CONF=load_test.conf
- echo "**** about to run locust ******"
- locust -f ./src/main.py --host $TARGET_HOST -u $USERS -t $TEST_DURATION -r $RAMPUP -s 1800 --headless --csv=./LoadTestsData_VPOS --csv-full-history --html=./LoadTestsReport_VPOS.html --stream-file ./data/stream_jsons/streams_vpos.json --database=csv
- Start-Sleep -s $SLEEP_TIME
variables:
LOAD_TEST_CONF: load_test.conf
PYTHON_VERSION: 3.8.0
TARGET_HOST: http://10.10.10.184:9000
tags:
- win2019
artifacts:
paths:
- ./LoadTests*
- public
only:
- schedules
after_script:
- ls src -r
- mkdir .public
- cp -r ./LoadTests* .public
- cp metrics.csv .public -ErrorAction SilentlyContinue
- mv .public public
When I tried with changing the Gitlab CI file to use requirements.txt:
Probably the python libraries you are using in your local environment are not the same you are using in gitlab. Run a pip list or pip freeze in your local machine and see which versions do you have there. Then pip install those in your gitlab script. A good practice is to have a requirements.txt or a setup.py file with specific versions rather than pulling the latest versions every time.
Probably the module you are developing doesn't have the __init__.py file and thus it cannot be found when imported from the external.
I have a Databricks ML Runtime cluster. I am trying to install fbprophet library using cluster init_script. I am following the example in the Databricks documentation.
#!/bin/bash
set -x
. /databricks/conda/etc/profile.d/conda.sh
conda info --envs
conda activate /databricks/python
conda install -y --offline /dbfs/conda_packages/linux-64/fbprophet.tar.bz2
But the init_script logs show that it cannot activate conda env, and cannot locate package from given path.
: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
bash: line 2: /databricks/conda/etc/profile.d/conda.sh
: No such file or directory
usage: conda [-h] [-V] command ...
conda: error: unrecognized arguments: --envs
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
PackagesNotFoundError: The following packages are not available from current channels:
- /dbfs/conda_packages/linux-64/fbprophet.tar.bz2
Current channels:
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/linux-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Can you please guide me what is wrong with the script, and how can I install the conda package from a path on DBFS.
Or is there any other way I can install conda packages. Because if I try to install using UI, or library API, it fails to install fbprophet package.
Regards
I use MacBook Air, with catalina OS. The conda version I use is 4.9.2.
I am trying to install pycafe with conda using the following syntax in my homebrew window:
conda install -c paulscherrerinstitute pycafe
And I get the following error:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- pycafe
Current channels:
- https://conda.anaconda.org/paulscherrerinstitute/osx-64
- https://conda.anaconda.org/paulscherrerinstitute/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Can you please help on how to install pycafe?
Package Unavailable for OS X
As mentioned in the comments, this particular package is only available through a user channel and they had only built it for the linux-64 platform and Python 3.5. If you're keen on avoiding having to compile this yourself (and I should note that the version build is still the latest), then I'd suggest going through Docker to build and host the environment.
Alternative: Docker Container
I put together a basic Dockerfile and YAML as a starter, but I'd expect there may be some additional configuration (e.g., setting environmental variables) that is needed.
pycafe.yaml
name: pycafe
channels:
- paulscherrerinstitute
- conda-forge
- defaults
dependencies:
- python=3.5
- pycafe
- cafe
- numpy
Dockerfile
ROM continuumio/miniconda3
SHELL ["/bin/bash", "--login", "-c"]
# Install PyCafe
COPY pycafe.yaml .
RUN conda env create -f pycafe.yaml -n pycafe \
&& conda config --set auto_activate_base false \
&& sed -i "s/conda activate base/conda activate pycafe/" ~/.bashrc \
&& conda clean -a -y
Placing both of these in a directory, one can build the Docker image with
docker build -t "pycafe:1.3.0" .
Then one could launch a Python session with
docker run -it pycafe:1.3.0 bash
python
I have a Python project for which I use tox to run the pytest-based tests. I am attempting to configure the project to build on CircleCI.
The tox.ini lists both Python 3.6 and 3.7 as environments:
envlist = py{36,37,},coverage
I can successfully run tox on a local machine within a conda virtual environment that uses Python version 3.7.
On CircleCI I am using a standard Python virtual environment since that is what is provided in the example ("getting started") configuration. The tox tests fail when tox attempts to create the Python 3.6 environment:
py36 create: /home/circleci/repo/.tox/py36
ERROR: InterpreterNotFound: python3.6
It appears that when you use this kind of virtual environment then tox can only find an interpreter of the same version, whereas if using a conda virtual environment it somehow knows how to cook up the environments as long as they're lower versions. At least for my case (Python 3.6 and 3.7 environments for tox running in a Python 3.7 conda environment), this works fine.
The CircleCI configuration file I'm currently using looks like this:
# Python CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-python/ for more details
#
version: 2
jobs:
build:
docker:
# specify the version you desire here
# CircleCI maintains a library of pre-built images
# documented at https://circleci.com/docs/2.0/circleci-images/
- image: circleci/python:3.7
working_directory: ~/repo
steps:
- checkout
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ checksum "requirements.txt" }}
# fallback to using the latest cache if no exact match is found
- v1-dependencies-
- run:
name: install dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip install -e .
pip install tox
- save_cache:
paths:
- ./venv
key: v1-dependencies-{{ checksum "requirements.txt" }}
# run tests with tox
- run:
name: run tests
command: |
. venv/bin/activate
tox
- store_artifacts:
path: test-reports
destination: test-reports
What is the best practice for testing for multiple environments with tox on CircleCI? Should I move to using conda rather than venv within CircleCI, and if so how would I add this? Or is there a way to stay with venv, maybe by modifying its environment creation command?
Edit
I have now discovered that this is not specific to CircleCI, as I get a similar error when running this tox on Travis CI. Also, I have confirmed that this works as advertised using a Python 3.7 virtual environment created using venv on my local machine, both py36 and py37 environments run successfully.
If you use the multi-python docker image it allows you to still use tox for testing in multiple different environments, for example
version: 2
jobs:
test:
docker:
- image: fkrull/multi-python
steps:
- checkout
- run:
name: Test
command: 'tox'
workflows:
version: 2
test:
jobs:
- test
I have worked this out although not completely as it involves abandoning tox and instead using pytest for running tests for each Python version environment in separate workflow jobs. The CircleCI configuration (.circleci/config.yml) looks like this:
version: 2
workflows:
version: 2
test:
jobs:
- test-3.6
- test-3.7
- test-3.8
jobs:
test-3.6: &test-template
docker:
- image: circleci/python:3.6
working_directory: ~/repo
steps:
- checkout
- restore_cache:
keys:
- v1-dependencies-{{ checksum "requirements.txt" }}
- v1-dependencies-
- run:
name: install dependencies
command: |
python3 -m venv venv
. venv/bin/activate
pip install -e .
pip install coverage
pip install pytest
- save_cache:
paths:
- ./venv
key: v1-dependencies-{{ checksum "requirements.txt" }}
- run:
name: run tests
command: |
. venv/bin/activate
coverage run -m pytest tests
# store artifacts (for example logs, binaries, etc)
# to be available in the web app or through the API
- store_artifacts:
path: test-reports
test-3.7:
<<: *test-template
docker:
- image: circleci/python:3.7
test-3.8:
<<: *test-template
docker:
- image: circleci/python:3.8
On my project, I have issues with my test scripts on Travis CI. I installed the packages that I needed using conda, and then ran my test scripts. The build fails because of import errors. How can I fix this?
My travis YAML file is as such:
language: python
python:
- "2.7"
- "3.4"
- "nightly"
# command to install dependencies
# setup anaconda
before_install:
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- chmod +x miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- conda update -y conda
# install packages
install:
- conda install -y numpy scipy matplotlib networkx biopython
- echo $PATH
- which python
# command to run tests
script: py.test
One of my test scripts requires BioPython, and the test script fails because it cannot find biopython.
__________________ ERROR collecting test_genotype_network.py ___________________
test_genotype_network.py:1: in <module>
import genotype_network as gn
genotype_network.py:1: in <module>
from Bio import SeqIO
E ImportError: No module named 'Bio'
Is there a way to fix this?
Turns out the problem was py.test - I didn't install it in the conda environment. Adding pytest to the packages to install when creating the new environment made those import errors go away.