Develop Reference

Use conda in linux

I install anaconda in my ubuntu , but I don't use Conda.
I have this error and I don't fix this.
mamal#DESKTOP-H4B5SN5:~$ conda create -n pars
conda: command not found
mamal#DESKTOP-H4B5SN5:~$ /boot/anaconda3/bin/conda create -n pars
Collecting package metadata (current_repodata.json): done
Solving environment: done
UnavailableInvalidChannel: The channel is not accessible or is invalid.
channel name: pkgs/main
channel url: https://repo.anaconda.com/pkgs/main
error code: 403
You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.

Install conda package manually Databricks ML runtime

I have a Databricks ML Runtime cluster. I am trying to install fbprophet library using cluster init_script. I am following the example in the Databricks documentation.
#!/bin/bash
set -x
. /databricks/conda/etc/profile.d/conda.sh
conda info --envs
conda activate /databricks/python
conda install -y --offline /dbfs/conda_packages/linux-64/fbprophet.tar.bz2
But the init_script logs show that it cannot activate conda env, and cannot locate package from given path.
: invalid option
set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
bash: line 2: /databricks/conda/etc/profile.d/conda.sh
: No such file or directory
usage: conda [-h] [-V] command ...
conda: error: unrecognized arguments: --envs
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
PackagesNotFoundError: The following packages are not available from current channels:
- /dbfs/conda_packages/linux-64/fbprophet.tar.bz2
Current channels:
- https://repo.anaconda.com/pkgs/main/linux-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/free/linux-64
- https://repo.anaconda.com/pkgs/free/noarch
- https://repo.anaconda.com/pkgs/r/linux-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Can you please guide me what is wrong with the script, and how can I install the conda package from a path on DBFS.
Or is there any other way I can install conda packages. Because if I try to install using UI, or library API, it fails to install fbprophet package.
Regards

Pytest does not find pyYAML in Travis CI tests that have PyPI and conda package dependencies

I am trying to set up automatic testing for my Python package using Travis CI. My Python package depends on Iris as well as other packages such as PyYAML, numpy, etc. It also depends on a PyPI package (ScriptEngine). Now, I would like to set up a Travis CI environment using conda (to install Iris) and pip (to install the PyPI package as well as checking the requirements for PyYAML and numpy). I would then like to install my package using pip install ..
To test if this works, I have written one simple Pytest test that imports PyYAML.
I am currently trying to do this using this .travis.yml file:
language: python
python:
- "3.6"
- "3.7"
- "3.8"
# command to install dependencies
install:
- sudo apt-get update
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- source "$HOME/miniconda/etc/profile.d/conda.sh"
- hash -r
- conda config --set always_yes yes --set changeps1 no
- conda update -q conda
# Useful for debugging any issues with conda
- conda info -a
- conda env create -f tests/test-environment.yml python=$TRAVIS_PYTHON_VERSION
- conda activate test-environment
- conda install pip
- conda install -c conda-forge iris
- pip install -r requirements.txt
- pip install .
# command to run tests
script: pytest
Note: This is the first time for me to really work with Travis CI. This script is a mixture of examples from the Conda docs as well as the Travis CI docs.
Pytest then fails to import PyYAML (although it gets installed because of the requirements.txt as well as the Iris dependencies):
Here is the confirmation from the logs that it got installed:
Requirement already satisfied: pyYAML>=5.1 in /home/travis/miniconda/envs/test-environment/lib/python3.8/site-packages (from ece-4-monitoring==0.1.0) (5.3.1)
And this is the Error from Pytest:
$ pytest
============================= test session starts ==============================
platform linux -- Python 3.7.1, pytest-4.3.1, py-1.7.0, pluggy-0.8.0
rootdir: /home/travis/build/valentinaschueller/ece-4-monitoring, inifile:
collected 1 item / 1 errors
==================================== ERRORS ====================================
_________________ ERROR collecting tests/test_file_handling.py _________________
ImportError while importing test module '/home/travis/build/valentinaschueller/sciptengine-tasks-ecearth/tests/test_file_handling.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_file_handling.py:3: in <module>
import helpers.file_handling as file_handling
helpers/file_handling.py:1: in <module>
import yaml
E ModuleNotFoundError: No module named 'yaml'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.12 seconds ============================
The command "pytest" exited with 2.
If I try this exact setup using a conda virtual environment locally on my computer, I do not get this problem. Why does this not work on the Travis CI virtual machine?

As cel suggested in their comment: I could fix the problem by explicitly requiring pytest in the requirements.txt.
It is not necessary to activate test-environment in the script part. However, a very helpful tip was to use echo $(which pytest) && pytest instead of just pytest to check if pytest is installed at /home/travis/miniconda/envs/test-environment/bin/pytest.

Install python modules on Elastic Beanstalk with conda

Problem
I am trying to install python packages that have C dependencies on AWS Elastic Beanstalk (namely : fbprophet and xgboost)
Elastic Beanstalk python installs packages from requirements.txt by default with pip or pipenv on Amazon Linux 2
However, fbprophet and xgboost have dependencies in C that need to be compiled before installing them with pip. conda comes with these libraries precompiled so they are a lot easier to install with conda.
What I have tried
Here is my attempt at installing them with conda using a .config file in .ebextensions folder :
commands:
00_download_conda:
command: 'wget http://repo.continuum.io/archive/Anaconda3-2020.02-Linux-x86_64.sh'
test: test ! -d /anaconda
01_install_conda:
command: 'bash Anaconda3-2020.02-Linux-x86_64.sh -b -f -p /anaconda'
test: test ! -d /anaconda
02_reload_bash:
command: 'source ~/.bashrc'
03_create_home:
command: 'mkdir -p /home/wsgi'
04_conda_env:
command: 'conda env create -f environment.yml'
05_activate_env:
command: 'conda activate demo_forecast'
However this does not work and throws this error :
[2020-04-21T18:18:22.285Z] INFO [3699] - [Application update app-8acc-200421_201612#4/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_0_test_empty_dash/Command 03_conda_env] : Activity execution failed, because: /bin/sh: conda: command not found
(ElasticBeanstalk::ExternalInvocationError)
[2020-04-21T18:18:22.285Z] INFO [3699] - [Application update app-8acc-200421_201612#4/AppDeployStage0/EbExtensionPreBuild/Infra-EmbeddedPreBuild/prebuild_0_test_empty_dash/Command 03_conda_env] : Activity failed.
So it seems that sourcing the .bashrc does not create the conda alias
I am aware of this question and its answer , however it is a little bit old and does not provide enough details for my case, because it does not go through with installing packages using conda.
Another way would be to try and install and compile the C dependencies before pip installing the requirement, but I had no success there for now.
Thank you for the help !

Multiple python versions with anaconda and travis

I have a python project that makes use of libraries that needs to be built. Given that I use anaconda. I want to create a plan for travis that would let me test against multiple python versions and I am not able to do it. Here is what I have:
I want to test it against multiple python versions (e.g. 2.7, 3.5, 3.6)
I have requirements.yml file which looks like following:
channels:
- kne
dependencies:
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
My .travis.yml contains:
language: python
# sudo false implies containerized builds
sudo: false
python:
- 3.5
- 3.4
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
- conda env create -n testenv -f environment.yml python=$TRAVIS_PYTHON_VERSION
- source activate testenv
install:
- python setup.py install
script:
- python --version
- python -m pytest --cov=.
- codecov
If I put python version into environment.yml it works fine but I can't use multiple python versions. For me, it seems to if -f is provided, it ignores any additional packages listed for conda env create.
Also, adding - conda install -n testenv python=$TRAVIS_PYTHON_VERSION after env creation does not work.
UnsatisfiableError: The following specifications were found to be in conflict:
- functools32 -> python 2.7.*
- python 3.5*
Use "conda info <package>" to see the dependencies for each package.
What am I supposed to do, in order make it work?
// If you would like to see more details, it is available here: https://travis-ci.org/mbednarski/Chiron/jobs/220644726

You can use sed to modify the python dependency in environment.yml file before creating the conda environment.
Include python in the environment.yml:
channels:
- kne
dependencies:
- python=3.6
- numpy
- pytest
- numpy
- scipy
- matplotlib
- seaborn
- pybox2d
- pip:
- gym
- codecov
- pytest
- pytest-cov
And then modify your .travis.yml:
before_install:
# Here we download miniconda and install the dependencies
- export MINICONDA=$HOME/miniconda
- export PATH="$MINICONDA/bin:$PATH"
- hash -r
- wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -f -p $MINICONDA
- conda config --set always_yes yes
- conda update conda
- conda info -a
- echo "Python version var"
- echo $TRAVIS_PYTHON_VERSION
# Edit the environment.yml file for the target Python version
- sed -i -E 's/(python=)(.*)/\1'$TRAVIS_PYTHON_VERSION'/' ./environment.yml
- conda env create -n testenv -f environment.yml
- source activate testenv
The sed regex will replace the text python=3.6 with the equivalent for the target python version.
BTW: I see in your repository you've worked around this by using multiple environment.yml files. This seems reasonable, even necessary for certain dependencies, but perhaps tedious to maintain for many Python versions.