I'm using databricks for Spark and one of the packages I want to install is cyipopt for Python. cyipopt documentation recommends installation of the package from conda-forge using the command
conda install -c conda-forge cyipopt
The problem is, databricks has recently disabled conda due to the some terms and conditions update of Anaconda, and only supports pip where cyipopt isn't available. So, I'm looking for alternate methods of installing this package with no luck so far. One of the things I tried is to install conda in databricks using:
%sh mkdir -p ./miniconda3
%sh wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ./miniconda3/miniconda.sh
%sh bash ./miniconda3/miniconda.sh -b -u -p ./miniconda3
%sh ./miniconda3/bin/conda install -c conda-forge cyipopt
The above commands installed conda successfully, but the last command to install cyipopt kept running for multiple hours when I tried it many many times, so I just assumed it failed.
Secondly, I manually downloaded all the tar.bz2 files of cyipopt from conda-forge and its dependencies and was able to manually install all of them successfully including cyipopt, but looks like they are not accessible by the default Python of databricks (likely because conda installed them in a different env).
Can I get conda to install packages in the same env as the default Python instead of in its own new environment? Or is there any other easier way to install cyipopt package in the default environment?
I got a solution from this article which talks about a similar process for Colab, https://towardsdatascience.com/conda-google-colab-75f7c867a522
Note that %sh in the below code is the magic command of databricks that is used to run shell commands.
## install conda
%sh mkdir -p ./miniconda3
# ## download miniconda compatible with below python
# ## from https://docs.conda.io/en/latest/miniconda.html
import sys
print(sys.version)
## change Miniconda download link relevant to your python version, the below link is for Python 3.8 only
%sh wget https://repo.anaconda.com/miniconda/Miniconda3-py38_4.12.0-Linux-x86_64.sh -O ./miniconda3/miniconda.sh
%sh bash ./miniconda3/miniconda.sh -b -u -p /usr/local
%sh which conda
%sh rm -rf ./miniconda3/miniconda.sh
%sh conda init bash
%sh conda init zsh
%sh conda --version
## update conda
## change python version below to your python version
%sh conda install --channel defaults conda python=3.8 --yes
%sh conda update --channel defaults --all --yes
%sh conda --version
## add conda packages path to sys.path so that Python can find those packages
import sys
sys.path
## change python version to your respective python version
_ = (sys.path.append("/usr/local/lib/python3.8/site-packages"))
## install your package
%sh conda install -c conda-forge cyipopt --yes
import cyipopt
Related
I have used prior versions of torchvision in my docker images in the past however now I am trying to install torchvision version 0.13.0 onto the image (OS = ubuntu 18.04) however anytime my requirements.txt file hits the line
torchvision==0.13.0
I get this error
Could not find a version that satisfies the requirement torchvision==0.13.0
I installed and used the same version on google colabs so I know the it is available, I just need a way to get it onto ubuntu via docker.
I also tried installing the same version directly on a ubuntu 18.04 VM (same error) so I think it is an issue with ubuntu.
I have tried
adding line pip install torchvision==0.13.0 to requirements.txt
Solved by installing torchvision via "RUN" command in dockerfile.
Adding the following lines in the dockerfile fixed the issue:
RUN apt-get update && apt-get install -y build-essential && apt-get install -y wget && apt-get clean
# Install miniconda
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && /bin/bash ~/miniconda.sh -b -p /opt/conda
# Put conda in path so we can use conda activate
ENV PATH=$CONDA_DIR/bin:$PATH
# Install torchvision
RUN conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
I am working on a project which requirements need python 3.7 and TensorFlow 2.3.1. The problem, I have a MacBook Pro with M1 chip. I was able to install and run TF 2.4.
However, I am running into more complicated compatibility issues.
Does anyone know how can I solve this?
M1 has a compatibility issue with TensorFlow. There is a workaround provided by Apple and other blogs. I have recently tried the same and have provided the summary below:
Tensorflow:
OS: BigSur(11.2.3)
Install command-line tools:
xcode-select --install
Install miniforge:
https://github.com/conda-forge/miniforge#miniforge3
After download run
chmod +x ~/<dir>/Miniforge3-MacOSX-arm64.sh
sh ~/<dir>/Miniforge3-MacOSX-arm64.sh
dir: directory to which miniforge is downloaded.
Or
brew install miniforge
After download change source to miniforge3
source ~/miniforge3/bin/activate
Download the environment.yml file from
https://github.com/mwidjaja1/DSOnMacARM
Setup a new conda environment using the yml file:
conda env create --file=environment.yml --name env_name
conda activate env_name
Install tensorflow dependencies:
conda install -c apple tensorflow-deps
Install base tensorflow:
python -m pip install tensorflow-macos
Install tensorflow-metal plugin:
python -m pip install tensorflow-metal
Possible Issues:
Miniforge uses conda-forge to install packages
Method to route channel to conda-forge
conda install -c conda-forge matplotlib
conda install or pip install might not work
Packages available for libraries in conda-forge:
https://anaconda.org/conda-forge
Only these will work.
Sources:
https://developer.apple.com/metal/tensorflow-plugin/
https://towardsdatascience.com/installing-tensorflow-on-the-m1-mac-410bb36b776
https://github.com/mwidjaja1/DSOnMacARM
I installed snorkel using conda and when I try to run - from snorkel.labeling import labeling_function it throws the following error - ModuleNotFoundError: No module named 'snorkel.labeling'.
I tried looking up for a solution on Github, but unfortunately, I couldn't follow through. I have also tried installing nb_conda_kernels, to make all your conda environments available in jupyterbut it didn't help. Also tried creating a separate env, installing snorkel and launching jupyter notebook from the snorkel's environment, and it didn't work either. Any form of help is much appreciated!
Thanks in advance!
Try these install instructions:
conda create --yes -n snorkel
conda activate snorkel
conda install pytorch==1.1.0 -c pytorch
conda install snorkel==0.9.0 -c conda-forge
Or these, listed here:
# [OPTIONAL] Activate a virtual environment
conda create --yes -n spam python=3.6
conda activate spam
# Install requirements (both shared and tutorial-specific)
pip install environment_kernels
# We specify PyTorch here to ensure compatibility, but it may not be necessary.
conda install pytorch==1.1.0 -c pytorch
conda install snorkel==0.9.5 -c conda-forge
pip install -r spam/requirements.txt
# Launch the Jupyter notebook interface
jupyter notebook spam
Please try the following:
For pip users
pip install snorkel
For conda users
conda install snorkel -c conda-forge
I'm building an Environment object in the Azure Machine Learning service using the Python SDK, and everything is working fine except one Python package that installs from a URL. I'm wondering how to deal with it. This works:
my_env = Environment.from_conda_specification("trident", './environment.yml')
..but the Docker build fails on one of the packages, which installs from a file.
[91mERROR: Could not find a version that satisfies the requirement detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7)) (from versions: none)
[0m[91mERROR: No matching distribution found for detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7))
[0m[91m
Here's how I would install that package manually:
python -m pip install detectron2 -f / https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html
and I have another package that should install from github, like this:
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
I'm pretty ignorant about yaml files: is there a way to include complicated syntax like that in a yaml file?
I'm hoping to not have to re-build the environment locally and install from it (which is an alternative option), because I would have to reinstall CUDA to do so.
Thanks
Updating because who likes downvotes and someone might find this useful.
AML uses same spec for installing Conda packages as per: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually
OP could have applied something like:
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
# works for regular pip packages
- docx
- gooey
# for github
- git+https://github.com/facebookresearch/detectron2.git
# and for wheels
- https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html
However, I found it much easier to use a Docker image to load Detectron2 onto a container for AzureML because of CUDA/CuDNN compatibility fun.
FROM mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04
RUN apt update && apt install git -y && rm -rf /var/lib/apt/lists/*
RUN /opt/miniconda/bin/conda update -n base -c defaults conda
RUN /opt/miniconda/bin/conda install -y cython=0.29.15 numpy=1.18.1
# Install cocoapi, required for drawing bounding boxes
RUN git clone https://github.com/cocodataset/cocoapi.git && cd cocoapi/PythonAPI && python setup.py build_ext install
RUN pip install --user tensorboard cython
RUN pip install --user torch==1.5+cu101 torchvision==0.6+cu101 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip install azureml-defaults
RUN pip install azureml-dataprep[fuse]
RUN pip install pandas pyarrow
RUN pip install opencv-python-headless
RUN pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html```
I would like to install python 3.5 packages so they would be available in Jupyter notebook with pyspark3 kernel.
I've tried to run the following script action:
#!/bin/bash
source /usr/bin/anaconda/envs/py35/bin/activate py35
sudo /usr/bin/anaconda/envs/py35/bin/conda install -y keras tensorflow theano gensim
but the packages get installed on python 2.7 and not in 3.5
You can use:
#!/usr/bin/env bash
/usr/bin/anaconda/envs/py35/bin/conda install -y ...
or (if conda doesn't have the package)
/usr/bin/anaconda/envs/py35/bin/pip install ...
According to the help information of the command conda help install, there is a parameter -n ENVIRONMENT to do the package installation on a specified environment like envs/py35, the explaination as below.
$ conda help install
.......
-n ENVIRONMENT, --name ENVIRONMENT
Name of environment (in
/home/sshuser/.conda/envs:/usr/bin/anaconda/envs).
So you can install these packages you want via the command as below.
sudo /usr/bin/anaconda/bin/conda install -y keras tensorflow theano gensim -n py35
Hope it helps.
Have you tried installing using pip?
In some cases where you have both Python 2 and Python 3, you have to run pip3 instead of just pip to invoke pip for Python 3.