I'm building an Environment object in the Azure Machine Learning service using the Python SDK, and everything is working fine except one Python package that installs from a URL. I'm wondering how to deal with it. This works:
my_env = Environment.from_conda_specification("trident", './environment.yml')
..but the Docker build fails on one of the packages, which installs from a file.
[91mERROR: Could not find a version that satisfies the requirement detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7)) (from versions: none)
[0m[91mERROR: No matching distribution found for detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7))
[0m[91m
Here's how I would install that package manually:
python -m pip install detectron2 -f / https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html
and I have another package that should install from github, like this:
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
I'm pretty ignorant about yaml files: is there a way to include complicated syntax like that in a yaml file?
I'm hoping to not have to re-build the environment locally and install from it (which is an alternative option), because I would have to reinstall CUDA to do so.
Thanks
Updating because who likes downvotes and someone might find this useful.
AML uses same spec for installing Conda packages as per: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually
OP could have applied something like:
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
# works for regular pip packages
- docx
- gooey
# for github
- git+https://github.com/facebookresearch/detectron2.git
# and for wheels
- https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html
However, I found it much easier to use a Docker image to load Detectron2 onto a container for AzureML because of CUDA/CuDNN compatibility fun.
FROM mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04
RUN apt update && apt install git -y && rm -rf /var/lib/apt/lists/*
RUN /opt/miniconda/bin/conda update -n base -c defaults conda
RUN /opt/miniconda/bin/conda install -y cython=0.29.15 numpy=1.18.1
# Install cocoapi, required for drawing bounding boxes
RUN git clone https://github.com/cocodataset/cocoapi.git && cd cocoapi/PythonAPI && python setup.py build_ext install
RUN pip install --user tensorboard cython
RUN pip install --user torch==1.5+cu101 torchvision==0.6+cu101 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip install azureml-defaults
RUN pip install azureml-dataprep[fuse]
RUN pip install pandas pyarrow
RUN pip install opencv-python-headless
RUN pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html```
Related
I have used prior versions of torchvision in my docker images in the past however now I am trying to install torchvision version 0.13.0 onto the image (OS = ubuntu 18.04) however anytime my requirements.txt file hits the line
torchvision==0.13.0
I get this error
Could not find a version that satisfies the requirement torchvision==0.13.0
I installed and used the same version on google colabs so I know the it is available, I just need a way to get it onto ubuntu via docker.
I also tried installing the same version directly on a ubuntu 18.04 VM (same error) so I think it is an issue with ubuntu.
I have tried
adding line pip install torchvision==0.13.0 to requirements.txt
Solved by installing torchvision via "RUN" command in dockerfile.
Adding the following lines in the dockerfile fixed the issue:
RUN apt-get update && apt-get install -y build-essential && apt-get install -y wget && apt-get clean
# Install miniconda
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && /bin/bash ~/miniconda.sh -b -p /opt/conda
# Put conda in path so we can use conda activate
ENV PATH=$CONDA_DIR/bin:$PATH
# Install torchvision
RUN conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
I am trying to run a model written in Python and packaged in Docker. The model is available here: https://github.com/pennsignals/chime_sims
The model has a requirements.txt file and a number of python files. Requirements file is supposed to install a number of python packages, including configargparse. The docker file runs, but when it gets to the python code that uses configargparse, I get an error that such a package does not exist.
I added a line that I think should install it:
RUN pip -m install --upgrade configargparse
and then I import it in the python code. But I still get an error stating that no such package exits.
Here is the Docker code:
FROM continuumio/miniconda3
WORKDIR /chime_sims
RUN conda update -yq -n base -c defaults conda
RUN conda create -yq -n chime_sims python=3.7 pip pandas matplotlib scipy numpy seaborn
COPY ./requirements.txt .
FROM python
RUN python -m pip install --upgrade pip
RUN pip -m install --upgrade configargparse
RUN pip freeze > requirements.txt .
RUN py -m pip install -r requirements.txt
And here is the list from requirements.txt:
ConfigArgParse
configargparse
gitpython
seaborn
numpy
pandas
gvar
lsqfit
I suspect I'll have problems installing gvar and lsqfit if I ever get configargparse to work. I had trouble installing those packages outside of docker, but it finally worked.
Any ideas on how I can fix it?
thank you,
i.
I'm trying to install some packages in a docker container, and there is a problem when installing from a requirements.txt file. This line:
RUN python3.8 -m pip install -r requirements.txt
fails with the error:
...
Collecting torch
Downloading torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl (735.5 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
torch from https://files.pythonhosted.org/packages/89/c1/72e9050d3e31e4df983f6e06799a1a4c896427c1e5645a6d810940944b60/torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl#sha256=fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787 (from -r requirements.txt (line 3)):
Expected sha256 fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787
Got d5466637c17c3ae0c81c00d93a0b7c8d8428cfd216f54953a11d0788ea7b74fb
The requirements.txt file is the following:
numpy
opencv-python
torch
However, when installing these packages one at a time everything works fine:
RUN python3.8 -m pip install numpy
RUN python3.8 -m pip install opencv-python
RUN python3.8 -m pip install torch
Any ideas how to solve this?
*** EDIT ***
Dockerfile up to that point:
FROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt ./
You could try a couple of things. Depending on your base image, you could run pip install in this way:
RUN pip install -r requirements.txt
Another option would be to change your requirements.txt such that it is version controlled. Then you can be sure you have compatible versions and is a good practice in general. Eg.:
torch==1.8.0
Try to run Docker again with without caches:
docker build -no-cache
Or you could check this answer:
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. when updating Django
I cannot figure out why when installing my python dependencies from requirements.txt pip don't complain, but when I do it from docker container, I got the following error message:
The requirements.txt content:
Flask~=1.1
grpcio
grpcio-tools
protobuf
iexfinance
numpy
pandas
pandas_datareader
pymongo
I've created my container like below:
docker run -it -p 8080:50051 -v ${pwd}:/app -w "/app" python:3.8-alpine
I've tried to install my dependencies using this command:
pip install -r requirements.txt
Bellow some screenshot:
Alpine Linux uses musl C, but most python wheel files are compiled for glib C. Therefore, packages that have extensions written in C/C++ need to be compiled. If you do not have a compiler installed, you will get an error.
Instead of installing a compiler and dependencies that packages might require at compile time, I suggest using a python Docker image that is not based on Alpine. For example, you can use python:3.8-slim or python:3.8, and python packages that ship Linux wheels will not have to be compiled. All of the packages listed in OP's requirements.txt can be installed from pre-compiled wheels if using python:3.8-slim.
So you can use these commands
docker run -it -p 8080:50051 -v ${pwd}:/app -w "/app" python:3.8-slim
pip install -r requirements.txt
If you are concerned about the size of the resulting image, you can also use the --no-cache-dir flag in pip install to disable caching.
The solution was to update alpine-SDK, which is a "meta-package" that pulls in the essential packages used to build new packages."
apk add --update alpine-sdk
I found the solution here:
Github: docker alpine issues
I've been trying to install Yaafe Library on my linux system, but I'm unable to do it as I can't compile the yaafe source using ccmake. Does anyone have the detailed step-by-step procedure for the same?
I tried to follow the instructions, which failed for me during compile. The ccmake can be replace by cmake. I could not install libhdf5-serial-1.8.4, because it was integrated in the main package.
Alternative approach
An alternative to yaafe would be librosa, which has the advantage of being available via PyPi. You install it via (assuming Debian/Ubuntu)
apt-get install pip
(for the PyPi client), and
pip install librosa
if you follow their advice and install scikits.samplerate, you also need libsamplerate0-dev:
apt-get install libsamplerate0-dev
The home page of the library includes a thorough manual for compiling yaafe.
I am citing the beginning here:
$ sudo apt-get install cmake cmake-curses-gui libargtable2-0 libargtable2-dev libsndfile1 libsndfile1-dev libmpg123-0 libmpg123-dev libfftw3-3 libfftw3-dev liblapack-dev libhdf5-serial-dev libhdf5-serial-1.8.4
$ mkdir build
$ cd build
$ ccmake -DCMAKE_PREFIX_PATH=<lib-path> -DCMAKE_INSTALL_PREFIX=<install-path> ..
see the rest there.
That's what I had to do in Ubuntu 14.04 to get it working:
sudo apt-get install build-essential -y
sudo apt-get install libeigen3-dev
sudo apt-get install cmake cmake-curses-gui libargtable2-0 libargtable2-dev libsndfile1 libsndfile1-dev libmpg123-0 libmpg123-dev libfftw3-3 libfftw3-dev liblapack-dev libhdf5-serial-dev libhdf5-7
Download Yaafe from here:https://github.com/Yaafe/Yaafe/archive/master.zip
Extract Yaafe-master.zip
Inside Yaafe-master directory:
mkdir build
cd build
ccmake ..
make
sudo make install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
Also add this path to your IDE (Enviromental Variables) if yaafe does not work with it.
I just installed it using Anaconda, and it was extremely easy! Just install Anaconda like the link tells you to. On the last step, I recommend you allow Anaconda to modify your PATH so that when you type python on the command line, it uses the Anaconda version of Python. Then restart your terminal, just to make sure it's using the Anaconda stuff you just installed.
Then, assuming you're using Ubuntu, you just need to type the following command:
conda install --channel https://conda.anaconda.org/Yaafe yaafe