Develop Reference

Dockerfile: pip install fails with requirements.txt (but succeeds with individual packages)

I'm trying to install some packages in a docker container, and there is a problem when installing from a requirements.txt file. This line:
RUN python3.8 -m pip install -r requirements.txt
fails with the error:
...
Collecting torch
Downloading torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl (735.5 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
torch from https://files.pythonhosted.org/packages/89/c1/72e9050d3e31e4df983f6e06799a1a4c896427c1e5645a6d810940944b60/torch-1.8.0-cp38-cp38-manylinux1_x86_64.whl#sha256=fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787 (from -r requirements.txt (line 3)):
Expected sha256 fa1e391cca3937d5dea31f31a1a80a01bd4a8062c039448c254bbf5a58eb0787
Got d5466637c17c3ae0c81c00d93a0b7c8d8428cfd216f54953a11d0788ea7b74fb
The requirements.txt file is the following:
numpy
opencv-python
torch
However, when installing these packages one at a time everything works fine:
RUN python3.8 -m pip install numpy
RUN python3.8 -m pip install opencv-python
RUN python3.8 -m pip install torch
Any ideas how to solve this?
*** EDIT ***
Dockerfile up to that point:
FROM public.ecr.aws/lambda/python:3.8
COPY requirements.txt ./

You could try a couple of things. Depending on your base image, you could run pip install in this way:
RUN pip install -r requirements.txt
Another option would be to change your requirements.txt such that it is version controlled. Then you can be sure you have compatible versions and is a good practice in general. Eg.:
torch==1.8.0
Try to run Docker again with without caches:
docker build -no-cache
Or you could check this answer:
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. when updating Django

Why is building a wheel for <package> with a (PEP 517) indication so slow in an alpine Linux docker environment

I'm installing numpy in an alpine Docker Python image, but it takes really very long when building the wheel at that point:
Building wheel for numpy (PEP 517) ... |
(the same appears for pandas e.g.)
What does this mean and why is it so slow?
I never faced a so slow install on Ubuntu, so I guess it may then be related to the alpine Linux environment.
Here is the Dockerfile:
FROM python:3.9.1-alpine3.12
WORKDIR /app
RUN python -m pip install --upgrade pip \
&& pip install -U setuptools wheel \
&& pip install -U numpy
Host machine is an Ubuntu 18.04 mid-range laptop.

not all docker images are born equal - each image packs s different set of packages.
that implies that it will take different effort to install required packages for running whatever you need (such as installing numpy)
you might like to read this

Azure ML Environment: install a package from a file?

I'm building an Environment object in the Azure Machine Learning service using the Python SDK, and everything is working fine except one Python package that installs from a URL. I'm wondering how to deal with it. This works:
my_env = Environment.from_conda_specification("trident", './environment.yml')
..but the Docker build fails on one of the packages, which installs from a file.
[91mERROR: Could not find a version that satisfies the requirement detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7)) (from versions: none)
[0m[91mERROR: No matching distribution found for detectron2==0.1.3+cu101 (from -r /azureml-environment-setup/condaenv.s5fi23rw.requirements.txt (line 7))
[0m[91m
Here's how I would install that package manually:
python -m pip install detectron2 -f / https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html
and I have another package that should install from github, like this:
pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'
I'm pretty ignorant about yaml files: is there a way to include complicated syntax like that in a yaml file?
I'm hoping to not have to re-build the environment locally and install from it (which is an alternative option), because I would have to reinstall CUDA to do so.
Thanks

Updating because who likes downvotes and someone might find this useful.
AML uses same spec for installing Conda packages as per: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#create-env-file-manually
OP could have applied something like:
# run: conda env create --file environment.yml
name: test-env
dependencies:
- python>=3.5
- anaconda
- pip
- pip:
# works for regular pip packages
- docx
- gooey
# for github
- git+https://github.com/facebookresearch/detectron2.git
# and for wheels
- https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html
However, I found it much easier to use a Docker image to load Detectron2 onto a container for AzureML because of CUDA/CuDNN compatibility fun.
FROM mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.1-cudnn7-ubuntu18.04
RUN apt update && apt install git -y && rm -rf /var/lib/apt/lists/*
RUN /opt/miniconda/bin/conda update -n base -c defaults conda
RUN /opt/miniconda/bin/conda install -y cython=0.29.15 numpy=1.18.1
# Install cocoapi, required for drawing bounding boxes
RUN git clone https://github.com/cocodataset/cocoapi.git && cd cocoapi/PythonAPI && python setup.py build_ext install
RUN pip install --user tensorboard cython
RUN pip install --user torch==1.5+cu101 torchvision==0.6+cu101 -f https://download.pytorch.org/whl/torch_stable.html
RUN pip install azureml-defaults
RUN pip install azureml-dataprep[fuse]
RUN pip install pandas pyarrow
RUN pip install opencv-python-headless
RUN pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/index.html```

pipenv and wheel/pre-compiled packages

I'm currently using a buildhost for third party packages running
$ pip3 wheel --wheel-dir=/root/wheelhouse -r /requirements.txt
After successful build I copy the directory /root/wheelhouse onto a new machine and install the compiled packages by running
$ pip3 install -r /requirements.txt --no-index --find-links=/root/wheelhouse
Is there something similar in pipenv?
Everything I found in combination with wheel are bug reports on GitHub.
Note that copying the venv directory is not an option. I'm using a docker container and want to install the packages system wide.

Cryptography Python Docker multistage build

I have a Python project that runs in a docker container and I am trying to convert to a multistage docker build process. My project depends on the cryptography package. My Dockerfile consists of:
# Base
FROM python:3.6 AS base
RUN pip install cryptography
# Production
FROM python:3.6-alpine
COPY --from=base /root/.cache /root/.cache
RUN pip install cryptography \
&& rm -rf /root/.cache
CMD python
Which I try to build with e.g:
docker build -t my-python-app .
This process works for a number of other Python requirements I have tested, such as pycrypto and psutil, but throws the following error for cryptography:
Step 5/6 : RUN pip install cryptography && rm -rf /root/.cache
---> Running in ebc15bd61d43
Collecting cryptography
Downloading cryptography-2.1.4.tar.gz (441kB)
Collecting idna>=2.1 (from cryptography)
Using cached idna-2.6-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography)
Using cached asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting six>=1.4.1 (from cryptography)
Using cached six-1.11.0-py2.py3-none-any.whl
Collecting cffi>=1.7 (from cryptography)
Downloading cffi-1.11.5.tar.gz (438kB)
Complete output from command python setup.py egg_info:
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.freenode.net.)
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-uyh9_v63/cffi/
Obviously I was hoping not to have to install any compiler on my production image. Do I need to copy across another directory other than /root/.cache?

There is no manylinux wheel for Alpine, so you need to compile it yourself. Below is pasted from documentation on installation. Install and remove build dependencies in the same command to only save the package to the docker image layer.
If you are on Alpine or just want to compile it yourself then
cryptography requires a compiler, headers for Python (if you’re not
using pypy), and headers for the OpenSSL and libffi libraries
available on your system.
Alpine Replace python3-dev with python-dev if you’re using Python 2.
$ sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev
If you get an error with openssl-dev you may have to use libressl-dev.
Docs can be found here

I hope, my answer will be useful.
You should use --user option for cryptography installing via pip in base stage. Example: RUN pip install --user cryptography. This option means, that all files will be installed in the .local directory of
the current user’s home directory.
COPY --from=base /root/.local /root/.local, because cryptography installed in /root/.local.
Thats all. Full example docker multistage
# Base
FROM python:3.6 AS base
RUN pip install --user cryptography
# Production
FROM python:3.6-alpine
COPY --from=base /root/.local /root/.local
RUN pip install cryptography \
&& rm -rf /root/.cache
CMD python