Develop Reference

ERROR: Could not find a version that satisfies the requirement while dockerizing a python project

I am trying to dockerize a python project. The problem is that docker is unable to find many of the packages that I am importing from requirements.txt file.
My OS is Xubuntu 20.04.
My docker version informations are as follow:
Client:
Version: 20.10.7
API version: 1.41
Go version: go1.13.8
Git commit: 20.10.7-0ubuntu1~20.04.1
Built: Wed Aug 4 22:52:25 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.8
Git commit: 20.10.7-0ubuntu1~20.04.1
Built: Wed Aug 4 19:07:47 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.2-0ubuntu1~20.04.2
GitCommit:
runc:
Version: 1.0.0~rc95-0ubuntu1~20.04.2
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
This is my dockerfile:
FROM pypy:3
WORKDIR /usr/src/app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD [ "pypy3", "./index.py" ]
The requirements.txt file is :
apturl==0.5.2
attrs==20.3.0
Automat==20.2.0
beautifulsoup4==4.9.3
blinker==1.4
blis==0.7.4
bs4==0.0.1
catalogue==2.0.3
catfish==1.4.13
certifi==2019.11.28
cffi==1.14.5
chardet==3.0.4
click==7.1.2
colorama==0.4.3
command-not-found==0.3
constantly==15.1.0
cryptography==3.4.7
cssselect==1.1.0
cupshelpers==1.0
cymem==2.0.5
dbus-python==1.2.16
defer==1.0.6
distro==1.4.0
distro-info===0.23ubuntu1
elasticsearch==7.13.0
entrypoints==0.3
h2==3.2.0
hpack==3.0.0
httplib2==0.14.0
hyperframe==5.2.0
hyperlink==21.0.0
idna==2.8
incremental==21.3.0
itemadapter==0.2.0
itemloaders==1.0.4
Jinja2==2.11.3
jmespath==0.10.0
joblib==1.0.1
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lightdm-gtk-greeter-settings==1.2.2
lxml==4.6.3
MarkupSafe==1.1.1
menulibre==2.2.1
mugshot==0.4.2
murmurhash==1.0.5
netifaces==0.10.4
numpy==1.20.2
oauthlib==3.1.0
olefile==0.46
onboard==1.4.1
packaging==20.9
pandas==1.2.4
parsel==1.6.0
pathy==0.5.2
pexpect==4.6.0
Pillow==7.0.0
preshed==3.0.5
priority==1.3.0
Protego==0.1.16
psutil==5.5.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycparser==2.20
pycups==1.9.73
pydantic==1.7.3
PyDispatcher==2.0.5
PyGObject==3.36.0
PyJWT==1.7.1
pymacaroons==0.13.0
pymongo==3.11.3
PyNaCl==1.3.0
pyOpenSSL==20.0.1
pyparsing==2.4.7
python-apt==2.0.0+ubuntu0.20.4.5
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
pytz==2021.1
PyYAML==5.3.1
queuelib==1.6.1
reportlab==3.5.34
requests==2.22.0
requests-unixsocket==0.2.0
scikit-learn==0.24.1
scipy==1.6.3
Scrapy==2.5.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
service-identity==18.1.0
sgt-launcher==0.2.5
simplejson==3.16.0
six==1.14.0
sklearn==0.0
smart-open==3.0.0
soupsieve==2.2.1
spacy==3.0.6
spacy-legacy==3.0.5
srsly==2.4.1
systemd-python==234
thinc==8.0.3
threadpoolctl==2.1.0
tqdm==4.60.0
Twisted==21.2.0
typer==0.3.2
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.8
w3lib==1.22.0
wadllib==1.3.3
wasabi==0.8.2
xcffib==0.8.1
xkit==0.0.0
zope.interface==5.4.0
I receive the error ERROR: Could not find a version that satisfies the requirement for the following packages:
apturl==0.5.2
catfish==1.4.13
command-not-found==0.3
cupshelpers==1.0
defer==1.0.6
distro-info===0.23ubuntu1
language-selector==0.1
lightdm-gtk-greeter-settings==1.2.2
menulibre==2.2.1
mugshot==0.4.2
onboard==1.4.1
PyGObject==3.36.0
python-apt==2.0.0+ubuntu0.20.4.5
python-debian===0.1.36ubuntu1
screen-resolution-extra==0.0.0
sgt-launcher==0.2.5
systemd-python==234
ubuntu-advantage-tools==27.0
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
xkit==0.0.0
(I tried to eliminate them one by one until the container could be compiled, but of course it did not run because of the absence of these packages).
I also tried to replace pypy by the normal python image, but I received the same error.
I tried to use the following dockerfile based on an ubuntu image :
FROM ubuntu:focal
SHELL ["/bin/bash", "-xe", "-c"]
ARG DEBIAN_FRONTEND=noninteractive
COPY . /code
ADD requirements.txt ./
RUN apt-get update -q \
&& apt-get install -y -q --no-install-recommends \
python3-wheel \
python3-pip \
gunicorn \
&& if [ -e requirements.txt ]; then \
python3 -m pip install --no-cache-dir \
--disable-pip-version-check \
-r requirements.txt; \
fi \
&& python3 -m pip install \
--no-cache-dir --disable-pip-version-check \
/code/ \
&& apt-get remove -y python3-pip python3-wheel \
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/* \
&& useradd _gunicorn --no-create-home --user-group
USER _gunicorn
WORKDIR /code
CMD ["gunicorn", \
"--bind", "0.0.0.0:8000", \
"hello_world:app"]
I also got the same result.
I tried to edit the docker DNS options by :
Adding DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" to the file /etc/default/docker.
Adding { "dns": ["192.168.1.254", "8.8.8.8"] } to the file /etc/docker/daemon.json.
I feel that I ran out of propositions :(. Does anyone have an idea of what can I do to make pip install all these packages in a python image container?
Thanks.

I think David Maze nailed it in the comments for some of the failures: a lot of the Python packages in the fail-list are installed via apt together with the Ubuntu packages they are part of. If you look them up (e.g. distro-info, ufw, command-not-found) on https://packages.ubuntu.com/, you'll see that many apt package ships with a Python library. (In fact, python-apt seems to be an outlier for being in PyPI as most of them are missing completely, and this answer starts with explaining why.)
The above only applies to packages that fail with the following exact message:
ERROR: Could not find a version that satisfies the
requirement <package-name> (from versions: none)
ERROR: No matching distribution found for <package-name>
Found it important to emphasize this because in my case (see at the bottom) this was the issue, and once I simply omitted the "Ubuntu-shipped packages", I just had to resolve other dependency issues, such as PyGObject (which was a tough one).
Also, there are other variants of the above error message (e.g., here and here), and this Stackoverflow thread, Could not find a version that satisfies the requirement <package>, has a lot of suggestions for those cases.
My case
Inherited a legacy Django app, and the dev did a pip freeze before leaving the project 2 years ago. I simply removed all the qualifiers (i.e., == + parts until the end of the line), and started the pip install process when I got these errors.
I have a theory that needs to be confirmed, but it was true in my case:
The pip freeze was issued on the production server, and it caught every Python package that could be pulled into the project (regardless whether it was used by the project or not).
So kind of like capturing the output of pip list vs. pip list --local.
I also had ufw on my list (among many others) that didn't make any sense as it is a firewall and this project was a simple, internal CMS. From then on, I kept hammering pip install, and crossed out errored out packages which were there because of (probably) unrelated Ubuntu installs.
(As for PyGObject and co., I was lucky that I used nix-shell to resolve dependency issues, so I only needed to issue
nix-shell -p cairo gobject-introspection
and it found the needed headers automatically. Even the database was dropped in a nix-shell.)

Install pydrill in Docker image

I have this docker file based on alpine that installs several packages with conda. At the end installs pydrill with pip as there's no conda installation.
from jcrist/alpine-dask
RUN /opt/conda/bin/conda update -n base -c defaults conda -y
RUN /opt/conda/bin/conda update dask
RUN /opt/conda/bin/conda install -c conda-forge dask-ml
RUN /opt/conda/bin/conda install scikit-learn -y
RUN /opt/conda/bin/conda install flask -y
RUN /opt/conda/bin/conda install waitress -y
RUN /opt/conda/bin/conda install gunicorn -y
RUN /opt/conda/bin/conda install pytest -y
RUN /opt/conda/bin/conda install apscheduler -y
RUN /opt/conda/bin/conda install matplotlib -y
RUN /opt/conda/bin/conda install pyodbc -y
USER root
RUN apk update
RUN apk add py-pip
RUN pip install pydrill
When I build the docker image everything works fine. But when I run the container the command line starts gunicorn, but it fails with the following message:
File "/code/app/service/cm/exec/run_drill.py", line 1, in <module>
from pydrill.client import PyDrill
ModuleNotFoundError: No module named 'pydrill'
Is this pip installation correct? This is the docker compose:
version: "3.0"
services:
web:
image: img-dask
volumes:
- vol_py_code:/code
- vol_dask_data:/data
- vol_dask_model:/model
ports:
- "5000:5000"
working_dir: /code
environment:
- app.config=/code/conf/py.app.json
- common.config=/code/conf/py.common.json
entrypoint:
- /opt/conda/bin/gunicorn
command:
- -b 0.0.0.0:5000
- --reload
- app.frontend.app:app
scheduler:
image: img-dask
ports:
- "8787:8787"
- "8786:8786"
entrypoint:
- /opt/conda/bin/dask-scheduler
worker:
image: img-dask
depends_on:
- scheduler
environment:
- PYTHONPATH=/code
- MODEL_PATH=/model/rfc_model.pkl
- PREPROCESSING_PATH=/model/data_columns.pkl
- SCHEDULER_ADDRESS=scheduler
- SCHEDULER_PORT=8786
volumes:
- vol_py_code:/code
- vol_dask_data:/data
- vol_dask_model:/model
entrypoint:
- /opt/conda/bin/dask-worker
command:
- scheduler:8786
volumes:
vol_py_code:
name: vol_py_code
vol_dask_data:
name: vol_dask_data
vol_dask_model:
name: vol_dask_model
UPDATE
If I run the command line inside the container, I can see that pydrill is installed, but my code does not see the library.
/code/conf # pip3 list
Package Version
---------- ---------
certifi 2020.12.5
chardet 4.0.0
idna 2.10
pip 18.1
pydrill 0.3.4
requests 2.25.1
setuptools 40.6.2
urllib3 1.26.4
You are using pip version 18.1, however version 21.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The problem is that pydrill and all other conda packages are in different environments. When the server starts, it doesn't see pydrill, only conda packages.
To fix the issue install pip itself in conda's environment:
from jcrist/alpine-dask
USER root
RUN /opt/conda/bin/conda create -p /pyenv -y
RUN /opt/conda/bin/conda install -p /pyenv dask scikit-learn flask waitress gunicorn \
pytest apscheduler matplotlib pyodbc -y
RUN /opt/conda/bin/conda install -p /pyenv -c conda-forge dask-ml -y
RUN /opt/conda/bin/conda install -p /pyenv pip -y
RUN /pyenv/bin/pip install pydrill

Can you try conda install pip instead of apk
Something like
from jcrist/alpine-dask
WORKDIR /opt/conda/bin
RUN conda update -n base -c defaults conda -y
RUN conda update dask
RUN install -c conda-forge dask-ml
RUN conda install stickit-learn flask waitress gunicorn \
pytest apscheduler matplotlib pydobc pip -y
RUN pip install pydrill

I have packaged pydrill for conda-forge, so you can simply conda install -c conda-forge pydrill.

Conda install dlib - PackageNotfoundError

Doing conda update --all and conda install dlib
gives this error:
.2.2 - haa95532_296.tar.zst . You probably need to delete and re - download or re - create this file . Message from libarchive was : \ n
\ nFile size could not be restored ' )
( myvenvpy ) ( base ) C : \ Users \ sir > Conda install dlib
Collecting package metadata ( current_repodata.json ) : done
Solving environment : failed with initial frozen solve . Retrying with flexible solve . collecting package metadata ( repodata.json ) : done
Solving environment : failed with initial frozen solve . Retrying with flexible solve .
PackagesNotFoundError : The following packages are not available from current channels :
dlib
Current channels :
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're looking for , navigate to https://anaconda.org and use the search bar at the top of the page .
( myvenvpy ) ( base ) C : \ Users \ sir >
Screenshot of error: Conda install dlib package not found error

you can also try this pip install dlib
or python3 -m pip install dlib
I think it will works for you

I have a dlib installer inside a Dockefile (for OpenCV) that looks like this:
sudo apt-get update && sudo apt-get install -y sudo && rm -rf /var/lib/apt/lists/*
sudo chmod -R 777 ./
sudo apt-get update && sudo apt-get install -y apt-utils build-essential cmake && sudo apt-get install -y libgtk-3-dev && sudo apt-get install -y libboost-all-dev && sudo apt-get install -y default-libmysqlclient-dev
pip install dlib
All of this because of the dependencies.

As per Anaconda: dlib
Try conda install -c conda-forge dlib

First of all check your python version. Upgrade or downgrade python accordingly .
install cmake :- pip install cmake
install dlib :- pip install dlib
for anaconda :-
install cmake :-conda install -c conda-forge cmake
install dlib :- conda install -c conda-forge dlib

Why does pip (using --find-links) still collect some dependencies from the internet despite a wheel existing for it in multi-stage docker build?

I have a multi-stage Dockerfile where I build wheels for my project's dependencies in a python image, then copy the wheels over to an alpine image and do a pip install -r ./wheels/requirements.txt --find-links ./wheels.
It seems to install from most wheels, except certain deps like numpy, spacy, and gensim where it reaches out to the internet to collect their zips/tars. Why can't pip find the links for them? The wheels are there. Normally it's whatever, but installing these deps on alpine takes a very long time.
Here's my Dockerfile:
FROM python:3.6.10 as builder
ENV PYTHONUNBUFFERED 1
WORKDIR /wheels
COPY ./requirements.txt /wheels/
RUN pip install -U pip \
&& pip wheel -r ./requirements.txt
FROM python:3.6.10-alpine
ENV PYTHONUNBUFFERED=1
RUN apk update
RUN apk add --no-cache \
--upgrade \
--repository http://dl-cdn.alpinelinux.org/alpine/edge/main \
make \
automake \
gcc \
g++ \
subversion \
python3-dev \
gettext \
libpq \
postgresql-client \
&& rm -rf /var/cache/apk/*
COPY --from=builder /wheels /wheels
RUN pip install -U pip \
&& pip install -r ./wheels/requirements.txt --find-links ./wheels \
&& rm -rf /wheels \
&& rm -rf /root/.cache/pip/*
WORKDIR /app
COPY ./ /app
COPY ./docker/entrypoint.sh /
ENTRYPOINT [ "/entrypoint.sh" ]
And my requirements.txt:
asgiref==3.2.3
Django==3.0.2
luhn==0.2.0
nltk==3.4.5
numpy==1.18.1
psycopg2==2.8.4
pytest==5.3.5
pytz==2019.3
spacy==2.2.3
sqlparse==0.3.0
yapf==0.29.0
gensim==3.8.1
Here's an example log output:
Looking in links: ./wheels
Processing /wheels/asgiref-3.2.3-py2.py3-none-any.whl
Processing /wheels/Django-3.0.2-py3-none-any.whl
Processing /wheels/luhn-0.2.0-py3-none-any.whl
Processing /wheels/nltk-3.4.5-py3-none-any.whl
Collecting numpy==1.18.1
Downloading numpy-1.18.1.zip (5.4 MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing wheel metadata: started
Preparing wheel metadata: finished with status 'done'
Processing /wheels/psycopg2-2.8.4-cp36-cp36m-linux_x86_64.whl
Processing /wheels/pytest-5.3.5-py3-none-any.whl
Processing /wheels/pytz-2019.3-py2.py3-none-any.whl
Collecting spacy==2.2.3
Downloading spacy-2.2.3.tar.gz (5.9 MB)
Basically I want numpy and spacy to be processed like Django and the other deps straight from the wheel.

numpy, spacy and genism are all packages combining Python and Cython and that interface with C/C++. In "plain" (glibc based) Linux distributions, these packages are precompiled to wheel binaries, which are downloaded and installed directly by pip.
However, on Alpine Linux, these packages are not available in their binary form, and must be compiled from source on the Alpine target during install.
The reason for this is that the wheel build system (PEP 517) does not support Alpine Linux. Linux wheel binaries are built for the manylinux target, which is not compatible with musl-libc based targets, such as Alpine and Void Linux. There's no wheel tag for Alpine, so even if you have a prebuilt Alpine wheel, you can't install its directly using pip. Instead, pip has to fetch the package source code and build it on the target.
The other packages are pure Python, so they are used as provided.
You can find more details on this issue, as well as the current status, in the following PyPa GitHub thread:
https://github.com/pypa/manylinux/issues/37

installing yfinance python package in docker container

I am trying to modify this docker compose setup to install a number of python packages including yfinance. In my mind there are three ways to install packages and i'd like to be able to use each one in docker:
install from pip requirements file
install from conda environments.yml file
install manually in environment by running install command (pip install yfinance or conda install -c conda-forge beautifulsoup4)
Here is a list of the problems I am having trying to modify this setup:
pip requirements file. - After altering this file the packages don't appear to be installed and instead the file is overridden with defaults on github.
conda environment file - Unable to create environment and install packages from environment.yml
manual install of package - entering a docker container shell using docker exec -it <containername> /bin/bash results in pip and conda returning command not found in bash.
Results so far:
All of the above methods have resulted in errors including 'command not found' or 'no module named yfinance' when importing in a notebook.
The only way I've been able to have any success so far is opening a notebook in the browser at localhost:8888 and creating a new notebook and running !pip install yfinance. However importing and executing the following code also results in an error making me think the package or dependencies haven't installed correctly.
import yfinance as yf
m = yf.Ticker("MSFT")
m.info
Here is my docker-compose file service
jupyter:
image: cmihai/jupyter:v1
container_name: 'joshnotebook'
hostname: jupyter
restart: 'always'
ports:
- '8888:8888'
env_file:
- notebook/config/jupyter.env
volumes:
- ${USERDIR}/docker/notebook:/home/user/notebook
build: ./notebook/services/jupyter
Here is my Dockerfile
FROM cmihai/python:v1
LABEL name="cmihai/jupyterlab" \
version="1" \
maintainer="Mihai Criveti <crivetimihai#gmail.com>" \
description="Anaconda Python with Jupyter Lab\
This container installs Anaconda Python 3 and Jupyter Lab."
SHELL ["/bin/bash", "-l", "-c"]
ENV DEBIAN_FRONTEND noninteractive
# PARAMETERS
ARG OS_PACKAGES="tzdata curl graphviz vim-nox gosu mc sqlite3 bash-completion sudo"
#ARG LATEX_PACKAGES="texlive texlive-latex-base texlive-latex-extra texlive-xetex texlive-generic-recommended texlive-fonts-extra texlive-fonts-recommended pandoc"
ARG CONDA_PACKAGES="jupyterlab pandas"
COPY condaenvironment.yml /tmp/condaenvironment.yml
# ROOT
USER root
# INSTALL OS PACKAGES
RUN apt-get update \
&& apt-get --no-install-recommends install --yes ${OS_PACKAGES} \ # ${LATEX_PACKAGES}\
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN conda install -c ranaroussi yfinance
# SWITCH TO REGULAR USER
RUN mkdir /home/user/notebook && chown -R user:user /home/user/notebook \
&& echo 'user ALL=NOPASSWD: ALL' >> /etc/sudoers
USER user
# INSTALL CONDA PACKAGES
RUN . ~/.profile \
&& unset SUDO_UID SUDO_GID SUDO_USER \
&& conda install --quiet --yes ${CONDA_PACKAGES} \
&& conda install -c anaconda requests \
&& conda install -c conda-forge lxml \
&& conda install -c conda-forge beautifulsoup4 \
&& conda install -c conda-forge sqlalchemy \
&& conda install -c anaconda mysql-connector-python \
&& conda install -c conda-forge seaborn \
&& conda install -c conda-forge altair \
&& conda clean -y -all
# INSTALL PIP PACKAGES
RUN . ~/.profile \
&& python -m pip install --no-cache-dir --upgrade pip \
&& python -m pip install yfinance
# WORKDIR
WORKDIR /home/user
# JUPYTERLAB EXTENSIONS
RUN jupyter labextension install jupyterlab-drawio \
&& jupyter labextension install jupyterlab_bokeh \
&& jupyter labextension install plotlywidget \
&& jupyter labextension install #jupyterlab/plotly-extension \
&& jupyter labextension install jupyterlab-chart-editor \
&& jupyter labextension install #jupyterlab/git \
&& jupyter serverextension enable --py jupyterlab_git \
&& jupyter labextension install #jupyterlab/latex \
&& jupyter labextension install #jupyterlab/toc \
&& jupyter labextension install #oriolmirosa/jupyterlab_materialdarker \
&& jupyter labextension install #jupyterlab/geojson-extension \
&& jupyter lab build
# COMMAND and ENTRYPOINT:
COPY start-jupyter.sh /home/user/start-jupyter.sh
CMD ["/home/user/start-jupyter.sh"]