Develop Reference

Pip install requirements PyWavelets & scipy takes forever and then fails

I try to build a docker container with a python script on a raspberry pi.
I' using the arm32v7/python latest image (3.12), but also tried it with 3.9 and 3.10 images.
In my Dockerfile I first update pip and then try to install my requirements. But when it tries to install PyWavelets or scipy it wont move further. I just get "installing build dependencies: still running".
Downloading PyWavelets-1.4.1.tar.gz (4.6 MB)
=> => # 4.6/4.6 MB 11.9 MB/s eta 0:00:00
=> => # Installing build dependencies: started
=> => # Installing build dependencies: still running...
After 30 mins it fails with
� Building wheel for ninja (pyproject.toml) did not run successfully.
#0 2857.1 exit code: 1
#0 2857.1 > [11 lines of output]
#0 2857.1 Traceback (most recent call last):
#0 2857.1 File "/tmp/pip-build-env-roddjnvw/overlay/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 612, in setup
#0 2857.1 cmkr = cmaker.CMaker(cmake_executable)
#0 2857.1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 2857.1 File "/tmp/pip-build-env-roddjnvw/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 149, in __init__
#0 2857.1 self.cmake_version = get_cmake_version(self.cmake_executable)
#0 2857.1 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#0 2857.1 File "/tmp/pip-build-env-roddjnvw/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 104, in get_cmake_version
#0 2857.1 raise SKBuildError(
#0 2857.1
#0 2857.1 Problem with the CMake installation, aborting build. CMake executable is cmake
#0 2857.1 [end of output]
I already tried to install them via their apt packages (python3-pywt, python3-scipy) and skip the ones in my requirements.txt, but then I get an error in the installer.
I also tried to use https://www.piwheels.org/simple as index url in my install command but then it wont find any scipy installation candidate.
My Dockerfile:
FROM arm32v7/python
RUN apt-get update && apt-get -y install vim python3-pywt python3-scipy
WORKDIR /app
COPY main.py /app/main.py
COPY requirements.txt /app/requirements.txt
RUN pip install --upgrade pip
#RUN pip install --index-url=https://www.piwheels.org/simple --no-cache-dir -r requirements.txt
RUN pip install -r requirements.txt
My requirements.txt
certifi==2022.12.7
charset-normalizer==2.1.1
idna==3.4
mysql-connector-python==8.0.25
numpy==1.24.0
Pillow==9.3.0
protobuf==3.20.1
python-dotenv==0.21.0
PyWavelets==1.4.1
requests==2.28.1
scipy==1.9.3
update-checker==0.18.0
urllib3==1.26.13
websocket-client==1.4.2

python, mariadb library and his connector inside a docker container

I'm trying to create a container for my python project that uses mariaDB. Here there are dockerfile's lines related to the problem:
FROM python:3.10.6-slim-buster
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && \
apt-get install -yq apt-utils && \
apt-get install -yq procps && \
apt-get install -yq nano && \
apt-get install -yq libmariadb3 libmariadb-dev
RUN rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir -r requirements.txt
In my requirements.txt I have:
mariadb==1.1.4
When I run the docker build it returns me this error:
Collecting mariadb==1.1.4
Downloading mariadb-1.1.4.zip (97 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.4/97.4 kB 10.8 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-2uzbllpk/mariadb_4588dbf4d5fc48b989ebd2605c2d7d1c/setup.py", line 27, in <module>
cfg = get_config(options)
File "/tmp/pip-install-2uzbllpk/mariadb_4588dbf4d5fc48b989ebd2605c2d7d1c/mariadb_posix.py", line 64, in get_config
print('MariaDB Connector/Python requires MariaDB Connector/C '
TypeError: not enough arguments for format string
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
As you can see, I installed the required packages for the connector via apt-get, the installation works without error, but I am still having this problem.
Do you have any idea?
Thank you.

Sorry, my bad - When formatting the code in 1.1.4 with Flake8 (PEP-8) the brackets around the tuple accidentally disappeared, this is already fixed in 1.1.5 (not released yet)
Minimum required Connector/C version is 3.2.4. Buster has MariaDB Server 10.3, corresponding C/C version is 3.1.x - so instead of buster you should bookworm (Server version 10.6.8, C/C version 3.2.8).

Error while installing psycopg2 using pip [duplicate]

This question already has answers here:
pg_config executable not found
(54 answers)
Closed 6 months ago.
I am trying to run docker in django using this command docker build -t myimage . Now the docker file tries to run the RUN pip install -r /app/requirements.txt --no-cache-dir but when ot gets to the Downloading psycopg2-2.9.3.tar.gz (380 kB) section, it throws the error.
NOTE: i do not have psycopg2 in my requirements.txt file only the psycopg2-binary.
requirements.txt file
...
dj-database-url==0.5.0
Django==3.2.7
django-filter==21.1
django-formset-js-improved==0.5.0.2
django-heroku==0.3.1
psycopg2-binary
python-decouple==3.5
...
Downloading pytz-2022.2.1-py2.py3-none-any.whl (500 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 500.6/500.6 kB 2.6 MB/s eta 0:00:00
Collecting psycopg2
Downloading psycopg2-2.9.3.tar.gz (380 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 380.6/380.6 kB 2.7 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
running egg_info
creating /tmp/pip-pip-egg-info-383i9hb2/psycopg2.egg-info
writing /tmp/pip-pip-egg-info-383i9hb2/psycopg2.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-383i9hb2/psycopg2.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-383i9hb2/psycopg2.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-383i9hb2/psycopg2.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.
For further information please check the 'doc/src/install.rst' file (also at
<https://www.psycopg.org/docs/install.html>).
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
The command '/bin/sh -c pip install -r /app/requirements.txt --no-cache-dir' returned a non-zero code: 1
Dockerfile
FROM python:3.8.13-slim-buster
WORKDIR /app
COPY ./my_app ./
RUN pip install --upgrade pip --no-cache-dir
RUN pip install -r /app/requirements.txt --no-cache-dir
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]
# CMD ["gunicorn", "main_app.wsgi:application", "--bind"]

You need to install the system dependencies (pg_*) if you want to use psycopg2, otherwise, you can use the all-in-one package that include them by remplacing psycopg2 by psycopg2-binary

Installing ruamel.yaml.clib with docker

I have a small project in django rest framework and I want to dockerize it. In my requirements.txt file there is a package called ruamel.yaml.clib==0.2.6. While downloading all other requirements is successfull, there is a problem when it tries to download this package.
#11 208.5 Collecting ruamel.yaml.clib==0.2.6
#11 208.7 Downloading ruamel.yaml.clib-0.2.6.tar.gz (180 kB)
#11 217.8 ERROR: Command errored out with exit status 1:
#11 217.8 command: /usr/local/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b8oectgw/ruamel-yaml-clib_517e9b3f18a94ebea71ec88fbaece43a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-b8oectgw/ruamel-yaml-clib_517e9b3f18a94ebea71ec88fbaece43a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-n2gr5j35
#11 217.8 cwd: /tmp/pip-install-b8oectgw/ruamel-yaml-clib_517e9b3f18a94ebea71ec88fbaece43a/
#11 217.8 Complete output (3 lines):
#11 217.8 sys.argv ['/tmp/pip-install-b8oectgw/ruamel-yaml-clib_517e9b3f18a94ebea71ec88fbaece43a/setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-n2gr5j35']
#11 217.8 test compiling /tmp/tmp_ruamel_erx3efla/test_ruamel_yaml.c -> test_ruamel_yaml compile error: /tmp/tmp_ruamel_erx3efla/test_ruamel_yaml.c
#11 217.8 Exception: command 'gcc' failed: No such file or directory
#11 217.8 ----------------------------------------
#11 217.8 WARNING: Discarding https://files.pythonhosted.org/packages/8b/25/08e5ad2431a028d0723ca5540b3af6a32f58f25e83c6dda4d0fcef7288a3/ruamel.yaml.clib-0.2.6.tar.gz#sha256=4ff604ce439abb20794f05613c374759ce10e3595d1867764dd1ae675b85acbd (from https://pypi.org/simple/ruamel-yaml-clib/) (requires-python:>=3.5). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
#11 217.8 ERROR: Could not find a version that satisfies the requirement ruamel.yaml.clib==0.2.6 (from versions: 0.1.0, 0.1.2, 0.2.0, 0.2.2, 0.2.3, 0.2.4, 0.2.6)
#11 217.8 ERROR: No matching distribution found for ruamel.yaml.clib==0.2.6
However, there is no problem when I download this package without docker. Any suggestions?
Here is Dockerfile:
FROM python:3-alpine
# set work directory
WORKDIR /usr/src/app
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# install dependencies
RUN pip install -U pip setuptools wheel ruamel.yaml ruamel.yaml.clib==0.2.6
COPY ./requirements.txt .
RUN pip install --default-timeout=100 -r requirements.txt
# copy project
COPY . .
Here is compose file
version: '3.8'
services:
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/usr/src/app
ports:
- 8000:8000
env_file:
- ./.env.dev
EDIT
As mentioned in comments by #Anthon, the problem was related to alpine. I used python:3.9-slim-buster instead in Dockerfile and problem solved!

I see that OP has moved on from Alpine to solve this issue, but for whatever reason you need/want/wish to stay on Alpine, you can install gcc, musl-dev, and python3-dev:
⋮
RUN apk add --no-cache gcc musl-dev python3-dev
⋮
RUN pip install ruamel.yaml.clib …

I think the problem is with the way your Dockerfile tries to install ruamel.yaml.clib. It should be installed using pip (just as documented for the ruamel.yaml).
I suggest you take it out of the requirements.txt and explicitly do a
pip install -U pip setuptools wheel ruamel.yaml.clib==0.2.6
in your Dockerfile instead. This should just get you the pre-compiled wheel instead of trying to compile ruamel.yaml.clib from source, which will not work if you don't have a C compiler installed (this is actually what docker complains about)
I have ruamel.yaml.clib running succesfully in multiple Docker containers (but I never use a requirements.txt)

Using a pip cache directory in docker builds

I'm hoping to get my pip install instructions inside my docker builds as fast as possible.
I've read many posts explaining how adding your requirements.txt before the rest of the app helps you take advantage of Docker's own image cache if your requirements.txt hasn't changed. But this is no help at all when dependencies do change, even slightly.
The next step would be if we could use a consistent pip cache directory. By default, pip will cache downloaded packages in ~/.cache/pip (on Linux), and so if you're ever installing the same version of a module that has been installed before anywhere on the system, it shouldn't need to go and download it again, but instead simply use the cached version. If we could leverage a shared cache directory for docker builds, this could help speed up dependency installs a lot.
However, there doesn't appear to be any simple way to mount a volume while running docker build. The build environment seems to be basically impenetrable. I found one article suggesting a genius but complex method of running an rsync server on the host and then, with a hack inside the build to get the host IP, rsyncing the pip cache in from the host. But I'm not relishing the idea of running an rsync server in Jenkins (which isn't the most secure platform at the best of times).
Does anyone know if there's any other way to achieve a shared cache volume more simply?

I suggest you to use buildkit, also see this.
Dockerfile:
# syntax = docker/dockerfile:experimental
FROM python:3.6-alpine
RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml
NOTE: # syntax = docker/dockerfile:experimental is a must，you have to add it at the beginning of Dockerfile to enable this feature.
1.
The first execute build:
export DOCKER_BUILDKIT=1
docker build --progress=plain -t abc:1 . --no-cache
The first log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:11:35.296107357 +0000 UTC
#9 1.955 Collecting pyyaml
#9 3.050 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 5.006 Building wheels for collected packages: pyyaml
#9 5.007 Building wheel for pyyaml (setup.py): started
#9 5.249 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 5.250 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=867daf35eab43c2d047ad737ea1e9eaeb4168b87501cd4d62c533f671208acaa
#9 5.250 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 5.267 Successfully built pyyaml
#9 5.274 Installing collected packages: pyyaml
#9 5.309 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:11:42.221146294 +0000 UTC
#9 duration: 6.925038937s
From above, you can see the first time, the build will download pyyaml from internet.
2.
The second execute build:
docker build --progress=plain -t abc:1 . --no-cache
The second log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:16:58.588157354 +0000 UTC
#9 1.786 Collecting pyyaml
#9 2.234 Installing collected packages: pyyaml
#9 2.270 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:17:01.933398002 +0000 UTC
#9 duration: 3.345240648s
From above, you can see the build no longer download package from internet, just use the cache. NOTE, this is not the traditional docker build cache as I have use --no-cache, it's /root/.cache/pip which I mount into build.
3.
The third execute build which delete buildkit cache:
docker builder prune
docker build --progress=plain -t abc:1 . --no-cache
The third log:
#9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install...
#9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5
#9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml"
#9 started: 2019-09-20 03:19:07.434792944 +0000 UTC
#9 1.894 Collecting pyyaml
#9 2.740 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
#9 3.319 Building wheels for collected packages: pyyaml
#9 3.319 Building wheel for pyyaml (setup.py): started
#9 3.560 Building wheel for pyyaml (setup.py): finished with status 'done'
#9 3.560 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=cea5bc4689e231df7915c2fc3abca225d4ee2e869a7540682aacb6d42eb17053
#9 3.560 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
#9 3.580 Successfully built pyyaml
#9 3.585 Installing collected packages: pyyaml
#9 3.622 Successfully installed pyyaml-5.1.2
#9completed: 2019-09-20 03:19:12.530742712 +0000 UTC
#9 duration: 5.095949768s
From above, you can see if delete buildkit cache, the package download again.
In a word, it will give you a shared cache between several times build, and this cache will only be mounted when image build. But, the image self will not have these cache, so avoid a lots of intermediate layer in image.
EDIT for folks who are using docker compose and are lazy to read the comments...:
You can also do this with docker-compose if you set
COMPOSE_DOCKER_CLI_BUILD=1. For example: COMPOSE_DOCKER_CLI_BUILD=1
DOCKER_BUILDKIT=1 docker-compose build –
UPDATE according to folk's question 2020/09/02:
I don't know from which version (my version now is 19.03.11), if not specify mode for cache directory, the cache won't be reused by next time build.
Don't know the detail reason, but you could add mode=0755, to Dockerfile to make it work again:
Dockerfile:
# syntax = docker/dockerfile:experimental
FROM python:3.6-alpine
RUN --mount=type=cache,mode=0755,target=/root/.cache/pip pip install pyyaml