I am developing an azure app function that will be using a custom image with docker.
This is my DockerFile:
FROM mcr.microsoft.com/azure-functions/python:3.0-python3.7
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
# Adding "apt-get install make" here
RUN apt-get update && apt-get install make && apt-get install -y gcc g++ && rm -rf /var/lib/apt/lists/*
RUN pip install numpy
RUN pip install pandas
RUN pip install scipy
RUN wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && \
tar -xvzf ta-lib-0.4.0-src.tar.gz && \
cd ta-lib/ && \
./configure --prefix=/usr && \
make && \
make install
RUN rm -R ta-lib ta-lib-0.4.0-src.tar.gz
COPY requirements.txt /
RUN pip install -r /requirements.txt
COPY . /home/site/wwwroot
Everything works fine when I am using my local docker desktop app.
When I am trying to deploy to azure using the VS-Code build it interface the docker file fails to build here is the output I get:
14:57:08 : building 'talib._ta_lib' extension
14:57:08 : creating build/temp.linux-x86_64-3.7
14:57:08 : creating build/temp.linux-x86_64-3.7/talib
14:57:08 : gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-install-rgqyohyf/ta-lib/.eggs/numpy-1.21.2-py3.7-linux-x86_64.egg/numpy/core/include -I/opt/python/3.7.9/include/python3.7m -c talib/_ta_lib.c -o build/temp.linux-x86_64-3.7/talib/_ta_lib.o
14:57:08 : talib/_ta_lib.c:613:28: fatal error: ta-lib/ta_defs.h: No such file or directory
14:57:08 : #include "ta-lib/ta_defs.h"
14:57:08 : ^
14:57:08 : compilation terminated.
14:57:09 : error: command 'gcc' failed with exit status 1
14:57:09 : ----------------------------------------
14:57:09 : ERROR: Failed building wheel for TA-Lib
14:57:09 : DEPRECATION: Could not build wheels for TA-Lib which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
14:57:16 : [11:57:16+0000] Running setup.py install for TA-Lib: started
14:57:16 : [11:57:16+0000] Running setup.py install for TA-Lib: finished with status 'error'
14:57:16 : ERROR: Command errored out with exit status 1:
I am not sure what to make of this error and why it works fine in docker.
Thanks
Amit
Related
I want to install pandas on docker image containing python. I used the code below to run a container:
docker run -p 8888:8888 -v /home/DATA/Project_NY/:/home/jovyan/work/Project_NY jupyter/scipy-notebook
I created a new notebook and then tried to install my requirements file doing pip install -r "requirements.txt" i got the error below and when I tried to pip install pandas inside that running container it works perfectly:
requirements.txt content
SQLAlchemy==1.2.2
pandas==0.25.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==0.8.4
matplotlib
seaborn
psycopg2-binary
the error is
building 'pandas._libs.algos' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/pandas
creating build/temp.linux-x86_64-3.9/pandas/_libs
gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -DNPY_NO_DEPRECATED_API=0 -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/conda/lib/python3.9/site-packages/numpy/core/include -I/opt/conda/include/python3.9 -c pandas/_libs/algos.c -o build/temp.linux-x86_64-3.9/pandas/_libs/algos.o -Wno-unused-function
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for pandas
It seems like the pandas version you are trying to install needs a custom build for the System you're using as Dockercontainer.
You should get the same error if you run pip install pandas==0.25.0 inside the container.
Either use a different version of pandas or install gcc in the container ( e.g.: for alpine, inside Dockerfile CMD apk add --no-cache --virtual .build-deps gcc).
EDIT: I think the 'jupyter/scipy-notebook'-image uses conda, so maybe try:
docker run -p 8888:8888 -v /home/DATA/Project_NY/:/home/jovyan/work/Project_NY jupyter/scipy-notebook conda install gcc
the conda install gcc in the end is executed inside the container.
I followed the instructions and ran the following to commands to install pykaldi:
git clone https://github.com/pykaldi/pykaldi.git
cd pykaldi
sudo apt-get install autoconf automake cmake curl g++ git graphviz libatlas3-base libtool make pkg-config subversion unzip wget zlib1g-dev
sudo apt install intel-mkl-64bit-2020.4-912
python3.7 -m pip install --upgrade pip setuptools
python3.7 -m pip install numpy pyparsing ninja==1.10.0
cd tools
sudo torify ./check_dependencies.sh /usr/bin/python3.7
sudo torify ./install_protobuf.sh /usr/bin/python3.7
sudo torify ./install_clif.sh /usr/bin/python3.7
sudo torify ./install_kaldi.sh
cd ..
python3.7 -m pip install setuptools
sudo apt-get install ninja-build
sudo python3.7 setup.py install
when it comes to the last line, I get the following error:
running install
running bdist_egg
running egg_info
writing pykaldi.egg-info/PKG-INFO
writing dependency_links to pykaldi.egg-info/dependency_links.txt
writing requirements to pykaldi.egg-info/requires.txt
writing top-level names to pykaldi.egg-info/top_level.txt
reading manifest file 'pykaldi.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pykaldi.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Using PYCLIF: /usr/local/bin/pyclif
Using CLIF_MATCHER: /usr/clang/bin/clif-matcher
-- Configuring done
-- Generating done
-- Build files have been written to: /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/build
[6/505] Building CXX object kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
FAILED: kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
/usr/bin/c++ -D_matrix_ext_EXPORTS -I../kaldi/lib -I../kaldi -Ikaldi -I../tools/kaldi/src -I/usr/include/python2.7 -I/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include -std=c++11 -I.. -isystem /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/tools/kaldi/tools/openfst-1.6.7/include -O1 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I/opt/intel/mkl/include -m64 -msse -msse2 -pthread -g -fPIC -Wno-maybe-uninitialized -fPIC -MD -MT kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -MF kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o.d -o kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -c ../kaldi/matrix/matrix-ext.cc
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:4:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/npy_common.h:386:9: error: ‘Py_hash_t’ does not name a type
typedef Py_hash_t npy_hash_t;
^~~~~~~~~
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:655:9: error: ‘npy_hash_t’ does not name a type; did you mean ‘npy_half’?
npy_hash_t hash;
^~~~~~~~~~
npy_half
[11/505] Building CXX object kaldi/chain/CMakeFiles/_chain_generic_numerator.dir/chain-generic-numerator-clifwrap.cc.o
ninja: build stopped: subcommand failed.
Command '['ninja', '-j', '6']' returned non-zero exit status 1.
I also tried different versions of PyKaldi, i.e. 1.8.0 and 1.10.0. But it shows the same error.
Edit 1: when I execute /usr/bin/ninja -j 6 I get the following error: ninja: error: loading 'build.ninja': No such file or directory
I can't install nmslib using pip in docker.
This is my Dockerfile:
FROM continuumio/anaconda3:4.4.0
MAINTAINER UNP, https://unp.education
EXPOSE 8000
RUN apt-get update && apt-get install -y apache2 \
apache2-dev \
emacs \
&& apt-get clean \
&& apt-get autoremove \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /var/www/devise-api/
COPY ./devise-api.wsgi /var/www/devise-api/devise-api.wsgi
COPY ./devise-api /var/www/devise-api/
RUN pip install -r requirements.txt
RUN /opt/conda/bin/mod_wsgi-express install-module
RUN mod_wsgi-express setup-server devise-api.wsgi --port=8000 \
--user www-data --group www-data \
--server-root=/etc/mod_wsgi-express-80
CMD /etc/mod_wsgi-express-80/apachectl start -D FOREGROUND
For now requirements.txt contains only the word nmslib
This is the output I get when running sudo docker build -t devise-api .:
Sending build context to Docker daemon 306.2MB
Step 1/11 : FROM continuumio/anaconda3:4.4.0
---> 795ad88c47ff
Step 2/11 : MAINTAINER UNP, https://unp.education
---> Using cache
---> cd5b1f7e6188
Step 3/11 : EXPOSE 8000
---> Using cache
---> 21ad868f0823
Step 4/11 : RUN apt-get update && apt-get install -y apache2 apache2-dev emacs && apt-get clean && apt-get autoremove && rm -rf /var/lib/apt/lists/*
---> Using cache
---> ebfc7c30b394
Step 5/11 : WORKDIR /var/www/devise-api/
---> Using cache
---> 8228e4b4d4fd
Step 6/11 : COPY ./devise-api.wsgi /var/www/devise-api/devise-api.wsgi
---> Using cache
---> c8cf5cfcf7dc
Step 7/11 : COPY ./devise-api /var/www/devise-api/
---> Using cache
---> 856e67f0b1de
Step 8/11 : RUN pip install -r requirements.txt
---> Running in 7260901af476
Collecting nmslib (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/e1/95/1f7c90d682b79398c5ee3f9296be8d2640fa41de24226bcf5473c801ada6/nmslib-1.7.3.6.tar.gz (255kB)
Collecting pybind11>=2.0 (from nmslib->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/f2/7c/e71995e59e108799800cb0fce6c4b4927914d7eada0723dd20bae3b51786/pybind11-2.2.4-py2.py3-none-any.whl (145kB)
Requirement already satisfied: numpy in /opt/conda/lib/python3.6/site-packages (from nmslib->-r requirements.txt (line 1))
Building wheels for collected packages: nmslib
Running setup.py bdist_wheel for nmslib: started
Running setup.py bdist_wheel for nmslib: finished with status 'error'
Complete output from command /opt/conda/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-4qfop8hr/nmslib/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpl9tnakclpip-wheel- --python-tag cp36:
running bdist_wheel
running build
running build_ext
creating tmp
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/include/python3.6m -c /tmp/tmp2pnwck3x.cpp -o tmp/tmp2pnwck3x.o -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/include/python3.6m -c /tmp/tmp5ktoxd0l.cpp -o tmp/tmp5ktoxd0l.o -fvisibility=hidden
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
building 'nmslib' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/nmslib
creating build/temp.linux-x86_64-3.6/nmslib/similarity_search
creating build/temp.linux-x86_64-3.6/nmslib/similarity_search/src
creating build/temp.linux-x86_64-3.6/nmslib/similarity_search/src/space
creating build/temp.linux-x86_64-3.6/nmslib/similarity_search/src/method
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./nmslib/similarity_search/include -I/opt/conda/include/python3.6m -I/root/.local/include/python3.6m -I/opt/conda/lib/python3.6/site-packages/numpy/core/include -I/opt/conda/include/python3.6m -c nmslib.cc -o build/temp.linux-x86_64-3.6/nmslib.o -O3 -march=native -fopenmp -DVERSION_INFO="1.7.3.6" -std=c++14 -fvisibility=hidden
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
nmslib.cc:16:31: fatal error: pybind11/pybind11.h: No such file or directory
#include <pybind11/pybind11.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Failed building wheel for nmslib
Running setup.py clean for nmslib
Failed to build nmslib
Installing collected packages: pybind11, nmslib
Running setup.py install for nmslib: started
Then it stays like this forever.
Does anyone know what I could try to fix this?
Thank you a lot in advance!
Best regards
F
I just faced the same issue and fixed it by adding RUN step before installing nmslib:
RUN pip install pip==9.0.3 pybind11
See: https://github.com/nmslib/nmslib/issues/307#issuecomment-384113900. Hope this helps in your case as well :-)
Solved Wow, these guys are fast... It's basically this https://github.com/pyca/cryptography/issues/2750 It turned out that a security update for openssl was released (DROWN Attack) and that update contained an unexpected function signature change which caused the incompatibility, so this was just bad luck for me.
I need to use pip install cryptography in a Docker container running Alpine Linux. Actually, it's another module, service_identity, but the problem resides in the cryptography module, which is a dependency.
I have the following Dockerfile
FROM alpine:3.3
RUN apk --update add build-base libffi-dev openssl-dev python-dev py-pip
RUN pip install cryptography
which fails with the following error
generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'
building '_openssl' extension
creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o
build/temp.linux-x86_64-2.7/_openssl.c:726:6: error: conflicting types for 'BIO_new_mem_buf'
BIO *BIO_new_mem_buf(void *, int);
^
In file included from /usr/include/openssl/asn1.h:65:0,
from build/temp.linux-x86_64-2.7/_openssl.c:434:
/usr/include/openssl/bio.h:692:6: note: previous declaration of 'BIO_new_mem_buf' was here
BIO *BIO_new_mem_buf(const void *buf, int len);
^
error: command 'gcc' failed with exit status 1
openssl 1.0.2g was released on 2016-03-01 (yesterday) and the alpine package already got updated to that version. Can it be related to this?
How can I resolve this issue? Maybe some environment variables which I can set?
Update I've been checking the GitHub Repo for openssl, and in fact BIO *BIO_new_mem_buf(void *buf, int len) of openssl/bio.h got changed to BIO *BIO_new_mem_buf(const void *buf, int len) during the 1.0.2f to 1.0.2g transition (search for "BIO_new_mem_buf" in https://github.com/openssl/openssl/compare/OpenSSL_1_0_2f...OpenSSL_1_0_2g). I don't know where this openssl/asn1.h is coming from, which is importing an outdated version of openssl/bio.h, as it does not look like the one in the openssl repo. Any ideas?
Ok, I see some are already working on this:
https://github.com/pyca/cryptography/issues/2750
For those who are still experiencing problems installing cryptography==2.1.4 in Alpine 3.7 like this:
writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
running build_ext
generating cffi module 'build/temp.linux-x86_64-2.7/_padding.c'
creating build/temp.linux-x86_64-2.7
generating cffi module 'build/temp.linux-x86_64-2.7/_constant_time.c'
generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'
building '_openssl' extension
creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o -Wconversion -Wno-error=sign-conversion
build/temp.linux-x86_64-2.7/_openssl.c:493:30: fatal error: openssl/opensslv.h: No such file or directory
#include <openssl/opensslv.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
Solution
Install these dependencies in the Alpine container:
$ apk add --no-cache libressl-dev musl-dev libffi-dev
To install these dependencies using a Dockerfile:
RUN apk add --no-cache \
libressl-dev \
musl-dev \
libffi-dev && \
pip install --no-cache-dir cryptography==2.1.4 && \
apk del \
libressl-dev \
musl-dev \
libffi-dev
Reference
Installation instructions for cryptography on Alpine can be found here:
https://cryptography.io/en/latest/installation/#building-cryptography-on-linux
A version from the time of writing is available on github
Here is the relevant portion:
Building cryptography on Linux
[skipping over the part for non-Alpine Linux] …
$ pip install cryptography
If you are on Alpine or just want to compile it yourself then
cryptography requires a compiler, headers for Python (if you're not
using pypy), and headers for the OpenSSL and libffi libraries
available on your system.
Alpine
Replace python3-dev with python-dev if you're using Python 2.
$ sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev
If you get an error with openssl-dev you may have to use libressl-dev.
If it fails because of Rust version, then following is recommended in cryptography's docs:
The Rust available by default in Alpine < 3.12 is older than the
minimum supported version. See the Rust installation instructions
for information about installing a newer Rust.
$ sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo
in my case, python3.8-alpine, adding cargo resolved.
Add this before install:
RUN apk -U upgrade
RUN apk add --no-cache libffi-dev openssl-dev
Alternatively use build-base:
RUN apk add --no-cache --upgrade --virtual .build-deps build-base
Details here: https://git.alpinelinux.org/aports/tree/main/build-base/APKBUILD?h=3.3-stable
Check if you are building for the right architecture !!
x86-64 or amd64 architecture runs similar softwares and the other category is
aarch64 or arm architecture chips like Apple Silicon M1 or your mobile phone cpu
I try to install couchbase for python, but I get the following error:
building 'couchbase._libcouchbase' extension
creating build/temp.linux-i686-2.7
creating build/temp.linux-i686-2.7/src
creating build/temp.linux-i686-2.7/src/viewrow
creating build/temp.linux-i686-2.7/src/contrib
creating build/temp.linux-i686-2.7/src/contrib/jsonsl
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes - fPIC -I/usr/include/python2.7 -c src/exceptions.c -o build/temp.linux-i686-2.7/src/exceptions.o
In file included from src/exceptions.c:17:0:
src/pycbc.h:25:36: fatal error: libcouchbase/couchbase.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
I installed the couchbase server and the c library, and I already had:
sudo apt-get install build-essential # for a C compiler
sudo apt-get install python-dev
So what's wrong ?
You should install libcouchbase first http://www.couchbase.com/communities/c/getting-started
wget -O- http://packages.couchbase.com/ubuntu/couchbase.key | sudo apt-key add -
sudo wget -O/etc/apt/sources.list.d/couchbase.list \
http://packages.couchbase.com/ubuntu/couchbase-ubuntu1204.list
sudo apt-get update
sudo apt-get install libcouchbase2-libevent libcouchbase-dev
What about debian wheezy?
libcouchbase2-libevent: Depends: libevent-1.4-2 (>= 1.4.13-stable) but it is not installable
libcouchbase2-core : Depends: libssl0.9.8 (>= 0.9.8k-1) but it is not installable
If libcoouchbase is not in the repo, clone the libcouchbase repo and follow the instructions from there to install it:
$ git clone git://github.com/couchbase/libcouchbase.git
$ cd libcouchbase && mkdir build && cd build
$ ../cmake/configure
$ make
$ sudo make install
In the end install the python library from pip:
sudo -H pip3 install couchbase