Build zeep in docker from alpine - python

First things first: I struggled a little building the zeep pacakge in a docker alpine image, but I finally got it working. I hope that this post can help others with zeep on alpine in docker.
The question: After I got the zeep build to work, I had the simple issue of copying my build results to the new image. I used tar and COPY FROM eventually, but I was wondering if there is a better way of getting my build results from my build image to my deployment image. It is not clear to me that I got all the dependencies, but the module loads successfully.
Ultimately, here's the Dockerfile that I settled on:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
RUN cd /usr/lib/python3.7/site-packages/ && tar zcf /packages.tgz *
FROM alpine AS deploy-stage
RUN apk add python3 libxml2 libxslt
RUN pip3 install --upgrade pip
COPY --from=build-stage /packages.tgz /
RUN cd /usr/lib/python3.7/site-packages/ && tar zxf /packages.tgz && rm /packages.tgz
The next portion of this question is to express some of the difficulty I had with getting zeep to build. I'm not asking for analysis here but rather I'm including these error messages to make this post more searchable for others having similar problems (in other words, stop reading here).
I used FROM centos to build a python docker image that used zeep (for SOAP). However, to shrink the image size (it was a 1G image), I wanted to use FROM alpine. Once I made the move to alpine, the pip-3 install zeep fails.
/bin/sh: pip-3: not found
Well, python3 on alpine calls pip pip3.
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/usr/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
unable to execute 'gcc': No such file or directory
Compile failed: command 'gcc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitg55bfhan.c -o tmp/xmlXPathInitg55bfhan.o
unable to execute 'cc': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cq2qcz1c/lxml/setup.py'"'"';
__file__='"'"'/tmp/pip-install-cq2qcz1c/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
install --record /tmp/pip-record-nmuulbim/install-record.txt
--single-version-externally-managed --compile Check the logs for full command output.
Apparently, the install of the zeep package needs to compile "something". As I understand it, docker will allow me to build stuff in one image and copy it to another image. This allows me to install the compiler, build zeep, and then copy the end results to my new image, throwing away all the tools I used to build it.
To correct the need for the compiler, I added the line RUN apk add build-base to my Dockerfile.
I next hit this error:
#include "libxml/xpath.h"
^~~~~~~~~~~~~~~~
compilation terminated.
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
Let me try to add the libxml2 lib. I added this line RUN apk add libxml2 and got this result:
ERROR: b'/bin/sh: xslt-config: not found\n'
** make sure the development packages of libxml2 and libxslt are installed **
However, that line is way up in the error results (and easy to miss). So, following that suggestion, I replaced the last add libxml2 with RUN apk add libxml2-dev libxslt-dev. Now that I had the development versions of those libraries installed, I hit this error:
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/usr/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
src/lxml/etree.c:97:10: fatal error: Python.h: No such file or directory
#include "Python.h"
It looks like I have to include the header files for python, too. I modified my apk add command to look like this: RUN apk add python3-dev libxml2-dev libxslt-dev. Now, putting it altogether (and combining apk add commands), I have this to build zeep. At this point, this is what my Dockerfile looks like:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
This builds an image that I can use to import zeep:
$ docker run -it f1ac77dacbda sh
/ # python3
Python 3.7.5 (default, Oct 17 2019, 12:25:15)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zeep
>>>
So that looks good. And the really good news is that my image size is 312 MB, down from 1.05 GB (my original centos image). Now, that I've built zeep, I want to discard all of the larger devel packages and just stick with the runtimes. I also need to grab my zeep package (and it's dependencies) and add them to my deployment image.
To be honest, I don't know how to accurately determine the zeep dependencies without spending a lot of time. So, I just wanted to copy zeep from the packages directory. I discovered my package location this way:
$ docker run -it 2b53acb67b25 sh
/ # python3 -m site
sys.path = [
'/',
'/usr/lib/python37.zip',
'/usr/lib/python3.7',
'/usr/lib/python3.7/lib-dynload',
'/usr/lib/python3.7/site-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
To copy my build results, I tried this command:
COPY --from=build-stage /usr/lib/python3.7/site-packages/zeep/ /usr/lib/python3.7/site-packages/zeep/
This copied my zeep package over, but not its dependency packages:
$ docker run -it 1292276a94da sh
/ # python3
Python 3.7.5 (default, Oct 17 2019, 12:25:15)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zeep
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/site-packages/zeep/__init__.py", line 1, in <module>
from zeep.client import CachingClient, Client # noqa
File "/usr/lib/python3.7/site-packages/zeep/client.py", line 4, in <module>
from zeep.settings import Settings
File "/usr/lib/python3.7/site-packages/zeep/settings.py", line 4, in <module>
import attr
ModuleNotFoundError: No module named 'attr'
This is now my question (repeated from the initial summary)- how do I accurately copy it's dependencies from my build-stage to the deploy stage?
While I could add a COPY FROM for the attr package and each package I subsequently discover to be missing, I opted to brute force copy them all. The COPY command doesn't appear to have any recursive abilities and the directory structures are not maintained. So, I opted to put everything into a tar ball and copy that over. So here's my final Dockerfile for building and deploying zeep:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
RUN cd /usr/lib/python3.7/site-packages/ && tar zcf /packages.tgz *
FROM alpine AS deploy-stage
RUN apk add python3 libxml2 libxslt
RUN pip3 install --upgrade pip
COPY --from=build-stage /packages.tgz /
RUN cd /usr/lib/python3.7/site-packages/ && tar zxf /packages.tgz && rm /packages.tgz
Is there a more accurate way of copying the build results over, capturing all the dependencies, and excluding unnecessary packages (perhaps there were some packages that got downloaded for build only).

Related

python : pandas install errors on container

I want to install pandas on docker image containing python. I used the code below to run a container:
docker run -p 8888:8888 -v /home/DATA/Project_NY/:/home/jovyan/work/Project_NY jupyter/scipy-notebook
I created a new notebook and then tried to install my requirements file doing pip install -r "requirements.txt" i got the error below and when I tried to pip install pandas inside that running container it works perfectly:
requirements.txt content
SQLAlchemy==1.2.2
pandas==0.25.0
docker==3.3.0
python-json-logger
sshtunnel==0.1.4
jupyter
jupytext==0.8.4
matplotlib
seaborn
psycopg2-binary
the error is
building 'pandas._libs.algos' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/pandas
creating build/temp.linux-x86_64-3.9/pandas/_libs
gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -DNPY_NO_DEPRECATED_API=0 -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/opt/conda/lib/python3.9/site-packages/numpy/core/include -I/opt/conda/include/python3.9 -c pandas/_libs/algos.c -o build/temp.linux-x86_64-3.9/pandas/_libs/algos.o -Wno-unused-function
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for pandas

It seems like the pandas version you are trying to install needs a custom build for the System you're using as Dockercontainer.
You should get the same error if you run pip install pandas==0.25.0 inside the container.
Either use a different version of pandas or install gcc in the container ( e.g.: for alpine, inside Dockerfile CMD apk add --no-cache --virtual .build-deps gcc).
EDIT: I think the 'jupyter/scipy-notebook'-image uses conda, so maybe try:
docker run -p 8888:8888 -v /home/DATA/Project_NY/:/home/jovyan/work/Project_NY jupyter/scipy-notebook conda install gcc
the conda install gcc in the end is executed inside the container.

Failed to install pykaldi on ubuntu 18.04

I followed the instructions and ran the following to commands to install pykaldi:
git clone https://github.com/pykaldi/pykaldi.git
cd pykaldi
sudo apt-get install autoconf automake cmake curl g++ git graphviz libatlas3-base libtool make pkg-config subversion unzip wget zlib1g-dev
sudo apt install intel-mkl-64bit-2020.4-912
python3.7 -m pip install --upgrade pip setuptools
python3.7 -m pip install numpy pyparsing ninja==1.10.0
cd tools
sudo torify ./check_dependencies.sh /usr/bin/python3.7
sudo torify ./install_protobuf.sh /usr/bin/python3.7
sudo torify ./install_clif.sh /usr/bin/python3.7
sudo torify ./install_kaldi.sh
cd ..
python3.7 -m pip install setuptools
sudo apt-get install ninja-build
sudo python3.7 setup.py install
when it comes to the last line, I get the following error:
running install
running bdist_egg
running egg_info
writing pykaldi.egg-info/PKG-INFO
writing dependency_links to pykaldi.egg-info/dependency_links.txt
writing requirements to pykaldi.egg-info/requires.txt
writing top-level names to pykaldi.egg-info/top_level.txt
reading manifest file 'pykaldi.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pykaldi.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Using PYCLIF: /usr/local/bin/pyclif
Using CLIF_MATCHER: /usr/clang/bin/clif-matcher
-- Configuring done
-- Generating done
-- Build files have been written to: /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/build
[6/505] Building CXX object kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
FAILED: kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
/usr/bin/c++ -D_matrix_ext_EXPORTS -I../kaldi/lib -I../kaldi -Ikaldi -I../tools/kaldi/src -I/usr/include/python2.7 -I/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include -std=c++11 -I.. -isystem /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/tools/kaldi/tools/openfst-1.6.7/include -O1 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I/opt/intel/mkl/include -m64 -msse -msse2 -pthread -g -fPIC -Wno-maybe-uninitialized -fPIC -MD -MT kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -MF kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o.d -o kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -c ../kaldi/matrix/matrix-ext.cc
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:4:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/npy_common.h:386:9: error: ‘Py_hash_t’ does not name a type
typedef Py_hash_t npy_hash_t;
^~~~~~~~~
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:655:9: error: ‘npy_hash_t’ does not name a type; did you mean ‘npy_half’?
npy_hash_t hash;
^~~~~~~~~~
npy_half
[11/505] Building CXX object kaldi/chain/CMakeFiles/_chain_generic_numerator.dir/chain-generic-numerator-clifwrap.cc.o
ninja: build stopped: subcommand failed.
Command '['ninja', '-j', '6']' returned non-zero exit status 1.
I also tried different versions of PyKaldi, i.e. 1.8.0 and 1.10.0. But it shows the same error.
Edit 1: when I execute /usr/bin/ninja -j 6 I get the following error: ninja: error: loading 'build.ninja': No such file or directory

Cannot "pip install cryptography" in Docker Alpine Linux 3.3 with OpenSSL 1.0.2g and Python 2.7

Solved Wow, these guys are fast... It's basically this https://github.com/pyca/cryptography/issues/2750 It turned out that a security update for openssl was released (DROWN Attack) and that update contained an unexpected function signature change which caused the incompatibility, so this was just bad luck for me.
I need to use pip install cryptography in a Docker container running Alpine Linux. Actually, it's another module, service_identity, but the problem resides in the cryptography module, which is a dependency.
I have the following Dockerfile
FROM alpine:3.3
RUN apk --update add build-base libffi-dev openssl-dev python-dev py-pip
RUN pip install cryptography
which fails with the following error
generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'
building '_openssl' extension
creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o
build/temp.linux-x86_64-2.7/_openssl.c:726:6: error: conflicting types for 'BIO_new_mem_buf'
BIO *BIO_new_mem_buf(void *, int);
^
In file included from /usr/include/openssl/asn1.h:65:0,
from build/temp.linux-x86_64-2.7/_openssl.c:434:
/usr/include/openssl/bio.h:692:6: note: previous declaration of 'BIO_new_mem_buf' was here
BIO *BIO_new_mem_buf(const void *buf, int len);
^
error: command 'gcc' failed with exit status 1
openssl 1.0.2g was released on 2016-03-01 (yesterday) and the alpine package already got updated to that version. Can it be related to this?
How can I resolve this issue? Maybe some environment variables which I can set?
Update I've been checking the GitHub Repo for openssl, and in fact BIO *BIO_new_mem_buf(void *buf, int len) of openssl/bio.h got changed to BIO *BIO_new_mem_buf(const void *buf, int len) during the 1.0.2f to 1.0.2g transition (search for "BIO_new_mem_buf" in https://github.com/openssl/openssl/compare/OpenSSL_1_0_2f...OpenSSL_1_0_2g). I don't know where this openssl/asn1.h is coming from, which is importing an outdated version of openssl/bio.h, as it does not look like the one in the openssl repo. Any ideas?
Ok, I see some are already working on this:
https://github.com/pyca/cryptography/issues/2750

For those who are still experiencing problems installing cryptography==2.1.4 in Alpine 3.7 like this:
writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
running build_ext
generating cffi module 'build/temp.linux-x86_64-2.7/_padding.c'
creating build/temp.linux-x86_64-2.7
generating cffi module 'build/temp.linux-x86_64-2.7/_constant_time.c'
generating cffi module 'build/temp.linux-x86_64-2.7/_openssl.c'
building '_openssl' extension
creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7
gcc -fno-strict-aliasing -Os -fomit-frame-pointer -g -DNDEBUG -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64-2.7/_openssl.c -o build/temp.linux-x86_64-2.7/build/temp.linux-x86_64-2.7/_openssl.o -Wconversion -Wno-error=sign-conversion
build/temp.linux-x86_64-2.7/_openssl.c:493:30: fatal error: openssl/opensslv.h: No such file or directory
#include <openssl/opensslv.h>
^
compilation terminated.
error: command 'gcc' failed with exit status 1
Solution
Install these dependencies in the Alpine container:
$ apk add --no-cache libressl-dev musl-dev libffi-dev
To install these dependencies using a Dockerfile:
RUN apk add --no-cache \
libressl-dev \
musl-dev \
libffi-dev && \
pip install --no-cache-dir cryptography==2.1.4 && \
apk del \
libressl-dev \
musl-dev \
libffi-dev
Reference
Installation instructions for cryptography on Alpine can be found here:
https://cryptography.io/en/latest/installation/#building-cryptography-on-linux
A version from the time of writing is available on github
Here is the relevant portion:
Building cryptography on Linux
[skipping over the part for non-Alpine Linux] …
$ pip install cryptography
If you are on Alpine or just want to compile it yourself then
cryptography requires a compiler, headers for Python (if you're not
using pypy), and headers for the OpenSSL and libffi libraries
available on your system.
Alpine
Replace python3-dev with python-dev if you're using Python 2.
$ sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev
If you get an error with openssl-dev you may have to use libressl-dev.

If it fails because of Rust version, then following is recommended in cryptography's docs:
The Rust available by default in Alpine < 3.12 is older than the
minimum supported version. See the Rust installation instructions
for information about installing a newer Rust.
$ sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo
in my case, python3.8-alpine, adding cargo resolved.

Add this before install:
RUN apk -U upgrade
RUN apk add --no-cache libffi-dev openssl-dev

Alternatively use build-base:
RUN apk add --no-cache --upgrade --virtual .build-deps build-base
Details here: https://git.alpinelinux.org/aports/tree/main/build-base/APKBUILD?h=3.3-stable

Check if you are building for the right architecture !!
x86-64 or amd64 architecture runs similar softwares and the other category is
aarch64 or arm architecture chips like Apple Silicon M1 or your mobile phone cpu

Problems with pip install numpy - RuntimeError: Broken toolchain: cannot link a simple C program

I'm trying to install numpy (and scipy and matplotlib) into a virturalenv.
I keep getting these errors though:
RuntimeError: Broken toolchain: cannot link a simple C program
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1
I have the command line tools for xcode installed
$ which gcc
/usr/bin/gcc
$ which cc
/usr/bin/cc
I'm on Mac OSX 10.9
Using a brew installed python
Edit
Yes, trying to install with pip.
The whole traceback is huge (>400 lines)
Here is a section of it:
C compiler: cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
cc: _configtest.c
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]
clang: note: this will be a hard error (cannot be downgraded to a warning) in the future
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/setup.py", line 192, in <module>
setup_package()
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/setup.py", line 185, in setup_package
configuration=configuration )
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/core.py", line 169, in setup
return old_setup(**new_attr)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/command/egg_info.py", line 10, in run
self.run_command("build_src")
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/command/build_src.py", line 153, in run
self.build_sources()
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/command/build_src.py", line 164, in build_sources
self.build_library_sources(*libname_info)
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/command/build_src.py", line 299, in build_library_sources
sources = self.generate_sources(sources, (lib_name, build_info))
File "/Users/bdhammel/Documents/research_programming/julia_env/build/numpy/numpy/distutils/command/build_src.py", line 386, in generate_sources
source = func(extension, build_dir)
File "numpy/core/setup.py", line 674, in get_mathlib_info
raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

For Docker (Alpine) and Python 3.x this worked for me:
RUN apk update
RUN apk add make automake gcc g++ subversion python3-dev

While it's ugly, it appears to work
sudo ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install --upgrade numpy
Note that if you are getting this error for a package other than numpy, (such as lxml) specify that package name instead of numpy at the end of the commnd.
I saw a similar issue someone was having with installing a gem
Ruby Gem install Json fails on Mavericks and Xcode 5.1 - unknown argument: '-multiply_definedsuppress'
This is only a temporary fix, at some point the compiler options will have to be fixed

The problem is that you are unable to compile.
First, make sure that you have accepted the new Terms and Conditions with Xcode. To do this, just open up xCode and accept.
Then, try installing gcc with
brew install gcc
Finally, try to install Numpy with
pip install numpy
Hope this helps.

If you don't want to use sudo (so permissions and things like that are preserved when using venv), you can add the ARCHFLAGS declaration to your .bash_profile, and run as normal. This worked for me with Mavericks and Xcode 5.1 using with venv:
In ~/.bash_profile:
export ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future
Then, just run the command:
pip install --upgrade numpy

If you are running a linux distribution, you may need a C compiler, especially if you see telltale log lines like sh: gcc: command not found. You can follow the instructions here, which I've summarized below:
Fedora, Red Hat, CentOS, or Scientific Linux
# yum groupinstall 'Development Tools'
Debian or Ubuntu Linux
$ sudo apt-get update
$ sudo apt-get install build-essential manpages-dev
Then you can try rerunning:
sudo pip install numpy

In my case this happened during a docker build. The problem was that the base image wasn't fixed to a specific python version and numpy couldn't compile with the new one.
FROM python:3-slim # BAD
After I changed it to the following it worked:
FROM python:3.8-slim # GOOD
Remember to fix your versions! :-)

I simply had to open XCode and accept the agreement and let it install the tools. I then went back to PyCharm and installed numpy again with no issues.

On Fedora 22 this was resolved by updating pip itself:
sudo pip install --upgrade pip

For fedora users that are having a similar problem try installing these packeges:
(if not using python3 use python-devel and pip instead of pip3)
sudo dnf install python3-devel
sudo dnf install make automake gcc gcc-c++ gcc-gfortran
sudo dnf install redhat-rpm-config
sudo dnf install subversion
then try
sudo pip3 install numpy

In some cases after OS X upgrades XCode, XCode will require the user (with administrative privileges) to accept a new license. Until the license is accepted, clang and gcc will issue an error when attempting to compile and link code. Or at least python packages.
If you launch XCode and accept the license, the errors no longer appear.
At least, this was the case for me.

The Error:
RuntimeError: Broken toolchain: cannot link a simple C program
Means it either can't find your C compiler or encountered problems linking a simple C program.
Demo docker walkthrough, to get Python2/pip to puke that error:
Keeping this demo confined inside a read-only docker image helps us be fearless in issuing OS-wrecking root level commands and start fresh every time:
#download a read only docker image that doesn't have gcc installed:
docker pull frolvlad/alpine-python2
#check the version of pip and python:
docker run --rm frolvlad/alpine-python2 pip --version
#pip 20.3.4 from /usr/lib/python2.7/site-packages/pip (python
Try to pip install numpy, you can't because pip needs gcc:
docker run --rm frolvlad/alpine-python2 pip install --user numpy
#...
#RuntimeError: Broken toolchain: cannot link a simple C program
gcc clearly not installed:
docker run --rm frolvlad/alpine-python2 which gcc
#empty output, gcc is not installed
So install it, now the binary /usr/bin/gcc exists:
docker run --rm frolvlad/alpine-python2 sh -c "apk update && apk add make automake gcc g++ subversion python2-dev && which gcc"
#/usr/bin/gcc
Apologies for stacking terminal commands but this is a read only image so everything has to be redone every time:
Now pip can use gcc and install numpy for python2:
docker run --rm frolvlad/alpine-python2 sh -c "apk update && apk add make automake gcc g++ subversion python2-dev && pip install numpy && python -c 'import numpy as np; print(np.__name__, np.__version__)'"
# ...
#Successfully installed numpy-1.16.6
#('numpy', '1.16.6')
We're in a good place so freeze the docker image and rename it.
docker images
#choose the most recent container id: 751420a548c1:latest and rename it:
docker commit 751420a548c1:latest frolvlad/alpine-python2_numpy
Now you have a new read only docker image with python2, pip and updated 1.16.6 numpy, ready to run your legacy python2 software like it's 2005:
docker run --rm frolvlad/alpine-python2_numpy python -c 'import numpy as np; print(np.__name__, np.__version__)'
#('numpy', '1.16.6')
Or to make the above commands simpler and sequential:
Use the -it flag on docker run which means open an interactive terminal from the docker image:
$ docker run -it frolvlad/alpine-python2_numpy sh
/ > python2 -c 'import numpy as np; print(np.__version__)'
1.16.6
/ > which gcc
/usr/bin/gcc
/ > python2 -c 'import numpy as np; print(np.zeros(3))'
[0. 0. 0.]
/ > echo "headward free now to rise"; exit;
As python2 packages fall further into ancient history, pip will have to be configured to point at your a private python2 package repository url.

I solved this by using Conda instead of pip in my Dockerfile:
FROM continuumio/miniconda3:latest

Old thread, by my problem was that I didn't have Xcode installed. The following solved it.
xcode-select --install

Installing PyCrypto on Ubuntu - fatal error on build

Having looked at other similar threads, I still can't get pycrypto running.
I'm trying to get it working on my Ubuntu laptop - but I couldn't manage it on my Windows PC either.
I downloaded pycrypto-2.6, extracted it, and ran
python setup.py build
but then this happened
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 - fomit-frame-pointer -Isrc/ -I/usr/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-?2.7/src/MD2.o
src/MD2.c:31:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Would appreciate any help.

You need to install the Python development files. I believe this will do it:
sudo apt-get install python-dev

On Ubuntu, I needed some other packages for it to succeed:
apt-get install autoconf g++ python2.7-dev
pip install pycrypto

On Ubuntu and if you use Python 3.x you will need:
sudo apt-get install gcc python3-dev
you probably already have gcc but just in case if you are trying this command from Dockerfile with base image python:3.6.4-slim-jessie then you will also need gcc.

August 2021
For python 3.8 users run
sudo apt-get install python3.8-dev
and try to install pycrypto again
pip install pycrypto

Categories

Resources