I followed the instructions and ran the following to commands to install pykaldi:
git clone https://github.com/pykaldi/pykaldi.git
cd pykaldi
sudo apt-get install autoconf automake cmake curl g++ git graphviz libatlas3-base libtool make pkg-config subversion unzip wget zlib1g-dev
sudo apt install intel-mkl-64bit-2020.4-912
python3.7 -m pip install --upgrade pip setuptools
python3.7 -m pip install numpy pyparsing ninja==1.10.0
cd tools
sudo torify ./check_dependencies.sh /usr/bin/python3.7
sudo torify ./install_protobuf.sh /usr/bin/python3.7
sudo torify ./install_clif.sh /usr/bin/python3.7
sudo torify ./install_kaldi.sh
cd ..
python3.7 -m pip install setuptools
sudo apt-get install ninja-build
sudo python3.7 setup.py install
when it comes to the last line, I get the following error:
running install
running bdist_egg
running egg_info
writing pykaldi.egg-info/PKG-INFO
writing dependency_links to pykaldi.egg-info/dependency_links.txt
writing requirements to pykaldi.egg-info/requires.txt
writing top-level names to pykaldi.egg-info/top_level.txt
reading manifest file 'pykaldi.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pykaldi.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Using PYCLIF: /usr/local/bin/pyclif
Using CLIF_MATCHER: /usr/clang/bin/clif-matcher
-- Configuring done
-- Generating done
-- Build files have been written to: /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/build
[6/505] Building CXX object kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
FAILED: kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
/usr/bin/c++ -D_matrix_ext_EXPORTS -I../kaldi/lib -I../kaldi -Ikaldi -I../tools/kaldi/src -I/usr/include/python2.7 -I/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include -std=c++11 -I.. -isystem /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/tools/kaldi/tools/openfst-1.6.7/include -O1 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I/opt/intel/mkl/include -m64 -msse -msse2 -pthread -g -fPIC -Wno-maybe-uninitialized -fPIC -MD -MT kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -MF kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o.d -o kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -c ../kaldi/matrix/matrix-ext.cc
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:4:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/npy_common.h:386:9: error: ‘Py_hash_t’ does not name a type
typedef Py_hash_t npy_hash_t;
^~~~~~~~~
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:655:9: error: ‘npy_hash_t’ does not name a type; did you mean ‘npy_half’?
npy_hash_t hash;
^~~~~~~~~~
npy_half
[11/505] Building CXX object kaldi/chain/CMakeFiles/_chain_generic_numerator.dir/chain-generic-numerator-clifwrap.cc.o
ninja: build stopped: subcommand failed.
Command '['ninja', '-j', '6']' returned non-zero exit status 1.
I also tried different versions of PyKaldi, i.e. 1.8.0 and 1.10.0. But it shows the same error.
Edit 1: when I execute /usr/bin/ninja -j 6 I get the following error: ninja: error: loading 'build.ninja': No such file or directory
Related
I am developing an azure app function that will be using a custom image with docker.
This is my DockerFile:
FROM mcr.microsoft.com/azure-functions/python:3.0-python3.7
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
# Adding "apt-get install make" here
RUN apt-get update && apt-get install make && apt-get install -y gcc g++ && rm -rf /var/lib/apt/lists/*
RUN pip install numpy
RUN pip install pandas
RUN pip install scipy
RUN wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz && \
tar -xvzf ta-lib-0.4.0-src.tar.gz && \
cd ta-lib/ && \
./configure --prefix=/usr && \
make && \
make install
RUN rm -R ta-lib ta-lib-0.4.0-src.tar.gz
COPY requirements.txt /
RUN pip install -r /requirements.txt
COPY . /home/site/wwwroot
Everything works fine when I am using my local docker desktop app.
When I am trying to deploy to azure using the VS-Code build it interface the docker file fails to build here is the output I get:
14:57:08 : building 'talib._ta_lib' extension
14:57:08 : creating build/temp.linux-x86_64-3.7
14:57:08 : creating build/temp.linux-x86_64-3.7/talib
14:57:08 : gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-install-rgqyohyf/ta-lib/.eggs/numpy-1.21.2-py3.7-linux-x86_64.egg/numpy/core/include -I/opt/python/3.7.9/include/python3.7m -c talib/_ta_lib.c -o build/temp.linux-x86_64-3.7/talib/_ta_lib.o
14:57:08 : talib/_ta_lib.c:613:28: fatal error: ta-lib/ta_defs.h: No such file or directory
14:57:08 : #include "ta-lib/ta_defs.h"
14:57:08 : ^
14:57:08 : compilation terminated.
14:57:09 : error: command 'gcc' failed with exit status 1
14:57:09 : ----------------------------------------
14:57:09 : ERROR: Failed building wheel for TA-Lib
14:57:09 : DEPRECATION: Could not build wheels for TA-Lib which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
14:57:16 : [11:57:16+0000] Running setup.py install for TA-Lib: started
14:57:16 : [11:57:16+0000] Running setup.py install for TA-Lib: finished with status 'error'
14:57:16 : ERROR: Command errored out with exit status 1:
I am not sure what to make of this error and why it works fine in docker.
Thanks
Amit
First things first: I struggled a little building the zeep pacakge in a docker alpine image, but I finally got it working. I hope that this post can help others with zeep on alpine in docker.
The question: After I got the zeep build to work, I had the simple issue of copying my build results to the new image. I used tar and COPY FROM eventually, but I was wondering if there is a better way of getting my build results from my build image to my deployment image. It is not clear to me that I got all the dependencies, but the module loads successfully.
Ultimately, here's the Dockerfile that I settled on:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
RUN cd /usr/lib/python3.7/site-packages/ && tar zcf /packages.tgz *
FROM alpine AS deploy-stage
RUN apk add python3 libxml2 libxslt
RUN pip3 install --upgrade pip
COPY --from=build-stage /packages.tgz /
RUN cd /usr/lib/python3.7/site-packages/ && tar zxf /packages.tgz && rm /packages.tgz
The next portion of this question is to express some of the difficulty I had with getting zeep to build. I'm not asking for analysis here but rather I'm including these error messages to make this post more searchable for others having similar problems (in other words, stop reading here).
I used FROM centos to build a python docker image that used zeep (for SOAP). However, to shrink the image size (it was a 1G image), I wanted to use FROM alpine. Once I made the move to alpine, the pip-3 install zeep fails.
/bin/sh: pip-3: not found
Well, python3 on alpine calls pip pip3.
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -Isrc -Isrc/lxml/includes -I/usr/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
unable to execute 'gcc': No such file or directory
Compile failed: command 'gcc' failed with exit status 1
creating tmp
cc -I/usr/include/libxml2 -c /tmp/xmlXPathInitg55bfhan.c -o tmp/xmlXPathInitg55bfhan.o
unable to execute 'cc': No such file or directory
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cq2qcz1c/lxml/setup.py'"'"';
__file__='"'"'/tmp/pip-install-cq2qcz1c/lxml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))'
install --record /tmp/pip-record-nmuulbim/install-record.txt
--single-version-externally-managed --compile Check the logs for full command output.
Apparently, the install of the zeep package needs to compile "something". As I understand it, docker will allow me to build stuff in one image and copy it to another image. This allows me to install the compiler, build zeep, and then copy the end results to my new image, throwing away all the tools I used to build it.
To correct the need for the compiler, I added the line RUN apk add build-base to my Dockerfile.
I next hit this error:
#include "libxml/xpath.h"
^~~~~~~~~~~~~~~~
compilation terminated.
*********************************************************************************
Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
*********************************************************************************
error: command 'gcc' failed with exit status 1
----------------------------------------
Let me try to add the libxml2 lib. I added this line RUN apk add libxml2 and got this result:
ERROR: b'/bin/sh: xslt-config: not found\n'
** make sure the development packages of libxml2 and libxslt are installed **
However, that line is way up in the error results (and easy to miss). So, following that suggestion, I replaced the last add libxml2 with RUN apk add libxml2-dev libxslt-dev. Now that I had the development versions of those libraries installed, I hit this error:
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -DCYTHON_CLINE_IN_TRACEBACK=0 -I/usr/include/libxml2 -Isrc -Isrc/lxml/includes -I/usr/include/python3.7m -c src/lxml/etree.c -o build/temp.linux-x86_64-3.7/src/lxml/etree.o -w
src/lxml/etree.c:97:10: fatal error: Python.h: No such file or directory
#include "Python.h"
It looks like I have to include the header files for python, too. I modified my apk add command to look like this: RUN apk add python3-dev libxml2-dev libxslt-dev. Now, putting it altogether (and combining apk add commands), I have this to build zeep. At this point, this is what my Dockerfile looks like:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
This builds an image that I can use to import zeep:
$ docker run -it f1ac77dacbda sh
/ # python3
Python 3.7.5 (default, Oct 17 2019, 12:25:15)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zeep
>>>
So that looks good. And the really good news is that my image size is 312 MB, down from 1.05 GB (my original centos image). Now, that I've built zeep, I want to discard all of the larger devel packages and just stick with the runtimes. I also need to grab my zeep package (and it's dependencies) and add them to my deployment image.
To be honest, I don't know how to accurately determine the zeep dependencies without spending a lot of time. So, I just wanted to copy zeep from the packages directory. I discovered my package location this way:
$ docker run -it 2b53acb67b25 sh
/ # python3 -m site
sys.path = [
'/',
'/usr/lib/python37.zip',
'/usr/lib/python3.7',
'/usr/lib/python3.7/lib-dynload',
'/usr/lib/python3.7/site-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True
To copy my build results, I tried this command:
COPY --from=build-stage /usr/lib/python3.7/site-packages/zeep/ /usr/lib/python3.7/site-packages/zeep/
This copied my zeep package over, but not its dependency packages:
$ docker run -it 1292276a94da sh
/ # python3
Python 3.7.5 (default, Oct 17 2019, 12:25:15)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zeep
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/site-packages/zeep/__init__.py", line 1, in <module>
from zeep.client import CachingClient, Client # noqa
File "/usr/lib/python3.7/site-packages/zeep/client.py", line 4, in <module>
from zeep.settings import Settings
File "/usr/lib/python3.7/site-packages/zeep/settings.py", line 4, in <module>
import attr
ModuleNotFoundError: No module named 'attr'
This is now my question (repeated from the initial summary)- how do I accurately copy it's dependencies from my build-stage to the deploy stage?
While I could add a COPY FROM for the attr package and each package I subsequently discover to be missing, I opted to brute force copy them all. The COPY command doesn't appear to have any recursive abilities and the directory structures are not maintained. So, I opted to put everything into a tar ball and copy that over. So here's my final Dockerfile for building and deploying zeep:
FROM alpine AS build-stage
RUN apk add python3-dev build-base libxml2-dev libxslt-dev
RUN pip3 install --upgrade pip
RUN pip3 install zeep
RUN cd /usr/lib/python3.7/site-packages/ && tar zcf /packages.tgz *
FROM alpine AS deploy-stage
RUN apk add python3 libxml2 libxslt
RUN pip3 install --upgrade pip
COPY --from=build-stage /packages.tgz /
RUN cd /usr/lib/python3.7/site-packages/ && tar zxf /packages.tgz && rm /packages.tgz
Is there a more accurate way of copying the build results over, capturing all the dependencies, and excluding unnecessary packages (perhaps there were some packages that got downloaded for build only).
I'm getting error installing Scrapy on my ubuntu box. I'm using pip to install Scrapy. I'm aware that it needs setuptools to be installed. I got that installed using the script provided in setuptools website.
reading manifest file 'Twisted.egg-info/SOURCES.txt'
writing manifest file 'Twisted.egg-info/SOURCES.txt'
creating build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport
copying twisted/internet/iocpreactor/iocpsupport/iocpsupport.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport
copying twisted/internet/iocpreactor/iocpsupport/winsock_pointers.c -> build/lib.linux-x86_64-2.7/twisted/internet/iocpreactor/iocpsupport
copying twisted/test/raiser.c -> build/lib.linux-x86_64-2.7/twisted/test
copying twisted/runner/portmap.c -> build/lib.linux-x86_64-2.7/twisted/runner
copying twisted/python/sendmsg.c -> build/lib.linux-x86_64-2.7/twisted/python
running build_ext
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c conftest.c -o conftest.o
building 'twisted.runner.portmap' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/twisted
creating build/temp.linux-x86_64-2.7/twisted/runner
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c twisted/runner/portmap.c -o build/temp.linux-x86_64-2.7/twisted/runner/portmap.o
twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directory
#include <Python.h>
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/Twisted/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-4QNuNV-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/Twisted
Storing debug log for failure in /root/.pip/pip.log
Any Idea where the process goes wrong? I've got gcc and g++ installed already.
From the error fatal error: Python.h: No such file or directory it looks like python development headers are not installed. Try this command and then try to install again.
sudo apt-get install python-dev
For installing libevent library apply this command,
sudo apt-get install libevent-dev
I figured this out on ubuntu 14.04 by (A) actually following the scrapy docs for installing on ubuntu, then (B) an error that a lot of others were getting, by following a S/O solution Error while starting new scrapy project
(A) http://doc.scrapy.org/en/1.0/topics/ubuntu.html#topics-ubuntu
then
(B)
sudo pip install pyasn1 --upgrade
Right after installing Fedora 23 that's what I did (may works on Ubuntu):
[root#x ~]# pip install scrapy
[root#x ~]# dnf install python-cffi
[root#x ~]# dnf install openssl-devel
[root#x ~]# dnf install gcc
[root#x ~]# dnf install redhat-rpm-config
[root#x ~]# dnf install libxml
[root#x ~]# dnf install libxml2-devel
[root#x ~]# dnf install libxml-devel
[root#x ~]# dnf install glib2-devel gnet2-devel
[root#x ~]# dnf install libxslt-devel
sudo apt install python3-scrapy
I have tried this ..and it worked for me.
(You should have the ssh installed)
I try to install couchbase for python, but I get the following error:
building 'couchbase._libcouchbase' extension
creating build/temp.linux-i686-2.7
creating build/temp.linux-i686-2.7/src
creating build/temp.linux-i686-2.7/src/viewrow
creating build/temp.linux-i686-2.7/src/contrib
creating build/temp.linux-i686-2.7/src/contrib/jsonsl
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes - fPIC -I/usr/include/python2.7 -c src/exceptions.c -o build/temp.linux-i686-2.7/src/exceptions.o
In file included from src/exceptions.c:17:0:
src/pycbc.h:25:36: fatal error: libcouchbase/couchbase.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
I installed the couchbase server and the c library, and I already had:
sudo apt-get install build-essential # for a C compiler
sudo apt-get install python-dev
So what's wrong ?
You should install libcouchbase first http://www.couchbase.com/communities/c/getting-started
wget -O- http://packages.couchbase.com/ubuntu/couchbase.key | sudo apt-key add -
sudo wget -O/etc/apt/sources.list.d/couchbase.list \
http://packages.couchbase.com/ubuntu/couchbase-ubuntu1204.list
sudo apt-get update
sudo apt-get install libcouchbase2-libevent libcouchbase-dev
What about debian wheezy?
libcouchbase2-libevent: Depends: libevent-1.4-2 (>= 1.4.13-stable) but it is not installable
libcouchbase2-core : Depends: libssl0.9.8 (>= 0.9.8k-1) but it is not installable
If libcoouchbase is not in the repo, clone the libcouchbase repo and follow the instructions from there to install it:
$ git clone git://github.com/couchbase/libcouchbase.git
$ cd libcouchbase && mkdir build && cd build
$ ../cmake/configure
$ make
$ sudo make install
In the end install the python library from pip:
sudo -H pip3 install couchbase
Having looked at other similar threads, I still can't get pycrypto running.
I'm trying to get it working on my Ubuntu laptop - but I couldn't manage it on my Windows PC either.
I downloaded pycrypto-2.6, extracted it, and ran
python setup.py build
but then this happened
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 - fomit-frame-pointer -Isrc/ -I/usr/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-?2.7/src/MD2.o
src/MD2.c:31:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Would appreciate any help.
You need to install the Python development files. I believe this will do it:
sudo apt-get install python-dev
On Ubuntu, I needed some other packages for it to succeed:
apt-get install autoconf g++ python2.7-dev
pip install pycrypto
On Ubuntu and if you use Python 3.x you will need:
sudo apt-get install gcc python3-dev
you probably already have gcc but just in case if you are trying this command from Dockerfile with base image python:3.6.4-slim-jessie then you will also need gcc.
August 2021
For python 3.8 users run
sudo apt-get install python3.8-dev
and try to install pycrypto again
pip install pycrypto