Develop Reference

Failed to install pykaldi on ubuntu 18.04

I followed the instructions and ran the following to commands to install pykaldi:
git clone https://github.com/pykaldi/pykaldi.git
cd pykaldi
sudo apt-get install autoconf automake cmake curl g++ git graphviz libatlas3-base libtool make pkg-config subversion unzip wget zlib1g-dev
sudo apt install intel-mkl-64bit-2020.4-912
python3.7 -m pip install --upgrade pip setuptools
python3.7 -m pip install numpy pyparsing ninja==1.10.0
cd tools
sudo torify ./check_dependencies.sh /usr/bin/python3.7
sudo torify ./install_protobuf.sh /usr/bin/python3.7
sudo torify ./install_clif.sh /usr/bin/python3.7
sudo torify ./install_kaldi.sh
cd ..
python3.7 -m pip install setuptools
sudo apt-get install ninja-build
sudo python3.7 setup.py install
when it comes to the last line, I get the following error:
running install
running bdist_egg
running egg_info
writing pykaldi.egg-info/PKG-INFO
writing dependency_links to pykaldi.egg-info/dependency_links.txt
writing requirements to pykaldi.egg-info/requires.txt
writing top-level names to pykaldi.egg-info/top_level.txt
reading manifest file 'pykaldi.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'pykaldi.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
Using PYCLIF: /usr/local/bin/pyclif
Using CLIF_MATCHER: /usr/clang/bin/clif-matcher
-- Configuring done
-- Generating done
-- Build files have been written to: /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/build
[6/505] Building CXX object kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
FAILED: kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o
/usr/bin/c++ -D_matrix_ext_EXPORTS -I../kaldi/lib -I../kaldi -Ikaldi -I../tools/kaldi/src -I/usr/include/python2.7 -I/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include -std=c++11 -I.. -isystem /home/soroushh/KhodnevisProjects/wav2vec-u/pykaldi/tools/kaldi/tools/openfst-1.6.7/include -O1 -Wall -Wno-sign-compare -Wno-unused-local-typedefs -Wno-deprecated-declarations -Winit-self -DKALDI_DOUBLEPRECISION=0 -DHAVE_EXECINFO_H=1 -DHAVE_CXXABI_H -DHAVE_MKL -I/opt/intel/mkl/include -m64 -msse -msse2 -pthread -g -fPIC -Wno-maybe-uninitialized -fPIC -MD -MT kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -MF kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o.d -o kaldi/matrix/CMakeFiles/_matrix_ext.dir/matrix-ext.cc.o -c ../kaldi/matrix/matrix-ext.cc
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:4:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/npy_common.h:386:9: error: ‘Py_hash_t’ does not name a type
typedef Py_hash_t npy_hash_t;
^~~~~~~~~
In file included from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12:0,
from /home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from ../kaldi/matrix/matrix-ext.cc:8:
/home/soroushh/.local/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:655:9: error: ‘npy_hash_t’ does not name a type; did you mean ‘npy_half’?
npy_hash_t hash;
^~~~~~~~~~
npy_half
[11/505] Building CXX object kaldi/chain/CMakeFiles/_chain_generic_numerator.dir/chain-generic-numerator-clifwrap.cc.o
ninja: build stopped: subcommand failed.
Command '['ninja', '-j', '6']' returned non-zero exit status 1.
I also tried different versions of PyKaldi, i.e. 1.8.0 and 1.10.0. But it shows the same error.
Edit 1: when I execute /usr/bin/ninja -j 6 I get the following error: ninja: error: loading 'build.ninja': No such file or directory

Pyenv broke after upgrading to Big Sur

Pyenv has not worked since upgrading to Big Sur. This is how it started:
❯ pyenv install 3.8.0
python-build: use openssl#1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.0.tar.xz...
-> https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
Installing Python-3.8.0...
python-build: use readline from homebrew
BUILD FAILED (OS X 11.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/_y/vdsjczjj3nvflspjcl8r68zr0000gq/T/python-build.20210131003659.3456
Results logged to /var/folders/_y/vdsjczjj3nvflspjcl8r68zr0000gq/T/python-build.20210131003659.3456.log
Last 10 log lines:
checking for python3.8... python3.8
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/var/folders/_y/vdsjczjj3nvflspjcl8r68zr0000gq/T/python-build.20210131003659.3456/Python-3.8.0':
configure: error: C compiler cannot create executables
See `config.log' for more details
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
So I ran xcode-select --install. It went through, though I don't see a xrun directory under /Library/Developer/CommandLineTools/usr/bin/.
I read many different github issues' pages, StackOverflow posts, and tried many different things including brew install zlib sqlite bzip2 libiconv libzip. This is where I am right now:
❯ pyenv install 3.8.0
python-build: use openssl#1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.8.0.tar.xz...
-> https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tar.xz
Installing Python-3.8.0...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
BUILD FAILED (OS X 11.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/_y/vdsjczjj3nvflspjcl8r68zr0000gq/T/python-build.20210201012936.8553
Results logged to /var/folders/_y/vdsjczjj3nvflspjcl8r68zr0000gq/T/python-build.20210201012936.8553.log
Last 10 log lines:
^
clang -fno-strict-aliasing -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -L brew --prefix bzip2/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -L /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -L brew --prefix bzip2/include -std=c99 -Werror=implicit-function-declaration -I./Include/internal -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/rayhanmuktader/.pyenv/versions/3.8.0/include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/rayhanmuktader/.pyenv/versions/3.8.0/include -DPy_BUILD_CORE_BUILTIN -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
clang: warning: argument unused during compilation: '-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-Lbrew' [-Wunused-command-line-argument]
c
lang: warning: argument unused during compilation: '-L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-Lbrew' [-Wunused-command-line-argument]
1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....
1 warning generated.
My .zshrc looks this at the moment:
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi
export PATH="$HOME/.poetry/bin:$PATH"
export LDFLAGS="-L $(xcrun --show-sdk-path)/usr/lib -L brew --prefix bzip2/lib"
export CFLAGS="-L $(xcrun --show-sdk-path)/usr/include -L brew --prefix bzip2/include"
I also tried SYSTEM_VERSION_COMPAT=1 pyenv install 3.8.0 but that didn't solve it either. If it helps anything, note that it hangs for an unusually long time at python-build: use zlib from xcode sdk line.
Thank you if read this far. Any help would be greatly appreciated.

This is what worked for me (kind of).
I followed step 1 and step 2 from https://koji-kanao.medium.com/install-python-3-8-0-via-pyenv-on-bigsur-b4246987a548
I upgraded to BigSur 11.2, installed XCode 12.5 beta from https://developer.apple.com/xcode/resources/. I also installed Command Line Tools for Xcode 12.5 beta from https://developer.apple.com/download/more/?=command%20line%20tools (I don't know if this was necessary)
My .zshrc looks like this:
eval "$(pyenv init -)"
fi
export PATH="$HOME/.poetry/bin:$PATH"
Then in order to install python 3.8.0 I ran:
$ CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" pyenv install --patch 3.8.0 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)
This actually installed it. HOWEVER, when I ran poetry install on an existing project, it started installing the package then it suddenly blew up with the following error:
• Installing pynacl (1.3.0): Failed
ERROR: Failed building wheel for PyNaCl
Failed to build PyNaCl
ERROR: Could not build wheels for PyNaCl which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 20.2.2; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/rayhanmuktader/Library/Caches/pypoetry/virtualenvs/desk-backend-kAA1Wc9W-py3.8/bin/python -m pip install --upgrade pip' command.
at ~/.poetry/lib/poetry/utils/env.py:1074 in _run
1070│ output = subprocess.check_output(
1071│ cmd, stderr=subprocess.STDOUT, **kwargs
1072│ )
1073│ except CalledProcessError as e:
→ 1074│ raise EnvCommandError(e, input=input_)
1075│
1076│ return decode(output)
1077│
1078│ def execute(self, bin, *args, **kwargs):
The solution to this problem was running $ /Users/rayhanmuktader/Library/Caches/pypoetry/virtualenvs/desk-backend-kAA1Wc9W-py3.8/bin/python -m pip install pynacl==1.3.0 to install the pynacl package. Then running poetry install again to continue the installation process. Rinse and repeat for every package it fails to install through poetry install.

Error installing mply library. Fatal error: gsl/gsl_sf.h: No such file or directory

I'm trying to install the latest version of mply (3.5.0) on xubuntu (ubuntu 17.10):
download the .tar.gz
Unzipped it
python setup.py build. Here I get the error displayed below.
also tried directly python setup.py install. same error.
tried installing gsl library by downloading the latest version, and following all the steps in https://coral.ise.lehigh.edu/jild13/2016/07/11/hello/. I have it installed in /home/gsl
still the same error
tried various other installations (sudo apt-get install libgsl0ldbl, sudo apt-get install libgs123, sudo apt-get install libgsl0-dev). Nothing works.
tried copying the folder /home/gsl/include/gsl inside the mply-3.5.0/mply/gsl folder . this slightly changes something, because now I get fatal error: gsl/gsl_sf_result.h: No such file or directory (So it finds gsl_sf.h, but not gsl_sf_result.h (both these files are present inside the /gsl/include/gsl folder).
tried with
python setup.py build_ext --include-dirs=/home/gls/include/gls
still nothing
tried to copy the /home/gsl/include/gsl folder inside my /usr/local/include:
cp -R /home/gsl/include/gsl /usr/local/include/gsl
still nothing.
Help!
Error message:
gcc -pthread -B /home/lorenzo/anaconda3/compiler_compat
-Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -I/home/lorenzo/anaconda3/include/python3.6m
-I/home/lorenzo/anaconda3/lib/python3.6/site-packages/numpy/core/include
-I/home/lorenzo/anaconda3/include/python3.6m -c mlpy/gsl/gsl.c
-o build/temp.linux-x86_64-3.6/mlpy/gsl/gsl.o
mlpy/gsl/gsl.c:223:10: fatal error: gsl/gsl_sf.h: No such file or directory
#include "gsl/gsl_sf.h"

SOLVED by re-installing gsl library using conda:
conda install gsl

pip install: Command clang failed [duplicate]

Python library gevent, version 0.13.6 (the current version on PyPI) will not pip install on OS X Lion, Python 2.7 (and probably others.) It works fine on Snow Leopard.
How can I get this library installed?
Bonus points if it can be done using pip install, rather than a manual or custom process, because then it will play nicely with automated builds.
Here is my pip install output:
pip install gevent
Downloading/unpacking gevent
Running setup.py egg_info for package gevent
Requirement already satisfied (use --upgrade to upgrade): greenlet in ./tl_env/lib/python2.7/site-packages (from gevent)
Installing collected packages: gevent
Running setup.py install for gevent
building 'gevent.core' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.6-intel-2.7/gevent/core.o
In file included from gevent/core.c:225:
gevent/libevent.h:9:19: error: event.h: No such file or directory
gevent/libevent.h:38:20: error: evhttp.h: No such file or directory
gevent/libevent.h:39:19: error: evdns.h: No such file or directory
gevent/core.c:361: error: field ‘ev’ has incomplete type
gevent/core.c:741: warning: parameter names (without types) in function declaration
gevent/core.c: In function ‘__pyx_f_6gevent_4core___event_handler’:
gevent/core.c:1619: error: ‘EV_READ’ undeclared (first use in this function)
gevent/core.c:1619: error: (Each undeclared identifier is reported only once
gevent/core.c:15376: warning: assignment makes pointer from integer without a cast
[... about 1000 more lines of compiler errors...]
gevent/core.c:15385: error: dereferencing pointer to incomplete type
gevent/core.c: In function ‘__pyx_pf_6gevent_4core_4http___init__’:
gevent/core.c:15559: warning: assignment makes pointer from integer without a cast
gevent/core.c: At top level:
gevent/core.c:21272: error: expected ‘)’ before ‘val’
lipo: can't figure out the architecture type of: /var/folders/s5/t94kn0p10hdgxzx9_9sprpg40000gq/T//cczk54q7.out
error: command 'gcc-4.2' failed with exit status 1
Complete output from command /Users/jacob/code/toplevel/tl_env/bin/python -c "import setuptools;__file__='/Users/jacob/code/toplevel/tl_env/build/gevent/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /var/folders/s5/t94kn0p10hdgxzx9_9sprpg40000gq/T/pip-s2hPd3-record/install-record.txt --install-headers /Users/jacob/code/toplevel/tl_env/bin/../include/site/python2.7:
running install
running build
running build_py
running build_ext
building 'gevent.core' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c gevent/core.c -o build/temp.macosx-10.6-intel-2.7/gevent/core.o

Don't post the entire thing! That's too much! 90% of the time, the first error is enough...
gevent/libevent.h:9:19: error: event.h: No such file or directory
This means that the library which provides the event.h header is not installed. The library is called libevent (website).
In general, compilation errors like these are a flaw in the build scripts. The build script should give an error message that libevent is not installed, and it is a bug that it did not do so.
To get libevent from MacPorts and then manually tell compiler with CFLAGS environment variable where to find event.h and libevent while running pip.
sudo port install libevent
CFLAGS="-I /opt/local/include -L /opt/local/lib" pip install gevent
You can also use homebrew for installing libevent : brew install libevent
(from David Wolever's comment)

CFLAGS='-std=c99' pip install gevent
See in: Can't install gevent OSX 10.11
on OS X 10.11, clang uses c11 as the default, so just turn it back to c99.

After a while, I realized that the paths for the CFLAGS variable mentioned above works when installing libevent from port, but not from brew. The following worked for me (on OSX Mavericks):
$ brew install libevent
$ export CFLAGS="-I /usr/local/Cellar/libevent/2.0.21/include -L /usr/local/Cellar/libevent/2.0.21/lib"
$ pip install gevent

This is the way I found the easiest:
install libevent using homebrew
$ brew install libevent
install gevent
$ pip install gevent
This was the only way I could get it to work.

Found this answer when looking for help installing on Snow Leopard, posting this in case someone else comes this way with the same problem.
I had libevent installed via macports.
export CFLAGS=-I/opt/local/include
export LDFLAGS=-L/opt/local/lib
sudo pip install gevent

I had libevent installed via brew and it failed too, what worked was similar to what Stephen done, but pointing to brew default install:
CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib pip install gevent

In case you install all from sources and use csh the following works on mac os 10.9
download latest stable http://libevent.org/ libevent-2.0.21-stable
./configure
make
sudo make install
virtualenv env
source env/bin/activate.csh
setenv CFLAGS "-I /usr/local/include -L /usr/local/lib"
pip install gevent

I use virtualenv and virtualenv wrapper, and so I wanted this to be self contained. I got gevent working like so:
Assuming you have virtual env setup, then:
workon {my_virtual_env}
Then download libevent and install it against the virtualenv.
curl -L -O https://github.com/downloads/libevent/libevent/libevent-2.0.21-stable.tar.gz
tar -xzf libevent-2.0.21-stable.tar.gz
cd libevent-2.0.21-stable
./configure --prefix="$VIRTUAL_ENV"
make && make install
I'm assuming you've got gcc 5+ installed (I use brew)
Hope this helps.

sudo pip install cython git+git://github.com/gevent/gevent.git#egg=gevent

I am using MacOs High Sierra (10.13.3)
First I did :
brew install libevent
I upgraded my pip version to pip-18.0.
then tried installing again with following :-
pip install gevent
it worked.

Installing PyCrypto on Ubuntu - fatal error on build

Having looked at other similar threads, I still can't get pycrypto running.
I'm trying to get it working on my Ubuntu laptop - but I couldn't manage it on my Windows PC either.
I downloaded pycrypto-2.6, extracted it, and ran
python setup.py build
but then this happened
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
building 'Crypto.Hash._MD2' extension
gcc -pthread -fno-strict-aliasing -fwrapv -Wall -Wstrict-prototypes -fPIC -std=c99 -O3 - fomit-frame-pointer -Isrc/ -I/usr/include/python2.7 -c src/MD2.c -o build/temp.linux-i686-?2.7/src/MD2.o
src/MD2.c:31:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
Would appreciate any help.

You need to install the Python development files. I believe this will do it:
sudo apt-get install python-dev

On Ubuntu, I needed some other packages for it to succeed:
apt-get install autoconf g++ python2.7-dev
pip install pycrypto

On Ubuntu and if you use Python 3.x you will need:
sudo apt-get install gcc python3-dev
you probably already have gcc but just in case if you are trying this command from Dockerfile with base image python:3.6.4-slim-jessie then you will also need gcc.

August 2021
For python 3.8 users run
sudo apt-get install python3.8-dev
and try to install pycrypto again
pip install pycrypto