I have a distutils setup script with an Extension section, which looks something like this:
from distutils.core import setup, Extension
my_module = Extension('my_module',
sources = ['my_file.c', 'my_other_file.c'])
setup (name = 'my_module',
version = '1.0',
description = 'My module',
ext_modules = [my_module])
Running setup.py build works fine on my Mac. When I move to a Debian machine, it fails:
error: Python/Python.h: No such file or directory
I have python2.6 and python2.6-dev installed, and the file is present at /usr/include/Python2.6.
The command it executes for the problem file:
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c my_module.c -o -build/XYZ/my_module.o
So it is passing in the location of the header file.
The only obvious difference between the Mac vs Linux environment is gcc-4.2 vs gcc-4.4 and Python 2.7 vs Python 2.6
Ideas?
EDIT:
In the C file in question:
#include <Python/Python.h>
#include <Python/structmember.h>
May be in your module, you need to include "Python.h" instead of "Python/Python.h"?
or you may try exporting include path, and try compiling again with gcc or g++?
export C_INCLUDE_PATH=/usr/include/python2.6:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/include/python2.6:$CPLUS_INCLUDE_PATH
In my case, I was missing python3-dev, sudo apt-get install python3-dev fixed it.
Related
Hit problem when convert python code to shared object by Cython.
setup file here:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize("hello.py")
)
So everything works fine on my Ubuntu desktop util transferred to CentOS.
Got error:
undefined symbol: PyUnicodeUCS4_DecodeUTF8
I googled and find there are many questions on this, but, almost all of them say the root cause is python with UCS2 or UCS4, and I understand this, didn't find one show the way to solve this.
IMO, ways to solve:
rebuild python to get the right version by "--enable-unicode=ucs4/ucs2"
But I need to reinstall all packages
Compile the code from another desktop whose python with the right UCS
Now, I wanna if there is way to set Cython to compile with specified UCS mode.
Any suggestions is great appreciated.
Thanks.
First, to answer your actual question:
I wanna if there is way to set Cython to compile with specified UCS mode.
You can build a separate python installation from source and link Cython against its headers. To find the headers, you can use the python-config tool (or python3-config for Python 3). It is usually located in the bin directory where the python executable is:
$ # system python on my machine (macos):
$ which python-config
/usr/bin/python-config
$ # python 3 installation
$ which python3-config
/Library/Frameworks/Python.framework/Versions/3.6/bin/python3-config
$ python-config --cflags
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE
$ python-config --ldflags
-L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl -framework CoreFoundation
Copy the output to the setup.py:
from setuptools import setup
from setuptools.extension import Extension
from Cython.Build import cythonize
cflags_ucs4 = [
'-I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m',
'-I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m',
...
]
ldflags_ucs4 = [
'-L/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin',
'-lpython3.6m',
...
]
cflags_ucs4 = [...]
ldflags_ucs2 = [...]
should_build_ucs2 = False # i.e. could be passed via sys.argv
if should_build_ucs2:
cflags = cflags_ucs2
ldflags = ldflags_ucs2
else:
cflags = cflags_ucs4
ldflags = ldflags_ucs4
extensions = [
Extension('hello.py', extra_compile_args=cflags, extra_link_args=ldflags),
]
setup(
ext_modules = cythonize(extensions)
)
However, I do not recommend doing that as you won't win anything by doing that - you will still need to build and distribute two separate packages (one for UCS2, another for UCS4) which is messy to maintain.
Instead, if you are building a wheel that should be installable on a wide range of Linux distros (what is most probably your actual goal), I would suggest to make your build compliable with PEP 513 (manylinux1 packages).I suggest you to read it through as it was very helpful for me when I faced the problem of distributing Linux-compliant wheels.
Now, one way to get a manylinux1-compliant wheel is to build the wheel on your machine, then running auditwheel to check for platform-specific issues and trying to resolve them:
$ pip install auditwheel
$ python setup.py bdist_wheel
$ # there should be now a mypkg-myver-cp36-cp36m-linux_x86_64.whl file in your dist directory
$ auditwheel show dist/mypkg-myver-cp36-cp36m-linux_x86_64.whl
$ # check what warnings auditwheel produced
$ # if there are warnings, try to repair them:
$ auditwheel repair dist/mypkg-myver-cp36-cp36m-linux_x86_64.whl
This should generate a wheel file named mypkg-myver-cp36-cp36m-manylinux1_x86_64.whl in a wheelhouse directory. Check again that everything is fine now by running auditwheel show wheelhouse/mypkg-myver-cp36-cp36m-manylinux1_x86_64.whl. If the wheel is now consistent with manylinux1, you can distribute it and it should work on most Linux distros (at least those with glibc; distros with musl like Alpine won't work, you will need to build a separate wheel if you want to support it).
What should you do if auditwheel can't repair your wheel? The best way is to pull a special docker container provided by PyPA for building manylinux1-compliant wheels (this is what I'm using myself):
$ docker pull https://quay.io/repository/pypa/manylinux1_x86_64
A wheel built inside this container will work on most of the Linux distros (excluding some exotic ones like Alpine).
I have created a python extension for a C program. In linux, using gcc, everything works correctly and the extension can be installed typing:
sudo python setup.py install
But when I try to use it in OS X:
GCC 4.7:
I have installed gcc 4.9 using macports and I have added this line in my setup.py file
import os
os.environ["CC"]="gcc-mp-4.9"
And when I type sudo python setup.py install
I obtain this error:
unrecognized command line option '-Wshorten-64-to-32'
I have been looking for the solution and everybody say "Use clang instead of gcc" to solve that issue.
Clang 3.8:
I have also installed clang 3.8 (3.5 is installed in os X but it doesn't have openmp) and I have modified the file setup.py:
import os
os.environ["CC"]="clang-mp-3.8"
And I obtain this error:
unknown argument: '-mno-fused-madd'
In some forums I have found a possible solution for this issue setting an empty value for CFLAGS:
sudo CFLAGS="" python setup.py install
But I obtain a new error:
library not found for -lgomp
I use -fopenmp but I do not why -fgomp is called. In some forums people say that I must use gcc instead of clang, so I am at the starting point again.
I would like to find a solution to easily isntall this extension in OS X because I would like to create an extension that can be easily installed by anyone.
I had a similar problem. Python is built with clang and uses clang-specific CFLAGS:
>>> import sysconfig
>>> sysconfig.get_config_var("CFLAGS")
'-fno-strict-aliasing -fno-common -dynamic -arch x86_64
-arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing
-fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall
-Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g
-fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE'
Distutils copies this information to the "UnixCCCompiler" instance, used to make the extension. However, as you found out, -Wshorten-64-to-32 is clang specific.
My solution was to modify how distutils builds extensions. The following removes that option from the list of command-line arguments to pass to the compiler before calling the real compilation code. (You code may not need to be so complicated. I support several compilers and configurations.)
def _is_gcc(compiler):
return "gcc" in compiler or "g++" in compiler
class build_ext_subclass( build_ext ):
def build_extensions(self):
c = self.compiler.compiler_type
if c == "unix":
compiler_args = self.compiler.compiler
c = compiler_args[0] # get the compiler name (argv0)
if _is_gcc(c):
names = [c, "gcc"]
# Fix up a problem on older Mac machines where Python
# was compiled with clang-specific options:
# error: unrecognized command line option '-Wshorten-64-to-32'
compiler_so_args = self.compiler.compiler_so
for args in (compiler_args, compiler_so_args):
if "-Wshorten-64-to-32" in args:
del args[args.index("-Wshorten-64-to-32")]
build_ext.build_extensions(self)
Then tell setup() to use this new subclass to build extensions:
setup(name = ...
cmdclass = {"build_ext": build_ext_subclass},
)
I build libyaml and install it into a local area:
yaml-0.1.5 $ ./configure --prefix=/usr/local/sqlminus
yaml-0.1.5 $ make install
yaml-0.1.5 $ ls -l /usr/local/sqlminus/include/yaml.h
-rw-r--r--# 1 mh admin 54225 Jan 5 09:05 /usr/local/sqlminus/include/yaml.h
But when I build PyYAML, it cannot find yaml.h.
PyYAML-3.11 $ /usr/local/sqlminus/bin/python setup.py build
checking if libyaml is compilable
gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -I/usr/local/sqlminus/include/python2.7
-c build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c
-o build/temp.macosx-10.4-x86_64-2.7/check_libyaml.o
build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:2:10:
fatal error: 'yaml.h'
file not found
#include <yaml.h>
^
1 error generated.
How can I tell PyYAML where I've installed libyaml?
(update) Based on dotslash's comment below, editing setup.cfg and adding these two lines made everything work smoothly.
include_dirs=/usr/local/sqlminus/include
library_dirs=/usr/local/sqlminus/lib
(end update)
I think you should install dependencies.
If you are using Ubuntu or Debian based system, you could search by this
apt-cache search libyaml
Then you may find there are some packages related.
I would suggest you try to install this: apt-get install libyaml-dev -y
If you are using Mac OS, you could change the source in file check_libyaml.c, tell it what the absolute path of yaml.h is.
Or just specify the path while compiling
python setup.py config --with-includepath=/path/to/your/install/of/python/includes/
Then go compiling.
More info can be found here.
Hope this be helpful.
Based on dotslash's comment, editing setup.cfg and adding these two lines made everything work smoothly:
include_dirs=/usr/local/sqlminus/include
library_dirs=/usr/local/sqlminus/lib
I am trying to install biopython 1.65 in debian. I have the dependencies Numpy and Scipy.
When I try to build it, it fails:
python setup.py build
running build
running build_py
running build_ext
building 'Bio.cpairwise2' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c Bio/cpairwise2module.c -o build/temp.linux-x86_64-2.7/Bio/cpairwise2module.o
Bio/cpairwise2module.c:12:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Somebody know how I can solve it?
Many thanks
#include "Python.h"
tells the preprocessor to search a local file, and if it doesn't exists there, the preprocessor changes it to
#include <Python.h>
which should be located in /usr/include/python2.7 (which is passed as argument to gcc). Many Linux distros don't have header files installed by default, so you have to install it manually.
Header files for python2.7 are shipped with package libpython2.7-dev
You can find which package to install by searching it with aptitude, synaptic or apt-cache search adding dev after the package name (in that case python dev); the name could be different than the installed one.
Had the same issue on Fedora, what helped was:
yum install python-devel
I've tried using pip install matplotlib and git clone then python setup.py install as described in the installation faq for Mac OS 10.7. But I get the same error:
[...]
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -DPY_ARRAY_UNIQUE_SYMBOL=MPL_ARRAY_API -DPYCXX_ISO_CPP_LIB=1 -I/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include -I. -I/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include/freetype2 -I./freetype2 -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/ft2font.cpp -o build/temp.macosx-10.7-intel-2.7/src/ft2font.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
In file included from src/ft2font.cpp:3:
src/ft2font.h:16:22: error: ft2build.h: No such file or directory
src/ft2font.h:17:10: error: #include expects "FILENAME" or <FILENAME>
src/ft2font.h:18:10: error: #include expects "FILENAME" or <FILENAME>
src/ft2font.h:19:10: error: #include expects "FILENAME" or <FILENAME>
src/ft2font.h:20:10: error: #include expects "FILENAME" or <FILENAME>
src/ft2font.h:21:10: error: #include expects "FILENAME" or <FILENAME>
In file included from src/ft2font.cpp:3:
src/ft2font.h:34: error: ‘FT_Bitmap’ has not been declared
src/ft2font.h:34: error: ‘FT_Int’ has not been declared
src/ft2font.h:34: error: ‘FT_Int’ has not been declared
src/ft2font.h:86: error: expected ‘,’ or ‘...’ before ‘&’ token
[...]
It seems like I'm missing some package installed in my system? Or there any other better way to install matplotlib?
Thanks!
UPDATE: by googling and search on SO I've found that I might lack the package freetype2, however, if I try to install it by homebrew I get a warning message:
[me # my mac]$ brew search freetype
Apple distributes freetype with OS X, you can find it in /usr/X11/lib.
However not all build scripts look here, so you may need to call ENV.x11
in your formula's install function.
I had this issue on Ubuntu server 12.04.
I had to install libfreetype6-dev and libpng-dev from the repositories. I was using a virtualenv and installing matplotlib using pip when I ran into this issue.
Hints that I needed to do this came from the warning messages that popup early in the matplotlib installation so keep an eye out for those messages which indicate a dependency is found, but not the headers.
Same error, the install worked on one of my Lion machines but not the other. Tracked it down to a missing pkg-config
$ brew install pkg-config
$ pip install -U 'http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.0/matplotlib-1.1.0.tar.gz/download'
(see also https://jholewinski.org/blog/installing-matplotlib-on-os-x-10-7-with-homebrew/ )
Thanks for the link above. I was able to get matplotlib working with some minor changes. I am documenting the specific error message I ran into, for future reference.
Env:
Mac OS X 10.7.4 (Lion) running stock python 2.7.1 (found in /usr/bin)
I started out trying to get matplotlib working against the default install. setup.py hinted at some problems specifically:
freetype2: found, but unknown version (no pkg-config)
* WARNING: Could not find 'freetype2' headers in any
* of '.', './freetype2'.
and
OPTIONAL BACKEND DEPENDENCIES
libpng: found, but unknown version (no pkg-config)
* Could not find 'libpng' headers in any of '.'
the setup step would fail with the following error message:
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/core/include/numpy/__multiarray_api.h:1187: warning: ‘int _import_array()’ defined but not used
lipo: can't open input file: /var/tmp//ccG28dDI.out (No such file or directory)
Googling for this file did not help with anything. I dug around a bit and decided it was not worth my time to go down the virtualenv route.
I ended up following the steps outlined in the jholewinski link above. Installed a new python 2.7.4 in /usr/local/ and followed the instructions there. (I re-installed pkg-config).
I was able to pull the latest version of matplotlib from git and it worked just fine.
Both the libpng (1.5.4) and freetype2 (13.2.7) libraries were present this time.