Develop Reference

waf check_python_headers fails with python installed with pyenv

I'm trying to build a project using the waf build tool.
The build fails in the configure step when checking the python headers.
I narrowed the problem down to this minimal example of my wscript:
def options(ctx):
ctx.load('compiler_c')
def configure(ctx):
ctx.load('compiler_c python')
ctx.check_python_version((2,7,0))
if int(ctx.env.PYTHON_VERSION[0]) == 2:
print ('→ Configuring for python2')
else:
print ('→ Configuring for python3')
ctx.check_python_headers(features='pyext')
I use python versions (3.6.12 & 3.7.9) that I installed with pyenv on Ubuntu 20.04.
I run that like this and it fails with an error that I cannot make sense of:
$ python waf configure
Setting top to : /home/myuser/waf-test
Setting out to : /home/myuser/waf-test/build
Checking for 'gcc' (C compiler) : /usr/bin/gcc
Checking for program 'python' : /home/myuser/.pyenv/versions/3.7.9/bin/python
Checking for python version >= 2.7.0 : 3.7.9
→ Configuring for python3
python-config : /home/myuser/.pyenv/versions/3.7.9/bin/python-config
Asking python-config for pyext '--cflags --libs --ldflags' flags : yes
Testing pyext configuration : Could not build python extensions
The configuration failed
(complete log in /home/myuser/waf-test/build/config.log)
$ tail -n 20 build/config.log
[1/2] Compiling build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/test.c
['/usr/bin/gcc', '-fPIC', '-g', '-fwrapv', '-O3', '-I../../../../.pyenv/versions/3.7.9/include/python3.7m', '-DPYTHONDIR="/usr/local/lib/python3.7/site-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3.7/site-packages"', '-DNDEBUG', '../test.c', '-c', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/test.c.1.o']
[2/2] Linking build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so
['/usr/bin/gcc', '-shared', 'test.c.1.o', '-o/home/myuser/waf-test/build/.conf_check_d3b505aa7ab58576b6d76a2fc3091b1f/testbuild/testprog.cpython-37m-x86_64-linux-gnu.so', '-Wl,-Bstatic', '-Wl,-Bdynamic', '-L/home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu', '-L/home/myuser/.pyenv/versions/3.7.9/lib', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm', '-lpython3.7m', '-lcrypt', '-lpthread', '-ldl', '-lutil', '-lm']
err: /usr/bin/ld: /home/myuser/.pyenv/versions/3.7.9/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.a(pylifecycle.o): relocation R_X86_64_PC32 against symbol `Py_VerboseFlag' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
from /home/myuser/waf-test: Test does not build: Traceback (most recent call last):
File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Configure.py", line 335, in run_build
bld.compile()
File "/home/myuser/waf-test/.waf3-2.0.20-36f5354d605298f6a89c09e0c7ef6c1d/waflib/Build.py", line 176, in compile
raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
-> task in 'testprog' failed with exit status 1 (run with -v to display more information)
Could not build python extensions
from /home/myuser/waf-test: The configuration failed
Note that this problem does not appear with python 3.8.6 that was also installed with pyenv. I can configure without errors.
My questions: How can it be that the python versions in pyenv differ in a way that waf checks fail in one version and not in another?
Since I simply install the versions with pyenv install 3.6.12, what do I need to change in the install so my build succeeds?

Well, the error is that, when creating the test program to check if pyext is correctly set, the compiler complains about the fact that pylifecycle.o included in libpython3.7m.a is non relocatable ie non compiled with the -fPIC flag.
As the code fragment generated by waf is compiled with -fPIC, it seems that the problem is that pylifecycle.o in libpython3.7m.a is not compiled with -fPIC, which makes sense as it is in a static library. Maybe the problem is that it uses the .a archive lib instead of the shared object one ie the .so.
Check if you have the shared object version of libpython ie libpython3.7m.so.

numpy/ndarrayobject.h: No such file or directory

I am trying to compile a project using cmake. This is the relevant CMakeLists.txt code snippet.
set(PYTHON3_NUMPY_INCLUDE_DIRS=/usr/local/lib/python3.6/dist-packages/numpy/core/include/)
set(PYTHON_INCLUDE_DIRS ${PYTHON_INCLUDE_DIRS} /usr/local/lib/python3.6/dist-packages/numpy/core/include)
find_package(Python COMPONENTS Interpreter Development)
find_package(PythonLibs REQUIRED)
...
include_directories( ${PYTHON_INCLUDE_DIRS} )
To ensure that it is working properly, I print the relevant info
execute_process ( COMMAND ${PYTHON_EXECUTABLE} -c "import numpy; print(numpy.get_include())" OUTPUT_VARIABLE NUMPY_INCLUDE OUTPUT_STRIP_TRAILING_WHITESPACE)
message (STATUS "NUMPY_INCLUDE: " ${NUMPY_INCLUDE})
This is how I compile
cmake -DPYTHON_EXECUTABLE=/usr/bin/python3 .
Here is my output of cmake, the relevant bits.
-- Found Python: /usr/bin/python3.6 (found version "3.6.9") found components: Interpreter Development
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found version "3.6.9")
-- NUMPY_INCLUDE: /usr/local/lib/python3.6/dist-packages/numpy/core/include
So far everything is perfect and numpy is found.
However, on running make, I get the following
fatal error: numpy/ndarrayobject.h: No such file or directory
#include <numpy/ndarrayobject.h>
From what I can see I've covered all bases, yet I'm getting this error. What do I do?

make pycaffe fatal error: 'Python.h' file not found

I compiled caffe on a mac running OSX 10.9.5 and I know trying to compile pycaffe. When I run make pycaffe in the caffe root folder, I get:
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
python/caffe/_caffe.cpp:1:10: fatal error: 'Python.h' file not found
#include <Python.h> // NOLINT(build/include_alpha)
^
1 error generated.
make: *** [python/caffe/_caffe.so] Error 1
how can I fix this?
Perhaps is something wrong with Makefile.config. How do I know what is my PYTHONPATH?

Looking at the comments, I see that you use Anaconda. In Makefile.config, you should uncomment the lines dedicated to Anaconda:
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python2.7 \
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
Python.h is in $(ANACONDA_HOME)/include/python2.7 as you can see running sudo find / -name 'Python.h'.

I met this problem too.
I have set the PYTHON_INCLUDE PATH
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python2.7
But it still can't find the Python.h
So I just give the include path manually to the compiler as follows:
export CPLUS_INCLUDE_PATH=/home/woolawren/anaconda2/include/python2.7/:$CPLUS_INCLUDE_PATH
if you don't use anaconda2, you can use:
export CPLUS_INCLUDE_PATH=/usr/include/python2.7:$CPLUS_INCLUDE_PATH
I have successfully done "make pycaffe" by doing this.

I just finished a tedious Caffe install on Arch Linux; hopefully my install notes (link below) will help others.
While specific to my Caffe install, those notes address the "Python.h" install error (this Question), as well as a downstream issue mentioned in another SO question,
Import caffe error.
https://stackoverflow.com/questions/28177298/import-caffe-error
My gist file (notes):
Caffe Installation Notes
https://gist.github.com/victoriastuart/fb2cb22209ccb2771963a25c06221213

I uncommented the below code in Makefile.config
PYTHON_INCLUDE := /usr/include/python3.5m \
/usr/lib/python3.5/dist-packages/numpy/core/include
Then did sudo make pycaffe.
It worked.

How can I build my C extensions with MinGW-w64 in Python?

So I have a few Python C extensions I have previously built for and used in 32 bit Python running in Win7. I have now however switched to 64 bit Python, and I am having issues building the C extension with MinGW-w64.
I made the changes to distutils as per this post, but I am getting some weird errors suggesting something is wrong:
$ python setup.py build
running build
running build_ext
building 'MyLib' extension
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -mdll -O -Wall -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c MyLib.c -o build\temp.win-amd64-2.7\Release\mylib.o
MyLib.c: In function 'initMyLib':
MyLib.c:631:5: warning: implicit declaration of function 'Py_InitModule4_64' [-Wimplicit-function-declaration]
writing build\temp.win-amd64-2.7\Release\MyLib.def
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -shared -s build\temp.win-amd64-2.7\Release\mylib.o build\temp.win-amd64-2.7\Release\MyLib.def -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -lpython27 -o build\lib.win-amd64-2.7\MyLib.pyd
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x13d): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1275): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1eef): undefined reference to `__imp_PyExc_ImportError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f38): undefined reference to `__imp_PyExc_AttributeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f4d): undefined reference to `__imp_PyCObject_Type'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f61): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1fc7): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1ffe): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x2042): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x206c): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x208a): more undefined references to `__imp_PyExc_RuntimeError' follow
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x20a7): undefined reference to `__imp_PyExc_ImportError'
collect2.exe: error: ld returned 1 exit status
error: command 'x86_64-w64-mingw32-gcc' failed with exit status 1
I have googled around quite a bit to find information, but it's not easy to find a definite answer. Could someone shed some light on this? What further changes should I do to be able to successfully build C extensions for 64 bit Python in Win7?
EDIT:
After some helpful pointers in cgohlke's comments below I managed to generate libpython27.a. However after following the advice on this post (2nd to last) I still had a the __imp_Py_InitModule4_64 error. After some serious Google-fu I managed to trip over this post telling me to rename the Py_InitModule4 line to Py_InitModule4_64. After that everything worked swimmingly.

This worked for me with Python 3.3 :
create static python lib from dll
python dll is usually in C:/Windows/System32; in msys shell:
gendef.exe python33.dll
dlltool.exe --dllname python33.dll --def python33.def --output-lib libpython33.a
mv libpython33.a C:/Python33/libs
use swig to generate wrappers
e.g., swig -c++ -python myExtension.i
wrapper MUST be compiled with MS_WIN64, or your computer will crash when you import the class in Python
g++ -c myExtension.cpp -I/other/includes
g++ -DMS_WIN64 -c myExtension_wrap.cxx -IC:/Python33/include
shared library
g++ -shared -o _myExtension.pyd myExtension.o myExtension_wrap.o -lPython33 -lOtherSharedLibs -LC:/Python33/libs -LC:/path/to/other/shared/libs
make sure all shared libs (gdal, OtherSharedLibs) are in your PATH
(windows does not use LD_LIBRARY_PATH or PYTHONPATH)
in Python, just: import myExtension
voila!

I realize this is an old question, but it is still the top search result. Today, in 2019, I was able to do this:
https://github.com/PetterS/quickjs/commit/67bc2428b8c0716538b4583f4f2b0a2a5a49106c
In short:
Make sure a 64-bit version of mingw-w64 is in the PATH.
Monkey-patch distutils:
import distutils.cygwinccompiler
distutils.cygwinccompiler.get_msvcr = lambda: []
Some differences in the shell w.r.t. escaping.
extra_link_args = ["-Wl,-Bstatic", "-lpthread"] in order to link statically and not have extra runtime deps.
pipenv run python setup.py build -c mingw32 now works.

Here is a example code for VC++ Build Tools
https://github.com/starnight/python-c-extension/tree/master/00-HelloWorld
You could try:
python setup.py -c mingw32
However this is not work for me.
My Solution is:
install Anaconda 64bit python 3.6
install mingw64
add mingw64/bin to PATH
compile dll from c file by
gcc -c libmypy.c -IC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\include
gcc -shared -o libmypy.dll libmypy.o -LC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\libs -lPython36
load dll file in .py script
from ctypes import *
m = cdll.LoadLibrary(r"C:\{path_to_dll}\libmypy.dll")
print(m.hello())

I created a monkey-patch for setuptools to let you to build_ext with mingw64 on Windows easily. See https://github.com/imba-tjd/mingw64ccompiler

I used this thread to wade through learning how to make a C extension, and since most of what I learned is in it, I thought I'd put the final discovery here too, so that someone else can find it if they are looking.
I wasn't trying to compile something big, just the example in Hetland's Beginning Python. Here is what I did (the example C pgm is called palindrome.c). I'm using Anaconda with python 3.7 in it, and the TDM-GCC version of MinGW64. I put all of the tools used into my Path, and all of the paths needed in PYTHONPATH, and the ..\Anaconda3 directory into PYTHON_HOME. I still ended up using explicit paths on some things.
I created the libpython37.a library with gendef.exe and dlltool.exe as Mark said above, and put it in ..\Anaconda3\libs.
I followed the prescription in Hetland:
gcc -c palindrome.c
gcc -I$PYTHON_HOME -I$PYTHON_HOME/Include -c palindrome_wrap.c
The second failed, the compiler couldn't find Python.h, the following worked:
gcc -I[somedirectories]\Anaconda3\Include -c palindrome_wrap.c
I then did, as many have said, including Hetland 3rd ed.,
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.dll
This did not work. Even with the Load Library cswu used (which I found elsewhere, too).
So I gendef'd _palindrome.dll and couldn't find the function in it, "is_palindrome" in the exports. I went through some of the SWIG documentation, and declared the function both in the %{ %} section and below it, both extern, that finally got the function extern'd in palindrome_wrap.c as it should have been. But no export, so I went back into palindrome.c and redeclared the function as:
declspec(dllexport) extern int __stdcall is_palindrome(char* text)
and redeclared it in palindrome.i in both places as above with this signature.
Partial success! It got listed in the Export section when I gendef'd _palindrome.dll and I could do cswu's call using Load Library. But still not do what Hetland says and do
import _palindrome
in Python.
Going back to all the sources again, I could not figure this out. I finally started reading the SWIG documentation from the beginning leaving no stone unturned -- Searching through the manual doesn't produce the place found.
At the end of Introduction Sec. 2.7 Incorporating Into a Build System, under the sample Make process, it says:
"The above example will generate native build files such as makefiles, nmake files and Visual Studio projects which will invoke SWIG and compile the generated C++ files into _example.so (UNIX) or _example.pyd (Windows). For other target languages on Windows a dll, instead of a .pyd file, is usually generated."
And that's the answer to the last problem:
The compile step for the dll should read:
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.pyd
(I didn't go back and change out my declspec declarations so I don't know whether they were necessary, so they were still there too).
I got a file, _palindrome.pyd
Which if in the PYTHONPATH (mine was local) works, and one can then do
import _palindrome
from _palindrome import is_palindrome
and use the exported, properly wrapped and packaged C function, compiled with TDM-GCC, in python as promised. gcc, which is MinGW64 in a different installation, knows how to do the .pyd file. I diffed the dll and pyd since they were the same byte length. They are not the same at hundreds of points.
Hope this helps someone else.

bjam `Unable to find file or target named 'libboost_python'`

What am I missing in my Boost.Python configuration/installation?
I'm trying to compile tutorial example, and I get error with libboost_python not found
cd /usr/share/doc/libboost1.42-doc/examples/libs/python/example/tutorial
bjam
error: Unable to find file or target named
error: 'libboost_python'
error: referred from project at
error: '.'
But the library is there, ldconfig.real has been run:
/usr/lib/libboost_python.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py26.a -> libboost_python-py26.a
/usr/lib/libboost_python-mt-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-mt-py27.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py26.a
/usr/lib/libboost_python-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py27.a
/usr/lib/libboost_python-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py27.so.1.42.0
/usr/lib/libboost_python.so -> libboost_python-py27.so
I'm using default libboost packages from Ubuntu 11.04.
My user-config.jam is
using python : 2.7 ;

I had a similar problem on ubuntu 12.04 where I installed all the boost libraries as a package. I found the solution here:
http://jayrambhia.wordpress.com/2012/06/25/configuring-boostpython-and-hello-boost/
It turns out that you do not need to use bjam at all. A makefile suffices. I will repeat the the solution from the above link here:
1.) Install the libboost-python package
2.) Create a hello world source file called 'hello_ext.c':
char const* greet()
{
return "hello, world";
}
#include<boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet",greet);
}
3.) Create a makefile:
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr /lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).c
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).c
4.) make
make
5.) Ready to use. In python:
import hello_ext
print hello_ext.greet()

Still not sure it that's the proper way, seems little hackish, but following helped:
In Jamroot file replaced
project
: requirements <library>libboost_python ;
with
project
: requirements <library>/usr/lib/libboost_python.so ;

You could have a site-config file with something like the following ;
using boost : 1.48 : <include>/usr/include/boost-1_48 <library>/usr/lib ;
(you need the < library > bit, not sure why)
then you can do stuff like.
project foo
: <library>/boost//python
Makes things easier in the long run, as you inevitably will have to change boost version at some point.