What am I missing in my Boost.Python configuration/installation?
I'm trying to compile tutorial example, and I get error with libboost_python not found
cd /usr/share/doc/libboost1.42-doc/examples/libs/python/example/tutorial
bjam
error: Unable to find file or target named
error: 'libboost_python'
error: referred from project at
error: '.'
But the library is there, ldconfig.real has been run:
/usr/lib/libboost_python.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py26.a -> libboost_python-py26.a
/usr/lib/libboost_python-mt-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-mt-py27.a -> libboost_python-py27.a
/usr/lib/libboost_python-mt-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py26.a
/usr/lib/libboost_python-py26.so -> libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py26.so.1.42.0
/usr/lib/libboost_python-py27.a
/usr/lib/libboost_python-py27.so -> libboost_python-py27.so.1.42.0
/usr/lib/libboost_python-py27.so.1.42.0
/usr/lib/libboost_python.so -> libboost_python-py27.so
I'm using default libboost packages from Ubuntu 11.04.
My user-config.jam is
using python : 2.7 ;
I had a similar problem on ubuntu 12.04 where I installed all the boost libraries as a package. I found the solution here:
http://jayrambhia.wordpress.com/2012/06/25/configuring-boostpython-and-hello-boost/
It turns out that you do not need to use bjam at all. A makefile suffices. I will repeat the the solution from the above link here:
1.) Install the libboost-python package
2.) Create a hello world source file called 'hello_ext.c':
char const* greet()
{
return "hello, world";
}
#include<boost/python.hpp>
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet",greet);
}
3.) Create a makefile:
PYTHON_VERSION = 2.7
PYTHON_INCLUDE = /usr/include/python$(PYTHON_VERSION)
# location of the Boost Python include files and library
BOOST_INC = /usr/include
BOOST_LIB = /usr/lib
# compile mesh classes
TARGET = hello_ext
$(TARGET).so: $(TARGET).o
g++ -shared -Wl,--export-dynamic $(TARGET).o -L$(BOOST_LIB) -lboost_python -L/usr /lib/python$(PYTHON_VERSION)/config -lpython$(PYTHON_VERSION) -o $(TARGET).so
$(TARGET).o: $(TARGET).c
g++ -I$(PYTHON_INCLUDE) -I$(BOOST_INC) -fPIC -c $(TARGET).c
4.) make
make
5.) Ready to use. In python:
import hello_ext
print hello_ext.greet()
Still not sure it that's the proper way, seems little hackish, but following helped:
In Jamroot file replaced
project
: requirements <library>libboost_python ;
with
project
: requirements <library>/usr/lib/libboost_python.so ;
You could have a site-config file with something like the following ;
using boost : 1.48 : <include>/usr/include/boost-1_48 <library>/usr/lib ;
(you need the < library > bit, not sure why)
then you can do stuff like.
project foo
: <library>/boost//python
Makes things easier in the long run, as you inevitably will have to change boost version at some point.
Related
I'm currently trying to write a 'setup.py' script that when the user installs the python package automatically compiles my C++ extension bound with 'pybind11'. In Windows, I haven't got any problem making it happen with the 'VS19 MSVC' compiler. But I'm trying to make it happen if the user has installed 'MinGW-w64' instead.
These are the package files:
**main.cpp**
#include <pybind11/pybind11.h>
int add(int i, int j) {
return i + j;
}
namespace py = pybind11;
PYBIND11_MODULE(pybind11_example, m) {
m.def("add", &add);
}
**setup.py**
from setuptools import setup, Extension
import pybind11
ext_modules = [
Extension(
'pybind11_example',
sources = ['main.cpp'],
include_dirs=[pybind11.get_include()],
language='c++'
),
]
setup(
name='pybind11_example',
ext_modules=ext_modules
)
Having the two files in the same folder and running from the command prompt:
python setup.py build
If the user has VS19 MSVC compiler installed it successfully generates **pybind11_example.pyd** that can be tested to work running with python:
import pybind11_example as m
print(m.add(1, 2))
But if the user has a Mingw-w64 compiler installed raises an error saying that Visual Studio 2015 is required.
Note that I can easily compile **main.cpp** in to **pybind11_example.pyd** manually with Mingw-w64 running:
g++ -static -shared -std=c++11 -DMS_WIN64 -fPIC -I C:\...\Python\Python38\Lib\site-packages\pybind11\include -I C:\ ... \Python\Python38\include -L C:\ ... \Python\Python38\libs main.cpp -o pybind11_example.pyd -lPython38
Is there a way to write **setup.py** in a way that if the user has Windows with a MinGW-w64 compiler automatically compile **main.cpp** into **pybind11_example.pyd** when installing the package without needing to make it manually?
Chek the answer to this question. They try to solve the opposite case, force msvc instead of mingw, but the approach with setup.cfg might help you.
And here the answer demonstrates how to specify command line parameters depending on the choice made by setup tools: if it is msvc then one set of parameters, and another set for mingw.
I belive the second approach should suite your needs - whichever compier is installled, you have the proper command line to build your module.
I am on MacOS 10.13.3 (17D47) with Python 2.7.14. I am in the process of building caffe w/ python. The project is CPU only. I am able to build caffe through make run test however, after attempting make pycaffe I get an error. This is the result:
touch python/caffe/proto/__init__.py
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
PROTOC (python) src/caffe/proto/caffe.proto
In file included from python/caffe/_caffe.cpp:17:
In file included from ./include/caffe/caffe.hpp:12:
./include/caffe/net.hpp:41:5: warning: unused typedef 'INVALID_REQUESTED_LOG_SEVERITY' [-Wunused-local-typedef]
LOG_EVERY_N(WARNING, 1000) << "DEPRECATED: ForwardPrefilled() "
^
/usr/local/include/glog/logging.h:943:30: note: expanded from macro 'LOG_EVERY_N'
INVALID_REQUESTED_LOG_SEVERITY); \
^
1 warning generated.
ld: library not found for -lboost_python
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [python/caffe/_caffe.so] Error 1
I have a Makefile.config that calls out boost-python from the brew install set as:
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/Cellar/boost-python/1.67.0/lib
My two initial questions are:
1)Am I not allowed to link to the boost library using the system python? 2)What is the correct way to tell make to grab boost-python?
I was actually having the same problem and couldn't find any answers for days. However, I found this question: Build caffe with Python ( cannot find -lboost_python3 ) .
That question is for python3, but I'm using python2.7 . Basically, I went into the Makefile (not Makefile.config), searched for boost_python, and changed it to boost_python27 . I did the following steps:
vim Makefile
use vim command :?boost_python (there should only be 1 occurrence)
changed PYTHON_LIBRARIES ?= boost_python python2.7
to PYTHON_LIBRARIES ?= boost_python27 python2.7
ran the command sudo make pycaffe
I hope this helps you!
Goto Makefile (and not the .config file) and change the name of the ld library to python27.
I am trying to cross-compile a simple SWIG Python extension on Linux for Windows (mingw32), using the distutils module.
The ultimate goal is to compile a Python wrapper for some library and being able to use it on Windows. Obviously I started with the most basic example and unfortunately it fails.
Here are the files I am using:
example.c
/* File : example.c */
/* A global variable */
double Foo = 3.0;
/* Compute the greatest common divisor of positive integers */
int gcd(int x, int y) {
int g;
g = y;
while (x > 0) {
g = x;
x = y % x;
y = g;
}
return g;
}
example.i - SWIG interface file
/* File : example.i */
%module example
%inline %{
extern int gcd(int x, int y);
extern double Foo;
%}
setup.py
# setup.py
import distutils
from distutils.core import setup, Extension
setup(name = "SWIG example",
version = "1.0",
ext_modules = [Extension("_example", ["example.i","example.c"])])
In order to compile using the native (Linux) gcc compiler, I am invoking:
python setup.py build
Everything works like a charm! Unfortunately when trying to specify the Windows target:
python setup.py build --compiler=mingw32
I get the error saying that gcc can't recognize -mdll switch:
running build
running build_ext
building '_example' extension
swigging example.i to example_wrap.c
swig -python -o example_wrap.c example.i
creating build
creating build/temp.linux-x86_64-2.7
gcc -mdll -O -Wall -I/home/jojek/anaconda/include/python2.7 -c example_wrap.c -o build/temp.linux-x86_64-2.7/example_wrap.o
gcc: error: unrecognized command line option ‘-mdll’
error: command 'gcc' failed with exit status 1
Fair enough, it makes perfect sense, since toolchain is not valid. I made sure that mingw32 is installed on my machine. By calling dpkg -L mingw32 I know that compiler is located in /usr/bin/i586-mingw32msvc-gcc.
My next step was to override the CC environmental variable with the actual path to my compiler. When I try to compile it again, then I am getting the following error with missing sys/select.h header file:
running build
running build_ext
building '_example' extension
swigging example.i to example_wrap.c
swig -python -o example_wrap.c example.i
creating build
creating build/temp.linux-x86_64-2.7
/usr/bin/i586-mingw32msvc-gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jojek/anaconda/include/python2.7 -c example_wrap.c -o build/temp.linux-x86_64-2.7/example_wrap.o
example_wrap.c:1: warning: -fPIC ignored for target (all code is position independent)
In file included from /home/jojek/anaconda/include/python2.7/Python.h:58,
from example_wrap.c:125:
/home/jojek/anaconda/include/python2.7/pyport.h:351:24: error: sys/select.h: No such file or directory
error: command '/usr/bin/i586-mingw32msvc-gcc' failed with exit status 1
Does anyone have an idea how to manage that task?
There's quite a bit going on behind the scenes when you compile Python modules using distutils. You're getting closer with each try in your question, however the problem you've now encountered is that you're using Linux header files with a Windows (cross) compiler. (sys/select.h isn't supported with mingw32, cygwin might be a different story though). In reality it's the lack of the configuration header file that's causing your cross compile to try and use the POSIX interfaces instead of Win32 alternatives.
My answer rewinds a few steps and starts out simply building the module by hand, using mingw32 on Linux and then we'll look at using distutils once we've proven that we have all that's required.
I'm also assuming that you don't have a Windows build box (or even a VM) available to simply build your extension on Windows natively since that's far simpler than cross compiling. If you're reading this and have the option to use a Windows box to build your Windows Python extensions, do that instead and save time and effort. That said it is possible to build Windows Python modules using only a Linux box.
Starting with mingw32 already installed and working on your Linux box (e.g. using the Debian/Ubuntu packages) the first step is to get the Windows header files (or configuration to be more specific). I'm assuming you're targeting the build that most people get when they type "python windows" into a search engine so I downloaded the Windows MSI installers from python.org and extracted them from there.
There are two things we want to get from the Python distribution:
python27.dll (Usually gets placed in c:\windows\system32 or c:\windows\syswow64)
The 'include' directory (Usually gets placed in c:\python27\include)
Under Linux there are a few different ways you can extract this. You could use Wine to install the MSI file. I used both cabextract and 7z with success in my testing though, for example with cabextract:
cabextract /tmp/python-2.7.10.msi -F '*.h'
cabextract /tmp/python-2.7.10.msi -F 'python27.dll'
(Note: if you use 7z you'll find the files you really want inside a second, inner archive named 'python').
At this point you could also extract the file 'libpython27.a' which usually lives inside c:\python27\libs\ however this file isn't sufficient or even useful for linking using mingw32.
Given the header files we've now got enough to compile our extension, although as noted above to get mingw32 to link against python27.dll we need to do a bit more work first. We're going to need a tool called pexports to list all the exported symbols in the Python DLL and let dlltool generate a stub library for mingw32 to link against. I downloaded pexports directly and then extracted it with:
tar xvf ~/Downloads/pexports-0.47-mingw32-bin.tar.xz
Once that's extracted we get a single Windows executable. I used Wine in my example here to run it directly; alternatively, you could extract the source, and build it as a tool to run natively on the Linux host:
tar xvf ~/Downloads/pexports-0.47-mingw32-src.tar.xz
(cd pexports-0.47 && ./configure && make)
or you could have duplicated the functionality of the tool using the Python module pefile (which runs fine cross platform) to extract the exports that we care about if you were looking to avoid using Wine as well.
Anyway with pexports you can generate a .def file that contains the information we need for dlltool:
wine bin/pexports.exe -v python27.dll > python27.def
or, (if you've built pexports as a native tool), simply:
./pexports-0.47/pexports -v python27.dll > python27.def
where python27.dll is what we extracted from the .msi file earlier.
(This was my pexports reference)
Once you've got the .def file you can use the mingw32 dlltool to generate a .a file that we'll use later to link our Python module against:
i586-mingw32msvc-dlltool -A --dllname python27.dll --def python27.def --output-lib libpython27.a
Now we've reached a point where we can think about running SWIG itself to generate the code for us to compile. I simplified your example interface even further to be just:
%module test
%inline %{
int gcd(int x, int y) {
int g;
g = y;
while (x > 0) {
g = x;
x = y % x;
y = g;
}
return g;
}
%}
And then ran SWIG on my Linux box as:
swig -Wall -python test.i
This generated test_wrap.c which I compiled with:
i586-mingw32msvc-gcc test_wrap.c -I../include -Wall -Wextra -shared -o _test.pyd ./libpython27.a
And there we have a Windows Python module built using just Linux.
To check it really runs I copied test.py and _test.pyd to a Windows box and then did:
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.gcd(1024, 512)
512
>>>
Now all that remains is to make sure distutils can find the right include files and libraries to link against by manipulating its paths.
For 64 bits, I couldnt make it work with pexports, so I used gendef to generate the python27.def file.
gendef is a tool which generate def files from DLLs. A def file is a list of symbols exported by a DLL. The primary use of this tool is to allow creation of a import library of DLLs created by non-GCC compilers. It can handle both x86 (win32) and amd64 (win64) executables.
https://sourceforge.net/p/mingw-w64/wiki2/gendef/
Hope It helps!
I compiled caffe on a mac running OSX 10.9.5 and I know trying to compile pycaffe. When I run make pycaffe in the caffe root folder, I get:
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
python/caffe/_caffe.cpp:1:10: fatal error: 'Python.h' file not found
#include <Python.h> // NOLINT(build/include_alpha)
^
1 error generated.
make: *** [python/caffe/_caffe.so] Error 1
how can I fix this?
Perhaps is something wrong with Makefile.config. How do I know what is my PYTHONPATH?
Looking at the comments, I see that you use Anaconda. In Makefile.config, you should uncomment the lines dedicated to Anaconda:
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
# $(ANACONDA_HOME)/include/python2.7 \
# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
Python.h is in $(ANACONDA_HOME)/include/python2.7 as you can see running sudo find / -name 'Python.h'.
I met this problem too.
I have set the PYTHON_INCLUDE PATH
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python2.7
But it still can't find the Python.h
So I just give the include path manually to the compiler as follows:
export CPLUS_INCLUDE_PATH=/home/woolawren/anaconda2/include/python2.7/:$CPLUS_INCLUDE_PATH
if you don't use anaconda2, you can use:
export CPLUS_INCLUDE_PATH=/usr/include/python2.7:$CPLUS_INCLUDE_PATH
I have successfully done "make pycaffe" by doing this.
I just finished a tedious Caffe install on Arch Linux; hopefully my install notes (link below) will help others.
While specific to my Caffe install, those notes address the "Python.h" install error (this Question), as well as a downstream issue mentioned in another SO question,
Import caffe error.
https://stackoverflow.com/questions/28177298/import-caffe-error
My gist file (notes):
Caffe Installation Notes
https://gist.github.com/victoriastuart/fb2cb22209ccb2771963a25c06221213
I uncommented the below code in Makefile.config
PYTHON_INCLUDE := /usr/include/python3.5m \
/usr/lib/python3.5/dist-packages/numpy/core/include
Then did sudo make pycaffe.
It worked.
I am trying to setup and compile the Hello World example for Boost.Python: http://www.boost.org/doc/libs/1_57_0/libs/python/doc/tutorial/doc/html/python/hello.html
I installed bjam, boost, boost-build, and boost-python from Homebrew:
brew install bjam
brew install boost
brew install boost-build
brew install boost-python
My python install is also via Homebrew. I am not sure how to properly modify the example Jamroot file so that it is compatible with my system setup. I changed the boost path to : /usr/local/Cellar/boost; but I'm not sure of the other paths that need to be changed. The current setup gives me the following error:
> bjam
notice: no Python configured in user-config.jam
notice: will use default configuration
Jamroot:26: in modules.load
*** argument error
* rule use-project ( id : where )
* called with: ( boost : /usr/local/Cellar/boost; project : requirements <library>/boost/python//boost_python <implicit-dependency>/boost//headers : usage-requirements <implicit-dependency>/boost//headers )
* extra argument project
/usr/local/share/boost-build/build/project.jam:1138:see definition of rule 'use-project' being called
/usr/local/share/boost-build/build/project.jam:311: in load-jamfile
/usr/local/share/boost-build/build/project.jam:64: in load
/usr/local/share/boost-build/build/project.jam:145: in project.find
/usr/local/share/boost-build/build-system.jam:535: in load
/usr/local/share/boost-build/kernel/modules.jam:289: in import
/usr/local/share/boost-build/kernel/bootstrap.jam:139: in boost-build
/usr/local/share/boost-build/boost-build.jam:8: in module scope
Summary
Don't use BJAM it's a waste of your time - I am assuming your interest in BJAM is a side-product of getting your code to actually work
Here is the quick-link to my github page where I do a hello_world example using namespace boost::python
See my github for linking multiple boost files into one import library
Longer Answer
I have exactly the same setup as you. I spent ages getting this to work as the documentation is really shady (as you know) and before you know it you go down some weird rabbit hole trying to hack make files and BJAM installations.
You can use a setup.py as you would normally with C code as follows...
Installation
You can obtain the correct boost-python through homebrew via the command:
brew install boost --with-python --build-from-source
I think brew install boost should work but it's a big install and life is short to do it twice
Boost code
Assume the following code in hello_ext.cpp
#include <boost/python.hpp>
char const* greet()
{
return "Greetings!";
}
BOOST_PYTHON_MODULE(hello_ext)
{
using namespace boost::python;
def("greet", greet);
}
Python setup
Then you can write setup.py file as
from distutils.core import setup
from distutils.extension import Extension
hello_ext = Extension(
'hello_ext',
sources=['hello_ext.cpp'],
libraries=['boost_python-mt'],
)
setup(
name='hello-world',
version='0.1',
ext_modules=[hello_ext])
Compiling
The following example can be used by:
python setup.py build_ext --inplace
which will create the following build/ directory and file:
build/
hello_ext.so
Running
This can be directly now called by python with:
In [1]: import hello_ext
In [2]: hello_ext.greet()
Out[2]: 'Greetings!'