I'm building a Rust binary (liblonlat_bng.dylib) on Travis CI, pulling it into a Cython extension (in the same dir as the Cython source .c/.pyx), and testing it, also on Travis CI (in a different repo and build). However, tests of the Python package are failing, and I'm not sure why:
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/travis/build/urschrei/convertbng/test/test_convertbng.py", line 15, in <module>
from convertbng.cutil import convert_bng as cconvert_bng
ImportError: dlopen(/Users/travis/build/urschrei/convertbng/convertbng/cutil.so, 2): Library not loaded: /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib
Referenced from: /Users/travis/build/urschrei/convertbng/convertbng/cutil.so
Reason: image not found
Here's the OSX build output for the package from Travis:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /Users/travis/build/urschrei/convertbng/venv/bin/python2.7 -c "import setuptools, tokenize;__file__='/Users/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/convertbng
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -Iconvertbng -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c convertbng/cutil.c -o build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -O3
[unused function warnings]
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/convertbng
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so
copying build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /Users/travis/build/urschrei/convertbng
Successfully installed convertbng
And here's the Linux Travis output, which successfully locates the dylib on a relative path. Note the $ORIGIN argument to -R, which can't be used on OSX:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /usr/bin/python -c "import setuptools, tokenize;__file__='/home/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iconvertbng -I/usr/include/python2.7 -c convertbng/cutil.c -o build/temp.linux-x86_64-2.7/convertbng/cutil.o -O3
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -Wl,-R$ORIGIN -llonlat_bng -o build/lib.linux-x86_64-2.7/convertbng/cutil.so
copying build/lib.linux-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /usr/local/lib/python2.7/dist-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /home/travis/build/urschrei/convertbng
Successfully installed convertbng
Other details:
Both my OS X and Travis are using XCode 7.3
The binaries are being built with the same commit both locally and on Travis
If I build the binary locally, then run setup.py build_ext --inplace on my package, the tests pass
I run the same install for Linux, using a .so built with the same commit on Travis, and the tests pass
If I copy the Travis-built .dylib to my local machine, then run setup.py build_ext --inplace, the tests fail with the same error as on Travis.
I'm very puzzled as to what's going on. Is there something I should be looking at in the binaries? The otool -l output differs slightly. For instance:
Local:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 1226.10.1
Travis:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 00:00:02 1970
current version 1225.1.1
Link to passing Linux job: https://travis-ci.org/urschrei/convertbng/jobs/136730347
Link to failing OSX job: https://travis-ci.org/urschrei/convertbng/jobs/136730348
UPDATE:
It's definitely because of the library name. if I use install_name_tool to change the dylib location to #loader_path/liblonlat_bng.dylib:
install_name_tool -change /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib #loader_path/liblonlat_bng.dylib convertbng/cutil.so
The Travis-built executable will run on my local machine. However, Travis seems to have a broken install_name_tool install, and I'm not managing to pass the correct invocation to cutil.so from setup.py. I've tried setting extra_link_args to ['-Wl,-rpath,'+'#loader_path/liblonlat_bng.dylib'], which generates
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.6-intel-2.7/convertbng/cutil.so -Wl,-rpath,#loader_path/liblonlat_bng.dylib
But my tests still fail if I do that.
UPDATE 2: I can fix the install_name of the dylib in the link phase when building with cargo:
RUSTFLAGS="-C link-args=-Wl,-install_name,#rpath/liblonlat_bng.dylib" cargo build --release
And this works. But I suspect my setup.py Extension setup is wrong:
# only append the runtime dir on Linux
rdirs = []
ldirs = []
if sys.platform != 'darwin':
# from http://stackoverflow.com/a/10252190/416626
# the $ORIGIN trick is not perfect, though
rdirs = ['$ORIGIN']
if sys.platform == 'darwin':
ldirs = ['-Wl,-rpath,'+'#loader_path/liblonlat_bng.dylib']
extensions = Extension("convertbng.cutil",
sources=["convertbng/cutil" + suffix],
libraries=["lonlat_bng"],
include_dirs=['.', 'convertbng'],
library_dirs=['.', 'convertbng'],
runtime_library_dirs=rdirs,
extra_compile_args=["-O3"],
extra_link_args=ldirs
)
I had the same problem and I believe it is a bug in Distutils. I've made a pull request to fix this and I've also proposed a workaround. See https://github.com/python/cpython/pull/12418
Can you try the following in your setup.py?
from Cython.Distutils.build_ext import new_build_ext as build_ext
# alternative:
# from distutils.command import build_ext
class my_build_ext(build_ext):
"""Workaround for rpath bug in distutils for OSX."""
def finalize_options(self):
super().finalize_options()
# Special treatment of rpath in case of OSX, to work around python
# distutils bug 36353. This constructs proper rpath arguments for clang.
# See https://bugs.python.org/issue36353
if sys.platform[:6] == "darwin":
for path in self.rpath:
for ext in self.extensions:
ext.extra_link_args.append("-Wl,-rpath," + path)
self.rpath[:] = []
setup(
cmdclass={'build_ext': my_build_ext}
# ...
)
There are two ways to fix this without using install_name_tool.
Assuming that your dylib is in the same directory as your Cython extension (In this case, cutil.so)
Compile your Rust binary with rpath support, and correctly set the rpath in your setup.py Extension instance:
Invoke RUSTFLAGS="-C rpath" cargo build --release.
The less flexible way is to compile your Rust binary and pass the path in via linker flags. This bakes in the path (albeit with dynamic refs), however:
cargo rustc --release -- -C link-args=-Wl,-install_name,#rpath/libname.dylib
After you've enabled rpath support in your Rust dylib, edit your setup.py:
Set (or add) extra_link_args=["-Wl,-rpath", "-Wl,#loader_path/"] in your Extension instance.
You'll know it's worked if you compile your Cython extension (e.g. using setup.py build_ext --inplace), then run otool -l on the resulting .so:
#rpath/liblonlat_bng.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
If you want your Rust dylib elsewhere relative to your Cython extension, you'll need to adjust your #rpath/ or #loader_path/ paths in setup.py accordingly.
Note: The foregoing works for relative paths on OS X only. On Linux, you can set extra_link_args=["-Wl,-rpath", "-Wl,$ORIGIN"] in your setup.py Extension instance (and adjust it accordingly if your Rust dylib is elsewhere, relative to your extension.)
I have problems building pywt from source on Ubuntu.
When I run python setup.py build, I get the following error:
running build
running build_py
running build_ext
Cython is not installed. Using compiled file: src/_pywt.pyx
building 'pywt._pywt' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPY_EXTENSION -Isrc -I/usr/local/include/python2.7 -c src/_pywt.c -o build/temp.linux-x86_64-2.7/src/_pywt.o
gcc: error: src/_pywt.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
When I run cython --version I get the following output:
Cython version 0.21.1
pywt can be installed as an Ubuntu package using apt-get:
sudo apt-get install python-pywt
I untarred it from: http://pypi.python.org/pypi/greenlet#downloads
(I'm in my virtual env. I sourced it, too, thats why I didn't sudo)
$ py setup.py install
/usr/lib/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'repository'
warnings.warn(msg)
running install
running bdist_egg
running egg_info
writing greenlet.egg-info/PKG-INFO
writing top-level names to greenlet.egg-info/top_level.txt
writing dependency_links to greenlet.egg-info/dependency_links.txt
reading manifest file 'greenlet.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'greenlet.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
running build_ext
building 'greenlet' extension
creating build
creating build/temp.linux-i686-2.6
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c greenlet.c -o build/temp.linux-i686-2.6/greenlet.o
In file included from greenlet.c:5:
greenlet.h:11: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
On Ubuntu you'd need to install the python-dev package. This provides the Python.h header:
% dpkg -S /usr/include/python2.6/Python.h
python2.6-dev: /usr/include/python2.6/Python.h
you don't have python-dev installed so there is no Python.h header.
I'm new to svn and python, but I really want to use a more recent version of xattr that includes more useful features than the original that came with the OS.
On OSX 10.5.6, I'm having trouble installing xattr-0.6.1.
From the local machine in Terminal:
$ svn co http://svn.red-bean.com/bob/xattr/releases/xattr-0.6.1/
The files are downloaded and placed in ~/xattr-0.6.1/
$ cd xattr-0.6.1/
$ sudo python setup.py install
---------------------------------------------------------------------------
This script requires setuptools version 0.6c11 to run (even to display
help). I will attempt to download it for you (from
http://pypi.python.org/packages/2.5/s/setuptools/), but
you may need to enable firewall access for this script first.
I will start the download in 15 seconds.
(Note: if this machine does not have network access, please obtain the file
http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg
and place it in this directory before rerunning this script.)
---------------------------------------------------------------------------
Downloading http://pypi.python.org/packages/2.5/s/setuptools/setuptools-0.6c11-py2.5.egg
running install
running bdist_egg
running egg_info
creating xattr.egg-info
writing xattr.egg-info/PKG-INFO
writing top-level names to xattr.egg-info/top_level.txt
writing dependency_links to xattr.egg-info/dependency_links.txt
writing entry points to xattr.egg-info/entry_points.txt
writing manifest file 'xattr.egg-info/SOURCES.txt'
writing manifest file 'xattr.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.5-i386/egg
running install_lib
running build_py
creating build
creating build/lib.macosx-10.5-i386-2.5
creating build/lib.macosx-10.5-i386-2.5/xattr
copying xattr/__init__.py -> build/lib.macosx-10.5-i386-2.5/xattr
copying xattr/constants.py -> build/lib.macosx-10.5-i386-2.5/xattr
copying xattr/tool.py -> build/lib.macosx-10.5-i386-2.5/xattr
running build_ext
building 'xattr._xattr' extension
creating build/temp.macosx-10.5-i386-2.5
creating build/temp.macosx-10.5-i386-2.5/xattr
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DMACOSX -I/usr/include/ffi -DENABLE_DTRACE -arch i386 -arch ppc -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c xattr/_xattr.c -o build/temp.macosx-10.5-i386-2.5/xattr/_xattr.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
To check the version of python:
$ python -V
Python 2.5.1
It appears to have installed the python update.
I try to run the xattr install again:
$ sudo python setup.py install
running install
running bdist_egg
running egg_info
writing xattr.egg-info/PKG-INFO
writing top-level names to xattr.egg-info/top_level.txt
writing dependency_links to xattr.egg-info/dependency_links.txt
writing entry points to xattr.egg-info/entry_points.txt
writing manifest file 'xattr.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.5-i386/egg
running install_lib
running build_py
running build_ext
building 'xattr._xattr' extension
gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -Os -Wall -Wstrict-prototypes -DMACOSX -I/usr/include/ffi -DENABLE_DTRACE -arch i386 -arch ppc -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c xattr/_xattr.c -o build/temp.macosx-10.5-i386-2.5/xattr/_xattr.o
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1
Am I missing an important fundamental element or is there something wrong with the setup.py configuration, or perhaps something else I may have overlooked?
It needs to compile a C file, so it needs gcc. Mac folks need to install XCode. Update As #jathanism mentions, you should have an XCode install DVD that came with your Mac (most likely version 3.2).
For example:
atlas% which gcc
/usr/bin/gcc
atlas% gcc -v
Target: i686-apple-darwin10
..
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)
I'm trying to install MySQLdb for python.
but when I run the setup, this is the error I get.
well I know why its giving all the missing file statements, but dont know where to change the bold marked location from.
Please help
gaurav-toshniwals-macbook-7:MySQL-python-1.2.3c1 gauravtoshniwal$ python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-fat-2.6/MySQLdb
running build_ext
building '_mysql' extension
gcc-4.0 -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -Dversion_info=(1,2,3,'gamma',1) -D__version__=1.2.3c1 **-I/Applications/MAMP/Library/include/mysql** -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c _mysql.c -o build/temp.macosx-10.3-fat-2.6/_mysql.o
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory
_mysql.c:38:19:_mysql.c:39:26: error: mysqld_error.h: No such file or directory
error: _mysql.c:40:20:mysql.h: No such file or directory
This path probably comes from mysql_config utility, edit your setup_posix.py and change the variable mysql_config.path to meet your correct mysql_config utility path.
The above suggestion is really helpful but it might also be worth mentioning that you must have mysql-devel installed as well for your build to be successful here.