I'm building a Rust binary (liblonlat_bng.dylib) on Travis CI, pulling it into a Cython extension (in the same dir as the Cython source .c/.pyx), and testing it, also on Travis CI (in a different repo and build). However, tests of the Python package are failing, and I'm not sure why:
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/travis/build/urschrei/convertbng/test/test_convertbng.py", line 15, in <module>
from convertbng.cutil import convert_bng as cconvert_bng
ImportError: dlopen(/Users/travis/build/urschrei/convertbng/convertbng/cutil.so, 2): Library not loaded: /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib
Referenced from: /Users/travis/build/urschrei/convertbng/convertbng/cutil.so
Reason: image not found
Here's the OSX build output for the package from Travis:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /Users/travis/build/urschrei/convertbng/venv/bin/python2.7 -c "import setuptools, tokenize;__file__='/Users/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/convertbng
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -Iconvertbng -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c convertbng/cutil.c -o build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -O3
[unused function warnings]
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/convertbng
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so
copying build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /Users/travis/build/urschrei/convertbng
Successfully installed convertbng
And here's the Linux Travis output, which successfully locates the dylib on a relative path. Note the $ORIGIN argument to -R, which can't be used on OSX:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /usr/bin/python -c "import setuptools, tokenize;__file__='/home/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iconvertbng -I/usr/include/python2.7 -c convertbng/cutil.c -o build/temp.linux-x86_64-2.7/convertbng/cutil.o -O3
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -Wl,-R$ORIGIN -llonlat_bng -o build/lib.linux-x86_64-2.7/convertbng/cutil.so
copying build/lib.linux-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /usr/local/lib/python2.7/dist-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /home/travis/build/urschrei/convertbng
Successfully installed convertbng
Other details:
Both my OS X and Travis are using XCode 7.3
The binaries are being built with the same commit both locally and on Travis
If I build the binary locally, then run setup.py build_ext --inplace on my package, the tests pass
I run the same install for Linux, using a .so built with the same commit on Travis, and the tests pass
If I copy the Travis-built .dylib to my local machine, then run setup.py build_ext --inplace, the tests fail with the same error as on Travis.
I'm very puzzled as to what's going on. Is there something I should be looking at in the binaries? The otool -l output differs slightly. For instance:
Local:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 1226.10.1
Travis:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 00:00:02 1970
current version 1225.1.1
Link to passing Linux job: https://travis-ci.org/urschrei/convertbng/jobs/136730347
Link to failing OSX job: https://travis-ci.org/urschrei/convertbng/jobs/136730348
UPDATE:
It's definitely because of the library name. if I use install_name_tool to change the dylib location to #loader_path/liblonlat_bng.dylib:
install_name_tool -change /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib #loader_path/liblonlat_bng.dylib convertbng/cutil.so
The Travis-built executable will run on my local machine. However, Travis seems to have a broken install_name_tool install, and I'm not managing to pass the correct invocation to cutil.so from setup.py. I've tried setting extra_link_args to ['-Wl,-rpath,'+'#loader_path/liblonlat_bng.dylib'], which generates
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.6-intel-2.7/convertbng/cutil.so -Wl,-rpath,#loader_path/liblonlat_bng.dylib
But my tests still fail if I do that.
UPDATE 2: I can fix the install_name of the dylib in the link phase when building with cargo:
RUSTFLAGS="-C link-args=-Wl,-install_name,#rpath/liblonlat_bng.dylib" cargo build --release
And this works. But I suspect my setup.py Extension setup is wrong:
# only append the runtime dir on Linux
rdirs = []
ldirs = []
if sys.platform != 'darwin':
# from http://stackoverflow.com/a/10252190/416626
# the $ORIGIN trick is not perfect, though
rdirs = ['$ORIGIN']
if sys.platform == 'darwin':
ldirs = ['-Wl,-rpath,'+'#loader_path/liblonlat_bng.dylib']
extensions = Extension("convertbng.cutil",
sources=["convertbng/cutil" + suffix],
libraries=["lonlat_bng"],
include_dirs=['.', 'convertbng'],
library_dirs=['.', 'convertbng'],
runtime_library_dirs=rdirs,
extra_compile_args=["-O3"],
extra_link_args=ldirs
)
I had the same problem and I believe it is a bug in Distutils. I've made a pull request to fix this and I've also proposed a workaround. See https://github.com/python/cpython/pull/12418
Can you try the following in your setup.py?
from Cython.Distutils.build_ext import new_build_ext as build_ext
# alternative:
# from distutils.command import build_ext
class my_build_ext(build_ext):
"""Workaround for rpath bug in distutils for OSX."""
def finalize_options(self):
super().finalize_options()
# Special treatment of rpath in case of OSX, to work around python
# distutils bug 36353. This constructs proper rpath arguments for clang.
# See https://bugs.python.org/issue36353
if sys.platform[:6] == "darwin":
for path in self.rpath:
for ext in self.extensions:
ext.extra_link_args.append("-Wl,-rpath," + path)
self.rpath[:] = []
setup(
cmdclass={'build_ext': my_build_ext}
# ...
)
There are two ways to fix this without using install_name_tool.
Assuming that your dylib is in the same directory as your Cython extension (In this case, cutil.so)
Compile your Rust binary with rpath support, and correctly set the rpath in your setup.py Extension instance:
Invoke RUSTFLAGS="-C rpath" cargo build --release.
The less flexible way is to compile your Rust binary and pass the path in via linker flags. This bakes in the path (albeit with dynamic refs), however:
cargo rustc --release -- -C link-args=-Wl,-install_name,#rpath/libname.dylib
After you've enabled rpath support in your Rust dylib, edit your setup.py:
Set (or add) extra_link_args=["-Wl,-rpath", "-Wl,#loader_path/"] in your Extension instance.
You'll know it's worked if you compile your Cython extension (e.g. using setup.py build_ext --inplace), then run otool -l on the resulting .so:
#rpath/liblonlat_bng.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
If you want your Rust dylib elsewhere relative to your Cython extension, you'll need to adjust your #rpath/ or #loader_path/ paths in setup.py accordingly.
Note: The foregoing works for relative paths on OS X only. On Linux, you can set extra_link_args=["-Wl,-rpath", "-Wl,$ORIGIN"] in your setup.py Extension instance (and adjust it accordingly if your Rust dylib is elsewhere, relative to your extension.)
I'm running a setup Python script for Pyosmium on OSX 10.11, Python version is 2.7:
python setup.py install
running install
running build
running build_py
running build_ext
building 'osmium._osmium' extension
gcc -fno-strict-aliasing -I/Users/stephen/anaconda/include -arch x86_64
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include
-I../libosmium/include -I/Users/stephen/anaconda/include/python2.7 -c
lib/osmium.cc -o build/temp.macosx-10.5-x86_64-2.7/lib/osmium.o -std=c++0x
-stdlib=libstdc++ -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
In file included from lib/osmium.cc:1:
../libosmium/include/osmium/visitor.hpp:36:10: fatal error: 'type_traits' file not found
As far as possible I've most available dependencies using Brew. And added required directories for supporting developments.
Though it's clear I am dealing with a level here that I am not at all familiar with.
My guess at the moment is the compiler that's been called - gcc, should this be Any ideas would be much appreciated!
Thank you,
Stephen.
I have problems building pywt from source on Ubuntu.
When I run python setup.py build, I get the following error:
running build
running build_py
running build_ext
Cython is not installed. Using compiled file: src/_pywt.pyx
building 'pywt._pywt' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DPY_EXTENSION -Isrc -I/usr/local/include/python2.7 -c src/_pywt.c -o build/temp.linux-x86_64-2.7/src/_pywt.o
gcc: error: src/_pywt.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
When I run cython --version I get the following output:
Cython version 0.21.1
pywt can be installed as an Ubuntu package using apt-get:
sudo apt-get install python-pywt
I'm a newbie to Django and I want to run it not with SQLite but with my current MySQL database set up through PHP. However, I am coding mainly on a Mac (the bane of my developer existence) and I have run into a problem with running:
python setup.py build
from my MySQL-python-1.2.4b4 folder and it's giving me the following error. I have tried Googling for the solution but most results show an error with XCode4 for Mac OS Lion or Mountain Lion and I'm running Snow Leopard. The results I found for Snow Leopard doesn't work either.
running develop
running egg_info
writing MySQL_python.egg-info/PKG-INFO
writing top-level names to MySQL_python.egg-info/top_level.txt
writing dependency_links to MySQL_python.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'MySQL_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'MySQL_python.egg-info/SOURCES.txt'
running build_ext
building '_mysql' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -Dversion_info=(1,2,4,'beta',4) -D__version__=1.2.4b4 -I/usr/local/mysql/include -I/System/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c _mysql.c -o build/temp.macosx-10.6-universal-2.6/_mysql.o -g -Os -arch x86_64 -fno-common -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL
unable to execute gcc-4.2: No such file or directory
error: command 'gcc-4.2' failed with exit status 1
Since you mention that xcodebuild is not found, the most likely explanation is that you have not yet installed a version of Xcode which includes various build tools, like the gcc C compiler needed to build the C extension module included in MySQL-python. Since you are running on OS X 10.6, Snow Leopard, the most recent version of Xcode for 10.6 that includes gcc-4.2 is Xcode 3.2.6 which you can download without cost from the Apple Developer site after free registration. See, for example, the directions here. Unfortunately, the download is very large. (More recent releases of Xcode for more recent OS X releases are packaged in smaller pieces but they are not supported on 10.6.) The good news is: you'll only have to do it once since Xcode for 10.6 is not likely to ever be updated again!
I'm currently trying to build and install the mySQLdb module for Python, but the command
python setup.py build
gives me the following error
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-intel-2.7/MySQLdb
error: could not delete 'build/lib.macosx-10.3-intel-2.7/MySQLdb/release.py': Permission denied
I verified that I'm a root user and when trying to execute the script using sudo, I then get a gcc-4.0 error:
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-fat-2.7/MySQLdb
running build_ext
building '_mysql' extension
gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -O3 -Dversion_info=(1,2,3,'final',0) -D__version__=1.2.3 -I/usr/local/mysql/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _mysql.c -o build/temp.macosx-10.3-fat-2.7/_mysql.o -Os -g -fno-common -fno-strict-aliasing -arch x86_64
unable to execute gcc-4.0: No such file or directory
error: command 'gcc-4.0' failed with exit status 1
Which is odd, because I'm using XCode 4 with Python 2.7. I've tried the easy_install and pip methods, both of which dont work and give me a permission denied error on release.py. I've chmodded that file to see if that was the problem but no luck. Thoughts?
Make sure that gcc-4.0 is in your PATH. Also, you can create an alias from gcc to gcc-4.0.
Take care about 32b and 64b versions. Mac OS X is a 64b operating system and you should right flags to make sure you're compiling for 64b architecture.
If you're on Windows, you can use a pre-compiled installer. Check this page for many such installers, including MySQLdb for various Python releases