I am trying to compile a package on Mac OSX 10.6.5. The package's install script relies on distutils. The problem is that the computer's default gcc is version 4.2 (I determined this by just running gcc --version in a terminal window) but when I run 'python setup.py build', I see from the output that the distutils is choosing gcc-4.0 instead of 4.2 This is a big problem because the code I am using require gcc >= 4.2. I do not have admin rights on this machine, so as a workaroud, I created some symlinks that send gcc-4.0 to gcc-4.2. The result is the code compiles, but the generated .so files do not work (when I try to import them in python, I get errors complaining about a missing init function in the shared object).
I have tried compiling this code on a different mac (10.6.6) and it works like a charm: distutils chooses 4.2 without being forced to do so and I can import the generated shared object without a problem. So, what I would like to do is to compile the code on my computer without having to do this symlink trickery...I just want distutils to choose 4.2 automatically as it should. I have tried taking the .so files that compile properly and transferring them to my computer, but that fails for a number of reasons (they are linked against libraries that are not present on my machine/are a different version that those installed).
Does anyone have any advice here?
Thanks,
Josh
To force distutils to use a separate compiler, you can redefine a few variables via the environment. First, find out what distutils is using as defaults:
>>> from distutils import sysconfig
>>> sysconfig.get_config_var('LDSHARED')
'gcc-4.0 -Wl,-F. -bundle -undefined dynamic_lookup'
>>> sysconfig.get_config_var('CC')
'gcc-4.0'
Next you need to redefine those, substituting in the version of gcc you'd like to use:
% LDSHARED="gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup" CC=gcc-4.2 \
/usr/bin/python setup.py build_ext
Keep in mind that the sysconfig defaults are pulled from the Makefile which was originally used to compile python, so fudging with them may produce unintended results:
>>> path = sysconfig.get_python_lib(plat_specific=1, standard_lib=1)
>>> os.path.join(path, 'config', 'Makefile')
'/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config/Makefile'
Have you tried setting a custom CC environment variable?
CC=gcc-4.2 python setup.py build
You don't say which versions of Python you are using but chances are that the default Python on the machine you are using is not the Apple-supplied Python 2.6.1, more likely one installed from python.org. Try running the build script with /usr/bin/python2.6 which should be the Apple-supplied Python 2.6; that does use gcc-4.2.
When you use Python's Distutils (typically by running a setup.py script or using easy_install), Distutils tries to make sure that any C extension modules in the package you are installing will be compiled with the same compiler version and with compatible compilation options (CPU archs, ABI, etc) as was used to build Python itself. Traditionally, most python.org installers for OS X provide a universal Python that works on multiple versions of OS X and multiple CPU archs. That's why they are built with gcc-4.0. If you need to use gcc-4.2 for some other library, then the safest thing is to use a Python that was built with gcc-4.2. The Apple-supplied system Pythons on OS X 10.6 are so built. Also, for the most recent releases of Python (2.7.1 and 3.2), python.org provides a second OS X installer variant for OS X 10.6 that is also built with gcc-4.2. But if you do not have admin access to your machine, that's not an option anyway.
You can see which python is being used by default by:
which python
You can either adjust your distutils.cfg (see here) or you can pass command line arguments as --compiler=gcc42 (not sure about the last one).
Related
I'm running clang on mac to compile a c file created by running a very simple program through cython, but the compiler always give me a "Python.h not found" fatal error. I've tried every solution I could find, reinstalling python 3.9, using the -I/path/to/headerfile method, and rewriting the include statement in the code to contain the full filepath, but nothing has worked. When I do include the full filepath, I get fatal error: 'cpython/initconfig.h' file not found. What could the issue possibly be, and how would I fix it? The program itself works fine in the standard python interpreter, pyinstaller, and nuitka.
Today I needed to embed Python into a C application developed on XCode with Clang compiler, and I've met the same problems and solved them. Let me share the tips:
1. Python not found error
You should install a Python framework to your MacOS, and attach it into Frameworks and Libraries project settings tab. Also, specify it's path in Framework Search Paths inside Build Settings tab if XCode hasn't done it automatically.
2. Undefined symbols / Implicit declaration errors
Verify you're using a modern version of Python framework. Versions prior to 3.6 had different ABI which leads to the mentioned errors.
3. cpython/initconfig.h file not found
Versions prior to ~3.9 had some header inclusion problem for the cpython/initconfig.h file (issues 40642, 39026).
Check the builds of Python that your OS currently has. I found Pythons installed in the following paths:
# Default 2.7 MacOSX installation is here
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/
/Library/Frameworks/
Or install a recent Python version. I recommend using "universal binary" build, see the docs.
I'm on OSX, trying to compile a shared library in C with distutils' setup.py (to use in python using ctypes). I'm new to distutils, but I'm having problems when the shared library I want to compile (libreboundx.so) depends on another shared library (librebound.so). Explicitly, in modify_orbits_direct.c I have
#include "rebound.h"
rebound.h is in directory /Users/dt/rebound/src/, and all the functions in rebound.h are in the shared library librebound.so, which is in /Users/dt/rebound/.
The linking with cc would look like.
cc -fPIC -shared reboundx.o -L/Users/dt/rebound -lrebound -o libreboundx.so
UPDATE: This situation looks exactly like the example at the end of Sec. 3 at https://docs.python.org/2/extending/building.html. I've updated my setup.py to mimic that one:
libreboundxmodule = Extension('libreboundx',
sources = [ 'src/reboundx.c',
'src/modify_orbits_direct.c'],
include_dirs = ['src', '/Users/dt/rebound/src'],
extra_compile_args=['-fstrict-aliasing', '-O3','-std=c99','-march=native', '-D_GNU_SOURCE', '-fPIC'],
library_dirs=['/Users/dt/rebound'],
libraries=['rebound'],
)
This installs fine when I run
pip install -e ./
Build output:
You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Obtaining file:///Users/dtamayo/Documents/workspace/reboundx
Installing collected packages: reboundx
Running setup.py develop for reboundx
Successfully installed reboundx-1.0
but when I try
import reboundx
in Python, I get an OSError: dlopen(libreboundx.so, 10): Symbol not found: _reb_boundary_particle_is_in_box, which is a function in the other library (librebound.so), which doesn't even get called in the code for libreboundx.so.
If I link the shared library with the cc command above, everything works, and I can use the shared library libreboundx.so perfectly fine in C. If I try to take the same libreboundx.so I compile with the cc command and stick it where setup.py would put it, then try to import reboundx in python, I instead get
OSError: dlopen(/Users/dtamayo/Documents/workspace/reboundx/reboundx/../libreboundx.so, 10): Library not loaded: librebound.so
Referenced from: /Users/dtamayo/Documents/workspace/reboundx/libreboundx.so
Reason: image not found
Could this be like an rpath issue, where at runtime libreboundx.so doesn't know where to look for librebound.so?
Thanks for all the suggestions. I should have specified in the question that in the end I want a solution that I could package for upload to PyPy so users can install with a single command. It should also run on both OSX and Linux, so I preferred solutions not involving install_name_tool.
I haven't been able to test it, but I think adding
runtime_library_dirs=['/Users/dt/rebound'],
next to library_dirs should fix the problem on Linux. Apparently this doesn't work on Mac, but you instead can use extra_link_args. Adding this below the libreboundxmodule definition posted above,
if platform.system() == 'Darwin':
extra_link_args.append('-Wl,-rpath,'+'/Users/dtamayo/Documents/workspace/rebound')
fixed my problem. I found the answer here: Python runtime_library_dirs doesn't work on Mac
So far as I can tell what happens is this:
In python.jam, it works out which version of Python I am using and which library directories to look in;
It adds -Wl-R arguments to the g++ command line to include those directories;
The ld command complains that it does not have a -R option.
So either (a) I have a defective version of ld, or (b) I need to tell bjam that it needs to use a different option (-rpath perhaps?) or that this option is not required.
I must be missing something—I am surely not the first person in history to try to build a Python extension with Boost on Mac OS X—but I can’t figure out where to look next. Any hints?
Update:
The command I am using is
bjam
If I do bjam --version, I get
Boost.Build V2 (Milestone 12)
Boost.Jam 03.1.18
The toolset used is whatever the default toolset is on Mac OS X.
The compiler is the default compiler on Mac OS X (with the developer tools installed), which is GCC version ‘i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664)’.
The linker is the default linker on Mac OS X, which is called ld or ld64, but which does not have the -R option that GNU ld has, so I assume it is a special version designed to allow for Mac OS X’s concept of frameworks or whatever. It does not have a --version option.
There is a Jamfile, which goes like this:
import python ;
python-extension _optimor :
bill_python.cpp
bill_record_python.cpp
.. etc ...
:
<cxxflags>-fPIC
... etc ...
<variant>debug:<define>DEBUG
<include>/usr/include/python2.6
<include>../
;
It builds OK on Ububtu GNU/Linux.
I am not interested in Boost or bjam per se; my only requirement to compile this extension so I can get on with developing the system of which this extension is a small but important part.
I can't tell which version of Boost you have.. BUt the most likely reason for the problem is that you are using the generic "gcc" toolset to build. There's a special toolset for building with the GCC variant that Apple uses in Xcode. Try building with bjam toolset=darwin instead.
I am trying to install Python 2.6 on Solaris by building the source on Solaris machine. I installed one this way and it appears that it is 32-bit. I downloaded some source tar ball as Linux or Unix for this purpose. Everything works well but I need 64-bit Python.
I looked up the Python download site and there is no separate installation for a 64-bit Python.
That makes me think that there must be some option while running configure and/or install commands to install Python. I tried reading README.txt of the installation but could not find any info. I am very new to installations on "Unix" like systems.
How can I install 64-bit Python on Solaris?
I would strongly suggest seeing if you can get away with the 32 bit version of Python. If your new to compiling stuff on Solaris, this will save you many headaches. However, it is possible, and I do have a working 64 bit version of Python. I'm using cc: Sun C 5.8 2005/10/13 to compile. Additionally, I've already compiled 64-bit version of readline and ncurses.
My configure line looks like this:
../Python-2.6.1/configure CCSHARED="-KPIC" LDSHARED="cc -xarch=generic64 -G -KPIC" LDFLAGS="-xarch=generic64 -L/opt/tools/lib -R/opt/tools/lib -L/opt/tools/ssl/lib -ltermcap -lz -R $ORIGIN/../lib" CC="cc" CPP="cc -xarch=generic64 -E -I/opt/tools/include -I/opt/tools/include/ncurses -I/opt/tools/include/readline" BASECFLAGS="-xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses" OPT="-xO5" CFLAGS="-xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses -I/opt/tools/include/readline" CXX="CC -xarch=generic64 -I/opt/tools/include -I/opt/tools/include/ncurses" --prefix=/opt/tools/python-2.6.1 --enable-64-bit --without-gcc --disable-ipv6 --with-ssl=openssl --with-ncurses --with-readline
Additionally, I modified these two lines in Modules/Setup.local to include the required locations:
readline readline.c -I/opt/tools/include/readline -L/opt/tools/lib -lreadline -ltermcap
_ssl _ssl.c -I/opt/tools/ssl/include -L/opt/tools/ssl/lib -lssl -lcrypto
Now, just pray you don't need to compile in some Sybase bindings or some other 64-bit libraries.
It's currently an acknowledged bug that Solaris 64-bit support is suboptimal, but that bug report looks to contain some flags that you might want to use. See also this mailing list posting.
I am the new maintainer for an in-house Python system that uses a set of 3rd-party shared C libraries via a shared library shim that is created using a combination of swig and a setup.py script. This has been working well until recently.
The 3rd-party shared C libraries were updated for new functionality and now I get the following run-time error, after a clean build, when I try to run our main Python program (which imports the generated shared library shim):
-sh-3.00$ python ams.py
ImportError: /usr/lib/libz4lnx.so: undefined symbol: stat
I found a discussion thread from 1999 that explains that the problem is that stat is not present in libc.so.6, but rather in libc_nonshared.a, and provides a solution: Link against the c library, by adding -lc to your build command line.
http://www.redhat.com/archives/pam-list/1999-February/msg00082.html
I've added 'c' to the list of libraries in the setup.py script, but this doesn't change my results. I suspect that this is because I am creating a shared library shim rather than an executable.
How can I satisfy the 3rd-party shared library's reference to stat, given my build environment?
My build system is:
-sh-3.00$ lsb_release -a
LSB Version: :core-3.0-ia32:core-3.0-noarch:graphics-3.0-ia32:graphics-3.0-noarch
Distributor ID: CentOS
Description: CentOS release 4.6 (Final)
Release: 4.6
Codename: Final
My gcc version is:
-sh-3.00$ gcc --version
gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-10)
My Python version is:
-sh-3.00$ python -V
Python 2.3.4
The solution was to create to a new Centos 5.3 VM and re-build and/or re-install components as needed.
As it turns out, while moving to Centos 5.3 was probably a good thing in the long run, the actual problem turns out to have been the way that libz4lnx was built on the DVD that I was originally using. In the process of moving to Centos 5.3, I also moved to a newer build of the libz4lnx library. Today, while testing something else, I used the library from the original DVD and got the exact same undefined symbol error when running the Python program. Switching back to the newest DVD (some two months newer) solved the problem again.