Develop Reference

Python C extension with openmp for OS X

I have created a python extension for a C program. In linux, using gcc, everything works correctly and the extension can be installed typing:
sudo python setup.py install
But when I try to use it in OS X:
GCC 4.7:
I have installed gcc 4.9 using macports and I have added this line in my setup.py file
import os
os.environ["CC"]="gcc-mp-4.9"
And when I type sudo python setup.py install
I obtain this error:
unrecognized command line option '-Wshorten-64-to-32'
I have been looking for the solution and everybody say "Use clang instead of gcc" to solve that issue.
Clang 3.8:
I have also installed clang 3.8 (3.5 is installed in os X but it doesn't have openmp) and I have modified the file setup.py:
import os
os.environ["CC"]="clang-mp-3.8"
And I obtain this error:
unknown argument: '-mno-fused-madd'
In some forums I have found a possible solution for this issue setting an empty value for CFLAGS:
sudo CFLAGS="" python setup.py install
But I obtain a new error:
library not found for -lgomp
I use -fopenmp but I do not why -fgomp is called. In some forums people say that I must use gcc instead of clang, so I am at the starting point again.
I would like to find a solution to easily isntall this extension in OS X because I would like to create an extension that can be easily installed by anyone.

I had a similar problem. Python is built with clang and uses clang-specific CFLAGS:
>>> import sysconfig
>>> sysconfig.get_config_var("CFLAGS")
'-fno-strict-aliasing -fno-common -dynamic -arch x86_64
-arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing
-fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall
-Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g
-fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE'
Distutils copies this information to the "UnixCCCompiler" instance, used to make the extension. However, as you found out, -Wshorten-64-to-32 is clang specific.
My solution was to modify how distutils builds extensions. The following removes that option from the list of command-line arguments to pass to the compiler before calling the real compilation code. (You code may not need to be so complicated. I support several compilers and configurations.)
def _is_gcc(compiler):
return "gcc" in compiler or "g++" in compiler
class build_ext_subclass( build_ext ):
def build_extensions(self):
c = self.compiler.compiler_type
if c == "unix":
compiler_args = self.compiler.compiler
c = compiler_args[0] # get the compiler name (argv0)
if _is_gcc(c):
names = [c, "gcc"]
# Fix up a problem on older Mac machines where Python
# was compiled with clang-specific options:
# error: unrecognized command line option '-Wshorten-64-to-32'
compiler_so_args = self.compiler.compiler_so
for args in (compiler_args, compiler_so_args):
if "-Wshorten-64-to-32" in args:
del args[args.index("-Wshorten-64-to-32")]
build_ext.build_extensions(self)
Then tell setup() to use this new subclass to build extensions:
setup(name = ...
cmdclass = {"build_ext": build_ext_subclass},
)

OSX: pyosmium setup error

I'm running a setup Python script for Pyosmium on OSX 10.11, Python version is 2.7:
python setup.py install
running install
running build
running build_py
running build_ext
building 'osmium._osmium' extension
gcc -fno-strict-aliasing -I/Users/stephen/anaconda/include -arch x86_64
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/include
-I../libosmium/include -I/Users/stephen/anaconda/include/python2.7 -c
lib/osmium.cc -o build/temp.macosx-10.5-x86_64-2.7/lib/osmium.o -std=c++0x
-stdlib=libstdc++ -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
In file included from lib/osmium.cc:1:
../libosmium/include/osmium/visitor.hpp:36:10: fatal error: 'type_traits' file not found
As far as possible I've most available dependencies using Brew. And added required directories for supporting developments.
Though it's clear I am dealing with a level here that I am not at all familiar with.
My guess at the moment is the compiler that's been called - gcc, should this be Any ideas would be much appreciated!
Thank you,
Stephen.

How to embed particular version of python interpreter in c in OS X

I want to embed python in C. But I find that the version of python interpreter which is embedded in my program is 2.7 (The default version on mac).
How could I specify particular version of python interpreter when I compile the c codes in mac os x. The gcc in os x is definitely different from in linux.
I have already installed python3 through HomeBrew.
Thanks a lot.
UPDATE:
I try to use python3.4-config --cflags and python3.4-config --ldflags to find out the required compiler and linker flags. Then I get these recommended flags when compiling & linking:
-I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m -Wno-unused-result -Werror=declaration-after-statement -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include
and
-L/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m -ldl -framework CoreFoundation -lpython3.4m
After this, I assemble these flags along with source file into gcc, and obtain an error:
Undefined symbols for architecture x86_64:
"_PyUnicodeUCS2_FromString", referenced from:
_main in py2-5d8da5.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
The C code which I test here comes from Python Documentation

I got the same errors when trying to do this tutorial on OSX. You don't need all of the flags that the config utility spits out. You definitely don't need the corefoundation framework if you're just doing the embedding tutorial. Just use the include directory for headers:
-I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m
, and the library to link to:
-L/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m -lpython3.4m
so here's a one-liner to compile and link:
gcc -I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m -I/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/include/python3.4m -L/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m -lpython3.4m /path/to/main.c -o /path/to/output/executable

Installing PIL on OS X Lion: initialization from incompatible pointer type

I want to run python's ndimage to do some image analysis. I have a 64-bit Mac running OSX Lion and Python 2.7. When I tried to run commands from ndimage commands I found out that I need to install PIL.
I downloaded that and unzipped it (for now into the downloads folder, is there a better place to do it?). I run setup.py and get these messages:
running install
running build
running build_py
running build_ext
--- using frameworks at /System/Library/Frameworks
building '_imaging' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -IlibImaging -I/Library/Frameworks/Python.framework/Versions/2.7/include -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _imaging.c -o build/temp.macosx-10.6-intel-2.7/_imaging.o
unable to execute gcc-4.2: No such file or directory
error: command 'gcc-4.2' failed with exit status 1
I read Failed to build PIL on Mac OS X 10.7 Lion and made sure that I have the 32/64 bit version of python 2.7 installed (Mac OS X 64-bit/32-bit x86-64/i386 Installer (2.7.2) for Mac OS X 10.6 and 10.7 ). I also have Xcode installed. Am I doing something stupid here?
Edit 1:
looking further, I have found this gcc-4.2 failed with exit status 1. I've tried entering this when I get the error:
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -g -O2 -DNDEBUG -g -O3 -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -IlibImaging -I/Library/Frameworks/Python.framework/Versions/2.7/include -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _imaging.c -o build/temp.macosx-10.6-intel-2.7/_imaging.o
and I get these messages:
_imaging.c:3017: warning: initialization from incompatible pointer type
_imaging.c:3077: warning: initialization from incompatible pointer type
_imaging.c:3017: warning: initialization from incompatible pointer type
_imaging.c:3077: warning: initialization from incompatible pointer type
Edit 2:
I'm not sure if this is the right way to go, but I found a post on installing pil on OSX Leopard and, following its advice, got rid of the -arch i386 part of the command and entered:
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch x86_64 -g -O2 -DNDEBUG -g -O3 -DHAVE_LIBZ -I/System/Library/Frameworks/Tcl.framework/Headers -I/System/Library/Frameworks/Tk.framework/Headers -IlibImaging -I/Library/Frameworks/Python.framework/Versions/2.7/include -I/usr/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _imaging.c -o build/temp.macosx-10.6-intel-2.7/_imaging.o
Now I only get two errors:
_imaging.c:3017: warning: initialization from incompatible pointer type
_imaging.c:3077: warning: initialization from incompatible pointer type
From what I can tell at this point the program poops out and doesn't finish compiling. Can anyone help me take it from here?

https://github.com/kennethreitz/osx-gcc-installer/downloads
Download GCC for lion. It solved all the error: command 'gcc-4.2' failed with exit status 1
problems.

I have had very good success with the MacPorts Python Imaging Library (PIL), Pandas, Numpy and other numerical analysis packages on both Lion and Mountain Lion.
Recently there were some significant upgrades with gcc for integration with the latest numerical Python modules on MacPorts. Looked like a very significant effort. I recommend MacPorts unless you are determined to hash through a native install of PIL on Lion.

I had compiling problems with PIL with Mountain Lion and python 2.7. I used Pillow instead, it's a friendly PIL fork with wider platform support.
regardins your setup questions:
I use macports for python 2.7 and other open source stuff on my mac (libpng, libjpeg, etc for PILLOW)
virtualenv for creating a virtual python environment (venv will be built-in into python 3.3)
I put source code under ~/src (instead Downloads folder)

What is the meaning of this ImportError when importing a Cython generated .so file?

I am walking through the Cython documentation and building each of the example apps. I'm a little stuck at Using C Libraries. After successfully building the .so file and attempting to import it in a python file called test.py the following error is thrown.
$ python3.2 test.py
Traceback (most recent call last):
File "test.py", line 12, in <module>
from queue import Queue
ImportError: dlopen(/Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so, 2): Symbol not found: _queue_free
Referenced from: /Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so
Expected in: flat namespace
in /Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so
The .so file sits right next to the test.py file. So, it seems as though it should be found.
This is running the latest version of Cython, with Python 3.2 on a OSX 10.6.
Any insights?
Edit - adding build command and output
$ python3.2 setup.py build_ext --inplace
running build_ext
cythoning queue.pyx to queue.c
building 'queue' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c queue.c -o build/temp.macosx-10.6-intel-3.2/queue.o
queue.c: In function ‘__pyx_f_5queue_5Queue_append’:
queue.c:627: warning: cast to pointer from integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_extend’:
queue.c:740: warning: cast to pointer from integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_peek’:
queue.c:813: warning: cast from pointer to integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_pop’:
queue.c:965: warning: cast from pointer to integer of different size
gcc-4.2 -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-3.2/queue.o -o
Edit 2 - adding "otool" cmd requested in comment
queue.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
Edit 3 - adding "nm" output
U ___stack_chk_fail
U ___stack_chk_guard
U _queue_free
U _queue_is_empty
U _queue_new
U _queue_peek_head
U _queue_pop_head
U _queue_push_tail
U dyld_stub_binder
grep cmd outputs this:
(undefined) external _queue_free (dynamically looked up)

EDIT:
Ah, you didn't mention you had a dependency on the code in libcalg. That stuff needs to be compiled and included when you build the cextension.
Just modify setup.py:
# setup.py
# ...
ext_modules = [Extension("queue", ["queue.pyx", "libcalg/queue.c"])]
# ...
We could step back and see if you can build a really simple example:
I've tried the following (3 files, myext.pyx, test.py, setup.py) and it appears to work fine. Of course I'm on OS X 10.7 so it's not exactly the same as your environment. To rule out differences perhaps you can copy these and build them as a sanity check.
Contents of myext.pyx:
# myext.pyx
def square(x):
return x * x
Contents of test.py
# test.py
from myext import square
print "%d squared is %d"%(4, square(4))
Contents of setup.py:
# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("myext", ["myext.pyx"])]
setup(
name = 'Hello world app',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
I build in the directory containing these 3 files:
cython_test$ /usr/bin/python setup.py build_ext --inplace
running build_ext
cythoning myext.pyx to myext.c
building 'myext' extension
creating build
creating build/temp.macosx-10.7-intel-2.7
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c myext.c -o build/temp.macosx-10.7-intel-2.7/myext.o
llvm-gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -Wl,-F. -arch i386 -arch x86_64 build/temp.macosx-10.7-intel-2.7/myext.o -o /Users/steder/SO/cython_test/myext.so
cython_test$ python test.py
4 squared is 16:
My environment has similar otool output and DYLD_LIBRARY_PATH is also unset but nm -m shows squared as defined.
Specifically:
00000000000011d0 (__DATA,__data) non-external ___pyx_k__square
00000000000011e0 (__DATA,__data) non-external ___pyx_mdef_5myext_square
0000000000001218 (__DATA,__bss) non-external ___pyx_n_s__square
0000000000000c80 (__TEXT,__text) non-external ___pyx_pf_5myext_square
Please give this a shot and see what it nm -m shows in your environment.