Develop Reference

Compiling a wrapper for a c library using Cython - Linker cant find .dylib of external c library on OSX

I have written a wrapper in Cython for an integration function from NAG (https://www.nag.co.uk/content/nag-library-c) c library.
It compiles using the python setup.py build --inplace, where setup file is:
from setuptools import Extension, setup
from Cython.Build import cythonize
import re
def loadMacros(headerFile):
""" Given a .h file, return dict of touples with macros """
regex = re.compile("#define +(\w+) *(\w*)")
with open(headerFile) as f:
macros = dict(map(lambda x: re.match(regex, x).groups(),
[l for l in f if re.match(regex, l)]))
# Remove recursive entries - Note this is not foolproof..
# while not set(macros.keys()).isdisjoint(macros.values()):
# for k, v in macros.items():
# if v in macros:
# macros[k] = macros[v]
return macros
nagHome = "/Users/hfmw1m17/NAG/nlmi627dbl" # "/opt/NAG/cll6a23dhl"
macros = loadMacros(nagHome + "/lp64/include/nag.h")
macros = list(macros.items())
e = Extension("nag_integrate",
define_macros=macros,
sources=["nag_integrate.pyx"],
include_dirs=[nagHome + "/lp64/include",nagHome + "/lp64/lib"],
library_dirs=[nagHome + "/lp64/lib"],libraries=["nag_nag"],extra_objects=[
nagHome+"/lp64/lib/libnag_nag.dylib"],runtime_library_dirs=[nagHome+"/lp64/lib/"],extra_link_args=['-Wl,-rpath']
)
setup(ext_modules=cythonize(e,annotate=True,language_level=3))enter code here
with output:
/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/bin/python3.7 setup.py build_ext --inplace
Compiling nag_integrate.pyx because it changed.
[1/1] Cythonizing nag_integrate.pyx
running build_ext
building 'nag_integrate' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/include -arch x86_64 -I/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/include -arch x86_64 -DNAG_H= -DNAG_MICROSOFT_THREAD_SAFE= -DNAG_THREAD_SAFE= -DNULLFN=0 -DNULLDFN=0 -DNAGERR_DEFAULT= -DNAGUSER_DEFAULT= -DNAGCOMM_NULL= -DNAGMESG_DEFAULT= -DE01_DEFAULT= -DE04_DEFAULT= -DG13_DEFAULT= -DH02_DEFAULT= -DINIT_FAIL= -DSET_FAIL= -DINIT_MESG= -DINIT_STREAM= -DRDUMMY= -DIDUMMY= -DINIT2DUMMY= -DVprintf= -DVfprintf= -DVsprintf= -DVscanf= -DVfscanf= -DVstrcpy= -DABS= -DFABS= -DSIGN= -DMAX= -DMIN= -DDROUND= -DROUND= -DSQZABS= -DCONJ= -DVCONJ= -DZMULT= -D_nag_expand= -Dnag_stringize= -I/Users/hfmw1m17/NAG/nlmi627dbl/lp64/include -I/Users/hfmw1m17/NAG/nlmi627dbl/lp64/lib -I/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/include/python3.7m -c nag_integrate.c -o build/temp.macosx-10.9-x86_64-3.7/nag_integrate.o
gcc -bundle -undefined dynamic_lookup -L/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/lib -arch x86_64 -L/Users/hfmw1m17/anaconda3/envs/TowingTankAcoustics/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.7/nag_integrate.o /Users/hfmw1m17/NAG/nlmi627dbl/lp64/lib/libnag_nag.dylib -L/Users/hfmw1m17/NAG/nlmi627dbl/lp64/lib -L/Users/hfmw1m17/NAG/nlmi627dbl/lp64/lib/ -lnag_nag -o build/lib.macosx-10.9-x86_64-3.7/nag_integrate.cpython-37m-darwin.so -Wl,-rpath
copying build/lib.macosx-10.9-x86_64-3.7/nag_integrate.cpython-37m-darwin.so ->
Process finished with exit code 0
However when i import a function from the .so object created i get the following error:
ImportError: dlopen(/Users/hfmw1m17/WaterTankISM/WaterTankISM/nag_integration/nag_integrate.cpython-37m-darwin.so, 2): Library not loaded: libnag_nag.dylib
Referenced from: /Users/hfmw1m17/WaterTankISM/WaterTankISM/nag_integration/nag_integrate.cpython-37m-darwin.so
Reason: image not found
libnag_nag.dylib is a dynamic library produced by NAG.
Using otool -L on the shared object of my wrapper results in:
libnag_nag.dylib (compatibility version 0.0.0, current version 27.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
I think its an issue with the linker not being able to find the dynamic library when compiling. Any suggestions on how to solve this problem?
many thanks

Solved using:
install_name_tool -add_rpath path_to_dylib_directory
after running python setup.py build --inplace.
Would like a way of doing this in the setup.py file if anyone can figure that out.
many thanks

Advice on creating Python libraries from C++?

I recently created a 3D and 2D force layout diagram visualizer in C++ using the OpenGL library (it uses some physics, sort of). Can someone give me some introductory pointers about making this useful as a Python library (issues or considerations and potential pitfalls I might run into)?

If I understand your question correctly you want to know how to write C extensions for use in Python. Here's a trivial simple example of how:
hello.c:
#include <Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] =
"helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld,
METH_NOARGS, helloworld_docs},
{NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,
"Extension module example!");
}
setup.py:
#!/usr/bin/env python
from setuptools import setup, Extension
setup(
name='helloworld',
version='1.0',
ext_modules=[
Extension('helloworld', ['hello.c'])
]
)
Build:
$ python setup.py develop
running develop
running egg_info
creating helloworld.egg-info
writing helloworld.egg-info/PKG-INFO
writing top-level names to helloworld.egg-info/top_level.txt
writing dependency_links to helloworld.egg-info/dependency_links.txt
writing manifest file 'helloworld.egg-info/SOURCES.txt'
reading manifest file 'helloworld.egg-info/SOURCES.txt'
writing manifest file 'helloworld.egg-info/SOURCES.txt'
running build_ext
building 'helloworld' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c hello.c -o build/temp.linux-x86_64-2.7/hello.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/hello.o -o build/lib.linux-x86_64-2.7/helloworld.so
copying build/lib.linux-x86_64-2.7/helloworld.so ->
Creating /home/prologic/.virtualenvs/hellopyc/lib/python2.7/site-packages/helloworld.egg-link (link to .)
Adding helloworld 1.0 to easy-install.pth file
Installed /home/prologic/tmp/hello-py-c
Processing dependencies for helloworld==1.0
Finished processing dependencies for helloworld==1.0
Test:
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import helloworld
>>> helloworld.helloworld()
'Hello, Python extensions!!'
>>>
Good luck!

While James Mills answer is fine, i would consider other option - use Boost.Python library to write python extensions. Here is a basic example. It looks like Boost.Python is not maintained (last update - 2009), but still it's a very good option. Right now i'm using it in one project and i've got to say that after a little difficult start, now it's really usefull and quite easy. Most important for me is that it handles reference counting(which is painful in Python/C API), allows to modify python objects inside c++ code and allows you to register converters for custom datatypes. So instead of writing:
MyClass test = convertPythonObjectToMyClass(pythonObject);
MyOtherClass test2 = convertPythonObjectMyOtherClass(pythonObject);
you can register your converters and than use this:
MyClass test = boost::python::extract<MyClass>(pythonObject);
MyOtherClass test2 = boost::python::extract<MyOtherClass>(pythonObject);
Of course you can consider other options as well - see this questions for more information about other solutions.

'ImportError' in Python extension module wrapping C library

(UPDATE 3 contains the questions I'd like to get answers to. UPDATE 2 refers to corrections I did trying to understand and fix this issue)
I'm trying to get a Python extension module to wrap a C library (in this case, it is only an example described in the book Python Cookbook (3er edition.) The problem is that I'm encountering the classic error
ImportError: dynamic module does not define init function (initsample)
when I'm trying to import the module.
The book itself has the following comment:
For all of the recipes that follow, assume that the preceding code is found in a file named
sample.c, that definitions are found in a file named sample.h and that it has been com‐
piled into a library libsample that can be linked to other C code. The exact details of
compilation and linking vary from system to system, but that is not the primary focus.
It is assumed that if you’re working with C code, you’ve already figured that out.
And I think I'm getting that wrong. This is what I'm doing: First of all, I have the following files:
➜ Sample ls
csample.pxd sample.c sample.h sample.pyx setup.py
The content of the files is according to this github repository containing the files described in the book. Assuming everything is copied correctly. I then compile sample.c
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
This creates sample.o and I proceed to create a shared library:
➜ Sample gcc -shared sample.o -o libsample.so
It creates a libsample.so and finally run the setup.py file:
➜ Sample python setup.py build_ext --inplace
running build_ext
skipping 'sample.c' Cython extension (up-to-date)
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/sample.so
However, I obtain the same error. What am I missing?
This is my setup.py (eventually, I copied a part to clean previous build):
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import sys
import os
import shutil
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("sample") and not(name.endswith(".pyx") or name.endswith(".pxd") or name.endswith(".h"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
ext_modules = [
Extension('sample',
['sample.pyx'],
libraries=['sample'],
library_dirs=['.'])]
setup(
name = 'Sample extension module',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
UPDATE 2
I was wondering if it is normal that after running
python setup.py build_ext --inplace
the file sample.c gets replaced with a cythonized version.
I assume that running setup.py is using cython sample.c, but I'm asking this because yesterday I was mistakenly compiling the cythonized version sample.c (after removing libsample.so, sample.o, sample.so that were created in a previous run) and after updating sample.pyx, I got this error:
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./sample.so: undefined symbol: divide
After this, I restored sample.c to its original content, removed all remaining files from previous builds (including temporal files) and crucially, I had to change sample.pyx in some way (e.g. adding a blank line.)
I noticed I have to do this to avoid the
ImportError: dynamic module does not define init function (initsample)
error. Then I proceeded as usual:
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
➜ Sample gcc -shared sample.o -o libsample.so
➜ Sample python setup.py build_ext --inplace
running build_ext
cythoning sample.pyx to sample.c
warning: sample.pyx:27:42: Use boundscheck(False) for faster access
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/Sample/sample.so
➜ Sample python
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libsample.so: cannot open shared object file: No such file or directory
This time I get cythoning sample.pyx to sample.c in the output above and a new error. However, libsample.so is in the Sample directory. What could be the issue here?
UPDATE 3:
I finally was able to import this module successfully but the last error
ImportError: libsample.so: cannot open shared object file: No such file or directory
implied that I needed to set the current directory in the environment variable LD_LIBRARY_PATH. However, I did it in a crude way:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/Sample
What is the correct way to do this? Maybe using some parameter in setup.py? Regarding the several issues I found, I'd like to get an answer as to why all of this happened:
sample.c gets replaced when running python setup.py build_ext --inplace Is this okay?
Do I need to restore sample.c every time I'm doing this procedure?
Do I need to change sample.pyx in order to get a correct output (instead of skipping 'sample.c' Cython extension (up-to-date) with a misleading output)
What is the correct way to solve ImportError: libsample.so: cannot open shared object file: No such file or directory?
A possible solution to 4. is to add in setup.py:
runtime_library_dirs=['./'],
However, this requires libsample.so to be located in the same directory as setup.py.

Including a compiled module in module that is wrapped with f2py (Minimum working example)?

I have tried, but am failing, to get a minimum working example. As I do not need to expose much of my fortran code to python, I don't need f2py to wrap large parts of it. Also, due to allocatable arrays being passed and derived types being used, I specifically want f2py to only wrap the interface module I created (in the following example 'main.f90'). But I am having problems to get the other modules which I compile separately to link to my main module.
The Code:
Note all source files are in a single directory.
I have created a fortran module I want to compile (libtest.f90):
module testmod
implicit none
contains
subroutine testsub(arr)
real, allocatable, intent(in) :: arr(:,:)
print *, 'testsub executed'
end subroutine testsub
end module testmod
and a fortran module that I want to wrap with f2py (main.f90):
module mainmod
use testmod
implicit none
contains
subroutine mainsub
real, allocatable :: arr(:,:)
call testsub(arr)
end subroutine main sub
end module mainmod
I use the following compile commands:
gfortran -c -fPIC libtest.f90
which generates 'libtest.o' and 'testmod.mod', and
f2py -c --fcompiler=gfortran -L. -I. -llibtest -m Main main.f90
Which gives me 'ld: library not found for -llibtest'.
I don't understand why this occurs, since it seems to work for others (F2PY doesn't find a module).
If I take out the -llibtest, I of course get (in my python script):
Traceback (most recent call last):
File "./script.py", line 7, in <module>
import Main
ImportError: dlopen(/Users/gmueller/Workspace/Minimum_PySpin/Main.so, 2): Symbol not found: ___testmod_MOD_testsub
Referenced from: /Users/gmueller/Workspace/Minimum_PySpin/Main.so
Expected in: dynamic lookup
Edit: Note I am on OSX(10.9.5), in case that makes some difference (but it shouldn't, since I do not need to pass any -shared (linux) or -dynamiclib (osx) to gfortran, right?).
Here the full output from f2py:
f2py -c --fcompiler=gfortran -L. -I. -llibtest -m Main main.f90
Unknown vendor: "gfortran"
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "Main" sources
f2py options: []
f2py:> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
Reading fortran codes...
Reading file 'main.f90' (format:free)
Post-processing...
Block: Main
Block: mainmod
In: :Main:main.f90:mainmod
get_useparameters: no module testmod info used by mainmod
Block: mainsub
In: :Main:main.f90:mainmod:mainsub
get_useparameters: no module testmod info used by mainsub
Post-processing (stage 2)...
Block: Main
Block: unknown_interface
Block: mainmod
Block: mainsub
Building modules...
Building module "Main"...
Constructing F90 module support for "mainmod"...
Constructing wrapper function "mainmod.mainsub"...
mainsub()
Wrote C/API module "Main" to file "/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c"
Fortran 90 wrappers are saved to "/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90"
adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c' to sources.
adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7' to include_dirs.
copying /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.c -> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
copying /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/f2py/src/fortranobject.h -> /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
adding '/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90' to sources.
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/local/bin/gfortran
customize Gnu95FCompiler using build_ext
building 'Main' extension
compiling C sources
C compiler: gcc -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8
creating /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
gcc: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.c:2:
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.h:13:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:
/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " \
^
1 warning generated.
gcc: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.c:17:
In file included from /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.h:13:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:17:
In file included from /usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1804:
/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " \
^
1 warning generated.
compiling Fortran 90 module sources
Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
extra options: '-J/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/ -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/'
gfortran:f90: main.f90
compiling Fortran sources
Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/local/bin/gfortran -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I. -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7 -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c'
extra options: '-J/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/ -I/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/'
gfortran:f90: /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.f90
/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/main.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.o -L. -L/usr/local/lib/gcc/x86_64-apple-darwin/4.8.2 -llibtest -lgfortran -o ./Main.so
ld: library not found for -llibtest
collect2: error: ld returned 1 exit status
ld: library not found for -llibtest
collect2: error: ld returned 1 exit status
error: Command "/usr/local/bin/gfortran -Wall -g -Wall -g -undefined dynamic_lookup -bundle /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Mainmodule.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/fortranobject.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/main.o /var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/var/folders/yg/65v1lbd153v1jlt1kb91vcbm0000gq/T/tmpJSf2e8/src.macosx-10.4-x86_64-2.7/Main-f2pywrappers2.o -L. -L/usr/local/lib/gcc/x86_64-apple-darwin/4.8.2 -llibtest -lgfortran -o ./Main.so" failed with exit status 1

Your command:
gfortran -c -fPIC libtest.f90
produces an object file with position independent code. This is a pre-requisite of a shared library, not a shared library.
If you want to use the object as is, you can modify your f2py invocation:
f2py -c --fcompiler=gfortran -I. libtest.o -m Main main.f90
This will link the object file and produce the file Main.cpython-33.so (the python version number may differ for you) and you can then import main in your python code.
If you would rather actually produce a shared object, you need to compile to a shared library. One way to do this is:
gfortran -shared -O2 -o libtest.so -fPIC libtest.f90
This produces libtest.so and now your original f2py command will work with one small change:
f2py -c --fcompiler=gfortran -L. -I. -ltest -m Main main.f90
The small change I am referring to is changing -llibtest to -ltest, as the -l option will add lib to the front of the library and .so to the end, e.g. -ltest will look for libtest.so. This produces Main.cpython-33.so with a dynamic link dependency to libtest.so, so you will need to distribute both shared libraries in order to use the python module.

Python, ImportError: undefined symbol: g_utf8_skip

There are like tens of similar questions on StackOverflow, but after several hours of lurking I finally gave up.
So I'm trying to write a C extension for Python. Let's call it mylib. Here is the header file:
mylib.h
#ifndef mylib_H
#define mylib_H
#include <Python.h>
< ... >
#include <glib.h>
< ... >
and setup.py:
from distutils.core import setup, Extension
include_list = [
"/usr/include/glib-2.0", "-lglib-2.0",
"/usr/lib/x86_64-linux-gnu/glib-2.0/include"
]
module = Extension('mylib', ['mylib.c'])
setup(name='mylib', version='1.0',
include_dirs=include_list,
ext_modules=[module])
If I run python setup.py install, I get the following (which I take as successful installation):
running install
running build
running build_ext
building 'mylib' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/glib-2.0 -I-lglib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/python2.7 -c mylib.c -o build/temp.linux-x86_64-2.7/mylib.o
mylib.c: In function ‘c_sound_utf8’:
mylib.c:117:5: warning: ‘g_unicode_canonical_decomposition’ is deprecated (declared at /usr/include/glib-2.0/glib/gunicode.h:627) [-Wdeprecated-declarations]
decomposition = g_unicode_canonical_decomposition(c_composed, &decomposition_len);
^
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/mylib.o -o build/lib.linux-x86_64-2.7/mylib.so
running install_lib
copying build/lib.linux-x86_64-2.7/mylib.so -> /usr/local/lib/python2.7/dist-packages
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/mylib-1.0.egg-info
Writing /usr/local/lib/python2.7/dist-packages/mylib-1.0.egg-info
But when I try to use mylib from inside Python, I get the following:
>>> import mylib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /usr/local/lib/python2.7/dist-packages/mylib.so: undefined symbol: g_utf8_skip
After rambling around StackOverflow for some time I got an idea that I should either 1. rebuild the needed library or 2. put all the links to needed library after all generated module names.
Rebuilding didn't work (or I did it the wrong way). As for placing links to the needed library after everything else - well, I didn't find out the way to make distutils change the order of links in its compile string. Is there a way?
I also tried providing extra_link_args/extra_compile_args to my extension (without any effect):
module = Extension('mylib', ['mylib.c'],
extra_link_args=["-Xlinker", "-export-dynamic"])
I felt pretty miserable and kept googling on. Then I found out about SWIG. I decided to try it by making another library, (uppercase) MYLIB (I changed filenames and all text occurences of mylib to MYLIB). I wrote a shell script:
#!/bin/bash
GLIB_IMPORT_OPTS="-I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -lglib-2.0"
PY_IMPORT_OPTS="-I/usr/include/python2.7/ -lpython2.7"
swig -Wall -python MYLIB.i
gcc -fPIC -Wall -c MYLIB.c $GLIB_IMPORT_OPTS
gcc -fPIC -Wall -shared MYLIB.o MYLIB_wrap.c -o _MYLIB.so $GLIB_IMPORT_OPTS -L. $PY_IMPORT_OPTS $GLIB_IMPORT_OPTS
When I ran this thing, everything worked fine (I could import the library and do stuff with it). Here, as you can see, links are at the very end of the compile line. So now I'm trying to understand: what did I miss with the distutils way? How can I make it work?

Well, actually I found the solution. A had to add library links to extra_link_args:
extra_link_args=["-I", "/usr/include/glib-2.0", "-l", "glib-2.0", "-I", "/usr/lib/x86_64-linux-gnu/glib-2.0/include"]
which appends them to the end of compile string.

I found adding -fPIC to "extra_compile_args" in the Extension constructor also helped. Like so:
my_module = Extension('modulename',
...
extra_compile_args=["-fPIC"]
sources = ['mycode.c'])