Develop Reference

Are there any working examples of how to use the scikit-build with f2py?

The scikit-build distribution provides usage examples of FindF2PY and UseF2PY, but they are incomplete, only providing a partial CMakeLists.txt file without the other required files. Based on the documentation I have not been able to make something that builds.
Following the examples in the scikit-build documentation, I created the following files:
CMakeLists.txt:
cmake_minimum_required(VERSION 3.10.2)
project(skbuild_test)
enable_language(Fortran)
find_package(F2PY REQUIRED)
add_f2py_target(f2py_test f2py_test.f90)
add_library(f2py_test MODULE f2py_test.f90)
install(TARGETS f2py_test LIBRARY DESTINATION f2py_test)
setup.py:
import setuptools
from skbuild import setup
requires=['numpy']
setup(
name="skbuild-test",
version='0.0.1',
description='Performs line integrals through SAMI3 grids',
author='John Haiducek',
requires=requires,
packages=['f2py_test']
)
f2py_test.f90:
module mod_f2py_test
implicit none
contains
subroutine f2py_test(a,b,c)
real(kind=8), intent(in)::a,b
real(kind=8), intent(out)::c
end subroutine f2py_test
end module mod_f2py_test
In addition, I created a directory f2py_test containing an empty init.py.
The output from python setup.py develop shows that scikit-build invokes CMake and compiles my Fortran code. However, it fails to find Python.h while compiling the f2py wrapper code:
[2/7] Building C object CMakeFiles/_f2...kages/numpy/f2py/src/fortranobject.c.o
FAILED: CMakeFiles/_f2py_runtime_library.dir/venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -O3 -DNDEBUG -arch x86_64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.14 -MD -MT CMakeFiles/_f2py_runtime_library.dir/venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c.o -MF CMakeFiles/_f2py_runtime_library.dir/venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c.o.d -o CMakeFiles/_f2py_runtime_library.dir/venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c.o -c ../../../venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c
In file included from ../../../venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.c:2:
../../../venv/lib/python3.8/site-packages/numpy/f2py/src/fortranobject.h:7:10: fatal error: 'Python.h' file not found
#include "Python.h"
^~~~~~~~~~
1 error generated.

First caveat, there may be a better way to do this, as I just figured out how to make scikit-build work by stumbling on your post and looking at documentation. Second caveat, I'm also learning cmake. So, there may be a better way.
There are a couple of things that you'll need to make your example here work. The big one is the second argument of add_f2py_target() isn't the source file. It is either the name of a pre-generated *.pyf, or to let f2py generate one, provide it an argument without the *.pyf extension. The other is adding the include directories for various components.
I made your example work with the following CMakeLists.txt:
cmake_minimum_required(VERSION 3.10.2)
project(skbuild_test)
enable_language(Fortran)
find_package(F2PY REQUIRED)
find_package(PythonLibs REQUIRED)
find_package(Python3 REQUIRED COMPONENTS NumPy)
#the following command either generates or points to an existing .pyf
#if provided an argument with .pyf extension, otherwise f2py generates one (not source code).
add_f2py_target(f2py_test f2py_test)
add_library(f2py_test MODULE f2py_test.f90)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(${_Python3_NumPy_INCLUDE_DIR})
target_link_libraries(f2py_test ${PYTHON_LIBRARIES})
install(TARGETS f2py_test LIBRARY DESTINATION f2py_test)

Creating a Python package for C++ code wrapped with pybind11 using setuptools

I have a C++ file called VBB.cpp which contains implementations of a few classes and I wrote Python bindings for those classes using the pybind11 library, these are located in bindings.cpp. I can successfully compile the code with:
g++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` bindings.cpp VBB.cpp -o VBB`python3-config --extension-suffix
And then use the C++ code from Python with import library.
I want to turn this into a Python package via setuptools. I just used the example setup.py file available at https://github.com/pybind/python_example and modified the Extension call with
Extension(
'VBB',
['src/bindings.cpp', 'src/VBB.cpp'],
include_dirs=[
# Path to pybind11 headers
get_pybind_include(),
get_pybind_include(user=True)
],
language='c++'
),
If I run the install script it compiles but if I try to run import VBB in Python, I get the following error:
ImportError: dynamic module does not define module export function (PyInit_VBB)
I'm new to using setuptools so I'm not sure if I'm doing something wrong. The example package from GitHub works without any issues.

Error when trying to cross-compile SWIG Python extension for mingw32 using distutils

I am trying to cross-compile a simple SWIG Python extension on Linux for Windows (mingw32), using the distutils module.
The ultimate goal is to compile a Python wrapper for some library and being able to use it on Windows. Obviously I started with the most basic example and unfortunately it fails.
Here are the files I am using:
example.c
/* File : example.c */
/* A global variable */
double Foo = 3.0;
/* Compute the greatest common divisor of positive integers */
int gcd(int x, int y) {
int g;
g = y;
while (x > 0) {
g = x;
x = y % x;
y = g;
}
return g;
}
example.i - SWIG interface file
/* File : example.i */
%module example
%inline %{
extern int gcd(int x, int y);
extern double Foo;
%}
setup.py
# setup.py
import distutils
from distutils.core import setup, Extension
setup(name = "SWIG example",
version = "1.0",
ext_modules = [Extension("_example", ["example.i","example.c"])])
In order to compile using the native (Linux) gcc compiler, I am invoking:
python setup.py build
Everything works like a charm! Unfortunately when trying to specify the Windows target:
python setup.py build --compiler=mingw32
I get the error saying that gcc can't recognize -mdll switch:
running build
running build_ext
building '_example' extension
swigging example.i to example_wrap.c
swig -python -o example_wrap.c example.i
creating build
creating build/temp.linux-x86_64-2.7
gcc -mdll -O -Wall -I/home/jojek/anaconda/include/python2.7 -c example_wrap.c -o build/temp.linux-x86_64-2.7/example_wrap.o
gcc: error: unrecognized command line option ‘-mdll’
error: command 'gcc' failed with exit status 1
Fair enough, it makes perfect sense, since toolchain is not valid. I made sure that mingw32 is installed on my machine. By calling dpkg -L mingw32 I know that compiler is located in /usr/bin/i586-mingw32msvc-gcc.
My next step was to override the CC environmental variable with the actual path to my compiler. When I try to compile it again, then I am getting the following error with missing sys/select.h header file:
running build
running build_ext
building '_example' extension
swigging example.i to example_wrap.c
swig -python -o example_wrap.c example.i
creating build
creating build/temp.linux-x86_64-2.7
/usr/bin/i586-mingw32msvc-gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/jojek/anaconda/include/python2.7 -c example_wrap.c -o build/temp.linux-x86_64-2.7/example_wrap.o
example_wrap.c:1: warning: -fPIC ignored for target (all code is position independent)
In file included from /home/jojek/anaconda/include/python2.7/Python.h:58,
from example_wrap.c:125:
/home/jojek/anaconda/include/python2.7/pyport.h:351:24: error: sys/select.h: No such file or directory
error: command '/usr/bin/i586-mingw32msvc-gcc' failed with exit status 1
Does anyone have an idea how to manage that task?

There's quite a bit going on behind the scenes when you compile Python modules using distutils. You're getting closer with each try in your question, however the problem you've now encountered is that you're using Linux header files with a Windows (cross) compiler. (sys/select.h isn't supported with mingw32, cygwin might be a different story though). In reality it's the lack of the configuration header file that's causing your cross compile to try and use the POSIX interfaces instead of Win32 alternatives.
My answer rewinds a few steps and starts out simply building the module by hand, using mingw32 on Linux and then we'll look at using distutils once we've proven that we have all that's required.
I'm also assuming that you don't have a Windows build box (or even a VM) available to simply build your extension on Windows natively since that's far simpler than cross compiling. If you're reading this and have the option to use a Windows box to build your Windows Python extensions, do that instead and save time and effort. That said it is possible to build Windows Python modules using only a Linux box.
Starting with mingw32 already installed and working on your Linux box (e.g. using the Debian/Ubuntu packages) the first step is to get the Windows header files (or configuration to be more specific). I'm assuming you're targeting the build that most people get when they type "python windows" into a search engine so I downloaded the Windows MSI installers from python.org and extracted them from there.
There are two things we want to get from the Python distribution:
python27.dll (Usually gets placed in c:\windows\system32 or c:\windows\syswow64)
The 'include' directory (Usually gets placed in c:\python27\include)
Under Linux there are a few different ways you can extract this. You could use Wine to install the MSI file. I used both cabextract and 7z with success in my testing though, for example with cabextract:
cabextract /tmp/python-2.7.10.msi -F '*.h'
cabextract /tmp/python-2.7.10.msi -F 'python27.dll'
(Note: if you use 7z you'll find the files you really want inside a second, inner archive named 'python').
At this point you could also extract the file 'libpython27.a' which usually lives inside c:\python27\libs\ however this file isn't sufficient or even useful for linking using mingw32.
Given the header files we've now got enough to compile our extension, although as noted above to get mingw32 to link against python27.dll we need to do a bit more work first. We're going to need a tool called pexports to list all the exported symbols in the Python DLL and let dlltool generate a stub library for mingw32 to link against. I downloaded pexports directly and then extracted it with:
tar xvf ~/Downloads/pexports-0.47-mingw32-bin.tar.xz
Once that's extracted we get a single Windows executable. I used Wine in my example here to run it directly; alternatively, you could extract the source, and build it as a tool to run natively on the Linux host:
tar xvf ~/Downloads/pexports-0.47-mingw32-src.tar.xz
(cd pexports-0.47 && ./configure && make)
or you could have duplicated the functionality of the tool using the Python module pefile (which runs fine cross platform) to extract the exports that we care about if you were looking to avoid using Wine as well.
Anyway with pexports you can generate a .def file that contains the information we need for dlltool:
wine bin/pexports.exe -v python27.dll > python27.def
or, (if you've built pexports as a native tool), simply:
./pexports-0.47/pexports -v python27.dll > python27.def
where python27.dll is what we extracted from the .msi file earlier.
(This was my pexports reference)
Once you've got the .def file you can use the mingw32 dlltool to generate a .a file that we'll use later to link our Python module against:
i586-mingw32msvc-dlltool -A --dllname python27.dll --def python27.def --output-lib libpython27.a
Now we've reached a point where we can think about running SWIG itself to generate the code for us to compile. I simplified your example interface even further to be just:
%module test
%inline %{
int gcd(int x, int y) {
int g;
g = y;
while (x > 0) {
g = x;
x = y % x;
y = g;
}
return g;
}
%}
And then ran SWIG on my Linux box as:
swig -Wall -python test.i
This generated test_wrap.c which I compiled with:
i586-mingw32msvc-gcc test_wrap.c -I../include -Wall -Wextra -shared -o _test.pyd ./libpython27.a
And there we have a Windows Python module built using just Linux.
To check it really runs I copied test.py and _test.pyd to a Windows box and then did:
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test
>>> test.gcd(1024, 512)
512
>>>
Now all that remains is to make sure distutils can find the right include files and libraries to link against by manipulating its paths.

For 64 bits, I couldnt make it work with pexports, so I used gendef to generate the python27.def file.
gendef is a tool which generate def files from DLLs. A def file is a list of symbols exported by a DLL. The primary use of this tool is to allow creation of a import library of DLLs created by non-GCC compilers. It can handle both x86 (win32) and amd64 (win64) executables.
https://sourceforge.net/p/mingw-w64/wiki2/gendef/
Hope It helps!

How to remove a compiler option that python disutils includes (language code=pic32 not recognized)

I'm trying to compile the msgpack-python python module with gcc (v4.7) on solaris 10. The python installed is 2.6.8. Distutils is automatically picking up a incorrect compiler option (-xcode=pic32) that I want to remove from command.
The full command that distutils is putting together is:
/opt/csw/bin/gcc-4.7 -DNDEBUG -O -O2 -pipe -mcpu=v9 -I/opt/csw/include -xcode=pic32 -I/opt/csw/include/python2.6 -c msgpack/_msgpack.c -o build/temp.solaris-2.10-sun4v-2.6/msgpack/_msgpack.o
but produces this error:
gcc-4.7: error: language code=pic32 not recognized
then fails. If I remove that -xcode=pic32 option and manually execute the above command the module compiles successfully.
I need to be able to do this in an automated fashion though (using a buildfarm to produce the packages). The question is, Without modifying or changing the current python or distutils, is there a way to "remove" this option that distutils is picking up, so I can have the python setup.py process build the module appropriately (i.e. without the pic32 option)?
Thanks

Do not compile with that gcc. -xcode=pic32 is Sun Studio complier command line parameter. It will lead to linking problems too, even if you compile OK. Compile with SUN CoolTools gcc which can understand such parameter, or use Oracle Solaris Studio for SPARC.
Some hints:
GCC produce very slow code for SPARC, that's for why SUN created Cool Tools.
You haven't to remove -xcode=pic32, but change for -m32 -fpic, when you insist on gcc-4.7
To get mature setup of OSS tool I'm using pkgsrc compiling with Studio Express to particular CPU (-xtarget=native)

You may also find luck by setting the follow env vars:
export CC=$gcc_dir_path # Example: /usr/bin/gcc
export CXX=$gxx_dir_path # Example: /usr/bin/g++
export CFLAGS=''
export CPPFLAGS=''
export CXXFLAGS=''
export LDFLAGS=''
Note: There is a difference between unset env var, and set-as-empty env var. I had build bugs with Python packages when my *FLAGS env vars were unset. (Calling gcc with option -xO2 was the cause.) Setting as empty did the trick.

How to statically link a library when compiling a python module extension

I would like to modify a setup.py file such that the command "python setup.py build" compiles a C-based extension module that is statically (rather than dynamically) linked to a library.
The extension is currently dynamically linked to a number of libraries. I would like to leave everything unchanged except for statically linking to just one library. I have successfully done this by manually modifying the call to gcc that distutils runs, although it required that I explicitly listed the dependent libraries.
Perhaps this is too much information, but for clarity this is the final linking command that was executed during the "python setup.py build" script:
gcc -pthread -shared -L/system/lib64 -L/system/lib/ -I/system/include build/temp.linux-x86_64-2.7/src/*.o -L/system/lib -L/usr/local/lib -L/usr/lib -ligraph -o build/lib.linux-x86_64-2.7/igraph/core.so
And this is my manual modification:
gcc -pthread -shared -L/system/lib64 -L/system/lib/ -I/system/include build/temp.linux-x86_64-2.7/src/*.o -L/system/lib -L/usr/local/lib -L/usr/lib /system/lib/libigraph.a -lxml2 -lz -lgmp -lstdc++ -lm -ldl -o build/lib.linux-x86_64-2.7/igraph/core.so
Section 2.3.4 of Distributing Python Modules discusses the specification of libraries, but only "library_dirs" is appropriate and those libraries are dynamically linked.
I'm using a Linux environment for development but the package will also be compiled and installed on Windows, so a portable solution is what I'm after.
Can someone tell me where to look for instructions, or how to modify the setup.py script? (Thanks in advance!)
I'm new to StackOverflow, so my apologies if I haven't correctly tagged this question, or if I have made some other error in this posting.

6 - 7 years later, static linking with Python extensions is still poorly documented. As the OP points out in a comment, the usage is OS dependend.
On Linux / Unix
Static libraries are linked just as object files and should go with the path and its extension into extra_objects.
On Windows
The compiler sees if the linked library is static or dynamic and the static library name goes to the libraries list and the directories to library_dir
Solution for both platforms
For the example below, I will use the same library scenario as OP, linking igraph static and z, xml2 and gmp dynamic. This solution is a bit hackish, but at least does for each platform the right thing.
static_libraries = ['igraph']
static_lib_dir = '/system/lib'
libraries = ['z', 'xml2', 'gmp']
library_dirs = ['/system/lib', '/system/lib64']
if sys.platform == 'win32':
libraries.extend(static_libraries)
library_dirs.append(static_lib_dir)
extra_objects = []
else: # POSIX
extra_objects = ['{}/lib{}.a'.format(static_lib_dir, l) for l in static_libraries]
ext = Extension('igraph.core',
sources=source_file_list,
libraries=libraries,
library_dirs=library_dirs,
include_dirs=include_dirs,
extra_objects=extra_objects)
On MacOS
I guess this works also for MacOS (using the else path) but I have not tested it.

If all else fails, there's always the little-documented extra_compile_args and extra_link_args options to the Extension builder. (See also here.)
You might need to hack in some OS-dependent code to get the right argument format for a particular platform though.

Any possibility that this might work?
g++ -Wl,-Bstatic -lfoo -Wl,-Bdynamic -lbar -Wl,--as-needed