Develop Reference

How do I import a module created with pybind11 on Ubuntu

I am trying to setup a CMake project that creates python bindings for its c++ functions using pybind11 on Ubuntu.
The directory structure is:
pybind_test
arithmetic.cpp
arithmetic.h
bindings.h
CMakeLists.txt
main.cpp
pybind11 (github repo clone)
Repo contents (https://github.com/pybind/pybind11)
The CMakeLists.txt file:
cmake_minimum_required(VERSION 3.10)
project(pybind_test)
set(CMAKE_CXX_STANDARD 17)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(pybind11/include/pybind11)
add_executable(pybind_test main.cpp arithmetic.cpp)
add_subdirectory(pybind11)
pybind11_add_module(arithmetic arithmetic.cpp)
target_link_libraries(pybind_test ${PYTHON_LIBRARIES})
The repository builds successfully and the file arithmetic.cpython-36m-x86_64-linux-gnu.so is produced. How do I import this shared object file into python?
The documentation in the pybind11 docs has this line
$ c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` example.cpp -o example`python3-config --extension-suffix`
but I want to build using CMake and I also don't want to have to specify extra include directories every time I run python to use this module.
How would I import this shared object file into python like a normal python module?
I am using Ubuntu 16.04.

If you open a terminal, go to the directory where arithmetic.cpython-36m-x86_64-linux-gnu.so is located and run python followed by import arithmetic the module will get imported just like any other module.
Another options is to use the method of
import sys
sys.path.insert(0, 'path/to/directory/where/so-file/is')
import arithmetic
With this method you can use both relative and absolute path.

Besides the solution of setting the path in the Python script that is presented by #super, you have two more generic solutions.
Setting PYTHONPATH
There is an environment variable in Linux (and macOS) called PYTHONPATH. If you add the path that contains your *.so to the PYTHONPATH before you call Python, Python will be able to find your library.
To do this:
export PYTHONPATH="/path/that/contains/your/so":"${PYTHONPATH}"
To apply this 'automatically' for every session you can add this line to ~/.bash_profile or ~/.bashrc (see the same reference). In that case, Python will always be able to find your library.
Copying your to a path already in Python's path
You can also 'install' the library. The usual way to do this is to create a setup.py file. If set up correctly you can build and install your library using
python setup.py build
python setup.py install
(Python will know where to put your library. You can 'customize' a bit with an option like --user to use your home-folder, but this doesn't seems to be of particular interest to you.)
The question remains: How to write setup.py? For your case you can actually call CMake. In fact there exists an example that does exactly that: pybind/cmake_example. You can basically copy-paste from there.

Python.h compiling error: collect2.exe Id returned 1 exit status

So, I've searched all around the web and I couldn't find anything.
I'm started with python, and I made a TKinter application. At first, I tried to compile it to an .exe file, but that didn't work. So now I'm embedding python in C++. Every single time I try to compile it (using Dev-C++), I get the error:
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x10): undefined reference to `__imp_Py_Initialize'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x25): undefined reference to `__imp_PyRun_SimpleStringFlags'
C:\Users\*****\AppData\Local\Temp\ccsqSJ5V.o [program-name].cpp:(.text+0x2e): undefined reference to `__imp_Py_Finalize'
F:\Documents\Videos\[program-name]\program\collect2.exe [Error] ld returned 1 exit status
F: stands for the flash drive.
In the Dev-c++ options, I added:
-Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include
So this is the full command:
g++.exe "F:\Documents\Videos\[program-name]\program\[program-name].cpp" -o "F:\Documents\Videos\[program-name]\program\[program-name].exe" -Wall -I\C:\Users\*****\AppData\Local\Programs\Python\Python35\include -I"C:\Programma's\Dev-C++\MinGW64\include" -I"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include" -I"C:\Programma's\Dev-C++\MinGW64\lib\gcc\x86_64-w64-mingw32\4.9.2\include\c++" -I"C:\Users\*****\AppData\Local\Programs\Python\Python35\include" -L"C:\Programma's\Dev-C++\MinGW64\lib" -L"C:\Programma's\Dev-C++\MinGW64\x86_64-w64-mingw32\lib" -static-libgcc
My question is: how do I deal with this?
And please note: I haven't got administrator privileges, I'm just a standard user.
Python version: 3.5.1
Dev-C++ version: 5.11
GCC version: I don't know, but I'll find it out if needed :)
EDIT: Because of the duplicate mark by NathanOliver, I'll ask it like this: which command line variables do I have to add to compile it?

The linker is complaining bacause it can't find a group of references about some python stuff, as a matter of fact I don't see any python library in the g++ invocation.
You need to had something like this -lpython3.5.1 to tell g++ to link against the python library (assuming python is installed in your system, else you will have to add the path to the library using the -L option).
More resources on g++ makefiles and linking:
https://www3.ntu.edu.sg/home/ehchua/programming/cpp/gcc_make.html

How to remove a compiler option that python disutils includes (language code=pic32 not recognized)

I'm trying to compile the msgpack-python python module with gcc (v4.7) on solaris 10. The python installed is 2.6.8. Distutils is automatically picking up a incorrect compiler option (-xcode=pic32) that I want to remove from command.
The full command that distutils is putting together is:
/opt/csw/bin/gcc-4.7 -DNDEBUG -O -O2 -pipe -mcpu=v9 -I/opt/csw/include -xcode=pic32 -I/opt/csw/include/python2.6 -c msgpack/_msgpack.c -o build/temp.solaris-2.10-sun4v-2.6/msgpack/_msgpack.o
but produces this error:
gcc-4.7: error: language code=pic32 not recognized
then fails. If I remove that -xcode=pic32 option and manually execute the above command the module compiles successfully.
I need to be able to do this in an automated fashion though (using a buildfarm to produce the packages). The question is, Without modifying or changing the current python or distutils, is there a way to "remove" this option that distutils is picking up, so I can have the python setup.py process build the module appropriately (i.e. without the pic32 option)?
Thanks

Do not compile with that gcc. -xcode=pic32 is Sun Studio complier command line parameter. It will lead to linking problems too, even if you compile OK. Compile with SUN CoolTools gcc which can understand such parameter, or use Oracle Solaris Studio for SPARC.
Some hints:
GCC produce very slow code for SPARC, that's for why SUN created Cool Tools.
You haven't to remove -xcode=pic32, but change for -m32 -fpic, when you insist on gcc-4.7
To get mature setup of OSS tool I'm using pkgsrc compiling with Studio Express to particular CPU (-xtarget=native)

You may also find luck by setting the follow env vars:
export CC=$gcc_dir_path # Example: /usr/bin/gcc
export CXX=$gxx_dir_path # Example: /usr/bin/g++
export CFLAGS=''
export CPPFLAGS=''
export CXXFLAGS=''
export LDFLAGS=''
Note: There is a difference between unset env var, and set-as-empty env var. I had build bugs with Python packages when my *FLAGS env vars were unset. (Calling gcc with option -xO2 was the cause.) Setting as empty did the trick.

How can I build my C extensions with MinGW-w64 in Python?

So I have a few Python C extensions I have previously built for and used in 32 bit Python running in Win7. I have now however switched to 64 bit Python, and I am having issues building the C extension with MinGW-w64.
I made the changes to distutils as per this post, but I am getting some weird errors suggesting something is wrong:
$ python setup.py build
running build
running build_ext
building 'MyLib' extension
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -mdll -O -Wall -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c MyLib.c -o build\temp.win-amd64-2.7\Release\mylib.o
MyLib.c: In function 'initMyLib':
MyLib.c:631:5: warning: implicit declaration of function 'Py_InitModule4_64' [-Wimplicit-function-declaration]
writing build\temp.win-amd64-2.7\Release\MyLib.def
c:\MinGW64\bin\x86_64-w64-mingw32-gcc.exe -shared -s build\temp.win-amd64-2.7\Release\mylib.o build\temp.win-amd64-2.7\Release\MyLib.def -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -lpython27 -o build\lib.win-amd64-2.7\MyLib.pyd
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x13d): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1275): undefined reference to `__imp_PyExc_ValueError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1eef): undefined reference to `__imp_PyExc_ImportError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f38): undefined reference to `__imp_PyExc_AttributeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f4d): undefined reference to `__imp_PyCObject_Type'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1f61): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1fc7): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x1ffe): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x2042): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x206c): undefined reference to `__imp_PyExc_RuntimeError'
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x208a): more undefined references to `__imp_PyExc_RuntimeError' follow
build\temp.win-amd64-2.7\Release\mylib.o:MyLib.c:(.text+0x20a7): undefined reference to `__imp_PyExc_ImportError'
collect2.exe: error: ld returned 1 exit status
error: command 'x86_64-w64-mingw32-gcc' failed with exit status 1
I have googled around quite a bit to find information, but it's not easy to find a definite answer. Could someone shed some light on this? What further changes should I do to be able to successfully build C extensions for 64 bit Python in Win7?
EDIT:
After some helpful pointers in cgohlke's comments below I managed to generate libpython27.a. However after following the advice on this post (2nd to last) I still had a the __imp_Py_InitModule4_64 error. After some serious Google-fu I managed to trip over this post telling me to rename the Py_InitModule4 line to Py_InitModule4_64. After that everything worked swimmingly.

This worked for me with Python 3.3 :
create static python lib from dll
python dll is usually in C:/Windows/System32; in msys shell:
gendef.exe python33.dll
dlltool.exe --dllname python33.dll --def python33.def --output-lib libpython33.a
mv libpython33.a C:/Python33/libs
use swig to generate wrappers
e.g., swig -c++ -python myExtension.i
wrapper MUST be compiled with MS_WIN64, or your computer will crash when you import the class in Python
g++ -c myExtension.cpp -I/other/includes
g++ -DMS_WIN64 -c myExtension_wrap.cxx -IC:/Python33/include
shared library
g++ -shared -o _myExtension.pyd myExtension.o myExtension_wrap.o -lPython33 -lOtherSharedLibs -LC:/Python33/libs -LC:/path/to/other/shared/libs
make sure all shared libs (gdal, OtherSharedLibs) are in your PATH
(windows does not use LD_LIBRARY_PATH or PYTHONPATH)
in Python, just: import myExtension
voila!

I realize this is an old question, but it is still the top search result. Today, in 2019, I was able to do this:
https://github.com/PetterS/quickjs/commit/67bc2428b8c0716538b4583f4f2b0a2a5a49106c
In short:
Make sure a 64-bit version of mingw-w64 is in the PATH.
Monkey-patch distutils:
import distutils.cygwinccompiler
distutils.cygwinccompiler.get_msvcr = lambda: []
Some differences in the shell w.r.t. escaping.
extra_link_args = ["-Wl,-Bstatic", "-lpthread"] in order to link statically and not have extra runtime deps.
pipenv run python setup.py build -c mingw32 now works.

Here is a example code for VC++ Build Tools
https://github.com/starnight/python-c-extension/tree/master/00-HelloWorld
You could try:
python setup.py -c mingw32
However this is not work for me.
My Solution is:
install Anaconda 64bit python 3.6
install mingw64
add mingw64/bin to PATH
compile dll from c file by
gcc -c libmypy.c -IC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\include
gcc -shared -o libmypy.dll libmypy.o -LC:\Users\{user_name}\Anaconda3\pkgs\python-3.6.4-h6538335_1\libs -lPython36
load dll file in .py script
from ctypes import *
m = cdll.LoadLibrary(r"C:\{path_to_dll}\libmypy.dll")
print(m.hello())

I created a monkey-patch for setuptools to let you to build_ext with mingw64 on Windows easily. See https://github.com/imba-tjd/mingw64ccompiler

I used this thread to wade through learning how to make a C extension, and since most of what I learned is in it, I thought I'd put the final discovery here too, so that someone else can find it if they are looking.
I wasn't trying to compile something big, just the example in Hetland's Beginning Python. Here is what I did (the example C pgm is called palindrome.c). I'm using Anaconda with python 3.7 in it, and the TDM-GCC version of MinGW64. I put all of the tools used into my Path, and all of the paths needed in PYTHONPATH, and the ..\Anaconda3 directory into PYTHON_HOME. I still ended up using explicit paths on some things.
I created the libpython37.a library with gendef.exe and dlltool.exe as Mark said above, and put it in ..\Anaconda3\libs.
I followed the prescription in Hetland:
gcc -c palindrome.c
gcc -I$PYTHON_HOME -I$PYTHON_HOME/Include -c palindrome_wrap.c
The second failed, the compiler couldn't find Python.h, the following worked:
gcc -I[somedirectories]\Anaconda3\Include -c palindrome_wrap.c
I then did, as many have said, including Hetland 3rd ed.,
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.dll
This did not work. Even with the Load Library cswu used (which I found elsewhere, too).
So I gendef'd _palindrome.dll and couldn't find the function in it, "is_palindrome" in the exports. I went through some of the SWIG documentation, and declared the function both in the %{ %} section and below it, both extern, that finally got the function extern'd in palindrome_wrap.c as it should have been. But no export, so I went back into palindrome.c and redeclared the function as:
declspec(dllexport) extern int __stdcall is_palindrome(char* text)
and redeclared it in palindrome.i in both places as above with this signature.
Partial success! It got listed in the Export section when I gendef'd _palindrome.dll and I could do cswu's call using Load Library. But still not do what Hetland says and do
import _palindrome
in Python.
Going back to all the sources again, I could not figure this out. I finally started reading the SWIG documentation from the beginning leaving no stone unturned -- Searching through the manual doesn't produce the place found.
At the end of Introduction Sec. 2.7 Incorporating Into a Build System, under the sample Make process, it says:
"The above example will generate native build files such as makefiles, nmake files and Visual Studio projects which will invoke SWIG and compile the generated C++ files into _example.so (UNIX) or _example.pyd (Windows). For other target languages on Windows a dll, instead of a .pyd file, is usually generated."
And that's the answer to the last problem:
The compile step for the dll should read:
gcc -shared palindrome.o palindrome_wrap.o [somedirectories]/Anaconda3/libs/libpython37.a -o _palindrome.pyd
(I didn't go back and change out my declspec declarations so I don't know whether they were necessary, so they were still there too).
I got a file, _palindrome.pyd
Which if in the PYTHONPATH (mine was local) works, and one can then do
import _palindrome
from _palindrome import is_palindrome
and use the exported, properly wrapped and packaged C function, compiled with TDM-GCC, in python as promised. gcc, which is MinGW64 in a different installation, knows how to do the .pyd file. I diffed the dll and pyd since they were the same byte length. They are not the same at hundreds of points.
Hope this helps someone else.

How to statically link a library when compiling a python module extension

I would like to modify a setup.py file such that the command "python setup.py build" compiles a C-based extension module that is statically (rather than dynamically) linked to a library.
The extension is currently dynamically linked to a number of libraries. I would like to leave everything unchanged except for statically linking to just one library. I have successfully done this by manually modifying the call to gcc that distutils runs, although it required that I explicitly listed the dependent libraries.
Perhaps this is too much information, but for clarity this is the final linking command that was executed during the "python setup.py build" script:
gcc -pthread -shared -L/system/lib64 -L/system/lib/ -I/system/include build/temp.linux-x86_64-2.7/src/*.o -L/system/lib -L/usr/local/lib -L/usr/lib -ligraph -o build/lib.linux-x86_64-2.7/igraph/core.so
And this is my manual modification:
gcc -pthread -shared -L/system/lib64 -L/system/lib/ -I/system/include build/temp.linux-x86_64-2.7/src/*.o -L/system/lib -L/usr/local/lib -L/usr/lib /system/lib/libigraph.a -lxml2 -lz -lgmp -lstdc++ -lm -ldl -o build/lib.linux-x86_64-2.7/igraph/core.so
Section 2.3.4 of Distributing Python Modules discusses the specification of libraries, but only "library_dirs" is appropriate and those libraries are dynamically linked.
I'm using a Linux environment for development but the package will also be compiled and installed on Windows, so a portable solution is what I'm after.
Can someone tell me where to look for instructions, or how to modify the setup.py script? (Thanks in advance!)
I'm new to StackOverflow, so my apologies if I haven't correctly tagged this question, or if I have made some other error in this posting.

6 - 7 years later, static linking with Python extensions is still poorly documented. As the OP points out in a comment, the usage is OS dependend.
On Linux / Unix
Static libraries are linked just as object files and should go with the path and its extension into extra_objects.
On Windows
The compiler sees if the linked library is static or dynamic and the static library name goes to the libraries list and the directories to library_dir
Solution for both platforms
For the example below, I will use the same library scenario as OP, linking igraph static and z, xml2 and gmp dynamic. This solution is a bit hackish, but at least does for each platform the right thing.
static_libraries = ['igraph']
static_lib_dir = '/system/lib'
libraries = ['z', 'xml2', 'gmp']
library_dirs = ['/system/lib', '/system/lib64']
if sys.platform == 'win32':
libraries.extend(static_libraries)
library_dirs.append(static_lib_dir)
extra_objects = []
else: # POSIX
extra_objects = ['{}/lib{}.a'.format(static_lib_dir, l) for l in static_libraries]
ext = Extension('igraph.core',
sources=source_file_list,
libraries=libraries,
library_dirs=library_dirs,
include_dirs=include_dirs,
extra_objects=extra_objects)
On MacOS
I guess this works also for MacOS (using the else path) but I have not tested it.

If all else fails, there's always the little-documented extra_compile_args and extra_link_args options to the Extension builder. (See also here.)
You might need to hack in some OS-dependent code to get the right argument format for a particular platform though.

Any possibility that this might work?
g++ -Wl,-Bstatic -lfoo -Wl,-Bdynamic -lbar -Wl,--as-needed