Develop Reference

Calling CPython extension from Go

I currently have a source files written in C++ and wrapped into a Python module (uses Boost and Parser libs).
I ported the C++ folder under go/src and .so files along with main.go
Program Structure
src
/main
main.go
network.so
/network
file1.cpp (this has a function **object DBdata::getTable())
file1.hpp (#define FILE1_H_)
main.go
package main
// #cgo pkg-config: python3
// #cgo CFLAGS : -I./ -I/usr/include/python3.6
// #cgo LDFLAGS: -L/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/usr/lib -lpython3.6 -lpthread -ldl -lutil -lm
// #include <Python.h>
import "C"
import "fmt"
func main() {
cmd := exec.Command("python", "-c", "import network; network.getTable()")
cmd.Dir = "/home/username/go/src/network" //directory where is my python code
out,err := cmd.CombinedOutput()
}
After building this main.go, I get the error as
/usr/include/boost/config/no_tr1/memory.hpp: fatal error: memory: No
such file or directory

>  #  include

>             ^~~~~~~~
> compilation terminated.
How to import .so as Python modules in Go?
Can Swig be used in this place?
What is the better approach to expose Python module to Go?

Issue fixed. The .so placed in /bin and the Go was build and could access the functions under network

Cythonized extension library dependency is incorrect

Checking dependencies with ldd on the *.so file produced by Cython, the dependencies contains myLib.o instead of libforcython.o
I do not grasp why it is trying to reach a myLib.o instead of libforcython as indicated in my setup.py .
During python execution of the module that produces an error similar to Cython unable to find shared object file . However contrary to the included links and answer, my problem does not seem to happen during the python initialization, but rather during the cythonization itself.
using these files:
example.pyx :
cdef extern from "myLib.h":
void testFunction ()
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
pythoncallsc_extension = Extension (
name = "pythoncallsc",
sources=["./example.pyx"],
libraries=["forcython"])
setup ( name = "pythoncallsc",
ext_modules = cythonize ([pythoncallsc_extension]))
When I look at the log generated by python3 setup.py build_ext --inplace I can then clearly see in the commandline launching gcc, and it contains:
... -lforcython -lpython3.7m ...
So clearly gcc is linking against my library libforcython.
The lib contains:
header myLib.h
generated libforcython.so file for the function void testFunction(void).
This lib is built separately and elsewhere in my system. I have checked the include and lib and they are clearly in the $PATH of my cython project.
The cythonization produces the library pythoncallsc.cpython-37m-x86_64-linux-gnu.so
But against all my expectations when I do:
ldd pythoncallsc.cpython-37m-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffee40fb000)
myLib.o => not found <=== HERE ???
libpython3.7m.so.1.0 => /path/to/lib...
libpthread.so.0 => ...
libc.so.6 => ...
...
Why is cython producing an output that depends on a myLib.o file and not on libforcython.so ?
This myLib.o file does not exists, and consequently that produces an error when I launch my module:
`ImportError: myLib.o: cannot open shared object file: No such file or directory`
Any clue ?

In case we have this error, we should check how the c library is built.
As suggested in the comments, the libforcython library was not built properly.
It was wrongly built using soname with gcc in the Makefile.
Correct:
gcc myLib.o -shared -fPIC -Wl,-soname,libforcython.so -o libforcython.so -lc
Incorrect:
gcc -shared -fPIC -Wl,-soname,myLib.o -o libforcython.so myLib.o -lc
In this case soname is not really useful because I don't use a version number for the library: a simpler use case.

Swig wrapping C++ to python is using default Python rather than Anaconda python in Mac OS X

I am trying to wrap c++ functions into python using swig. I am using following commands
swig -c++ -python helloworld.i
g++ -O2 -fPIC -c hello.cpp
g++ -O2 -fPIC -c helloworld_wrap.cxx -I//anaconda/include/python2.7
g++ -lpython -dynamclib hello.o helloworld_wrap.o -o _helloworld.so
with hello.cpp being initial file with functions and helloworld.i being file with wrapper. These commands creates the library helloworldbut I can only import it through default python in /usr/bin/python
If I try to import it through python installed through anaconda it gives following error:
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
Can you tell me how can I wrap the codes with python from anaconda?
Found a solution : Python.h not found using swig and Anaconda Python
In above question, the top answer gives an explanation of using disutilsand a set up file in python to build the library. This works wonders :)
The next problem I am having for wrapping simple class:
my class code from [example] (http://web.mit.edu/svn/src/swig-1.3.25/Examples/python/class/index.html)
/* File : example.h */
class Shape {
public:
Shape() {
nshapes++;
}
virtual ~Shape() {
nshapes--;
};
double x, y;
void move(double dx, double dy);
virtual double area() = 0;
virtual double perimeter() = 0;
static int nshapes;
};
class Circle : public Shape {
private:
double radius;
public:
Circle(double r) : radius(r) { };
virtual double area();
virtual double perimeter();
};
class Square : public Shape {
private:
double width;
public:
Square(double w) : width(w) { };
virtual double area();
virtual double perimeter();
};
My setup.py file :
#setup.py file:
from setuptools import setup, Extension
setup(name='example',
version='0.1',
ext_modules=[Extension('_example', ['example.h', 'example.i'],
swig_opts=['-c++'],
)],
)
Code I am using to wrap :
python setup.py build_ext --inplace
Error message:
running build_ext
building '_example' extension
swigging example.i to example_wrap.cpp
swig -python -c++ -o example_wrap.cpp example.i
error: unknown file type '.h' (from 'example.h')
Can you suggest what is wrong here. I suppose it is not recognizing '.h' file but as it is header file, I thought it could be kept as it is. Also if my setup.py file is correct or not? I am just trying to follow example for simple wrapping, there is no simple tutorial online apparently.
I can also ask this question on other different question but thought just continuing here for now.

Answer by Warren Weckesser in similar question
. I used the setup.py file as suggested in the answer and added the path to the library to sys.path and it work wonders :)
Use the option -I/Users/myuser/anaconda/include/python2.7 in the gcc command. (That's assuming you are using python 2.7. Change the name to match the version of python that you are using.) You can use the command python-config --cflags to get the full set of recommended compilation flags:
$ python-config --cflags
-I/Users/myuser/anaconda/include/python2.7 -I/Users/myuser/anaconda/include/python2.7 -fno-strict-aliasing -I/Users/myuser/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
However, to build the extension module, I recommend using a simple setup script, such as the following setup.py, and let distutils figure out all the compiling and linking options for you.
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example_wrap.c', 'example.c'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
Then you can run:
$ swig -python example.i
$ python setup.py build_ext --inplace
(Take a look at the compiler commands that are echoed to the terminal when setup.py is run.)
distutils knows about SWIG, so instead of including example_wrap.c in the list of source files, you can include example.i, and swig will be run automatically by the setup script:
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_example', sources=['example.c', 'example.i'])
setup(name='example', ext_modules=[example_module], py_modules=["example"])
With the above version of setup.py, you can build the extension module with the single command
$ python setup.py build_ext --inplace
Once you've built the extension module, you should be able to use it in python:
>>> import example
>>> example.fact(5)
120
If you'd rather not use the script setup.py, here's a set of commands that worked for me:
$ swig -python example.i
$ gcc -c -I/Users/myuser/anaconda/include/python2.7 example.c example_wrap.c
$ gcc -bundle -undefined dynamic_lookup -L/Users/myuser/anaconda/lib example.o example_wrap.o -o _example.so
Note: I'm using Mac OS X 10.9.4:
$ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.1 (clang-503.0.40) (based on LLVM 3.4svn)
Target: x86_64-apple-darwin13.3.0
Thread model: posix

Cython module doesn't work

I'm trying to produce a simple fibonacci algorithm with Cython.
I have fib.pyx:
def fib(int n):
cdef int i
cdef double a=0.0, b=1.0
for i in range(n):
a, b = a + b, a
return a
and setup.py in the same folder:
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize('fib.pyx'))
Then I open cmd and cd my way to this folder and build the code with (I have [http://www.microsoft.com/en-us/download/details.aspx?id=44266](this compiler) :
python setup.py build
Which produces this result:
C:\Users\MyUserName\Documents\Python Scripts\Cython>python setup.py build
Compiling fib.pyx because it changed.
Cythonizing fib.pyx
running build
running build_ext
building 'fib' extension
creating build
creating build\temp.win-amd64-2.7
creating build\temp.win-amd64-2.7\Release
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -mdll -O -Wall -IC:\Anaconda\include -IC:
\Anaconda\PC -c fib.c -o build\temp.win-amd64-2.7\Release\fib.o
writing build\temp.win-amd64-2.7\Release\fib.def
creating build\lib.win-amd64-2.7
C:\Anaconda\Scripts\gcc.bat -DMS_WIN64 -shared -s build\temp.win-amd64-2.7\Relea
se\fib.o build\temp.win-amd64-2.7\Release\fib.def -LC:\Anaconda\libs -LC:\Anacon
da\PCbuild\amd64 -lpython27 -lmsvcr90 -o build\lib.win-amd64-2.7\fib.pyd
So it seems the compiling worked and I should be able to import this module with
import fib
ImportError: No module named fib
Where did I go wrong?
Edit:
os.getcwd()
Out[6]: 'C:\\Users\\MyUserName\\Documents\\Python Scripts\\Cython\\build\\temp.win-amd64-2.7\\Release'
In [7]: import fib
Traceback (most recent call last):
File "<ipython-input-7-6c0ab2f0a4e0>", line 1, in <module>
import fib
ImportError: No module named fib

The compiling worked however the library was placed under: build\lib.win-amd64-2.7 so you have to either copy/move the file in the current directory or change your current directory to that one.
You can see this in the last part of the command run:
-o build\lib.win-amd64-2.7\fib.pyd
The -o option stands for output and specifies where it should put the final compiled file.

Use
python setup.py build_ext --inplace
--inplace flag will put your pyd in the working directory

Create a DLL exposing python code

Can I use cython to create a shared library with exported C functions that have python code as the core? Like wrapping Python with C??
It is to be used in plugins.
tk

Using Cython, you can write function declared as C ones with the cdef keyword (and public... important!), with Python inner code:
yourext.pyx
cdef int public func1(unsigned long l, float f):
print(f) # some python code
Note: in the following is assumed that we are working in the root of drive D:\
Building (setup.py)
from distutils.core import setup
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
name = 'My app',
ext_modules = cythonize("yourext.pyx"),
)
Then run python setup.py build_ext --inplace
After running the setup.py (if you are using distutils), you'll get 2 files of interest:
yourext.h
yourext.c
Looking into the .c will show you that func1 is a C function, in the end.
Those two files are all we need to do the rest.
C main program for testing
// test.c
#include "Python.h"
#include "yourext.h"
main()
{
Py_Initialize(); // start python interpreter
inityourext(); // run module yourext
func1(12, 3.0); // Lets use shared library...
Py_Finalize();
}
As we don't use the extension (.pyd) by itself, we need to make a little trick/hack in the header file to disable the "DLL behavior". Add the following at the beginning of "yourext.h":
#undef DL_IMPORT # Undefines DL_IMPORT macro
#define DL_IMPORT(t) t # Redefines it to do nothing...
__PYX_EXTERN_C DL_IMPORT(int) func1(unsigned long, float);
Compiling "yourext" as a shared library
gcc -shared yourext.c -IC:\Python27\include -LC:\Python27\libs -lpython27 -o libyourext.dll
Then compiling our test program (linking to the DLL)
gcc test.c -IC:\Python27\include -LC:\Python27\libs -LD:\ -lpython27 -lyourext -o test.exe
Finally, run the program
$ test
3.0
This is not obvious, and there is many other ways to achieve the same thing, but this works (have a look to boost::python, ..., other solutions may better fit your needs).
I hope this answers a little bit your question or, at least, gave you an idea...