This question is based on this one, but with a twist. I want to compile a python function using cython, then call it in C. In this simple case, it works, but when I try to import another python module from my python code, it doesn't work.
world.pyx
def world():
return 10
hello.pyx
import world
def hello():
ret = 5 + world()
return ret
cdef public int call_hello():
return hello()
main.c
#include <Python.h>
#include "hello.h"
int main() {
Py_Initialize();
PyInit_hello();
int v = call_hello();
printf("the value is %d\n",v);
Py_Finalize();
return 0;
}
cython -3 --force world.pyx hello.pyx
clang -c world.c hello.c main.c -I/Users/aneben/.pyenv/versions/3.6.1/Python.framework/Versions/3.6/include/python3.6m -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
clang -L/Users/aneben/.pyenv/versions/3.6.1/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin -lpython3.6m -ldl -framework CoreFoundation world.o hello.o main.o -o main
If I simply move the world function into hello.pyx, then it works fine. But it doesn't work as written when I import it.
./main
NameError: name 'hello' is not defined
Exception ignored in: 'hello.call_hello'
import world imports the module, not the function.
In order to call world() from a different module, you must use world.world()
One solution seems to be to add a call to PyInit_world(); in the main function. Cython adds this function to world.c, even though in this case it does not generate a header file, so we get a warning, but it works.
Is there a way to have hello.c call PyInit_world() automatically, so that main.c doesn't need to know about that? In that way, the C include hierarchy would mirror the python one.
Related
QUESTION
When creating a Python extension in C++ that uses Armadillo, I get the errors:
A) In Mac OS Mojave 10.14.4, Python 3.7.5:
Traceback (most recent call last):
File "./py_program.py", line 5, in <module>
import cmodule
ImportError: dlopen(/Users/angel/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cmodule.cpython-37m-darwin.so, 2): Symbol not found: __ZTWN4arma23arma_rng_cxx11_instanceE
Referenced from: /Users/angel/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cmodule.cpython-37m-darwin.so
Expected in: flat namespace in /Users/angel/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cmodule.cpython-37m-darwin.so
B) In Ubuntu 20, Python 3.8.2:
Traceback (most recent call last):
File "./py_program.py", line 5, in <module>
import cmodule
ImportError: /usr/local/lib/python3.8/dist-packages/cmodule.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN4arma23arma_rng_cxx11_instanceE
Both of them due to the use of arma::arma_rng::randn<double>(), see below.
How can I fix it?
DETAILS
I want py_program.py to import the C++ module (extension) defined in cmodule.cpp.
Following the documentation https://docs.python.org/3/extending/extending.html , I have the files py_program.py, setup.py and cmodule.cpp.
Their contents are:
py_program.py
#!/usr/bin/env python3
"""Import and use cmodule."""
import cmodule
cmodule.printvol(3.)
setup.py
"""To install the module defined in cmodule.cpp."""
from distutils.core import setup, Extension
setup(name='cmodule', version='1.0', \
ext_modules=[
Extension(
'cmodule', ['cmodule.cpp'],
extra_compile_args=['-std=c++11'],
language='c++')],
)
cmodule.cpp
/* Module to be used in Python.
All Python stuff follows Sec. 1.8 of
https://docs.python.org/3/extending/extending.html */
#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <armadillo>
double f()
// Fails at using Armadillo.
// The module works if I delete this function.
{
double rn_y = arma::arma_rng::randn<double>();
return rn_y;
}
arma::cx_double g()
// Succeeds at using Armadillo.
{
arma::cx_double value(0., 1.);
return value;
}
static PyObject *
cmodule_printvol(PyObject *self, PyObject *args)
// A method of the module.
{
double voltage;
if (!PyArg_ParseTuple(args, "d", &voltage))
return NULL;
printf("voltage is %f.\n", voltage);
Py_RETURN_NONE;
}
static PyMethodDef cmodule_methods[] = {
// Declare the modules methods.
{"printvol", cmodule_printvol, METH_VARARGS, "Print voltage."},
{NULL, NULL, 0, NULL} /* sentinel */
};
static struct PyModuleDef cmodule = {
// Create the module.
PyModuleDef_HEAD_INIT,
"diff",
NULL,
-1,
cmodule_methods
};
PyMODINIT_FUNC
PyInit_cmodule(void)
// Initialize the module.
{
return PyModule_Create(&cmodule);
}
I run them as follows:
python setup.py install
python py_program.py
In Ubuntu, the output of python3 setup.py install is:
running install
running build
running build_ext
building 'cmodule' extension
creating build
creating build/temp.linux-x86_64-3.8
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c cmodule.cpp -o build/temp.linux-x86_64-3.8/cmodule.o -std=c++11
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/cmodule.o -o build/lib.linux-x86_64-3.8/cmodule.cpython-38-x86_64-linux-gnu.so
running install_lib
copying build/lib.linux-x86_64-3.8/cmodule.cpython-38-x86_64-linux-gnu.so -> /usr/local/lib/python3.8/dist-packages
running install_egg_info
Removing /usr/local/lib/python3.8/dist-packages/cmodule-1.0.egg-info
Writing /usr/local/lib/python3.8/dist-packages/cmodule-1.0.egg-info
What causes the problem is
double rn_y = arma::arma_rng::randn<double>();
Actually, if I delete the function f(), I get no error.
Notice that Armadillo is loaded successfully, since g() uses it no problem.
What is happening?
In setup.py, the argument libraries=['armadillo'] to Extension() fixes the problem:
"""To install the module defined in cmodule.cpp."""
from distutils.core import setup, Extension
setup(name='cmodule', version='1.0', \
ext_modules=[
Extension(
'cmodule', ['cmodule.cpp'],
extra_compile_args=['-std=c++11'],
libraries=['armadillo'], // this solves the problem
language='c++')],
)
Mysteriously, without it, arma:: can be used correctly. But not 'submodules' like arma::arma_rng.
This solution is general: the same problem happens with other libraries. Actually, I reproduced the same (and made it work) with the GNU Scientific Library (libraries=['gsl']).
I recently created a 3D and 2D force layout diagram visualizer in C++ using the OpenGL library (it uses some physics, sort of). Can someone give me some introductory pointers about making this useful as a Python library (issues or considerations and potential pitfalls I might run into)?
If I understand your question correctly you want to know how to write C extensions for use in Python. Here's a trivial simple example of how:
hello.c:
#include <Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] =
"helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld,
METH_NOARGS, helloworld_docs},
{NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,
"Extension module example!");
}
setup.py:
#!/usr/bin/env python
from setuptools import setup, Extension
setup(
name='helloworld',
version='1.0',
ext_modules=[
Extension('helloworld', ['hello.c'])
]
)
Build:
$ python setup.py develop
running develop
running egg_info
creating helloworld.egg-info
writing helloworld.egg-info/PKG-INFO
writing top-level names to helloworld.egg-info/top_level.txt
writing dependency_links to helloworld.egg-info/dependency_links.txt
writing manifest file 'helloworld.egg-info/SOURCES.txt'
reading manifest file 'helloworld.egg-info/SOURCES.txt'
writing manifest file 'helloworld.egg-info/SOURCES.txt'
running build_ext
building 'helloworld' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c hello.c -o build/temp.linux-x86_64-2.7/hello.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/hello.o -o build/lib.linux-x86_64-2.7/helloworld.so
copying build/lib.linux-x86_64-2.7/helloworld.so ->
Creating /home/prologic/.virtualenvs/hellopyc/lib/python2.7/site-packages/helloworld.egg-link (link to .)
Adding helloworld 1.0 to easy-install.pth file
Installed /home/prologic/tmp/hello-py-c
Processing dependencies for helloworld==1.0
Finished processing dependencies for helloworld==1.0
Test:
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import helloworld
>>> helloworld.helloworld()
'Hello, Python extensions!!'
>>>
Good luck!
While James Mills answer is fine, i would consider other option - use Boost.Python library to write python extensions. Here is a basic example. It looks like Boost.Python is not maintained (last update - 2009), but still it's a very good option. Right now i'm using it in one project and i've got to say that after a little difficult start, now it's really usefull and quite easy. Most important for me is that it handles reference counting(which is painful in Python/C API), allows to modify python objects inside c++ code and allows you to register converters for custom datatypes. So instead of writing:
MyClass test = convertPythonObjectToMyClass(pythonObject);
MyOtherClass test2 = convertPythonObjectMyOtherClass(pythonObject);
you can register your converters and than use this:
MyClass test = boost::python::extract<MyClass>(pythonObject);
MyOtherClass test2 = boost::python::extract<MyOtherClass>(pythonObject);
Of course you can consider other options as well - see this questions for more information about other solutions.
There are like tens of similar questions on StackOverflow, but after several hours of lurking I finally gave up.
So I'm trying to write a C extension for Python. Let's call it mylib. Here is the header file:
mylib.h
#ifndef mylib_H
#define mylib_H
#include <Python.h>
< ... >
#include <glib.h>
< ... >
and setup.py:
from distutils.core import setup, Extension
include_list = [
"/usr/include/glib-2.0", "-lglib-2.0",
"/usr/lib/x86_64-linux-gnu/glib-2.0/include"
]
module = Extension('mylib', ['mylib.c'])
setup(name='mylib', version='1.0',
include_dirs=include_list,
ext_modules=[module])
If I run python setup.py install, I get the following (which I take as successful installation):
running install
running build
running build_ext
building 'mylib' extension
creating build
creating build/temp.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/glib-2.0 -I-lglib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/python2.7 -c mylib.c -o build/temp.linux-x86_64-2.7/mylib.o
mylib.c: In function ‘c_sound_utf8’:
mylib.c:117:5: warning: ‘g_unicode_canonical_decomposition’ is deprecated (declared at /usr/include/glib-2.0/glib/gunicode.h:627) [-Wdeprecated-declarations]
decomposition = g_unicode_canonical_decomposition(c_composed, &decomposition_len);
^
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/mylib.o -o build/lib.linux-x86_64-2.7/mylib.so
running install_lib
copying build/lib.linux-x86_64-2.7/mylib.so -> /usr/local/lib/python2.7/dist-packages
running install_egg_info
Removing /usr/local/lib/python2.7/dist-packages/mylib-1.0.egg-info
Writing /usr/local/lib/python2.7/dist-packages/mylib-1.0.egg-info
But when I try to use mylib from inside Python, I get the following:
>>> import mylib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /usr/local/lib/python2.7/dist-packages/mylib.so: undefined symbol: g_utf8_skip
After rambling around StackOverflow for some time I got an idea that I should either 1. rebuild the needed library or 2. put all the links to needed library after all generated module names.
Rebuilding didn't work (or I did it the wrong way). As for placing links to the needed library after everything else - well, I didn't find out the way to make distutils change the order of links in its compile string. Is there a way?
I also tried providing extra_link_args/extra_compile_args to my extension (without any effect):
module = Extension('mylib', ['mylib.c'],
extra_link_args=["-Xlinker", "-export-dynamic"])
I felt pretty miserable and kept googling on. Then I found out about SWIG. I decided to try it by making another library, (uppercase) MYLIB (I changed filenames and all text occurences of mylib to MYLIB). I wrote a shell script:
#!/bin/bash
GLIB_IMPORT_OPTS="-I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -lglib-2.0"
PY_IMPORT_OPTS="-I/usr/include/python2.7/ -lpython2.7"
swig -Wall -python MYLIB.i
gcc -fPIC -Wall -c MYLIB.c $GLIB_IMPORT_OPTS
gcc -fPIC -Wall -shared MYLIB.o MYLIB_wrap.c -o _MYLIB.so $GLIB_IMPORT_OPTS -L. $PY_IMPORT_OPTS $GLIB_IMPORT_OPTS
When I ran this thing, everything worked fine (I could import the library and do stuff with it). Here, as you can see, links are at the very end of the compile line. So now I'm trying to understand: what did I miss with the distutils way? How can I make it work?
Well, actually I found the solution. A had to add library links to extra_link_args:
extra_link_args=["-I", "/usr/include/glib-2.0", "-l", "glib-2.0", "-I", "/usr/lib/x86_64-linux-gnu/glib-2.0/include"]
which appends them to the end of compile string.
I found adding -fPIC to "extra_compile_args" in the Extension constructor also helped. Like so:
my_module = Extension('modulename',
...
extra_compile_args=["-fPIC"]
sources = ['mycode.c'])
I am working on a python/fortran interface using f2py.
My fortran code has some subroutines whose list of arguments are longer than one line, so I have used the standard fortran rule to break a line, i.e.:
SUBROUTINE mutation(it,pop,pm,pmg,typem,xmin,xmax,newfx,nbvar,
$popsize,tip,nouvpop,nbnew)
When I try to build the python module with f2py, using the command :
f2py3 -c forFunct.f -m mga
I get this :
...
Block: unknown_subroutine
...
Constructing wrapper function "unknown_subroutine"...
unknown_subroutine()
and eventually
error: Command "gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-
prototypes -fPIC -I/tmp/tmp1tw75l/src.linux-x86_64-3.3
-I/Produits/publics/x86_64.Linux.RH6/python/3.3.3/lib/python3.3/site-
packages/numpy/core/include -
I/Produits/publics/x86_64.Linux.RH6/python/3.3.3/include/python3.3m -c
/tmp/tmp1tw75l/src.linux-x86_64-3.3/mgamodule.c -o
/tmp/tmp1tw75l/tmp/tmp1tw75l/src.linux-x86_64-3.3/mgamodule.o"
failed with exit status 1
Do you know any solution to this problem? I am sure that it comes from the multiple lines arguments, is it supported by f2py?
Thank you
Francesco
How exactly do you break the lines? It is not clear from your snippet, but it looks like it is wrong. If I do it in the real standard way, with positioning any character in the 6th column, it works.
1234567
SUBROUTINE mutation(it,pop,pm,pmg,typem,xmin,xmax,newfx,nbvar,
$ popsize,tip,nouvpop,nbnew)
END
I'm trying to call some python code from c and I'm trying to follow the cython user guide on how to do that.
I've got a cython script, modulename.pyx that looks like this (taken from the user guide):
cdef public struct Bunny: # public type declaration
int vorpalness
cdef public int spam # public variable declaration
cdef public void grail(Bunny b): # public function declaration
print "Ready the holy hand grenade"
And I've got a c program, main.c, that looks like this (also from the guide):
#include <Python.h>
#include "modulename.h"
int main() {
Py_Initialize();
initmodulename();
struct Bunny b;
grail(b);
Py_Finalize();
}
I've tried to get this to compile in various ways with no luck. I can get the .pyx file compiled to c well enough with cython modulename.pyx, which produces modulename.c and modulename.h. The trouble comes when I want to compile and link those generated files along with my c program.
Here is one example of what I've been trying:
cc -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c main.c modulename.h modulename.c
Which gave me these errors:
modulename.h:25:32: error: expected function body after function declarator
__PYX_EXTERN_C DL_IMPORT(void) grail(struct Bunny);
^
modulename.h:27:31: error: expected function body after function declarator
__PYX_EXTERN_C DL_IMPORT(int) spam;
^
modulename.h:32:1: error: unknown type name 'PyMODINIT_FUNC'
PyMODINIT_FUNC initmodulename(void);
^
3 errors generated.
I guess what I'm looking for is some guidance on getting the example to work or perhaps a pointer towards some example code that calls python code from c with cython.
Alright, so I've figured out the issue. The problem was that I included modulename.h in the argument for the compiler. After I removed that, everyhing ran without any modification to either main.c or modulename.pyx. Also, not all those arguments were necessary for the compiler. The simplest way to compile the above code on my machine (Mac OS 10.9) is:
Compile:
cc -c main.c modulename.c -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
Link:
cc -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl main.o modulename.o -o main