I'm trying to embed Cython code into C following O'reilly Cython book chapter 8. I found this paragraph on Cython's documentation but still don't know what should I do:
If the C code wanting to use these functions is part of more than one shared library or executable, then import_modulename() function needs to be called in each of the shared libraries which use these functions. If you crash with a segmentation fault (SIGSEGV on linux) when calling into one of these api calls, this is likely an indication that the shared library which contains the api call which is generating the segmentation fault does not call the import_modulename() function before the api call which crashes.
I'm running Python 3.4, Cython 0.23 and GCC 5 on OS X. The source code are transcendentals.pyx and main.c:
main.c
#include "transcendentals_api.h"
#include <math.h>
#include <stdio.h>
int main(int argc, char **argv)
{
Py_SetPythonHome(L"/Users/spacegoing/anaconda");
Py_Initialize();
import_transcendentals();
printf("pi**e: %f\n", pow(get_pi(), get_e()));
Py_Finalize();
return 0;
}
transcendentals.pyx
cdef api double get_pi():
return 3.1415926
cdef api double get_e():
print("calling get_e()")
return 2.718281828
I'm compiling those files using setup.py and Makefile:
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules=cythonize([
Extension("transcendentals", ["transcendentals.pyx"])
])
)
Makefile
python-config=/Users/spacegoing/anaconda/bin/python3-config
ldflags:=$(shell $(python-config) --ldflags)
cflags:=$(shell $(python-config) --cflags)
a.out: main.c transcendentals.so
gcc-5 $(cflags) $(ldflags) transcendentals.c main.c
transcendentals.so: setup.py transcendentals.pyx
python setup.py build_ext --inplace
cython transcendentals.pyx
clean:
rm -r a.out a.out.dSYM build transcendentals.[ch] transcendentals.so transcendentals_api.h
However, I came to error Segmentation fault: 11. Any idea can help with this? Thanks!
In that Makefile there is
transcendentals.so: setup.py transcendentals.pyx
python setup.py build_ext --inplace
Unless python refers to /Users/spacegoing/anaconda/bin/python3 it should be replaced since the module may be compiled for wrong python version, and cannot thus be loaded.
In main.c there is call import_transcendentals() that does not check the return value i.e. if the import fails or succeeds. In case of failure, get_pi() and get_e() point to invalid memory locations and trying to call them causes a segmentation fault.
Also, the module has to be located somewhere where it can be found. It seems that when embedding, the current directory is not searched for python modules. PYTHONPATH environment variable could be changed to include the directory where transcendentals.so is located.
The following is an altenative way of embedding the code to the C program and sidesteps the import issues since the module code is linked to the executable.
Essentially, a call to PyInit_transcendentals() is missing.
File transcendentals.h will be generated when the cython functions are defined public i.e.
cdef public api double get_pi():
...
cdef public api double get_e():
Your main.c should have the include directives
#include <Python.h>
#include "transcendentals.h"
and then in main
Py_Initialize();
PyInit_transcendentals();
There should be no #include "transcendentals_api.h" and no import_transcendentals()
The first reason is that according to the documentation
However, note that you should include either modulename.h or
modulename_api.h in a given C file, not both, otherwise you may get
conflicting dual definitions.
The second reason is, that since transcendentals.c is linked to the program in
gcc $(cflags) $(ldflags) transcendentals.c main.c
there is no reason to import transcendentals module. The module has to be initialized though, PyInit_transcendentals() does that for Python 3
Related
I have a simple example python extenstion I want to use from C/C++. The code is as follows
example.pxy:
from numpy import random
cdef public void somefunc():
print(random.randint(500))
setup.py:
from setuptools import setup
cfom Cython.Build import cythonize
import numpy
setup(
ext_modules=cythonize("example.pyx"),
zip_safe=False,
include_dirs=[numpy.get_include()]
)
Running python3 setup.py build_ext --inplace --compiler="mingw32" -DMS_WIN64 then creates example.c, example.h and example.cp310-win_amd64.pyd. The C++ code I am using to call someFunc is:
example.cpp:
#include <Python.h>
#include "example.h"
int main()
{
Py_Initialize();
somefunc();
return 0;
}
This I compile using g++ example.cpp -DMS_WIN64. But that command seems to be incomplete. There are sill objects left that need to be linked, namely the ones from the example.pyx. How do I do this? I do not see any generated .dllor.lib` or similar.
Additionally, if I use #include "exmaple.c" in
example.cpp, I get a very long list of missing symbols from the linker. The objects are all named __im_Py*.
I am using MINGW64 on Windows 10. The python installation I am trying to link against is a regular python installation from the system. I have an environment variable CPLUS_INCLUDE_PATH=C:\Program Files\Python310\include.
I am running Python 3.4.4 :: Anaconda 4.0.0 (x86_64) on OS X Yosemite. My Cython's version is 0.23.4. I'm trying to embed some very trivial Cython code test.pyx into C code testcc.c. The problem is, if I use python2.7-config then everything works well (Python 2.7 is the built-in version on OS X). However if I use python3.4-config the following errors raised:
Undefined symbols for architecture x86_64:
"_inittest", referenced from:
_main in testcc-b22dcf.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
I have to use Python3 since all my other codes are written in it. Please help me solve this problem.
The following are my source files:
test.pyx:
cdef public void pythonAdd(int[] a):
a[1] = 5
a[0] = 4
testcc.c:
#include "Python.h"
#include "test.h"
#include <math.h>
#include <stdio.h>
int main(int argc, char **argv) {
Py_Initialize();
inittest();
int a [2] = {0 , 0};
pythonAdd(a);
printf("fist: %d, second: %d", a[0], a[1]);
Py_Finalize();
return 0;
}
And Compiling those two files using following setup.py:
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("testc", sources=["test.pyx"])
setup(name="testc", ext_modules=cythonize(ext))
The following is the command I compile those c files:
ldflags:=$(shell $(python3.4-config) --ldflags)
cflags:=$(shell $(python3.4-config) --cflags)
python setup.py build_ext --inplace
cython test.pyx
gcc $(cflags) $(ldflags) test.c testcc.c -o cysvm.out
Update:
I changed the inittest() to PyInit_test() as Jim suggested. The code compiles successfully. However when I run ./cysvm.out the following errors occured:
./cysvm.out
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'
Current thread 0x00007fff772f5300 (most recent call first):
Update
I solved this by adding the following line before Py_Initialize(); in my c code:
Py_SetPythonHome(L"/PATH/to/python3");
This is probably due to the fact that in Python 3.x initialization of modules is not performed by calling init<module_name> but rather with PyInit_<module_name> (See PEP 3121). So, if you are linking with Python 3.x and executing via 3.x you need to change the initialization call.
In short, changing the call that initializes the module from:
inittest();
To:
PyInit_test();
and recompiling, should do the trick.
As for your second problem, an alternate solution other than using Py_SetPythonHome is setting PYTHONHOME to the output of python3.4-config --exec-prefix (or sys.exec_prefix) prior to compilation.
As my title kind of says, I'm trying to develop a C extension for Python. I followed this tutorial here and I ran the setup.py script. How ever when I run the python interpreter and try to import my newly created module, I get a linker error undefined symbol: py_BuildValue. I also tryed to compile it my self and I got the same errors plus an error saying Py_InitModule3 is undefined. I have installed both python3.2-dev and python3-dev. Here is my test.c code:
#include <python3.2/Python.h>
static PyObject* Test(PyObject* self){
return py_BuildValue("s","This is a test and my first trip into the world of python bindings!!");
}
static PyMethodDef test_funcs[] ={{"testExtensions",(PyCFunction)Test, METH_NOARGS,"This is my First Extension!!"},{NULL}};
void initTest(void){
Py_InitModule3("Test", test_funcs, "Extension module example!");
}
And my setup.py code:
from distutils.core import setup, Extension
setup(name='Test', version='1.0', \
ext_modules=[Extension('Test', ['test.c'])])
That's because the function is called Py_BuildValue rather than py_BuildValue. C is case sensitive. If you check further up in your compile messages, you probably also have a warning there about the function being implicitly declared.
So I'd like to call some python code from c via cython. I've managed to call cython code from c. And I can also call python code from cython. But when I add it all together, some things are missing.
Here is my python code (quacker.pyx):
def quack():
print "Quack!"
Here is my cython "bridge" (caller.pyx):
from quacker import quack
cdef public void call_quack():
quack()
And here is the c code (main.c):
#include <Python.h>
#include "caller.h"
int main() {
Py_Initialize();
initcaller();
call_quack();
Py_Finalize();
return 0;
}
When I run this I get this exception:
Exception NameError: "name 'quack' is not defined" in 'caller.call_quack' ignored
The missing pieces I'm suspecting:
I haven't called initquacker()
I haven't included quacker.h
Cython didn't produce any quacker.h - only quacker.c
caller.c doesn't import quacker.h or call initquacker()
I'm not really sure that it's even possible to do what I'm trying to do, but it seems to me that it ought to be. I'd love to hear any input you might have.
Edit:
This is how I cythonize / compile / link / run:
$ cython *.pyx
$ cc -c *.c -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
$ cc -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7 -ldl *.o -o main
$ ./main
If you rename the quacker.pyx to quacker.py, everything is actually correct. The only problem is that your program won't search for python modules in the current directory, resulting in the output:
Exception NameError: "name 'quack' is not defined" in 'caller.call_quack' ignored
If you add the current directory to the PYTHONPATH environment variable however, the output becomes the one you'd expect:
$ PYTHONPATH=".:$PYTHONPATH" ./main
Quack!
When running the python shell, according to the documentation the current directory (or the directory containing the script) is added to the sys.path variable automatically, but when creating a simple program using Py_Initialize and Py_Finalize this does not seem to happen. Since the PYTHONPATH variable is also used to populate the sys.path python variable, the workaround above produces the correct result.
Alternatively, below the Py_Intialize line, you could add an empty string to sys.path as follows by just executing some python code, specified as a string:
PyRun_SimpleString("import sys\nsys.path.insert(0,'')");
After recompiling, just running ./main should then work.
Edit
It's actually interesting to see what's going on if you run the code as specified in the question, so without renaming the quacker.pyx file. In that case, the initcaller() function tries to import the quacker module, but since no quacker.py or quacker.pyc exists, the module cannot be found, and the initcaller() function produces an error.
Now, this error is reported the python way, by raising an exception. But the code in the main.c file doesn't check for this. I'm no expert in this, but in my tests adding the following code below initcaller() seemed to work:
if (PyErr_Occurred())
{
PyErr_Print();
return -1;
}
The output of the program then becomes the following:
Traceback (most recent call last):
File "caller.pyx", line 1, in init caller (caller.c:836)
from quacker import quack
ImportError: No module named quacker
By calling the initquacker() function before initcaller(), the module name quacker already gets registered so the import call that's done inside initcaller() will detect that it's already loaded and the call will succeed.
In case there's anyone wondering how would it work in Python 3, here's my solution after struggling a bit as a Cython newbie.
main.c
#include <Python.h>
#include "caller.h"
int
main()
{
PyImport_AppendInittab("caller", PyInit_caller);
Py_Initialize();
PyImport_ImportModule("caller");
call_quack();
Py_Finalize();
return 0;
}
caller.pyx
# cython: language_level=3
import sys
sys.path.insert(0, '')
from quacker import quack
cdef public void call_quack():
quack()
quacker.py
def quack():
print("Quack!")
Finally, here's the Makefile that compiles everything:
target=main
cybridge=caller
CC=gcc
CFLAGS= `python3-config --cflags`
LDFLAGS=`python3-config --ldflags`
all:
cython $(cybridge).pyx
$(CC) $(CFLAGS) -c *.c
$(CC) $(LDFLAGS) *.o -o $(target)
clean:
rm -f $(cybridge).{c,h,o} $(target).o $(target)
rm -rf __pycache__
Maybe this is not what you want but I got it working by the following changes:
in quacker.pyx I added
cdef public int i
To force Cython to generate the .h file.
An then in the main:
#include <Python.h>
#include "caller.h"
#include "quacker.h"
int main() {
Py_Initialize();
initquacker();
initcaller();
call_quack();
Py_Finalize();
return 0;
}
I needed to do this using CMake and ended up recreating this sample. You can find the repository with complete working example here.
You can build and run the example using either Docker on the CLI or a Visual Studio devcontainer.
Can I use cython to create a shared library with exported C functions that have python code as the core? Like wrapping Python with C??
It is to be used in plugins.
tk
Using Cython, you can write function declared as C ones with the cdef keyword (and public... important!), with Python inner code:
yourext.pyx
cdef int public func1(unsigned long l, float f):
print(f) # some python code
Note: in the following is assumed that we are working in the root of drive D:\
Building (setup.py)
from distutils.core import setup
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext': build_ext},
name = 'My app',
ext_modules = cythonize("yourext.pyx"),
)
Then run python setup.py build_ext --inplace
After running the setup.py (if you are using distutils), you'll get 2 files of interest:
yourext.h
yourext.c
Looking into the .c will show you that func1 is a C function, in the end.
Those two files are all we need to do the rest.
C main program for testing
// test.c
#include "Python.h"
#include "yourext.h"
main()
{
Py_Initialize(); // start python interpreter
inityourext(); // run module yourext
func1(12, 3.0); // Lets use shared library...
Py_Finalize();
}
As we don't use the extension (.pyd) by itself, we need to make a little trick/hack in the header file to disable the "DLL behavior". Add the following at the beginning of "yourext.h":
#undef DL_IMPORT # Undefines DL_IMPORT macro
#define DL_IMPORT(t) t # Redefines it to do nothing...
__PYX_EXTERN_C DL_IMPORT(int) func1(unsigned long, float);
Compiling "yourext" as a shared library
gcc -shared yourext.c -IC:\Python27\include -LC:\Python27\libs -lpython27 -o libyourext.dll
Then compiling our test program (linking to the DLL)
gcc test.c -IC:\Python27\include -LC:\Python27\libs -LD:\ -lpython27 -lyourext -o test.exe
Finally, run the program
$ test
3.0
This is not obvious, and there is many other ways to achieve the same thing, but this works (have a look to boost::python, ..., other solutions may better fit your needs).
I hope this answers a little bit your question or, at least, gave you an idea...