I have a working cpp project which builds fine with cmake. Now I need to create a Python wrapper for the same. So I chose cython to bridge the gap between C++ and Python.
Instead of writing the CMakeLists.txt logic in pyx file, I tried compiling the library with cmake itself and later wrap the Cython wrapper around it. Following is the setup.py file:
import os
import sys
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
try:
build_dir = os.path.join(os.getcwd(), "build")
print("building bobcore.a")
if not os.path.exists(build_dir):
os.makedirs(build_dir)
if not os.path.exists(build_dir):
raise Exception("Not able to create `build` directory")
os.chdir(build_dir)
assert os.system("cmake ..") == 0
assert os.system("make -j4") == 0
os.chdir(os.path.abspath(os.path.join(os.getcwd(), os.pardir)))
except:
if not os.path.exists("build/bobcli"):
print("Error building external library, please create libsample.a manually.")
sys.exit(1)
_ext_modules = cythonize([
Extension("call_core",
sources=["call_core.pyx"],
language="c++",
extra_compile_args=["-std=c++11", "-lpthread", "-lz", "-ldl"],
extra_link_args=["-std=c++11"],
include_dirs=[os.path.join(os.getcwd(), "src")], # path to .h file(s)
library_dirs=[os.path.join(os.getcwd(), "build")], # path to .a or .so file(s)
libraries=["bobcli"] # name of pre-built .so file.
)
])
setup(
name='Demos',
ext_modules=_ext_modules,
)
Here is the call_core.pyx file:
cdef extern from "src/api/Api.h" namespace "cli":
cdef cppclass Api:
int getVal()
cdef Api *api = new Api()
def getHelp():
return api.getVal()
del api
Now this file works fine if I implement the getVal() method in the header file itself. But as soon as I move the implementation to .cpp file, Cython compiler shows the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: /usr/local/lib/python2.7/dist-packages/call_core.so: undefined symbol: _ZN3cli9Api6getValEv
NOTE: The above snippets work perfectly for the functions with their implementation in the .h files.
Related
I have a problem in a bigger project but I managed to isolate it into a smallest possible example. Please take a look.
The def.c file that contains a definition of a struct
typedef struct {
unsigned char data[64];
} somestruct;
a build.py file that builds the cffi wrapper
import os
from cffi import FFI, VerificationError
basepath = os.path.abspath(os.path.dirname(__file__))
ffi = FFI()
# the def.c is both header and source
with open(basepath + '/def.c', 'rt') as fid:
_source = fid.read()
ffi.cdef(_source)
ffi.set_source(
"some_examples",
_source,
extra_compile_args=['-g']
)
ffi.compile()
the file run.py that attempts to instantiate an empty struct
from some_examples import ffi, lib
ffi.new('struct somestruct *')
How to run
python build.py
python run.py
what I get
Traceback (most recent call last):
File "run.py", line 3, in <module>
ffi.new('struct somestruct *')
ffi.error: undefined struct/union name
struct somestruct *
^
I researched it a bit, this answer was helpful but it indicated that definition must be in cdef() which is the case here. Anyone has some ideas why is somestruct undefined?
I'm trying to incorporate a c++ extension as a submodule into an existing python library via cmake. Building the C++ extension works fine and importing it as a python module works, but not as the submodule of the header library.
I have the following directory structure:
frontend/
foo.py
bar.py
backend/
backend.cpp
The extension is bound to a python module via pybind:
PYBIND11_MODULE(backend, m)
{
m.doc() = "backend c++ implementation"; // optional module docstring
m.def("method", &method, "The method I want to call from python.");
}
In the CMakeLists.txt, the relevant line is:
pybind11_add_module(backend "frontend/backend/backend.cpp")
I've followed the instructions form here and here to write the setup.py script. I guess the most important lines look like this:
from setuptools import setup, Extension, find_packages
from setuptools.command.build_ext import build_ext
from setuptools.command.test import test as TestCommand
class CMakeExtension(Extension):
def __init__(self, name, sourcedir=".", sources=[]):
Extension.__init__(self, name, sources=[])
class CMakeBuild(build_ext):
def run(self):
build_directory = os.path.abspath(self.build_temp)
if not os.path.exists(self.build_temp):
os.makedirs(self.build_temp)
cmake_list_dir = os.path.abspath(os.path.dirname(__file__))
print("-" * 10, "Running CMake prepare", "-" * 40)
subprocess.check_call(
["cmake", cmake_list_dir], cwd=self.build_temp,
)
print("-" * 10, "Building extensions", "-" * 40)
cmake_cmd = ["cmake", "--build", "."] + self.build_args
subprocess.check_call(cmake_cmd, cwd=self.build_temp)
# Move from build temp to final position
for ext in self.extensions:
self.move_output(ext)
def move_output(self, ext):
build_temp = Path(self.build_temp).resolve()
dest_path = Path(self.get_ext_fullpath(ext.name)).resolve()
source_path = build_temp / self.get_ext_filename(ext.name)
dest_directory = dest_path.parents[0]
dest_directory.mkdir(parents=True, exist_ok=True)
self.copy_file(source_path, dest_path)
extensions = [CMakeExtension("backend")]
setup(
name="frontend",
packages=["frontend"],
ext_modules=extensions,
cmdclass=dict(build_ext=CMakeBuild),
)
But this does not make backend a submodule of frontend, but instead a module on its own. So this works:
from backend import method
But to avoid naming issues with other libraries, what I would like to have is this:
from frontend.backend import method
Changing the naming in the pybinding or in the extension call to extensions = [CMakeExtension("frontend.backend")] does unfortunately not resolve my problem, the setup does not find the backend.<platform>.so shared library then, because it looks for frontend/backend.<platform>.so, which does not exist. How could I resolve this issue?
I think I've resolved the issue with the following lines:
Change in the setup.py file:
ext_modules = [
Extension(
"frontend.backend", sources=["frontend/backend/backend.cpp"]
)
]
Change in the CMakeLists.txt file:
pybind11_add_module(backend "frontend/backend/backend.cpp")
set_target_properties( backend
PROPERTIES
ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
)
The shared library object backend.platform.so must be located in the frontend directory. Neither the pybind module name nor the sourcefile .cpp should contain any "." in the names, because the get_ext_fullpath() method from build_ext will split by dots. Only the frontend directory containts an init.py file.
I'd like to build a static Cython library using distutils. I don't care about having it be a true Python extension module that can be import'ed. I just want to compile the code and put the objects in a static library. The code to create a dynamic library is very simple,
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
cmdclass = {'build_ext':build_ext},
ext_modules = [Extension("test",["test.pyx"])]
)
Is there a simple way to make it static instead?
Distutils is very limited and not set up for static builds. I would advise you to use something else to compile the static library part of your project.
If your use case is to call into Cython code from other C code, then you want to use the public or api declarations along with your cdef declared functions and variables in your Cython code. Cython will allow the so-declared objects to be called from external C code, and it will generate a .h file alongside the .c file for you.
Fyi, this works using numpy distutils, but obviously is nowhere near the simplicity or probably portability of the original code for a shared library,
from Cython.Compiler.Main import compile
from numpy.distutils.misc_util import Configuration
compile('test.pyx')
config = Configuration(...)
config.add_installed_library('test',
['test.c'],
'test',
{'include_dirs':[get_python_inc()]})
Assuming that you have a sources, include_dirs and build_dir in your setup.py, this is how you can build a static library
from distutils.ccompiler import new_compiler
from sysconfig import get_paths
import os
project_name = "slimey_project"
source = ['source1.c']
include_dirs = ['include']
build_dir = os.path.join(os.path.dirname(__file__), 'build')
class StaticLib(Command):
description = 'build static lib'
user_options = [] # do not remove, needs to be stubbed out!
python_info = get_paths()
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
# Create compiler with default options
c = new_compiler()
# Optionally add include directories etc.
for d in include_dirs:
c.add_include_dir(d)
c.add_include_dir(self.python_info['include'])
# Compile into .o files
objects = c.compile(sources)
# Create static or shared library
c.create_static_lib(objects, project_name, output_dir=build_dir)
Source: https://gist.github.com/udnaan/d549950a33fd82d13f9e6ba4aae82964
I'm trying to teach myself Cython but having problems accessing numpy. The problems seem to be when I use 'cimport'.
For example when importing the following function:
cimport numpy
def cy_sum(x):
cdef numpy.ndarray[int, ndim=1] arr = x
cdef int i, s = 0
for i in range(arr.shape[0]):
s += arr[i]
return s
I get error:
/Users/Daniel/.pyxbld/temp.macosx-10.6-x86_64-2.7/pyrex/test.c:314:10: fatal error: 'numpy/arrayobject.h' file not found
include "numpy/arrayobject.h"
1 error generated.
Any simple instructions on how to resolve this would be much appreciated!
Ok, as has been pointed out above, similar questions have been asked before.
Eg: Make distutils look for numpy header files in the correct place.
I got my code to work by using a setup file with the line
include_dirs = [np.get_include()],
As in:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy as np
ext = [Extension( "cy_test", sources=["cy_test.pyx"] )]
setup(
name = "testing",
cmdclass={'build_ext' : build_ext},
include_dirs = [np.get_include()],
ext_modules=ext
)
I haven't yet tried using pyximport so I'm not sure whether another fix is required for that.
I use a custom random number generator build with Cython. I don't understand why, but it no longer works... I guess it is related to Python 2.7, or maybe a new version of Cython.
In dcmtrand.pyx, I have:
...
import dcmt
...
cdef class RandomState:
...
def __reduce__(self):
return (dcmt.__RandomState_ctor, (), self.get_state())
...
dcmt is a folder. In it, I have init.py file:
from dcmtrand import *
def __RandomState_ctor():
return RandomState.__new__(RandomState)
I compile it using
python setup.py build_ext --inplace
then I copy resulting dcmtrand.so file into dcmt folder, and I move dcmt folder into my project.
Now, if I import dcmt, everything is ok:
import dcmt
import cPickle
dc = dcmt.DynamicCreator(5)
a = dc[0]
cPickle.dumps(a)
But if I want to put dcmt into a subpackage, it no longer works:
from prng import dcmt
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "prng/dcmt/__init__.py", line 1, in <module>
from dcmtrand import *
File "dcmtrand.pyx", line 10, in init dcmtrand (dcmtrand.c:6955)
ImportError: No module named dcmt
To make it work, I need to add prng to Python path.
Why it is no longer working? How to make it work again?
Yes, you have 3 choices:
Set PYTHONPATH to have dmct in it: PYTHONPATH=$PYTHONPATH:prng
Use sys.path:
from os.path import dirname, join
import sys
sys.path.append(join(dirname(__file__), 'prng')
Do the same inclusion as in your .py: from prng import dcmt