Develop Reference

How to cythonize a Python-ROS2 package in order to hide the source code?

I have a ROS2-python (rclpy) package with multiple subpackages that work together. I would like to cythonize the package in order to hide the source code and just leave the executables to do the job. Any idea how could I do that with the setup.py of the package and the launch files ?

So I would answer my own question in case someone faced the same issue. For cythonizing your ros2 package in python the best way is adding the required cython components to the setup.py of each package separately. I have marked the cython parts (with <--) for a better understranding.
Please note that after building the package using colcon, a copy of the python source codes would be stored under the install directory produced by colcon. You can easily delete them manually and the package should run with the generated c and shared object files.
from glob import glob
from setuptools import setup
from Cython.Build import cythonize <--
import os
package_name = 'your_package_name_here'
files = package_name + "/*.py"
setup(
ext_modules=cythonize(files,compiler_directives={'language_level' : "3"},force=True,quiet=True), <--
name=package_name,
version='0.0.0',
packages=[package_name],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
(os.path.join('share', package_name), glob('launch/*.launch.py'))
],
install_requires=['setuptools', "wheel", "Cython"], <--
zip_safe=True,
maintainer='NAME',
maintainer_email='EMAIL',
description='TODO: Package description',
license='TODO: License declaration',
tests_require=['pytest'],
entry_points={
'console_scripts': [
'SOME CODE HERE',
],
},
)

`python setup.py sdist` with cython extension: .pyx' doesn't match any files

I'm maintaining a python package which includes a cython-based c extention.
The source code is on github: https://github.com/vlkit/vlkit.
Here is my setup.py:
import os
from setuptools import setup, find_packages
from distutils.core import Extension
try:
import numpy
except ImportError: # We do not have numpy installed
os.system("pip install numpy")
try:
from Cython.Build import cythonize
except ImportError: # We do not have Cython installed
os.system("pip install Cython")
import numpy
from Cython.Build import cythonize
__version__ = "0.1.0-b3"
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/nms_ext.pyx"],
include_dirs=[numpy.get_include()])
]
setup(name='vltools',
version=__version__,
description='vision and learning tools',
url='https://github.com/vltools/vltools',
author_email='a#b.c',
license='MIT',
packages=find_packages(),
ext_modules=cythonize(exts),
zip_safe=False,
data_files=[("data", ["data/imagenet1000_clsidx_to_labels.txt"])]
)
When building locally with python setup build && python setup.py install, everything goes smoothly.
However, when I'm trying to create a source distribution with python setup.py sdist and then install from the generated dist/vltools-0.1.0b3.tar.gz it run into an error:
ValueError: 'vltools/nms/nms_ext.pyx' doesn't match any files
In my understanding, the one actually required for installation is nms_ext.c which is indeed inside the generated dist/vltools-0.1.0b3.tar.gz.
However, in my setup.py it's "nms_ext.pyx" in the sources:
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/nms_ext.pyx"],
include_dirs=[numpy.get_include()])
]
So what should I do when creating a source distribution with python setup.py sdist?

There are several things to fix or improve in your setup script.
Declaring build and runtime dependencies
Instead of running pip install {numpy,cython} where a lot of things can go wrong, a proper way of declaring dependencies is to pass them to setup() in the setup_requires/install_requires args. Cython is required only at the build stage, but not when vltools is installed and imported - it will be added to setup_requires. numpy is required to both build and run the package, so it is included in both lists:
setup(
...,
setup_requires=["cython", "numpy"],
install_requires=["numpy"],
)
This has the advantage that Cython won't be installed. It will be downloaded, used for building and then removed.
You can (and should) extend the install_requires list with other packages vltools requires, for example scipy, pillow etc.
Deferring Cython and numpy imports
To ensure the package is installable on a system that has neither Cython nor numpy installed beforehand, we have to defer their imports. I won't go into much detail here; check out Add numpy.get_include() argument to setuptools without preinstalled numpy for more details. In the custom impl of the build command below, I defer both cythonizing and extending extension headers with numpy includes:
class build(build_orig):
def finalize_options(self):
super().finalize_options()
__builtins__.__NUMPY_SETUP__ = False
import numpy
for extension in self.distribution.ext_modules:
extension.include_dirs.append(numpy.get_include())
from Cython.Build import cythonize
self.distribution.ext_modules = cythonize(self.distribution.ext_modules,
language_level=3)
Register the custom command impl via cmdclass:
setup(
...,
cmdclass={"build": build},
)
Packaging nms.h
Currently, the nms.h is not being added to your source dist, making the installation impossible. This is easily fixed by including it via e.g. package_data:
setup(
...,
package_data={"vltools.nms": ["nms.h"]},
)
Now you also should add the parent dir of nms.h to include_dirs:
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/_nms_ext.c", "vltools/nms/nms_ext.pyx"],
include_dirs=["vltools/nms"])]
Full setup script
import os
from setuptools import setup, find_packages
from setuptools import Extension
from distutils.command.build import build as build_orig
__version__ = "0.1.0b3"
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/_nms_ext.c", "vltools/nms/nms_ext.pyx"],
include_dirs=["vltools/nms"])]
class build(build_orig):
def finalize_options(self):
super().finalize_options()
__builtins__.__NUMPY_SETUP__ = False
import numpy
for extension in self.distribution.ext_modules:
extension.include_dirs.append(numpy.get_include())
from Cython.Build import cythonize
self.distribution.ext_modules = cythonize(self.distribution.ext_modules,
language_level=3)
setup(name='vltools',
version=__version__,
description='vision and learning tools',
url='https://github.com/vltools/vltools',
author_email='kz#kaizhao.net',
license='MIT',
packages=find_packages(),
ext_modules=exts,
setup_requires=["cython", "numpy"],
install_requires=["numpy"],
zip_safe=False,
data_files=[("data", ["data/imagenet1000_clsidx_to_labels.txt"])],
package_data={"vltools.nms": ["nms.h"]},
cmdclass={"build": build},
)
P.S. Two years have passed and here we meet again ;-)

As #hoefling suggested, I've updated my setup.py.
However, there is still an error when installing from .tar.gz file that it cannot find 'vltools/nms/nms_ext.pyx'.
setup.py hoefling posted:
import os
from setuptools import setup, find_packages
from setuptools import Extension
from distutils.command.build import build as build_orig
__version__ = "0.1.0b3"
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/_nms_ext.c", "vltools/nms/nms_ext.pyx"],
include_dirs=["vltools/nms"])]
class build(build_orig):
def finalize_options(self):
super().finalize_options()
__builtins__.__NUMPY_SETUP__ = False
import numpy
for extension in self.distribution.ext_modules:
extension.include_dirs.append(numpy.get_include())
from Cython.Build import cythonize
self.distribution.ext_modules = cythonize(self.distribution.ext_modules,
language_level=3)
setup(name='vltools',
version=__version__,
description='vision and learning tools',
url='https://github.com/vltools/vltools',
author_email='kz#kaizhao.net',
license='MIT',
packages=find_packages(),
ext_modules=exts,
setup_requires=["cython", "numpy"],
install_requires=["numpy"],
zip_safe=False,
data_files=[("data", ["data/imagenet1000_clsidx_to_labels.txt"])],
package_data={"vltools.nms": ["nms.h"]},
cmdclass={"build": build},
)
When installing with python setup.py sdist && cd dist && pip install vltools-0.1.0b3.tar.gz, there is an error saying:
ValueError: 'vltools/nms/nms_ext.pyx' doesn't match any files
Becase the pyx file was not packed into tar.gz.
To have 'vltools/nms/nms_ext.pyx' in the tar.gz file, I add it into the package_data list:
package_data={"vltools.nms": ["nms.h", "nms_ext.pyx"]}
Finally, my full setup.py is:
import os
from setuptools import setup, find_packages
from setuptools import Extension
from distutils.command.build import build as build_orig
__version__ = "0.1.0b3"
exts = [Extension(name='vltools.nms.nms_ext',
sources=["vltools/nms/_nms_ext.c", "vltools/nms/nms_ext.pyx"],
include_dirs=["vltools/nms"])]
class build(build_orig):
def finalize_options(self):
super().finalize_options()
__builtins__.__NUMPY_SETUP__ = False
import numpy
for extension in self.distribution.ext_modules:
extension.include_dirs.append(numpy.get_include())
from Cython.Build import cythonize
self.distribution.ext_modules = cythonize(self.distribution.ext_modules,
language_level=3)
setup(name='vltools',
version=__version__,
description='vision and learning tools',
url='https://github.com/vltools/vltools',
author_email='kz#kaizhao.net',
license='MIT',
packages=find_packages(),
ext_modules=exts,
setup_requires=["cython", "numpy"],
install_requires=["numpy"],
zip_safe=False,
data_files=[("data", ["data/imagenet1000_clsidx_to_labels.txt"])],
package_data={"vltools.nms": ["nms.h", "nms_ext.pyx"]},
cmdclass={"build": build},
)

Use Setuptools with C Extension as part of package

I wrote a C Extension to access an error message for a camera from a proprietary library. The structure is
setup.py
dicamsdk\
|---__init__.py
|---control.py
|---pcoError.c
with setup.py
from setuptools import setup, Extension, find_packages
from dicamsdk.control import __version__
pcoErrorModule = Extension("dicamsdk.pcoError",
sources=["dicamsdk/pcoError.c"],
include_dirs=['C:\Program Files (x86)'
'\Digital Camera Toolbox'
'\Sensicam SDK\include'],
define_macros=[("PCO_ERRT_H_CREATE_OBJECT", None)],
)
setup(
name="pydicamsdk",
platforms=["win-amd64", 'win32'],
license="GNU GPLv3",
ext_modules=[pcoErrorModule],
packages=find_packages(),
version=__version__
)
and the control.py intends to import the compiled C Extension with
from . import pcoError
When I try to build (or install) the package I always receive the error ImportError: cannot import name 'pcoError'.
The only way it seems to work is to comment out the import in control.py and build the C Extension with setup.py build_ext --inplace. Just with the compiled present I can build/install my library.
Is there a solution to be implemented in my setup.py to compile my extension in first place to enable a simple installation?

A more proper solution based on your own answer:
Your version is presumably for the entire project, not just the control module. It is standard to define __version__ in __init__.py. In that case, your import in setup.py would look like from dicamsdk import __version__. This will cause no conflicts, unless you do something foolish like import all your modules automatically from the package root.

The problem was with the import at the beginning of setup.py. The import in line 2 (from dicamsdk.control import __version__) forced the code check of the module.
When removed, the install or build of the packages runs correctly.

Cython Compilation Error: dynamic module does not define module export function

I am building a package in Cython. I am using the following as the structure for setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy
import scipy
extensions = [
Extension("xxxxx",["xxxx/xxxxx.pyx"],
include_dirs=[numpy.get_include(),"."]),
Extension("nnls",["xxxxx/xxxxx.pyx"],
include_dirs=[numpy.get_include(),"."]),
]
setup(
name='xxxxxx',
version='0.0.0',
description='''********''',
url='xxxxxxx',
author='xxxxx',
author_email='xxxxx',
packages=[
'xxxxx',
],
install_requires=[
'cython',
'numpy',
'scipy',
],
ext_modules=cythonize(extensions),
)
However, I am getting an error upon installation in Python 3. It is working in Python 2 however, it is not compiling in Python 3 having the following error:
dynamic module does not define module export function
How can I solve this problem? Is the structure of the setup.py the reason why this is not compiling?

You need to call setup.py with Python 3 (python3 setup.py build_ext, maybe --inplace). It's because Python 3 defines a different name for the init function called when the module starts, and so you need to build it using Python 3 to ensure the correct name is generated.
See dynamic module does not define init function (PyInit_fuzzy) and How to specify Python 3 source in Cython's setup.py? for slightly more detail (it's bordering on a duplicate of these questions, but isn't quite in my view)

I experienced this and found that I had to use the same name of .pyx as the module name, e.g.
makefile:
# (default)
# INSTALL_DIR:=/usr/lib/python3.6/site-packages
# (my venv)
INSTALL_DIR:=/home/<username>/python3_venv/lib/python3.6/site-packages
all:
sudo python3 setup_myproj.py install --install-lib ${INSTALL_DIR}
setup_myproj.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("myproj",
sources=["myproj.pyx", "myCppProjFacade.cpp"],
<etc>
language="c++"
)
setup(name="myproj",
version="0.0.1",
ext_modules=cythonize(ext))
client module, run after installing to venv
import myproj as myCppProjWrapper
...
I also found that if the "myproj" names are different, under <python-lib-dir>/<python-vers>/site-packages the .so and .egg-info names are different and the client fails to load it.
In addition I found that the client's environment does not need to have the cython package installed.

I had the same error for torchvision. FIxed it by downgrading the installation versions:
pip install torch==1.2.0+cu92 torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

How should I structure a Python package that contains Cython code

I'd like to make a Python package containing some Cython code. I've got the the Cython code working nicely. However, now I want to know how best to package it.
For most people who just want to install the package, I'd like to include the .c file that Cython creates, and arrange for setup.py to compile that to produce the module. Then the user doesn't need Cython installed in order to install the package.
But for people who may want to modify the package, I'd also like to provide the Cython .pyx files, and somehow also allow for setup.py to build them using Cython (so those users would need Cython installed).
How should I structure the files in the package to cater for both these scenarios?
The Cython documentation gives a little guidance. But it doesn't say how to make a single setup.py that handles both the with/without Cython cases.

I've done this myself now, in a Python package simplerandom (BitBucket repo - EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).
This method relies on the fact that building a .pyx file with Cython.Distutils.build_ext (at least with Cython version 0.14) always seems to create a .c file in the same directory as the source .pyx file.
Here is a cut-down version of setup.py which I hope shows the essentials:
from distutils.core import setup
from distutils.extension import Extension
try:
from Cython.Distutils import build_ext
except ImportError:
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
]
setup(
name='mypackage',
...
cmdclass=cmdclass,
ext_modules=ext_modules,
...
)
I also edited MANIFEST.in to ensure that mycythonmodule.c is included in a source distribution (a source distribution that is created with python setup.py sdist):
...
recursive-include cython *
...
I don't commit mycythonmodule.c to version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_ext first, to ensure that mycythonmodule.c is present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.

Adding to Craig McQueen's answer: see below for how to override the sdist command to have Cython automatically compile your source files before creating a source distribution.
That way your run no risk of accidentally distributing outdated C sources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.
from distutils.command.sdist import sdist as _sdist
...
class sdist(_sdist):
def run(self):
# Make sure the compiled Cython files in the distribution are up-to-date
from Cython.Build import cythonize
cythonize(['cython/mycythonmodule.pyx'])
_sdist.run(self)
cmdclass['sdist'] = sdist

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules
It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn’t want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.
This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:
from distutils.core import setup
from distutils.extension import Extension
setup(
ext_modules = [Extension("example", ["example.c"])]
)

The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.
Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html
What that documentation doesn't do is say how to make this conditional, but
try:
from Cython.distutils import build_ext
except ImportError:
from distutils.command import build_ext
Should handle it.

Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.
A possible example: https://github.com/douban/greenify/blob/master/setup.py
Update(2017-01-05):
Since setuptools 18.0, there's no need to use setuptools_cython. Here is an example to build Cython project from scratch without setuptools_cython.

All other answers either rely on
distutils
importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requires and importing it.
A modern solution is to use setuptools instead, see this answer (automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.py with requirements handling, an entry point, and a cython module could look like this:
from setuptools import setup, Extension
with open('requirements.txt') as f:
requirements = f.read().splitlines()
setup(
name='MyPackage',
install_requires=requirements,
setup_requires=[
'setuptools>=18.0', # automatically handles Cython extensions
'cython>=0.28.4',
],
entry_points={
'console_scripts': [
'mymain = mypackage.main:main',
],
},
ext_modules=[
Extension(
'mypackage.my_cython_module',
sources=['mypackage/my_cython_module.pyx'],
),
],
)

The simple hack I came up with:
from distutils.core import setup
try:
from Cython.Build import cythonize
except ImportError:
from pip import pip
pip.main(['install', 'cython'])
from Cython.Build import cythonize
setup(…)
Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.

This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.
Givig structure like this:
__init__.py
setup.py
test.py
subdir/
__init__.py
anothertest.py
setup.py
from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
'test',
'subdir.anothertest',
)
cmdclass = {'build_ext': build_ext}
# for modules in main dir
ext_modules = [
Extension(
ext,
[ext + ".py"],
)
for ext in ext_names if ext.find('.') < 0]
# for modules in subdir ONLY ONE LEVEL DOWN!!
# modify it if you need more !!!
ext_modules += [
Extension(
ext,
["/".join(ext.split('.')) + ".py"],
)
for ext in ext_names if ext.find('.') > 0]
setup(
name='name',
ext_modules=ext_modules,
cmdclass=cmdclass,
packages=["base", "base.subdir"],
)
# Build --------------------------
# python setup.py build_ext --inplace
Happy compiling ;)

The easiest way I found using only setuptools instead of the feature limited distutils is
from setuptools import setup
from setuptools.extension import Extension
try:
from Cython.Build import cythonize
except ImportError:
use_cython = False
else:
use_cython = True
ext_modules = []
if use_cython:
ext_modules += cythonize('package/cython_module.pyx')
else:
ext_modules += [Extension('package.cython_module',
['package/cython_modules.c'])]
setup(name='package_name', ext_modules=ext_modules)

I think I found a pretty good way of doing this by providing a custom build_ext command. The idea is the following:
I add the numpy headers by overriding finalize_options() and doing import numpy in the body of the function, which nicely avoids the problem of numpy not being available before setup() installs it.
If cython is available on the system, it hooks into the command's check_extensions_list() method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension() method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.
Here's the code:
import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext
try:
import Cython.Build
HAVE_CYTHON = True
except ImportError:
HAVE_CYTHON = False
class BuildExtWithNumpy(build_ext):
def check_cython(self, ext):
c_sources = []
for fname in ext.sources:
cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
c_sources.append(cname)
if matches and dep_util.newer(fname, cname):
if HAVE_CYTHON:
return ext
raise RuntimeError("Cython and C module unavailable")
ext.sources = c_sources
return ext
def check_extensions_list(self, extensions):
extensions = [self.check_cython(ext) for ext in extensions]
return build_ext.check_extensions_list(self, extensions)
def finalize_options(self):
import numpy as np
build_ext.finalize_options(self)
self.include_dirs.append(np.get_include())
This allows one to just write the setup() arguments without worrying about imports and whether one has cython available:
setup(
# ...
ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
setup_requires=['numpy'],
cmdclass={'build_ext': BuildExtWithNumpy}
)