Develop Reference

Python/C++ Extension, Undefined symbol error when linking a library

I made a project that has glfw as a library, my directory looks like this:
main_dir
|--include
|--|--glfw_binder.h
|--src
|--|--glfw_binder.cpp
|--lib
|--|--glfw
|--|--|--src
|--|--|--|--libglfw.so
|--|--|--|--libglfw.so.3
|--|--|--|--libglfw.so.3.3
|--|--|--|--...
|--|--|--include
|--|--|-- ...
|--main.cpp
|--setup.py
|--test.py
I've installed glfw from its website and simply extracted it to lib/ directory. Then I ran cmake -G "Unix Makefiles" followed by make to build it.
When I try to run it as pure C++ with this cmake file it runs just fine, it displays the glfw window:
cmake_minimum_required(VERSION 3.0)
project(Test)
add_subdirectory(lib/glfw)
find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_DIRS})
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
add_executable(Test main.cpp src/glfw_binder.cpp)
target_link_libraries(Test PRIVATE
${PYTHON_LIBRARIES}
glfw
)
And here is what my setup.py looks like:
from setuptools import setup, Extension, find_packages
module1 = Extension('test',
sources = ['main.cpp', 'src/glfw_binder.cpp'],
include_dirs=["include", "lib/glfw/include"],
library_dirs=["lib/glfw"],
libraries=["lib/glfw"],
extra_link_args=["lib/glfw"]
)
setup (name = 'python_test',
version = '1.0',
description = 'This is a demo package',
packages=find_packages("glfw"),
ext_modules = [module1]
)
Then I run python3 setup.py build followed by sudo python3 setup.py install. It creates the so files just fine. Then I run this test code:
import test
test.create_glfw_window()
When I run this script via python3 test.py I got this error:
Traceback (most recent call last):
File "/home/turgut/Desktop/TestDir/Python-Binder-Test/test.py", line 3, in <module>
import test
ImportError: /usr/local/lib/python3.10/dist-packages/python_test-1.0-py3.10-linux-x86_64.egg/test.cpython-310-x86_64-linux-gnu.so: undefined symbol: glfwDestroyWindow
It's not specific to glfwDestroyWindow it says this for all glfw functions. How am I supposed to link this?
Update: I've added these lines:
packages=['glfw'],
package_dir={'glfw': 'lib/glfw'},
#package_data={'glfw': ['lib/glfw/src/*.so']},
package_data={'glfw': ['lib/glfw/src/libglfw.so', "lib/glfw/src/libglfw.so.3", "lib/glfw/src/libglfw.so.3.3"]},
Now it gives me the following error:
ImportError: libglfw.so.3: cannot open shared object file: No such file or directory
Even though I've specified all the .so files together. I don't think this is the way to go but I thought it's worth mentioning.
Better update:
I've removed the above update and added these two lines to my extension:
include_dirs=["include", "lib/glfw/include"],
#extra_link_args=[ "-lm", "-lGL", "-lGLU", "-lglfw"],
library_dirs=["lib/glfw"],
libraries=['glfw']
and now I'm getting this error which I think is closer to what I'm looking for:
/usr/bin/ld: cannot find -lglfw: No such file or directory
collect2: error: ld returned 1 exit status
error: command '/usr/bin/x86_64-linux-gnu-g++' failed with exit code 1

For anyone facing a similar issue, you need to specify runtime_library_dirs so the setup.py file looks like this:
from setuptools import setup, Extension, find_packages
module1 = Extension('nerveblox',
sources = ['main.cpp', 'src/nerveblox_VM.cpp'],
include_dirs=["include", "lib/glfw/include"],
#extra_link_args=[ "-lm", "-lGL", "-lGLU", "-lglfw"],
library_dirs=["lib/glfw/src"],
libraries=['glfw', 'GL'],
runtime_library_dirs=['lib/glfw/src']
)
setup (
name = 'Nerveblox_python_test',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1],
install_requires=[
'importlib-metadata; python_version == "3.8"',
],
)

Cythonized Python.Net code cannot find system assemblies

When I compile with Cython the Python code which uses Python.NET to access .NET assemblies, it can't find those assemblies:
ModuleNotFoundError: No module named 'System'
Without compilation it works ok.
For demo code, I used https://github.com/pythonnet/pythonnet/blob/master/demo/helloform.py
My setup.py file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
ext_modules = [
Extension(
'helloform',
sources = ['helloform.py'],
language = 'c++'
)
]
setup(
name = 'helloform',
ext_modules = cythonize(ext_modules),
)
Then I build it with python setup.py build_ext --inplace.
I wanted to load compiled module from Python prompt with import helloform but it failed with
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "helloform.py", line 8, in init helloform
ModuleNotFoundError: No module named 'System'

This answer is untested - I don't think I can easily set up an environment to test so it's a bit of a guess. If it doesn't work I'll remove it.
This probably is a bug, and if you want it fixed in the longer term you should report it. Cython does try to be compatible with Python wherever possible.... A quick investigate suggests that Python.NET overrides the built-in __import__ function. Cython looks to lookup and use this function in Python 2, but not in Python 3. This is no longer the preferred way of customizing import behaviour (but is still supported). I'd guess it would work in Cython + Python 2?
As a workaround you should probably just run the import statements in Python. There's two obvious ways to do it:
Write a small separate module just containing the import statements, then in Cython import from that module:
from import_module import WinForms, Size, Point
Run the import statements in exec; extract the values out of the global dict you pass it:
import_dict = {}
exec("""import clr
# etc...
""", import_dict) # pass a dict in as `globals`
WinForms = import_dict['WinForms']
# etc.

python cmd error: invalid command 'y'

Python - When parsing the following command-line arguments...
sys.argv = ('%s %s build_ext --inplace' % ('python', sys.argv[0]))
...for Cython inside my buildscript "myscript_pyd_setup.py" for "myscript.pyd" that uses the "myscript.py" file I get the following error (e1):
python error 1: invalid command 'y'
The python command-line option python -- help revealed no corresponding option 'y'. This makes sense otherwise the error would not exist. Searching SO, Python docs and the web resulted in unrelated articles about Tkinter. As I do absolutely nothing with Tkinter, as far as I know it, I'm wondering if this error is from the python command-line interpreter at all?
My effort:
What I tried to accomplish is parsing arguments sys.argv.append('build_ext --inplace') to python command-line interpreter when running the "buildscript" from within the editor (Komodo edit 11.x) but it returns also with an error (e2) similar to the version from command-line or another editor.
python error 2: invalid command name 'build_ext --inplace'
Running "myscript_pyd_setup.py" with sys.argv.append('build_ext') builds the required *.pyd file just fine. What I don't understand is why its choking on --infile. This works for py2exe and pyinstaller.
The objective:
To run the setup-script of "myscript.py" from within the editor and not having to flip back-and-forth to the command-line editor for compiling *.py > *.pyd when I changed code inside "myscript.py" and want to see the result quickly.
Note: parsing command-line python myscript_pyd_setup.py build_ext --inplace works fine when sys.argv... is commented-out in the "buildscript"!
A third option was to use cythonize in combo with the "myscript.py" and "myscript.pyd" files but that showed a copyfile error for "myscript.pyd". Relevant but not for the above asked "error = y" question.
Any thoughts and help on how to automate this part to prevent RSI are more than welcome! Thx.
My "myscript.py" example code:
import sys, time, os
#...snippet...
def print_me():
text = "bar(man), yes, Hello, how do you do Mr. foo?"
return text
if __name__ == '__main__':
#...snippet...
print_me()
The myscript_pyd_setup.py:
# myscript.py
try:
from setuptools import setup
from setuptools import Extension
except ImportError:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
#from Cython.Build import cythonize
import sys
print 'sys.argv[0] : %s' % sys.argv[0]
#sys.argv = ('%s %s build_ext --inplace' % ('python', sys.argv[0]))
sys.argv.append('build_ext --inplace')
ext_modules = [Extension("myscript",['myscript.py'])]
#ext_modules = cythonize("myscript.py")
#setup: "name" and "cmdclass" are commented-out when using cythonize.
setup(
name= 'XYZ model class',
cmdclass = {'build_ext': build_ext},
include_dirs = [],
ext_modules = ext_modules)

sys.argv is a list of arguments. When you append 'build_ext --inplace', you literally append that as a single argument (as if you've passed it enclosed in quotes from the shell command line). That is what happened in the second case.
In the first one, you've assigned string back to sys.argv. But string is also a sequence, so your commands ran as if called with (argv[1:]): ['y', 't', `h`, ...]
In either case you ended up with an option/sub-command unknown the the argument parser.

Cython Compilation Error: dynamic module does not define module export function

I am building a package in Cython. I am using the following as the structure for setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy
import scipy
extensions = [
Extension("xxxxx",["xxxx/xxxxx.pyx"],
include_dirs=[numpy.get_include(),"."]),
Extension("nnls",["xxxxx/xxxxx.pyx"],
include_dirs=[numpy.get_include(),"."]),
]
setup(
name='xxxxxx',
version='0.0.0',
description='''********''',
url='xxxxxxx',
author='xxxxx',
author_email='xxxxx',
packages=[
'xxxxx',
],
install_requires=[
'cython',
'numpy',
'scipy',
],
ext_modules=cythonize(extensions),
)
However, I am getting an error upon installation in Python 3. It is working in Python 2 however, it is not compiling in Python 3 having the following error:
dynamic module does not define module export function
How can I solve this problem? Is the structure of the setup.py the reason why this is not compiling?

You need to call setup.py with Python 3 (python3 setup.py build_ext, maybe --inplace). It's because Python 3 defines a different name for the init function called when the module starts, and so you need to build it using Python 3 to ensure the correct name is generated.
See dynamic module does not define init function (PyInit_fuzzy) and How to specify Python 3 source in Cython's setup.py? for slightly more detail (it's bordering on a duplicate of these questions, but isn't quite in my view)

I experienced this and found that I had to use the same name of .pyx as the module name, e.g.
makefile:
# (default)
# INSTALL_DIR:=/usr/lib/python3.6/site-packages
# (my venv)
INSTALL_DIR:=/home/<username>/python3_venv/lib/python3.6/site-packages
all:
sudo python3 setup_myproj.py install --install-lib ${INSTALL_DIR}
setup_myproj.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
ext = Extension("myproj",
sources=["myproj.pyx", "myCppProjFacade.cpp"],
<etc>
language="c++"
)
setup(name="myproj",
version="0.0.1",
ext_modules=cythonize(ext))
client module, run after installing to venv
import myproj as myCppProjWrapper
...
I also found that if the "myproj" names are different, under <python-lib-dir>/<python-vers>/site-packages the .so and .egg-info names are different and the client fails to load it.
In addition I found that the client's environment does not need to have the cython package installed.

I had the same error for torchvision. FIxed it by downgrading the installation versions:
pip install torch==1.2.0+cu92 torchvision==0.4.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html

How should I structure a Python package that contains Cython code

I'd like to make a Python package containing some Cython code. I've got the the Cython code working nicely. However, now I want to know how best to package it.
For most people who just want to install the package, I'd like to include the .c file that Cython creates, and arrange for setup.py to compile that to produce the module. Then the user doesn't need Cython installed in order to install the package.
But for people who may want to modify the package, I'd also like to provide the Cython .pyx files, and somehow also allow for setup.py to build them using Cython (so those users would need Cython installed).
How should I structure the files in the package to cater for both these scenarios?
The Cython documentation gives a little guidance. But it doesn't say how to make a single setup.py that handles both the with/without Cython cases.

I've done this myself now, in a Python package simplerandom (BitBucket repo - EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).
This method relies on the fact that building a .pyx file with Cython.Distutils.build_ext (at least with Cython version 0.14) always seems to create a .c file in the same directory as the source .pyx file.
Here is a cut-down version of setup.py which I hope shows the essentials:
from distutils.core import setup
from distutils.extension import Extension
try:
from Cython.Distutils import build_ext
except ImportError:
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
]
setup(
name='mypackage',
...
cmdclass=cmdclass,
ext_modules=ext_modules,
...
)
I also edited MANIFEST.in to ensure that mycythonmodule.c is included in a source distribution (a source distribution that is created with python setup.py sdist):
...
recursive-include cython *
...
I don't commit mycythonmodule.c to version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_ext first, to ensure that mycythonmodule.c is present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.

Adding to Craig McQueen's answer: see below for how to override the sdist command to have Cython automatically compile your source files before creating a source distribution.
That way your run no risk of accidentally distributing outdated C sources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.
from distutils.command.sdist import sdist as _sdist
...
class sdist(_sdist):
def run(self):
# Make sure the compiled Cython files in the distribution are up-to-date
from Cython.Build import cythonize
cythonize(['cython/mycythonmodule.pyx'])
_sdist.run(self)
cmdclass['sdist'] = sdist

http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules
It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn’t want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.
This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:
from distutils.core import setup
from distutils.extension import Extension
setup(
ext_modules = [Extension("example", ["example.c"])]
)

The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.
Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html
What that documentation doesn't do is say how to make this conditional, but
try:
from Cython.distutils import build_ext
except ImportError:
from distutils.command import build_ext
Should handle it.

Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.
A possible example: https://github.com/douban/greenify/blob/master/setup.py
Update(2017-01-05):
Since setuptools 18.0, there's no need to use setuptools_cython. Here is an example to build Cython project from scratch without setuptools_cython.

All other answers either rely on
distutils
importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requires and importing it.
A modern solution is to use setuptools instead, see this answer (automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.py with requirements handling, an entry point, and a cython module could look like this:
from setuptools import setup, Extension
with open('requirements.txt') as f:
requirements = f.read().splitlines()
setup(
name='MyPackage',
install_requires=requirements,
setup_requires=[
'setuptools>=18.0', # automatically handles Cython extensions
'cython>=0.28.4',
],
entry_points={
'console_scripts': [
'mymain = mypackage.main:main',
],
},
ext_modules=[
Extension(
'mypackage.my_cython_module',
sources=['mypackage/my_cython_module.pyx'],
),
],
)

The simple hack I came up with:
from distutils.core import setup
try:
from Cython.Build import cythonize
except ImportError:
from pip import pip
pip.main(['install', 'cython'])
from Cython.Build import cythonize
setup(…)
Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.

This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.
Givig structure like this:
__init__.py
setup.py
test.py
subdir/
__init__.py
anothertest.py
setup.py
from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
'test',
'subdir.anothertest',
)
cmdclass = {'build_ext': build_ext}
# for modules in main dir
ext_modules = [
Extension(
ext,
[ext + ".py"],
)
for ext in ext_names if ext.find('.') < 0]
# for modules in subdir ONLY ONE LEVEL DOWN!!
# modify it if you need more !!!
ext_modules += [
Extension(
ext,
["/".join(ext.split('.')) + ".py"],
)
for ext in ext_names if ext.find('.') > 0]
setup(
name='name',
ext_modules=ext_modules,
cmdclass=cmdclass,
packages=["base", "base.subdir"],
)
# Build --------------------------
# python setup.py build_ext --inplace
Happy compiling ;)

The easiest way I found using only setuptools instead of the feature limited distutils is
from setuptools import setup
from setuptools.extension import Extension
try:
from Cython.Build import cythonize
except ImportError:
use_cython = False
else:
use_cython = True
ext_modules = []
if use_cython:
ext_modules += cythonize('package/cython_module.pyx')
else:
ext_modules += [Extension('package.cython_module',
['package/cython_modules.c'])]
setup(name='package_name', ext_modules=ext_modules)

I think I found a pretty good way of doing this by providing a custom build_ext command. The idea is the following:
I add the numpy headers by overriding finalize_options() and doing import numpy in the body of the function, which nicely avoids the problem of numpy not being available before setup() installs it.
If cython is available on the system, it hooks into the command's check_extensions_list() method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension() method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.
Here's the code:
import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext
try:
import Cython.Build
HAVE_CYTHON = True
except ImportError:
HAVE_CYTHON = False
class BuildExtWithNumpy(build_ext):
def check_cython(self, ext):
c_sources = []
for fname in ext.sources:
cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
c_sources.append(cname)
if matches and dep_util.newer(fname, cname):
if HAVE_CYTHON:
return ext
raise RuntimeError("Cython and C module unavailable")
ext.sources = c_sources
return ext
def check_extensions_list(self, extensions):
extensions = [self.check_cython(ext) for ext in extensions]
return build_ext.check_extensions_list(self, extensions)
def finalize_options(self):
import numpy as np
build_ext.finalize_options(self)
self.include_dirs.append(np.get_include())
This allows one to just write the setup() arguments without worrying about imports and whether one has cython available:
setup(
# ...
ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
setup_requires=['numpy'],
cmdclass={'build_ext': BuildExtWithNumpy}
)