I coerced Cython to use clang by specifying the CC environment variable:
import os
os.environ['CC'] = 'clang'
I have a standard build:
EXT_MODULES = [Extension('example.src.ex',
sources=['example/src/ex.pyx'])]
setup_info = dist(
...
ext_modules=cythonize(EXT_MODULES,
compiler_directives={'language_level': '3'}),
...
)
setup(**setup_info)
However it seems like Cython is somehow using both clang and gcc in different parts of the build step, in particular it's using gcc to build the shared libraries:
running build_ext
building 'example.src.ex' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/example
creating build/temp.linux-x86_64-3.8/example/src
clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.8 -c example/src/ex.c -o build/temp.linux-x86_64-3.8/example/src/ex.o
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/example
creating build/lib.linux-x86_64-3.8/example/src
gcc -shared -Wl,--strip-all build/temp.linux-x86_64-3.8/example/src/ex.o -L/usr/local/lib -o build/lib.linux-x86_64-3.8/example/src/ex.cpython-38-x86_64-linux-gnu.so
How do I get Cython to use clang on both steps?
I realized distutils needs an override of the linker as well
os.environ['LDSHARED'] = 'clang -shared'
Related
In my current project, I'm extensively using Cython. I have many separated setup.py files to build Cython code (*.pyx) with no issues at all, (python 3.8 and gcc 8.3) with the command:
python setup.py build_ext --inplace
Here is a straightforward example:
# =============================================================
# Imports:
# =============================================================
import os
import sys
from distutils.core import setup
import setuptools
from distutils.extension import Extension
from Cython.Build import build_ext
from Cython.Build import cythonize
import numpy
import os.path
import io
import shutil
# =============================================================
# Modules:
# =============================================================
ext_modules = [
Extension("cc1", ["cc1.pyx"],
extra_compile_args=['-O3','-w','-fopenmp'],
extra_link_args=['-fopenmp','-ffast-math','-march=native'],
include_dirs=[numpy.get_include()],
language='c++'),
Extension("cc2", ["cc2.pyx"],
extra_compile_args=['-O3','-w','-fopenmp'],
extra_link_args=['-fopenmp','-ffast-math','-march=native'],
include_dirs=[numpy.get_include()],
language='c++'),
]
# =============================================================
# Class:
# =============================================================
class BuildExt(build_ext):
def build_extensions(self):
if '-Wstrict-prototypes' in self.compiler.compiler_so:
self.compiler.compiler_so.remove('-Wstrict-prototypes')
super().build_extensions()
# =============================================================
# Main:
# =============================================================
for e in ext_modules:
e.cython_directives = {'embedsignature': True,'boundscheck': False,'wraparound': False,'linetrace': True, 'language_level': "3"}
setup(
name='jackProject',
version='0.1.0',
author='Jack',
ext_modules=ext_modules,
cmdclass = {'build_ext': BuildExt},
)
Everything works fine, and while the script is in execution, I can check the status on my console, as instance:
running build_ext
cythoning cc1.pyx to cc1.cpp
cythoning cc2.pyx to cc2.cpp
building 'cc1' extension
creating build
creating build/temp.linux-x86_64-3.8
gcc -pthread -B /home/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/anaconda3/include/python3.8 -c cc1.cpp -o build/temp.linux-x86_64-3.8/cc1.o -O3 -w -fopenmp
gcc -pthread -B /home/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/anaconda3/include/python3.8 -c cc1.cpp -o build/temp.linux-x86_64-3.8/cc1.o -O3 -w -fopenmp
g++ -pthread -shared -B /home/anaconda3/compiler_compat -L/home/anaconda3/lib -Wl,-rpath=/home/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/cc1.o -o /home/Desktop/DRAFT_CODING/Python/return_many_values_with_cyt/cc1.cpython-38-x86_64-linux-gnu.so -fopenmp -ffast-math -march=native
building 'cc2' extension
gcc -pthread -B /home/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/anaconda3/lib/python3.8/site-packages/numpy/core/include -I/home/anaconda3/include/python3.8 -c cc2.cpp -o build/temp.linux-x86_64-3.8/cc2.o -O3 -w -fopenmp
g++ -pthread -shared -B /home/anaconda3/compiler_compat -L/home/anaconda3/lib -Wl,-rpath=/home/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/cc2.o -o /home/Desktop/DRAFT_CODING/Python/return_many_values_with_cyt/cc2.cpython-38-x86_64-linux-gnu.so -fopenmp -ffast-math -march=native
Now, that I have transferred my project into my Raspberry Pi 4, everything works fine as well, but when I start the setuptools, by launching the same command as usual on the remote terminal, for the same setup.py, this time nothing is written on the terminal, while the script is cythoninzing the source code.
Launch python with the 'verbose' option is not the best...
In this case my virtualenv is based on Python 3.9. with gcc 10.21.
Can anyone tell me what is happening or which is the culprit?
How can I capture again the classic output messages I have always seen until today?
It is not the end of the world, but it was convenient to follow the status of the execution, especially when creating many shared objects at the same time.
I am trying to build python extension with Cython with the following project structure:
include/
algorithm/
LessonplanScheduler.hpp
src/
LessonplanScheduler.cpp
GenAlgorithm.cpp
GenAlgorithm.hpp
LessonplanGenAlgorithm.cpp
LessonplanGenAlgorithm.cpp
__init__.py
algorithm.pyx
LessonplanScheduler.pxd
setup.py
setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name='Lessonplan scheduler algorithm',
ext_modules=cythonize(
[
Extension(
name="algorithm",
include_dirs=["./include/algorithm"],
library_dirs=["./src"],
sources=["algorithm.pyx", "algorithm.cpp"],
)
],
language_level=3
),
requires=['Cython'],
)
After running python ./setup.py build_ext --inplace, I am getting build folder in project root and algorithm.cpp also there. But .so file is not built because of multiple definition errors:
Compiling algorithm.pyx because it changed.
[1/1] Cythonizing algorithm.pyx
running build_ext
building 'algorithm' extension
creating build
creating build/temp.linux-x86_64-3.7
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I. -I./include/algorithm -I/usr/local/include/python3.7m -c algorithm.cpp -o build/temp.linux-x86_64-3.7/algorithm.o
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I. -I./include/algorithm -I/usr/local/include/python3.7m -c algorithm.cpp -o build/temp.linux-x86_64-3.7/algorithm.o
creating build/lib.linux-x86_64-3.7
g++ -pthread -shared build/temp.linux-x86_64-3.7/algorithm.o build/temp.linux-x86_64-3.7/algorithm.o -L./src -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/algorithm.cpython-37m-x86_64-linux-gnu.so
/usr/bin/ld: build/temp.linux-x86_64-3.7/algorithm.o:(.bss+0x0): multiple definition of `__pyx_module_is_main_algorithm'; build/temp.linux-x86_64-3.7/algorithm.o:(.bss+0x0): first defined here
/usr/bin/ld: build/temp.linux-x86_64-3.7/algorithm.o: in function `PyInit_algorithm':
/backend/backend/algorithm/algorithm.cpp:1966: multiple definition of `PyInit_algorithm'; build/temp.linux-x86_64-3.7/algorithm.o:/backend/backend/algorithm/algorithm.cpp:1966: first defined here
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
The problem, as I suppose, lives in algorithm.cpp file, so it is probably caused by wrong build configuration. What should I do to fix the problem and what would be the best way to build the project?
I transfered to Linux from win10. It should be all easy and etc to deal with libs and building on linux... but this simple code do not build. I know how to use cython commands to build (from its tutorial) and it worker in python with msvc=2008, 2013, 2015. But this damn linux is somehow not so nice here ;/ and wont build
helloWorld.pyx
#cython: language_level=2
print "hello world!"
Or (same error)
helloWorld.pyx
#cython: language_level=3
print("hello world!")
Im building it with (this ${fileDirname} is from visual code task):
zsh -c ". activate py35a && gcc --version && python ${fileDirname}/setup.py build_ext --inplace"
setup.py
from __init__ import *
try:
from setuptools import setup
from setuptools import Extension
except ImportError:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Compiler.Options import directive_defaults
from Cython.Build import cythonize
import numpy
import os,sys
from sys import platform
path=current_path()
os.chdir(path)
print("CWD: %s"%path)
#this is one of my old functions (like from being a NOOB), it was making error, now I changed it (so it do not add second file with .cpp extension)...
def sources(*lst):
lst=list(lst)
for n in range(len(lst)):
lst[n]+='.pyx'
return lst
ext_modules=[
Extension("C_helloWorld",
sources=sources("helloWorld"),
),
]
for extension in ext_modules:
extension.include_dirs=[numpy.get_include()]
extension.language='c++'
#extension.extra_compile_args=['-g']
#extension.extra_link_args=['-g', '-pthread']
setup(
name = 'simple test of cython',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)
This is error I get:
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
No module named 'adjustText'
CWD: /home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/
Compiling helloWorld.pyx because it changed.
[1/1] Cythonizing helloWorld.pyx
running build_ext
building 'C_helloWorld' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib/python3.5/site-packages/numpy/core/include -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/include/python3.5m -c helloWorld.cpp -o build/temp.linux-x86_64-3.5/helloWorld.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib/python3.5/site-packages/numpy/core/include -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/include/python3.5m -c helloWorld.cpp -o build/temp.linux-x86_64-3.5/helloWorld.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
g++ -pthread -shared -L/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib -Wl,-rpath=/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib,--no-as-needed build/temp.linux-x86_64-3.5/helloWorld.o build/temp.linux-x86_64-3.5/helloWorld.o -L/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib -lpython3.5m -o /home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/C_helloWorld.cpython-35m-x86_64-linux-gnu.so -pthread
build/temp.linux-x86_64-3.5/helloWorld.o: In function `PyInit_helloWorld':
/home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/helloWorld.cpp:698: multiple definition of `PyInit_helloWorld'
build/temp.linux-x86_64-3.5/helloWorld.o:/home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/helloWorld.cpp:698: first defined here
build/temp.linux-x86_64-3.5/helloWorld.o: In function `PyInit_helloWorld':
/home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/helloWorld.cpp:698: multiple definition of `__pyx_module_is_main_CTO__A2_CROSS_SECTIONS__helloWorld'
build/temp.linux-x86_64-3.5/helloWorld.o:/home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/helloWorld.cpp:698: first defined here
collect2: error: ld returned 1 exit status
error: command 'g++' failed with exit status 1
Solved by changing function sources( it appears, that I was totally my error, not connected with linux and cython, shame me ;/)
Now it gives:
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
3.5.2 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
No module named 'adjustText'
CWD: /home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/
Compiling hello_world.pyx because it changed.
[1/1] Cythonizing hello_world.pyx
running build_ext
building 'c_hello_world' extension
gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib/python3.5/site-packages/numpy/core/include -I/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/include/python3.5m -c hello_world.c -o build/temp.linux-x86_64-3.5/hello_world.o
gcc -pthread -shared -L/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib -Wl,-rpath=/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib,--no-as-needed build/temp.linux-x86_64-3.5/hello_world.o -L/home/robgrzel/ProgramFiles/Programming/Python/Miniconda3/envs/py35a/lib -lpython3.5m -o /home/robgrzel/SHARED/GOOGLE/WORKSPACE/PROJECTS/SHIPPING/CTO/A2_CROSS_SECTIONS/c_hello_world.cpython-35m-x86_64-linux-gnu.so -pthread
Tada... But this extension name is bad: c_hello_world.cpython-35m-x86_64-linux-gnu.so
I want use Twisted in Python, but when I installing ,in comes this error, how to handle it?
....
running build_ext
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c conftest.c -o conftest.o
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c conftest.c -o conftest.o
building 'twisted.runner.portmap' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c twisted/runner/portmap.c -o build/temp.linux-i686-2.7/twisted/runner/portmap.o
twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
You are missing the python development headers, needed to build packages that need to compile extensions.
If you are building on OSX, make sure you either install a prebuilt mac python package or if building python from source, use the framework flag when configuring. And also make sure you have XCODE installed so that you have a compiler.
If you are building on Linux, you probably need to install the python devel headers. For instance on Ubuntu you would need: apt-get install build-essential python-dev.
Once you have the python development headers, twisted should be able to find them when you build with that python interpreter.
I am walking through the Cython documentation and building each of the example apps. I'm a little stuck at Using C Libraries. After successfully building the .so file and attempting to import it in a python file called test.py the following error is thrown.
$ python3.2 test.py
Traceback (most recent call last):
File "test.py", line 12, in <module>
from queue import Queue
ImportError: dlopen(/Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so, 2): Symbol not found: _queue_free
Referenced from: /Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so
Expected in: flat namespace
in /Users/jeremy/Development/labs/python/cython_lib_wrapper/queue.so
The .so file sits right next to the test.py file. So, it seems as though it should be found.
This is running the latest version of Cython, with Python 3.2 on a OSX 10.6.
Any insights?
Edit - adding build command and output
$ python3.2 setup.py build_ext --inplace
running build_ext
cythoning queue.pyx to queue.c
building 'queue' extension
gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 -isysroot /Developer/SDKs/MacOSX10.6.sdk -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -I/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m -c queue.c -o build/temp.macosx-10.6-intel-3.2/queue.o
queue.c: In function ‘__pyx_f_5queue_5Queue_append’:
queue.c:627: warning: cast to pointer from integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_extend’:
queue.c:740: warning: cast to pointer from integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_peek’:
queue.c:813: warning: cast from pointer to integer of different size
queue.c: In function ‘__pyx_f_5queue_5Queue_pop’:
queue.c:965: warning: cast from pointer to integer of different size
gcc-4.2 -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -isysroot /Developer/SDKs/MacOSX10.6.sdk -g build/temp.macosx-10.6-intel-3.2/queue.o -o
Edit 2 - adding "otool" cmd requested in comment
queue.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
Edit 3 - adding "nm" output
U ___stack_chk_fail
U ___stack_chk_guard
U _queue_free
U _queue_is_empty
U _queue_new
U _queue_peek_head
U _queue_pop_head
U _queue_push_tail
U dyld_stub_binder
grep cmd outputs this:
(undefined) external _queue_free (dynamically looked up)
EDIT:
Ah, you didn't mention you had a dependency on the code in libcalg. That stuff needs to be compiled and included when you build the cextension.
Just modify setup.py:
# setup.py
# ...
ext_modules = [Extension("queue", ["queue.pyx", "libcalg/queue.c"])]
# ...
We could step back and see if you can build a really simple example:
I've tried the following (3 files, myext.pyx, test.py, setup.py) and it appears to work fine. Of course I'm on OS X 10.7 so it's not exactly the same as your environment. To rule out differences perhaps you can copy these and build them as a sanity check.
Contents of myext.pyx:
# myext.pyx
def square(x):
return x * x
Contents of test.py
# test.py
from myext import square
print "%d squared is %d"%(4, square(4))
Contents of setup.py:
# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules = [Extension("myext", ["myext.pyx"])]
setup(
name = 'Hello world app',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
I build in the directory containing these 3 files:
cython_test$ /usr/bin/python setup.py build_ext --inplace
running build_ext
cythoning myext.pyx to myext.c
building 'myext' extension
creating build
creating build/temp.macosx-10.7-intel-2.7
llvm-gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c myext.c -o build/temp.macosx-10.7-intel-2.7/myext.o
llvm-gcc-4.2 -Wl,-F. -bundle -undefined dynamic_lookup -Wl,-F. -arch i386 -arch x86_64 build/temp.macosx-10.7-intel-2.7/myext.o -o /Users/steder/SO/cython_test/myext.so
cython_test$ python test.py
4 squared is 16:
My environment has similar otool output and DYLD_LIBRARY_PATH is also unset but nm -m shows squared as defined.
Specifically:
00000000000011d0 (__DATA,__data) non-external ___pyx_k__square
00000000000011e0 (__DATA,__data) non-external ___pyx_mdef_5myext_square
0000000000001218 (__DATA,__bss) non-external ___pyx_n_s__square
0000000000000c80 (__TEXT,__text) non-external ___pyx_pf_5myext_square
Please give this a shot and see what it nm -m shows in your environment.