Develop Reference

How to get Cython to switch a compiler? [duplicate]

I want to wrap a test project containing C++ and OpenMP code with Cython, and build it with distutils via a setup.py file. The content of my file looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
modules = [Extension("Interface",
["Interface.pyx", "Parallel.cpp"],
language = "c++",
extra_compile_args=["-fopenmp"],
extra_link_args=["-fopenmp"])]
for e in modules:
e.cython_directives = {"embedsignature" : True}
setup(name="Interface",
cmdclass={"build_ext": build_ext},
ext_modules=modules)
The -fopenmp flag is used with gcc to compile and link against OpenMP. However, if I just invoke
cls ~/workspace/CythonOpenMP/src $ python3 setup.py build
this flag is not recognized, because the compiler is clang:
running build
running build_ext
skipping 'Interface.cpp' Cython extension (up-to-date)
building 'Interface' extension
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Interface.cpp -o build/temp.macosx-10.8-x86_64-3.3/Interface.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
cc -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c Parallel.cpp -o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -fopenmp
clang: warning: argument unused during compilation: '-fopenmp'
Parallel.cpp:24:10: warning: unknown pragma ignored [-Wunknown-pragmas]
#pragma omp parallel for
^
1 warning generated.
c++ -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-3.3/Interface.o build/temp.macosx-10.8-x86_64-3.3/Parallel.o -o build/lib.macosx-10.8-x86_64-3.3/Interface.so -fopenmp
ld: library not found for -lgomp
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'c++' failed with exit status 1
I've unsucessfully tried to specify gcc:
cls ~/workspace/CythonOpenMP/src $ python3 setup.py build --compiler=g++-4.7
running build
running build_ext
error: don't know how to compile C/C++ code on platform 'posix' with 'g++-4.7' compiler
How can I tell distutils to use gcc?

Try setting the "CC" environment variable from inside the setup.py with os.environ.

I just took a look at the distutils source, and the --compiler option expects "unix", "msvc", "cygwin", "mingw32", "bcpp", or "emx". It checks the compiler name you want by checking the CC environment variable. Try calling build like this:
CC=gcc python setup.py build
You don't need to set CXX, it doesn't check for that.

Just in case some others are facing the same problem under Windows (where CC environment variable wouldn't have any effect) :
Create file "C:\Python27\Lib\distutils\distutils.cfg" and write this inside :
Code :
[build]
compiler = mingw32
Remove all instances of "-mno-cygwin" gcc option from file "C:\Python27\Lib\distutils\cygwinccompiler.py" :
This :
self.set_executables(compiler='gcc -mno-cygwin -O -Wall',
compiler_so='gcc -mno-cygwin -mdll -O -Wall',
compiler_cxx='g++ -mno-cygwin -O -Wall',
linker_exe='gcc -mno-cygwin',
linker_so='%s -mno-cygwin %s %s'
% (self.linker_dll, shared_option,
entry_point))
Becomes this :
self.set_executables(compiler='gcc -O -Wall',
compiler_so='gcc -mdll -O -Wall',
compiler_cxx='g++ -O -Wall',
linker_exe='gcc',
linker_so='%s %s %s'
% (self.linker_dll, shared_option,
entry_point))
The second point can be necessary in case you are using a recent version of gcc, where the deprecated option -mno-cygwin has been removed.
Hope this will help even if it is not directly related to the OP real needs (but still related to the question's title...)

According to this wiki, Python versions after 3.4 do not support MinGW anymore.
CPython 3.7 for Windows is compiled with MSC v.1916. When I try to use above-mentioned method with distutils.cfg, I then get an error from distutils: Unknown MS Compiler Version 1916. Looks like it has a hardcoded table of msvcr libraries in its cygwincompiler.py file (which is also responsible for MinGW), and last version known to that file is 1600 from VS2010 / MSVC 10.0.

Try this:
http://mail.python.org/pipermail/distutils-sig/2002-August/002944.html
In short, it appears that you should try: python setup.py build --compiler=g++ first.

On linux while using distutils.ccompiler do
os.environ('CC')='gcc' and then call distutils.sysconfig.customize_compiler(compiler)
It will do the job.

Problem when linking against Python 3.8 libs in gcc

I am trying to compile the example C application embedding Python (from here https://docs.python.org/3.8/extending/embedding.html)
#define PY_SSIZE_T_CLEAN
#include <Python.h>
int
main(int argc, char *argv[])
{
wchar_t *program = Py_DecodeLocale(argv[0], NULL);
if (program == NULL) {
fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
exit(1);
}
Py_SetProgramName(program); /* optional but recommended */
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print('Today is', ctime(time()))\n");
if (Py_FinalizeEx() < 0) {
exit(120);
}
PyMem_RawFree(program);
return 0;
}
I'm using gcc options generated by python3.8-config.
Thus, compile and link are done as follows:
gcc $(python3.8-config --cflags) -c embePy.c -o embePy.o
gcc $(python3.8-config --ldflags) -o embePy.o
Compiling goes smoothly, whereas linking throws an error:
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/Scrt1.o: in function `_start':
(.text+0x24): undefined reference to `main'
collect2: error: ld returned 1 exit status
Just in case:
$ /usr/bin/python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-CoVRmP/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security -DNDEBUG -g -fwrapv -O3 -Wall
$ /usr/bin/python3-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl -lutil -lm -lm
I don't have experience with manual control of gcc's options, all what I've done was compilation from inside IDE's with automatically set flags. Can anybody help me to identify the problem? Thank you.

I solved the problem, thank #deamentiaemundi and the post from here: https://stackoverflow.com/a/27672776/9256844
To avoid "relocation R_X86_64_32", compile with -fPIE:
gcc $(python3.8-config --cflags) -fPIE -c embePy.c -o embePy.o
To link against the Python libs, I had to put my object file before the python's flags AND manually add -lpython3.8:
gcc embePy.o $(python3.8-config --ldflags) -lpython3.8 -o embePy
Looks like my version of Python mistakenly outputs duplicated flag -lm instead of -lpython3.8 (Python 3.8.10):
$ python3.8-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl -lutil -lm -lm

Calling cv::cuda::HOG from python

I need to access the cuda modules of opencv in C++ from python. I tried wrapping the c++ functions using Boost however I get the following error.
ImportError: ../build/lib.linux-x86_64-2.7/pyBoostHOG.so: undefined
symbol: _ZN2cv4cuda6Stream4NullEv.
Is there any way by which I can access the opencv cuda modules from python?
Thanks in advance..

I was facing the same error. And this is how I was compiling the .so file:
g++ `pkg-config opencv4 --cflags --libs` -fPIC -c flow.cpp
g++ -Wall -O3 -shared -o flow.so flow.o `pkg-config opencv4 --cflags --libs`
The error was resolved when I changed my make file to the following:
g++ `pkg-config opencv4 --cflags --libs` -fPIC -c flow.cpp
g++ -Wall -O3 -shared flow.o -o flow.so `pkg-config opencv4 --cflags --libs`

I had the same error.
I figured out that I must include opencv2/core/cuda.hpp file (symbol _ZN2cv4cuda6Stream4NullEv reffers to: cv::cuda::Stream::Null()
But this doesn't solve the problem, because when I tried to make wrap thought swig and include these file, new error:
.../include/opencv2/core/cuda.hpp:317: Error: Syntax error in input(1).
is displayed.
Compiling another program using cv::cuda::Stream::Null() cause not error.

g++ linking and swig

I have a cpp file with functions that I'm using in python with SWIG. I use the following commands to compile the source and create the file to use with python.
swig -c++ -python mini.i
g++ -O2 -c mini.cpp -I/usr/include/python2.4 -I/usr/lib/python2.4
g++ -O2 -c mini_wrap.cxx -I/usr/include/python2.4 -I/usr/lib/python2.4
g++ -shared mini.o mini_wrap.o -o _mini.so
I'm trying now to use GSL in my source cpp source file. If I was just compiling the GSL file I would do
g++ -lgsl -lgslcblas -lm -o mini.o mini.cpp
I've tried adding the -lgsl -lgslcblas -lm to the lines for the swig compile but I get
g++: -lgsl: linker input file unused because linking not done
g++: -lgslcblas: linker input file unused because linking not done
g++: -lm: linker input file unused because linking not done
How can I link the gsl libraries? Thanks

Swig does no linking, as the warning message states. Put the -lgsl etc. on the link command, which is your last g++ command.

Linking GSL (or other library) statically into a shared library

Note: Despite the mentioning of Python in the following there is a good chance for my problem not to be Python related at all. If I am not mistaken the “module” I mention is equivalent to a C library—at least for the concerns of my problem.
On Debian I am trying to create a Python module with C, which in turn uses the GSL. The following Makefile successfully compiles it:
CC = gcc -Wall -fPIC -O3
NAME = meinzeug
matrizenwuerfler: $(SRC)
$(CC) -o $(NAME).o -I/usr/lib/python2.5/site-packages/numpy/core/include -I/usr/include/python2.5 -c $(NAME).c
$(CC) -shared -o $(NAME).so -lgsl -lgslcblas -lm $(NAME).o
Because this module is supposed to be used by (Linux) machines other than mine, I want the GSL to be included into the module (or be shipped with it).
However, if I add -static as option to the last line of the Makefile, I get the following error:
gcc -Wall -fPIC -O3 -shared -static -o meinzeug.so -lgsl -lgslcblas -lm meinzeug.o
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/4.3.2/crtbeginT.o: relocation R_X86_64_32 against `__DTOR_END__' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/4.3.2/crtbeginT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
Adding -Wl,-Bstatic before the library linking results in a different error:
gcc -Wall -fPIC -O3 -shared -o meinzeug.so -Wl,-Bstatic -lgsl -lgslcblas -lm meinzeug.o
/usr/bin/ld: cannot find -lgcc_s
collect2: ld returned 1 exit status
Other Stuff, that did not work: Recompiling GSL with fPIC, -static-libgcc, permutating the options.
What I did not try yet, is compiling gcc with fPIC or similar.

Try
gcc -Wall -fPIC -O3 -shared -o meinzeug.so /usr/lib/libgsl.a -lm meinzeug.
as you cannot do
gcc -Wall -fPIC -O3 -shared -static ... # shared and static at the same time ?
so you would provide the static library of GSL alongside with your code.
At the end of the day, I would punt and keep the dependency on the GSL. Just about everybody has it, and the API is pretty stable.

The ordering of the library calls is important. For me, it meant sending the /usr/lib/libgsl.a to the end of the command. That solved it.