I have a setup.py file that is very similar to the one shown here: https://stackoverflow.com/a/49866324/4080129. It looks like this:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy
sources = ["hs/subfolder/detect.pyx",
"hs/subfolder/Donline.cpp",
"hs/subfolder/Handler.cpp",
"hs/subfolder/Process.cpp",
"hs/subfolder/Filter.cpp",
"hs/subfolder/Localize.cpp"]
exts = [Extension(name='hs.detect',
sources=sources,
extra_compile_args=['-std=c++11', '-O3'],
include_dirs=[numpy.get_include()])]
setup(
ext_modules=cythonize(exts),
include_dirs=[numpy.get_include()]
)
There's a package with some pure-Python, and a submodule that contains Cython files. The setup.py is in the parent folder, not in the Cython one:
setup.py
hs/
some_python.py
subfolder/
detect.pyx
Donline.cpp
...etc
Now, setup.py correctly compiles all the files module/submodule/file1.cpp etc. and saves the build to build/temp.linux-x86_64-3.6/module/submodule/file1.o .
However, just after that, it tries to compile a file called file1.cpp, which doesn't exist (the correct one is module/submodule/file1.cpp, and has already been compiled).
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ihs/subfolder -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/python3.6/site-packages/numpy/core/include -I/[...]/include -I/disk/scratch/miniconda/envs/my_default/include/python3.6m -c Donline.cpp -o build/temp.linux-x86_64-3.6/Donline.o -std=c++11 -O3
gcc: error: Donline.cpp: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4
I'm very confused, this completely prevents my code from compiling...
It turns out the .pyx file contains a line
# distutils: sources = Donline.cpp Handler.cpp Process.cpp Filter.cpp Localize.cpp
which tells distutils what to compile. I wasn't aware of it, and since it looks an awful lot like a commented-out line, I didn't realise it was there.
Cython tries to compile also these, other than the ones contained in the setup.pyfile, i.e. neither of the two sources list overrides the other. Apparently, these sources, despite being listed in the pyx file, which is in a subfolder, are expected to be in paths relative to the file where the setup.py file is, or perhaps relative to the folder I'm calling python from.
Anyway, removing the line solved the issue.
Related
I am an Ultra Linux newbie, and I am trying to install this program and when I try to build the python wrapper I'd get this
~/Downloads/DeepMimic-master/DeepMimicCore$ make python
clang++ -c -g -std=c++11 -O3 -Wall -fPIC -I./ -I../../libraries/eigen -I../../libraries/bullet3/src -I/usr/include/python3.6m -I/usr/lib/ -lpython3.6m -o objs/Main.o Main.cpp
clang: warning: -lpython3.6m: 'linker' input unused [-Wunused-command-line-argument]
In file included from Main.cpp:3:
In file included from ./DeepMimicCore.h:3:
In file included from ./util/ArgParser.h:6:
./util/MathUtil.h:5:10: fatal error: 'Eigen/Dense' file not found
#include "Eigen/Dense"
You're missing a dependency, Eigen, which is listed under 'Dependencies' in the DeepMimic readme.
I see this problem has been encountered before:
fatal error: Eigen/Dense: No such file or directory
Looks like that program depends on Eigen. Try download Eigen and putting it in the appropriate directory.
Eigen is a template library, so you just have to download it, unzip it, and copy the folder called Eigen inside of the directory of the program.
Eigen website
Is it possible to disable the creation of shared objects with distutils.core.Extension? I want to stop the compiler before linking (i.e. g++ -c ...).
I am swigging a native file, which creates an object file and a python file. I have other code to compile that I'll later link with this object file, so I don't want this to proceed after the .o compilation.
$ python setup.py build
running build
....
building 'foo' extension
swigging src/foobar.i to src/foobar.cpp
swig -python -c++ -o src/foobar.cpp src/foobar.i
I want to stop here, but it continues.
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -Isrc -I/usr/include/python2.7 -c src/foobar.cpp -o build/temp.linux-x86_64-2.7/src/foobar.o
g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/src/foobar.o -o build/lib.linux-x86_64-2.7/foobar.so
Do I need to use the CCompiler class directly? Or is there a way to wrangle the Extension class?
23 ext_modules=[
24 # Swig
25 Extension(
26 name='foobar',
27 sources=['src/foobar.i'],
28 include_dirs=['src'],
29 swig_opts=['-c++'],
30 ),
31 ]
It is not possible to stop the linking step without modifying the underlying ccompiler object. One could theoretically override the link_shared_object function of the underlying ccompiler to do nothing (See the build_ext source).
However, to answer the original intent behind this question, the C/C++ files can be passed to the Extension with the Swig interface file without needing to compile them independently and link later. It is not necessary to separate the swig file generation and the library compilation.
You could do something like this:
from distutils.command import build_ext
def cmd_ex(command_subclass):
orig_ext = command_subclass.build_extension
def build_ext(self, ext):
sources = self.swig_sources(list(ext.sources), ext)
command_subclass.build_extension = build_ext
return command_subclass
#cmd_ex
class build_ext_ex(build_ext):
pass
setup(
name = ...,
cmdclass = {'build_ext': build_ext_ex},
ext_modules = ...
)
to override the default behavior of distutils command.
Setuptools – run custom code in setup.py
I'm trying to compile a Python wrapper to a small C++ library I've written. I've written the following setup.py script to try to use setuptools to compile the wrapper:
from setuptools import setup, Extension
import numpy as np
import os
atmcmodule = Extension(
'atmc',
include_dirs=[np.get_include(), '/usr/local/include'],
libraries=['mcopt', 'c++'], # my C++ library is at ./build/libmcopt.a
library_dirs=[os.path.abspath('./build')],
sources=['atmcmodule.cpp'],
language='c++',
extra_compile_args=['-std=c++11', '-v'],
)
setup(name='tracking',
version='0.1',
description='Particle tracking and MC optimizer module',
ext_modules=[atmcmodule],
)
However, when I run python setup.py build on OS X El Capitan, clang complains about not finding some C++ standard library headers:
In file included from atmcmodule.cpp:7:
In file included from ./mcopt.h:11:
In file included from ./arma_include.h:4:
/usr/local/include/armadillo:54:12: fatal error: 'initializer_list' file not found
#include <initializer_list>
^
1 error generated.
error: command 'gcc' failed with exit status 1
Passing the -v flag to the compiler shows that it is searching the following include paths:
#include <...> search starts here:
/Users/[username]/miniconda3/include
/Users/[username]/miniconda3/lib/python3.4/site-packages/numpy/core/include
/usr/local/include
/Users/[username]/miniconda3/include/python3.4m
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/c++/4.2.1
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/c++/4.2.1/backward
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.0/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks (framework directory)
End of search list.
This apparently doesn't include the path to the C++ standard library headers. If I compile a small test C++ source with the -v option, I can see that clang++ normally also searches the path /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1, and if I include this path in the include_dirs option for Extension in my setup.py script, then the extension module compiles correctly and works. However, hard-coding this path into the script doesn't seem like a good solution since this module also needs to work on Linux.
So, my question is how do I properly make setuptools include the required headers?
Update (11/22/2015)
As setuptools tries to compile the extension, it prints the first command it's running:
gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/[username]/miniconda3/include -arch x86_64 -I/Users/[username]/miniconda3/lib/python3.4/site-packages/numpy/core/include -I/Users/[username]/Documents/Code/ar40-aug15/monte_carlo/mcopt -I/usr/local/include -I/Users/[username]/miniconda3/include/python3.4m -c /Users/[username]/Documents/Code/ar40-aug15/monte_carlo/atmc/atmcmodule.cpp -o build/temp.macosx-10.5-x86_64-3.4/Users/[username]/Documents/Code/ar40-aug15/monte_carlo/atmc/atmcmodule.o -std=c++11 -fopenmp -v
If I paste this command into a terminal and run it myself, the extension compiles successfully. So I suspect either setuptools is modifying some environment variables I'm not aware of, or it's lying a little about the commands it's actually running.
Setuptools tries to compile C/C++ extension modules with the same flags used to compile the Python interpreter. After checking the flags used to compile my Python install (from Anaconda), I found it was compiling for a minimum Mac OS X version of 10.5. This seems to make it use the GCC libstdc++ instead of clang's libc++ (which supports C++11).
This can be fixed by either setting the environment variable MACOSX_DEPLOYMENT_TARGET to 10.9 (or later), or adding '-mmacosx-version-min=10.9' to extra_compile_args.
(UPDATE 3 contains the questions I'd like to get answers to. UPDATE 2 refers to corrections I did trying to understand and fix this issue)
I'm trying to get a Python extension module to wrap a C library (in this case, it is only an example described in the book Python Cookbook (3er edition.) The problem is that I'm encountering the classic error
ImportError: dynamic module does not define init function (initsample)
when I'm trying to import the module.
The book itself has the following comment:
For all of the recipes that follow, assume that the preceding code is found in a file named
sample.c, that definitions are found in a file named sample.h and that it has been com‐
piled into a library libsample that can be linked to other C code. The exact details of
compilation and linking vary from system to system, but that is not the primary focus.
It is assumed that if you’re working with C code, you’ve already figured that out.
And I think I'm getting that wrong. This is what I'm doing: First of all, I have the following files:
➜ Sample ls
csample.pxd sample.c sample.h sample.pyx setup.py
The content of the files is according to this github repository containing the files described in the book. Assuming everything is copied correctly. I then compile sample.c
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
This creates sample.o and I proceed to create a shared library:
➜ Sample gcc -shared sample.o -o libsample.so
It creates a libsample.so and finally run the setup.py file:
➜ Sample python setup.py build_ext --inplace
running build_ext
skipping 'sample.c' Cython extension (up-to-date)
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/sample.so
However, I obtain the same error. What am I missing?
This is my setup.py (eventually, I copied a part to clean previous build):
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import sys
import os
import shutil
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("sample") and not(name.endswith(".pyx") or name.endswith(".pxd") or name.endswith(".h"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
ext_modules = [
Extension('sample',
['sample.pyx'],
libraries=['sample'],
library_dirs=['.'])]
setup(
name = 'Sample extension module',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
)
UPDATE 2
I was wondering if it is normal that after running
python setup.py build_ext --inplace
the file sample.c gets replaced with a cythonized version.
I assume that running setup.py is using cython sample.c, but I'm asking this because yesterday I was mistakenly compiling the cythonized version sample.c (after removing libsample.so, sample.o, sample.so that were created in a previous run) and after updating sample.pyx, I got this error:
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./sample.so: undefined symbol: divide
After this, I restored sample.c to its original content, removed all remaining files from previous builds (including temporal files) and crucially, I had to change sample.pyx in some way (e.g. adding a blank line.)
I noticed I have to do this to avoid the
ImportError: dynamic module does not define init function (initsample)
error. Then I proceeded as usual:
➜ Sample gcc -c -fPIC -I/path/to/python2.7 sample.c
➜ Sample gcc -shared sample.o -o libsample.so
➜ Sample python setup.py build_ext --inplace
running build_ext
cythoning sample.pyx to sample.c
warning: sample.pyx:27:42: Use boundscheck(False) for faster access
building 'sample' extension
creating build
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/path/to/include/python2.7 -c sample.c -o build/temp.linux-x86_64-2.7/sample.o
gcc -pthread -shared build/temp.linux-x86_64-2.7/sample.o -L. -L/path/to/lib -lsample -lpython2.7 -o /path/to/Sample/sample.so
➜ Sample python
>>> import sample
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libsample.so: cannot open shared object file: No such file or directory
This time I get cythoning sample.pyx to sample.c in the output above and a new error. However, libsample.so is in the Sample directory. What could be the issue here?
UPDATE 3:
I finally was able to import this module successfully but the last error
ImportError: libsample.so: cannot open shared object file: No such file or directory
implied that I needed to set the current directory in the environment variable LD_LIBRARY_PATH. However, I did it in a crude way:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/Sample
What is the correct way to do this? Maybe using some parameter in setup.py? Regarding the several issues I found, I'd like to get an answer as to why all of this happened:
sample.c gets replaced when running python setup.py build_ext --inplace Is this okay?
Do I need to restore sample.c every time I'm doing this procedure?
Do I need to change sample.pyx in order to get a correct output (instead of skipping 'sample.c' Cython extension (up-to-date) with a misleading output)
What is the correct way to solve ImportError: libsample.so: cannot open shared object file: No such file or directory?
A possible solution to 4. is to add in setup.py:
runtime_library_dirs=['./'],
However, this requires libsample.so to be located in the same directory as setup.py.
I am trying to read and learn the pytho2.5.6 source code, and try to recompile the code everytime i make some change on it.
But it doesn't seem to work when I comile the source code via the following commands:
./configure --prefix=/path/to/somewhere/that/don't/messup/myenv/
make
After make, the console just show errors like this:
cc1: error: unrecognized command line option "-Wno-long-double"
I search the google , and it seems like the gcc(4.2.1) that i am using is not compatiable with the python2.5.6.
some info of my sys:
gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Xcode 4.6.2(installed via the xcode command cline tool )
python2.5.6 source code(download from python.org)
I have try some solutions like these:
1.download diff python code version , like 2.5.4, and do the steps again.
2.after configure,modify the Makefile,and remove the --Wno-long-double,but new errors showed like below:
➜ Python-2.5.6 make
gcc -c -fno-strict-aliasing -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o Python/mactoolboxglue.o Python/mactoolboxglue.c
In file included from Include/Python.h:57,
from Python/mactoolboxglue.c:26:
Include/pyport.h:547: warning: ‘struct winsize’ declared inside parameter list
Include/pyport.h:547: warning: its scope is only this definition or declaration, which is probably not what you want
Include/pyport.h:548: warning: ‘struct winsize’ declared inside parameter list
In file included from /System/Library/Frameworks/CoreFoundation.framework/Headers/CFBase.h:67,
from /System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:38,
from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:18,
from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20,
from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:18,
from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20,
from Include/pymactoolbox.h:10,
from Python/mactoolboxglue.c:27:
/usr/include/AvailabilityMacros.h:109:14: warning: #warning Building for Intel with Mac OS X Deployment Target < 10.4 is invalid.
In file included from Python/mactoolboxglue.c:27:
.......
Python/mactoolboxglue.c: In function ‘MediaObj_Convert’:
Python/mactoolboxglue.c:431: error: ‘cobj’ undeclared (first use in this function)
Python/mactoolboxglue.c:431: error: too many arguments to function ‘PyMacGluePtr_MediaObj_Convert’
make: *** [Python/mactoolboxglue.o] Error 1
So, i wonder if there are some way to intall gcc4.0 manually?
PS:the reason why i use python2.5.6 is because i read a book called which is written in chinese.Or maybe the best solution is to read the 2.7 source code =.=....hmmm.....