When I compile a cython .pyx file from IdleX the build shell window pops up with a bunch of warnings to close again after less than a second.
I think pyximport uses distutils to build. How can I write the gcc warnings to a file or have the output delay or wait for keypress?
You can add a .pyxbld file to specify Cython build settings.
Say you are trying to compile yourmodule.pyx, simply create a file in the same directory named yourmodule.pyxbld containing:
def make_ext(modname, pyxfilename):
from distutils.extension import Extension
ext = Extension(name = modname,
sources=[pyxfilename])
return ext
def make_setup_args():
return dict(script_args=['--verbose'])
The --verbose flag makes pyximport print gcc's output.
Note that you can easily add extra compiler and linker flags. For example, to use Cython's prange() function you must compile and link against the OpenMP library, this is specified using keywords to the Extension class:
ext = Extension(name = modname,
sources=[pyxfilename],
extra_compile_args=['-fopenmp'],
extra_link_args=['-fopenmp'])
I haven't done anything with cython myself but I guess you could use a commandline for the building. That way you would see all the messages until you close the actual commandline unless some really fatal error happens.
Yes, here is how you could do to get the output in a file (in case the warnings overflow the command prompt) :
This is for Windows, but you can easily adapt for *unices :
SET PYTHONPATH=%PYTHONPATH%;../../../DEPENDENCIES/Cython-0.18
SET PATH=%PATH%;../../../DEPENDENCIES/Cython-0.18/bin
python setup.py build_ext --inplace > output.log
Related
I have read How to enable `--embed` with cythonize? and Cython --embed flag in setup.py but this method does not work on Windows. Let's say we have:
# app.py
print("hello")
# build.py
import setuptools # allows us to avoid calling vcvarsall.bat, see https://stackoverflow.com/a/53172602/
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.embed = "main"
setup(ext_modules=cythonize(r'app.py', build_dir="release"), script_args=['build'], options={'build': {'build_lib': 'release'}})
Running this build.py script on Python 3.8 for Windows does not produce an app.exe file like it would with the command line command:
cython app.py --embed
Instead, it produces a .pyd file.
How to use cythonize + embed from a Python script, producing a .exe, on Windows?
Solved: in fact the problem did not come from cythonize itself, but from the fact distutils.core.setup(...) is configured to compile+link into a .pyd instead of a .exe.
Here is the solution:
from distutils._msvccompiler import MSVCCompiler # "from distutils.msvccompiler" did not work for me!
from Cython.Compiler import Options
Options.embed = "main"
cythonize(r'src\app.py', build_dir="build")
compiler = MSVCCompiler()
compiler.compile([r"build\src\app.c"], include_dirs=["C:/Python38/include"])
compiler.link_executable([r"build\src\app.obj"], "app", libraries=["python38"], library_dirs=["C:/Python38/libs"], output_dir="release", extra_preargs=["/NOIMPLIB", "/NOEXP"])
The .exe will be in the release folder.
(Note: I also upgraded Cython to the latest version 0.29.30, it might have helped as well)
toolsTep.py
def HelloWord():
print('hello word')
testpythran.py
from calaTools.toolsTep import *
#pythran export callOtherPyFiles()
def callOtherPyFiles():
HelloWord()
complie pythran testpythran.py
CRITICAL :
I am in trouble. Your input file does not seem to match Pythran's constraints...
testpythran.py:
None:None error: Module 'calaTools.toolsTep' not found.
when two function in save file and it can find ,in diffrent file it occured those errors
When distributing a Python application with Pythran modules, you can either:
declare the module as a regular Python module. After all, they are 100% Python compatible.
declare them as a PythranExtension and Pythran will compile them:
from distutils.core import setup
These two lines are required to be able to use pythran in the setup.py
import setuptools
setuptools.dist.Distribution(dict(setup_requires='pythran'))
from pythran.dist import PythranExtension, PythranBuildExt
setup(...,
ext_modules=[PythranExtension("mymodule", ["mymodule.py"])],
cmdclass={"build_ext": PythranBuildExt})
PythranBuildExt is optional, but necessary to build extensions with different C++ compilers. It derives from distuil’s build_ext by default, but you can change its base class by using PythranBuildExt[base_cls] instead.
all configuration options supported in .pythranrc can also be passed through the optional config argument, in the form of a list, e.g. config=['compiler.blas=openblas']
from pythran doc.https://pythran.readthedocs.io/en/latest/MANUAL.html
I'm trying to cythonize some python modules (on windows) and want the resulting .c files stored in a different directory.
I tried something like this:
from Cython.Build import cythonize
module_list = cythonize(r'C:\myproject\module.py',
force=True,
build_dir=r'C:\myproject\compiled_modules')
The compilation is done but the file module.c is created in C:\myproject and not in C:\myproject\compiled_modules as expected. What am I doing wrong?
I'm not sure the cythonize function can be coerced into doing that. I would just use the command line:
/path/to/cython yourmod.py -o compiled_modules/yourmod.c
Edit:
This command can be called from a python script using the subprocess module:
import subprocess
subprocess.check_call(["/path/to/cython", "yourmod.py", "-o", "compiled_modules/yourmod.c"])
Edit 2:
According to this Sage developer, relocating the c/cpp files is not supported by cython, so it looks like either compiling from the command line or using a normal configuration and then moving the c/cpp files are your only options.
I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.
I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?
Thanks
Sure !
(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)
The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :
1. The Distutils setup file :
Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...
# setup.py file
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("myext",
sources=["myext.pyx",
"SomeAdditionalCppClass1.cpp",
"SomeAdditionalCppClass2.cpp"
],
libraries=["externlib"], # refers to "libexternlib.so"
language="c++", # remove this if C and not C++
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=["-DSOME_DEFINE_OPT",
"-L./some/extra/dependency/dir/"]
)
]
)
Note : Your external .so file is linked via the libraries option :
libraries=["externlib"] # Without the 'lib' prefix and the '.so' extension...
Note : the sources option can be used to get some additional source files compiled.
Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).
2. Then run it as follows :
After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :
# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc" \
CXX="g++" \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15" \
LDFLAGS="-L./some/path/to/externlib/" \
python setup.py build_ext --inplace
Where :
libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/
Note : CFLAGS could also have been setup using the extra_compile_args option :
extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]
Note : LDFLAGS could also have been setup using the extra_link_args option :
extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]
Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.
3. After that, you're good to go :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/
# Run some script requiring "myext.so"
python somescript.py
Where your freshly created Python extension can be imported by its name :
# somescript.py
import myext
from myext import PySomeFeature
...
Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).
Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :
PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;
Well, hope I covered the main points...
I would like to create a setup.py script for a python package with several submodules that depend on both cython and f2py. I have attempted to use setuptools and numpy.distutils, but have so far failed:
Using setuptools
I am able to compile my cython extensions (and create an installation for the rest of the package) using setuptools. I have, however, been unable to figure out how to use setuptools to generate the f2py extension. After extensive searching, I only found rather old messages like this one that state that f2py modules must be compiled using numpy.distutils.
Using numpy.distutils
I am able to compile my f2py extensions (and create an installation for the rest of the package) using numpy.distutils. I have, however, been unable to figure out how to get numpy.distutils to compile my cython extensions as it always attempts to use pyrex to compile it (and I am using extensions specific to cython) recent. I have done a search to figure out how to get numpy.distutils for cython files and - at least as of a year ago - they recommend applying a monkey patch to numpy.distutils. It seems applying such a monkey patch also restricts the options that can be passed to Cython.
My question is: what is the recommended way to write a setup.py script for packages that depend on both f2py and cython? Is applying a patch to numpy.distutils really the way to go still?
You can just call each separately in your setup.py as in
http://answerpot.com/showthread.php?601643-cython%20and%20f2py
# Cython extension
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
ext_modules = [Extension( 'cext', ['cext.pyx'] )],
cmdclass = {'build_ext': build_ext},
script_args = ['build_ext', '--inplace'],
)
# Fortran extension
from numpy.distutils.core import setup, Extension
setup(
ext_modules = [Extension( 'fext', ['fext.f90'] )],
)
Your calling context (I think they call this namespace, not sure)
has to change as to what the current object Extension and function
setup() is.
The first setup() call, it's the distutils.extension.Extension
and distutils.core.setup()
The second setup() call, it's the numpy.distutils.core.Extension
and numpy.distutils.core.setup()
Turns out this is no longer true. With both setuptools and distutils (at least the numpy version) it is possible to have extensions with C, Cython and f2py. The only caveat is that to compile f2py modules one must always use numpy.distutils for both the setup and Extension functions. But setuptools can still be used for the installation (for example, allowing the installation of a developer version with python setup.py develop).
To use distutils exclusively you use the following:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
To use setuptools, you need to import it before the distutils imports:
import setuptools
And then the rest of the code is identical:
from numpy import get_include
from Cython.Build import cythonize
NAME = 'my_package'
NUMPY_INC = get_include()
extensions = [
Extension(name=NAME + ".my_cython_ext",
include_dirs=[NUMPY_INC, "my_c_dir"]
sources=["my_cython_ext.pyx", "my_c_dir/my_ext_c_file.c"]),
Extension(name=NAME + ".my_f2py_ext",
sources=["my_f2py_ext.f"]),
]
extensions = cythonize(extensions)
setup(..., ext_modules=extensions)
Obviously you need to put all your other stuff in the setup() call. In the above I assume that you'll use numpy with Cython, along with an external C file (my_ext_c_file.c) that will be at my_c_dir/, and that the f2py module is only one Fortran file. Adjust as needed.