Develop Reference

how to use pythran add a myfunction from other py files?

toolsTep.py
def HelloWord():
print('hello word')
testpythran.py
from calaTools.toolsTep import *
#pythran export callOtherPyFiles()
def callOtherPyFiles():
HelloWord()
complie pythran testpythran.py
CRITICAL :
I am in trouble. Your input file does not seem to match Pythran's constraints...
testpythran.py:
None:None error: Module 'calaTools.toolsTep' not found.
when two function in save file and it can find ,in diffrent file it occured those errors

When distributing a Python application with Pythran modules, you can either:
declare the module as a regular Python module. After all, they are 100% Python compatible.
declare them as a PythranExtension and Pythran will compile them:
from distutils.core import setup
These two lines are required to be able to use pythran in the setup.py
import setuptools
setuptools.dist.Distribution(dict(setup_requires='pythran'))
from pythran.dist import PythranExtension, PythranBuildExt
setup(...,
ext_modules=[PythranExtension("mymodule", ["mymodule.py"])],
cmdclass={"build_ext": PythranBuildExt})
PythranBuildExt is optional, but necessary to build extensions with different C++ compilers. It derives from distuil’s build_ext by default, but you can change its base class by using PythranBuildExt[base_cls] instead.
all configuration options supported in .pythranrc can also be passed through the optional config argument, in the form of a list, e.g. config=['compiler.blas=openblas']
from pythran doc.https://pythran.readthedocs.io/en/latest/MANUAL.html

Building Python-C Extension using CFFI, but Setuptools does not include custom header files in build

I'm trying to use the CFFI package in Python to create a Python interface for already existing C-code.
I am able to compile a C library by following this blog post. Now I want to make it so that this python library is available without any fancy updates to the sys.path.
I found that maybe creating a distribution through Python's setuptools setup() function would accomplish this and I got it to mostly work by creating a setup.py file as such
import os
import sys
from setuptools import setup, find_packages
os.chdir(os.path.dirname(sys.argv[0]) or ".")
setup(
name="SobelFilterTest",
version="0.1",
description="An example project using Python's CFFI",
packages=find_packages(),
install_requires=["cffi>=1.0.0"],
setup_requires=["cffi>=1.0.0"],
cffi_modules=[
"./src/build_sobel.py:ffi",
"./src/build_file_operations.py:ffi",
],
)
, but I run into this error
build/temp.linux-x86_64-3.5/_sobel.c:492:19: fatal error: sobel.h: No such file or directory
From what I can tell, the problem is that the sobel.h file does not get uploaded into the build folder created by setuptools.setup(). I looked for suggestions of what to do including using Extensions() and writing a MANIFEST.in file, and both seem to add a relative path to the correct header files:
MANIFEST.in
setup.py
SobelFilterTest.egg-info/PKG-INFO
SobelFilterTest.egg-info/SOURCES.txt
SobelFilterTest.egg-info/dependency_links.txt
SobelFilterTest.egg-info/requires.txt
SobelFilterTest.egg-info/top_level.txt
src/file_operations.h
src/macros.h
src/sobel.h
But I still get the same error message. Is there a correct way to go about adding the header file to the build folder? Thanks!

It's actually not pip that is missing the .h file, but rather the compiler (like gcc). Therefore it's not about adding the missing file to setup, but rather make sure that cffi can find it. One way (like mentioned in the comments) is to make it available to the compiler through environment variables, but there is another way.
When setting the source with cffi you can add directories for the compiler like this:
from cffi import FFI
ffibuilder = FFI()
ffibuilder.set_source("<YOUR SOURCE HERE>", include_dirs=["./src"])
# ... Rest of your code
"""

Cython unable to find shared object file

I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?

I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.

I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory

Using Cython To Link Python To A Shared Library

I am trying to integrate a third party library written in C with my python application using Cython. I have all of the python code written for a test. I am having trouble finding an example for setting this up.
I have a pyd/pyx file I created manually. The third party has given me a header file (*.h) and a shared library (*.so). As far as I can tell, there are no other dependencies. Can someone provide an example of how to set this up using Cython and disutils?
Thanks

Sure !
(In the following, I assume that you already know how to deal with cimport and the interactions between .pxd and .pyx. If this is not completely the case, just ask and I will develop that part as well)
The sample (grabbed from a C++ project of mine, but a C project would work pretty much the same) :
1. The Distutils setup file :
Assuming that the extension to be created will be called myext and the 3rd party shared library is libexternlib.so (note the lib* prefix, here)...
# setup.py file
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# clean previous build
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("myext") and not(name.endswith(".pyx") or name.endswith(".pxd"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# build "myext.so" python extension to be added to "PYTHONPATH" afterwards...
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("myext",
sources=["myext.pyx",
"SomeAdditionalCppClass1.cpp",
"SomeAdditionalCppClass2.cpp"
],
libraries=["externlib"], # refers to "libexternlib.so"
language="c++", # remove this if C and not C++
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=["-DSOME_DEFINE_OPT",
"-L./some/extra/dependency/dir/"]
)
]
)
Note : Your external .so file is linked via the libraries option :
libraries=["externlib"] # Without the 'lib' prefix and the '.so' extension...
Note : the sources option can be used to get some additional source files compiled.
Important : myext.pxd (do not confound with .pyd - Windows stuff) and myext.pyx should be in the same directory. At compile time the definition file, if it exists, is processed first (more).
2. Then run it as follows :
After having changed directory to the one containing your myext.pxd, your myext.pyx, as well as the above setup.py script :
# setup.sh
# Make the "myext" Python Module ("myext.so")
CC="gcc" \
CXX="g++" \
CFLAGS="-I./some/path/to/includes/ -I../../../DEPENDENCIES/python2.7/inc -I../../../DEPENDENCIES/gsl-1.15" \
LDFLAGS="-L./some/path/to/externlib/" \
python setup.py build_ext --inplace
Where :
libexternlib.so is assumed to be located at ./some/path/to/externlib/
yourheader.h is assumed to be located at ./some/path/to/includes/
Note : CFLAGS could also have been setup using the extra_compile_args option :
extra_compile_args=["-I./some/path/to/includes/", "-fopenmp", "-O3"]
Note : LDFLAGS could also have been setup using the extra_link_args option :
extra_link_args=["-L./some/path/to/externlib/", "-DSOME_DEFINE_OPT", "-L./some/extra/dependency/dir/"]
Once distutils is done with the build, you get some new files, specially the myext.cpp, myext.h and most importantly, the myext.so.
3. After that, you're good to go :
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:./some/path/to/externlib/
export PYTHONPATH=$PYTHONPATH:./some/path/to/myext/
# Run some script requiring "myext.so"
python somescript.py
Where your freshly created Python extension can be imported by its name :
# somescript.py
import myext
from myext import PySomeFeature
...
Note about Optimization : By default -O2 is used for compiling the extension, but this can be overloaded (see above setup where -O3 is specified).
Note about Cython paths : If Cython was installed in a custom directory, you might want to add it to your environment, before all :
PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18 export PYTHONPATH;
PATH=$PATH:../../../DEPENDENCIES/Cython-0.18/bin; export PATH;
Well, hope I covered the main points...

Get hold of warnings from cython pyximport compile (distutils build output?)

When I compile a cython .pyx file from IdleX the build shell window pops up with a bunch of warnings to close again after less than a second.
I think pyximport uses distutils to build. How can I write the gcc warnings to a file or have the output delay or wait for keypress?

You can add a .pyxbld file to specify Cython build settings.
Say you are trying to compile yourmodule.pyx, simply create a file in the same directory named yourmodule.pyxbld containing:
def make_ext(modname, pyxfilename):
from distutils.extension import Extension
ext = Extension(name = modname,
sources=[pyxfilename])
return ext
def make_setup_args():
return dict(script_args=['--verbose'])
The --verbose flag makes pyximport print gcc's output.
Note that you can easily add extra compiler and linker flags. For example, to use Cython's prange() function you must compile and link against the OpenMP library, this is specified using keywords to the Extension class:
ext = Extension(name = modname,
sources=[pyxfilename],
extra_compile_args=['-fopenmp'],
extra_link_args=['-fopenmp'])

I haven't done anything with cython myself but I guess you could use a commandline for the building. That way you would see all the messages until you close the actual commandline unless some really fatal error happens.

Yes, here is how you could do to get the output in a file (in case the warnings overflow the command prompt) :
This is for Windows, but you can easily adapt for *unices :
SET PYTHONPATH=%PYTHONPATH%;../../../DEPENDENCIES/Cython-0.18
SET PATH=%PATH%;../../../DEPENDENCIES/Cython-0.18/bin
python setup.py build_ext --inplace > output.log