Develop Reference

Cython embed on Windows

I have read How to enable `--embed` with cythonize? and Cython --embed flag in setup.py but this method does not work on Windows. Let's say we have:
# app.py
print("hello")
# build.py
import setuptools # allows us to avoid calling vcvarsall.bat, see https://stackoverflow.com/a/53172602/
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.embed = "main"
setup(ext_modules=cythonize(r'app.py', build_dir="release"), script_args=['build'], options={'build': {'build_lib': 'release'}})
Running this build.py script on Python 3.8 for Windows does not produce an app.exe file like it would with the command line command:
cython app.py --embed
Instead, it produces a .pyd file.
How to use cythonize + embed from a Python script, producing a .exe, on Windows?

Solved: in fact the problem did not come from cythonize itself, but from the fact distutils.core.setup(...) is configured to compile+link into a .pyd instead of a .exe.
Here is the solution:
from distutils._msvccompiler import MSVCCompiler # "from distutils.msvccompiler" did not work for me!
from Cython.Compiler import Options
Options.embed = "main"
cythonize(r'src\app.py', build_dir="build")
compiler = MSVCCompiler()
compiler.compile([r"build\src\app.c"], include_dirs=["C:/Python38/include"])
compiler.link_executable([r"build\src\app.obj"], "app", libraries=["python38"], library_dirs=["C:/Python38/libs"], output_dir="release", extra_preargs=["/NOIMPLIB", "/NOEXP"])
The .exe will be in the release folder.
(Note: I also upgraded Cython to the latest version 0.29.30, it might have helped as well)

Cythonized Python.Net code cannot find system assemblies

When I compile with Cython the Python code which uses Python.NET to access .NET assemblies, it can't find those assemblies:
ModuleNotFoundError: No module named 'System'
Without compilation it works ok.
For demo code, I used https://github.com/pythonnet/pythonnet/blob/master/demo/helloform.py
My setup.py file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
ext_modules = [
Extension(
'helloform',
sources = ['helloform.py'],
language = 'c++'
)
]
setup(
name = 'helloform',
ext_modules = cythonize(ext_modules),
)
Then I build it with python setup.py build_ext --inplace.
I wanted to load compiled module from Python prompt with import helloform but it failed with
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "helloform.py", line 8, in init helloform
ModuleNotFoundError: No module named 'System'

This answer is untested - I don't think I can easily set up an environment to test so it's a bit of a guess. If it doesn't work I'll remove it.
This probably is a bug, and if you want it fixed in the longer term you should report it. Cython does try to be compatible with Python wherever possible.... A quick investigate suggests that Python.NET overrides the built-in __import__ function. Cython looks to lookup and use this function in Python 2, but not in Python 3. This is no longer the preferred way of customizing import behaviour (but is still supported). I'd guess it would work in Cython + Python 2?
As a workaround you should probably just run the import statements in Python. There's two obvious ways to do it:
Write a small separate module just containing the import statements, then in Cython import from that module:
from import_module import WinForms, Size, Point
Run the import statements in exec; extract the values out of the global dict you pass it:
import_dict = {}
exec("""import clr
# etc...
""", import_dict) # pass a dict in as `globals`
WinForms = import_dict['WinForms']
# etc.

In a setup.py involving Cython, if install_requires, then how can from library import something?

This doesn't make sense to me. How can I use the setup.py to install Cython and then also use the setup.py to compile a library proxy?
import sys, imp, os, glob
from setuptools import setup
from Cython.Build import cythonize # this isn't installed yet
setup(
name='mylib',
version='1.0',
package_dir={'mylib': 'mylib', 'mylib.tests': 'tests'},
packages=['mylib', 'mylib.tests'],
ext_modules = cythonize("mylib_proxy.pyx"), #how can we call cythonize here?
install_requires=['cython'],
test_suite='tests',
)
Later:
python setup.py build
Traceback (most recent call last):
File "setup.py", line 3, in <module>
from Cython.Build import cythonize
ImportError: No module named Cython.Build
It's because cython isn't installed yet.
What's odd is that a great many projects are written this way. A quick github search reveals as much: https://github.com/search?utf8=%E2%9C%93&q=install_requires+cython&type=Code

As I understand it, this is where PEP 518 comes in - also see some clarifications by one of its authors.
The idea is that you add yet another file to your Python project / package: pyproject.toml. It is supposed to contain information on build environment dependencies (among other stuff, long term). pip (or just any other package manager) could look into this file and before running setup.py (or any other build script) install the required build environment. A pyproject.toml could therefore look like this:
[build-system]
requires = ["setuptools", "wheel", "Cython"]
It is a fairly recent development and, as of yet (January 2019), it is not finalized / approved by the Python community, though (limited) support was added to pip in May 2017 / the 10.0 release.

One solution to this is to not make Cython a build requirement, and instead distribute the Cython generated C files with your package. I'm sure there is a simpler example somewhere, but this is what pandas does - it conditionally imports Cython, and if not present can be built from the c files.
https://github.com/pandas-dev/pandas/blob/3ff845b4e81d4dde403c29908f5a9bbfe4a87788/setup.py#L433
Edit: The doc link from #danny has an easier to follow example.
http://docs.cython.org/en/latest/src/reference/compilation.html#distributing-cython-modules

When you use setuptool, you should add cython to setup_requires (and also to install_requires if cython is used by installation), i.e.
# don't import cython, it isn't yet there
from setuptools import setup, Extension
# use Extension, rather than cythonize (it is not yet available)
cy_extension = Extension(name="mylib_proxy", sources=["mylib_proxy.pyx"])
setup(
name='mylib',
...
ext_modules = [cy_extension],
setup_requires=["cython"],
...
)
Cython isn't imported (it is not yet available when setup.pystarts), but setuptools.Extension is used instead of cythonize to add cython-extension to the setup.
It should work now. The reason: setuptools will try to import cython, after setup_requires are fulfilled:
...
try:
# Attempt to use Cython for building extensions, if available
from Cython.Distutils.build_ext import build_ext as _build_ext
# Additionally, assert that the compiler module will load
# also. Ref #1229.
__import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext
...
It becomes more complicated, if your Cython-extension uses numpy, but also this is possible - see this SO post.

It doesn't make sense in general. It is, as you suspect, an attempt to use something that (possibly) has yet to be installed. If tested on a system that already has the dependency installed, you might not notice this defect. But run it on a system where your dependency is absent, and you will certainly notice.
There is another setup() keyword argument, setup_requires, that can appear to be parallel in form and use to install_requires, but this is an illusion. Whereas install_requires triggers a lovely ballet of automatic installation in environments that lack the dependencies it names, setup_requires is more documentation than automation. It won't auto-install, and certainly not magically jump back in time to auto-install modules that have already been called for in import statements.
There's more on this at the setuptools docs, but the quick answer is that you're right to be confused by a module that is trying to auto-install its own setup pre-requisites.
For a practical workaround, try installing cython separately, and then run this setup. While it won't fix the metaphysical illusions of this setup script, it will resolve the requirements and let you move on.

How can I check for which command is run in setup.py?

I would like to know how to make some code in setup.py conditional on which command (e.g. install or upload) was run.
Specifically, I'd like to have:
An easy way to add "hacks" such as ignoring a particular file in install, but no other commands.
A recommended/canonical way to add hooks such as running tests before installing.
I have tried reading the distutils documentation, but it's pretty sparse on details – the distutils.command[.foo] modules are completely undocumented.
For the first point I can check sys.argv like mentioned in this question, but that doesn't work when multiple commands are run, like:
python setup.py sdist bdist upload
so it isn't applicable in general.

You can override the command instead:
from distutils.command.install import install
from distutils.core import setup
def run_file(path):
with open(path, 'r') as f:
exec(f.read())
class myinstall(install): # subclass distutils's install command
def finalize_options(self): # called after option parsing
# call base class function
install.finalize_options(self)
# super won't work because distutils under Python 2 uses old-style classes
# ignore a module
self.distribution.py_modules.remove('mymodule')
def run(self): # called to run a command
# run tests first
run_file('path/to/test.py')
# ^ remember to make sure the module is in sys.path
# run the real commands
install.run(self)
setup(
name='abc',
py_modules=['mymodule'],
cmdclass={'install': myinstall}
# ^ override the install command
)

setup.py for packages that depend on both cython and f2py

I would like to create a setup.py script for a python package with several submodules that depend on both cython and f2py. I have attempted to use setuptools and numpy.distutils, but have so far failed:
Using setuptools
I am able to compile my cython extensions (and create an installation for the rest of the package) using setuptools. I have, however, been unable to figure out how to use setuptools to generate the f2py extension. After extensive searching, I only found rather old messages like this one that state that f2py modules must be compiled using numpy.distutils.
Using numpy.distutils
I am able to compile my f2py extensions (and create an installation for the rest of the package) using numpy.distutils. I have, however, been unable to figure out how to get numpy.distutils to compile my cython extensions as it always attempts to use pyrex to compile it (and I am using extensions specific to cython) recent. I have done a search to figure out how to get numpy.distutils for cython files and - at least as of a year ago - they recommend applying a monkey patch to numpy.distutils. It seems applying such a monkey patch also restricts the options that can be passed to Cython.
My question is: what is the recommended way to write a setup.py script for packages that depend on both f2py and cython? Is applying a patch to numpy.distutils really the way to go still?

You can just call each separately in your setup.py as in
http://answerpot.com/showthread.php?601643-cython%20and%20f2py
# Cython extension
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
ext_modules = [Extension( 'cext', ['cext.pyx'] )],
cmdclass = {'build_ext': build_ext},
script_args = ['build_ext', '--inplace'],
)
# Fortran extension
from numpy.distutils.core import setup, Extension
setup(
ext_modules = [Extension( 'fext', ['fext.f90'] )],
)
Your calling context (I think they call this namespace, not sure)
has to change as to what the current object Extension and function
setup() is.
The first setup() call, it's the distutils.extension.Extension
and distutils.core.setup()
The second setup() call, it's the numpy.distutils.core.Extension
and numpy.distutils.core.setup()

Turns out this is no longer true. With both setuptools and distutils (at least the numpy version) it is possible to have extensions with C, Cython and f2py. The only caveat is that to compile f2py modules one must always use numpy.distutils for both the setup and Extension functions. But setuptools can still be used for the installation (for example, allowing the installation of a developer version with python setup.py develop).
To use distutils exclusively you use the following:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
To use setuptools, you need to import it before the distutils imports:
import setuptools
And then the rest of the code is identical:
from numpy import get_include
from Cython.Build import cythonize
NAME = 'my_package'
NUMPY_INC = get_include()
extensions = [
Extension(name=NAME + ".my_cython_ext",
include_dirs=[NUMPY_INC, "my_c_dir"]
sources=["my_cython_ext.pyx", "my_c_dir/my_ext_c_file.c"]),
Extension(name=NAME + ".my_f2py_ext",
sources=["my_f2py_ext.f"]),
]
extensions = cythonize(extensions)
setup(..., ext_modules=extensions)
Obviously you need to put all your other stuff in the setup() call. In the above I assume that you'll use numpy with Cython, along with an external C file (my_ext_c_file.c) that will be at my_c_dir/, and that the f2py module is only one Fortran file. Adjust as needed.