Develop Reference

Cython embed on Windows

I have read How to enable `--embed` with cythonize? and Cython --embed flag in setup.py but this method does not work on Windows. Let's say we have:
# app.py
print("hello")
# build.py
import setuptools # allows us to avoid calling vcvarsall.bat, see https://stackoverflow.com/a/53172602/
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.embed = "main"
setup(ext_modules=cythonize(r'app.py', build_dir="release"), script_args=['build'], options={'build': {'build_lib': 'release'}})
Running this build.py script on Python 3.8 for Windows does not produce an app.exe file like it would with the command line command:
cython app.py --embed
Instead, it produces a .pyd file.
How to use cythonize + embed from a Python script, producing a .exe, on Windows?

Solved: in fact the problem did not come from cythonize itself, but from the fact distutils.core.setup(...) is configured to compile+link into a .pyd instead of a .exe.
Here is the solution:
from distutils._msvccompiler import MSVCCompiler # "from distutils.msvccompiler" did not work for me!
from Cython.Compiler import Options
Options.embed = "main"
cythonize(r'src\app.py', build_dir="build")
compiler = MSVCCompiler()
compiler.compile([r"build\src\app.c"], include_dirs=["C:/Python38/include"])
compiler.link_executable([r"build\src\app.obj"], "app", libraries=["python38"], library_dirs=["C:/Python38/libs"], output_dir="release", extra_preargs=["/NOIMPLIB", "/NOEXP"])
The .exe will be in the release folder.
(Note: I also upgraded Cython to the latest version 0.29.30, it might have helped as well)

cython setuptools change output filename

I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?

As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.

An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.

Cythonized Python.Net code cannot find system assemblies

When I compile with Cython the Python code which uses Python.NET to access .NET assemblies, it can't find those assemblies:
ModuleNotFoundError: No module named 'System'
Without compilation it works ok.
For demo code, I used https://github.com/pythonnet/pythonnet/blob/master/demo/helloform.py
My setup.py file:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
ext_modules = [
Extension(
'helloform',
sources = ['helloform.py'],
language = 'c++'
)
]
setup(
name = 'helloform',
ext_modules = cythonize(ext_modules),
)
Then I build it with python setup.py build_ext --inplace.
I wanted to load compiled module from Python prompt with import helloform but it failed with
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "helloform.py", line 8, in init helloform
ModuleNotFoundError: No module named 'System'

This answer is untested - I don't think I can easily set up an environment to test so it's a bit of a guess. If it doesn't work I'll remove it.
This probably is a bug, and if you want it fixed in the longer term you should report it. Cython does try to be compatible with Python wherever possible.... A quick investigate suggests that Python.NET overrides the built-in __import__ function. Cython looks to lookup and use this function in Python 2, but not in Python 3. This is no longer the preferred way of customizing import behaviour (but is still supported). I'd guess it would work in Cython + Python 2?
As a workaround you should probably just run the import statements in Python. There's two obvious ways to do it:
Write a small separate module just containing the import statements, then in Cython import from that module:
from import_module import WinForms, Size, Point
Run the import statements in exec; extract the values out of the global dict you pass it:
import_dict = {}
exec("""import clr
# etc...
""", import_dict) # pass a dict in as `globals`
WinForms = import_dict['WinForms']
# etc.

Cython compiler error

I haven't been able to find much about this, but when attempting to compile one of my python scripts with cython, it gives me this error:
Error compiling Cython file:
------------------------------------------------------------
...
import traceback #line:24
import bcrypt #line:25
Y =str (os .path .dirname (os .path .abspath (__file__ )))#line:28
IH =open #line:29
IA =str #line:30
IK =print #line:31
^
------------------------------------------------------------
headlessobfu.pyx:29:4: Expected an identifier or literal
Traceback (most recent call last):
File "setup.py", line 5, in <module>
ext_modules = cythonize("headlessobfu.pyx")
File "C:\Users\justi\AppData\Local\Programs\Python\Python36-32\lib\site-packages\Cython\Build\Dependencies.py", line 1026, in cythonize
cythonize_one(*args)
File "C:\Users\justi\AppData\Local\Programs\Python\Python36-32\lib\site-packages\Cython\Build\Dependencies.py", line 1146, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: headlessobfu.pyx
I haven't been able to find the exact cause of this error. I was able to get simple scripts to compile just fine. The python runs just fine on it's own. Is it a problem with my python formatting?
Here is the command line argument I am running:
py setup.py build_ext --inplace
If anyone has a solution please let me know. Thanks.

By default Cython assumes Python 2 syntax, even when you're using Python 3 (edit 2021: Cython 3 will change this and largely assume Python 3 syntax by default). Here's the minimum, complete example you should have created:
cy_print_test.pyx
x = print
If I compile it with Cython
cython cy_print_test.pyx
I get the error message that you do. print is not an identifier or literal because under Python 2 syntax it is a special statement.
However, if I compile it with Cython set to use Python 3 syntax:
cython -3 cy_print_test.pyx
it works fine - under Python 3 syntax print is a function and so this makes perfect sense.
Alternative ways of getting Python 3 syntax would be to add the following line to the start of your .pyx file
#cython: language_level=3
or to specify it as a compiler directive in setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize('cy_print_test.pyx', compiler_directives={'language_level': 3}),
)
(note that your setup.py should also form part of the minimum, complete example in your question)

Ok so this has to do with cython and python formatting. The code has to be totally obfuscated and with no errors or the compiler won't work. So work through the compiler and fix each error as it arises. Hope this helps.

we need to install the required dependencies, and I guess you are missing out the python3-dev.
Try these commands on terminal (cmd)
$ pip3 install cython
$ sudo apt install update
$ sudo apt install build-essential
$ sudo apt install python3-dev
then run the $ python3 setup.py build_ext --inplace again. it should work.

In a setup.py involving Cython, if install_requires, then how can from library import something?

This doesn't make sense to me. How can I use the setup.py to install Cython and then also use the setup.py to compile a library proxy?
import sys, imp, os, glob
from setuptools import setup
from Cython.Build import cythonize # this isn't installed yet
setup(
name='mylib',
version='1.0',
package_dir={'mylib': 'mylib', 'mylib.tests': 'tests'},
packages=['mylib', 'mylib.tests'],
ext_modules = cythonize("mylib_proxy.pyx"), #how can we call cythonize here?
install_requires=['cython'],
test_suite='tests',
)
Later:
python setup.py build
Traceback (most recent call last):
File "setup.py", line 3, in <module>
from Cython.Build import cythonize
ImportError: No module named Cython.Build
It's because cython isn't installed yet.
What's odd is that a great many projects are written this way. A quick github search reveals as much: https://github.com/search?utf8=%E2%9C%93&q=install_requires+cython&type=Code

As I understand it, this is where PEP 518 comes in - also see some clarifications by one of its authors.
The idea is that you add yet another file to your Python project / package: pyproject.toml. It is supposed to contain information on build environment dependencies (among other stuff, long term). pip (or just any other package manager) could look into this file and before running setup.py (or any other build script) install the required build environment. A pyproject.toml could therefore look like this:
[build-system]
requires = ["setuptools", "wheel", "Cython"]
It is a fairly recent development and, as of yet (January 2019), it is not finalized / approved by the Python community, though (limited) support was added to pip in May 2017 / the 10.0 release.

One solution to this is to not make Cython a build requirement, and instead distribute the Cython generated C files with your package. I'm sure there is a simpler example somewhere, but this is what pandas does - it conditionally imports Cython, and if not present can be built from the c files.
https://github.com/pandas-dev/pandas/blob/3ff845b4e81d4dde403c29908f5a9bbfe4a87788/setup.py#L433
Edit: The doc link from #danny has an easier to follow example.
http://docs.cython.org/en/latest/src/reference/compilation.html#distributing-cython-modules

When you use setuptool, you should add cython to setup_requires (and also to install_requires if cython is used by installation), i.e.
# don't import cython, it isn't yet there
from setuptools import setup, Extension
# use Extension, rather than cythonize (it is not yet available)
cy_extension = Extension(name="mylib_proxy", sources=["mylib_proxy.pyx"])
setup(
name='mylib',
...
ext_modules = [cy_extension],
setup_requires=["cython"],
...
)
Cython isn't imported (it is not yet available when setup.pystarts), but setuptools.Extension is used instead of cythonize to add cython-extension to the setup.
It should work now. The reason: setuptools will try to import cython, after setup_requires are fulfilled:
...
try:
# Attempt to use Cython for building extensions, if available
from Cython.Distutils.build_ext import build_ext as _build_ext
# Additionally, assert that the compiler module will load
# also. Ref #1229.
__import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext
...
It becomes more complicated, if your Cython-extension uses numpy, but also this is possible - see this SO post.

It doesn't make sense in general. It is, as you suspect, an attempt to use something that (possibly) has yet to be installed. If tested on a system that already has the dependency installed, you might not notice this defect. But run it on a system where your dependency is absent, and you will certainly notice.
There is another setup() keyword argument, setup_requires, that can appear to be parallel in form and use to install_requires, but this is an illusion. Whereas install_requires triggers a lovely ballet of automatic installation in environments that lack the dependencies it names, setup_requires is more documentation than automation. It won't auto-install, and certainly not magically jump back in time to auto-install modules that have already been called for in import statements.
There's more on this at the setuptools docs, but the quick answer is that you're right to be confused by a module that is trying to auto-install its own setup pre-requisites.
For a practical workaround, try installing cython separately, and then run this setup. While it won't fix the metaphysical illusions of this setup script, it will resolve the requirements and let you move on.