Develop Reference

AoT Compiler for Python

I want to get my Python script working on a bare metal device like microcontroller WITHOUT the need for an interpreter. I know there are already JIT compilers for Python like PyPy, and interpreters like CPython.
However, existing interpreters I've seen (such as CPython) take up large memory (in MB range).
Is there an AOT compiler for Python (i.e. compiling directly to native hardware through intermediary like LLVM)?
I assume such a compiler would enable Python to run much faster compared to existing implementations AND with lower memory footprint. If there is, I wonder why that solution hasn't been popularized.

As you already mentioned Cython is an option (However, it is true that the result is big due since the C runtime need to implement the Python functionality together with your program).
With regards to LLVM there was a project by Google named unladen swallow. However, that project is mostly abandoned. You can find some information about it here
Basically it was an attempt to bring LLVM optimizations into the runtime of Cython. E.g JITTING Python code.
Another old alternative was shed skin which compiles Python to C++. Some information about it can be found here.
Yet another option similar to shed skin is to restrict yourself to a subset of the Python language and use micropython.

An alternative would be to use GraalVM with Truffle AOT with Python.
It's basically Python running on an optimized AOT for jvm.
The project looks promising. You can chek this link here:
https://www.graalvm.org/22.2/graalvm-as-a-platform/language-implementation-framework/AOT/

Recently came across codon,
Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead. Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with (and sometimes better than) that of C/C++.

Does WebAssembly run faster if written in C as opposed to Python?

There's a long list of languages that can be compiled into Wasm. Is there any performance gain from writing in something like C or Rust over Python? Or is it all the same since it is being compiled to Wasm?

Short answer: Yes, because Python, the language itself, is not compiled to Wasm, but its interpreter.
Saying Python supports Wasm does not always means the same. Firstly, Python is NOT a compiled language, it's a script language. Don't expect a script language will be compiled to a native (or Wasm) language because it is not meant to work that way.
Then how Python supports Wasm? Python interpreters/runtimes like cpython, which is written in C, are compiled to Wasm. There are two popular Python runtimes that supports Python: pyodide and Wasm port for micropython (there are a lot of efforts to run Python in a browser besides the two). Both of them are interpreters that translate Python to their own bytecode and then execute bytecode in Wasm. Of course there will be huge performance penalties just like cpython in the native environment.

Compiling to WebAssembly is basically just simulating a special form of assembly targeting virtual hardware. When you read "can compile language X" into Wasm, it doesn't always mean the language literally compiles directly to Wasm. In the case of Python, to my knowledge, it means "they compiled Python interpreters to Wasm" (e.g. CPython, PyPy), so the whole Python interpreter is Wasm, but it still interprets Python source code files normally, it doesn't convert them to special Wasm modules or anything. Which means all the overhead of the Python interpreter is there, on top of the overhead of the Wasm engine, etc.
So yes, C and Rust (which can target Wasm directly by swapping out the compiler backend) will still run faster than Python code targeting CPython compiled to Wasm, for the same reasons. Tools that speed up Python when run natively (e.g. Cython, raw CPython C extensions, etc.) may also work in Wasm to get the same speed ups, but it's not a free "Compile slow interpreted language to Wasm and become fast compiled language"; computers aren't that smart yet.

What does translate mean in pypy?

I'm reading pypy's document, which has a section called Translating the PyPy Python interpreter. But i don't understand what does the word translate mean. Is it the same as compile?
The document says:
First download a pre-built PyPy for your architecture which you will use to translate your Python interpreter.
Is the pre-built PyPy here refer to the source code? Because there is no pypy/goal directory in the binary I have downloaded. If so, there is something wrong with the document. It is misleading.
Is the pypy-c created in the translation the same thing as bin/pypy in the binary?

i don't understand what does the word translate mean. Is it the same as compile?
What "translation" means is described in detail in The RPython Toolchain. There's also some higher-level introductory information in the Coding Guide and FAQ.
Summarizing their summary:
Compile and import the complete RPython program.
Dynamically analyze the program and annotate it with flow graphs.
Compile the flow graphs into lower-level flow graphs.
Optimize the compiled flow graphs.
Analyze the compiled and optimized flow graphs.
Generate C source from the flow graphs and analysis.
Compile and link the C source into a native executable.
So, step 1 uses the normal Python compiler, step 7 uses the normal C compiler (and linker), and steps 3 and 4 are similar to the kind of thing an optimizing compiler normally does. But calling the overall process "compilation" would be misleading. (Also, people would probably interpret it to mean something akin to what Shedskin does, which is definitely not right.)
Is the pypy-c created in the translation the same thing as bin/pypy in the binary?
What ends up in a binary distribution is basically the same as if you run the install process on the translation goal. So, yes, goal/pypy-c and bin/pypy are effectively the same thing.
Is the pre-built PyPy here refer to the source code?
No. It refers to a bin/pypy from a binary distribution. As the docs say, you can actually use any Python 2.6+, including CPython, or a goal/pypy-c left over from a previous build, etc. However, the translator will probably run fastest on the standard PyPy binary distribution, so that's what you should use unless you have a good reason to do otherwise.

Let me give you what I can - PyPy is several things:
A fast implementation of Python using a Just-In-Time compiler (written in RPython)
The RPython JIT compiler compiler
When the docs talk about translating the interpreter they are talking about generating a JIT compiler for Python out of the RPython implementation of the Python compiler.
Python Compiler (Written in RPython)
|--[RPython to JIT compiler compiler]-->
PyPy (JIT'ed Python Interpreter)
The key thing to note is that "compiler compiler" is not a typo. RPython is part of a toolchain used to generate JIT compilers. Rather than writing a compiler for your language and then writing a JIT layer for your compiler (which can be difficult and time consuming) instead you implement your language in RPython and the RPython translation toolchain writes you a JIT compiler for your language.
The easiest way to think about this is to imagine that the PyPy team hadn't written their own JIT compiler compiler. Imagine instead that Topaz (JIT Ruby) came first and that team had written a JIT compiler compiler in Ruby (we'll call it RRuby). The PyPy team would have then written the PyPy compiler in RRuby (instead, since PyPy came first, the Topaz team is implementing their JIT Ruby compiler in RPython).

How to update an older C extension for Python 2.x to Python 3.x

I'm wanting to use an extension for Python that I found here, but I'm using Python 3.1 and when I attempt to compile the C extension included in the package (_wincon), it does not compile due to all the syntax errors. Unfortunately, it was written for 2.x versions of Python and as such includes methods such as PyMember_Get and PyMember_Set, which are no longer part of Python. My problem is that I have not gotten around to learning C and as such have not been able to figure out how to modify the code to use syntax that is still valid in Python 3.1. (There were also a couple macros such as staticforward that need fixing, but I'm assuming those just need to be changed to static.) Therefore: how do I go about fixing this?
(Note that I have indeed looked into various other Windows console interfaces for Python such as the win32con extension in PyWin32), but none of them fit my needs as much as this one appears to.)

I don't believe there is any magic bullet to make the C sources for a Python extension coded for some old-ish version of Python 2, into valid C sources for one coded for Python 3 -- it takes understanding of C and of how the C API has changed, and of what exactly the extension is doing in each part of its code. Believe me, if we had known of some magic bullet way to do it susbtantially without such human knowledge and understanding, we'd have included a "code generator" (like 2to3 for Python sources -- and even that has substantial limitations!) for such C code translation.
In practice, even though Python 3.1 is per se a mature and production-ready language, you should not yet migrate your code (or write a totally new app) in Python 3.1 if you need some Python 2.* extension that you're unable to port -- stick with 2.6 until the extensions you require are available (or you have learned enough C to port them yourself -- or rewrite them in Cython, which DOES smoothly support Python 2.* and 3.*, with, I believe, just a modicum of care on the programmer's part).

Is it possible to compile Python natively (beyond pyc byte code)?

I wonder if it is possible to create an executable module from a Python script. I need to have the most performance and the flexibility of Python script, without needing to run in the Python environment. I would use this code to load on demand user modules to customize my application.

There's pyrex that compiles python like source to python extension modules
rpython which allows you to compile python with some restrictions to various backends like C, LLVM, .Net etc.
There's also shed-skin which translates python to C++, but I can't say if it's any good.
PyPy implements a JIT compiler which attempts to optimize runtime by translating pieces of what's running at runtime to machine code, if you write for the PyPy interpreter that might be a feasible path.
The same author that is working on JIT in PyPy wrote psyco previously which optimizes python in the CPython interpreter.

You can use something like py2exe to compile your python script into an exe, or Freeze for a linux binary.
see: How can I create a directly-executable cross-platform GUI app using Python?

I've had a lot of success using Cython, which is based on and extends pyrex:
Cython is a language that makes
writing C extensions for the Python
language as easy as Python itself.
Cython is based on the well-known
Pyrex, but supports more cutting edge
functionality and optimizations.
The Cython language is very close to
the Python language, but Cython
additionally supports calling C
functions and declaring C types on
variables and class attributes. This
allows the compiler to generate very
efficient C code from Cython code.
This makes Cython the ideal language
for wrapping for external C libraries,
and for fast C modules that speed up
the execution of Python code.

I think you can use jython to compile python to Java bytecode, and then compile that with GCJ.