I am looking to compile python source code into a binary. I generally use pypy for performance reasons (my application runs significantly faster using pypy). I found this answer from 2012 stating compiling pypy into binary isn't possible using any of the existing tools (e.g. pyinstaller). Compile PyPy to Exe. I wasn't quite able to follow what the top response was explaining as a solution.
I am wondering:
Is there now a reasonably easy way to compile python code to binary using pypy as the interpreter?
I don't actually need to use pypy, I just use it for the performance gains. Is there a reasonable way to avoid using pypy but still get similar performance gains? This would still need to support compiling to binary.
As an additional note, I use the 2.7 syntax.
you can take a look to nuitka ( http://nuitka.net/doc/user-manual.html)
nuitka --standalone --recurse-on --python-version=2.7 main.py
Related
I want to get my Python script working on a bare metal device like microcontroller WITHOUT the need for an interpreter. I know there are already JIT compilers for Python like PyPy, and interpreters like CPython.
However, existing interpreters I've seen (such as CPython) take up large memory (in MB range).
Is there an AOT compiler for Python (i.e. compiling directly to native hardware through intermediary like LLVM)?
I assume such a compiler would enable Python to run much faster compared to existing implementations AND with lower memory footprint. If there is, I wonder why that solution hasn't been popularized.
As you already mentioned Cython is an option (However, it is true that the result is big due since the C runtime need to implement the Python functionality together with your program).
With regards to LLVM there was a project by Google named unladen swallow. However, that project is mostly abandoned. You can find some information about it here
Basically it was an attempt to bring LLVM optimizations into the runtime of Cython. E.g JITTING Python code.
Another old alternative was shed skin which compiles Python to C++. Some information about it can be found here.
Yet another option similar to shed skin is to restrict yourself to a subset of the Python language and use micropython.
An alternative would be to use GraalVM with Truffle AOT with Python.
It's basically Python running on an optimized AOT for jvm.
The project looks promising. You can chek this link here:
https://www.graalvm.org/22.2/graalvm-as-a-platform/language-implementation-framework/AOT/
Recently came across codon,
Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead. Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with (and sometimes better than) that of C/C++.
I would like to know if there are any documented performance differences between a Python interpreter that I can install from an rpm (or using yum) and a Python interpreter compiled from sources (with a priori well set flags for compilations).
I am using a Redhat 6.3 machine as Django/Apache/Mod_WSGI production server. I have already properly compiled everything in different setups and in different orders. However, I usually keep the build-dev dependencies on such machine. For some various ego-related (and more or less practical) reasons, I would like to use Python-2.7.3. By default, Redhat comes with Python-2.6.6. I think I could go with it but it would hurt me somehow (I would have to drop and find a replacement for a few libraries and my ego).
However, besides my ego and dependencies, I would like to know what would be the impact in terms of performance for a Django server.
If you compile with the exact same flags that were used to compile the RPM version, you will get a binary that's exactly as fast. And you can get those flags by looking at the RPM's spec file.
However, you can sometimes do better than the pre-built version. For example, you can let the compiler optimize for your specific CPU, instead of for "general 386 compatible" (or whatever the RPM was optimized for). Of course if you don't know what you're doing (or are doing it on purpose), it's always possible to build something slower than the pre-built version, too.
Meanwhile, 2.7.3 is faster in a few areas than 2.6.6. Most of them usually won't affect you, but if they do, they'll probably be a big win.
Finally, for the vast majority of Python code, the speed of the Python interpreter itself isn't relevant to your overall performance or scalability. (And when it is, you probably want to try PyPy, Jython, or IronPython to replace CPython.) This is especially true for a WSGI service. If you're not doing anything slow, Apache will probably be the bottleneck. If you are doing anything slow, it's probably something I/O bound and well outside of Python's control (like reading files).
Ultimately, the only way you can know how much gain you get is by trying it both ways and performance testing. But if you just want a rule of thumb, I'd say expect a 0% gain, and be pleasantly surprised if you get lucky.
Is there a difference in python programming while using just python and while using pypy compiler? I wanted to try using pypy so that my program execution time becomes faster. Does all the syntax that work in python works in pypy too? If there is no difference, can you tell me how can i install pypy on debian lunux and some usage examples on pypy? Google does not contain much info on pypy other than its description.
From the pypy features page:
PyPy 1.9 implements Python 2.7.2 and runs on Intel x86 (IA-32) and
x86_64 platforms, with ARM and PPC being underway. It supports all of
the core language, passing the Python test suite.
This means that pretty much any code that you've written in Python 2.7 will work. The only exceptions worth mentioning are some python extensions written in C, such as numpy.
Installation should be fairly easy, you can download a linux binary from here. Then simply extract the interpreter. From this point, you can run your python programs similar to how you would run them with the normal python interpreter.
At the command line, instead of:
python my_program.py
Use:
path/to/where/you/installed/pypy my_program.py
For examples of how/why you might want to use pypy, check out this video from PyCon 2012.
pypy is a compliant alternative implementation of the python language. This means that there are few (intentional) differences. One of the few differences is pypy does not use reference counting. This means, for instance, you have to manually close your files, they will not be automatically closed when your file variable goes out of scope as in CPython.
I am looking to bring speed improvements to an existing application and I'm looking for advice on my possible options. The application is written in Python, uses wxPython, and is packaged with py2exe (I only target windows platforms). Parts of the application are computationally intensive and run too slowly in interpreted Python. I am not familiar with C, so porting parts of the code over is not really an option for me.
So my question is basically do I have a clear picture of my options as I outline below, or am I approaching this from the wrong direction?
Running with pypy: Today I started experimenting with Pypy - the results are exciting, in that I can run large parts of the code from the pypy interpreter and I'm seeing 5x+ speed improvements with no code changes. However, if I understand correctly, (a) Pypy with wxpython support is still a work in progress, and (b) I cannot compile it down to an exe for distribution anyway. So unless I'm mistaken, this seems like a no-go for me? There's no way to package things up so parts of it are executed with pypy?
Converting code to RPython, translating with pypy So the next option seems to be actually rewriting parts of the code to the pypy restricted language, which seems like a pretty large job. But if I do that, parts of the code can then be compiled to an executable (?) and then I can access the code through ctypes (?).
Other restricted options Shedskin seems to be a popular alternative here, does this fit my requirements better? Other options seem to be Cpython, Psyco, and Unladen, but they are all superseded or no longer maintained.
Using PyPy indeed rules out py2exe and similar tools, at least until one is ported (AFAIK there is no active work on that). Still, as PyPy binaries do not need to be installed, you might get away with a more complicated distribution that includes both your Python source code and a PyPy binary+stdlib and uses a small wrapper (batch file, executable) to ease launching. I can't comment on whether WxPython on PyPy is mature enough to be used, but perhaps someone on pypy-dev, wxpython-dev or either one's IRC channel can give a recommendation if you describe your situation.
Translating your code into RPython does not seem viable to me. The translation toolchain is not really a tool for general purpose development, and producing a C dll for embedding/ctypes seems nontrivial. Also, RPython code really is low-level, making your Python code restricted enough may amount to rewriting half of it.
As for other restricted options: You seem to mix up CPython (the original Python interpreter written in C) with Cython (a compiler for a Python-like language that emits C code suitable for CPython extension modules). Both projects are active. I'm not very familiar with Shedskin, but it seems to be a tool for developing whole programs, with little or no interaction with non-restricted Python code. Cython seems a much better fit: Although it requires manual type annotations and lower-level code to achieve really good performance, it's trivial to use from Python: The very purpose of the project is producing extension modules.
I would definitely look into Cython, I've been playing with it some and have seen speedups of ~100x over pure python. Use the profile module to find the bottlenecks first. Usually the loops are the biggest chances to increase speed when going to Cython. You should also look into seeing if you can use array/vector operations in Numpy instead of loops, if so that can also give extreme performance boosts. For instance:
a = range(1000000)
for i in range(len(a)):
a[i] += 5
is slow, real slow. On the other hand:
a = numpy.arange(10000000)
a = a +5
is fast, real fast.
Correction: shedskin can be used to generare extention modules, as well as whole programs.
I have a few queries regarding python
Why is there no python compiler to create native code? I have found py2exe etc but they just pack a python interpreter along with them and hence, it is again the interpreter executing the code.
Is it not possible to create a python compiler like a LISP compiler and hence the code will execute faster(compared to C++)?
Thanks,
Vinay
Nuitka – Python Compiler
What it is
I thought there ought to be possible to use a compiler for Python, a better compiler than what CPython already has with its bytecode. This is what Nuitka is supposed to be.
It is my attempt to translate pure Python not into bytecode, but into machine code (via C++ compiler), while using libpython at run time. And then to do compile time and also run time analysis to speculatively execute things in a faster mode if certain expectations are met.
Question 1:
Nuitka (Direct Python code to C++)
ShedSkin (Compiles implicitly statically typed Python to C++, stand-alone programs or
extension modules)
Cython (From a superset of Python to C
extensions. Cython comes from Pyrex)
Question 2:
Not sure if I get it correctly but maybe the answer is:
psyco (A Just in time compiler (JIT) for Python Code, the
predecessor of the PyPy JIT )
The nearest equivalents for Python are cython and pypy.
There is, sort of.
See Cython -- I haven't had a chance to fully explore it yet, but as best as I can tell, it directly compiles Python code. You can also use (optional) static typing -- it won't be vanilla Python anymore, but it can lead to a speed boost, if you do it right. Also, see this: Can Cython compile to an EXE?
It might be because I don't have much experience with Lisp, but I'm not entirely sure by what you mean by 'create a Python compiler like a Lisp compiler'.
Numba is a newer Python compiler based on NumPy & LLVM, which falls back to CPython.