I wrote a python/c extension file lda_model.c
and I added setup.py:
from setuptools import setup, Extension
modules = [Extension('c_lda_model', sources=["lda_model.c"])]
setup(ext_modules=modules)
Now I have to compile the C code by
python setup.py build
before running python code where call the C code.
Is there any way to automatically compile the invoked C extension,
while running the python code?
Not with the standard way of writing extensions, which is what you just do.
However, there are a couple other approaches of writing native code extensions to Python which do compilation in execution time.
One such example is Weave that comes with Scipy - there are others ways if you look for.
Related
My goal is to have the ability to call functions in C++ with meaningful arguments.
I can't do that with just subprocess.call because then I go into main(int argc,char** argv) and I have a bunch of strings to deal with. I do not want to have to parse matrices out of strings.
I'm trying to use Cython because that seems like the reasonable thing to do. But although there are a good amount of guides for getting Cython running most of them are for 2.7, and it's rare to see two advise the same thing.
My question basically is does anybody here know how to get Cython running on Py3.5? Or know of a guide or something? I'm lost.
Okay so I had a pretty silly mistake, was compiling with msvs, spent so much time trying to get mingw to work but forget that, 'msvc' does the trick. For any passersby if you're on 3.5+ you should be using Visual Studio 2015. After installing cython with 'pip3 install cython', create a setup.py file where you put this
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize(
"TestCython.pyx", # our Cython source
#sources=["Rectangle.cpp"], # additional source file(s)
language="c++", # generate C++ code
))
Create a .pyx file (let's say 'TestCython.pyx') where you write whatever you want for example (let's say 'print("Hello World")'. The sources argument for cythonize is optional.
Then cd into where your .pyx and .py file is and run
'python setup.py build_ext --inplace --compiler=msvc'
This should compile to a .cpp and then .pyd file (the latter is the one you will use). If you just had the hello world that will get printed out as soon as you import TestCython.
Refer to the docs and google for anything else. ;)
I'm building a standalone executable using Cython on Linux.
I have the following code:
import psycopg2 as pg
conn = pg.connect('dbname=**** user=**** password=****')
cur = conn.cursor()
cur.execute('SELECT version()')
print(cur.fetchone())
The problem is when the machine does not have the Python package psycopg2 installed, throws the following exception:
Traceback (most recent call last):
File "test.py", line 2, in init test (test.c:872)
import psycopg2 as pg
ImportError: No module named 'psycopg2'
Im building using the --embed cython flag.
How can I make Cython to compile that particular package too?
From my experience, it is not that straightforward to create a standalone executable from multiples python files (yours or from dependencies like psycopg2).
I would say there are a couple of approaches here I would try:
The first one would be cython_freeze https://github.com/cython/cython/tree/master/Demos/freeze I do not use it myself, so I cannot tell much.
The second one is to use pyinstaller to create such executable. It takes as input the .py or .pyc files and embed them into one executable, together with the python interpreter and required dependencies, so you don't have to install anything on the target machine. Note, however, that your code will run as interpreted python and can be easily decompiled and inspected.
If you really need to compile (cythonize) your code, then you can first cythonize() and the build with setup() your extensions, then run pyinstaller as above (taking care that it doesnt find the .py or .pyc files, just the .pyd or .so extensions) to generate the standalone executable. In both cases, pyinstaller will collect all your dependencies and embed them in the executable (even if it fails, you can tell pyinstaller to embed them with hidden_imports).
There are surely other approaches, like py2exe, but when I researched and played with several technologies some months ago, pyinstaller was the best option for me. I do the process in win, linux and mac without many changes.
EDIT: I didn't realize that the example is python 3. Pyinstaller only works for 2.x now.
--embed means the Python interpreter is embedded in your executable. It does not mean independence from Python. It does not do what you think. It sounds more like you need a tool like py2exe, py2app or pyfreeze.
Nuitka is the tool that you need.
You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.
Right now Nuitka is a good replacement for the Python interpreter and compiles every construct that CPython 2.6, 2.7, 3.2, 3.3 and 3.4 offer. It translates the Python into a C++ program that then uses "libpython" to execute in the same way as CPython does, in a very compatible way.
It is somewhat faster than CPython already, but currently it doesn't make all the optimizations possible, but a 258% factor on pystone is a good start (number is from version 0.3.11).
I downloaded a pyx and c file from the internet and I am just wondering how can i incorporate it into python? The documentation on Cython is fairly vague and mainly focus on generating pyx/c from py file. It would be great if you could give me some solid examples on how to do this properly. Many thanks
The Cython executable turns a .pyx file into a .c file. You then use your favorite C build tool to compile it into a shared library (e.g. an .so file on Linux). Then poof, you have an extension module. Note that you'll need to figure out all the extra arguments to your C compiler like the header paths for Python and numpy. These all depend very heavily on not only your OS but also the particulars of how you've installed Python and SciPy on it.
If this all sounds a bit scary, see if the .pyx file is simple enough that you can use pyximport to handle all the messy compilation for you. If external libraries are needed, you'll probably need to construct a .pyxbld file.
Cython is a compiler: it converts a .pyx into a .c, which you then build to a .so or .pyd. Take a look at the cython docs on compilation. You will probably want to use pyximport module technique if you want to modify the code and experiment, and then use a setup.py when done and you need a final version of your Cython .pyx module.
I'm a Matlab and C++ user, and have recently discovered python (spyder) as a possible replacement for both. One of the main benefits I thought python had was the ability to work in interpreter mode, and then seamlessly translate it into fast compiled code once I'm satisfied with the result. The interpreted environment is great for prototyping, analyzing data while stopped at a breakpoint, throwing plots and images all around, etc.
I started looking into Cython, and I don't fully understand the programming flow. Lets say you have a .py code you'd like to speed up - Do you have to write a .pyx file from scratch? Can you run a .pyx file in interpreted mode as if it were a regular .py file (before compiling)? How do you debug the code in a .pyx file?
I don't have too much experience with Cython, but judging from this entry in their documentation, the recommended workflow is to have a setup.py file with the following lines:
from distutils.core import setup
from Cython.Build import cythonize
setup(name='Hello world app', ext_modules=cythonize("hello.pyx"))
Here hello.pyx is just an example file, you will have to replace the string to reference your Python script.
Afterwards you will be able to call
python setup.py build_ext --inplace
which will compile your code and leave you with a new file. Now, as long as that file is in the same directory, you can easily import what you defined in your file, just like with any other module. E.g., suppose you compiled a file hello.pyx with the function f, you could write:
from hello import f
and then proceed to use f.
Now, regarding your other questions. .pyx seems to just indicate, that this should be Cython code, there is no real difference. Using the method with a setup.py script as described above, you could also reference a file with the ending .py. However, Python won't allow you to import from .pyx files, only from the files created after compiling.
As to how you would debug code in a .pyx file, I don't have enough information on that, though you could probably just debug the non-compiled file like a .py file.
I wonder if it is possible to create an executable module from a Python script. I need to have the most performance and the flexibility of Python script, without needing to run in the Python environment. I would use this code to load on demand user modules to customize my application.
There's pyrex that compiles python like source to python extension modules
rpython which allows you to compile python with some restrictions to various backends like C, LLVM, .Net etc.
There's also shed-skin which translates python to C++, but I can't say if it's any good.
PyPy implements a JIT compiler which attempts to optimize runtime by translating pieces of what's running at runtime to machine code, if you write for the PyPy interpreter that might be a feasible path.
The same author that is working on JIT in PyPy wrote psyco previously which optimizes python in the CPython interpreter.
You can use something like py2exe to compile your python script into an exe, or Freeze for a linux binary.
see: How can I create a directly-executable cross-platform GUI app using Python?
I've had a lot of success using Cython, which is based on and extends pyrex:
Cython is a language that makes
writing C extensions for the Python
language as easy as Python itself.
Cython is based on the well-known
Pyrex, but supports more cutting edge
functionality and optimizations.
The Cython language is very close to
the Python language, but Cython
additionally supports calling C
functions and declaring C types on
variables and class attributes. This
allows the compiler to generate very
efficient C code from Cython code.
This makes Cython the ideal language
for wrapping for external C libraries,
and for fast C modules that speed up
the execution of Python code.
I think you can use jython to compile python to Java bytecode, and then compile that with GCJ.