My goal is to have the ability to call functions in C++ with meaningful arguments.
I can't do that with just subprocess.call because then I go into main(int argc,char** argv) and I have a bunch of strings to deal with. I do not want to have to parse matrices out of strings.
I'm trying to use Cython because that seems like the reasonable thing to do. But although there are a good amount of guides for getting Cython running most of them are for 2.7, and it's rare to see two advise the same thing.
My question basically is does anybody here know how to get Cython running on Py3.5? Or know of a guide or something? I'm lost.
Okay so I had a pretty silly mistake, was compiling with msvs, spent so much time trying to get mingw to work but forget that, 'msvc' does the trick. For any passersby if you're on 3.5+ you should be using Visual Studio 2015. After installing cython with 'pip3 install cython', create a setup.py file where you put this
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize(
"TestCython.pyx", # our Cython source
#sources=["Rectangle.cpp"], # additional source file(s)
language="c++", # generate C++ code
))
Create a .pyx file (let's say 'TestCython.pyx') where you write whatever you want for example (let's say 'print("Hello World")'. The sources argument for cythonize is optional.
Then cd into where your .pyx and .py file is and run
'python setup.py build_ext --inplace --compiler=msvc'
This should compile to a .cpp and then .pyd file (the latter is the one you will use). If you just had the hello world that will get printed out as soon as you import TestCython.
Refer to the docs and google for anything else. ;)
Related
I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.
Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.
I need to make some modifications to scikit-learn, including changes to the cython code.
I haven't worked on cython before, so could do with some guidance - so far I have got all the dependencies going in a python virtualenv, and cloned and installed the sklearn git.
Now, what is a good workflow for modifying the .pyx files? Should I make modifications and then reinstall to see the effects? Or build instead?
Is there any way to avoid recompiling all the stuff that is unchanged?
I have heard of import pyximport; pyximport.install() but for me this throws a compile error with sklearn -> is there a way to ensure it uses the same options as the Makefile which runs successfully?
In general I am looking for guidance on how to modify a large cython project without spending decades waiting for unmodified files to recompile.
You could simply run,
python setup.py develop
after each modification. Unlike the install command this will not copy any files and only creates a symbolic link to the working directory. It will also automatically build all the necessary extensions in place, in an equivalent of
python setup.py build_ext --inplace
If you change a Cython file in your project, only those files will be recompiled next time you run the develop command.
The pyximport module is nice for standalone Cython functions. However, for a more complex project, with multiple files, the above approach would probably be simpler.
I wrote a python/c extension file lda_model.c
and I added setup.py:
from setuptools import setup, Extension
modules = [Extension('c_lda_model', sources=["lda_model.c"])]
setup(ext_modules=modules)
Now I have to compile the C code by
python setup.py build
before running python code where call the C code.
Is there any way to automatically compile the invoked C extension,
while running the python code?
Not with the standard way of writing extensions, which is what you just do.
However, there are a couple other approaches of writing native code extensions to Python which do compilation in execution time.
One such example is Weave that comes with Scipy - there are others ways if you look for.
I downloaded a pyx and c file from the internet and I am just wondering how can i incorporate it into python? The documentation on Cython is fairly vague and mainly focus on generating pyx/c from py file. It would be great if you could give me some solid examples on how to do this properly. Many thanks
The Cython executable turns a .pyx file into a .c file. You then use your favorite C build tool to compile it into a shared library (e.g. an .so file on Linux). Then poof, you have an extension module. Note that you'll need to figure out all the extra arguments to your C compiler like the header paths for Python and numpy. These all depend very heavily on not only your OS but also the particulars of how you've installed Python and SciPy on it.
If this all sounds a bit scary, see if the .pyx file is simple enough that you can use pyximport to handle all the messy compilation for you. If external libraries are needed, you'll probably need to construct a .pyxbld file.
Cython is a compiler: it converts a .pyx into a .c, which you then build to a .so or .pyd. Take a look at the cython docs on compilation. You will probably want to use pyximport module technique if you want to modify the code and experiment, and then use a setup.py when done and you need a final version of your Cython .pyx module.
I'm a Matlab and C++ user, and have recently discovered python (spyder) as a possible replacement for both. One of the main benefits I thought python had was the ability to work in interpreter mode, and then seamlessly translate it into fast compiled code once I'm satisfied with the result. The interpreted environment is great for prototyping, analyzing data while stopped at a breakpoint, throwing plots and images all around, etc.
I started looking into Cython, and I don't fully understand the programming flow. Lets say you have a .py code you'd like to speed up - Do you have to write a .pyx file from scratch? Can you run a .pyx file in interpreted mode as if it were a regular .py file (before compiling)? How do you debug the code in a .pyx file?
I don't have too much experience with Cython, but judging from this entry in their documentation, the recommended workflow is to have a setup.py file with the following lines:
from distutils.core import setup
from Cython.Build import cythonize
setup(name='Hello world app', ext_modules=cythonize("hello.pyx"))
Here hello.pyx is just an example file, you will have to replace the string to reference your Python script.
Afterwards you will be able to call
python setup.py build_ext --inplace
which will compile your code and leave you with a new file. Now, as long as that file is in the same directory, you can easily import what you defined in your file, just like with any other module. E.g., suppose you compiled a file hello.pyx with the function f, you could write:
from hello import f
and then proceed to use f.
Now, regarding your other questions. .pyx seems to just indicate, that this should be Cython code, there is no real difference. Using the method with a setup.py script as described above, you could also reference a file with the ending .py. However, Python won't allow you to import from .pyx files, only from the files created after compiling.
As to how you would debug code in a .pyx file, I don't have enough information on that, though you could probably just debug the non-compiled file like a .py file.