How do I properly include cython source files in a python meson project managed by the meson build system?
Cython as a first class language is still in progress (I'm the author of that work), right now the right way is with a generator or custom target, then compile an extension module:
pyx_c = custom_target(
'cython_file.c',
output : 'cython_file.c',
input : 'cython_file.pyx',
command : [cython, '#INPUT#', '-o', '#OUTPUT#'],
)
import('python').find_installation().extension_module(
'my_extension_module'
pyx_c,
install : true
)
Here's an example from the meson test suite, which is pretty close to my example above.
EDIT: cython as a first class language has landed. So if you can rely on Meson 0.59.0, you can just do:
import('python').find_installation().extension_module(
'my_extension_module'
'cython_file.pyx',
install : true
)
The simplest way I found (completely code-side) was to add them like any other source file and when you need to import them in a python file
just add
from pyximport import install
install(language_level=3)
before importing.
Best way is #dcbaker's though. I wrote an example pygobject cython application to show that here.
Related
The library I'm working on generates python files according to an executable (which turns ANTLRv4 .g4 files into python files), and I have the following install step:
import os
import subprocess
from setuptools import setup
from setuptools.command.install import install
class AntlrInstallCommand(install):
def run(self):
output_dir = compile_grammar()
print(f"Compiled ANTLRv4 grammar in {output_dir}")
install.run(self)
def compile_grammar():
parser_dir = os.path.join(os.path.dirname(__file__), "my_project/grammar")
subprocess.check_output(
["antlr", "MyGrammar.g4", "-Dlanguage=Python3", "-visitor", "-o", "gen"],
cwd=parser_dir,
)
# The result is created in the subfolder `gen`
return os.path.join(parser_dir, "gen")
setup(
...
install_requires=[
"setuptools",
"antlr4-python3-runtime==4.9.2",
...
],
cmdclass={"install": AntlrInstallCommand},
license="MIT",
python_requires=">=3.6",
)
Which works great if I'm pip install'ing the project on a machine that has antlr installed (since I'm calling it via subprocess).
Ideally, attempting to do this on a machine that doesn't have antlr installed would first install the executable(with the correct version) in either a system directory like /usr/bin, or whatever relevant python bin directory we're working in, but right now it errors out with the following message(which is expected):
running install
error: [Errno 2] No such file or directory: 'antlr'
----------------------------------------
ERROR: Failed building wheel for my_project
I see a couple of solutions each with slight caveats:
sympy uses ANTLR, but it requires the user to install antlr first. See here
setuptools-antlr allows me to download an antlr jar as a giant blob in a python package, and then I can invoke it here. However, the version doesn't match mine (which is 4.9.2).
java2python precompiles the files for me and writes them into the github repo. However, these files are extremely large and are very hard to read as they're autogenerated. If I slightly modify the grammar and don't modify the parser it would also lead to unexpected bugs. As a result, I would like to hide this complexity from the repository as it's tangential to development.
If I can get the right version of the antlr binary and be able to invoke it at install time, that would be optimal. Otherwise I'm okay with picking one of these alternatives. Any suggestions for either case would be appreciated.
I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?
As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.
An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.
I'm trying to use the CFFI package in Python to create a Python interface for already existing C-code.
I am able to compile a C library by following this blog post. Now I want to make it so that this python library is available without any fancy updates to the sys.path.
I found that maybe creating a distribution through Python's setuptools setup() function would accomplish this and I got it to mostly work by creating a setup.py file as such
import os
import sys
from setuptools import setup, find_packages
os.chdir(os.path.dirname(sys.argv[0]) or ".")
setup(
name="SobelFilterTest",
version="0.1",
description="An example project using Python's CFFI",
packages=find_packages(),
install_requires=["cffi>=1.0.0"],
setup_requires=["cffi>=1.0.0"],
cffi_modules=[
"./src/build_sobel.py:ffi",
"./src/build_file_operations.py:ffi",
],
)
, but I run into this error
build/temp.linux-x86_64-3.5/_sobel.c:492:19: fatal error: sobel.h: No such file or directory
From what I can tell, the problem is that the sobel.h file does not get uploaded into the build folder created by setuptools.setup(). I looked for suggestions of what to do including using Extensions() and writing a MANIFEST.in file, and both seem to add a relative path to the correct header files:
MANIFEST.in
setup.py
SobelFilterTest.egg-info/PKG-INFO
SobelFilterTest.egg-info/SOURCES.txt
SobelFilterTest.egg-info/dependency_links.txt
SobelFilterTest.egg-info/requires.txt
SobelFilterTest.egg-info/top_level.txt
src/file_operations.h
src/macros.h
src/sobel.h
But I still get the same error message. Is there a correct way to go about adding the header file to the build folder? Thanks!
It's actually not pip that is missing the .h file, but rather the compiler (like gcc). Therefore it's not about adding the missing file to setup, but rather make sure that cffi can find it. One way (like mentioned in the comments) is to make it available to the compiler through environment variables, but there is another way.
When setting the source with cffi you can add directories for the compiler like this:
from cffi import FFI
ffibuilder = FFI()
ffibuilder.set_source("<YOUR SOURCE HERE>", include_dirs=["./src"])
# ... Rest of your code
"""
I'm trying to compile some project I've found on the web.
the project is wrapping some Fortran code into python object.
The author of that project wrote I need to run the setup.py file.
I've done that. It contains the following piece of code:
ext = Extension(name = "GaussianFitter._Fitter",
sources = ["GaussianFitter/src/lmdif.f",
"GaussianFitter/src/splev.f",
"GaussianFitter/src/gaussian.f90"],
and obviously all these "f" files are existing.
When I install that setup.py file it seams to do some job succesfully but the first line in the script which uses that _Fitter:
import _Fitter
doesn't work.
I'm newby with python, and have no experiment with Fortran at all, so please forgive my ignorance.
Edit: project available here:
https://github.com/ardiloot/GaussianFitter
Thanks!
Finally made it, so here's my for-begginers tutorial:
note - since it's for beginners - it sets all in the global scope.
install mingw32 from:
http://www.mingw.org/
Add it's bin path to system PATH.
add a setup.cfg file containing the following text:
[build]
compiler=mingw32
locate it in %pythondir%\Lib\distutils
since setup.py file contains links to .c or .f files (c++/fortran/c) in order to make command (python setup.py install) be able to locate these files - run the command when working directory (i.e. cd c:....\) is set.
Ignore red error line in code when you import that module. It is in the LIB directory, just try to run...
EDIT:
If still not managing to import the already built f files - building it in the following way might help:
python setup.py build_ext --inplace
which generates an object (*.pyd), this helped me after it, for any reason, stopped working.
I've compiled the python wrapper of nanomsg and I want to create a python installer for the package.The package can be created by running
python setup.py bdist --format=wininst
However I would like nanomsg.dll/nanomsg.so to be included in the installer/package but I haven't found any documentation regarding this issue.
As stated in the documentation here one needs to add the following code to his setup.py script:
setup(
name='nanomsg',
version=__version__,
packages=[str('nanomsg'), str('_nanomsg_ctypes'), str('nanomsg_wrappers')],
data_files=[(
'lib\\site-packages\\', ["C:\\Dev\\external\\nanomsg\\x86\\Release\\nanomsg.dll"]
)],
)