I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?
As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.
An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.
Related
Given a package random.whl containing hello.py:
print("Hello World!")
Is there a way to create a setup.py, setup.cfg or pyproject.toml, that when executed, will install the package in such a way that hello.py will be executed every time Python is started?
pip install random.whl
python unrelated.py # First prints "Hello World", then continues on.
I know it's possible to hook on readline.py that Python automatically loads, but is there a different and less "hacky" way to achieve it?
Some impossible ways that I thought of:
Running a post-install script on a .whl distribution (post-install is only avaiable on sdist).
Modifying PYTHONSTARTUP env variable or copying files.
Changing the import machinery.
While being a security risk, a method achieving it is good for implementing debuggers or auditing tools without requiring a change in either pre-compiled or post-compiled Python code, or used for penetration testing in side-channel attacks.
So far, using sitecustomize.py and publishing an sdist with a custom install command was the most reliable, and worked in virtual environments unlike usercustomize.py or .pth files.
Relevant setup.py code:
import sys
import os
import setuptools
import sysconfig
from setuptools.command.install import install
class PreInstall(install):
def run(self):
site_packages_dir = sysconfig.get_path("purelib")
sitecustomize_path = os.path.join(site_packages_dir, "sitecustomize.py")
if os.path.exists(sitecustomize_path):
raise FileExistsError("Site customize file already exists. "
"Please remove it before installing.")
install.run(self)
setuptools.setup(
name='...',
version='0.0.1',
py_modules=["sitecustomize"],
cmdclass={'install': PreInstall,},
)
It still doesn't work with .whl distributions as it might overwrite an existing sitecustomize without being able to check.
The library I'm working on generates python files according to an executable (which turns ANTLRv4 .g4 files into python files), and I have the following install step:
import os
import subprocess
from setuptools import setup
from setuptools.command.install import install
class AntlrInstallCommand(install):
def run(self):
output_dir = compile_grammar()
print(f"Compiled ANTLRv4 grammar in {output_dir}")
install.run(self)
def compile_grammar():
parser_dir = os.path.join(os.path.dirname(__file__), "my_project/grammar")
subprocess.check_output(
["antlr", "MyGrammar.g4", "-Dlanguage=Python3", "-visitor", "-o", "gen"],
cwd=parser_dir,
)
# The result is created in the subfolder `gen`
return os.path.join(parser_dir, "gen")
setup(
...
install_requires=[
"setuptools",
"antlr4-python3-runtime==4.9.2",
...
],
cmdclass={"install": AntlrInstallCommand},
license="MIT",
python_requires=">=3.6",
)
Which works great if I'm pip install'ing the project on a machine that has antlr installed (since I'm calling it via subprocess).
Ideally, attempting to do this on a machine that doesn't have antlr installed would first install the executable(with the correct version) in either a system directory like /usr/bin, or whatever relevant python bin directory we're working in, but right now it errors out with the following message(which is expected):
running install
error: [Errno 2] No such file or directory: 'antlr'
----------------------------------------
ERROR: Failed building wheel for my_project
I see a couple of solutions each with slight caveats:
sympy uses ANTLR, but it requires the user to install antlr first. See here
setuptools-antlr allows me to download an antlr jar as a giant blob in a python package, and then I can invoke it here. However, the version doesn't match mine (which is 4.9.2).
java2python precompiles the files for me and writes them into the github repo. However, these files are extremely large and are very hard to read as they're autogenerated. If I slightly modify the grammar and don't modify the parser it would also lead to unexpected bugs. As a result, I would like to hide this complexity from the repository as it's tangential to development.
If I can get the right version of the antlr binary and be able to invoke it at install time, that would be optimal. Otherwise I'm okay with picking one of these alternatives. Any suggestions for either case would be appreciated.
This doesn't make sense to me. How can I use the setup.py to install Cython and then also use the setup.py to compile a library proxy?
import sys, imp, os, glob
from setuptools import setup
from Cython.Build import cythonize # this isn't installed yet
setup(
name='mylib',
version='1.0',
package_dir={'mylib': 'mylib', 'mylib.tests': 'tests'},
packages=['mylib', 'mylib.tests'],
ext_modules = cythonize("mylib_proxy.pyx"), #how can we call cythonize here?
install_requires=['cython'],
test_suite='tests',
)
Later:
python setup.py build
Traceback (most recent call last):
File "setup.py", line 3, in <module>
from Cython.Build import cythonize
ImportError: No module named Cython.Build
It's because cython isn't installed yet.
What's odd is that a great many projects are written this way. A quick github search reveals as much: https://github.com/search?utf8=%E2%9C%93&q=install_requires+cython&type=Code
As I understand it, this is where PEP 518 comes in - also see some clarifications by one of its authors.
The idea is that you add yet another file to your Python project / package: pyproject.toml. It is supposed to contain information on build environment dependencies (among other stuff, long term). pip (or just any other package manager) could look into this file and before running setup.py (or any other build script) install the required build environment. A pyproject.toml could therefore look like this:
[build-system]
requires = ["setuptools", "wheel", "Cython"]
It is a fairly recent development and, as of yet (January 2019), it is not finalized / approved by the Python community, though (limited) support was added to pip in May 2017 / the 10.0 release.
One solution to this is to not make Cython a build requirement, and instead distribute the Cython generated C files with your package. I'm sure there is a simpler example somewhere, but this is what pandas does - it conditionally imports Cython, and if not present can be built from the c files.
https://github.com/pandas-dev/pandas/blob/3ff845b4e81d4dde403c29908f5a9bbfe4a87788/setup.py#L433
Edit: The doc link from #danny has an easier to follow example.
http://docs.cython.org/en/latest/src/reference/compilation.html#distributing-cython-modules
When you use setuptool, you should add cython to setup_requires (and also to install_requires if cython is used by installation), i.e.
# don't import cython, it isn't yet there
from setuptools import setup, Extension
# use Extension, rather than cythonize (it is not yet available)
cy_extension = Extension(name="mylib_proxy", sources=["mylib_proxy.pyx"])
setup(
name='mylib',
...
ext_modules = [cy_extension],
setup_requires=["cython"],
...
)
Cython isn't imported (it is not yet available when setup.pystarts), but setuptools.Extension is used instead of cythonize to add cython-extension to the setup.
It should work now. The reason: setuptools will try to import cython, after setup_requires are fulfilled:
...
try:
# Attempt to use Cython for building extensions, if available
from Cython.Distutils.build_ext import build_ext as _build_ext
# Additionally, assert that the compiler module will load
# also. Ref #1229.
__import__('Cython.Compiler.Main')
except ImportError:
_build_ext = _du_build_ext
...
It becomes more complicated, if your Cython-extension uses numpy, but also this is possible - see this SO post.
It doesn't make sense in general. It is, as you suspect, an attempt to use something that (possibly) has yet to be installed. If tested on a system that already has the dependency installed, you might not notice this defect. But run it on a system where your dependency is absent, and you will certainly notice.
There is another setup() keyword argument, setup_requires, that can appear to be parallel in form and use to install_requires, but this is an illusion. Whereas install_requires triggers a lovely ballet of automatic installation in environments that lack the dependencies it names, setup_requires is more documentation than automation. It won't auto-install, and certainly not magically jump back in time to auto-install modules that have already been called for in import statements.
There's more on this at the setuptools docs, but the quick answer is that you're right to be confused by a module that is trying to auto-install its own setup pre-requisites.
For a practical workaround, try installing cython separately, and then run this setup. While it won't fix the metaphysical illusions of this setup script, it will resolve the requirements and let you move on.
I have created a package called clearplot that wraps around matplotlib. I have also created a nice font that I want to distribute with my package. I consulted this section of the Python Packaging User guide, and determined that I should use the data_files keyword. I chose data_files instead of package_data since I need to install the font in a matplotlib directory that is outside of my package.
Here is my first, flawed, attempt at a setup.py file:
from distutils.core import setup
import os, sys
import matplotlib as mpl
#Find where matplotlib stores its True Type fonts
mpl_data_dir = os.path.dirname(mpl.matplotlib_fname())
mpl_ttf_dir = os.path.join(mpl_data_dir, 'fonts', 'ttf')
setup(
...(edited for brevity)...
install_requires = ['matplotlib >= 1.4.0, !=1.4.3', 'numpy >= 1.6'],
data_files = [
(mpl_ttf_dir, ['./font_files/TeXGyreHeros-txfonts/TeXGyreHerosTXfonts-Regular.ttf']),
(mpl_ttf_dir, ['./font_files/TeXGyreHeros-txfonts/TeXGyreHerosTXfonts-Italic.ttf'])]
)
#Try to delete matplotlib's fontList cache
mpl_cache_dir = mpl.get_cachedir()
mpl_cache_dir_ls = os.listdir(mpl_cache_dir)
if 'fontList.cache' in mpl_cache_dir_ls:
fontList_path = os.path.join(mpl_cache_dir, 'fontList.cache')
os.remove(fontList_path)
There are two issues with this setup.py:
I attempt to import matplotlib before setup() has a chance to install it. This is an obvious booboo, but I needed to know where mpl_ttf_dir was before I ran setup().
As mentioned here, wheel distributions do not support absolute paths for data_files. I didn't think this would be a problem because I thought I would just use a sdist distribution. (sdists do allow absolute paths.) Then I came to find out that pip 7.0 (and later) converts all packages to wheel distributions, even if the distribution was originally created as a sdist.
I was quite annoyed by issue #2, but, since then, I found out that absolute paths are bad because they do not work with virtualenv. Thus, I am now willing to change my approach, but what do I do?
The only idea I have is to distribute the font as package_data first and then move the font to the proper location afterwards using the os module. Is that a kosher method?
Thanks to #benjaoming's answer and this blog post, here is what I came up with:
from setuptools import setup
from setuptools.command.install import install
import warnings
#Set up the machinery to install custom fonts. Subclass the setup tools install
#class in order to run custom commands during installation.
class move_ttf(install):
def run(self):
"""
Performs the usual install process and then copies the True Type fonts
that come with clearplot into matplotlib's True Type font directory,
and deletes the matplotlib fontList.cache
"""
#Perform the usual install process
install.run(self)
#Try to install custom fonts
try:
import os, shutil
import matplotlib as mpl
import clearplot as cp
#Find where matplotlib stores its True Type fonts
mpl_data_dir = os.path.dirname(mpl.matplotlib_fname())
mpl_ttf_dir = os.path.join(mpl_data_dir, 'fonts', 'ttf')
#Copy the font files to matplotlib's True Type font directory
#(I originally tried to move the font files instead of copy them,
#but it did not seem to work, so I gave up.)
cp_ttf_dir = os.path.join(os.path.dirname(cp.__file__), 'true_type_fonts')
for file_name in os.listdir(cp_ttf_dir):
if file_name[-4:] == '.ttf':
old_path = os.path.join(cp_ttf_dir, file_name)
new_path = os.path.join(mpl_ttf_dir, file_name)
shutil.copyfile(old_path, new_path)
print "Copying " + old_path + " -> " + new_path
#Try to delete matplotlib's fontList cache
mpl_cache_dir = mpl.get_cachedir()
mpl_cache_dir_ls = os.listdir(mpl_cache_dir)
if 'fontList.cache' in mpl_cache_dir_ls:
fontList_path = os.path.join(mpl_cache_dir, 'fontList.cache')
os.remove(fontList_path)
print "Deleted the matplotlib fontList.cache"
except:
warnings.warn("WARNING: An issue occured while installing the custom fonts for clearplot.")
setup(...
#Specify the dependencies and versions
install_requires = ['matplotlib >= 1.4.0, !=1.4.3', 'numpy >= 1.6'],
#Specify any non-python files to be distributed with the package
package_data = {'' : ['color_maps/*.csv', 'true_type_fonts/*.ttf']},
#Specify the custom install class
cmdclass={'install' : move_ttf}
)
This solves both problem #1 (it installs matplotlib before it imports it) and problem #2 (it works with wheels).
The only idea I have is to distribute the font as package_data first and then move the font to the proper location afterwards using the os module. Is that a kosher method?
I would consider doing exactly this. I know your package may not be an obvious candidate for virtualenvs, but consider that python packages may be installed only to a user-writable location. Thus, copying the font when you first run your programme and detecting the correct location, might prompt you to do stuff in a better manner than possible through setup.py, stuff like: Elevate privileges through a password prompt in case it's needed, ask for a different location in case you fail to detect it, prompt if you are over-writing existing system files etc.
I once tried arguing that Python packages should be able to place stuff in /etc, but I realized the benefits were small compared to just creating a proper native package for the target OS, i.e. a debian package for Debian or a .exe installer for Windows.
The bottom line is that wheel and setuptools are not package managers for your entire OS, but just for what's in some local site-packages/.
I hope this answer gives you enough background to avoid data_files. One last good reason: Making it work across distutils, setuptools, and wheel is a no-go.
I use the following script to distribute a module containing pure python code.
from distutils.core import setup, Extension
import os
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
)
The problem I have is, I modified one of the files that uses an external library (a .so file), which I need to ship along with the existing module. I was suggested to use package_data to include the library. I modified the script to the following.
from distutils.core import setup, Extension
import os
data_dir = os.path.abspath('../lib64/')
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
package_data={'mtester':[data_dir+'mhelper.so']},
)
The problem is, adding package_data did not make any difference. This is not installing the mhelper.so in any location (neither in site-packages nor in site-packages/mtester).
System info: Fedora 10, 64 bit, python 2.5 (Yes it is ancient. But it is our build machine, and it needs to stay that way to maintain backward compatibility)
Any suggestions that would help me resolve this would be well appreciated!
Unfortunately package_data looks for files relative to the top of the package. One fix is to move the helper library under the module dir with the rest of the code:
% mv lib64/mhelper.so module/
Then modify the package_data argument accordingly:
package_data = {'mtester': ['mhelper.so']}
...
Then test:
% python setup.py bdist
% tar tf dist/mtester-0.1.linux-x86_64.tar.gz | grep mhelper
./usr/local/lib/python2.5/dist-packages/mtester/mhelper.so