I have created a package called clearplot that wraps around matplotlib. I have also created a nice font that I want to distribute with my package. I consulted this section of the Python Packaging User guide, and determined that I should use the data_files keyword. I chose data_files instead of package_data since I need to install the font in a matplotlib directory that is outside of my package.
Here is my first, flawed, attempt at a setup.py file:
from distutils.core import setup
import os, sys
import matplotlib as mpl
#Find where matplotlib stores its True Type fonts
mpl_data_dir = os.path.dirname(mpl.matplotlib_fname())
mpl_ttf_dir = os.path.join(mpl_data_dir, 'fonts', 'ttf')
setup(
...(edited for brevity)...
install_requires = ['matplotlib >= 1.4.0, !=1.4.3', 'numpy >= 1.6'],
data_files = [
(mpl_ttf_dir, ['./font_files/TeXGyreHeros-txfonts/TeXGyreHerosTXfonts-Regular.ttf']),
(mpl_ttf_dir, ['./font_files/TeXGyreHeros-txfonts/TeXGyreHerosTXfonts-Italic.ttf'])]
)
#Try to delete matplotlib's fontList cache
mpl_cache_dir = mpl.get_cachedir()
mpl_cache_dir_ls = os.listdir(mpl_cache_dir)
if 'fontList.cache' in mpl_cache_dir_ls:
fontList_path = os.path.join(mpl_cache_dir, 'fontList.cache')
os.remove(fontList_path)
There are two issues with this setup.py:
I attempt to import matplotlib before setup() has a chance to install it. This is an obvious booboo, but I needed to know where mpl_ttf_dir was before I ran setup().
As mentioned here, wheel distributions do not support absolute paths for data_files. I didn't think this would be a problem because I thought I would just use a sdist distribution. (sdists do allow absolute paths.) Then I came to find out that pip 7.0 (and later) converts all packages to wheel distributions, even if the distribution was originally created as a sdist.
I was quite annoyed by issue #2, but, since then, I found out that absolute paths are bad because they do not work with virtualenv. Thus, I am now willing to change my approach, but what do I do?
The only idea I have is to distribute the font as package_data first and then move the font to the proper location afterwards using the os module. Is that a kosher method?
Thanks to #benjaoming's answer and this blog post, here is what I came up with:
from setuptools import setup
from setuptools.command.install import install
import warnings
#Set up the machinery to install custom fonts. Subclass the setup tools install
#class in order to run custom commands during installation.
class move_ttf(install):
def run(self):
"""
Performs the usual install process and then copies the True Type fonts
that come with clearplot into matplotlib's True Type font directory,
and deletes the matplotlib fontList.cache
"""
#Perform the usual install process
install.run(self)
#Try to install custom fonts
try:
import os, shutil
import matplotlib as mpl
import clearplot as cp
#Find where matplotlib stores its True Type fonts
mpl_data_dir = os.path.dirname(mpl.matplotlib_fname())
mpl_ttf_dir = os.path.join(mpl_data_dir, 'fonts', 'ttf')
#Copy the font files to matplotlib's True Type font directory
#(I originally tried to move the font files instead of copy them,
#but it did not seem to work, so I gave up.)
cp_ttf_dir = os.path.join(os.path.dirname(cp.__file__), 'true_type_fonts')
for file_name in os.listdir(cp_ttf_dir):
if file_name[-4:] == '.ttf':
old_path = os.path.join(cp_ttf_dir, file_name)
new_path = os.path.join(mpl_ttf_dir, file_name)
shutil.copyfile(old_path, new_path)
print "Copying " + old_path + " -> " + new_path
#Try to delete matplotlib's fontList cache
mpl_cache_dir = mpl.get_cachedir()
mpl_cache_dir_ls = os.listdir(mpl_cache_dir)
if 'fontList.cache' in mpl_cache_dir_ls:
fontList_path = os.path.join(mpl_cache_dir, 'fontList.cache')
os.remove(fontList_path)
print "Deleted the matplotlib fontList.cache"
except:
warnings.warn("WARNING: An issue occured while installing the custom fonts for clearplot.")
setup(...
#Specify the dependencies and versions
install_requires = ['matplotlib >= 1.4.0, !=1.4.3', 'numpy >= 1.6'],
#Specify any non-python files to be distributed with the package
package_data = {'' : ['color_maps/*.csv', 'true_type_fonts/*.ttf']},
#Specify the custom install class
cmdclass={'install' : move_ttf}
)
This solves both problem #1 (it installs matplotlib before it imports it) and problem #2 (it works with wheels).
The only idea I have is to distribute the font as package_data first and then move the font to the proper location afterwards using the os module. Is that a kosher method?
I would consider doing exactly this. I know your package may not be an obvious candidate for virtualenvs, but consider that python packages may be installed only to a user-writable location. Thus, copying the font when you first run your programme and detecting the correct location, might prompt you to do stuff in a better manner than possible through setup.py, stuff like: Elevate privileges through a password prompt in case it's needed, ask for a different location in case you fail to detect it, prompt if you are over-writing existing system files etc.
I once tried arguing that Python packages should be able to place stuff in /etc, but I realized the benefits were small compared to just creating a proper native package for the target OS, i.e. a debian package for Debian or a .exe installer for Windows.
The bottom line is that wheel and setuptools are not package managers for your entire OS, but just for what's in some local site-packages/.
I hope this answer gives you enough background to avoid data_files. One last good reason: Making it work across distutils, setuptools, and wheel is a no-go.
Related
Given a package random.whl containing hello.py:
print("Hello World!")
Is there a way to create a setup.py, setup.cfg or pyproject.toml, that when executed, will install the package in such a way that hello.py will be executed every time Python is started?
pip install random.whl
python unrelated.py # First prints "Hello World", then continues on.
I know it's possible to hook on readline.py that Python automatically loads, but is there a different and less "hacky" way to achieve it?
Some impossible ways that I thought of:
Running a post-install script on a .whl distribution (post-install is only avaiable on sdist).
Modifying PYTHONSTARTUP env variable or copying files.
Changing the import machinery.
While being a security risk, a method achieving it is good for implementing debuggers or auditing tools without requiring a change in either pre-compiled or post-compiled Python code, or used for penetration testing in side-channel attacks.
So far, using sitecustomize.py and publishing an sdist with a custom install command was the most reliable, and worked in virtual environments unlike usercustomize.py or .pth files.
Relevant setup.py code:
import sys
import os
import setuptools
import sysconfig
from setuptools.command.install import install
class PreInstall(install):
def run(self):
site_packages_dir = sysconfig.get_path("purelib")
sitecustomize_path = os.path.join(site_packages_dir, "sitecustomize.py")
if os.path.exists(sitecustomize_path):
raise FileExistsError("Site customize file already exists. "
"Please remove it before installing.")
install.run(self)
setuptools.setup(
name='...',
version='0.0.1',
py_modules=["sitecustomize"],
cmdclass={'install': PreInstall,},
)
It still doesn't work with .whl distributions as it might overwrite an existing sitecustomize without being able to check.
The library I'm working on generates python files according to an executable (which turns ANTLRv4 .g4 files into python files), and I have the following install step:
import os
import subprocess
from setuptools import setup
from setuptools.command.install import install
class AntlrInstallCommand(install):
def run(self):
output_dir = compile_grammar()
print(f"Compiled ANTLRv4 grammar in {output_dir}")
install.run(self)
def compile_grammar():
parser_dir = os.path.join(os.path.dirname(__file__), "my_project/grammar")
subprocess.check_output(
["antlr", "MyGrammar.g4", "-Dlanguage=Python3", "-visitor", "-o", "gen"],
cwd=parser_dir,
)
# The result is created in the subfolder `gen`
return os.path.join(parser_dir, "gen")
setup(
...
install_requires=[
"setuptools",
"antlr4-python3-runtime==4.9.2",
...
],
cmdclass={"install": AntlrInstallCommand},
license="MIT",
python_requires=">=3.6",
)
Which works great if I'm pip install'ing the project on a machine that has antlr installed (since I'm calling it via subprocess).
Ideally, attempting to do this on a machine that doesn't have antlr installed would first install the executable(with the correct version) in either a system directory like /usr/bin, or whatever relevant python bin directory we're working in, but right now it errors out with the following message(which is expected):
running install
error: [Errno 2] No such file or directory: 'antlr'
----------------------------------------
ERROR: Failed building wheel for my_project
I see a couple of solutions each with slight caveats:
sympy uses ANTLR, but it requires the user to install antlr first. See here
setuptools-antlr allows me to download an antlr jar as a giant blob in a python package, and then I can invoke it here. However, the version doesn't match mine (which is 4.9.2).
java2python precompiles the files for me and writes them into the github repo. However, these files are extremely large and are very hard to read as they're autogenerated. If I slightly modify the grammar and don't modify the parser it would also lead to unexpected bugs. As a result, I would like to hide this complexity from the repository as it's tangential to development.
If I can get the right version of the antlr binary and be able to invoke it at install time, that would be optimal. Otherwise I'm okay with picking one of these alternatives. Any suggestions for either case would be appreciated.
I am using cython to cross-compile external python module. I am using python3.6 on the host and python3.5 on the target. Also I am compiling on x86_64 for target aarch64.
My setup.py looks like:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
import builder_config
import os
os.environ["PATH"] = builder_config.PATH
os.environ["CC"] = builder_config.COMPILER
os.environ["LDSHARED"] = builder_config.COMPILER + " -lpython3.5m -shared"
os.environ["CFLAGS"] = builder_config.CFLAGS
os.environ["LDFLAGS"] = builder_config.LDFLAGS
os.environ["ARCH"] = "aarch64"
setup(
ext_modules = cythonize((Extension("my_ext", ["file1.pyx", "file2.pyx", "file3.pyx", "file4.pyx", "file5.pyx"]))),
)
When I run python3.6 setup.py build_ext -i I get a file named: my_ext.cpython-36m-x86_64-linux-gnu.so
My problem is that on the target the library will not be loaded unless the name is changed to:
my_ext.cpython-35m-aarch64-linux-gnu.so
How can I change the generated filename?
As stated in the comments, what you are trying to achieve is unsafe.
You can work around the architecture tag with the environment variable _PYTHON_HOST_PLATFORM (e.g. you can change it in your sitecustomize.py). But, if the modules are actually incompatible (and they most likely are), you will only get core dumps later on.
I don't think you can work around the major Python version.
In order to come back to safer grounds, I would try to rely on portable solutions. For example, it doesn't look official, but we can find some articles on the web about Conda and aarch64 (e.g. you can look for 'Archiconda'). One more time, you wouldn't be able to simply copy the conda environments from one machine to another, but, you can freeze these environments (via a 'conda export') and build similar ones on the target machine.
An option is to upgrade the target interpreter to v3.6 if that's possible for you.
Another option is to install v3.5 on the machine you're using to build with that interpreter. It's pretty uncomplicated to get several different versions of the python interpreter installed on the same machine. I don't know your specifics so I can't provide any links but I'm sure a quick search will get you what you need.
Scripts generated by zc.buildout using zc.recipe.egg, on our <package>/bin/ directory look like this:
#! <python shebang> -S
import sys
sys.path[0:0] = [
... # some paths derived from the eggs
... # some other paths included with zc.recipe.egg `extra-path`
]
# some user initialization code from zc.recipe.egg `initialization`
# import function, call function
What I have not been able to was to find a way to programmatically prepend a path at the sys.path construction introduced in every script. Is this possible?
Why: I have a version of my python project installed globally and another version of it installed locally (off-buildout tree). I want to be able to switch between these two versions.
Note: Clearly, one can use the zc.recipe.egg/initialization property to add something like:
initialization = sys.path[0:0] = [ /add/path/to/my/eggs ]
But, is there any other way? Extra points for an example!
Finally, I got a working environment by creating my own buildout recipe that you can find here: https://github.com/idiap/local.bob.recipe. The file that contains the recipe is this one: https://github.com/idiap/local.bob.recipe/blob/master/config.py. There are lots of checks which are specific to our software at the class constructor and some extra improvements as well, but don't get bothered with that. The "real meat (TM)" is on the install() method of that class. It goes like this more or less:
egg_link = os.path.join(self.buildout['buildout']['eggs-directory'], 'external-package.egg-link')
f = open(egg_link, 'wt')
f.write(self.options['install-directory'] + '\n')
f.close()
self.options.created(egg_link)
return self.options.created()
This will do the trick. My external (CMake-based) package now only has to create the right .egg-info file in parallel with the python package(s) it builds. Than, I can tie, using the above recipe, the usage of a specific package installation like this:
[buildout]
parts = external_package python
develop = .
eggs = my_project
external_package
recipe.as.above
[external_package]
recipe = recipe.as.above:config
install-directory = ../path/to/my/local/package/build
[python]
recipe = zc.recipe.egg
interpreter = python
eggs = ${buildout:eggs}
If you wish to switch installations, just change the install-directory property above. If you wish to use the default installation available system wide, just remove altogether the recipe.as.above constructions from your buildout.cfg file. Buildout will just find the global installation w/o requiring any extra configuration. Uninstallation will work properly as well. So, switching between builds will just work.
Here is a fully working buildout .cfg file that we use here: https://github.com/idiap/bob.project.example/blob/master/localbob.cfg
The question is: Is there an easier way to achieve the same w/o having this external recipe?
Well, what you miss is probably the most useful buildout extension, mr.developer.
Typically the package, let's say foo.bar will be in some repo, let's say git.
Your buildout will look like
[buildout]
extensions = mr.developer
[sources]
foo.bar = git git#github.com:foo/foo.bar.git
If you don't have your package in a repo, you can use fs instead of git, have a look at the documentation for details.
Activating the "local" version is done by
./bin/develop a foo.bar
Deactivating by
./bin/develop d foo.bar
There are quite a few other things you can do with mr.developer, do check it out!
I use the following script to distribute a module containing pure python code.
from distutils.core import setup, Extension
import os
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
)
The problem I have is, I modified one of the files that uses an external library (a .so file), which I need to ship along with the existing module. I was suggested to use package_data to include the library. I modified the script to the following.
from distutils.core import setup, Extension
import os
data_dir = os.path.abspath('../lib64/')
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
package_data={'mtester':[data_dir+'mhelper.so']},
)
The problem is, adding package_data did not make any difference. This is not installing the mhelper.so in any location (neither in site-packages nor in site-packages/mtester).
System info: Fedora 10, 64 bit, python 2.5 (Yes it is ancient. But it is our build machine, and it needs to stay that way to maintain backward compatibility)
Any suggestions that would help me resolve this would be well appreciated!
Unfortunately package_data looks for files relative to the top of the package. One fix is to move the helper library under the module dir with the rest of the code:
% mv lib64/mhelper.so module/
Then modify the package_data argument accordingly:
package_data = {'mtester': ['mhelper.so']}
...
Then test:
% python setup.py bdist
% tar tf dist/mtester-0.1.linux-x86_64.tar.gz | grep mhelper
./usr/local/lib/python2.5/dist-packages/mtester/mhelper.so