I've got a python module that calls an external binary, built from C source.
The source for that external executable is part of my python module, distributed as a .tar.gz file.
Is there a way of unzipping, then compiling that external executable, and installing it using setuptools/setup.py?
What I'd like to achieve is:
installing that binary into virtual environments
manage compilation/installation of the binary using setup.py install, setup.py build etc.
making the binary part of my python module, so that it can be distributed as a wheel without external dependencies
Solved in the end by modifying setup.py to add additional handlers for commands which did the installation.
An example of a setup.py which does this might be:
import os
from setuptools import setup
from setuptools.command.install import install
import subprocess
def get_virtualenv_path():
"""Used to work out path to install compiled binaries to."""
if hasattr(sys, 'real_prefix'):
return sys.prefix
if hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix:
return sys.prefix
if 'conda' in sys.prefix:
return sys.prefix
return None
def compile_and_install_software():
"""Used the subprocess module to compile/install the C software."""
src_path = './some_c_package/'
# compile the software
cmd = "./configure CFLAGS='-03 -w -fPIC'"
venv = get_virtualenv_path()
if venv:
cmd += ' --prefix=' + os.path.abspath(venv)
subprocess.check_call(cmd, cwd=src_path, shell=True)
# install the software (into the virtualenv bin dir if present)
subprocess.check_call('make install', cwd=src_path, shell=True)
class CustomInstall(install):
"""Custom handler for the 'install' command."""
def run(self):
compile_and_install_software()
super().run()
setup(name='foo',
# ...other settings skipped...
cmdclass={'install': CustomInstall})
Now when python setup.py install is called, the custom CustomInstall class is used, this then compiles and installs software before the normal install steps are run.
You can also do similar for any other steps you're interested in (e.g. build/develop/bdist_egg etc.).
An alternative is to make the compile_and_install_software() function a subclass of setuptools.Command, and create a fully fledged setuptools command for it.
This is more complicated, but lets you do things like specify it as a subcommand of another command (to e.g. avoid executing it twice), and to pass custom options in to it on the command line.
Related
Is it possible to specify a post-install Python script file as part of the setuptools setup.py file so that a user can run the command:
python setup.py install
on a local project file archive, or
pip install <name>
for a PyPI project and the script will be run at the completion of the standard setuptools install? I am looking to perform post-install tasks that can be coded in a single Python script file (e.g. deliver a custom post-install message to the user, pull additional data files from a different remote source repository).
I came across this SO answer from several years ago that addresses the topic and it sounds as though the consensus at that time was that you need to create an install subcommand. If that is still the case, would it be possible for someone to provide an example of how to do this so that it is not necessary for the user to enter a second command to run the script?
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This solution is more transparent:
You will make a few additions to setup.py and there is no need for an extra file.
Also you need to consider two different post-installations; one for development/editable mode and the other one for install mode.
Add these two classes that includes your post-install script to setup.py:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
class PostDevelopCommand(develop):
"""Post-installation for development mode."""
def run(self):
develop.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
and insert cmdclass argument to setup() function in setup.py:
setup(
...
cmdclass={
'develop': PostDevelopCommand,
'install': PostInstallCommand,
},
...
)
You can even call shell commands during installation, like in this example which does pre-installation preparation:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
from subprocess import check_call
class PreDevelopCommand(develop):
"""Pre-installation for development mode."""
def run(self):
check_call("apt-get install this-package".split())
develop.run(self)
class PreInstallCommand(install):
"""Pre-installation for installation mode."""
def run(self):
check_call("apt-get install this-package".split())
install.run(self)
setup(
...
P.S. there are no any pre-install entry points available on setuptools. Read this discussion if you are wondering why there is none.
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This is the only strategy that has worked for me when the post-install script requires that the package dependencies have already been installed:
import atexit
from setuptools.command.install import install
def _post_install():
print('POST INSTALL')
class new_install(install):
def __init__(self, *args, **kwargs):
super(new_install, self).__init__(*args, **kwargs)
atexit.register(_post_install)
setuptools.setup(
cmdclass={'install': new_install},
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
A solution could be to include a post_setup.py in setup.py's directory. post_setup.py will contain a function which does the post-install and setup.py will only import and launch it at the appropriate time.
In setup.py:
from distutils.core import setup
from distutils.command.install_data import install_data
try:
from post_setup import main as post_install
except ImportError:
post_install = lambda: None
class my_install(install_data):
def run(self):
install_data.run(self)
post_install()
if __name__ == '__main__':
setup(
...
cmdclass={'install_data': my_install},
...
)
In post_setup.py:
def main():
"""Do here your post-install"""
pass
if __name__ == '__main__':
main()
With the common idea of launching setup.py from its directory, you will be able to import post_setup.py else it will launch an empty function.
In post_setup.py, the if __name__ == '__main__': statement allows you to manually launch post-install from command line.
Combining the answers from #Apalala, #Zulu and #mertyildiran; this worked for me in a Python 3.5 environment:
import atexit
import os
import sys
from setuptools import setup
from setuptools.command.install import install
class CustomInstall(install):
def run(self):
def _post_install():
def find_module_path():
for p in sys.path:
if os.path.isdir(p) and my_name in os.listdir(p):
return os.path.join(p, my_name)
install_path = find_module_path()
# Add your post install code here
atexit.register(_post_install)
install.run(self)
setup(
cmdclass={'install': CustomInstall},
...
This also gives you access the to the installation path of the package in install_path, to do some shell work on.
I think the easiest way to perform the post-install, and keep the requirements, is to decorate the call to setup(...):
from setup tools import setup
def _post_install(setup):
def _post_actions():
do_things()
_post_actions()
return setup
setup = _post_install(
setup(
name='NAME',
install_requires=['...
)
)
This will run setup() when declaring setup. Once done with the requirements installation, it will run the _post_install() function, which will run the inner function _post_actions().
If using atexit, there is no need to create a new cmdclass. You can simply create your atexit register right before the setup() call. It does the same thing.
Also, if you need dependencies to be installed first, this does not work with pip install since your atexit handler will be called before pip moves the packages into place.
I wasn't able to solve a problem with any presented recommendations, so here is what helped me.
You can call function, that you want to run after installation just after setup() in setup.py, like that:
from setuptools import setup
def _post_install():
<your code>
setup(...)
_post_install()
Given a package random.whl containing hello.py:
print("Hello World!")
Is there a way to create a setup.py, setup.cfg or pyproject.toml, that when executed, will install the package in such a way that hello.py will be executed every time Python is started?
pip install random.whl
python unrelated.py # First prints "Hello World", then continues on.
I know it's possible to hook on readline.py that Python automatically loads, but is there a different and less "hacky" way to achieve it?
Some impossible ways that I thought of:
Running a post-install script on a .whl distribution (post-install is only avaiable on sdist).
Modifying PYTHONSTARTUP env variable or copying files.
Changing the import machinery.
While being a security risk, a method achieving it is good for implementing debuggers or auditing tools without requiring a change in either pre-compiled or post-compiled Python code, or used for penetration testing in side-channel attacks.
So far, using sitecustomize.py and publishing an sdist with a custom install command was the most reliable, and worked in virtual environments unlike usercustomize.py or .pth files.
Relevant setup.py code:
import sys
import os
import setuptools
import sysconfig
from setuptools.command.install import install
class PreInstall(install):
def run(self):
site_packages_dir = sysconfig.get_path("purelib")
sitecustomize_path = os.path.join(site_packages_dir, "sitecustomize.py")
if os.path.exists(sitecustomize_path):
raise FileExistsError("Site customize file already exists. "
"Please remove it before installing.")
install.run(self)
setuptools.setup(
name='...',
version='0.0.1',
py_modules=["sitecustomize"],
cmdclass={'install': PreInstall,},
)
It still doesn't work with .whl distributions as it might overwrite an existing sitecustomize without being able to check.
I am trying to create a directory upon a package installation. The function to create the directory, by itself, successfully creates it. Additionally, when I run "python3.7 setup.py install", the directory is created.
Why does this not work when using pip though? I don't see any errors. When I added print statements, I do not see them.
I have chosen to use setuptools' 'bdist_egg' function instead of the 'install' function for reasons found in here:
Running custom setuptools build during install
from sys import platform
from setuptools import setup
from os import mkdir, chmod, path
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
class OverrideInstall(_bdist_egg):
def run(self):
_bdist_egg.run(self)
# create log directory
log = "/var/log/FOO"
mode = 0o777
if not path.exists(log):
mkdir(log)
chmod(log, mode)
setup(
name='cox-nams',
version='FOO',
description='FOO',
<-- output omitted for brevity / security>
cmdclass={"bdist_egg": OverrideInstall},
)
Apparently not supported with pip install.
I'm trying to provide a bash completion script for my CLI tool that is written in Python. According to the Python Packaging Authority, data_files in setup.py is exactly what I need:
Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.
So I added the completion file like this:
data_files=[
('/usr/share/bash-completion/completions', ['completion/dotenv']),
],
and try to test it with:
pip install -e .
In my virtual environment. However, the completion script gets not installed. Did I forgot something or is pip broken? The full project can be found here
I had the same issue and I have implemented a workaround.
It seems to me that python setup.py develop or (pip install -e .) does not run the same function than python setup.py install.
In fact, I have noticed by looking in the source code that python setup.py install run build_py :
https://github.com/python/cpython/blob/master/Lib/distutils/command/build_py.py#L134
https://github.com/pypa/setuptools/blob/master/setuptools/command/build_py.py
After a few digging I have opted to override the develop command as follow. The following code is python3.6:
""" SetupTool Entry Point """
import sys
from pathlib import Path
from shutil import copy2
from setuptools import find_packages, setup
from setuptools.command.develop import develop
# create os_data_files that will be used by the default install command
os_data_files = [
(
f"{sys.prefix}/config", # providing absolute path, sys.prefix will be different in venv
[
"src/my_package/config/properties.env",
],
),
]
def build_package_data():
""" implement the necessary function for develop """
for dest_dir, filenames in os_data_files:
for filename in filenames:
print(
"CUSTOM SETUP.PY (build_package_data): copy %s to %s"
% (filename, dest_dir)
)
copy2(filename, dest_dir)
def make_dirstruct():
""" Set the the logging path """
for subdir in ["config"]:
print("CUSTOM SETUP.PY (make_dirstruct): creating %s" % subdir)
(Path(BASE_DIR) / subdir).mkdir(parents=True, exist_ok=True)
class CustomDevelopCommand(develop):
""" Customized setuptools install command """
def run(self):
develop.run(self)
make_dirstruct()
build_package_data()
# provide the relevant information for stackoverflow
setup(
package_dir={"": "src"},
packages=find_packages("src"),
data_files=os_data_files,
cmdclass={"develop": CustomDevelopCommand},
)
This question already has answers here:
How can I Install a Python module within code?
(12 answers)
Closed 2 years ago.
Can I download and install Python modules from PyPi strictly inside a script, without using a shell at all?
I use a non-standard Python environment, Autodesk Maya's Python interpreter. This does not come with "easy_install," and there is no "shell," only a python script interpreter invoked by the main Maya executable. Copying and pasting ez_setup.py's contents into the script editor window and running it correctly installs an easy_install somewhere into Maya's directory, but the script incorrectly records the Python interpreter as "...maya.exe" instead of "...mayapy.exe" Furthermore, using easy_install requires a shell.
The objective is to deliver a Python script that, e.g., installs NumPy into the Maya Python system. This could be accomplished by dropping eggs into the site-packages directory, but that requires manual user intervention. Anything an end user has to do outside the Maya environment is essentially untouchable, especially messing with the file system. But messing with the filesystem through a script? That's fine.
Is there something more elegant than ez_setup.py + editing the resulting easy_install...py's + subprocess calls? I feel like this is a basic feature. I see documentation online for programmatic module installation through pip... but pip needs to be installed first!
What is the most elegant way to install a module strictly within the confines of a script?
Installing easy_install for Maya on windows.
Download ez_setup.py.
open windows cmd elevated (start, type cmd, rmb click on it ->run as administrator)
change the cmd directory to x:\maya install dir\bin
example: cd c:\Program Files\MayaXX\bin
execute following command mayapy x:\WhereYouSaved\ez_setup.py
Now easy install should be set up properly. You may want to still do following steps:
cd x:\maya install dir\python\scripts
rename all files in this folder to start with ma
example: for %i in (*) do ren %i ma%i
add this folder to your path
hit win+e
rmb my computer and choose properties
Advanced system settings -> Environment variables
search variable path edit it and append ;x:\maya install dir\python\scripts
Now you can call maeasy_install pythonModule from cmd for installing stuff. Also you can call following inside Maya to install modules:
from setuptools.command import easy_install
easy_install.main( ["pythonModule"] )
NOTE: If Maya is installed in program files then you can not really install stuff without elevating. Unless you change disk permissions to the Maya python directory.
#!/usr/bin/env python
from __future__ import print_function
REQUIREMENTS = [ 'distribute', 'version', 'Cython', 'sortedcollection' ]
try:
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection
except:
import os, pip
pip_args = [ '-vvv' ]
proxy = os.environ['http_proxy']
if proxy:
pip_args.append('--proxy')
pip_args.append(proxy)
pip_args.append('install')
for req in REQUIREMENTS:
pip_args.append( req )
print('Installing requirements: ' + str(REQUIREMENTS))
pip.main(initial_args = pip_args)
# do it again
from setuptools import find_packages
from distutils.core import setup
from Cython.Distutils import build_ext as cython_build
import sortedcollection
To make it work, open the ez_setup.py file and simply add an s after http at this line:
DEFAULT_URL = "http://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]
so that it becomes
DEFAULT_URL = "https://pypi.python.org/packages/%s/s/setuptools/" % sys.version[:3]