Is it possible to specify a post-install Python script file as part of the setuptools setup.py file so that a user can run the command:
python setup.py install
on a local project file archive, or
pip install <name>
for a PyPI project and the script will be run at the completion of the standard setuptools install? I am looking to perform post-install tasks that can be coded in a single Python script file (e.g. deliver a custom post-install message to the user, pull additional data files from a different remote source repository).
I came across this SO answer from several years ago that addresses the topic and it sounds as though the consensus at that time was that you need to create an install subcommand. If that is still the case, would it be possible for someone to provide an example of how to do this so that it is not necessary for the user to enter a second command to run the script?
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This solution is more transparent:
You will make a few additions to setup.py and there is no need for an extra file.
Also you need to consider two different post-installations; one for development/editable mode and the other one for install mode.
Add these two classes that includes your post-install script to setup.py:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
class PostDevelopCommand(develop):
"""Post-installation for development mode."""
def run(self):
develop.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
and insert cmdclass argument to setup() function in setup.py:
setup(
...
cmdclass={
'develop': PostDevelopCommand,
'install': PostInstallCommand,
},
...
)
You can even call shell commands during installation, like in this example which does pre-installation preparation:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
from subprocess import check_call
class PreDevelopCommand(develop):
"""Pre-installation for development mode."""
def run(self):
check_call("apt-get install this-package".split())
develop.run(self)
class PreInstallCommand(install):
"""Pre-installation for installation mode."""
def run(self):
check_call("apt-get install this-package".split())
install.run(self)
setup(
...
P.S. there are no any pre-install entry points available on setuptools. Read this discussion if you are wondering why there is none.
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This is the only strategy that has worked for me when the post-install script requires that the package dependencies have already been installed:
import atexit
from setuptools.command.install import install
def _post_install():
print('POST INSTALL')
class new_install(install):
def __init__(self, *args, **kwargs):
super(new_install, self).__init__(*args, **kwargs)
atexit.register(_post_install)
setuptools.setup(
cmdclass={'install': new_install},
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
A solution could be to include a post_setup.py in setup.py's directory. post_setup.py will contain a function which does the post-install and setup.py will only import and launch it at the appropriate time.
In setup.py:
from distutils.core import setup
from distutils.command.install_data import install_data
try:
from post_setup import main as post_install
except ImportError:
post_install = lambda: None
class my_install(install_data):
def run(self):
install_data.run(self)
post_install()
if __name__ == '__main__':
setup(
...
cmdclass={'install_data': my_install},
...
)
In post_setup.py:
def main():
"""Do here your post-install"""
pass
if __name__ == '__main__':
main()
With the common idea of launching setup.py from its directory, you will be able to import post_setup.py else it will launch an empty function.
In post_setup.py, the if __name__ == '__main__': statement allows you to manually launch post-install from command line.
Combining the answers from #Apalala, #Zulu and #mertyildiran; this worked for me in a Python 3.5 environment:
import atexit
import os
import sys
from setuptools import setup
from setuptools.command.install import install
class CustomInstall(install):
def run(self):
def _post_install():
def find_module_path():
for p in sys.path:
if os.path.isdir(p) and my_name in os.listdir(p):
return os.path.join(p, my_name)
install_path = find_module_path()
# Add your post install code here
atexit.register(_post_install)
install.run(self)
setup(
cmdclass={'install': CustomInstall},
...
This also gives you access the to the installation path of the package in install_path, to do some shell work on.
I think the easiest way to perform the post-install, and keep the requirements, is to decorate the call to setup(...):
from setup tools import setup
def _post_install(setup):
def _post_actions():
do_things()
_post_actions()
return setup
setup = _post_install(
setup(
name='NAME',
install_requires=['...
)
)
This will run setup() when declaring setup. Once done with the requirements installation, it will run the _post_install() function, which will run the inner function _post_actions().
If using atexit, there is no need to create a new cmdclass. You can simply create your atexit register right before the setup() call. It does the same thing.
Also, if you need dependencies to be installed first, this does not work with pip install since your atexit handler will be called before pip moves the packages into place.
I wasn't able to solve a problem with any presented recommendations, so here is what helped me.
You can call function, that you want to run after installation just after setup() in setup.py, like that:
from setuptools import setup
def _post_install():
<your code>
setup(...)
_post_install()
Related
I am trying to create a directory upon a package installation. The function to create the directory, by itself, successfully creates it. Additionally, when I run "python3.7 setup.py install", the directory is created.
Why does this not work when using pip though? I don't see any errors. When I added print statements, I do not see them.
I have chosen to use setuptools' 'bdist_egg' function instead of the 'install' function for reasons found in here:
Running custom setuptools build during install
from sys import platform
from setuptools import setup
from os import mkdir, chmod, path
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
class OverrideInstall(_bdist_egg):
def run(self):
_bdist_egg.run(self)
# create log directory
log = "/var/log/FOO"
mode = 0o777
if not path.exists(log):
mkdir(log)
chmod(log, mode)
setup(
name='cox-nams',
version='FOO',
description='FOO',
<-- output omitted for brevity / security>
cmdclass={"bdist_egg": OverrideInstall},
)
Apparently not supported with pip install.
I have some .proto gRPC files I want to compile as part of the setup.py script. This requires running from grpc_tools import protoc and calling protoc before setup(args). The goal is to compile and install the pb files from pip install pkgname.
E.g.
# setup.py
# generate our pb2 files in the temp directory structure
compile_protobufs(pkgname)
# this will package the generated files and put them in site-packages or .whl
setup(
name=pkgname,
install_requires=['grpcio-tools', ...],
...
)
This works as intended, I get the pb files in my site-packages or in the wheel without them having to exist in the source folder. However, this pattern means I cannot naively pip install pkgname from scratch, as the step compile_protobufs depends on grpcio-tools, which does not get installed until setup().
I could use setup_requires, but that is on the chopping block. I could just install the dependencies first (right now I use RUN pip install -r build-require.txt && pip install pkgname/ ), but it still seems like there ought to be a cleaner way.
Am I even going about this pattern correctly or am I missing some packaging idiom?
My criteria:
Generally this is run inside a container, so minimizing external deps
I want the _pb2.py files regenerated each time I pip install
These files need to also make their way into any .whl or tar.
Looks like it is already documented here:
https://github.com/grpc/grpc/tree/master/tools/distrib/python/grpcio_tools#usage
So your setup.py could look like this:
#!/usr/bin/env python3
import distutils.command.install
import setuptools
class build_package_protos(setuptools.Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
from grpc_tools import command
command.build_package_protos(self.distribution.package_dir[''])
class install(distutils.command.install.install):
_sub_command = ('build_package_protos', None,)
_sub_commands = distutils.command.install.install.sub_commands
sub_commands = [_sub_command] + _sub_commands
def setup():
setuptools.setup(
# see 'setup.cfg'
cmdclass={
'build_package_protos': build_package_protos,
'install': install,
},
setup_requires=[
'grpcio-tools',
],
)
if __name__ == '__main__':
setup()
I'm trying to provide a bash completion script for my CLI tool that is written in Python. According to the Python Packaging Authority, data_files in setup.py is exactly what I need:
Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that. It is mostly useful if you need to install files which are used by other programs, which may be unaware of Python packages.
So I added the completion file like this:
data_files=[
('/usr/share/bash-completion/completions', ['completion/dotenv']),
],
and try to test it with:
pip install -e .
In my virtual environment. However, the completion script gets not installed. Did I forgot something or is pip broken? The full project can be found here
I had the same issue and I have implemented a workaround.
It seems to me that python setup.py develop or (pip install -e .) does not run the same function than python setup.py install.
In fact, I have noticed by looking in the source code that python setup.py install run build_py :
https://github.com/python/cpython/blob/master/Lib/distutils/command/build_py.py#L134
https://github.com/pypa/setuptools/blob/master/setuptools/command/build_py.py
After a few digging I have opted to override the develop command as follow. The following code is python3.6:
""" SetupTool Entry Point """
import sys
from pathlib import Path
from shutil import copy2
from setuptools import find_packages, setup
from setuptools.command.develop import develop
# create os_data_files that will be used by the default install command
os_data_files = [
(
f"{sys.prefix}/config", # providing absolute path, sys.prefix will be different in venv
[
"src/my_package/config/properties.env",
],
),
]
def build_package_data():
""" implement the necessary function for develop """
for dest_dir, filenames in os_data_files:
for filename in filenames:
print(
"CUSTOM SETUP.PY (build_package_data): copy %s to %s"
% (filename, dest_dir)
)
copy2(filename, dest_dir)
def make_dirstruct():
""" Set the the logging path """
for subdir in ["config"]:
print("CUSTOM SETUP.PY (make_dirstruct): creating %s" % subdir)
(Path(BASE_DIR) / subdir).mkdir(parents=True, exist_ok=True)
class CustomDevelopCommand(develop):
""" Customized setuptools install command """
def run(self):
develop.run(self)
make_dirstruct()
build_package_data()
# provide the relevant information for stackoverflow
setup(
package_dir={"": "src"},
packages=find_packages("src"),
data_files=os_data_files,
cmdclass={"develop": CustomDevelopCommand},
)
Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?
For example, in my main.py:
import os, sys
import MultipartPostHandler
def main():
# do stuff here
But MultipartPostHandler is not yet installed, so what I want is to have it installed first before
actually running main.py... but in an automated manner. When I say automatically, I mean I will just invoke the script one time to start the dependency installation, then to be followed by actual functionalities of the main script.
(somehow, a little bit similar with maven. But I just need the installation part)
I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.
Any ideas are greatly appreciated. Thanks in advance!
Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?
A good way is to use setuptools and explicitly list them in install_requires.
Since you are providing a main function, you also probably want to provide entry_points.
I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.
That is usually not a problem. It is very common to first install everything with a requirements.txt file and pip install -r requirements.txt. Plus if you list dependencies you can then have reasonable expectations that it will be there when your function is called and not rely on try/except ImporError. It is a reasonable approach to expect required dependencies to be present and only use try/except for optional dependencies.
setuptools 101:
create a tree structure like this:
$ tree
.
├── mymodule
│ ├── __init__.py
│ └── script.py
└── setup.py
your code will go under mymodule; let's imagine some code that does a simple task:
# module/script.py
def main():
try:
import requests
print 'requests is present. kudos!'
except ImportError:
raise RuntimeError('how the heck did you install this?')
and here is a relevant setup:
# setup.py
from setuptools import setup
setup(
name='mymodule',
packages=['mymodule'],
entry_points={
'console_scripts' : [
'mycommand = mymodule.script:main',
]
},
install_requires=[
'requests',
]
)
This would make your main available as a command, and this would also take care of installing the dependencies you need (e.g requests)
~tmp damien$ virtualenv test && source test/bin/activate && pip install mymodule/
New python executable in test/bin/python
Installing setuptools, pip...done.
Unpacking ./mymodule
Running setup.py (path:/var/folders/cs/nw44s66532x_rdln_cjbkmpm000lk_/T/pip-9uKQFC-build/setup.py) egg_info for package from file:///tmp/mymodule
Downloading/unpacking requests (from mymodule==0.0.0)
Using download cache from /Users/damien/.pip_download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2Fr%2Frequests%2Frequests-2.4.1-py2.py3-none-any.whl
Installing collected packages: requests, mymodule
Running setup.py install for mymodule
Installing mycommand script to /tmp/test/bin
Successfully installed requests mymodule
Cleaning up...
(test)~tmp damien$ mycommand
requests is present. kudos!
more useful commands with argparse:
If you want to use argparse then...
# module/script.py
import argparse
def foobar(args):
# ...
def main():
parser = argparse.ArgumentParser()
# parser.add_argument(...)
args = parser.parse_args()
foobar(args)
There's a few ways to do this. One way is to surround the import statement with a try...except ImportError block and then have some Python code that installs the package if the ImportError exception is raised, so something like:
try:
import MultipartPostHandler
except ImportError:
# code that installs MultipartPostHandler and then imports it
I don't think this approach is very clean. Plus if there are other unrelated importing issues, that won't be detected here. A better approach might be to have a bash script that checks to see if the module is installed:
pip freeze | grep MultipartPostHandler
and if not, installs the module:
pip install MultipartPostHandler
Then we can safely run the original Python code.
EDIT: Actually, I like FLORET's answer better. The imp module is exactly what you want.
You should use the imp module. Here's a example:
import imp
import httplib2
import sys
try:
import MultipartPostHandler
except ImportError:
# Here you download
http = httplib2.Http()
response, content = http.request('http://where_your_file_is.com/here')
if response.status == 200:
# Don't forget the right managment
with open('MultipartPostHandler.py', 'w') as f:
f.write(content)
file, pathname, description = imp.find_module('MultipartPostHandler')
MultipartPostHandler = imp.load_module('MultipartPostHandler', file, pathname, description)
else:
sys.exit('Unable to download the file')
For a full approach, use a queue:
download_list = []
try:
import FirstModule
except ImportError:
download_list.append('FirstModule')
try:
import SecondModule
except ImportError:
download_list.append('SecondModule')
if download_list:
# Here do the routine to dowload, install and load_modules
# THe main routine
def main():
the_response_is(42)
You can download binaries with open(file_content, 'wb')
I hope it help
BR
I would like to know how to make some code in setup.py conditional on which command (e.g. install or upload) was run.
Specifically, I'd like to have:
An easy way to add "hacks" such as ignoring a particular file in install, but no other commands.
A recommended/canonical way to add hooks such as running tests before installing.
I have tried reading the distutils documentation, but it's pretty sparse on details – the distutils.command[.foo] modules are completely undocumented.
For the first point I can check sys.argv like mentioned in this question, but that doesn't work when multiple commands are run, like:
python setup.py sdist bdist upload
so it isn't applicable in general.
You can override the command instead:
from distutils.command.install import install
from distutils.core import setup
def run_file(path):
with open(path, 'r') as f:
exec(f.read())
class myinstall(install): # subclass distutils's install command
def finalize_options(self): # called after option parsing
# call base class function
install.finalize_options(self)
# super won't work because distutils under Python 2 uses old-style classes
# ignore a module
self.distribution.py_modules.remove('mymodule')
def run(self): # called to run a command
# run tests first
run_file('path/to/test.py')
# ^ remember to make sure the module is in sys.path
# run the real commands
install.run(self)
setup(
name='abc',
py_modules=['mymodule'],
cmdclass={'install': myinstall}
# ^ override the install command
)