Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?
For example, in my main.py:
import os, sys
import MultipartPostHandler
def main():
# do stuff here
But MultipartPostHandler is not yet installed, so what I want is to have it installed first before
actually running main.py... but in an automated manner. When I say automatically, I mean I will just invoke the script one time to start the dependency installation, then to be followed by actual functionalities of the main script.
(somehow, a little bit similar with maven. But I just need the installation part)
I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.
Any ideas are greatly appreciated. Thanks in advance!
Is there a way to include/invoke python module(s) (dependencies) installation first, before running the actual/main script?
A good way is to use setuptools and explicitly list them in install_requires.
Since you are providing a main function, you also probably want to provide entry_points.
I already know the basics of setuptools. The problem is I may have to call the installation (setup.py) and the main script (main.py) separately.
That is usually not a problem. It is very common to first install everything with a requirements.txt file and pip install -r requirements.txt. Plus if you list dependencies you can then have reasonable expectations that it will be there when your function is called and not rely on try/except ImporError. It is a reasonable approach to expect required dependencies to be present and only use try/except for optional dependencies.
setuptools 101:
create a tree structure like this:
$ tree
.
├── mymodule
│ ├── __init__.py
│ └── script.py
└── setup.py
your code will go under mymodule; let's imagine some code that does a simple task:
# module/script.py
def main():
try:
import requests
print 'requests is present. kudos!'
except ImportError:
raise RuntimeError('how the heck did you install this?')
and here is a relevant setup:
# setup.py
from setuptools import setup
setup(
name='mymodule',
packages=['mymodule'],
entry_points={
'console_scripts' : [
'mycommand = mymodule.script:main',
]
},
install_requires=[
'requests',
]
)
This would make your main available as a command, and this would also take care of installing the dependencies you need (e.g requests)
~tmp damien$ virtualenv test && source test/bin/activate && pip install mymodule/
New python executable in test/bin/python
Installing setuptools, pip...done.
Unpacking ./mymodule
Running setup.py (path:/var/folders/cs/nw44s66532x_rdln_cjbkmpm000lk_/T/pip-9uKQFC-build/setup.py) egg_info for package from file:///tmp/mymodule
Downloading/unpacking requests (from mymodule==0.0.0)
Using download cache from /Users/damien/.pip_download_cache/https%3A%2F%2Fpypi.python.org%2Fpackages%2F2.7%2Fr%2Frequests%2Frequests-2.4.1-py2.py3-none-any.whl
Installing collected packages: requests, mymodule
Running setup.py install for mymodule
Installing mycommand script to /tmp/test/bin
Successfully installed requests mymodule
Cleaning up...
(test)~tmp damien$ mycommand
requests is present. kudos!
more useful commands with argparse:
If you want to use argparse then...
# module/script.py
import argparse
def foobar(args):
# ...
def main():
parser = argparse.ArgumentParser()
# parser.add_argument(...)
args = parser.parse_args()
foobar(args)
There's a few ways to do this. One way is to surround the import statement with a try...except ImportError block and then have some Python code that installs the package if the ImportError exception is raised, so something like:
try:
import MultipartPostHandler
except ImportError:
# code that installs MultipartPostHandler and then imports it
I don't think this approach is very clean. Plus if there are other unrelated importing issues, that won't be detected here. A better approach might be to have a bash script that checks to see if the module is installed:
pip freeze | grep MultipartPostHandler
and if not, installs the module:
pip install MultipartPostHandler
Then we can safely run the original Python code.
EDIT: Actually, I like FLORET's answer better. The imp module is exactly what you want.
You should use the imp module. Here's a example:
import imp
import httplib2
import sys
try:
import MultipartPostHandler
except ImportError:
# Here you download
http = httplib2.Http()
response, content = http.request('http://where_your_file_is.com/here')
if response.status == 200:
# Don't forget the right managment
with open('MultipartPostHandler.py', 'w') as f:
f.write(content)
file, pathname, description = imp.find_module('MultipartPostHandler')
MultipartPostHandler = imp.load_module('MultipartPostHandler', file, pathname, description)
else:
sys.exit('Unable to download the file')
For a full approach, use a queue:
download_list = []
try:
import FirstModule
except ImportError:
download_list.append('FirstModule')
try:
import SecondModule
except ImportError:
download_list.append('SecondModule')
if download_list:
# Here do the routine to dowload, install and load_modules
# THe main routine
def main():
the_response_is(42)
You can download binaries with open(file_content, 'wb')
I hope it help
BR
Related
Is it possible to specify a post-install Python script file as part of the setuptools setup.py file so that a user can run the command:
python setup.py install
on a local project file archive, or
pip install <name>
for a PyPI project and the script will be run at the completion of the standard setuptools install? I am looking to perform post-install tasks that can be coded in a single Python script file (e.g. deliver a custom post-install message to the user, pull additional data files from a different remote source repository).
I came across this SO answer from several years ago that addresses the topic and it sounds as though the consensus at that time was that you need to create an install subcommand. If that is still the case, would it be possible for someone to provide an example of how to do this so that it is not necessary for the user to enter a second command to run the script?
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This solution is more transparent:
You will make a few additions to setup.py and there is no need for an extra file.
Also you need to consider two different post-installations; one for development/editable mode and the other one for install mode.
Add these two classes that includes your post-install script to setup.py:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
class PostDevelopCommand(develop):
"""Post-installation for development mode."""
def run(self):
develop.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
class PostInstallCommand(install):
"""Post-installation for installation mode."""
def run(self):
install.run(self)
# PUT YOUR POST-INSTALL SCRIPT HERE or CALL A FUNCTION
and insert cmdclass argument to setup() function in setup.py:
setup(
...
cmdclass={
'develop': PostDevelopCommand,
'install': PostInstallCommand,
},
...
)
You can even call shell commands during installation, like in this example which does pre-installation preparation:
from setuptools import setup
from setuptools.command.develop import develop
from setuptools.command.install import install
from subprocess import check_call
class PreDevelopCommand(develop):
"""Pre-installation for development mode."""
def run(self):
check_call("apt-get install this-package".split())
develop.run(self)
class PreInstallCommand(install):
"""Pre-installation for installation mode."""
def run(self):
check_call("apt-get install this-package".split())
install.run(self)
setup(
...
P.S. there are no any pre-install entry points available on setuptools. Read this discussion if you are wondering why there is none.
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
This is the only strategy that has worked for me when the post-install script requires that the package dependencies have already been installed:
import atexit
from setuptools.command.install import install
def _post_install():
print('POST INSTALL')
class new_install(install):
def __init__(self, *args, **kwargs):
super(new_install, self).__init__(*args, **kwargs)
atexit.register(_post_install)
setuptools.setup(
cmdclass={'install': new_install},
Note: The solution below only works when installing a source distribution zip or tarball, or installing in editable mode from a source tree. It will not work when installing from a binary wheel (.whl)
A solution could be to include a post_setup.py in setup.py's directory. post_setup.py will contain a function which does the post-install and setup.py will only import and launch it at the appropriate time.
In setup.py:
from distutils.core import setup
from distutils.command.install_data import install_data
try:
from post_setup import main as post_install
except ImportError:
post_install = lambda: None
class my_install(install_data):
def run(self):
install_data.run(self)
post_install()
if __name__ == '__main__':
setup(
...
cmdclass={'install_data': my_install},
...
)
In post_setup.py:
def main():
"""Do here your post-install"""
pass
if __name__ == '__main__':
main()
With the common idea of launching setup.py from its directory, you will be able to import post_setup.py else it will launch an empty function.
In post_setup.py, the if __name__ == '__main__': statement allows you to manually launch post-install from command line.
Combining the answers from #Apalala, #Zulu and #mertyildiran; this worked for me in a Python 3.5 environment:
import atexit
import os
import sys
from setuptools import setup
from setuptools.command.install import install
class CustomInstall(install):
def run(self):
def _post_install():
def find_module_path():
for p in sys.path:
if os.path.isdir(p) and my_name in os.listdir(p):
return os.path.join(p, my_name)
install_path = find_module_path()
# Add your post install code here
atexit.register(_post_install)
install.run(self)
setup(
cmdclass={'install': CustomInstall},
...
This also gives you access the to the installation path of the package in install_path, to do some shell work on.
I think the easiest way to perform the post-install, and keep the requirements, is to decorate the call to setup(...):
from setup tools import setup
def _post_install(setup):
def _post_actions():
do_things()
_post_actions()
return setup
setup = _post_install(
setup(
name='NAME',
install_requires=['...
)
)
This will run setup() when declaring setup. Once done with the requirements installation, it will run the _post_install() function, which will run the inner function _post_actions().
If using atexit, there is no need to create a new cmdclass. You can simply create your atexit register right before the setup() call. It does the same thing.
Also, if you need dependencies to be installed first, this does not work with pip install since your atexit handler will be called before pip moves the packages into place.
I wasn't able to solve a problem with any presented recommendations, so here is what helped me.
You can call function, that you want to run after installation just after setup() in setup.py, like that:
from setuptools import setup
def _post_install():
<your code>
setup(...)
_post_install()
Given a package random.whl containing hello.py:
print("Hello World!")
Is there a way to create a setup.py, setup.cfg or pyproject.toml, that when executed, will install the package in such a way that hello.py will be executed every time Python is started?
pip install random.whl
python unrelated.py # First prints "Hello World", then continues on.
I know it's possible to hook on readline.py that Python automatically loads, but is there a different and less "hacky" way to achieve it?
Some impossible ways that I thought of:
Running a post-install script on a .whl distribution (post-install is only avaiable on sdist).
Modifying PYTHONSTARTUP env variable or copying files.
Changing the import machinery.
While being a security risk, a method achieving it is good for implementing debuggers or auditing tools without requiring a change in either pre-compiled or post-compiled Python code, or used for penetration testing in side-channel attacks.
So far, using sitecustomize.py and publishing an sdist with a custom install command was the most reliable, and worked in virtual environments unlike usercustomize.py or .pth files.
Relevant setup.py code:
import sys
import os
import setuptools
import sysconfig
from setuptools.command.install import install
class PreInstall(install):
def run(self):
site_packages_dir = sysconfig.get_path("purelib")
sitecustomize_path = os.path.join(site_packages_dir, "sitecustomize.py")
if os.path.exists(sitecustomize_path):
raise FileExistsError("Site customize file already exists. "
"Please remove it before installing.")
install.run(self)
setuptools.setup(
name='...',
version='0.0.1',
py_modules=["sitecustomize"],
cmdclass={'install': PreInstall,},
)
It still doesn't work with .whl distributions as it might overwrite an existing sitecustomize without being able to check.
I am trying to create a directory upon a package installation. The function to create the directory, by itself, successfully creates it. Additionally, when I run "python3.7 setup.py install", the directory is created.
Why does this not work when using pip though? I don't see any errors. When I added print statements, I do not see them.
I have chosen to use setuptools' 'bdist_egg' function instead of the 'install' function for reasons found in here:
Running custom setuptools build during install
from sys import platform
from setuptools import setup
from os import mkdir, chmod, path
from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
class OverrideInstall(_bdist_egg):
def run(self):
_bdist_egg.run(self)
# create log directory
log = "/var/log/FOO"
mode = 0o777
if not path.exists(log):
mkdir(log)
chmod(log, mode)
setup(
name='cox-nams',
version='FOO',
description='FOO',
<-- output omitted for brevity / security>
cmdclass={"bdist_egg": OverrideInstall},
)
Apparently not supported with pip install.
So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.req.
Since it is common practice to use the parse_requirements function in your setup.py to install all the dependencies out of a requirements file I now wonder if this practice should change since it is now lying under _internal?
Or what is actually best practice without using parse_requirements?
First, I believe parsing requirements.txt to fill the list of dependencies in package metadata is not a good idea. The requirements.txt file and the list of "install dependencies" are two different concepts, they are not interchangeable. It should be the other way around, the list of dependencies in package metadata should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. For example with a tool such as pip-compile. See the notes at the bottom of this answer.
But everyone has different needs, that lead to different workflows. So with that said... There are 3 possibilities to handle this, depending on where you want your project's package metadata to be written: pyproject.toml, setup.cfg, or setup.py.
Words of caution!
If you insist on having the list of dependencies in package metadata be read from a requirements.txt file then make sure that this requirements.txt file is included in the "source distribution" (sdist) otherwise installation will fail, for obvious reasons.
These techniques will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not supported and it will cause a failure.
pyproject.toml
[project]
# ...
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
# ...
dependencies = requirements.txt
setup.cfg
Since setuptools version 62.6 it is possible to write something like this in setup.cfg:
[options]
install_requires = file: requirements.txt
setup.py
It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.
It could more or less look like this:
#!/usr/bin/env python
import pathlib
import pkg_resources
import setuptools
with pathlib.Path('requirements.txt').open() as requirements_txt:
install_requires = [
str(requirement)
for requirement
in pkg_resources.parse_requirements(requirements_txt)
]
setuptools.setup(
install_requires=install_requires,
)
Notes:
https://github.com/pypa/setuptools/issues/1951#issuecomment-1431345869
https://caremad.io/posts/2013/07/setup-vs-requirement/
https://setuptools.pypa.io/en/latest/history.html#v62-6-0
See also this other answer: https://stackoverflow.com/a/59971469
The solution of Scrotch only works until pip 19.0.3, in the pip >= 20 versions the PipSession module was refactored. Here is a solution for the imports that works for all pip versions:
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements
EDIT: modified to support pip>= 19.0.3
I don't agree with the accepted answer. The setup.py file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt file. I would do something like this -
try:
# pip >=20
from pip._internal.network.session import PipSession
from pip._internal.req import parse_requirements
except ImportError:
try:
# 10.0.0 <= pip <= 19.3.1
from pip._internal.download import PipSession
from pip._internal.req import parse_requirements
except ImportError:
# pip <= 9.0.3
from pip.download import PipSession
from pip.req import parse_requirements
requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())
if __name__ == '__main__':
setup(
...
install_requires=[str(requirement.requirement) for requirement in requirements],
...
)
Throw in all your requirements in requirements.txt under project root directory.
What I figured out the right way to do is adding the dependencies in the setup.py like:
REQUIRED_PACKAGES = [
'boto3==1.7.33'
]
if __name__ == '__main__':
setup(
...
install_requires=REQUIRED_PACKAGES,
...
)
and just have a . in your requirements.txt. It will then automatically install all packages from the setup.py if you install from the file.
#sinoroc is correct that you generally do not want to populate your setup() function using your requirements.txt. They are for different things: requirements.txt produces a maximal, reproducible environment with hard versions to aid in deployments, help other developers etc.. Setup dependencies are a minimal, permissive list to allow users to install things.
However, sometimes you want to install an application using pip, or collaborators insist on using the requirements file, so I wrote a package which helps you do this: https://pypi.org/project/extreqs/
with open("requirements.txt") as f:
dependencies = [line for line in f if "==" in line]
setup(
install_requires=dependencies
)
I'm trying to install a python package I've developed using the develop command of setuptools.
[sidenote: There is a bewilderingly vast quantity of information about this on the web (distutils, distutils2, setuptools, distribute). setuptools and develop are, as far as I can tell, the most modern/best practice way to use a piece of code that's in development. Perhaps I am wrong.]
Here's what I did:
(1) I placed an empty __init__.py in the directory with my Python code.
(2) I made a setup.py:
from setuptools import setup, find_packages
setup(name = "STEM_pytools",
version = "0.1",
packages = find_packages(),
author = "Timothy W. Hilton",
author_email = "my#email.address",
description = "visualization and data pre-/post-processing tools for STEM",
license = "",
keywords = "STEM",
url = "")
(3) I ran
python setup.py develop
That seemed to proceed without problems.
However, when I try to use the package, I get:
>>> import STEM_pytools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named STEM_pytools
The same thing happens with the install command: it's output looks ok, then "No module named STEM_pytools". I'm tearing my hair out. Any suggestions appreciated!
I solved the problem, although I still don't entirely understand why it works now and did not work before. It seems my setup.py and the directory structure of my project were not interacting successfully.
This is the directory structure that worked with my setup.py:
STEMpytools/
setup.py
stem_pytools/
__init__.py
source1.py
source2.py
...
sourceN.py
This directory structure did not work, at least when paired with my setup.py:
STEMpytools/
setup.py
__init__.py
source1.py
source2.py
...
sourceN.py
This explanation helped me a lot: http://bashelton.com/2009/04/setuptools-tutorial/
Now, from the python interpreter, these both work:
import stem_pytools
import stem_pytools.source1
Experimenting on my system suggests it is necessary to place __init__.py and the package source code in a subdirectory one level below the root directory that contains setup.py. I'm not sure from the setuptools and distutils documentation why this is the case.