I have a python package fsspec. This library provides a way to register external filesystem (backend integration). One way to register the required filesystem is to add it to the entry_points in setuptools.
To accomplish the task, we can manually add the entry_points in the setup.py and then install the package. But what I am looking for is a way to programmatically make the entry pre/post installation of the fsspec package.
I don't believe there is a simple way to update the metadata that normally gets written into your system during a package install. Digging through the code of setuptools or importlib.metadata might tell you how. It may be, that the Distribution object exposed by metadata can be edited in memory.
The point of the approach is, that fsspec does not need to import anything (aside from importlib.metadata itself) to know what extra implementations are available.
However, you might get far enough by using fsspec.register_implementation, which is be done dynamically at runtime. The only downside is, that your package will need to be imported and the function called before you try to access the protocol it provides.
Related
How to get pip to clean up better?
I have a pip module that does two things I would like un-done:
a) remove a dependent module dynamically specified during install - think sister binary package with os and python version in name. Those are now obsolete since I've learned the manylinux (and soon the universal) wheel dance.
b) remove license, usage, and telemetry files created while running the module.
Is there a hook or something that I can add a bit of logic and fileio too?
Seems option a was designed for but never implemented with the obsoletes keyword found at https://setuptools.pypa.io/en/latest/references/keywords.htm
I have a project that needs some DevOps TLC, so I am finally building my installation script. This will eventually be a package that will be install-able by pip locally, but may not end up in PyPI.
It has a dependency for a module called u2py. It is this package, created for U2 Database operations, not this package, for... something else. The one I want is only ever installed by a 3rd party vendor (Rocket), the one I don't want is in PyPI.
What should be the expected behavior of my package in this case? I will include a blurb about this in my readme doc, but is that sufficient?
I've thought about throwing an exception to identify when the wrong package is present, but that makes me feel weird. It seems that maybe the most pythonic thing is to NOT add this to my install script, and blindly assume import u2py results in a module I can use. If it quacks like a duck, parses DynArrays like a duck, and call()s SUBROUTINEs like a duck, then it's a duck, right? Otherwise, if there is an error the user will just go and actually read the docs.
I've looked a classifiers, but not sure if they apply here.
Ideally there would be a way at install-time (in setup.py) to detect whether the package is being installed into a "u2 environment" or not, and could fail the installation (with an appropriate error message) if that's the case.
With this solution, you won't be able to provide built distributions (wheels) since they don't execute the setup.py file at install-time, but just publishing source distributions should be fine.
It's a case where it would be nice if Python projects had namespaces (pip install com.rocket.u2py and import com.rocket.u2py as u2py).
From my point of view there are 2 aspects to consider: at the project level, at the package level.
1. project (distribution package)
I believe it is a bad practice to force alternative download sources onto the end user of your project. By default, pip should download from PyPI and nowhere else, unless the user decides it themselves (via --find-links or similar options, which you could instruct your users to do in your documentation).
Since it is such a niche dependency, I think I would simply not add it to install_requires. I would assume the end users of your project know about the dependency already and are able to install it themselves directly.
Also I don't believe it is possible to check reliably at install-time if the correct dependency is installed, since setup.py does not always run (overriding the bdist_wheel command can help, but probably not 100% effective).
2. package (importable package)
I am not sure some specific action is needed. The code would most likely fail sooner or later naturally, because of module or function is not importable. Which might be okay-ish, maybe?
But probably detecting if the dependency is installed (and it is the correct one), is relatively easy and would provide a better user experience. Either check that some specific modules or functions are importable. Or inspect the meta-data (import importlib_metadata; importlib_metadata.distribution('u2py').metadata['Author']).
In case of an application, I would try to fail gracefully as soon as possible. In case of a library I would try to find one strategic spot to place the check and raise a custom exception (CannotFindU2pyException).
Links:
Prevent pip from caching a package
https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata
https://github.com/pypa/pip/issues/4187#issuecomment-415067034
Equivalent for `--find-links` in `setup.py`
You can specify the url to the package in install_requires using setuptools (requires pip version 18.1 or greater).
Requirement specifiers
Example
setup.py
import setuptools
setuptools.setup(
name='MyPackage',
version='1.0.0',
# ...
install_requires=[
'requests # https://github.com/psf/requests/archive/v2.22.0.zip'
]
# ...
)
and do python setup.py install
Also
Since version 19.1, pip also supports direct references like so:
SomeProject # file:///somewhere/...
Ref
https://www.python.org/dev/peps/pep-0508/
https://github.com/pypa/pip/pull/4175
I'm using setuptools to install a Python module that I'm working on. In addition to numpy, scipy, ..., whose presence I can assure with install_requires = [...], my module also depends on a Python module - let's call it specialmodule - that is a Python interface to a program that is neither an egg, nor a single .py-file or a VCS repo (so Dependencies that aren’t in PyPI is not applicable). The program is written in C++ and has a Python interface, and can either be built from source after cloning from git, or obtained as a tar archive.
Is there a way to use setuptools to check the existence of this module (which is in PYTHONPATH), and if it can not be found, display some message to the user that the module is missing (and if possible, also some instructions on how to get it)?
Edit: Also, if there is a more elegant way to do this with a different approach than with setuptools, I'd be glad to hear! But I would really like to check directly on installation, not during runtime of my module.
I'm doing a Python program (for use in a Linux environment) and I want it to support version actualizations of the program. I read somewhere I should add a line like "__ Version __" but I'm not really sure where and how to put it. What I want to achieve is not having to erase my whole program every time I want to install a new version of it.
thanks!
I highly recommend you to use setuptools instead of manually versioning it. It is the de-facto stanrad now and very simple to use. All you need to do is to create a setup.py in your projects root directory:
from setuptools import setup, find_packages
setup(name='your_programms_name',
version='major.minor.patch',
packages=find_packages(),
)
and then just run:
python setup.py sdist
and then there will be eggs under your dist folder.
What you actually want to do is make your Python program into a package using distutils.
You would create a setup.py. A simple example would look something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
This is the most standard way to version and distribute Python code. If your code is open source you can even register it with the cheese shop^M^M^M^MPyPi and install it on any machine in the world with a simple pip install mypackage.
It depends what you want to be versioned.
Single modules with independent versions, or the whole program/package.
You could theoretically add a __version__ string to every class and do dynamic imports, by testing these variables.
If you want to version the main, or whole program you could add a __version__ string somewhere at the top of your __init__.py file and import this string into you setup.py when generating packages. This way you wouldn't have to manually edit multiple files and setup.py could be left mostly untouched.
Also consider using a string, not a number or tuple. See PEP 396.
If you are only concerned with versions on your local machine, I would suggest becoming familiar with Git. Information about version control in Git can be found here.
If you are concerned with version control on a module that others would use, information can be found here.
What's the appropriate way of constructing a Python package via disutils when that Python package relies on a large system library?
I found this similar question, but it refers to an installable Python app, not a generic package.
I've written a package that relies on OpenCV. I'm only concerned with supporting Linux distros, but most distros either don't provide OpenCV or provide a version that's too old to use. Unfortunately, OpenCV is to large and cumbersome (and depends on several other system libraries) to include in the package and compile during the build step.
My current approach is to simply do nothing special in my setup.py and just import its Python modules in a try/except, showing a detailed error message if the import fails. Is there a better way?
You could use zc.buildout: http://www.buildout.org/
You should be able to extend the buildout config for your project from this one: https://github.com/cpsaltis/opencv-buildout