How to get pip to clean up better?
I have a pip module that does two things I would like un-done:
a) remove a dependent module dynamically specified during install - think sister binary package with os and python version in name. Those are now obsolete since I've learned the manylinux (and soon the universal) wheel dance.
b) remove license, usage, and telemetry files created while running the module.
Is there a hook or something that I can add a bit of logic and fileio too?
Seems option a was designed for but never implemented with the obsoletes keyword found at https://setuptools.pypa.io/en/latest/references/keywords.htm
Related
How should a file myModule.cpython-35m-x86_64-linux-gnu.so be imported in python? Is it possible?
I tried the regular way:
import myModule
and the interpreter says:
`ModuleNotFoundError: No module named 'myModule'`
This is a software that I can't install in the cluster that I am working at so I just extracted the .deb package and it does not have a wheel file or structure to install.
It is problematic to use a C-extension built for one Python version in another Python version. Normally (at least for Python3) there is a mechanism in place to differentiate C-extensions for different Python versions, so they can co-exist in the same directory.
In your example, the suffix is cpython-35m-x86_64-linux-gnu so this C-extension will be picked up by a CPython3.5 on a x86_64 Linux. If you try to import this extension with another Python-version or on another plattform, the module isn't visible and ModuleNotFoundError is raised.
It is possible to see, which suffixes are accepted by the current Python version, e.g. via:
>>> import _imp
>>>_imp.extension_suffixes()
['.cpython-36m-x86_64-linux-gnu.so', '.abi3.so', '.so']
A possibility is to use the stable C-API which could be used with multiple Python versions without recompilation. Cython start to support it in version 3.0 (see this PR), see also this SO-post about setuptools and stable C-API.
One might want to be clever and rename the extension to simple .so, so it can be picked up by the Finder - this can/does work for some Python-version combinations on some platforms for some extension - yet this approach cannot be sustained in the long run and is not the right thing to do.
The right thing to do, is to build the C-extension for/with the right Python-version on the right OS/platform or to use the right wheel (or use stable C-API).
In general, a C-extension built for a python-version (let's say PythonA.B) cannot be used by another Python version (let's say PythonC.D), because those extensions/modules are linked against a special Python-library and the needed functionality might no longer/not yet be present in the library of another version.
This different to *.py-files and more similar to *.pyc-files which cannot be used with a different version.
While PEP-3147 regulates the suffices of *.pyc-files, PEP-3149 does the same for the C-extensions. PEP-3149 is however not the state-of-the-art, as some of the problems where fixed only in Python3.5, the whole discussion can be found here.
I have a project that needs some DevOps TLC, so I am finally building my installation script. This will eventually be a package that will be install-able by pip locally, but may not end up in PyPI.
It has a dependency for a module called u2py. It is this package, created for U2 Database operations, not this package, for... something else. The one I want is only ever installed by a 3rd party vendor (Rocket), the one I don't want is in PyPI.
What should be the expected behavior of my package in this case? I will include a blurb about this in my readme doc, but is that sufficient?
I've thought about throwing an exception to identify when the wrong package is present, but that makes me feel weird. It seems that maybe the most pythonic thing is to NOT add this to my install script, and blindly assume import u2py results in a module I can use. If it quacks like a duck, parses DynArrays like a duck, and call()s SUBROUTINEs like a duck, then it's a duck, right? Otherwise, if there is an error the user will just go and actually read the docs.
I've looked a classifiers, but not sure if they apply here.
Ideally there would be a way at install-time (in setup.py) to detect whether the package is being installed into a "u2 environment" or not, and could fail the installation (with an appropriate error message) if that's the case.
With this solution, you won't be able to provide built distributions (wheels) since they don't execute the setup.py file at install-time, but just publishing source distributions should be fine.
It's a case where it would be nice if Python projects had namespaces (pip install com.rocket.u2py and import com.rocket.u2py as u2py).
From my point of view there are 2 aspects to consider: at the project level, at the package level.
1. project (distribution package)
I believe it is a bad practice to force alternative download sources onto the end user of your project. By default, pip should download from PyPI and nowhere else, unless the user decides it themselves (via --find-links or similar options, which you could instruct your users to do in your documentation).
Since it is such a niche dependency, I think I would simply not add it to install_requires. I would assume the end users of your project know about the dependency already and are able to install it themselves directly.
Also I don't believe it is possible to check reliably at install-time if the correct dependency is installed, since setup.py does not always run (overriding the bdist_wheel command can help, but probably not 100% effective).
2. package (importable package)
I am not sure some specific action is needed. The code would most likely fail sooner or later naturally, because of module or function is not importable. Which might be okay-ish, maybe?
But probably detecting if the dependency is installed (and it is the correct one), is relatively easy and would provide a better user experience. Either check that some specific modules or functions are importable. Or inspect the meta-data (import importlib_metadata; importlib_metadata.distribution('u2py').metadata['Author']).
In case of an application, I would try to fail gracefully as soon as possible. In case of a library I would try to find one strategic spot to place the check and raise a custom exception (CannotFindU2pyException).
Links:
Prevent pip from caching a package
https://docs.python.org/3/library/importlib.metadata.html#distribution-metadata
https://github.com/pypa/pip/issues/4187#issuecomment-415067034
Equivalent for `--find-links` in `setup.py`
You can specify the url to the package in install_requires using setuptools (requires pip version 18.1 or greater).
Requirement specifiers
Example
setup.py
import setuptools
setuptools.setup(
name='MyPackage',
version='1.0.0',
# ...
install_requires=[
'requests # https://github.com/psf/requests/archive/v2.22.0.zip'
]
# ...
)
and do python setup.py install
Also
Since version 19.1, pip also supports direct references like so:
SomeProject # file:///somewhere/...
Ref
https://www.python.org/dev/peps/pep-0508/
https://github.com/pypa/pip/pull/4175
I maintain a Python utility that allows bpy to be installable as a Python module. Due to the hugeness of the spurce code, and the length of time it takes to download the libraries, I have chosen to provide this module as a wheel.
Unfortunately, platform differences and Blender runtime expectations makes support for this tricky at times.
Currently, one of my big goals is to get the Blender addon scripts directory to install into the correct location. The directory (simply named after the version of Blender API) has to exist in the same directory as the Python executable.
Unfortunately the way that setuptools works (or at least the way that I have it configured) the 2.79 directory is not always placed as a sibling to the Python executable. It fails on Windows platforms outside of virtual environments.
However, I noticed in setuptools documentation that you can specify eager_resources that supposedly guarantees the location of extracted files.
https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-resource-extraction
https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resource-extraction
There was a lot of hand waving and jargon in the documentation, and 0 examples. I'm really confused as to how to structure my setup.py file in order to guarantee the resource extraction. Currently, I just label the whole 2.79 directory as "scripts" in my setuptools Extension and ship it.
Is there a way to write my setup.py and package my module so as to guarantee the 2.79 directory's location is the same as the currently running python executable when someone runs
py -3.6.8-32 -m pip install bpy
Besides simply "hacking it in"? I was considering writing a install_requires module that would simply move it if possible but that is mangling with the user's file system and kind of hacky. However it's the route I am going to go if this proves impossible.
Here is the original issue for anyone interested.
https://github.com/TylerGubala/blenderpy/issues/13
My build process is identical to the process descsribed in my answer here
https://stackoverflow.com/a/51575996/6767685
Maybe try the data_files option of distutils/setuptools.
You could start by adding data_files=[('mydata', ['setup.py'],)], to your setuptools.setup function call. Build a wheel, then install it and see if you can find mydata/setup.py somewhere in your sys.prefix.
In your case the difficult part will be to compute the actual target directory (mydata in this example). It will depend on the platform (Linux, Windows, etc.), if it's in a virtual environment or not, if it's a global or local install (not actually feasible with wheels currently, see update below) and so on.
Finally of course, check that everything gets removed cleanly on uninstall. It's a bit unnecessary when working with virtual environments, but very important in case of a global installation.
Update
Looks like your use case requires a custom step at install time of your package (since the location of the binary for the Python interpreter relative to sys.prefix can not be known in advance). This can not be done currently with wheels. You have seen it yourself in this discussion.
Knowing this, my recommendation would be to follow the advice from Jan Vlcinsky in his comment for his answer to this question:
Post install script after installing a wheel.
Add an extra setuptools console entry point to your package (let's call it bpyconfigure).
Instruct the users of your package to run it immediately after installing your package (pip install bpy && bpyconfigure).
The purpose of bpyconfigure should be clearly stated (in the documentation and maybe also as a notice shown in the console right after starting bpyconfigure) since it would write into locations of the file system where pip install does not usually write.
bpyconfigure should figure out where is the Python interpreter, and where to write the extra data.
The extra data to write should be packaged as package_data, so that it can be found with pkg_resources.
Of course bpyconfigure --uninstall should be available as well!
I'm using setuptools to install a Python module that I'm working on. In addition to numpy, scipy, ..., whose presence I can assure with install_requires = [...], my module also depends on a Python module - let's call it specialmodule - that is a Python interface to a program that is neither an egg, nor a single .py-file or a VCS repo (so Dependencies that aren’t in PyPI is not applicable). The program is written in C++ and has a Python interface, and can either be built from source after cloning from git, or obtained as a tar archive.
Is there a way to use setuptools to check the existence of this module (which is in PYTHONPATH), and if it can not be found, display some message to the user that the module is missing (and if possible, also some instructions on how to get it)?
Edit: Also, if there is a more elegant way to do this with a different approach than with setuptools, I'd be glad to hear! But I would really like to check directly on installation, not during runtime of my module.
How can I check is some package is installed in my system. My system is Linux, but even better if it could works in other OSs. I mean OS specific package (like could be *.rpm or *.deb).
Is there any python module or script that could do it?
To find out whether you've installed a .deb, .rpm, etc. package, you need to use the appropriate tools for your packaging system.
APT has a Python wrapper named python-apt in Debian, or just apt at PyPI.
RPM has a whole slew of Python tools—in fact, most of Redhat's installer ecosystem is built on Python, and you should already have the rpm module installed. Read Programming RPM with Python (or, better, search for a newer version…) before looking for a high-level wrapper, so you understand what you're actually doing; it's only a couple lines of code even with the low-level interface.
As far as I know, nobody has wrapped these up in a universal tool for every packaging format and database that any linux distro has ever used (and, even if they had, that wouldn't do you much good on linux systems that don't use a packaging system). But if you just want to handle a handful of popular systems, python-apt and either Redhat's own tools or search PyPI for RPM, and that will cover almost everything you care about.
Alternatively, pkg-config is the closest thing to a universal notion of "packages installed on this system". Every linux system will have it (and most other non-Windows systems), but not every package registers with pkg-config. Still, if this is what you're looking for, pkgconfig is the Python answer.
The word "package" has a half-dozen similar but incompatible meanings, but the fact that you said "package or module" implies you specifically want to know about Python packages and modules, as in the things you can import.
In which case, the way to test it is to import them.
Manually, do this:
$ python
>>> import foo
ImportError: No module named foo
Well, foo isn't installed.
Programmatically:
try:
import foo
except ImportError:
# do whatever you wanted if foo is missing
Note that this doesn't actually tell you foo is missing, just that it couldn't be imported. In a simple "test whether you have this" script, that's generally what you want to actually check for. But what if you really want to check "is installed (even if broken)"?
In recent Python (I think 3.4+), the ImportError will have additional information in it that you can access—name for the name you were trying to import, path if it was found, etc. However, this is one of those cases where EAFP may not be better than LBYL. You can use importlib to search for the module without trying to import it, like this:
spec = importlib.util.find_spec('foo')
What if you're using an older Python? There are similar features going back to 3.2, but not quite as nice, and if you're using 2.7, there's really nothing worth using, because the import machinery wasn't exposed very well.
For that case (and many, many other cool things related to package installation), use setuptools—which isn't in the stdlib, but a huge number of third-party packages depend on it (until recently it was the cornerstone of Python package installation, even if unofficially):
pkg_resources.get_distribution('foo')
However, that looks for a distutils/setuptools/PyPI package, not a Python module or package. There's a lot of overlap there, but they're not exactly the same thing. For a simple example, when you pip install more-itertools, you get the more-itertools PyPI package, which installs the more_itertools Python package into your site-packages.