Develop Reference

How can a pre-built Python-installation be modified to work in another directory?

For a project shipping with a pre-built customized Python distribution I need to be able to compile packages from source using pip (within the installed environment).
This is what the file system structure for two installations of the final product might look like:
/opt
├── my-program-v1
│   ├── some-files
│   ├── custom-python-3.9
├── my-program-v2
│   ├── some-files
│   ├── custom-python-3.11
Since the readily installed program (together with its Python installation) might be installed to any directory, pip needs a way to find the header files for the used Python installation.
Just copying the whole Python installation to the desired directory will result in errors when trying to pip install a package that needs to be built from source (e.g. ibm_db):
$ python3 -m pip install ibm_db
Collecting ibm_db
Downloading ibm_db-3.1.4.tar.gz (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 9.5 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Detected 64-bit Python
Detected platform = linux, uname = x86_64
Downloading https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz
Downloading DSDriver from url = https://public.dhe.ibm.com/ibmdl/export/pub/software/data/db2/drivers/odbc_cli/linuxx64_odbc_cli.tar.gz
Pre-requisite check [gcc] : Passed
No Python.h header file detected.
Please install python-devel or python3-devel as instructed in "https://github.com/ibmdb/python-ibmdb/blob/master/README.md#KnownIssues" and continue with the installation
[end of output]
Usually, when you build Python from source you would set the final destination folder using configure --prefix=/opt/my-program-v2, which would be stored inside the sysconfigdata of that readily built Python installation.
But how can this be done when you need to pre-build Python and don't know the installation directory in advance?
One possible way is to manually modify the generated sysconfig-data, (usually found in lib/<python-version>/_sysconfigdata__linux_x86_64-linux-gnu.py) but this is not fun (due to the used format) and introduces a step outside of the CI.
How do others do that? Is there an intended way to place/relocate a prebuilt installation of python?

Limit package to CPython - setup.cfg

I'm using pyproject.toml and setup.cfg per PEP-517.
I have my setup.cfg as follows:
[metadata]
name = Package
version = 1.0.0
[options]
py_modules = ...
python_requires = >=3
I wish to limit the package to work only on CPython, as it solves an implementation-specific issue.
I've tried using the environment markers, but unfortunately they don't work on the python_requires field:
python_requires = >=3; implementation_name == "cpython"
Is there any way to achieve what I wish without resorting to the deprecated setup.py?

Unfortunately, you can't restrict the implementation via environment markers and the package metadata doesn't define any suitable field. However, the implementation can be restricted via the language implementation tag in wheel metadata; the format of the tag is defined in PEP 425. The CPython implementation is abbreviated with cp, python_requires = >=3 from your config implies cp3 for the full tag. Set the tag in a [bdist_wheel] section in setup.cfg:
[metadata]
...
[options]
...
[bdist_wheel]
python_tag=cp3
This is not special to bdist_wheel; any distutils subcommand can have a section in setup.cfg where one can persist its command line options. Read through Writing the Setup Configuration File for more details on that.
To answer your question from the comments:
To be honest I haven't even found the online documentation for the python-tag command. Am I missing something?
This is indeed somewhat hidden; wheel's docs don't provide a reference for the bdist_wheel's options. Usually, you list them via python setup.py bdist_wheel --help (just as with any other subcommand), but since you don't have the setup script, you can fake one:
$ python -c "from setuptools import setup; setup()" bdist_wheel --help
Options for 'bdist_wheel' command:
...
--python-tag Python implementation compatibility tag (default: 'py3')
A built wheel file will now have the cp3-none-any suffix (and the cp3-none-any tag in wheel metadata), and will be rejected by other implementations:
$ pypy3 -m pip install meowpkg-0.0.1-cp3-none-any.whl
ERROR: meowpkg-0.0.1-cp3-none-any.whl is not a supported wheel on this platform.
This will also work with source dists since pip will always build a wheel from PEP 517-compliant sdists (and not resort to legacy setup.py install), so this is also safe to use with source dists. Example:
$ python -m build
$ pip uninstall -y wheel # for testing only
WARNING: Skipping wheel as it is not installed.
$ pip install dist/meowpkg-0.0.1.tar.gz
Processing ./dist/meowpkg-0.0.1.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Building wheels for collected packages: meowpkg
Building wheel for meowpkg (PEP 517) ... done
Created wheel for meowpkg: filename=meowpkg-0.0.1-cp3-none-any.whl size=1005 sha256=d87571afcdeda6be74a182b9e3e1ce6035a4aea4bb173c291900a85e53232983
Stored in directory: /home/oleg.hoefling/.cache/pip/wheels/1a/8c/43/90d6b484adcf2515d25503b0c47f14565cadf0e891a597e23e
Successfully built meowpkg
Installing collected packages: meowpkg
Attempting uninstall: meowpkg
Found existing installation: meowpkg 0.0.1
Uninstalling meowpkg-0.0.1:
Successfully uninstalled meowpkg-0.0.1
Successfully installed meowpkg-0.0.1

extras_require with readthedocs and poetry

I use poetry to build my package with cython extensions, so install package flag is enabled. I'd like to host documentation via readthedocs. Poetry uses build.py file to generate setup.py with poetry build command. However readthedocs does not support build.py so I provided setup.py with following contents:
from setuptools import setup
from build import *
global setup_kwargs
setup_kwargs = {}
build(setup_kwargs)
setup(**setup_kwargs)
To get rid of requirements.txt file in docs folder, I would like to add extras_require parameter to setup, so the last line in setup.py:
setup(extras_require={'docs': ['toml']}, **setup_kwargs)
Readthedocs calls /<path>/envs/latest/bin/python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[docs]
WARNING: does not provide the extra 'docs' and importing toml from docs/conf.py fails
If I add extras to pyproject.toml:
[tool.poetry.extras]
docs = ['toml']
Warning disappears but still rtd fails to import toml.
My .readthedocs.yml:
version: 2
sphinx:
configuration: docs/conf.py
formats: all
python:
version: 3.7
install:
- method: pip
path: .
extra_requirements:
- docs
submodules:
include: all

How can I review the dependencies for a package in PyPI? [duplicate]

Without going through with the installation, I want to quickly see all the packages that pip install would install.

Check out my project johnnydep!
Installation:
pip install johnnydep
Usage example:
$ johnnydep requests
name summary
------------------------- ----------------------------------------------------------------------
requests Python HTTP for Humans.
├── certifi>=2017.4.17 Python package for providing Mozilla's CA Bundle.
├── chardet<3.1.0,>=3.0.2 Universal encoding detector for Python 2 and 3
├── idna<2.7,>=2.5 Internationalized Domain Names in Applications (IDNA)
└── urllib3<1.23,>=1.21.1 HTTP library with thread-safe connection pooling, file post, and more.
A more complex tree:
$ johnnydep ipython
name summary
-------------------------------- -----------------------------------------------------------------------------
ipython IPython: Productive Interactive Computing
├── appnope Disable App Nap on OS X 10.9
├── decorator Better living through Python with decorators
├── jedi>=0.10 An autocompletion tool for Python that can be used for text editors.
│ └── parso==0.1.1 A Python Parser
├── pexpect Pexpect allows easy control of interactive console applications.
│ └── ptyprocess>=0.5 Run a subprocess in a pseudo terminal
├── pickleshare Tiny 'shelve'-like database with concurrency support
├── prompt-toolkit<2.0.0,>=1.0.4 Library for building powerful interactive command lines in Python
│ ├── six>=1.9.0 Python 2 and 3 compatibility utilities
│ └── wcwidth Measures number of Terminal column cells of wide-character codes
├── pygments Pygments is a syntax highlighting package written in Python.
├── setuptools>=18.5 Easily download, build, install, upgrade, and uninstall Python packages
├── simplegeneric>0.8 Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
└── traitlets>=4.2 Traitlets Python config system
├── decorator Better living through Python with decorators
├── ipython-genutils Vestigial utilities from IPython
└── six Python 2 and 3 compatibility utilities

This was tested with pip versions 8.1.2, 9.0.1, 10.0.1, and 18.1.
To get the output without cluttering your current directory on Linux use
pip download [package] -d /tmp --no-binary :all: -v
-d tells pip the directory that download should put files in.
Better, just use this script with the argument being the package name to get only the dependencies as output:
#!/bin/sh
PACKAGE=$1
pip download $PACKAGE -d /tmp --no-binary :all:-v 2>&1 \
| grep Collecting \
| cut -d' ' -f2 \
| grep -Ev "$PACKAGE(~|=|\!|>|<|$)"
Also available here.

If and only if the package is installed, you can use pip show <package>. Look for the Requires: field at the end of the output. Clearly, this breaks your requirement but might be useful nonetheless.
For example:
$ pip --version
pip 7.1.0 [...]
$ pip show pytest
---
Metadata-Version: 2.0
Name: pytest
Version: 2.7.2
Summary: pytest: simple powerful testing with Python
Home-page: http://pytest.org
Author: Holger Krekel, Benjamin Peterson, Ronny Pfannschmidt, Floris Bruynooghe and others
Author-email: holger at merlinux.eu
License: MIT license
Location: /home/usr/.tox/develop/lib/python2.7/site-packages
Requires: py

Note: the feature used in this answer was deprecated in 2014 and removed in 2015. Please see other answers that apply to modern pip.
The closest you can get with pip directly is by using the --no-install argument:
pip install --no-install <package>
For example, this is the output when installing celery:
Downloading/unpacking celery
Downloading celery-2.5.5.tar.gz (945Kb): 945Kb downloaded
Running setup.py egg_info for package celery
no previously-included directories found matching 'tests/*.pyc'
no previously-included directories found matching 'docs/*.pyc'
no previously-included directories found matching 'contrib/*.pyc'
no previously-included directories found matching 'celery/*.pyc'
no previously-included directories found matching 'examples/*.pyc'
no previously-included directories found matching 'bin/*.pyc'
no previously-included directories found matching 'docs/.build'
no previously-included directories found matching 'docs/graffles'
no previously-included directories found matching '.tox/*'
Downloading/unpacking anyjson>=0.3.1 (from celery)
Downloading anyjson-0.3.3.tar.gz
Running setup.py egg_info for package anyjson
Downloading/unpacking kombu>=2.1.8,<2.2.0 (from celery)
Downloading kombu-2.1.8.tar.gz (273Kb): 273Kb downloaded
Running setup.py egg_info for package kombu
Downloading/unpacking python-dateutil>=1.5,<2.0 (from celery)
Downloading python-dateutil-1.5.tar.gz (233Kb): 233Kb downloaded
Running setup.py egg_info for package python-dateutil
Downloading/unpacking amqplib>=1.0 (from kombu>=2.1.8,<2.2.0->celery)
Downloading amqplib-1.0.2.tgz (58Kb): 58Kb downloaded
Running setup.py egg_info for package amqplib
Successfully downloaded celery anyjson kombu python-dateutil amqplib
Admittedly, this does leave some cruft around in the form of temporary files, but it does accomplish the goal. If you're doing this with virtualenv (which you should be), the cleanup is as easy as removing the <virtualenv root>/build directory.

I quote an alternative solution from #onnovalkering:
PyPi provides a JSON endpoint with package metadata:
>>> import requests
>>> url = 'https://pypi.org/pypi/{}/json'
>>> json = requests.get(url.format('pandas')).json()
>>> json['info']['requires_dist']
['numpy (>=1.9.0)', 'pytz (>=2011k)', 'python-dateutil (>=2.5.0)']
>>> json['info']['requires_python']
'>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'
For a specific package version, add an additional version segment to
the URL:
https://pypi.org/pypi/pandas/0.22.0/json
Also if you are using conda (as suggested by #ShpielMeister), you can use:
conda info package==X.X.X
to display information, including dependencies for a particular version or:
conda info package
to display information, including dependencies about all supported versions of that package.

Use pipdeptree ( pip install pipdeptree). Needs the package to be installed.
$ pipdeptree -p pandas
pandas==1.2.2
- numpy [required: >=1.16.5, installed: 1.19.5]
- python-dateutil [required: >=2.7.3, installed: 2.8.1]
- six [required: >=1.5, installed: 1.15.0]
- pytz [required: >=2017.3, installed: 2021.1]
Use johnnydep (pip install johnnydep). Slower because it download the wheels of the packages.
$ johnnydep pandas
2021-06-09 11:01:21 [info ] init johnnydist [johnnydep.lib] dist=pandas parent=None
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.16.5 parent=pandas
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.7.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2017.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.7.3
name summary
-------------------------- -----------------------------------------------------------------------
pandas Powerful data structures for data analysis, time series, and statistics
├── numpy>=1.16.5 NumPy is the fundamental package for array computing with Python.
├── python-dateutil>=2.7.3 Extensions to the standard Python datetime module
│ └── six>=1.5 Python 2 and 3 compatibility utilities
└── pytz>=2017.3 World timezone definitions, modern and historical

Just an addendum on how to use johnnydep.
If you want to have the dependencies in requirements.txt format:
johnnydep pandas --output-format=pinned
pandas==1.4.1
numpy==1.22.3
python-dateutil==2.8.2
pytz==2021.3
six==1.16.0
Above list will be written to stdout while the informational messages during collection are written to stderr (if you want to capture the output using bash or subprocess).
Calling it from Python instead of the CLI:
import sys, johnnydep.cli
sys.argv = ["", "pandas"]
johnnydep.cli.main()

I think these answers are outdated and there's a better solution now. Original post here:
How can I get a list of packages pip command will install without DOWNLOADING or INSTALLING the packages?
To generate requirements.txt for packages listed in install_requires in your setup.cfg or setup.py, you would need to install pip-tools.
pip install pip-tools
pip-compile
To generate a requirements.txt file that includes packages specified under extras_requires for tests and dev:
pip-compile --extra tests --extra devrequirements.txt file with packages listed under
Furthermore, you can also use requirements.in file instead of setup.cfg or setup.py to list your requirements.
pip-compile requirements.in

The command pip install <package> --download <path> should be used, as mentioned in comments by #radtek, since as of 7.0.0 (2015-05-21), --no-install is removed from pip. This will download the dependencies needed into <path>.

Another option is to use a helper script similar to this one which uses the pip.req.parse_requirements API to parse requirements.txt files and a distutils.core.setup replacement to parse setup.py files.

Installation of a python package via setup.py

I have made a bug fix in a popular python package fastparquet https://github.com/dask/fastparquet and attempting to try it out on my machine. The package comes with a setup.py and so I have attempted to use that to install:
fastparquet $ python setup.py install
This completed and appears to have done the requisite installation of files to /usr/local/lib/python2.7/site-packages:
..
Adding python-dateutil 2.6.0 to easy-install.pth file
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for fastparquet==0.1.0
However when we examine that location - specifically for the api.py file that was updated - it is still the original github version:
11:35:38/fastparquet $ll /usr/local/lib/python2.7/site-packages/fastparquet/api.py
-rw-r--r-- 1 sboesch admin 27294 Jun 19 17:18 /usr/local/lib/python2.7/site-packages/fastparquet/api.py
After having looked a bit into setup.py it seems that were a standard approach used across most python packages: https://docs.python.org/2/distutils/setupscript.html.
So then
am I misunderstanding how the install works /what it does?