Develop Reference

`poetry install` using packages on my system?

I'm still getting used to poetry so I'm hoping that I'm doing something wrong. For some reason, after running poetry install, it looks like not all the packages that I depend on end up in the virtual environment. See the below output:
$ poetry run pip show boto3
Name: boto3
Version: 1.18.29
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: /home/stian/.cache/pypoetry/virtualenvs/some_virtualenv-py3.8/lib/python3.8/site-packages
Requires: s3transfer, jmespath, botocore
Required-by: moto
$ poetry run pip show pluggy
Name: pluggy
Version: 0.13.1
Summary: plugin and hook calling mechanisms for python
Home-page: https://github.com/pytest-dev/pluggy
Author: Holger Krekel
Author-email: holger#merlinux.eu
License: MIT license
Location: /nix/store/l2i2x3g16g6dbwsw4iii614dh3xqgg9m-python3.8-pluggy-0.13.1/lib/python3.8/site-packages
Requires:
Required-by: tox, pytest
Why does this happen? I'm using NixOS, but I'm hoping that's not the cause. To be clear: What I wish is for the "Location" for the pluggy package to also be within the "/home/stian/.cache/pypoetry/virtualenvs/some_virtualenv-py3.8/lib/python3.8/site-packages", as it is for the boto3 package.

How can I review the dependencies for a package in PyPI? [duplicate]

Without going through with the installation, I want to quickly see all the packages that pip install would install.

Check out my project johnnydep!
Installation:
pip install johnnydep
Usage example:
$ johnnydep requests
name summary
------------------------- ----------------------------------------------------------------------
requests Python HTTP for Humans.
├── certifi>=2017.4.17 Python package for providing Mozilla's CA Bundle.
├── chardet<3.1.0,>=3.0.2 Universal encoding detector for Python 2 and 3
├── idna<2.7,>=2.5 Internationalized Domain Names in Applications (IDNA)
└── urllib3<1.23,>=1.21.1 HTTP library with thread-safe connection pooling, file post, and more.
A more complex tree:
$ johnnydep ipython
name summary
-------------------------------- -----------------------------------------------------------------------------
ipython IPython: Productive Interactive Computing
├── appnope Disable App Nap on OS X 10.9
├── decorator Better living through Python with decorators
├── jedi>=0.10 An autocompletion tool for Python that can be used for text editors.
│ └── parso==0.1.1 A Python Parser
├── pexpect Pexpect allows easy control of interactive console applications.
│ └── ptyprocess>=0.5 Run a subprocess in a pseudo terminal
├── pickleshare Tiny 'shelve'-like database with concurrency support
├── prompt-toolkit<2.0.0,>=1.0.4 Library for building powerful interactive command lines in Python
│ ├── six>=1.9.0 Python 2 and 3 compatibility utilities
│ └── wcwidth Measures number of Terminal column cells of wide-character codes
├── pygments Pygments is a syntax highlighting package written in Python.
├── setuptools>=18.5 Easily download, build, install, upgrade, and uninstall Python packages
├── simplegeneric>0.8 Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
└── traitlets>=4.2 Traitlets Python config system
├── decorator Better living through Python with decorators
├── ipython-genutils Vestigial utilities from IPython
└── six Python 2 and 3 compatibility utilities

This was tested with pip versions 8.1.2, 9.0.1, 10.0.1, and 18.1.
To get the output without cluttering your current directory on Linux use
pip download [package] -d /tmp --no-binary :all: -v
-d tells pip the directory that download should put files in.
Better, just use this script with the argument being the package name to get only the dependencies as output:
#!/bin/sh
PACKAGE=$1
pip download $PACKAGE -d /tmp --no-binary :all:-v 2>&1 \
| grep Collecting \
| cut -d' ' -f2 \
| grep -Ev "$PACKAGE(~|=|\!|>|<|$)"
Also available here.

If and only if the package is installed, you can use pip show <package>. Look for the Requires: field at the end of the output. Clearly, this breaks your requirement but might be useful nonetheless.
For example:
$ pip --version
pip 7.1.0 [...]
$ pip show pytest
---
Metadata-Version: 2.0
Name: pytest
Version: 2.7.2
Summary: pytest: simple powerful testing with Python
Home-page: http://pytest.org
Author: Holger Krekel, Benjamin Peterson, Ronny Pfannschmidt, Floris Bruynooghe and others
Author-email: holger at merlinux.eu
License: MIT license
Location: /home/usr/.tox/develop/lib/python2.7/site-packages
Requires: py

Note: the feature used in this answer was deprecated in 2014 and removed in 2015. Please see other answers that apply to modern pip.
The closest you can get with pip directly is by using the --no-install argument:
pip install --no-install <package>
For example, this is the output when installing celery:
Downloading/unpacking celery
Downloading celery-2.5.5.tar.gz (945Kb): 945Kb downloaded
Running setup.py egg_info for package celery
no previously-included directories found matching 'tests/*.pyc'
no previously-included directories found matching 'docs/*.pyc'
no previously-included directories found matching 'contrib/*.pyc'
no previously-included directories found matching 'celery/*.pyc'
no previously-included directories found matching 'examples/*.pyc'
no previously-included directories found matching 'bin/*.pyc'
no previously-included directories found matching 'docs/.build'
no previously-included directories found matching 'docs/graffles'
no previously-included directories found matching '.tox/*'
Downloading/unpacking anyjson>=0.3.1 (from celery)
Downloading anyjson-0.3.3.tar.gz
Running setup.py egg_info for package anyjson
Downloading/unpacking kombu>=2.1.8,<2.2.0 (from celery)
Downloading kombu-2.1.8.tar.gz (273Kb): 273Kb downloaded
Running setup.py egg_info for package kombu
Downloading/unpacking python-dateutil>=1.5,<2.0 (from celery)
Downloading python-dateutil-1.5.tar.gz (233Kb): 233Kb downloaded
Running setup.py egg_info for package python-dateutil
Downloading/unpacking amqplib>=1.0 (from kombu>=2.1.8,<2.2.0->celery)
Downloading amqplib-1.0.2.tgz (58Kb): 58Kb downloaded
Running setup.py egg_info for package amqplib
Successfully downloaded celery anyjson kombu python-dateutil amqplib
Admittedly, this does leave some cruft around in the form of temporary files, but it does accomplish the goal. If you're doing this with virtualenv (which you should be), the cleanup is as easy as removing the <virtualenv root>/build directory.

I quote an alternative solution from #onnovalkering:
PyPi provides a JSON endpoint with package metadata:
>>> import requests
>>> url = 'https://pypi.org/pypi/{}/json'
>>> json = requests.get(url.format('pandas')).json()
>>> json['info']['requires_dist']
['numpy (>=1.9.0)', 'pytz (>=2011k)', 'python-dateutil (>=2.5.0)']
>>> json['info']['requires_python']
'>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'
For a specific package version, add an additional version segment to
the URL:
https://pypi.org/pypi/pandas/0.22.0/json
Also if you are using conda (as suggested by #ShpielMeister), you can use:
conda info package==X.X.X
to display information, including dependencies for a particular version or:
conda info package
to display information, including dependencies about all supported versions of that package.

Use pipdeptree ( pip install pipdeptree). Needs the package to be installed.
$ pipdeptree -p pandas
pandas==1.2.2
- numpy [required: >=1.16.5, installed: 1.19.5]
- python-dateutil [required: >=2.7.3, installed: 2.8.1]
- six [required: >=1.5, installed: 1.15.0]
- pytz [required: >=2017.3, installed: 2021.1]
Use johnnydep (pip install johnnydep). Slower because it download the wheels of the packages.
$ johnnydep pandas
2021-06-09 11:01:21 [info ] init johnnydist [johnnydep.lib] dist=pandas parent=None
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.16.5 parent=pandas
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.7.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2017.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.7.3
name summary
-------------------------- -----------------------------------------------------------------------
pandas Powerful data structures for data analysis, time series, and statistics
├── numpy>=1.16.5 NumPy is the fundamental package for array computing with Python.
├── python-dateutil>=2.7.3 Extensions to the standard Python datetime module
│ └── six>=1.5 Python 2 and 3 compatibility utilities
└── pytz>=2017.3 World timezone definitions, modern and historical

Just an addendum on how to use johnnydep.
If you want to have the dependencies in requirements.txt format:
johnnydep pandas --output-format=pinned
pandas==1.4.1
numpy==1.22.3
python-dateutil==2.8.2
pytz==2021.3
six==1.16.0
Above list will be written to stdout while the informational messages during collection are written to stderr (if you want to capture the output using bash or subprocess).
Calling it from Python instead of the CLI:
import sys, johnnydep.cli
sys.argv = ["", "pandas"]
johnnydep.cli.main()

I think these answers are outdated and there's a better solution now. Original post here:
How can I get a list of packages pip command will install without DOWNLOADING or INSTALLING the packages?
To generate requirements.txt for packages listed in install_requires in your setup.cfg or setup.py, you would need to install pip-tools.
pip install pip-tools
pip-compile
To generate a requirements.txt file that includes packages specified under extras_requires for tests and dev:
pip-compile --extra tests --extra devrequirements.txt file with packages listed under
Furthermore, you can also use requirements.in file instead of setup.cfg or setup.py to list your requirements.
pip-compile requirements.in

The command pip install <package> --download <path> should be used, as mentioned in comments by #radtek, since as of 7.0.0 (2015-05-21), --no-install is removed from pip. This will download the dependencies needed into <path>.

Another option is to use a helper script similar to this one which uses the pip.req.parse_requirements API to parse requirements.txt files and a distutils.core.setup replacement to parse setup.py files.

Licenses for dependencies with Pyinstaller

I have an application I have written in Python, that is being packaged with Pyinstaller.
From what I understand (and I am not a lawyer) and from the Pyinstaller FAQ bundling the source code and distributing with my own license seems like it is OK (points 1 and 2). I have no intention of modifying the Pyinstaller source in anyway, so I don't believe there are any issues here (relating to the Pyinstaller GPL license, points 4 and 5).
My question is specifically about my dependencies, beyond the standard library. pip list for my Virtual Environment gives
(venv) pip list
Package Version
-------------- ---------
altgraph 0.16.1
colorama 0.4.1
et-xmlfile 1.0.1
future 0.18.2
jdcal 1.4.1
nose 1.3.7
numpy 1.17.4
openpyxl 3.0.0
pefile 2019.4.18
Pillow 7.0.0
pip 19.0.3
PyInstaller 3.5
pywin32-ctypes 0.2.0
setuptools 40.8.0
Using numpy as an example pip show numpy
(venv) pip show numpy
Name: numpy
Version: 1.17.4
Summary: NumPy is the fundamental package for array computing with Python.
Home-page: https://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: c:\dev\tool\venv\lib\site-packages
Requires:
Required-by:
And if I look at the BSD license file
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
Does this mean that I need to copy the numpy LICENSE.txt file, and distribute it along with my distributed application (and the same for all other dependencies)? Does this example constitute redistribution in binary form?, or have I misunderstood the meaning here?
It seems like any project with more than a few dependencies, that was using Pyinstaller would have a very large task to keep on top of these dependencies, and so it made me wonder if I was understanding these concepts correctly.

Yep, that's what it means. This is why e.g. your car, your TV, etc. have a "Open Source Licenses" view somewhere in the menus.
On a related note, I've written a tool, license-grep that generates a collated list of the various licenses used by projects in various languages.

How to get Ansible on Fedora 30 to see modules

I installed Ansible 2.8.2 using dnf on Fedora 30. I have an Ansible plug-in that requires a library. Using pip3 install I installed the required library.
When I run ansible-playbook directly, I see a ModuleNotFoundError for that module.
But if I run python3 /usr/bin/ansible-playbook, the module is found.
How can I get Ansible as installed by dnf to see this library?
Edit: further info: as installed from dnf, the main Ansible script has a shebang for /usr/bin/python3 -s. If I remove the -s, this problem is solved.
What's the benefit that the repo maintainers were seeking in adding this -s flag?
Is there a case to be made for asking the repo maintainers to omit the flag?
How can I get pip3 to install the library I need into a directory that will be seen when the -s flag is in effect?
Edit: Here's the output of ansible --version, and thanks for asking.
ansible 2.8.2
config file = /home/jdashton/proj/ansible-ccharacter/ansible.cfg
configured module search path = ['/home/jdashton/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.7.4 (default, Jul 9 2019, 16:32:37) [GCC 9.1.1 20190503 (Red Hat 9.1.1-1)]
This is different from the suggested duplicate question because that question describes a "remote" task being run on localhost, in which case Ansible uses the default "remote" python interpreter. In this question, the library is being called from a local plugin, under Ansible's own python process. The -s flag in the shebang line at the top of the /usr/bin/ansible script is preventing Ansible from seeing some local libraries.
Edit: Following the suggestion from #zigarn I tried removing the library and reinstalling it as root. This resulted in the library being reinstalled into the same directory /usr/local/lib/... as before. Is there a way to get pip3 to install into the system library?
Here are the commands I attempted:
# pip3 uninstall tenacity
Uninstalling tenacity-5.0.4:
Would remove:
/usr/local/lib/python3.7/site-packages/tenacity-5.0.4.dist-info/*
/usr/local/lib/python3.7/site-packages/tenacity/*
Proceed (y/n)? y
Successfully uninstalled tenacity-5.0.4
# pip3 install tenacity
Collecting tenacity
Using cached https://files.pythonhosted.org/packages/6a/93/dfcf5b1b46ab29196274b78dcba69fab5e54b6dc303a7eed90a79194d277/tenacity-5.0.4-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3.7/site-packages (from tenacity) (1.12.0)
Installing collected packages: tenacity
Successfully installed tenacity-5.0.4
# pip show tenacity
Name: tenacity
Version: 5.0.4
Summary: Retry code until it succeeeds
Home-page: https://github.com/jd/tenacity
Author: Julien Danjou
Author-email: julien#danjou.info
License: Apache 2.0
Location: /usr/local/lib/python3.7/site-packages
Requires: six
Required-by:
Also, for clarity, here is the source for the Ansible installation:
# dnf list ansible
Last metadata expiration check: 0:09:48 ago on Mon 12 Aug 2019 11:12:58 AM EDT.
Installed Packages
ansible.noarch 2.8.2-1.fc30 #updates

According to the documentation -s option of python is "Don’t add the user site-packages directory to sys.path."
So I would guess that you did pip3 install as your user instead as root, so the library was installed at your user side instead of system wide.
Try to reinstall the library with pip3 as root and it should be ok.

Is there a way to list pip dependencies/requirements?

Without going through with the installation, I want to quickly see all the packages that pip install would install.

Check out my project johnnydep!
Installation:
pip install johnnydep
Usage example:
$ johnnydep requests
name summary
------------------------- ----------------------------------------------------------------------
requests Python HTTP for Humans.
├── certifi>=2017.4.17 Python package for providing Mozilla's CA Bundle.
├── chardet<3.1.0,>=3.0.2 Universal encoding detector for Python 2 and 3
├── idna<2.7,>=2.5 Internationalized Domain Names in Applications (IDNA)
└── urllib3<1.23,>=1.21.1 HTTP library with thread-safe connection pooling, file post, and more.
A more complex tree:
$ johnnydep ipython
name summary
-------------------------------- -----------------------------------------------------------------------------
ipython IPython: Productive Interactive Computing
├── appnope Disable App Nap on OS X 10.9
├── decorator Better living through Python with decorators
├── jedi>=0.10 An autocompletion tool for Python that can be used for text editors.
│ └── parso==0.1.1 A Python Parser
├── pexpect Pexpect allows easy control of interactive console applications.
│ └── ptyprocess>=0.5 Run a subprocess in a pseudo terminal
├── pickleshare Tiny 'shelve'-like database with concurrency support
├── prompt-toolkit<2.0.0,>=1.0.4 Library for building powerful interactive command lines in Python
│ ├── six>=1.9.0 Python 2 and 3 compatibility utilities
│ └── wcwidth Measures number of Terminal column cells of wide-character codes
├── pygments Pygments is a syntax highlighting package written in Python.
├── setuptools>=18.5 Easily download, build, install, upgrade, and uninstall Python packages
├── simplegeneric>0.8 Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
└── traitlets>=4.2 Traitlets Python config system
├── decorator Better living through Python with decorators
├── ipython-genutils Vestigial utilities from IPython
└── six Python 2 and 3 compatibility utilities

This was tested with pip versions 8.1.2, 9.0.1, 10.0.1, and 18.1.
To get the output without cluttering your current directory on Linux use
pip download [package] -d /tmp --no-binary :all: -v
-d tells pip the directory that download should put files in.
Better, just use this script with the argument being the package name to get only the dependencies as output:
#!/bin/sh
PACKAGE=$1
pip download $PACKAGE -d /tmp --no-binary :all:-v 2>&1 \
| grep Collecting \
| cut -d' ' -f2 \
| grep -Ev "$PACKAGE(~|=|\!|>|<|$)"
Also available here.

If and only if the package is installed, you can use pip show <package>. Look for the Requires: field at the end of the output. Clearly, this breaks your requirement but might be useful nonetheless.
For example:
$ pip --version
pip 7.1.0 [...]
$ pip show pytest
---
Metadata-Version: 2.0
Name: pytest
Version: 2.7.2
Summary: pytest: simple powerful testing with Python
Home-page: http://pytest.org
Author: Holger Krekel, Benjamin Peterson, Ronny Pfannschmidt, Floris Bruynooghe and others
Author-email: holger at merlinux.eu
License: MIT license
Location: /home/usr/.tox/develop/lib/python2.7/site-packages
Requires: py

Note: the feature used in this answer was deprecated in 2014 and removed in 2015. Please see other answers that apply to modern pip.
The closest you can get with pip directly is by using the --no-install argument:
pip install --no-install <package>
For example, this is the output when installing celery:
Downloading/unpacking celery
Downloading celery-2.5.5.tar.gz (945Kb): 945Kb downloaded
Running setup.py egg_info for package celery
no previously-included directories found matching 'tests/*.pyc'
no previously-included directories found matching 'docs/*.pyc'
no previously-included directories found matching 'contrib/*.pyc'
no previously-included directories found matching 'celery/*.pyc'
no previously-included directories found matching 'examples/*.pyc'
no previously-included directories found matching 'bin/*.pyc'
no previously-included directories found matching 'docs/.build'
no previously-included directories found matching 'docs/graffles'
no previously-included directories found matching '.tox/*'
Downloading/unpacking anyjson>=0.3.1 (from celery)
Downloading anyjson-0.3.3.tar.gz
Running setup.py egg_info for package anyjson
Downloading/unpacking kombu>=2.1.8,<2.2.0 (from celery)
Downloading kombu-2.1.8.tar.gz (273Kb): 273Kb downloaded
Running setup.py egg_info for package kombu
Downloading/unpacking python-dateutil>=1.5,<2.0 (from celery)
Downloading python-dateutil-1.5.tar.gz (233Kb): 233Kb downloaded
Running setup.py egg_info for package python-dateutil
Downloading/unpacking amqplib>=1.0 (from kombu>=2.1.8,<2.2.0->celery)
Downloading amqplib-1.0.2.tgz (58Kb): 58Kb downloaded
Running setup.py egg_info for package amqplib
Successfully downloaded celery anyjson kombu python-dateutil amqplib
Admittedly, this does leave some cruft around in the form of temporary files, but it does accomplish the goal. If you're doing this with virtualenv (which you should be), the cleanup is as easy as removing the <virtualenv root>/build directory.

I quote an alternative solution from #onnovalkering:
PyPi provides a JSON endpoint with package metadata:
>>> import requests
>>> url = 'https://pypi.org/pypi/{}/json'
>>> json = requests.get(url.format('pandas')).json()
>>> json['info']['requires_dist']
['numpy (>=1.9.0)', 'pytz (>=2011k)', 'python-dateutil (>=2.5.0)']
>>> json['info']['requires_python']
'>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*'
For a specific package version, add an additional version segment to
the URL:
https://pypi.org/pypi/pandas/0.22.0/json
Also if you are using conda (as suggested by #ShpielMeister), you can use:
conda info package==X.X.X
to display information, including dependencies for a particular version or:
conda info package
to display information, including dependencies about all supported versions of that package.

Use pipdeptree ( pip install pipdeptree). Needs the package to be installed.
$ pipdeptree -p pandas
pandas==1.2.2
- numpy [required: >=1.16.5, installed: 1.19.5]
- python-dateutil [required: >=2.7.3, installed: 2.8.1]
- six [required: >=1.5, installed: 1.15.0]
- pytz [required: >=2017.3, installed: 2021.1]
Use johnnydep (pip install johnnydep). Slower because it download the wheels of the packages.
$ johnnydep pandas
2021-06-09 11:01:21 [info ] init johnnydist [johnnydep.lib] dist=pandas parent=None
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=numpy>=1.16.5 parent=pandas
2021-06-09 11:01:22 [info ] init johnnydist [johnnydep.lib] dist=python-dateutil>=2.7.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=pytz>=2017.3 parent=pandas
2021-06-09 11:01:23 [info ] init johnnydist [johnnydep.lib] dist=six>=1.5 parent=python-dateutil>=2.7.3
name summary
-------------------------- -----------------------------------------------------------------------
pandas Powerful data structures for data analysis, time series, and statistics
├── numpy>=1.16.5 NumPy is the fundamental package for array computing with Python.
├── python-dateutil>=2.7.3 Extensions to the standard Python datetime module
│ └── six>=1.5 Python 2 and 3 compatibility utilities
└── pytz>=2017.3 World timezone definitions, modern and historical

Just an addendum on how to use johnnydep.
If you want to have the dependencies in requirements.txt format:
johnnydep pandas --output-format=pinned
pandas==1.4.1
numpy==1.22.3
python-dateutil==2.8.2
pytz==2021.3
six==1.16.0
Above list will be written to stdout while the informational messages during collection are written to stderr (if you want to capture the output using bash or subprocess).
Calling it from Python instead of the CLI:
import sys, johnnydep.cli
sys.argv = ["", "pandas"]
johnnydep.cli.main()

I think these answers are outdated and there's a better solution now. Original post here:
How can I get a list of packages pip command will install without DOWNLOADING or INSTALLING the packages?
To generate requirements.txt for packages listed in install_requires in your setup.cfg or setup.py, you would need to install pip-tools.
pip install pip-tools
pip-compile
To generate a requirements.txt file that includes packages specified under extras_requires for tests and dev:
pip-compile --extra tests --extra devrequirements.txt file with packages listed under
Furthermore, you can also use requirements.in file instead of setup.cfg or setup.py to list your requirements.
pip-compile requirements.in

The command pip install <package> --download <path> should be used, as mentioned in comments by #radtek, since as of 7.0.0 (2015-05-21), --no-install is removed from pip. This will download the dependencies needed into <path>.

Another option is to use a helper script similar to this one which uses the pip.req.parse_requirements API to parse requirements.txt files and a distutils.core.setup replacement to parse setup.py files.