Identifying the dependency relationship for python packages installed with pip - python

When I do a pip freeze I see large number of Python packages that I didn't explicitly install, e.g.
$ pip freeze
Cheetah==2.4.3
GnuPGInterface==0.3.2
Landscape-Client==11.01
M2Crypto==0.20.1
PAM==0.4.2
PIL==1.1.7
PyYAML==3.09
Twisted-Core==10.2.0
Twisted-Web==10.2.0
(etc.)
Is there a way for me to determine why pip installed these particular dependent packages? In other words, how do I determine the parent package that had these packages as dependencies?
For example, I might want to use Twisted and I don't want to depend on a package until I know more about not accidentally uninstalling it or upgrading it.


You could try pipdeptree which displays dependencies as a tree structure e.g.:
$ pipdeptree
Lookupy==0.1
wsgiref==0.1.2
argparse==1.2.1
psycopg2==2.5.2
Flask-Script==0.6.6
- Flask [installed: 0.10.1]
- Werkzeug [required: >=0.7, installed: 0.9.4]
- Jinja2 [required: >=2.4, installed: 2.7.2]
- MarkupSafe [installed: 0.18]
- itsdangerous [required: >=0.21, installed: 0.23]
alembic==0.6.2
- SQLAlchemy [required: >=0.7.3, installed: 0.9.1]
- Mako [installed: 0.9.1]
- MarkupSafe [required: >=0.9.2, installed: 0.18]
ipython==2.0.0
slugify==0.0.1
redis==2.9.1
To get it run:
pip install pipdeptree
EDIT: as noted by #Esteban in the comments you can also list the tree in reverse with -r or for a single package with -p <package_name> so to find what installed Werkzeug you could run:
$ pipdeptree -r -p Werkzeug
Werkzeug==0.11.15
- Flask==0.12 [requires: Werkzeug>=0.7]


The pip show command will show what packages are required for the specified package (note that the specified package must already be installed):
$ pip show specloud
Package: specloud
Version: 0.4.4
Requires:
nose
figleaf
pinocchio
pip show was introduced in pip version 1.4rc5


As I recently said on a hn thread, I'll recommend the following:
Have a commented requirements.txt file with your main dependencies:
## this is needed for whatever reason
package1
Install your dependencies: pip install -r requirements.txt.
Now you get the full list of your dependencies with pip freeze -r requirements.txt:
## this is needed for whatever reason
package1==1.2.3
## The following requirements were added by pip --freeze:
package1-dependency1==1.2.3
package1-dependency1==1.2.3
This allows you to keep your file structure with comments, nicely separating your dependencies from the dependencies of your dependencies. This way you'll have a much nicer time the day you need to remove one of them :)
Note the following:
You can have a clean requirements.raw with version control to rebuild your full requirements.txt.
Beware of git urls being replaced by egg names in the process.
The dependencies of your dependencies are still alphabetically sorted so you don't directly know which one was required by which package but at this point you don't really need it.
Use pip install --no-install <package_name> to list specific requirements.
Use virtualenv if you don't.


You may also use a one line command which pipes the packages in requirements to pip show.
cut -d'=' -f1 requirements.txt | xargs pip show


The following command will show requirements of all installed packages:
pip3 freeze | awk '{print $1}' | cut -d '=' -f1 | xargs pip3 show


First of all pip freeze displays all currently installed packages Python, not necessarily using PIP.
Secondly Python packages do contain the information about dependent packages as well as required versions. You can see the dependencies of particular pkg using the methods described here. When you're upgrading a package the installer script like PIP will handle the upgrade of dependencies for you.
To solve updating of packages i recommend using PIP requirements files. You can define what packages and versions you need, and install them at once using pip install.


(workaround, not true answer)
Had the same problem, with lxml not installing and me wanting to know who needed lxml. Not who lxml needed. Ended up bypassing the issue by.
noting where my site packages were being put.
go there and recursive grep for the import (the last grep's --invert-match serves to remove lxml's own files from consideration).
Yes, not an answer as to how to use pip to do it, but I didn't get any success out of the suggestions here, for whatever reason.
site-packages me$ egrep -i --include=*.py -r -n lxml . | grep import | grep --invert-match /lxml/


I wrote a quick script to solve this problem. The following script will display the parent (dependant) package(s) for any given package. This way you can be sure it is safe to upgrade or install any particular package. It can be used as follows: dependants.py PACKAGENAME
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Find dependants of a Python package"""
import logging
import pip
import pkg_resources
import sys
__program__ = 'dependants.py'
def get_dependants(target_name):
for package in pip._internal.utils.misc.get_installed_distributions():
for requirement_package in package.requires():
requirement_name = requirement_package.project_name
if requirement_name == target_name:
yield package.project_name
# configure logging
logging.basicConfig(format='%(levelname)s: %(message)s',
level=logging.INFO)
try:
target_name = sys.argv[1]
except IndexError:
logging.error('missing package name')
sys.exit(1)
try:
pkg_resources.get_distribution(target_name)
except pkg_resources.DistributionNotFound:
logging.error("'%s' is not a valid package", target_name)
sys.exit(1)
print(list(get_dependants(target_name)))


You have two options here.
The first will output all top-level packages, excluding sub packages. Note that this will also exclude for example requests, even if you want to have it explicitly installed
pip3 list --not-required --format freeze --exclude pip --exclude setuptools
The second option is to print the packages based on the existing requirements.txt file.
pip3 freeze -r requirements.txt
This will generate a file in the format:
existing-package==1.0.0
## The following requirements were added by pip freeze:
dependency-package==1.0.0
You can remove all the additionally added packages by using sed:
pip3 freeze -r requirements.txt | sed -n '/## The following requirements were added by pip freeze:/q;p'


With GraphVis as seen on tv
If you like graphs you can use graphviz (Documentation)
pip install graphviz
Then do something like this:
#! /usr/bin/env python3
import graphviz
import pkg_resources
GRAPH_NAME = "pipdeps"
def init_grph():
grph = graphviz.Digraph(GRAPH_NAME,
node_attr={'color': 'lightblue2', 'style': 'filled'})
# This does not seem to be interpreted on websites I tested
grph.attr(engine='neato')
return grph
def add_pip_dependencies_to_graph(grph):
l_packages = [p for p in pkg_resources.working_set]
for package in l_packages:
name = package.key
for dep in package.requires():
grph.edge(dep.key, name)
def main():
grph = init_grph()
add_pip_dependencies_to_graph(grph)
print(grph.source)
# grph.view()
main()
This prints a dotgraph/digraph/graphviz (idk 🤷‍♀️)
You can view it on an online graphviz visualiser (e.g. )
If you have a graphic interface (😮) you can use grph.view()
hth

Related

How can I check that the installed packages match Pipfile.lock using pipenv?

In my tests, I would like to run a command to make sure that the installed packages in my virtual environment match the packages found in Pipfile.lock.
Is there a command like this?
pipenv checkifinstalled || exit 1

This problem can be reduced down to these two steps:
Convert Pipfile.lock into a requirements.txt file (in the format generated by pip freeze).
This is easily done by running pipenv requirements (or pipenv requirements --dev). (Note that this command is supported in pipenv >= 2022.4.8)
Check that the installed packages match the generated requirements.txt file.
Solutions to this are found under this question: Check if my Python has all required packages
Implementation:
Here is how I put it all together in a test:
import pkg_resources
import subprocess
import unittest
class DependencyChecks(unittest.TestCase):
def test_requirements_installed(self):
requirements_lines = subprocess.check_output(["pipenv", "requirements", "--dev"], text=True).splitlines()
req_lines = [line for line in requirements_lines if not line.startswith("-i ")]
requirements = pkg_resources.parse_requirements(req_lines)
for requirement in requirements:
req = str(requirement)
with self.subTest(requirement=req):
pkg_resources.require(req)

I'm getting the following Import Error when importing the sutime module - what does it mean?

I'm getting this error:
ImportError: cannot import name 'SUTime' from partially initialized module 'sutime' (most likely due to a circular import)
when importing the sutime module as:
from sutime import SUTime
as suggested in the sutime GitHub example: https://github.com/FraBle/python-sutime
Context: sutime is a Python library for parsing date/time from a natural language input, developed by the amazing team over at Stanford CoreNLP.
Note: I've already run the pre-req installs as well:
>> pip install setuptools_scm jpype1 # install pre-reqs
>> pip install sutime
>> # use package pom.xml to install all Java dependencies via Maven into ./jars
>> mvn dependency:copy-dependencies -DoutputDirectory=./jars

It's not true that a circular import is the most likely cause of your error. A failed, incomplete, or in some way strange installation is more likely. Try this:
pip uninstall sutime
pip uninstall jpype1
pip uninstall setuptools_scm
pip3 install setuptools_scm jpype1 # note: pip3
pip3 install sutime
Then, in the python-sutime directory, enter this command:
./test.sh
It should output lots of log lines, and the last line but one should be similar to this:
======================== 5 passed, 2 warnings in 13.06s ========================
In the same directory you can enter and run the Python script from the Example section of the README. It should output many log lines before the reported output. There may be a way to avoid that, but anyway it happens only once, when the script starts.

Clone the git repo. -
!git clone https://github.com/FraBle/python-sutime.git
Go to the python-sutime/sutime in the cloned repo. There is a pom.xml file. Open the terminal and issue the following command.
mvn dependency:copy-dependencies -DoutputDirectory=./jars -P english
Now you can simply import the sutime from the sutime.py script in the cloned repo folder.
If you want to use sutime from anywhere, install sutime using...
pip install sutime
and replace the /usr/local/lib/python3.6/dist-packages/sutime folder with the sutime folder you get after step 2.

Pex: Could not satisfy all requirements

I'm trying to package a python virtual environment using pex, but can't seem to shake off the "Could not satisfy all requirements for..." error. This is either me being a total python newb question, or an ask to help me find out to figure out what these pesky requirements are that pex couldn't satisfy.
Here's the error message I'm seeing:
(env-rba-deploy)my_machine:env-rba-deploy my_user$ pex -v --disable-cache -r <(pip freeze) -o foo.pex
Could not satisfy all requirements for rba-deploy==1.6.0:
rba-deploy==1.6.0
The python code I'm trying to package is here:
My setup.py looks like this:
#!/usr/bin/env python
from distutils.core import setup
setup(name='rba-deploy',
version='v1.6.0',
description='blah',
author='Dude',
author_email='blah',
url='https://www.foo.bar.baz',
package_dir = {'':'lib'},
packages=['rba','rba/response']
)
Here is the pip list and pex version:
(env-rba-deploy)my_computer:env-rba-deploy my_user$ pex --version
pex 1.0.3
(env-rba-deploy)my_computer:env-rba-deploy my_user$ pip list
pip (7.1.2)
rba-deploy (1.6.0)
setuptools (18.2)
wheel (0.24.0)
What the heck am I missing?

The '-r' for requirements is going to try and download your package from pypi but if you haven't uploaded it then it won't work.
Instead of the pip freeze command just point at setup.py directory using a '.' or './'
pex -v --disable-cache -o foo.pex ./

Could not find a version that satisfies the requirement <package>

I'm installing several Python packages in Ubuntu 12.04 using the following requirements.txt file:
numpy>=1.8.2,<2.0.0
matplotlib>=1.3.1,<2.0.0
scipy>=0.14.0,<1.0.0
astroML>=0.2,<1.0
scikit-learn>=0.14.1,<1.0.0
rpy2>=2.4.3,<3.0.0
and these two commands:
$ pip install --download=/tmp -r requirements.txt
$ pip install --user --no-index --find-links=/tmp -r requirements.txt
(the first one downloads the packages and the second one installs them).
The process is frequently stopped with the error:
Could not find a version that satisfies the requirement <package> (from matplotlib<2.0.0,>=1.3.1->-r requirements.txt (line 2)) (from versions: )
No matching distribution found for <package> (from matplotlib<2.0.0,>=1.3.1->-r requirements.txt (line 2))
which I fix manually with:
pip install --user <package>
and then run the second pip install command again.
But that only works for that particular package. When I run the second pip install command again, the process is stopped now complaining about another required package and I need to repeat the process again, ie: install the new required package manually (with the command above) and then run the second pip install command.
So far I've had to manually install six, pytz, nose, and now it's complaining about needing mock.
Is there a way to tell pip to automatically install all needed dependencies so I don't have to do it manually one by one?
Add: This only happens in Ubuntu 12.04 BTW. In Ubuntu 14.04 the pip install commands applied on the requirements.txt file work without issues.

Although it doesn't really answers this specific question. Others got the same error message with this mistake.
For those who like me initial forgot the -r: Use pip install -r requirements.txt the -r is essential for the command.
The original answer:
https://stackoverflow.com/a/42876654/10093070

I had installed python3 but my python in /usr/bin/python was still the old 2.7 version
This worked (<pkg> was pyserial in my case):
python3 -m pip install <pkg>

This approach (having all dependencies in a directory and not downloading from an index) only works when the directory contains all packages. The directory should therefore contain all dependencies but also all packages that those dependencies depend on (e.g., six, pytz etc).
You should therefore manually include these in requirements.txt (so that the first step downloads them explicitly) or you should install all packages using PyPI and then pip freeze > requirements.txt to store the list of all packages needed.

Just a reminder to whom google this error and come here.
Let's say I get this error:
$ python3 example.py
Traceback (most recent call last):
File "example.py", line 7, in <module>
import aalib
ModuleNotFoundError: No module named 'aalib'
Since it mentions aalib, I was thought to try aalib:
$ python3.8 -m pip install aalib
ERROR: Could not find a version that satisfies the requirement aalib (from versions: none)
ERROR: No matching distribution found for aalib
But it actually wrong package name, ensure pip search(service disabled at the time of writing), or google, or search on pypi site to get the accurate package name:
Then install successfully:
$ python3.8 -m pip install python-aalib
Collecting python-aalib
Downloading python-aalib-0.3.2.tar.gz (14 kB)
...
As pip --help stated:
$ python3.8 -m pip --help
...
-v, --verbose Give more output. Option is additive, and can be used up to 3 times.
To have a systematic way to figure out the root causes instead of rely on luck, you can append -vvv option of pip command to see details, e.g.:
$ python3.8 -u -m pip install aalib -vvv
User install by explicit request
Created temporary directory: /tmp/pip-ephem-wheel-cache-b3ghm9eb
Created temporary directory: /tmp/pip-req-tracker-ygwnj94r
Initialized build tracking at /tmp/pip-req-tracker-ygwnj94r
Created build tracker: /tmp/pip-req-tracker-ygwnj94r
Entered build tracker: /tmp/pip-req-tracker-ygwnj94r
Created temporary directory: /tmp/pip-install-jfurrdbb
1 location(s) to search for versions of aalib:
* https://pypi.org/simple/aalib/
Fetching project page and analyzing links: https://pypi.org/simple/aalib/
Getting page https://pypi.org/simple/aalib/
Found index url https://pypi.org/simple
Getting credentials from keyring for https://pypi.org/simple
Getting credentials from keyring for pypi.org
Looking up "https://pypi.org/simple/aalib/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/aalib/ HTTP/1.1" 404 13
[hole] Status code 404 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/aalib/: 404 Client Error: Not Found for url: https://pypi.org/simple/aalib/ - skipping
Given no hashes to check 0 links for project 'aalib': discarding no candidates
ERROR: Could not find a version that satisfies the requirement aalib (from versions: none)
Cleaning up...
Removed build tracker: '/tmp/pip-req-tracker-ygwnj94r'
ERROR: No matching distribution found for aalib
Exception information:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/_internal/cli/base_command.py", line 186, in _main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/_internal/commands/install.py", line 357, in run
resolver.resolve(requirement_set)
File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
discovered_reqs.extend(self._resolve_one(requirement_set, req))
File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/usr/lib/python3/dist-packages/pip/_internal/legacy_resolve.py", line 281, in _get_abstract_dist_for
req.populate_link(self.finder, upgrade_allowed, require_hashes)
File "/usr/lib/python3/dist-packages/pip/_internal/req/req_install.py", line 249, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/usr/lib/python3/dist-packages/pip/_internal/index/package_finder.py", line 926, in find_requirement
raise DistributionNotFound(
pip._internal.exceptions.DistributionNotFound: No matching distribution found for aalib
From above log, there is pretty obvious the URL https://pypi.org/simple/aalib/ 404 not found. Then you can guess the possible reasons which cause that 404, i.e. wrong package name. Another thing is I can modify relevant python files of pip modules to further debug with above log. To edit .whl file, you can use wheel command to unpack and pack.

After 2 hours of searching, I found a way to fix it with just one line of command. You need to know the version of the package (Just search up PACKAGE version).
Command:
python3 -m pip install --pre --upgrade PACKAGE==VERSION.VERSION.VERSION

Below command worked for me -
python -m pip install flask

Not always, but in some cases the package already exists. For example - getpass. It is not listed by "pip list" but it can be imported and used:
If I try to pip install getpass I get the following error:
"Could not find a version that satisfies the requirement getpass"

Try installing flask through the powershell using the following command.
pip install --isolated Flask
This will allow installation to avoide environment variables and user configuration.

If you facing this issue at the workplace. This might be the solution for you.
pip install -U <package_name> --user --proxy=<your proxy>

Pip install from pypi.org.
pip install -U -i https://pypi.org/simple package

One possible error, pip package requires python intepreter which you are not using.
I ran into the same problem, it occurred only when I ran commands from my Docker image (or Dockerfile). Finally many hours later I managed to solve it by updating my python intepreter. Pointed out that my pip-package required python>=3,7 but my Docker image was using python 3.6.
Tip: To check out if you have similar problem, just check pip package requirements and your python version. Private pip package intepreter requirements are wrote down inside setup.py or setup.cfg. Public pip packages are usuially hosted in pypi.org where you can just check intepreter requirements with your browser. To check your python intepreter version just write for example python --version or python3 --version in your console
General problem description
As other answers point out there can also be other requirements that you are not satisfying and that is why pip can not found suitable package version for you. All the requirements are wrote down in pip package documentation and can be easily readed from https://pypi.org/project/graphene-django/your-package

I got this error while installing awscli on Windows 10 in anaconda (python 3.7).
While troubleshooting, I went to the answer https://stackoverflow.com/a/49991357/6862405 and then to https://stackoverflow.com/a/54582701/6862405. Finally found that I need to install the libraries PyOpenSSL, cryptography, enum34, idna and ipaddress. After installing these (using simply pip install command), I was able to install awscli.

When I lost my internet connection, I had this error.

Since it's a pretty annoying problem that may stuck beginners for a long period of time, here I write a complete guild.
if you are running pip install PACKAGE or python -m pip install PACKAGE, and a no matching version found error reported, here's how to solve the problem.
search your package on browser, for example my package is pycypto, here I search pycypto pypi
find your package, open the link on pypi, click download file
open a python shell, import any of your installed package, for example, I have installed Pillow before.
>>> import PIL
>>> PIL.__path__
['/Applications/MAMP/htdocs/canvas/src/zzd/env/lib/python3.7/site-packages/PIL']
PACKAGE.__path__ function will gives you the side packages path where all packages should go into.
PLUS:
if you have no idea what packages you installed before, run pip list to get a list of installed packages.
after we obtain the path, open a shell, cd to the path
cd /Applications/MAMP/htdocs/canvas/src/zzd/env/lib/python3.7/site-packages/
open
unzip the downloaded file, drag it into site-packages.
cd into the downloaded directory, and run setup.py to install
cd pycrypto-2.6.1
python setup.py install
Then you should be able to import and use the package in python.

Same error in slightly different circumstances, on MacOs. Apparently setuptools versions past 45 can expose some issues and this command got me past it:
pip3 install setuptools==45

If the package is local, don't miss the relative path.
E.g.
pip install ./<pkg>
finally worked in my case, while
pip install <pkg>
yielded:
ERROR: Could not find a version that satisfies the requirement <pkg> (from versions: none)
ERROR: No matching distribution found for <pkg>

I had a problem installing pandas-1.4.3, and the problem was my python patch version. pandas-1.4.3 required python version 3.8.13 and did not work with 3.8.9:
python install -r requirements.txt # or pip install pandas==1.4.3
# -> Could not find a version that satisfies...
conda activate my_project # creates a virtual env for a new python version
conda install python=3.8.13 # installing the new python version
python --version # displays 3.8.13
pip install -r python/requirements.txt
# -> pandas installed as expected

Search in google if you find some other version of that package available
use that for example
I was getting errors using the glob so I used glob2 instead

Pip freeze does not show repository paths for requirements file

I've created an environment and added a package django-paramfield via git:
$ pip install git+https://bitbucket.org/DataGreed/django-paramfield.git
Downloading/unpacking git+https://bitbucket.org/DataGreed/django-paramfield.git
Cloning https://bitbucket.org/DataGreed/django-paramfield.git to /var/folders/9Z/9ZQZ1Q3WGMOW+JguzcBKNU+++TI/-Tmp-/pip-49Eokm-build
Unpacking objects: 100% (29/29), done.
Running setup.py egg_info for package from git+https://bitbucket.org/DataGreed/django-paramfield.git
Installing collected packages: paramfield
Running setup.py install for paramfield
Successfully installed paramfield
Cleaning up...
But when i want to create a requirements file, i see only the package name:
$ pip freeze
paramfield==0.1
wsgiref==0.1.2
How can I make it output the whole string git+https://bitbucket.org/DataGreed/django-paramfield.git instead of just a package name? The package isn't in PyPi.
UPD: perhaps, it has to do something with setup.py? Should I change it somehow to reflect repo url?
UPD2: I found quite a similar question in stackoverflow, but the author was not sure how did he manage to resolve an issue and the accepted answer doesn't give a good hint unfortunately, though judging from the author's commentary it has something to do with the setup.py file.
UPD3: I've tried to pass download_url in setup.py and installing package via pip with this url, but he problem persists.

A simple but working workaround would be to install the package with the -e flag like pip install -e git+https://bitbucket.org/DataGreed/django-paramfield.git#egg=django-paramfield.
Then pip freeze shows the full source path of the package. It's not the best way it should be fixed in pip but it's working. The trade off -e (editing flag) is that pip clones the git/hg repo into /path/to/venv/src/packagename and run python setup.py deploy instead of clone it into a temp dir and run python setup.py install and remove the temp dir after the setup of the package.

Here's a script that will do that:
#!/usr/bin/env python
from subprocess import check_output
from pkg_resources import get_distribution
def download_url(package):
dist = get_distribution(package)
for line in dist._get_metadata('PKG-INFO'):
if line.startswith('Download-URL:'):
return line.split(':', 1)[1]
def main(argv=None):
import sys
from argparse import ArgumentParser
argv = argv or sys.argv
parser = ArgumentParser(
description='show download urls for installed packages')
parser.parse_args(argv[1:])
for package in check_output(['pip', 'freeze']).splitlines():
print('{}: {}'.format(package, download_url(package) or 'UNKNOWN'))
if __name__ == '__main__':
main()

This is an old question but I have just worked through this same issue and the resolution
Simply add the path to the repo (git in my case) to the requirements fie instead of the package name
i.e.
...
celery==3.0.19
# chunkdata isn't available on PyPi
https://github.com/aaronmccall/chunkdata/zipball/master
distribute==0.6.34
...
Worked like a charm deplying on heroku

Categories

Resources