In my tests, I would like to run a command to make sure that the installed packages in my virtual environment match the packages found in Pipfile.lock.
Is there a command like this?
pipenv checkifinstalled || exit 1
This problem can be reduced down to these two steps:
Convert Pipfile.lock into a requirements.txt file (in the format generated by pip freeze).
This is easily done by running pipenv requirements (or pipenv requirements --dev). (Note that this command is supported in pipenv >= 2022.4.8)
Check that the installed packages match the generated requirements.txt file.
Solutions to this are found under this question: Check if my Python has all required packages
Implementation:
Here is how I put it all together in a test:
import pkg_resources
import subprocess
import unittest
class DependencyChecks(unittest.TestCase):
def test_requirements_installed(self):
requirements_lines = subprocess.check_output(["pipenv", "requirements", "--dev"], text=True).splitlines()
req_lines = [line for line in requirements_lines if not line.startswith("-i ")]
requirements = pkg_resources.parse_requirements(req_lines)
for requirement in requirements:
req = str(requirement)
with self.subTest(requirement=req):
pkg_resources.require(req)
Running pip install seems to create the directory structure + files specified in data_files in /usr/local
However, if I run:
import sys
sys.prefix
I get the string /usr.
Is there any way to figure out where pip installed the data_files for a specific package in a distribution/OS agnostic way ?
Note: I am installing a package from a github repostiroy instead of pypi so maybe this results in the different behavior ?
I believe you should work with sysconfig.
First try:
path/to/pythonX.Y -m sysconfig
And then try its get_path function:
import sysconfig
data_path_str = sysconfig.get_path('data')
print("data_path_str", data_path_str)
Please before flagging as duplicate, I have tried a bunch of solutions
including one here
but no luck
I have created a simple tool to do some tasks and was able to package it successfully.
When trying to install it, I get the desired effect when I use python setup.py install but pip install package_name just installs the package but no post installation script.
Here is part of my code;
setup.py
from distutils import setup
from app.scripts import *
setup(
#Application name
name = "my-app-name",
version = "my-app-version",
author = "my-name",
author_email = "my-email",
packages = ['app'],
include_package_data = True,
license = 'MIT',
url = "https://my-url",
description = "description",
install_requires = ["flake8"],
cmdclass = {
"install":Post_install
}
)
scripts.py
from distutils.command.install import install
import os
class Post_install(install):
#staticmethod
def func():
return True
def run(self):
install.run(self)
#Pre install actions
if Post_install.func():
print("Bingo")
else:
print("Failed")
Thanks :)
PS I run pip install after uploading the package.
Install the package directly from your GitHub repository:
pip install -vvv git+url/for/github/repo#my-branch
You mentioned in the chat that you'd like to add this package to your requirements.txt file. See this question for details:
-e git://github.com/path/to/project
Former answer (rejected by the OP):
I managed to recreate the issue you're having. It seems to be a matter of pip install silencing or redirecting output (as indicated by an answer to this question).
The solution is to add the option -vvv after pip install. I'm guessing the v stands for verbose.
I am trying to install a Python package and I get a dependency error but I am sure I have fulfilled that requirement.
It says that it can't find libdickinson.so, but this library is already installed (system wide) and its files are in /user/local/lib/. What am I doing wrong?
This is my console output:
(iwidget)chris#mint-desktop ~ $ pip install pthelma
Downloading/unpacking pthelma
Downloading pthelma-0.7.2.tar.gz (50kB): 50kB downloaded
Running setup.py egg_info for package pthelma
libdickinson.so: cannot open shared object file: No such file or directory
Please make sure you have installed dickinson
(see http://dickinson.readthedocs.org/).
Complete output from command python setup.py egg_info:
libdickinson.so: cannot open shared object file: No such file or directory
Please make sure you have installed dickinson
(see http://dickinson.readthedocs.org/).
----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/chris/.virtualenvs/iwidget/build/pthelma
Storing complete log in /home/chris/.pip/pip.log
(iwidget)chris#mint-desktop ~ $ ls /usr/local/lib/
libdickinson.a libdickinson.la libdickinson.so libdickinson.so.0 libdickinson.so.0.0.0 python2.7/ python3.2/ site_ruby/
(iwidget)chris#mint-desktop ~ $
Also try the above command as superuser:
sudo pip install pthelma
and just go through the thread given below:
Why can't Python find shared objects that are in directories in sys.path?
Try building it yourself and installing from the GIT repo:
git clone https://github.com/openmeteo/pthelma.git
Also, try running it as super user (pip).
sudo pip install pthelma
It looks like it can't see the libdickinson.so file but if you're confident it's installed and setup correctly you can, as I said, try cloning the source and building it that way.
When I do a pip freeze I see large number of Python packages that I didn't explicitly install, e.g.
$ pip freeze
Cheetah==2.4.3
GnuPGInterface==0.3.2
Landscape-Client==11.01
M2Crypto==0.20.1
PAM==0.4.2
PIL==1.1.7
PyYAML==3.09
Twisted-Core==10.2.0
Twisted-Web==10.2.0
(etc.)
Is there a way for me to determine why pip installed these particular dependent packages? In other words, how do I determine the parent package that had these packages as dependencies?
For example, I might want to use Twisted and I don't want to depend on a package until I know more about not accidentally uninstalling it or upgrading it.
You could try pipdeptree which displays dependencies as a tree structure e.g.:
$ pipdeptree
Lookupy==0.1
wsgiref==0.1.2
argparse==1.2.1
psycopg2==2.5.2
Flask-Script==0.6.6
- Flask [installed: 0.10.1]
- Werkzeug [required: >=0.7, installed: 0.9.4]
- Jinja2 [required: >=2.4, installed: 2.7.2]
- MarkupSafe [installed: 0.18]
- itsdangerous [required: >=0.21, installed: 0.23]
alembic==0.6.2
- SQLAlchemy [required: >=0.7.3, installed: 0.9.1]
- Mako [installed: 0.9.1]
- MarkupSafe [required: >=0.9.2, installed: 0.18]
ipython==2.0.0
slugify==0.0.1
redis==2.9.1
To get it run:
pip install pipdeptree
EDIT: as noted by #Esteban in the comments you can also list the tree in reverse with -r or for a single package with -p <package_name> so to find what installed Werkzeug you could run:
$ pipdeptree -r -p Werkzeug
Werkzeug==0.11.15
- Flask==0.12 [requires: Werkzeug>=0.7]
The pip show command will show what packages are required for the specified package (note that the specified package must already be installed):
$ pip show specloud
Package: specloud
Version: 0.4.4
Requires:
nose
figleaf
pinocchio
pip show was introduced in pip version 1.4rc5
As I recently said on a hn thread, I'll recommend the following:
Have a commented requirements.txt file with your main dependencies:
## this is needed for whatever reason
package1
Install your dependencies: pip install -r requirements.txt.
Now you get the full list of your dependencies with pip freeze -r requirements.txt:
## this is needed for whatever reason
package1==1.2.3
## The following requirements were added by pip --freeze:
package1-dependency1==1.2.3
package1-dependency1==1.2.3
This allows you to keep your file structure with comments, nicely separating your dependencies from the dependencies of your dependencies. This way you'll have a much nicer time the day you need to remove one of them :)
Note the following:
You can have a clean requirements.raw with version control to rebuild your full requirements.txt.
Beware of git urls being replaced by egg names in the process.
The dependencies of your dependencies are still alphabetically sorted so you don't directly know which one was required by which package but at this point you don't really need it.
Use pip install --no-install <package_name> to list specific requirements.
Use virtualenv if you don't.
You may also use a one line command which pipes the packages in requirements to pip show.
cut -d'=' -f1 requirements.txt | xargs pip show
The following command will show requirements of all installed packages:
pip3 freeze | awk '{print $1}' | cut -d '=' -f1 | xargs pip3 show
First of all pip freeze displays all currently installed packages Python, not necessarily using PIP.
Secondly Python packages do contain the information about dependent packages as well as required versions. You can see the dependencies of particular pkg using the methods described here. When you're upgrading a package the installer script like PIP will handle the upgrade of dependencies for you.
To solve updating of packages i recommend using PIP requirements files. You can define what packages and versions you need, and install them at once using pip install.
(workaround, not true answer)
Had the same problem, with lxml not installing and me wanting to know who needed lxml. Not who lxml needed. Ended up bypassing the issue by.
noting where my site packages were being put.
go there and recursive grep for the import (the last grep's --invert-match serves to remove lxml's own files from consideration).
Yes, not an answer as to how to use pip to do it, but I didn't get any success out of the suggestions here, for whatever reason.
site-packages me$ egrep -i --include=*.py -r -n lxml . | grep import | grep --invert-match /lxml/
I wrote a quick script to solve this problem. The following script will display the parent (dependant) package(s) for any given package. This way you can be sure it is safe to upgrade or install any particular package. It can be used as follows: dependants.py PACKAGENAME
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""Find dependants of a Python package"""
import logging
import pip
import pkg_resources
import sys
__program__ = 'dependants.py'
def get_dependants(target_name):
for package in pip._internal.utils.misc.get_installed_distributions():
for requirement_package in package.requires():
requirement_name = requirement_package.project_name
if requirement_name == target_name:
yield package.project_name
# configure logging
logging.basicConfig(format='%(levelname)s: %(message)s',
level=logging.INFO)
try:
target_name = sys.argv[1]
except IndexError:
logging.error('missing package name')
sys.exit(1)
try:
pkg_resources.get_distribution(target_name)
except pkg_resources.DistributionNotFound:
logging.error("'%s' is not a valid package", target_name)
sys.exit(1)
print(list(get_dependants(target_name)))
You have two options here.
The first will output all top-level packages, excluding sub packages. Note that this will also exclude for example requests, even if you want to have it explicitly installed
pip3 list --not-required --format freeze --exclude pip --exclude setuptools
The second option is to print the packages based on the existing requirements.txt file.
pip3 freeze -r requirements.txt
This will generate a file in the format:
existing-package==1.0.0
## The following requirements were added by pip freeze:
dependency-package==1.0.0
You can remove all the additionally added packages by using sed:
pip3 freeze -r requirements.txt | sed -n '/## The following requirements were added by pip freeze:/q;p'
With GraphVis as seen on tv
If you like graphs you can use graphviz (Documentation)
pip install graphviz
Then do something like this:
#! /usr/bin/env python3
import graphviz
import pkg_resources
GRAPH_NAME = "pipdeps"
def init_grph():
grph = graphviz.Digraph(GRAPH_NAME,
node_attr={'color': 'lightblue2', 'style': 'filled'})
# This does not seem to be interpreted on websites I tested
grph.attr(engine='neato')
return grph
def add_pip_dependencies_to_graph(grph):
l_packages = [p for p in pkg_resources.working_set]
for package in l_packages:
name = package.key
for dep in package.requires():
grph.edge(dep.key, name)
def main():
grph = init_grph()
add_pip_dependencies_to_graph(grph)
print(grph.source)
# grph.view()
main()
This prints a dotgraph/digraph/graphviz (idk 🤷♀️)
You can view it on an online graphviz visualiser (e.g. )
If you have a graphic interface (😮) you can use grph.view()
hth