I have a script that runs pip install -r requirements.txt -t folder and executes scripts in "folder" for me.
Every time I run it pip install runs, collects all the packages and installs them with setup.py. It looks like it's not making any changes for packages that are already there and up to date but it still takes a while to collect them and run setup.py for each one. Often I'm not modifying the packages.
How can I have it skip ones that are already there? is there a way to speed this up apart from just not running pip install every time?
You can apparently select the upgrade strategy like this:
python -m pip install --requirement requirements.txt --upgrade --upgrade-strategy=only-if-needed
Source: Ignoring some requirements when installing pip requirements
In my case it still doesn't completely do what I expect since dependencies of the updated packages will still be reinstalled, disregarding whether they are already installed and the version matches.
But it should do the trick for most cases.
You could take a look at :
-I, --ignore-installed Ignore the installed packages (reinstalling instead).
or
--upgrade-strategy Determines how dependency upgrading should be handled [default: only-if-needed]. "eager" -
dependencies are upgraded regardless of whether the currently
installed version satisfies the requirements of the upgraded
package(s). "only-if-needed" - are upgraded only when they do not
satisfy the requirements of the upgraded package(s).
Never get into this problem before, but
https://pip.pypa.io/en/stable/man/commands/install/?highlight=--ignore-installed
it might help you.
Or you can do a pre-setup script that make some check before.
Related
my requirements.txt:
google-api-python-client>=1.6.2
httplib2==0.9.2
...
In this case google-api-python-client installs the httplib2=0.14.0. But I would be overwritten to the older version on the next line.
Of course, in this case, I could act. But there are some cases that a lib depends on another, that depends on another, etc.
So I'm wondering whether there's a way to tell pip to always keep the most recent version of the libs.
like: pip install -r --keep-most-recent requirements.txt
This would mean that pip should install the latest version of the requirements and their dependencies no matter what the version constraints in requirements.txt are, is that right? Impossible as far as I know.
Not sure exactly what you are trying to achieve but maybe you would prefer working with one of these tools:
poetry update
pip-compile --upgrade
pipenv update
When I run pip freeze I see (among other expected packages) pkg-resources==0.0.0. I have seen a few posts mentioning this package (including this one), but none explaining what it is, or why it is included in the output of pip freeze. The main reason I am wondering is out of curiosity, but also, it seems to break things in some cases when trying to install packages with a requirements.txt file generated with pip freeze that includes the pkg-resources==0.0.0 line (for example when Travis CI tries to install dependencies through pip and finds this line).
What is pkg-resources, and is it OK to remove this line from requirements.txt?
Update:
I have found that this line only seems to exist in the output of pip freeze when I am in a virtualenv. I am still not sure what it is or what it does, but I will investigate further knowing that it is likely related to virtualenv.
According to https://github.com/pypa/pip/issues/4022, this is a bug resulting from Ubuntu providing incorrect metadata to pip. So, no there does not seem to be a good reason for this behaviour. I filed a follow-up bug with Ubuntu. https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463
To backup the previous answer, it should be safe to remove that line from your requirements.txt. Here is an example Make file stanza that safely freezes your package list (drop in your Makefile and run with make freeze):
freeze:
pip freeze | grep -v "pkg-resources" > requirements.txt
edit 2022 July 06:
I have been informed that the package name differs depending on the system in use (pkg-resources vs pkg_resources). Please see the comments attached to this answer for differences in usage between different versions of Debian/Ubuntu. As pkg-resources is the historically correct package name at the time this was posted (almost 6 years ago) for the system in question, it will remain unchanged in this answer.
As for the part of your question "is it OK to remove this line?":
I have the same issue here developing on an ubuntu 16.04 with that very line in the requirements. When deploying on a debian 8.5 running "pip install -r requirements.txt" pip complains that pkg-resources is "not found" but there is a global package installed "python-pkg-resources" so the dependency should be satisfied. Same on ubuntu: The package exists there as well.
As stated here it seems to be some "implicitly installed package".
So: If you are on a Debian/Ubuntu having python-pkg-resources installed it should be safe to remove that line. I did so and everything is running fine. However since I am no expert on this you should keep in mind that this might lead to complications when deploying on another machine.
found this answer in this link: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463
by: Louis Bouchard (louis) wrote on 2019-11-16:
It worked for me. But Iḿ not an expert so, if someone undestend it better, would be great if explained it.
Hello,
for what it's worth, the problem comes from the debianized version of virtualenv which uses a debundled version of pkg_resource which gets added into the virtualenv at creation time :
$ virtualenv .
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/caribou/git/quividi/test/bin/python2
Also creating executable in /home/caribou/git/quividi/test/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
$ pip freeze
pkg-resources==0.0.0
Using the pip installed version of virtualenv can be a workable workaround :
$ sudo apt -y purge python3-virtualenv virtualenv tox
$ pip install virtualenv
$ virtualenv .
pip install virtualenv
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/c5/97/00dd42a0fc41e9016b23f07ec7f657f636cb672fad9cf72b80f8f65c6a46/virtualenv-16.7.7-py2.py3-none-any.whl (3.4MB)
100% |████████████████████████████████| 3.4MB 351kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-16.7.7
$ virtualenv .
New python executable in /home/caribou/git/quividi/test/bin/python
Installing setuptools, pip, wheel...
done.
$ source bin/activate
$ pip freeze
$
When I install anything with setup.py or with pip on OS X, it is installed only into an older Python version. What is the best way to solve this?
Here is written that a solution is to modify the first line of /usr/local/share/python/pip, but I do not know how to access that file.
Try: /path/to/your/python -m pip install {yourlib}
Long answer:
For more information on what's going on, you can run which python or which pip. This shows you exactly which python/pip is being run. Even if you install another pip for another python, running the new pip may not install the package on the new python. If you look at the output from pip, you may see something like:
$ pip install pyodbc
Requirement already satisfied (use --upgrade to upgrade): pyodbc in /some/other/python/path/pyodbc-3.0.7-py2.7-linux-x86_64.egg
Cleaning up...
If your previous python path (echo $PYTHONPATH) can be seen by your new pip, or if it's linked in ~/.local, then pip will find the package and think it's already installed. If you uninstall the package on the old version, then the reinstall should install to the right location if you're using the right pip.
Response to follow-up question in comments - When pip isn't found by bash:
There are quite a few solutions:
The above will work
If you want to be able to type pip, then putting alias pip='/path/to/your/python -m pip' or simply alias pip='/path/to/your/new/pip' will work
Uninstalling and reinstalling the new python/pip package should put pip in your default namespace. Likely your old pip was around when you first installed, so it didn't overwrite.
Play with your $PATH. The most complicated, but probably the cleanest answer. I have a ~/bin/ in my path, to which I add symlinks for anything I want in my standard path.
pip install --upgrade -r requirements.txt
repeats the installation process for all the previously installed dependencies which can be a pain when I have a huge list of dependencies (like more than 30?)
Isn't there a way to check an updated requirements.txt and install only specific dependencies that have been included into the requirements.txt file since the previous installation attempt?
I find this to be a real shortcoming of pip (or using pip in virtualenv for that matter). Do not like the repetitive installation nature of pip at all.
As mentioned by Piotr in the comments above, if "--upgrade" is not included in the command, already installed python packages are left alone.
This Python package install using pip or easy_install from repos points out a very interesting features of pip.
However, sometimes you just want it to install the source distribution; this is particularly true when
you are running in a virtualenv (so you don't care about messing up the python path, since you are deliberating doing it in an env),
when you are not the developer of that particular package, and you don't want to have it "editable",
when you cannot pip install package-name because the package is not in any index,
when there is no tar.gz available.
Thanks for your answers!
Have you tried just omitting the --editable? If I run
pip install hg+http://bitbucket.org/carljm/django-markitup/
it clones the repo to a temporary build directory and installs normally (via setup.py install rather than setup.py develop).
Of course, if you then freeze this environment, the generated requirement will not be fulfillable. If you need this, then just use --editable (there's really not much difference, works fine even if you don't actually need to edit the package) or just run your own instance of something like chishop and upload the sdists you need to it, then use the -i or --extra-index-url option.