Develop Reference

Pipenv package hash does not match lock file

We have a lock file which has not changed since April 2021. Recently, we have started seeing the following error on pipenv install --deploy:
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
gunicorn==20.1.0 from https://files.pythonhosted.org/packages/e4/dd/5b190393e6066286773a67dfcc2f9492058e9b57c4867a95f1ba5caf0a83/gunicorn-20.1.0-py3-none-any.whl (from -r /tmp/pipenv-g7_1pdnq-requirements/pipenv-d64a8p6k-hashed-reqs.txt (line 32)):
Expected sha256 e0a968b5ba15f8a328fdfd7ab1fcb5af4470c28aaf7e55df02a99bc13138e6e8
Got 9dcc4547dbb1cb284accfb15ab5667a0e5d1881cc443e0677b4882a4067a807e
We have opened an issue in the project GitHub https://github.com/benoitc/gunicorn/issues/2889
We believe that it would be unsafe to use this new version without confirmation it is correct and safe in case someone has maliciously updated the package in the package repository.
Is there a way we can grab the wheel file from a previous docker build and force that to be used for the time being so we can safely build with the existing version and checksum?
Thanks

Thanks to #Ouroborus for the answer:
e0... is for the .tar.gz (source) package, 9d... is for the .whl package. (See the "view hashes" links on PyPI's gunicorn files page) I'm not sure why your systems are choosing to download the wheel now when they downloaded the source previously. However, those are both valid hashes for that module and version.

Is there a way to exclude a specific version when installing a package from pypi using pip?

In my local pypi server let's say I have 3 versions of the package example like below:
example==20200903
example==20200904
example==202009089 # I need to exclude this
example==20200909
As you can see I have used date to manage our versioning, but in the middle of the versioning we have a package that has a version like 202009089 so it always match as it has a bigger number and the versioning gets broken. Is there a way to exclude that specific version when installing via pip install and install the latest version except 202009089?

One approach would be to number future versions using a new epoch (PEP440)
For example
version='1!20200910
another option is to delete the offending package from your internal pypi
another option is to select example!=202009089 (the bad version) or pin using example==... (some good version)

You can:
pip install "example<202000000"
It will pick the last version before the erroneous one: 202009089.
Related answer: https://stackoverflow.com/a/8811418/4709400

Increase Version number if Travis at github was successful

I wrote a simple script in Python.
Now I would like travis to check my code. After travis was successful, the version number should get increased.
Up to now my script has no version number yet. I can store it anywhere where it makes sense for the auto-increment workflow.
How to do this for Python code?
Update
It works now:
run tests
bumpversion
push tag to master
Unfortunately travis does not support "after-all". This means if I want to run the tests for several Python versions, I have no way to bumpversion after the tests of all python versions were successful.
In my case I will check on Python2.7 only until travis resolved this issue: https://github.com/travis-ci/travis-ci/issues/929
Here is my simple script: https://github.com/guettli/compare-with-remote
Solved :-)
It works now:
Developer pushes to github
Travis-CI runs
If all tests are successful bumpversion increases the version
The new version in setup.py get's pushed to the github repo
A new release of the python package gets uploaded to pypi with the tool twine.
I explain the way I do CI with github, travis and pypi here: https://github.com/guettli/github-travis-bumpversion-pypi

If you accept having extra commit for your versioning, you could add this script in continuous_integration/increment_version.py
import os
import pkg_resources
if __name__ == "__main__":
version = pkg_resources.get_distribution("compare_with_remote").version
split_version = version.split('.')
try:
split_version[-1] = str(int(split_version[-1]) + 1)
except ValueError:
# do something about the letters in the last field of version
pass
new_version = '.'.join(split_version)
os.system("sed -i \"s/version='[0-9.]\+'/version='{}'/\" setup.py"
.format(new_version))
os.system("git add -u")
os.system("git commit -m '[ci skip] Increase version to {}'"
.format(new_version))
os.system("git push")
And change your .travis.yml to
after_success:
- python continuous_integration/increment_version.py
I am not sure about how to make the git push part work as it need some testing with the repo rights but I assume that you could probably setup something to allow travis to push in your repo. you could look into that post for instance.
Also note that I used python to perform the git operations but they can be added as extra line in the after_success field:
after_success:
- python continuous_integration/increment_version.py
- git add -u
- git commit -m "[ci skip] version changed"
- git push
I just find it convenient to put the version number in the commit msg.
Also, it is very important to add [ci skip] in the commit message to avoid infinite increment. Maybe it would be safer to trigger the version change on a specific commit msg tag.

Not Python-specific, but this tutorial explains auto-incrementing version numbers by adding .travis.yaml entries which update git tags with each successful build. It seems like a good balance between manual and auto-versioning.
While the tutorial does use npm's package.json for the initial version-check, you could implement a simple equivalent (in Python or otherwise).

Assuming that,
only commits with successful Travis CI builds get merged into the master branch (e.g. using Pull Requests)
the package is always installed with pip from a git repo using, for instance,
pip install git+https://github.com/user/package.git
For auto-incrementing version number, one could simply define the version as the number of commits in the master branch. This can be done with the following lines in setup.py,
minor_version = check_output(['git', 'rev-list',
'--count', 'master']).decode('latin-1').strip()
setup(version='0.{}'.format(minor_version), ...)

Deploy Ablog to github pages

I've just started using the ABlog plugin for sphinx to create a static-site blog.
Is it easy to change ablog deploy to deploy to a different location,
e.g. ../username.github.io/ instead of ./username.github.io/?
I have my ABlog project under source control in a git repository. Creating my username.github.io inside the current ABlog project creates a repo inside a repo and this causes errors (also I don't want to store the built site along with the ABlog repository -- although I could add a .gitignore).

Is it easy to change ablog deploy to deploy to a different location,
e.g. ../username.github.io/ instead of ./username.github.io/?
For ABlog ≥ 0.8.0, yes
For ablog-0.8.0 and above, you can use the -p option to specify a github repo location other than the default (<location of conf.py>/<your username>.github.io):
ablog deploy -p /the/path/for/your/local/github/pages/repo
i.e., in your case
ablog deploy -p ../username.github.io/
How to install the most recent ABlog version
Until version 0.8.0 is available on pypi, you can tell pip to install ablog directly from git:
pip install git+https://github.com/abakan/ablog.git
For Ablog < 0.8.0, no
For versions prior to 0.8.0, the old version of this answer applies:
With the current implementation of ABlog-internal function
ablog_deploy,
the location of the target repository cannot be changed:
String gitdir (holding the path where the local repository will be
created) is set
to
<confdir>/<github_pages option>.github.io but the `github_pages` option is also [used to choose the remote
repository](https://github.com/abakan/ablog/blob/0ed765d95a23ad7dce48c755773ac60dd08cf319/ablog/commands.py#L338),
so passing something else than the GitHub account name will make the
process fail.
Manipulating confdir would be difficult and would result in the
configuration
file
not being found and probably a bunch of other side effects.
However, if you're willing to modify ABlog's source code, it would not
be hard to adapt the assignment of gitdir as you see fit (maybe
introducing another option) to produce the decided effect. (E.g., make
it use confdir if your new option hasn't been set, and have it use
your new option instead if that option has been set.)

How pip determine a python package version

When I use pip to install a package from source, it will generates a version number for the package which I can see using 'pip show '. But I can't find out how that version number is generated and I can't find the version string from the source code. Can someone tell me how the version is generated?

The version number that pip uses comes from the setup.py (if you pip install a file, directory, repo, etc.) and/or the information in the PyPI index (if you pip install a package name). (Since these two must be identical, it doesn't really matter which.)
It's recommended that packages make the same string available as a __version__ attribute on their top-level module/package(s) at runtime that they put in their setup, but that isn't required, and not every package does.
And if the package doesn't expose its version, there's really no way for you to get it. (Well, unless you want to grub through the pip data trying to figure out which package owns a module and then get its version.)
Here's an example:
In the source code for bs4 (BeautifulSoup4), the setup.py file has this line:
version = "4.3.2",
That's the version that's used, directly or indirectly, by pip.
Then, inside bs4/__init__.py, there's this line:
__version__ = "4.3.2"
That means that Leonard Richardson is a nice guy who follows the recommendations, so I can import bs4; print(bs4.__version__) and get back the same version string that pip show beautifulsoup4 gives me.
But, as you can see, they're two completely different strings in completely different files. If he wasn't nice, they could be totally different, or the second one could be missing, or named something different.
The OpenStack people came up with a nifty library named PBR that helps you manage version numbers. You can read the linked doc page for the full details, but the basic idea is that it either generates the whole version number for you out of git, or verifies your specified version number (in the metadata section of setup.cfg) and appends the dev build number out of git. (This relies on you using Semantic Versioning in your git repo.)

Instead of specifying the version number in code, tools such as setuptools-scm may use tags from version control. Sometimes the magic is not directly visible. For example PyScaffold uses it, but in the project's root folder's __init__.py one may just see:
import pkg_resources
try:
__version__ = pkg_resources.get_distribution(__name__).version
except:
__version__ = "unknown"
If, for example, the highest version tag in Git is 6.10.0, then pip install -e . will generate a local version number such as 6.10.0.post0.dev23+ngc376c3c (c376c3c being the short hash of the last commit) or 6.10.0.post0.dev23+ngc376c3c.dirty (if it has uncommitted changes).

For more complicated strings such as 4.0.0rc1, they are usually hand edited in the PKG-INFO file. Such as:
# cat ./<package-name>.egg-info/PKG-INFO
...
Version: 4.0.0rc1
...
This make it unfeasible to obtain it from within any python code.