When I use pip to install a package from source, it will generates a version number for the package which I can see using 'pip show '. But I can't find out how that version number is generated and I can't find the version string from the source code. Can someone tell me how the version is generated?
The version number that pip uses comes from the setup.py (if you pip install a file, directory, repo, etc.) and/or the information in the PyPI index (if you pip install a package name). (Since these two must be identical, it doesn't really matter which.)
It's recommended that packages make the same string available as a __version__ attribute on their top-level module/package(s) at runtime that they put in their setup, but that isn't required, and not every package does.
And if the package doesn't expose its version, there's really no way for you to get it. (Well, unless you want to grub through the pip data trying to figure out which package owns a module and then get its version.)
Here's an example:
In the source code for bs4 (BeautifulSoup4), the setup.py file has this line:
version = "4.3.2",
That's the version that's used, directly or indirectly, by pip.
Then, inside bs4/__init__.py, there's this line:
__version__ = "4.3.2"
That means that Leonard Richardson is a nice guy who follows the recommendations, so I can import bs4; print(bs4.__version__) and get back the same version string that pip show beautifulsoup4 gives me.
But, as you can see, they're two completely different strings in completely different files. If he wasn't nice, they could be totally different, or the second one could be missing, or named something different.
The OpenStack people came up with a nifty library named PBR that helps you manage version numbers. You can read the linked doc page for the full details, but the basic idea is that it either generates the whole version number for you out of git, or verifies your specified version number (in the metadata section of setup.cfg) and appends the dev build number out of git. (This relies on you using Semantic Versioning in your git repo.)
Instead of specifying the version number in code, tools such as setuptools-scm may use tags from version control. Sometimes the magic is not directly visible. For example PyScaffold uses it, but in the project's root folder's __init__.py one may just see:
import pkg_resources
try:
__version__ = pkg_resources.get_distribution(__name__).version
except:
__version__ = "unknown"
If, for example, the highest version tag in Git is 6.10.0, then pip install -e . will generate a local version number such as 6.10.0.post0.dev23+ngc376c3c (c376c3c being the short hash of the last commit) or 6.10.0.post0.dev23+ngc376c3c.dirty (if it has uncommitted changes).
For more complicated strings such as 4.0.0rc1, they are usually hand edited in the PKG-INFO file. Such as:
# cat ./<package-name>.egg-info/PKG-INFO
...
Version: 4.0.0rc1
...
This make it unfeasible to obtain it from within any python code.
Related
I am creating a module, henceforth called mymodule, which I distribute using a pyproject.toml. This file contains a version number. I would like to write this version number in the logfile of mymodule. In mymodule I use the following snippet (in __init__.py) to obtain the version:
import importlib.metadata
__version__ = importlib.metadata.version(__package__)
del importlib.metadata
However this version is wrong. This appears to be the highest version which I have ever installed. For reference the command python3 -m pip show mypackage does actually show the correct version after installing the module locally. I struggle to explain this difference. Can anyone think of a cause of this discrepancy?
I also ran importlib.metadata.version(mypackage) which returned the same incorrect version.
The problem was related to left over build artifacts from using setup.py. importlib and pkg_resources will detect these artifacts in a local installation and pip will not. Deleting the mypackage.egg-info directory fixed the issue.
In my local pypi server let's say I have 3 versions of the package example like below:
example==20200903
example==20200904
example==202009089 # I need to exclude this
example==20200909
As you can see I have used date to manage our versioning, but in the middle of the versioning we have a package that has a version like 202009089 so it always match as it has a bigger number and the versioning gets broken. Is there a way to exclude that specific version when installing via pip install and install the latest version except 202009089?
One approach would be to number future versions using a new epoch (PEP440)
For example
version='1!20200910
another option is to delete the offending package from your internal pypi
another option is to select example!=202009089 (the bad version) or pin using example==... (some good version)
You can:
pip install "example<202000000"
It will pick the last version before the erroneous one: 202009089.
Related answer: https://stackoverflow.com/a/8811418/4709400
pytest appears to be using old source code and failing tests because of it. I'm not sure how to update it.
Test code:
from nba_stats import league
class TestLeaders():
def test_default():
leaders = league.Leaders()
print(leaders)
Source code (league.py):
from nba_stats.nba_api import NbaAPI
from nba_stats import constants
class Leaders:
...
When I run pytest on my parent directory, I get an error that refers to an old import statement.
_____________________________ ERROR collecting test/test_league.py ______________________________
ImportError while importing test module '/home/mfb/src/nba_stats/test/test_league.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_league.py:1: in <module>
from nba_stats import league
../../../.virtualenvs/nba_stats_dev/lib/python3.6/site-packages/nba_stats/league.py:1: in <module>
from nba_stats import _api_scrape, _get_json
E ImportError: cannot import name '_api_scrape'
I tried resetting my virtualenvironment and also reinstalling my package via pip. What do I need to do to tell it to see the new import statement and why is this happening?
Edit: Deleting my virtual environment completely and then creating a new one seemed to fix it, but it seems to be a recurring issue with any further source code changes. Surely there must be a way to not have to reset my virtualenvironment each time?
Looks like you installed that package (possibly as a dependency through something else if not directly) and also have it cloned locally for development. You can look into local editable installs (https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs), but personally, I prefer to make the test refer directly to the package under which it is being run, since then it can be used "as-is" after cloning it. Do this by modifying sys.path in your test_league.py. Ie., assuming it has a structure with the python code under python/nba_stats, in the parent directory of `test
sys.path = [os.path.join(os.pardir, 'python')] + sys.path
at the top of test_league.py. This puts your local package up front and import will consider it first.
EDIT:
Since you tried and it still did not work (please do make sure that the snippet above does point to the local python package as in the actual structure; the above is just a common one but you may have a different structure), here is how you can see which directories are considered in order, and which are eventually selected:
python -vv -m pytest -svx
You will be able to see if there are spurious directories in sys.path, whether the one tried (as in the snippet above) matches as expected or not, any left-over .pyc files that get picked up, etc.
EDITv2: Since you stated that python -m pytest works, but pytest not, have a look where that pytest executable is coming from with which pytest. Likely it's a system one that refers to a different python then the one in your virtualenv. To see which python it picks up, do:
cat `which python`
and look at the top line.
If that is not the same as what which python gives you (with your desired virtualenv activated), you may have to install pytest for that current virtualenv (python -m pip install pytest).
The Problem
I am working on a project that uses a package in beta with multiple versions (package name: psychxr). After some confusing error messages about missing modules, I have discovered that depending on where I source my installation from I get different package contents.
If I use pip to install psychxr, I get an ovr sub-package. However, if I install from source (via official github repository), I get a libovr sub-package. Is there a way I can rename the source package such that I can get both modules? Alternatively, is there a better way to go about this? Although the packages complete roughly the same task, their implementations are noticeably different, and I'd like access to both.
CMD output of >>>python -c help('psychxr')
Version 1 (OVR)
NAME
psychxr
PACKAGE CONTENTS
ovr (package)
VERSION
0.1.4
Version 2 (LIBOVR)
NAME
psychxr
PACKAGE CONTENTS
libovr (package)
VERSION
0.2.0
Post Script: I do apologize for any misuse of terminology, or illegibility. I'm fairly new to both python and cmd in windows.
Some packages are imported with a string which is different from the name of the package on PyPI, e.g.:
$ pip list | grep -i "yaml\|qt"
PyYAML 3.13
QtPy 1.5.2
pyyaml (pip instal pyyaml), but import yaml
qtpy (pip install qtpy), yes import is qtpy but package is QtPy
Several tools can't not handle that, e.g sphinx:
$ make html
WARNING: autodoc: failed to import module 'wireshark' from module 'logcollector.plugins'; the following exception was raised:
No module named 'qtpy'
I don't remember it right now, but same is for tools which scan the requirements.txt file and print warnings that the yaml package isn't installed (but it is and its name is pyyaml).
There are multiple reasons why authors choose to use different names in different environments:
Drop-in replacements: Sometimes it is helpful when you can install a fork and keep the rest of your code the same. I guess the most famous example is pyyaml / yaml. I did it when I created propy3 which can be used as a drop-in replacement for propy. I would say that this is also what happened with pillow.
Convenience: beautifulsoup4 can be imported as bs4 (+ package parking for bs4)
Lost credentials: I don't know of an example where the import name was changed as well, but I think for flask-restx the package name and the import name were changed.
A word of caution
As Ziyad Edher has pointed out in a related discussion, typosquatting is an issue on PyPI (source). If you add packages with different names, this gets more likely.
Other examples
Name in the docs vs "import" package name vs pypi package name vs anaconda packages vs Debian:
scikit-learn vs sklearn vs scikit-learn vs scikit-learn vs python-sklearn and python3-sklearn
OpenCV-Pyton vs cv2 vs opencv-python vs py-opencv vs python-opencv
PyTables vs tables vs tables vs pytables vs python-tables
Because these two concepts are not really related.
One is a python concept of package/module names, the other one a package manager concept.
Look at a simple packaging command with zip:
zip -r MyCoolTool.zip tool.py
The Tool is named tool, which probably is not unique and if you do not know that its MyCoolTool you do not know which tool it is. When I upload it somewhere I name it MyCoolTool, so you now a more unique name, that may be a bit more descriptive.
The other point is, that a pip package may include more modules than just one. PyYAML could for example include a second python module yaml2xml in addtion to yaml.
Finally there can be several implementations. PyYAML sounds like a pure python implementation. Now assume you need a really fast parser, then you may program CYAML with a C-backend, but the same interface at the name yaml.
In case of sphinx you can mock 3rd party packages with: autodoc_mock_imports