I am creating a module, henceforth called mymodule, which I distribute using a pyproject.toml. This file contains a version number. I would like to write this version number in the logfile of mymodule. In mymodule I use the following snippet (in __init__.py) to obtain the version:
import importlib.metadata
__version__ = importlib.metadata.version(__package__)
del importlib.metadata
However this version is wrong. This appears to be the highest version which I have ever installed. For reference the command python3 -m pip show mypackage does actually show the correct version after installing the module locally. I struggle to explain this difference. Can anyone think of a cause of this discrepancy?
I also ran importlib.metadata.version(mypackage) which returned the same incorrect version.
The problem was related to left over build artifacts from using setup.py. importlib and pkg_resources will detect these artifacts in a local installation and pip will not. Deleting the mypackage.egg-info directory fixed the issue.
Related
pytest appears to be using old source code and failing tests because of it. I'm not sure how to update it.
Test code:
from nba_stats import league
class TestLeaders():
def test_default():
leaders = league.Leaders()
print(leaders)
Source code (league.py):
from nba_stats.nba_api import NbaAPI
from nba_stats import constants
class Leaders:
...
When I run pytest on my parent directory, I get an error that refers to an old import statement.
_____________________________ ERROR collecting test/test_league.py ______________________________
ImportError while importing test module '/home/mfb/src/nba_stats/test/test_league.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_league.py:1: in <module>
from nba_stats import league
../../../.virtualenvs/nba_stats_dev/lib/python3.6/site-packages/nba_stats/league.py:1: in <module>
from nba_stats import _api_scrape, _get_json
E ImportError: cannot import name '_api_scrape'
I tried resetting my virtualenvironment and also reinstalling my package via pip. What do I need to do to tell it to see the new import statement and why is this happening?
Edit: Deleting my virtual environment completely and then creating a new one seemed to fix it, but it seems to be a recurring issue with any further source code changes. Surely there must be a way to not have to reset my virtualenvironment each time?
Looks like you installed that package (possibly as a dependency through something else if not directly) and also have it cloned locally for development. You can look into local editable installs (https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs), but personally, I prefer to make the test refer directly to the package under which it is being run, since then it can be used "as-is" after cloning it. Do this by modifying sys.path in your test_league.py. Ie., assuming it has a structure with the python code under python/nba_stats, in the parent directory of `test
sys.path = [os.path.join(os.pardir, 'python')] + sys.path
at the top of test_league.py. This puts your local package up front and import will consider it first.
EDIT:
Since you tried and it still did not work (please do make sure that the snippet above does point to the local python package as in the actual structure; the above is just a common one but you may have a different structure), here is how you can see which directories are considered in order, and which are eventually selected:
python -vv -m pytest -svx
You will be able to see if there are spurious directories in sys.path, whether the one tried (as in the snippet above) matches as expected or not, any left-over .pyc files that get picked up, etc.
EDITv2: Since you stated that python -m pytest works, but pytest not, have a look where that pytest executable is coming from with which pytest. Likely it's a system one that refers to a different python then the one in your virtualenv. To see which python it picks up, do:
cat `which python`
and look at the top line.
If that is not the same as what which python gives you (with your desired virtualenv activated), you may have to install pytest for that current virtualenv (python -m pip install pytest).
When I use pip to install a package from source, it will generates a version number for the package which I can see using 'pip show '. But I can't find out how that version number is generated and I can't find the version string from the source code. Can someone tell me how the version is generated?
The version number that pip uses comes from the setup.py (if you pip install a file, directory, repo, etc.) and/or the information in the PyPI index (if you pip install a package name). (Since these two must be identical, it doesn't really matter which.)
It's recommended that packages make the same string available as a __version__ attribute on their top-level module/package(s) at runtime that they put in their setup, but that isn't required, and not every package does.
And if the package doesn't expose its version, there's really no way for you to get it. (Well, unless you want to grub through the pip data trying to figure out which package owns a module and then get its version.)
Here's an example:
In the source code for bs4 (BeautifulSoup4), the setup.py file has this line:
version = "4.3.2",
That's the version that's used, directly or indirectly, by pip.
Then, inside bs4/__init__.py, there's this line:
__version__ = "4.3.2"
That means that Leonard Richardson is a nice guy who follows the recommendations, so I can import bs4; print(bs4.__version__) and get back the same version string that pip show beautifulsoup4 gives me.
But, as you can see, they're two completely different strings in completely different files. If he wasn't nice, they could be totally different, or the second one could be missing, or named something different.
The OpenStack people came up with a nifty library named PBR that helps you manage version numbers. You can read the linked doc page for the full details, but the basic idea is that it either generates the whole version number for you out of git, or verifies your specified version number (in the metadata section of setup.cfg) and appends the dev build number out of git. (This relies on you using Semantic Versioning in your git repo.)
Instead of specifying the version number in code, tools such as setuptools-scm may use tags from version control. Sometimes the magic is not directly visible. For example PyScaffold uses it, but in the project's root folder's __init__.py one may just see:
import pkg_resources
try:
__version__ = pkg_resources.get_distribution(__name__).version
except:
__version__ = "unknown"
If, for example, the highest version tag in Git is 6.10.0, then pip install -e . will generate a local version number such as 6.10.0.post0.dev23+ngc376c3c (c376c3c being the short hash of the last commit) or 6.10.0.post0.dev23+ngc376c3c.dirty (if it has uncommitted changes).
For more complicated strings such as 4.0.0rc1, they are usually hand edited in the PKG-INFO file. Such as:
# cat ./<package-name>.egg-info/PKG-INFO
...
Version: 4.0.0rc1
...
This make it unfeasible to obtain it from within any python code.
The default version of python (ie, one that opens on typing "python" in command line) is 2.6 on my server. I also have 2.7 installed. How do I import a module, in this case numpy, in python2.7? When I try to import right now, it gives me an error -
ImportError: No module named numpy
Is there any workaround, apart from downloading the package and doing a build install?
I agree with the comment above. You will have to compile it separately. I once tried a hack of importing and modifying the sys.path however I ran into issues with the .so files.
I'm confused about the way Python, on OS X, uses the packages in 'Extras' and 'site-packages'. In particular, I'm confused about the what I see in these directories, and how duplicate packages in the directories affect each other, and where I should put packages that I install.
I had assumed that 'Extras'1 was the location for packages that are not part of core Python, but that were nonetheless distributed with a given platform. For example, OS X, is distributed with PyObjC, twisted, and numpy, among others; and these are found in 'Extras'. I had also assumed that 'site-packages'2 is where packages that I subsequently install go, and that the directory is empty or absent on a "fresh" machine. Furthermore I'd assumed that installing a new package would leave the version in 'Extras' alone, and place any updates I do in 'site-packages', where they would "mask" those in 'Extras'. (So for example, the default installation could simply be restored by deleting 'site-packages'.)
What confuses me is that the contents of my 'Extras' directory is dwindling. Compared with the contents on a fresh OS X (10.8.2) install, the 'Extras' for my current configuration is missing
altgraph
altgraph-0.9-py2.7.egg-info
dateutil
macholib
macholib-1.4.2-py2.7.egg-info
modulegraph
modulegraph-0.9.1-py2.7.egg-info
numpy #though numpy-1.6.1-py2.7.egg-info is there
pkg_resources.py # though .pyc is there
py2app
py2app-0.6.3-py2.7.egg-info
setuptools # though setuptools-0.6c12dev_r88846-py2.7.egg-info is there
site.py # though .pyc and .pyo are there
xattr
xattr-0.6.2-py2.7.egg-info
zope
zope.interface-3.5.1-py2.7.egg-info
and has two additional files not found in the fresh install
pkg_resources.py.OLD.1356069438.31
setuptools-0.6c12dev_r88846-py2.7.egg-info.OLD.1356069438.31
setuptools.OLD.1356069438.31
To the best of my recollection, these are all packages that I've updated myself, and versions of all are present in 'site-packages'.
Is the contents of the 'Extras' directory supposed to "dwindle" in this way? Does updating a package that is in 'Extras' "move" it to 'site-packages', or should the 'Extras' versions be left alone and simply "masked" by those in 'site-packages'?
1:/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
2: /Library/Python/2.7/site-packages/
OSX (and Linux) does not seem to like to install extra python packages in the default installation directory of python distribution.
pip can search (like python itself does) for installed modules, and if you do an upgrade will de-install the old version and install the new version in the place you asked for, defaulting to site-packages. If you closely look at the output of pip it will tell you what it removed and what it installed.
So only if you tell pip to install to the Extras directory, or would have a pip patched to do so for OSX by default, that would work.
It is actually a good thing the older versions are removed, as otherwise the version of a module you import would be depending on your search path (sys.path) and that is kind of error prone.
Whether it is supposed to dwindle can be argued, but it is a normal result of the way you do your upgrades with pip.
Getting the following kinds of warnings when running most python scripts in the command line:
/Library/Python/2.6/site-packages/virtualenvwrapper/hook_loader.py:16: UserWarning: Module
pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Library/Python/2.6/site-packages is being added to sys.path
import pkg_resources
/Library/Python/2.6/site-packages/virtualenvwrapper/hook_loader.py:16: UserWarning: Module site was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site.pyc, but /Library/Python/2.6/site-packages is being added to sys.path
import pkg_resources
I think it has to do with a combination of using distribute and virtualenv, but wanted to check if anyone else has run in to this or would know how to go about fixing it.
Perhaps use the virtualenv option --no-site-packages so you won't see any system site-packages within your virtual environment. Having items installed both in your virtualenv and on the system root may be the cause of this issue.
Using --no-site-packages when creating your virtualenv prevents any conflict between system packages. I almost always use that option when creating a new virtualenv to prevent any conflicts. Though I may have several copies of libraries, at least they don't mess with each other.
The python equivalent of putting a bit of electrical tape over the check engine light would be to use the -W command line flag or by adding a warning filter.
In my case reinstalling of anything did not help. There were some orphaned .pyc files (specifically pkg_resources.pyc) left in /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
sudo find . -type f -name "*.pyc" -delete
made it work. This link helped me to track down the problem.
I had this sort of Python packaging hell visit today too.
Running Python 2.7.3 on Ubuntu, using namespace packages and using zc.buildout.
Finally, updating system wide Distribute from older version 0.6.30 to latest version 0.6.35 resolved the problem.
If the warning shows in a program you are modifying, try it this way (examply with pytz):
try:
import pytz
except ImportError:
from pkg_resources import require
require('pytz')