Develop Reference

Error when trying to upgrade a python package

I have git pulled a python package (I think that is the correct terminology) that I already had installed and working.
So now it is up to date, apparently I needed to update it using pip. I did the following:
pip install the_package --upgrade
But I got the following error message:
Exception:
Traceback (most recent call last):
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/req/req_set.py", line 666, in _prepare_file
check_dist_requires_python(dist)
File "/opt/work/python/2.7.13/lib/python2.7/site-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
feed_parser.feed(metadata)
File "/opt/work/python/2.7.13/lib/python2.7/email/feedparser.py", line 177, in feed
self._input.push(data)
File "/opt/work/python/2.7.13/lib/python2.7/email/feedparser.py", line 99, in push
parts = data.splitlines(True)
AttributeError: 'NoneType' object has no attribute 'splitlines'
I'm not sure what the problem is. Could someone please point me in the right direction?

I can't say much about the error you're seeing, but my best guess is that the version of the package from git is interfering with the pip installed one. Perhaps something went wrong when trying to upgrade starting with the gitHub version (or wherever it came from) and not with an officially published one.
If a package is available through PyPI you shouldn't need to do anything besides pip install <somepackage>. I would recommend removing all files downloaded via git and those modules in your python lib so you can start over completely. From there just use pip install <packagename>.
If you're not already in a python virtual environment I would recommend creating one too, so that these things are easier to fix.
Edit: I just found elsewhere on this site that you can use pip show <packagename> to show the install location of the package if you're not in a venv (or if you are).

ImportError: No module named py31compat

i am trying to install gensim using
sudo -H pip install --upgrade gensim
but it is giving me this error :
File "setup.py", line 301, in <module>
include_package_data=True,
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command /install.py", line 67, in run
self.do_egg_install()
File "/usr/local/lib/python2.7/dist-packages/setuptools/command /install.py", line 98, in do_egg_install
easy_install = self.distribution.get_command_class('easy_install')
File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 576, in get_command_class
self.cmdclass[command] = cmdclass = ep.load()
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2229, in load
return self.resolve()
File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2235, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 47, in <module>
from setuptools.sandbox import run_setup
File "/usr/local/lib/python2.7/dist-packages/setuptools/sandbox.py", line 15, in <module>
import pkg_resources.py31compat
ImportError: No module named py31compat
please help me, in installing gensim. i googled it, but i am not able to find the solution.

It looks like setuptoopls is missing. Try
$sudo pip install setuptools

Notice that setuptools come from /usr/local/lib whereas pkg_resources come from /usr/lib. You must likely have an older pkg_resources through the package system and then a newer setuptools installed by pip.
Delete (or move away) the /usr/local/lib/python2.7/dist-packages/setuptools directory.

I encountered a very similar error upon an Ansible run of a refactored OpenShift playbook. The error was discovered once all of the plays were parsed.
"ERROR! Unexpected Exception, this is probably a bug: cannot import name py31compat"
I am using ...
Ansible 2.5.3
Python2-2.7.15
I ran the following command to list the location of all of the packages and libraries associated with Python 2. I checked available paths to ensure I had one version of Ansible and one copy of each package installed.
$ python -c 'import sys; print sys.path'
'py31compat' is not a separate module one can download. It is included with the Setuptools package. It was determined I had two packages installed of different versions, both out-of-date. I uninstalled both via pip and installed the latest setuptools version available. No luck.
My solution was to uninstall Ansible and reinstall the same version. This resolved the issue.

In my case i have odoo v10 installed and for any weird reason update or installed a library that stuck my server and not longer to start with XOE.
The error was:
cannot import name py31compat
After 1 hour trying many differents things i decide uninstall:
sudo pip uninstall setuptools
Uninstalling setuptools-40.8.0: Would remove:
/usr/local/bin/easy_install
/usr/local/bin/easy_install-2.7
/usr/local/lib/python2.7/dist-packages/easy_install.py
/usr/local/lib/python2.7/dist-packages/pkg_resources/*
/usr/local/lib/python2.7/dist-packages/setuptools-40.8.0.dist-info/**
And my service odoo server starts and works fine again!

How to fix ".dist-info directory not found" in my package?

I have a Python 2 package that I'm trying to upgrade to Python 3. It was written by someone who used to work on the same team I'm on now but who is no longer with the company, and unfortunately nobody left on the team is able to help out.
After running 2to3 on the files of the package, I ran python setup.py sdist to create a package, placed the package in a local repository, then attempted to use pip install to install the package. It ended up erroring out with the following:
Exception:
Traceback (most recent call last):
File "/home/user/project/lib/python3.5/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/home/user/project/lib/python3.5/site-packages/pip/commands/install.py", line 297, in run
root=options.root_path,
File "/home/user/project/lib/python3.5/site-packages/pip/req/req_set.py", line 622, in install
**kwargs
File "/home/user/project/lib/python3.5/site-packages/pip/req/req_install.py", line 808, in install
self.move_wheel_files(self.source_dir, root=root)
File "/home/user/project/lib/python3.5/site-packages/pip/req/req_install.py", line 1003, in move_wheel_files
isolated=self.isolated,
File "/home/user/project/lib/python3.5/site-packages/pip/wheel.py", line 340, in move_wheel_files
assert info_dir, "%s .dist-info directory not found" % req
AssertionError: my-package-name .dist-info directory not found
The old Python 2 version of the package didn't have anything called .dist-info in the .tgz archive, and it installed just fine. Does anyone know what's going on here and how to fix it?

In my case this was resolved by deleting AppData\Local\pip\Cache folder (windows). Should be fairly similar for other operating systems.

How to access a module installed by setuptools' install_requires within setup.py?

I'm writing a setup.py to install my package reboundx, which has a single dependency, rebound. My package builds an extension libreboundx.so that needs to link to librebound.so in setup.py
rebxExt = Extension('libreboundx', libraries=['rebound'], library_dirs = [rebound_path]...)
I'd like to be able to use install_requires in the setup(...) call to build the reboundx module to make sure the right version of rebound is installed. Is there any way to resolve the circularity?
If rebound is not installed, I would somehow need setuptools to detect this through install_requires, install rebound, and THEN find the right paths and build the extension libreboundx.so.

You should use the setup_requires argument to setup(). From the docs,
setup_requires
A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands. This argument is needed if you are using distutils extensions as part of your build process; for example, extensions that process setup() arguments and turn them into EGG-INFO metadata files.
(Note: projects listed in setup_requires will NOT be automatically installed on the system where the setup script is being run. They are simply downloaded to the ./.eggs directory if they’re not locally available already. If you want them to be installed, as well as being available when the setup script is run, you should add them to install_requires and setup_requires.)
https://pythonhosted.org/setuptools/setuptools.html#new-and-changed-setup-keywords
Edit:
It should create an egg directory for each entry in setup_requires. However, I just tried this with rebound and it actually fails to build under easy install.
$> python2 setup.py install
install_dir .
warning: no files found matching 'src/rebound.h'
src/rebound.c:38:21: fatal error: rebound.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
File "setup.py", line 187, in <module>
url="http://www.mathics.github.io/", # project home page, if any
File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
_setup_distribution = dist = klass(attrs)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 221, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 544, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 786, in best_match
return self.obtain(req, installer) # try and download/install
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/pkg_resources.py", line 798, in obtain
return installer(requirement)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/dist.py", line 293, in fetch_build_egg
return cmd.easy_install(req)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 582, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 612, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 802, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1079, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.7/site-packages/distribute-0.6.14-py2.7.egg/setuptools/command/easy_install.py", line 1070, in run_setup
raise DistutilsError("Setup script exited with %s" % (v.args[0],))
distutils.errors.DistutilsError: Setup script exited with error: command 'gcc' failed with exit status 1

(If you like this approach, I can elaborate further)
Consider this example. I use ctypes to load a shared library. Then get the version of the library using a function call.
>>> import ctypes
>>> x = ctypes.cdll.LoadLibrary("libc.so.6")
>>> x.gnu_get_libc_version
<_FuncPtr object at 0x7f9a08e3e460>
>>> getversion = x.gnu_get_libc_version
>>> getversion.restype = ctypes.c_char_p
>>> getversion()
'2.19'
You could do something similar with rebound.so. Use a try statement to load librebound.
If it is the wrong version or it is not found in the standard search path for libraries, then compile librebound.
#setup.py
import ctypes
try:
x = ctypes.cdll.LoadLibrary("librebound.so")
x.restype = ctypes.c_char_p
version = x.get_version()
major,minor = version.split(".")
if int(major) < REQUIRED_VERSION:
raise False, "LIBREBOUND VERSION %s IS NOT SUPPORTED"%(str(version))
except:
pass
#invoke rebound's makefile
#probably install the new library to lib path.
#link to new rebound
rebxExt = Extension('libreboundx', libraries=['rebound'], library_dirs = [rebound_path]...)

It seems setuptools does make this possible through the setup_requires keyword, as pointed out by sn6uv. However, I found it difficult to find documentation and/or examples that showed how to use it. From what I did find, it also seemed like it might be a bit delicate/complicated for the uninitiated.
For one, when using pip, the dependencies in setup_requires are installed differently than everything else (see Install package which has setup_requires from local source distributions). It also seems like if you need to import the dependency within setup.py, you need to subclass the install command in order to delay the import until it's available (deep in a thread on the distutils mailing list arguing over setup_requires, Donald Stufft talks about this and links to an example: https://mail.python.org/pipermail/distutils-sig/2015-March/025882.html).
So I gave up and implemented something similar to the manual check suggested by goCards. Just thought I'd post the links for the more ambitious who come across this.

Unable to upgrade Python on Red Hat Linux

I have Python 2.6.6. to run a certain script on my Red Hat Linux VM. Need to upgrade to 3.4.1. Tried
easy_install --upgrade python
But I always get
bash-4.1# easy_install --upgrade python
Searching for python
Reading http://pypi.python.org/simple/python/
Reading http://www.python.org
Reading http://www.python.org/2.3
Reading http://www.python.org/2.4
Reading http://www.python.org/2.4.1
Reading http://www.python.org/2.5
Reading http://www.python.org/download/
Best match: Python 3.4.1
Downloading https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz
Processing Python-3.4.1.tgz
Running Python-3.4.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Qd_HVP/Python-3.4.1/egg-dist-tmp-iqaG4H
Traceback (most recent call last):
File "/usr/bin/easy_install", line 9, in <module>
load_entry_point('distribute==0.6.10', 'console_scripts', 'easy_install')()
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 1715, in main
with_ei_usage(lambda:
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 1696, in with_ei_usage
return f()
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 1719, in <lambda>
distclass=DistributionWithoutHelpCommands, **kw
File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
cmd_obj.run()
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 236, in run
self.easy_install(spec, not self.no_deps)
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 472, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 502, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 681, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 958, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/usr/lib/python2.6/site-packages/setuptools/command/easy_install.py", line 947, in run_setup
run_setup(setup_script, args)
File "/usr/lib/python2.6/site-packages/setuptools/sandbox.py", line 29, in run_setup
lambda: execfile(
File "/usr/lib/python2.6/site-packages/setuptools/sandbox.py", line 70, in run
return func()
File "/usr/lib/python2.6/site-packages/setuptools/sandbox.py", line 31, in <lambda>
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 1865
exec(f.read(), globals(), fficonfig)
SyntaxError: unqualified exec is not allowed in function 'configure_ctypes' it contains a nested function with free variables
bash-4.1# SyntaxError: unqualified exec is not allowed in function 'configure_ctypes' it contains a nested function with free variables
And of course I can't delete Python 2.6.6 from the machine as a lot of system scripts depend on it. Any ideas what to do?

I generally advise against manually upgrading packages on an RPM-managed distribution such as RHEL. If an upgrade is needed, it should only be done by upgrading via rpm. There is a reason why the rpm tools exists, and why it is used. There's more than one reason, actually. It serves an important purpose, and manually installing or upgrading packages completely subverts it.
If it's absolutely necessary, the following procedure should be as follows:
Grab the source RPM for Red Hat's python package.
Grab the source for the new version of python.
Figure out what patches in Red Hat's python RPM, if any, are still applicable to the new version of python.
Use the spec file from the python RPM to build rpms with a new version of python.
Hope that step #4 works. If so, yay! Install it. If not, figure out why not, patch the spec file, as needed, go back to step #4.
You now installed a new version of python. Check that none of the other RHEL rpms that use python break with the new version. If not, yay! You're done. If something is broken, figure out what to do about it.
Many RHEL system management tools and scripts use Python. It's not entirely out of the question that something ends up being broken, as a result of the new python package being installed.
The whole reason for using RHEL is to have a commercially-supported, stable Linux distribution whose all components have been tested for interoperability. Upgrading random parts of it misses the whole point of having RHEL; typically you upgrade to an entire new release of RHEL, instead of the individual packages; but, to each their own...

Red Hat supplies multiple versions of Python (and other packages) that can be installed in parallel as part of software collections. They leave the system /usr/bin/ intact and are supported by Red Hat.
You need to:
* enable the RHSCL and optional software repos
* then you can yum install rh-python36 or python27 (2.7.13)
* Use scl enable rh-python 36 bash to add python to your path.
See How to install Python 3 on RHEL. It covers many tips for working with multiple versions of Python, Python virtual environments, and software collections.

You will need to find a source for Python-3.4 packages, or build it yourself.
A quick search finds http://wiki.guibin.info/?p=133 for building Python-3.4 yourself.
With some more searching, you may be able to find a source for RPM packages, but of course it's up to you how much you trust those packages -- I wouldn't use a random RPM in production!