Develop Reference

Why doesn't virtualenv contain a module for pip when it creates a python environment for me?

I might have multiple problems here but they both deal with virtualenv and python3 (3.6.13).
Problem 1:
I had installed virtualenv==20.14.1.
However, it's complaining that there's a version conflict for some reason.
$ python3 -V
Python 3.6.13
$ virtualenv -h
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 573, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 891, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 782, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.VersionConflict: (virtualenv 20.14.1 (/home/mylogin/.local/lib/python3.6/site-packages), Requirement.parse('virtualenv==15.0.1'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/virtualenv", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3266, in <module>
#_call_aside
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3241, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 3279, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 575, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 588, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/__init__.py", line 777, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'virtualenv==15.0.1' distribution was not found and is required by the application
$
Problem 2:
I downgraded virtualenv to be 15.0.1 like it said it needed but then it complains that there's a missing module called dataclasses. For some reason, my co-workers don't have this problem but we can't figure out what is different between my setup and their setup.
$ virtualenv mydir -p python3.6
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /tmp/mydir/bin/python3.6
Also creating executable in /tmp/mydir/bin/python
Installing setuptools, pip, wheel...done.
/tmp/mydir$ source bin/activate
(mydir) myhost:/tmp/mydir$ python -V
Python 3.6.13
(mydir) myhost:/tmp/mydir$ python -m pip -V
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/mydir/lib/python3.6/site-packages/pip/__main__.py", line 29, in <module>
from pip._internal.cli.main import main as _main
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/main.py", line 9, in <module>
from pip._internal.cli.autocompletion import autocomplete
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
from pip._internal.cli.main_parser import create_main_parser
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
from pip._internal.cli import cmdoptions
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
from pip._internal.cli.parser import ConfigOptionParser
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/cli/parser.py", line 12, in <module>
from pip._internal.configuration import Configuration, ConfigurationError
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/configuration.py", line 20, in <module>
from pip._internal.exceptions import (
File "/tmp/mydir/lib/python3.6/site-packages/pip/_internal/exceptions.py", line 14, in <module>
from pip._vendor.rich.console import Console, ConsoleOptions, RenderResult
File "/tmp/mydir/lib/python3.6/site-packages/pip/_vendor/rich/console.py", line 7, in <module>
from dataclasses import dataclass, field
ModuleNotFoundError: No module named 'dataclasses'
This seems to be a chicken and the egg problem. I would run pip to install this missing module but pip is the program that is complaining about this missing module so I can't install it. Like I said above, none of my co-workers I asked know what the problem is.
For debugging purposes, I installed python 2.7.12 and virtualenv==20.14.1 to see what would happen. Apparently things work fine with python 2.7.
$ pip freeze | grep virtual
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
virtualenv==20.14.1
virtualenv-clone==0.5.3
virtualenvwrapper==4.8.4
myhost:/tmp$ virtualenv zzz -p python
Running virtualenv with interpreter /usr/bin/python
New python executable in /tmp/zzz/bin/python
Installing setuptools, pip, wheel...done.
myhost:/tmp$ cd zzz
myhost:/tmp/zzz$ source bin/activate
(zzz) myhost:/tmp/zzz$ python -V
Python 2.7.12
(zzz) myhost:/tmp/zzz$ pip -V
pip 20.3.4 from /tmp/zzz/local/lib/python2.7/site-packages/pip (python 2.7)
(zzz) myhost:/tmp/zzz$ pip freeze
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
(zzz) myhost:/tmp/zzz$
So the questions are
Why do I need to use virtualenv 15.0.1? Why can't I use a newer version of virtualenv?
How come python3 is complaining about the missing dataclasses module but python2 is ok?

Like mentioned in this pypi docs; https://pypi.org/project/dataclasses/ dataclasses only exist in 3.7 and above, which i would recommend to use instead, otherwise, you could also download this package in hopes that it fixes your problem. Are you limited to a python version?

ModuleNotFoundError: No module named 'Cython' when installing pyarrow

I am trying to install packages from requirements.txt in a virtualenv. However, when pip comes to installing pyarrow, it fails with the following:
Collecting pyarrow==0.9.0 (from -r requirements.txt (line 41))
Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-dgdvn2c1/pyarrow/setup.py", line 29, in <module>
from Cython.Distutils import build_ext as _build_ext
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-dgdvn2c1/pyarrow/
I am using Python 3.6.3 and Pip 18.0
What I tried:
Most of the SO answers ask to check if python and pip refer to the venv python and I am sure it does.
$ which python
/Users/ME/repos/myproject/venv/bin/python
(venv)
I also tried installing cython via pip and even then installation fails albeit with a different error.
How do I solve this?
Edit:
Stacktrace after installing Cython:
Collecting pyarrow==0.9.0 (from -r requirements.txt (line 41))
Using cached https://files.pythonhosted.org/packages/be/2d/11751c477e4e7f4bb07ac7584aafabe0d0608c170e4bff67246d695ebdbe/pyarrow-0.9.0.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/setup.py", line 508, in <module>
url="https://arrow.apache.org/"
File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 131, in setup
return distutils.core.setup(**attrs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/dist.py", line 370, in __init__
k: v for k, v in attrs.items()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py", line 281, in __init__
self.finalize_options()
File "/Users/ME/Work/myproject/venv/lib/python3.6/site-packages/setuptools/dist.py", line 529, in finalize_options
ep.load()(self, ep.name, value)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/integration.py", line 23, in version_keyword
dist.metadata.version = get_version(**value)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 135, in get_version
parsed_version = _do_parse(config)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 77, in _do_parse
parse_result = _call_entrypoint_fn(config, config.parse)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 40, in _call_entrypoint_fn
return fn(config.absolute_root)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/setup.py", line 462, in parse_version
return version_from_scm(root)
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 28, in version_from_scm
return _version_from_entrypoint(root, "setuptools_scm.parse_scm")
File "/private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/.eggs/setuptools_scm-3.0.5-py3.6.egg/setuptools_scm/__init__.py", line 44, in _version_from_entrypoint
for ep in iter_matching_entrypoints(config.absolute_root, entrypoint):
AttributeError: 'str' object has no attribute 'absolute_root'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/tj/nzhk3hh12074tfyhtgpkw0zm0000gn/T/pip-install-98s3ff2r/pyarrow/

This is a known issue (#2163) with pyarrow that has been closed without a fix.
It's worth noting that there is no pyarrow 0.9.0 wheel available for MacOS in pypi. It was made available in a subsequent 0.9.0.post1 release.
The following should work with Python 3.6 on MacOS:
pip install pyarrow==0.9.0.post1

Installing pip for arcpy

I'm attempting to install pip for arcpy (arcgis 10.2 on windows 7). Running get-pip.py results in the following error message:
X:\python>python get-pip.py
Traceback (most recent call last):
File "get-pip.py", line 20061, in <module>
main()
File "get-pip.py", line 194, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip
File "c:\temp\tmpou5fje\pip.zip\pip\__init__.py", line 26, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\utils\__init__.py", line 27, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\pkg_resources\__init__.py", line 73, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\packaging\specifiers.py", line 275, in <module>
File "c:\temp\tmpou5fje\pip.zip\pip\_vendor\packaging\specifiers.py", line 373, in Specifier
File "C:\Python27\ArcGIS10.2\Lib\re.py", line 190, in compile
return _compile(pattern, flags)
File "C:\Python27\ArcGIS10.2\Lib\re.py", line 242, in _compile
raise error, v # invalid expression
sre_constants.error: nothing to repeat
Using an administrator command prompt doesn't help. My real goal is to get win32com working under arcpy. I usual just copy the appropriate directories out of c:\python27\lib\site-packages to c:\python27\arcgis10.2\lib\site-packages to install a package under arcpy (why doesn't arcpy come with pip?) but that's not working for win32com, presumably do to a missing dll or other windows specific file.

I would recommend the following:
Get the setuptools module
Get the pip module`
And then run the following in command line (assuming windows)
path-to-python path-to-setuptools install
path-to-python path-to-pip install
I work on a closed network (away from the interwebs of old) and cannot use get-pip.py so I find it best to simply download the actual modules and hard install.
Keep us posted!

Copy get_pip.py to "C:\Python27\ArcGIS10.2", then perform command "python get-pip.py" in the directory.
Note that keep network connected in the process, so that auto-download and setup setuptools,wheels,etc.
Hope that can help you.

Try opening a CMD prompt and typing:
C:\Python27\ArcGIS10.2\python.exe -m pip install -U pip

getting DistributionNotFound error, but package exists

I am running pyjade for Django templates as I find it much easier to write in, but I am trying to convert it to html with the built in converter. When I run pyjade -c django input.jade output.html I get the error pkg_resources.DistributionNotFound: six. However, the package six does in fact exist, and I can import it on python, and when I run pip install six --upgrade it says it is already up-to-date.
Here is the full traceback:
Traceback (most recent call last):
File "/usr/local/bin/pyjade", line 5, in <module>
from pkg_resources import load_entry_point
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2707, in <module>
working_set.require(__requires__)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 686, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 584, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: six
What would be causing this error?

Re-install via easy_setup (vs pip) solved it for me on OS X Mavericks:
sudo easy_install six
Hope that helps you!

An old setuptools was my culprit.
pip install -U setuptools

Why can't I install the openstack nova client on OS X?

I am attempting to install the openstack nova client on my Mac (10.4.8)
nova = https://github.com/openstack/python-novaclient#command-line-api
python --version
Python 2.7.2
I successfully got nova installed (after installing pip)
When I run the client, I get the following error
foo#bar-macbook-pro:~$ nova
Traceback (most recent call last):
File "/usr/local/bin/nova", line 6, in <module>
from novaclient.shell import main
File "/Library/Python/2.7/site-packages/novaclient/__init__.py", line 15, in <module>
import pbr.version
ImportError: No module named pbr.version
In my research, I have found conflicting information about pbr, some say it is required for nova, while others say it isn't required for nova.
https://github.com/rackspace/pyrax/issues/121
When I attempt to install pbr, I see the following error.
foo#bar-macbook-pro:~$ sudo python ~/Downloads/pbr/setup.py install
Traceback (most recent call last):
File "setup.py", line 22, in <module>
**util.cfg_to_args())
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/util.py", line 241, in cfg_to_args
pbr.hooks.setup_hook(config)
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/__init__.py", line 27, in setup_hook
metadata_config.run()
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/base.py", line 29, in run
self.hook()
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/hooks/metadata.py", line 28, in hook
self.config['name'], self.config.get('version', None))
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 817, in get_version
version = _get_version_from_git(pre_version)
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 776, in _get_version_from_git
"git --git-dir=\"" + git_dir + "\" describe --always").replace(
File "/Volumes/WDBlack750/spencerowen/Downloads/pbr/pbr/packaging.py", line 220, in _run_shell_command
stderr=err_location)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1228, in _execute_child
raise child_exception
TypeError: must be encoded string without NULL bytes, not str
Is there anything apparent that would explain why I can not get the library installed?
Surely I must not be the first person to try and install nova on OS X.

Over a year later, I finally got this working on OS X Yosemite
sudo pip install python-novaclient
I did not have to install pbr.

I did the install based on venv:
virtualenv venv_name
source venv_name/bin/activate
pip install python-novaclient fabric
In my case, I had a mixup in which python I was using by way of fabric being installed globally.
Prior: rf -rf all my virtualenvs, rf -rf all references to novaclient (locally, and globally), and deleted a global install of fabric which was calling the novaclient.
Also as a precaution, I do not install pip globally, and only use it without sudo in virtualenvs.