Develop Reference

Why is mechanize not installing properly via pip on RPi? (python 3.9)

I can't get to successfully install the package 'mechanize' on a Raspberry Pi (so, ARM chip) with Debian Bullseye, python 3.9 in a virtualenv.
When I look in the virtualenv's sitepackages folder, indeed the mechanize package only has .dist-info file, but not a mechanize.py file or mechanize package folder. See below.
Question: Am I overlooking something?
Details (1): Installing mechanize with pip
(venvtest) pi#mango:~/MyProject/myproj $ pip --version
pip 20.3.4 from /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages/pip (python 3.9)
(venvtest) pi#mango:~/MyProject/myproj $ pip install mechanize
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting mechanize
Using cached https://www.piwheels.org/simple/mechanize/mechanize-0.4.8-py2.py3-none-any.whl (5.1 kB)
Requirement already satisfied: html5lib>=0.999999999 in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from mechanize) (1.1)
Requirement already satisfied: webencodings in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from html5lib>=0.999999999->mechanize) (0.5.1)
Requirement already satisfied: six>=1.9 in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from html5lib>=0.999999999->mechanize) (1.16.0)
Installing collected packages: mechanize
Successfully installed mechanize-0.4.8
(venvtest) pi#mango:~/MyProject/myproj $ pip list
Package Version
------------- -------
html5lib 1.1
mechanize 0.4.8
pip 20.3.4
pkg-resources 0.0.0
setuptools 44.1.1
six 1.16.0
webencodings 0.5.1
wheel 0.34.2
Details (2): Package folder missing in sitelibs
(venvtest) pi#mango:~/MyProject/myproj/ $ ls venvtest/lib/python3.9/site-packages
easy_install.py pip-20.3.4.dist-info __pycache__ six.py wheel
html5lib pip-20.3.4.virtualenv setuptools _virtualenv.pth wheel-0.34.2.dist-info
html5lib-1.1.dist-info pkg_resources setuptools-44.1.1.dist-info _virtualenv.py wheel-0.34.2.virtualenv
mechanize-0.4.8.dist-info pkg_resources-0.0.0.dist-info setuptools-44.1.1.virtualenv webencodings
pip pkg_resources-0.0.0.virtualenv six-1.16.0.dist-info webencodings-0.5.1.dist-info
Details (3): As a consequence, importing mechanize doesn't work
(venvtest) pi#mango:~/MyProject/myproj $ python
Python 3.9.2 (default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mechanize
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mechanize'
>>> exit()

I've found this thread : https://github.community/t/python-modulenotfounderror-although-module-is-installed-aiohttp/15425 , does this help ?

It appears to be an issue with version 0.4.8. Installing version 0.4.7 works fine:
pip install mechanize==0.4.7
as can be seen:
(venvtest) pi#mango:~/MyProj/myproj $ python
Python 3.9.2 (default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mechanize
>>> print(mechanize.__version__)
(0, 4, 7, None, None)
I've opened a ticket at the mechanize github repository: https://github.com/python-mechanize/mechanize/issues/76

Pip succeeds but ImportError with Pandas - even inside virtualenv

I believe I'm following commonly perscribed means of installing a python package. First I activate the virtual environment:
[]$ source virt_env/bin/activate
Then I check to see that the python and pip versions and locations are correct:
(virt_env) []$ which python
~/virt_env/bin/python
(virt_env) []]$ which pip
~/virt_env/bin/pip
Now I attempt to install with pip and it succeeds
(virt_env) []$ pip install pandas
Collecting pandas
Using cached https://files.pythonhosted.org/packages/b7/e3/f52d484244105fa3d558ce8217a5190cd3d40536076bef66d92d01566325/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.9.0 (from pandas)
Using cached https://files.pythonhosted.org/packages/9e/eb/c9eda9f4865d669e0bb37ce5c780e86c63daa54ca827b95a171429012d08/numpy-1.15.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: python-dateutil>=2.5.0 in ./virt_env/lib/python2.7/dist-packages (from pandas) (2.7.3)
Requirement already satisfied: pytz>=2011k in ./virt_env/lib/python2.7/dist-packages (from pandas) (2018.5)
Requirement already satisfied: six>=1.5 in ./virt_env/lib/python2.7/dist-packages (from python-dateutil>=2.5.0->pandas) (1.11.0)
Installing collected packages: numpy, pandas
Successfully installed numpy-1.15.3 pandas-0.23.4
However, when I try to import it in my python session, I cannot find pandas
(virt_env) []$ python
Python 2.7.14 (default, May 2 2018, 18:31:34)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pandas
Is this not the correct way to install? Looking in ./virt_env/lib/python2.7/dist-packages I see that pandas is no where to be found. Am I doing anything wrong?
I have tried doing python -m pip install pandas as well with no success.
I am on an Amazon ec2 instance running Amazon Linux.
Edit: python -m site returns the following:
sys.path = [
'',
'/home/myuser/virt_env/local/lib/python2.7/dist-packages',
'/home/myuser/virt_env/local/lib/python2.7/dist-packages',
'/home/myuser/virt_env/lib/python2.7/dist-packages',
'/home/myuser/virt_env/local/lib64/python2.7/site-packages',
'/home/myuser/virt_env/local/lib/python2.7/site-packages',
'/home/myuser/virt_env/lib64/python2.7',
'/home/myuser/virt_env/lib/python2.7',
'/home/myuser/virt_env/lib64/python2.7/site-packages',
'/home/myuser/virt_env/lib/python2.7/site-packages',
'/home/myuser/virt_env/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7',
'/usr/lib/python2.7',
]
USER_BASE: '/home/myuser/.local' (doesn't exist)
USER_SITE: '/home/muser/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
I am also able to install things if I use
pip install --install-option="--prefix=/home/myuser/virt_env" pandas
and then move the resulting files to the correct location.

Python Modules Not In Correct Location

We have CentOS 7 servers in a Hadoop cluster. Python and pip should have been installed on all of the server the same way because it was done with Ansible. But for some reason, there are some servers where Python cannot use the modules that were installed. For instance pandas was install, but when in python3.6, I get a 'No module named pandas' error.
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>
If I try installing pandas again, I get messages that the reuirements are already satisfied.
$ sudo pip3.6 install pandas
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages
Requirement already satisfied: python-dateutil>=2 in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: pytz>=2011k in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib64/python3.6/site-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2->pandas)
It appears that the python 3.6 sys.path is using a different location for site-packages.
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
It may all stem from pip being installed in an unusual location.
$ pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)
Pip and the modules seem to be located in the same place, correctly on servers that DO work as follows:
$ pip3.6 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:17:20)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> exit()
$ pip3.6 show pandas
Name: pandas
Version: 0.22.0
Summary: Powerful data structures for data analysis, time series,and statistics
Home-page: http://pandas.pydata.org
Author: The PyData Development Team
Author-email: pydata#googlegroups.com
License: BSD
Location: /usr/local/lib/python3.6/site-packages
Requires: numpy, pytz, python-dateutil
So my question is, how do I get this fixed and how do I keep this from happening in the first place? I don't seem to find a way to configure Pip and I don't find any way to specify the installation location for modules.
I tried to be thorough with the information provided, but I'm sure I missed something.
Thanks,
Anthony
Well, the Pip documentation for the config pointed me to the --target option for pip install. That allowed me to specify the location where I thought the packages should go - /usr/local/lib/python3.6/site-packages (as noted in the sys.path). That allowed the import to find the pandas package, but fail on the import of some basic system modules, builtin and binascii. It seems like python itself is kinda hosed.
# pip3.6 install --target /usr/local/lib/python3.6/site-packages pandas
Collecting pandas
Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy>=1.9.0 (from pandas)
Downloading numpy-1.14.2-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
100% |████████████████████████████████| 12.2MB 92kB/s
Collecting python-dateutil>=2 (from pandas)
Downloading python_dateutil-2.7.2-py2.py3-none-any.whl (212kB)
100% |████████████████████████████████| 215kB 2.4MB/s
Collecting pytz>=2011k (from pandas)
Using cached pytz-2018.3-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.14.2 pandas-0.22.0 python-dateutil-2.7.2 pytz-2018.3 six-1.11.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 48, in <module>
import __builtin__ as builtins
ModuleNotFoundError: No module named '__builtin__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 23, in <module>
from pandas.compat.numpy import *
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 61, in <module>
import http.client as httplib
File "/usr/local/lib/python3.6/http/client.py", line 71, in <module>
import email.parser
File "/usr/local/lib/python3.6/email/parser.py", line 12, in <module>
from email.feedparser import FeedParser, BytesFeedParser
File "/usr/local/lib/python3.6/email/feedparser.py", line 27, in <module>
from email._policybase import compat32
File "/usr/local/lib/python3.6/email/_policybase.py", line 7, in <module>
from email import header
File "/usr/local/lib/python3.6/email/header.py", line 14, in <module>
import binascii
ModuleNotFoundError: No module named 'binascii'
>>> exit()
Ugh...

You need to be very careful using pip when having multiple Python distributions. Instead of using pip3.6', try usingpython3.6 -m pip` to install pandas:
python3.6 -m pip install pandas
(prepend sudo if you really need to). In this way you are invoking pip through Python itself, and so you are guaranteed to get the pip that belongs with the Python you want.

You can try to set the location for pip in $HOME/.pip/pip.conf
As per PIP Docs:
Configuration Config file
pip allows you to set all command line option defaults in a standard
ini style config file.
The names and locations of the configuration files vary slightly
across platforms. You may have per-user, per-virtualenv or site-wide
(shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME
environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There are also a legacy per-user configuration file which is also
respected, these are located at:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the
environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Site-wide:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a "pip" subdirectory of any of the paths set in the
environment variable XDG_CONFIG_DIRS (if it exists), for example
/etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Site-wide configuration is not supported on Windows Vista
If multiple configuration files are found by pip then they are
combined in the following order:
Firstly the site-wide file is read, then
The per-user file is read, and finally
The virtualenv-specific file is read.
Each file read overrides any values read from previous files, so if
the global timeout is specified in both the site-wide file and the
per-user file then the latter value is the one that will be used.

Short answer, but I would recommend using Python Virtual Environments. Makes managing Python versions and pip packages very easy.
https://docs.python.org/3/tutorial/venv.html
This way you can have many different virtualized Python environments and activate them for whatever script, Ansible playbook, etc you are running. Another great feature is you can create a requirements file to share with others, so if they want to run your code they can simply install from the requirements file which guarantees they have the right version of Python and associated packages installed.

Specifying which python version to pip install to

I'm having troubles when trying to download and run packages within python, using pip install. Often pip will tell me that I have the package downloaded, but when I try to use the function it gives me an error and says 'no such file or directory'. It seems to me that I am using 2.7.10 and that is where pip is trying to install to, but I can't get why this isn't working
Peters-MBP:~ pete$ python import plotly
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'import': [Errno 2] No such file or directory
Peters-MBP:~ pete$ pip install plotly
Requirement already satisfied (use --upgrade to upgrade): plotly in ./Library/Python/2.7/lib/python/site-packages
Collecting pytz (from plotly)
Using cached pytz-2016.3-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests in ./Library/Python/2.7/lib/python/site-packages (from plotly)
Collecting six (from plotly)
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six
Successfully installed pytz-2016.3 six-1.10.0
You are using pip version 8.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Peters-MBP:~ pete$ python --version
Python 2.7.10
Peters-MBP:~ pete$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Try to upgrade 'pip install --upgrade pip'
Then try to upgrade also plotly and check it again.
Please post if the plotly module is visible in the Python27 installation folder.
In addition you should go inside the "python interpreter" to import a module.
Just try to
> python
import plotly

You can’t just import a module on the command line like that(!!!):
$ python import plotly
Instead, do this:
First start the interactive Python interpreter:
$ python
Then, at the Python prompt, import your installed module:
>>> import plotly
Or, of course, put the import statement in a script and execute that script.
Edit: If you try to run python install plotly at the shell prompt, as you did, the Python interpreter will look for a script called install in the current directory, in order to execute it. Since no such script exists, it gives the error can't open file 'import': [Errno 2] No such file or directory.

Python interpteter can't find module in virtualenv, but pip sees it and won't install

I'm trying to use the pytools module within the virtualenv created by Nervana for their Neon deep learning package, but can't seem to either find pytools or pip it. When I enter my virtualenv, I see this behavior:
me#ARL--M6800:~/Downloads/neon$ source .venv/bin/activate
(.venv) me#ARL--M6800:~/Downloads/neon$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pytools
>>>
>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/dist-packages', '/home/me/Downloads/neon',
'/home/me/Downloads/neon/.venv/lib/python2.7',
'/home/me/Downloads/neon/.venv/lib/python2.7/plat-x86_64-linux-gnu',
'/home/me/Downloads/neon/.venv/lib/python2.7/lib-tk',
'/home/me/Downloads/neon/.venv/lib/python2.7/lib-old',
'/home/me/Downloads/neon/.venv/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/home/me/Downloads/neon/.venv/local/lib/python2.7/site-packages',
'/home/me/Downloads/neon/.venv/lib/python2.7/site-packages']
(.venv) me#ARL--M6800:~/Downloads/neon$ pip install pytools
Requirement already satisfied (use --upgrade to upgrade): pytools in
/usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg
Requirement already satisfied (use --upgrade to upgrade):
decorator>=3.2.0 in /usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): appdirs>=1.4.0
in /usr/local/lib/python2.7/dist-packages/appdirs-1.4.0-py2.7.egg (from
pytools)
Requirement already satisfied (use --upgrade to upgrade): six>=1.8.0 in
/usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6.0 in
/usr/local/lib/python2.7/dist-packages (from pytools)
So, I can't import pytools becauseit isn't on my sys.path. According to pip, it is installed in the /usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg directory, which leaves me with 3 questions:
First: Why can my virtualenv see my system-wide packages? I thought the default was not to see them. When I look at the Makefile used to create the virtualenv, I see this
# where our installed python packages will live
VIRTUALENV_DIR := .venv
VIRTUALENV_EXE := virtualenv -p python2.7 # use pyvenv for python3 install
ACTIVATE := $(VIRTUALENV_DIR)/bin/activate
which should give me default behavior.
Second: Why are there egg directories in my dist-packages dir? Doesn't this make it harder to find those modules? (Though apparently, the sys.path for my system environment python has been updated to search in the egg dir. When/How???)
Third: What is an efficient way of fixing things so that my virtualenv will have access to pytools?
(I would've numbered my list instead of First/Second/Third, but then the Makefile code I inserted didn't format well)

First of all, I don't know anything about Nervana's Neon.
I get the same sys.path when the virtualenv is created with root privileges:
dm#Z580:~$ sudo virtualenv test1 -p python2.7
[sudo] password for dm:
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in test1/bin/python2.7
Also creating executable in test1/bin/python
Installing setuptools, pip, wheel...done.
dm#Z580:~$ source test1/bin/activate
(test1)dm#Z580:~$ which python
/home/dm/test1/bin/python
(test1)dm#Z580:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
'/home/dm/test1/lib/python2.7',
'/home/dm/test1/lib/python2.7/plat-x86_64-linux-gnu',
'/home/dm/test1/lib/python2.7/lib-tk',
'/home/dm/test1/lib/python2.7/lib-old',
'/home/dm/test1/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/home/dm/test1/local/lib/python2.7/site-packages',
'/home/dm/test1/lib/python2.7/site-packages']
I would remove the current virtualenv, recreate it without sudo and then install all requirements with the virtualenv's pip.
cd ~/Downloads
rm -rf neon/.venv/
virtualenv neon/.venv/ -p python2.7
source neon/.venv/bin/activate
pip install -r neon/requirements.txt

For what it's worth, I uninstalled pytools using pip in my main environment. Then, went into my virtual environment and used pip to install pytools. Finally, went back to my main environment and reinstalled pytools using pip install.
Now things are working. (Though oddly, pytools is no longer installed in a '.egg' directory. I think this is better, but am now even more puzzled as to why/when things get installed into .egg directories)