Develop Reference

Why is mechanize not installing properly via pip on RPi? (python 3.9)

I can't get to successfully install the package 'mechanize' on a Raspberry Pi (so, ARM chip) with Debian Bullseye, python 3.9 in a virtualenv.
When I look in the virtualenv's sitepackages folder, indeed the mechanize package only has .dist-info file, but not a mechanize.py file or mechanize package folder. See below.
Question: Am I overlooking something?
Details (1): Installing mechanize with pip
(venvtest) pi#mango:~/MyProject/myproj $ pip --version
pip 20.3.4 from /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages/pip (python 3.9)
(venvtest) pi#mango:~/MyProject/myproj $ pip install mechanize
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting mechanize
Using cached https://www.piwheels.org/simple/mechanize/mechanize-0.4.8-py2.py3-none-any.whl (5.1 kB)
Requirement already satisfied: html5lib>=0.999999999 in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from mechanize) (1.1)
Requirement already satisfied: webencodings in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from html5lib>=0.999999999->mechanize) (0.5.1)
Requirement already satisfied: six>=1.9 in /home/pi/MyProject/myproj/venvtest/lib/python3.9/site-packages (from html5lib>=0.999999999->mechanize) (1.16.0)
Installing collected packages: mechanize
Successfully installed mechanize-0.4.8
(venvtest) pi#mango:~/MyProject/myproj $ pip list
Package Version
------------- -------
html5lib 1.1
mechanize 0.4.8
pip 20.3.4
pkg-resources 0.0.0
setuptools 44.1.1
six 1.16.0
webencodings 0.5.1
wheel 0.34.2
Details (2): Package folder missing in sitelibs
(venvtest) pi#mango:~/MyProject/myproj/ $ ls venvtest/lib/python3.9/site-packages
easy_install.py pip-20.3.4.dist-info __pycache__ six.py wheel
html5lib pip-20.3.4.virtualenv setuptools _virtualenv.pth wheel-0.34.2.dist-info
html5lib-1.1.dist-info pkg_resources setuptools-44.1.1.dist-info _virtualenv.py wheel-0.34.2.virtualenv
mechanize-0.4.8.dist-info pkg_resources-0.0.0.dist-info setuptools-44.1.1.virtualenv webencodings
pip pkg_resources-0.0.0.virtualenv six-1.16.0.dist-info webencodings-0.5.1.dist-info
Details (3): As a consequence, importing mechanize doesn't work
(venvtest) pi#mango:~/MyProject/myproj $ python
Python 3.9.2 (default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mechanize
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'mechanize'
>>> exit()

I've found this thread : https://github.community/t/python-modulenotfounderror-although-module-is-installed-aiohttp/15425 , does this help ?

It appears to be an issue with version 0.4.8. Installing version 0.4.7 works fine:
pip install mechanize==0.4.7
as can be seen:
(venvtest) pi#mango:~/MyProj/myproj $ python
Python 3.9.2 (default, Mar 12 2021, 04:06:34)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mechanize
>>> print(mechanize.__version__)
(0, 4, 7, None, None)
I've opened a ticket at the mechanize github repository: https://github.com/python-mechanize/mechanize/issues/76

Python Modules Not In Correct Location

We have CentOS 7 servers in a Hadoop cluster. Python and pip should have been installed on all of the server the same way because it was done with Ansible. But for some reason, there are some servers where Python cannot use the modules that were installed. For instance pandas was install, but when in python3.6, I get a 'No module named pandas' error.
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>
If I try installing pandas again, I get messages that the reuirements are already satisfied.
$ sudo pip3.6 install pandas
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages
Requirement already satisfied: python-dateutil>=2 in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: pytz>=2011k in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib64/python3.6/site-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2->pandas)
It appears that the python 3.6 sys.path is using a different location for site-packages.
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
It may all stem from pip being installed in an unusual location.
$ pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)
Pip and the modules seem to be located in the same place, correctly on servers that DO work as follows:
$ pip3.6 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:17:20)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> exit()
$ pip3.6 show pandas
Name: pandas
Version: 0.22.0
Summary: Powerful data structures for data analysis, time series,and statistics
Home-page: http://pandas.pydata.org
Author: The PyData Development Team
Author-email: pydata#googlegroups.com
License: BSD
Location: /usr/local/lib/python3.6/site-packages
Requires: numpy, pytz, python-dateutil
So my question is, how do I get this fixed and how do I keep this from happening in the first place? I don't seem to find a way to configure Pip and I don't find any way to specify the installation location for modules.
I tried to be thorough with the information provided, but I'm sure I missed something.
Thanks,
Anthony
Well, the Pip documentation for the config pointed me to the --target option for pip install. That allowed me to specify the location where I thought the packages should go - /usr/local/lib/python3.6/site-packages (as noted in the sys.path). That allowed the import to find the pandas package, but fail on the import of some basic system modules, builtin and binascii. It seems like python itself is kinda hosed.
# pip3.6 install --target /usr/local/lib/python3.6/site-packages pandas
Collecting pandas
Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy>=1.9.0 (from pandas)
Downloading numpy-1.14.2-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
100% |████████████████████████████████| 12.2MB 92kB/s
Collecting python-dateutil>=2 (from pandas)
Downloading python_dateutil-2.7.2-py2.py3-none-any.whl (212kB)
100% |████████████████████████████████| 215kB 2.4MB/s
Collecting pytz>=2011k (from pandas)
Using cached pytz-2018.3-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.14.2 pandas-0.22.0 python-dateutil-2.7.2 pytz-2018.3 six-1.11.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 48, in <module>
import __builtin__ as builtins
ModuleNotFoundError: No module named '__builtin__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 23, in <module>
from pandas.compat.numpy import *
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 61, in <module>
import http.client as httplib
File "/usr/local/lib/python3.6/http/client.py", line 71, in <module>
import email.parser
File "/usr/local/lib/python3.6/email/parser.py", line 12, in <module>
from email.feedparser import FeedParser, BytesFeedParser
File "/usr/local/lib/python3.6/email/feedparser.py", line 27, in <module>
from email._policybase import compat32
File "/usr/local/lib/python3.6/email/_policybase.py", line 7, in <module>
from email import header
File "/usr/local/lib/python3.6/email/header.py", line 14, in <module>
import binascii
ModuleNotFoundError: No module named 'binascii'
>>> exit()
Ugh...

You need to be very careful using pip when having multiple Python distributions. Instead of using pip3.6', try usingpython3.6 -m pip` to install pandas:
python3.6 -m pip install pandas
(prepend sudo if you really need to). In this way you are invoking pip through Python itself, and so you are guaranteed to get the pip that belongs with the Python you want.

You can try to set the location for pip in $HOME/.pip/pip.conf
As per PIP Docs:
Configuration Config file
pip allows you to set all command line option defaults in a standard
ini style config file.
The names and locations of the configuration files vary slightly
across platforms. You may have per-user, per-virtualenv or site-wide
(shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME
environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There are also a legacy per-user configuration file which is also
respected, these are located at:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the
environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Site-wide:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a "pip" subdirectory of any of the paths set in the
environment variable XDG_CONFIG_DIRS (if it exists), for example
/etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Site-wide configuration is not supported on Windows Vista
If multiple configuration files are found by pip then they are
combined in the following order:
Firstly the site-wide file is read, then
The per-user file is read, and finally
The virtualenv-specific file is read.
Each file read overrides any values read from previous files, so if
the global timeout is specified in both the site-wide file and the
per-user file then the latter value is the one that will be used.

Short answer, but I would recommend using Python Virtual Environments. Makes managing Python versions and pip packages very easy.
https://docs.python.org/3/tutorial/venv.html
This way you can have many different virtualized Python environments and activate them for whatever script, Ansible playbook, etc you are running. Another great feature is you can create a requirements file to share with others, so if they want to run your code they can simply install from the requirements file which guarantees they have the right version of Python and associated packages installed.

Installed Pillow but still get : ModuleNotFoundError: No module named 'PIL'

I was trying to build Fisherfaces(https://github.com/bytefish/facerec) on my Mac(OS X 10.12.6), and I successfully built its dependencies including sklearn, numpy, scipy, matplotlib except Pillow.
So I tried :
pip3 install Pillow
at first I failed out of some network issue, so I tried again, and at last, I get this from terminal:
Installing collected packages: pillow
Successfully installed pillow-5.0.0
but then I tried on terminal:
TreaserdeMacBook-Pro:scripts treasersmac$ python3
Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import PIL
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'PIL'
and when I use python instead of python3, still get the same question.
Then I check the pip3 list:
TreaserdeMacBook-Pro:scripts treasersmac$ pip3 list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
cycler (0.10.0)
Django (2.0.1)
image (1.5.17)
matplotlib (2.0.0)
nose (1.3.7)
numpy (1.12.0)
Pillow (5.0.0)
pip (9.0.1)
pygal (1.7.0)
pyparsing (2.1.10)
python-dateutil (2.6.0)
pytz (2016.10)
requests (2.13.0)
scikit-learn (0.19.1)
setuptools (36.5.0)
six (1.10.0)
wheel (0.30.0)
seemed OK.
Then I checked path:
>>> import sys
>>> print(sys.path)
['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/treasersmac/Library/Python/3.6/lib/python/site-packages', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages']
and PIL or Pillow was in:
/usr/local/lib/python3.6/site-packages
but:
TreaserdeMacBook-Pro:scripts treasersmac$ cd /usr/local/lib/python3.6/site-packages
TreaserdeMacBook-Pro:site-packages treasersmac$ ls
Django-2.0.1.dist-info numpy-1.13.3-py3.6.egg-info
PIL pip
Pillow-5.0.0.dist-info pip-9.0.1-py3.6.egg-info
__pycache__ pkg_resources
cv2.cpython-36m-darwin.so scikit_learn-0.19.1.dist-info
django setuptools
easy_install.py setuptools-36.5.0-py3.6.egg-info
homebrew-numpy-nose.pth sitecustomize.py
image sklearn
image-1.5.17.dist-info wheel
numpy wheel-0.30.0-py3.6.egg-info
I can import sklearn and sklearn can not be found in sys.path printed before.
I searched StackOverFlow but only to find that questions raised about PIL or Pillow were almost years ago, and not the same situation as mine.
(I think I used pip3 successfully installed Pillow..)
I'm new here, and urging for an answer...

Module installs but cannot be imported

I'm new to Python and I'm trying to run a Python2 script on a Mac that depends on a module, which I have installed but according to the script can't be found. Why is the following failing?
Home:Dev m$ python -V
Python 2.7.10
Home:Dev m$ sudo pip install coincurve==5.2.0 pysha3
Password:
The directory '/Users/m/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/m/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting coincurve==5.2.0
Downloading coincurve-5.2.0-cp27-cp27m-macosx_10_6_intel.whl (157kB)
100% |████████████████████████████████| 163kB 1.3MB/s
Collecting pysha3
Downloading pysha3-1.0.2.tar.gz (829kB)
100% |████████████████████████████████| 829kB 692kB/s
Requirement already satisfied: cffi>=1.3.0 in /usr/local/lib/python2.7/site-packages (from coincurve==5.2.0)
Requirement already satisfied: asn1crypto in /usr/local/lib/python2.7/site-packages (from coincurve==5.2.0)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/site-packages (from cffi>=1.3.0->coincurve==5.2.0)
Installing collected packages: coincurve, pysha3
Running setup.py install for pysha3 ... done
Successfully installed coincurve-5.2.0 pysha3-1.0.2
Home:Dev m$ pip show coincurve
Name: coincurve
Version: 5.2.0
Summary: Cross-platform Python CFFI bindings for libsecp256k1
Home-page: https://github.com/ofek/coincurve
Author: Ofek Lev
Author-email: ofekmeister#gmail.com
License: MIT/Apache-2.0
Location: /usr/local/lib/python2.7/site-packages
Requires: cffi, asn1crypto
Home:Dev m$ python
Python 2.7.10 (default, Jul 15 2017, 17:16:57)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import coincurve
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named coincurve
>>>

The problem is that you are not actually in the python 2.7 shell. In some Linux distributions, you have to specify the python version to get into its shell. So you run the following command:
python2
Instead of:
python
Because the above command automatically opens the python 3.x shell.

Specifying which python version to pip install to

I'm having troubles when trying to download and run packages within python, using pip install. Often pip will tell me that I have the package downloaded, but when I try to use the function it gives me an error and says 'no such file or directory'. It seems to me that I am using 2.7.10 and that is where pip is trying to install to, but I can't get why this isn't working
Peters-MBP:~ pete$ python import plotly
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'import': [Errno 2] No such file or directory
Peters-MBP:~ pete$ pip install plotly
Requirement already satisfied (use --upgrade to upgrade): plotly in ./Library/Python/2.7/lib/python/site-packages
Collecting pytz (from plotly)
Using cached pytz-2016.3-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests in ./Library/Python/2.7/lib/python/site-packages (from plotly)
Collecting six (from plotly)
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six
Successfully installed pytz-2016.3 six-1.10.0
You are using pip version 8.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Peters-MBP:~ pete$ python --version
Python 2.7.10
Peters-MBP:~ pete$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Try to upgrade 'pip install --upgrade pip'
Then try to upgrade also plotly and check it again.
Please post if the plotly module is visible in the Python27 installation folder.
In addition you should go inside the "python interpreter" to import a module.
Just try to
> python
import plotly

You can’t just import a module on the command line like that(!!!):
$ python import plotly
Instead, do this:
First start the interactive Python interpreter:
$ python
Then, at the Python prompt, import your installed module:
>>> import plotly
Or, of course, put the import statement in a script and execute that script.
Edit: If you try to run python install plotly at the shell prompt, as you did, the Python interpreter will look for a script called install in the current directory, in order to execute it. Since no such script exists, it gives the error can't open file 'import': [Errno 2] No such file or directory.