We have CentOS 7 servers in a Hadoop cluster. Python and pip should have been installed on all of the server the same way because it was done with Ansible. But for some reason, there are some servers where Python cannot use the modules that were installed. For instance pandas was install, but when in python3.6, I get a 'No module named pandas' error.
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
>>>
If I try installing pandas again, I get messages that the reuirements are already satisfied.
$ sudo pip3.6 install pandas
Requirement already satisfied: pandas in /usr/lib64/python3.6/site-packages
Requirement already satisfied: python-dateutil>=2 in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: pytz>=2011k in /usr/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: numpy>=1.9.0 in /usr/lib64/python3.6/site-packages (from pandas)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.6/site-packages (from python-dateutil>=2->pandas)
It appears that the python 3.6 sys.path is using a different location for site-packages.
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
It may all stem from pip being installed in an unusual location.
$ pip3.6 --version
pip 9.0.1 from /usr/lib/python3.6/site-packages (python 3.6)
Pip and the modules seem to be located in the same place, correctly on servers that DO work as follows:
$ pip3.6 --version
pip 9.0.1 from /usr/local/lib/python3.6/site-packages (python 3.6)
$ python3.6 -m site
sys.path = [
'/home/avalenti',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages',
]
$ python3.6
Python 3.6.2 (default, Aug 2 2017, 14:17:20)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> exit()
$ pip3.6 show pandas
Name: pandas
Version: 0.22.0
Summary: Powerful data structures for data analysis, time series,and statistics
Home-page: http://pandas.pydata.org
Author: The PyData Development Team
Author-email: pydata#googlegroups.com
License: BSD
Location: /usr/local/lib/python3.6/site-packages
Requires: numpy, pytz, python-dateutil
So my question is, how do I get this fixed and how do I keep this from happening in the first place? I don't seem to find a way to configure Pip and I don't find any way to specify the installation location for modules.
I tried to be thorough with the information provided, but I'm sure I missed something.
Thanks,
Anthony
Well, the Pip documentation for the config pointed me to the --target option for pip install. That allowed me to specify the location where I thought the packages should go - /usr/local/lib/python3.6/site-packages (as noted in the sys.path). That allowed the import to find the pandas package, but fail on the import of some basic system modules, builtin and binascii. It seems like python itself is kinda hosed.
# pip3.6 install --target /usr/local/lib/python3.6/site-packages pandas
Collecting pandas
Using cached pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy>=1.9.0 (from pandas)
Downloading numpy-1.14.2-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
100% |████████████████████████████████| 12.2MB 92kB/s
Collecting python-dateutil>=2 (from pandas)
Downloading python_dateutil-2.7.2-py2.py3-none-any.whl (212kB)
100% |████████████████████████████████| 215kB 2.4MB/s
Collecting pytz>=2011k (from pandas)
Using cached pytz-2018.3-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Using cached six-1.11.0-py2.py3-none-any.whl
Installing collected packages: numpy, six, python-dateutil, pytz, pandas
Successfully installed numpy-1.14.2 pandas-0.22.0 python-dateutil-2.7.2 pytz-2018.3 six-1.11.0
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# python3.6
Python 3.6.2 (default, Aug 2 2017, 14:51:00)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 48, in <module>
import __builtin__ as builtins
ModuleNotFoundError: No module named '__builtin__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/pandas/__init__.py", line 23, in <module>
from pandas.compat.numpy import *
File "/usr/local/lib/python3.6/site-packages/pandas/compat/__init__.py", line 61, in <module>
import http.client as httplib
File "/usr/local/lib/python3.6/http/client.py", line 71, in <module>
import email.parser
File "/usr/local/lib/python3.6/email/parser.py", line 12, in <module>
from email.feedparser import FeedParser, BytesFeedParser
File "/usr/local/lib/python3.6/email/feedparser.py", line 27, in <module>
from email._policybase import compat32
File "/usr/local/lib/python3.6/email/_policybase.py", line 7, in <module>
from email import header
File "/usr/local/lib/python3.6/email/header.py", line 14, in <module>
import binascii
ModuleNotFoundError: No module named 'binascii'
>>> exit()
Ugh...
You need to be very careful using pip when having multiple Python distributions. Instead of using pip3.6', try usingpython3.6 -m pip` to install pandas:
python3.6 -m pip install pandas
(prepend sudo if you really need to). In this way you are invoking pip through Python itself, and so you are guaranteed to get the pip that belongs with the Python you want.
You can try to set the location for pip in $HOME/.pip/pip.conf
As per PIP Docs:
Configuration Config file
pip allows you to set all command line option defaults in a standard
ini style config file.
The names and locations of the configuration files vary slightly
across platforms. You may have per-user, per-virtualenv or site-wide
(shared amongst all users) configuration:
Per-user:
On Unix the default configuration file is: $HOME/.config/pip/pip.conf which respects the XDG_CONFIG_HOME
environment variable.
On macOS the configuration file is $HOME/Library/Application Support/pip/pip.conf.
On Windows the configuration file is %APPDATA%\pip\pip.ini.
There are also a legacy per-user configuration file which is also
respected, these are located at:
On Unix and macOS the configuration file is: $HOME/.pip/pip.conf
On Windows the configuration file is: %HOME%\pip\pip.ini
You can set a custom path location for this config file using the
environment variable PIP_CONFIG_FILE.
Inside a virtualenv:
On Unix and macOS the file is $VIRTUAL_ENV/pip.conf
On Windows the file is: %VIRTUAL_ENV%\pip.ini
Site-wide:
On Unix the file may be located in /etc/pip.conf. Alternatively it may be in a "pip" subdirectory of any of the paths set in the
environment variable XDG_CONFIG_DIRS (if it exists), for example
/etc/xdg/pip/pip.conf.
On macOS the file is: /Library/Application Support/pip/pip.conf
On Windows XP the file is: C:\Documents and Settings\All Users\Application Data\pip\pip.ini
On Windows 7 and later the file is hidden, but writeable at C:\ProgramData\pip\pip.ini
Site-wide configuration is not supported on Windows Vista
If multiple configuration files are found by pip then they are
combined in the following order:
Firstly the site-wide file is read, then
The per-user file is read, and finally
The virtualenv-specific file is read.
Each file read overrides any values read from previous files, so if
the global timeout is specified in both the site-wide file and the
per-user file then the latter value is the one that will be used.
Short answer, but I would recommend using Python Virtual Environments. Makes managing Python versions and pip packages very easy.
https://docs.python.org/3/tutorial/venv.html
This way you can have many different virtualized Python environments and activate them for whatever script, Ansible playbook, etc you are running. Another great feature is you can create a requirements file to share with others, so if they want to run your code they can simply install from the requirements file which guarantees they have the right version of Python and associated packages installed.
Related
I believe I'm following commonly perscribed means of installing a python package. First I activate the virtual environment:
[]$ source virt_env/bin/activate
Then I check to see that the python and pip versions and locations are correct:
(virt_env) []$ which python
~/virt_env/bin/python
(virt_env) []]$ which pip
~/virt_env/bin/pip
Now I attempt to install with pip and it succeeds
(virt_env) []$ pip install pandas
Collecting pandas
Using cached https://files.pythonhosted.org/packages/b7/e3/f52d484244105fa3d558ce8217a5190cd3d40536076bef66d92d01566325/pandas-0.23.4-cp27-cp27mu-manylinux1_x86_64.whl
Collecting numpy>=1.9.0 (from pandas)
Using cached https://files.pythonhosted.org/packages/9e/eb/c9eda9f4865d669e0bb37ce5c780e86c63daa54ca827b95a171429012d08/numpy-1.15.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: python-dateutil>=2.5.0 in ./virt_env/lib/python2.7/dist-packages (from pandas) (2.7.3)
Requirement already satisfied: pytz>=2011k in ./virt_env/lib/python2.7/dist-packages (from pandas) (2018.5)
Requirement already satisfied: six>=1.5 in ./virt_env/lib/python2.7/dist-packages (from python-dateutil>=2.5.0->pandas) (1.11.0)
Installing collected packages: numpy, pandas
Successfully installed numpy-1.15.3 pandas-0.23.4
However, when I try to import it in my python session, I cannot find pandas
(virt_env) []$ python
Python 2.7.14 (default, May 2 2018, 18:31:34)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named pandas
Is this not the correct way to install? Looking in ./virt_env/lib/python2.7/dist-packages I see that pandas is no where to be found. Am I doing anything wrong?
I have tried doing python -m pip install pandas as well with no success.
I am on an Amazon ec2 instance running Amazon Linux.
Edit: python -m site returns the following:
sys.path = [
'',
'/home/myuser/virt_env/local/lib/python2.7/dist-packages',
'/home/myuser/virt_env/local/lib/python2.7/dist-packages',
'/home/myuser/virt_env/lib/python2.7/dist-packages',
'/home/myuser/virt_env/local/lib64/python2.7/site-packages',
'/home/myuser/virt_env/local/lib/python2.7/site-packages',
'/home/myuser/virt_env/lib64/python2.7',
'/home/myuser/virt_env/lib/python2.7',
'/home/myuser/virt_env/lib64/python2.7/site-packages',
'/home/myuser/virt_env/lib/python2.7/site-packages',
'/home/myuser/virt_env/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7',
'/usr/lib/python2.7',
]
USER_BASE: '/home/myuser/.local' (doesn't exist)
USER_SITE: '/home/muser/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
I am also able to install things if I use
pip install --install-option="--prefix=/home/myuser/virt_env" pandas
and then move the resulting files to the correct location.
This has me pretty confused. I've installed pyad using pip and everything seems fine:
C:\WINDOWS\system32>pip install pyad
Collecting pyad
Using cached pyad-0.5.16.tar.gz
Requirement already satisfied (use --upgrade to upgrade): setuptools in c:\python35\lib\site-packages (from pyad)
Requirement already satisfied (use --upgrade to upgrade): pywin32 in c:\python35\lib\site-packages (from pyad)
Installing collected packages: pyad
Running setup.py install for pyad ... done
Successfully installed pyad-0.5.16
But when I try to use it, I get an error that complains about not finding adbase:
C:\WINDOWS\system32>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec 6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyad import aduser
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\site-packages\pyad\__init__.py", line 1, in <module>
from adbase import set_defaults as pyad_setdefaults
ImportError: No module named 'adbase'
>>> import pyad
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python35\lib\site-packages\pyad\__init__.py", line 1, in <module>
from adbase import set_defaults as pyad_setdefaults
ImportError: No module named 'adbase'
This is odd, because if I try to uninstall pyad or if I check the site-packages directory, adbase is definitely there:
C:\WINDOWS\system32>pip uninstall pyad
Uninstalling pyad-0.5.16:
c:\python35\lib\site-packages\pyad-0.5.16-py3.5.egg-info
c:\python35\lib\site-packages\pyad\__init__.py
c:\python35\lib\site-packages\pyad\__pycache__\__init__.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adcomputer.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\addomain.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adgroup.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adquery.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\adsearch.cpython-35.pyc
c:\python35\lib\site-packages\pyad\__pycache__\pyad.cpython-35.pyc
c:\python35\lib\site-packages\pyad\adbase.py
c:\python35\lib\site-packages\pyad\adcomputer.py
c:\python35\lib\site-packages\pyad\adcontainer.py
c:\python35\lib\site-packages\pyad\addomain.py
c:\python35\lib\site-packages\pyad\adgroup.py
c:\python35\lib\site-packages\pyad\adobject.py
c:\python35\lib\site-packages\pyad\adquery.py
c:\python35\lib\site-packages\pyad\adsearch.py
c:\python35\lib\site-packages\pyad\aduser.py
c:\python35\lib\site-packages\pyad\pyad.py
c:\python35\lib\site-packages\pyad\pyadconstants.py
c:\python35\lib\site-packages\pyad\pyadexceptions.py
c:\python35\lib\site-packages\pyad\pyadutils.py
Proceed (y/n)?
pyad directory contents
I'm really not sure what else to try. I've run everything under an elevated command prompt, so it's not a permissions issue. I even tried downloading pyad and installing it using setup.py, but I had the same problem with that. adbase is definitely there, and I can't figure out why Python isn't finding it.
That's a bug on pyad part. They're importing adbase as if it were a standalone module or package, and that's why it does not work. The proper way to fix this would be to change the import to an absolute import from pyad.adbase import ... or relative from .adbase import ....
However, if you check the master branch on Github, you will see that they have actually fixed it. But that's not all, if you check their setup.py you'll see that the version on Github is 0.5.15, while the last version on PyPI, which is the one you have installed, is 0.5.16. Weird.
I suggest you to install the package directly from Github, and that should take care of the problem. To do that, first uninstall pyad and then run
pip install https://github.com/zakird/pyad/archive/master.zip
I'm having troubles when trying to download and run packages within python, using pip install. Often pip will tell me that I have the package downloaded, but when I try to use the function it gives me an error and says 'no such file or directory'. It seems to me that I am using 2.7.10 and that is where pip is trying to install to, but I can't get why this isn't working
Peters-MBP:~ pete$ python import plotly
/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: can't open file 'import': [Errno 2] No such file or directory
Peters-MBP:~ pete$ pip install plotly
Requirement already satisfied (use --upgrade to upgrade): plotly in ./Library/Python/2.7/lib/python/site-packages
Collecting pytz (from plotly)
Using cached pytz-2016.3-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests in ./Library/Python/2.7/lib/python/site-packages (from plotly)
Collecting six (from plotly)
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six
Successfully installed pytz-2016.3 six-1.10.0
You are using pip version 8.1.0, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Peters-MBP:~ pete$ python --version
Python 2.7.10
Peters-MBP:~ pete$ python
Python 2.7.10 (v2.7.10:15c95b7d81dc, May 23 2015, 09:33:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Try to upgrade 'pip install --upgrade pip'
Then try to upgrade also plotly and check it again.
Please post if the plotly module is visible in the Python27 installation folder.
In addition you should go inside the "python interpreter" to import a module.
Just try to
> python
import plotly
You can’t just import a module on the command line like that(!!!):
$ python import plotly
Instead, do this:
First start the interactive Python interpreter:
$ python
Then, at the Python prompt, import your installed module:
>>> import plotly
Or, of course, put the import statement in a script and execute that script.
Edit: If you try to run python install plotly at the shell prompt, as you did, the Python interpreter will look for a script called install in the current directory, in order to execute it. Since no such script exists, it gives the error can't open file 'import': [Errno 2] No such file or directory.
I'm running virtualenv on Windows 7 and I created a virtual environment for python 2.7 and installed some modules but I can't seem to import it.
1) venv\scripts\activate
2) pip2 install requests
(I the following message: get Requirement already satisfied (use --upgrade to upgrade): requests in c:\users\dan\desktop\python\bulkphotos\venv\lib\site-packages)
3) when I try to import this on the interactive shell, I get the following error
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>py
Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named requests
4) I've tried searching SO but the other answers seem to be applicable for a mac (e.g. accidentally installing via sudo). When I do pip2 list, the module is present, so I don't understand why I can't access it.
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>pip2 list
google-api-python-client (1.4.1)
httplib2 (0.9.1)
oauth2client (1.4.12)
pip (7.1.0)
pyasn1 (0.1.8)
pyasn1-modules (0.0.7)
requests (2.7.0)
rsa (3.2)
setuptools (18.0.1)
simplejson (3.8.0)
six (1.9.0)
uritemplate (0.6)
wheel (0.24.0)
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>pip2 show requests
---
Metadata-Version: 2.0
Name: requests
Version: 2.7.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me#kennethreitz.com
License: Apache 2.0
Location: c:\users\dan\desktop\python\bulkphotos\venv\lib\site-packages
Requires:
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>
Does anyone know how to solve this problem?
EDIT
The results of running sys.path on the interactive shell
['', 'C:\\windows\\system32\\python27.zip', 'C:\\Python27\\DLLs', 'C:\\Python27\\lib', 'C:\\Python27\\lib\\plat-win', 'C:\\Python27\\lib\\lib-tk', 'C:\\Python27', 'C:\\Python27\\lib\\site-packages']
The results of running where py and where python
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>where py
C:\Windows\py.exe
(venv) C:\Users\Dan\Desktop\Python\bulkphotos>where python
C:\Users\Dan\Desktop\Python\bulkphotos\venv\Scripts\python.exe
C:\Python27\python.exe
C:\Python34\python.exe
The Windows Python launcher PEP 397 is not aware of the active virtual env. See PEP 486 for details.
Try to launch Python interpreter by executing python instead of py.
I dont know why these packages are not imported,
But you can debug it by viewing your pythonpath using,
python -c "import sys; print sys.path"
You can append optional path to sys.path.
I installed psycopg2 via pip, but my programs are having trouble finding it.
So, I tried to install psycopg2 via pip again:
user#ubuntu:~/Desktop/progFolder$ sudo pip install psycopg2
Requirement already satisfied (use --upgrade to upgrade): psycopg2 in /usr/local/lib/python2.7/dist-packages
Cleaning up...
Then I tried to use a program that imports it:
user#ubuntu:~/Desktop/progFolder$ python myProg.py
Traceback (most recent call last):
File "myProg.py", line 6, in <module>
import psycopg2
ImportError: No module named psycopg2
And I have tried just importing directly in python:
user#ubuntu:~/Desktop/progFolder$ python
Python 2.7.5 (default, Nov 9 2014, 14:14:12)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psycopg2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named psycopg2
So I printed my python path.
>>> import sys
>>> print sys.path
['', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages']
And noticed that the path does contain the path to psycopg2.
psycopg2 in /usr/local/lib/python2.7/dist-packages
So, I have no idea on why this is happening. Any help would be appreciated.
UPDATE:
I have done
>>>help()
>>>modules
And psycopg2 was not listed among the other modules. (this does not help me but may help you help me)
Your pip looks ok (that is, it's the system/default one). Your Python executable, however, is something that didn't come by default with 14.04 LTS (e.g., on my 14.04 system, it's /usr/bin/python). Did you install that Python yourself? Then you need to install (and use) the corresponding pip as well. (Normally, Python would have come with a pip installation, but apparently in this case, it didn't.)
pip can be fairly simple installed from its installation instructions.
Though first, verify that
you did install /usr/local/bin/python yourself. That is, it didn't come with some other piece of software that you installed and that, along the way, decided to install Python there.
you want to use /usr/local/bin/python (I guess it is a more recent version of Python 2.7; the default 14.04 LTS one appears to be 2.6.7 as of 2015-08-03).
From your python path print, it looks like it doesn't have /usr/local/lib/python2.7/dist-packages included in it. You can add it in one way by:
sys.path.insert(0, "/usr/local/lib/python2.7/dist-packages")