Develop Reference

pip installed module but python gives Import error [duplicate]

During the past years, I have installed many Python libraries with various Python versions. To make them ready to work immediately, I installed them blindly without control. Currently they're causing problems when I tried to install pynest which invokes numpy, scipy and matplotlib. After struggling, I am going to clean and reinstall Python and the libraries.
After investigation, I found Python 2.5/2.6/2.7/3.2 on my system, and each of them has some copies or other things at: (my OS == Mac OS X 10.7.5 Lion)
/Library/Frameworks/
/opt/local/Library/Frameworks/
/opt/local/bin/
/Applications/
/usr/local/bin/
/usr/bin/
/System/Library/Frameworks/
I know I'm crazy to have these. Now I have removed all these except the things in /System/Libarary/Frameworks (I never remove any thing from /System/Library/). After the clean work, which python now gives /usr/bin/python which links to /System/Library/Frameworks.
Now, is it a clear environment for me to reinstall python? How to double check that there's no other versions existing? How should I reinstall them to guarantee that they and their libraries won't be everywhere and have many copies again?
I want to install a clean Python 2.7 onto a proper location, and make my system know exactly where it is and never install any libraries somewhere else. Please give me some advice that how to manage it like in a professional way.
For your information, here is my current $PATH, I think it should be modified:
/opt/local/bin:/opt/local/sbin:/opt/nest/lib/python2.7/site-packages:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/django/bin:/usr/X11/bin:/opt/local/bin:/opt/local/sbin:/usr/local/lib/python2.7/site-packages:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/3.2/bin
Please let me know If you need more information. Thank you!
UPDATE:
I'm rethinking profoudly why it becomes so crazy. I believe it's because I installed things via:
easy_install / macports / homebrew / fink / pip sometimes;
.dmg sometimes;
.pkg sometimes;
compile source code sometimes;
and they made things at different locations. I wonder what's the mechanism behind these ways? How do they choose target location? How to prevent them from messing things up?

Why did it get messed up?
There're a couples of different way to install Python, as the update of OP says, and they locate files in different locations. For example, macports puts things into /opt/local/, while homebrew puts things into /usr/local/. Also, Mac OS X brings a few python versions with itself. So, if you install python many times via different ways, you will get many python versions existing independently on your system.
What problem does it cause?
I don't know exactly. I guess the problem is that if you have many versions of python, then which one to use and where to find packages will be determined by the path order in your system PATH and the PYTHONPATH respectively. So you may lose control of where to install python modules. Consider that if you run sudo python setup.py install to install a module (it finds python by the root's PATH) and then try to import the module by python -c "import it" (this time it finds python by your PATH), maybe something will go wrong. This is my guess, I didn't validate it. But in my own case, something did go wrong.
How to avoid this?
I think the principle would be that be aware of that different ways and tools install things independently to different locations, so use them mindfully.
Unless you intend to, don't install the same thing twice via different
ways. (If you intend to do it for python, you might want to check out virtualenv)
Keep an eye on the path order in your PATH and consider if it's
correct.
When installing modules, be clear which python (or pip) is
running and where the module is installed.
So, how did I solve my own case?
Since it had been messing up already and seemed to be very hard to cure, so finally I solved this question by a full OS re-installation, and started to follow the DOs-and-DONTs above. For the installation of the scientific environment with python (numpy/scipy/matplotlib, which had shown problems to make me ask this question), I found this tutorial was extremely helpful. So, problem solved finally.

Here is what was confusing me and how I solved it.
$ which python
/usr/bin/python
$ which python3
/usr/local/bin/python3
$ ls /usr/local/bin/python
ls: /usr/local/bin/python: No such file or directory
So notice I didn't have a HomeBrew installation of python2.7, but did have the python3 installation. The version under /usr/bin/python is using the system default. You can tell based on the module search path:
$ /usr/bin/python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
`enter code here`Type "help", "copyright", "credits" or "license" for
more information.
>>> import sys
>>> sys.path
['', '/Library/Python/2.7/...
Notice the '/Library/Python'... that's Mac OS's version of python. But I want to stay strictly on a user installed version (i.e. HomeBrew).
So here's what I did to fix this:
$ brew install python
...
Warning: python 2.7.13 is already installed, it's just not linked.
You can use `brew link python` to link this version.
$ brew link --overwrite python
$ which python
/usr/local/bin/python
$ python
Python 2.7.10 (default, Feb 7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/Cellar/python/2.7.13...
Its no longer /Library/.. but /usr/local.
Now its finding all of my pip installed modules! Problem solved!
UPDATE:
After updating brew to version 1.5.4, it seems the symbolic links were removed. And now you have to add this to your path:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Read the Caveats section in 'brew info python':
==> Caveats
This formula installs a python2 executable to /usr/local/bin.
If you wish to have this formula's python executable in your PATH then add
the following to ~/.bash_profile:
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
Pip and setuptools have been installed. To update them
pip2 install --upgrade pip setuptools
You can install Python packages with
pip2 install <package>
They will install into the site-package directory
/usr/local/lib/python2.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python.html

In order to install a python distributions into specific folder, you can use the --prefix scheme during python installation. Using the prefix scheme, you can for example install Python 2.7 into the folder /opt/py27. Now, in order to use the new installed Python distribution you have to: cleanup you PATH and LD_LIBRARY_PATH:
Remove all 'old' Python paths and
configure (according to my example) the environment variables like this:
PATH: Add /opt/py27/bin
LD_LIBRARY_PATH: Add /opt/py27/lib
That's it.
(In case you need multiple environments of Python installed at the same time, I'd suggest to have a look at virtualenv)

tl;dr
brew install python
Symptoms
I had similar issues with python programs not finding dependencies.
My python3 version was a broken symlink.
My pip was pointing to a python 3.8
And my pip3 was pointing to 3.9
python -V was outputting some python 2.7 version
python3 -V was outputting some python3.8 version
Solution
I ran brew install python and it fixed all my problems.

Directing Python to look in another folder for modules

I'm still pretty new to python so forgive me if this is extremely simple or extremely the wrong way of thinking about this.
I have python 2.7 installed. From what I understand when I run the following code, it lists the directories where it looks for modules.
Python 2.7.12 (default, Oct 11 2016, 14:42:58)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print '\n'.join(sys.path)
/usr/local/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/readline
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
I have another directory that appears to have a bunch of python modules I installed in it. "/Library/Python/2.7/site-packages"
I guess I need to do one of two things:
(1) direct python to look in this additional folder for modules. How do I do that?
(2) install the modules in one of the folders it is already directing to. I've been using pip to install modules, and I think pip is installing to this additional director. How do I check if pip is installing to this folder? How do I change where pip install packages to?
Thanks!

This stuff is setup by site.py. You can find out which site.py by entering the interpreter with:
python -v
Alternatively, import site in the interactive interpreter and check the site.__file__ attribute.
There's also a helpful script in site.py which you can run with
python -m site
What you want to see in the output there is your user site, something like
USER_BASE: '/home/<your_username>/.local' (exists)
USER_SITE: '/home/<your_username>/.local/lib/python2.7/site-packages' (exists)
ENABLE_USER_SITE: True
When you pip install a package, use pip install --user, which will install modules to your user space. Don't install stuff with sudo pip install. Don't munge the sys.path manually like the "quick fix" from jmd_dk suggests. Do it properly.
If, after reading the site documentation and PEP370, you are still having trouble setting up the user site correctly, then it is acceptable to add a line like this in your bash profile:
export PYTHONPATH=/home/<your_username>/.local/lib/python2.7/site-packages
If you have both a Python 3 and a Python 2 installation, beware that the PYTHONPATH environment variable will be seen by both interpreters. In this case, it is strongly recommend to enable the site.USER_SITE separately for each interpreter (or use virtual environments) to provide adequate namespacing of installed packages.

Installing VTK for Python

I'm trying to install VTK module for python, I am however unsuccesful in doing so. I have downloaded a VTK tar-file, but I'm unable to extract it. I'm capable of extracting other tar-files, so there must be something specific with this file I suppose.
This is my error:
gzip: stdin: invalid compressed data--format violated
tar: Child returned status 1
tar: Error is not recoverable: exiting now
I hope somebody can help me with this.

The answer depends on the operating system you are using. This will be a lot easier if you can find a package or installer for your specific operating system and/or distribution.
Linux
If you are using Linux then look for the corresponding package in the distribution's package manager. For example, on Ubuntu Linux you should be able to install it using the following command:
sudo apt-get install python-vtk
Microsoft Windows
If you are using Microsoft Windows, the easiest way would be to install Python(x,y). It comes with VTK support.
Additionally, Anaconda also includes VTK package as well as support for virtual environments. It might be a good option for some folks.
Mac OS X
If you are using Mac OS X, try installing everything via MacPorts.
As #Nil mentioned in comments below, a standalone python interface to VTK is now provided by VTK developers. You may download it for Windows, Darwin, and Linux from here.
As mentioned by #Nil, VTK used to offer vtkpython binaries on their download page. However, they've dropped this since VTK-8.x.x as mentioned here:
Sorry, about that. We decided to drop the vtkpython binaries for 8. I want to focus our energies on supporting python wheel installs instead. There’s no timeline yet for a complete solution but we’ve made some good progress toward that recently here: https://github.com/jcfr/VTKPythonPackage.
Thus, the recommended way of installing vtkpython now is (see this page):
$ python -m pip install --upgrade pip
$ python -m pip install vtk

on Ubuntu, maybe this post will be helpful:
http://kazenotaiyo.blogspot.jp/2010/06/installing-vtk-in-ubuntu-and-making.html
The easiest way
The first and easiest is to just install the packages with the Aptitude Package Manager:
sudo apt-get install libvtk5-dev python-vtk
If you want the newest version
If you want the newest version VTK, you can also build it yourself:
Make sure CMake is installed:
sudo apt-get install cmake
Download the VTK source from the Downloads page.
Untar it:
tar xvzf vtk-5.6.0.tar.gz
Create an Out-Of-Source build and configure with CMake:
mkdir VTK_BUILD
cd VTK_BUILD
ccmake ../VTK
Make sure you enable python wrapping and set your install prefix to where you want the package to go. The default /usr/local works fine.
sudo make -j 8 install
(the -j 8 for make just makes the build process parallel assuming you've got the processors for it)
You now have VTK installed. Congrats! if you try to run vtkpython though, you'll get an error:
vtkpython: error while loading shared libraries: libvtksys.so.5.6: cannot open shared object file: No such file or directory
To fix this, append these lines to your .bash_profile, .bashrc, or .profile file in your home directory:
# add vtk paths
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/lib/vtk-5.6"
PYTHONPATH="$PYTHONPATH:/usr/local/lib/vtk-5.6"
You'll need to reset your terminal now.
That sets up your library and python paths for the vtkpython executable.

http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk Try this! Works for windows !

I have installed vtk without a problem under win7 via pip:
> pip install vtk
Collecting vtk
Downloading vtk-8.1.0-cp36-cp36m-win_amd64.whl (24.4MB)
100% |████████████████████████████████| 24.4MB 56kB/s
Installing collected packages: vtk
Successfully installed vtk-8.1.0
With Anacond python:
> python
Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:25:24) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

MacOS only:
See How to Install Mayavi on MacOS
An abridged version of this for MacOS is the following: (but I recommend the full procedure based on above link, but note that it installs Mayavi too)
The following steps seem to work on MacOS:
brew install vtk
pip install vtk
This installs vtk#9.1. You may want to install brew install vtk#8.2 instead.
Tested on:
Python: 3.9.13, MacOS: 12.4 Monterey
PS. As mentioned before, this answer may be incomplete (you may need QT too, bu tI am not sure). For a complete one including Mayavi, see My answer here . I suggest following the steps there.
I didn't update all content s here because I am not sure which steps are required if you only need VTK (not Mayavi). For example, I don't know whether you need QT or not.

Python unable to find lxml module

I wrote a script some times ago that contain
from lxml import etree
But, unfortunatly it is not working anymore.
In doubt i checked installation with :
sudo apt-get install python-lxml
sudo pip install lxml
sudo apt-get install libxml2-dev
sudo apt-get install libxslt1-dev
I checked if it could be my python version with :
me#pc:~$ python
Python 2.7.3 (default, Sep 14 2012, 14:11:57)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named lxml
My os is ubuntu 12.04.1 LTS with Python 2.7.3.
All seems fine. I can't see what could be the problem.

Your solution cited in edit, which use the xml.etree instead of lxml.etree is not the better way to do it, as these module have known incompatibilities, and mainly because lxml is certainly more optimised.
A good way to make a clean environment available is to use virtualenv :
$ virtualenv myproject
$ cd myproject
$ ./bin/pip install lxml # Repeat this with other dependencies
[wait for download and compiling]
Then, use ./bin/python to execute your script.
The advantages of this method are :
you can have different versions of dependencies between your system and your project
even if you break everything in your virtual environment, you will not compromised the rest of your system
you do not need root privileges to make an installation
As a reference, a more powerful but slightly more complex way to achieve this is to use buildout, but it can looks like hunting flies with a bazooka if you just want to execute a simple one-file script.

Solved the problem.
It seems that a software that i installed messed out my python path. The python that i was using when calling python in a terminal was the one installed by the software, and the one called by my script was the one installed on my system.
So, i just removed the python path of the software from the path variable of bash.

I had this problem when running tests in PyCharm. I could import lxml when using the terminal and virtual environment, but using the same virtual environment to run the tests resulted in the ImportError.
In my case upgrading from lxml 3.3.5 to 4.2.5 fixed the problem.

This likely has something do to with your Python environment. Here are three troubleshooting methods, from least to most cumbersome
1) Re-install via pip
pip uninstall lxml
pip install lxml
2) Re-Install via IDE
uninstalling it from the terminal (pip uninstall lxml)
adding from lxml import etree at the beginning of a python file
letting my IDE (PyCharm) show a missing import error
hovering over the error and installing it from the IDE via the error-fixing pop-up (not terminal)
3) Re-create virtualenv
Assuming you're working in a virtual environment (if not, shame to you and your family): delete and create a new environment.

if youre dealing with conda, i made the "discovery, that my conda install lxml does not work propperly after "successfully" installing
try uninstalling and reinstalling with pip, that worked for me
conda remove lxml
pip install lxml

For Python 3.6, it is very tricky.
I have the same problem and try to install the latest version, lxml 3.8. But I still get the error. And then I use lxml 3.7.3, then it is ok.

Can python.org releases play nice with Apple framework builds?

(Advanced apologies for the lack or proper links; the system won't allow me to add more than two.)
Unfortunately, I've learnt the hard way that you shouldn't mess with the default Python installations in Mac OS X (specifically, 10.6.8).
After using the python.org installers for 2.6.6 (http://www.python.org/getit/releases/2.6.6/) and 2.5.4 (http://www.python.org/getit/releases/2.5.4/), I have Python versions which are more mature than those provided by Apple (which is great for development), but have broken core system functionality (which is bad for just about everything else.) The most visible breaks so far have been when trying to run namebench (https://code.google.com/p/namebench/), Blink (http://icanblink.com/) and Mercurial (https://www.mercurial-scm.org/). From what I can gather, it's down to the paths.
In a default installation, the paths should resemble something those outlined in this question. Instead, mine look like this:
$ /usr/bin/python2.6 -V
Python 2.6.6
$ which python
/usr/bin/python
$ python
Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> for i in sys.path:
... print i
...
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pip-1.0.2-py2.6.egg
/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
(It's a similar story for 2.5 but, for simplicity, I'll stick with 2.6.)
The issue seems to be the non-inclusion of:
/Library/Python/2.6/site-packages/*.egg
/Library/Python/2.6/site-packages
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python
This explains why Blink and namebench can't find PyObjC (Is PyObjC pre-installed on OSX SL?) and Mercurial can't find its modules (in /Library/Python/2.6)
Issue #4865 in the Python bug tracker partially addresses this particular problem, which has since been fixed for versions 2.7 and greater. But because it's "a feature request and not a bugfix", has not been back-ported to 2.5 and 2.6. Even then, looking at the committed fix (http://hg.python.org/lookup/r70778), I'm not sure it addresses the lack of references to the "Extras" directories.
I understand that I can manually add paths to Python by manually altering site.py. I could also make use of the PYTHONPATH environment variable. I'd rather not cause any further damage by altering site.py, and changes PYTHONPATH would only be valid for scripts/applications run from the shell using my user account.
Can I get these more recent versions of Python to reference the same paths as the default framework installations? If so, what is the best way to go about it? If not, is there an accepted method of rolling back to the system defaults?

You misunderstand how Python installations work on OS X. Each Python instance has its own site-packages directory. The standard location for framework installers is within the framework at ./lib/pythonx.y/site-packages. So for the python.org installers which install into /Library/Frameworks/Python.framework, you will find its 2.6 site-packages here:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
Apple makes some modifications to the versions of Python that it ships with OS X releases. From OS X 10.5 on, the system Pythons are installed at:
/System/Library/Frameworks/Python.framework
and Apple chooses to include some extra 3rd-party packages in the non-standard ./Extras directory within each version. It also uses a non-standard location for each version's site-packages directory. They are created in /Library/Python/, presumably so that user-installed packages do not modify anything in /System/Library. So for the Apple-supplied Python 2.6, its site-packages directory is:
/Library/Python/2.6/site-packages
and can be thought of as having been extended by the packages in ./Extras.
Each Python instance has a separate site-packages directory. It is not intended and often not possible to share packages across site-packages of different instances even of the same minor version of Python, i.e. 2.6. The most obvious problem is that there are often differences in the C compiler version, the OS X ABI (MACOSX_DEPLOYMENT_TARGET), the SDK version, and/or the CPU architectures used to build the Python interpreters and which are subsequently used by Distutils to build C extension modules included in 3rd-party packages.
In Mac OS X 10.6, the Apple-supplied Python is built using gcc-4.2 and targeted just for OSX 10.6 and includes 3 CPU architectures (i386, x86_64, and ppc). The python.org installers for Python 2.6 are built to run on older systems as well, so have a target of 10.3 and later, use gcc-4.0, and are 32-bit only (i386 and ppc). So, in general, you cannot run C extension modules built for the one Python with the other.
This means that, in general, you need to install separate copies of 3rd-party packages you need with and for each Python, if they are not already included with that Python. This includes basic items like easy_install (from setuptools or Distribute). The system Pythons in 10.5+ include easy_install version in /usr/bin for them. If you install a python.org Python, you'll need to install a separate version for it; by default, the easy_install command will be installed in that Python's ./bin directory in its framework; that is the Distutils default location. That's why it is recommended that you add this directory to your shell PATH (and the python.org installer for Python 2 do that automatically by default).
The change introduced by Issue4865 is not really a good solution and may fail with C extension modules. I would not depend on it remaining in Python in future versions.
Also installing a python.org Python in no way breaks anything in the system Python because they are completely independent installations using different file system locations. The only thing that may change is which instance of Python is invoked when you type a particular name. That is controlled primarily by the search order of your shell PATH environment variable. As noted, the python.org installer by default changes that order but the system Python is still readily available by using its absolute path /usr/bin/python2.6. Or you can revert the changes to shell profile, for instance, .bash_profile.
$ echo $PATH
/Library/Frameworks/Python.framework/Versions/2.6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/Library/Frameworks/Python.framework/Versions/2.6/bin/python
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ python -V
Python 2.6.6
$ python2.6 -V
Python 2.6.6
$ /usr/bin/python -V
Python 2.6.1
$ /usr/bin/python2.6 -V
Python 2.6.1
#
# remove python.org Python 2.6 from PATH
#
$ export PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
$ which python python2.6
/usr/bin/python
/usr/bin/python2.6
$ python -V
Python 2.6.1
$ python2.6 -V
Python 2.6.1